← All benches

All benches: Methodology

How the page All benches puts the benches of the three sites and Answer Check side by side. It uses only the files each bench publishes, bench-scores.json, in one schema for the three sites; a bench whose file is not live says "not published yet".

One scale: 0 = random guessing, 100 = perfect

Combined score

The plain mean, equal weights, of Reading · Turkish (all 2,198 Turkish questions, reading and grammar together, the Reading site's own score), Logic & Math (the Logic & Math site's pooled score, logic-math in its bench-scores.json) and Backgammon, unchanged by the splits into Turkish Reading and Turkish Grammar and into Logic and Math (project lead, #4173, #4234, #4239): the parts are views of their own, never averaged in. Only a model with a score on all three gets one; the others appear in each bench's tab. English Reading, Vision and Answer Check are not part of it.

A row gets a combined score only if one model answered every bench: the same checkpoint (one repo, one set of weights) or the same served version of a hosted model. A hosted service that changed its model between the benches' runs keeps its score on each bench, but gets no combined score:

The page labels Combined "3 benchmark groups" and shows under each combined score its three parts: TR, Reading · Turkish (all 2,198 questions); L&M, Logic & Math (pooled); BG, Backgammon.

Decision Index

The plain mean, equal weights, of English Reading, Logic, Math and Answer Check, each on its own 0 to 100 scale with chance = 0 (Ümit #10906). Only a model with a score on all four gets one, and none where one model did not answer every bench, as for Combined. The page labels it "4 benches" and shows under each score its four parts: EN, English Reading; Logic, Logic; Math, Math; AC, Answer Check. The address ?view=index (or ?score=index) opens the page on it. "Decision Index" is a working name.

Answer Check: does the text give this answer?

This benchmark tests whether a model can classify a machine-written English summary, a question and its reference answer into three categories: the summary states the reference answer, states a conflicting answer, or does not answer the question. It measures grading of summaries, not the quality of the grader's own free-form answers.

Initial cohort: 2,069 consensus-labelled pairs from summaries of books, research papers, legal texts and agent sessions. Repository code is excluded. Raw texts, questions, reference answers, quotations and item-level predictions are not published.

Reference labels come from independent Claude Opus 5.5 and GPT-6.1 Sol judgments, not human annotation. For labels that assert an answer, a supporting quotation must be found in the summary. Disputed or invalid judgments are excluded. Within this cohort, 2,325 pairs were sampled and 256 were excluded. Sampling deliberately enriched cases that an earlier grader called conflicting answers; this is not a natural class distribution. Excluding disagreements may remove difficult cases. The summaries are English and machine-written; these results should not be generalized to Turkish or human-written answers without further testing. d1-ultra was previously selected in part for agreement with Opus on a separate sample, so agreement with this judge may favor it.

Accuracy is correct first outcomes divided by the frozen cohort size; missing or invalid terminal outputs count wrong. The displayed score is (accuracy − 1/3) / (2/3) × 100. Zero means uniform random choice among three labels, 100 means every pair correct, and below-chance scores may be negative. ECE uses 10 equal-width probability bins and is null when usable option probabilities are absent.

75 models have completed this cohort so far, 55 decision models and 20 chat models: 13 through their makers' APIs and 62 on our own GPUs (open decision models and chat models; a chat model's thinking setting is its row's tag). These numbers apply only to this cohort and request setting. Additional models receive rows only after completion on the same frozen cohort.

Requests contain one question each. Each score is one full run's: a decision model's first run, a chat model's run under answer rule v2 (below), except MiniCPM5-2B (thinking on)'s, LFM3-1.7B (thinking on)'s, Gemma 4 E4B (thinking on)'s, LFM3-1.7B (thinking off)'s, MiniCPM5-2B (thinking off)'s, Qwen3.5-4B (thinking off)'s, Gemma 4 E2B (thinking off)'s, Qwen3.5-2B (thinking off)'s and Raon-1 nano (thinking off)'s, the mean of three full runs with seeds 0, 1 and 2, with their lowest and highest beside it (a range, not a confidence interval).

How a chat model's answer is read (answer rule v2, from 15 Nov). A chat model writes its reasoning and then its choice. The label is the last marker 'ANSWER: X' in the visible reply. The word ANSWER may be in any letter case; X must be an uppercase A, B or C. The marker does not need to be on the final line. The model's separate reasoning text is never read. A reply with no such marker counts wrong. Each chat row's notes give how many of its replies had no marker and how many of those were cut off at the token limit.

Why the rule changed. The first rule (v1) accepted the word only in uppercase, 'ANSWER:'. LFM3-1.7B with thinking on writes 'Answer: X', so v1 counted about 40% of its replies wrong although they named an option. That was found by rerunning the model and keeping every reply. Rule v2 differs from v1 only in the letter case of the word ANSWER. On every kept reply, v2 changed none of the labels v1 had read; it only reads replies v1 missed. A reply with 'ANSWER: A' followed by a later 'Answer: C' would now read C.

Fresh runs, not rescoring. The earlier chat runs did not keep their replies, so they could not be read again. The 10 existing chat rows were therefore replaced by fresh runs with the same model, sampling settings and number of requests at a time, keeping every reply. Three thinking modes not run before were added the same way. The seed trios' servers follow the settings recorded for the earlier runs. For single runs the earlier runs' own server flags were not recorded, so the fresh runs use a server from the same launcher. The earlier runs and their scores stay in the history. A replaced row's change has two parts: the fresh run itself (a new sample from the model; under 2 points for every row) and the new rule (a large gain for LFM3-1.7B thinking on; small gains for LFM3-1.7B thinking off and for Qwen3.5-2B thinking on and off; none for the others).

Interrupted attempts. Two runs had an earlier attempt that stopped part-way: Qwen3.5-2B thinking on and Gemma 4 E4B thinking on. Those attempts are kept apart and never merged. Their rows note how many pairs they recorded and how many further requests may have been answered but not recorded. Each published run started again from the first pair, so its outcomes are that run's recorded outputs, not guaranteed to be the model's first outputs.

Test requests before a run. Before some decision models' scored runs, a few published pairs (one to three) were sent to the model as test requests, to check that its server answered with probabilities that sum to one. Those test outputs were printed, not kept. The scored row for each such pair comes from the run's own, later request, so for those pairs the recorded output is not the model's first output on the pair. The rows concerned say so in their notes and give the count. The count comes from our own record of the test calls; requests that others may have sent to servers they run are not covered.

Earlier scoring of the same material. Before Answer Check existed, two of the API decision models, d1-ultra and Gemini Decision, were used or tested as scorers in the compactor project. There they scored compaction texts and question-bank questions from the same held-out pool this benchmark's pairs were drawn from, in a different request form (all of a text's questions in one request). How many of the 2,069 published pairs they saw that way has not been counted, so their rows make no claim that these pairs were new to them.

Answer Check is a separate benchmark tab. Combined scoring is unchanged.

The Answer Check page's further columns come from the grading agent's metrics.json:

A cell without a score

Each says why, the reason on hover and on keyboard focus: Not run, the model is not on that bench; Not applicable, the score does not apply: Combined for a model without all 3 benchmark groups, or where one model did not answer every bench, and a bench whose file is not published yet. Sorting by a column puts such cells last.

Learning from experience

"Did it improve?": the points a model gains from its own experience, as a bench's file gives them (improvement_points). A column of its own, never part of the combined score.

When we write "tied"

No interval bars and no ranges: where two scores are too close to tell apart, the page says tied. The rule, the same everywhere on the page:

Calibration error

ECE: the answers sorted into 10 equal-width bins of the chosen option's probability; in each bin the gap between the mean probability and the share right, weighted by the answers in it; shown in points (×100), lower is better ↓. Reading · Turkish's is its Text tab run's, as the Text tab gives it; Turkish Reading's and Turkish Grammar's are the same run's, computed the same way over the part's answers that carry a probability; English Reading's is its first English run's, computed the same way (none for a run without a probability); Vision's the page-image run's; the other benches' as their files give them. On the Combined view the chart uses the mean of the benches' calibration errors: the mean over the benches in Combined (Reading · Turkish, Logic & Math, Backgammon) that give the model one, so a model without one on a bench gets the mean of the other two; a model with fewer than two is left out of that chart.

Latency

One pinned machine for every model: spark2, a DGX Spark (GB10), one request at a time, Reading · Turkish requests (the Text tab's runs); the median and the 95th percentile (nearest rank) of the time per request, in milliseconds. A run's machine is read from the host its requests went to, not from the hardware text. Models timed elsewhere, on another machine or through an API, read "not timed on the pinned machine yet".

Type and size

The type keys (All · Chat models · Decision models; Decision models when the page opens) and the class keys (All and the Reading site's Leaderboard's classes, the ones no model here falls in left out) choose the models in view, the two combined: the chart, the table under it, Head to head's choices, Full results and the charts follow them. A second tap on a pressed key goes back to All, as on the three sites. The address keeps the choice: ?models=all or chat (the Reading site's form) and ?class= a class. Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. A model of this site is classed as on the Leaderboard (Site.model_class); a model only on the other sites' benches by its file's params_total, else Undisclosed, as the files give no list price.

Size

Total parameters (and the active count for a mixture of experts) as the bench files give them, in billions; "not disclosed" where the maker states none. The table's size is the Leaderboard's: the total only, MoE after a mixture of experts, 2× after two models run together, empty where undisclosed.

One model across benches

Two rows are one model (the three sites' rule) when both have a Hugging Face repo and the repos match (in any case, without a commit, and without the org where one side gives none); else, unless both have repos that differ, when their model_key match: the three sites' join key, the model's name without its setting or route, lower case, hyphens for spaces, "-with-web" for a setup with web search, which never joins the plain one; a Reading model_key the three sites' shared names table maps to another (its row's source_model_key: Reading's qwen-3.5-397b-a17b is qwen3.5-397b-a17b) joins as that one. A repo on one side never keeps rows apart, and the route never splits a model (Ümit #4270). Else, for a row another site names differently, by the model_key a fixed list gives its name (the alias list, page_config.json all_benches.aliases: Backgammon's "Decider v1.2 27B" is Perplexity Decider 27B v1.2, its "Mercury Decide" is Mercury Decide, and so on); else, for a row without model_key, by the name in lower case without the maker's name, setting and route words, in any word order ("Decider v1.2 27B" meets "Perplexity Decider 27B v1.2" with the tag "API"). A row never joins a model already on its bench. The model shows the name of this site's row, else Logic & Math's. Where the benches ran different setups of one model (our hardware's checkpoint and a hosted API, or two checkpoints), the table under the chart says which bench ran which, and so does the model's page. Each bench's commit is on the model's page: the benches may have run different commits of one repo.

Their score

Jev Decision Index 0.4.1, "Full score", as of 6 November 2026 (source), for its 15 open decision models: a sanity check beside ours, never shown as our score. Jev 1.14 is a reference bar: TypeSafe's hosted Jev 1.14, the model the index's entries reproduce: a reference bar, with no index score of its own here.

Colours, marks and tags

Each bar and dot takes its maker's colour from maker_colours.json, the one colour map of the three sites, with the maker's mark before the model's name in the ranking (a plain shape where the maker's mark is not freely licensed). Makers without a colour there yet read grey: none. Every model shows by its maker's official name. Beside it, a small tag says how the setup ran where its bench's file gives one (the file's tag field: setting · route · snapshot, as "API · with web" or "high · chat"); two rows of one model differ only by their tag, and rows join by the rule above, never by their names. Then a second small tag, the training method, where our setup records state it (fine-tune, post-trained, LoRA, blend …); blank where they do not.

Sources

The build reads the other sites' files from these addresses and keeps what it read in benches.json.