All benches: Methodology
How the page All benches puts the benches of the three sites and Answer Check side by side. It uses only the files each bench publishes, bench-scores.json, in one schema for the three sites; a bench whose file is not live says "not published yet".
One scale: 0 = random guessing, 100 = perfect
- Turkish Reading and Turkish Grammar: the Reading site's 656 Turkish reading questions (units 1 to 6) and its 1,542 grammar questions (units 7 to 20), 5 options each, as its Text tab asks them, each part scored on its own questions: Score = (accuracy − 1/k) / (1 − 1/k) × 100 with k = 5: 20% right reads 0, every question right reads 100. Below 0 means worse than guessing. The row is the setup the Text tab's Leaderboard shows, the same run on both parts.
- English Reading: the 547 reading questions kept after translation into English (the Reading site's English reading page), 5 options each, the same score with k = 5. A model's row is its first English run of the setup its Turkish row shows (else of another setup of the same row, in the Leaderboard's order); the models run only in English have a row here and none on the Turkish benches.
- Vision: the 763 questions that depend on the printed page, asked with the page image alone (the Reading site's Vision tab, condition B), k = each question's number of options. Models that read only text are not on it.
- Logic and Math: the Logic & Math site's two parts, each in a file of its own: the same correction, on its own questions (k = 5), as its file states.
- Backgammon: 0 = the error rate of a player that picks a random legal move, 100 = no error, as its file states.
- Answer Check: 2,069 pairs of a machine-written English summary, a question and its reference answer, 3 labels each, the same correction with k = 3: one label in three at random reads 0, every pair right reads 100 (Answer Check).
- Higher is better ↑ on every score; lower is better ↓ for calibration error and latency. Size has no better direction.
- A score is not a share of right answers: the chart's head and the table's caption name its unit, "Score 0–100, chance = 0", and a score never takes a % sign. Raw accuracy, in each bench's table, keeps its %.
Combined score
The plain mean, equal weights, of Reading · Turkish (all 2,198 Turkish questions, reading and grammar together, the Reading site's own score), Logic & Math (the Logic & Math site's pooled score, logic-math in its bench-scores.json) and Backgammon, unchanged by the splits into Turkish Reading and Turkish Grammar and into Logic and Math (project lead, #4173, #4234, #4239): the parts are views of their own, never averaged in. Only a model with a score on all three gets one; the others appear in each bench's tab. English Reading, Vision and Answer Check are not part of it.
A row gets a combined score only if one model answered every bench: the same checkpoint (one repo, one set of weights) or the same served version of a hosted model. A hosted service that changed its model between the benches' runs keeps its score on each bench, but gets no combined score:
- GLiDE: Fastino's API served a changed model between our benches' runs (Reading 6 Oct, Logic & Math 7 Nov, Backgammon 20 Oct to 7 Nov), so its bench scores are not one model's.
The page labels Combined "3 benchmark groups" and shows under each combined score its three parts: TR, Reading · Turkish (all 2,198 questions); L&M, Logic & Math (pooled); BG, Backgammon.
Decision Index
The plain mean, equal weights, of English Reading, Logic, Math and Answer Check, each on its own 0 to 100 scale with chance = 0 (Ümit #10906). Only a model with a score on all four gets one, and none where one model did not answer every bench, as for Combined. The page labels it "4 benches" and shows under each score its four parts: EN, English Reading; Logic, Logic; Math, Math; AC, Answer Check. The address ?view=index (or ?score=index) opens the page on it. "Decision Index" is a working name.
Answer Check: does the text give this answer?
This benchmark tests whether a model can classify a machine-written English summary, a question and its reference answer into three categories: the summary states the reference answer, states a conflicting answer, or does not answer the question. It measures grading of summaries, not the quality of the grader's own free-form answers.
Initial cohort: 2,069 consensus-labelled pairs from summaries of books, research papers, legal texts and agent sessions. Repository code is excluded. Raw texts, questions, reference answers, quotations and item-level predictions are not published.
Reference labels come from independent Claude Opus 5.5 and GPT-6.1 Sol judgments, not human annotation. For labels that assert an answer, a supporting quotation must be found in the summary. Disputed or invalid judgments are excluded. Within this cohort, 2,325 pairs were sampled and 256 were excluded. Sampling deliberately enriched cases that an earlier grader called conflicting answers; this is not a natural class distribution. Excluding disagreements may remove difficult cases. The summaries are English and machine-written; these results should not be generalized to Turkish or human-written answers without further testing. d1-ultra was previously selected in part for agreement with Opus on a separate sample, so agreement with this judge may favor it.
Accuracy is correct first outcomes divided by the frozen cohort size; missing or invalid terminal outputs count wrong. The displayed score is (accuracy − 1/3) / (2/3) × 100. Zero means uniform random choice among three labels, 100 means every pair correct, and below-chance scores may be negative. ECE uses 10 equal-width probability bins and is null when usable option probabilities are absent.
75 models have completed this cohort so far, 55 decision models and 20 chat models: 13 through their makers' APIs and 62 on our own GPUs (open decision models and chat models; a chat model's thinking setting is its row's tag). These numbers apply only to this cohort and request setting. Additional models receive rows only after completion on the same frozen cohort.
Requests contain one question each. Each score is one full run's: a decision model's first run, a chat model's run under answer rule v2 (below), except MiniCPM5-2B (thinking on)'s, LFM3-1.7B (thinking on)'s, Gemma 4 E4B (thinking on)'s, LFM3-1.7B (thinking off)'s, MiniCPM5-2B (thinking off)'s, Qwen3.5-4B (thinking off)'s, Gemma 4 E2B (thinking off)'s, Qwen3.5-2B (thinking off)'s and Raon-1 nano (thinking off)'s, the mean of three full runs with seeds 0, 1 and 2, with their lowest and highest beside it (a range, not a confidence interval).
How a chat model's answer is read (answer rule v2, from 15 Nov). A chat model writes its reasoning and then its choice. The label is the last marker 'ANSWER: X' in the visible reply. The word ANSWER may be in any letter case; X must be an uppercase A, B or C. The marker does not need to be on the final line. The model's separate reasoning text is never read. A reply with no such marker counts wrong. Each chat row's notes give how many of its replies had no marker and how many of those were cut off at the token limit.
Why the rule changed. The first rule (v1) accepted the word only in uppercase, 'ANSWER:'. LFM3-1.7B with thinking on writes 'Answer: X', so v1 counted about 40% of its replies wrong although they named an option. That was found by rerunning the model and keeping every reply. Rule v2 differs from v1 only in the letter case of the word ANSWER. On every kept reply, v2 changed none of the labels v1 had read; it only reads replies v1 missed. A reply with 'ANSWER: A' followed by a later 'Answer: C' would now read C.
Fresh runs, not rescoring. The earlier chat runs did not keep their replies, so they could not be read again. The 10 existing chat rows were therefore replaced by fresh runs with the same model, sampling settings and number of requests at a time, keeping every reply. Three thinking modes not run before were added the same way. The seed trios' servers follow the settings recorded for the earlier runs. For single runs the earlier runs' own server flags were not recorded, so the fresh runs use a server from the same launcher. The earlier runs and their scores stay in the history. A replaced row's change has two parts: the fresh run itself (a new sample from the model; under 2 points for every row) and the new rule (a large gain for LFM3-1.7B thinking on; small gains for LFM3-1.7B thinking off and for Qwen3.5-2B thinking on and off; none for the others).
Interrupted attempts. Two runs had an earlier attempt that stopped part-way: Qwen3.5-2B thinking on and Gemma 4 E4B thinking on. Those attempts are kept apart and never merged. Their rows note how many pairs they recorded and how many further requests may have been answered but not recorded. Each published run started again from the first pair, so its outcomes are that run's recorded outputs, not guaranteed to be the model's first outputs.
Test requests before a run. Before some decision models' scored runs, a few published pairs (one to three) were sent to the model as test requests, to check that its server answered with probabilities that sum to one. Those test outputs were printed, not kept. The scored row for each such pair comes from the run's own, later request, so for those pairs the recorded output is not the model's first output on the pair. The rows concerned say so in their notes and give the count. The count comes from our own record of the test calls; requests that others may have sent to servers they run are not covered.
Earlier scoring of the same material. Before Answer Check existed, two of the API decision models, d1-ultra and Gemini Decision, were used or tested as scorers in the compactor project. There they scored compaction texts and question-bank questions from the same held-out pool this benchmark's pairs were drawn from, in a different request form (all of a text's questions in one request). How many of the 2,069 published pairs they saw that way has not been counted, so their rows make no claim that these pairs were new to them.
Answer Check is a separate benchmark tab. Combined scoring is unchanged.
The Answer Check page's further columns come from the grading agent's metrics.json:
- Latency: client-side wall time per request, request sent to answer parsed; compare only rows with the same requests_in_flight. Each self-hosted row ran on its own hardware, our RTX PRO 6000 or a DGX Spark (in its notes and its box); runs on different hardware are not on an equal footing, even at the same requests in flight.
- List price per 1,000: billed input tokens x the provider's list price (price_source, price_date), per 1,000 questions; for a chat model billed per token through OpenRouter, input and output (reasoning included) tokens at its list prices; not an invoice; null on own GPU and on a subscription route, and null with its rate, source and date where no public price page backs that exact model's rate (checked 13 Nov): unverified, not zero.
- GPU time per 1,000: measured wall time of the whole run on one GPU (model load excluded) over all pairs it answered, per 1,000 questions: a rate, not a cohort share; own GPU only, not comparable with API dollars: a rate over the whole run, its wall time divided by all the pairs it answered, at the run's requests in flight; not a measurement of these pairs alone. It is measured on the row's own hardware, named under its value: a DGX Spark's seconds and an RTX PRO 6000's are not like for like.
- Wrong answers caught: recall_other: texts stating a conflicting answer that were flagged 'other', of the 329 such pairs; false alarms: gold 'reference' or 'none' pairs labelled 'other', over false_alarm_pairs; invalid replies are not counted as correct rejections, so 1 - false_alarm_rate is not an accuracy; read it beside invalid (1,740).
- Recall by label: of all gold pairs with that label, the share labelled so; an invalid reply counts as missed (the reference answer 942 pairs, a conflicting one 329, none 798); in each row's notes on its wrong answers caught.
- Invalid: replies without a usable label; for chat, no last matching ANSWER marker in the visible output under answer rule v2 (keyword in any letter case, option an uppercase A, B or C); counted wrong.
- Three seeds: a row with 'runs' covers three full runs with seeds 0, 1, 2 under its first run's id: recalls and false_alarm_rate are the mean of the three, latency is over all their requests, gpu_seconds_per_1000 the mean rate; 'invalid' is null and each seed's count is in invalid_per_seed.
- Measurement gap: present only where a run was interrupted and resumed: latency and gpu_seconds_per_1000 are then null, with the reason; scores are unaffected.
- Probabilities: this run's records hold per-option probabilities for every pair; says nothing about other runs of the model; no confidence is derived for chat models; calibration_error stays in bench-scores.json.
A cell without a score
Each says why, the reason on hover and on keyboard focus: Not run, the model is not on that bench; Not applicable, the score does not apply: Combined for a model without all 3 benchmark groups, or where one model did not answer every bench, and a bench whose file is not published yet. Sorting by a column puts such cells last.
Learning from experience
"Did it improve?": the points a model gains from its own experience, as a bench's file gives them (improvement_points). A column of its own, never part of the combined score.
When we write "tied"
No interval bars and no ranges: where two scores are too close to tell apart, the page says tied. The rule, the same everywhere on the page:
- On one bench, two models are tied when their scores differ by less than 100 ÷ √n points, n being the smaller of their two question counts on that bench (Backgammon: scored decisions; Answer Check: pairs). On Turkish Reading's 656 questions that is 3.90 points; on Turkish Grammar's 1,542 2.55; on English Reading's 547 4.28; on Vision's 763 3.62; on Reading · Turkish's 2,198, in the Combined score, 2.13; on Answer Check's 2,069 pairs 2.20. The margin is about the size of the noise a score on n questions carries, so a smaller bench needs a larger gap.
- On the combined score, a mean of three, the three benches' margins add up as noise does: √(m₁² + m₂² + m₃²) ÷ 3. On the Decision Index, a mean of four, the same: √(m₁² + m₂² + m₃² + m₄²) ÷ 4 (1.51 points at the full question counts).
- The ranking marks a bar "tied" when it is within the margin of the bar in view before it; Head to head says "tied" for a bench where the two are within it; each bench's table writes "tied with" the row above. Ties are not chained: A tied with B and B with C does not tie A with C.
- The table under the chart numbers its rows in a Rank column: the place by the score chosen among the models in view, 1, 2, 3 without repeats; models with the same score keep the table's order, which claims no difference. Each bench's table numbers its rows the same way, and "tied with" there marks the margin above. No bench's file carries the all-pairs test the Reading site's Leaderboard draws its tiers from, so there are no tiers here.
Calibration error
ECE: the answers sorted into 10 equal-width bins of the chosen option's probability; in each bin the gap between the mean probability and the share right, weighted by the answers in it; shown in points (×100), lower is better ↓. Reading · Turkish's is its Text tab run's, as the Text tab gives it; Turkish Reading's and Turkish Grammar's are the same run's, computed the same way over the part's answers that carry a probability; English Reading's is its first English run's, computed the same way (none for a run without a probability); Vision's the page-image run's; the other benches' as their files give them. On the Combined view the chart uses the mean of the benches' calibration errors: the mean over the benches in Combined (Reading · Turkish, Logic & Math, Backgammon) that give the model one, so a model without one on a bench gets the mean of the other two; a model with fewer than two is left out of that chart.
Latency
One pinned machine for every model: spark2, a DGX Spark (GB10), one request at a time, Reading · Turkish requests (the Text tab's runs); the median and the 95th percentile (nearest rank) of the time per request, in milliseconds. A run's machine is read from the host its requests went to, not from the hardware text. Models timed elsewhere, on another machine or through an API, read "not timed on the pinned machine yet".
Type and size
The type keys (All · Chat models · Decision models; Decision models when the page opens) and the class keys (All and the Reading site's Leaderboard's classes, the ones no model here falls in left out) choose the models in view, the two combined: the chart, the table under it, Head to head's choices, Full results and the charts follow them. A second tap on a pressed key goes back to All, as on the three sites. The address keeps the choice: ?models=all or chat (the Reading site's form) and ?class= a class. Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. A model of this site is classed as on the Leaderboard (Site.model_class); a model only on the other sites' benches by its file's params_total, else Undisclosed, as the files give no list price.
Size
Total parameters (and the active count for a mixture of experts) as the bench files give them, in billions; "not disclosed" where the maker states none. The table's size is the Leaderboard's: the total only, MoE after a mixture of experts, 2× after two models run together, empty where undisclosed.
One model across benches
Two rows are one model (the three sites' rule) when both have a Hugging Face repo and the repos match (in any case, without a commit, and without the org where one side gives none); else, unless both have repos that differ, when their model_key match: the three sites' join key, the model's name without its setting or route, lower case, hyphens for spaces, "-with-web" for a setup with web search, which never joins the plain one; a Reading model_key the three sites' shared names table maps to another (its row's source_model_key: Reading's qwen-3.5-397b-a17b is qwen3.5-397b-a17b) joins as that one. A repo on one side never keeps rows apart, and the route never splits a model (Ümit #4270). Else, for a row another site names differently, by the model_key a fixed list gives its name (the alias list, page_config.json all_benches.aliases: Backgammon's "Decider v1.2 27B" is Perplexity Decider 27B v1.2, its "Mercury Decide" is Mercury Decide, and so on); else, for a row without model_key, by the name in lower case without the maker's name, setting and route words, in any word order ("Decider v1.2 27B" meets "Perplexity Decider 27B v1.2" with the tag "API"). A row never joins a model already on its bench. The model shows the name of this site's row, else Logic & Math's. Where the benches ran different setups of one model (our hardware's checkpoint and a hosted API, or two checkpoints), the table under the chart says which bench ran which, and so does the model's page. Each bench's commit is on the model's page: the benches may have run different commits of one repo.
Their score
Jev Decision Index 0.4.1, "Full score", as of 6 November 2026 (source), for its 15 open decision models: a sanity check beside ours, never shown as our score. Jev 1.14 is a reference bar: TypeSafe's hosted Jev 1.14, the model the index's entries reproduce: a reference bar, with no index score of its own here.
Colours, marks and tags
Each bar and dot takes its maker's colour from maker_colours.json, the one colour map of the three sites, with the maker's mark before the model's name in the ranking (a plain shape where the maker's mark is not freely licensed). Makers without a colour there yet read grey: none. Every model shows by its maker's official name. Beside it, a small tag says how the setup ran where its bench's file gives one (the file's tag field: setting · route · snapshot, as "API · with web" or "high · chat"); two rows of one model differ only by their tag, and rows join by the rule above, never by their names. Then a second small tag, the training method, where our setup records state it (fine-tune, post-trained, LoRA, blend …); blank where they do not.
Sources
- Reading · Turkish (in Combined only): https://zekiworks.github.io/analysis/decision-models/bench-scores.json — published, 116 rows, updated 16 November 2026
- Turkish Reading: https://zekiworks.github.io/analysis/decision-models/bench-scores.json — published, 116 rows, updated 16 November 2026
- Turkish Grammar: https://zekiworks.github.io/analysis/decision-models/bench-scores.json — published, 116 rows, updated 16 November 2026
- English Reading: https://zekiworks.github.io/analysis/decision-models/bench-scores.json — published, 122 rows, updated 16 November 2026
- Vision: https://zekiworks.github.io/analysis/decision-models/bench-scores.json — published, 22 rows, updated 16 November 2026
- Logic: https://zekiworks.github.io/analysis/logic/bench-scores-logic.json — published, 104 rows, updated 16 November 2026
- Math: https://zekiworks.github.io/analysis/logic/bench-scores-math.json — published, 105 rows, updated 16 November 2026
- Logic & Math (in Combined only): https://zekiworks.github.io/analysis/logic/bench-scores.json — published, 103 rows, updated 16 November 2026
- Backgammon: https://zekiworks.github.io/analysis/backgammon/bench-scores.json — published, 57 rows, updated 16 November 2026
- Answer Check: https://zekiworks.github.io/analysis/grading/bench-scores.json — published, 75 rows, updated 16 November 2026
The build reads the other sites' files from these addresses and keeps what it read in benches.json.