Data

Everything the pages show comes from these files. They hold aggregate numbers and every answer, but not the questions: the publisher owns them.

Results version 0bdaa447 (what changed: the results changelog); picks.json 5e6555a0126d. The code that ran the benchmark and rebuilds these files is in the repository; its runner README explains how someone who owns the book rebuilds the question bank.

Results changelog

What changed in each results version that went live (results_version in results.json), newest first. It starts with results 0eb1e828, the release that added Liquid d1-2B; earlier versions are in the repository's history.

Results 0bdaa447 (3 November 2026)

• Text tab: Gemini 4 Argon at thinking level low, added at Ümit's request, the setting it played the backgammon paired test with: one run over the 2,198 scored questions through the Gemini API, 16 questions per request, the current prompt (run 557): 2,118 right (96.4%), 0 invalid. Its cost, $3.93, is at Google's regular price ($4 input, $20 output per 1M tokens; introductory $2 and $10 until 30 November 2026). A chat model: the decision-model lists are unchanged.
• Following from one more Text tab setup: the paired test of neighbouring setups compares 74 pairs (Holm over 74; still 11 pass), and the comparisons over every Text tab setup include it: answers at 95%+ confidence, and the same number of answers in Turkish (Holm over 73 instead of 72; each model's own p unchanged). No other run changed.

Results fb40b559 (3 November 2026)

• Hallucination tab, new section “Answers at 95%+ confidence” (results.json high_confidence), from runs already published: each decision model's Text tab run, keeping only the answers whose chosen option has a probability of at least 95% (an answer without a score never counts), with the wrong ones among them. Liquid d1-ultra keeps 549 answers with none wrong (run 199), and 538, 542, 534 and 550 with none wrong in its repeat runs 200 to 203; SubQ Snap 1.1 keeps 665 with 18 wrong (run 346), Elephant-1 393 with 13 (run 373) and Celeris-1 Decision 123 with none. For the three models Artificial Analysis's Decision Index puts level at 65, the share wrong in each of their five runs at 50%, 70%, 80%, 90%, 95% and 99%: at every whole-percent cutoff up to 98%, d1-ultra's worst run has a lower share wrong than the best run of SubQ Snap 1.1 and of Elephant-1; at 99% all three are at or near 0. In English, at 95%, d1-ultra has 1 wrong answer in each of its 5 runs, question 280 each time.
• Turkish vs English tab, new section: Liquid d1-ultra against each of the other 16 models on the 547 kept English questions (english_reading.against_reference): the exact McNemar test on the questions only one of the two got right, with Holm's correction over the 16. Too close to call: Fastino GLiDE v1.2, Mammoth-1, vLLM-SR d3, Drex Max v1, vLLM-SR d3-ultra, Perplexity Decider 27B v1.2 and Drex Max 1.1 (adjusted p 1) and Liquid d1 (0.069); behind d1-ultra: the other eight, from Jev 1.14.0 (0.020) down, including SubQ Snap 1.1 (0.010) and Elephant-1 (0.008).
• AA page (aa-decision-index/): “Our five reply charts”, the five cards of our thread on X, replace “Our four reply charts”, each with its alt text from the thread, with a note: Artificial Analysis does not name its GLiDE version; the rest of the page uses Fastino's hosted GLiDE, while the cards show GLiDE v1.2, which we ran on our GPU. Every number on the cards is a published one.
• No other number changed.

Results 8d77fddf (3 November 2026)

• Turkish vs English tab, risk and coverage and the main table's “Bilmiyorum” column: vLLM-SR d3, vLLM-SR d3-ultra and Drex Max v1 have no English run with “I don't know”, so they are outside the comparison with “Bilmiyorum”, but their Hallucination tab runs cover the same 547 kept questions; as Zeki pointed out, their Turkish “Bilmiyorum” points now come from those runs counted on the kept questions instead of reading Not measured: d3 (run 358) 494 right, 53 wrong, 0 “Bilmiyorum”; d3-ultra (363) 500, 46, 1; Drex Max v1 (367) 492, 53, 2 (english_reading.coverage, whose definitions say so). Their English points and every comparison of the two stay Not measured. No other number changed.

Results 8f69ed8d (3 November 2026)

• Turkish vs English tab, “I don't know” with only the language different, at Ümit's request: on the 547 kept questions, each model's first English run with “I don't know” against its Turkish run with the option also written “I don't know” (the Hallucination tab's option-word run, counted on the kept questions), in results.json (english_reading.dont_know_word); the two runs' instructions differ only in the word that names the language. Seven models have both runs; none of the seven differences in wrong answers is significant after Holm's correction over the seven (exact McNemar; smallest adjusted p 0.21, SubQ Snap 1.1). The main table's “I don't know” columns now show this comparison, leading with Answered and Wrong of answers, then non-hallucination, with the Turkish “Bilmiyorum” runs in a column of their own; a new section shows it with two charts and a table. The comparison with “Bilmiyorum” stays as published. No other number changed.

Results 6cc17354 (3 November 2026)

• Turkish vs English tab: vLLM-SR d3, vLLM-SR d3-ultra and Drex Max v1 added at Ümit's request, one English run each on the 547 kept questions (554, 550 and 552), with the same server, checkpoint and settings as their Text tab runs (259, 269 and 322) on the same kind of GPU; one run each because their repeated Turkish runs changed no answers. Right in Turkish against English: d3 91.0% against 89.8%, d3-ultra 91.4% against 89.4%, Drex Max v1 91.4% against 89.6% (exact McNemar p 0.32, 0.18 and 0.19; adjusted p 1 each, with Holm's correction now over 17 models). Microsoft-Decision-1's difference stays the only significant one; the other adjusted p-values rise with the family (Liquid d1-ultra from 0.084 to 0.105, Perplexity Decider Flash v1.2 from 0.055 to 0.068). The three leave the list of models not run by decision, which is now empty. They have no English run with “I don't know”, so that comparison and its charts are unchanged; in risk and coverage, each has its Turkish “I don't know” point and its other native points read Not measured.
• Hallucination tab, the same number of answers in English: the three join the family (17 models, Holm over 16). Of the 496 answers, d3 keeps 29 wrong, d3-ultra 31 and Drex Max v1 28 (Liquid d1-ultra 29); none differs significantly, and the other models' verdicts are unchanged (6 significantly more, the rest could be chance). No other number changed.

Results 40598194 (3 November 2026)

• Hallucination tab, the same number of answers: every setup's curve can now be seen, as Zeki asked. Under Info, one small plot per Turkish setup (all 73) shows its curve against Liquid d1-ultra's, with its runs with the option marked, or "Not measured" where it has none; the two main plots are unchanged. In results.json, each entry of dont_know_same_answers carries wrong_by_rank (its five-option run's scored answers in selection order, "1" wrong and "0" right), from which every curve is drawn; it replaces the curves' point pairs, so the file is 1.9 MB smaller. No number changed.

Results 30f574b0 (3 November 2026)

• Hallucination tab, the same number of answers, corrected as Zeki asked: each model now keeps an actual selection of exactly k answers, equal scores taken in question-id order and never by whether the answer is right, so every count is a whole number (the proportional share of tied answers is gone). Hosted GLiDE goes through its Text tab run 129, so its count is 325 wrong of 1,582 (300 with run 517, which stays in the confidence cut-off comparison); its markers come from runs made with Fastino's glide_v2 engine, as the page says. The curves have one point per number of answers, from 1 to the run's scored answers, in English too (no longer the Turkish vs English tab's curves, which keep their own rule). 33 Turkish counts and 2 English ones move (Jev 1.14.0 44 instead of 43.5 and GPT-6 Luna 58 instead of 58.3 in English); p moves only for Kimi K3.1 Pro (run 320; 0.051, Holm 0.10) and GLM-5.3-Flash (run 160; 0.71) in Turkish and Jev 1.14.0 (0.024, Holm 0.12) in English, and some intervals shift slightly. The verdicts stay: in Turkish all 36 other decision models keep significantly more wrong answers than Liquid d1-ultra, and of the 36 chat and general models 19 significantly fewer, 15 more and 2 could be chance; in English 6 of the 13 others significantly more. No other number changed.

Results 5fbf4cb3 (3 November 2026)

• Hallucination tab, the same number of answers: at Ümit's request, the Turkish comparison now covers every setup of the Text tab (73, decision models and chat and general models alike; hosted GLiDE through its five-option run 517), with Holm's correction over the 72 besides Liquid d1-ultra; English is unchanged (14 models, Holm over 13). Each model's p is as before, the resamples being the same; the adjusted p of the 10 models published in 64cea0e4 rises from 0.002 to 0.014, still significant. All 36 other decision models keep significantly more wrong answers than Liquid d1-ultra's 167 of 1,582; of the 36 chat and general models, 19 keep significantly fewer (from GPT-6 Astra's 4.4 expected wrong to Claude Sonnet 5.5's 114.3), 15 significantly more, and for 2 (Kimi K3.1 Pro and GLM-5.3-Flash) the difference could be chance. The plot keeps the 11 models with runs with the option. No other number changed.

Results 64cea0e4 (3 November 2026)

• Hallucination tab, new section: the same number of answers, in results.json (dont_know_same_answers), from runs already published. Each model's five-option run keeps its k most confident answers, k being Liquid d1-ultra's answered count with “I don't know” (1,582 of the 2,198 Turkish questions, run 511; 496 of the 547 kept English questions, run 449), and tied scores at the k-th answer count in proportion. In Turkish, Liquid d1-ultra's five-option run keeps 167 wrong of 1,582 (10.6%), and all 10 other models keep significantly more, 7.7 to 26.5 points more (paired bootstrap over questions with 10,000 resamples, Holm's correction over the 10; hosted GLiDE with its five-option run 517). In English, it keeps 29 wrong of 496 (5.8%); 6 of the 13 others keep significantly more (Inception Mercury Decide, Microsoft-Decision-1, Elephant-1, Celeris-1 Decision, GPT-6 Luna and Perplexity Decider Flash v1.2) and none significantly fewer. Each model's curve of wrong of answers against answered is drawn with its runs with the option marked.
• Every “I don't know” table (the Hallucination tab's, the option word's, the English questions' on both tabs and the four models') now leads with Answered ((right + wrong) / questions; higher is better) and Wrong of answers (wrong / (right + wrong); lower is better), worked out from the published counts; the non-hallucination rate follows with its formula. No other number changed.

Results ce52c83e (3 November 2026)

Analyses of the existing runs only, approved by Ümit; no new run, and no published measurement changed.

• Hallucination tab: each model's run with the option written “I don't know” (the 2,198 Turkish questions) against its five-option run held back by its own confidence at the same attempt rate, with the tab's derived method (the cut-off nearest the option run's attempt rate, the higher one on a tie), in results.json (dont_know_option_word.vs_confidence). Hosted GLiDE is compared with its new five-option run 517, not run 129. Each cut-off reached the option run's answered count exactly. Confidence left fewer wrong answers for all 7 models that chose the option on at least 1% of the questions (9 to 49 fewer); with the exact McNemar test of wrong answers and Holm's correction over those 7, only Liquid d1-ultra's difference is significant (216 wrong with “I don't know”, 167 by confidence; adjusted p 0.00046). Mammoth-1's, significant with “Bilmiyorum”, is not with “I don't know” (adjusted p 0.078). The banner and What we found now give both versions.
• Turkish vs English tab, new section: risk and coverage on the 547 kept questions, in results.json (english_reading.coverage): for each of the 14 compared models, the native operating points with Turkish “Bilmiyorum”, the Turkish wording control with “I don't know” and English “I don't know” (coverage, error rate among answered, right, wrong and abstained; Not measured where a model has no such run), and a confidence curve per language from its five-option runs, plotted per language. Full-bank runs are counted on the 547 questions only.
• The footer's “updated” date is now the day the results went out, 3 November 2026 (it was the leaderboard's newest run, 1 November). Here that is run 548's finish time; from the next release that changes the results, it is the export time, kept by any re-export that changes nothing.

Results 612d3cdf (3 November 2026)

• Turkish vs English tab and the Hallucination tab's English-option section: Celeris-1 Decision's English run with “I don't know” (548), in results.json (english_reading.dont_know.english_only): 418 right, 37 wrong and 92 “I don't know” of the 547 kept questions (non-hallucination rate 71.3%, attempt rate 83.2%). It has no Turkish run with “Bilmiyorum” on these questions, so its Turkish side, the paired difference and the test read Not measured; it is in no Holm family. The comparison of the 13 models with runs in both languages and the four-model comparison are unchanged. No other number changed.

Results aa79451f (3 November 2026)

• Turkish vs English tab: Celeris-1 Decision added, at Ümit's request. One English run on the 547 kept questions (546), through the same API and settings as its Text tab run 150, whose responses name the same model, celeris-1-decision: 87.4% right in Turkish against 84.1% in English (42 right only in Turkish, 24 only in English; exact McNemar p = 0.036, adjusted p = 0.36 with Holm's correction over the 14 compared models). Its repeated Turkish runs changed answers, so the five-run rule would give it five English runs; it has one, as the tab says. With 14 models in the correction instead of 13, the other adjusted p-values rise; only Microsoft-Decision-1's difference is significant, as before. Earlier entries (5c638c7d, 3ad29607) gave a missing key as the reason Celeris-1 Decision was not run: that was our error, as the key was there.
• Clef is still not run: its own model code rounds each option's probability to 4 decimal places, and an allowance that would let its runs pass our probability check awaits a decision. Apart from the adjusted p-values above, no number changed.

Results 47420478 (2 November 2026)

• Hallucination tab, English-option section: the four models of Ümit's follow-up (Liquid d1-ultra, SubQ Snap 1.1, Elephant-1 and Mammoth-1) with Holm's correction over these four, in results.json (dont_know_english_option). The same first runs and the same exact McNemar test of wrong answers as the comparison of all 13 models; only the correction's family differs. None of the four differences is significant over the four either: the smallest adjusted p is Liquid d1-ultra's, 0.077 (0.21 over the 13). The comparison of the 13 models is unchanged. No other number changed.

Results 3ad29607 (2 November 2026)

• English reading: four more models on our GPUs, each run as its Text tab run (the same loader, checkpoint and settings): Mammoth-1, Fastino GLiDE v1.2, Perplexity Decider 27B v1.2 and Perplexity Decider Flash v1.2, with one English reading run and one English “I don't know” run each (and, for Decider Flash v1.2, which has no Hallucination tab run, one Turkish “Bilmiyorum” run on the kept questions). With 13 models in Holm's correction, only Microsoft-Decision-1's accuracy difference is significant (higher in English); in the “I don't know” comparison only Perplexity Decider Flash v1.2's difference in wrong answers is (more wrong answers in Turkish). The three Perplexity and Fastino models made their Text tab runs on a DGX Spark and their English runs on our RTX PRO 6000 GPU; the page says so. Not run: Clef (its server reports probabilities rounded to 4 decimal places, so its English smoke run failed our probability check, and no allowance for that rounding has been decided) and Celeris-1 Decision (no key).
• Hallucination tab, new section: the option's word. The same 2,198 Turkish questions with only the sixth option changed to “I don't know” and the sentence naming it (prompt version fb17d583a3af), one run for each of the tab's 11 models. With the English word, “I don't know” left significantly fewer wrong answers for Elephant-1 (non-hallucination rate 49.3% against 34.8% with “Bilmiyorum”), Drex Max 1.1 (54.1% against 38.2%), Fastino GLiDE v1.2 (49.9% against 39.5%) and Mammoth-1 (73.2% against 68.7%) (exact McNemar test of wrong answers, Holm's correction over the 11); for the other seven the difference could be chance. Drex Max 1.1's run (513) stopped once on HTTP 529 (Nace.AI overloaded) and was resumed.
• Hallucination tab, new section: hosted GLiDE against a new five-option run. Run 129 (6 October) predates Fastino's change of the hosted model, so the confidence comparison was redone with a new five-option run, 517 (2 November): 68.1% right; held back by its confidence on the same 521 questions, it left 351 wrong answers against 356 with “Bilmiyorum” (a derived non-hallucination rate of 59.7% against 59.4% with the option; exact McNemar p = 0.76). The table's row is left as published, against run 129, and run 517 is kept out of the Text tab.
• English reading tab, new table: the languages each model's card or docs list (19 models, checked 2 November). Only Elephant-1 and Mammoth-1 name a language (English, by their cards' language: en tag), Jev's docs call English its primary language, and no card names Turkish.
• Files: results.json gains dont_know_option_word and hosted_glide_rerun, and english_reading its new rows; answers.json gains dont_know_word_runs, rerun_runs and dont_know_word_prompts, the new English runs in their lists, and the fields dont_know_word and rerun (null) on every other run; runner/ gains --dont-know-word and --rerun; page_config.json gains model_languages. No published number changed apart from the English reading tab's adjusted p-values, now over 13 models.

Results 7dc0f9c7 (2 November 2026)

• English reading: Elephant-1 added, the only one of the nine models on our GPUs to run in English. One English reading run (499; same server, image and checkpoint as its Text tab runs) against its Text tab run 373 on the 547 kept questions: 85.4% right in English against 83.5% in Turkish (39 right only in English, 29 only in Turkish; adjusted p = 1). With nine models in Holm's correction instead of eight, Liquid d1-ultra's difference (exact McNemar p = 0.007) is no longer significant after the correction: its adjusted p is 0.056 (0.049 over eight). Microsoft-Decision-1 stays significant (adjusted p < 0.0001); the other adjusted p-values rise with the larger family.
• With an “I don't know” option: Elephant-1's English run (501) against its Hallucination tab run (382) counted on the kept questions: a non-hallucination rate of 25.6% in English at an attempt rate of 95.8%, against 15.5% at 97.3% in Turkish (31 wrong only in English, 46 only in Turkish; adjusted p = 0.66). No difference in wrong answers is significant after Holm's correction over nine; the smallest adjusted p is now 0.073 (Liquid d1).
• Elephant-1 leaves the list of models not run by decision; eight remain. Apart from the adjusted p-values of the two comparisons, no other number changed.

Results 34bac8a0 (2 November 2026)

• English reading tab: a new section headed With an “I don't know” option, the Hallucination tab's test in English. The same 547 kept questions with a sixth option, “I don't know” (the instructions end “If you do not know the answer, choose I don't know.”), asked of the eight compared models in 28 runs (five each for Liquid d1-ultra, Liquid d1, Jev 1.14.0, Microsoft-Decision-1 and Mercury Decide; one each for SubQ Snap 1.1, Drex Max 1.1 and GPT-6 Luna), against the same models' “Bilmiyorum” answers on the same questions in Turkish: the Hallucination tab's published runs counted on the kept questions for Liquid d1-ultra, SubQ Snap 1.1 and Drex Max 1.1, and 21 new runs on the kept questions for the other five (Liquid d1 asked as d1 in both languages). Six of the eight had a higher non-hallucination rate in English than in Turkish and two a lower one; no difference in wrong answers between the two languages is significant after Holm's correction (exact McNemar; the smallest adjusted p is 0.065, for Liquid d1). Each rate is read with its attempt rate. The section changes no other number.
• Files: results.json gains english_reading.dont_know; answers.json gains english_dont_know_runs (28 runs), english_dont_know_turkish_runs (21 runs) and english_dont_know_prompts; runner/ gains the English “I don't know” mode (--english with --dont-know, prompt version 413d6775fc58). No other number changed.

Results 5c638c7d (2 November 2026)

• New tab, “English reading” (english-reading/): the Text tab's 656 reading questions (units 1 to 6; the grammar questions left out) translated into English by Claude Opus 5.5 (effort high, through the Claude Code CLI), which in a separate request flagged the questions whose answer depends on Turkish wording. The 109 flagged questions are left out and 547 kept; no person checked the translations. Eight decision models answered the kept questions in English as on the Text tab (one question per request, each model's own settings), with the instructions naming English instead of Turkish and the options under “Options:” instead of “Seçenekler:”, in 28 runs: five each for Liquid d1-ultra, Liquid d1, Jev 1.14.0, Microsoft-Decision-1 and Mercury Decide, whose repeated Turkish runs changed answers, and one each for SubQ Snap 1.1, Drex Max 1.1 and GPT-6 Luna. Each model's first English run against its Text tab run on the same 547 questions (exact McNemar test; Holm's correction over the eight): Microsoft-Decision-1 scored higher in English (85.7% against 76.4%, adjusted p < 0.0001) and Liquid d1-ultra higher in Turkish (93.6% against 90.7%, adjusted p = 0.049); for the other six the difference could be chance. Liquid d1 was asked as d1, because Liquid's API no longer serves the d1:free name its Turkish runs used. Not compared: the hosted Fastino GLiDE, because since 10 October Fastino's API has answered with its glide_v2 engine, not the glide engine of its Turkish run. Not run: Perplexity Decider 27B v1.1 and Celeris-1 Decision. Not run by decision: the nine models that run on our GPUs. The tab sits apart from the leaderboard and changes none of its numbers.
• New and changed files: english-reading.json, the track's public record (the translator, both prompts, each reading question's flag reasons, whether it is kept and the SHA-256 of its English text; no question text, and the translations themselves are not published); results.json gains english_reading; answers.json gains english_runs and english_prompts, and every other run an english field (null); runner/ gains the runner's --english mode, run_english.sh and translate_reading.py. No other number changed.

Results 973d93d3 (2 November 2026)

• Hallucination tab: each model's run with “Bilmiyorum” against its five-option run held back by its own confidence on as many questions, as wrong answers on the same questions (exact McNemar test; Holm's correction over the 7 models that chose “Bilmiyorum” on at least 1% of the questions), in results.json (dont_know.models[].vs_confidence). Confidence left significantly fewer wrong answers for Liquid d1-ultra and Mammoth-1 (adjusted p = 0.016 each); for the other 5 the difference could be chance. The tab says so at the top and under What we found. SubQ Snap Preview stays not run, now by decision. The charts on the Text tab show the 20 most accurate decision models by default, instead of 8. No other number changed.

Results 8cacd512 (2 November 2026)

• Shorter notes under three leaderboard rows. Elephant-1 and Mammoth-1: "Probabilities rounded to 6 decimals by the model's own loader." SubQ Snap 1.1: "Its probability check used an experimental 32-bit rule of ours."; the note on run 346's two pauses is no longer on the leaderboard and stays on that run's page. The three models' full notes keep the same facts, reworded to open with those sentences. No number changed.

Results 07bac50e (2 November 2026)

• SubQ Snap 1.1, run 346: its note corrected. The retry it mentions belonged to the first paused request, not the second. Without the two paused requests, its 90th-percentile request time is 0.387 s, not 0.388 s, while its median stays 0.378 s; the note had called both unchanged. Its first pause now reads 12 minutes (it lasted 11 minutes 44 seconds) instead of 11, rounded like the second. No number changed.

Results 2c430f54 (1 November 2026)

• Added SubQ Snap 1.1 (subq-snap-1.1, public preview) through SubQ's Decisions API. Text: five runs over the 2,198 scored questions (runs 346, 387, 388, 389 and 390), 63.9% right, 12th of 37 decision models, the five runs identical. Page images: condition A from run 346 (58.1%), B (45.7%), C (62.9%) and D with each of the four OCR models (49.7% to 54.3%), over the 763 page-image questions (runs 62 to 67). Its probabilities arrive as 32-bit floats. As an experimental rule of ours for this model only, the check before its runs accepted sums within 32-bit rounding. Run 346 was paused twice while that rule's approval was confirmed; its note gives the effect on its request times. The SubQ Snap Preview rows are unchanged.
• "I don't know option" tab: SubQ Snap 1.1 added (run 392).
• From the new row: the summary's "Better than a flat guess" line now names SubQ Snap 1.1 with Perplexity Decider 27B v1.1 (31.5% each) after Open-Jev 27B v1.1. SubQ Snap Preview's places on the Routing tab move down one in front of Opus 5.5 low (seventeenth) and Astra low (fifteenth). One neighbour pair is replaced by two, and 16 comparisons have new Holm-adjusted p-values, because the correction now covers 2,628 pairs of runs instead of 2,556. Comparison letters gain the new row's letter where it joins a group; no tie between existing rows changed. The short chart labels now read "Snap Preview" and "Snap 1.1". No other existing number changed.

Results 74c13f91 (1 November 2026)

• Added Base Labs' Elephant-1 and Mammoth-1 (open weights): baselabs/elephant-1 at b3f8a93 and baselabs/mammoth-1 at 61cec44 on our GPU, through their model cards' loader (qev's DecisionEngine with the checkpoints' serve.patch). Text: five runs each over the 2,198 scored questions, each model's five runs identical. Mammoth-1 (runs 374, 377, 380, 383, 384): 67.1% right, 8th of 36 decision models. Elephant-1 (runs 373, 375, 376, 378, 379): 52.4% right, 26th of 36. Their loader rounds probabilities to 6 decimals, and the check before their runs allowed for it. Their cost is at Baseten's list prices for the same models. They are not on the Page images tab: their runtime reads images only through an engine without the attention the models were trained with.
• New tab, "I don't know option" (dont-know/): the Text tab's questions with a sixth option, "Bilmiyorum" (I don't know), always last and never correct. It covers ten decision models in 18 runs, with accuracy, attempt rate, non-hallucination rate and the mean probability of "Bilmiyorum". It sits apart from the leaderboard and changes none of its numbers.
• From the two new rows: the summary's accuracy line now names Mammoth-1 after Drex Max v1 and 1.1. SubQ Snap's places on the Routing tab move down one (tenth in front of Sonnet 5.5 high, sixteenth in front of Opus 5.5 low, fourteenth in front of Astra low). The paired tests between ranking neighbours pair each new row with the rows on either side of it, in place of two earlier neighbour pairs. 17 other comparisons have new Holm-adjusted p-values, because the correction now covers 2,556 pairs of runs instead of 2,415. No other existing number changed.

Results 25290875 (1 November 2026)

• Liquid d1-2B (open weights): a note beside its row on how its request times were measured (one GPU with no other known work while its runs took place, not monitored throughout), and its four extra Page images passes in each of conditions B and C, identical to the table's runs 60 and 61, published in vision-repeats.json and described on the Page images page. No number changed.

Results 0eb1e828 (1 November 2026)

• Added Liquid d1-2B (open weights): LiquidAI/d1-2B at 10e7671 on our GPU, through its model card's own call. Text: five runs over the 2,198 scored questions (runs 340 to 344), 29.4% right, 29th of 34 decision models, the five runs identical. Page images: conditions B and C over the 763 page-image questions (runs 60 and 61). Its cost is not measured (†).

Answer key changelog

Every setup is scored on answer key version 2. Question bank SHA-256 797ad6c868bc1c9485bc54203f58839ad995b0443b7149f6a9e36cf0d07df4d7; key fingerprint cd7f998de262e355cd28f596465a51f4158211646ee0cbfbc85f5d8cdeced20a.

Every setup in this benchmark is scored on one frozen answer key. This file lists every change to that key.

How a fix works. A fix to the key needs a check against the book's own key, or a second person who reviews the question and its options without seeing the models' answers. Each fix creates a new key version, adds a row to this file (question ID, old answer, new answer, reason, date) and rescores every run.

To report a wrong key, see CONTRIBUTING.md.

How to identify a version. results.json names the key version under answer_key, with the SHA-256 of the question bank and the key's fingerprint: the SHA-256 of [[id, key, excluded], ...] over every question in answers.json, as compact JSON in question order (answer_key_fingerprint in benchmark_metrics.py).

reproduce.py recomputes the fingerprint and checks every exported answer against the key. export_results.py stops when the answers are graded against a key that is not frozen there.

Version 2 (frozen 13 October 2026)

• Question bank: questions-v2.sqlite, SHA-256 797ad6c868bc1c9485bc54203f58839ad995b0443b7149f6a9e36cf0d07df4d7.
• Key fingerprint: cd7f998de262e355cd28f596465a51f4158211646ee0cbfbc85f5d8cdeced20a.
• 2,210 questions with a key; 2,198 scored.

Version 2 is the key after the review of 4 October 2026. The review checked 25 disputed keys, confirmed 13 and removed these 12 questions from scoring.

Their printed keys stay in the data, but they count in no result. Topic, page, test and number locate each question in the book.

Later fixes

None yet. Each fix adds a row here and a section for its new key version.

No rows yet.

To report a wrong key, use the answer key correction form.

How to cite

Cite the results version, the answer key version and the commit of the repository you used.

# Generated by build_page.py from results.json. Do not edit by hand: run the build again.
cff-version: 1.2.0
message: "If you use these results, please cite them with the results version, the answer key version and the commit of zekiworks/analysis you used."
title: "When should you trust an AI model to make a decision?"
abstract: "We tested 55 models, 37 of them decision models, on a reading comprehension and grammar test in native Turkish (not translated): 2,198 questions."
type: dataset
authors:
  - name: "Hilde"
version: "results 0bdaa447, answer key version 2"
date-released: "2026-11-03"
url: "https://zekiworks.github.io/analysis/decision-models/"
repository-code: "https://github.com/zekiworks/analysis/tree/main/decision-models"
license: MIT
keywords:
  - "decision models"
  - "confidence"
  - "calibration"
  - "Turkish"

Licences

• Code, results and answers: MIT licence.
• The questions: owned by the book's publisher and not published. Question IDs, pages and printed numbers locate each one in the book.