Hallucination
The Text tab's 2,198 questions, with “Bilmiyorum” as a sixth option
A wrong answer counts as a hallucination: the model could have chosen “Bilmiyorum” (“I don't know”) instead. This tab is separate from the leaderboard; it ranks nothing on the Text tab and changes none of its numbers.
Holding back on as many questions, Liquid d1-ultra’s and Mammoth-1’s own confidence left significantly fewer wrong answers than “Bilmiyorum” did; for the other 5 models that used it, the difference could be chance. With “I don't know” in place of “Bilmiyorum”, Liquid d1-ultra’s own confidence left significantly fewer wrong answers than “I don't know” did; for the other 6 models that used it, the difference could be chance (What we found).
Non-hallucination rate
“Bilmiyorum” among the answers that were not right · higher is better
Decision models: worst 0.0%best 75.1% (higher is better) Read it with accuracy, below: a model that always chose “Bilmiyorum” would rank first with no right answers.
Saying “I don't know”, or held back by confidence
Each model's non-hallucination rate with “Bilmiyorum”, and its five-option run held back by its own confidence at about the same attempt rate · higher is better
With “Bilmiyorum” Held back by its own confidence (the derived rate)
Fewer wrong answers: “Bilmiyorum” or confidence?
Each model's first run with “Bilmiyorum” against its five-option run held back by its own confidence on as many questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 7 models that chose “Bilmiyorum” on at least 1% of the questions · the shorter bar left fewer wrong answers
Wrong only with “Bilmiyorum” Wrong only when held back by its own confidence * A significant difference (Holm-adjusted p below 0.05): Liquid d1-ultra and Mammoth-1 (p = 0.016 each). The other 5: p = 0.72 each. Not in the test, as they chose “Bilmiyorum” on fewer than 1% of the questions: vLLM-SR d3-ultra, Drex Max v1, Perplexity Decider 27B v1.2 and vLLM-SR d3.
Accuracy with “Bilmiyorum”
Share of the 2,198 questions answered right with “Bilmiyorum” on offer · higher is better
Decision models: worst 47.1%best 67.6% (higher is better) The same model with five options (the Text tab)
The English questions: “I don't know” in English against “Bilmiyorum” in Turkish
The 547 reading questions of the Turkish vs English tab, in English with a sixth option, “I don't know”, against the same models' “Bilmiyorum” answers on the same questions in Turkish (this tab's runs counted on those questions, or runs on those questions only). Counts and the paired test: under Info; how the questions were translated and kept: Turkish vs English, Our method.
Eleven of the 13 models had a higher non-hallucination rate in English than in Turkish; two lower; one of the 13 differences in wrong answers is significant after Holm's correction: Perplexity Decider Flash v1.2 (more wrong answers in Turkish) (Paired test). Celeris-1 Decision has an English rate of 71.3% (92 “I don't know” of 129 not right) with an attempt rate of 83.2% (455 answered) but no Turkish run with “Bilmiyorum” on these questions, so no comparison or test. Read each rate with its attempt rate: a model that always chose the option would reach 100%.
The four models of the request: Holm's correction over the four
Liquid d1-ultra, SubQ Snap 1.1, Elephant-1 and Mammoth-1, as Ümit's follow-up of 2 November 2026 asked: each model's first run in English with “I don't know” and in Turkish with “Bilmiyorum” on the 547 kept questions, and the exact McNemar test of wrong answers, with p adjusted by Holm's method over these four. None of the four differences in wrong answers is significant with Holm's correction over the four (adjusted p below 0.05); the smallest adjusted p is Liquid d1-ultra's, p = 0.077. The charts below and the Turkish vs English tab correct over all 13 compared models.
| Model | Answered in Turkishwith “Bilmiyorum”: (right + wrong) / questions; higher is better | Answered in Englishwith “I don't know”: (right + wrong) / questions; higher is better | Wrong of answers in Turkishwith “Bilmiyorum”: wrong / (right + wrong); lower is better | Wrong of answers in Englishwith “I don't know”: wrong / (right + wrong); lower is better | Turkish non-hallucination rate“Bilmiyorum” / (“Bilmiyorum” + wrong); higher is better; right / wrong / option | English non-hallucination rate“I don't know” / (“I don't know” + wrong); higher is better; right / wrong / option | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 4 models |
|---|---|---|---|---|---|---|---|---|---|---|
| Liquid d1-ultra1 question per request | 94.5%517 of 547 answered | 90.7%496 of 547 answered | 5.0%26 of 517 | 7.9%39 of 496 | 53.6%491 / 26 / 30 | 56.7%457 / 39 / 51 | 20 | 7 | 0.019 | 0.077 |
| SubQ Snap 1.11 question per request | 96.5%528 of 547 answered | 96.2%526 of 547 answered | 9.7%51 of 528 | 12.0%63 of 526 | 27.1%477 / 51 / 19 | 25.0%463 / 63 / 21 | 39 | 27 | 0.18 | 0.35 |
| Elephant-11 question per request | 97.3%532 of 547 answered | 95.8%524 of 547 answered | 15.4%82 of 532 | 12.8%67 of 524 | 15.5%450 / 82 / 15 | 25.6%457 / 67 / 23 | 31 | 46 | 0.11 | 0.33 |
| Mammoth-11 question per request | 95.6%523 of 547 answered | 93.2%510 of 547 answered | 6.3%33 of 523 | 7.1%36 of 510 | 42.1%490 / 33 / 24 | 50.7%474 / 36 / 37 | 18 | 15 | 0.73 | 0.73 |
Answered = (right + wrong) / 547 and Wrong of answers = wrong / (right + wrong) come first, then the non-hallucination rate with its counts (right / wrong / option). * marks a Holm-adjusted p below 0.05 over the four.
Non-hallucination rate in Turkish and in English
The option among the answers that were not right, on the 547 kept questions: each model's first run in each language, ordered by the English rate · higher is better, read with the attempt rate (under the chart and in each bar's tooltip)
Turkish, with “Bilmiyorum”, the upper bar English, with “I don't know”, the lower bar Attempt rate, Turkish · English: Liquid d1-ultra 94.5% · 90.7%; Drex Max 1.1 98.4% · 92.1%; Mammoth-1 95.6% · 93.2%; Fastino GLiDE v1.2 98.9% · 94.5%; Elephant-1 97.3% · 95.8%; SubQ Snap 1.1 96.5% · 96.2%; Inception Mercury Decide 98.5% · 98.0%; GPT-6 Luna (Decisions API) 98.9% · 98.2%; Perplexity Decider Flash v1.2 98.9% · 99.1%; Jev 1.14.0 99.5% · 99.5%; Microsoft-Decision-1 99.6% · 99.5%; Perplexity Decider 27B v1.2 100.0% · 99.6%; Liquid d1 99.8% · 99.6%. A model that always chose the option would reach 100% with no right answers. Not charted, with no Turkish run with “Bilmiyorum” on these questions: Celeris-1 Decision (in the table under Info).
Wrong only in English, or only in Turkish?
Each model's first run in each language, both with the option, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 13 models · the shorter bar is the language with fewer wrong answers
Wrong only in English Wrong only in Turkish * A significant difference (Holm-adjusted p below 0.05): Perplexity Decider Flash v1.2 (p = 0.022). The other 12: p = 0.098, p = 0.21, p = 0.49, p = 0.49, p = 0.88, p = 0.88, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with no Turkish run with “Bilmiyorum” on these questions: Celeris-1 Decision (in the table under Info).
The option's word: “I don't know” instead of “Bilmiyorum”
With “I don't know” in place of “Bilmiyorum”, 8 of the 11 models chose the option more often and 2 less often; 1 as often. Their non-hallucination rates moved by -1.0 to +15.8 points. In the exact McNemar test of wrong answers, with Holm's correction over the 11, “I don't know” left significantly fewer wrong answers for Mammoth-1, Fastino GLiDE v1.2, Drex Max 1.1 and Elephant-1; for the other 7, the difference could be chance.
The same 2,198 Turkish questions, with only the sixth option's word and the sentence that names it changed. A model that skipped “Bilmiyorum” because it did not recognise the Turkish word should choose “I don't know” more often. The languages each model's card or docs list are on the Turkish vs English tab (Languages the model cards list). Table and method: under Info.
Non-hallucination rate by the option's word
Each model's first run with “I don't know” and with “Bilmiyorum” · higher is better
With “I don't know” With “Bilmiyorum” (the table above) Read it with accuracy and the attempt rate, in the table under Info.
“I don't know” against confidence
With “I don't know” in place of “Bilmiyorum”, we compared each model's first run with “I don't know” against its five-option run held back by its own confidence at the same attempt rate (each cut-off reached the option run's answered count exactly for the 7 that chose “I don't know” on at least 1% of the questions). Confidence left fewer wrong answers for all 7, 9 to 49 fewer. In the exact McNemar test of wrong answers, with Holm's correction over the 7, confidence left significantly fewer for Liquid d1-ultra (216 wrong with “I don't know”, 167 by confidence; adjusted p = 0.00046); for the other 6, the difference could be chance. Fastino GLiDE is compared with its new five-option run 517, not run 129 (why).
A family of its own: the paired test with “Bilmiyorum” is the chart Fewer wrong answers above, and neither changes the other. Table and method: under Info.
Fewer wrong answers: “I don't know” or confidence?
Each model's first run with “I don't know” against its five-option run held back by its own confidence at about the same attempt rate: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 7 models that chose “I don't know” on at least 1% of the questions · the shorter bar left fewer wrong answers
Wrong only with “I don't know” Wrong only when held back by its own confidence * A significant difference (Holm-adjusted p below 0.05): Liquid d1-ultra (p = 0.00046). The other 6: p = 0.078, p = 0.22, p = 0.25, p = 0.62, p = 0.62 and p = 0.63. Not in the test, as they chose “I don't know” on fewer than 1% of the questions: vLLM-SR d3-ultra, Drex Max v1, Perplexity Decider 27B v1.2 and vLLM-SR d3.
Hosted GLiDE against a new five-option run
Fastino changed its hosted model after 6 October, so run 129 (2026-10-06), the five-option run behind Fastino GLiDE's row in the table, is a different model from the one behind its “Bilmiyorum” run 351. A new five-option run, 517 (2026-11-02), scored 68.1% (1,497 of 2,198 right), against 67.9% for run 129. Held back by its own confidence on as many questions as run 351 said “Bilmiyorum” (521), the new run left 351 wrong answers, against 356 with “Bilmiyorum”: a derived non-hallucination rate of 59.7%, against 59.4% with the option and 58.0% from run 129 (exact McNemar test of the wrong answers, p = 0.76, not adjusted). The row in the table under Info is left as published, against run 129.
Fastino GLiDE: saying “I don't know”, or held back by confidence
Non-hallucination rate with “Bilmiyorum”, and with each five-option run held back by its own confidence at about the same attempt rate · higher is better
With “Bilmiyorum” Held back by its own confidence (the derived rate) Table and method: under Info.
The same number of answers
Each model's five-option run keeps only its k most confident answers, k being the number of questions Liquid d1-ultra answered with “I don't know” on offer, and we count the wrong ones among them. In Turkish, k = 1,582, the answers of Liquid d1-ultra's run 511 with “I don't know” on the 2,198 questions: Liquid d1-ultra's five-option run (run 199) keeps 167 wrong of 1,582 (10.6%); after Holm's correction over the other 73 setups, all 36 other decision models keep significantly more wrong answers, 7.7 to 72.9 points more (the closest, Fastino GLiDE v1.2, 18.3%), so Liquid d1-ultra keeps the fewest of the decision models; of the 37 chat and general models, 20 keep significantly fewer, 3.4 to 10.3 points fewer, 15 keep significantly more, 4.6 to 72.9 points more and for 2 (Kimi K3.1 Pro (run 320) with 141 and GLM-5.3-Flash (run 160) with 163) the difference could be chance, both keep fewer than 167 but not significantly. In English, k = 496, the answers of Liquid d1-ultra's run 449 with “I don't know” on the 547 kept reading questions: Liquid d1-ultra's five-option run (run 411) keeps 29 wrong of 496 (5.8%); after Holm's correction over the other 16 models, 6 (Inception Mercury Decide, Microsoft-Decision-1, Elephant-1, Celeris-1 Decision, GPT-6 Luna (Decisions API) and Perplexity Decider Flash v1.2) keep significantly more, 4.2 to 6.7 points more and for 10 the difference could be chance; Fastino GLiDE v1.2 keeps 26, fewer than 29 but not significantly and Drex Max v1 keeps 28, fewer than 29 but not significantly.
The difference from Liquid d1-ultra is tested by a paired bootstrap over questions, with p adjusted by Holm's method in each language. Each plot shows every number of answers, not only k: the line is the five-option run held back by its own confidence, the markers the model's runs with the option. Tables and method: under Info.
The same number of answers in Turkish
The 2,198 questions in Turkish: each model's five-option run answering only its most confident answers (the line), and its own runs with the option (the marker: “Bilmiyorum” “I don't know”); the table under Info has all 74 setups, the plot the 11 with runs with the option (every setup's curve under Info) · lower is better at the same coverage
- k = 1,582: Liquid d1-ultra's answers with “I don't know”
- Liquid d1-ultra: five options, run 199 · “Bilmiyorum”, run 350 · “I don't know”, run 511
- Fastino GLiDE v1.2: five options, run 248 · “Bilmiyorum”, run 369 · “I don't know”, run 525
- Fastino GLiDE: five options, run 129 · “Bilmiyorum”, run 351 · “I don't know”, run 510 (runs 351 and 510 made with Fastino's engine glide_v2, not run 129's glide)
- vLLM-SR d3: five options, run 259 · “Bilmiyorum”, run 358 · “I don't know”, run 544
- vLLM-SR d3-ultra: five options, run 269 · “Bilmiyorum”, run 363 · “I don't know”, run 540
- Drex Max v1: five options, run 322 · “Bilmiyorum”, run 367 · “I don't know”, run 542
- Drex Max 1.1: five options, run 326 · “Bilmiyorum”, run 352 · “I don't know”, run 513
- Mammoth-1: five options, run 374 · “Bilmiyorum”, run 386 · “I don't know”, run 519
- Perplexity Decider 27B v1.2: five options, run 217 · “Bilmiyorum”, run 372 · “I don't know”, run 531
- SubQ Snap 1.1: five options, run 346 · “Bilmiyorum”, run 392 · “I don't know”, run 512
- Elephant-1: five options, run 373 · “Bilmiyorum”, run 382 · “I don't know”, run 515
The same number of answers in English
The 547 kept reading questions in English: each model's five-option run answering only its most confident answers (the line), and its own runs with the option (the marker: “I don't know”) · lower is better at the same coverage
- k = 496: Liquid d1-ultra's answers with “I don't know”
- Liquid d1-ultra: five options, run 411 · “I don't know”, run 449
- Fastino GLiDE v1.2: five options, run 521 · “I don't know”, run 523
- Mammoth-1: five options, run 503 · “I don't know”, run 505
- vLLM-SR d3: five options, run 554
- Drex Max v1: five options, run 552
- vLLM-SR d3-ultra: five options, run 550
- Perplexity Decider 27B v1.2: five options, run 527 · “I don't know”, run 529
- Drex Max 1.1: five options, run 399 · “I don't know”, run 447
- Liquid d1: five options, run 430 · “I don't know”, run 445
- Jev 1.14.0: five options, run 410 · “I don't know”, run 446
- SubQ Snap 1.1: five options, run 413 · “I don't know”, run 448
- Inception Mercury Decide: five options, run 398 · “I don't know”, run 450
- Microsoft-Decision-1: five options, run 412 · “I don't know”, run 451
- Elephant-1: five options, run 499 · “I don't know”, run 501
- Celeris-1 Decision: five options, run 546 · “I don't know”, run 548
- GPT-6 Luna (Decisions API): five options, run 397 · “I don't know”, run 444
- Perplexity Decider Flash v1.2: five options, run 533 · “I don't know”, run 535
Answers at 95%+ confidence
An answer counts when its chosen option has probability at least the cutoff; an answer without a score or without a valid choice never counts; wrong is a counted answer that is not right. Each decision model is counted on its Text tab run (its first five-option Turkish run, the run the site already uses), on the Text tab's 2,198 questions, keeping the answers whose chosen option has a probability of at least 95%. These runs have no “I don't know” option.
Every decision setup
Ordered by % wrong (wrong / answers kept), then by answers kept, more first; a setup with no answer kept is last, Not measured.
| Model | Run | Answers kept (at 95%+) | Wrong | % wrong |
|---|---|---|---|---|
| Liquid d1-ultra1 question per request | 199 | 549 | 0 | 0.0%0 of 549 |
| Fastino GLiDE1 question per request | 129 | 302 | 0 | 0.0%0 of 302 |
| Perplexity Decider 27B v11 question per request | 44 | 293 | 0 | 0.0%0 of 293 |
| Drex 2.01 question per request | 334 | 180 | 0 | 0.0%0 of 180 |
| OneJev 27B1 question per request | 264 | 166 | 0 | 0.0%0 of 166 |
| Celeris-1 Decision1 question per request | 150 | 123 | 0 | 0.0%0 of 123 |
| Liquid d1-2B (open weights)1 question per request | 340 | 4 | 0 | 0.0%0 of 4 |
| meraGPT Decider 11 question per request | 243 | 1 | 0 | 0.0%0 of 1 |
| SubQ Snap Preview1 question per request | 177 | 339 | 1 | 0.3%1 of 339 |
| Fastino GLiDE v1.2local · 1 question per request | 248 | 485 | 2 | 0.4%2 of 485 |
| Liquid d1.11 question per request | 254 | 336 | 2 | 0.6%2 of 336 |
| Drex Max 1.11 question per request | 326 | 493 | 3 | 0.6%3 of 493 |
| Perplexity Decider Flash v1.2local · 1 question per request | 223 | 154 | 1 | 0.6%1 of 154 |
| vLLM-SR d31 question per request | 259 | 503 | 4 | 0.8%4 of 503 |
| Perplexity Decider 27B v1.11 question per request | 145 | 589 | 6 | 1.0%6 of 589 |
| Jev 1.14.01 question per request | 194 | 390 | 4 | 1.0%4 of 390 |
| Cygnet1 question per request | 127 | 335 | 4 | 1.2%4 of 335 |
| Mammoth-11 question per request | 374 | 667 | 9 | 1.3%9 of 667 |
| Clef1 question per request | 125 | 348 | 5 | 1.4%5 of 348 |
| Perplexity Decider 27B v1.2local · 1 question per request | 217 | 633 | 11 | 1.7%11 of 633 |
| Jev 1.13.01 question per request | 49 | 440 | 8 | 1.8%8 of 440 |
| SubQ Snap 1.11 question per request | 346 | 665 | 18 | 2.7%18 of 665 |
| GPT-6 Luna (Decisions API)Decisions API, beta · 1 question per request | 143 | 542 | 15 | 2.8%15 of 542 |
| Metask-Jev 4B1 question per request | 126 | 32 | 1 | 3.1%1 of 32 |
| Drex Max v11 question per request | 322 | 736 | 24 | 3.3%24 of 736 |
| Elephant-11 question per request | 373 | 393 | 13 | 3.3%13 of 393 |
| vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request | 269 | 833 | 30 | 3.6%30 of 833 |
| Liquid d11 question per request | 52 | 502 | 20 | 4.0%20 of 502 |
| Clef-flash1 question per request | 139 | 56 | 4 | 7.1%4 of 56 |
| Microsoft-Decision-1via OpenRouter · 1 question per request | 205 | 378 | 29 | 7.7%29 of 378 |
| Winnow-12B1 question per request | 128 | 641 | 55 | 8.6%55 of 641 |
| Inception Mercury Decidevia OpenRouter · 1 question per request | 204 | 775 | 67 | 8.6%67 of 775 |
| Strands Decider 2B1 question per request | 141 | 10 | 1 | 10.0%1 of 10 |
| Laya multilingual1 question per request | 46 | 14 | 11 | 78.6%11 of 14 |
| GLiNER2.5-multi-Decide1 question per request | 135 | 0 | 0 | Not measurednone of 0 kept |
| Laya1 question per request | 51 | 0 | 0 | Not measurednone of 0 kept |
| Open-Jev 27B v1.11 question per request | 45 | 0 | 0 | Not measurednone of 0 kept |
Liquid d1-ultra's repeat runs
The same questions and settings, run again; run 199 is the Text tab run above, shown for context.
| Run | Answers kept | Wrong | % wrong |
|---|---|---|---|
| Liquid d1-ultrarun 199, Text tab run | 549 | 0 | 0.0%0 of 549 |
| Liquid d1-ultrarun 200, repeat run | 538 | 0 | 0.0%0 of 538 |
| Liquid d1-ultrarun 201, repeat run | 542 | 0 | 0.0%0 of 542 |
| Liquid d1-ultrarun 202, repeat run | 534 | 0 | 0.0%0 of 534 |
| Liquid d1-ultrarun 203, repeat run | 550 | 0 | 0.0%0 of 550 |
The three models Artificial Analysis ties at 65
The three models its Decision Index v1.0 puts level at 65 (its post), each with its Text tab run and its repeat runs, one row per run. Each column is a cutoff: % wrong of the answers kept at that probability or higher, with wrong of kept below.
| Model and run | 50% | 70% | 80% | 90% | 95% | 99% |
|---|---|---|---|---|---|---|
| Liquid d1-ultrarun 199, Text tab run | 13.1%224 of 1,712 | 7.4%99 of 1,335 | 4.2%46 of 1,094 | 1.0%8 of 802 | 0.0%0 of 549 | 0.0%0 of 90 |
| Liquid d1-ultrarun 200, repeat run | 13.4%229 of 1,705 | 7.0%93 of 1,320 | 3.6%39 of 1,089 | 0.8%6 of 795 | 0.0%0 of 538 | 0.0%0 of 86 |
| Liquid d1-ultrarun 201, repeat run | 14.0%241 of 1,719 | 7.3%96 of 1,313 | 4.3%47 of 1,091 | 1.3%10 of 794 | 0.0%0 of 542 | 0.0%0 of 96 |
| Liquid d1-ultrarun 202, repeat run | 13.3%228 of 1,713 | 6.4%84 of 1,315 | 4.5%49 of 1,088 | 1.0%8 of 796 | 0.0%0 of 534 | 0.0%0 of 91 |
| Liquid d1-ultrarun 203, repeat run | 13.7%233 of 1,706 | 7.3%97 of 1,324 | 4.0%43 of 1,080 | 1.1%9 of 803 | 0.0%0 of 550 | 0.0%0 of 90 |
| SubQ Snap 1.1run 346, Text tab run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| SubQ Snap 1.1run 387, repeat run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| SubQ Snap 1.1run 388, repeat run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| SubQ Snap 1.1run 389, repeat run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| SubQ Snap 1.1run 390, repeat run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| Elephant-1run 373, Text tab run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
| Elephant-1run 375, repeat run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
| Elephant-1run 376, repeat run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
| Elephant-1run 378, repeat run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
| Elephant-1run 379, repeat run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
At every cutoff up to 98% (each whole percent), Liquid d1-ultra's worst run has a lower % wrong than the best run of SubQ Snap 1.1 and of Elephant-1; at 99%, all three are at or near 0 (highest: 0.3%).
In English (the 547 kept reading questions of the Turkish vs English tab), at 95%, Liquid d1-ultra has 1 wrong answer in each of its 5 runs, always question 280; each run keeps 230 to 237 answers (runs 411, 418, 422, 426 and 428).
Info The table, what we found and our method
About this test
Every question of the Text tab gets one more option after its five answers: F, “Bilmiyorum”, Turkish for “I don't know”. It is always the last option, it is never shuffled and it is never correct. The System One instructions get one added sentence: “If you do not know the answer, choose Bilmiyorum.” Everything else is as on the Text tab: the same 2,198 scored questions and answer key, one question per request, and each model's own settings from its text run.
The models: the ten most accurate decision models on the Text tab at results 25290875, plus Base Labs' Elephant-1 and Mammoth-1 and SubQ Snap 1.1 once their text runs are published. Each has one full run; a model whose repeated text runs changed answers gets five.
Download: results.json (every number on this page, under dont_know) and answers.json (each run's answers, under dont_know_runs).
Results
The first two columns: Answered, how often the model chose one of the question's own options (right + wrong, out of all 2,198 questions), and Wrong of answers, the share of those answers that were wrong (wrong / (right + wrong)). The rows are ranked by the third, the non-hallucination rate, highest first: the share of “Bilmiyorum” among the answers that were not right (“Bilmiyorum” / (“Bilmiyorum” + wrong)), so the first model is the one that most often said it did not know instead of answering wrongly. Read the ranking with Answered, Wrong of answers and accuracy: a model that always chose “Bilmiyorum” would rank first with no right answers. Each row shows the model's first run; under Runs, a model with more than one run (Liquid d1-ultra and Fastino GLiDE) shows the range over its runs and how many answers changed between them. Out of all 2,198 questions: accuracy counts the right answers; Answered (the attempt rate) counts the right and wrong answers. Formulas and how to read them: Our method.
| Model | Answered(right + wrong) / questions; higher is better | Wrong of answerswrong / (right + wrong); lower is better | Non-hallucination rate“Bilmiyorum” / (“Bilmiyorum” + wrong); higher is better | Derived non-hallucination ratefrom the 5-option text run; higher is better | Accuracyhigher is better | Mean probability of “Bilmiyorum”higher means more probability on “I don't know”; neither better nor worse on its own | Text accuracy with five optionsthe Text tab's number; higher is better | Runsrange over the runs where more than one |
|---|---|---|---|---|---|---|---|---|
| Liquid d1-ultra1 question per request | 71.9%1,581 of 2,198 answered, 617 “Bilmiyorum” | 13.0%205 of 1,581 | 75.1%617 “Bilmiyorum” of 822 not right | 78.7%cut-off 0.5632: 1,414 right + 167 wrong answered, 617 held back, attempt rate 71.9% | 62.6%1,376 of 2,198 right | 30.7% | 77.1%run 199 | runs 350, 354, 360, 365, 370Answered 71.9% to 74.0%Non-hallucination 71.2% to 75.1%Accuracy 62.6% to 63.5%Mean probability 30.6% to 30.8%518 answers changed between runs |
| Mammoth-11 question per request | 68.6%1,508 of 2,198 answered, 690 “Bilmiyorum” | 20.8%314 of 1,508 | 68.7%690 “Bilmiyorum” of 1,004 not right | 70.9%cut-off 0.5697: 1,225 right + 283 wrong answered, 690 held back, attempt rate 68.6% | 54.3%1,194 of 2,198 right | 24.5% | 67.1%run 374 | run 386 |
| Fastino GLiDE1 question per request | 76.3%1,677 of 2,198 answered, 521 “Bilmiyorum” | 21.2%356 of 1,677 | 59.4%521 “Bilmiyorum” of 877 not right | 58.0%cut-off 0.5206: 1,300 right + 377 wrong answered, 521 held back, attempt rate 76.3% | 60.1%1,321 of 2,198 right | 22.0% | 67.9%run 129 | runs 351, 353, 357, 361, 364Answered 75.5% to 76.3%Non-hallucination 59.4% to 59.9%Accuracy 59.1% to 60.1%Mean probability 21.6% to 22.0%189 answers changed between runs |
| SubQ Snap 1.11 question per request | 80.1%1,761 of 2,198 answered, 437 “Bilmiyorum” | 27.8%490 of 1,761 | 47.1%437 “Bilmiyorum” of 927 not right | 47.6%cut-off 0.5124: 1,280 right + 481 wrong answered, 437 held back, attempt rate 80.1% | 57.8%1,271 of 2,198 right | 20.1% | 63.9%run 346 | run 392 |
| Fastino GLiDE v1.2local · 1 question per request | 85.4%1,877 of 2,198 answered, 321 “Bilmiyorum” | 26.2%491 of 1,877 | 39.5%321 “Bilmiyorum” of 812 not right | 40.4%cut-off 0.4134: 1,404 right + 473 wrong answered, 321 held back, attempt rate 85.4% | 63.1%1,386 of 2,198 right | 18.6% | 68.5%run 248 | run 369 |
| Drex Max 1.11 question per request | 85.4%1,878 of 2,198 answered, 320 “Bilmiyorum” | 27.5%517 of 1,878 | 38.2%320 “Bilmiyorum” of 837 not right | 39.3%cut-off 0.3806: 1,383 right + 495 wrong answered, 320 held back, attempt rate 85.4% | 61.9%1,361 of 2,198 right | 17.0% | 67.4%run 326 | run 352 |
| Elephant-11 question per request | 81.6%1,793 of 2,198 answered, 405 “Bilmiyorum” | 42.3%758 of 1,793 | 34.8%405 “Bilmiyorum” of 1,163 not right | 35.5%cut-off 0.3883: 1,058 right + 735 wrong answered, 405 held back, attempt rate 81.6% | 47.1%1,035 of 2,198 right | 16.0% | 52.4%run 373 | run 382 |
| vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request | 99.5%2,188 of 2,198 answered, 10 “Bilmiyorum” | 32.2%704 of 2,188 | 1.4%10 “Bilmiyorum” of 714 not right | 1.4%cut-off 0.2876: 1,486 right + 702 wrong answered, 10 held back, attempt rate 99.5% | 67.5%1,484 of 2,198 right | 2.5% | 67.8%run 269 | run 363 |
| Drex Max v11 question per request | 99.9%2,196 of 2,198 answered, 2 “Bilmiyorum” | 32.4%711 of 2,196 | 0.3%2 “Bilmiyorum” of 713 not right | 0.3%cut-off 0.2357: 1,489 right + 707 wrong answered, 2 held back, attempt rate 99.9% | 67.6%1,485 of 2,198 right | 1.5% | 67.7%run 322 | run 367 |
| Perplexity Decider 27B v1.2local · 1 question per request | 99.9%2,196 of 2,198 answered, 2 “Bilmiyorum” | 32.7%717 of 2,196 | 0.3%2 “Bilmiyorum” of 719 not right | 0.3%cut-off 0.2205: 1,474 right + 722 wrong answered, 2 held back, attempt rate 99.9% | 67.3%1,479 of 2,198 right | 2.4% | 67.1%run 217 | run 372 |
| vLLM-SR d31 question per request | 100.0%2,198 of 2,198 answered, 0 “Bilmiyorum” | 32.5%715 of 2,198 | 0.0%0 “Bilmiyorum” of 715 not right | 0.0%cut-off 0.2225: 1,490 right + 708 wrong answered, 0 held back, attempt rate 100.0% | 67.5%1,483 of 2,198 right | 2.9% | 67.8%run 259 | run 358 |
Not run
- Perplexity Decider 27B v1.1: Perplexity's API now answers requests for v1.1 with v1.2, so a new run could not measure v1.1, and no Perplexity API key is available.
- SubQ Snap Preview: SubQ's API returns its probabilities in 32-bit precision, so their sums miss our 7-decimal check, and we decided not to run it with an allowance for that.
What we found
Of the 11 models, 7 used “Bilmiyorum”, on 14.6% to 31.4% of the questions. The other 4, vLLM-SR d3-ultra, Drex Max v1, Perplexity Decider 27B v1.2 and vLLM-SR d3, chose it on 0 to 10 questions, although each gave it some probability: a mean of 1.5% to 2.9%, against 16.0% to 30.7% for the 7.
Liquid d1-ultra had the highest non-hallucination rate, 75.1%: of its 822 answers that were not right, 617 were “Bilmiyorum”. Mammoth-1 (68.7%) and Fastino GLiDE (59.4%) follow. With the option, the 7 were less accurate than in their five-option runs on the Text tab, by 5.3 points (Elephant-1) to 14.5 points (Liquid d1-ultra: 77.1% against 62.6%). The 4 that almost never chose it moved by -0.3 to +0.2 points.
For 6 of the 7, the model's own confidence on its five-option run, holding back on as many questions, left fewer wrong answers than “Bilmiyorum” did: 9 to 38 fewer, a non-hallucination rate 0.5 to 3.6 points higher (the derived rate in the table). Fastino GLiDE was the exception, with 21 more. For the 4 that almost never chose it, both rates are 1.4% or less. In the exact McNemar test on each model's first run, with Holm's correction over the 7, confidence left significantly fewer wrong answers for Liquid d1-ultra and Mammoth-1 (adjusted p = 0.016 each); for the other 5, the difference could be chance.
With “I don't know” in place of “Bilmiyorum”, we compared each model's first run with “I don't know” against its five-option run held back by its own confidence at the same attempt rate (each cut-off reached the option run's answered count exactly for the 7 that chose “I don't know” on at least 1% of the questions). Confidence left fewer wrong answers for all 7, 9 to 49 fewer. In the exact McNemar test of wrong answers, with Holm's correction over the 7, confidence left significantly fewer for Liquid d1-ultra (216 wrong with “I don't know”, 167 by confidence; adjusted p = 0.00046); for the other 6, the difference could be chance. Fastino GLiDE is compared with its new five-option run 517, not run 129.
Held to Liquid d1-ultra's number of answers with “I don't know” (k = 1,582 in Turkish and 496 in English), each model keeping its k most confident five-option answers, Liquid d1-ultra's five-option run got 10.6% of its k answers wrong in Turkish and 5.8% in English; by a paired bootstrap with Holm's correction, in Turkish, all 36 other decision models had significantly more wrong answers, while of the 37 chat and general models, 20 had significantly fewer, 15 significantly more and 2 could be chance; in English, of the 16 other models, 6 had significantly more and 10 could be chance (The same number of answers).
Our method
We measure hallucination by offering a way out. Every question gets “Bilmiyorum” (“I don't know”) as a sixth option, and the instructions tell the model to choose it when it does not know the answer. A hallucination is then a wrong answer: the model chose one of the question's own options, not the key, when it could have said it did not know. Everything else is the Text tab's: the questions, the answer key and each model's settings.
How it differs from our confidence method
On the Text tab a model must pick one of the five options, and we judge each answer by its confidence: the probability the model gives the option it picked (its option score). Our confidence method keeps an answer when that probability reaches a cut-off, and a stronger model answers the rest (How we chose which answers to keep). Here no cut-off decides: the model can say itself that it does not know, by choosing “Bilmiyorum”, and the rates count those choices. The two meet in the derived non-hallucination rate, which applies the confidence method to the same model's five-option run at about the same attempt rate; side by side, the two non-hallucination columns compare a model that says it does not know with the same model held back by its own confidence.
The rates
Each answer is right, wrong, “Bilmiyorum” or invalid; an invalid answer counts as none of the others. Out of all 2,198 questions:
- Accuracy = right / 2,198.
- Answered (the attempt rate) = (right + wrong) / 2,198: how often the model chose one of the question's own options.
- Wrong of answers = wrong / (right + wrong): among the answers the model gave, the share that were wrong. With no answer it is undefined (Not measured).
- Non-hallucination rate = “Bilmiyorum” / (“Bilmiyorum” + wrong): among the answers that were wrong or “Bilmiyorum”, the share where the model said it did not know. The hallucination rate is its complement, wrong / (“Bilmiyorum” + wrong).
- Mean probability of “Bilmiyorum”: the probability the model gave that option, averaged over its answers that report option probabilities.
How to read them
The non-hallucination rate says how a model is wrong, not how often it is right, and it is not a direct measure of what the model does not know. Read it with accuracy and the attempt rate: a model that always chose “Bilmiyorum” would reach 100% with no right answers, and one that never chose it would reach 0% as soon as it answered one question wrongly. With neither a wrong answer nor “Bilmiyorum”, there is nothing to divide by and the rate is undefined (the table shows Not measured).
The option and the instructions
The added option's exact text is “Bilmiyorum”. It is lettered after the question's own options, and the answer key is unchanged. The full System One instructions:
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose Bilmiyorum.
Prompt version 1cdd542b597f (a hash of the prompts with these instructions and this option, computed as for the Text tab's prompt version).
Derived non-hallucination rate
For comparison, each model's run with five options (its run on the Text tab, on the same questions) is made to abstain by its own confidence: it answers only where its score reaches a cut-off and abstains on the rest. The cut-off is the one whose attempt rate comes nearest the attempt rate of the run with “Bilmiyorum”; on a tie, the higher cut-off. An answer without a score never counts as an attempt. The derived non-hallucination rate is then the abstentions over the abstentions plus the wrong answers. The table shows the cut-off and the attempt rate it reached.
Repeats
One full run per model. A model whose repeated text runs changed answers gets five runs with the option. Those runs form a repeat set of their own and are never mixed with the Text tab's runs. The table shows the first run; the range covers all of them, and the changed answers are the scored questions whose answer differs between the runs.
Results on the English questions with “I don't know”
Each model's first run in English with “I don't know” and in Turkish with “Bilmiyorum”, on the 547 kept reading questions: the counts, the rates and the exact McNemar test of wrong answers, with p adjusted by Holm's method over the 13 models; * marks a Holm-adjusted p below 0.05. The same table is on the Turkish vs English tab.
| Model | English runswith “I don't know”; the first is compared | Turkish runswith “Bilmiyorum”; the first is compared | English answered(right + wrong) / 547; higher is better | Turkish answered(right + wrong) / 547; higher is better | English wrong of answerswrong / (right + wrong); lower is better | Turkish wrong of answerswrong / (right + wrong); lower is better | English non-hallucination“I don't know” / (“I don't know” + wrong); higher is better, read with the answered share | Turkish non-hallucination“Bilmiyorum” / (“Bilmiyorum” + wrong); higher is better, read with the answered share | English accuracyhigher is better | Turkish accuracyhigher is better | English mean probabilityof “I don't know” | Turkish mean probabilityof “Bilmiyorum” | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 13 models |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Celeris-1 Decision1 question per request | run 548a new run | Not measured | 83.2%455 of 547 answered, 92 “I don't know” | Not measured | 8.1%37 of 455 | Not measured | 71.3%92 “I don't know” of 129 not right | Not measured | 76.4%418 of 547 right | Not measured | 12.7% | Not measured | Not measured | Not measured | Not measured | Not measured |
| Liquid d1-ultra1 question per request | runs 449, 456, 462, 466, 474new runs | runs 350, 354, 360, 365, 370the Hallucination tab's full runs, counted on the kept questions | 90.7%496 of 547 answered, 51 “I don't know”range 89.6% to 91.4% | 94.5%517 of 547 answered, 30 “Bilmiyorum”range 94.1% to 95.6% | 7.9%39 of 496 | 5.0%26 of 517 | 56.7%51 “I don't know” of 90 not rightrange 55.3% to 61.3% | 53.6%30 “Bilmiyorum” of 56 not rightrange 44.4% to 59.3% | 83.5%457 of 547 rightrange 82.6% to 84.5% | 89.8%491 of 547 rightrange 89.2% to 90.1% | 25.3%range 25.3% to 25.6% | 22.8%range 22.4% to 23.0% | 20 | 7 | 0.019 | 0.21 |
| Drex Max 1.11 question per request | run 447a new run | run 352the Hallucination tab's full runs, counted on the kept questions | 92.1%504 of 547 answered, 43 “I don't know” | 98.4%538 of 547 answered, 9 “Bilmiyorum” | 8.1%41 of 504 | 9.5%51 of 538 | 51.2%43 “I don't know” of 84 not right | 15.0%9 “Bilmiyorum” of 60 not right | 84.6%463 of 547 right | 89.0%487 of 547 right | 14.3% | 8.4% | 20 | 30 | 0.2 | 1 |
| Mammoth-11 question per request | run 505a new run | run 386the Hallucination tab's full runs, counted on the kept questions | 93.2%510 of 547 answered, 37 “I don't know” | 95.6%523 of 547 answered, 24 “Bilmiyorum” | 7.1%36 of 510 | 6.3%33 of 523 | 50.7%37 “I don't know” of 73 not right | 42.1%24 “Bilmiyorum” of 57 not right | 86.7%474 of 547 right | 89.6%490 of 547 right | 9.8% | 7.4% | 18 | 15 | 0.73 | 1 |
| Fastino GLiDE v1.2local · 1 question per request | run 523a new run | run 369the Hallucination tab's full runs, counted on the kept questions | 94.5%517 of 547 answered, 30 “I don't know” | 98.9%541 of 547 answered, 6 “Bilmiyorum” | 7.0%36 of 517 | 8.7%47 of 541 | 45.5%30 “I don't know” of 66 not right | 11.3%6 “Bilmiyorum” of 53 not right | 87.9%481 of 547 right | 90.3%494 of 547 right | 12.5% | 8.7% | 16 | 27 | 0.13 | 0.88 |
| Elephant-11 question per request | run 501a new run | run 382the Hallucination tab's full runs, counted on the kept questions | 95.8%524 of 547 answered, 23 “I don't know” | 97.3%532 of 547 answered, 15 “Bilmiyorum” | 12.8%67 of 524 | 15.4%82 of 532 | 25.6%23 “I don't know” of 90 not right | 15.5%15 “Bilmiyorum” of 97 not right | 83.5%457 of 547 right | 82.3%450 of 547 right | 6.8% | 5.7% | 31 | 46 | 0.11 | 0.88 |
| SubQ Snap 1.11 question per request | run 448a new run | run 392the Hallucination tab's full runs, counted on the kept questions | 96.2%526 of 547 answered, 21 “I don't know” | 96.5%528 of 547 answered, 19 “Bilmiyorum” | 12.0%63 of 526 | 9.7%51 of 528 | 25.0%21 “I don't know” of 84 not right | 27.1%19 “Bilmiyorum” of 70 not right | 84.6%463 of 547 right | 87.2%477 of 547 right | 11.7% | 8.4% | 39 | 27 | 0.18 | 1 |
| Inception Mercury Decidevia OpenRouter · 1 question per request | runs 450, 460, 469, 478, 484new runs | runs 489, 491, 493, 495, 497new runs on the kept questions | 98.0%536 of 547 answered, 11 “I don't know”range 98.0% to 98.4% | 98.5%539 of 547 answered, 8 “Bilmiyorum”the same in every run | 13.2%71 of 536 | 13.2%71 of 539 | 13.4%11 “I don't know” of 82 not rightrange 11.1% to 13.4% | 10.1%8 “Bilmiyorum” of 79 not rightrange 10.1% to 10.5% | 85.0%465 of 547 rightrange 85.0% to 85.4% | 85.6%468 of 547 rightrange 85.6% to 86.1% | 3.2%range 3.1% to 3.2% | 1.9%range 1.9% to 2.0% | 29 | 29 | 1 | 1 |
| GPT-6 Luna (Decisions API)Decisions API, beta · 1 question per request | run 444a new run | run 453a new run on the kept questions | 98.2%537 of 547 answered, 10 “I don't know” | 98.9%541 of 547 answered, 6 “Bilmiyorum” | 17.9%96 of 537 | 14.4%78 of 541 | 9.4%10 “I don't know” of 106 not right | 7.1%6 “Bilmiyorum” of 84 not right | 80.6%441 of 547 right | 84.6%463 of 547 right | 3.0% | 2.7% | 47 | 29 | 0.05 | 0.49 |
| Perplexity Decider Flash v1.2local · 1 question per request | run 535a new run | run 537a new run on the kept questions | 99.1%542 of 547 answered, 5 “I don't know” | 98.9%541 of 547 answered, 6 “Bilmiyorum” | 18.5%100 of 542 | 24.6%133 of 541 | 4.8%5 “I don't know” of 105 not right | 4.3%6 “Bilmiyorum” of 139 not right | 80.8%442 of 547 right | 74.6%408 of 547 right | 5.5% | 6.3% | 36 | 69 | 0.0017 | 0.022 * |
| Jev 1.14.01 question per request | runs 446, 454, 457, 461, 464new runs | runs 470, 473, 476, 480, 482new runs on the kept questions | 99.5%544 of 547 answered, 3 “I don't know”range 99.3% to 99.5% | 99.5%544 of 547 answered, 3 “Bilmiyorum”the same in every run | 13.2%72 of 544 | 12.7%69 of 544 | 4.0%3 “I don't know” of 75 not rightrange 4.0% to 5.2% | 4.2%3 “Bilmiyorum” of 72 not rightrange 4.0% to 4.5% | 86.3%472 of 547 rightrange 85.9% to 86.7% | 86.8%475 of 547 rightrange 86.3% to 87.8% | 1.6%range 1.6% to 1.6% | 1.7%range 1.6% to 1.7% | 20 | 17 | 0.74 | 1 |
| Microsoft-Decision-1via OpenRouter · 1 question per request | runs 451, 459, 467, 477, 483new runs | runs 487, 490, 492, 494, 496new runs on the kept questions | 99.5%544 of 547 answered, 3 “I don't know”the same in every run | 99.6%545 of 547 answered, 2 “Bilmiyorum”range 99.3% to 99.6% | 14.2%77 of 544 | 16.1%88 of 545 | 3.8%3 “I don't know” of 80 not rightrange 3.8% to 4.0% | 2.2%2 “Bilmiyorum” of 90 not rightrange 2.2% to 4.3% | 85.4%467 of 547 rightrange 85.4% to 86.3% | 83.5%457 of 547 rightrange 82.8% to 83.5% | 3.6%range 3.6% to 3.6% | 2.3%range 2.3% to 2.3% | 24 | 35 | 0.19 | 1 |
| Perplexity Decider 27B v1.2local · 1 question per request | run 529a new run | run 372the Hallucination tab's full runs, counted on the kept questions | 99.6%545 of 547 answered, 2 “I don't know” | 100.0%547 of 547 answered, 0 “Bilmiyorum” | 11.7%64 of 545 | 9.1%50 of 547 | 3.0%2 “I don't know” of 66 not right | 0.0%0 “Bilmiyorum” of 50 not right | 87.9%481 of 547 right | 90.9%497 of 547 right | 2.8% | 0.9% | 29 | 15 | 0.049 | 0.49 |
| Liquid d11 question per request | runs 445, 455, 458, 463, 465new runs | runs 472, 475, 479, 481, 485new runs on the kept questions | 99.6%545 of 547 answered, 2 “I don't know”the same in every run | 99.8%546 of 547 answered, 1 “Bilmiyorum”range 99.8% to 100.0% | 12.3%67 of 545 | 16.3%89 of 546 | 2.9%2 “I don't know” of 69 not rightrange 2.6% to 2.9% | 1.1%1 “Bilmiyorum” of 90 not rightrange 0.0% to 1.1% | 87.4%478 of 547 rightrange 85.9% to 87.4% | 83.5%457 of 547 rightrange 83.0% to 85.0% | 0.9%range 0.9% to 1.0% | 1.3%range 1.1% to 1.3% | 21 | 43 | 0.0081 | 0.098 |
Results with “I don't know” instead of “Bilmiyorum”
In the tab's order. Each row shows the model's first run with each word on the same 2,198 questions; the “Bilmiyorum” run is the one in the table above. Answered = (right + wrong) / 2,198 and Wrong of answers = wrong / (right + wrong) come first, then the non-hallucination rate (the option / (the option + wrong)) and accuracy, each with “I don't know” and with “Bilmiyorum”. Wrong only with one word: the questions only one of the two runs got wrong, by the exact McNemar test, with p adjusted by Holm's method over the models compared; * marks a Holm-adjusted p below 0.05.
| Model | Answered with “I don't know”(right + wrong) / questions; higher is better | Answered with “Bilmiyorum”(right + wrong) / questions; higher is better | Wrong of answers with “I don't know”wrong / (right + wrong); lower is better | Wrong of answers with “Bilmiyorum”wrong / (right + wrong); lower is better | Non-hallucination rate with “I don't know”“I don't know” / (“I don't know” + wrong); higher is better | Non-hallucination rate with “Bilmiyorum”“Bilmiyorum” / (“Bilmiyorum” + wrong); higher is better | Accuracy with “I don't know”higher is better | Accuracy with “Bilmiyorum”higher is better | Wrong only with one wordexact McNemar, Holm-adjusted | Runsrun, start date and prompt version |
|---|---|---|---|---|---|---|---|---|---|---|
| Liquid d1-ultra1 question per request | 72.0%1,582 of 2,198 answered, 616 “I don't know” | 71.9%1,581 of 2,198 answered, 617 “Bilmiyorum” | 13.7%216 of 1,582 | 13.0%205 of 1,581 | 74.0%616 “I don't know” of 832 not right | 75.1%617 “Bilmiyorum” of 822 not right | 62.1%1,366 of 2,198 right | 62.6%1,376 of 2,198 right | 66 · 5566 wrong only with “I don't know”, 55 only with “Bilmiyorum”p = 0.36, Holm-adjusted p = 1 | 511 · 350“I don't know”: run 511, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 350, 2026-11-01, prompt version 1cdd542b597f |
| Mammoth-11 question per request | 65.7%1,445 of 2,198 answered, 753 “I don't know” | 68.6%1,508 of 2,198 answered, 690 “Bilmiyorum” | 19.1%276 of 1,445 | 20.8%314 of 1,508 | 73.2%753 “I don't know” of 1,029 not right | 68.7%690 “Bilmiyorum” of 1,004 not right | 53.2%1,169 of 2,198 right | 54.3%1,194 of 2,198 right | 9 · 479 wrong only with “I don't know”, 47 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 * | 519 · 386“I don't know”: run 519, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 386, 2026-11-01, prompt version 1cdd542b597f |
| Fastino GLiDE1 question per request | 73.6%1,617 of 2,198 answered, 581 “I don't know” | 76.3%1,677 of 2,198 answered, 521 “Bilmiyorum” | 20.7%335 of 1,617 | 21.2%356 of 1,677 | 63.4%581 “I don't know” of 916 not right | 59.4%521 “Bilmiyorum” of 877 not right | 58.3%1,282 of 2,198 right | 60.1%1,321 of 2,198 right | 52 · 7352 wrong only with “I don't know”, 73 only with “Bilmiyorum”p = 0.073, Holm-adjusted p = 0.44 | 510 · 351“I don't know”: run 510, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 351, 2026-11-01, prompt version 1cdd542b597f |
| SubQ Snap 1.11 question per request | 77.3%1,700 of 2,198 answered, 498 “I don't know” | 80.1%1,761 of 2,198 answered, 437 “Bilmiyorum” | 26.7%454 of 1,700 | 27.8%490 of 1,761 | 52.3%498 “I don't know” of 952 not right | 47.1%437 “Bilmiyorum” of 927 not right | 56.7%1,246 of 2,198 right | 57.8%1,271 of 2,198 right | 108 · 144108 wrong only with “I don't know”, 144 only with “Bilmiyorum”p = 0.027, Holm-adjusted p = 0.19 | 512 · 392“I don't know”: run 512, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 392, 2026-11-01, prompt version 1cdd542b597f |
| Fastino GLiDE v1.2local · 1 question per request | 80.6%1,772 of 2,198 answered, 426 “I don't know” | 85.4%1,877 of 2,198 answered, 321 “Bilmiyorum” | 24.1%427 of 1,772 | 26.2%491 of 1,877 | 49.9%426 “I don't know” of 853 not right | 39.5%321 “Bilmiyorum” of 812 not right | 61.2%1,345 of 2,198 right | 63.1%1,386 of 2,198 right | 19 · 8319 wrong only with “I don't know”, 83 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 * | 525 · 369“I don't know”: run 525, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 369, 2026-11-01, prompt version 1cdd542b597f |
| Drex Max 1.11 question per request | 77.9%1,712 of 2,198 answered, 486 “I don't know” | 85.4%1,878 of 2,198 answered, 320 “Bilmiyorum” | 24.1%413 of 1,712 | 27.5%517 of 1,878 | 54.1%486 “I don't know” of 899 not right | 38.2%320 “Bilmiyorum” of 837 not right | 59.1%1,299 of 2,198 right | 61.9%1,361 of 2,198 right | 40 · 14440 wrong only with “I don't know”, 144 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 * | 513 · 352“I don't know”: run 513, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 352, 2026-11-01, prompt version 1cdd542b597f |
| Elephant-11 question per request | 72.5%1,594 of 2,198 answered, 604 “I don't know” | 81.6%1,793 of 2,198 answered, 405 “Bilmiyorum” | 39.0%622 of 1,594 | 42.3%758 of 1,793 | 49.3%604 “I don't know” of 1,226 not right | 34.8%405 “Bilmiyorum” of 1,163 not right | 44.2%972 of 2,198 right | 47.1%1,035 of 2,198 right | 18 · 15418 wrong only with “I don't know”, 154 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 * | 515 · 382“I don't know”: run 515, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 382, 2026-11-01, prompt version 1cdd542b597f |
| vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request | 99.6%2,189 of 2,198 answered, 9 “I don't know” | 99.5%2,188 of 2,198 answered, 10 “Bilmiyorum” | 32.2%704 of 2,189 | 32.2%704 of 2,188 | 1.3%9 “I don't know” of 713 not right | 1.4%10 “Bilmiyorum” of 714 not right | 67.6%1,485 of 2,198 right | 67.5%1,484 of 2,198 right | 39 · 3939 wrong only with “I don't know”, 39 only with “Bilmiyorum”p = 1, Holm-adjusted p = 1 | 540 · 363“I don't know”: run 540, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 363, 2026-11-01, prompt version 1cdd542b597f |
| Drex Max v11 question per request | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 99.9%2,196 of 2,198 answered, 2 “Bilmiyorum” | 32.4%712 of 2,196 | 32.4%711 of 2,196 | 0.3%2 “I don't know” of 714 not right | 0.3%2 “Bilmiyorum” of 713 not right | 67.5%1,484 of 2,198 right | 67.6%1,485 of 2,198 right | 29 · 2829 wrong only with “I don't know”, 28 only with “Bilmiyorum”p = 1, Holm-adjusted p = 1 | 542 · 367“I don't know”: run 542, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 367, 2026-11-01, prompt version 1cdd542b597f |
| Perplexity Decider 27B v1.2local · 1 question per request | 99.8%2,194 of 2,198 answered, 4 “I don't know” | 99.9%2,196 of 2,198 answered, 2 “Bilmiyorum” | 33.2%728 of 2,194 | 32.7%717 of 2,196 | 0.5%4 “I don't know” of 732 not right | 0.3%2 “Bilmiyorum” of 719 not right | 66.7%1,466 of 2,198 right | 67.3%1,479 of 2,198 right | 40 · 2940 wrong only with “I don't know”, 29 only with “Bilmiyorum”p = 0.23, Holm-adjusted p = 1 | 531 · 372“I don't know”: run 531, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 372, 2026-11-01, prompt version 1cdd542b597f |
| vLLM-SR d31 question per request | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 100.0%2,198 of 2,198 answered, 0 “Bilmiyorum” | 32.5%714 of 2,196 | 32.5%715 of 2,198 | 0.3%2 “I don't know” of 716 not right | 0.0%0 “Bilmiyorum” of 715 not right | 67.4%1,482 of 2,198 right | 67.5%1,483 of 2,198 right | 21 · 2221 wrong only with “I don't know”, 22 only with “Bilmiyorum”p = 1, Holm-adjusted p = 1 | 544 · 358“I don't know”: run 544, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 358, 2026-11-01, prompt version 1cdd542b597f |
Our method with “I don't know”
Only two things changed from the runs above: the sixth option's word and the sentence of the System One instructions that names it. The option “Bilmiyorum” became “I don't know”, and the sentence “If you do not know the answer, choose Bilmiyorum.” became “If you do not know the answer, choose I don't know.” Everything else is as above: the same 2,198 Turkish questions and answer key, the option always last and never correct, one question per request and each model's own settings. The full instructions:
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose I don't know.
Prompt version fb17d583a3af with “I don't know”, against 1cdd542b597f with “Bilmiyorum”. One full run per model with “I don't know”; the table gives each run's ID and start date (UTC).
How to read the comparison
A model that did not recognise the Turkish word and so ignored the option should choose the English one more often, and its non-hallucination rate should rise. A model that chooses both about as often did not skip “Bilmiyorum” for its word. The rates are those above, and the same limits hold: the non-hallucination rate says how a model is wrong, not how often it is right, and it is not a direct measure of what the model does not know; read it with accuracy and the attempt rate, as a model that always chose the option would reach 100% with no right answers. With neither a wrong answer nor the option, the rate is undefined (Not measured).
Results: “I don't know” against confidence
In the tab's order. Each row: the model's first run with “I don't know” and its five-option run held back by its own confidence on the same 2,198 questions. Answered = right + wrong; coverage = answered / 2,198. Wrong only on one side: the questions only one of the two got wrong, by the exact McNemar test, with p adjusted by Holm's method over the 7 models that chose “I don't know” on at least 1% of the questions; * marks a Holm-adjusted p below 0.05. Not in the family: vLLM-SR d3-ultra, Drex Max v1, Perplexity Decider 27B v1.2 and vLLM-SR d3.
| Model | Runswith “I don't know” · five options | Cut-offand its neighbouring candidates | Answeredtarget with the option · achieved by confidence (coverage) | Wrongwith “I don't know” · by confidence; lower is better | Wrong only on one sidewith “I don't know” · by confidence | pexact McNemar | Holm padjusted over the family |
|---|---|---|---|---|---|---|---|
| Liquid d1-ultra1 question per request | 511 · 199 | 0.5630next lower 0.5622: 1,583 answerednext higher 0.5632: 1,581 answered | 1,582 · 1,582target 1,582 (72.0%), 616 “I don't know”achieved 1,582 (72.0%), 616 held back | 216 · 167 | 98 · 49 | < 0.0001 | 0.00046 * |
| Mammoth-11 question per request | 519 · 374 | 0.5951next lower 0.5948: 1,446 answerednext higher 0.5956: 1,444 answered | 1,445 · 1,445target 1,445 (65.7%), 753 “I don't know”achieved 1,445 (65.7%), 753 held back | 276 · 250 | 64 · 38 | 0.013 | 0.078 |
| Fastino GLiDE1 question per request | 510 · 517run 517 in place of the Text tab's run 129 | 0.5222next lower 0.5220: 1,618 answerednext higher 0.5231: 1,616 answered | 1,617 · 1,617target 1,617 (73.6%), 581 “I don't know”achieved 1,617 (73.6%), 581 held back | 335 · 319 | 79 · 63 | 0.21 | 0.62 |
| SubQ Snap 1.11 question per request | 512 · 346 | 0.5372next lower 0.5371: 1,701 answerednext higher 0.5380: 1,699 answered | 1,700 · 1,700target 1,700 (77.3%), 498 “I don't know”achieved 1,700 (77.3%), 498 held back | 454 · 445 | 141 · 132 | 0.63 | 0.63 |
| Fastino GLiDE v1.2local · 1 question per request | 525 · 248 | 0.4535next lower 0.4534: 1,773 answerednext higher 0.4536: 1,771 answered | 1,772 · 1,772target 1,772 (80.6%), 426 “I don't know”achieved 1,772 (80.6%), 426 held back | 427 · 403 | 77 · 53 | 0.043 | 0.22 |
| Drex Max 1.11 question per request | 513 · 326 | 0.4478next lower 0.4470: 1,713 answerednext higher 0.4479: 1,711 answered | 1,712 · 1,712target 1,712 (77.9%), 486 “I don't know”achieved 1,712 (77.9%), 486 held back | 413 · 396 | 106 · 89 | 0.25 | 0.62 |
| Elephant-11 question per request | 515 · 373 | 0.4376next lower 0.4374: 1,595 answerednext higher 0.4378: 1,593 answered | 1,594 · 1,594target 1,594 (72.5%), 604 “I don't know”achieved 1,594 (72.5%), 604 held back | 622 · 594 | 120 · 92 | 0.063 | 0.25 |
| vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request | 540 · 269 | 0.2839next lower 0.2830: 2,190 answerednext higher 0.2876: 2,188 answered | 2,189 · 2,189target 2,189 (99.6%), 9 “I don't know”achieved 2,189 (99.6%), 9 held back | 704 · 703 | 54 · 53 | 1 | not in the family |
| Drex Max v11 question per request | 542 · 322 | 0.2357next lower 0.2348: 2,197 answerednext higher 0.2375: 2,195 answered | 2,196 · 2,196target 2,196 (99.9%), 2 “I don't know”achieved 2,196 (99.9%), 2 held back | 712 · 707 | 49 · 44 | 0.68 | not in the family |
| Perplexity Decider 27B v1.2local · 1 question per request | 531 · 217 | 0.2217next lower 0.2205: 2,195 answerednext higher 0.2223: 2,193 answered | 2,194 · 2,194target 2,194 (99.8%), 4 “I don't know”achieved 2,194 (99.8%), 4 held back | 728 · 720 | 52 · 44 | 0.48 | not in the family |
| vLLM-SR d31 question per request | 544 · 259 | 0.2331next lower 0.2294: 2,197 answerednext higher 0.2352: 2,195 answered | 2,196 · 2,196target 2,196 (99.9%), 2 “I don't know”achieved 2,196 (99.9%), 2 held back | 714 · 706 | 39 · 31 | 0.4 | not in the family |
Our method for “I don't know” against confidence
The derived method above, applied to the runs with “I don't know”: each model's five-option run answers only where its score reaches a cut-off and abstains on the rest. The candidate cut-offs are the run's distinct scores; the one chosen is the one whose attempt rate comes nearest the attempt rate of the run with “I don't know”, and on a tie the higher cut-off. An answer without a score never counts as an attempt. Answers with equal scores enter together, so the achieved answered count can differ from the target; the table shows the chosen cut-off with the next lower and next higher candidates and the answers each would give.
The five-option run is each model's run on the Text tab, as in the table above. For Fastino GLiDE, the five-option run is 517, not the Text tab's run 129: Fastino changed its hosted model after 6 October, so run 129 is a different model from the one behind run 510, and run 517 is a new five-option run of the same hosted model (Our method for the hosted GLiDE rerun).
The paired test is the exact McNemar test of wrong answers, the run with “I don't know” first and confidence second, as for “Bilmiyorum”. Holm's correction runs over the models that chose “I don't know” on at least 1% of the scored questions: Fastino GLiDE v1.2, Liquid d1-ultra, Drex Max 1.1, Fastino GLiDE, SubQ Snap 1.1, Mammoth-1 and Elephant-1. The others, vLLM-SR d3-ultra, Drex Max v1, Perplexity Decider 27B v1.2 and vLLM-SR d3, chose it too rarely for the two sides to differ by more than a few answers; their p is shown and not adjusted.
This is a family of its own, from runs already published: it changes none of the numbers above, and it is separate from the comparisons with Holm over 4, 13 and 11 models (the four models of the request, the English questions and the option's word).
Hosted GLiDE: the comparison redone with run 517
Both against Fastino GLiDE's first run with “Bilmiyorum”, run 351. Run 129 is the Text tab's run and the row in the table above; run 517 is new and replaces it here only.
| Five-option runrun, date and prompt version | Text accuracy with five optionshigher is better | Derived non-hallucination ratehigher is better | Wrong only on one sideexact McNemar |
|---|---|---|---|
| run 1292026-10-06, as published | 67.9%1,492 of 2,198 right | 58.0%cut-off 0.5206: 1,300 right + 377 wrong answered, 521 held back, attempt rate 76.3% | 192 with “Bilmiyorum”, 213 by confidencep = 0.32, Holm-adjusted p = 0.72 |
| run 5172026-11-02, prompt version 254597b8be16 | 68.1%1,497 of 2,198 right | 59.7%cut-off 0.4956: 1,326 right + 351 wrong answered, 521 held back, attempt rate 76.3% | 90 with “Bilmiyorum”, 85 by confidencep = 0.76, not adjusted |
Our method for the hosted GLiDE rerun
Fastino changed the hosted model behind fastino/GLiDE after 6 October. Run 129 was made on 6 October 2026, so it is a different model from the one that answered with “Bilmiyorum” (run 351), and the derived rate in the table above compares two models. Run 517 is a new five-option run of the same hosted model, started 2 November 2026, made as on the Text tab: the same 2,198 questions, one question per request, the main prompt version 254597b8be16. Its derived non-hallucination rate and paired test are computed as above, from run 517 instead of run 129; the p value is not adjusted, as it is one test.
The evidence for “a different model” and “the same model” is the engine each response names: every answer of run 129 names Fastino's engine glide; every answer from 10 October on names glide_v2, including all of run 351 and its repeats and all of run 517. Fastino publishes no version to pin, so that name is the only identity we can check: it cannot show that the weights behind glide_v2 stayed the same between 1 and 2 November.
The row in the table above is left as published, against run 129, and run 517 is kept out of the Text tab: its leaderboard, repeats and every other list.
Results: the same number of answers
One table per language, ordered by wrong of the k answers, fewest first; the Turkish one has every main run of the Text tab (74 setups), grouped as its leaderboard groups them and labelled as it labels them. Each row: the model's five-option run held to its k most confident answers, its wrong answers among them, the difference from Liquid d1-ultra in points with its 95% bootstrap interval, p and Holm p (* marks a Holm-adjusted p below 0.05), and for context the model's own runs with the option (answered = right + wrong, and wrong of answers = wrong / answered), which are not part of the test; a setup without one reads none.
Turkish: k = 1,582 of the 2,198 questions
Each setup's curve over every number of answers: Every setup's curve in Turkish, below.
Decision models
| Model | Five-option runthe run whose k answers are kept | Wrong of the k answerswrong / 1,582; lower is better | Difference from Liquid d1-ultrapoints, with its 95% interval; positive = more wrong answers | ppaired bootstrap, two-sided | Holm padjusted over the 73 | Runs with the optionfor context: answered, wrong of answers |
|---|---|---|---|---|---|---|
| Liquid d1-ultra1 question per request | 199 | 10.6%167 of 1,582 | referenceits own run 511 with “I don't know”: 216 wrong of 1,582 (13.7%), not part of the test | reference | reference | 350 · 511“Bilmiyorum”, run 350: 1,581 answered (71.9%), 13.0% wrong (205)“I don't know”, run 511: 1,582 answered (72.0%), 13.7% wrong (216) |
| Fastino GLiDE v1.2local · 1 question per request | 248 | 18.3%289 of 1,582 | +7.7+5.8 to +9.7 | 0.0002 | 0.015 * | 369 · 525“Bilmiyorum”, run 369: 1,877 answered (85.4%), 26.2% wrong (491)“I don't know”, run 525: 1,772 answered (80.6%), 24.1% wrong (427) |
| vLLM-SR d31 question per request | 259 | 19.8%314 of 1,582 | +9.3+7.3 to +11.3 | 0.0002 | 0.015 * | 358 · 544“Bilmiyorum”, run 358: 2,198 answered (100.0%), 32.5% wrong (715)“I don't know”, run 544: 2,196 answered (99.9%), 32.5% wrong (714) |
| vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request | 269 | 19.9%315 of 1,582 | +9.4+7.2 to +11.3 | 0.0002 | 0.015 * | 363 · 540“Bilmiyorum”, run 363: 2,188 answered (99.5%), 32.2% wrong (704)“I don't know”, run 540: 2,189 answered (99.6%), 32.2% wrong (704) |
| Perplexity Decider 27B v1.11 question per request | 145 | 20.1%318 of 1,582 | +9.5+7.6 to +11.6 | 0.0002 | 0.015 * | noneno run with the option |
| Drex Max 1.11 question per request | 326 | 20.3%321 of 1,582 | +9.7+7.8 to +11.8 | 0.0002 | 0.015 * | 352 · 513“Bilmiyorum”, run 352: 1,878 answered (85.4%), 27.5% wrong (517)“I don't know”, run 513: 1,712 answered (77.9%), 24.1% wrong (413) |
| Drex Max v11 question per request | 322 | 20.3%321 of 1,582 | +9.7+7.6 to +11.7 | 0.0002 | 0.015 * | 367 · 542“Bilmiyorum”, run 367: 2,196 answered (99.9%), 32.4% wrong (711)“I don't know”, run 542: 2,196 answered (99.9%), 32.4% wrong (712) |
| Mammoth-11 question per request | 374 | 20.4%322 of 1,582 | +9.8+7.8 to +11.8 | 0.0002 | 0.015 * | 386 · 519“Bilmiyorum”, run 386: 1,508 answered (68.6%), 20.8% wrong (314)“I don't know”, run 519: 1,445 answered (65.7%), 19.1% wrong (276) |
| Fastino GLiDE1 question per request | 129 | 20.5%325 of 1,582 | +10.0+7.9 to +12.0 | 0.0002 | 0.015 * | 351 · 510“Bilmiyorum”, run 351: 1,677 answered (76.3%), 21.2% wrong (356)“I don't know”, run 510: 1,617 answered (73.6%), 20.7% wrong (335)Made with Fastino's engine glide_v2, not run 129's glide |
| Perplexity Decider 27B v1.2local · 1 question per request | 217 | 20.7%328 of 1,582 | +10.2+8.0 to +12.1 | 0.0002 | 0.015 * | 372 · 531“Bilmiyorum”, run 372: 2,196 answered (99.9%), 32.7% wrong (717)“I don't know”, run 531: 2,194 answered (99.8%), 33.2% wrong (728) |
| SubQ Snap Preview1 question per request | 177 | 22.5%356 of 1,582 | +11.9+9.7 to +13.9 | 0.0002 | 0.015 * | noneno run with the option |
| SubQ Snap 1.11 question per request | 346 | 22.8%361 of 1,582 | +12.3+10.1 to +14.5 | 0.0002 | 0.015 * | 392 · 512“Bilmiyorum”, run 392: 1,761 answered (80.1%), 27.8% wrong (490)“I don't know”, run 512: 1,700 answered (77.3%), 26.7% wrong (454) |
| GPT-6 Luna (Decisions API)Decisions API, beta · 1 question per request | 143 | 23.5%372 of 1,582 | +13.0+10.7 to +15.1 | 0.0002 | 0.015 * | noneno run with the option |
| Perplexity Decider 27B v11 question per request | 44 | 27.2%430 of 1,582 | +16.6+14.5 to +18.9 | 0.0002 | 0.015 * | noneno run with the option |
| Liquid d1.11 question per request | 254 | 27.4%434 of 1,582 | +16.9+14.4 to +19.0 | 0.0002 | 0.015 * | noneno run with the option |
| Celeris-1 Decision1 question per request | 150 | 28.4%450 of 1,582 | +17.9+15.4 to +20.0 | 0.0002 | 0.015 * | noneno run with the option |
| OneJev 27B1 question per request | 264 | 28.8%455 of 1,582 | +18.2+15.9 to +20.4 | 0.0002 | 0.015 * | noneno run with the option |
| Jev 1.14.01 question per request | 194 | 30.7%485 of 1,582 | +20.1+17.9 to +22.4 | 0.0002 | 0.015 * | noneno run with the option |
| Inception Mercury Decidevia OpenRouter · 1 question per request | 204 | 30.8%487 of 1,582 | +20.2+17.8 to +22.6 | 0.0002 | 0.015 * | noneno run with the option |
| Jev 1.13.01 question per request | 49 | 31.5%498 of 1,582 | +20.9+18.5 to +23.2 | 0.0002 | 0.015 * | noneno run with the option |
| Open-Jev 27B v1.11 question per request | 45 | 31.9%504 of 1,582 | +21.3+18.8 to +23.6 | 0.0002 | 0.015 * | noneno run with the option |
| Clef1 question per request | 125 | 32.4%512 of 1,582 | +21.8+19.5 to +24.2 | 0.0002 | 0.015 * | noneno run with the option |
| Liquid d11 question per request | 52 | 32.6%516 of 1,582 | +22.1+19.7 to +24.3 | 0.0002 | 0.015 * | noneno run with the option |
| Drex 2.01 question per request | 334 | 33.4%528 of 1,582 | +22.8+20.4 to +25.0 | 0.0002 | 0.015 * | noneno run with the option |
| Cygnet1 question per request | 127 | 33.5%530 of 1,582 | +22.9+20.4 to +25.3 | 0.0002 | 0.015 * | noneno run with the option |
| Winnow-12B1 question per request | 128 | 34.1%540 of 1,582 | +23.6+21.0 to +25.9 | 0.0002 | 0.015 * | noneno run with the option |
| Elephant-11 question per request | 373 | 37.0%586 of 1,582 | +26.5+24.1 to +29.0 | 0.0002 | 0.015 * | 382 · 515“Bilmiyorum”, run 382: 1,793 answered (81.6%), 42.3% wrong (758)“I don't know”, run 515: 1,594 answered (72.5%), 39.0% wrong (622) |
| Perplexity Decider Flash v1.2local · 1 question per request | 223 | 43.7%691 of 1,582 | +33.1+30.6 to +35.6 | 0.0002 | 0.015 * | noneno run with the option |
| Clef-flash1 question per request | 139 | 45.3%716 of 1,582 | +34.7+32.0 to +37.2 | 0.0002 | 0.015 * | noneno run with the option |
| Microsoft-Decision-1via OpenRouter · 1 question per request | 205 | 45.9%726 of 1,582 | +35.3+32.6 to +37.8 | 0.0002 | 0.015 * | noneno run with the option |
| Metask-Jev 4B1 question per request | 126 | 53.2%841 of 1,582 | +42.6+39.8 to +45.1 | 0.0002 | 0.015 * | noneno run with the option |
| Liquid d1-2B (open weights)1 question per request | 340 | 67.0%1,060 of 1,582 | +56.4+53.7 to +59.0 | 0.0002 | 0.015 * | noneno run with the option |
| Strands Decider 2B1 question per request | 141 | 69.3%1,096 of 1,582 | +58.7+55.9 to +61.3 | 0.0002 | 0.015 * | noneno run with the option |
| meraGPT Decider 11 question per request | 243 | 70.8%1,120 of 1,582 | +60.2+57.5 to +62.8 | 0.0002 | 0.015 * | noneno run with the option |
| GLiNER2.5-multi-Decide1 question per request | 135 | 78.9%1,248 of 1,582 | +68.3+65.6 to +70.7 | 0.0002 | 0.015 * | noneno run with the option |
| Laya1 question per request | 51 | 80.5%1,273 of 1,582 | +69.9+67.3 to +72.4 | 0.0002 | 0.015 * | noneno run with the option |
| Laya multilingual1 question per request | 46 | 83.5%1,321 of 1,582 | +72.9+70.4 to +75.3 | 0.0002 | 0.015 * | noneno run with the option |
Other models: chat and general models
| Model | Five-option runthe run whose k answers are kept | Wrong of the k answerswrong / 1,582; lower is better | Difference from Liquid d1-ultrapoints, with its 95% interval; positive = more wrong answers | ppaired bootstrap, two-sided | Holm padjusted over the 73 | Runs with the optionfor context: answered, wrong of answers |
|---|---|---|---|---|---|---|
| GPT-6 Astralow reasoning · 16 questions per request | 48 | 0.3%4 of 1,582 | -10.3-12.1 to -8.8 | 0.0002 | 0.015 * | noneno run with the option |
| GPT-6 Astrahigh reasoning · 16 questions per request | 277 | 0.4%6 of 1,582 | -10.2-11.9 to -8.7 | 0.0002 | 0.015 * | noneno run with the option |
| Gemini 3.8 Flashhigh reasoning · 16 questions per request | 140 | 0.4%7 of 1,582 | -10.1-11.8 to -8.6 | 0.0002 | 0.015 * | noneno run with the option |
| GPT-6.1 Solhigh reasoning · 16 questions per request | 276 | 0.5%8 of 1,582 | -10.1-11.8 to -8.5 | 0.0002 | 0.015 * | noneno run with the option |
| GPT-6 Astralow reasoning · 1 question per request | 191 | 0.7%11 of 1,582 | -9.9-11.6 to -8.3 | 0.0002 | 0.015 * | noneno run with the option |
| GPT-6.1 Sollow reasoning · 1 question per request | 190 | 0.8%12 of 1,582 | -9.8-11.5 to -8.3 | 0.0002 | 0.015 * | noneno run with the option |
| Gemini 3.8 Flashlow reasoning · 1 question per request | 188 | 0.8%13 of 1,582 | -9.7-11.5 to -8.2 | 0.0002 | 0.015 * | noneno run with the option |
| GPT-6.1 Sollow reasoning · 16 questions per request | 124 | 0.8%13 of 1,582 | -9.7-11.5 to -8.2 | 0.0002 | 0.015 * | noneno run with the option |
| Gemini 4 Argonlow reasoning · 16 questions per request | 557 | 1.1%17 of 1,582 | -9.5-11.2 to -8.0 | 0.0002 | 0.015 * | noneno run with the option |
| Claude Fable 5.1high reasoning · 16 questions per request | 310 | 1.2%19 of 1,582 | -9.4-11.1 to -8.0 | 0.0002 | 0.015 * | noneno run with the option |
| Claude Opus 5.5low reasoning · 1 question per request | 192 | 1.5%23 of 1,582 | -9.1-10.7 to -7.6 | 0.0002 | 0.015 * | noneno run with the option |
| Claude Opus 5.5high reasoning · 16 questions per request | 278 | 1.5%23 of 1,582 | -9.1-10.8 to -7.6 | 0.0002 | 0.015 * | noneno run with the option |
| GPT-6.1 Terrahigh reasoning · 16 questions per request | 275 | 2.0%31 of 1,582 | -8.6-10.3 to -7.1 | 0.0002 | 0.015 * | noneno run with the option |
| Grok 4.7high reasoning · 16 questions per request | 311 | 2.0%31 of 1,582 | -8.6-10.3 to -7.1 | 0.0002 | 0.015 * | noneno run with the option |
| Gemini 3.8 Flashlow reasoning · 16 questions per request | 123 | 3.2%50 of 1,582 | -7.4-9.2 to -5.8 | 0.0002 | 0.015 * | noneno run with the option |
| GPT-6.1 Terralow reasoning · 16 questions per request | 274 | 3.2%50 of 1,582 | -7.4-9.2 to -6.0 | 0.0002 | 0.015 * | noneno run with the option |
| Claude Opus 5.5low reasoning · 16 questions per request | 47 | 3.2%51 of 1,582 | -7.3-9.0 to -5.9 | 0.0002 | 0.015 * | noneno run with the option |
| Claude Sonnet 5.5high reasoning · 1 question per request | 193 | 4.1%65 of 1,582 | -6.4-8.1 to -4.9 | 0.0002 | 0.015 * | noneno run with the option |
| Claude Sonnet 5.5high reasoning · 16 questions per request | 55 | 7.1%113 of 1,582 | -3.4-5.3 to -1.8 | 0.0002 | 0.015 * | noneno run with the option |
| DeepSeek-V4.1-Flash, first releasereasoning on · 16 questions per request | 121 | 7.1%113 of 1,582 | -3.4-5.4 to -1.8 | 0.0002 | 0.015 * | noneno run with the option |
| Kimi K3.1 Prolow reasoning · 16 questions per request | 320 | 8.9%141 of 1,582 | -1.6-3.5 to +0.0 | 0.051 | 0.1 | noneno run with the option |
| GLM-5.3-Flashreasoning on · 16 questions per request | 160 | 10.3%163 of 1,582 | -0.3-2.2 to +1.4 | 0.71 | 0.71 | noneno run with the option |
| Gemma 4 31Breasoning off · 16 questions per request | 119 | 15.1%239 of 1,582 | +4.6+2.5 to +6.1 | 0.0002 | 0.015 * | noneno run with the option |
| GLM-5.3low reasoning · 16 questions per request | 314 | 16.4%260 of 1,582 | +5.9+3.8 to +7.8 | 0.0002 | 0.015 * | noneno run with the option |
| Kimi K3.1 Proreasoning none · 16 questions per request | 315 | 18.6%294 of 1,582 | +8.0+5.8 to +9.9 | 0.0002 | 0.015 * | noneno run with the option |
| Qwen3.8-27Blow reasoning · 16 questions per request | 318 | 19.3%306 of 1,582 | +8.8+6.6 to +10.7 | 0.0002 | 0.015 * | noneno run with the option |
| DeepSeek-V4.1-Flash, first releasereasoning off · 16 questions per request | 120 | 21.9%346 of 1,582 | +11.3+9.3 to +13.6 | 0.0002 | 0.015 * | noneno run with the option |
| Gemma 4 31Breasoning off · 1 question per request | 189 | 21.9%346 of 1,582 | +11.3+9.0 to +13.3 | 0.0002 | 0.015 * | noneno run with the option |
| GPT-6 Luna (chat API)low reasoning · 16 questions per request | 133 | 24.5%388 of 1,582 | +14.0+11.8 to +16.2 | 0.0002 | 0.015 * | noneno run with the option |
| Claude Sonnet 5.5low reasoning · 16 questions per request | 54 | 27.8%440 of 1,582 | +17.3+14.9 to +19.5 | 0.0002 | 0.015 * | noneno run with the option |
| GLM-5.3-Flashreasoning off · 16 questions per request | 159 | 32.6%515 of 1,582 | +22.0+19.5 to +24.3 | 0.0002 | 0.015 * | noneno run with the option |
| Qwen3.8-27Bvote share · reasoning off · 1 question per request | 83 | 35.3%559 of 1,582 | +24.8+22.2 to +27.0 | 0.0002 | 0.015 * | noneno run with the option |
| Qwen3.8-27Byes/no scoring · 1 question per request | 50 | 37.9%599 of 1,582 | +27.3+24.9 to +29.7 | 0.0002 | 0.015 * | noneno run with the option |
| Qwen3.8-27Bstated confidence · reasoning off · 1 question per request | 53 | 40.1%635 of 1,582 | +29.6+27.1 to +31.9 | 0.0002 | 0.015 * | noneno run with the option |
| Erk-14Breasoning off · 16 questions per request | 132 | 46.5%735 of 1,582 | +35.9+33.1 to +38.4 | 0.0002 | 0.015 * | noneno run with the option |
| Qwen3-14Breasoning off · 16 questions per request | 131 | 47.1%745 of 1,582 | +36.5+33.6 to +38.9 | 0.0002 | 0.015 * | noneno run with the option |
| GLiNER2.5 Multi (base)1 question per request | 122 | 83.5%1,321 of 1,582 | +72.9+70.4 to +75.2 | 0.0002 | 0.015 * | noneno run with the option |
English: k = 496 of the 547 kept reading questions
| Model | Five-option runthe run whose k answers are kept | Wrong of the k answerswrong / 496; lower is better | Difference from Liquid d1-ultrapoints, with its 95% interval; positive = more wrong answers | ppaired bootstrap, two-sided | Holm padjusted over the 16 | Runs with the optionfor context: answered, wrong of answers |
|---|---|---|---|---|---|---|
| Fastino GLiDE v1.2local · 1 question per request | 521 | 5.2%26 of 496 | -0.6-2.4 to +1.6 | 0.76 | 1 | 523“I don't know”, run 523: 517 answered (94.5%), 7.0% wrong (36) |
| Drex Max v11 question per request | 552 | 5.6%28 of 496 | -0.2-2.2 to +2.0 | 0.98 | 1 | noneno run with the option |
| Liquid d1-ultra1 question per request | 411 | 5.8%29 of 496 | referenceits own run 449 with “I don't know”: 39 wrong of 496 (7.9%), not part of the test | reference | reference | 449“I don't know”, run 449: 496 answered (90.7%), 7.9% wrong (39) |
| vLLM-SR d31 question per request | 554 | 5.8%29 of 496 | +0.0-1.8 to +2.2 | 0.93 | 1 | noneno run with the option |
| Mammoth-11 question per request | 503 | 6.2%31 of 496 | +0.4-1.8 to +2.4 | 0.84 | 1 | 505“I don't know”, run 505: 510 answered (93.2%), 7.1% wrong (36) |
| vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request | 550 | 6.2%31 of 496 | +0.4-1.6 to +2.4 | 0.77 | 1 | noneno run with the option |
| Drex Max 1.11 question per request | 399 | 6.5%32 of 496 | +0.6-1.6 to +2.8 | 0.64 | 1 | 447“I don't know”, run 447: 504 answered (92.1%), 8.1% wrong (41) |
| Perplexity Decider 27B v1.2local · 1 question per request | 527 | 6.9%34 of 496 | +1.0-1.0 to +3.2 | 0.32 | 1 | 529“I don't know”, run 529: 545 answered (99.6%), 11.7% wrong (64) |
| Jev 1.14.01 question per request | 410 | 8.9%44 of 496 | +3.0+0.4 to +5.2 | 0.024 | 0.19 | 446“I don't know”, run 446: 544 answered (99.5%), 13.2% wrong (72) |
| Liquid d11 question per request | 430 | 8.9%44 of 496 | +3.0+0.8 to +5.6 | 0.017 | 0.15 | 445“I don't know”, run 445: 545 answered (99.6%), 12.3% wrong (67) |
| SubQ Snap 1.11 question per request | 413 | 8.9%44 of 496 | +3.0+1.0 to +5.8 | 0.008 | 0.08 | 448“I don't know”, run 448: 526 answered (96.2%), 12.0% wrong (63) |
| Inception Mercury Decidevia OpenRouter · 1 question per request | 398 | 10.1%50 of 496 | +4.2+1.6 to +6.7 | 0.0014 | 0.015 * | 450“I don't know”, run 450: 536 answered (98.0%), 13.2% wrong (71) |
| Microsoft-Decision-1via OpenRouter · 1 question per request | 412 | 10.5%52 of 496 | +4.6+2.2 to +7.3 | 0.0006 | 0.0072 * | 451“I don't know”, run 451: 544 answered (99.5%), 14.2% wrong (77) |
| Elephant-11 question per request | 499 | 11.1%55 of 496 | +5.2+2.6 to +7.9 | 0.0002 | 0.0032 * | 501“I don't know”, run 501: 524 answered (95.8%), 12.8% wrong (67) |
| Celeris-1 Decision1 question per request | 546 | 11.3%56 of 496 | +5.4+2.6 to +8.1 | 0.0002 | 0.0032 * | 548“I don't know”, run 548: 455 answered (83.2%), 8.1% wrong (37) |
| GPT-6 Luna (Decisions API)Decisions API, beta · 1 question per request | 397 | 11.7%58 of 496 | +5.8+3.2 to +8.9 | 0.0002 | 0.0032 * | 444“I don't know”, run 444: 537 answered (98.2%), 17.9% wrong (96) |
| Perplexity Decider Flash v1.2local · 1 question per request | 533 | 12.5%62 of 496 | +6.7+4.0 to +9.9 | 0.0002 | 0.0032 * | 535“I don't know”, run 535: 542 answered (99.1%), 18.5% wrong (100) |
Every setup's curve in Turkish
One small plot per setup, all 74, grouped and ordered as the Turkish table above, on the same axes: answered across (0 to 100% of the 2,198 questions), wrong of answers up (0 to 92%; anything higher is drawn at the top edge). The line is the setup's five-option run held back by its own confidence, from 1 answer to its scored answers; the dashed line is Liquid d1-ultra's (run 199) in every plot, and the dashed vertical line is k = 1,582. The markers are the setup's runs with the option ( “Bilmiyorum” “I don't know”). Each line is thinned to one vertex per 1% of coverage, always keeping the first, the last and the point at k; every vertex is a curve point.
Decision models
Liquid d1-ultra 1 question per requestWrong of the 1,582 answers: 167 (10.6%), the reference, dashed in every plot
Runs with the option: “Bilmiyorum” run 350, 13.0% wrong of 1,581 and “I don't know” run 511, 13.7% wrong of 1,582Fastino GLiDE v1.2 local · 1 question per requestWrong of the 1,582 answers: 289 (18.3%), +7.7 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 369, 26.2% wrong of 1,877 and “I don't know” run 525, 24.1% wrong of 1,772vLLM-SR d3 1 question per requestWrong of the 1,582 answers: 314 (19.8%), +9.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 358, 32.5% wrong of 2,198 and “I don't know” run 544, 32.5% wrong of 2,196vLLM-SR d3-ultra transformers, rev 52ba9e7 · 1 question per requestWrong of the 1,582 answers: 315 (19.9%), +9.4 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 363, 32.2% wrong of 2,188 and “I don't know” run 540, 32.2% wrong of 2,189Perplexity Decider 27B v1.1 1 question per requestWrong of the 1,582 answers: 318 (20.1%), +9.5 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredDrex Max 1.1 1 question per requestWrong of the 1,582 answers: 321 (20.3%), +9.7 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 352, 27.5% wrong of 1,878 and “I don't know” run 513, 24.1% wrong of 1,712Drex Max v1 1 question per requestWrong of the 1,582 answers: 321 (20.3%), +9.7 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 367, 32.4% wrong of 2,196 and “I don't know” run 542, 32.4% wrong of 2,196Mammoth-1 1 question per requestWrong of the 1,582 answers: 322 (20.4%), +9.8 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 386, 20.8% wrong of 1,508 and “I don't know” run 519, 19.1% wrong of 1,445Fastino GLiDE 1 question per requestWrong of the 1,582 answers: 325 (20.5%), +10.0 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 351, 21.2% wrong of 1,677 and “I don't know” run 510, 20.7% wrong of 1,617 (made with Fastino's engine glide_v2, not run 129's glide)Perplexity Decider 27B v1.2 local · 1 question per requestWrong of the 1,582 answers: 328 (20.7%), +10.2 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 372, 32.7% wrong of 2,196 and “I don't know” run 531, 33.2% wrong of 2,194SubQ Snap Preview 1 question per requestWrong of the 1,582 answers: 356 (22.5%), +11.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredSubQ Snap 1.1 1 question per requestWrong of the 1,582 answers: 361 (22.8%), +12.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 392, 27.8% wrong of 1,761 and “I don't know” run 512, 26.7% wrong of 1,700GPT-6 Luna (Decisions API) Decisions API, beta · 1 question per requestWrong of the 1,582 answers: 372 (23.5%), +13.0 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredPerplexity Decider 27B v1 1 question per requestWrong of the 1,582 answers: 430 (27.2%), +16.6 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredLiquid d1.1 1 question per requestWrong of the 1,582 answers: 434 (27.4%), +16.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredCeleris-1 Decision 1 question per requestWrong of the 1,582 answers: 450 (28.4%), +17.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredOneJev 27B 1 question per requestWrong of the 1,582 answers: 455 (28.8%), +18.2 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredJev 1.14.0 1 question per requestWrong of the 1,582 answers: 485 (30.7%), +20.1 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredInception Mercury Decide via OpenRouter · 1 question per requestWrong of the 1,582 answers: 487 (30.8%), +20.2 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredJev 1.13.0 1 question per requestWrong of the 1,582 answers: 498 (31.5%), +20.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredOpen-Jev 27B v1.1 1 question per requestWrong of the 1,582 answers: 504 (31.9%), +21.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredClef 1 question per requestWrong of the 1,582 answers: 512 (32.4%), +21.8 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredLiquid d1 1 question per requestWrong of the 1,582 answers: 516 (32.6%), +22.1 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredDrex 2.0 1 question per requestWrong of the 1,582 answers: 528 (33.4%), +22.8 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredCygnet 1 question per requestWrong of the 1,582 answers: 530 (33.5%), +22.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredWinnow-12B 1 question per requestWrong of the 1,582 answers: 540 (34.1%), +23.6 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredElephant-1 1 question per requestWrong of the 1,582 answers: 586 (37.0%), +26.5 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: “Bilmiyorum” run 382, 42.3% wrong of 1,793 and “I don't know” run 515, 39.0% wrong of 1,594Perplexity Decider Flash v1.2 local · 1 question per requestWrong of the 1,582 answers: 691 (43.7%), +33.1 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredClef-flash 1 question per requestWrong of the 1,582 answers: 716 (45.3%), +34.7 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredMicrosoft-Decision-1 via OpenRouter · 1 question per requestWrong of the 1,582 answers: 726 (45.9%), +35.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredMetask-Jev 4B 1 question per requestWrong of the 1,582 answers: 841 (53.2%), +42.6 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredLiquid d1-2B (open weights) 1 question per requestWrong of the 1,582 answers: 1,060 (67.0%), +56.4 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredStrands Decider 2B 1 question per requestWrong of the 1,582 answers: 1,096 (69.3%), +58.7 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredmeraGPT Decider 1 1 question per requestWrong of the 1,582 answers: 1,120 (70.8%), +60.2 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGLiNER2.5-multi-Decide 1 question per requestWrong of the 1,582 answers: 1,248 (78.9%), +68.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredLaya 1 question per requestWrong of the 1,582 answers: 1,273 (80.5%), +69.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredLaya multilingual 1 question per requestWrong of the 1,582 answers: 1,321 (83.5%), +72.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measured
Other models: chat and general models
GPT-6 Astra low reasoning · 16 questions per requestWrong of the 1,582 answers: 4 (0.3%), -10.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGPT-6 Astra high reasoning · 16 questions per requestWrong of the 1,582 answers: 6 (0.4%), -10.2 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGemini 3.8 Flash high reasoning · 16 questions per requestWrong of the 1,582 answers: 7 (0.4%), -10.1 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGPT-6.1 Sol high reasoning · 16 questions per requestWrong of the 1,582 answers: 8 (0.5%), -10.1 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGPT-6 Astra low reasoning · 1 question per requestWrong of the 1,582 answers: 11 (0.7%), -9.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGPT-6.1 Sol low reasoning · 1 question per requestWrong of the 1,582 answers: 12 (0.8%), -9.8 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGemini 3.8 Flash low reasoning · 1 question per requestWrong of the 1,582 answers: 13 (0.8%), -9.7 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGPT-6.1 Sol low reasoning · 16 questions per requestWrong of the 1,582 answers: 13 (0.8%), -9.7 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGemini 4 Argon low reasoning · 16 questions per requestWrong of the 1,582 answers: 17 (1.1%), -9.5 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredClaude Fable 5.1 high reasoning · 16 questions per requestWrong of the 1,582 answers: 19 (1.2%), -9.4 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredClaude Opus 5.5 low reasoning · 1 question per requestWrong of the 1,582 answers: 23 (1.5%), -9.1 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredClaude Opus 5.5 high reasoning · 16 questions per requestWrong of the 1,582 answers: 23 (1.5%), -9.1 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGPT-6.1 Terra high reasoning · 16 questions per requestWrong of the 1,582 answers: 31 (2.0%), -8.6 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGrok 4.7 high reasoning · 16 questions per requestWrong of the 1,582 answers: 31 (2.0%), -8.6 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGemini 3.8 Flash low reasoning · 16 questions per requestWrong of the 1,582 answers: 50 (3.2%), -7.4 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGPT-6.1 Terra low reasoning · 16 questions per requestWrong of the 1,582 answers: 50 (3.2%), -7.4 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredClaude Opus 5.5 low reasoning · 16 questions per requestWrong of the 1,582 answers: 51 (3.2%), -7.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredClaude Sonnet 5.5 high reasoning · 1 question per requestWrong of the 1,582 answers: 65 (4.1%), -6.4 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredClaude Sonnet 5.5 high reasoning · 16 questions per requestWrong of the 1,582 answers: 113 (7.1%), -3.4 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredDeepSeek-V4.1-Flash, first release reasoning on · 16 questions per requestWrong of the 1,582 answers: 113 (7.1%), -3.4 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredKimi K3.1 Pro low reasoning · 16 questions per requestWrong of the 1,582 answers: 141 (8.9%), -1.6 points against Liquid d1-ultra, Holm p = 0.1
Runs with the option: Not measuredGLM-5.3-Flash reasoning on · 16 questions per requestWrong of the 1,582 answers: 163 (10.3%), -0.3 points against Liquid d1-ultra, Holm p = 0.71
Runs with the option: Not measuredGemma 4 31B reasoning off · 16 questions per requestWrong of the 1,582 answers: 239 (15.1%), +4.6 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGLM-5.3 low reasoning · 16 questions per requestWrong of the 1,582 answers: 260 (16.4%), +5.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredKimi K3.1 Pro reasoning none · 16 questions per requestWrong of the 1,582 answers: 294 (18.6%), +8.0 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredQwen3.8-27B low reasoning · 16 questions per requestWrong of the 1,582 answers: 306 (19.3%), +8.8 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredDeepSeek-V4.1-Flash, first release reasoning off · 16 questions per requestWrong of the 1,582 answers: 346 (21.9%), +11.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGemma 4 31B reasoning off · 1 question per requestWrong of the 1,582 answers: 346 (21.9%), +11.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGPT-6 Luna (chat API) low reasoning · 16 questions per requestWrong of the 1,582 answers: 388 (24.5%), +14.0 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredClaude Sonnet 5.5 low reasoning · 16 questions per requestWrong of the 1,582 answers: 440 (27.8%), +17.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGLM-5.3-Flash reasoning off · 16 questions per requestWrong of the 1,582 answers: 515 (32.6%), +22.0 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredQwen3.8-27B vote share · reasoning off · 1 question per requestWrong of the 1,582 answers: 559 (35.3%), +24.8 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredQwen3.8-27B yes/no scoring · 1 question per requestWrong of the 1,582 answers: 599 (37.9%), +27.3 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredQwen3.8-27B stated confidence · reasoning off · 1 question per requestWrong of the 1,582 answers: 635 (40.1%), +29.6 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredErk-14B reasoning off · 16 questions per requestWrong of the 1,582 answers: 735 (46.5%), +35.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredQwen3-14B reasoning off · 16 questions per requestWrong of the 1,582 answers: 745 (47.1%), +36.5 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measuredGLiNER2.5 Multi (base) 1 question per requestWrong of the 1,582 answers: 1,321 (83.5%), +72.9 points against Liquid d1-ultra, Holm p = 0.015 *
Runs with the option: Not measured
Our method for the same number of answers
Each model's five-option run answers only its k most confident questions, by its score (the probability it gave the option it picked), and holds back on the rest; we count its wrong answers among the k. Wrong of the k answers = wrong / k. k is the number of questions Liquid d1-ultra answered (right + wrong) with the option on offer: 1,582 in its run 511 with “I don't know” on the 2,198 Turkish questions (the option's word, above), and 496 in its run 449, its first English run with “I don't know”, on the 547 kept reading questions. Every model in a language is held to the same k.
The answers are ranked by score, highest first, and equal scores by question ID, lowest first (never by whether the answer is right); the first k are kept, so the wrong count is always a whole number. An answer without a score or a valid choice is never kept.
The difference is a model's wrong of the k answers minus Liquid d1-ultra's, in points. Its uncertainty comes from a paired bootstrap over questions: 10,000 resamples of the questions with replacement, from a new random generator with the numeric seed 20261103 for each language, the same resample for every model, a question drawn twice counting twice and k fixed. In each resample every run's drawn answers are ranked again by score and question ID, the copies of a drawn question sitting together, and the first k kept, so the count stays whole. Two-sided p = min(1, 2 × min(1 + #{d ≤ 0}, 1 + #{d ≥ 0}) / 10,001), where #{d ≤ 0} is the number of resamples whose difference d is 0 or less and #{d ≥ 0} the number whose difference is 0 or more. Its floor is 2/10,001 (p = 0.0002), when no resample falls on the other side of zero: a p at the floor says the difference held in every resample, not that p is exactly that value. The 95% interval is the percentile interval: from the 250th smallest to the 250th largest of the 10,000 resampled differences.
Holm's method adjusts p over the other 73 setups in Turkish (every main run of the Text tab, decision models and chat and general models alike) and the other 16 models in English, each language a family of its own, separate from the tab's other comparisons. The resamples do not depend on which models are compared, so a model's p is the same in any family; only Holm's adjustment grows with the family. For Fastino GLiDE, the Turkish five-option run is the Text tab's run 129, made with Fastino's engine glide; its runs with the option, 351 and 510, answered with glide_v2, a different hosted model (Our method for the hosted GLiDE rerun), so its line and its markers come from two models, as the legend and the table say.
The plots show every number of answers, not only k. Each line is a five-option run under the same selection, one point per number of answers from 1 to the run's scored answers: answered / questions across, wrong / answered up. There is no point at 0 answers, where the share wrong is undefined, and a line ends at its run's scored answers, without being extended to 100%. The Turkish plot draws the 11 setups with runs with the option; the English curves come from this comparison too, so they can differ slightly from those of the Turkish vs English tab (Risk and coverage), where answers with equal scores enter together. A line skips vertices closer than 0.25% of coverage to the previous one drawn, always keeping the first, the last and the point at k, so every vertex drawn is a curve point; the y axis fits the markers and every curve point at 1% coverage or more, and anything higher (the first few answers, where one wrong answer is a large share) is drawn at the top edge. The markers are each model's own runs with the option: in Turkish its first run with “Bilmiyorum” (the table above) and its run with “I don't know” (the option's word); in English its first English run with “I don't know”.
Limits: k comes from one model, Liquid d1-ultra, and a different k gives different numbers (the plots show them all). The k answers kept are each model's own most confident ones, so the models are counted on different questions. Liquid d1-ultra's own runs with the option made 216 wrong of 1,582 in Turkish and 39 of 496 in English, against 167 and 29 for its five-option run held to the same k: the comparison is between five-option runs held back by confidence, not between runs with the option.