Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Provisional: Drex Max v1.1, whose maker publishes only a promotional price. Left out here: the setups only this site ran (Erk-14B, GLiNER2.5 Multi, GLiNER2.5-multi-Decide, Laya, Laya Multilingual and Qwen3-14B); they show under Class: All. On this page a class narrows the rows of the tables and bar charts; the sentences, counts, tests and line plots still cover every model of the type in view.
Hallucination
The Text tab's 2,198 questions, with “I don't know” as a sixth option
Non-hallucination, answered and error rate
Holding back by confidence
The answers each setup gave at 99% confidence or more, in its five-option run on the Text tab, the bar their share of the 2,198 questions; how many of them were wrong; and its ranking score (AUROC: how often a right answer has higher confidence than a wrong one, 0.5 chance, 1 perfect). No wrong answer at 99% first, by answers kept, then by the share wrong among the kept answers; a setup with no answers at 99% last · more kept and fewer wrong is better
Decision models
Confidence: option probabilities.
Karpo Perplexity Gemini Mammoth vLLM-SR Fastino REAX Drex Elephant KnowLine Copernicus Liquid Logits Turion Jev Verdin StartLux Valen vera Cygnet OneJev Sieve SubQ Winnow Dexy Adjudex Chinese Decision Pointer Laya Cerno Clef GLiClass Jeffan Metask Open-Jev Strands
Chat models
Confidence: the confidence the model states for 38; option probabilities for 8; agreement across ten answers for 1.
Inkling GLM Qwen Claude Muse Gemini GPT DeepSeek Erk MiMo Kimi MiniMax Gemma Fastino Grok Confidence not measured (a plain answer line, no confidence): LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, Qwen3.5-4B · reasoning off · chat and Qwen3.5-4B · reasoning on · chat.
Holding back with an “I don't know” option
“I don't know” among the answers that were not right, with it offered as a sixth option on the 2,198 questions; beside it the share answered (right + wrong of 2,198) and the error rate among the answers (wrong of right + wrong), in the table's order · higher non-hallucination is better, read with the share answered; a model that always chose “I don't know” would reach 100% with no right answers
Decision models
Chose “I don't know” on fewer than 1% of the 2,198 questions: Turion Jev 31B, 15 times; Gemini Decision 1 · 20261111, 10 times; StartLux-Decision-v1.1-4B-Instant, 21 times; Open-Jev-27B-v1.1, 14 times; d3-ultra, 10 times; Copernicus-31B, 5 times; Adjudex 9B v5, 8 times; Metask-Jev-4B, 10 times; Perplexity Decider 27B v1.3, 5 times; Karpo Decision 31B, 3 times; KnowLine-9B-Gen1, 5 times; Verdin-4B-v1.1, 5 times; REAX 1.3, 2 times; Perplexity Decider 27B v1.2 · API, 3 times; Logits Sigma-1 Duo, 2 times; Drex Max, 2 times; Perplexity Decider 27B v1.2 · DGX Spark, 2 times; KnowLine-31B-Gen1.5, 1 time; Cerno 9B v1.2, 2 times; KnowLine-12B-Gen1, 1 time; OneJev-27B, 1 time; Valen-Decision-1.5-9B, 1 time; KnowLine-31B-Gen2, 0 times; Sieve 31B v3, 0 times; d3, 0 times. Their rate here is low (3.3% or less) because they almost never chose the option; the part Holding back by confidence shows how their confidence holds back wrong answers.
Non-hallucination Answered Error rate
Chat models
Non-hallucination Answered Error rate
Saying “I don't know”, or held back by confidence
Each model's non-hallucination rate with “I don't know”, and its five-option run held back by its own confidence at about the same attempt rate · higher is better; beside them the run with the option's share answered and error rate among the answers
Decision models
With “I don't know” Held back by its own confidence (the derived rate)
Chat models
With “I don't know” Held back by its own confidence (the derived rate) Held back by confidence not measured (a plain answer line, no confidence): LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, Qwen3.5-4B · reasoning off · chat and Qwen3.5-4B · reasoning on · chat.
Fewer wrong answers: “I don't know” or confidence?
Each decision model's first run with “I don't know” against its five-option run held back by its own confidence on as many questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 38 decision models that chose “I don't know” on at least 1% of the questions · the shorter bar left fewer wrong answers
Wrong only with “I don't know” Wrong only when held back by its own confidence No difference is significant (Holm-adjusted p below 0.05). Adjusted: p = 0.085, p = 0.098, p = 0.48, p = 0.72, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not in the test, as they chose “I don't know” on fewer than 1% of the questions: Turion Jev 31B, Gemini Decision 1 · 20261111, StartLux-Decision-v1.1-4B-Instant, Open-Jev-27B-v1.1, d3-ultra, Copernicus-31B, Adjudex 9B v5, Metask-Jev-4B, Perplexity Decider 27B v1.3, Karpo Decision 31B, KnowLine-9B-Gen1, Verdin-4B-v1.1, REAX 1.3, Perplexity Decider 27B v1.2 · API, Logits Sigma-1 Duo, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark, KnowLine-31B-Gen1.5, Cerno 9B v1.2, KnowLine-12B-Gen1, OneJev-27B, Valen-Decision-1.5-9B, KnowLine-31B-Gen2, Sieve 31B v3 and d3. Chat models are not in this test.
Accuracy with “I don't know”
Share of the 2,198 questions answered right with “I don't know” on offer · higher is better
Decision models
Decision models: worst 1.2%best 81.5% (higher is better) The same model with five options (the Text tab)
Chat models
Chat models: worst 5.0%best 98.2% (higher is better) The same model with five options (the Text tab)
Non-hallucination, answered and error rate
Holding back by confidence
The answers each setup gave at 99% confidence or more, in its five-option run on the Text tab, the bar their share of the 2,198 questions; how many of them were wrong; and its ranking score (AUROC: how often a right answer has higher confidence than a wrong one, 0.5 chance, 1 perfect). No wrong answer at 99% first, by answers kept, then by the share wrong among the kept answers; a setup with no answers at 99% last · more kept and fewer wrong is better
Chat models
Confidence: the confidence the model states for 38; option probabilities for 8; agreement across ten answers for 1.
Inkling GLM Qwen Claude Muse Gemini GPT DeepSeek Erk MiMo Kimi MiniMax Gemma Fastino Grok Confidence not measured (a plain answer line, no confidence): LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, Qwen3.5-4B · reasoning off · chat and Qwen3.5-4B · reasoning on · chat.
Holding back with an “I don't know” option
“I don't know” among the answers that were not right, with it offered as a sixth option on the 2,198 questions; beside it the share answered (right + wrong of 2,198) and the error rate among the answers (wrong of right + wrong), in the table's order · higher non-hallucination is better, read with the share answered; a model that always chose “I don't know” would reach 100% with no right answers
Chat models
Non-hallucination Answered Error rate
Saying “I don't know”, or held back by confidence
Each model's non-hallucination rate with “I don't know”, and its five-option run held back by its own confidence at about the same attempt rate · higher is better; beside them the run with the option's share answered and error rate among the answers
Chat models
With “I don't know” Held back by its own confidence (the derived rate) Held back by confidence not measured (a plain answer line, no confidence): LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, Qwen3.5-4B · reasoning off · chat and Qwen3.5-4B · reasoning on · chat.
Accuracy with “I don't know”
Share of the 2,198 questions answered right with “I don't know” on offer · higher is better
Chat models
Chat models: worst 5.0%best 98.2% (higher is better) The same model with five options (the Text tab)
Non-hallucination, answered and error rate
Holding back by confidence
The answers each setup gave at 99% confidence or more, in its five-option run on the Text tab, the bar their share of the 2,198 questions; how many of them were wrong; and its ranking score (AUROC: how often a right answer has higher confidence than a wrong one, 0.5 chance, 1 perfect). No wrong answer at 99% first, by answers kept, then by the share wrong among the kept answers; a setup with no answers at 99% last · more kept and fewer wrong is better
Decision models
Confidence: option probabilities.
Karpo Perplexity Gemini Mammoth vLLM-SR Fastino REAX Drex Elephant KnowLine Copernicus Liquid Logits Turion Jev Verdin StartLux Valen vera Cygnet OneJev Sieve SubQ Winnow Dexy Adjudex Chinese Decision Pointer Laya Cerno Clef GLiClass Jeffan Metask Open-Jev Strands
Holding back with an “I don't know” option
“I don't know” among the answers that were not right, with it offered as a sixth option on the 2,198 questions; beside it the share answered (right + wrong of 2,198) and the error rate among the answers (wrong of right + wrong), in the table's order · higher non-hallucination is better, read with the share answered; a model that always chose “I don't know” would reach 100% with no right answers
Decision models
Chose “I don't know” on fewer than 1% of the 2,198 questions: Turion Jev 31B, 15 times; Gemini Decision 1 · 20261111, 10 times; StartLux-Decision-v1.1-4B-Instant, 21 times; Open-Jev-27B-v1.1, 14 times; d3-ultra, 10 times; Copernicus-31B, 5 times; Adjudex 9B v5, 8 times; Metask-Jev-4B, 10 times; Perplexity Decider 27B v1.3, 5 times; Karpo Decision 31B, 3 times; KnowLine-9B-Gen1, 5 times; Verdin-4B-v1.1, 5 times; REAX 1.3, 2 times; Perplexity Decider 27B v1.2 · API, 3 times; Logits Sigma-1 Duo, 2 times; Drex Max, 2 times; Perplexity Decider 27B v1.2 · DGX Spark, 2 times; KnowLine-31B-Gen1.5, 1 time; Cerno 9B v1.2, 2 times; KnowLine-12B-Gen1, 1 time; OneJev-27B, 1 time; Valen-Decision-1.5-9B, 1 time; KnowLine-31B-Gen2, 0 times; Sieve 31B v3, 0 times; d3, 0 times. Their rate here is low (3.3% or less) because they almost never chose the option; the part Holding back by confidence shows how their confidence holds back wrong answers.
Non-hallucination Answered Error rate
Saying “I don't know”, or held back by confidence
Each model's non-hallucination rate with “I don't know”, and its five-option run held back by its own confidence at about the same attempt rate · higher is better; beside them the run with the option's share answered and error rate among the answers
Decision models
With “I don't know” Held back by its own confidence (the derived rate)
Fewer wrong answers: “I don't know” or confidence?
Each decision model's first run with “I don't know” against its five-option run held back by its own confidence on as many questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 38 decision models that chose “I don't know” on at least 1% of the questions · the shorter bar left fewer wrong answers
Wrong only with “I don't know” Wrong only when held back by its own confidence No difference is significant (Holm-adjusted p below 0.05). Adjusted: p = 0.085, p = 0.098, p = 0.48, p = 0.72, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not in the test, as they chose “I don't know” on fewer than 1% of the questions: Turion Jev 31B, Gemini Decision 1 · 20261111, StartLux-Decision-v1.1-4B-Instant, Open-Jev-27B-v1.1, d3-ultra, Copernicus-31B, Adjudex 9B v5, Metask-Jev-4B, Perplexity Decider 27B v1.3, Karpo Decision 31B, KnowLine-9B-Gen1, Verdin-4B-v1.1, REAX 1.3, Perplexity Decider 27B v1.2 · API, Logits Sigma-1 Duo, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark, KnowLine-31B-Gen1.5, Cerno 9B v1.2, KnowLine-12B-Gen1, OneJev-27B, Valen-Decision-1.5-9B, KnowLine-31B-Gen2, Sieve 31B v3 and d3.
Accuracy with “I don't know”
Share of the 2,198 questions answered right with “I don't know” on offer · higher is better
Decision models
Decision models: worst 1.2%best 81.5% (higher is better) The same model with five options (the Text tab)
A wrong answer counts as a hallucination: the model could have chosen “I don't know” instead.
The English questions: the English option against the Turkish option
The 547 reading questions of the Turkish vs English tab, in English with a sixth option, “I don't know”, against the same models' answers with the Turkish option on the same questions in Turkish (this tab's runs counted on those questions, or runs on those questions only). Counts and the paired test: under Info; how the questions were translated and kept: Turkish vs English, Our method.
46 of the 67 models with both runs had a higher non-hallucination rate in English than in Turkish; 17 lower; four the same; 12 of the 67 differences in wrong answers are significant after Holm's correction: d1-2B (more wrong answers in Turkish), GLiDE 0.4B (more wrong answers in English), Laya Multilingual (more wrong answers in Turkish), OneJev1.6-0.8B (more wrong answers in Turkish), GLiDE 2B (more wrong answers in Turkish), GLiDE 0.8B (more wrong answers in Turkish), GLiClass v4 Edge (more wrong answers in Turkish), StartLux-Decision-v1.1-2B-Instant (more wrong answers in Turkish), GLiNER2.5-multi-Decide (more wrong answers in Turkish), KnowLine-0.8B-Gen1 (more wrong answers in Turkish), StartLux-Decision-v1.1-0.8B-Instant (more wrong answers in Turkish) and Metask-Jev-4B (more wrong answers in Turkish) (Paired test). celeris-1.1-decision has an English rate of 74.8% (98 “I don't know” of 131 not right) with an attempt rate of 82.1% (449 answered) but no Turkish run with the option on these questions, so no comparison or test. celeris-1-decision has an English rate of 71.3% (92 “I don't know” of 129 not right) with an attempt rate of 83.2% (455 answered) but no Turkish run with the option on these questions, so no comparison or test. KnowLine-2B-Gen1 has an English rate of 22.0% (38 “I don't know” of 173 not right) with an attempt rate of 93.1% (509 answered) but no Turkish run with the option on these questions, so no comparison or test. Read each rate with its attempt rate: a model that always chose the option would reach 100%.
The four models of the request: Holm's correction over the four
d1-ultra, SubQ Snap 1.1, Elephant-1 and Mammoth-1, as Ümit's follow-up of 2 November 2026 asked: each model's first run in English and in Turkish, each with the “I don't know” option in its language, on the 547 kept questions, and the exact McNemar test of wrong answers, with p adjusted by Holm's method over these four. None of the four differences in wrong answers is significant with Holm's correction over the four (adjusted p below 0.05); the smallest adjusted p is d1-ultra's, p = 0.077. The charts below and the Turkish vs English tab correct over all 67 compared models.
| Model | English non-hallucinationof answers not right, % “I don't know” · higher is better; right / wrong / option | Answered in Englishnot “I don't know”: (right + wrong) / 547 | Error rate in Englishwith “I don't know”, of its answers: wrong / (right + wrong) · lower is better | Turkish non-hallucinationof answers not right, % the Turkish option · higher is better; right / wrong / option | Answered in Turkishnot the Turkish option: (right + wrong) / 547 | Error rate in Turkishwith the Turkish option, of its answers: wrong / (right + wrong) · lower is better | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 4 models |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient | 56.7%457 / 39 / 51 | 90.7%496 of 547 answered | 7.9%39 of 496 | 53.6%491 / 26 / 30 | 94.5%517 of 547 answered | 5.0%26 of 517 | 20 | 7 | 0.019 | 0.077 |
| SubQ Snap 1.1 Closed · Efficient | 25.0%463 / 63 / 21 | 96.2%526 of 547 answered | 12.0%63 of 526 | 27.1%477 / 51 / 19 | 96.5%528 of 547 answered | 9.7%51 of 528 | 39 | 27 | 0.18 | 0.35 |
| Elephant-1 Open · 9.0B | 25.6%457 / 67 / 23 | 95.8%524 of 547 answered | 12.8%67 of 524 | 15.5%450 / 82 / 15 | 97.3%532 of 547 answered | 15.4%82 of 532 | 31 | 46 | 0.11 | 0.33 |
| Mammoth-1 Open · 27B | 50.7%474 / 36 / 37 | 93.2%510 of 547 answered | 7.1%36 of 510 | 42.1%490 / 33 / 24 | 95.6%523 of 547 answered | 6.3%33 of 523 | 18 | 15 | 0.73 | 0.73 |
In English, then in Turkish: the non-hallucination rate with its counts (right / wrong / option), Answered = (right + wrong) / 547 and the error rate = wrong / (right + wrong). * marks a Holm-adjusted p below 0.05 over the four.
Non-hallucination rate in Turkish and in English
The option among the answers that were not right, on the 547 kept questions: each model's first run in each language, ordered by the English rate · higher is better, read with the attempt rate (under the chart and in each bar's tooltip)
Turkish option, the upper bar English option, the lower bar, striped Numbers on the right: Turkish · English. Liquid Fastino Drex Mammoth Laya OneJev Elephant SubQ GLiClass Open-Jev StartLux Perplexity Winnow Inception Cygnet KnowLine GPT Gemini Turion Jev vLLM-SR vera REAX Clef Logits Jev Microsoft Chinese Decision Pointer Cerno Sieve Verdin Metask Valen Dexy Strands Adjudex Copernicus Jeffan Karpo Attempt rate, Turkish · English: d1.2-ultra 88.8% · 80.1%; d1.2 90.3% · 89.2%; d1-2B 70.6% · 72.4%; d1-ultra 94.5% · 90.7%; GLiDE 0.4B 13.2% · 72.0%; Drex Max v1.1 98.4% · 92.1%; Mammoth-1 95.6% · 93.2%; GLiDE · v1.2 98.9% · 94.5%; Laya Multilingual 77.3% · 62.0%; GLiDE 27B 98.4% · 96.0%; OneJev1.6-0.8B 79.5% · 86.5%; GLiDE 2B 87.9% · 91.4%; Elephant-1 97.3% · 95.8%; SubQ Snap 1.1 96.5% · 96.2%; GLiDE 0.8B 86.5% · 91.0%; GLiClass v4 Edge 89.4% · 90.5%; Laya 90.3% · 84.8%; Open-Jev-27B-v1.1 98.9% · 97.1%; StartLux-Decision-v1.1-2B-Instant 94.0% · 94.1%; GLiDE 9B 98.4% · 97.4%; GLiNER2.5-multi-Decide 95.6% · 87.2%; Perplexity Decider 27B v1.2 · API · with web 99.5% · 98.2%; Winnow-12B 98.5% · 97.3%; Mercury Decide 98.5% · 98.0%; Cygnet 99.1% · 98.2%; KnowLine-0.8B-Gen1 97.3% · 95.2%; GPT-6 Luna · Decisions API 98.9% · 98.2%; Gemini Decision 1 · 20261111 99.1% · 99.1%; Perplexity Decider 27B v1.3 100.0% · 99.1%; Turion Jev 31B 99.6% · 99.3%; d3-ultra 99.8% · 99.1%; vera-medium-omni 99.1% · 98.7%; Perplexity Decider Flash v1.3 98.4% · 98.7%; REAX 1.3 99.8% · 99.5%; Clef-Flash 97.8% · 99.1%; StartLux-Decision-v1.1-0.8B-Instant 96.0% · 97.8%; Perplexity Decider Flash v1.2 98.9% · 99.1%; Perplexity Decider 27B v1.2 · API 100.0% · 99.5%; Logits Sigma-1 Duo 100.0% · 99.6%; Jev 1.14 99.5% · 99.5%; OneJev-27B 99.8% · 99.5%; Microsoft-Decision-1 99.6% · 99.5%; Chinese Decision Pointer 4B 99.3% · 99.3%; Drex Max 99.6% · 99.6%; Perplexity Decider 27B v1.2 · DGX Spark 100.0% · 99.6%; d1 99.8% · 99.6%; StartLux-Decision-v1.1-9B-Instant 99.8% · 99.6%; Cerno 9B v1.2 100.0% · 99.6%; Sieve 31B v3 100.0% · 99.8%; Verdin-4B-v1.1 99.8% · 99.6%; d3 100.0% · 99.8%; Metask-Jev-4B 99.6% · 99.6%; Clef 99.8% · 99.8%; KnowLine-12B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen2 100.0% · 99.8%; Valen-Decision-1.5-9B 100.0% · 99.8%; Dexy-Decision-9B-V2 99.8% · 99.8%; KnowLine 4B Gen5.5 99.5% · 99.8%; Strands Decider 2B 99.1% · 99.6%; Adjudex 9B v5 100.0% · 100.0%; Copernicus-31B 99.8% · 100.0%; Jeffan-9B-1211 99.5% · 100.0%; Karpo Decision 31B 99.8% · 100.0%; KnowLine-31B-Gen1.5 100.0% · 100.0%; KnowLine-31B-Gen2 100.0% · 100.0%; StartLux-Decision-v1.1-4B-Instant 99.6% · 100.0%. A model that always chose the option would reach 100% with no right answers. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision and KnowLine-2B-Gen1 (in the table under Info).
Wrong only in Turkish, or only in English?
Each model's first run in each language, both with the option, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the decision models · the shorter bar is the language with fewer wrong answers
Wrong only in Turkish, the upper bar Wrong only in English, the lower bar, striped Numbers on the right: Turkish · English. GLiClass Liquid KnowLine StartLux Fastino Laya Metask OneJev Perplexity Verdin vera Clef Valen Chinese Decision Pointer Strands GPT Sieve Karpo Elephant Turion Jev Gemini Cerno SubQ Microsoft Drex Adjudex vLLM-SR Logits REAX Copernicus Mammoth Jev Open-Jev Cygnet Dexy Jeffan Inception Winnow * A significant difference (Holm-adjusted p below 0.05): GLiClass v4 Edge, d1-2B, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, StartLux-Decision-v1.1-2B-Instant, Laya Multilingual, GLiDE 2B, Metask-Jev-4B, OneJev1.6-0.8B, GLiNER2.5-multi-Decide and GLiDE 0.8B (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p = 0.0026 and p = 0.0079). The other 55: p = 0.091, p = 0.099, p = 0.23, p = 0.28, p = 0.33, p = 0.4, p = 0.4, p = 0.71, p = 0.88, p = 0.88, p = 0.9, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision and KnowLine-2B-Gen1 (in the table under Info).
The option's word: “I don't know” instead of “Bilmiyorum”
With “I don't know” in place of “Bilmiyorum”, 9 of the 13 models chose the option more often and 3 less often; 1 as often. Their non-hallucination rates moved by -3.0 to +15.8 points. In the exact McNemar test of wrong answers, with Holm's correction over the 13, “I don't know” left significantly fewer wrong answers for Mammoth-1, GLiDE · v1.2, Drex Max v1.1 and Elephant-1; for the other 9, the difference could be chance. Not run: Copernicus-31B, KnowLine-31B-Gen2, KnowLine-31B-Gen1.5, Turion Jev 31B, REAX 1.3, Karpo Decision 31B, Gemini Decision 1, Logits Sigma-1 Duo, Sieve 31B v3, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.2 · API, GLiDE 27B, KnowLine-12B-Gen1, OneJev-27B, Jeffan-9B-1211, Clef, Open-Jev-27B-v1.1, Cygnet, Cerno 9B v1.2, KnowLine-9B-Gen1, Winnow-12B, Adjudex 9B v5, KnowLine-9B-Gen2, Dexy-Decision-9B-V2, StartLux-Decision-v1.1-9B-Instant, Perplexity Decider Flash v1.3, GLiDE 9B, Valen-Decision-1.5-9B, StartLux-Decision-v1.1-4B-Instant, Verdin-4B-v1.1, KnowLine 4B Gen5.5, vera-medium-omni, Chinese Decision Pointer 4B, Clef-Flash, Perplexity Decider Flash v1.2, Metask-Jev-4B, StartLux-Decision-v1.1-2B-Instant, GLiDE 2B, StartLux-Decision-v1.1-0.8B-Instant, KnowLine-0.8B-Gen1, Strands Decider 2B, GLiDE 0.8B, OneJev1.6-0.8B, GLiNER2.5-multi-Decide, GLiClass v4 Edge, d1-2B, Laya Multilingual, Laya and GLiDE 0.4B (why).
The same 2,198 Turkish questions, with only the sixth option's word and the sentence that names it changed. A model that skipped “Bilmiyorum” because it did not recognise the Turkish word should choose “I don't know” more often. The languages each model's card or docs list are on the Turkish vs English tab (Languages the model cards list). Table and method: under Info.
Non-hallucination rate by the option's word
Each model's first run with “I don't know” and with “Bilmiyorum” · higher is better
With “I don't know” With “Bilmiyorum” (the table above) Read it with accuracy and the attempt rate, in the table under Info.
“I don't know” against confidence
With “I don't know” in place of “Bilmiyorum”, we compared each model's first run with “I don't know” against its five-option run held back by its own confidence at the same attempt rate (each cut-off reached the option run's answered count exactly for the 9 that chose “I don't know” on at least 1% of the questions). Confidence left fewer wrong answers for all 9, 9 to 49 fewer. In the exact McNemar test of wrong answers, with Holm's correction over the 9, confidence left significantly fewer for d1-ultra (216 wrong with “I don't know”, 167 by confidence; adjusted p = 0.00059); for the other 8, the difference could be chance. GLiDE · API is compared with its new five-option run 517, not run 129 (why).
A family of its own: the paired test with “Bilmiyorum” is the chart Fewer wrong answers above, and neither changes the other. Table and method: under Info.
Fewer wrong answers: “I don't know” or confidence?
Each model's first run with “I don't know” against its five-option run held back by its own confidence at about the same attempt rate: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 9 models that chose “I don't know” on at least 1% of the questions · the shorter bar left fewer wrong answers
Wrong only with “I don't know” Wrong only when held back by its own confidence * A significant difference (Holm-adjusted p below 0.05): d1-ultra (p = 0.00059). The other 8: p = 0.1, p = 0.3, p = 0.38, p = 1, p = 1, p = 1, p = 1 and p = 1. Not in the test, as they chose “I don't know” on fewer than 1% of the questions: d3-ultra, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark and d3.
Hosted GLiDE against a new five-option run
Fastino changed its hosted model after 6 October, so run 129 (2026-10-06), the five-option run behind GLiDE · API's row in the table, is a different model from the one behind its “I don't know” run 351. A new five-option run, 517 (2026-11-02), scored 68.1% (1,497 of 2,198 right), against 67.9% for run 129. Held back by its own confidence on as many questions as run 351 said “I don't know” (521), the new run left 351 wrong answers, against 356 with “I don't know”: a derived non-hallucination rate of 59.7%, against 59.4% with the option and 58.0% from run 129 (exact McNemar test of the wrong answers, p = 0.76, not adjusted). The row in the table under Info is left as published, against run 129.
GLiDE · API: saying “I don't know”, or held back by confidence
Non-hallucination rate with “I don't know”, and with each five-option run held back by its own confidence at about the same attempt rate · higher is better
With “I don't know” Held back by its own confidence (the derived rate) Table and method: under Info.
The same number of answers
Each model's five-option run keeps only its k most confident answers, k being the number of questions d1-ultra answered with “I don't know” on offer, and we count the wrong ones among them. In Turkish, k = 1,582, the answers of d1-ultra's run 511 with “I don't know” on the 2,198 questions: d1-ultra's five-option run (run 199) keeps 167 wrong of 1,582 (10.6%); after Holm's correction over the other 126 setups, of the 76 other decision models, 4 (Copernicus-31B, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2 and Turion Jev 31B) keep significantly fewer, 2.8 to 4.6 points fewer, 67 keep significantly more, 5.8 to 72.9 points more and for 5 (REAX 1.3 with 151, Karpo Decision 31B with 146, Gemini Decision 1 · 20261111 with 156, Logits Sigma-1 Duo with 209 and Sieve 31B v3 with 183) the difference could be chance, 3 of them keeping fewer than 167 but not significantly; of the 50 chat and general models, 27 keep significantly fewer, 3.4 to 10.3 points fewer, 19 keep significantly more, 4.6 to 72.9 points more and for 4 (Kimi K3.1 Pro · low · chat with 141, Qwen3.5-397B-A17B · high · chat with 158, Inkling-Small · high · chat · Jul 2026 with 181 and GLM-5.3-Flash · reasoning on · chat with 163) the difference could be chance, 3 of them keeping fewer than 167 but not significantly. Qwen3.5-4B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Gemma 4 E4B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Claude Haiku 5.5 · max · chat is left out: its run has 1,414 scored answers, fewer than k = 1,582; Qwen3.5-4B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-2.6B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-8B-A1B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; MiniCPM5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by. In English, k = 496, the answers of d1-ultra's run 449 with “I don't know” on the 547 kept reading questions: d1-ultra's five-option run (run 411) keeps 29 wrong of 496 (5.8%); after Holm's correction over the other 123 models, 17 keep significantly fewer, 3.8 to 5.4 points fewer, 43 keep significantly more, 4.4 to 79.2 points more and for 63 the difference could be chance; 22 of them keep fewer than 29 but not significantly. Qwen3.5-4B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Gemma 4 E4B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-4B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-2.6B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-8B-A1B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; MiniCPM5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by.
The difference from d1-ultra is tested by a paired bootstrap over questions, with p adjusted by Holm's method in each language. Each plot shows every number of answers, not only k: the line is the five-option run held back by its own confidence, the markers the model's runs with the option. Tables and method: under Info.
The same number of answers in Turkish
The 2,198 questions in Turkish: each model's five-option run answering only its most confident answers (the line), and its own runs with the option (the marker: “Bilmiyorum” “I don't know”); the table under Info has all 137 setups, the plot the 63 with runs with the option (every setup's curve under Info) · lower is better at the same coverage
- k = 1,582: d1-ultra's answers with “I don't know”
- Copernicus-31B: five options, run 1165 · “Bilmiyorum”, run 1172
- KnowLine-31B-Gen1.5: five options, run 957 · “Bilmiyorum”, run 959
- KnowLine-31B-Gen2: five options, run 1314 · “Bilmiyorum”, run 1331
- Turion Jev 31B: five options, run 1273 · “Bilmiyorum”, run 1285
- REAX 1.3: five options, run 827 · “Bilmiyorum”, run 833
- Karpo Decision 31B: five options, run 1156 · “Bilmiyorum”, run 1163
- Gemini Decision 1 · 20261111: five options, run 1021 · “Bilmiyorum”, run 1023
- d1-ultra: five options, run 199 · “Bilmiyorum”, run 350 · “I don't know”, run 511
- Logits Sigma-1 Duo: five options, run 1221 · “Bilmiyorum”, run 1233
- Sieve 31B v3: five options, run 1241 · “Bilmiyorum”, run 1253
- GLiDE 27B: five options, run 899 · “Bilmiyorum”, run 905
- Perplexity Decider 27B v1.2 · API · with web: five options, run 640 · “Bilmiyorum”, run 642
- GLiDE · v1.2: five options, run 248 · “Bilmiyorum”, run 369 · “I don't know”, run 525
- Perplexity Decider 27B v1.3: five options, run 1134 · “Bilmiyorum”, run 1141
- d1.2-ultra: five options, run 1347 · “Bilmiyorum”, run 1367 · “I don't know”, run 1378
- GLiDE · API: five options, run 129 · “Bilmiyorum”, run 351 · “I don't know”, run 510 (runs 351 and 510 made with Fastino's engine glide_v2, not run 129's glide)
- d3: five options, run 259 · “Bilmiyorum”, run 358 · “I don't know”, run 544
- d3-ultra: five options, run 269 · “Bilmiyorum”, run 363 · “I don't know”, run 540
- Drex Max: five options, run 322 · “Bilmiyorum”, run 367 · “I don't know”, run 542
- Drex Max v1.1: five options, run 326 · “Bilmiyorum”, run 352 · “I don't know”, run 513
- Mammoth-1: five options, run 374 · “Bilmiyorum”, run 386 · “I don't know”, run 519
- Perplexity Decider 27B v1.2 · DGX Spark: five options, run 217 · “Bilmiyorum”, run 372 · “I don't know”, run 531
- Perplexity Decider 27B v1.2 · API: five options, run 633 · “Bilmiyorum”, run 641
- SubQ Snap 1.1: five options, run 346 · “Bilmiyorum”, run 392 · “I don't know”, run 512
- KnowLine-12B-Gen1: five options, run 1309 · “Bilmiyorum”, run 1318
- d1.2: five options, run 1346 · “Bilmiyorum”, run 1365 · “I don't know”, run 1377
- OneJev-27B: five options, run 264 · “Bilmiyorum”, run 782
- Clef: five options, run 125 · “Bilmiyorum”, run 788
- Jeffan-9B-1211: five options, run 1192 · “Bilmiyorum”, run 1199
- Cygnet: five options, run 127 · “Bilmiyorum”, run 757
- Open-Jev-27B-v1.1: five options, run 45 · “Bilmiyorum”, run 768
- Winnow-12B: five options, run 128 · “Bilmiyorum”, run 750
- KnowLine-9B-Gen1: five options, run 1174 · “Bilmiyorum”, run 1181
- Cerno 9B v1.2: five options, run 1201 · “Bilmiyorum”, run 1204
- KnowLine-9B-Gen2: five options, run 1306 · “Bilmiyorum”, run 1325
- Adjudex 9B v5: five options, run 1183 · “Bilmiyorum”, run 1190
- Elephant-1: five options, run 373 · “Bilmiyorum”, run 382 · “I don't know”, run 515
- GLiDE 9B: five options, run 907 · “Bilmiyorum”, run 913
- Dexy-Decision-9B-V2: five options, run 1255 · “Bilmiyorum”, run 1262
- StartLux-Decision-v1.1-9B-Instant: five options, run 1290 · “Bilmiyorum”, run 1297
- Perplexity Decider Flash v1.3: five options, run 1123 · “Bilmiyorum”, run 1130
- vera-medium-omni: five options, run 1264 · “Bilmiyorum”, run 1271
- KnowLine 4B Gen5.5: five options, run 965 · “Bilmiyorum”, run 967
- StartLux-Decision-v1.1-4B-Instant: five options, run 1236 · “Bilmiyorum”, run 1239
- Perplexity Decider Flash v1.2: five options, run 223 · “Bilmiyorum”, run 791
- Valen-Decision-1.5-9B: five options, run 1275 · “Bilmiyorum”, run 1282
- Verdin-4B-v1.1: five options, run 1244 · “Bilmiyorum”, run 1251
- Chinese Decision Pointer 4B: five options, run 1206 · “Bilmiyorum”, run 1213
- Clef-Flash: five options, run 139 · “Bilmiyorum”, run 664
- Metask-Jev-4B: five options, run 126 · “Bilmiyorum”, run 743
- GLiDE 2B: five options, run 915 · “Bilmiyorum”, run 921
- StartLux-Decision-v1.1-2B-Instant: five options, run 977 · “Bilmiyorum”, run 979
- d1-2B: five options, run 340 · “Bilmiyorum”, run 712
- GLiDE 0.8B: five options, run 923 · “Bilmiyorum”, run 929
- OneJev1.6-0.8B: five options, run 1029 · “Bilmiyorum”, run 1031
- StartLux-Decision-v1.1-0.8B-Instant: five options, run 1015 · “Bilmiyorum”, run 1017
- Strands Decider 2B: five options, run 141 · “Bilmiyorum”, run 706
- KnowLine-0.8B-Gen1: five options, run 1005 · “Bilmiyorum”, run 1007
- GLiClass v4 Edge: five options, run 1046 · “Bilmiyorum”, run 1048
- GLiDE 0.4B: five options, run 931 · “Bilmiyorum”, run 937
- GLiNER2.5-multi-Decide: five options, run 135 · “Bilmiyorum”, run 718
- Laya: five options, run 51 · “Bilmiyorum”, run 730
- Laya Multilingual: five options, run 46 · “Bilmiyorum”, run 737
The same number of answers in English
The 547 kept reading questions in English: each model's five-option run answering only its most confident answers (the line), and its own runs with the option (the marker: “I don't know”); not drawn: Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat and Qwen3.5-2B · reasoning off · chat, not measured (a plain answer line, no confidence) · lower is better at the same coverage
- k = 496: d1-ultra's answers with “I don't know”
- Claude Fable 5.2 · max · chat: five options, run 971
- Gemini 4 Hydrogen · high · chat: five options, run 1107
- GPT-6 Astra · low · chat: five options, run 616
- GPT-6 Astra · high · chat: five options, run 615
- GPT-6.1 Terra · high · chat: five options, run 620
- GPT-6.1 Terra · low · chat: five options, run 621
- Claude Fable 5.1 · high · chat: five options, run 580
- Gemini 3.8 Flash · high · chat: five options, run 583
- Gemini 4 Argon · low · chat: five options, run 593
- GPT-6 Astra · medium · chat: five options, run 624
- GPT-6.1 Sol · low · chat: five options, run 619
- GPT-6 Astra · low · chat · 1 per request: five options, run 622
- GPT-6.1 Sol · high · chat: five options, run 618
- Gemini 3.8 Flash · low · chat · 1 per request: five options, run 589
- GPT-6.1 Sol · low · chat · 1 per request: five options, run 623
- Claude Opus 5.5 · low · chat · 1 per request: five options, run 592
- Claude Opus 5.5 · high · chat: five options, run 587
- Qwen4-Max · high · chat: five options, run 1334
- Gemini 3.8 Flash · low · chat: five options, run 578
- Gemini 4 Carbon · high · chat: five options, run 1055
- KnowLine-31B-Gen1.5: five options, run 953 · “I don't know”, run 955
- Grok 4.7 · high · chat: five options, run 579
- Claude Opus 5.5 · low · chat: five options, run 588
- Copernicus-31B: five options, run 1168 · “I don't know”, run 1170
- Claude Haiku 5.5 · max · chat: five options, run 949
- DeepSeek-V4.1-Flash · reasoning on · chat · 1017: five options, run 808
- REAX 1.3: five options, run 829 · “I don't know”, run 831
- Kimi K3.1 Pro · low · chat: five options, run 584
- KnowLine-31B-Gen2: five options, run 1327 · “I don't know”, run 1329
- Karpo Decision 31B: five options, run 1159 · “I don't know”, run 1161
- Claude Sonnet 5.5 · high · chat · 1 per request: five options, run 594
- Kimi K3.1 Pro · no reasoning · chat: five options, run 582
- Qwen3.5-397B-A17B · high · chat: five options, run 871
- Gemini Decision 1 · 20261111: five options, run 1009 · “I don't know”, run 1019
- Logits Sigma-1 Duo: five options, run 1229 · “I don't know”, run 1231
- GLiDE 27B: five options, run 901 · “I don't know”, run 903
- d1-ultra: five options, run 411 · “I don't know”, run 449
- Sieve 31B v3: five options, run 1064 · “I don't know”, run 1066
- Turion Jev 31B: five options, run 1075 · “I don't know”, run 1077
- Gemma 4 31B · reasoning off · chat: five options, run 774
- Perplexity Decider 27B v1.2 · API · with web: five options, run 639 · “I don't know”, run 644
- GLiDE · v1.2: five options, run 521 · “I don't know”, run 523
- d1.2-ultra: five options, run 1359 · “I don't know”, run 1363
- Mammoth-1: five options, run 503 · “I don't know”, run 505
- Gemma 4 31B · reasoning off · chat · 1 per request: five options, run 783
- d3: five options, run 554 · “I don't know”, run 793
- Drex Max: five options, run 552 · “I don't know”, run 798
- GLM-5.3-Flash · reasoning on · chat: five options, run 823
- Muse Ember · max · chat: five options, run 942
- Perplexity Decider 27B v1.3: five options, run 1137 · “I don't know”, run 1139
- d3-ultra: five options, run 550 · “I don't know”, run 795
- d1.2: five options, run 1357 · “I don't know”, run 1361
- MiMo-V2.6-Flash · high · chat: five options, run 878
- Qwen3.8-27B · low · chat: five options, run 667
- Perplexity Decider 27B v1.2 · DGX Spark: five options, run 527 · “I don't know”, run 529
- Drex Max v1.1: five options, run 399 · “I don't know”, run 447
- Perplexity Decider 27B v1: five options, run 626
- Perplexity Decider 27B v1.2 · API: five options, run 637 · “I don't know”, run 643
- Inkling-Small · high · chat · Jul 2026: five options, run 867
- d1.1: five options, run 565
- GLM-5.3 · low · chat: five options, run 577
- Jeffan-9B-1211: five options, run 1195 · “I don't know”, run 1197
- GLM-5.3-Flash · reasoning off · chat: five options, run 820
- DeepSeek-V4.1-Flash · reasoning off · chat · 1017: five options, run 804
- Qwen3.8 Flash · no reasoning · chat: five options, run 986
- d1: five options, run 430 · “I don't know”, run 445
- StartLux-Decision-v1.1-9B-Instant: five options, run 1293 · “I don't know”, run 1295
- Clef: five options, run 613 · “I don't know”, run 785
- KnowLine-12B-Gen1: five options, run 1313 · “I don't know”, run 1316
- GLiDE 9B: five options, run 909 · “I don't know”, run 911
- GPT-6 Luna · low · chat: five options, run 617
- OneJev-27B: five options, run 778 · “I don't know”, run 780
- Claude Sonnet 5.5 · high · chat: five options, run 590
- Jev 1.14: five options, run 410 · “I don't know”, run 446
- Cerno 9B v1.2: five options, run 1085 · “I don't know”, run 1087
- Drex 2.0: five options, run 566
- Jev 1.13: five options, run 564
- KnowLine-9B-Gen1: five options, run 1177 · “I don't know”, run 1179
- SubQ Snap 1.1: five options, run 413 · “I don't know”, run 448
- SubQ Snap Preview: five options, run 595
- Adjudex 9B v5: five options, run 1186 · “I don't know”, run 1188
- Mercury Decide: five options, run 398 · “I don't know”, run 450
- Microsoft-Decision-1: five options, run 412 · “I don't know”, run 451
- Elephant-1: five options, run 499 · “I don't know”, run 501
- Open-Jev-27B-v1.1: five options, run 760 · “I don't know”, run 763
- KnowLine-9B-Gen2: five options, run 1321 · “I don't know”, run 1323
- Qwen3.8-27B · stated confidence · reasoning off · chat: five options, run 672
- Qwen3.8-27B · yes/no scoring: five options, run 696
- MiniMax-M3.1-Flash · high · chat: five options, run 863
- celeris-1-decision: five options, run 546 · “I don't know”, run 548
- GPT-6 Luna · Decisions API: five options, run 397 · “I don't know”, run 444
- Claude Sonnet 5.5 · low · chat: five options, run 591
- Cygnet: five options, run 753 · “I don't know”, run 755
- Qwen3.8-27B · vote share · reasoning off · chat: five options, run 678
- Winnow-12B: five options, run 746 · “I don't know”, run 748
- Perplexity Decider Flash v1.3: five options, run 1126 · “I don't know”, run 1128
- Valen-Decision-1.5-9B: five options, run 1278 · “I don't know”, run 1280
- Dexy-Decision-9B-V2: five options, run 1258 · “I don't know”, run 1260
- vera-medium-omni: five options, run 1267 · “I don't know”, run 1269
- StartLux-Decision-v1.1-4B-Instant: five options, run 1081 · “I don't know”, run 1083
- Clef-Flash: five options, run 614 · “I don't know”, run 660
- Verdin-4B-v1.1: five options, run 1247 · “I don't know”, run 1249
- Perplexity Decider Flash v1.2: five options, run 533 · “I don't know”, run 535
- KnowLine 4B Gen5.5: five options, run 961 · “I don't know”, run 963
- Chinese Decision Pointer 4B: five options, run 1209 · “I don't know”, run 1211
- Qwen3-14B · reasoning off · chat: five options, run 770
- Metask-Jev-4B: five options, run 739 · “I don't know”, run 741
- Erk-14B · reasoning off · chat: five options, run 762
- KnowLine-2B-Gen1: five options, run 1057 · “I don't know”, run 1059
- GLiDE 2B: five options, run 917 · “I don't know”, run 919
- StartLux-Decision-v1.1-2B-Instant: five options, run 973 · “I don't know”, run 975
- d1-2B: five options, run 708 · “I don't know”, run 710
- GLiDE 0.8B: five options, run 925 · “I don't know”, run 927
- StartLux-Decision-v1.1-0.8B-Instant: five options, run 1011 · “I don't know”, run 1013
- GLiDE 0.4B: five options, run 933 · “I don't know”, run 935
- OneJev1.6-0.8B: five options, run 1025 · “I don't know”, run 1027
- KnowLine-0.8B-Gen1: five options, run 1001 · “I don't know”, run 1003
- GLiClass v4 Edge: five options, run 1042 · “I don't know”, run 1044
- Decider 1: five options, run 596
- Strands Decider 2B: five options, run 702 · “I don't know”, run 704
- Laya: five options, run 726 · “I don't know”, run 728
- GLiNER2.5-multi-Decide: five options, run 714 · “I don't know”, run 716
- GLiNER2.5 Multi: five options, run 720
- Laya Multilingual: five options, run 732 · “I don't know”, run 734
Answers at 95%+ confidence
An answer counts when its chosen option has probability at least the cutoff; an answer without a score or without a valid choice never counts; wrong is a counted answer that is not right. Each decision model is counted on its Text tab run (its first five-option Turkish run, the run the site already uses), on the Text tab's 2,198 questions, keeping the answers whose chosen option has a probability of at least 95%. These runs have no “I don't know” option.
Every decision setup
Ordered by % wrong (wrong / answers kept), then by answers kept, more first; a setup that keeps no answers is last (No answers at this threshold: its % wrong is undefined).
| Model | Run | Answers kept (at 95%+) | Wrong | % wrong |
|---|---|---|---|---|
| d1-ultra Closed · Efficient | 199 | 549 | 0 | 0.0%0 of 549 |
| GLiDE 27B Open · 27B | 899 | 548 | 0 | 0.0%0 of 548 |
| Copernicus-31B Open · 31B | 1165 | 483 | 0 | 0.0%0 of 483 |
| KnowLine-31B-Gen2 Open · 31B | 1314 | 456 | 0 | 0.0%0 of 456 |
| d1.2-ultra Closed · Efficient | 1347 | 389 | 0 | 0.0%0 of 389 |
| Logits Sigma-1 Duo Open · 57B | 1221 | 351 | 0 | 0.0%0 of 351 |
| GLiDE API Closed · Efficient | 129 | 302 | 0 | 0.0%0 of 302 |
| Perplexity Decider 27B v1 Open · 27B | 44 | 293 | 0 | 0.0%0 of 293 |
| GLiDE 9B Open · 7.9B | 907 | 208 | 0 | 0.0%0 of 208 |
| Drex 2.0 Open · 35B MoE · 3.0B active | 334 | 180 | 0 | 0.0%0 of 180 |
| OneJev-27B Open · 27B | 264 | 166 | 0 | 0.0%0 of 166 |
| celeris-1-decision Closed · Efficient | 150 | 123 | 0 | 0.0%0 of 123 |
| KnowLine-12B-Gen1 Open · 12B | 1309 | 15 | 0 | 0.0%0 of 15 |
| KnowLine-9B-Gen2 Open · 9.0B | 1306 | 10 | 0 | 0.0%0 of 10 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 977 | 7 | 0 | 0.0%0 of 7 |
| Jeffan-9B-1211 Open · 9.0B | 1192 | 6 | 0 | 0.0%0 of 6 |
| d1-2B Open · 1.9B | 340 | 4 | 0 | 0.0%0 of 4 |
| GLiClass v4 Edge Open · 0.4B | 1046 | 2 | 0 | 0.0%0 of 2 |
| Decider 1 Closed · Efficient | 243 | 1 | 0 | 0.0%0 of 1 |
| REAX 1.3 Open · 57B | 827 | 657 | 1 | 0.2%1 of 657 |
| KnowLine-31B-Gen1.5 Open · 31B | 957 | 628 | 1 | 0.2%1 of 628 |
| SubQ Snap Preview Closed · Efficient | 177 | 339 | 1 | 0.3%1 of 339 |
| Turion Jev 31B Open · 31B | 1273 | 304 | 1 | 0.3%1 of 304 |
| d1.2 Closed · Efficient | 1346 | 269 | 1 | 0.4%1 of 269 |
| GLiDE v1.2 Open · 27B | 248 | 485 | 2 | 0.4%2 of 485 |
| d1.1 Closed · Efficient | 254 | 336 | 2 | 0.6%2 of 336 |
| Drex Max v1.1 Closed · Efficient (provisional) | 326 | 493 | 3 | 0.6%3 of 493 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 223 | 154 | 1 | 0.6%1 of 154 |
| Gemini Decision 1 20261111 Closed · Efficient | 1021 | 766 | 6 | 0.8%6 of 766 |
| d3 Open · 26B | 259 | 503 | 4 | 0.8%4 of 503 |
| Perplexity Decider 27B v1.3 Open · 27B | 1134 | 611 | 5 | 0.8%5 of 611 |
| Karpo Decision 31B Open · 31B | 1156 | 870 | 8 | 0.9%8 of 870 |
| Perplexity Decider 27B v1.1 Open · 27B | 145 | 589 | 6 | 1.0%6 of 589 |
| Jev 1.14 Closed · Efficient | 194 | 390 | 4 | 1.0%4 of 390 |
| Valen-Decision-1.5-9B Open · 9.0B | 1275 | 92 | 1 | 1.1%1 of 92 |
| Sieve 31B v3 Open · 31B | 1241 | 727 | 8 | 1.1%8 of 727 |
| Cygnet Open · 12B | 127 | 335 | 4 | 1.2%4 of 335 |
| Mammoth-1 Open · 27B | 374 | 667 | 9 | 1.3%9 of 667 |
| vera-medium-omni Open · 4.2B | 1264 | 72 | 1 | 1.4%1 of 72 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 640 | 641 | 9 | 1.4%9 of 641 |
| Clef Open · 27B | 125 | 348 | 5 | 1.4%5 of 348 |
| Perplexity Decider 27B v1.2 API Open · 27B | 633 | 610 | 10 | 1.6%10 of 610 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 217 | 633 | 11 | 1.7%11 of 633 |
| Jev 1.13 Closed · Efficient | 49 | 440 | 8 | 1.8%8 of 440 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 1236 | 156 | 3 | 1.9%3 of 156 |
| KnowLine-9B-Gen1 Open · 9.0B | 1174 | 300 | 6 | 2.0%6 of 300 |
| Dexy-Decision-9B-V2 Open · 9.0B | 1255 | 293 | 6 | 2.0%6 of 293 |
| Cerno 9B v1.2 Open · 9.0B | 1201 | 139 | 3 | 2.2%3 of 139 |
| KnowLine 4B Gen5.5 Open · 4.0B | 965 | 274 | 6 | 2.2%6 of 274 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 1290 | 219 | 5 | 2.3%5 of 219 |
| SubQ Snap 1.1 Closed · Efficient | 346 | 665 | 18 | 2.7%18 of 665 |
| GPT-6 Luna Decisions API Closed · Efficient | 143 | 542 | 15 | 2.8%15 of 542 |
| Metask-Jev-4B Open · 4.5B | 126 | 32 | 1 | 3.1%1 of 32 |
| Perplexity Decider Flash v1.3 Open · 4.0B | 1123 | 246 | 8 | 3.3%8 of 246 |
| Drex Max Open · 27B | 322 | 736 | 24 | 3.3%24 of 736 |
| Elephant-1 Open · 9.0B | 373 | 393 | 13 | 3.3%13 of 393 |
| d3-ultra Open · 26B | 269 | 833 | 30 | 3.6%30 of 833 |
| Adjudex 9B v5 Open · 9.0B | 1183 | 357 | 13 | 3.6%13 of 357 |
| KnowLine-2B-Gen1 Open · 2.0B | 1079 | 26 | 1 | 3.8%1 of 26 |
| d1 Closed · Efficient | 52 | 502 | 20 | 4.0%20 of 502 |
| Verdin-4B-v1.1 Open · 4.0B | 1244 | 202 | 11 | 5.4%11 of 202 |
| Clef-Flash Open · 9.0B | 139 | 56 | 4 | 7.1%4 of 56 |
| Chinese Decision Pointer 4B Open · 4.0B | 1206 | 193 | 14 | 7.3%14 of 193 |
| Microsoft-Decision-1 Closed · Efficient | 205 | 378 | 29 | 7.7%29 of 378 |
| Winnow-12B Open · 12B | 128 | 641 | 55 | 8.6%55 of 641 |
| Mercury Decide Closed · Efficient | 204 | 775 | 67 | 8.6%67 of 775 |
| Strands Decider 2B Open · 2.0B | 141 | 10 | 1 | 10.0%1 of 10 |
| GLiDE 2B Open · 1.9B | 915 | 101 | 11 | 10.9%11 of 101 |
| GLiDE 0.8B Open · 0.8B | 923 | 85 | 17 | 20.0%17 of 85 |
| OneJev1.6-0.8B Open · 0.8B | 1029 | 7 | 2 | 28.6%2 of 7 |
| KnowLine-0.8B-Gen1 Open · 0.8B | 1005 | 2 | 1 | 50.0%1 of 2 |
| Laya Multilingual Open · 0.3B | 46 | 14 | 11 | 78.6%11 of 14 |
| GLiDE 0.4B Open · 0.4B | 931 | No answers at this threshold | – | – |
| GLiNER2.5-multi-Decide Open · 0.3B | 135 | No answers at this threshold | – | – |
| Laya Open · 0.4B | 51 | No answers at this threshold | – | – |
| Open-Jev-27B-v1.1 Open · 27B | 45 | No answers at this threshold | – | – |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 1015 | No answers at this threshold | – | – |
d1-ultra's repeat runs
The same questions and settings, run again; run 199 is the Text tab run above, shown for context.
| Run | Answers kept | Wrong | % wrong |
|---|---|---|---|
| d1-ultra Closed · Efficientrun 199, Text tab run | 549 | 0 | 0.0%0 of 549 |
| d1-ultra Closed · Efficientrun 200, repeat run | 538 | 0 | 0.0%0 of 538 |
| d1-ultra Closed · Efficientrun 201, repeat run | 542 | 0 | 0.0%0 of 542 |
| d1-ultra Closed · Efficientrun 202, repeat run | 534 | 0 | 0.0%0 of 534 |
| d1-ultra Closed · Efficientrun 203, repeat run | 550 | 0 | 0.0%0 of 550 |
The three models Artificial Analysis ties at 65
The three models its Decision Index v1.0 puts level at 65 (its post), each with its Text tab run and its repeat runs, one row per run. Each column is a cutoff: % wrong of the answers kept at that probability or higher, with wrong of kept below.
| Model and run | 50% | 70% | 80% | 90% | 95% | 99% |
|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficientrun 199, Text tab run | 13.1%224 of 1,712 | 7.4%99 of 1,335 | 4.2%46 of 1,094 | 1.0%8 of 802 | 0.0%0 of 549 | 0.0%0 of 90 |
| d1-ultra Closed · Efficientrun 200, repeat run | 13.4%229 of 1,705 | 7.0%93 of 1,320 | 3.6%39 of 1,089 | 0.8%6 of 795 | 0.0%0 of 538 | 0.0%0 of 86 |
| d1-ultra Closed · Efficientrun 201, repeat run | 14.0%241 of 1,719 | 7.3%96 of 1,313 | 4.3%47 of 1,091 | 1.3%10 of 794 | 0.0%0 of 542 | 0.0%0 of 96 |
| d1-ultra Closed · Efficientrun 202, repeat run | 13.3%228 of 1,713 | 6.4%84 of 1,315 | 4.5%49 of 1,088 | 1.0%8 of 796 | 0.0%0 of 534 | 0.0%0 of 91 |
| d1-ultra Closed · Efficientrun 203, repeat run | 13.7%233 of 1,706 | 7.3%97 of 1,324 | 4.0%43 of 1,080 | 1.1%9 of 803 | 0.0%0 of 550 | 0.0%0 of 90 |
| SubQ Snap 1.1 Closed · Efficientrun 346, Text tab run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| SubQ Snap 1.1 Closed · Efficientrun 387, repeat run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| SubQ Snap 1.1 Closed · Efficientrun 388, repeat run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| SubQ Snap 1.1 Closed · Efficientrun 389, repeat run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| SubQ Snap 1.1 Closed · Efficientrun 390, repeat run | 27.9%499 of 1,791 | 17.3%229 of 1,323 | 12.4%138 of 1,117 | 6.2%52 of 839 | 2.7%18 of 665 | 0.3%1 of 299 |
| Elephant-1 Open · 9.0Brun 373, Text tab run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
| Elephant-1 Open · 9.0Brun 375, repeat run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
| Elephant-1 Open · 9.0Brun 376, repeat run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
| Elephant-1 Open · 9.0Brun 378, repeat run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
| Elephant-1 Open · 9.0Brun 379, repeat run | 33.2%455 of 1,370 | 21.3%195 of 917 | 14.0%102 of 726 | 6.9%35 of 510 | 3.3%13 of 393 | 0.0%0 of 155 |
At every cutoff up to 98% (each whole percent), d1-ultra's worst run has a lower % wrong than the best run of SubQ Snap 1.1 and of Elephant-1; at 99%, all three are at or near 0 (highest: 0.3%).
In English (the 547 kept reading questions of the Turkish vs English tab), at 95%, d1-ultra has 1 wrong answer in each of its 5 runs, always question 280; each run keeps 230 to 237 answers (runs 411, 418, 422, 426 and 428).
Info The table, what we found and our method
About this test
Every question of the Text tab gets one more option after its five answers: F, “I don't know”, written in Turkish in the prompt (“Bilmiyorum”). It is always the last option, it is never shuffled and it is never correct. The System One instructions get one added sentence: “If you do not know the answer, choose Bilmiyorum.” A chat model's instructions end with its own sentence instead: “Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.” Everything else is as on the Text tab: the same 2,198 scored questions and answer key, and each model's own settings from its text run, its questions per request too (one for every decision model).
The models: the ten most accurate decision models on the Text tab at results 25290875, plus Base Labs' Elephant-1 and Mammoth-1 and SubQ Snap 1.1 once their text runs are published. With them, every chat model setup of the Text tab as its run finishes: 56 so far, one still running in Turkish. Each has one full run; a model whose repeated text runs changed answers gets five.
Download: results.json (every number on this page, under dont_know) and answers.json (each run's answers, under dont_know_runs).
Results
The rows are ranked by the first column, the non-hallucination rate, highest first: the share of “I don't know” among the answers that were not right (“I don't know” / (“I don't know” + wrong)), so the first model is the one that most often said it did not know instead of answering wrongly. Beside it: Answered, how often the model chose one of the question's own options (right + wrong, out of all 2,198 questions), and the error rate, the share of those answers that were wrong (wrong / (right + wrong)). Read the ranking with Answered, the error rate and accuracy: a model that always chose “I don't know” would rank first with no right answers. Each row shows the model's first run; under Runs, a model with more than one run (d1-ultra and GLiDE · API) shows the range over its runs and how many answers changed between them. Out of all 2,198 questions: accuracy counts the right answers; Answered (the attempt rate) counts the right and wrong answers. Formulas and how to read them: Our method.
Decision models
| Model | Non-hallucinationof answers not right, % “I don't know” · higher is better | Answerednot “I don't know”: (right + wrong) / 2,198 | Error rateof its answers: wrong / (right + wrong) · lower is better | Derived non-hallucination ratefrom the 5-option text run; higher is better | Accuracyhigher is better | Mean probability of “I don't know”higher means more probability on the option; neither better nor worse on its own | Text accuracy with five optionsthe Text tab's number; higher is better | Runsrange over the runs where more than one |
|---|---|---|---|---|---|---|---|---|
| GLiDE 0.4B Open · 0.4B | 95.3%2,070 “I don't know” of 2,171 not right | 5.8%128 of 2,198 answered, 2,070 “I don't know” | 78.9%101 of 128 | 95.4%cut-off 0.4443: 29 right + 99 wrong answered, 2,070 held back, attempt rate 5.8% | 1.2%27 of 2,198 right | 39.1% | 22.2%run 931 | run 937 |
| d1.2-ultra Closed · Efficient | 87.3%996 “I don't know” of 1,141 not right | 54.7%1,202 of 2,198 answered, 996 “I don't know” | 12.1%145 of 1,202 | 87.8%cut-off 0.5947: 1,064 right + 138 wrong answered, 996 held back, attempt rate 54.7% | 48.1%1,057 of 2,198 right | 34.6% | 68.1%run 1347 | run 1367 |
| d1.2 Closed · Efficient | 79.4%973 “I don't know” of 1,226 not right | 55.7%1,225 of 2,198 answered, 973 “I don't know” | 20.7%253 of 1,225 | 81.3%cut-off 0.4936: 1,001 right + 224 wrong answered, 973 held back, attempt rate 55.7% | 44.2%972 of 2,198 right | 31.5% | 61.7%run 1346 | run 1365 |
| d1-ultra Closed · Efficient | 75.1%617 “I don't know” of 822 not right | 71.9%1,581 of 2,198 answered, 617 “I don't know” | 13.0%205 of 1,581 | 78.7%cut-off 0.5632: 1,414 right + 167 wrong answered, 617 held back, attempt rate 71.9% | 62.6%1,376 of 2,198 right | 30.7% | 77.1%run 199 | runs 350, 354, 360, 365, 370Answered 71.9% to 74.0%Non-hallucination 71.2% to 75.1%Accuracy 62.6% to 63.5%Mean probability 30.6% to 30.8%518 answers changed between runs |
| Mammoth-1 Open · 27B | 68.7%690 “I don't know” of 1,004 not right | 68.6%1,508 of 2,198 answered, 690 “I don't know” | 20.8%314 of 1,508 | 70.9%cut-off 0.5697: 1,225 right + 283 wrong answered, 690 held back, attempt rate 68.6% | 54.3%1,194 of 2,198 right | 24.5% | 67.1%run 374 | run 386 |
| d1-2B Open · 1.9B | 65.5%1,209 “I don't know” of 1,846 not right | 45.0%989 of 2,198 answered, 1,209 “I don't know” | 64.4%637 of 989 | 66.5%cut-off 0.3129: 381 right + 608 wrong answered, 1,209 held back, attempt rate 45.0% | 16.0%352 of 2,198 right | 26.3% | 29.4%run 340 | run 712 |
| GLiDE API Closed · Efficient | 59.4%521 “I don't know” of 877 not right | 76.3%1,677 of 2,198 answered, 521 “I don't know” | 21.2%356 of 1,677 | 58.0%cut-off 0.5206: 1,300 right + 377 wrong answered, 521 held back, attempt rate 76.3% | 60.1%1,321 of 2,198 right | 22.0% | 67.9%run 129 | runs 351, 353, 357, 361, 364Answered 75.5% to 76.3%Non-hallucination 59.4% to 59.9%Accuracy 59.1% to 60.1%Mean probability 21.6% to 22.0%189 answers changed between runs |
| SubQ Snap 1.1 Closed · Efficient | 47.1%437 “I don't know” of 927 not right | 80.1%1,761 of 2,198 answered, 437 “I don't know” | 27.8%490 of 1,761 | 47.6%cut-off 0.5124: 1,280 right + 481 wrong answered, 437 held back, attempt rate 80.1% | 57.8%1,271 of 2,198 right | 20.1% | 63.9%run 346 | run 392 |
| OneJev1.6-0.8B Open · 0.8B | 43.4%754 “I don't know” of 1,739 not right | 65.7%1,444 of 2,198 answered, 754 “I don't know” | 68.2%985 of 1,444 | 43.5%cut-off 0.2972: 463 right + 981 wrong answered, 754 held back, attempt rate 65.7% | 20.9%459 of 2,198 right | 22.2% | 29.1%run 1029 | run 1031 |
| GLiDE 27B Open · 27B | 39.8%291 “I don't know” of 732 not right | 86.8%1,907 of 2,198 answered, 291 “I don't know” | 23.1%441 of 1,907 | 39.9%cut-off 0.4133: 1,469 right + 438 wrong answered, 291 held back, attempt rate 86.8% | 66.7%1,466 of 2,198 right | 16.8% | 71.1%run 899 | run 905 |
| GLiDE v1.2 Open · 27B | 39.5%321 “I don't know” of 812 not right | 85.4%1,877 of 2,198 answered, 321 “I don't know” | 26.2%491 of 1,877 | 40.4%cut-off 0.4134: 1,404 right + 473 wrong answered, 321 held back, attempt rate 85.4% | 63.1%1,386 of 2,198 right | 18.6% | 68.5%run 248 | run 369 |
| GLiDE 2B Open · 1.9B | 38.5%612 “I don't know” of 1,590 not right | 72.2%1,586 of 2,198 answered, 612 “I don't know” | 61.7%978 of 1,586 | 38.7%cut-off 0.4078: 618 right + 968 wrong answered, 612 held back, attempt rate 72.2% | 27.7%608 of 2,198 right | 23.5% | 35.3%run 915 | run 921 |
| Drex Max v1.1 Closed · Efficient (provisional) | 38.2%320 “I don't know” of 837 not right | 85.4%1,878 of 2,198 answered, 320 “I don't know” | 27.5%517 of 1,878 | 39.3%cut-off 0.3806: 1,383 right + 495 wrong answered, 320 held back, attempt rate 85.4% | 61.9%1,361 of 2,198 right | 17.0% | 67.4%run 326 | run 352 |
| Elephant-1 Open · 9.0B | 34.8%405 “I don't know” of 1,163 not right | 81.6%1,793 of 2,198 answered, 405 “I don't know” | 42.3%758 of 1,793 | 35.5%cut-off 0.3883: 1,058 right + 735 wrong answered, 405 held back, attempt rate 81.6% | 47.1%1,035 of 2,198 right | 16.0% | 52.4%run 373 | run 382 |
| GLiDE 0.8B Open · 0.8B | 29.2%480 “I don't know” of 1,643 not right | 78.2%1,718 of 2,198 answered, 480 “I don't know” | 67.7%1,163 of 1,718 | 28.9%cut-off 0.3650: 536 right + 1,182 wrong answered, 480 held back, attempt rate 78.2% | 25.3%555 of 2,198 right | 19.9% | 29.3%run 923 | run 929 |
| Laya Open · 0.4B | 28.3%536 “I don't know” of 1,891 not right | 75.6%1,662 of 2,198 answered, 536 “I don't know” | 81.5%1,355 of 1,662 | 28.5%cut-off 0.2527: 316 right + 1,345 wrong answered, 537 held back, attempt rate 75.6% | 14.0%307 of 2,198 right | 18.6% | 19.0%run 51 | run 730 |
| GLiDE 9B Open · 7.9B | 24.1%277 “I don't know” of 1,149 not right | 87.4%1,921 of 2,198 answered, 277 “I don't know” | 45.4%872 of 1,921 | 24.8%cut-off 0.3391: 1,081 right + 840 wrong answered, 277 held back, attempt rate 87.4% | 47.7%1,049 of 2,198 right | 15.2% | 52.0%run 907 | run 913 |
| GLiClass v4 Edge Open · 0.4B | 22.3%398 “I don't know” of 1,784 not right | 81.9%1,800 of 2,198 answered, 398 “I don't know” | 77.0%1,386 of 1,800 | 22.6%cut-off 0.2423: 435 right + 1,365 wrong answered, 398 held back, attempt rate 81.9% | 18.8%414 of 2,198 right | 14.5% | 23.3%run 1046 | run 1048 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 21.5%333 “I don't know” of 1,546 not right | 84.8%1,865 of 2,198 answered, 333 “I don't know” | 65.0%1,213 of 1,865 | 21.8%cut-off 0.2682: 669 right + 1,196 wrong answered, 333 held back, attempt rate 84.8% | 29.7%652 of 2,198 right | 13.2% | 34.3%run 977 | run 979 |
| Laya Multilingual Open · 0.3B | 14.8%280 “I don't know” of 1,888 not right | 87.3%1,918 of 2,198 answered, 280 “I don't know” | 83.8%1,608 of 1,918 | 15.0%cut-off 0.2794: 331 right + 1,587 wrong answered, 280 held back, attempt rate 87.3% | 14.1%310 of 2,198 right | 14.6% | 17.9%run 46 | run 737 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 14.7%180 “I don't know” of 1,224 not right | 91.8%2,018 of 2,198 answered, 180 “I don't know” | 51.7%1,044 of 2,018 | 15.2%cut-off 0.2883: 1,014 right + 1,004 wrong answered, 180 held back, attempt rate 91.8% | 44.3%974 of 2,198 right | 12.5% | 47.8%run 223 | run 791 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 12.4%199 “I don't know” of 1,608 not right | 90.9%1,999 of 2,198 answered, 199 “I don't know” | 70.5%1,409 of 1,999 | 12.3%cut-off 0.2318: 585 right + 1,414 wrong answered, 199 held back, attempt rate 90.9% | 26.8%590 of 2,198 right | 9.4% | 28.6%run 1015 | run 1017 |
| Chinese Decision Pointer 4B Open · 4.0B | 11.2%136 “I don't know” of 1,218 not right | 93.8%2,062 of 2,198 answered, 136 “I don't know” | 52.5%1,082 of 2,062 | 11.3%cut-off 0.3048: 994 right + 1,068 wrong answered, 136 held back, attempt rate 93.8% | 44.6%980 of 2,198 right | 6.8% | 46.7%run 1206 | run 1213 |
| Strands Decider 2B Open · 2.0B | 9.2%150 “I don't know” of 1,633 not right | 93.2%2,048 of 2,198 answered, 150 “I don't know” | 72.4%1,483 of 2,048 | 9.2%cut-off 0.2554: 574 right + 1,474 wrong answered, 150 held back, attempt rate 93.2% | 25.7%565 of 2,198 right | 8.8% | 27.5%run 141 | run 706 |
| Clef-Flash Open · 9.0B | 9.0%110 “I don't know” of 1,218 not right | 95.0%2,088 of 2,198 answered, 110 “I don't know” | 53.1%1,108 of 2,088 | 9.0%cut-off 0.2929: 981 right + 1,107 wrong answered, 110 held back, attempt rate 95.0% | 44.6%980 of 2,198 right | 9.1% | 45.7%run 139 | run 664 |
| vera-medium-omni Open · 4.2B | 8.9%105 “I don't know” of 1,180 not right | 95.2%2,093 of 2,198 answered, 105 “I don't know” | 51.4%1,075 of 2,093 | 9.2%cut-off 0.2717: 1,055 right + 1,038 wrong answered, 105 held back, attempt rate 95.2% | 46.3%1,018 of 2,198 right | 9.1% | 49.1%run 1264 | run 1271 |
| Clef Open · 27B | 7.4%71 “I don't know” of 965 not right | 96.8%2,127 of 2,198 answered, 71 “I don't know” | 42.0%894 of 2,127 | 7.4%cut-off 0.3189: 1,238 right + 889 wrong answered, 71 held back, attempt rate 96.8% | 56.1%1,233 of 2,198 right | 10.1% | 57.3%run 125 | run 788 |
| Perplexity Decider Flash v1.3 Open · 4.0B | 6.5%75 “I don't know” of 1,146 not right | 96.6%2,123 of 2,198 answered, 75 “I don't know” | 50.4%1,071 of 2,123 | 6.7%cut-off 0.2769: 1,074 right + 1,049 wrong answered, 75 held back, attempt rate 96.6% | 47.9%1,052 of 2,198 right | 9.1% | 49.7%run 1123 | run 1130 |
| KnowLine-0.8B-Gen1 Open · 0.8B | 6.4%104 “I don't know” of 1,619 not right | 95.3%2,094 of 2,198 answered, 104 “I don't know” | 72.3%1,515 of 2,094 | 6.4%cut-off 0.2348: 578 right + 1,516 wrong answered, 104 held back, attempt rate 95.3% | 26.3%579 of 2,198 right | 8.1% | 27.1%run 1005 | run 1007 |
| GLiNER2.5-multi-Decide Open · 0.3B | 6.4%114 “I don't know” of 1,780 not right | 94.8%2,084 of 2,198 answered, 114 “I don't know” | 79.9%1,666 of 2,084 | 6.4%cut-off 0.2553: 426 right + 1,658 wrong answered, 114 held back, attempt rate 94.8% | 19.0%418 of 2,198 right | 10.7% | 20.2%run 135 | run 718 |
| KnowLine 4B Gen5.5 Open · 4.0B | 4.7%55 “I don't know” of 1,176 not right | 97.5%2,143 of 2,198 answered, 55 “I don't know” | 52.3%1,121 of 2,143 | 4.8%cut-off 0.2688: 1,051 right + 1,092 wrong answered, 55 held back, attempt rate 97.5% | 46.5%1,022 of 2,198 right | 2.8% | 48.5%run 965 | run 967 |
| Jeffan-9B-1211 Open · 9.0B | 4.3%41 “I don't know” of 958 not right | 98.1%2,157 of 2,198 answered, 41 “I don't know” | 42.5%917 of 2,157 | 4.3%cut-off 0.2414: 1,248 right + 909 wrong answered, 41 held back, attempt rate 98.1% | 56.4%1,240 of 2,198 right | 5.8% | 57.0%run 1192 | run 1199 |
| Winnow-12B Open · 12B | 4.3%43 “I don't know” of 1,008 not right | 98.0%2,155 of 2,198 answered, 43 “I don't know” | 44.8%965 of 2,155 | 4.3%cut-off 0.2923: 1,209 right + 946 wrong answered, 43 held back, attempt rate 98.0% | 54.1%1,190 of 2,198 right | 3.5% | 55.5%run 128 | run 750 |
| Dexy-Decision-9B-V2 Open · 9.0B | 3.4%37 “I don't know” of 1,083 not right | 98.3%2,161 of 2,198 answered, 37 “I don't know” | 48.4%1,046 of 2,161 | 3.4%cut-off 0.2579: 1,122 right + 1,039 wrong answered, 37 held back, attempt rate 98.3% | 50.7%1,115 of 2,198 right | 2.7% | 51.6%run 1255 | run 1262 |
| Turion Jev 31B Open · 31B | 3.3%15 “I don't know” of 451 not right | 99.3%2,183 of 2,198 answered, 15 “I don't know” | 20.0%436 of 2,183 | 3.4%cut-off 0.2468: 1,755 right + 428 wrong answered, 15 held back, attempt rate 99.3% | 79.5%1,747 of 2,198 right | 3.1% | 80.1%run 1273 | run 1285 |
| Cygnet Open · 12B | 3.2%32 “I don't know” of 989 not right | 98.5%2,166 of 2,198 answered, 32 “I don't know” | 44.2%957 of 2,166 | 3.3%cut-off 0.2955: 1,217 right + 949 wrong answered, 32 held back, attempt rate 98.5% | 55.0%1,209 of 2,198 right | 5.1% | 55.7%run 127 | run 757 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 3.2%23 “I don't know” of 725 not right | 99.0%2,175 of 2,198 answered, 23 “I don't know” | 32.3%702 of 2,175 | 3.3%cut-off 0.2556: 1,505 right + 670 wrong answered, 23 held back, attempt rate 99.0% | 67.0%1,473 of 2,198 right | 5.6% | 68.7%run 640 | run 642 |
| KnowLine-9B-Gen2 Open · 9.0B | 2.6%27 “I don't know” of 1,057 not right | 98.8%2,171 of 2,198 answered, 27 “I don't know” | 47.4%1,030 of 2,171 | 2.6%cut-off 0.2371: 1,162 right + 1,009 wrong answered, 27 held back, attempt rate 98.8% | 51.9%1,141 of 2,198 right | 6.2% | 53.3%run 1306 | run 1325 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 2.2%24 “I don't know” of 1,093 not right | 98.9%2,174 of 2,198 answered, 24 “I don't know” | 49.2%1,069 of 2,174 | 2.2%cut-off 0.2431: 1,124 right + 1,050 wrong answered, 24 held back, attempt rate 98.9% | 50.3%1,105 of 2,198 right | 4.3% | 51.2%run 1290 | run 1297 |
| Gemini Decision 1 20261111 Closed · Efficient | 1.9%10 “I don't know” of 515 not right | 99.5%2,188 of 2,198 answered, 10 “I don't know” | 23.1%505 of 2,188 | 2.0%cut-off 0.2613: 1,702 right + 486 wrong answered, 10 held back, attempt rate 99.5% | 76.6%1,683 of 2,198 right | 5.5% | 77.5%run 1021 | run 1023 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 1.8%21 “I don't know” of 1,168 not right | 99.0%2,177 of 2,198 answered, 21 “I don't know” | 52.7%1,147 of 2,177 | 1.8%cut-off 0.2403: 1,048 right + 1,129 wrong answered, 21 held back, attempt rate 99.0% | 46.9%1,030 of 2,198 right | 5.5% | 47.9%run 1236 | run 1239 |
| Open-Jev-27B-v1.1 Open · 27B | 1.4%14 “I don't know” of 979 not right | 99.4%2,184 of 2,198 answered, 14 “I don't know” | 44.2%965 of 2,184 | 1.4%cut-off 0.2077: 1,222 right + 962 wrong answered, 14 held back, attempt rate 99.4% | 55.5%1,219 of 2,198 right | 12.7% | 55.7%run 45 | run 768 |
| d3-ultra Open · 26B | 1.4%10 “I don't know” of 714 not right | 99.5%2,188 of 2,198 answered, 10 “I don't know” | 32.2%704 of 2,188 | 1.4%cut-off 0.2876: 1,486 right + 702 wrong answered, 10 held back, attempt rate 99.5% | 67.5%1,484 of 2,198 right | 2.5% | 67.8%run 269 | run 363 |
| Copernicus-31B Open · 31B | 1.2%5 “I don't know” of 407 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 18.3%402 of 2,193 | 1.2%cut-off 0.2499: 1,796 right + 397 wrong answered, 5 held back, attempt rate 99.8% | 81.5%1,791 of 2,198 right | 2.0% | 81.8%run 1165 | run 1172 |
| Adjudex 9B v5 Open · 9.0B | 0.8%8 “I don't know” of 1,029 not right | 99.6%2,190 of 2,198 answered, 8 “I don't know” | 46.6%1,021 of 2,190 | 0.8%cut-off 0.2339: 1,167 right + 1,023 wrong answered, 8 held back, attempt rate 99.6% | 53.2%1,169 of 2,198 right | 1.6% | 53.1%run 1183 | run 1190 |
| Metask-Jev-4B Open · 4.5B | 0.8%10 “I don't know” of 1,325 not right | 99.5%2,188 of 2,198 answered, 10 “I don't know” | 60.1%1,315 of 2,188 | 0.8%cut-off 0.2132: 892 right + 1,296 wrong answered, 10 held back, attempt rate 99.5% | 39.7%873 of 2,198 right | 6.3% | 40.8%run 126 | run 743 |
| Perplexity Decider 27B v1.3 Open · 27B | 0.7%5 “I don't know” of 716 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 32.4%711 of 2,193 | 0.7%cut-off 0.2424: 1,504 right + 689 wrong answered, 5 held back, attempt rate 99.8% | 67.4%1,482 of 2,198 right | 3.2% | 68.4%run 1134 | run 1141 |
| Karpo Decision 31B Open · 31B | 0.6%3 “I don't know” of 478 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 21.6%475 of 2,195 | 0.6%cut-off 0.2622: 1,710 right + 485 wrong answered, 3 held back, attempt rate 99.9% | 78.3%1,720 of 2,198 right | 1.0% | 77.8%run 1156 | run 1163 |
| KnowLine-9B-Gen1 Open · 9.0B | 0.5%5 “I don't know” of 1,001 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 45.4%996 of 2,193 | 0.5%cut-off 0.2281: 1,198 right + 995 wrong answered, 5 held back, attempt rate 99.8% | 54.5%1,197 of 2,198 right | 2.1% | 54.5%run 1174 | run 1181 |
| Verdin-4B-v1.1 Open · 4.0B | 0.4%5 “I don't know” of 1,174 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 53.3%1,169 of 2,193 | 0.4%cut-off 0.2269: 1,026 right + 1,167 wrong answered, 5 held back, attempt rate 99.8% | 46.6%1,024 of 2,198 right | 3.2% | 46.8%run 1244 | run 1251 |
| REAX 1.3 Open · 57B | 0.4%2 “I don't know” of 477 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 21.6%475 of 2,196 | 0.4%cut-off 0.2462: 1,712 right + 484 wrong answered, 2 held back, attempt rate 99.9% | 78.3%1,721 of 2,198 right | 1.9% | 77.9%run 827 | run 833 |
| Perplexity Decider 27B v1.2 API Open · 27B | 0.4%3 “I don't know” of 729 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 33.1%726 of 2,195 | 0.4%cut-off 0.2252: 1,457 right + 738 wrong answered, 3 held back, attempt rate 99.9% | 66.8%1,469 of 2,198 right | 2.7% | 66.3%run 633 | run 641 |
| Logits Sigma-1 Duo Open · 57B | 0.4%2 “I don't know” of 545 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 24.7%543 of 2,196 | 0.4%cut-off 0.2460: 1,649 right + 547 wrong answered, 2 held back, attempt rate 99.9% | 75.2%1,653 of 2,198 right | 1.9% | 75.0%run 1221 | run 1233 |
| Drex Max Open · 27B | 0.3%2 “I don't know” of 713 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 32.4%711 of 2,196 | 0.3%cut-off 0.2357: 1,489 right + 707 wrong answered, 2 held back, attempt rate 99.9% | 67.6%1,485 of 2,198 right | 1.5% | 67.7%run 322 | run 367 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 0.3%2 “I don't know” of 719 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 32.7%717 of 2,196 | 0.3%cut-off 0.2205: 1,474 right + 722 wrong answered, 2 held back, attempt rate 99.9% | 67.3%1,479 of 2,198 right | 2.4% | 67.1%run 217 | run 372 |
| KnowLine-31B-Gen1.5 Open · 31B | 0.2%1 “I don't know” of 412 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 18.7%411 of 2,197 | 0.2%cut-off 0.2517: 1,797 right + 400 wrong answered, 1 held back, attempt rate 100.0% | 81.3%1,786 of 2,198 right | 0.6% | 81.8%run 957 | run 959 |
| Cerno 9B v1.2 Open · 9.0B | 0.2%2 “I don't know” of 1,000 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 45.4%998 of 2,196 | 0.2%cut-off 0.2371: 1,197 right + 999 wrong answered, 2 held back, attempt rate 99.9% | 54.5%1,198 of 2,198 right | 2.8% | 54.5%run 1201 | run 1204 |
| KnowLine-12B-Gen1 Open · 12B | 0.1%1 “I don't know” of 805 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 36.6%804 of 2,197 | 0.1%cut-off 0.2147: 1,373 right + 824 wrong answered, 1 held back, attempt rate 100.0% | 63.4%1,393 of 2,198 right | 1.1% | 62.5%run 1309 | run 1318 |
| OneJev-27B Open · 27B | 0.1%1 “I don't know” of 893 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 40.6%892 of 2,197 | 0.1%cut-off 0.2094: 1,310 right + 887 wrong answered, 1 held back, attempt rate 100.0% | 59.4%1,305 of 2,198 right | 2.1% | 59.6%run 264 | run 782 |
| Valen-Decision-1.5-9B Open · 9.0B | 0.1%1 “I don't know” of 1,158 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 52.7%1,157 of 2,197 | 0.1%cut-off 0.2162: 1,031 right + 1,166 wrong answered, 1 held back, attempt rate 100.0% | 47.3%1,040 of 2,198 right | 4.8% | 47.0%run 1275 | run 1282 |
| KnowLine-31B-Gen2 Open · 31B | 0.0%0 “I don't know” of 408 not right | 100.0%2,198 of 2,198 answered, 0 “I don't know” | 18.6%408 of 2,198 | 0.0%cut-off 0.2259: 1,785 right + 413 wrong answered, 0 held back, attempt rate 100.0% | 81.4%1,790 of 2,198 right | 0.2% | 81.2%run 1314 | run 1331 |
| Sieve 31B v3 Open · 31B | 0.0%0 “I don't know” of 553 not right | 100.0%2,198 of 2,198 answered, 0 “I don't know” | 25.2%553 of 2,198 | 0.0%cut-off 0.2191: 1,630 right + 568 wrong answered, 0 held back, attempt rate 100.0% | 74.8%1,645 of 2,198 right | 0.9% | 74.2%run 1241 | run 1253 |
| d3 Open · 26B | 0.0%0 “I don't know” of 715 not right | 100.0%2,198 of 2,198 answered, 0 “I don't know” | 32.5%715 of 2,198 | 0.0%cut-off 0.2225: 1,490 right + 708 wrong answered, 0 held back, attempt rate 100.0% | 67.5%1,483 of 2,198 right | 2.9% | 67.8%run 259 | run 358 |
Chat models
| Model | Non-hallucinationof answers not right, % “I don't know” · higher is better | Answerednot “I don't know”: (right + wrong) / 2,198 | Error rateof its answers: wrong / (right + wrong) · lower is better | Derived non-hallucination ratefrom the 5-option text run; higher is better | Accuracyhigher is better | Mean probability of “I don't know”higher means more probability on the option; neither better nor worse on its own | Text accuracy with five optionsthe Text tab's number; higher is better | Runsrange over the runs where more than one |
|---|---|---|---|---|---|---|---|---|
| LFM3-1.7B reasoning off · chat Open · 1.7B | 76.2%1,433 “I don't know” of 1,881 not right | 32.7%718 of 2,198 answered, 1,433 “I don't know”, 47 invalid | 62.4%448 of 718 | Not measureda plain answer line, no confidence | 12.3%270 of 2,198 right | Not measureda plain answer line, no confidence | 28.3%run 1227 | run 1234 |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 67.0%1,197 “I don't know” of 1,786 not right | 45.5%1,001 of 2,198 answered, 1,197 “I don't know” | 58.8%589 of 1,001 | Not measureda plain answer line, no confidence | 18.7%412 of 2,198 right | Not measureda plain answer line, no confidence | 32.4%run 1089 | run 1090 |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 56.0%878 “I don't know” of 1,568 not right | 42.4%932 of 2,198 answered, 878 “I don't know”, 388 invalid | 74.0%690 of 932 | Not measureda plain answer line, no confidence | 11.0%242 of 2,198 right | Not measureda plain answer line, no confidence | 21.0%run 1288 | run 1298 |
| Grok 4.7 high · chat Closed · Workhorse | 29.9%56 “I don't know” of 187 not right | 97.5%2,142 of 2,198 answered, 56 “I don't know” | 6.1%131 of 2,142 | 26.9%cut-off 0.4200: 2,005 right + 141 wrong answered, 52 held back, attempt rate 97.6% | 91.5%2,011 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 92.7%run 311 | run 675 |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 29.7%307 “I don't know” of 1,034 not right | 86.0%1,891 of 2,198 answered, 307 “I don't know” | 38.4%727 of 1,891 | Not measureda plain answer line, no confidence | 53.0%1,164 of 2,198 right | Not measureda plain answer line, no confidence | 57.6%run 1040 | run 1049 |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 25.4%113 “I don't know” of 445 not right | 20.1%441 of 2,198 answered, 113 “I don't know”, 1,644 invalid | 75.3%332 of 441 | Not measureda plain answer line, no confidence | 5.0%109 of 2,198 right | Not measureda plain answer line, no confidence | 5.6%run 1299 | run 1304 |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 24.7%372 “I don't know” of 1,506 not right | 81.6%1,793 of 2,198 answered, 372 “I don't know”, 33 invalid | 63.2%1,134 of 1,793 | Not measureda plain answer line, no confidence | 30.0%659 of 2,198 right | Not measureda plain answer line, no confidence | 33.9%run 1037 | run 1038 |
| Claude Haiku 5.5 max · chat Closed · Efficient | 22.0%39 “I don't know” of 177 not right | 67.7%1,487 of 2,198 answered, 39 “I don't know”, 672 invalid | 9.3%138 of 1,487 | 83.1%cut-off 0.1500: 1,254 right + 160 wrong answered, 784 held back, attempt rate 64.3% | 61.4%1,349 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 57.1%run 948 | run 950 |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 20.8%315 “I don't know” of 1,517 not right | 75.6%1,662 of 2,198 answered, 315 “I don't know”, 221 invalid | 72.3%1,202 of 1,662 | Not measureda plain answer line, no confidence | 20.9%460 of 2,198 right | Not measureda plain answer line, no confidence | 25.4%run 1218 | run 1219 |
| Gemini 4 Carbon high · chat Closed · Flagship | 17.9%7 “I don't know” of 39 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 1.5%32 of 2,191 | 16.2%cut-off 0.7200: 2,161 right + 31 wrong answered, 6 held back, attempt rate 99.7% | 98.2%2,159 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 98.5%run 1108 | run 1121 |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 17.4%8 “I don't know” of 46 not right | 99.1%2,179 of 2,198 answered, 8 “I don't know”, 11 invalid | 1.7%38 of 2,179 | 37.5%cut-off 0.7200: 2,150 right + 30 wrong answered, 18 held back, attempt rate 99.2% | 97.4%2,141 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 98.1%run 1114 | run 1117 |
| GPT-6.1 Sol high · chat Closed · Workhorse | 15.4%8 “I don't know” of 52 not right | 99.6%2,190 of 2,198 answered, 8 “I don't know” | 2.0%44 of 2,190 | 16.7%cut-off 0.6500: 2,144 right + 45 wrong answered, 9 held back, attempt rate 99.6% | 97.6%2,146 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.8%run 276 | run 802 |
| GPT-6 Astra high · chat Closed · Flagship | 14.3%7 “I don't know” of 49 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 1.9%42 of 2,191 | 14.3%cut-off 0.5300: 2,149 right + 42 wrong answered, 7 held back, attempt rate 99.7% | 97.8%2,149 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 98.0%run 277 | run 818 |
| GPT-6 Astra medium · chat Closed · Flagship | 12.7%7 “I don't know” of 55 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 2.2%48 of 2,191 | 11.9%cut-off 0.5200: 2,139 right + 52 wrong answered, 7 held back, attempt rate 99.7% | 97.5%2,143 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.5%run 610 | run 817 |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 11.9%7 “I don't know” of 59 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 2.4%52 of 2,191 | 11.7%cut-off 0.6200: 2,138 right + 53 wrong answered, 7 held back, attempt rate 99.7% | 97.3%2,139 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.4%run 191 | run 819 |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 11.6%8 “I don't know” of 69 not right | 99.6%2,190 of 2,198 answered, 8 “I don't know” | 2.8%61 of 2,190 | 10.5%cut-off 0.5600: 2,122 right + 68 wrong answered, 8 held back, attempt rate 99.6% | 96.9%2,129 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 96.7%run 190 | run 810 |
| GPT-6 Astra low · chat Closed · Flagship | 10.5%6 “I don't know” of 57 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 2.3%51 of 2,192 | 10.9%cut-off 0.5000: 2,143 right + 49 wrong answered, 6 held back, attempt rate 99.7% | 97.4%2,141 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.6%run 48 | run 816 |
| GPT-6.1 Sol low · chat Closed · Workhorse | 9.2%6 “I don't know” of 65 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 2.7%59 of 2,192 | 7.9%cut-off 0.4200: 2,122 right + 70 wrong answered, 6 held back, attempt rate 99.7% | 97.0%2,133 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 96.7%run 124 | run 799 |
| GPT-6.1 Terra high · chat Closed · Workhorse | 7.3%9 “I don't know” of 124 not right | 99.6%2,189 of 2,198 answered, 9 “I don't know” | 5.3%115 of 2,189 | 6.9%cut-off 0.4300: 2,067 right + 122 wrong answered, 9 held back, attempt rate 99.6% | 94.4%2,074 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 94.3%run 275 | run 807 |
| Gemini 4 Argon low · chat Closed · Flagship | 7.0%6 “I don't know” of 86 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 3.6%80 of 2,192 | 7.4%cut-off 0.7500: 2,117 right + 75 wrong answered, 6 held back, attempt rate 99.7% | 96.1%2,112 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 96.4%run 557 | run 679 |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 6.9%84 “I don't know” of 1,216 not right | 95.4%2,096 of 2,198 answered, 84 “I don't know”, 18 invalid | 54.0%1,132 of 2,096 | Not measureda plain answer line, no confidence | 43.9%964 of 2,198 right | Not measureda plain answer line, no confidence | 46.2%run 1146 | run 1147 |
| GLiNER2.5 Multi Open · 0.3B | 6.3%116 “I don't know” of 1,835 not right | 94.7%2,082 of 2,198 answered, 116 “I don't know” | 82.6%1,719 of 2,082 | 6.3%cut-off 0.3186: 362 right + 1,720 wrong answered, 116 held back, attempt rate 94.7% | 16.5%363 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 17.5%run 122 | run 724 |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 6.2%4 “I don't know” of 64 not right | 99.8%2,194 of 2,198 answered, 4 “I don't know” | 2.7%60 of 2,194 | 9.2%cut-off 0.5500: 2,133 right + 59 wrong answered, 6 held back, attempt rate 99.7% | 97.1%2,134 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.1%run 188 | run 691 |
| Claude Fable 5.1 high · chat Closed · Flagship | 6.0%5 “I don't know” of 83 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 3.6%78 of 2,193 | 8.0%cut-off 0.3500: 2,111 right + 80 wrong answered, 7 held back, attempt rate 99.7% | 96.2%2,115 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 96.1%run 310 | run 695 |
| Qwen3.8-27B low · chat Open · 27B | 5.9%31 “I don't know” of 524 not right | 97.9%2,151 of 2,198 answered, 31 “I don't know”, 16 invalid | 22.9%493 of 2,151 | 8.6%cut-off 0.2018: 1,641 right + 509 wrong answered, 48 held back, attempt rate 97.8% | 75.4%1,658 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 74.7%run 318 | run 666 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 5.5%41 “I don't know” of 745 not right | 97.6%2,145 of 2,198 answered, 41 “I don't know”, 12 invalid | 32.8%704 of 2,145 | 6.3%cut-off 0.4500: 1,450 right + 701 wrong answered, 47 held back, attempt rate 97.9% | 65.6%1,441 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 66.5%run 862 | run 864 |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 4.7%38 “I don't know” of 802 not right | 98.3%2,160 of 2,198 answered, 38 “I don't know” | 35.4%764 of 2,160 | 2.7%cut-off 0.7500: 1,379 right + 797 wrong answered, 22 held back, attempt rate 99.0% | 63.5%1,396 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 63.1%run 985 | run 987 |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 4.5%2 “I don't know” of 44 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 1.9%42 of 2,196 | 4.8%cut-off 0.8500: 2,156 right + 40 wrong answered, 2 held back, attempt rate 99.9% | 98.0%2,154 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 98.1%run 140 | run 681 |
| Gemma 4 31B reasoning off · chat Open · 31B | 4.1%24 “I don't know” of 592 not right | 98.9%2,174 of 2,198 answered, 24 “I don't know” | 26.1%568 of 2,174 | 4.0%cut-off 0.5583: 1,596 right + 578 wrong answered, 24 held back, attempt rate 98.9% | 73.1%1,606 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 73.1%run 119 | run 776 |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 3.5%20 “I don't know” of 577 not right | 96.7%2,125 of 2,198 answered, 20 “I don't know”, 53 invalid | 26.2%557 of 2,125 | 11.7%cut-off 0.4000: 1,583 right + 543 wrong answered, 72 held back, attempt rate 96.7% | 71.3%1,568 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 72.1%run 877 | run 879 |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 3.3%30 “I don't know” of 911 not right | 98.4%2,163 of 2,198 answered, 30 “I don't know”, 5 invalid | 40.7%881 of 2,163 | Not measureda plain answer line, no confidence | 58.3%1,282 of 2,198 right | Not measureda plain answer line, no confidence | 58.9%run 1149 | run 1150 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 3.1%15 “I don't know” of 483 not right | 99.3%2,183 of 2,198 answered, 15 “I don't know” | 21.4%468 of 2,183 | 4.7%cut-off 0.6200: 1,754 right + 423 wrong answered, 21 held back, attempt rate 99.0% | 78.0%1,715 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 80.0%run 866 | run 868 |
| GPT-6.1 Terra low · chat Closed · Workhorse | 2.9%6 “I don't know” of 205 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 9.1%199 of 2,192 | 2.9%cut-off 0.5500: 1,994 right + 198 wrong answered, 6 held back, attempt rate 99.7% | 90.7%1,993 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 90.9%run 274 | run 801 |
| Claude Opus 5.5 high · chat Closed · Flagship | 2.7%3 “I don't know” of 111 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 4.9%108 of 2,195 | 0.9%cut-off 0.2500: 2,087 right + 110 wrong answered, 1 held back, attempt rate 100.0% | 94.9%2,087 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 95.0%run 278 | run 773 |
| Muse Ember max · chat Closed · Efficient | 2.1%6 “I don't know” of 287 not right | 97.5%2,144 of 2,198 answered, 6 “I don't know”, 48 invalid | 13.1%281 of 2,144 | 17.0%cut-off 0.3000: 1,881 right + 263 wrong answered, 54 held back, attempt rate 97.5% | 84.8%1,863 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 85.6%run 939 | run 940 |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 2.0%7 “I don't know” of 350 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 15.7%343 of 2,191 | 0.8%cut-off 0.2000: 1,818 right + 377 wrong answered, 3 held back, attempt rate 99.9% | 84.1%1,848 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 82.8%run 320 | run 686 |
| Qwen3-14B reasoning off · chat Open · 15B | 1.9%23 “I don't know” of 1,194 not right | 99.0%2,175 of 2,198 answered, 23 “I don't know” | 53.8%1,171 of 2,175 | 1.9%cut-off 0.3774: 1,001 right + 1,174 wrong answered, 23 held back, attempt rate 99.0% | 45.7%1,004 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 45.7%run 131 | run 772 |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 1.9%14 “I don't know” of 745 not right | 99.4%2,184 of 2,198 answered, 14 “I don't know” | 33.5%731 of 2,184 | 1.9%cut-off 0.5606: 1,464 right + 720 wrong answered, 14 held back, attempt rate 99.4% | 66.1%1,453 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 66.8%run 189 | run 789 |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 1.9%3 “I don't know” of 160 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 7.2%157 of 2,195 | 2.7%cut-off 0.8000: 2,049 right + 145 wrong answered, 4 held back, attempt rate 99.8% | 92.7%2,038 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 93.3%run 123 | run 676 |
| Claude Opus 5.5 low · chat Closed · Flagship | 1.8%3 “I don't know” of 171 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 7.7%168 of 2,195 | 0.9%cut-off 0.2500: 1,987 right + 209 wrong answered, 2 held back, attempt rate 99.9% | 92.2%2,027 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 90.4%run 47 | run 683 |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 1.5%6 “I don't know” of 396 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 17.8%390 of 2,192 | 1.3%cut-off 0.5000: 1,816 right + 377 wrong answered, 5 held back, attempt rate 99.8% | 82.0%1,802 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 82.7%run 870 | run 872 |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 1.3%13 “I don't know” of 1,010 not right | 99.4%2,185 of 2,198 answered, 13 “I don't know” | 45.6%997 of 2,185 | 2.5%cut-off 0.4000: 1,188 right + 985 wrong answered, 25 held back, attempt rate 98.9% | 54.0%1,188 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 54.3%run 83 | run 682 |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 1.3%2 “I don't know” of 156 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 7.0%154 of 2,196 | 17.7%cut-off 0.8354: 2,017 right + 149 wrong answered, 32 held back, attempt rate 98.5% | 92.9%2,042 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 91.8%run 585 | run 811 |
| GLM-5.3 low · chat Open · Workhorse | 1.1%6 “I don't know” of 525 not right | 99.0%2,176 of 2,198 answered, 6 “I don't know”, 16 invalid | 23.9%519 of 2,176 | 3.2%cut-off 0.3500: 1,634 right + 546 wrong answered, 18 held back, attempt rate 99.2% | 75.4%1,657 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 74.5%run 314 | run 656 |
| GPT-6 Luna low · chat Closed · Efficient | 1.0%7 “I don't know” of 735 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 33.2%728 of 2,191 | 1.0%cut-off 0.5700: 1,499 right + 692 wrong answered, 7 held back, attempt rate 99.7% | 66.6%1,463 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 68.3%run 133 | run 796 |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 0.8%2 “I don't know” of 246 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 11.1%244 of 2,196 | 1.3%cut-off 0.2500: 1,959 right + 236 wrong answered, 3 held back, attempt rate 99.9% | 88.8%1,952 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 89.2%run 193 | run 812 |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 0.7%1 “I don't know” of 138 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 6.2%137 of 2,197 | 0.7%cut-off 0.3000: 2,051 right + 146 wrong answered, 1 held back, attempt rate 100.0% | 93.7%2,060 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 93.3%run 192 | run 803 |
| Qwen4-Max high · chat Closed · Workhorse | 0.7%1 “I don't know” of 140 not right | 99.2%2,181 of 2,198 answered, 1 “I don't know”, 16 invalid | 6.4%139 of 2,181 | 17.6%cut-off 0.2500: 2,016 right + 150 wrong answered, 32 held back, attempt rate 98.5% | 92.9%2,042 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 91.7%run 1336 | run 1339 |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 0.7%2 “I don't know” of 284 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 12.8%282 of 2,196 | 0.3%cut-off 0.1500: 1,869 right + 328 wrong answered, 1 held back, attempt rate 100.0% | 87.1%1,914 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 85.0%run 55 | run 800 |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 0.6%5 “I don't know” of 774 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 35.1%769 of 2,193 | 0.4%cut-off 0.3000: 1,348 right + 847 wrong answered, 3 held back, attempt rate 99.9% | 64.8%1,424 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 61.4%run 54 | run 690 |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 0.6%6 “I don't know” of 1,006 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 45.6%1,000 of 2,192 | 0.4%cut-off 0.9500: 1,190 right + 1,004 wrong answered, 4 held back, attempt rate 99.8% | 54.2%1,192 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 54.2%run 53 | run 674 |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 0.6%2 “I don't know” of 338 not right | 91.9%2,020 of 2,198 answered, 2 “I don't know”, 176 invalid | 16.6%336 of 2,020 | 37.1%cut-off 0.4500: 1,715 right + 304 wrong answered, 179 held back, attempt rate 91.9% | 76.6%1,684 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 78.8%run 160 | run 825 |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 0.5%3 “I don't know” of 624 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 28.3%621 of 2,195 | 0.5%cut-off 0.5500: 1,598 right + 597 wrong answered, 3 held back, attempt rate 99.9% | 71.6%1,574 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 72.7%run 315 | run 653 |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 0.3%3 “I don't know” of 903 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 41.0%900 of 2,195 | 0.1%cut-off 0.4000: 1,270 right + 927 wrong answered, 1 held back, attempt rate 100.0% | 58.9%1,295 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 57.8%run 159 | run 822 |
| Erk-14B reasoning off · chat Open · 14B | 0.3%4 “I don't know” of 1,217 not right | 99.8%2,194 of 2,198 answered, 4 “I don't know” | 55.3%1,213 of 2,194 | 0.3%cut-off 0.2288: 988 right + 1,206 wrong answered, 4 held back, attempt rate 99.8% | 44.6%981 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 45.0%run 132 | run 765 |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 0.0%0 “I don't know” of 729 not right | 100.0%2,198 of 2,198 answered, 0 “I don't know” | 33.2%729 of 2,198 | 0.0%cut-off 0.2528: 1,451 right + 747 wrong answered, 0 held back, attempt rate 100.0% | 66.8%1,469 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 66.0%run 558 | run 806 |
Chat models
| Model | Non-hallucinationof answers not right, % “I don't know” · higher is better | Answerednot “I don't know”: (right + wrong) / 2,198 | Error rateof its answers: wrong / (right + wrong) · lower is better | Derived non-hallucination ratefrom the 5-option text run; higher is better | Accuracyhigher is better | Mean probability of “I don't know”higher means more probability on the option; neither better nor worse on its own | Text accuracy with five optionsthe Text tab's number; higher is better | Runsrange over the runs where more than one |
|---|---|---|---|---|---|---|---|---|
| LFM3-1.7B reasoning off · chat Open · 1.7B | 76.2%1,433 “I don't know” of 1,881 not right | 32.7%718 of 2,198 answered, 1,433 “I don't know”, 47 invalid | 62.4%448 of 718 | Not measureda plain answer line, no confidence | 12.3%270 of 2,198 right | Not measureda plain answer line, no confidence | 28.3%run 1227 | run 1234 |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 67.0%1,197 “I don't know” of 1,786 not right | 45.5%1,001 of 2,198 answered, 1,197 “I don't know” | 58.8%589 of 1,001 | Not measureda plain answer line, no confidence | 18.7%412 of 2,198 right | Not measureda plain answer line, no confidence | 32.4%run 1089 | run 1090 |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 56.0%878 “I don't know” of 1,568 not right | 42.4%932 of 2,198 answered, 878 “I don't know”, 388 invalid | 74.0%690 of 932 | Not measureda plain answer line, no confidence | 11.0%242 of 2,198 right | Not measureda plain answer line, no confidence | 21.0%run 1288 | run 1298 |
| Grok 4.7 high · chat Closed · Workhorse | 29.9%56 “I don't know” of 187 not right | 97.5%2,142 of 2,198 answered, 56 “I don't know” | 6.1%131 of 2,142 | 26.9%cut-off 0.4200: 2,005 right + 141 wrong answered, 52 held back, attempt rate 97.6% | 91.5%2,011 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 92.7%run 311 | run 675 |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 29.7%307 “I don't know” of 1,034 not right | 86.0%1,891 of 2,198 answered, 307 “I don't know” | 38.4%727 of 1,891 | Not measureda plain answer line, no confidence | 53.0%1,164 of 2,198 right | Not measureda plain answer line, no confidence | 57.6%run 1040 | run 1049 |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 25.4%113 “I don't know” of 445 not right | 20.1%441 of 2,198 answered, 113 “I don't know”, 1,644 invalid | 75.3%332 of 441 | Not measureda plain answer line, no confidence | 5.0%109 of 2,198 right | Not measureda plain answer line, no confidence | 5.6%run 1299 | run 1304 |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 24.7%372 “I don't know” of 1,506 not right | 81.6%1,793 of 2,198 answered, 372 “I don't know”, 33 invalid | 63.2%1,134 of 1,793 | Not measureda plain answer line, no confidence | 30.0%659 of 2,198 right | Not measureda plain answer line, no confidence | 33.9%run 1037 | run 1038 |
| Claude Haiku 5.5 max · chat Closed · Efficient | 22.0%39 “I don't know” of 177 not right | 67.7%1,487 of 2,198 answered, 39 “I don't know”, 672 invalid | 9.3%138 of 1,487 | 83.1%cut-off 0.1500: 1,254 right + 160 wrong answered, 784 held back, attempt rate 64.3% | 61.4%1,349 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 57.1%run 948 | run 950 |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 20.8%315 “I don't know” of 1,517 not right | 75.6%1,662 of 2,198 answered, 315 “I don't know”, 221 invalid | 72.3%1,202 of 1,662 | Not measureda plain answer line, no confidence | 20.9%460 of 2,198 right | Not measureda plain answer line, no confidence | 25.4%run 1218 | run 1219 |
| Gemini 4 Carbon high · chat Closed · Flagship | 17.9%7 “I don't know” of 39 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 1.5%32 of 2,191 | 16.2%cut-off 0.7200: 2,161 right + 31 wrong answered, 6 held back, attempt rate 99.7% | 98.2%2,159 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 98.5%run 1108 | run 1121 |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 17.4%8 “I don't know” of 46 not right | 99.1%2,179 of 2,198 answered, 8 “I don't know”, 11 invalid | 1.7%38 of 2,179 | 37.5%cut-off 0.7200: 2,150 right + 30 wrong answered, 18 held back, attempt rate 99.2% | 97.4%2,141 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 98.1%run 1114 | run 1117 |
| GPT-6.1 Sol high · chat Closed · Workhorse | 15.4%8 “I don't know” of 52 not right | 99.6%2,190 of 2,198 answered, 8 “I don't know” | 2.0%44 of 2,190 | 16.7%cut-off 0.6500: 2,144 right + 45 wrong answered, 9 held back, attempt rate 99.6% | 97.6%2,146 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.8%run 276 | run 802 |
| GPT-6 Astra high · chat Closed · Flagship | 14.3%7 “I don't know” of 49 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 1.9%42 of 2,191 | 14.3%cut-off 0.5300: 2,149 right + 42 wrong answered, 7 held back, attempt rate 99.7% | 97.8%2,149 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 98.0%run 277 | run 818 |
| GPT-6 Astra medium · chat Closed · Flagship | 12.7%7 “I don't know” of 55 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 2.2%48 of 2,191 | 11.9%cut-off 0.5200: 2,139 right + 52 wrong answered, 7 held back, attempt rate 99.7% | 97.5%2,143 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.5%run 610 | run 817 |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 11.9%7 “I don't know” of 59 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 2.4%52 of 2,191 | 11.7%cut-off 0.6200: 2,138 right + 53 wrong answered, 7 held back, attempt rate 99.7% | 97.3%2,139 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.4%run 191 | run 819 |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 11.6%8 “I don't know” of 69 not right | 99.6%2,190 of 2,198 answered, 8 “I don't know” | 2.8%61 of 2,190 | 10.5%cut-off 0.5600: 2,122 right + 68 wrong answered, 8 held back, attempt rate 99.6% | 96.9%2,129 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 96.7%run 190 | run 810 |
| GPT-6 Astra low · chat Closed · Flagship | 10.5%6 “I don't know” of 57 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 2.3%51 of 2,192 | 10.9%cut-off 0.5000: 2,143 right + 49 wrong answered, 6 held back, attempt rate 99.7% | 97.4%2,141 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.6%run 48 | run 816 |
| GPT-6.1 Sol low · chat Closed · Workhorse | 9.2%6 “I don't know” of 65 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 2.7%59 of 2,192 | 7.9%cut-off 0.4200: 2,122 right + 70 wrong answered, 6 held back, attempt rate 99.7% | 97.0%2,133 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 96.7%run 124 | run 799 |
| GPT-6.1 Terra high · chat Closed · Workhorse | 7.3%9 “I don't know” of 124 not right | 99.6%2,189 of 2,198 answered, 9 “I don't know” | 5.3%115 of 2,189 | 6.9%cut-off 0.4300: 2,067 right + 122 wrong answered, 9 held back, attempt rate 99.6% | 94.4%2,074 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 94.3%run 275 | run 807 |
| Gemini 4 Argon low · chat Closed · Flagship | 7.0%6 “I don't know” of 86 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 3.6%80 of 2,192 | 7.4%cut-off 0.7500: 2,117 right + 75 wrong answered, 6 held back, attempt rate 99.7% | 96.1%2,112 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 96.4%run 557 | run 679 |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 6.9%84 “I don't know” of 1,216 not right | 95.4%2,096 of 2,198 answered, 84 “I don't know”, 18 invalid | 54.0%1,132 of 2,096 | Not measureda plain answer line, no confidence | 43.9%964 of 2,198 right | Not measureda plain answer line, no confidence | 46.2%run 1146 | run 1147 |
| GLiNER2.5 Multi Open · 0.3B | 6.3%116 “I don't know” of 1,835 not right | 94.7%2,082 of 2,198 answered, 116 “I don't know” | 82.6%1,719 of 2,082 | 6.3%cut-off 0.3186: 362 right + 1,720 wrong answered, 116 held back, attempt rate 94.7% | 16.5%363 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 17.5%run 122 | run 724 |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 6.2%4 “I don't know” of 64 not right | 99.8%2,194 of 2,198 answered, 4 “I don't know” | 2.7%60 of 2,194 | 9.2%cut-off 0.5500: 2,133 right + 59 wrong answered, 6 held back, attempt rate 99.7% | 97.1%2,134 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 97.1%run 188 | run 691 |
| Claude Fable 5.1 high · chat Closed · Flagship | 6.0%5 “I don't know” of 83 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 3.6%78 of 2,193 | 8.0%cut-off 0.3500: 2,111 right + 80 wrong answered, 7 held back, attempt rate 99.7% | 96.2%2,115 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 96.1%run 310 | run 695 |
| Qwen3.8-27B low · chat Open · 27B | 5.9%31 “I don't know” of 524 not right | 97.9%2,151 of 2,198 answered, 31 “I don't know”, 16 invalid | 22.9%493 of 2,151 | 8.6%cut-off 0.2018: 1,641 right + 509 wrong answered, 48 held back, attempt rate 97.8% | 75.4%1,658 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 74.7%run 318 | run 666 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 5.5%41 “I don't know” of 745 not right | 97.6%2,145 of 2,198 answered, 41 “I don't know”, 12 invalid | 32.8%704 of 2,145 | 6.3%cut-off 0.4500: 1,450 right + 701 wrong answered, 47 held back, attempt rate 97.9% | 65.6%1,441 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 66.5%run 862 | run 864 |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 4.7%38 “I don't know” of 802 not right | 98.3%2,160 of 2,198 answered, 38 “I don't know” | 35.4%764 of 2,160 | 2.7%cut-off 0.7500: 1,379 right + 797 wrong answered, 22 held back, attempt rate 99.0% | 63.5%1,396 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 63.1%run 985 | run 987 |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 4.5%2 “I don't know” of 44 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 1.9%42 of 2,196 | 4.8%cut-off 0.8500: 2,156 right + 40 wrong answered, 2 held back, attempt rate 99.9% | 98.0%2,154 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 98.1%run 140 | run 681 |
| Gemma 4 31B reasoning off · chat Open · 31B | 4.1%24 “I don't know” of 592 not right | 98.9%2,174 of 2,198 answered, 24 “I don't know” | 26.1%568 of 2,174 | 4.0%cut-off 0.5583: 1,596 right + 578 wrong answered, 24 held back, attempt rate 98.9% | 73.1%1,606 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 73.1%run 119 | run 776 |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 3.5%20 “I don't know” of 577 not right | 96.7%2,125 of 2,198 answered, 20 “I don't know”, 53 invalid | 26.2%557 of 2,125 | 11.7%cut-off 0.4000: 1,583 right + 543 wrong answered, 72 held back, attempt rate 96.7% | 71.3%1,568 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 72.1%run 877 | run 879 |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 3.3%30 “I don't know” of 911 not right | 98.4%2,163 of 2,198 answered, 30 “I don't know”, 5 invalid | 40.7%881 of 2,163 | Not measureda plain answer line, no confidence | 58.3%1,282 of 2,198 right | Not measureda plain answer line, no confidence | 58.9%run 1149 | run 1150 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 3.1%15 “I don't know” of 483 not right | 99.3%2,183 of 2,198 answered, 15 “I don't know” | 21.4%468 of 2,183 | 4.7%cut-off 0.6200: 1,754 right + 423 wrong answered, 21 held back, attempt rate 99.0% | 78.0%1,715 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 80.0%run 866 | run 868 |
| GPT-6.1 Terra low · chat Closed · Workhorse | 2.9%6 “I don't know” of 205 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 9.1%199 of 2,192 | 2.9%cut-off 0.5500: 1,994 right + 198 wrong answered, 6 held back, attempt rate 99.7% | 90.7%1,993 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 90.9%run 274 | run 801 |
| Claude Opus 5.5 high · chat Closed · Flagship | 2.7%3 “I don't know” of 111 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 4.9%108 of 2,195 | 0.9%cut-off 0.2500: 2,087 right + 110 wrong answered, 1 held back, attempt rate 100.0% | 94.9%2,087 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 95.0%run 278 | run 773 |
| Muse Ember max · chat Closed · Efficient | 2.1%6 “I don't know” of 287 not right | 97.5%2,144 of 2,198 answered, 6 “I don't know”, 48 invalid | 13.1%281 of 2,144 | 17.0%cut-off 0.3000: 1,881 right + 263 wrong answered, 54 held back, attempt rate 97.5% | 84.8%1,863 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 85.6%run 939 | run 940 |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 2.0%7 “I don't know” of 350 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 15.7%343 of 2,191 | 0.8%cut-off 0.2000: 1,818 right + 377 wrong answered, 3 held back, attempt rate 99.9% | 84.1%1,848 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 82.8%run 320 | run 686 |
| Qwen3-14B reasoning off · chat Open · 15B | 1.9%23 “I don't know” of 1,194 not right | 99.0%2,175 of 2,198 answered, 23 “I don't know” | 53.8%1,171 of 2,175 | 1.9%cut-off 0.3774: 1,001 right + 1,174 wrong answered, 23 held back, attempt rate 99.0% | 45.7%1,004 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 45.7%run 131 | run 772 |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 1.9%14 “I don't know” of 745 not right | 99.4%2,184 of 2,198 answered, 14 “I don't know” | 33.5%731 of 2,184 | 1.9%cut-off 0.5606: 1,464 right + 720 wrong answered, 14 held back, attempt rate 99.4% | 66.1%1,453 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 66.8%run 189 | run 789 |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 1.9%3 “I don't know” of 160 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 7.2%157 of 2,195 | 2.7%cut-off 0.8000: 2,049 right + 145 wrong answered, 4 held back, attempt rate 99.8% | 92.7%2,038 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 93.3%run 123 | run 676 |
| Claude Opus 5.5 low · chat Closed · Flagship | 1.8%3 “I don't know” of 171 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 7.7%168 of 2,195 | 0.9%cut-off 0.2500: 1,987 right + 209 wrong answered, 2 held back, attempt rate 99.9% | 92.2%2,027 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 90.4%run 47 | run 683 |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 1.5%6 “I don't know” of 396 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 17.8%390 of 2,192 | 1.3%cut-off 0.5000: 1,816 right + 377 wrong answered, 5 held back, attempt rate 99.8% | 82.0%1,802 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 82.7%run 870 | run 872 |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 1.3%13 “I don't know” of 1,010 not right | 99.4%2,185 of 2,198 answered, 13 “I don't know” | 45.6%997 of 2,185 | 2.5%cut-off 0.4000: 1,188 right + 985 wrong answered, 25 held back, attempt rate 98.9% | 54.0%1,188 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 54.3%run 83 | run 682 |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 1.3%2 “I don't know” of 156 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 7.0%154 of 2,196 | 17.7%cut-off 0.8354: 2,017 right + 149 wrong answered, 32 held back, attempt rate 98.5% | 92.9%2,042 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 91.8%run 585 | run 811 |
| GLM-5.3 low · chat Open · Workhorse | 1.1%6 “I don't know” of 525 not right | 99.0%2,176 of 2,198 answered, 6 “I don't know”, 16 invalid | 23.9%519 of 2,176 | 3.2%cut-off 0.3500: 1,634 right + 546 wrong answered, 18 held back, attempt rate 99.2% | 75.4%1,657 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 74.5%run 314 | run 656 |
| GPT-6 Luna low · chat Closed · Efficient | 1.0%7 “I don't know” of 735 not right | 99.7%2,191 of 2,198 answered, 7 “I don't know” | 33.2%728 of 2,191 | 1.0%cut-off 0.5700: 1,499 right + 692 wrong answered, 7 held back, attempt rate 99.7% | 66.6%1,463 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 68.3%run 133 | run 796 |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 0.8%2 “I don't know” of 246 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 11.1%244 of 2,196 | 1.3%cut-off 0.2500: 1,959 right + 236 wrong answered, 3 held back, attempt rate 99.9% | 88.8%1,952 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 89.2%run 193 | run 812 |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 0.7%1 “I don't know” of 138 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 6.2%137 of 2,197 | 0.7%cut-off 0.3000: 2,051 right + 146 wrong answered, 1 held back, attempt rate 100.0% | 93.7%2,060 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 93.3%run 192 | run 803 |
| Qwen4-Max high · chat Closed · Workhorse | 0.7%1 “I don't know” of 140 not right | 99.2%2,181 of 2,198 answered, 1 “I don't know”, 16 invalid | 6.4%139 of 2,181 | 17.6%cut-off 0.2500: 2,016 right + 150 wrong answered, 32 held back, attempt rate 98.5% | 92.9%2,042 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 91.7%run 1336 | run 1339 |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 0.7%2 “I don't know” of 284 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 12.8%282 of 2,196 | 0.3%cut-off 0.1500: 1,869 right + 328 wrong answered, 1 held back, attempt rate 100.0% | 87.1%1,914 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 85.0%run 55 | run 800 |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 0.6%5 “I don't know” of 774 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 35.1%769 of 2,193 | 0.4%cut-off 0.3000: 1,348 right + 847 wrong answered, 3 held back, attempt rate 99.9% | 64.8%1,424 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 61.4%run 54 | run 690 |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 0.6%6 “I don't know” of 1,006 not right | 99.7%2,192 of 2,198 answered, 6 “I don't know” | 45.6%1,000 of 2,192 | 0.4%cut-off 0.9500: 1,190 right + 1,004 wrong answered, 4 held back, attempt rate 99.8% | 54.2%1,192 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 54.2%run 53 | run 674 |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 0.6%2 “I don't know” of 338 not right | 91.9%2,020 of 2,198 answered, 2 “I don't know”, 176 invalid | 16.6%336 of 2,020 | 37.1%cut-off 0.4500: 1,715 right + 304 wrong answered, 179 held back, attempt rate 91.9% | 76.6%1,684 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 78.8%run 160 | run 825 |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 0.5%3 “I don't know” of 624 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 28.3%621 of 2,195 | 0.5%cut-off 0.5500: 1,598 right + 597 wrong answered, 3 held back, attempt rate 99.9% | 71.6%1,574 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 72.7%run 315 | run 653 |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 0.3%3 “I don't know” of 903 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 41.0%900 of 2,195 | 0.1%cut-off 0.4000: 1,270 right + 927 wrong answered, 1 held back, attempt rate 100.0% | 58.9%1,295 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 57.8%run 159 | run 822 |
| Erk-14B reasoning off · chat Open · 14B | 0.3%4 “I don't know” of 1,217 not right | 99.8%2,194 of 2,198 answered, 4 “I don't know” | 55.3%1,213 of 2,194 | 0.3%cut-off 0.2288: 988 right + 1,206 wrong answered, 4 held back, attempt rate 99.8% | 44.6%981 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 45.0%run 132 | run 765 |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 0.0%0 “I don't know” of 729 not right | 100.0%2,198 of 2,198 answered, 0 “I don't know” | 33.2%729 of 2,198 | 0.0%cut-off 0.2528: 1,451 right + 747 wrong answered, 0 held back, attempt rate 100.0% | 66.8%1,469 of 2,198 right | No probabilitiesa chat model states a confidence, not option probabilities | 66.0%run 558 | run 806 |
Decision models
| Model | Non-hallucinationof answers not right, % “I don't know” · higher is better | Answerednot “I don't know”: (right + wrong) / 2,198 | Error rateof its answers: wrong / (right + wrong) · lower is better | Derived non-hallucination ratefrom the 5-option text run; higher is better | Accuracyhigher is better | Mean probability of “I don't know”higher means more probability on the option; neither better nor worse on its own | Text accuracy with five optionsthe Text tab's number; higher is better | Runsrange over the runs where more than one |
|---|---|---|---|---|---|---|---|---|
| GLiDE 0.4B Open · 0.4B | 95.3%2,070 “I don't know” of 2,171 not right | 5.8%128 of 2,198 answered, 2,070 “I don't know” | 78.9%101 of 128 | 95.4%cut-off 0.4443: 29 right + 99 wrong answered, 2,070 held back, attempt rate 5.8% | 1.2%27 of 2,198 right | 39.1% | 22.2%run 931 | run 937 |
| d1.2-ultra Closed · Efficient | 87.3%996 “I don't know” of 1,141 not right | 54.7%1,202 of 2,198 answered, 996 “I don't know” | 12.1%145 of 1,202 | 87.8%cut-off 0.5947: 1,064 right + 138 wrong answered, 996 held back, attempt rate 54.7% | 48.1%1,057 of 2,198 right | 34.6% | 68.1%run 1347 | run 1367 |
| d1.2 Closed · Efficient | 79.4%973 “I don't know” of 1,226 not right | 55.7%1,225 of 2,198 answered, 973 “I don't know” | 20.7%253 of 1,225 | 81.3%cut-off 0.4936: 1,001 right + 224 wrong answered, 973 held back, attempt rate 55.7% | 44.2%972 of 2,198 right | 31.5% | 61.7%run 1346 | run 1365 |
| d1-ultra Closed · Efficient | 75.1%617 “I don't know” of 822 not right | 71.9%1,581 of 2,198 answered, 617 “I don't know” | 13.0%205 of 1,581 | 78.7%cut-off 0.5632: 1,414 right + 167 wrong answered, 617 held back, attempt rate 71.9% | 62.6%1,376 of 2,198 right | 30.7% | 77.1%run 199 | runs 350, 354, 360, 365, 370Answered 71.9% to 74.0%Non-hallucination 71.2% to 75.1%Accuracy 62.6% to 63.5%Mean probability 30.6% to 30.8%518 answers changed between runs |
| Mammoth-1 Open · 27B | 68.7%690 “I don't know” of 1,004 not right | 68.6%1,508 of 2,198 answered, 690 “I don't know” | 20.8%314 of 1,508 | 70.9%cut-off 0.5697: 1,225 right + 283 wrong answered, 690 held back, attempt rate 68.6% | 54.3%1,194 of 2,198 right | 24.5% | 67.1%run 374 | run 386 |
| d1-2B Open · 1.9B | 65.5%1,209 “I don't know” of 1,846 not right | 45.0%989 of 2,198 answered, 1,209 “I don't know” | 64.4%637 of 989 | 66.5%cut-off 0.3129: 381 right + 608 wrong answered, 1,209 held back, attempt rate 45.0% | 16.0%352 of 2,198 right | 26.3% | 29.4%run 340 | run 712 |
| GLiDE API Closed · Efficient | 59.4%521 “I don't know” of 877 not right | 76.3%1,677 of 2,198 answered, 521 “I don't know” | 21.2%356 of 1,677 | 58.0%cut-off 0.5206: 1,300 right + 377 wrong answered, 521 held back, attempt rate 76.3% | 60.1%1,321 of 2,198 right | 22.0% | 67.9%run 129 | runs 351, 353, 357, 361, 364Answered 75.5% to 76.3%Non-hallucination 59.4% to 59.9%Accuracy 59.1% to 60.1%Mean probability 21.6% to 22.0%189 answers changed between runs |
| SubQ Snap 1.1 Closed · Efficient | 47.1%437 “I don't know” of 927 not right | 80.1%1,761 of 2,198 answered, 437 “I don't know” | 27.8%490 of 1,761 | 47.6%cut-off 0.5124: 1,280 right + 481 wrong answered, 437 held back, attempt rate 80.1% | 57.8%1,271 of 2,198 right | 20.1% | 63.9%run 346 | run 392 |
| OneJev1.6-0.8B Open · 0.8B | 43.4%754 “I don't know” of 1,739 not right | 65.7%1,444 of 2,198 answered, 754 “I don't know” | 68.2%985 of 1,444 | 43.5%cut-off 0.2972: 463 right + 981 wrong answered, 754 held back, attempt rate 65.7% | 20.9%459 of 2,198 right | 22.2% | 29.1%run 1029 | run 1031 |
| GLiDE 27B Open · 27B | 39.8%291 “I don't know” of 732 not right | 86.8%1,907 of 2,198 answered, 291 “I don't know” | 23.1%441 of 1,907 | 39.9%cut-off 0.4133: 1,469 right + 438 wrong answered, 291 held back, attempt rate 86.8% | 66.7%1,466 of 2,198 right | 16.8% | 71.1%run 899 | run 905 |
| GLiDE v1.2 Open · 27B | 39.5%321 “I don't know” of 812 not right | 85.4%1,877 of 2,198 answered, 321 “I don't know” | 26.2%491 of 1,877 | 40.4%cut-off 0.4134: 1,404 right + 473 wrong answered, 321 held back, attempt rate 85.4% | 63.1%1,386 of 2,198 right | 18.6% | 68.5%run 248 | run 369 |
| GLiDE 2B Open · 1.9B | 38.5%612 “I don't know” of 1,590 not right | 72.2%1,586 of 2,198 answered, 612 “I don't know” | 61.7%978 of 1,586 | 38.7%cut-off 0.4078: 618 right + 968 wrong answered, 612 held back, attempt rate 72.2% | 27.7%608 of 2,198 right | 23.5% | 35.3%run 915 | run 921 |
| Drex Max v1.1 Closed · Efficient (provisional) | 38.2%320 “I don't know” of 837 not right | 85.4%1,878 of 2,198 answered, 320 “I don't know” | 27.5%517 of 1,878 | 39.3%cut-off 0.3806: 1,383 right + 495 wrong answered, 320 held back, attempt rate 85.4% | 61.9%1,361 of 2,198 right | 17.0% | 67.4%run 326 | run 352 |
| Elephant-1 Open · 9.0B | 34.8%405 “I don't know” of 1,163 not right | 81.6%1,793 of 2,198 answered, 405 “I don't know” | 42.3%758 of 1,793 | 35.5%cut-off 0.3883: 1,058 right + 735 wrong answered, 405 held back, attempt rate 81.6% | 47.1%1,035 of 2,198 right | 16.0% | 52.4%run 373 | run 382 |
| GLiDE 0.8B Open · 0.8B | 29.2%480 “I don't know” of 1,643 not right | 78.2%1,718 of 2,198 answered, 480 “I don't know” | 67.7%1,163 of 1,718 | 28.9%cut-off 0.3650: 536 right + 1,182 wrong answered, 480 held back, attempt rate 78.2% | 25.3%555 of 2,198 right | 19.9% | 29.3%run 923 | run 929 |
| Laya Open · 0.4B | 28.3%536 “I don't know” of 1,891 not right | 75.6%1,662 of 2,198 answered, 536 “I don't know” | 81.5%1,355 of 1,662 | 28.5%cut-off 0.2527: 316 right + 1,345 wrong answered, 537 held back, attempt rate 75.6% | 14.0%307 of 2,198 right | 18.6% | 19.0%run 51 | run 730 |
| GLiDE 9B Open · 7.9B | 24.1%277 “I don't know” of 1,149 not right | 87.4%1,921 of 2,198 answered, 277 “I don't know” | 45.4%872 of 1,921 | 24.8%cut-off 0.3391: 1,081 right + 840 wrong answered, 277 held back, attempt rate 87.4% | 47.7%1,049 of 2,198 right | 15.2% | 52.0%run 907 | run 913 |
| GLiClass v4 Edge Open · 0.4B | 22.3%398 “I don't know” of 1,784 not right | 81.9%1,800 of 2,198 answered, 398 “I don't know” | 77.0%1,386 of 1,800 | 22.6%cut-off 0.2423: 435 right + 1,365 wrong answered, 398 held back, attempt rate 81.9% | 18.8%414 of 2,198 right | 14.5% | 23.3%run 1046 | run 1048 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 21.5%333 “I don't know” of 1,546 not right | 84.8%1,865 of 2,198 answered, 333 “I don't know” | 65.0%1,213 of 1,865 | 21.8%cut-off 0.2682: 669 right + 1,196 wrong answered, 333 held back, attempt rate 84.8% | 29.7%652 of 2,198 right | 13.2% | 34.3%run 977 | run 979 |
| Laya Multilingual Open · 0.3B | 14.8%280 “I don't know” of 1,888 not right | 87.3%1,918 of 2,198 answered, 280 “I don't know” | 83.8%1,608 of 1,918 | 15.0%cut-off 0.2794: 331 right + 1,587 wrong answered, 280 held back, attempt rate 87.3% | 14.1%310 of 2,198 right | 14.6% | 17.9%run 46 | run 737 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 14.7%180 “I don't know” of 1,224 not right | 91.8%2,018 of 2,198 answered, 180 “I don't know” | 51.7%1,044 of 2,018 | 15.2%cut-off 0.2883: 1,014 right + 1,004 wrong answered, 180 held back, attempt rate 91.8% | 44.3%974 of 2,198 right | 12.5% | 47.8%run 223 | run 791 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 12.4%199 “I don't know” of 1,608 not right | 90.9%1,999 of 2,198 answered, 199 “I don't know” | 70.5%1,409 of 1,999 | 12.3%cut-off 0.2318: 585 right + 1,414 wrong answered, 199 held back, attempt rate 90.9% | 26.8%590 of 2,198 right | 9.4% | 28.6%run 1015 | run 1017 |
| Chinese Decision Pointer 4B Open · 4.0B | 11.2%136 “I don't know” of 1,218 not right | 93.8%2,062 of 2,198 answered, 136 “I don't know” | 52.5%1,082 of 2,062 | 11.3%cut-off 0.3048: 994 right + 1,068 wrong answered, 136 held back, attempt rate 93.8% | 44.6%980 of 2,198 right | 6.8% | 46.7%run 1206 | run 1213 |
| Strands Decider 2B Open · 2.0B | 9.2%150 “I don't know” of 1,633 not right | 93.2%2,048 of 2,198 answered, 150 “I don't know” | 72.4%1,483 of 2,048 | 9.2%cut-off 0.2554: 574 right + 1,474 wrong answered, 150 held back, attempt rate 93.2% | 25.7%565 of 2,198 right | 8.8% | 27.5%run 141 | run 706 |
| Clef-Flash Open · 9.0B | 9.0%110 “I don't know” of 1,218 not right | 95.0%2,088 of 2,198 answered, 110 “I don't know” | 53.1%1,108 of 2,088 | 9.0%cut-off 0.2929: 981 right + 1,107 wrong answered, 110 held back, attempt rate 95.0% | 44.6%980 of 2,198 right | 9.1% | 45.7%run 139 | run 664 |
| vera-medium-omni Open · 4.2B | 8.9%105 “I don't know” of 1,180 not right | 95.2%2,093 of 2,198 answered, 105 “I don't know” | 51.4%1,075 of 2,093 | 9.2%cut-off 0.2717: 1,055 right + 1,038 wrong answered, 105 held back, attempt rate 95.2% | 46.3%1,018 of 2,198 right | 9.1% | 49.1%run 1264 | run 1271 |
| Clef Open · 27B | 7.4%71 “I don't know” of 965 not right | 96.8%2,127 of 2,198 answered, 71 “I don't know” | 42.0%894 of 2,127 | 7.4%cut-off 0.3189: 1,238 right + 889 wrong answered, 71 held back, attempt rate 96.8% | 56.1%1,233 of 2,198 right | 10.1% | 57.3%run 125 | run 788 |
| Perplexity Decider Flash v1.3 Open · 4.0B | 6.5%75 “I don't know” of 1,146 not right | 96.6%2,123 of 2,198 answered, 75 “I don't know” | 50.4%1,071 of 2,123 | 6.7%cut-off 0.2769: 1,074 right + 1,049 wrong answered, 75 held back, attempt rate 96.6% | 47.9%1,052 of 2,198 right | 9.1% | 49.7%run 1123 | run 1130 |
| KnowLine-0.8B-Gen1 Open · 0.8B | 6.4%104 “I don't know” of 1,619 not right | 95.3%2,094 of 2,198 answered, 104 “I don't know” | 72.3%1,515 of 2,094 | 6.4%cut-off 0.2348: 578 right + 1,516 wrong answered, 104 held back, attempt rate 95.3% | 26.3%579 of 2,198 right | 8.1% | 27.1%run 1005 | run 1007 |
| GLiNER2.5-multi-Decide Open · 0.3B | 6.4%114 “I don't know” of 1,780 not right | 94.8%2,084 of 2,198 answered, 114 “I don't know” | 79.9%1,666 of 2,084 | 6.4%cut-off 0.2553: 426 right + 1,658 wrong answered, 114 held back, attempt rate 94.8% | 19.0%418 of 2,198 right | 10.7% | 20.2%run 135 | run 718 |
| KnowLine 4B Gen5.5 Open · 4.0B | 4.7%55 “I don't know” of 1,176 not right | 97.5%2,143 of 2,198 answered, 55 “I don't know” | 52.3%1,121 of 2,143 | 4.8%cut-off 0.2688: 1,051 right + 1,092 wrong answered, 55 held back, attempt rate 97.5% | 46.5%1,022 of 2,198 right | 2.8% | 48.5%run 965 | run 967 |
| Jeffan-9B-1211 Open · 9.0B | 4.3%41 “I don't know” of 958 not right | 98.1%2,157 of 2,198 answered, 41 “I don't know” | 42.5%917 of 2,157 | 4.3%cut-off 0.2414: 1,248 right + 909 wrong answered, 41 held back, attempt rate 98.1% | 56.4%1,240 of 2,198 right | 5.8% | 57.0%run 1192 | run 1199 |
| Winnow-12B Open · 12B | 4.3%43 “I don't know” of 1,008 not right | 98.0%2,155 of 2,198 answered, 43 “I don't know” | 44.8%965 of 2,155 | 4.3%cut-off 0.2923: 1,209 right + 946 wrong answered, 43 held back, attempt rate 98.0% | 54.1%1,190 of 2,198 right | 3.5% | 55.5%run 128 | run 750 |
| Dexy-Decision-9B-V2 Open · 9.0B | 3.4%37 “I don't know” of 1,083 not right | 98.3%2,161 of 2,198 answered, 37 “I don't know” | 48.4%1,046 of 2,161 | 3.4%cut-off 0.2579: 1,122 right + 1,039 wrong answered, 37 held back, attempt rate 98.3% | 50.7%1,115 of 2,198 right | 2.7% | 51.6%run 1255 | run 1262 |
| Turion Jev 31B Open · 31B | 3.3%15 “I don't know” of 451 not right | 99.3%2,183 of 2,198 answered, 15 “I don't know” | 20.0%436 of 2,183 | 3.4%cut-off 0.2468: 1,755 right + 428 wrong answered, 15 held back, attempt rate 99.3% | 79.5%1,747 of 2,198 right | 3.1% | 80.1%run 1273 | run 1285 |
| Cygnet Open · 12B | 3.2%32 “I don't know” of 989 not right | 98.5%2,166 of 2,198 answered, 32 “I don't know” | 44.2%957 of 2,166 | 3.3%cut-off 0.2955: 1,217 right + 949 wrong answered, 32 held back, attempt rate 98.5% | 55.0%1,209 of 2,198 right | 5.1% | 55.7%run 127 | run 757 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 3.2%23 “I don't know” of 725 not right | 99.0%2,175 of 2,198 answered, 23 “I don't know” | 32.3%702 of 2,175 | 3.3%cut-off 0.2556: 1,505 right + 670 wrong answered, 23 held back, attempt rate 99.0% | 67.0%1,473 of 2,198 right | 5.6% | 68.7%run 640 | run 642 |
| KnowLine-9B-Gen2 Open · 9.0B | 2.6%27 “I don't know” of 1,057 not right | 98.8%2,171 of 2,198 answered, 27 “I don't know” | 47.4%1,030 of 2,171 | 2.6%cut-off 0.2371: 1,162 right + 1,009 wrong answered, 27 held back, attempt rate 98.8% | 51.9%1,141 of 2,198 right | 6.2% | 53.3%run 1306 | run 1325 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 2.2%24 “I don't know” of 1,093 not right | 98.9%2,174 of 2,198 answered, 24 “I don't know” | 49.2%1,069 of 2,174 | 2.2%cut-off 0.2431: 1,124 right + 1,050 wrong answered, 24 held back, attempt rate 98.9% | 50.3%1,105 of 2,198 right | 4.3% | 51.2%run 1290 | run 1297 |
| Gemini Decision 1 20261111 Closed · Efficient | 1.9%10 “I don't know” of 515 not right | 99.5%2,188 of 2,198 answered, 10 “I don't know” | 23.1%505 of 2,188 | 2.0%cut-off 0.2613: 1,702 right + 486 wrong answered, 10 held back, attempt rate 99.5% | 76.6%1,683 of 2,198 right | 5.5% | 77.5%run 1021 | run 1023 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 1.8%21 “I don't know” of 1,168 not right | 99.0%2,177 of 2,198 answered, 21 “I don't know” | 52.7%1,147 of 2,177 | 1.8%cut-off 0.2403: 1,048 right + 1,129 wrong answered, 21 held back, attempt rate 99.0% | 46.9%1,030 of 2,198 right | 5.5% | 47.9%run 1236 | run 1239 |
| Open-Jev-27B-v1.1 Open · 27B | 1.4%14 “I don't know” of 979 not right | 99.4%2,184 of 2,198 answered, 14 “I don't know” | 44.2%965 of 2,184 | 1.4%cut-off 0.2077: 1,222 right + 962 wrong answered, 14 held back, attempt rate 99.4% | 55.5%1,219 of 2,198 right | 12.7% | 55.7%run 45 | run 768 |
| d3-ultra Open · 26B | 1.4%10 “I don't know” of 714 not right | 99.5%2,188 of 2,198 answered, 10 “I don't know” | 32.2%704 of 2,188 | 1.4%cut-off 0.2876: 1,486 right + 702 wrong answered, 10 held back, attempt rate 99.5% | 67.5%1,484 of 2,198 right | 2.5% | 67.8%run 269 | run 363 |
| Copernicus-31B Open · 31B | 1.2%5 “I don't know” of 407 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 18.3%402 of 2,193 | 1.2%cut-off 0.2499: 1,796 right + 397 wrong answered, 5 held back, attempt rate 99.8% | 81.5%1,791 of 2,198 right | 2.0% | 81.8%run 1165 | run 1172 |
| Adjudex 9B v5 Open · 9.0B | 0.8%8 “I don't know” of 1,029 not right | 99.6%2,190 of 2,198 answered, 8 “I don't know” | 46.6%1,021 of 2,190 | 0.8%cut-off 0.2339: 1,167 right + 1,023 wrong answered, 8 held back, attempt rate 99.6% | 53.2%1,169 of 2,198 right | 1.6% | 53.1%run 1183 | run 1190 |
| Metask-Jev-4B Open · 4.5B | 0.8%10 “I don't know” of 1,325 not right | 99.5%2,188 of 2,198 answered, 10 “I don't know” | 60.1%1,315 of 2,188 | 0.8%cut-off 0.2132: 892 right + 1,296 wrong answered, 10 held back, attempt rate 99.5% | 39.7%873 of 2,198 right | 6.3% | 40.8%run 126 | run 743 |
| Perplexity Decider 27B v1.3 Open · 27B | 0.7%5 “I don't know” of 716 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 32.4%711 of 2,193 | 0.7%cut-off 0.2424: 1,504 right + 689 wrong answered, 5 held back, attempt rate 99.8% | 67.4%1,482 of 2,198 right | 3.2% | 68.4%run 1134 | run 1141 |
| Karpo Decision 31B Open · 31B | 0.6%3 “I don't know” of 478 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 21.6%475 of 2,195 | 0.6%cut-off 0.2622: 1,710 right + 485 wrong answered, 3 held back, attempt rate 99.9% | 78.3%1,720 of 2,198 right | 1.0% | 77.8%run 1156 | run 1163 |
| KnowLine-9B-Gen1 Open · 9.0B | 0.5%5 “I don't know” of 1,001 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 45.4%996 of 2,193 | 0.5%cut-off 0.2281: 1,198 right + 995 wrong answered, 5 held back, attempt rate 99.8% | 54.5%1,197 of 2,198 right | 2.1% | 54.5%run 1174 | run 1181 |
| Verdin-4B-v1.1 Open · 4.0B | 0.4%5 “I don't know” of 1,174 not right | 99.8%2,193 of 2,198 answered, 5 “I don't know” | 53.3%1,169 of 2,193 | 0.4%cut-off 0.2269: 1,026 right + 1,167 wrong answered, 5 held back, attempt rate 99.8% | 46.6%1,024 of 2,198 right | 3.2% | 46.8%run 1244 | run 1251 |
| REAX 1.3 Open · 57B | 0.4%2 “I don't know” of 477 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 21.6%475 of 2,196 | 0.4%cut-off 0.2462: 1,712 right + 484 wrong answered, 2 held back, attempt rate 99.9% | 78.3%1,721 of 2,198 right | 1.9% | 77.9%run 827 | run 833 |
| Perplexity Decider 27B v1.2 API Open · 27B | 0.4%3 “I don't know” of 729 not right | 99.9%2,195 of 2,198 answered, 3 “I don't know” | 33.1%726 of 2,195 | 0.4%cut-off 0.2252: 1,457 right + 738 wrong answered, 3 held back, attempt rate 99.9% | 66.8%1,469 of 2,198 right | 2.7% | 66.3%run 633 | run 641 |
| Logits Sigma-1 Duo Open · 57B | 0.4%2 “I don't know” of 545 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 24.7%543 of 2,196 | 0.4%cut-off 0.2460: 1,649 right + 547 wrong answered, 2 held back, attempt rate 99.9% | 75.2%1,653 of 2,198 right | 1.9% | 75.0%run 1221 | run 1233 |
| Drex Max Open · 27B | 0.3%2 “I don't know” of 713 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 32.4%711 of 2,196 | 0.3%cut-off 0.2357: 1,489 right + 707 wrong answered, 2 held back, attempt rate 99.9% | 67.6%1,485 of 2,198 right | 1.5% | 67.7%run 322 | run 367 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 0.3%2 “I don't know” of 719 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 32.7%717 of 2,196 | 0.3%cut-off 0.2205: 1,474 right + 722 wrong answered, 2 held back, attempt rate 99.9% | 67.3%1,479 of 2,198 right | 2.4% | 67.1%run 217 | run 372 |
| KnowLine-31B-Gen1.5 Open · 31B | 0.2%1 “I don't know” of 412 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 18.7%411 of 2,197 | 0.2%cut-off 0.2517: 1,797 right + 400 wrong answered, 1 held back, attempt rate 100.0% | 81.3%1,786 of 2,198 right | 0.6% | 81.8%run 957 | run 959 |
| Cerno 9B v1.2 Open · 9.0B | 0.2%2 “I don't know” of 1,000 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 45.4%998 of 2,196 | 0.2%cut-off 0.2371: 1,197 right + 999 wrong answered, 2 held back, attempt rate 99.9% | 54.5%1,198 of 2,198 right | 2.8% | 54.5%run 1201 | run 1204 |
| KnowLine-12B-Gen1 Open · 12B | 0.1%1 “I don't know” of 805 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 36.6%804 of 2,197 | 0.1%cut-off 0.2147: 1,373 right + 824 wrong answered, 1 held back, attempt rate 100.0% | 63.4%1,393 of 2,198 right | 1.1% | 62.5%run 1309 | run 1318 |
| OneJev-27B Open · 27B | 0.1%1 “I don't know” of 893 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 40.6%892 of 2,197 | 0.1%cut-off 0.2094: 1,310 right + 887 wrong answered, 1 held back, attempt rate 100.0% | 59.4%1,305 of 2,198 right | 2.1% | 59.6%run 264 | run 782 |
| Valen-Decision-1.5-9B Open · 9.0B | 0.1%1 “I don't know” of 1,158 not right | 100.0%2,197 of 2,198 answered, 1 “I don't know” | 52.7%1,157 of 2,197 | 0.1%cut-off 0.2162: 1,031 right + 1,166 wrong answered, 1 held back, attempt rate 100.0% | 47.3%1,040 of 2,198 right | 4.8% | 47.0%run 1275 | run 1282 |
| KnowLine-31B-Gen2 Open · 31B | 0.0%0 “I don't know” of 408 not right | 100.0%2,198 of 2,198 answered, 0 “I don't know” | 18.6%408 of 2,198 | 0.0%cut-off 0.2259: 1,785 right + 413 wrong answered, 0 held back, attempt rate 100.0% | 81.4%1,790 of 2,198 right | 0.2% | 81.2%run 1314 | run 1331 |
| Sieve 31B v3 Open · 31B | 0.0%0 “I don't know” of 553 not right | 100.0%2,198 of 2,198 answered, 0 “I don't know” | 25.2%553 of 2,198 | 0.0%cut-off 0.2191: 1,630 right + 568 wrong answered, 0 held back, attempt rate 100.0% | 74.8%1,645 of 2,198 right | 0.9% | 74.2%run 1241 | run 1253 |
| d3 Open · 26B | 0.0%0 “I don't know” of 715 not right | 100.0%2,198 of 2,198 answered, 0 “I don't know” | 32.5%715 of 2,198 | 0.0%cut-off 0.2225: 1,490 right + 708 wrong answered, 0 held back, attempt rate 100.0% | 67.5%1,483 of 2,198 right | 2.9% | 67.8%run 259 | run 358 |
Not run
- Perplexity Decider 27B v1.1: Perplexity's API now answers requests for v1.1 with v1.2, so a new run could not measure v1.1.
- SubQ Snap Preview: SubQ's API returns its probabilities in 32-bit precision, so their sums miss our 7-decimal check. An allowance for that was approved on 5 November 2026 for its English runs; a Turkish run with “Bilmiyorum” was not part of that request.
Running
Running: a chat model's runs with the option go English first, then Turkish, and each is added as it finishes. The English run of each is on the Turkish vs English tab.
- Claude Fable 5.2 max · chat
Running
Running: a chat model's runs with the option go English first, then Turkish, and each is added as it finishes. The English run of each is on the Turkish vs English tab.
- Claude Fable 5.2 max · chat
What we found
Of the 63 models, 38 used “I don't know”, on 1.0% to 94.2% of the questions. The other 25, Turion Jev 31B, Gemini Decision 1 · 20261111, StartLux-Decision-v1.1-4B-Instant, Open-Jev-27B-v1.1, d3-ultra, Copernicus-31B, Adjudex 9B v5, Metask-Jev-4B, Perplexity Decider 27B v1.3, Karpo Decision 31B, KnowLine-9B-Gen1, Verdin-4B-v1.1, REAX 1.3, Perplexity Decider 27B v1.2 · API, Logits Sigma-1 Duo, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark, KnowLine-31B-Gen1.5, Cerno 9B v1.2, KnowLine-12B-Gen1, OneJev-27B, Valen-Decision-1.5-9B, KnowLine-31B-Gen2, Sieve 31B v3 and d3, chose it on 0 to 21 questions, although each gave it some probability: a mean of 0.2% to 12.7%, against 2.7% to 39.1% for the 38.
GLiDE 0.4B had the highest non-hallucination rate, 95.3%: of its 2,171 answers that were not right, 2,070 were “I don't know”. It answered 5.8% of the questions, with an error rate of 78.9% among those answers. d1.2-ultra (87.3%) and d1.2 (79.4%) follow. With the option, the 38 were less accurate than in their five-option runs on the Text tab, by 0.6 points (Jeffan-9B-1211) to 21.0 points (GLiDE 0.4B: 22.2% against 1.2%). The 25 that almost never chose it moved by -1.0 to +0.9 points.
For 34 of the 38, the model's own confidence on its five-option run, holding back on as many questions (for Laya, 537 against 536: tied scores at the cut-off allow no nearer count), left fewer wrong answers than “I don't know” did: 1 to 40 fewer, a non-hallucination rate 0.0 to 3.6 points higher (the derived rate in the table). GLiDE · API, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant and KnowLine-0.8B-Gen1 were the exceptions, with 21, 19, 5 and 1 more. For the 25 that almost never chose it, both rates are 3.4% or less. In the exact McNemar test on each model's first run, with Holm's correction over the 38, none of these differences is significant: each could be chance.
With “I don't know” in place of “Bilmiyorum”, we compared each model's first run with “I don't know” against its five-option run held back by its own confidence at the same attempt rate (each cut-off reached the option run's answered count exactly for the 9 that chose “I don't know” on at least 1% of the questions). Confidence left fewer wrong answers for all 9, 9 to 49 fewer. In the exact McNemar test of wrong answers, with Holm's correction over the 9, confidence left significantly fewer for d1-ultra (216 wrong with “I don't know”, 167 by confidence; adjusted p = 0.00059); for the other 8, the difference could be chance. GLiDE · API is compared with its new five-option run 517, not run 129.
Held to d1-ultra's number of answers with “I don't know” (k = 1,582 in Turkish and 496 in English), each model keeping its k most confident five-option answers, d1-ultra's five-option run got 10.6% of its k answers wrong in Turkish and 5.8% in English; by a paired bootstrap with Holm's correction, in Turkish, of the 76 other decision models, 4 had significantly fewer, 67 significantly more and 5 could be chance, while of the 50 chat and general models, 27 had significantly fewer, 19 significantly more and 4 could be chance (Qwen3.5-4B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Gemma 4 E4B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Claude Haiku 5.5 · max · chat is left out: its run has 1,414 scored answers, fewer than k = 1,582; Qwen3.5-4B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-2.6B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-8B-A1B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; MiniCPM5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by); in English, of the 123 other models, 17 had significantly fewer, 43 significantly more and 63 could be chance (Qwen3.5-4B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Gemma 4 E4B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-4B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-2.6B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-8B-A1B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; MiniCPM5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by) (The same number of answers).
Chat models: 56 setups have a run with “I don't know” so far, with a non-hallucination rate from 0.0% (DeepSeek-V4.1-Flash · reasoning off · chat · 1017) to 76.2% (LFM3-1.7B · reasoning off · chat), against 0.0% to 95.3% for the 63 decision models. One more chat model setup is still running in Turkish (Running, under Results).
Our method
We measure hallucination by offering a way out. Every question gets “I don't know” as a sixth option, and the instructions tell the model to choose it when it does not know the answer. A hallucination is then a wrong answer: the model chose one of the question's own options, not the key, when it could have said it did not know. Everything else is the Text tab's: the questions, the answer key and each model's settings.
How it differs from our confidence method
On the Text tab a model must pick one of the five options, and we judge each answer by its confidence: the probability the model gives the option it picked (its option score), or for a chat model the confidence it states. A chat model asked for a plain answer line (Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM3-1.7B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat and MiniCPM5-2B · reasoning off · chat) states none, so its derived rate is not measured. Our confidence method keeps an answer when that confidence reaches a cut-off, and a stronger model answers the rest (How we chose which answers to keep). Here no cut-off decides: the model can say itself that it does not know, by choosing “I don't know”, and the rates count those choices. The two meet in the derived non-hallucination rate, which applies the confidence method to the same model's five-option run at about the same attempt rate; side by side, the two non-hallucination columns compare a model that says it does not know with the same model held back by its own confidence.
The rates
Each answer is right, wrong, “I don't know” or invalid; an invalid answer counts as none of the others. Out of all 2,198 questions:
- Accuracy = right / 2,198.
- Answered (the attempt rate) = (right + wrong) / 2,198: how often the model chose one of the question's own options.
- Error rate = wrong / (right + wrong): among the answers the model gave, the share that were wrong. With no answer it is undefined.
- Non-hallucination rate = “I don't know” / (“I don't know” + wrong): among the answers that were wrong or “I don't know”, the share where the model said it did not know. The hallucination rate is its complement, wrong / (“I don't know” + wrong).
- Mean probability of “I don't know”: the probability the model gave that option, averaged over its answers that report option probabilities. A chat model states a confidence and reports no option probabilities. A chat model asked for a plain answer line reports neither: not measured.
How to read them
The non-hallucination rate says how a model is wrong, not how often it is right, and it is not a direct measure of what the model does not know. Read it with accuracy and the attempt rate: a model that always chose “I don't know” would reach 100% with no right answers, and one that never chose it would reach 0% as soon as it answered one question wrongly. With neither a wrong answer nor “I don't know”, there is nothing to divide by and the rate is undefined (the table shows Undefined).
The option and the instructions
The added option's exact text in the prompt is “Bilmiyorum”. It is lettered after the question's own options, and the answer key is unchanged. The full System One instructions of a decision model:
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose Bilmiyorum.
Prompt version 1cdd542b597f (a hash of the prompts with these instructions and this option, computed as for the Text tab's prompt version).
A chat model asked for a plain answer line (Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM3-1.7B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat and MiniCPM5-2B · reasoning off · chat) is asked as on the Text tab: one question per request, a user message with one instruction and the question, no system prompt and no answer schema, its answer the letter on the reply's last “CEVAP: X” line. With the option, F is among the letters of its instruction and one sentence is added at its end: “Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.” Its instruction in full ({letters} is the question's own letters):
Aşağıdaki çoktan seçmeli soruyu çöz. Yanıtının son satırına yalnızca 'CEVAP: X' yaz (X = {letters}). Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.
Prompt version 711479895d19 (the plain answer line's prompts with the sentence and the option, hashed the same way). It states no confidence and reports no option probabilities, so its derived non-hallucination rate and its mean probability of “I don't know” are not measured.
Any other chat model is asked as on the Text tab, with the option after each question's own and one sentence added at the end of each instruction of its prompts: “Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.” Its system prompt and instruction when it states a confidence, in full:
You are taking a Turkish multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.
Aşağıdaki çoktan seçmeli soruların her birini yanıtlayın. Açıklama yapmadan, her soru için doğru seçeneğin harfini ve cevabınızın doğru olduğundan ne kadar emin olduğunuzu gösteren 0 ile 1 arasında bir güven puanını belirtilen JSON biçiminde birlikte döndürün. Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.
Prompt version d3c64e8c862e (the chat models' prompts with the sentence and the option, hashed the same way).
Derived non-hallucination rate
For comparison, each model's run with five options (its run on the Text tab, on the same questions) is made to abstain by its own confidence: it answers only where its score (a decision model's option score, a chat model's stated confidence) reaches a cut-off and abstains on the rest. The cut-off is the one whose attempt rate comes nearest the attempt rate of the run with “I don't know”; on a tie, the higher cut-off. An answer without a score never counts as an attempt. The derived non-hallucination rate is then the abstentions over the abstentions plus the wrong answers. The table shows the cut-off and the attempt rate it reached. A chat model asked for a plain answer line (Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM3-1.7B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat and MiniCPM5-2B · reasoning off · chat) states none, so its derived rate is not measured. The paired test of the two is among the decision models.
Repeats
One full run per model. A model whose repeated text runs changed answers gets five runs with the option. Those runs form a repeat set of their own and are never mixed with the Text tab's runs. The table shows the first run; the range covers all of them, and the changed answers are the scored questions whose answer differs between the runs.
Results on the English questions with “I don't know”
Each model's first run in English and in Turkish, each with the “I don't know” option in its language, on the 547 kept reading questions: the counts, the rates and the exact McNemar test of wrong answers, with p adjusted by Holm's method over the 67 models; * marks a Holm-adjusted p below 0.05. The same table is on the Turkish vs English tab.
| Model | English non-hallucinationof answers not right, % “I don't know” · higher is better, read with the answered share | English answerednot “I don't know”: (right + wrong) / 547 | English error rateof its answers: wrong / (right + wrong) · lower is better | Turkish non-hallucinationof answers not right, % the Turkish option · higher is better, read with the answered share | Turkish answerednot the Turkish option: (right + wrong) / 547 | Turkish error rateof its answers: wrong / (right + wrong) · lower is better | English runswith “I don't know”; the first is compared | Turkish runswith the Turkish option; the first is compared | English accuracyhigher is better | Turkish accuracyhigher is better | English mean probabilityof “I don't know” | Turkish mean probabilityof the Turkish option | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 67 decision models |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Efficient | 89.3%109 “I don't know” of 122 not right | 80.1%438 of 547 answered, 109 “I don't know” | 3.0%13 of 438 | 76.2%61 “I don't know” of 80 not right | 88.8%486 of 547 answered, 61 “I don't know” | 3.9%19 of 486 | run 1363a new run | run 1367the Hallucination tab's full runs, counted on the kept questions | 77.7%425 of 547 right | 85.4%467 of 547 right | 26.0% | 17.6% | 6 | 12 | 0.24 | 1 |
| celeris-1.1-decision Closed · Efficient | 74.8%98 “I don't know” of 131 not right | 82.1%449 of 547 answered, 98 “I don't know” | 7.3%33 of 449 | Not run | Not run | Not run | run 884a new run | Not run | 76.1%416 of 547 right | Not run | 13.3% | Not run | Not run | Not run | Not run | Not run |
| celeris-1-decision Closed · Efficient | 71.3%92 “I don't know” of 129 not right | 83.2%455 of 547 answered, 92 “I don't know” | 8.1%37 of 455 | Not run | Not run | Not run | run 548a new run | Not run | 76.4%418 of 547 right | Not run | 12.7% | Not run | Not run | Not run | Not run | Not run |
| d1.2 Closed · Efficient | 61.5%59 “I don't know” of 96 not right | 89.2%488 of 547 answered, 59 “I don't know” | 7.6%37 of 488 | 60.2%53 “I don't know” of 88 not right | 90.3%494 of 547 answered, 53 “I don't know” | 7.1%35 of 494 | run 1361a new run | run 1365the Hallucination tab's full runs, counted on the kept questions | 82.4%451 of 547 right | 83.9%459 of 547 right | 19.4% | 17.1% | 21 | 19 | 0.87 | 1 |
| d1-2B Open · 1.9B | 60.9%151 “I don't know” of 248 not right | 72.4%396 of 547 answered, 151 “I don't know” | 24.5%97 of 396 | 43.8%161 “I don't know” of 368 not right | 70.6%386 of 547 answered, 161 “I don't know” | 53.6%207 of 386 | run 710a new run | run 712the Hallucination tab's full runs, counted on the kept questions | 54.7%299 of 547 right | 32.7%179 of 547 right | 26.9% | 23.7% | 38 | 148 | < 0.0001 | < 0.0001 * |
| d1-ultra Closed · Efficient | 56.7%51 “I don't know” of 90 not rightrange 55.3% to 61.3% | 90.7%496 of 547 answered, 51 “I don't know”range 89.6% to 91.4% | 7.9%39 of 496 | 53.6%30 “I don't know” of 56 not rightrange 44.4% to 59.3% | 94.5%517 of 547 answered, 30 “I don't know”range 94.1% to 95.6% | 5.0%26 of 517 | runs 449, 456, 462, 466, 474new runs | runs 350, 354, 360, 365, 370the Hallucination tab's full runs, counted on the kept questions | 83.5%457 of 547 rightrange 82.6% to 84.5% | 89.8%491 of 547 rightrange 89.2% to 90.1% | 25.3%range 25.3% to 25.6% | 22.8%range 22.4% to 23.0% | 20 | 7 | 0.019 | 0.88 |
| GLiDE 0.4B Open · 0.4B | 53.1%153 “I don't know” of 288 not right | 72.0%394 of 547 answered, 153 “I don't know” | 34.3%135 of 394 | 89.6%475 “I don't know” of 530 not right | 13.2%72 of 547 answered, 475 “I don't know” | 76.4%55 of 72 | run 935a new run | run 937the Hallucination tab's full runs, counted on the kept questions | 47.3%259 of 547 right | 3.1%17 of 547 right | 23.3% | 35.2% | 125 | 45 | < 0.0001 | < 0.0001 * |
| Drex Max v1.1 Closed · Efficient (provisional) | 51.2%43 “I don't know” of 84 not right | 92.1%504 of 547 answered, 43 “I don't know” | 8.1%41 of 504 | 15.0%9 “I don't know” of 60 not right | 98.4%538 of 547 answered, 9 “I don't know” | 9.5%51 of 538 | run 447a new run | run 352the Hallucination tab's full runs, counted on the kept questions | 84.6%463 of 547 right | 89.0%487 of 547 right | 14.3% | 8.4% | 20 | 30 | 0.2 | 1 |
| Mammoth-1 Open · 27B | 50.7%37 “I don't know” of 73 not right | 93.2%510 of 547 answered, 37 “I don't know” | 7.1%36 of 510 | 42.1%24 “I don't know” of 57 not right | 95.6%523 of 547 answered, 24 “I don't know” | 6.3%33 of 523 | run 505a new run | run 386the Hallucination tab's full runs, counted on the kept questions | 86.7%474 of 547 right | 89.6%490 of 547 right | 9.8% | 7.4% | 18 | 15 | 0.73 | 1 |
| GLiDE v1.2 Open · 27B | 45.5%30 “I don't know” of 66 not right | 94.5%517 of 547 answered, 30 “I don't know” | 7.0%36 of 517 | 11.3%6 “I don't know” of 53 not right | 98.9%541 of 547 answered, 6 “I don't know” | 8.7%47 of 541 | run 523a new run | run 369the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.3%494 of 547 right | 12.5% | 8.7% | 16 | 27 | 0.13 | 1 |
| Laya Multilingual Open · 0.3B | 42.6%208 “I don't know” of 488 not right | 62.0%339 of 547 answered, 208 “I don't know” | 82.6%280 of 339 | 25.7%124 “I don't know” of 483 not right | 77.3%423 of 547 answered, 124 “I don't know” | 84.9%359 of 423 | run 734a new run | run 737the Hallucination tab's full runs, counted on the kept questions | 10.8%59 of 547 right | 11.7%64 of 547 right | 34.7% | 21.6% | 51 | 130 | < 0.0001 | < 0.0001 * |
| GLiDE 27B Open · 27B | 37.9%22 “I don't know” of 58 not right | 96.0%525 of 547 answered, 22 “I don't know” | 6.9%36 of 525 | 18.8%9 “I don't know” of 48 not right | 98.4%538 of 547 answered, 9 “I don't know” | 7.2%39 of 538 | run 903a new run | run 905the Hallucination tab's full runs, counted on the kept questions | 89.4%489 of 547 right | 91.2%499 of 547 right | 9.3% | 6.5% | 15 | 18 | 0.73 | 1 |
| OneJev1.6-0.8B Open · 0.8B | 28.4%74 “I don't know” of 261 not right | 86.5%473 of 547 answered, 74 “I don't know” | 39.5%187 of 473 | 30.7%112 “I don't know” of 365 not right | 79.5%435 of 547 answered, 112 “I don't know” | 58.2%253 of 435 | run 1027a new run | run 1031the Hallucination tab's full runs, counted on the kept questions | 52.3%286 of 547 right | 33.3%182 of 547 right | 10.7% | 15.6% | 55 | 121 | < 0.0001 | < 0.0001 * |
| GLiDE 2B Open · 1.9B | 26.3%47 “I don't know” of 179 not right | 91.4%500 of 547 answered, 47 “I don't know” | 26.4%132 of 500 | 24.0%66 “I don't know” of 275 not right | 87.9%481 of 547 answered, 66 “I don't know” | 43.5%209 of 481 | run 919a new run | run 921the Hallucination tab's full runs, counted on the kept questions | 67.3%368 of 547 right | 49.7%272 of 547 right | 12.5% | 15.4% | 48 | 125 | < 0.0001 | < 0.0001 * |
| Elephant-1 Open · 9.0B | 25.6%23 “I don't know” of 90 not right | 95.8%524 of 547 answered, 23 “I don't know” | 12.8%67 of 524 | 15.5%15 “I don't know” of 97 not right | 97.3%532 of 547 answered, 15 “I don't know” | 15.4%82 of 532 | run 501a new run | run 382the Hallucination tab's full runs, counted on the kept questions | 83.5%457 of 547 right | 82.3%450 of 547 right | 6.8% | 5.7% | 31 | 46 | 0.11 | 1 |
| SubQ Snap 1.1 Closed · Efficient | 25.0%21 “I don't know” of 84 not right | 96.2%526 of 547 answered, 21 “I don't know” | 12.0%63 of 526 | 27.1%19 “I don't know” of 70 not right | 96.5%528 of 547 answered, 19 “I don't know” | 9.7%51 of 528 | run 448a new run | run 392the Hallucination tab's full runs, counted on the kept questions | 84.6%463 of 547 right | 87.2%477 of 547 right | 11.7% | 8.4% | 39 | 27 | 0.18 | 1 |
| KnowLine-2B-Gen1 Open · 2.0B | 22.0%38 “I don't know” of 173 not right | 93.1%509 of 547 answered, 38 “I don't know” | 26.5%135 of 509 | Not run | Not run | Not run | run 1059a new run | Not run | 68.4%374 of 547 right | Not run | 7.3% | Not run | Not run | Not run | Not run | Not run |
| GLiDE 0.8B Open · 0.8B | 21.7%49 “I don't know” of 226 not right | 91.0%498 of 547 answered, 49 “I don't know” | 35.5%177 of 498 | 24.3%74 “I don't know” of 304 not right | 86.5%473 of 547 answered, 74 “I don't know” | 48.6%230 of 473 | run 927a new run | run 929the Hallucination tab's full runs, counted on the kept questions | 58.7%321 of 547 right | 44.4%243 of 547 right | 14.8% | 15.4% | 68 | 121 | 0.00014 | 0.0079 * |
| GLiClass v4 Edge Open · 0.4B | 21.0%52 “I don't know” of 248 not right | 90.5%495 of 547 answered, 52 “I don't know” | 39.6%196 of 495 | 14.1%58 “I don't know” of 410 not right | 89.4%489 of 547 answered, 58 “I don't know” | 72.0%352 of 489 | run 1044a new run | run 1048the Hallucination tab's full runs, counted on the kept questions | 54.7%299 of 547 right | 25.0%137 of 547 right | 11.8% | 12.0% | 48 | 204 | < 0.0001 | < 0.0001 * |
| Laya Open · 0.4B | 18.4%83 “I don't know” of 450 not right | 84.8%464 of 547 answered, 83 “I don't know” | 79.1%367 of 464 | 12.0%53 “I don't know” of 443 not right | 90.3%494 of 547 answered, 53 “I don't know” | 78.9%390 of 494 | run 728a new run | run 730the Hallucination tab's full runs, counted on the kept questions | 17.7%97 of 547 right | 19.0%104 of 547 right | 14.2% | 11.1% | 97 | 120 | 0.14 | 1 |
| Open-Jev-27B-v1.1 Open · 27B | 17.4%16 “I don't know” of 92 not right | 97.1%531 of 547 answered, 16 “I don't know” | 14.3%76 of 531 | 7.1%6 “I don't know” of 84 not right | 98.9%541 of 547 answered, 6 “I don't know” | 14.4%78 of 541 | run 763a new run | run 768the Hallucination tab's full runs, counted on the kept questions | 83.2%455 of 547 right | 84.6%463 of 547 right | 15.2% | 13.2% | 26 | 28 | 0.89 | 1 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 16.9%32 “I don't know” of 189 not right | 94.1%515 of 547 answered, 32 “I don't know” | 30.5%157 of 515 | 12.3%33 “I don't know” of 268 not right | 94.0%514 of 547 answered, 33 “I don't know” | 45.7%235 of 514 | run 975a new run | run 979the Hallucination tab's full runs, counted on the kept questions | 65.4%358 of 547 right | 51.0%279 of 547 right | 6.7% | 7.6% | 47 | 125 | < 0.0001 | < 0.0001 * |
| GLiDE 9B Open · 7.9B | 16.7%14 “I don't know” of 84 not right | 97.4%533 of 547 answered, 14 “I don't know” | 13.1%70 of 533 | 9.8%9 “I don't know” of 92 not right | 98.4%538 of 547 answered, 9 “I don't know” | 15.4%83 of 538 | run 911a new run | run 913the Hallucination tab's full runs, counted on the kept questions | 84.6%463 of 547 right | 83.2%455 of 547 right | 7.8% | 6.9% | 25 | 38 | 0.13 | 1 |
| GLiNER2.5-multi-Decide Open · 0.3B | 16.2%70 “I don't know” of 433 not right | 87.2%477 of 547 answered, 70 “I don't know” | 76.1%363 of 477 | 5.5%24 “I don't know” of 436 not right | 95.6%523 of 547 answered, 24 “I don't know” | 78.8%412 of 523 | run 716a new run | run 718the Hallucination tab's full runs, counted on the kept questions | 20.8%114 of 547 right | 20.3%111 of 547 right | 13.2% | 10.4% | 46 | 95 | < 0.0001 | 0.0026 * |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 15.2%10 “I don't know” of 66 not right | 98.2%537 of 547 answered, 10 “I don't know” | 10.4%56 of 537 | 5.6%3 “I don't know” of 54 not right | 99.5%544 of 547 answered, 3 “I don't know” | 9.4%51 of 544 | run 644a new run | run 642the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.1%493 of 547 right | 4.6% | 2.2% | 26 | 21 | 0.56 | 1 |
| Winnow-12B Open · 12B | 15.0%15 “I don't know” of 100 not right | 97.3%532 of 547 answered, 15 “I don't know” | 16.0%85 of 532 | 8.6%8 “I don't know” of 93 not right | 98.5%539 of 547 answered, 8 “I don't know” | 15.8%85 of 539 | run 748a new run | run 750the Hallucination tab's full runs, counted on the kept questions | 81.7%447 of 547 right | 83.0%454 of 547 right | 3.1% | 1.8% | 32 | 32 | 1 | 1 |
| Mercury Decide Closed · Efficient | 13.4%11 “I don't know” of 82 not rightrange 11.1% to 13.4% | 98.0%536 of 547 answered, 11 “I don't know”range 98.0% to 98.4% | 13.2%71 of 536 | 10.1%8 “I don't know” of 79 not rightrange 10.1% to 10.5% | 98.5%539 of 547 answered, 8 “I don't know”the same in every run | 13.2%71 of 539 | runs 450, 460, 469, 478, 484new runs | runs 489, 491, 493, 495, 497new runs on the kept questions | 85.0%465 of 547 rightrange 85.0% to 85.4% | 85.6%468 of 547 rightrange 85.6% to 86.1% | 3.2%range 3.1% to 3.2% | 1.9%range 1.9% to 2.0% | 29 | 29 | 1 | 1 |
| Cygnet Open · 12B | 11.2%10 “I don't know” of 89 not right | 98.2%537 of 547 answered, 10 “I don't know” | 14.7%79 of 537 | 5.8%5 “I don't know” of 86 not right | 99.1%542 of 547 answered, 5 “I don't know” | 14.9%81 of 542 | run 755a new run | run 757the Hallucination tab's full runs, counted on the kept questions | 83.7%458 of 547 right | 84.3%461 of 547 right | 3.5% | 2.8% | 28 | 30 | 0.9 | 1 |
| KnowLine-0.8B-Gen1 Open · 0.8B | 10.7%26 “I don't know” of 243 not right | 95.2%521 of 547 answered, 26 “I don't know” | 41.7%217 of 521 | 4.4%15 “I don't know” of 340 not right | 97.3%532 of 547 answered, 15 “I don't know” | 61.1%325 of 532 | run 1003a new run | run 1007the Hallucination tab's full runs, counted on the kept questions | 55.6%304 of 547 right | 37.8%207 of 547 right | 5.8% | 6.0% | 49 | 157 | < 0.0001 | < 0.0001 * |
| GPT-6 Luna Decisions API Closed · Efficient | 9.4%10 “I don't know” of 106 not right | 98.2%537 of 547 answered, 10 “I don't know” | 17.9%96 of 537 | 7.1%6 “I don't know” of 84 not right | 98.9%541 of 547 answered, 6 “I don't know” | 14.4%78 of 541 | run 444a new run | run 453a new run on the kept questions | 80.6%441 of 547 right | 84.6%463 of 547 right | 3.0% | 2.7% | 47 | 29 | 0.05 | 1 |
| Gemini Decision 1 20261111 Closed · Efficient | 9.1%5 “I don't know” of 55 not right | 99.1%542 of 547 answered, 5 “I don't know” | 9.2%50 of 542 | 10.9%5 “I don't know” of 46 not right | 99.1%542 of 547 answered, 5 “I don't know” | 7.6%41 of 542 | run 1019a new run | run 1023the Hallucination tab's full runs, counted on the kept questions | 89.9%492 of 547 right | 91.6%501 of 547 right | 4.6% | 4.2% | 20 | 11 | 0.15 | 1 |
| Perplexity Decider 27B v1.3 Open · 27B | 8.8%5 “I don't know” of 57 not right | 99.1%542 of 547 answered, 5 “I don't know” | 9.6%52 of 542 | 0.0%0 “I don't know” of 52 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.5%52 of 547 | run 1139a new run | run 1141the Hallucination tab's full runs, counted on the kept questions | 89.6%490 of 547 right | 90.5%495 of 547 right | 3.8% | 1.7% | 24 | 24 | 1 | 1 |
| Turion Jev 31B Open · 31B | 8.5%4 “I don't know” of 47 not right | 99.3%543 of 547 answered, 4 “I don't know” | 7.9%43 of 543 | 5.7%2 “I don't know” of 35 not right | 99.6%545 of 547 answered, 2 “I don't know” | 6.1%33 of 545 | run 1077a new run | run 1285the Hallucination tab's full runs, counted on the kept questions | 91.4%500 of 547 right | 93.6%512 of 547 right | 2.3% | 1.6% | 21 | 11 | 0.11 | 1 |
| d3-ultra Open · 26B | 8.2%5 “I don't know” of 61 not right | 99.1%542 of 547 answered, 5 “I don't know” | 10.3%56 of 542 | 2.1%1 “I don't know” of 47 not right | 99.8%546 of 547 answered, 1 “I don't know” | 8.4%46 of 546 | run 795a new run | run 363the Hallucination tab's full runs, counted on the kept questions | 88.8%486 of 547 right | 91.4%500 of 547 right | 2.8% | 1.0% | 33 | 23 | 0.23 | 1 |
| vera-medium-omni Open · 4.2B | 7.4%7 “I don't know” of 94 not right | 98.7%540 of 547 answered, 7 “I don't know” | 16.1%87 of 540 | 4.2%5 “I don't know” of 118 not right | 99.1%542 of 547 answered, 5 “I don't know” | 20.8%113 of 542 | run 1269a new run | run 1271the Hallucination tab's full runs, counted on the kept questions | 82.8%453 of 547 right | 78.4%429 of 547 right | 5.7% | 5.3% | 26 | 52 | 0.0043 | 0.23 |
| Perplexity Decider Flash v1.3 Open · 4.0B | 7.2%7 “I don't know” of 97 not right | 98.7%540 of 547 answered, 7 “I don't know” | 16.7%90 of 540 | 7.6%9 “I don't know” of 119 not right | 98.4%538 of 547 answered, 9 “I don't know” | 20.4%110 of 538 | run 1128a new run | run 1130the Hallucination tab's full runs, counted on the kept questions | 82.3%450 of 547 right | 78.2%428 of 547 right | 5.1% | 5.0% | 37 | 57 | 0.049 | 1 |
| REAX 1.3 Open · 57B | 6.7%3 “I don't know” of 45 not right | 99.5%544 of 547 answered, 3 “I don't know” | 7.7%42 of 544 | 2.8%1 “I don't know” of 36 not right | 99.8%546 of 547 answered, 1 “I don't know” | 6.4%35 of 546 | run 831a new run | run 833the Hallucination tab's full runs, counted on the kept questions | 91.8%502 of 547 right | 93.4%511 of 547 right | 1.9% | 0.7% | 20 | 13 | 0.3 | 1 |
| Clef-Flash Open · 9.0B | 5.3%5 “I don't know” of 95 not right | 99.1%542 of 547 answered, 5 “I don't know” | 16.6%90 of 542 | 9.4%12 “I don't know” of 127 not right | 97.8%535 of 547 answered, 12 “I don't know” | 21.5%115 of 535 | run 660a new run | run 664the Hallucination tab's full runs, counted on the kept questions | 82.6%452 of 547 right | 76.8%420 of 547 right | 3.1% | 4.8% | 27 | 52 | 0.0066 | 0.33 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 5.2%12 “I don't know” of 229 not right | 97.8%535 of 547 answered, 12 “I don't know” | 40.6%217 of 535 | 6.4%22 “I don't know” of 342 not right | 96.0%525 of 547 answered, 22 “I don't know” | 61.0%320 of 525 | run 1013a new run | run 1017the Hallucination tab's full runs, counted on the kept questions | 58.1%318 of 547 right | 37.5%205 of 547 right | 6.6% | 7.3% | 48 | 151 | < 0.0001 | < 0.0001 * |
| Perplexity Decider Flash v1.2 Open · 4.0B | 4.8%5 “I don't know” of 105 not right | 99.1%542 of 547 answered, 5 “I don't know” | 18.5%100 of 542 | 4.3%6 “I don't know” of 139 not right | 98.9%541 of 547 answered, 6 “I don't know” | 24.6%133 of 541 | run 535a new run | run 537a new run on the kept questions | 80.8%442 of 547 right | 74.6%408 of 547 right | 5.5% | 6.3% | 36 | 69 | 0.0017 | 0.091 |
| Perplexity Decider 27B v1.2 API Open · 27B | 4.5%3 “I don't know” of 66 not right | 99.5%544 of 547 answered, 3 “I don't know” | 11.6%63 of 544 | 0.0%0 “I don't know” of 51 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.3%51 of 547 | run 643a new run | run 641the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.7%496 of 547 right | 3.1% | 1.1% | 27 | 15 | 0.088 | 1 |
| Logits Sigma-1 Duo Open · 57B | 4.3%2 “I don't know” of 46 not right | 99.6%545 of 547 answered, 2 “I don't know” | 8.1%44 of 545 | 0.0%0 “I don't know” of 36 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.6%36 of 547 | run 1231a new run | run 1233the Hallucination tab's full runs, counted on the kept questions | 91.6%501 of 547 right | 93.4%511 of 547 right | 1.8% | 0.8% | 25 | 17 | 0.28 | 1 |
| Jev 1.14 Closed · Efficient | 4.0%3 “I don't know” of 75 not rightrange 4.0% to 5.2% | 99.5%544 of 547 answered, 3 “I don't know”range 99.3% to 99.5% | 13.2%72 of 544 | 4.2%3 “I don't know” of 72 not rightrange 4.0% to 4.5% | 99.5%544 of 547 answered, 3 “I don't know”the same in every run | 12.7%69 of 544 | runs 446, 454, 457, 461, 464new runs | runs 470, 473, 476, 480, 482new runs on the kept questions | 86.3%472 of 547 rightrange 85.9% to 86.7% | 86.8%475 of 547 rightrange 86.3% to 87.8% | 1.6%range 1.6% to 1.6% | 1.7%range 1.6% to 1.7% | 20 | 17 | 0.74 | 1 |
| OneJev-27B Open · 27B | 4.0%3 “I don't know” of 75 not right | 99.5%544 of 547 answered, 3 “I don't know” | 13.2%72 of 544 | 1.6%1 “I don't know” of 64 not right | 99.8%546 of 547 answered, 1 “I don't know” | 11.5%63 of 546 | run 780a new run | run 782the Hallucination tab's full runs, counted on the kept questions | 86.3%472 of 547 right | 88.3%483 of 547 right | 2.3% | 1.3% | 34 | 25 | 0.3 | 1 |
| Microsoft-Decision-1 Closed · Efficient | 3.8%3 “I don't know” of 80 not rightrange 3.8% to 4.0% | 99.5%544 of 547 answered, 3 “I don't know”the same in every run | 14.2%77 of 544 | 2.2%2 “I don't know” of 90 not rightrange 2.2% to 4.3% | 99.6%545 of 547 answered, 2 “I don't know”range 99.3% to 99.6% | 16.1%88 of 545 | runs 451, 459, 467, 477, 483new runs | runs 487, 490, 492, 494, 496new runs on the kept questions | 85.4%467 of 547 rightrange 85.4% to 86.3% | 83.5%457 of 547 rightrange 82.8% to 83.5% | 3.6%range 3.6% to 3.6% | 2.3%range 2.3% to 2.3% | 24 | 35 | 0.19 | 1 |
| Chinese Decision Pointer 4B Open · 4.0B | 3.7%4 “I don't know” of 108 not right | 99.3%543 of 547 answered, 4 “I don't know” | 19.2%104 of 543 | 3.0%4 “I don't know” of 133 not right | 99.3%543 of 547 answered, 4 “I don't know” | 23.8%129 of 543 | run 1211a new run | run 1213the Hallucination tab's full runs, counted on the kept questions | 80.3%439 of 547 right | 75.7%414 of 547 right | 2.1% | 1.9% | 41 | 66 | 0.02 | 0.9 |
| Drex Max Open · 27B | 3.2%2 “I don't know” of 62 not right | 99.6%545 of 547 answered, 2 “I don't know” | 11.0%60 of 545 | 3.6%2 “I don't know” of 55 not right | 99.6%545 of 547 answered, 2 “I don't know” | 9.7%53 of 545 | run 798a new run | run 367the Hallucination tab's full runs, counted on the kept questions | 88.7%485 of 547 right | 89.9%492 of 547 right | 1.7% | 0.6% | 28 | 21 | 0.39 | 1 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 3.0%2 “I don't know” of 66 not right | 99.6%545 of 547 answered, 2 “I don't know” | 11.7%64 of 545 | 0.0%0 “I don't know” of 50 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.1%50 of 547 | run 529a new run | run 372the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.9%497 of 547 right | 2.8% | 0.9% | 29 | 15 | 0.049 | 1 |
| d1 Closed · Efficient | 2.9%2 “I don't know” of 69 not rightrange 2.6% to 2.9% | 99.6%545 of 547 answered, 2 “I don't know”the same in every run | 12.3%67 of 545 | 1.1%1 “I don't know” of 90 not rightrange 0.0% to 1.1% | 99.8%546 of 547 answered, 1 “I don't know”range 99.8% to 100.0% | 16.3%89 of 546 | runs 445, 455, 458, 463, 465new runs | runs 472, 475, 479, 481, 485new runs on the kept questions | 87.4%478 of 547 rightrange 85.9% to 87.4% | 83.5%457 of 547 rightrange 83.0% to 85.0% | 0.9%range 0.9% to 1.0% | 1.3%range 1.1% to 1.3% | 21 | 43 | 0.0081 | 0.4 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 2.6%2 “I don't know” of 76 not right | 99.6%545 of 547 answered, 2 “I don't know” | 13.6%74 of 545 | 1.0%1 “I don't know” of 96 not right | 99.8%546 of 547 answered, 1 “I don't know” | 17.4%95 of 546 | run 1295a new run | run 1297the Hallucination tab's full runs, counted on the kept questions | 86.1%471 of 547 right | 82.4%451 of 547 right | 1.8% | 1.4% | 26 | 47 | 0.019 | 0.88 |
| Cerno 9B v1.2 Open · 9.0B | 2.5%2 “I don't know” of 79 not right | 99.6%545 of 547 answered, 2 “I don't know” | 14.1%77 of 545 | 0.0%0 “I don't know” of 91 not right | 100.0%547 of 547 answered, 0 “I don't know” | 16.6%91 of 547 | run 1087a new run | run 1204the Hallucination tab's full runs, counted on the kept questions | 85.6%468 of 547 right | 83.4%456 of 547 right | 2.2% | 1.6% | 34 | 48 | 0.15 | 1 |
| Sieve 31B v3 Open · 31B | 2.2%1 “I don't know” of 46 not right | 99.8%546 of 547 answered, 1 “I don't know” | 8.2%45 of 546 | 0.0%0 “I don't know” of 33 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.0%33 of 547 | run 1066a new run | run 1253the Hallucination tab's full runs, counted on the kept questions | 91.6%501 of 547 right | 94.0%514 of 547 right | 0.8% | 0.3% | 23 | 11 | 0.058 | 1 |
| Verdin-4B-v1.1 Open · 4.0B | 2.1%2 “I don't know” of 96 not right | 99.6%545 of 547 answered, 2 “I don't know” | 17.2%94 of 545 | 0.8%1 “I don't know” of 128 not right | 99.8%546 of 547 answered, 1 “I don't know” | 23.3%127 of 546 | run 1249a new run | run 1251the Hallucination tab's full runs, counted on the kept questions | 82.4%451 of 547 right | 76.6%419 of 547 right | 3.5% | 2.2% | 37 | 70 | 0.0018 | 0.099 |
| d3 Open · 26B | 1.7%1 “I don't know” of 59 not right | 99.8%546 of 547 answered, 1 “I don't know” | 10.6%58 of 546 | 0.0%0 “I don't know” of 53 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.7%53 of 547 | run 793a new run | run 358the Hallucination tab's full runs, counted on the kept questions | 89.2%488 of 547 right | 90.3%494 of 547 right | 2.9% | 1.1% | 24 | 19 | 0.54 | 1 |
| Metask-Jev-4B Open · 4.5B | 1.6%2 “I don't know” of 125 not right | 99.6%545 of 547 answered, 2 “I don't know” | 22.6%123 of 545 | 1.1%2 “I don't know” of 189 not right | 99.6%545 of 547 answered, 2 “I don't know” | 34.3%187 of 545 | run 741a new run | run 743the Hallucination tab's full runs, counted on the kept questions | 77.1%422 of 547 right | 65.4%358 of 547 right | 3.8% | 4.1% | 34 | 98 | < 0.0001 | < 0.0001 * |
| Clef Open · 27B | 1.3%1 “I don't know” of 76 not right | 99.8%546 of 547 answered, 1 “I don't know” | 13.7%75 of 546 | 1.5%1 “I don't know” of 68 not right | 99.8%546 of 547 answered, 1 “I don't know” | 12.3%67 of 546 | run 785a new run | run 788the Hallucination tab's full runs, counted on the kept questions | 86.1%471 of 547 right | 87.6%479 of 547 right | 2.4% | 3.6% | 26 | 18 | 0.29 | 1 |
| KnowLine-12B-Gen1 Open · 12B | 1.3%1 “I don't know” of 77 not right | 99.8%546 of 547 answered, 1 “I don't know” | 13.9%76 of 546 | 0.0%0 “I don't know” of 67 not right | 100.0%547 of 547 answered, 0 “I don't know” | 12.2%67 of 547 | run 1316a new run | run 1318the Hallucination tab's full runs, counted on the kept questions | 85.9%470 of 547 right | 87.8%480 of 547 right | 1.3% | 0.6% | 26 | 17 | 0.22 | 1 |
| KnowLine-9B-Gen1 Open · 9.0B | 1.3%1 “I don't know” of 77 not right | 99.8%546 of 547 answered, 1 “I don't know” | 13.9%76 of 546 | 0.0%0 “I don't know” of 98 not right | 100.0%547 of 547 answered, 0 “I don't know” | 17.9%98 of 547 | run 1179a new run | run 1181the Hallucination tab's full runs, counted on the kept questions | 85.9%470 of 547 right | 82.1%449 of 547 right | 1.6% | 0.7% | 18 | 40 | 0.0054 | 0.28 |
| KnowLine-9B-Gen2 Open · 9.0B | 1.3%1 “I don't know” of 79 not right | 99.8%546 of 547 answered, 1 “I don't know” | 14.3%78 of 546 | 0.0%0 “I don't know” of 95 not right | 100.0%547 of 547 answered, 0 “I don't know” | 17.4%95 of 547 | run 1323a new run | run 1325the Hallucination tab's full runs, counted on the kept questions | 85.6%468 of 547 right | 82.6%452 of 547 right | 3.8% | 3.5% | 23 | 40 | 0.043 | 1 |
| Valen-Decision-1.5-9B Open · 9.0B | 1.2%1 “I don't know” of 85 not right | 99.8%546 of 547 answered, 1 “I don't know” | 15.4%84 of 546 | 0.0%0 “I don't know” of 110 not right | 100.0%547 of 547 answered, 0 “I don't know” | 20.1%110 of 547 | run 1280a new run | run 1282the Hallucination tab's full runs, counted on the kept questions | 84.5%462 of 547 right | 79.9%437 of 547 right | 3.0% | 2.7% | 32 | 58 | 0.008 | 0.4 |
| Dexy-Decision-9B-V2 Open · 9.0B | 1.1%1 “I don't know” of 91 not right | 99.8%546 of 547 answered, 1 “I don't know” | 16.5%90 of 546 | 1.1%1 “I don't know” of 91 not right | 99.8%546 of 547 answered, 1 “I don't know” | 16.5%90 of 546 | run 1260a new run | run 1262the Hallucination tab's full runs, counted on the kept questions | 83.4%456 of 547 right | 83.4%456 of 547 right | 0.7% | 0.6% | 38 | 38 | 1 | 1 |
| KnowLine 4B Gen5.5 Open · 4.0B | 0.9%1 “I don't know” of 106 not right | 99.8%546 of 547 answered, 1 “I don't know” | 19.2%105 of 546 | 2.5%3 “I don't know” of 121 not right | 99.5%544 of 547 answered, 3 “I don't know” | 21.7%118 of 544 | run 963a new run | run 967the Hallucination tab's full runs, counted on the kept questions | 80.6%441 of 547 right | 77.9%426 of 547 right | 1.3% | 1.0% | 40 | 53 | 0.21 | 1 |
| Strands Decider 2B Open · 2.0B | 0.7%2 “I don't know” of 303 not right | 99.6%545 of 547 answered, 2 “I don't know” | 55.2%301 of 545 | 1.5%5 “I don't know” of 332 not right | 99.1%542 of 547 answered, 5 “I don't know” | 60.3%327 of 542 | run 704a new run | run 706the Hallucination tab's full runs, counted on the kept questions | 44.6%244 of 547 right | 39.3%215 of 547 right | 3.8% | 4.9% | 49 | 75 | 0.024 | 1 |
| Adjudex 9B v5 Open · 9.0B | 0.0%0 “I don't know” of 77 not right | 100.0%547 of 547 answered, 0 “I don't know” | 14.1%77 of 547 | 0.0%0 “I don't know” of 89 not right | 100.0%547 of 547 answered, 0 “I don't know” | 16.3%89 of 547 | run 1188a new run | run 1190the Hallucination tab's full runs, counted on the kept questions | 85.9%470 of 547 right | 83.7%458 of 547 right | 0.6% | 0.3% | 32 | 44 | 0.21 | 1 |
| Copernicus-31B Open · 31B | 0.0%0 “I don't know” of 39 not right | 100.0%547 of 547 answered, 0 “I don't know” | 7.1%39 of 547 | 2.9%1 “I don't know” of 34 not right | 99.8%546 of 547 answered, 1 “I don't know” | 6.0%33 of 546 | run 1170a new run | run 1172the Hallucination tab's full runs, counted on the kept questions | 92.9%508 of 547 right | 93.8%513 of 547 right | 1.0% | 0.7% | 19 | 13 | 0.38 | 1 |
| Jeffan-9B-1211 Open · 9.0B | 0.0%0 “I don't know” of 71 not right | 100.0%547 of 547 answered, 0 “I don't know” | 13.0%71 of 547 | 4.1%3 “I don't know” of 74 not right | 99.5%544 of 547 answered, 3 “I don't know” | 13.1%71 of 544 | run 1197a new run | run 1199the Hallucination tab's full runs, counted on the kept questions | 87.0%476 of 547 right | 86.5%473 of 547 right | 3.7% | 3.7% | 31 | 31 | 1 | 1 |
| Karpo Decision 31B Open · 31B | 0.0%0 “I don't know” of 42 not right | 100.0%547 of 547 answered, 0 “I don't know” | 7.7%42 of 547 | 3.2%1 “I don't know” of 31 not right | 99.8%546 of 547 answered, 1 “I don't know” | 5.5%30 of 546 | run 1161a new run | run 1163the Hallucination tab's full runs, counted on the kept questions | 92.3%505 of 547 right | 94.3%516 of 547 right | 0.6% | 0.3% | 25 | 13 | 0.073 | 1 |
| KnowLine-31B-Gen1.5 Open · 31B | 0.0%0 “I don't know” of 37 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.8%37 of 547 | 0.0%0 “I don't know” of 34 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.2%34 of 547 | run 955a new run | run 959the Hallucination tab's full runs, counted on the kept questions | 93.2%510 of 547 right | 93.8%513 of 547 right | 0.2% | 0.2% | 16 | 13 | 0.71 | 1 |
| KnowLine-31B-Gen2 Open · 31B | 0.0%0 “I don't know” of 40 not right | 100.0%547 of 547 answered, 0 “I don't know” | 7.3%40 of 547 | 0.0%0 “I don't know” of 29 not right | 100.0%547 of 547 answered, 0 “I don't know” | 5.3%29 of 547 | run 1329a new run | run 1331the Hallucination tab's full runs, counted on the kept questions | 92.7%507 of 547 right | 94.7%518 of 547 right | 0.2% | 0.1% | 20 | 9 | 0.061 | 1 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 0.0%0 “I don't know” of 94 not right | 100.0%547 of 547 answered, 0 “I don't know” | 17.2%94 of 547 | 1.7%2 “I don't know” of 120 not right | 99.6%545 of 547 answered, 2 “I don't know” | 21.7%118 of 545 | run 1083a new run | run 1239the Hallucination tab's full runs, counted on the kept questions | 82.8%453 of 547 right | 78.1%427 of 547 right | 3.2% | 2.9% | 33 | 57 | 0.015 | 0.71 |
Results with “I don't know” instead of “Bilmiyorum”
In the tab's order. Each row shows the model's first run with each word on the same 2,198 questions; the “Bilmiyorum” run is the one in the table above. With “I don't know”, then with “Bilmiyorum”: the non-hallucination rate (the option / (the option + wrong)), Answered = (right + wrong) / 2,198 and the error rate = wrong / (right + wrong); then accuracy with each. Wrong only with one word: the questions only one of the two runs got wrong, by the exact McNemar test, with p adjusted by Holm's method over the models compared; * marks a Holm-adjusted p below 0.05.
| Model | Non-hallucination with “I don't know”of answers not right, % “I don't know” · higher is better | Answered with “I don't know”not “I don't know”: (right + wrong) / 2,198 | Error rate with “I don't know”of its answers: wrong / (right + wrong) · lower is better | Non-hallucination with “Bilmiyorum”of answers not right, % “Bilmiyorum” · higher is better | Answered with “Bilmiyorum”not “Bilmiyorum”: (right + wrong) / 2,198 | Error rate with “Bilmiyorum”of its answers: wrong / (right + wrong) · lower is better | Accuracy with “I don't know”higher is better | Accuracy with “Bilmiyorum”higher is better | Wrong only with one wordexact McNemar, Holm-adjusted | Runsrun, start date and prompt version |
|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Efficient | 88.1%1,018 “I don't know” of 1,155 not right | 53.7%1,180 of 2,198 answered, 1,018 “I don't know” | 11.6%137 of 1,180 | 87.3%996 “Bilmiyorum” of 1,141 not right | 54.7%1,202 of 2,198 answered, 996 “Bilmiyorum” | 12.1%145 of 1,202 | 47.5%1,043 of 2,198 right | 48.1%1,057 of 2,198 right | 33 · 4133 wrong only with “I don't know”, 41 only with “Bilmiyorum”p = 0.42, Holm-adjusted p = 1 | 1378 · 1367“I don't know”: run 1378, 2026-11-16, prompt version fb17d583a3af“Bilmiyorum”: run 1367, 2026-11-16, prompt version 1cdd542b597f |
| d1.2 Closed · Efficient | 76.4%901 “I don't know” of 1,180 not right | 59.0%1,297 of 2,198 answered, 901 “I don't know” | 21.5%279 of 1,297 | 79.4%973 “Bilmiyorum” of 1,226 not right | 55.7%1,225 of 2,198 answered, 973 “Bilmiyorum” | 20.7%253 of 1,225 | 46.3%1,018 of 2,198 right | 44.2%972 of 2,198 right | 95 · 6995 wrong only with “I don't know”, 69 only with “Bilmiyorum”p = 0.051, Holm-adjusted p = 0.4 | 1377 · 1365“I don't know”: run 1377, 2026-11-16, prompt version fb17d583a3af“Bilmiyorum”: run 1365, 2026-11-16, prompt version 1cdd542b597f |
| d1-ultra Closed · Efficient | 74.0%616 “I don't know” of 832 not right | 72.0%1,582 of 2,198 answered, 616 “I don't know” | 13.7%216 of 1,582 | 75.1%617 “Bilmiyorum” of 822 not right | 71.9%1,581 of 2,198 answered, 617 “Bilmiyorum” | 13.0%205 of 1,581 | 62.1%1,366 of 2,198 right | 62.6%1,376 of 2,198 right | 66 · 5566 wrong only with “I don't know”, 55 only with “Bilmiyorum”p = 0.36, Holm-adjusted p = 1 | 511 · 350“I don't know”: run 511, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 350, 2026-11-01, prompt version 1cdd542b597f |
| Mammoth-1 Open · 27B | 73.2%753 “I don't know” of 1,029 not right | 65.7%1,445 of 2,198 answered, 753 “I don't know” | 19.1%276 of 1,445 | 68.7%690 “Bilmiyorum” of 1,004 not right | 68.6%1,508 of 2,198 answered, 690 “Bilmiyorum” | 20.8%314 of 1,508 | 53.2%1,169 of 2,198 right | 54.3%1,194 of 2,198 right | 9 · 479 wrong only with “I don't know”, 47 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 * | 519 · 386“I don't know”: run 519, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 386, 2026-11-01, prompt version 1cdd542b597f |
| GLiDE API Closed · Efficient | 63.4%581 “I don't know” of 916 not right | 73.6%1,617 of 2,198 answered, 581 “I don't know” | 20.7%335 of 1,617 | 59.4%521 “Bilmiyorum” of 877 not right | 76.3%1,677 of 2,198 answered, 521 “Bilmiyorum” | 21.2%356 of 1,677 | 58.3%1,282 of 2,198 right | 60.1%1,321 of 2,198 right | 52 · 7352 wrong only with “I don't know”, 73 only with “Bilmiyorum”p = 0.073, Holm-adjusted p = 0.51 | 510 · 351“I don't know”: run 510, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 351, 2026-11-01, prompt version 1cdd542b597f |
| SubQ Snap 1.1 Closed · Efficient | 52.3%498 “I don't know” of 952 not right | 77.3%1,700 of 2,198 answered, 498 “I don't know” | 26.7%454 of 1,700 | 47.1%437 “Bilmiyorum” of 927 not right | 80.1%1,761 of 2,198 answered, 437 “Bilmiyorum” | 27.8%490 of 1,761 | 56.7%1,246 of 2,198 right | 57.8%1,271 of 2,198 right | 108 · 144108 wrong only with “I don't know”, 144 only with “Bilmiyorum”p = 0.027, Holm-adjusted p = 0.25 | 512 · 392“I don't know”: run 512, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 392, 2026-11-01, prompt version 1cdd542b597f |
| GLiDE v1.2 Open · 27B | 49.9%426 “I don't know” of 853 not right | 80.6%1,772 of 2,198 answered, 426 “I don't know” | 24.1%427 of 1,772 | 39.5%321 “Bilmiyorum” of 812 not right | 85.4%1,877 of 2,198 answered, 321 “Bilmiyorum” | 26.2%491 of 1,877 | 61.2%1,345 of 2,198 right | 63.1%1,386 of 2,198 right | 19 · 8319 wrong only with “I don't know”, 83 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 * | 525 · 369“I don't know”: run 525, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 369, 2026-11-01, prompt version 1cdd542b597f |
| Drex Max v1.1 Closed · Efficient (provisional) | 54.1%486 “I don't know” of 899 not right | 77.9%1,712 of 2,198 answered, 486 “I don't know” | 24.1%413 of 1,712 | 38.2%320 “Bilmiyorum” of 837 not right | 85.4%1,878 of 2,198 answered, 320 “Bilmiyorum” | 27.5%517 of 1,878 | 59.1%1,299 of 2,198 right | 61.9%1,361 of 2,198 right | 40 · 14440 wrong only with “I don't know”, 144 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 * | 513 · 352“I don't know”: run 513, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 352, 2026-11-01, prompt version 1cdd542b597f |
| Elephant-1 Open · 9.0B | 49.3%604 “I don't know” of 1,226 not right | 72.5%1,594 of 2,198 answered, 604 “I don't know” | 39.0%622 of 1,594 | 34.8%405 “Bilmiyorum” of 1,163 not right | 81.6%1,793 of 2,198 answered, 405 “Bilmiyorum” | 42.3%758 of 1,793 | 44.2%972 of 2,198 right | 47.1%1,035 of 2,198 right | 18 · 15418 wrong only with “I don't know”, 154 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 * | 515 · 382“I don't know”: run 515, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 382, 2026-11-01, prompt version 1cdd542b597f |
| d3-ultra Open · 26B | 1.3%9 “I don't know” of 713 not right | 99.6%2,189 of 2,198 answered, 9 “I don't know” | 32.2%704 of 2,189 | 1.4%10 “Bilmiyorum” of 714 not right | 99.5%2,188 of 2,198 answered, 10 “Bilmiyorum” | 32.2%704 of 2,188 | 67.6%1,485 of 2,198 right | 67.5%1,484 of 2,198 right | 39 · 3939 wrong only with “I don't know”, 39 only with “Bilmiyorum”p = 1, Holm-adjusted p = 1 | 540 · 363“I don't know”: run 540, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 363, 2026-11-01, prompt version 1cdd542b597f |
| Drex Max Open · 27B | 0.3%2 “I don't know” of 714 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 32.4%712 of 2,196 | 0.3%2 “Bilmiyorum” of 713 not right | 99.9%2,196 of 2,198 answered, 2 “Bilmiyorum” | 32.4%711 of 2,196 | 67.5%1,484 of 2,198 right | 67.6%1,485 of 2,198 right | 29 · 2829 wrong only with “I don't know”, 28 only with “Bilmiyorum”p = 1, Holm-adjusted p = 1 | 542 · 367“I don't know”: run 542, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 367, 2026-11-01, prompt version 1cdd542b597f |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 0.5%4 “I don't know” of 732 not right | 99.8%2,194 of 2,198 answered, 4 “I don't know” | 33.2%728 of 2,194 | 0.3%2 “Bilmiyorum” of 719 not right | 99.9%2,196 of 2,198 answered, 2 “Bilmiyorum” | 32.7%717 of 2,196 | 66.7%1,466 of 2,198 right | 67.3%1,479 of 2,198 right | 40 · 2940 wrong only with “I don't know”, 29 only with “Bilmiyorum”p = 0.23, Holm-adjusted p = 1 | 531 · 372“I don't know”: run 531, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 372, 2026-11-01, prompt version 1cdd542b597f |
| d3 Open · 26B | 0.3%2 “I don't know” of 716 not right | 99.9%2,196 of 2,198 answered, 2 “I don't know” | 32.5%714 of 2,196 | 0.0%0 “Bilmiyorum” of 715 not right | 100.0%2,198 of 2,198 answered, 0 “Bilmiyorum” | 32.5%715 of 2,198 | 67.4%1,482 of 2,198 right | 67.5%1,483 of 2,198 right | 21 · 2221 wrong only with “I don't know”, 22 only with “Bilmiyorum”p = 1, Holm-adjusted p = 1 | 544 · 358“I don't know”: run 544, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 358, 2026-11-01, prompt version 1cdd542b597f |
Not run
- Copernicus-31B: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- KnowLine-31B-Gen2: added to the Hallucination tab on 16 November 2026 (run on our GPU; Ümit #9814), after the option-word check; not asked with “I don't know” yet.
- KnowLine-31B-Gen1.5: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- Turion Jev 31B: added to the Hallucination tab on 15 November 2026 (run on our DGX Spark), after the option-word check; not asked with “I don't know” yet.
- REAX 1.3: added to the Hallucination tab on 6 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- Karpo Decision 31B: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616), after the option-word check; not asked with “I don't know” yet.
- Gemini Decision 1: added to the Hallucination tab on 11 November 2026 (Google's Decisions API through OpenRouter), after the option-word check; not asked with “I don't know” yet.
- Logits Sigma-1 Duo: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- Sieve 31B v3: added to the Hallucination tab on 15 November 2026 (run on our DGX Spark), after the option-word check; not asked with “I don't know” yet.
- Perplexity Decider 27B v1.3: added to the Hallucination tab on 14 November 2026 (run on our GPU; Ümit #7470), after the option-word check; not asked with “I don't know” yet.
- Perplexity Decider 27B v1.2: added to the Hallucination tab on 5 November 2026 (Ümit #1229), after the option-word check; not asked with “I don't know” yet.
- Perplexity Decider 27B v1.2: added to the Hallucination tab on 5 November 2026 (Ümit #1229), after the option-word check; not asked with “I don't know” yet.
- GLiDE 27B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- KnowLine-12B-Gen1: added to the Hallucination tab on 16 November 2026 (run on our GPU; Ümit #9814), after the option-word check; not asked with “I don't know” yet.
- OneJev-27B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- Jeffan-9B-1211: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- Clef: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- Open-Jev-27B-v1.1: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- Cygnet: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- Cerno 9B v1.2: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- KnowLine-9B-Gen1: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- Winnow-12B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- Adjudex 9B v5: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- KnowLine-9B-Gen2: added to the Hallucination tab on 16 November 2026 (run on our GPU; Ümit #9814), after the option-word check; not asked with “I don't know” yet.
- Dexy-Decision-9B-V2: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- StartLux-Decision-v1.1-9B-Instant: added to the Hallucination tab on 15 November 2026 (run on our DGX Spark), after the option-word check; not asked with “I don't know” yet.
- Perplexity Decider Flash v1.3: added to the Hallucination tab on 14 November 2026 (run on our GPU; Ümit #7470), after the option-word check; not asked with “I don't know” yet.
- GLiDE 9B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- Valen-Decision-1.5-9B: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- StartLux-Decision-v1.1-4B-Instant: added to the Hallucination tab on 15 November 2026 (run on our DGX Spark), after the option-word check; not asked with “I don't know” yet.
- Verdin-4B-v1.1: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- KnowLine 4B Gen5.5: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- vera-medium-omni: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- Chinese Decision Pointer 4B: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
- Clef-Flash: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- Perplexity Decider Flash v1.2: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- Metask-Jev-4B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- StartLux-Decision-v1.1-2B-Instant: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- GLiDE 2B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- StartLux-Decision-v1.1-0.8B-Instant: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- KnowLine-0.8B-Gen1: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- Strands Decider 2B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- GLiDE 0.8B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- OneJev1.6-0.8B: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- GLiNER2.5-multi-Decide: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- GLiClass v4 Edge: added to the Hallucination tab on 12 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
- d1-2B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- Laya Multilingual: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- Laya: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
- GLiDE 0.4B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
Our method with “I don't know”
Only two things changed from the runs above: the sixth option's word and the sentence of the System One instructions that names it. The option “Bilmiyorum” became “I don't know”, and the sentence “If you do not know the answer, choose Bilmiyorum.” became “If you do not know the answer, choose I don't know.” Everything else is as above: the same 2,198 Turkish questions and answer key, the option always last and never correct, one question per request and each model's own settings. The full instructions:
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose I don't know.
Prompt version fb17d583a3af with “I don't know”, against 1cdd542b597f with “Bilmiyorum”. One full run per model with “I don't know”; the table gives each run's ID and start date (UTC).
How to read the comparison
A model that did not recognise the Turkish word and so ignored the option should choose the English one more often, and its non-hallucination rate should rise. A model that chooses both about as often did not skip “Bilmiyorum” for its word. The rates are those above, and the same limits hold: the non-hallucination rate says how a model is wrong, not how often it is right, and it is not a direct measure of what the model does not know; read it with accuracy and the attempt rate, as a model that always chose the option would reach 100% with no right answers. With neither a wrong answer nor the option, the rate is undefined.
Results: “I don't know” against confidence
In the tab's order. Each row: the model's first run with “I don't know” and its five-option run held back by its own confidence on the same 2,198 questions. Answered = right + wrong; coverage = answered / 2,198. Wrong only on one side: the questions only one of the two got wrong, by the exact McNemar test, with p adjusted by Holm's method over the 9 models that chose “I don't know” on at least 1% of the questions; * marks a Holm-adjusted p below 0.05. Not in the family: d3-ultra, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark and d3.
| Model | Runswith “I don't know” · five options | Cut-offand its neighbouring candidates | Answeredtarget with the option · achieved by confidence (coverage) | Wrongwith “I don't know” · by confidence; lower is better | Wrong only on one sidewith “I don't know” · by confidence | pexact McNemar | Holm padjusted over the family |
|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Efficient | 1378 · 1347 | 0.6035next lower 0.6034: 1,181 answerednext higher 0.6038: 1,179 answered | 1,180 · 1,180target 1,180 (53.7%), 1,018 “I don't know”achieved 1,180 (53.7%), 1,018 held back | 137 · 128 | 52 · 43 | 0.41 | 1 |
| d1.2 Closed · Efficient | 1377 · 1346 | 0.4689next lower 0.4687: 1,298 answerednext higher 0.4690: 1,296 answered | 1,297 · 1,297target 1,297 (59.0%), 901 “I don't know”achieved 1,297 (59.0%), 901 held back | 279 · 262 | 107 · 90 | 0.25 | 1 |
| d1-ultra Closed · Efficient | 511 · 199 | 0.5630next lower 0.5622: 1,583 answerednext higher 0.5632: 1,581 answered | 1,582 · 1,582target 1,582 (72.0%), 616 “I don't know”achieved 1,582 (72.0%), 616 held back | 216 · 167 | 98 · 49 | < 0.0001 | 0.00059 * |
| Mammoth-1 Open · 27B | 519 · 374 | 0.5951next lower 0.5948: 1,446 answerednext higher 0.5956: 1,444 answered | 1,445 · 1,445target 1,445 (65.7%), 753 “I don't know”achieved 1,445 (65.7%), 753 held back | 276 · 250 | 64 · 38 | 0.013 | 0.1 |
| GLiDE API Closed · Efficient | 510 · 517run 517 in place of the Text tab's run 129 | 0.5222next lower 0.5220: 1,618 answerednext higher 0.5231: 1,616 answered | 1,617 · 1,617target 1,617 (73.6%), 581 “I don't know”achieved 1,617 (73.6%), 581 held back | 335 · 319 | 79 · 63 | 0.21 | 1 |
| SubQ Snap 1.1 Closed · Efficient | 512 · 346 | 0.5372next lower 0.5371: 1,701 answerednext higher 0.5380: 1,699 answered | 1,700 · 1,700target 1,700 (77.3%), 498 “I don't know”achieved 1,700 (77.3%), 498 held back | 454 · 445 | 141 · 132 | 0.63 | 1 |
| GLiDE v1.2 Open · 27B | 525 · 248 | 0.4535next lower 0.4534: 1,773 answerednext higher 0.4536: 1,771 answered | 1,772 · 1,772target 1,772 (80.6%), 426 “I don't know”achieved 1,772 (80.6%), 426 held back | 427 · 403 | 77 · 53 | 0.043 | 0.3 |
| Drex Max v1.1 Closed · Efficient (provisional) | 513 · 326 | 0.4478next lower 0.4470: 1,713 answerednext higher 0.4479: 1,711 answered | 1,712 · 1,712target 1,712 (77.9%), 486 “I don't know”achieved 1,712 (77.9%), 486 held back | 413 · 396 | 106 · 89 | 0.25 | 1 |
| Elephant-1 Open · 9.0B | 515 · 373 | 0.4376next lower 0.4374: 1,595 answerednext higher 0.4378: 1,593 answered | 1,594 · 1,594target 1,594 (72.5%), 604 “I don't know”achieved 1,594 (72.5%), 604 held back | 622 · 594 | 120 · 92 | 0.063 | 0.38 |
| d3-ultra Open · 26B | 540 · 269 | 0.2839next lower 0.2830: 2,190 answerednext higher 0.2876: 2,188 answered | 2,189 · 2,189target 2,189 (99.6%), 9 “I don't know”achieved 2,189 (99.6%), 9 held back | 704 · 703 | 54 · 53 | 1 | not in the family |
| Drex Max Open · 27B | 542 · 322 | 0.2357next lower 0.2348: 2,197 answerednext higher 0.2375: 2,195 answered | 2,196 · 2,196target 2,196 (99.9%), 2 “I don't know”achieved 2,196 (99.9%), 2 held back | 712 · 707 | 49 · 44 | 0.68 | not in the family |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 531 · 217 | 0.2217next lower 0.2205: 2,195 answerednext higher 0.2223: 2,193 answered | 2,194 · 2,194target 2,194 (99.8%), 4 “I don't know”achieved 2,194 (99.8%), 4 held back | 728 · 720 | 52 · 44 | 0.48 | not in the family |
| d3 Open · 26B | 544 · 259 | 0.2331next lower 0.2294: 2,197 answerednext higher 0.2352: 2,195 answered | 2,196 · 2,196target 2,196 (99.9%), 2 “I don't know”achieved 2,196 (99.9%), 2 held back | 714 · 706 | 39 · 31 | 0.4 | not in the family |
Our method for “I don't know” against confidence
The derived method above, applied to the runs with “I don't know”: each model's five-option run answers only where its score reaches a cut-off and abstains on the rest. The candidate cut-offs are the run's distinct scores; the one chosen is the one whose attempt rate comes nearest the attempt rate of the run with “I don't know”, and on a tie the higher cut-off. An answer without a score never counts as an attempt. Answers with equal scores enter together, so the achieved answered count can differ from the target; the table shows the chosen cut-off with the next lower and next higher candidates and the answers each would give.
The five-option run is each model's run on the Text tab, as in the table above. For GLiDE · API, the five-option run is 517, not the Text tab's run 129: Fastino changed its hosted model after 6 October, so run 129 is a different model from the one behind run 510, and run 517 is a new five-option run of the same hosted model (Our method for the hosted GLiDE rerun).
The paired test is the exact McNemar test of wrong answers, the run with “I don't know” first and confidence second, as for “Bilmiyorum”. Holm's correction runs over the models that chose “I don't know” on at least 1% of the scored questions: GLiDE · v1.2, d1-ultra, Drex Max v1.1, GLiDE · API, SubQ Snap 1.1, Mammoth-1, d1.2-ultra, Elephant-1 and Liquid d1.2. The others, d3-ultra, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark and d3, chose it too rarely for the two sides to differ by more than a few answers; their p is shown and not adjusted.
This is a family of its own, from runs already published: it changes none of the numbers above, and it is separate from the comparisons with Holm over 4, 13 and 11 models (the four models of the request, the English questions and the option's word).
Hosted GLiDE: the comparison redone with run 517
Both against GLiDE · API's first run with “I don't know”, run 351. Run 129 is the Text tab's run and the row in the table above; run 517 is new and replaces it here only.
| Five-option runrun, date and prompt version | Text accuracy with five optionshigher is better | Derived non-hallucination ratehigher is better | Wrong only on one sideexact McNemar |
|---|---|---|---|
| run 1292026-10-06, as published | 67.9%1,492 of 2,198 right | 58.0%cut-off 0.5206: 1,300 right + 377 wrong answered, 521 held back, attempt rate 76.3% | 192 with “I don't know”, 213 by confidencep = 0.32, Holm-adjusted p = 1 |
| run 5172026-11-02, prompt version 254597b8be16 | 68.1%1,497 of 2,198 right | 59.7%cut-off 0.4956: 1,326 right + 351 wrong answered, 521 held back, attempt rate 76.3% | 90 with “I don't know”, 85 by confidencep = 0.76, not adjusted |
Our method for the hosted GLiDE rerun
Fastino changed the hosted model behind fastino/GLiDE after 6 October. Run 129 was made on 6 October 2026, so it is a different model from the one that answered with “I don't know” (run 351), and the derived rate in the table above compares two models. Run 517 is a new five-option run of the same hosted model, started 2 November 2026, made as on the Text tab: the same 2,198 questions, one question per request, the main prompt version 254597b8be16. Its derived non-hallucination rate and paired test are computed as above, from run 517 instead of run 129; the p value is not adjusted, as it is one test.
The evidence for “a different model” and “the same model” is the engine each response names: every answer of run 129 names Fastino's engine glide; every answer from 10 October on names glide_v2, including all of run 351 and its repeats and all of run 517. Fastino publishes no version to pin, so that name is the only identity we can check: it cannot show that the weights behind glide_v2 stayed the same between 1 and 2 November.
The row in the table above is left as published, against run 129, and run 517 is kept out of the Text tab: its leaderboard, repeats and every other list.
Results: the same number of answers
One table per language, ordered by wrong of the k answers, fewest first; the Turkish one has every main run of the Text tab (137 setups), grouped as its leaderboard groups them and labelled as it labels them. Each row: the model's five-option run held to its k most confident answers, its wrong answers among them, the difference from d1-ultra in points with its 95% bootstrap interval, p and Holm p (* marks a Holm-adjusted p below 0.05), and for context the model's own runs with the option (answered = right + wrong, and wrong of answers = wrong / answered), which are not part of the test; a setup without one reads none.
Turkish: k = 1,582 of the 2,198 questions
Each setup's curve over every number of answers: Every setup's curve in Turkish, below.
Decision models
| Model | Five-option runthe run whose k answers are kept | Wrong of the k answerswrong / 1,582; lower is better | Difference from d1-ultrapoints, with its 95% interval; positive = more wrong answers | ppaired bootstrap, two-sided | Holm padjusted over the 126 | Runs with the optionfor context: answered, wrong of answers |
|---|---|---|---|---|---|---|
| Copernicus-31B Open · 31B | 1165 | 6.0%95 of 1,582 | -4.6-6.3 to -3.2 | 0.0002 | 0.025 * | 1172“Bilmiyorum”, run 1172: 2,193 answered (99.8%), 18.3% wrong (402) |
| KnowLine-31B-Gen1.5 Open · 31B | 957 | 6.6%105 of 1,582 | -3.9-5.6 to -2.5 | 0.0002 | 0.025 * | 959“Bilmiyorum”, run 959: 2,197 answered (100.0%), 18.7% wrong (411) |
| Turion Jev 31B Open · 31B | 1273 | 7.4%117 of 1,582 | -3.2-4.7 to -1.5 | 0.0002 | 0.025 * | 1285“Bilmiyorum”, run 1285: 2,183 answered (99.3%), 20.0% wrong (436) |
| KnowLine-31B-Gen2 Open · 31B | 1314 | 7.7%122 of 1,582 | -2.8-4.7 to -1.5 | 0.0002 | 0.025 * | 1331“Bilmiyorum”, run 1331: 2,198 answered (100.0%), 18.6% wrong (408) |
| Karpo Decision 31B Open · 31B | 1156 | 9.2%146 of 1,582 | -1.3-3.1 to +0.2 | 0.084 | 0.59 | 1163“Bilmiyorum”, run 1163: 2,195 answered (99.9%), 21.6% wrong (475) |
| REAX 1.3 Open · 57B | 827 | 9.5%151 of 1,582 | -1.0-2.7 to +0.5 | 0.18 | 1 | 833“Bilmiyorum”, run 833: 2,196 answered (99.9%), 21.6% wrong (475) |
| Gemini Decision 1 20261111 Closed · Efficient | 1021 | 9.9%156 of 1,582 | -0.7-2.5 to +1.0 | 0.45 | 1 | 1023“Bilmiyorum”, run 1023: 2,188 answered (99.5%), 23.1% wrong (505) |
| d1-ultra Closed · Efficient | 199 | 10.6%167 of 1,582 | referenceits own run 511 with “I don't know”: 216 wrong of 1,582 (13.7%), not part of the test | reference | reference | 350 · 511“Bilmiyorum”, run 350: 1,581 answered (71.9%), 13.0% wrong (205)“I don't know”, run 511: 1,582 answered (72.0%), 13.7% wrong (216) |
| Sieve 31B v3 Open · 31B | 1241 | 11.6%183 of 1,582 | +1.0-0.9 to +2.8 | 0.32 | 1 | 1253“Bilmiyorum”, run 1253: 2,198 answered (100.0%), 25.2% wrong (553) |
| Logits Sigma-1 Duo Open · 57B | 1221 | 13.2%209 of 1,582 | +2.7+0.6 to +4.2 | 0.0092 | 0.083 | 1233“Bilmiyorum”, run 1233: 2,196 answered (99.9%), 24.7% wrong (543) |
| GLiDE 27B Open · 27B | 899 | 16.4%259 of 1,582 | +5.8+3.7 to +7.6 | 0.0002 | 0.025 * | 905“Bilmiyorum”, run 905: 1,907 answered (86.8%), 23.1% wrong (441) |
| GLiDE v1.2 Open · 27B | 248 | 18.3%289 of 1,582 | +7.7+5.8 to +9.7 | 0.0002 | 0.025 * | 369 · 525“Bilmiyorum”, run 369: 1,877 answered (85.4%), 26.2% wrong (491)“I don't know”, run 525: 1,772 answered (80.6%), 24.1% wrong (427) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 640 | 19.8%313 of 1,582 | +9.2+7.1 to +11.1 | 0.0002 | 0.025 * | 642“Bilmiyorum”, run 642: 2,175 answered (99.0%), 32.3% wrong (702) |
| d3 Open · 26B | 259 | 19.8%314 of 1,582 | +9.3+7.3 to +11.3 | 0.0002 | 0.025 * | 358 · 544“Bilmiyorum”, run 358: 2,198 answered (100.0%), 32.5% wrong (715)“I don't know”, run 544: 2,196 answered (99.9%), 32.5% wrong (714) |
| d3-ultra Open · 26B | 269 | 19.9%315 of 1,582 | +9.4+7.2 to +11.3 | 0.0002 | 0.025 * | 363 · 540“Bilmiyorum”, run 363: 2,188 answered (99.5%), 32.2% wrong (704)“I don't know”, run 540: 2,189 answered (99.6%), 32.2% wrong (704) |
| Perplexity Decider 27B v1.1 Open · 27B | 145 | 20.1%318 of 1,582 | +9.5+7.6 to +11.6 | 0.0002 | 0.025 * | noneno run with the option |
| Perplexity Decider 27B v1.3 Open · 27B | 1134 | 20.2%320 of 1,582 | +9.7+7.7 to +11.6 | 0.0002 | 0.025 * | 1141“Bilmiyorum”, run 1141: 2,193 answered (99.8%), 32.4% wrong (711) |
| Drex Max Open · 27B | 322 | 20.3%321 of 1,582 | +9.7+7.6 to +11.7 | 0.0002 | 0.025 * | 367 · 542“Bilmiyorum”, run 367: 2,196 answered (99.9%), 32.4% wrong (711)“I don't know”, run 542: 2,196 answered (99.9%), 32.4% wrong (712) |
| Drex Max v1.1 Closed · Efficient (provisional) | 326 | 20.3%321 of 1,582 | +9.7+7.8 to +11.8 | 0.0002 | 0.025 * | 352 · 513“Bilmiyorum”, run 352: 1,878 answered (85.4%), 27.5% wrong (517)“I don't know”, run 513: 1,712 answered (77.9%), 24.1% wrong (413) |
| Mammoth-1 Open · 27B | 374 | 20.4%322 of 1,582 | +9.8+7.8 to +11.8 | 0.0002 | 0.025 * | 386 · 519“Bilmiyorum”, run 386: 1,508 answered (68.6%), 20.8% wrong (314)“I don't know”, run 519: 1,445 answered (65.7%), 19.1% wrong (276) |
| d1.2-ultra Closed · Efficient | 1347 | 20.4%323 of 1,582 | +9.9+7.9 to +11.8 | 0.0002 | 0.025 * | 1367 · 1378“Bilmiyorum”, run 1367: 1,202 answered (54.7%), 12.1% wrong (145)“I don't know”, run 1378: 1,180 answered (53.7%), 11.6% wrong (137) |
| GLiDE API Closed · Efficient | 129 | 20.5%325 of 1,582 | +10.0+7.9 to +12.0 | 0.0002 | 0.025 * | 351 · 510“Bilmiyorum”, run 351: 1,677 answered (76.3%), 21.2% wrong (356)“I don't know”, run 510: 1,617 answered (73.6%), 20.7% wrong (335)Made with Fastino's engine glide_v2, not run 129's glide |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 217 | 20.7%328 of 1,582 | +10.2+8.0 to +12.1 | 0.0002 | 0.025 * | 372 · 531“Bilmiyorum”, run 372: 2,196 answered (99.9%), 32.7% wrong (717)“I don't know”, run 531: 2,194 answered (99.8%), 33.2% wrong (728) |
| Perplexity Decider 27B v1.2 API Open · 27B | 633 | 20.8%329 of 1,582 | +10.2+8.0 to +12.3 | 0.0002 | 0.025 * | 641“Bilmiyorum”, run 641: 2,195 answered (99.9%), 33.1% wrong (726) |
| SubQ Snap Preview Closed · Efficient | 177 | 22.5%356 of 1,582 | +11.9+9.7 to +13.9 | 0.0002 | 0.025 * | noneno run with the option |
| SubQ Snap 1.1 Closed · Efficient | 346 | 22.8%361 of 1,582 | +12.3+10.1 to +14.5 | 0.0002 | 0.025 * | 392 · 512“Bilmiyorum”, run 392: 1,761 answered (80.1%), 27.8% wrong (490)“I don't know”, run 512: 1,700 answered (77.3%), 26.7% wrong (454) |
| GPT-6 Luna Decisions API Closed · Efficient | 143 | 23.5%372 of 1,582 | +13.0+10.7 to +15.1 | 0.0002 | 0.025 * | noneno run with the option |
| KnowLine-12B-Gen1 Open · 12B | 1309 | 26.4%418 of 1,582 | +15.9+13.7 to +18.1 | 0.0002 | 0.025 * | 1318“Bilmiyorum”, run 1318: 2,197 answered (100.0%), 36.6% wrong (804) |
| Perplexity Decider 27B v1 Open · 27B | 44 | 27.2%430 of 1,582 | +16.6+14.5 to +18.9 | 0.0002 | 0.025 * | noneno run with the option |
| d1.1 Closed · Efficient | 254 | 27.4%434 of 1,582 | +16.9+14.4 to +19.0 | 0.0002 | 0.025 * | noneno run with the option |
| d1.2 Closed · Efficient | 1346 | 27.6%436 of 1,582 | +17.0+14.7 to +19.2 | 0.0002 | 0.025 * | 1365 · 1377“Bilmiyorum”, run 1365: 1,225 answered (55.7%), 20.7% wrong (253)“I don't know”, run 1377: 1,297 answered (59.0%), 21.5% wrong (279) |
| celeris-1-decision Closed · Efficient | 150 | 28.4%450 of 1,582 | +17.9+15.4 to +20.0 | 0.0002 | 0.025 * | noneno run with the option |
| OneJev-27B Open · 27B | 264 | 28.8%455 of 1,582 | +18.2+15.9 to +20.4 | 0.0002 | 0.025 * | 782“Bilmiyorum”, run 782: 2,197 answered (100.0%), 40.6% wrong (892) |
| Jev 1.14 Closed · Efficient | 194 | 30.7%485 of 1,582 | +20.1+17.9 to +22.4 | 0.0002 | 0.025 * | noneno run with the option |
| Mercury Decide Closed · Efficient | 204 | 30.8%487 of 1,582 | +20.2+17.8 to +22.6 | 0.0002 | 0.025 * | noneno run with the option |
| Jev 1.13 Closed · Efficient | 49 | 31.5%498 of 1,582 | +20.9+18.5 to +23.2 | 0.0002 | 0.025 * | noneno run with the option |
| Open-Jev-27B-v1.1 Open · 27B | 45 | 31.9%504 of 1,582 | +21.3+18.8 to +23.6 | 0.0002 | 0.025 * | 768“Bilmiyorum”, run 768: 2,184 answered (99.4%), 44.2% wrong (965) |
| Clef Open · 27B | 125 | 32.4%512 of 1,582 | +21.8+19.5 to +24.2 | 0.0002 | 0.025 * | 788“Bilmiyorum”, run 788: 2,127 answered (96.8%), 42.0% wrong (894) |
| d1 Closed · Efficient | 52 | 32.6%516 of 1,582 | +22.1+19.7 to +24.3 | 0.0002 | 0.025 * | noneno run with the option |
| Jeffan-9B-1211 Open · 9.0B | 1192 | 32.8%519 of 1,582 | +22.3+19.8 to +24.5 | 0.0002 | 0.025 * | 1199“Bilmiyorum”, run 1199: 2,157 answered (98.1%), 42.5% wrong (917) |
| Drex 2.0 Open · 35B MoE · 3.0B active | 334 | 33.4%528 of 1,582 | +22.8+20.4 to +25.0 | 0.0002 | 0.025 * | noneno run with the option |
| Cygnet Open · 12B | 127 | 33.5%530 of 1,582 | +22.9+20.4 to +25.3 | 0.0002 | 0.025 * | 757“Bilmiyorum”, run 757: 2,166 answered (98.5%), 44.2% wrong (957) |
| Winnow-12B Open · 12B | 128 | 34.1%540 of 1,582 | +23.6+21.0 to +25.9 | 0.0002 | 0.025 * | 750“Bilmiyorum”, run 750: 2,155 answered (98.0%), 44.8% wrong (965) |
| KnowLine-9B-Gen1 Open · 9.0B | 1174 | 35.0%553 of 1,582 | +24.4+22.0 to +26.9 | 0.0002 | 0.025 * | 1181“Bilmiyorum”, run 1181: 2,193 answered (99.8%), 45.4% wrong (996) |
| Cerno 9B v1.2 Open · 9.0B | 1201 | 35.3%559 of 1,582 | +24.8+22.4 to +27.1 | 0.0002 | 0.025 * | 1204“Bilmiyorum”, run 1204: 2,196 answered (99.9%), 45.4% wrong (998) |
| Adjudex 9B v5 Open · 9.0B | 1183 | 36.5%578 of 1,582 | +26.0+23.6 to +28.3 | 0.0002 | 0.025 * | 1190“Bilmiyorum”, run 1190: 2,190 answered (99.6%), 46.6% wrong (1,021) |
| KnowLine-9B-Gen2 Open · 9.0B | 1306 | 36.7%580 of 1,582 | +26.1+23.8 to +28.6 | 0.0002 | 0.025 * | 1325“Bilmiyorum”, run 1325: 2,171 answered (98.8%), 47.4% wrong (1,030) |
| Elephant-1 Open · 9.0B | 373 | 37.0%586 of 1,582 | +26.5+24.1 to +29.0 | 0.0002 | 0.025 * | 382 · 515“Bilmiyorum”, run 382: 1,793 answered (81.6%), 42.3% wrong (758)“I don't know”, run 515: 1,594 answered (72.5%), 39.0% wrong (622) |
| GLiDE 9B Open · 7.9B | 907 | 37.9%600 of 1,582 | +27.4+24.8 to +29.6 | 0.0002 | 0.025 * | 913“Bilmiyorum”, run 913: 1,921 answered (87.4%), 45.4% wrong (872) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 1290 | 39.2%620 of 1,582 | +28.6+26.1 to +31.0 | 0.0002 | 0.025 * | 1297“Bilmiyorum”, run 1297: 2,174 answered (98.9%), 49.2% wrong (1,069) |
| Dexy-Decision-9B-V2 Open · 9.0B | 1255 | 39.4%623 of 1,582 | +28.8+26.2 to +31.2 | 0.0002 | 0.025 * | 1262“Bilmiyorum”, run 1262: 2,161 answered (98.3%), 48.4% wrong (1,046) |
| vera-medium-omni Open · 4.2B | 1264 | 42.0%665 of 1,582 | +31.5+28.8 to +33.9 | 0.0002 | 0.025 * | 1271“Bilmiyorum”, run 1271: 2,093 answered (95.2%), 51.4% wrong (1,075) |
| Perplexity Decider Flash v1.3 Open · 4.0B | 1123 | 42.2%667 of 1,582 | +31.6+29.1 to +34.1 | 0.0002 | 0.025 * | 1130“Bilmiyorum”, run 1130: 2,123 answered (96.6%), 50.4% wrong (1,071) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 1236 | 42.6%674 of 1,582 | +32.0+29.4 to +34.4 | 0.0002 | 0.025 * | 1239“Bilmiyorum”, run 1239: 2,177 answered (99.0%), 52.7% wrong (1,147) |
| KnowLine 4B Gen5.5 Open · 4.0B | 965 | 43.0%681 of 1,582 | +32.5+29.8 to +34.8 | 0.0002 | 0.025 * | 967“Bilmiyorum”, run 967: 2,143 answered (97.5%), 52.3% wrong (1,121) |
| Perplexity Decider Flash v1.2 Open · 4.0B | 223 | 43.7%691 of 1,582 | +33.1+30.6 to +35.6 | 0.0002 | 0.025 * | 791“Bilmiyorum”, run 791: 2,018 answered (91.8%), 51.7% wrong (1,044) |
| Valen-Decision-1.5-9B Open · 9.0B | 1275 | 44.1%698 of 1,582 | +33.6+30.9 to +36.0 | 0.0002 | 0.025 * | 1282“Bilmiyorum”, run 1282: 2,197 answered (100.0%), 52.7% wrong (1,157) |
| Verdin-4B-v1.1 Open · 4.0B | 1244 | 44.8%708 of 1,582 | +34.2+31.5 to +36.7 | 0.0002 | 0.025 * | 1251“Bilmiyorum”, run 1251: 2,193 answered (99.8%), 53.3% wrong (1,169) |
| Clef-Flash Open · 9.0B | 139 | 45.3%716 of 1,582 | +34.7+32.0 to +37.2 | 0.0002 | 0.025 * | 664“Bilmiyorum”, run 664: 2,088 answered (95.0%), 53.1% wrong (1,108) |
| Chinese Decision Pointer 4B Open · 4.0B | 1206 | 45.7%723 of 1,582 | +35.1+32.5 to +37.6 | 0.0002 | 0.025 * | 1213“Bilmiyorum”, run 1213: 2,062 answered (93.8%), 52.5% wrong (1,082) |
| Microsoft-Decision-1 Closed · Efficient | 205 | 45.9%726 of 1,582 | +35.3+32.6 to +37.8 | 0.0002 | 0.025 * | noneno run with the option |
| Metask-Jev-4B Open · 4.5B | 126 | 53.2%841 of 1,582 | +42.6+39.8 to +45.1 | 0.0002 | 0.025 * | 743“Bilmiyorum”, run 743: 2,188 answered (99.5%), 60.1% wrong (1,315) |
| KnowLine-2B-Gen1 Open · 2.0B | 1079 | 58.2%920 of 1,582 | +47.6+44.8 to +50.2 | 0.0002 | 0.025 * | noneno run with the option |
| GLiDE 2B Open · 1.9B | 915 | 61.0%965 of 1,582 | +50.4+47.6 to +53.2 | 0.0002 | 0.025 * | 921“Bilmiyorum”, run 921: 1,586 answered (72.2%), 61.7% wrong (978) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 977 | 61.4%972 of 1,582 | +50.9+48.1 to +53.5 | 0.0002 | 0.025 * | 979“Bilmiyorum”, run 979: 1,865 answered (84.8%), 65.0% wrong (1,213) |
| d1-2B Open · 1.9B | 340 | 67.0%1,060 of 1,582 | +56.4+53.7 to +59.0 | 0.0002 | 0.025 * | 712“Bilmiyorum”, run 712: 989 answered (45.0%), 64.4% wrong (637) |
| OneJev1.6-0.8B Open · 0.8B | 1029 | 67.8%1,072 of 1,582 | +57.2+54.5 to +59.8 | 0.0002 | 0.025 * | 1031“Bilmiyorum”, run 1031: 1,444 answered (65.7%), 68.2% wrong (985) |
| GLiDE 0.8B Open · 0.8B | 923 | 68.3%1,080 of 1,582 | +57.7+54.9 to +60.1 | 0.0002 | 0.025 * | 929“Bilmiyorum”, run 929: 1,718 answered (78.2%), 67.7% wrong (1,163) |
| Strands Decider 2B Open · 2.0B | 141 | 69.3%1,096 of 1,582 | +58.7+55.9 to +61.3 | 0.0002 | 0.025 * | 706“Bilmiyorum”, run 706: 2,048 answered (93.2%), 72.4% wrong (1,483) |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 1015 | 69.4%1,098 of 1,582 | +58.8+56.1 to +61.3 | 0.0002 | 0.025 * | 1017“Bilmiyorum”, run 1017: 1,999 answered (90.9%), 70.5% wrong (1,409) |
| KnowLine-0.8B-Gen1 Open · 0.8B | 1005 | 70.0%1,107 of 1,582 | +59.4+56.6 to +61.9 | 0.0002 | 0.025 * | 1007“Bilmiyorum”, run 1007: 2,094 answered (95.3%), 72.3% wrong (1,515) |
| Decider 1 Closed · Efficient | 243 | 70.8%1,120 of 1,582 | +60.2+57.5 to +62.8 | 0.0002 | 0.025 * | noneno run with the option |
| GLiClass v4 Edge Open · 0.4B | 1046 | 75.7%1,197 of 1,582 | +65.1+62.3 to +67.5 | 0.0002 | 0.025 * | 1048“Bilmiyorum”, run 1048: 1,800 answered (81.9%), 77.0% wrong (1,386) |
| GLiDE 0.4B Open · 0.4B | 931 | 76.5%1,211 of 1,582 | +66.0+63.3 to +68.5 | 0.0002 | 0.025 * | 937“Bilmiyorum”, run 937: 128 answered (5.8%), 78.9% wrong (101) |
| GLiNER2.5-multi-Decide Open · 0.3B | 135 | 78.9%1,248 of 1,582 | +68.3+65.6 to +70.7 | 0.0002 | 0.025 * | 718“Bilmiyorum”, run 718: 2,084 answered (94.8%), 79.9% wrong (1,666) |
| Laya Open · 0.4B | 51 | 80.5%1,273 of 1,582 | +69.9+67.3 to +72.4 | 0.0002 | 0.025 * | 730“Bilmiyorum”, run 730: 1,662 answered (75.6%), 81.5% wrong (1,355) |
| Laya Multilingual Open · 0.3B | 46 | 83.5%1,321 of 1,582 | +72.9+70.4 to +75.3 | 0.0002 | 0.025 * | 737“Bilmiyorum”, run 737: 1,918 answered (87.3%), 83.8% wrong (1,608) |
Other models: chat and general models
| Model | Five-option runthe run whose k answers are kept | Wrong of the k answerswrong / 1,582; lower is better | Difference from d1-ultrapoints, with its 95% interval; positive = more wrong answers | ppaired bootstrap, two-sided | Holm padjusted over the 126 | Runs with the optionfor context: answered, wrong of answers |
|---|---|---|---|---|---|---|
| Gemini 4 Carbon high · chat Closed · Flagship | 1108 | 0.3%4 of 1,582 | -10.3-12.1 to -8.8 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6 Astra low · chat Closed · Flagship | 48 | 0.3%4 of 1,582 | -10.3-12.1 to -8.8 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Fable 5.2 max · chat Closed · Flagship | 980 | 0.3%5 of 1,582 | -10.2-12.0 to -8.8 | 0.0002 | 0.025 * | noneno run with the option |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 1114 | 0.3%5 of 1,582 | -10.2-12.0 to -8.7 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6 Astra high · chat Closed · Flagship | 277 | 0.4%6 of 1,582 | -10.2-11.9 to -8.7 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6 Astra medium · chat Closed · Flagship | 610 | 0.4%6 of 1,582 | -10.2-11.9 to -8.7 | 0.0002 | 0.025 * | noneno run with the option |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 140 | 0.4%7 of 1,582 | -10.1-11.8 to -8.6 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Sol high · chat Closed · Workhorse | 276 | 0.5%8 of 1,582 | -10.1-11.8 to -8.5 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 191 | 0.7%11 of 1,582 | -9.9-11.6 to -8.3 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 190 | 0.8%12 of 1,582 | -9.8-11.5 to -8.3 | 0.0002 | 0.025 * | noneno run with the option |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 188 | 0.8%13 of 1,582 | -9.7-11.5 to -8.2 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Sol low · chat Closed · Workhorse | 124 | 0.8%13 of 1,582 | -9.7-11.5 to -8.2 | 0.0002 | 0.025 * | noneno run with the option |
| Gemini 4 Argon low · chat Closed · Flagship | 557 | 1.1%17 of 1,582 | -9.5-11.2 to -8.0 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Fable 5.1 high · chat Closed · Flagship | 310 | 1.2%19 of 1,582 | -9.4-11.1 to -8.0 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Opus 5.5 high · chat Closed · Flagship | 278 | 1.5%23 of 1,582 | -9.1-10.8 to -7.6 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 192 | 1.5%23 of 1,582 | -9.1-10.7 to -7.6 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Terra high · chat Closed · Workhorse | 275 | 2.0%31 of 1,582 | -8.6-10.3 to -7.1 | 0.0002 | 0.025 * | noneno run with the option |
| Grok 4.7 high · chat Closed · Workhorse | 311 | 2.0%31 of 1,582 | -8.6-10.3 to -7.1 | 0.0002 | 0.025 * | noneno run with the option |
| Qwen4-Max high · chat Closed · Workhorse | 1336 | 2.0%31 of 1,582 | -8.6-10.4 to -7.1 | 0.0002 | 0.025 * | noneno run with the option |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 123 | 3.2%50 of 1,582 | -7.4-9.2 to -5.8 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Terra low · chat Closed · Workhorse | 274 | 3.2%50 of 1,582 | -7.4-9.2 to -6.0 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Opus 5.5 low · chat Closed · Flagship | 47 | 3.2%51 of 1,582 | -7.3-9.0 to -5.9 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 193 | 4.1%65 of 1,582 | -6.4-8.1 to -4.9 | 0.0002 | 0.025 * | noneno run with the option |
| Muse Ember max · chat Closed · Efficient | 939 | 5.4%86 of 1,582 | -5.1-7.0 to -3.4 | 0.0002 | 0.025 * | noneno run with the option |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 585 | 5.5%87 of 1,582 | -5.1-7.0 to -3.4 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 55 | 7.1%113 of 1,582 | -3.4-5.3 to -1.8 | 0.0002 | 0.025 * | noneno run with the option |
| DeepSeek-V4.1-Flash reasoning on · chat · first release Open · 552B MoE · 16B active | 121 | 7.1%113 of 1,582 | -3.4-5.4 to -1.8 | 0.0002 | 0.025 * | noneno run with the option |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 320 | 8.9%141 of 1,582 | -1.6-3.5 to +0.0 | 0.051 | 0.41 | noneno run with the option |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 870 | 10.0%158 of 1,582 | -0.6-2.5 to +1.1 | 0.45 | 1 | noneno run with the option |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 160 | 10.3%163 of 1,582 | -0.3-2.2 to +1.4 | 0.71 | 1 | noneno run with the option |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 866 | 11.4%181 of 1,582 | +0.9-1.1 to +2.7 | 0.45 | 1 | noneno run with the option |
| Gemma 4 31B reasoning off · chat Open · 31B | 119 | 15.1%239 of 1,582 | +4.6+2.5 to +6.1 | 0.0002 | 0.025 * | noneno run with the option |
| GLM-5.3 low · chat Open · Workhorse | 314 | 16.4%260 of 1,582 | +5.9+3.8 to +7.8 | 0.0002 | 0.025 * | noneno run with the option |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 315 | 18.6%294 of 1,582 | +8.0+5.8 to +9.9 | 0.0002 | 0.025 * | noneno run with the option |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 877 | 18.9%299 of 1,582 | +8.3+6.1 to +10.4 | 0.0002 | 0.025 * | noneno run with the option |
| Qwen3.8-27B low · chat Open · 27B | 318 | 19.6%310 of 1,582 | +9.0+6.7 to +10.9 | 0.0002 | 0.025 * | noneno run with the option |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 558 | 21.8%345 of 1,582 | +11.3+8.9 to +13.3 | 0.0002 | 0.025 * | noneno run with the option |
| DeepSeek-V4.1-Flash reasoning off · chat · first release Open · 552B MoE · 16B active | 120 | 21.9%346 of 1,582 | +11.3+9.3 to +13.6 | 0.0002 | 0.025 * | noneno run with the option |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 189 | 21.9%346 of 1,582 | +11.3+9.0 to +13.3 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6 Luna low · chat Closed · Efficient | 133 | 24.5%388 of 1,582 | +14.0+11.8 to +16.2 | 0.0002 | 0.025 * | noneno run with the option |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 862 | 25.7%406 of 1,582 | +15.1+12.8 to +17.4 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 54 | 27.8%440 of 1,582 | +17.3+14.9 to +19.5 | 0.0002 | 0.025 * | noneno run with the option |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 985 | 29.2%462 of 1,582 | +18.6+16.4 to +20.7 | 0.0002 | 0.025 * | noneno run with the option |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 159 | 32.6%515 of 1,582 | +22.0+19.5 to +24.3 | 0.0002 | 0.025 * | noneno run with the option |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 83 | 35.3%559 of 1,582 | +24.8+22.2 to +27.0 | 0.0002 | 0.025 * | noneno run with the option |
| Qwen3.8-27B yes/no scoring Open · 27B | 50 | 37.9%599 of 1,582 | +27.3+24.9 to +29.7 | 0.0002 | 0.025 * | noneno run with the option |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 53 | 40.1%635 of 1,582 | +29.6+27.1 to +31.9 | 0.0002 | 0.025 * | noneno run with the option |
| Erk-14B reasoning off · chat Open · 14B | 132 | 46.5%735 of 1,582 | +35.9+33.1 to +38.4 | 0.0002 | 0.025 * | noneno run with the option |
| Qwen3-14B reasoning off · chat Open · 15B | 131 | 47.1%745 of 1,582 | +36.5+33.6 to +38.9 | 0.0002 | 0.025 * | noneno run with the option |
| GLiNER2.5 Multi Open · 0.3B | 122 | 83.5%1,321 of 1,582 | +72.9+70.4 to +75.2 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Haiku 5.5 max · chat Closed · Efficient | 948 | —left out: its run has 1,414 scored answers, fewer than k = 1,582 | —left out, fewer than k scored answers | — | — | noneno run with the option |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 1040 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 1037 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 1288 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| LFM3-1.7B reasoning off · chat Open · 1.7B | 1227 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 1089 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 1299 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 1218 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 1146 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 1149 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
English: k = 496 of the 547 kept reading questions
| Model | Five-option runthe run whose k answers are kept | Wrong of the k answerswrong / 496; lower is better | Difference from d1-ultrapoints, with its 95% interval; positive = more wrong answers | ppaired bootstrap, two-sided | Holm padjusted over the 123 | Runs with the optionfor context: answered, wrong of answers |
|---|---|---|---|---|---|---|
| Claude Fable 5.2 max · chat Closed · Flagship | 971 | 0.4%2 of 496 | -5.4-7.7 to -3.2 | 0.0002 | 0.025 * | noneno run with the option |
| Gemini 4 Carbon high · chat Closed · Flagship | 1055 | 0.4%2 of 496 | -5.4-7.5 to -3.4 | 0.0002 | 0.025 * | noneno run with the option |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 1107 | 0.6%3 of 496 | -5.2-7.5 to -3.2 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6 Astra low · chat Closed · Flagship | 616 | 0.8%4 of 496 | -5.0-7.1 to -3.0 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6 Astra high · chat Closed · Flagship | 615 | 1.0%5 of 496 | -4.8-6.9 to -2.8 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6 Astra medium · chat Closed · Flagship | 624 | 1.0%5 of 496 | -4.8-6.7 to -2.6 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Sol high · chat Closed · Workhorse | 618 | 1.2%6 of 496 | -4.6-6.7 to -2.6 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Sol low · chat Closed · Workhorse | 619 | 1.2%6 of 496 | -4.6-6.7 to -2.6 | 0.0002 | 0.025 * | noneno run with the option |
| Gemini 4 Argon low · chat Closed · Flagship | 593 | 1.4%7 of 496 | -4.4-6.5 to -2.4 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 622 | 1.4%7 of 496 | -4.4-6.7 to -2.4 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 623 | 1.6%8 of 496 | -4.2-6.5 to -2.2 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Terra high · chat Closed · Workhorse | 620 | 1.6%8 of 496 | -4.2-6.0 to -2.2 | 0.0002 | 0.025 * | noneno run with the option |
| GPT-6.1 Terra low · chat Closed · Workhorse | 621 | 1.6%8 of 496 | -4.2-6.2 to -2.2 | 0.0002 | 0.025 * | noneno run with the option |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 583 | 1.8%9 of 496 | -4.0-6.2 to -2.0 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Fable 5.1 high · chat Closed · Flagship | 580 | 2.0%10 of 496 | -3.8-6.0 to -2.0 | 0.0004 | 0.028 * | noneno run with the option |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 592 | 2.0%10 of 496 | -3.8-5.6 to -1.8 | 0.0004 | 0.028 * | noneno run with the option |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 589 | 2.0%10 of 496 | -3.8-5.8 to -1.8 | 0.0004 | 0.028 * | noneno run with the option |
| Claude Opus 5.5 high · chat Closed · Flagship | 587 | 2.6%13 of 496 | -3.2-5.0 to -1.2 | 0.002 | 0.12 | noneno run with the option |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 578 | 2.8%14 of 496 | -3.0-4.8 to -1.0 | 0.005 | 0.24 | noneno run with the option |
| Grok 4.7 high · chat Closed · Workhorse | 579 | 2.8%14 of 496 | -3.0-5.0 to -1.0 | 0.0022 | 0.13 | noneno run with the option |
| KnowLine-31B-Gen1.5 Open · 31B | 953 | 2.8%14 of 496 | -3.0-4.8 to -1.2 | 0.0028 | 0.15 | 955“I don't know”, run 955: 547 answered (100.0%), 6.8% wrong (37) |
| Qwen4-Max high · chat Closed · Workhorse | 1334 | 2.8%14 of 496 | -3.0-5.0 to -1.2 | 0.0018 | 0.11 | noneno run with the option |
| Copernicus-31B Open · 31B | 1168 | 3.2%16 of 496 | -2.6-4.4 to -1.0 | 0.0032 | 0.17 | 1170“I don't know”, run 1170: 547 answered (100.0%), 7.1% wrong (39) |
| KnowLine-31B-Gen2 Open · 31B | 1327 | 3.2%16 of 496 | -2.6-4.4 to -0.6 | 0.012 | 0.46 | 1329“I don't know”, run 1329: 547 answered (100.0%), 7.3% wrong (40) |
| REAX 1.3 Open · 57B | 829 | 3.2%16 of 496 | -2.6-4.2 to -0.6 | 0.011 | 0.46 | 831“I don't know”, run 831: 544 answered (99.5%), 7.7% wrong (42) |
| Claude Opus 5.5 low · chat Closed · Flagship | 588 | 3.8%19 of 496 | -2.0-4.2 to +0.0 | 0.053 | 1 | noneno run with the option |
| Claude Haiku 5.5 max · chat Closed · Efficient | 949 | 4.0%20 of 496 | -1.8-4.0 to +0.4 | 0.12 | 1 | noneno run with the option |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 594 | 4.4%22 of 496 | -1.4-3.4 to +0.8 | 0.3 | 1 | noneno run with the option |
| Karpo Decision 31B Open · 31B | 1159 | 4.4%22 of 496 | -1.4-3.2 to +0.6 | 0.24 | 1 | 1161“I don't know”, run 1161: 547 answered (100.0%), 7.7% wrong (42) |
| Logits Sigma-1 Duo Open · 57B | 1229 | 4.6%23 of 496 | -1.2-2.8 to +0.6 | 0.28 | 1 | 1231“I don't know”, run 1231: 545 answered (99.6%), 8.1% wrong (44) |
| Turion Jev 31B Open · 31B | 1075 | 4.6%23 of 496 | -1.2-3.0 to +0.8 | 0.3 | 1 | 1077“I don't know”, run 1077: 543 answered (99.3%), 7.9% wrong (43) |
| d1.2-ultra Closed · Efficient | 1359 | 4.8%24 of 496 | -1.0-3.0 to +1.2 | 0.42 | 1 | 1363“I don't know”, run 1363: 438 answered (80.1%), 3.0% wrong (13) |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 584 | 4.8%24 of 496 | -1.0-3.2 to +1.2 | 0.42 | 1 | noneno run with the option |
| Sieve 31B v3 Open · 31B | 1064 | 4.8%24 of 496 | -1.0-3.2 to +1.0 | 0.38 | 1 | 1066“I don't know”, run 1066: 546 answered (99.8%), 8.2% wrong (45) |
| Gemini Decision 1 20261111 Closed · Efficient | 1009 | 5.0%25 of 496 | -0.8-2.8 to +1.4 | 0.51 | 1 | 1019“I don't know”, run 1019: 542 answered (99.1%), 9.2% wrong (50) |
| GLiDE v1.2 Open · 27B | 521 | 5.2%26 of 496 | -0.6-2.4 to +1.6 | 0.76 | 1 | 523“I don't know”, run 523: 517 answered (94.5%), 7.0% wrong (36) |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 871 | 5.2%26 of 496 | -0.6-2.4 to +1.6 | 0.78 | 1 | noneno run with the option |
| GLiDE 27B Open · 27B | 901 | 5.4%27 of 496 | -0.4-2.8 to +1.4 | 0.6 | 1 | 903“I don't know”, run 903: 525 answered (96.0%), 6.9% wrong (36) |
| Drex Max Open · 27B | 552 | 5.6%28 of 496 | -0.2-2.2 to +2.0 | 0.98 | 1 | 798“I don't know”, run 798: 545 answered (99.6%), 11.0% wrong (60) |
| d1-ultra Closed · Efficient | 411 | 5.8%29 of 496 | referenceits own run 449 with “I don't know”: 39 wrong of 496 (7.9%), not part of the test | reference | reference | 449“I don't know”, run 449: 496 answered (90.7%), 7.9% wrong (39) |
| d3 Open · 26B | 554 | 5.8%29 of 496 | +0.0-1.8 to +2.2 | 0.93 | 1 | 793“I don't know”, run 793: 546 answered (99.8%), 10.6% wrong (58) |
| Gemma 4 31B reasoning off · chat Open · 31B | 774 | 5.8%29 of 496 | +0.0-2.2 to +2.0 | 0.99 | 1 | noneno run with the option |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 582 | 6.0%30 of 496 | +0.2-2.2 to +2.2 | 1 | 1 | noneno run with the option |
| d1.2 Closed · Efficient | 1357 | 6.2%31 of 496 | +0.4-1.6 to +2.6 | 0.74 | 1 | 1361“I don't know”, run 1361: 488 answered (89.2%), 7.6% wrong (37) |
| d3-ultra Open · 26B | 550 | 6.2%31 of 496 | +0.4-1.6 to +2.4 | 0.77 | 1 | 795“I don't know”, run 795: 542 answered (99.1%), 10.3% wrong (56) |
| Mammoth-1 Open · 27B | 503 | 6.2%31 of 496 | +0.4-1.8 to +2.4 | 0.84 | 1 | 505“I don't know”, run 505: 510 answered (93.2%), 7.1% wrong (36) |
| Drex Max v1.1 Closed · Efficient (provisional) | 399 | 6.5%32 of 496 | +0.6-1.6 to +2.8 | 0.64 | 1 | 447“I don't know”, run 447: 504 answered (92.1%), 8.1% wrong (41) |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 823 | 6.9%34 of 496 | +1.0-1.4 to +3.4 | 0.43 | 1 | noneno run with the option |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 527 | 6.9%34 of 496 | +1.0-1.0 to +3.2 | 0.32 | 1 | 529“I don't know”, run 529: 545 answered (99.6%), 11.7% wrong (64) |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 878 | 7.1%35 of 496 | +1.2-1.0 to +3.4 | 0.35 | 1 | noneno run with the option |
| Perplexity Decider 27B v1 Open · 27B | 626 | 7.1%35 of 496 | +1.2-0.8 to +3.6 | 0.22 | 1 | noneno run with the option |
| Perplexity Decider 27B v1.3 Open · 27B | 1137 | 7.1%35 of 496 | +1.2-1.0 to +3.2 | 0.34 | 1 | 1139“I don't know”, run 1139: 542 answered (99.1%), 9.6% wrong (52) |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 808 | 7.3%36 of 496 | +1.4-1.0 to +3.8 | 0.3 | 1 | noneno run with the option |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 783 | 7.3%36 of 496 | +1.4-0.8 to +3.6 | 0.26 | 1 | noneno run with the option |
| Muse Ember max · chat Closed · Efficient | 942 | 7.3%36 of 496 | +1.4-0.6 to +4.0 | 0.19 | 1 | noneno run with the option |
| Perplexity Decider 27B v1.2 API Open · 27B | 637 | 7.3%36 of 496 | +1.4-0.6 to +3.8 | 0.17 | 1 | 643“I don't know”, run 643: 544 answered (99.5%), 11.6% wrong (63) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 639 | 7.5%37 of 496 | +1.6-0.4 to +3.8 | 0.16 | 1 | 644“I don't know”, run 644: 537 answered (98.2%), 10.4% wrong (56) |
| d1.1 Closed · Efficient | 565 | 8.5%42 of 496 | +2.6+0.2 to +4.8 | 0.038 | 1 | noneno run with the option |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 590 | 8.7%43 of 496 | +2.8+0.2 to +5.8 | 0.042 | 1 | noneno run with the option |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 804 | 8.7%43 of 496 | +2.8+0.4 to +5.0 | 0.028 | 0.95 | noneno run with the option |
| KnowLine-12B-Gen1 Open · 12B | 1313 | 8.7%43 of 496 | +2.8+0.4 to +5.0 | 0.021 | 0.76 | 1316“I don't know”, run 1316: 546 answered (99.8%), 13.9% wrong (76) |
| Clef Open · 27B | 613 | 8.9%44 of 496 | +3.0+0.6 to +5.2 | 0.014 | 0.56 | 785“I don't know”, run 785: 546 answered (99.8%), 13.7% wrong (75) |
| d1 Closed · Efficient | 430 | 8.9%44 of 496 | +3.0+0.8 to +5.6 | 0.017 | 0.62 | 445“I don't know”, run 445: 545 answered (99.6%), 12.3% wrong (67) |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 867 | 8.9%44 of 496 | +3.0+0.6 to +5.4 | 0.014 | 0.56 | noneno run with the option |
| Jev 1.14 Closed · Efficient | 410 | 8.9%44 of 496 | +3.0+0.4 to +5.2 | 0.024 | 0.83 | 446“I don't know”, run 446: 544 answered (99.5%), 13.2% wrong (72) |
| SubQ Snap 1.1 Closed · Efficient | 413 | 8.9%44 of 496 | +3.0+1.0 to +5.8 | 0.008 | 0.36 | 448“I don't know”, run 448: 526 answered (96.2%), 12.0% wrong (63) |
| SubQ Snap Preview Closed · Efficient | 595 | 8.9%44 of 496 | +3.0+1.0 to +5.8 | 0.008 | 0.36 | noneno run with the option |
| Jev 1.13 Closed · Efficient | 564 | 9.1%45 of 496 | +3.2+1.2 to +5.6 | 0.004 | 0.2 | noneno run with the option |
| GLiDE 9B Open · 7.9B | 909 | 9.3%46 of 496 | +3.4+1.0 to +6.0 | 0.0098 | 0.42 | 911“I don't know”, run 911: 533 answered (97.4%), 13.1% wrong (70) |
| Jeffan-9B-1211 Open · 9.0B | 1195 | 9.3%46 of 496 | +3.4+0.8 to +6.0 | 0.01 | 0.43 | 1197“I don't know”, run 1197: 547 answered (100.0%), 13.0% wrong (71) |
| OneJev-27B Open · 27B | 778 | 9.5%47 of 496 | +3.6+1.2 to +6.0 | 0.005 | 0.24 | 780“I don't know”, run 780: 544 answered (99.5%), 13.2% wrong (72) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 1293 | 9.5%47 of 496 | +3.6+1.2 to +6.5 | 0.0068 | 0.31 | 1295“I don't know”, run 1295: 545 answered (99.6%), 13.6% wrong (74) |
| Adjudex 9B v5 Open · 9.0B | 1186 | 9.7%48 of 496 | +3.8+1.2 to +6.5 | 0.0038 | 0.19 | 1188“I don't know”, run 1188: 547 answered (100.0%), 14.1% wrong (77) |
| Open-Jev-27B-v1.1 Open · 27B | 760 | 9.7%48 of 496 | +3.8+1.4 to +6.5 | 0.0028 | 0.15 | 763“I don't know”, run 763: 531 answered (97.1%), 14.3% wrong (76) |
| Qwen3.8-27B low · chat Open · 27B | 667 | 9.7%48 of 496 | +3.8+1.4 to +6.5 | 0.0024 | 0.13 | noneno run with the option |
| GLM-5.3 low · chat Open · Workhorse | 577 | 9.9%49 of 496 | +4.0+1.4 to +6.5 | 0.0032 | 0.17 | noneno run with the option |
| KnowLine-9B-Gen1 Open · 9.0B | 1177 | 9.9%49 of 496 | +4.0+1.2 to +6.7 | 0.0044 | 0.22 | 1179“I don't know”, run 1179: 546 answered (99.8%), 13.9% wrong (76) |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 820 | 10.1%50 of 496 | +4.2+1.6 to +6.9 | 0.0014 | 0.087 | noneno run with the option |
| Mercury Decide Closed · Efficient | 398 | 10.1%50 of 496 | +4.2+1.6 to +6.7 | 0.0014 | 0.087 | 450“I don't know”, run 450: 536 answered (98.0%), 13.2% wrong (71) |
| Drex 2.0 Open · 35B MoE · 3.0B active | 566 | 10.3%51 of 496 | +4.4+2.0 to +7.1 | 0.0004 | 0.028 * | noneno run with the option |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 678 | 10.3%51 of 496 | +4.4+2.0 to +7.3 | 0.0004 | 0.028 * | noneno run with the option |
| Cerno 9B v1.2 Open · 9.0B | 1085 | 10.5%52 of 496 | +4.6+1.8 to +6.9 | 0.0014 | 0.087 | 1087“I don't know”, run 1087: 545 answered (99.6%), 14.1% wrong (77) |
| Microsoft-Decision-1 Closed · Efficient | 412 | 10.5%52 of 496 | +4.6+2.2 to +7.3 | 0.0006 | 0.039 * | 451“I don't know”, run 451: 544 answered (99.5%), 14.2% wrong (77) |
| KnowLine-9B-Gen2 Open · 9.0B | 1321 | 10.9%54 of 496 | +5.0+2.4 to +7.9 | 0.001 | 0.063 | 1323“I don't know”, run 1323: 546 answered (99.8%), 14.3% wrong (78) |
| Qwen3.8-27B yes/no scoring Open · 27B | 696 | 10.9%54 of 496 | +5.0+2.4 to +7.7 | 0.0002 | 0.025 * | noneno run with the option |
| Elephant-1 Open · 9.0B | 499 | 11.1%55 of 496 | +5.2+2.6 to +7.9 | 0.0002 | 0.025 * | 501“I don't know”, run 501: 524 answered (95.8%), 12.8% wrong (67) |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 986 | 11.1%55 of 496 | +5.2+2.6 to +7.9 | 0.0002 | 0.025 * | noneno run with the option |
| celeris-1-decision Closed · Efficient | 546 | 11.3%56 of 496 | +5.4+2.6 to +8.1 | 0.0002 | 0.025 * | 548“I don't know”, run 548: 455 answered (83.2%), 8.1% wrong (37) |
| GPT-6 Luna low · chat Closed · Efficient | 617 | 11.3%56 of 496 | +5.4+2.6 to +8.5 | 0.0006 | 0.039 * | noneno run with the option |
| Valen-Decision-1.5-9B Open · 9.0B | 1278 | 11.5%57 of 496 | +5.6+3.2 to +8.7 | 0.0002 | 0.025 * | 1280“I don't know”, run 1280: 546 answered (99.8%), 15.4% wrong (84) |
| GPT-6 Luna Decisions API Closed · Efficient | 397 | 11.7%58 of 496 | +5.8+3.2 to +8.9 | 0.0002 | 0.025 * | 444“I don't know”, run 444: 537 answered (98.2%), 17.9% wrong (96) |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 863 | 11.7%58 of 496 | +5.8+3.4 to +8.7 | 0.0002 | 0.025 * | noneno run with the option |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 591 | 11.9%59 of 496 | +6.0+3.2 to +9.3 | 0.0002 | 0.025 * | noneno run with the option |
| Perplexity Decider Flash v1.3 Open · 4.0B | 1126 | 11.9%59 of 496 | +6.0+3.2 to +9.1 | 0.0002 | 0.025 * | 1128“I don't know”, run 1128: 540 answered (98.7%), 16.7% wrong (90) |
| vera-medium-omni Open · 4.2B | 1267 | 11.9%59 of 496 | +6.0+3.2 to +8.7 | 0.0002 | 0.025 * | 1269“I don't know”, run 1269: 540 answered (98.7%), 16.1% wrong (87) |
| Cygnet Open · 12B | 753 | 12.1%60 of 496 | +6.2+3.6 to +8.9 | 0.0002 | 0.025 * | 755“I don't know”, run 755: 537 answered (98.2%), 14.7% wrong (79) |
| Winnow-12B Open · 12B | 746 | 12.1%60 of 496 | +6.2+3.6 to +9.1 | 0.0002 | 0.025 * | 748“I don't know”, run 748: 532 answered (97.3%), 16.0% wrong (85) |
| Dexy-Decision-9B-V2 Open · 9.0B | 1258 | 12.3%61 of 496 | +6.5+3.4 to +9.3 | 0.0002 | 0.025 * | 1260“I don't know”, run 1260: 546 answered (99.8%), 16.5% wrong (90) |
| Perplexity Decider Flash v1.2 Open · 4.0B | 533 | 12.5%62 of 496 | +6.7+4.0 to +9.9 | 0.0002 | 0.025 * | 535“I don't know”, run 535: 542 answered (99.1%), 18.5% wrong (100) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 1081 | 12.5%62 of 496 | +6.7+4.0 to +10.1 | 0.0002 | 0.025 * | 1083“I don't know”, run 1083: 547 answered (100.0%), 17.2% wrong (94) |
| Verdin-4B-v1.1 Open · 4.0B | 1247 | 12.7%63 of 496 | +6.9+3.8 to +9.9 | 0.0002 | 0.025 * | 1249“I don't know”, run 1249: 545 answered (99.6%), 17.2% wrong (94) |
| Clef-Flash Open · 9.0B | 614 | 13.5%67 of 496 | +7.7+4.6 to +10.7 | 0.0002 | 0.025 * | 660“I don't know”, run 660: 542 answered (99.1%), 16.6% wrong (90) |
| Chinese Decision Pointer 4B Open · 4.0B | 1209 | 14.9%74 of 496 | +9.1+6.2 to +12.5 | 0.0002 | 0.025 * | 1211“I don't know”, run 1211: 543 answered (99.3%), 19.2% wrong (104) |
| KnowLine 4B Gen5.5 Open · 4.0B | 961 | 14.9%74 of 496 | +9.1+6.2 to +12.3 | 0.0002 | 0.025 * | 963“I don't know”, run 963: 546 answered (99.8%), 19.2% wrong (105) |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 672 | 15.1%75 of 496 | +9.3+6.3 to +12.1 | 0.0002 | 0.025 * | noneno run with the option |
| Qwen3-14B reasoning off · chat Open · 15B | 770 | 15.5%77 of 496 | +9.7+6.5 to +13.1 | 0.0002 | 0.025 * | noneno run with the option |
| Metask-Jev-4B Open · 4.5B | 739 | 18.1%90 of 496 | +12.3+9.1 to +15.5 | 0.0002 | 0.025 * | 741“I don't know”, run 741: 545 answered (99.6%), 22.6% wrong (123) |
| Erk-14B reasoning off · chat Open · 14B | 762 | 23.6%117 of 496 | +17.7+14.1 to +21.6 | 0.0002 | 0.025 * | noneno run with the option |
| KnowLine-2B-Gen1 Open · 2.0B | 1057 | 24.8%123 of 496 | +19.0+15.1 to +23.2 | 0.0002 | 0.025 * | 1059“I don't know”, run 1059: 509 answered (93.1%), 26.5% wrong (135) |
| GLiDE 2B Open · 1.9B | 917 | 25.2%125 of 496 | +19.4+15.7 to +23.0 | 0.0002 | 0.025 * | 919“I don't know”, run 919: 500 answered (91.4%), 26.4% wrong (132) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 973 | 26.0%129 of 496 | +20.2+16.5 to +24.2 | 0.0002 | 0.025 * | 975“I don't know”, run 975: 515 answered (94.1%), 30.5% wrong (157) |
| d1-2B Open · 1.9B | 708 | 29.4%146 of 496 | +23.6+19.8 to +27.8 | 0.0002 | 0.025 * | 710“I don't know”, run 710: 396 answered (72.4%), 24.5% wrong (97) |
| GLiDE 0.4B Open · 0.4B | 933 | 36.5%181 of 496 | +30.6+26.6 to +35.1 | 0.0002 | 0.025 * | 935“I don't know”, run 935: 394 answered (72.0%), 34.3% wrong (135) |
| GLiDE 0.8B Open · 0.8B | 925 | 36.7%182 of 496 | +30.8+26.4 to +35.1 | 0.0002 | 0.025 * | 927“I don't know”, run 927: 498 answered (91.0%), 35.5% wrong (177) |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 1011 | 37.5%186 of 496 | +31.7+27.2 to +35.9 | 0.0002 | 0.025 * | 1013“I don't know”, run 1013: 535 answered (97.8%), 40.6% wrong (217) |
| KnowLine-0.8B-Gen1 Open · 0.8B | 1001 | 37.9%188 of 496 | +32.1+27.6 to +36.5 | 0.0002 | 0.025 * | 1003“I don't know”, run 1003: 521 answered (95.2%), 41.7% wrong (217) |
| OneJev1.6-0.8B Open · 0.8B | 1025 | 37.9%188 of 496 | +32.1+27.8 to +36.5 | 0.0002 | 0.025 * | 1027“I don't know”, run 1027: 473 answered (86.5%), 39.5% wrong (187) |
| GLiClass v4 Edge Open · 0.4B | 1042 | 39.3%195 of 496 | +33.5+29.0 to +37.7 | 0.0002 | 0.025 * | 1044“I don't know”, run 1044: 495 answered (90.5%), 39.6% wrong (196) |
| Decider 1 Closed · Efficient | 596 | 42.3%210 of 496 | +36.5+32.1 to +40.9 | 0.0002 | 0.025 * | noneno run with the option |
| Strands Decider 2B Open · 2.0B | 702 | 51.4%255 of 496 | +45.6+40.9 to +50.4 | 0.0002 | 0.025 * | 704“I don't know”, run 704: 545 answered (99.6%), 55.2% wrong (301) |
| Laya Open · 0.4B | 726 | 76.8%381 of 496 | +71.0+66.5 to +75.4 | 0.0002 | 0.025 * | 728“I don't know”, run 728: 464 answered (84.8%), 79.1% wrong (367) |
| GLiNER2.5-multi-Decide Open · 0.3B | 714 | 78.2%388 of 496 | +72.4+68.3 to +76.4 | 0.0002 | 0.025 * | 716“I don't know”, run 716: 477 answered (87.2%), 76.1% wrong (363) |
| GLiNER2.5 Multi Open · 0.3B | 720 | 83.3%413 of 496 | +77.4+73.8 to +81.5 | 0.0002 | 0.025 * | noneno run with the option |
| Laya Multilingual Open · 0.3B | 732 | 85.1%422 of 496 | +79.2+75.2 to +83.1 | 0.0002 | 0.025 * | 734“I don't know”, run 734: 339 answered (62.0%), 82.6% wrong (280) |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 889 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 1034 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 886 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| LFM3-1.7B reasoning off · chat Open · 1.7B | 1223 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 1061 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 1094 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 1215 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 1143 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 1072 | Not measureda plain answer line, no confidence | —left out, not measured | — | — | noneno run with the option |
Every setup's curve in Turkish
One small plot per setup, all 137, grouped and ordered as the Turkish table above, on the same axes: answered across (0 to 100% of the 2,198 questions), wrong of answers up (0 to 92%; anything higher is drawn at the top edge). The line is the setup's five-option run held back by its own confidence, from 1 answer to its scored answers; the dashed line is d1-ultra's (run 199) in every plot, and the dashed vertical line is k = 1,582. The markers are the setup's runs with the option ( “Bilmiyorum” “I don't know”). Each line is thinned to one vertex per 1% of coverage, always keeping the first, the last and the point at k; every vertex is a curve point.
Decision models
Copernicus-31B Open · 31B Wrong of the 1,582 answers: 95 (6.0%), -4.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1172, 18.3% wrong of 2,193KnowLine-31B-Gen1.5 Open · 31B Wrong of the 1,582 answers: 105 (6.6%), -3.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 959, 18.7% wrong of 2,197Turion Jev 31B Open · 31B Wrong of the 1,582 answers: 117 (7.4%), -3.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1285, 20.0% wrong of 2,183KnowLine-31B-Gen2 Open · 31B Wrong of the 1,582 answers: 122 (7.7%), -2.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1331, 18.6% wrong of 2,198Karpo Decision 31B Open · 31B Wrong of the 1,582 answers: 146 (9.2%), -1.3 points against d1-ultra, Holm p = 0.59
Runs with the option: “Bilmiyorum” run 1163, 21.6% wrong of 2,195REAX 1.3 Open · 57B Wrong of the 1,582 answers: 151 (9.5%), -1.0 points against d1-ultra, Holm p = 1
Runs with the option: “Bilmiyorum” run 833, 21.6% wrong of 2,196Gemini Decision 1 20261111 Closed · Efficient Wrong of the 1,582 answers: 156 (9.9%), -0.7 points against d1-ultra, Holm p = 1
Runs with the option: “Bilmiyorum” run 1023, 23.1% wrong of 2,188d1-ultra Closed · Efficient Wrong of the 1,582 answers: 167 (10.6%), the reference, dashed in every plot
Runs with the option: “Bilmiyorum” run 350, 13.0% wrong of 1,581 and “I don't know” run 511, 13.7% wrong of 1,582Sieve 31B v3 Open · 31B Wrong of the 1,582 answers: 183 (11.6%), +1.0 points against d1-ultra, Holm p = 1
Runs with the option: “Bilmiyorum” run 1253, 25.2% wrong of 2,198Logits Sigma-1 Duo Open · 57B Wrong of the 1,582 answers: 209 (13.2%), +2.7 points against d1-ultra, Holm p = 0.083
Runs with the option: “Bilmiyorum” run 1233, 24.7% wrong of 2,196GLiDE 27B Open · 27B Wrong of the 1,582 answers: 259 (16.4%), +5.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 905, 23.1% wrong of 1,907GLiDE v1.2 Open · 27B Wrong of the 1,582 answers: 289 (18.3%), +7.7 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 369, 26.2% wrong of 1,877 and “I don't know” run 525, 24.1% wrong of 1,772Perplexity Decider 27B v1.2 API · with web Open · 27B Wrong of the 1,582 answers: 313 (19.8%), +9.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 642, 32.3% wrong of 2,175d3 Open · 26B Wrong of the 1,582 answers: 314 (19.8%), +9.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 358, 32.5% wrong of 2,198 and “I don't know” run 544, 32.5% wrong of 2,196d3-ultra Open · 26B Wrong of the 1,582 answers: 315 (19.9%), +9.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 363, 32.2% wrong of 2,188 and “I don't know” run 540, 32.2% wrong of 2,189Perplexity Decider 27B v1.1 Open · 27B Wrong of the 1,582 answers: 318 (20.1%), +9.5 points against d1-ultra, Holm p = 0.025 *
Runs with the option: Not run: Perplexity's API now answers requests for v1.1 with v1.2, so a new run could not measure v1.1.Perplexity Decider 27B v1.3 Open · 27B Wrong of the 1,582 answers: 320 (20.2%), +9.7 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1141, 32.4% wrong of 2,193Drex Max Open · 27B Wrong of the 1,582 answers: 321 (20.3%), +9.7 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 367, 32.4% wrong of 2,196 and “I don't know” run 542, 32.4% wrong of 2,196Drex Max v1.1 Closed · Efficient (provisional) Wrong of the 1,582 answers: 321 (20.3%), +9.7 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 352, 27.5% wrong of 1,878 and “I don't know” run 513, 24.1% wrong of 1,712Mammoth-1 Open · 27B Wrong of the 1,582 answers: 322 (20.4%), +9.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 386, 20.8% wrong of 1,508 and “I don't know” run 519, 19.1% wrong of 1,445d1.2-ultra Closed · Efficient Wrong of the 1,582 answers: 323 (20.4%), +9.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1367, 12.1% wrong of 1,202 and “I don't know” run 1378, 11.6% wrong of 1,180GLiDE API Closed · Efficient Wrong of the 1,582 answers: 325 (20.5%), +10.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 351, 21.2% wrong of 1,677 and “I don't know” run 510, 20.7% wrong of 1,617 (made with Fastino's engine glide_v2, not run 129's glide)Perplexity Decider 27B v1.2 DGX Spark Open · 27B Wrong of the 1,582 answers: 328 (20.7%), +10.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 372, 32.7% wrong of 2,196 and “I don't know” run 531, 33.2% wrong of 2,194Perplexity Decider 27B v1.2 API Open · 27B Wrong of the 1,582 answers: 329 (20.8%), +10.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 641, 33.1% wrong of 2,195SubQ Snap Preview Closed · Efficient Wrong of the 1,582 answers: 356 (22.5%), +11.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: Not run: SubQ's API returns its probabilities in 32-bit precision, so their sums miss our 7-decimal check. An allowance for that was approved on 5 November 2026 for its English runs; a Turkish run with “Bilmiyorum” was not part of that request.SubQ Snap 1.1 Closed · Efficient Wrong of the 1,582 answers: 361 (22.8%), +12.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 392, 27.8% wrong of 1,761 and “I don't know” run 512, 26.7% wrong of 1,700GPT-6 Luna Decisions API Closed · Efficient Wrong of the 1,582 answers: 372 (23.5%), +13.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetKnowLine-12B-Gen1 Open · 12B Wrong of the 1,582 answers: 418 (26.4%), +15.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1318, 36.6% wrong of 2,197Perplexity Decider 27B v1 Open · 27B Wrong of the 1,582 answers: 430 (27.2%), +16.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetd1.1 Closed · Efficient Wrong of the 1,582 answers: 434 (27.4%), +16.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetd1.2 Closed · Efficient Wrong of the 1,582 answers: 436 (27.6%), +17.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1365, 20.7% wrong of 1,225 and “I don't know” run 1377, 21.5% wrong of 1,297celeris-1-decision Closed · Efficient Wrong of the 1,582 answers: 450 (28.4%), +17.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetOneJev-27B Open · 27B Wrong of the 1,582 answers: 455 (28.8%), +18.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 782, 40.6% wrong of 2,197Jev 1.14 Closed · Efficient Wrong of the 1,582 answers: 485 (30.7%), +20.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetMercury Decide Closed · Efficient Wrong of the 1,582 answers: 487 (30.8%), +20.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetJev 1.13 Closed · Efficient Wrong of the 1,582 answers: 498 (31.5%), +20.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetOpen-Jev-27B-v1.1 Open · 27B Wrong of the 1,582 answers: 504 (31.9%), +21.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 768, 44.2% wrong of 2,184Clef Open · 27B Wrong of the 1,582 answers: 512 (32.4%), +21.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 788, 42.0% wrong of 2,127d1 Closed · Efficient Wrong of the 1,582 answers: 516 (32.6%), +22.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetJeffan-9B-1211 Open · 9.0B Wrong of the 1,582 answers: 519 (32.8%), +22.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1199, 42.5% wrong of 2,157Drex 2.0 Open · 35B MoE · 3.0B active Wrong of the 1,582 answers: 528 (33.4%), +22.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetCygnet Open · 12B Wrong of the 1,582 answers: 530 (33.5%), +22.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 757, 44.2% wrong of 2,166Winnow-12B Open · 12B Wrong of the 1,582 answers: 540 (34.1%), +23.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 750, 44.8% wrong of 2,155KnowLine-9B-Gen1 Open · 9.0B Wrong of the 1,582 answers: 553 (35.0%), +24.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1181, 45.4% wrong of 2,193Cerno 9B v1.2 Open · 9.0B Wrong of the 1,582 answers: 559 (35.3%), +24.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1204, 45.4% wrong of 2,196Adjudex 9B v5 Open · 9.0B Wrong of the 1,582 answers: 578 (36.5%), +26.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1190, 46.6% wrong of 2,190KnowLine-9B-Gen2 Open · 9.0B Wrong of the 1,582 answers: 580 (36.7%), +26.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1325, 47.4% wrong of 2,171Elephant-1 Open · 9.0B Wrong of the 1,582 answers: 586 (37.0%), +26.5 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 382, 42.3% wrong of 1,793 and “I don't know” run 515, 39.0% wrong of 1,594GLiDE 9B Open · 7.9B Wrong of the 1,582 answers: 600 (37.9%), +27.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 913, 45.4% wrong of 1,921StartLux-Decision-v1.1-9B-Instant Open · 9.0B Wrong of the 1,582 answers: 620 (39.2%), +28.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1297, 49.2% wrong of 2,174Dexy-Decision-9B-V2 Open · 9.0B Wrong of the 1,582 answers: 623 (39.4%), +28.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1262, 48.4% wrong of 2,161vera-medium-omni Open · 4.2B Wrong of the 1,582 answers: 665 (42.0%), +31.5 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1271, 51.4% wrong of 2,093Perplexity Decider Flash v1.3 Open · 4.0B Wrong of the 1,582 answers: 667 (42.2%), +31.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1130, 50.4% wrong of 2,123StartLux-Decision-v1.1-4B-Instant Open · 4.0B Wrong of the 1,582 answers: 674 (42.6%), +32.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1239, 52.7% wrong of 2,177KnowLine 4B Gen5.5 Open · 4.0B Wrong of the 1,582 answers: 681 (43.0%), +32.5 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 967, 52.3% wrong of 2,143Perplexity Decider Flash v1.2 Open · 4.0B Wrong of the 1,582 answers: 691 (43.7%), +33.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 791, 51.7% wrong of 2,018Valen-Decision-1.5-9B Open · 9.0B Wrong of the 1,582 answers: 698 (44.1%), +33.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1282, 52.7% wrong of 2,197Verdin-4B-v1.1 Open · 4.0B Wrong of the 1,582 answers: 708 (44.8%), +34.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1251, 53.3% wrong of 2,193Clef-Flash Open · 9.0B Wrong of the 1,582 answers: 716 (45.3%), +34.7 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 664, 53.1% wrong of 2,088Chinese Decision Pointer 4B Open · 4.0B Wrong of the 1,582 answers: 723 (45.7%), +35.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1213, 52.5% wrong of 2,062Microsoft-Decision-1 Closed · Efficient Wrong of the 1,582 answers: 726 (45.9%), +35.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetMetask-Jev-4B Open · 4.5B Wrong of the 1,582 answers: 841 (53.2%), +42.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 743, 60.1% wrong of 2,188KnowLine-2B-Gen1 Open · 2.0B Wrong of the 1,582 answers: 920 (58.2%), +47.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGLiDE 2B Open · 1.9B Wrong of the 1,582 answers: 965 (61.0%), +50.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 921, 61.7% wrong of 1,586StartLux-Decision-v1.1-2B-Instant Open · 2.0B Wrong of the 1,582 answers: 972 (61.4%), +50.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 979, 65.0% wrong of 1,865d1-2B Open · 1.9B Wrong of the 1,582 answers: 1,060 (67.0%), +56.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 712, 64.4% wrong of 989OneJev1.6-0.8B Open · 0.8B Wrong of the 1,582 answers: 1,072 (67.8%), +57.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1031, 68.2% wrong of 1,444GLiDE 0.8B Open · 0.8B Wrong of the 1,582 answers: 1,080 (68.3%), +57.7 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 929, 67.7% wrong of 1,718Strands Decider 2B Open · 2.0B Wrong of the 1,582 answers: 1,096 (69.3%), +58.7 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 706, 72.4% wrong of 2,048StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B Wrong of the 1,582 answers: 1,098 (69.4%), +58.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1017, 70.5% wrong of 1,999KnowLine-0.8B-Gen1 Open · 0.8B Wrong of the 1,582 answers: 1,107 (70.0%), +59.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1007, 72.3% wrong of 2,094Decider 1 Closed · Efficient Wrong of the 1,582 answers: 1,120 (70.8%), +60.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGLiClass v4 Edge Open · 0.4B Wrong of the 1,582 answers: 1,197 (75.7%), +65.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 1048, 77.0% wrong of 1,800GLiDE 0.4B Open · 0.4B Wrong of the 1,582 answers: 1,211 (76.5%), +66.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 937, 78.9% wrong of 128GLiNER2.5-multi-Decide Open · 0.3B Wrong of the 1,582 answers: 1,248 (78.9%), +68.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 718, 79.9% wrong of 2,084Laya Open · 0.4B Wrong of the 1,582 answers: 1,273 (80.5%), +69.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 730, 81.5% wrong of 1,662Laya Multilingual Open · 0.3B Wrong of the 1,582 answers: 1,321 (83.5%), +72.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: “Bilmiyorum” run 737, 83.8% wrong of 1,918
Other models: chat and general models
Gemini 4 Carbon high · chat Closed · Flagship Wrong of the 1,582 answers: 4 (0.3%), -10.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6 Astra low · chat Closed · Flagship Wrong of the 1,582 answers: 4 (0.3%), -10.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetClaude Fable 5.2 max · chat Closed · Flagship Wrong of the 1,582 answers: 5 (0.3%), -10.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGemini 4 Hydrogen high · chat Closed · Workhorse Wrong of the 1,582 answers: 5 (0.3%), -10.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6 Astra high · chat Closed · Flagship Wrong of the 1,582 answers: 6 (0.4%), -10.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6 Astra medium · chat Closed · Flagship Wrong of the 1,582 answers: 6 (0.4%), -10.2 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGemini 3.8 Flash high · chat Closed · Workhorse Wrong of the 1,582 answers: 7 (0.4%), -10.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6.1 Sol high · chat Closed · Workhorse Wrong of the 1,582 answers: 8 (0.5%), -10.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6 Astra low · chat · 1 per request Closed · Flagship Wrong of the 1,582 answers: 11 (0.7%), -9.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6.1 Sol low · chat · 1 per request Closed · Workhorse Wrong of the 1,582 answers: 12 (0.8%), -9.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGemini 3.8 Flash low · chat · 1 per request Closed · Workhorse Wrong of the 1,582 answers: 13 (0.8%), -9.7 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6.1 Sol low · chat Closed · Workhorse Wrong of the 1,582 answers: 13 (0.8%), -9.7 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGemini 4 Argon low · chat Closed · Flagship Wrong of the 1,582 answers: 17 (1.1%), -9.5 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetClaude Fable 5.1 high · chat Closed · Flagship Wrong of the 1,582 answers: 19 (1.2%), -9.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetClaude Opus 5.5 high · chat Closed · Flagship Wrong of the 1,582 answers: 23 (1.5%), -9.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetClaude Opus 5.5 low · chat · 1 per request Closed · Flagship Wrong of the 1,582 answers: 23 (1.5%), -9.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6.1 Terra high · chat Closed · Workhorse Wrong of the 1,582 answers: 31 (2.0%), -8.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGrok 4.7 high · chat Closed · Workhorse Wrong of the 1,582 answers: 31 (2.0%), -8.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetQwen4-Max high · chat Closed · Workhorse Wrong of the 1,582 answers: 31 (2.0%), -8.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGemini 3.8 Flash low · chat Closed · Workhorse Wrong of the 1,582 answers: 50 (3.2%), -7.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6.1 Terra low · chat Closed · Workhorse Wrong of the 1,582 answers: 50 (3.2%), -7.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetClaude Opus 5.5 low · chat Closed · Flagship Wrong of the 1,582 answers: 51 (3.2%), -7.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetClaude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse Wrong of the 1,582 answers: 65 (4.1%), -6.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetMuse Ember max · chat Closed · Efficient Wrong of the 1,582 answers: 86 (5.4%), -5.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetDeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active Wrong of the 1,582 answers: 87 (5.5%), -5.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetClaude Sonnet 5.5 high · chat Closed · Workhorse Wrong of the 1,582 answers: 113 (7.1%), -3.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetDeepSeek-V4.1-Flash reasoning on · chat · first release Open · 552B MoE · 16B active Wrong of the 1,582 answers: 113 (7.1%), -3.4 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetKimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active Wrong of the 1,582 answers: 141 (8.9%), -1.6 points against d1-ultra, Holm p = 0.41
Runs with the option: No run with the option yetQwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active Wrong of the 1,582 answers: 158 (10.0%), -0.6 points against d1-ultra, Holm p = 1
Runs with the option: No run with the option yetGLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active Wrong of the 1,582 answers: 163 (10.3%), -0.3 points against d1-ultra, Holm p = 1
Runs with the option: No run with the option yetInkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Wrong of the 1,582 answers: 181 (11.4%), +0.9 points against d1-ultra, Holm p = 1
Runs with the option: No run with the option yetGemma 4 31B reasoning off · chat Open · 31B Wrong of the 1,582 answers: 239 (15.1%), +4.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGLM-5.3 low · chat Open · Workhorse Wrong of the 1,582 answers: 260 (16.4%), +5.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetKimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active Wrong of the 1,582 answers: 294 (18.6%), +8.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetMiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active Wrong of the 1,582 answers: 299 (18.9%), +8.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetQwen3.8-27B low · chat Open · 27B Wrong of the 1,582 answers: 310 (19.6%), +9.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetDeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active Wrong of the 1,582 answers: 345 (21.8%), +11.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetDeepSeek-V4.1-Flash reasoning off · chat · first release Open · 552B MoE · 16B active Wrong of the 1,582 answers: 346 (21.9%), +11.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGemma 4 31B reasoning off · chat · 1 per request Open · 31B Wrong of the 1,582 answers: 346 (21.9%), +11.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGPT-6 Luna low · chat Closed · Efficient Wrong of the 1,582 answers: 388 (24.5%), +14.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetMiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active Wrong of the 1,582 answers: 406 (25.7%), +15.1 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetClaude Sonnet 5.5 low · chat Closed · Workhorse Wrong of the 1,582 answers: 440 (27.8%), +17.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetQwen3.8 Flash no reasoning · chat Closed · Efficient Wrong of the 1,582 answers: 462 (29.2%), +18.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active Wrong of the 1,582 answers: 515 (32.6%), +22.0 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetQwen3.8-27B vote share · reasoning off · chat Open · 27B Wrong of the 1,582 answers: 559 (35.3%), +24.8 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetQwen3.8-27B yes/no scoring Open · 27B Wrong of the 1,582 answers: 599 (37.9%), +27.3 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetQwen3.8-27B stated confidence · reasoning off · chat Open · 27B Wrong of the 1,582 answers: 635 (40.1%), +29.6 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetErk-14B reasoning off · chat Open · 14B Wrong of the 1,582 answers: 735 (46.5%), +35.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetQwen3-14B reasoning off · chat Open · 15B Wrong of the 1,582 answers: 745 (47.1%), +36.5 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetGLiNER2.5 Multi Open · 0.3B Wrong of the 1,582 answers: 1,321 (83.5%), +72.9 points against d1-ultra, Holm p = 0.025 *
Runs with the option: No run with the option yetClaude Haiku 5.5 max · chat Closed · Efficient Wrong of the 1,582 answers: — (left out: its run has 1,414 scored answers, fewer than k), not tested
Runs with the option: No run with the option yetGemma 4 E4B default reasoning · chat Open · 8.0B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
Runs with the option: No run with the option yetLFM2.5-2.6B default reasoning · chat Open · 2.7B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
Runs with the option: No run with the option yetLFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
Runs with the option: No run with the option yetLFM3-1.7B reasoning off · chat Open · 1.7B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
Runs with the option: No run with the option yetLFM3-1.7B reasoning on · chat Open · 1.7B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
Runs with the option: No run with the option yetMiniCPM5-2B reasoning off · chat Open · 2.5B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
Runs with the option: No run with the option yetQwen3.5-2B reasoning off · chat Open · 2.0B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
Runs with the option: No run with the option yetQwen3.5-4B reasoning off · chat Open · 4.0B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
Runs with the option: No run with the option yetQwen3.5-4B reasoning on · chat Open · 4.0B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
Runs with the option: No run with the option yet
Our method for the same number of answers
Each model's five-option run answers only its k most confident questions, by its score (the probability it gave the option it picked), and holds back on the rest; we count its wrong answers among the k. Wrong of the k answers = wrong / k. k is the number of questions d1-ultra answered (right + wrong) with the option on offer: 1,582 in its run 511 with “I don't know” on the 2,198 Turkish questions (the option's word, above), and 496 in its run 449, its first English run with “I don't know”, on the 547 kept reading questions. Every model in a language is held to the same k.
The answers are ranked by score, highest first, and equal scores by question ID, lowest first (never by whether the answer is right); the first k are kept, so the wrong count is always a whole number. An answer without a score or a valid choice is never kept.
The difference is a model's wrong of the k answers minus d1-ultra's, in points. Its uncertainty comes from a paired bootstrap over questions: 10,000 resamples of the questions with replacement, from a new random generator with the numeric seed 20261103 for each language, the same resample for every model, a question drawn twice counting twice and k fixed. In each resample every run's drawn answers are ranked again by score and question ID, the copies of a drawn question sitting together, and the first k kept, so the count stays whole. Two-sided p = min(1, 2 × min(1 + #{d ≤ 0}, 1 + #{d ≥ 0}) / 10,001), where #{d ≤ 0} is the number of resamples whose difference d is 0 or less and #{d ≥ 0} the number whose difference is 0 or more. Its floor is 2/10,001 (p = 0.0002), when no resample falls on the other side of zero: a p at the floor says the difference held in every resample, not that p is exactly that value. The 95% interval is the percentile interval: from the 250th smallest to the 250th largest of the 10,000 resampled differences.
Holm's method adjusts p over the other 126 setups in Turkish (every main run of the Text tab, decision models and chat and general models alike) and the other 123 models in English, each language a family of its own, separate from the tab's other comparisons. The resamples do not depend on which models are compared, so a model's p is the same in any family; only Holm's adjustment grows with the family. A run with fewer than k scored answers keeps no k answers: it is left out of the bootstrap and of Holm's adjustment, and its line ends before k. A setup asked for a plain answer line states no confidence and reports no option probabilities, so its answers have no score to rank by: not measured, it is left out of the bootstrap and of Holm's adjustment and has no line. For GLiDE · API, the Turkish five-option run is the Text tab's run 129, made with Fastino's engine glide; its runs with the option, 351 and 510, answered with glide_v2, a different hosted model (Our method for the hosted GLiDE rerun), so its line and its markers come from two models, as the legend and the table say.
The plots show every number of answers, not only k. Each line is a five-option run under the same selection, one point per number of answers from 1 to the run's scored answers: answered / questions across, wrong / answered up. There is no point at 0 answers, where the share wrong is undefined, and a line ends at its run's scored answers, without being extended to 100%. The Turkish plot draws the 63 setups with runs with the option; the English curves come from this comparison too, so they can differ slightly from those of the Turkish vs English tab (Risk and coverage), where answers with equal scores enter together. A line skips vertices closer than 0.25% of coverage to the previous one drawn, always keeping the first, the last and the point at k, so every vertex drawn is a curve point; the y axis fits the markers and every curve point at 1% coverage or more, and anything higher (the first few answers, where one wrong answer is a large share) is drawn at the top edge. The markers are each model's own runs with the option: in Turkish its first run with “Bilmiyorum” (the table above) and its run with “I don't know” (the option's word); in English its first English run with “I don't know”.
Limits: k comes from one model, d1-ultra, and a different k gives different numbers (the plots show them all). The k answers kept are each model's own most confident ones, so the models are counted on different questions. d1-ultra's own runs with the option made 216 wrong of 1,582 in Turkish and 39 of 496 in English, against 167 and 29 for its five-option run held to the same k: the comparison is between five-option runs held back by confidence, not between runs with the option.