Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Provisional: Drex Max v1.1, whose maker publishes only a promotional price. Left out here: the setups only this site ran (Erk-14B, GLiNER2.5 Multi, GLiNER2.5-multi-Decide, Laya, Laya Multilingual and Qwen3-14B); they show under Class: All. On this page a class narrows the rows of the tables and bar charts; the sentences, counts, tests and line plots still cover every model of the type in view.
Turkish vs English
The Text tab's reading questions, in Turkish and in English
Accuracy in Turkish and in English · 547 reading questions, the English machine-translated
Share of the 547 kept reading questions answered right: each of the 133 setups' Text tab run and its first English run, ordered by English accuracy · higher is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Claude Gemini GPT Qwen KnowLine Grok Copernicus DeepSeek REAX Kimi Karpo Logits Fastino Liquid Sieve Turion Jev Gemma Perplexity Mammoth vLLM-SR Drex GLM Muse MiMo Inkling Jeffan StartLux Clef OneJev Jev Cerno SubQ Adjudex Inception Microsoft Elephant Open-Jev MiniMax Celeris Cygnet Winnow Valen Dexy vera Verdin Chinese Decision Pointer Metask Erk MiniCPM GLiClass meraGPT Strands Laya
Right only in Turkish, or only in English?
Each setup's first English run against its Text tab run on the 547 kept questions: the questions only one of the two got right, by the exact McNemar test with p adjusted by Holm's method over all 133 English setups · the longer bar is the language with more right answers
Right only in Turkish, the upper bar Right only in English, the lower bar, striped Numbers on the right: Turkish · English. MiniCPM Liquid Fastino GLiClass Qwen StartLux meraGPT KnowLine OneJev Microsoft Metask Gemini Strands Kimi Perplexity SubQ GPT Claude Clef Verdin Valen Turion Jev Chinese Decision Pointer Sieve vera Karpo DeepSeek Celeris Mammoth Grok GLM Logits Copernicus Erk Gemma MiMo Cerno vLLM-SR Drex REAX Elephant Adjudex MiniMax Cygnet Laya Muse Jev Open-Jev Inception Inkling Jeffan Winnow Dexy * A significant difference (Holm-adjusted p below 0.05): MiniCPM5-2B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat, GLiDE 0.4B, GLiClass v4 Edge, LFM3-1.7B · reasoning off · chat, Qwen3.5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, d1-2B, LFM3-1.7B · reasoning on · chat, StartLux-Decision-v1.1-0.8B-Instant, Decider 1, KnowLine-0.8B-Gen1, OneJev1.6-0.8B, GLiDE 2B, KnowLine-2B-Gen1, StartLux-Decision-v1.1-2B-Instant, GLiDE 0.8B, Microsoft-Decision-1, Metask-Jev-4B, Qwen3.5-4B · reasoning off · chat and Gemini 4 Carbon · high · chat (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p = 0.00011, p = 0.018 and p = 0.031). The other 112: p = 0.14, p = 0.28, p = 0.46, p = 0.73, p = 0.76, p = 0.76, p = 0.83, p = 0.83, p = 0.85, p = 0.91, p = 0.97, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1.
Accuracy in Turkish and in English · 547 reading questions, the English machine-translated
Share of the 547 kept reading questions answered right: each of the 58 chat model setups' Text tab run and its first English run, ordered by English accuracy · higher is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Claude Gemini GPT Qwen Grok DeepSeek Kimi Gemma GLM Muse MiMo Inkling MiniMax Liquid Erk MiniCPM Fastino
Right only in Turkish, or only in English?
Each setup's first English run against its Text tab run on the 547 kept questions: the questions only one of the two got right, by the exact McNemar test with p adjusted by Holm's method over all 133 English setups · the longer bar is the language with more right answers
Right only in Turkish, the upper bar Right only in English, the lower bar, striped Numbers on the right: Turkish · English. MiniCPM Liquid Qwen Gemini Kimi GPT Claude DeepSeek Grok GLM Erk Gemma MiMo MiniMax Muse Inkling Fastino * A significant difference (Holm-adjusted p below 0.05): MiniCPM5-2B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat, LFM3-1.7B · reasoning off · chat, Qwen3.5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning on · chat, Qwen3.5-4B · reasoning off · chat and Gemini 4 Carbon · high · chat (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p = 0.018 and p = 0.031). The other 50: p = 0.28, p = 0.83, p = 0.83, p = 0.85, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1.
Accuracy in Turkish and in English · 547 reading questions, the English machine-translated
Share of the 547 kept reading questions answered right: each of the 75 decision models' Text tab run and its first English run, ordered by English accuracy · higher is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. KnowLine Copernicus REAX Karpo Gemini Logits Fastino Liquid Sieve Turion Jev Perplexity Mammoth vLLM-SR Drex Jeffan StartLux Clef OneJev Jev Cerno SubQ Adjudex Inception Microsoft Elephant Open-Jev Celeris GPT Cygnet Winnow Valen Dexy vera Verdin Chinese Decision Pointer Metask GLiClass meraGPT Strands Laya
Right only in Turkish, or only in English?
Each setup's first English run against its Text tab run on the 547 kept questions: the questions only one of the two got right, by the exact McNemar test with p adjusted by Holm's method over all 133 English setups · the longer bar is the language with more right answers
Right only in Turkish, the upper bar Right only in English, the lower bar, striped Numbers on the right: Turkish · English. Fastino GLiClass Liquid StartLux meraGPT KnowLine OneJev Microsoft Metask Strands Perplexity SubQ Clef Verdin Valen Turion Jev Chinese Decision Pointer Sieve vera Karpo Celeris Mammoth Logits Gemini Copernicus GPT Cerno vLLM-SR Drex REAX Elephant Adjudex Cygnet Laya Jev Open-Jev Inception Jeffan Winnow Dexy * A significant difference (Holm-adjusted p below 0.05): GLiDE 0.4B, GLiClass v4 Edge, d1-2B, StartLux-Decision-v1.1-0.8B-Instant, Decider 1, KnowLine-0.8B-Gen1, OneJev1.6-0.8B, GLiDE 2B, KnowLine-2B-Gen1, StartLux-Decision-v1.1-2B-Instant, GLiDE 0.8B, Microsoft-Decision-1 and Metask-Jev-4B (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001 and p = 0.00011). The other 62: p = 0.14, p = 0.46, p = 0.73, p = 0.76, p = 0.76, p = 0.91, p = 0.97, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1.
KnowLine-31B-Gen1.5 against each decision model in English
In English, against KnowLine-31B-Gen1.5, 11 are too close to call and 63 are behind it (of the 74 other decision models, after Holm's correction).
| Model | Runfirst English run | Only KnowLine-31B-Gen1.5 rightthe other not right | Only the other rightKnowLine-31B-Gen1.5 not right | pexact McNemar | Holm pover the 74 | Verdictby Holm p, 0.05 |
|---|---|---|---|---|---|---|
| Copernicus-31B Open · 31B | run 1168 | 7 | 3 | 0.34 | 0.57 | too close to call |
| REAX 1.3 Open · 57B | run 829 | 14 | 8 | 0.29 | 0.57 | too close to call |
| Karpo Decision 31B Open · 31B | run 1159 | 16 | 8 | 0.15 | 0.45 | too close to call |
| KnowLine-31B-Gen2 Open · 31B | run 1327 | 8 | 1 | 0.039 | 0.17 | too close to call |
| Gemini Decision 1 20261111 Closed · Efficient | run 1009 | 24 | 10 | 0.024 | 0.17 | too close to call |
| Logits Sigma-1 Duo Open · 57B | run 1229 | 24 | 10 | 0.024 | 0.17 | too close to call |
| GLiDE 27B Open · 27B | run 901 | 27 | 12 | 0.024 | 0.17 | too close to call |
| d1-ultra Closed · Efficient | run 411 | 26 | 10 | 0.011 | 0.091 | too close to call |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | run 639 | 28 | 11 | 0.0095 | 0.085 | too close to call |
| GLiDE v1.2 Open · 27B | run 521 | 30 | 12 | 0.0079 | 0.079 | too close to call |
| d1.2-ultra Closed · Efficient | run 1359 | 28 | 10 | 0.0051 | 0.056 | too close to call |
| Mammoth-1 Open · 27B | run 503 | 28 | 9 | 0.0026 | 0.032 | behind KnowLine-31B-Gen1.5 |
| Sieve 31B v3 Open · 31B | run 1064 | 21 | 5 | 0.0025 | 0.032 | behind KnowLine-31B-Gen1.5 |
| Turion Jev 31B Open · 31B | run 1075 | 20 | 4 | 0.0015 | 0.022 | behind KnowLine-31B-Gen1.5 |
| d3-ultra Open · 26B | run 550 | 34 | 11 | 0.00082 | 0.012 | behind KnowLine-31B-Gen1.5 |
| d3 Open · 26B | run 554 | 29 | 8 | 0.00075 | 0.012 | behind KnowLine-31B-Gen1.5 |
| Drex Max Open · 27B | run 552 | 31 | 9 | 0.00068 | 0.012 | behind KnowLine-31B-Gen1.5 |
| Perplexity Decider 27B v1.3 Open · 27B | run 1137 | 30 | 8 | 0.00047 | 0.0085 | behind KnowLine-31B-Gen1.5 |
| d1.2 Closed · Efficient | run 1357 | 35 | 10 | 0.00025 | 0.0047 | behind KnowLine-31B-Gen1.5 |
| Drex Max v1.1 Closed · Efficient (provisional) | run 399 | 38 | 10 | < 0.0001 | 0.0012 | behind KnowLine-31B-Gen1.5 |
| Jeffan-9B-1211 Open · 9.0B | run 1195 | 47 | 14 | < 0.0001 | 0.00057 | behind KnowLine-31B-Gen1.5 |
| d1.1 Closed · Efficient | run 565 | 38 | 9 | < 0.0001 | 0.00055 | behind KnowLine-31B-Gen1.5 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | run 527 | 33 | 6 | < 0.0001 | 0.00033 | behind KnowLine-31B-Gen1.5 |
| Perplexity Decider 27B v1 Open · 27B | run 626 | 34 | 6 | < 0.0001 | 0.00021 | behind KnowLine-31B-Gen1.5 |
| Perplexity Decider 27B v1.2 API Open · 27B | run 637 | 34 | 6 | < 0.0001 | 0.00021 | behind KnowLine-31B-Gen1.5 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | run 1293 | 49 | 12 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Cerno 9B v1.2 Open · 9.0B | run 1085 | 58 | 16 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| GLiDE 9B Open · 7.9B | run 909 | 52 | 12 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| KnowLine-12B-Gen1 Open · 12B | run 1313 | 48 | 10 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| d1 Closed · Efficient | run 430 | 45 | 8 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| KnowLine-9B-Gen1 Open · 9.0B | run 1177 | 54 | 12 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Adjudex 9B v5 Open · 9.0B | run 1186 | 56 | 13 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Clef Open · 27B | run 613 | 45 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| OneJev-27B Open · 27B | run 778 | 47 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| KnowLine-9B-Gen2 Open · 9.0B | run 1321 | 58 | 12 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Microsoft-Decision-1 Closed · Efficient | run 412 | 52 | 9 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Drex 2.0 Open · 35B MoE · 3.0B active | run 566 | 50 | 8 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Jev 1.14 Closed · Efficient | run 410 | 48 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| SubQ Snap 1.1 Closed · Efficient | run 413 | 49 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| SubQ Snap Preview Closed · Efficient | run 595 | 49 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Elephant-1 Open · 9.0B | run 499 | 54 | 9 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Mercury Decide Closed · Efficient | run 398 | 50 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Jev 1.13 Closed · Efficient | run 564 | 48 | 6 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Open-Jev-27B-v1.1 Open · 27B | run 760 | 52 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Perplexity Decider Flash v1.3 Open · 4.0B | run 1126 | 67 | 13 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Winnow-12B Open · 12B | run 746 | 65 | 12 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Valen-Decision-1.5-9B Open · 9.0B | run 1278 | 66 | 12 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Cygnet Open · 12B | run 753 | 64 | 11 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| celeris-1-decision Closed · Efficient | run 546 | 62 | 10 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Dexy-Decision-9B-V2 Open · 9.0B | run 1258 | 67 | 12 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | run 1081 | 74 | 15 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| GPT-6 Luna Decisions API Closed · Efficient | run 397 | 61 | 9 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| vera-medium-omni Open · 4.2B | run 1267 | 66 | 11 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Perplexity Decider Flash v1.2 Open · 4.0B | run 533 | 76 | 12 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Verdin-4B-v1.1 Open · 4.0B | run 1247 | 71 | 9 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Clef-Flash Open · 9.0B | run 614 | 69 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| KnowLine 4B Gen5.5 Open · 4.0B | run 961 | 81 | 11 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Chinese Decision Pointer 4B Open · 4.0B | run 1209 | 83 | 11 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Metask-Jev-4B Open · 4.5B | run 739 | 97 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| KnowLine-2B-Gen1 Open · 2.0B | run 1057 | 132 | 10 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| GLiDE 2B Open · 1.9B | run 917 | 138 | 9 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | run 973 | 140 | 9 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| d1-2B Open · 1.9B | run 708 | 159 | 6 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| GLiDE 0.8B Open · 0.8B | run 925 | 186 | 6 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | run 1011 | 194 | 8 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| GLiDE 0.4B Open · 0.4B | run 933 | 195 | 8 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| GLiClass v4 Edge Open · 0.4B | run 1042 | 202 | 9 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| KnowLine-0.8B-Gen1 Open · 0.8B | run 1001 | 197 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| OneJev1.6-0.8B Open · 0.8B | run 1025 | 191 | 4 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Decider 1 Closed · Efficient | run 596 | 216 | 9 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Strands Decider 2B Open · 2.0B | run 702 | 270 | 7 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Laya Open · 0.4B | run 726 | 398 | 11 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| GLiNER2.5-multi-Decide Open · 0.3B | run 714 | 402 | 6 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
| Laya Multilingual Open · 0.3B | run 732 | 432 | 4 | < 0.0001 | < 0.0001 | behind KnowLine-31B-Gen1.5 |
Ordered by p, highest first. On the 547 kept reading questions, KnowLine-31B-Gen1.5's first English run (run 953) against each other decision model's first English run: the exact McNemar test on the questions only one of the two got right, with p adjusted by Holm's method over the 74 decision models. Verdict: too close to call when the Holm-adjusted p is 0.05 or more; otherwise behind KnowLine-31B-Gen1.5 when KnowLine-31B-Gen1.5 has more questions only it got right, ahead of it when the other model has more. Each model's name links its model page's English section, as in the tables above; Paired test compares each model with itself in Turkish.
d1-ultra against each decision model in English is for decision models only.
Limit: a model translated the questions and no person checked them (Our method).
Turkish vs English
| Model | Accuracy | Answered with “I don't know” | Wrong of answers with “I don't know” | Non-hallucination with “I don't know” | Turkish with “Bilmiyorum”wrong of answers, lower is better; answered and non-hallucination under it | Cost | Latency | Rank by accuracy | Languages the model card listsand its source | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Turkishhigher is better | Englishhigher is better | Differencepoints higher in English than in Turkish (negative: higher in Turkish) | Turkish(right + wrong) / 547; higher is better | English(right + wrong) / 547; higher is better | Turkishwrong / (right + wrong); lower is better | Englishwrong / (right + wrong); lower is better | Turkish“I don't know” / (“I don't know” + wrong); higher is better | English“I don't know” / (“I don't know” + wrong); higher is better | Differencenon-hallucination points higher in English than in Turkish (negative: higher in Turkish) | Turkish$ per 1,000 questionsper 1M questions; lower is better | English$ per 1,000 questionsper 1M questions; lower is better | Turkishmedian seconds per request; lower is better | Englishmedian seconds per request; lower is better | Turkishamong the 133; lower is better | Englishamong the 133; lower is better | |||
| Claude Fable 5.2 max · chat Closed · Flagship | 97.8% | 98.0% | +0.2 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 1.8%10 of 543 | Not rundecision models only | 28.6%4 “I don't know” of 14 not right | Not rundecision models only | Queuedits run with the option has not started yet | 53.38153,381subscription, priced as the API | 52.94252,942subscription, priced as the API | 81.116 questions per request, 1 in flight | 74.016 questions per request, 1 in flight | 1 | 1 | other languages, not Turkishplatform.claude.com |
| Gemini 4 Carbon high · chat Closed · Flagship | 97.6% | 94.3% | -3.3 * | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 2.2%12 of 543 | Not rundecision models only | 25.0%4 “I don't know” of 16 not right | Not rundecision models only | 2.0%11 of 543answered 99.3%, non-hallucination 26.7% | 31.30931,309subscription, priced as the API | 38.05338,053subscription, priced as the API | 217.716 questions per request, 1 in flight | 287.816 questions per request, 1 in flight | 2 | 19 | not stateddeepmind.google |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 97.1% | 96.7% | -0.4 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 2.9%16 of 543 | Not rundecision models only | 20.0%4 “I don't know” of 20 not right | Not rundecision models only | 1.8%10 of 543answered 99.3%, non-hallucination 28.6% | 4.8064,806subscription, priced as the API | 4.9274,927subscription, priced as the API | 31.716 questions per request, 1 in flight | 28.416 questions per request, 1 in flight | 3 | 2 | not stateddeepmind.google |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 97.1% | 95.6% | -1.5 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 4.2%23 of 543 | Not rundecision models only | 14.8%4 “I don't know” of 27 not right | Not rundecision models only | 3.5%19 of 543answered 99.3%, non-hallucination 17.4% | 26.73826,738subscription, priced as the API | 28.65328,653subscription, priced as the API | 6.411 question per request, 1 in flight | 6.431 question per request, 1 in flight | 4 | 12 | not stateddevelopers.openai.com |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 96.9% | 96.0% | -0.9 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 3.3%18 of 545 | Not rundecision models only | 10.0%2 “I don't know” of 20 not right | Not rundecision models only | 2.4%13 of 545answered 99.6%, non-hallucination 13.3% | 2.4192,419at list prices | 3.6953,695at list prices | 53.716 questions per request, 4 in flight | 104.616 questions per request, 4 in flight | 5 | 7 | not stateddeepmind.google |
| Gemini 4 Argon low · chat Closed · Flagship | 96.9% | 96.0% | -0.9 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 4.1%22 of 542 | Not rundecision models only | 18.5%5 “I don't know” of 27 not right | Not rundecision models only | 2.4%13 of 543answered 99.3%, non-hallucination 23.5% | 1.7861,786at list prices | 1.9001,900at list prices | 5.4616 questions per request, 1 in flight | 6.0316 questions per request, 1 in flight | 6 | 8 | not stateddeepmind.google |
| GPT-6 Astra high · chat Closed · Flagship | 96.9% | 96.2% | -0.7 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 3.1%17 of 543 | Not rundecision models only | 19.0%4 “I don't know” of 21 not right | Not rundecision models only | 3.1%17 of 543answered 99.3%, non-hallucination 19.0% | 6.0666,066subscription, priced as the API | 6.3346,334subscription, priced as the API | 21.716 questions per request, 1 in flight | 15.616 questions per request, 1 in flight | 7 | 4 | not stateddevelopers.openai.com |
| GPT-6 Astra low · chat Closed · Flagship | 96.7% | 96.3% | -0.4 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 3.3%18 of 543 | Not rundecision models only | 18.2%4 “I don't know” of 22 not right | Not rundecision models only | 2.6%14 of 543answered 99.3%, non-hallucination 22.2% | 5.3785,378subscription, priced as the API | 5.9185,918subscription, priced as the API | 13.016 questions per request, 1 in flight, measured on a sample (200 questions) | 11.516 questions per request, 1 in flight | 8 | 3 | not stateddevelopers.openai.com |
| GPT-6 Astra medium · chat Closed · Flagship | 96.7% | 96.0% | -0.7 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 3.7%20 of 543 | Not rundecision models only | 16.7%4 “I don't know” of 24 not right | Not rundecision models only | 2.9%16 of 543answered 99.3%, non-hallucination 20.0% | 5.4475,447subscription, priced as the API | 4.6964,696subscription, priced as the API | 15.316 questions per request, 1 in flight | 10.716 questions per request, 1 in flight | 9 | 9 | not stateddevelopers.openai.com |
| GPT-6.1 Terra high · chat Closed · Workhorse | 96.3% | 96.2% | -0.2 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 3.7%20 of 543 | Not rundecision models only | 16.7%4 “I don't know” of 24 not right | Not rundecision models only | 3.5%19 of 542answered 99.1%, non-hallucination 20.8% | 0.515515subscription, priced as the API | 0.509509subscription, priced as the API | 7.1016 questions per request, 1 in flight | 7.9416 questions per request, 1 in flight | 10 | 5 | not stateddevelopers.openai.com |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 96.2% | 95.2% | -0.9 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 4.4%24 of 545 | Not rundecision models only | 7.7%2 “I don't know” of 26 not right | Not rundecision models only | 3.3%18 of 544answered 99.5%, non-hallucination 14.3% | 0.963963at list prices | 0.858858at list prices | 2.721 question per request, 1 in flight | 2.851 question per request, 1 in flight | 11 | 14 | not stateddeepmind.google |
| GPT-6.1 Sol high · chat Closed · Workhorse | 96.2% | 95.6% | -0.5 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 3.7%20 of 543 | Not rundecision models only | 16.7%4 “I don't know” of 24 not right | Not rundecision models only | 2.9%16 of 543answered 99.3%, non-hallucination 20.0% | 1.6221,622subscription, priced as the API | 1.4271,427subscription, priced as the API | 24.716 questions per request, 1 in flight | 16.916 questions per request, 1 in flight | 12 | 13 | not stateddevelopers.openai.com |
| GPT-6.1 Sol low · chat Closed · Workhorse | 95.8% | 95.8% | +0.0 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 4.2%23 of 543 | Not rundecision models only | 14.8%4 “I don't know” of 27 not right | Not rundecision models only | 3.3%18 of 543answered 99.3%, non-hallucination 18.2% | 1.1301,130subscription, priced as the API | 1.2191,219subscription, priced as the API | 13.316 questions per request, 1 in flight | 10.316 questions per request, 1 in flight | 13 | 11 | not stateddevelopers.openai.com |
| Claude Opus 5.5 high · chat Closed · Flagship | 95.6% | 94.5% | -1.1 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 5.3%29 of 545 | Not rundecision models only | 6.5%2 “I don't know” of 31 not right | Not rundecision models only | 4.2%23 of 544answered 99.5%, non-hallucination 11.5% | 6.5096,509subscription, priced as the API | 4.5194,519subscription, priced as the API | 22.116 questions per request, 1 in flight | 11.316 questions per request, 1 in flight | 14 | 17 | other languages, not Turkishplatform.claude.com |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 95.6% | 95.2% | -0.4 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 5.0%27 of 543 | Not rundecision models only | 12.9%4 “I don't know” of 31 not right | Not rundecision models only | 3.5%19 of 543answered 99.3%, non-hallucination 17.4% | 5.5695,569subscription, priced as the API | 6.1136,113subscription, priced as the API | 6.611 question per request, 1 in flight | 5.591 question per request, 1 in flight | 15 | 15 | not stateddevelopers.openai.com |
| GPT-6.1 Terra low · chat Closed · Workhorse | 95.6% | 96.2% | +0.5 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 4.1%22 of 543 | Not rundecision models only | 15.4%4 “I don't know” of 26 not right | Not rundecision models only | 4.2%23 of 543answered 99.3%, non-hallucination 14.8% | 0.426426subscription, priced as the API | 0.491491subscription, priced as the API | 4.2416 questions per request, 1 in flight | 7.2916 questions per request, 1 in flight | 16 | 6 | not stateddevelopers.openai.com |
| Claude Fable 5.1 high · chat Closed · Flagship | 95.4% | 96.0% | +0.5 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 4.6%25 of 545 | Not rundecision models only | 7.4%2 “I don't know” of 27 not right | Not rundecision models only | 4.2%23 of 543answered 99.3%, non-hallucination 14.8% | 17.04217,042subscription, priced as the API | 11.53111,531subscription, priced as the API | 30.316 questions per request, 1 in flight | 15.616 questions per request, 1 in flight | 17 | 10 | not statedplatform.claude.com |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 95.4% | 94.7% | -0.7 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 5.7%31 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 31 not right | Not rundecision models only | 4.6%25 of 546answered 99.8%, non-hallucination 3.8% | 11.61011,610subscription, priced as the API | 10.09410,094subscription, priced as the API | 3.711 question per request, 1 in flight | 3.501 question per request, 1 in flight | 18 | 16 | other languages, not Turkishplatform.claude.com |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 95.2% | 94.3% | -0.9 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 5.1%28 of 546 | Not rundecision models only | 3.4%1 “I don't know” of 29 not right | Not rundecision models only | 4.4%24 of 544answered 99.5%, non-hallucination 11.1% | 0.269269at list prices | 0.288288at list prices | 2.1916 questions per request, 1 in flight | 4.1916 questions per request, 1 in flight | 19 | 20 | not stateddeepmind.google |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 95.2% | 92.3% | -2.9 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 7.0%38 of 544 | Not rundecision models only | 7.3%3 “I don't know” of 41 not right | Not rundecision models only | 6.6%36 of 544answered 99.5%, non-hallucination 7.7% | 2.0482,048at list prices | 0.984984at list prices | 44.916 questions per request, 4 in flight | 16.616 questions per request, 4 in flight | 20 | 28 | not statedhuggingface.co |
| Claude Opus 5.5 low · chat Closed · Flagship | 95.1% | 93.1% | -2.0 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 7.5%41 of 545 | Not rundecision models only | 4.7%2 “I don't know” of 43 not right | Not rundecision models only | 5.3%29 of 545answered 99.6%, non-hallucination 6.5% | 4.9314,931subscription, priced as the API | 1.0871,087subscription, priced as the API | 13.416 questions per request, 1 in flight, measured on a sample (200 questions) | 7.6416 questions per request, 1 in flight | 21 | 23 | other languages, not Turkishplatform.claude.com |
| Grok 4.7 high · chat Closed · Workhorse | 95.1% | 93.4% | -1.6 | Not rundecision models only | 97.8%535 of 547 answered | Not rundecision models only | 5.8%31 of 535 | Not rundecision models only | 27.9%12 “I don't know” of 43 not right | Not rundecision models only | 3.9%21 of 539answered 98.5%, non-hallucination 27.6% | 14.29214,292subscription, priced as the API | 6.8516,851subscription, priced as the API | 401.916 questions per request, 4 in flight | 197.216 questions per request, 4 in flight | 22 | 22 | not statedmedia.x.ai |
| Karpo Decision 31B Open · 31B | 94.9% | 92.1% | -2.7 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 7.7%42 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 42 not right | Not rundecision models only | 5.5%30 of 546answered 99.8%, non-hallucination 3.2% | 0.105105from GPU time | 0.145145from GPU time | 0.1591 question per request, 1 in flight | 0.2071 question per request, 1 in flight | 23 | 30 | not statedhuggingface.co |
| Qwen4-Max high · chat Closed · Workhorse | 94.7% | 94.5% | -0.2 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 5.3%29 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 29 not right | Not rundecision models only | 5.1%28 of 546answered 99.8%, non-hallucination 3.4% | 2.5332,533at list prices | 1.1191,119at list prices | 91.516 questions per request, 4 in flight | 29.516 questions per request, 4 in flight | 24 | 18 | not statedopenrouter.ai |
| Copernicus-31B Open · 31B | 94.5% | 92.9% | -1.6 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 7.1%39 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 39 not right | Not rundecision models only | 6.0%33 of 546answered 99.8%, non-hallucination 2.9% | 0.134134from GPU time | 0.142142from GPU time | 0.1841 question per request, 1 in flight | 0.1881 question per request, 1 in flight | 25 | 24 | English (the card's language: en tag; its text names no language)huggingface.co |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 94.5% | 92.5% | -2.0 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 7.2%39 of 544 | Not rundecision models only | 7.1%3 “I don't know” of 42 not right | Not rundecision models only | 7.0%38 of 546answered 99.8%, non-hallucination 2.6% | 1.2781,278at the maker's list price | 0.421421at the maker's list price | 486.916 questions per request, 8 in flight | 189.416 questions per request, 8 in flight | 26 | 26 | not statedhuggingface.co |
| KnowLine-31B-Gen2 Open · 31B | 94.5% | 92.3% | -2.2 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 7.3%40 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 40 not right | Not rundecision models only | 5.3%29 of 547answered 100.0%, non-hallucination 0.0% | 0.120120from GPU time | 0.127127from GPU time | 0.1681 question per request, 1 in flight | 0.1711 question per request, 1 in flight | 27 | 29 | other languages, not Turkishhuggingface.co |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 94.3% | 91.6% | -2.7 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 6.9%38 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 38 not right | Not rundecision models only | 6.6%36 of 546answered 99.8%, non-hallucination 2.7% | 7.7487,748subscription, priced as the API | 5.5135,513subscription, priced as the API | 4.641 question per request, 1 in flight | 3.021 question per request, 1 in flight | 28 | 31 | other languages, not Turkishplatform.claude.com |
| KnowLine-31B-Gen1.5 Open · 31B | 94.3% | 93.6% | -0.7 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 6.8%37 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 37 not right | Not rundecision models only | 6.2%34 of 547answered 100.0%, non-hallucination 0.0% | No list price | No list price | 1.051 question per request, 1 in flight | 1.121 question per request, 1 in flight | 29 | 21 | other languages, not Turkishhuggingface.co |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 94.0% | 91.4% | -2.6 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 9.0%49 of 543 | Not rundecision models only | 7.5%4 “I don't know” of 53 not right | Not rundecision models only | 7.2%39 of 544answered 99.5%, non-hallucination 7.1% | 0.853853at list prices | 0.735735at list prices | 5.7516 questions per request, 4 in flight | 6.3116 questions per request, 4 in flight | 30 | 32 | not statedhuggingface.co |
| REAX 1.3 Open · 57B | 94.0% | 92.5% | -1.5 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 7.7%42 of 544 | Not rundecision models only | 6.7%3 “I don't know” of 45 not right | Not rundecision models only | 6.4%35 of 546answered 99.8%, non-hallucination 2.8% | 0.185185from GPU time | 0.169169from GPU time | 0.2501 question per request, 1 in flight | 0.2291 question per request, 1 in flight | 31 | 27 | English (the card's language: en tag; its text names no language)huggingface.co |
| d1-ultra Closed · Efficient | 93.6% | 90.7% | -2.9 | 95.1%520 of 547 answered | 90.7%496 of 547 answered | 5.4%28 of 520 | 7.9%39 of 496 | 49.1%27 “I don't know” of 55 not right | 56.7%51 “I don't know” of 90 not right | +7.6 | 5.0%26 of 517answered 94.5%, non-hallucination 53.6% | 0.01414.34at list prices | 0.01413.98at list prices | 0.2231 question per request, 1 in flight | 0.2201 question per request, 1 in flight | 32 | 37 | not stateddocs.liquid.ai |
| Turion Jev 31B Open · 31B | 93.6% | 90.7% | -2.9 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 7.9%43 of 543 | Not rundecision models only | 8.5%4 “I don't know” of 47 not right | Not rundecision models only | 6.1%33 of 545answered 99.6%, non-hallucination 5.7% | No list price | No list price | 0.4051 question per request, 1 in flight | 0.4451 question per request, 1 in flight | 33 | 38 | not statedhuggingface.co |
| Logits Sigma-1 Duo Open · 57B | 93.4% | 91.0% | -2.4 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 8.1%44 of 545 | Not rundecision models only | 4.3%2 “I don't know” of 46 not right | Not rundecision models only | 6.6%36 of 547answered 100.0%, non-hallucination 0.0% | 0.181181from GPU time | 0.191191from GPU time | 0.2501 question per request, 1 in flight | 0.2631 question per request, 1 in flight | 34 | 34 | not statedhuggingface.co |
| Sieve 31B v3 Open · 31B | 93.4% | 90.7% | -2.7 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 8.2%45 of 546 | Not rundecision models only | 2.2%1 “I don't know” of 46 not right | Not rundecision models only | 6.0%33 of 547answered 100.0%, non-hallucination 0.0% | No list price | No list price | 1.001 question per request, 1 in flight | 1.061 question per request, 1 in flight | 35 | 39 | other languages, not Turkishhuggingface.co |
| Gemini Decision 1 20261111 Closed · Efficient | 92.9% | 91.0% | -1.8 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 9.2%50 of 542 | Not rundecision models only | 9.1%5 “I don't know” of 55 not right | Not rundecision models only | 7.6%41 of 542answered 99.1%, non-hallucination 10.9% | 0.03737.19at list prices | 0.04039.51at list prices | 0.2271 question per request, 1 in flight | 0.2351 question per request, 1 in flight | 36 | 35 | not statedaistudio.google.com |
| Mammoth-1 Open · 27B | 92.9% | 90.1% | -2.7 | 95.4%522 of 547 answered | 93.2%510 of 547 answered | 6.1%32 of 522 | 7.1%36 of 510 | 43.9%25 “I don't know” of 57 not right | 50.7%37 “I don't know” of 73 not right | +6.8 | 6.3%33 of 523answered 95.6%, non-hallucination 42.1% | 0.04039.62at the maker's list price | 0.04040.07at the maker's list price | 0.1691 question per request, 1 in flight | 0.1891 question per request, 1 in flight | 37 | 44 | English (the card's language: en tag; its text names no language)huggingface.co |
| Gemma 4 31B reasoning off · chat Open · 31B | 92.5% | 90.5% | -2.0 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 9.7%53 of 545 | Not rundecision models only | 3.6%2 “I don't know” of 55 not right | Not rundecision models only | 7.7%42 of 545answered 99.6%, non-hallucination 4.5% | 0.04848.22at a host's list price for the same weights | 0.05352.81at a host's list price for the same weights | 3.9716 questions per request, 1 in flight | 3.1916 questions per request, 1 in flight | 38 | 40 | not statedhuggingface.co |
| d3-ultra Open · 26B | 91.4% | 89.4% | -2.0 | 99.6%545 of 547 answered | 99.1%542 of 547 answered | 9.4%51 of 545 | 10.3%56 of 542 | 3.8%2 “I don't know” of 53 not right | 8.2%5 “I don't know” of 61 not right | +4.4 | 8.4%46 of 546answered 99.8%, non-hallucination 2.1% | 0.216216from GPU time | 0.106106from GPU time | 0.3331 question per request, 1 in flight | 0.1491 question per request, 1 in flight | 39 | 51 | not statedhuggingface.co |
| Drex Max Open · 27B | 91.4% | 89.6% | -1.8 | 99.8%546 of 547 answered | 99.6%545 of 547 answered | 9.0%49 of 546 | 11.0%60 of 545 | 2.0%1 “I don't know” of 50 not right | 3.2%2 “I don't know” of 62 not right | +1.2 | 9.7%53 of 545answered 99.6%, non-hallucination 3.6% | 0.02323.29at the maker's list price | 0.02423.56at the maker's list price | 0.0821 question per request, 1 in flight | 0.0811 question per request, 1 in flight | 40 | 47 | not statedhuggingface.co |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 91.4% | 88.7% | -2.7 | 99.8%546 of 547 answered | 99.6%545 of 547 answered | 9.3%51 of 546 | 11.7%64 of 545 | 1.9%1 “I don't know” of 52 not right | 3.0%2 “I don't know” of 66 not right | +1.1 | 9.1%50 of 547answered 100.0%, non-hallucination 0.0% | 0.0087.76at the maker's list price | 0.0087.85at the maker's list price | 0.5801 question per request, 1 in flight | 0.1261 question per request, 1 in flight | 41 | 55 | not statedhuggingface.co |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 91.4% | 91.4% | +0.0 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 8.3%45 of 544 | Not rundecision models only | 6.2%3 “I don't know” of 48 not right | Not rundecision models only | 7.9%43 of 543answered 99.3%, non-hallucination 8.5% | 2.4112,411at list prices | 1.7421,742at list prices | 251.316 questions per request, 4 in flight | 162.916 questions per request, 4 in flight | 42 | 33 | not statedhuggingface.co |
| GLiDE 27B Open · 27B | 91.2% | 90.9% | -0.4 | Not rundecision models only | 96.0%525 of 547 answered | Not rundecision models only | 6.9%36 of 525 | Not rundecision models only | 37.9%22 “I don't know” of 58 not right | Not rundecision models only | 7.2%39 of 538answered 98.4%, non-hallucination 18.8% | No list price | No list price | 0.5421 question per request, 1 in flight | 0.5451 question per request, 1 in flight | 43 | 36 | not statedhuggingface.co |
| d3 Open · 26B | 91.0% | 89.8% | -1.3 | 100.0%547 of 547 answered | 99.8%546 of 547 answered | 9.5%52 of 547 | 10.6%58 of 546 | 0.0%0 “I don't know” of 52 not right | 1.7%1 “I don't know” of 59 not right | +1.7 | 9.7%53 of 547answered 100.0%, non-hallucination 0.0% | 0.07070.21from GPU time | 0.05555.35from GPU time | 0.0921 question per request, 1 in flight | 0.0741 question per request, 1 in flight | 44 | 46 | not statedhuggingface.co |
| Perplexity Decider 27B v1.2 API Open · 27B | 91.0% | 88.5% | -2.6 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 11.6%63 of 544 | Not rundecision models only | 4.5%3 “I don't know” of 66 not right | Not rundecision models only | 9.3%51 of 547answered 100.0%, non-hallucination 0.0% | 0.0087.70at list prices | 0.0087.79at list prices | 0.2541 question per request, 1 in flight | 0.2561 question per request, 1 in flight | 45 | 56 | not statedhuggingface.co |
| d1.2-ultra Closed · Efficient | 90.9% | 90.3% | -0.5 | 89.0%487 of 547 answered | 80.1%438 of 547 answered | 3.7%18 of 487 | 3.0%13 of 438 | 76.9%60 “I don't know” of 78 not right | 89.3%109 “I don't know” of 122 not right | +12.4 | 3.9%19 of 486answered 88.8%, non-hallucination 76.2% | 0.01414.34at list prices | 0.01413.98at list prices | 0.2201 question per request, 1 in flight | 0.2211 question per request, 1 in flight | 46 | 42 | not statedwww.liquid.ai |
| Muse Ember max · chat Closed · Efficient | 90.9% | 89.6% | -1.3 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 9.9%54 of 544 | Not rundecision models only | 5.3%3 “I don't know” of 57 not right | Not rundecision models only | 8.7%47 of 543answered 99.3%, non-hallucination 7.8% | 0.650650at list prices | 0.245245at list prices | 30.916 questions per request, 4 in flight | 11.616 questions per request, 4 in flight | 47 | 48 | not stateddev.meta.ai |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 90.7% | 90.5% | -0.2 | Not rundecision models only | 98.2%537 of 547 answered | Not rundecision models only | 10.4%56 of 537 | Not rundecision models only | 15.2%10 “I don't know” of 66 not right | Not rundecision models only | 9.4%51 of 544answered 99.5%, non-hallucination 5.6% | ≤ 1.034≤ 1,034at list prices | ≤ 1.036≤ 1,036at list prices | 0.4821 question per request, 1 in flight | 0.4841 question per request, 1 in flight | 48 | 41 | not statedhuggingface.co |
| Perplexity Decider 27B v1.3 Open · 27B | 90.7% | 89.6% | -1.1 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 9.6%52 of 542 | Not rundecision models only | 8.8%5 “I don't know” of 57 not right | Not rundecision models only | 9.5%52 of 547answered 100.0%, non-hallucination 0.0% | 0.08181.18from GPU time | 0.08382.63from GPU time | 0.1101 question per request, 1 in flight | 0.1151 question per request, 1 in flight | 49 | 49 | not statedhuggingface.co |
| Qwen3.8-27B low · chat Open · 27B | 90.7% | 89.0% | -1.6 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 9.7%53 of 545 | Not rundecision models only | 3.6%2 “I don't know” of 55 not right | Not rundecision models only | 8.8%48 of 546answered 99.8%, non-hallucination 2.0% | 3.0953,095at the maker's list price | 0.853853at the maker's list price | 648.416 questions per request, 4 in flight | 71.716 questions per request, 4 in flight | 50 | 52 | not statedhuggingface.co |
| GLiDE v1.2 Open · 27B | 90.5% | 90.3% | -0.2 | 98.4%538 of 547 answered | 94.5%517 of 547 answered | 8.7%47 of 538 | 7.0%36 of 517 | 16.1%9 “I don't know” of 56 not right | 45.5%30 “I don't know” of 66 not right | +29.4 | 8.7%47 of 541answered 98.9%, non-hallucination 11.3% | 0.01414.37at the maker's list price | 0.01514.55at the maker's list price | 0.4131 question per request, 1 in flight | 0.1041 question per request, 1 in flight | 51 | 43 | not statedhuggingface.co |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 90.3% | 89.9% | -0.4 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 10.8%59 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 59 not right | Not rundecision models only | 9.5%52 of 546answered 99.8%, non-hallucination 1.9% | 0.06868.19at a host's list price for the same weights | 0.07070.17at a host's list price for the same weights | 0.2981 question per request, 1 in flight | 0.2991 question per request, 1 in flight | 52 | 45 | not statedhuggingface.co |
| Drex Max v1.1 Closed · Efficient (provisional) | 90.1% | 88.5% | -1.6 | 96.7%529 of 547 answered | 92.1%504 of 547 answered | 9.1%48 of 529 | 8.1%41 of 504 | 27.3%18 “I don't know” of 66 not right | 51.2%43 “I don't know” of 84 not right | +23.9 | 9.5%51 of 538answered 98.4%, non-hallucination 15.0% | 0.02323.11at list prices | 0.02323.38at list prices | 0.3131 question per request, 1 in flight | 0.3351 question per request, 1 in flight | 53 | 57 | not statedconsole.nace.ai |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 90.1% | 89.6% | -0.5 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 11.2%61 of 546 | Not rundecision models only | 1.6%1 “I don't know” of 62 not right | Not rundecision models only | 10.1%55 of 545answered 99.6%, non-hallucination 3.5% | 0.543543at the maker's list price | 0.08181.38at the maker's list price | 102.616 questions per request, 8 in flight | 33.116 questions per request, 8 in flight | 54 | 50 | other languages, not Turkishhuggingface.co |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 89.9% | 87.0% | -2.9 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 12.9%70 of 543 | Not rundecision models only | 5.4%4 “I don't know” of 74 not right | Not rundecision models only | 10.4%57 of 547answered 100.0%, non-hallucination 0.0% | 0.09898.47at the maker's list price | 0.07979.31at the maker's list price | 5.0816 questions per request, 1 in flight | 6.5916 questions per request, 1 in flight | 55 | 64 | not statedhuggingface.co |
| SubQ Snap Preview Closed · Efficient | 89.9% | 85.9% | -4.0 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.01919.41at list prices | 0.01919.48at list prices | 0.3891 question per request, 1 in flight | 0.3811 question per request, 1 in flight | 56 | 76 | not stateddocs.subq.ai |
| Claude Haiku 5.5 max · chat Closed · Efficient | 89.4% | 92.7% | +3.3 | Not rundecision models only | 96.3%527 of 547 answered | Not rundecision models only | 8.9%47 of 527 | Not rundecision models only | 7.8%4 “I don't know” of 51 not right | Not rundecision models only | 5.5%29 of 525answered 96.0%, non-hallucination 17.1% | 2.9652,965at list prices | 1.2611,261at list prices | 417.016 questions per request, 4 in flight | 148.916 questions per request, 4 in flight | 57 | 25 | other languages, not Turkishplatform.claude.com |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 89.4% | 88.3% | -1.1 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 11.0%60 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 60 not right | Not rundecision models only | 11.3%62 of 547answered 100.0%, non-hallucination 0.0% | 1.1371,137at list prices | 0.425425at list prices | 97.716 questions per request, 4 in flight | 25.316 questions per request, 4 in flight | 58 | 59 | other languages, not Turkishhuggingface.co |
| GLM-5.3 low · chat Open · Workhorse | 89.2% | 87.8% | -1.5 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 10.3%56 of 545 | Not rundecision models only | 3.4%2 “I don't know” of 58 not right | Not rundecision models only | 11.2%61 of 544answered 99.5%, non-hallucination 4.7% | 0.814814at list prices | 0.408408at list prices | 17.116 questions per request, 4 in flight | 6.8416 questions per request, 4 in flight | 59 | 61 | other languages, not Turkishhuggingface.co |
| SubQ Snap 1.1 Closed · Efficient | 89.2% | 85.9% | -3.3 | 96.5%528 of 547 answered | 96.2%526 of 547 answered | 8.5%45 of 528 | 12.0%63 of 526 | 29.7%19 “I don't know” of 64 not right | 25.0%21 “I don't know” of 84 not right | -4.7 | 9.7%51 of 528answered 96.5%, non-hallucination 27.1% | 0.01515.41at list prices | 0.01615.59at list prices | 0.3781 question per request, 1 in flight | 0.3811 question per request, 1 in flight | 60 | 77 | not stateddocs.subq.ai |
| Perplexity Decider 27B v1 Open · 27B | 89.0% | 88.5% | -0.5 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.01615.54at list prices | 0.01615.59at list prices | 0.2161 question per request, 1 in flight | 0.2471 question per request, 1 in flight | 61 | 58 | not statedhuggingface.co |
| d1.2 Closed · Efficient | 88.8% | 89.0% | +0.2 | 91.6%501 of 547 answered | 89.2%488 of 547 answered | 8.4%42 of 501 | 7.6%37 of 488 | 52.3%46 “I don't know” of 88 not right | 61.5%59 “I don't know” of 96 not right | +9.2 | 7.1%35 of 494answered 90.3%, non-hallucination 60.2% | 0.0076.83at list prices | 0.0076.66at list prices | 0.2131 question per request, 1 in flight | 0.2191 question per request, 1 in flight | 62 | 53 | not statedwww.liquid.ai |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 88.8% | 87.0% | -1.8 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 12.3%67 of 543 | Not rundecision models only | 5.6%4 “I don't know” of 71 not right | Not rundecision models only | 10.1%55 of 544answered 99.5%, non-hallucination 5.2% | 0.04747.13at list prices | 0.05049.69at list prices | 10.016 questions per request, 4 in flight | 10.116 questions per request, 4 in flight | 63 | 65 | not statedhuggingface.co |
| GPT-6 Luna low · chat Closed · Efficient | 88.7% | 86.3% | -2.4 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 12.7%69 of 542 | Not rundecision models only | 6.8%5 “I don't know” of 74 not right | Not rundecision models only | 11.9%65 of 545answered 99.6%, non-hallucination 3.0% | 0.04848.01subscription, priced as the API | 0.05857.70subscription, priced as the API | 7.7116 questions per request, 1 in flight | 7.7216 questions per request, 1 in flight | 64 | 71 | not stateddevelopers.openai.com |
| d1.1 Closed · Efficient | 88.3% | 88.3% | +0.0 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.0076.83at list prices | 0.0076.66at list prices | 0.1951 question per request, 1 in flight | 0.1731 question per request, 1 in flight | 65 | 60 | not stateddocs.liquid.ai |
| Clef Open · 27B | 88.1% | 86.7% | -1.5 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 13.7%75 of 546 | Not rundecision models only | 1.3%1 “I don't know” of 76 not right | Not rundecision models only | 12.3%67 of 546answered 99.8%, non-hallucination 1.5% | 0.110110at the maker's list price | 0.111111at the maker's list price | 0.2051 question per request, 1 in flight | 0.2201 question per request, 1 in flight | 66 | 69 | not statedhuggingface.co |
| OneJev-27B Open · 27B | 87.8% | 86.3% | -1.5 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 13.2%72 of 544 | Not rundecision models only | 4.0%3 “I don't know” of 75 not right | Not rundecision models only | 11.5%63 of 546answered 99.8%, non-hallucination 1.6% | 0.09999.09from GPU time | 0.106106from GPU time | 0.1251 question per request, 1 in flight | 0.1301 question per request, 1 in flight | 67 | 72 | not statedhuggingface.co |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 87.6% | 86.1% | -1.5 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 10.8%59 of 544 | Not rundecision models only | 4.8%3 “I don't know” of 62 not right | Not rundecision models only | 9.7%53 of 545answered 99.6%, non-hallucination 3.6% | 3.6543,654subscription, priced as the API | 1.9981,998subscription, priced as the API | 20.516 questions per request, 1 in flight, measured on a sample (200 questions) | 3.8516 questions per request, 1 in flight | 68 | 74 | other languages, not Turkishplatform.claude.com |
| KnowLine-12B-Gen1 Open · 12B | 87.6% | 86.7% | -0.9 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 13.9%76 of 546 | Not rundecision models only | 1.3%1 “I don't know” of 77 not right | Not rundecision models only | 12.2%67 of 547answered 100.0%, non-hallucination 0.0% | 0.05252.28from GPU time | 0.05756.86from GPU time | 0.0711 question per request, 1 in flight | 0.0781 question per request, 1 in flight | 69 | 70 | other languages, not Turkishhuggingface.co |
| celeris-1-decision Closed · Efficient | 87.4% | 84.1% | -3.3 | Not rundecision models only | 83.2%455 of 547 answered | Not rundecision models only | 8.1%37 of 455 | Not rundecision models only | 71.3%92 “I don't know” of 129 not right | Not rundecision models only | Not run | 0.01515.28at list prices | 0.01616.09at list prices | 0.4011 question per request, 1 in flight | 0.3901 question per request, 1 in flight | 70 | 91 | not stateddocs.celeris.ai |
| Jev 1.14 Closed · Efficient | 87.2% | 86.1% | -1.1 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 13.2%72 of 544 | Not rundecision models only | 4.0%3 “I don't know” of 75 not right | Not rundecision models only | 12.7%69 of 544answered 99.5%, non-hallucination 4.2% | 0.02928.84at list prices | 0.02726.64at list prices | 0.2111 question per request, 1 in flight | 0.1811 question per request, 1 in flight | 71 | 75 | "English is the primary training language and where accuracy is currently best. Other languages, including CJK scripts, are handled but not equally well"docs.typesafe.ai |
| Mercury Decide Closed · Efficient | 87.0% | 85.7% | -1.3 | Not rundecision models only | 98.0%536 of 547 answered | Not rundecision models only | 13.2%71 of 536 | Not rundecision models only | 13.4%11 “I don't know” of 82 not right | Not rundecision models only | 13.2%71 of 539answered 98.5%, non-hallucination 10.1% | 0.0087.71at list prices | 0.0087.79at list prices | 0.3131 question per request, 1 in flight | 0.3101 question per request, 1 in flight | 72 | 82 | not statedopenrouter.ai |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 86.8% | 89.0% | +2.2 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 10.6%58 of 546 | Not rundecision models only | 1.7%1 “I don't know” of 59 not right | Not rundecision models only | 12.7%69 of 544answered 99.5%, non-hallucination 4.2% | 2.2192,219at list prices | 0.298298at list prices | 21.41 question per request, 32 in flight | 6.671 question per request, 8 in flight | 73 | 54 | other languages, not Turkishhuggingface.co |
| GPT-6 Luna Decisions API Closed · Efficient | 86.7% | 84.1% | -2.6 | Not rundecision models only | 98.2%537 of 547 answered | Not rundecision models only | 17.9%96 of 537 | Not rundecision models only | 9.4%10 “I don't know” of 106 not right | Not rundecision models only | 14.4%78 of 541answered 98.9%, non-hallucination 7.1% | 0.04040.44at list prices | 0.04141.09at list prices | 0.1911 question per request, 4 in flight | 0.3231 question per request, 4 in flight | 74 | 92 | not stateddevelopers.openai.com |
| Jeffan-9B-1211 Open · 9.0B | 86.5% | 87.6% | +1.1 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 13.0%71 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 71 not right | Not rundecision models only | 13.1%71 of 544answered 99.5%, non-hallucination 4.1% | 0.04444.29from GPU time | 0.04645.85from GPU time | 0.0641 question per request, 1 in flight | 0.0671 question per request, 1 in flight | 75 | 62 | not statedhuggingface.co |
| Open-Jev-27B-v1.1 Open · 27B | 86.5% | 85.4% | -1.1 | Not rundecision models only | 97.1%531 of 547 answered | Not rundecision models only | 14.3%76 of 531 | Not rundecision models only | 17.4%16 “I don't know” of 92 not right | Not rundecision models only | 14.4%78 of 541answered 98.9%, non-hallucination 7.1% | 0.271271from GPU time | 0.351351from GPU time | 0.3861 question per request, 1 in flight | 0.4951 question per request, 1 in flight | 76 | 85 | not statedhuggingface.co |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 86.3% | 84.5% | -1.8 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 15.6%85 of 544 | Not rundecision models only | 3.4%3 “I don't know” of 88 not right | Not rundecision models only | 16.5%90 of 545answered 99.6%, non-hallucination 1.1% | 0.814814at list prices | 0.193193at list prices | 112.616 questions per request, 4 in flight | 26.716 questions per request, 4 in flight | 77 | 90 | not statedhuggingface.co |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 86.3% | 84.8% | -1.5 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 14.3%78 of 544 | Not rundecision models only | 3.7%3 “I don't know” of 81 not right | Not rundecision models only | 13.3%72 of 543answered 99.3%, non-hallucination 5.3% | 0.234234at the maker's list price | 0.212212at the maker's list price | 0.4331 question per request, 1 in flight | 0.3901 question per request, 1 in flight | 78 | 88 | not statedhuggingface.co |
| Jev 1.13 Closed · Efficient | 86.1% | 85.9% | -0.2 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.02928.85at list prices | 0.02726.64at list prices | 0.2431 question per request, 1 in flight | 0.1731 question per request, 1 in flight | 79 | 78 | not stateddocs.typesafe.ai |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 85.9% | 83.9% | -2.0 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 15.4%84 of 544 | Not rundecision models only | 3.4%3 “I don't know” of 87 not right | Not rundecision models only | 13.8%75 of 543answered 99.3%, non-hallucination 5.1% | 0.825825at the maker's list price | 0.645645at the maker's list price | 1.621 question per request, 1 in flight | 1.701 question per request, 1 in flight | 80 | 93 | not statedhuggingface.co |
| Cygnet Open · 12B | 85.6% | 83.9% | -1.6 | Not rundecision models only | 98.2%537 of 547 answered | Not rundecision models only | 14.7%79 of 537 | Not rundecision models only | 11.2%10 “I don't know” of 89 not right | Not rundecision models only | 14.9%81 of 542answered 99.1%, non-hallucination 5.8% | 0.03939.39at a host's list price for the same weights | 0.04141.42at a host's list price for the same weights | 0.0361 question per request, 1 in flight | 0.0431 question per request, 1 in flight | 81 | 94 | not statedhuggingface.co |
| d1 Closed · Efficient | 85.0% | 86.8% | +1.8 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 12.3%67 of 545 | Not rundecision models only | 2.9%2 “I don't know” of 69 not right | Not rundecision models only | 16.3%89 of 546answered 99.8%, non-hallucination 1.1% | 0.0077.43free tier, at the list price | 0.0076.66at list prices | 0.3051 question per request, 1 in flight | 0.1801 question per request, 1 in flight | 82 | 67 | not stateddocs.liquid.ai |
| Winnow-12B Open · 12B | 85.0% | 83.9% | -1.1 | Not rundecision models only | 97.3%532 of 547 answered | Not rundecision models only | 16.0%85 of 532 | Not rundecision models only | 15.0%15 “I don't know” of 100 not right | Not rundecision models only | 15.8%85 of 539answered 98.5%, non-hallucination 8.6% | 0.04141.40from GPU time | 0.05049.59from GPU time | 0.0591 question per request, 1 in flight | 0.0701 question per request, 1 in flight | 83 | 95 | not statedhuggingface.co |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 84.8% | 83.9% | -0.9 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 15.8%86 of 543 | Not rundecision models only | 4.4%4 “I don't know” of 90 not right | Not rundecision models only | 15.1%82 of 542answered 99.1%, non-hallucination 5.7% | 1.8961,896subscription, priced as the API | 0.354354subscription, priced as the API | 3.8716 questions per request, 1 in flight, measured on a sample (200 questions) | 3.6816 questions per request, 1 in flight | 84 | 96 | other languages, not Turkishplatform.claude.com |
| Drex 2.0 Open · 35B MoE · 3.0B active | 84.8% | 85.9% | +1.1 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.01919.15at list prices | 0.01919.42at list prices | 0.3251 question per request, 1 in flight | 0.2831 question per request, 1 in flight | 85 | 79 | not statedhuggingface.co |
| Qwen3.8-27B yes/no scoring Open · 27B | 84.6% | 84.8% | +0.2 | Not rundecision models only | Not runyes/no scoring cannot offer it | Not rundecision models only | Not runyes/no scoring cannot offer it | Not rundecision models only | Not runyes/no scoring cannot offer it | Not rundecision models only | Not runyes/no scoring cannot offer it | 0.704704at the maker's list price | 0.741741at the maker's list price | 0.2801 question per request, 1 in flight | 0.4131 question per request, 1 in flight | 86 | 89 | not statedhuggingface.co |
| GLiDE 9B Open · 7.9B | 84.3% | 86.3% | +2.0 | Not rundecision models only | 97.4%533 of 547 answered | Not rundecision models only | 13.1%70 of 533 | Not rundecision models only | 16.7%14 “I don't know” of 84 not right | Not rundecision models only | 15.4%83 of 538answered 98.4%, non-hallucination 9.8% | No list price | No list price | 0.2071 question per request, 1 in flight | 0.2071 question per request, 1 in flight | 87 | 73 | not statedhuggingface.co |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 84.3% | 87.2% | +2.9 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 12.3%67 of 543 | Not rundecision models only | 5.6%4 “I don't know” of 71 not right | Not rundecision models only | 15.1%82 of 544answered 99.5%, non-hallucination 3.5% | 0.04747.28at the maker's list price | 0.04646.07at the maker's list price | 1.0716 questions per request, 1 in flight | 3.3716 questions per request, 1 in flight | 88 | 63 | other languages, not Turkishhuggingface.co |
| Adjudex 9B v5 Open · 9.0B | 83.9% | 85.7% | +1.8 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 14.1%77 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 77 not right | Not rundecision models only | 16.3%89 of 547answered 100.0%, non-hallucination 0.0% | 0.02626.30from GPU time | 0.02524.93from GPU time | 0.0381 question per request, 1 in flight | 0.0341 question per request, 1 in flight | 89 | 83 | not statedhuggingface.co |
| Cerno 9B v1.2 Open · 9.0B | 83.5% | 85.9% | +2.4 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 14.1%77 of 545 | Not rundecision models only | 2.5%2 “I don't know” of 79 not right | Not rundecision models only | 16.6%91 of 547answered 100.0%, non-hallucination 0.0% | 0.03636.02from GPU time | No list price | 0.0511 question per request, 1 in flight | 0.2111 question per request, 1 in flight | 90 | 80 | not statedhuggingface.co |
| Elephant-1 Open · 9.0B | 83.5% | 85.4% | +1.8 | 95.6%523 of 547 answered | 95.8%524 of 547 answered | 15.7%82 of 523 | 12.8%67 of 524 | 22.6%24 “I don't know” of 106 not right | 25.6%23 “I don't know” of 90 not right | +2.9 | 15.4%82 of 532answered 97.3%, non-hallucination 15.5% | 0.01413.87at the maker's list price | 0.01414.02at the maker's list price | 0.0741 question per request, 1 in flight | 0.0701 question per request, 1 in flight | 91 | 86 | English (the card's language: en tag; its text names no language)huggingface.co |
| Dexy-Decision-9B-V2 Open · 9.0B | 83.4% | 83.5% | +0.2 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 16.5%90 of 546 | Not rundecision models only | 1.1%1 “I don't know” of 91 not right | Not rundecision models only | 16.5%90 of 546answered 99.8%, non-hallucination 1.1% | 0.02323.03from GPU time | 0.02525.41from GPU time | 0.0311 question per request, 1 in flight | 0.0361 question per request, 1 in flight | 92 | 99 | not statedhuggingface.co |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 83.0% | 86.8% | +3.8 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 13.6%74 of 545 | Not rundecision models only | 2.6%2 “I don't know” of 76 not right | Not rundecision models only | 17.4%95 of 546answered 99.8%, non-hallucination 1.0% | No list price | No list price | 0.1541 question per request, 1 in flight | 0.1561 question per request, 1 in flight | 93 | 68 | other languages, not Turkishhuggingface.co |
| KnowLine-9B-Gen2 Open · 9.0B | 82.8% | 85.2% | +2.4 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 14.3%78 of 546 | Not rundecision models only | 1.3%1 “I don't know” of 79 not right | Not rundecision models only | 17.4%95 of 547answered 100.0%, non-hallucination 0.0% | 0.03535.40from GPU time | 0.03939.48from GPU time | 0.0481 question per request, 1 in flight | 0.0561 question per request, 1 in flight | 94 | 87 | other languages, not Turkishhuggingface.co |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 82.8% | 87.0% | +4.2 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 15.2%83 of 545 | Not rundecision models only | 2.4%2 “I don't know” of 85 not right | Not rundecision models only | 16.2%88 of 542answered 99.1%, non-hallucination 2.2% | 2.9982,998from GPU time | 1.3901,390from GPU time | 140.31 question per request, 32 in flight | 58.31 question per request, 32 in flight | 95 | 66 | not statedhuggingface.co |
| KnowLine-9B-Gen1 Open · 9.0B | 82.4% | 85.9% | +3.5 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 13.9%76 of 546 | Not rundecision models only | 1.3%1 “I don't know” of 77 not right | Not rundecision models only | 17.9%98 of 547answered 100.0%, non-hallucination 0.0% | 0.04140.69from GPU time | 0.04241.55from GPU time | 0.0571 question per request, 1 in flight | 0.0591 question per request, 1 in flight | 96 | 81 | other languages, not Turkishhuggingface.co |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 80.6% | 82.4% | +1.8 | Not rundecision models only | 97.8%535 of 547 answered | Not rundecision models only | 17.2%92 of 535 | Not rundecision models only | 9.8%10 “I don't know” of 102 not right | Not rundecision models only | 18.4%96 of 522answered 95.4%, non-hallucination 20.7% | 0.307307from GPU time | 0.199199from GPU time | 13.41 question per request, 32 in flight | 8.391 question per request, 32 in flight | 97 | 102 | not statedhuggingface.co |
| vera-medium-omni Open · 4.2B | 79.5% | 83.5% | +4.0 | Not rundecision models only | 98.7%540 of 547 answered | Not rundecision models only | 16.1%87 of 540 | Not rundecision models only | 7.4%7 “I don't know” of 94 not right | Not rundecision models only | 20.8%113 of 542answered 99.1%, non-hallucination 4.2% | 0.02120.72from GPU time | 0.03736.63from GPU time | 0.0291 question per request, 1 in flight | 0.0541 question per request, 1 in flight | 98 | 100 | English (the card's language: en tag; its text names no language)huggingface.co |
| Valen-Decision-1.5-9B Open · 9.0B | 79.0% | 83.7% | +4.8 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 15.4%84 of 546 | Not rundecision models only | 1.2%1 “I don't know” of 85 not right | Not rundecision models only | 20.1%110 of 547answered 100.0%, non-hallucination 0.0% | 0.04040.49from GPU time | 0.04140.55from GPU time | 0.0591 question per request, 1 in flight | 0.0591 question per request, 1 in flight | 99 | 97 | not statedhuggingface.co |
| Perplexity Decider Flash v1.3 Open · 4.0B | 78.8% | 83.7% | +4.9 | Not rundecision models only | 98.7%540 of 547 answered | Not rundecision models only | 16.7%90 of 540 | Not rundecision models only | 7.2%7 “I don't know” of 97 not right | Not rundecision models only | 20.4%110 of 538answered 98.4%, non-hallucination 7.6% | 0.02524.69from GPU time | 0.02323.17from GPU time | 0.0331 question per request, 1 in flight | 0.0331 question per request, 1 in flight | 100 | 98 | not statedhuggingface.co |
| KnowLine 4B Gen5.5 Open · 4.0B | 78.6% | 80.8% | +2.2 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 19.2%105 of 546 | Not rundecision models only | 0.9%1 “I don't know” of 106 not right | Not rundecision models only | 21.7%118 of 544answered 99.5%, non-hallucination 2.5% | No list price | No list price | 0.0991 question per request, 1 in flight | 0.0991 question per request, 1 in flight | 101 | 106 | other languages, not Turkishhuggingface.co |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 78.4% | 82.8% | +4.4 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 17.2%94 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 94 not right | Not rundecision models only | 21.7%118 of 545answered 99.6%, non-hallucination 1.7% | No list price | No list price | 0.0741 question per request, 1 in flight | 0.0751 question per request, 1 in flight | 102 | 101 | other languages, not Turkishhuggingface.co |
| Clef-Flash Open · 9.0B | 77.7% | 82.3% | +4.6 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 16.6%90 of 542 | Not rundecision models only | 5.3%5 “I don't know” of 95 not right | Not rundecision models only | 21.5%115 of 535answered 97.8%, non-hallucination 9.4% | 0.01717.37at the maker's list price | 0.01817.53at the maker's list price | 0.0751 question per request, 1 in flight | 0.1401 question per request, 1 in flight | 103 | 103 | not statedhuggingface.co |
| Verdin-4B-v1.1 Open · 4.0B | 77.5% | 82.3% | +4.8 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 17.2%94 of 545 | Not rundecision models only | 2.1%2 “I don't know” of 96 not right | Not rundecision models only | 23.3%127 of 546answered 99.8%, non-hallucination 0.8% | 0.03434.21from GPU time | 0.03434.28from GPU time | 0.0491 question per request, 1 in flight | 0.0521 question per request, 1 in flight | 104 | 104 | other languages, not Turkishhuggingface.co |
| Qwen3-14B reasoning off · chat Open · 15B | 76.8% | 79.7% | +2.9 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 19.8%108 of 545 | Not rundecision models only | 1.8%2 “I don't know” of 110 not right | Not rundecision models only | 22.3%122 of 546answered 99.8%, non-hallucination 0.8% | 0.03232.31at a host's list price for the same weights | 0.03131.17at a host's list price for the same weights | 2.8516 questions per request, 1 in flight | 2.8616 questions per request, 1 in flight | 105 | 109 | not statedhuggingface.co |
| Perplexity Decider Flash v1.2 Open · 4.0B | 76.6% | 81.9% | +5.3 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 18.5%100 of 542 | Not rundecision models only | 4.8%5 “I don't know” of 105 not right | Not rundecision models only | 24.6%133 of 541answered 98.9%, non-hallucination 4.3% | 0.0043.88at the maker's list price | 0.0043.93at the maker's list price | 0.1321 question per request, 1 in flight | 0.0371 question per request, 1 in flight | 106 | 105 | not statedhuggingface.co |
| Microsoft-Decision-1 Closed · Efficient | 76.4% | 85.7% | +9.3 * | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 14.2%77 of 544 | Not rundecision models only | 3.8%3 “I don't know” of 80 not right | Not rundecision models only | 16.1%88 of 545answered 99.6%, non-hallucination 2.2% | 0.01413.82at list prices | 0.01414.01at list prices | 0.2861 question per request, 1 in flight | 0.3151 question per request, 1 in flight | 107 | 84 | not statedopenrouter.ai |
| Chinese Decision Pointer 4B Open · 4.0B | 75.7% | 80.4% | +4.8 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 19.2%104 of 543 | Not rundecision models only | 3.7%4 “I don't know” of 108 not right | Not rundecision models only | 23.8%129 of 543answered 99.3%, non-hallucination 3.0% | 0.02524.58from GPU time | 0.02727.21from GPU time | 0.0351 question per request, 1 in flight | 0.0381 question per request, 1 in flight | 108 | 107 | other languages, not Turkishhuggingface.co |
| Erk-14B reasoning off · chat Open · 14B | 75.3% | 72.0% | -3.3 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 25.9%141 of 544 | Not rundecision models only | 2.1%3 “I don't know” of 144 not right | Not rundecision models only | 23.6%129 of 546answered 99.8%, non-hallucination 0.8% | 0.106106from GPU time | 0.123123from GPU time | 2.3916 questions per request, 1 in flight | 2.8616 questions per request, 1 in flight | 109 | 112 | Turkish listedhuggingface.co |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 72.0% | 80.4% | +8.4 * | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 19.5%106 of 543 | Not rundecision models only | 3.6%4 “I don't know” of 110 not right | Not rundecision models only | 26.0%140 of 538answered 98.4%, non-hallucination 3.4% | 0.894894from GPU time | 0.309309from GPU time | 32.21 question per request, 32 in flight | 9.741 question per request, 32 in flight | 110 | 108 | not statedhuggingface.co |
| Metask-Jev-4B Open · 4.5B | 67.1% | 77.1% | +10.1 * | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 22.6%123 of 545 | Not rundecision models only | 1.6%2 “I don't know” of 125 not right | Not rundecision models only | 34.3%187 of 545answered 99.6%, non-hallucination 1.1% | 0.06362.82from GPU time | 0.05959.11from GPU time | 0.0901 question per request, 1 in flight | 0.0851 question per request, 1 in flight | 111 | 110 | Turkish listedhuggingface.co |
| KnowLine-2B-Gen1 Open · 2.0B | 58.0% | 71.3% | +13.3 * | Not rundecision models only | 93.1%509 of 547 answered | Not rundecision models only | 26.5%135 of 509 | Not rundecision models only | 22.0%38 “I don't know” of 173 not right | Not rundecision models only | Not run | No list price | No list price | 0.0621 question per request, 1 in flight | 0.0621 question per request, 1 in flight | 112 | 114 | other languages, not Turkishhuggingface.co |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 55.8% | 69.7% | +13.9 * | Not rundecision models only | 94.1%515 of 547 answered | Not rundecision models only | 30.5%157 of 515 | Not rundecision models only | 16.9%32 “I don't know” of 189 not right | Not rundecision models only | 45.7%235 of 514answered 94.0%, non-hallucination 12.3% | No list price | No list price | 0.0401 question per request, 1 in flight | 0.0401 question per request, 1 in flight | 113 | 118 | other languages, not Turkishhuggingface.co |
| GLiDE 2B Open · 1.9B | 54.3% | 70.0% | +15.7 * | Not rundecision models only | 91.4%500 of 547 answered | Not rundecision models only | 26.4%132 of 500 | Not rundecision models only | 26.3%47 “I don't know” of 179 not right | Not rundecision models only | 43.5%209 of 481answered 87.9%, non-hallucination 24.0% | No list price | No list price | 0.0341 question per request, 1 in flight | 0.0351 question per request, 1 in flight | 114 | 117 | not statedhuggingface.co |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 48.1% | 71.1%historical paired baseline, run 1061 | +23.0 * | Not rundecision models only | 95.2%521 of 547 answered | Not rundecision models only | 27.1%141 of 521 | Not rundecision models only | 15.6%26 “I don't know” of 167 not right | Not rundecision models only | 47.6%210 of 441answered 80.6%, non-hallucination 33.5% | 1.2771,277at the maker's list price | 0.387387at the maker's list price | 24.21 question per request, 32 in flight | 5.241 question per request, 32 in flight | 115 | 115 | other languages, not Turkishhuggingface.co |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 47.9% | 73.9% | +26.0 * | Not rundecision models only | 95.6%523 of 547 answered | Not rundecision models only | 21.6%113 of 523 | Not rundecision models only | 15.7%21 “I don't know” of 134 not right | Not rundecision models only | 46.2%229 of 496answered 90.7%, non-hallucination 15.8% | 2.7612,761at the maker's list price | 0.964964at the maker's list price | 51.21 question per request, 32 in flight | 15.01 question per request, 32 in flight | 116 | 111 | other languages, not Turkishhuggingface.co |
| GLiDE 0.8B Open · 0.8B | 46.1% | 60.7% | +14.6 * | Not rundecision models only | 91.0%498 of 547 answered | Not rundecision models only | 35.5%177 of 498 | Not rundecision models only | 21.7%49 “I don't know” of 226 not right | Not rundecision models only | 48.6%230 of 473answered 86.5%, non-hallucination 24.3% | No list price | No list price | 0.0221 question per request, 1 in flight | 0.0231 question per request, 1 in flight | 117 | 122 | not statedhuggingface.co |
| OneJev1.6-0.8B Open · 0.8B | 42.8% | 59.4% | +16.6 * | Not rundecision models only | 86.5%473 of 547 answered | Not rundecision models only | 39.5%187 of 473 | Not rundecision models only | 28.4%74 “I don't know” of 261 not right | Not rundecision models only | 58.2%253 of 435answered 79.5%, non-hallucination 30.7% | No list price | No list price | 0.0171 question per request, 1 in flight | 0.0181 question per request, 1 in flight | 118 | 124 | not statedhuggingface.co |
| d1-2B Open · 1.9B | 41.9% | 65.6% | +23.8 * | Not rundecision models only | 72.4%396 of 547 answered | Not rundecision models only | 24.5%97 of 396 | Not rundecision models only | 60.9%151 “I don't know” of 248 not right | Not rundecision models only | 53.6%207 of 386answered 70.6%, non-hallucination 43.8% | 0.01313.17from GPU time | 0.0088.08from GPU time | 0.0191 question per request, 1 in flight | 0.0111 question per request, 1 in flight | 119 | 121 | other languages, not Turkishhuggingface.co |
| KnowLine-0.8B-Gen1 Open · 0.8B | 40.2% | 58.9% | +18.6 * | Not rundecision models only | 95.2%521 of 547 answered | Not rundecision models only | 41.7%217 of 521 | Not rundecision models only | 10.7%26 “I don't know” of 243 not right | Not rundecision models only | 61.1%325 of 532answered 97.3%, non-hallucination 4.4% | No list price | No list price | 0.0331 question per request, 1 in flight | 0.0331 question per request, 1 in flight | 120 | 126 | other languages, not Turkishhuggingface.co |
| LFM3-1.7B reasoning off · chat Open · 1.7B | 40.2% | 72.9%historical paired baseline, run 1223 | +32.7 * | Not rundecision models only | 96.7%529 of 547 answered | Not rundecision models only | 25.9%137 of 529 | Not rundecision models only | 11.6%18 “I don't know” of 155 not right | Not rundecision models only | 55.6%173 of 311answered 56.9%, non-hallucination 55.4% | 0.646646at the maker's list price | 0.355355at the maker's list price | 4.901 question per request, 32 in flight | 4.691 question per request, 32 in flight | 121 | 113 | other languages, not Turkishhuggingface.co |
| Strands Decider 2B Open · 2.0B | 39.1% | 45.5% | +6.4 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 55.2%301 of 545 | Not rundecision models only | 0.7%2 “I don't know” of 303 not right | Not rundecision models only | 60.3%327 of 542answered 99.1%, non-hallucination 1.5% | 0.02020.12from GPU time | 0.01818.15from GPU time | 0.0261 question per request, 1 in flight | 0.0261 question per request, 1 in flight | 122 | 129 | not statedhuggingface.co |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 37.3% | 59.6% | +22.3 * | Not rundecision models only | 97.8%535 of 547 answered | Not rundecision models only | 40.6%217 of 535 | Not rundecision models only | 5.2%12 “I don't know” of 229 not right | Not rundecision models only | 61.0%320 of 525answered 96.0%, non-hallucination 6.4% | No list price | No list price | 0.0281 question per request, 1 in flight | 0.0281 question per request, 1 in flight | 123 | 123 | other languages, not Turkishhuggingface.co |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 36.7% | 67.3% | +30.5 * | Not rundecision models only | 96.3%527 of 547 answered | Not rundecision models only | 29.6%156 of 527 | Not rundecision models only | 10.3%18 “I don't know” of 174 not right | Not rundecision models only | 61.2%279 of 456answered 83.4%, non-hallucination 16.5% | 0.268268from GPU time | 0.280280from GPU time | 8.841 question per request, 32 in flight | 9.271 question per request, 32 in flight | 124 | 120 | not statedhuggingface.co |
| Decider 1 Closed · Efficient | 36.6% | 55.8% | +19.2 * | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.01514.76at list prices | 0.01514.95at list prices | 0.3821 question per request, 1 in flight | 0.6991 question per request, 1 in flight | 125 | 128 | not statedmeragpt.com |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 27.1% | 70.4% | +43.3 * | Not rundecision models only | 90.5%495 of 547 answered | Not rundecision models only | 28.7%142 of 495 | Not rundecision models only | 26.4%51 “I don't know” of 193 not right | Not rundecision models only | 67.9%195 of 287answered 52.5%, non-hallucination 50.8% | 1.0501,050from GPU time | 0.517517from GPU time | 39.21 question per request, 32 in flight | 19.41 question per request, 32 in flight | 126 | 116 | other languages, not Turkishhuggingface.co |
| GLiClass v4 Edge Open · 0.4B | 26.9% | 58.3% | +31.4 * | Not rundecision models only | 90.5%495 of 547 answered | Not rundecision models only | 39.6%196 of 495 | Not rundecision models only | 21.0%52 “I don't know” of 248 not right | Not rundecision models only | 72.0%352 of 489answered 89.4%, non-hallucination 14.1% | No list price | No list price | 0.0251 question per request, 1 in flight | 0.0171 question per request, 1 in flight | 127 | 127 | other languages, not Turkishhuggingface.co |
| GLiDE 0.4B Open · 0.4B | 25.8% | 59.4% | +33.6 * | Not rundecision models only | 72.0%394 of 547 answered | Not rundecision models only | 34.3%135 of 394 | Not rundecision models only | 53.1%153 “I don't know” of 288 not right | Not rundecision models only | 76.4%55 of 72answered 13.2%, non-hallucination 89.6% | No list price | No list price | 0.0161 question per request, 1 in flight | 0.0141 question per request, 1 in flight | 128 | 125 | not statedhuggingface.co |
| GLiNER2.5-multi-Decide Open · 0.3B | 20.7% | 21.2% | +0.5 | Not rundecision models only | 87.2%477 of 547 answered | Not rundecision models only | 76.1%363 of 477 | Not rundecision models only | 16.2%70 “I don't know” of 433 not right | Not rundecision models only | 78.8%412 of 523answered 95.6%, non-hallucination 5.5% | 0.0066.27from GPU time | 0.0076.73from GPU time | 0.0091 question per request, 1 in flight | 0.0091 question per request, 1 in flight | 129 | 131 | other languages, not Turkishhuggingface.co |
| Laya Open · 0.4B | 20.5% | 22.9% | +2.4 | Not rundecision models only | 84.8%464 of 547 answered | Not rundecision models only | 79.1%367 of 464 | Not rundecision models only | 18.4%83 “I don't know” of 450 not right | Not rundecision models only | 78.9%390 of 494answered 90.3%, non-hallucination 12.0% | 0.0099.00from GPU time | 0.0087.84from GPU time | 0.0111 question per request, 1 in flight | 0.0111 question per request, 1 in flight | 130 | 130 | other languages, not Turkishhuggingface.co |
| Laya Multilingual Open · 0.3B | 17.0% | 15.4% | -1.6 | Not rundecision models only | 62.0%339 of 547 answered | Not rundecision models only | 82.6%280 of 339 | Not rundecision models only | 42.6%208 “I don't know” of 488 not right | Not rundecision models only | 84.9%359 of 423answered 77.3%, non-hallucination 25.7% | 0.0087.74from GPU time | 0.01312.62from GPU time | 0.0091 question per request, 1 in flight | 0.0181 question per request, 1 in flight | 131 | 133 | not statedhuggingface.co |
| GLiNER2.5 Multi Open · 0.3B | 16.8% | 16.3% | -0.5 | Not rundecision models only | 83.2%455 of 547 answered | Not rundecision models only | 83.1%378 of 455 | Not rundecision models only | 19.6%92 “I don't know” of 470 not right | Not rundecision models only | 82.5%428 of 519answered 94.9%, non-hallucination 6.1% | 0.0065.88at the maker's list price | 0.0088.39at the maker's list price | 0.0321 question per request, 1 in flight | 0.0091 question per request, 1 in flight | 132 | 132 | other languages, not Turkishhuggingface.co |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 7.1% | 67.6%historical paired baseline, run 1094 | +60.5 * | Not rundecision models only | 89.2%488 of 547 answered | Not rundecision models only | 33.6%164 of 488 | Not rundecision models only | 16.8%33 “I don't know” of 197 not right | Not rundecision models only | 65.8%75 of 114answered 20.8%, non-hallucination 22.7% | 5.4645,464from GPU time | 0.429429from GPU time | 172.01 question per request, 32 in flight | 7.291 question per request, 32 in flight | 133 | 119 | other languages, not Turkishhuggingface.co |
Ordered by Turkish accuracy, highest first. * marks a difference that is significant after Holm's correction: its paired test (the exact McNemar test, with p adjusted by Holm's method over all 133 English setups) has a Holm-adjusted p below 0.05. Accuracy: right answers over the 547 kept reading questions, the first English run against the Text tab run (Accuracy, Paired test). The “I don't know” columns: each model's first English run with “I don't know” against its Turkish run with the option also written “I don't know” (the Hallucination tab's option-word run, counted on these questions), so only the language differs (Only the language differs). Answered: (right + wrong) / 547; Wrong of answers: wrong / (right + wrong); Non-hallucination: “I don't know” / (“I don't know” + wrong); the difference's paired test is of wrong answers, with p adjusted by Holm's method over the 12 models with both runs. Turkish with “Bilmiyorum”: the Turkish side of the published comparison with “Bilmiyorum” (With “I don't know”) or, for a model outside it, its Hallucination tab run counted on these questions; its wrong of answers with its answered share and non-hallucination rate under it. Not run: no such run for a decision model (the Turkish “I don't know” run is for decision models only); Running: a chat model's run still running. Cost and latency: the Text tab run and the first English run, cost in $ per 1,000 questionsper 1M questions and the median time per request with the request size and the requests in flight under it (Cost, Latency). No list price: a setup on our own hardware whose maker lists no price for the same model. Rank: by accuracy among the 133 setups in each language, 1 the most accurate, tied setups sharing a rank. LFM3-1.7B, LFM3-1.7B and MiniCPM5-2B are run with 3 sampling seeds: until all are in, the number is the first English run's, named with the seeds in so far; then it is the mean of the runs' accuracies, ranked by it, with their lowest and highest (min–max, a range, not a confidence interval) under it. The paired comparison with Turkish and the “I don't know” runs stay on their own runs. Languages: what each model's card or its maker's docs list, with the source (Languages the model cards list); for a setup outside that table, whether its card lists Turkish, as the Turkish leaderboards' ‡ marks read it; Not checked: no card checked yet. The full tables are under Info.
Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. No list price: on our own GPUs, with no estimate. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches.
Turkish vs English
| Model | Accuracy | Cost | Latency | Rank by accuracy | |||||
|---|---|---|---|---|---|---|---|---|---|
| Turkishhigher is better | Englishhigher is better | Differencepoints higher in English than in Turkish (negative: higher in Turkish) | Turkish$ per 1,000 questionsper 1M questions; lower is better | English$ per 1,000 questionsper 1M questions; lower is better | Turkishmedian seconds per request; lower is better | Englishmedian seconds per request; lower is better | Turkishamong the 58; lower is better | Englishamong the 58; lower is better | |
| Claude Fable 5.2 max · chat Closed · Flagship | 97.8% | 98.0% | +0.2 | 53.38153,381subscription, priced as the API | 52.94252,942subscription, priced as the API | 81.116 questions per request, 1 in flight | 74.016 questions per request, 1 in flight | 1 | 1 |
| Gemini 4 Carbon high · chat Closed · Flagship | 97.6% | 94.3% | -3.3 * | 31.30931,309subscription, priced as the API | 38.05338,053subscription, priced as the API | 217.716 questions per request, 1 in flight | 287.816 questions per request, 1 in flight | 2 | 19 |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 97.1% | 96.7% | -0.4 | 4.8064,806subscription, priced as the API | 4.9274,927subscription, priced as the API | 31.716 questions per request, 1 in flight | 28.416 questions per request, 1 in flight | 3 | 2 |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 97.1% | 95.6% | -1.5 | 26.73826,738subscription, priced as the API | 28.65328,653subscription, priced as the API | 6.411 question per request, 1 in flight | 6.431 question per request, 1 in flight | 4 | 12 |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 96.9% | 96.0% | -0.9 | 2.4192,419at list prices | 3.6953,695at list prices | 53.716 questions per request, 4 in flight | 104.616 questions per request, 4 in flight | 5 | 7 |
| Gemini 4 Argon low · chat Closed · Flagship | 96.9% | 96.0% | -0.9 | 1.7861,786at list prices | 1.9001,900at list prices | 5.4616 questions per request, 1 in flight | 6.0316 questions per request, 1 in flight | 6 | 8 |
| GPT-6 Astra high · chat Closed · Flagship | 96.9% | 96.2% | -0.7 | 6.0666,066subscription, priced as the API | 6.3346,334subscription, priced as the API | 21.716 questions per request, 1 in flight | 15.616 questions per request, 1 in flight | 7 | 4 |
| GPT-6 Astra low · chat Closed · Flagship | 96.7% | 96.3% | -0.4 | 5.3785,378subscription, priced as the API | 5.9185,918subscription, priced as the API | 13.016 questions per request, 1 in flight, measured on a sample (200 questions) | 11.516 questions per request, 1 in flight | 8 | 3 |
| GPT-6 Astra medium · chat Closed · Flagship | 96.7% | 96.0% | -0.7 | 5.4475,447subscription, priced as the API | 4.6964,696subscription, priced as the API | 15.316 questions per request, 1 in flight | 10.716 questions per request, 1 in flight | 9 | 9 |
| GPT-6.1 Terra high · chat Closed · Workhorse | 96.3% | 96.2% | -0.2 | 0.515515subscription, priced as the API | 0.509509subscription, priced as the API | 7.1016 questions per request, 1 in flight | 7.9416 questions per request, 1 in flight | 10 | 5 |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 96.2% | 95.2% | -0.9 | 0.963963at list prices | 0.858858at list prices | 2.721 question per request, 1 in flight | 2.851 question per request, 1 in flight | 11 | 14 |
| GPT-6.1 Sol high · chat Closed · Workhorse | 96.2% | 95.6% | -0.5 | 1.6221,622subscription, priced as the API | 1.4271,427subscription, priced as the API | 24.716 questions per request, 1 in flight | 16.916 questions per request, 1 in flight | 12 | 13 |
| GPT-6.1 Sol low · chat Closed · Workhorse | 95.8% | 95.8% | +0.0 | 1.1301,130subscription, priced as the API | 1.2191,219subscription, priced as the API | 13.316 questions per request, 1 in flight | 10.316 questions per request, 1 in flight | 13 | 11 |
| Claude Opus 5.5 high · chat Closed · Flagship | 95.6% | 94.5% | -1.1 | 6.5096,509subscription, priced as the API | 4.5194,519subscription, priced as the API | 22.116 questions per request, 1 in flight | 11.316 questions per request, 1 in flight | 14 | 17 |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 95.6% | 95.2% | -0.4 | 5.5695,569subscription, priced as the API | 6.1136,113subscription, priced as the API | 6.611 question per request, 1 in flight | 5.591 question per request, 1 in flight | 15 | 15 |
| GPT-6.1 Terra low · chat Closed · Workhorse | 95.6% | 96.2% | +0.5 | 0.426426subscription, priced as the API | 0.491491subscription, priced as the API | 4.2416 questions per request, 1 in flight | 7.2916 questions per request, 1 in flight | 16 | 6 |
| Claude Fable 5.1 high · chat Closed · Flagship | 95.4% | 96.0% | +0.5 | 17.04217,042subscription, priced as the API | 11.53111,531subscription, priced as the API | 30.316 questions per request, 1 in flight | 15.616 questions per request, 1 in flight | 17 | 10 |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 95.4% | 94.7% | -0.7 | 11.61011,610subscription, priced as the API | 10.09410,094subscription, priced as the API | 3.711 question per request, 1 in flight | 3.501 question per request, 1 in flight | 18 | 16 |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 95.2% | 94.3% | -0.9 | 0.269269at list prices | 0.288288at list prices | 2.1916 questions per request, 1 in flight | 4.1916 questions per request, 1 in flight | 19 | 20 |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 95.2% | 92.3% | -2.9 | 2.0482,048at list prices | 0.984984at list prices | 44.916 questions per request, 4 in flight | 16.616 questions per request, 4 in flight | 20 | 25 |
| Claude Opus 5.5 low · chat Closed · Flagship | 95.1% | 93.1% | -2.0 | 4.9314,931subscription, priced as the API | 1.0871,087subscription, priced as the API | 13.416 questions per request, 1 in flight, measured on a sample (200 questions) | 7.6416 questions per request, 1 in flight | 21 | 22 |
| Grok 4.7 high · chat Closed · Workhorse | 95.1% | 93.4% | -1.6 | 14.29214,292subscription, priced as the API | 6.8516,851subscription, priced as the API | 401.916 questions per request, 4 in flight | 197.216 questions per request, 4 in flight | 22 | 21 |
| Qwen4-Max high · chat Closed · Workhorse | 94.7% | 94.5% | -0.2 | 2.5332,533at list prices | 1.1191,119at list prices | 91.516 questions per request, 4 in flight | 29.516 questions per request, 4 in flight | 23 | 18 |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 94.5% | 92.5% | -2.0 | 1.2781,278at the maker's list price | 0.421421at the maker's list price | 486.916 questions per request, 8 in flight | 189.416 questions per request, 8 in flight | 24 | 24 |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 94.3% | 91.6% | -2.7 | 7.7487,748subscription, priced as the API | 5.5135,513subscription, priced as the API | 4.641 question per request, 1 in flight | 3.021 question per request, 1 in flight | 25 | 26 |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 94.0% | 91.4% | -2.6 | 0.853853at list prices | 0.735735at list prices | 5.7516 questions per request, 4 in flight | 6.3116 questions per request, 4 in flight | 26 | 27 |
| Gemma 4 31B reasoning off · chat Open · 31B | 92.5% | 90.5% | -2.0 | 0.04848.22at a host's list price for the same weights | 0.05352.81at a host's list price for the same weights | 3.9716 questions per request, 1 in flight | 3.1916 questions per request, 1 in flight | 27 | 29 |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 91.4% | 91.4% | +0.0 | 2.4112,411at list prices | 1.7421,742at list prices | 251.316 questions per request, 4 in flight | 162.916 questions per request, 4 in flight | 28 | 28 |
| Muse Ember max · chat Closed · Efficient | 90.9% | 89.6% | -1.3 | 0.650650at list prices | 0.245245at list prices | 30.916 questions per request, 4 in flight | 11.616 questions per request, 4 in flight | 29 | 31 |
| Qwen3.8-27B low · chat Open · 27B | 90.7% | 89.0% | -1.6 | 3.0953,095at the maker's list price | 0.853853at the maker's list price | 648.416 questions per request, 4 in flight | 71.716 questions per request, 4 in flight | 30 | 33 |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 90.3% | 89.9% | -0.4 | 0.06868.19at a host's list price for the same weights | 0.07070.17at a host's list price for the same weights | 0.2981 question per request, 1 in flight | 0.2991 question per request, 1 in flight | 31 | 30 |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 90.1% | 89.6% | -0.5 | 0.543543at the maker's list price | 0.08181.38at the maker's list price | 102.616 questions per request, 8 in flight | 33.116 questions per request, 8 in flight | 32 | 32 |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 89.9% | 87.0% | -2.9 | 0.09898.47at the maker's list price | 0.07979.31at the maker's list price | 5.0816 questions per request, 1 in flight | 6.5916 questions per request, 1 in flight | 33 | 38 |
| Claude Haiku 5.5 max · chat Closed · Efficient | 89.4% | 92.7% | +3.3 | 2.9652,965at list prices | 1.2611,261at list prices | 417.016 questions per request, 4 in flight | 148.916 questions per request, 4 in flight | 34 | 23 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 89.4% | 88.3% | -1.1 | 1.1371,137at list prices | 0.425425at list prices | 97.716 questions per request, 4 in flight | 25.316 questions per request, 4 in flight | 35 | 35 |
| GLM-5.3 low · chat Open · Workhorse | 89.2% | 87.8% | -1.5 | 0.814814at list prices | 0.408408at list prices | 17.116 questions per request, 4 in flight | 6.8416 questions per request, 4 in flight | 36 | 36 |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 88.8% | 87.0% | -1.8 | 0.04747.13at list prices | 0.05049.69at list prices | 10.016 questions per request, 4 in flight | 10.116 questions per request, 4 in flight | 37 | 39 |
| GPT-6 Luna low · chat Closed · Efficient | 88.7% | 86.3% | -2.4 | 0.04848.01subscription, priced as the API | 0.05857.70subscription, priced as the API | 7.7116 questions per request, 1 in flight | 7.7216 questions per request, 1 in flight | 38 | 41 |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 87.6% | 86.1% | -1.5 | 3.6543,654subscription, priced as the API | 1.9981,998subscription, priced as the API | 20.516 questions per request, 1 in flight, measured on a sample (200 questions) | 3.8516 questions per request, 1 in flight | 39 | 42 |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 86.8% | 89.0% | +2.2 | 2.2192,219at list prices | 0.298298at list prices | 21.41 question per request, 32 in flight | 6.671 question per request, 8 in flight | 40 | 34 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 86.3% | 84.5% | -1.8 | 0.814814at list prices | 0.193193at list prices | 112.616 questions per request, 4 in flight | 26.716 questions per request, 4 in flight | 41 | 45 |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 86.3% | 84.8% | -1.5 | 0.234234at the maker's list price | 0.212212at the maker's list price | 0.4331 question per request, 1 in flight | 0.3901 question per request, 1 in flight | 42 | 43 |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 85.9% | 83.9% | -2.0 | 0.825825at the maker's list price | 0.645645at the maker's list price | 1.621 question per request, 1 in flight | 1.701 question per request, 1 in flight | 43 | 46 |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 84.8% | 83.9% | -0.9 | 1.8961,896subscription, priced as the API | 0.354354subscription, priced as the API | 3.8716 questions per request, 1 in flight, measured on a sample (200 questions) | 3.6816 questions per request, 1 in flight | 44 | 47 |
| Qwen3.8-27B yes/no scoring Open · 27B | 84.6% | 84.8% | +0.2 | 0.704704at the maker's list price | 0.741741at the maker's list price | 0.2801 question per request, 1 in flight | 0.4131 question per request, 1 in flight | 45 | 44 |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 84.3% | 87.2% | +2.9 | 0.04747.28at the maker's list price | 0.04646.07at the maker's list price | 1.0716 questions per request, 1 in flight | 3.3716 questions per request, 1 in flight | 46 | 37 |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 82.8% | 87.0% | +4.2 | 2.9982,998from GPU time | 1.3901,390from GPU time | 140.31 question per request, 32 in flight | 58.31 question per request, 32 in flight | 47 | 40 |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 80.6% | 82.4% | +1.8 | 0.307307from GPU time | 0.199199from GPU time | 13.41 question per request, 32 in flight | 8.391 question per request, 32 in flight | 48 | 48 |
| Qwen3-14B reasoning off · chat Open · 15B | 76.8% | 79.7% | +2.9 | 0.03232.31at a host's list price for the same weights | 0.03131.17at a host's list price for the same weights | 2.8516 questions per request, 1 in flight | 2.8616 questions per request, 1 in flight | 49 | 50 |
| Erk-14B reasoning off · chat Open · 14B | 75.3% | 72.0% | -3.3 | 0.106106from GPU time | 0.123123from GPU time | 2.3916 questions per request, 1 in flight | 2.8616 questions per request, 1 in flight | 50 | 52 |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 72.0% | 80.4% | +8.4 * | 0.894894from GPU time | 0.309309from GPU time | 32.21 question per request, 32 in flight | 9.741 question per request, 32 in flight | 51 | 49 |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 48.1% | 71.1%historical paired baseline, run 1061 | +23.0 * | 1.2771,277at the maker's list price | 0.387387at the maker's list price | 24.21 question per request, 32 in flight | 5.241 question per request, 32 in flight | 52 | 54 |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 47.9% | 73.9% | +26.0 * | 2.7612,761at the maker's list price | 0.964964at the maker's list price | 51.21 question per request, 32 in flight | 15.01 question per request, 32 in flight | 53 | 51 |
| LFM3-1.7B reasoning off · chat Open · 1.7B | 40.2% | 72.9%historical paired baseline, run 1223 | +32.7 * | 0.646646at the maker's list price | 0.355355at the maker's list price | 4.901 question per request, 32 in flight | 4.691 question per request, 32 in flight | 54 | 53 |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 36.7% | 67.3% | +30.5 * | 0.268268from GPU time | 0.280280from GPU time | 8.841 question per request, 32 in flight | 9.271 question per request, 32 in flight | 55 | 57 |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 27.1% | 70.4% | +43.3 * | 1.0501,050from GPU time | 0.517517from GPU time | 39.21 question per request, 32 in flight | 19.41 question per request, 32 in flight | 56 | 55 |
| GLiNER2.5 Multi Open · 0.3B | 16.8% | 16.3% | -0.5 | 0.0065.88at the maker's list price | 0.0088.39at the maker's list price | 0.0321 question per request, 1 in flight | 0.0091 question per request, 1 in flight | 57 | 58 |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 7.1% | 67.6%historical paired baseline, run 1094 | +60.5 * | 5.4645,464from GPU time | 0.429429from GPU time | 172.01 question per request, 32 in flight | 7.291 question per request, 32 in flight | 58 | 56 |
Ordered by Turkish accuracy, highest first. * marks a difference that is significant after Holm's correction: its paired test (the exact McNemar test, with p adjusted by Holm's method over all 133 English setups) has a Holm-adjusted p below 0.05. Accuracy: right answers over the 547 kept reading questions, the first English run against the Text tab run (Accuracy, Paired test). Cost and latency: the Text tab run and the first English run, cost in $ per 1,000 questionsper 1M questions and the median time per request with the request size and the requests in flight under it (Cost, Latency). Rank: by accuracy among the 58 chat model setups in each language, 1 the most accurate, tied setups sharing a rank. LFM3-1.7B, LFM3-1.7B and MiniCPM5-2B are run with 3 sampling seeds: until all are in, the number is the first English run's, named with the seeds in so far; then it is the mean of the runs' accuracies, ranked by it, with their lowest and highest (min–max, a range, not a confidence interval) under it. The paired comparison with Turkish and the “I don't know” runs stay on their own runs. The full tables are under Info.
Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending.
Turkish vs English
| Model | Accuracy | Answered with “I don't know” | Wrong of answers with “I don't know” | Non-hallucination with “I don't know” | Turkish with “Bilmiyorum”wrong of answers, lower is better; answered and non-hallucination under it | Cost | Latency | Rank by accuracy | Languages the model card listsand its source | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Turkishhigher is better | Englishhigher is better | Differencepoints higher in English than in Turkish (negative: higher in Turkish) | Turkish(right + wrong) / 547; higher is better | English(right + wrong) / 547; higher is better | Turkishwrong / (right + wrong); lower is better | Englishwrong / (right + wrong); lower is better | Turkish“I don't know” / (“I don't know” + wrong); higher is better | English“I don't know” / (“I don't know” + wrong); higher is better | Differencenon-hallucination points higher in English than in Turkish (negative: higher in Turkish) | Turkish$ per 1,000 questionsper 1M questions; lower is better | English$ per 1,000 questionsper 1M questions; lower is better | Turkishmedian seconds per request; lower is better | Englishmedian seconds per request; lower is better | Turkishamong the 75; lower is better | Englishamong the 75; lower is better | |||
| Karpo Decision 31B Open · 31B | 94.9% | 92.1% | -2.7 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 7.7%42 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 42 not right | Not rundecision models only | 5.5%30 of 546answered 99.8%, non-hallucination 3.2% | 0.105105from GPU time | 0.145145from GPU time | 0.1591 question per request, 1 in flight | 0.2071 question per request, 1 in flight | 1 | 5 | not statedhuggingface.co |
| Copernicus-31B Open · 31B | 94.5% | 92.9% | -1.6 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 7.1%39 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 39 not right | Not rundecision models only | 6.0%33 of 546answered 99.8%, non-hallucination 2.9% | 0.134134from GPU time | 0.142142from GPU time | 0.1841 question per request, 1 in flight | 0.1881 question per request, 1 in flight | 2 | 2 | English (the card's language: en tag; its text names no language)huggingface.co |
| KnowLine-31B-Gen2 Open · 31B | 94.5% | 92.3% | -2.2 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 7.3%40 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 40 not right | Not rundecision models only | 5.3%29 of 547answered 100.0%, non-hallucination 0.0% | 0.120120from GPU time | 0.127127from GPU time | 0.1681 question per request, 1 in flight | 0.1711 question per request, 1 in flight | 3 | 4 | other languages, not Turkishhuggingface.co |
| KnowLine-31B-Gen1.5 Open · 31B | 94.3% | 93.6% | -0.7 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 6.8%37 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 37 not right | Not rundecision models only | 6.2%34 of 547answered 100.0%, non-hallucination 0.0% | No list price | No list price | 1.051 question per request, 1 in flight | 1.121 question per request, 1 in flight | 4 | 1 | other languages, not Turkishhuggingface.co |
| REAX 1.3 Open · 57B | 94.0% | 92.5% | -1.5 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 7.7%42 of 544 | Not rundecision models only | 6.7%3 “I don't know” of 45 not right | Not rundecision models only | 6.4%35 of 546answered 99.8%, non-hallucination 2.8% | 0.185185from GPU time | 0.169169from GPU time | 0.2501 question per request, 1 in flight | 0.2291 question per request, 1 in flight | 5 | 3 | English (the card's language: en tag; its text names no language)huggingface.co |
| d1-ultra Closed · Efficient | 93.6% | 90.7% | -2.9 | 95.1%520 of 547 answered | 90.7%496 of 547 answered | 5.4%28 of 520 | 7.9%39 of 496 | 49.1%27 “I don't know” of 55 not right | 56.7%51 “I don't know” of 90 not right | +7.6 | 5.0%26 of 517answered 94.5%, non-hallucination 53.6% | 0.01414.34at list prices | 0.01413.98at list prices | 0.2231 question per request, 1 in flight | 0.2201 question per request, 1 in flight | 6 | 9 | not stateddocs.liquid.ai |
| Turion Jev 31B Open · 31B | 93.6% | 90.7% | -2.9 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 7.9%43 of 543 | Not rundecision models only | 8.5%4 “I don't know” of 47 not right | Not rundecision models only | 6.1%33 of 545answered 99.6%, non-hallucination 5.7% | No list price | No list price | 0.4051 question per request, 1 in flight | 0.4451 question per request, 1 in flight | 7 | 10 | not statedhuggingface.co |
| Logits Sigma-1 Duo Open · 57B | 93.4% | 91.0% | -2.4 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 8.1%44 of 545 | Not rundecision models only | 4.3%2 “I don't know” of 46 not right | Not rundecision models only | 6.6%36 of 547answered 100.0%, non-hallucination 0.0% | 0.181181from GPU time | 0.191191from GPU time | 0.2501 question per request, 1 in flight | 0.2631 question per request, 1 in flight | 8 | 6 | not statedhuggingface.co |
| Sieve 31B v3 Open · 31B | 93.4% | 90.7% | -2.7 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 8.2%45 of 546 | Not rundecision models only | 2.2%1 “I don't know” of 46 not right | Not rundecision models only | 6.0%33 of 547answered 100.0%, non-hallucination 0.0% | No list price | No list price | 1.001 question per request, 1 in flight | 1.061 question per request, 1 in flight | 9 | 11 | other languages, not Turkishhuggingface.co |
| Gemini Decision 1 20261111 Closed · Efficient | 92.9% | 91.0% | -1.8 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 9.2%50 of 542 | Not rundecision models only | 9.1%5 “I don't know” of 55 not right | Not rundecision models only | 7.6%41 of 542answered 99.1%, non-hallucination 10.9% | 0.03737.19at list prices | 0.04039.51at list prices | 0.2271 question per request, 1 in flight | 0.2351 question per request, 1 in flight | 10 | 7 | not statedaistudio.google.com |
| Mammoth-1 Open · 27B | 92.9% | 90.1% | -2.7 | 95.4%522 of 547 answered | 93.2%510 of 547 answered | 6.1%32 of 522 | 7.1%36 of 510 | 43.9%25 “I don't know” of 57 not right | 50.7%37 “I don't know” of 73 not right | +6.8 | 6.3%33 of 523answered 95.6%, non-hallucination 42.1% | 0.04039.62at the maker's list price | 0.04040.07at the maker's list price | 0.1691 question per request, 1 in flight | 0.1891 question per request, 1 in flight | 11 | 15 | English (the card's language: en tag; its text names no language)huggingface.co |
| d3-ultra Open · 26B | 91.4% | 89.4% | -2.0 | 99.6%545 of 547 answered | 99.1%542 of 547 answered | 9.4%51 of 545 | 10.3%56 of 542 | 3.8%2 “I don't know” of 53 not right | 8.2%5 “I don't know” of 61 not right | +4.4 | 8.4%46 of 546answered 99.8%, non-hallucination 2.1% | 0.216216from GPU time | 0.106106from GPU time | 0.3331 question per request, 1 in flight | 0.1491 question per request, 1 in flight | 12 | 19 | not statedhuggingface.co |
| Drex Max Open · 27B | 91.4% | 89.6% | -1.8 | 99.8%546 of 547 answered | 99.6%545 of 547 answered | 9.0%49 of 546 | 11.0%60 of 545 | 2.0%1 “I don't know” of 50 not right | 3.2%2 “I don't know” of 62 not right | +1.2 | 9.7%53 of 545answered 99.6%, non-hallucination 3.6% | 0.02323.29at the maker's list price | 0.02423.56at the maker's list price | 0.0821 question per request, 1 in flight | 0.0811 question per request, 1 in flight | 13 | 17 | not statedhuggingface.co |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 91.4% | 88.7% | -2.7 | 99.8%546 of 547 answered | 99.6%545 of 547 answered | 9.3%51 of 546 | 11.7%64 of 545 | 1.9%1 “I don't know” of 52 not right | 3.0%2 “I don't know” of 66 not right | +1.1 | 9.1%50 of 547answered 100.0%, non-hallucination 0.0% | 0.0087.76at the maker's list price | 0.0087.85at the maker's list price | 0.5801 question per request, 1 in flight | 0.1261 question per request, 1 in flight | 14 | 21 | not statedhuggingface.co |
| GLiDE 27B Open · 27B | 91.2% | 90.9% | -0.4 | Not rundecision models only | 96.0%525 of 547 answered | Not rundecision models only | 6.9%36 of 525 | Not rundecision models only | 37.9%22 “I don't know” of 58 not right | Not rundecision models only | 7.2%39 of 538answered 98.4%, non-hallucination 18.8% | No list price | No list price | 0.5421 question per request, 1 in flight | 0.5451 question per request, 1 in flight | 15 | 8 | not statedhuggingface.co |
| d3 Open · 26B | 91.0% | 89.8% | -1.3 | 100.0%547 of 547 answered | 99.8%546 of 547 answered | 9.5%52 of 547 | 10.6%58 of 546 | 0.0%0 “I don't know” of 52 not right | 1.7%1 “I don't know” of 59 not right | +1.7 | 9.7%53 of 547answered 100.0%, non-hallucination 0.0% | 0.07070.21from GPU time | 0.05555.35from GPU time | 0.0921 question per request, 1 in flight | 0.0741 question per request, 1 in flight | 16 | 16 | not statedhuggingface.co |
| Perplexity Decider 27B v1.2 API Open · 27B | 91.0% | 88.5% | -2.6 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 11.6%63 of 544 | Not rundecision models only | 4.5%3 “I don't know” of 66 not right | Not rundecision models only | 9.3%51 of 547answered 100.0%, non-hallucination 0.0% | 0.0087.70at list prices | 0.0087.79at list prices | 0.2541 question per request, 1 in flight | 0.2561 question per request, 1 in flight | 17 | 22 | not statedhuggingface.co |
| d1.2-ultra Closed · Efficient | 90.9% | 90.3% | -0.5 | 89.0%487 of 547 answered | 80.1%438 of 547 answered | 3.7%18 of 487 | 3.0%13 of 438 | 76.9%60 “I don't know” of 78 not right | 89.3%109 “I don't know” of 122 not right | +12.4 | 3.9%19 of 486answered 88.8%, non-hallucination 76.2% | 0.01414.34at list prices | 0.01413.98at list prices | 0.2201 question per request, 1 in flight | 0.2211 question per request, 1 in flight | 18 | 13 | not statedwww.liquid.ai |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 90.7% | 90.5% | -0.2 | Not rundecision models only | 98.2%537 of 547 answered | Not rundecision models only | 10.4%56 of 537 | Not rundecision models only | 15.2%10 “I don't know” of 66 not right | Not rundecision models only | 9.4%51 of 544answered 99.5%, non-hallucination 5.6% | ≤ 1.034≤ 1,034at list prices | ≤ 1.036≤ 1,036at list prices | 0.4821 question per request, 1 in flight | 0.4841 question per request, 1 in flight | 19 | 12 | not statedhuggingface.co |
| Perplexity Decider 27B v1.3 Open · 27B | 90.7% | 89.6% | -1.1 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 9.6%52 of 542 | Not rundecision models only | 8.8%5 “I don't know” of 57 not right | Not rundecision models only | 9.5%52 of 547answered 100.0%, non-hallucination 0.0% | 0.08181.18from GPU time | 0.08382.63from GPU time | 0.1101 question per request, 1 in flight | 0.1151 question per request, 1 in flight | 20 | 18 | not statedhuggingface.co |
| GLiDE v1.2 Open · 27B | 90.5% | 90.3% | -0.2 | 98.4%538 of 547 answered | 94.5%517 of 547 answered | 8.7%47 of 538 | 7.0%36 of 517 | 16.1%9 “I don't know” of 56 not right | 45.5%30 “I don't know” of 66 not right | +29.4 | 8.7%47 of 541answered 98.9%, non-hallucination 11.3% | 0.01414.37at the maker's list price | 0.01514.55at the maker's list price | 0.4131 question per request, 1 in flight | 0.1041 question per request, 1 in flight | 21 | 14 | not statedhuggingface.co |
| Drex Max v1.1 Closed · Efficient (provisional) | 90.1% | 88.5% | -1.6 | 96.7%529 of 547 answered | 92.1%504 of 547 answered | 9.1%48 of 529 | 8.1%41 of 504 | 27.3%18 “I don't know” of 66 not right | 51.2%43 “I don't know” of 84 not right | +23.9 | 9.5%51 of 538answered 98.4%, non-hallucination 15.0% | 0.02323.11at list prices | 0.02323.38at list prices | 0.3131 question per request, 1 in flight | 0.3351 question per request, 1 in flight | 22 | 23 | not statedconsole.nace.ai |
| SubQ Snap Preview Closed · Efficient | 89.9% | 85.9% | -4.0 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.01919.41at list prices | 0.01919.48at list prices | 0.3891 question per request, 1 in flight | 0.3811 question per request, 1 in flight | 23 | 34 | not stateddocs.subq.ai |
| SubQ Snap 1.1 Closed · Efficient | 89.2% | 85.9% | -3.3 | 96.5%528 of 547 answered | 96.2%526 of 547 answered | 8.5%45 of 528 | 12.0%63 of 526 | 29.7%19 “I don't know” of 64 not right | 25.0%21 “I don't know” of 84 not right | -4.7 | 9.7%51 of 528answered 96.5%, non-hallucination 27.1% | 0.01515.41at list prices | 0.01615.59at list prices | 0.3781 question per request, 1 in flight | 0.3811 question per request, 1 in flight | 24 | 35 | not stateddocs.subq.ai |
| Perplexity Decider 27B v1 Open · 27B | 89.0% | 88.5% | -0.5 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.01615.54at list prices | 0.01615.59at list prices | 0.2161 question per request, 1 in flight | 0.2471 question per request, 1 in flight | 25 | 24 | not statedhuggingface.co |
| d1.2 Closed · Efficient | 88.8% | 89.0% | +0.2 | 91.6%501 of 547 answered | 89.2%488 of 547 answered | 8.4%42 of 501 | 7.6%37 of 488 | 52.3%46 “I don't know” of 88 not right | 61.5%59 “I don't know” of 96 not right | +9.2 | 7.1%35 of 494answered 90.3%, non-hallucination 60.2% | 0.0076.83at list prices | 0.0076.66at list prices | 0.2131 question per request, 1 in flight | 0.2191 question per request, 1 in flight | 26 | 20 | not statedwww.liquid.ai |
| d1.1 Closed · Efficient | 88.3% | 88.3% | +0.0 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.0076.83at list prices | 0.0076.66at list prices | 0.1951 question per request, 1 in flight | 0.1731 question per request, 1 in flight | 27 | 25 | not stateddocs.liquid.ai |
| Clef Open · 27B | 88.1% | 86.7% | -1.5 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 13.7%75 of 546 | Not rundecision models only | 1.3%1 “I don't know” of 76 not right | Not rundecision models only | 12.3%67 of 546answered 99.8%, non-hallucination 1.5% | 0.110110at the maker's list price | 0.111111at the maker's list price | 0.2051 question per request, 1 in flight | 0.2201 question per request, 1 in flight | 28 | 29 | not statedhuggingface.co |
| OneJev-27B Open · 27B | 87.8% | 86.3% | -1.5 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 13.2%72 of 544 | Not rundecision models only | 4.0%3 “I don't know” of 75 not right | Not rundecision models only | 11.5%63 of 546answered 99.8%, non-hallucination 1.6% | 0.09999.09from GPU time | 0.106106from GPU time | 0.1251 question per request, 1 in flight | 0.1301 question per request, 1 in flight | 29 | 31 | not statedhuggingface.co |
| KnowLine-12B-Gen1 Open · 12B | 87.6% | 86.7% | -0.9 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 13.9%76 of 546 | Not rundecision models only | 1.3%1 “I don't know” of 77 not right | Not rundecision models only | 12.2%67 of 547answered 100.0%, non-hallucination 0.0% | 0.05252.28from GPU time | 0.05756.86from GPU time | 0.0711 question per request, 1 in flight | 0.0781 question per request, 1 in flight | 30 | 30 | other languages, not Turkishhuggingface.co |
| celeris-1-decision Closed · Efficient | 87.4% | 84.1% | -3.3 | Not rundecision models only | 83.2%455 of 547 answered | Not rundecision models only | 8.1%37 of 455 | Not rundecision models only | 71.3%92 “I don't know” of 129 not right | Not rundecision models only | Not run | 0.01515.28at list prices | 0.01616.09at list prices | 0.4011 question per request, 1 in flight | 0.3901 question per request, 1 in flight | 31 | 46 | not stateddocs.celeris.ai |
| Jev 1.14 Closed · Efficient | 87.2% | 86.1% | -1.1 | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 13.2%72 of 544 | Not rundecision models only | 4.0%3 “I don't know” of 75 not right | Not rundecision models only | 12.7%69 of 544answered 99.5%, non-hallucination 4.2% | 0.02928.84at list prices | 0.02726.64at list prices | 0.2111 question per request, 1 in flight | 0.1811 question per request, 1 in flight | 32 | 33 | "English is the primary training language and where accuracy is currently best. Other languages, including CJK scripts, are handled but not equally well"docs.typesafe.ai |
| Mercury Decide Closed · Efficient | 87.0% | 85.7% | -1.3 | Not rundecision models only | 98.0%536 of 547 answered | Not rundecision models only | 13.2%71 of 536 | Not rundecision models only | 13.4%11 “I don't know” of 82 not right | Not rundecision models only | 13.2%71 of 539answered 98.5%, non-hallucination 10.1% | 0.0087.71at list prices | 0.0087.79at list prices | 0.3131 question per request, 1 in flight | 0.3101 question per request, 1 in flight | 33 | 40 | not statedopenrouter.ai |
| GPT-6 Luna Decisions API Closed · Efficient | 86.7% | 84.1% | -2.6 | Not rundecision models only | 98.2%537 of 547 answered | Not rundecision models only | 17.9%96 of 537 | Not rundecision models only | 9.4%10 “I don't know” of 106 not right | Not rundecision models only | 14.4%78 of 541answered 98.9%, non-hallucination 7.1% | 0.04040.44at list prices | 0.04141.09at list prices | 0.1911 question per request, 4 in flight | 0.3231 question per request, 4 in flight | 34 | 47 | not stateddevelopers.openai.com |
| Jeffan-9B-1211 Open · 9.0B | 86.5% | 87.6% | +1.1 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 13.0%71 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 71 not right | Not rundecision models only | 13.1%71 of 544answered 99.5%, non-hallucination 4.1% | 0.04444.29from GPU time | 0.04645.85from GPU time | 0.0641 question per request, 1 in flight | 0.0671 question per request, 1 in flight | 35 | 26 | not statedhuggingface.co |
| Open-Jev-27B-v1.1 Open · 27B | 86.5% | 85.4% | -1.1 | Not rundecision models only | 97.1%531 of 547 answered | Not rundecision models only | 14.3%76 of 531 | Not rundecision models only | 17.4%16 “I don't know” of 92 not right | Not rundecision models only | 14.4%78 of 541answered 98.9%, non-hallucination 7.1% | 0.271271from GPU time | 0.351351from GPU time | 0.3861 question per request, 1 in flight | 0.4951 question per request, 1 in flight | 36 | 43 | not statedhuggingface.co |
| Jev 1.13 Closed · Efficient | 86.1% | 85.9% | -0.2 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.02928.85at list prices | 0.02726.64at list prices | 0.2431 question per request, 1 in flight | 0.1731 question per request, 1 in flight | 37 | 36 | not stateddocs.typesafe.ai |
| Cygnet Open · 12B | 85.6% | 83.9% | -1.6 | Not rundecision models only | 98.2%537 of 547 answered | Not rundecision models only | 14.7%79 of 537 | Not rundecision models only | 11.2%10 “I don't know” of 89 not right | Not rundecision models only | 14.9%81 of 542answered 99.1%, non-hallucination 5.8% | 0.03939.39at a host's list price for the same weights | 0.04141.42at a host's list price for the same weights | 0.0361 question per request, 1 in flight | 0.0431 question per request, 1 in flight | 38 | 48 | not statedhuggingface.co |
| d1 Closed · Efficient | 85.0% | 86.8% | +1.8 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 12.3%67 of 545 | Not rundecision models only | 2.9%2 “I don't know” of 69 not right | Not rundecision models only | 16.3%89 of 546answered 99.8%, non-hallucination 1.1% | 0.0077.43free tier, at the list price | 0.0076.66at list prices | 0.3051 question per request, 1 in flight | 0.1801 question per request, 1 in flight | 39 | 27 | not stateddocs.liquid.ai |
| Winnow-12B Open · 12B | 85.0% | 83.9% | -1.1 | Not rundecision models only | 97.3%532 of 547 answered | Not rundecision models only | 16.0%85 of 532 | Not rundecision models only | 15.0%15 “I don't know” of 100 not right | Not rundecision models only | 15.8%85 of 539answered 98.5%, non-hallucination 8.6% | 0.04141.40from GPU time | 0.05049.59from GPU time | 0.0591 question per request, 1 in flight | 0.0701 question per request, 1 in flight | 40 | 49 | not statedhuggingface.co |
| Drex 2.0 Open · 35B MoE · 3.0B active | 84.8% | 85.9% | +1.1 | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.01919.15at list prices | 0.01919.42at list prices | 0.3251 question per request, 1 in flight | 0.2831 question per request, 1 in flight | 41 | 37 | not statedhuggingface.co |
| GLiDE 9B Open · 7.9B | 84.3% | 86.3% | +2.0 | Not rundecision models only | 97.4%533 of 547 answered | Not rundecision models only | 13.1%70 of 533 | Not rundecision models only | 16.7%14 “I don't know” of 84 not right | Not rundecision models only | 15.4%83 of 538answered 98.4%, non-hallucination 9.8% | No list price | No list price | 0.2071 question per request, 1 in flight | 0.2071 question per request, 1 in flight | 42 | 32 | not statedhuggingface.co |
| Adjudex 9B v5 Open · 9.0B | 83.9% | 85.7% | +1.8 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 14.1%77 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 77 not right | Not rundecision models only | 16.3%89 of 547answered 100.0%, non-hallucination 0.0% | 0.02626.30from GPU time | 0.02524.93from GPU time | 0.0381 question per request, 1 in flight | 0.0341 question per request, 1 in flight | 43 | 41 | not statedhuggingface.co |
| Cerno 9B v1.2 Open · 9.0B | 83.5% | 85.9% | +2.4 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 14.1%77 of 545 | Not rundecision models only | 2.5%2 “I don't know” of 79 not right | Not rundecision models only | 16.6%91 of 547answered 100.0%, non-hallucination 0.0% | 0.03636.02from GPU time | No list price | 0.0511 question per request, 1 in flight | 0.2111 question per request, 1 in flight | 44 | 38 | not statedhuggingface.co |
| Elephant-1 Open · 9.0B | 83.5% | 85.4% | +1.8 | 95.6%523 of 547 answered | 95.8%524 of 547 answered | 15.7%82 of 523 | 12.8%67 of 524 | 22.6%24 “I don't know” of 106 not right | 25.6%23 “I don't know” of 90 not right | +2.9 | 15.4%82 of 532answered 97.3%, non-hallucination 15.5% | 0.01413.87at the maker's list price | 0.01414.02at the maker's list price | 0.0741 question per request, 1 in flight | 0.0701 question per request, 1 in flight | 45 | 44 | English (the card's language: en tag; its text names no language)huggingface.co |
| Dexy-Decision-9B-V2 Open · 9.0B | 83.4% | 83.5% | +0.2 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 16.5%90 of 546 | Not rundecision models only | 1.1%1 “I don't know” of 91 not right | Not rundecision models only | 16.5%90 of 546answered 99.8%, non-hallucination 1.1% | 0.02323.03from GPU time | 0.02525.41from GPU time | 0.0311 question per request, 1 in flight | 0.0361 question per request, 1 in flight | 46 | 52 | not statedhuggingface.co |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 83.0% | 86.8% | +3.8 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 13.6%74 of 545 | Not rundecision models only | 2.6%2 “I don't know” of 76 not right | Not rundecision models only | 17.4%95 of 546answered 99.8%, non-hallucination 1.0% | No list price | No list price | 0.1541 question per request, 1 in flight | 0.1561 question per request, 1 in flight | 47 | 28 | other languages, not Turkishhuggingface.co |
| KnowLine-9B-Gen2 Open · 9.0B | 82.8% | 85.2% | +2.4 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 14.3%78 of 546 | Not rundecision models only | 1.3%1 “I don't know” of 79 not right | Not rundecision models only | 17.4%95 of 547answered 100.0%, non-hallucination 0.0% | 0.03535.40from GPU time | 0.03939.48from GPU time | 0.0481 question per request, 1 in flight | 0.0561 question per request, 1 in flight | 48 | 45 | other languages, not Turkishhuggingface.co |
| KnowLine-9B-Gen1 Open · 9.0B | 82.4% | 85.9% | +3.5 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 13.9%76 of 546 | Not rundecision models only | 1.3%1 “I don't know” of 77 not right | Not rundecision models only | 17.9%98 of 547answered 100.0%, non-hallucination 0.0% | 0.04140.69from GPU time | 0.04241.55from GPU time | 0.0571 question per request, 1 in flight | 0.0591 question per request, 1 in flight | 49 | 39 | other languages, not Turkishhuggingface.co |
| vera-medium-omni Open · 4.2B | 79.5% | 83.5% | +4.0 | Not rundecision models only | 98.7%540 of 547 answered | Not rundecision models only | 16.1%87 of 540 | Not rundecision models only | 7.4%7 “I don't know” of 94 not right | Not rundecision models only | 20.8%113 of 542answered 99.1%, non-hallucination 4.2% | 0.02120.72from GPU time | 0.03736.63from GPU time | 0.0291 question per request, 1 in flight | 0.0541 question per request, 1 in flight | 50 | 53 | English (the card's language: en tag; its text names no language)huggingface.co |
| Valen-Decision-1.5-9B Open · 9.0B | 79.0% | 83.7% | +4.8 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 15.4%84 of 546 | Not rundecision models only | 1.2%1 “I don't know” of 85 not right | Not rundecision models only | 20.1%110 of 547answered 100.0%, non-hallucination 0.0% | 0.04040.49from GPU time | 0.04140.55from GPU time | 0.0591 question per request, 1 in flight | 0.0591 question per request, 1 in flight | 51 | 50 | not statedhuggingface.co |
| Perplexity Decider Flash v1.3 Open · 4.0B | 78.8% | 83.7% | +4.9 | Not rundecision models only | 98.7%540 of 547 answered | Not rundecision models only | 16.7%90 of 540 | Not rundecision models only | 7.2%7 “I don't know” of 97 not right | Not rundecision models only | 20.4%110 of 538answered 98.4%, non-hallucination 7.6% | 0.02524.69from GPU time | 0.02323.17from GPU time | 0.0331 question per request, 1 in flight | 0.0331 question per request, 1 in flight | 52 | 51 | not statedhuggingface.co |
| KnowLine 4B Gen5.5 Open · 4.0B | 78.6% | 80.8% | +2.2 | Not rundecision models only | 99.8%546 of 547 answered | Not rundecision models only | 19.2%105 of 546 | Not rundecision models only | 0.9%1 “I don't know” of 106 not right | Not rundecision models only | 21.7%118 of 544answered 99.5%, non-hallucination 2.5% | No list price | No list price | 0.0991 question per request, 1 in flight | 0.0991 question per request, 1 in flight | 53 | 58 | other languages, not Turkishhuggingface.co |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 78.4% | 82.8% | +4.4 | Not rundecision models only | 100.0%547 of 547 answered | Not rundecision models only | 17.2%94 of 547 | Not rundecision models only | 0.0%0 “I don't know” of 94 not right | Not rundecision models only | 21.7%118 of 545answered 99.6%, non-hallucination 1.7% | No list price | No list price | 0.0741 question per request, 1 in flight | 0.0751 question per request, 1 in flight | 54 | 54 | other languages, not Turkishhuggingface.co |
| Clef-Flash Open · 9.0B | 77.7% | 82.3% | +4.6 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 16.6%90 of 542 | Not rundecision models only | 5.3%5 “I don't know” of 95 not right | Not rundecision models only | 21.5%115 of 535answered 97.8%, non-hallucination 9.4% | 0.01717.37at the maker's list price | 0.01817.53at the maker's list price | 0.0751 question per request, 1 in flight | 0.1401 question per request, 1 in flight | 55 | 55 | not statedhuggingface.co |
| Verdin-4B-v1.1 Open · 4.0B | 77.5% | 82.3% | +4.8 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 17.2%94 of 545 | Not rundecision models only | 2.1%2 “I don't know” of 96 not right | Not rundecision models only | 23.3%127 of 546answered 99.8%, non-hallucination 0.8% | 0.03434.21from GPU time | 0.03434.28from GPU time | 0.0491 question per request, 1 in flight | 0.0521 question per request, 1 in flight | 56 | 56 | other languages, not Turkishhuggingface.co |
| Perplexity Decider Flash v1.2 Open · 4.0B | 76.6% | 81.9% | +5.3 | Not rundecision models only | 99.1%542 of 547 answered | Not rundecision models only | 18.5%100 of 542 | Not rundecision models only | 4.8%5 “I don't know” of 105 not right | Not rundecision models only | 24.6%133 of 541answered 98.9%, non-hallucination 4.3% | 0.0043.88at the maker's list price | 0.0043.93at the maker's list price | 0.1321 question per request, 1 in flight | 0.0371 question per request, 1 in flight | 57 | 57 | not statedhuggingface.co |
| Microsoft-Decision-1 Closed · Efficient | 76.4% | 85.7% | +9.3 * | Not rundecision models only | 99.5%544 of 547 answered | Not rundecision models only | 14.2%77 of 544 | Not rundecision models only | 3.8%3 “I don't know” of 80 not right | Not rundecision models only | 16.1%88 of 545answered 99.6%, non-hallucination 2.2% | 0.01413.82at list prices | 0.01414.01at list prices | 0.2861 question per request, 1 in flight | 0.3151 question per request, 1 in flight | 58 | 42 | not statedopenrouter.ai |
| Chinese Decision Pointer 4B Open · 4.0B | 75.7% | 80.4% | +4.8 | Not rundecision models only | 99.3%543 of 547 answered | Not rundecision models only | 19.2%104 of 543 | Not rundecision models only | 3.7%4 “I don't know” of 108 not right | Not rundecision models only | 23.8%129 of 543answered 99.3%, non-hallucination 3.0% | 0.02524.58from GPU time | 0.02727.21from GPU time | 0.0351 question per request, 1 in flight | 0.0381 question per request, 1 in flight | 59 | 59 | other languages, not Turkishhuggingface.co |
| Metask-Jev-4B Open · 4.5B | 67.1% | 77.1% | +10.1 * | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 22.6%123 of 545 | Not rundecision models only | 1.6%2 “I don't know” of 125 not right | Not rundecision models only | 34.3%187 of 545answered 99.6%, non-hallucination 1.1% | 0.06362.82from GPU time | 0.05959.11from GPU time | 0.0901 question per request, 1 in flight | 0.0851 question per request, 1 in flight | 60 | 60 | Turkish listedhuggingface.co |
| KnowLine-2B-Gen1 Open · 2.0B | 58.0% | 71.3% | +13.3 * | Not rundecision models only | 93.1%509 of 547 answered | Not rundecision models only | 26.5%135 of 509 | Not rundecision models only | 22.0%38 “I don't know” of 173 not right | Not rundecision models only | Not run | No list price | No list price | 0.0621 question per request, 1 in flight | 0.0621 question per request, 1 in flight | 61 | 61 | other languages, not Turkishhuggingface.co |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 55.8% | 69.7% | +13.9 * | Not rundecision models only | 94.1%515 of 547 answered | Not rundecision models only | 30.5%157 of 515 | Not rundecision models only | 16.9%32 “I don't know” of 189 not right | Not rundecision models only | 45.7%235 of 514answered 94.0%, non-hallucination 12.3% | No list price | No list price | 0.0401 question per request, 1 in flight | 0.0401 question per request, 1 in flight | 62 | 63 | other languages, not Turkishhuggingface.co |
| GLiDE 2B Open · 1.9B | 54.3% | 70.0% | +15.7 * | Not rundecision models only | 91.4%500 of 547 answered | Not rundecision models only | 26.4%132 of 500 | Not rundecision models only | 26.3%47 “I don't know” of 179 not right | Not rundecision models only | 43.5%209 of 481answered 87.9%, non-hallucination 24.0% | No list price | No list price | 0.0341 question per request, 1 in flight | 0.0351 question per request, 1 in flight | 63 | 62 | not statedhuggingface.co |
| GLiDE 0.8B Open · 0.8B | 46.1% | 60.7% | +14.6 * | Not rundecision models only | 91.0%498 of 547 answered | Not rundecision models only | 35.5%177 of 498 | Not rundecision models only | 21.7%49 “I don't know” of 226 not right | Not rundecision models only | 48.6%230 of 473answered 86.5%, non-hallucination 24.3% | No list price | No list price | 0.0221 question per request, 1 in flight | 0.0231 question per request, 1 in flight | 64 | 65 | not statedhuggingface.co |
| OneJev1.6-0.8B Open · 0.8B | 42.8% | 59.4% | +16.6 * | Not rundecision models only | 86.5%473 of 547 answered | Not rundecision models only | 39.5%187 of 473 | Not rundecision models only | 28.4%74 “I don't know” of 261 not right | Not rundecision models only | 58.2%253 of 435answered 79.5%, non-hallucination 30.7% | No list price | No list price | 0.0171 question per request, 1 in flight | 0.0181 question per request, 1 in flight | 65 | 67 | not statedhuggingface.co |
| d1-2B Open · 1.9B | 41.9% | 65.6% | +23.8 * | Not rundecision models only | 72.4%396 of 547 answered | Not rundecision models only | 24.5%97 of 396 | Not rundecision models only | 60.9%151 “I don't know” of 248 not right | Not rundecision models only | 53.6%207 of 386answered 70.6%, non-hallucination 43.8% | 0.01313.17from GPU time | 0.0088.08from GPU time | 0.0191 question per request, 1 in flight | 0.0111 question per request, 1 in flight | 66 | 64 | other languages, not Turkishhuggingface.co |
| KnowLine-0.8B-Gen1 Open · 0.8B | 40.2% | 58.9% | +18.6 * | Not rundecision models only | 95.2%521 of 547 answered | Not rundecision models only | 41.7%217 of 521 | Not rundecision models only | 10.7%26 “I don't know” of 243 not right | Not rundecision models only | 61.1%325 of 532answered 97.3%, non-hallucination 4.4% | No list price | No list price | 0.0331 question per request, 1 in flight | 0.0331 question per request, 1 in flight | 67 | 69 | other languages, not Turkishhuggingface.co |
| Strands Decider 2B Open · 2.0B | 39.1% | 45.5% | +6.4 | Not rundecision models only | 99.6%545 of 547 answered | Not rundecision models only | 55.2%301 of 545 | Not rundecision models only | 0.7%2 “I don't know” of 303 not right | Not rundecision models only | 60.3%327 of 542answered 99.1%, non-hallucination 1.5% | 0.02020.12from GPU time | 0.01818.15from GPU time | 0.0261 question per request, 1 in flight | 0.0261 question per request, 1 in flight | 68 | 72 | not statedhuggingface.co |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 37.3% | 59.6% | +22.3 * | Not rundecision models only | 97.8%535 of 547 answered | Not rundecision models only | 40.6%217 of 535 | Not rundecision models only | 5.2%12 “I don't know” of 229 not right | Not rundecision models only | 61.0%320 of 525answered 96.0%, non-hallucination 6.4% | No list price | No list price | 0.0281 question per request, 1 in flight | 0.0281 question per request, 1 in flight | 69 | 66 | other languages, not Turkishhuggingface.co |
| Decider 1 Closed · Efficient | 36.6% | 55.8% | +19.2 * | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | Not rundecision models only | Not run | 0.01514.76at list prices | 0.01514.95at list prices | 0.3821 question per request, 1 in flight | 0.6991 question per request, 1 in flight | 70 | 71 | not statedmeragpt.com |
| GLiClass v4 Edge Open · 0.4B | 26.9% | 58.3% | +31.4 * | Not rundecision models only | 90.5%495 of 547 answered | Not rundecision models only | 39.6%196 of 495 | Not rundecision models only | 21.0%52 “I don't know” of 248 not right | Not rundecision models only | 72.0%352 of 489answered 89.4%, non-hallucination 14.1% | No list price | No list price | 0.0251 question per request, 1 in flight | 0.0171 question per request, 1 in flight | 71 | 70 | other languages, not Turkishhuggingface.co |
| GLiDE 0.4B Open · 0.4B | 25.8% | 59.4% | +33.6 * | Not rundecision models only | 72.0%394 of 547 answered | Not rundecision models only | 34.3%135 of 394 | Not rundecision models only | 53.1%153 “I don't know” of 288 not right | Not rundecision models only | 76.4%55 of 72answered 13.2%, non-hallucination 89.6% | No list price | No list price | 0.0161 question per request, 1 in flight | 0.0141 question per request, 1 in flight | 72 | 68 | not statedhuggingface.co |
| GLiNER2.5-multi-Decide Open · 0.3B | 20.7% | 21.2% | +0.5 | Not rundecision models only | 87.2%477 of 547 answered | Not rundecision models only | 76.1%363 of 477 | Not rundecision models only | 16.2%70 “I don't know” of 433 not right | Not rundecision models only | 78.8%412 of 523answered 95.6%, non-hallucination 5.5% | 0.0066.27from GPU time | 0.0076.73from GPU time | 0.0091 question per request, 1 in flight | 0.0091 question per request, 1 in flight | 73 | 74 | other languages, not Turkishhuggingface.co |
| Laya Open · 0.4B | 20.5% | 22.9% | +2.4 | Not rundecision models only | 84.8%464 of 547 answered | Not rundecision models only | 79.1%367 of 464 | Not rundecision models only | 18.4%83 “I don't know” of 450 not right | Not rundecision models only | 78.9%390 of 494answered 90.3%, non-hallucination 12.0% | 0.0099.00from GPU time | 0.0087.84from GPU time | 0.0111 question per request, 1 in flight | 0.0111 question per request, 1 in flight | 74 | 73 | other languages, not Turkishhuggingface.co |
| Laya Multilingual Open · 0.3B | 17.0% | 15.4% | -1.6 | Not rundecision models only | 62.0%339 of 547 answered | Not rundecision models only | 82.6%280 of 339 | Not rundecision models only | 42.6%208 “I don't know” of 488 not right | Not rundecision models only | 84.9%359 of 423answered 77.3%, non-hallucination 25.7% | 0.0087.74from GPU time | 0.01312.62from GPU time | 0.0091 question per request, 1 in flight | 0.0181 question per request, 1 in flight | 75 | 75 | not statedhuggingface.co |
Ordered by Turkish accuracy, highest first. * marks a difference that is significant after Holm's correction: its paired test (the exact McNemar test, with p adjusted by Holm's method over all 133 English setups) has a Holm-adjusted p below 0.05. Accuracy: right answers over the 547 kept reading questions, the first English run against the Text tab run (Accuracy, Paired test). The “I don't know” columns: each model's first English run with “I don't know” against its Turkish run with the option also written “I don't know” (the Hallucination tab's option-word run, counted on these questions), so only the language differs (Only the language differs). Answered: (right + wrong) / 547; Wrong of answers: wrong / (right + wrong); Non-hallucination: “I don't know” / (“I don't know” + wrong); the difference's paired test is of wrong answers, with p adjusted by Holm's method over the 12 models with both runs. Turkish with “Bilmiyorum”: the Turkish side of the published comparison with “Bilmiyorum” (With “I don't know”) or, for a model outside it, its Hallucination tab run counted on these questions; its wrong of answers with its answered share and non-hallucination rate under it. Not run: no such run for a decision model (the Turkish “I don't know” run is for decision models only); Running: a chat model's run still running. Cost and latency: the Text tab run and the first English run, cost in $ per 1,000 questionsper 1M questions and the median time per request with the request size and the requests in flight under it (Cost, Latency). No list price: a setup on our own hardware whose maker lists no price for the same model. Rank: by accuracy among the 75 decision models in each language, 1 the most accurate, tied setups sharing a rank. Languages: what each model's card or its maker's docs list, with the source (Languages the model cards list); for a setup outside that table, whether its card lists Turkish, as the Turkish leaderboards' ‡ marks read it; Not checked: no card checked yet. The full tables are under Info.
Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. No list price: on our own GPUs, with no estimate. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches.
English-only setups (no Turkish run, so in no paired test and no Holm adjustment): Gemma 4 12B Unified · default reasoning · chat: English 86.8% (475 of 547 right), Turkish queued (in logic's GPU 1 queue (Ümit #6527, logic #6955)), celeris-1.1-decision: English 83.7% (458 of 547 right), Turkish Not run, LFM2.5-24B-A2B · default reasoning · chat: English 82.6% (452 of 547 right), Turkish queued (not scheduled yet: needs the model served again (Ümit #6527)), MiniCPM5-2B · reasoning on · chat: English 78.1% (the mean of 3 seeds, 76.6%–79.0% min–max of 3 seeds), Turkish Not run, Gemma 4 E2B · reasoning on · chat: English 76.2% (417 of 547 right), Turkish Not run, Qwen3.5-2B · reasoning on · chat: English 75.9% (415 of 547 right), Turkish Not run, Gemma 4 E2B · default reasoning · chat: English 74.6% (408 of 547 right), Turkish queued (in logic's GPU 1 queue (Ümit #6527, logic #6955)), Granite-4.2-3B · reasoning on · chat: English 74.4% (407 of 547 right), Turkish paused (its Text run stopped at 741 of 2,198 so that other setups go first on logic's GPU 1; it resumes when logic loads Granite again (logic #7481, #7599)) and Raon-1 nano · reasoning off · chat: English 53.7% (294 of 547 right), Turkish queued (asked on its next load in logic's GPU 1 queue, whichever way its smoke came out (run 1154: 3 of 20 replies at the 32,768-token limit, 5 without a letter; Ümit #6515 via Zeki #8211)).
English-only setups (no Turkish run, so in no paired test and no Holm adjustment): Gemma 4 12B Unified · default reasoning · chat: English 86.8% (475 of 547 right), Turkish queued (in logic's GPU 1 queue (Ümit #6527, logic #6955)), LFM2.5-24B-A2B · default reasoning · chat: English 82.6% (452 of 547 right), Turkish queued (not scheduled yet: needs the model served again (Ümit #6527)), MiniCPM5-2B · reasoning on · chat: English 78.1% (the mean of 3 seeds, 76.6%–79.0% min–max of 3 seeds), Turkish Not run, Gemma 4 E2B · reasoning on · chat: English 76.2% (417 of 547 right), Turkish Not run, Qwen3.5-2B · reasoning on · chat: English 75.9% (415 of 547 right), Turkish Not run, Gemma 4 E2B · default reasoning · chat: English 74.6% (408 of 547 right), Turkish queued (in logic's GPU 1 queue (Ümit #6527, logic #6955)), Granite-4.2-3B · reasoning on · chat: English 74.4% (407 of 547 right), Turkish paused (its Text run stopped at 741 of 2,198 so that other setups go first on logic's GPU 1; it resumes when logic loads Granite again (logic #7481, #7599)) and Raon-1 nano · reasoning off · chat: English 53.7% (294 of 547 right), Turkish queued (asked on its next load in logic's GPU 1 queue, whichever way its smoke came out (run 1154: 3 of 20 replies at the 32,768-token limit, 5 without a letter; Ümit #6515 via Zeki #8211)).
English-only setups (no Turkish run, so in no paired test and no Holm adjustment): celeris-1.1-decision: English 83.7% (458 of 547 right), Turkish Not run.
With an “I don't know” option
The same 547 kept questions with a sixth option, “I don't know”, in English and in Turkish, as on the Hallucination tab, asked of the same models (how).
62 of the 123 models with both runs had a higher non-hallucination rate in English than in Turkish; 55 lower; six the same; 18 of the 123 differences in wrong answers are significant after Holm's correction within each group (decision models, chat models): d1-2B (more wrong answers in Turkish), GLiDE 0.4B (more wrong answers in English), Laya Multilingual (more wrong answers in Turkish), OneJev1.6-0.8B (more wrong answers in Turkish), LFM2.5-8B-A1B · default reasoning · chat (more wrong answers in Turkish), GLiDE 2B (more wrong answers in Turkish), GLiDE 0.8B (more wrong answers in Turkish), GLiClass v4 Edge (more wrong answers in Turkish), GLiNER2.5 Multi (more wrong answers in Turkish), StartLux-Decision-v1.1-2B-Instant (more wrong answers in Turkish), MiniCPM5-2B · reasoning off · chat (more wrong answers in English), GLiNER2.5-multi-Decide (more wrong answers in Turkish), LFM2.5-2.6B · default reasoning · chat (more wrong answers in Turkish), LFM3-1.7B · reasoning on · chat (more wrong answers in Turkish), KnowLine-0.8B-Gen1 (more wrong answers in Turkish), Qwen3.5-2B · reasoning off · chat (more wrong answers in Turkish), StartLux-Decision-v1.1-0.8B-Instant (more wrong answers in Turkish) and Metask-Jev-4B (more wrong answers in Turkish) (Paired test). celeris-1.1-decision has an English rate of 74.8% (98 “I don't know” of 131 not right) with an attempt rate of 82.1% (449 answered) but no Turkish run with the option on these questions, so no comparison or test. celeris-1-decision has an English rate of 71.3% (92 “I don't know” of 129 not right) with an attempt rate of 83.2% (455 answered) but no Turkish run with the option on these questions, so no comparison or test. Raon-1 nano · reasoning off · chat has an English rate of 26.2% (71 “I don't know” of 271 not right) with an attempt rate of 86.8% (475 answered) but no Turkish run with the option on these questions, so no comparison or test. KnowLine-2B-Gen1 has an English rate of 22.0% (38 “I don't know” of 173 not right) with an attempt rate of 93.1% (509 answered) but no Turkish run with the option on these questions, so no comparison or test. MiniCPM5-2B · reasoning on · chat has an English rate of 19.7% (28 “I don't know” of 142 not right) with an attempt rate of 94.1% (515 answered) but no Turkish run with the option on these questions, so no comparison or test. Granite-4.2-3B · reasoning on · chat has an English rate of 16.1% (23 “I don't know” of 143 not right) with an attempt rate of 95.8% (524 answered) but no Turkish run with the option on these questions, so no comparison or test. LFM2.5-24B-A2B · default reasoning · chat has an English rate of 13.2% (14 “I don't know” of 106 not right) with an attempt rate of 97.4% (533 answered) but no Turkish run with the option on these questions, so no comparison or test. Gemma 4 E2B · default reasoning · chat has an English rate of 11.5% (17 “I don't know” of 148 not right) with an attempt rate of 96.5% (528 answered) but no Turkish run with the option on these questions, so no comparison or test. Gemma 4 E2B · reasoning on · chat has an English rate of 11.3% (17 “I don't know” of 150 not right) with an attempt rate of 96.3% (527 answered) but no Turkish run with the option on these questions, so no comparison or test. Qwen3.5-2B · reasoning on · chat has an English rate of 9.9% (14 “I don't know” of 142 not right) with an attempt rate of 97.1% (531 answered) but no Turkish run with the option on these questions, so no comparison or test. Gemma 4 12B Unified · default reasoning · chat has an English rate of 5.5% (4 “I don't know” of 73 not right) with an attempt rate of 98.9% (541 answered) but no Turkish run with the option on these questions, so no comparison or test. Still running in Turkish, so charted with one side and in no test: Claude Fable 5.2 · max · chat. Read each rate with its attempt rate: a model that always chose the option would reach 100%.
With the option written “I don't know” in Turkish too, so that only the language differs, the decision models' comparison is in the next section.
Non-hallucination rate in Turkish and in English
The option among the answers that were not right, on the 547 kept questions: each model's first run in each language, ordered by the English rate · higher is better, read with the attempt rate (under the chart and in each bar's tooltip)
Turkish option, the upper bar English option, the lower bar, striped Numbers on the right: Turkish · English. Liquid Fastino Drex Mammoth Laya Claude OneJev Grok Elephant Gemini SubQ GLiClass GPT Open-Jev StartLux MiniCPM Perplexity Winnow Inception Cygnet KnowLine Qwen Gemma Turion Jev vLLM-SR Kimi vera DeepSeek REAX GLM Clef Muse Logits Jev Microsoft Chinese Decision Pointer MiniMax Cerno Sieve Erk Verdin MiMo Metask Valen Dexy Strands Adjudex Copernicus Inkling Jeffan Karpo Attempt rate, Turkish · English: d1.2-ultra 88.8% · 80.1%; d1.2 90.3% · 89.2%; d1-2B 70.6% · 72.4%; d1-ultra 94.5% · 90.7%; GLiDE 0.4B 13.2% · 72.0%; Drex Max v1.1 98.4% · 92.1%; Mammoth-1 95.6% · 93.2%; GLiDE · v1.2 98.9% · 94.5%; Laya Multilingual 77.3% · 62.0%; GLiDE 27B 98.4% · 96.0%; Claude Fable 5.2 · max · chat running · 99.3%; OneJev1.6-0.8B 79.5% · 86.5%; Grok 4.7 · high · chat 98.5% · 97.8%; LFM2.5-8B-A1B · default reasoning · chat 52.5% · 90.5%; GLiDE 2B 87.9% · 91.4%; Elephant-1 97.3% · 95.8%; Gemini 4 Carbon · high · chat 99.3% · 99.3%; SubQ Snap 1.1 96.5% · 96.2%; GLiDE 0.8B 86.5% · 91.0%; GLiClass v4 Edge 89.4% · 90.5%; Gemini 4 Hydrogen · high · chat 99.3% · 99.3%; GLiNER2.5 Multi 94.9% · 83.2%; GPT-6 Astra · high · chat 99.3% · 99.3%; Gemini 4 Argon · low · chat 99.3% · 99.1%; Laya 90.3% · 84.8%; GPT-6 Astra · low · chat 99.3% · 99.3%; Open-Jev-27B-v1.1 98.9% · 97.1%; StartLux-Decision-v1.1-2B-Instant 94.0% · 94.1%; MiniCPM5-2B · reasoning off · chat 20.8% · 89.2%; GLiDE 9B 98.4% · 97.4%; GPT-6 Astra · medium · chat 99.3% · 99.3%; GPT-6.1 Sol · high · chat 99.3% · 99.3%; GPT-6.1 Terra · high · chat 99.1% · 99.3%; GLiNER2.5-multi-Decide 95.6% · 87.2%; LFM2.5-2.6B · default reasoning · chat 90.7% · 95.6%; LFM3-1.7B · reasoning on · chat 80.6% · 95.2%; GPT-6.1 Terra · low · chat 99.3% · 99.3%; Perplexity Decider 27B v1.2 · API · with web 99.5% · 98.2%; Winnow-12B 98.5% · 97.3%; GPT-6 Astra · low · chat · 1 per request 99.3% · 99.3%; GPT-6.1 Sol · low · chat 99.3% · 99.3%; Mercury Decide 98.5% · 98.0%; GPT-6.1 Sol · low · chat · 1 per request 99.3% · 99.3%; LFM3-1.7B · reasoning off · chat 56.9% · 96.7%; Cygnet 99.1% · 98.2%; KnowLine-0.8B-Gen1 97.3% · 95.2%; Qwen3.5-2B · reasoning off · chat 83.4% · 96.3%; Gemini 3.8 Flash · high · chat 99.6% · 99.6%; Gemma 4 E4B · default reasoning · chat 95.4% · 97.8%; GPT-6 Luna · Decisions API 98.9% · 98.2%; Gemini Decision 1 · 20261111 99.1% · 99.1%; Perplexity Decider 27B v1.3 100.0% · 99.1%; Turion Jev 31B 99.6% · 99.3%; d3-ultra 99.8% · 99.1%; Claude Haiku 5.5 · max · chat 96.0% · 96.3%; Gemini 3.8 Flash · low · chat · 1 per request 99.5% · 99.6%; Kimi K3.1 Pro · no reasoning · chat 99.5% · 99.3%; vera-medium-omni 99.1% · 98.7%; Claude Fable 5.1 · high · chat 99.3% · 99.6%; Kimi K3.1 Pro · low · chat 99.5% · 99.5%; Perplexity Decider Flash v1.3 98.4% · 98.7%; DeepSeek-V4.1-Flash · reasoning on · chat · 1017 99.8% · 99.5%; GPT-6 Luna · low · chat 99.6% · 99.1%; REAX 1.3 99.8% · 99.5%; Claude Opus 5.5 · high · chat 99.5% · 99.6%; Qwen3.5-397B-A17B · high · chat 99.3% · 99.5%; GLM-5.3-Flash · reasoning off · chat 99.5% · 99.3%; Qwen3.8 Flash · no reasoning · chat 99.5% · 99.3%; DeepSeek-V4.1-Flash · reasoning off · chat · 1017 100.0% · 99.3%; Clef-Flash 97.8% · 99.1%; Muse Ember · max · chat 99.3% · 99.5%; StartLux-Decision-v1.1-0.8B-Instant 96.0% · 97.8%; Claude Sonnet 5.5 · high · chat 99.6% · 99.5%; Perplexity Decider Flash v1.2 98.9% · 99.1%; Claude Opus 5.5 · low · chat 99.6% · 99.6%; Perplexity Decider 27B v1.2 · API 100.0% · 99.5%; Claude Sonnet 5.5 · low · chat 99.1% · 99.3%; Logits Sigma-1 Duo 100.0% · 99.6%; Jev 1.14 99.5% · 99.5%; OneJev-27B 99.8% · 99.5%; Microsoft-Decision-1 99.6% · 99.5%; Chinese Decision Pointer 4B 99.3% · 99.3%; Qwen3.8-27B · stated confidence · reasoning off · chat 99.3% · 99.5%; Gemma 4 31B · reasoning off · chat 99.6% · 99.6%; Qwen3.5-4B · reasoning off · chat 98.4% · 99.3%; Qwen3.8-27B · low · chat 99.8% · 99.6%; Gemini 3.8 Flash · low · chat 99.5% · 99.8%; GLM-5.3 · low · chat 99.5% · 99.6%; Qwen3.8-27B · vote share · reasoning off · chat 99.3% · 99.5%; MiniMax-M3.1-Flash · high · chat 99.6% · 99.5%; Drex Max 99.6% · 99.6%; Perplexity Decider 27B v1.2 · DGX Spark 100.0% · 99.6%; d1 99.8% · 99.6%; StartLux-Decision-v1.1-9B-Instant 99.8% · 99.6%; Cerno 9B v1.2 100.0% · 99.6%; Qwen3.5-4B · reasoning on · chat 99.1% · 99.6%; Sieve 31B v3 100.0% · 99.8%; Erk-14B · reasoning off · chat 99.8% · 99.5%; Verdin-4B-v1.1 99.8% · 99.6%; Qwen3-14B · reasoning off · chat 99.8% · 99.6%; d3 100.0% · 99.8%; MiMo-V2.6-Flash · high · chat 99.5% · 99.8%; GLM-5.3-Flash · reasoning on · chat 99.6% · 99.8%; Metask-Jev-4B 99.6% · 99.6%; Clef 99.8% · 99.8%; KnowLine-12B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen2 100.0% · 99.8%; Valen-Decision-1.5-9B 100.0% · 99.8%; Dexy-Decision-9B-V2 99.8% · 99.8%; KnowLine 4B Gen5.5 99.5% · 99.8%; Strands Decider 2B 99.1% · 99.6%; Adjudex 9B v5 100.0% · 100.0%; Claude Opus 5.5 · low · chat · 1 per request 99.8% · 100.0%; Claude Sonnet 5.5 · high · chat · 1 per request 99.8% · 100.0%; Copernicus-31B 99.8% · 100.0%; Gemma 4 31B · reasoning off · chat · 1 per request 99.8% · 100.0%; Inkling-Small · high · chat · Jul 2026 100.0% · 100.0%; Jeffan-9B-1211 99.5% · 100.0%; Karpo Decision 31B 99.8% · 100.0%; KnowLine-31B-Gen1.5 100.0% · 100.0%; KnowLine-31B-Gen2 100.0% · 100.0%; Qwen4-Max · high · chat 99.8% · 100.0%; StartLux-Decision-v1.1-4B-Instant 99.6% · 100.0%. A model that always chose the option would reach 100% with no right answers. Running: a chat model's runs with the option go English first, then Turkish, and each is added as it finishes. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision, Raon-1 nano · reasoning off · chat, KnowLine-2B-Gen1, MiniCPM5-2B · reasoning on · chat, Granite-4.2-3B · reasoning on · chat, LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E2B · default reasoning · chat, Gemma 4 E2B · reasoning on · chat, Qwen3.5-2B · reasoning on · chat and Gemma 4 12B Unified · default reasoning · chat (in the table under Info).
Wrong only in Turkish, or only in English?
Each model's first run in each language, both with the option, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method within each group (decision models, chat models) · the shorter bar is the language with fewer wrong answers
Wrong only in Turkish, the upper bar Wrong only in English, the lower bar, striped Numbers on the right: Turkish · English. GLiClass Qwen Liquid KnowLine StartLux MiniCPM Fastino Laya Metask OneJev Perplexity Verdin Claude vera Clef Valen Chinese Decision Pointer GPT Strands Sieve Gemini Karpo Grok GLM DeepSeek Elephant Turion Jev Cerno Gemma Kimi MiMo SubQ Microsoft Drex Adjudex vLLM-SR Logits REAX Erk Muse Copernicus MiniMax Mammoth Jev Open-Jev Cygnet Inkling Dexy Jeffan Inception Winnow * A significant difference (Holm-adjusted p below 0.05): GLiClass v4 Edge, Qwen3.5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, d1-2B, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-0.8B-Instant, MiniCPM5-2B · reasoning off · chat, GLiDE 0.4B, StartLux-Decision-v1.1-2B-Instant, Laya Multilingual, GLiDE 2B, Metask-Jev-4B, OneJev1.6-0.8B, LFM3-1.7B · reasoning on · chat, GLiNER2.5 Multi, GLiNER2.5-multi-Decide, GLiDE 0.8B and LFM2.5-8B-A1B · default reasoning · chat (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p = 0.0015, p = 0.0026, p = 0.0079 and p = 0.018). The other 105: p = 0.091, p = 0.099, p = 0.11, p = 0.23, p = 0.23, p = 0.28, p = 0.33, p = 0.4, p = 0.4, p = 0.57, p = 0.71, p = 0.88, p = 0.88, p = 0.9, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision, Raon-1 nano · reasoning off · chat, KnowLine-2B-Gen1, MiniCPM5-2B · reasoning on · chat, Granite-4.2-3B · reasoning on · chat, LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E2B · default reasoning · chat, Gemma 4 E2B · reasoning on · chat, Qwen3.5-2B · reasoning on · chat and Gemma 4 12B Unified · default reasoning · chat (in the table under Info).
With an “I don't know” option
The same 547 kept questions with a sixth option, “I don't know”, in English and in Turkish, as on the Hallucination tab, asked of the same models (how).
Sixteen of the 56 models with both runs had a higher non-hallucination rate in English than in Turkish; 38 lower; two the same; six of the 56 differences in wrong answers are significant after Holm's correction: LFM2.5-8B-A1B · default reasoning · chat (more wrong answers in Turkish), GLiNER2.5 Multi (more wrong answers in Turkish), MiniCPM5-2B · reasoning off · chat (more wrong answers in English), LFM2.5-2.6B · default reasoning · chat (more wrong answers in Turkish), LFM3-1.7B · reasoning on · chat (more wrong answers in Turkish) and Qwen3.5-2B · reasoning off · chat (more wrong answers in Turkish) (Paired test). Raon-1 nano · reasoning off · chat has an English rate of 26.2% (71 “I don't know” of 271 not right) with an attempt rate of 86.8% (475 answered) but no Turkish run with the option on these questions, so no comparison or test. MiniCPM5-2B · reasoning on · chat has an English rate of 19.7% (28 “I don't know” of 142 not right) with an attempt rate of 94.1% (515 answered) but no Turkish run with the option on these questions, so no comparison or test. Granite-4.2-3B · reasoning on · chat has an English rate of 16.1% (23 “I don't know” of 143 not right) with an attempt rate of 95.8% (524 answered) but no Turkish run with the option on these questions, so no comparison or test. LFM2.5-24B-A2B · default reasoning · chat has an English rate of 13.2% (14 “I don't know” of 106 not right) with an attempt rate of 97.4% (533 answered) but no Turkish run with the option on these questions, so no comparison or test. Gemma 4 E2B · default reasoning · chat has an English rate of 11.5% (17 “I don't know” of 148 not right) with an attempt rate of 96.5% (528 answered) but no Turkish run with the option on these questions, so no comparison or test. Gemma 4 E2B · reasoning on · chat has an English rate of 11.3% (17 “I don't know” of 150 not right) with an attempt rate of 96.3% (527 answered) but no Turkish run with the option on these questions, so no comparison or test. Qwen3.5-2B · reasoning on · chat has an English rate of 9.9% (14 “I don't know” of 142 not right) with an attempt rate of 97.1% (531 answered) but no Turkish run with the option on these questions, so no comparison or test. Gemma 4 12B Unified · default reasoning · chat has an English rate of 5.5% (4 “I don't know” of 73 not right) with an attempt rate of 98.9% (541 answered) but no Turkish run with the option on these questions, so no comparison or test. Still running in Turkish, so charted with one side and in no test: Claude Fable 5.2 · max · chat. Read each rate with its attempt rate: a model that always chose the option would reach 100%.
Non-hallucination rate in Turkish and in English
The option among the answers that were not right, on the 547 kept questions: each model's first run in each language, ordered by the English rate · higher is better, read with the attempt rate (under the chart and in each bar's tooltip)
Turkish option, the upper bar English option, the lower bar, striped Numbers on the right: Turkish · English. Claude Grok Liquid Gemini Fastino GPT MiniCPM Qwen Gemma Kimi DeepSeek GLM Muse MiniMax Erk MiMo Inkling Attempt rate, Turkish · English: Claude Fable 5.2 · max · chat running · 99.3%; Grok 4.7 · high · chat 98.5% · 97.8%; LFM2.5-8B-A1B · default reasoning · chat 52.5% · 90.5%; Gemini 4 Carbon · high · chat 99.3% · 99.3%; Gemini 4 Hydrogen · high · chat 99.3% · 99.3%; GLiNER2.5 Multi 94.9% · 83.2%; GPT-6 Astra · high · chat 99.3% · 99.3%; Gemini 4 Argon · low · chat 99.3% · 99.1%; GPT-6 Astra · low · chat 99.3% · 99.3%; MiniCPM5-2B · reasoning off · chat 20.8% · 89.2%; GPT-6 Astra · medium · chat 99.3% · 99.3%; GPT-6.1 Sol · high · chat 99.3% · 99.3%; GPT-6.1 Terra · high · chat 99.1% · 99.3%; LFM2.5-2.6B · default reasoning · chat 90.7% · 95.6%; LFM3-1.7B · reasoning on · chat 80.6% · 95.2%; GPT-6.1 Terra · low · chat 99.3% · 99.3%; GPT-6 Astra · low · chat · 1 per request 99.3% · 99.3%; GPT-6.1 Sol · low · chat 99.3% · 99.3%; GPT-6.1 Sol · low · chat · 1 per request 99.3% · 99.3%; LFM3-1.7B · reasoning off · chat 56.9% · 96.7%; Qwen3.5-2B · reasoning off · chat 83.4% · 96.3%; Gemini 3.8 Flash · high · chat 99.6% · 99.6%; Gemma 4 E4B · default reasoning · chat 95.4% · 97.8%; Claude Haiku 5.5 · max · chat 96.0% · 96.3%; Gemini 3.8 Flash · low · chat · 1 per request 99.5% · 99.6%; Kimi K3.1 Pro · no reasoning · chat 99.5% · 99.3%; Claude Fable 5.1 · high · chat 99.3% · 99.6%; Kimi K3.1 Pro · low · chat 99.5% · 99.5%; DeepSeek-V4.1-Flash · reasoning on · chat · 1017 99.8% · 99.5%; GPT-6 Luna · low · chat 99.6% · 99.1%; Claude Opus 5.5 · high · chat 99.5% · 99.6%; Qwen3.5-397B-A17B · high · chat 99.3% · 99.5%; GLM-5.3-Flash · reasoning off · chat 99.5% · 99.3%; Qwen3.8 Flash · no reasoning · chat 99.5% · 99.3%; DeepSeek-V4.1-Flash · reasoning off · chat · 1017 100.0% · 99.3%; Muse Ember · max · chat 99.3% · 99.5%; Claude Sonnet 5.5 · high · chat 99.6% · 99.5%; Claude Opus 5.5 · low · chat 99.6% · 99.6%; Claude Sonnet 5.5 · low · chat 99.1% · 99.3%; Qwen3.8-27B · stated confidence · reasoning off · chat 99.3% · 99.5%; Gemma 4 31B · reasoning off · chat 99.6% · 99.6%; Qwen3.5-4B · reasoning off · chat 98.4% · 99.3%; Qwen3.8-27B · low · chat 99.8% · 99.6%; Gemini 3.8 Flash · low · chat 99.5% · 99.8%; GLM-5.3 · low · chat 99.5% · 99.6%; Qwen3.8-27B · vote share · reasoning off · chat 99.3% · 99.5%; MiniMax-M3.1-Flash · high · chat 99.6% · 99.5%; Qwen3.5-4B · reasoning on · chat 99.1% · 99.6%; Erk-14B · reasoning off · chat 99.8% · 99.5%; Qwen3-14B · reasoning off · chat 99.8% · 99.6%; MiMo-V2.6-Flash · high · chat 99.5% · 99.8%; GLM-5.3-Flash · reasoning on · chat 99.6% · 99.8%; Claude Opus 5.5 · low · chat · 1 per request 99.8% · 100.0%; Claude Sonnet 5.5 · high · chat · 1 per request 99.8% · 100.0%; Gemma 4 31B · reasoning off · chat · 1 per request 99.8% · 100.0%; Inkling-Small · high · chat · Jul 2026 100.0% · 100.0%; Qwen4-Max · high · chat 99.8% · 100.0%. A model that always chose the option would reach 100% with no right answers. Running: a chat model's runs with the option go English first, then Turkish, and each is added as it finishes. Not charted, with no Turkish run with the option on these questions: Raon-1 nano · reasoning off · chat, MiniCPM5-2B · reasoning on · chat, Granite-4.2-3B · reasoning on · chat, LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E2B · default reasoning · chat, Gemma 4 E2B · reasoning on · chat, Qwen3.5-2B · reasoning on · chat and Gemma 4 12B Unified · default reasoning · chat (in the table under Info).
Wrong only in Turkish, or only in English?
Each model's first run in each language, both with the option, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the chat models · the shorter bar is the language with fewer wrong answers
Wrong only in Turkish, the upper bar Wrong only in English, the lower bar, striped Numbers on the right: Turkish · English. Qwen Liquid MiniCPM Fastino Claude GPT Gemini Grok GLM DeepSeek Gemma Kimi MiMo Erk Muse MiniMax Inkling * A significant difference (Holm-adjusted p below 0.05): Qwen3.5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, GLiNER2.5 Multi and LFM2.5-8B-A1B · default reasoning · chat (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p = 0.0015 and p = 0.018). The other 50: p = 0.11, p = 0.23, p = 0.57, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with no Turkish run with the option on these questions: Raon-1 nano · reasoning off · chat, MiniCPM5-2B · reasoning on · chat, Granite-4.2-3B · reasoning on · chat, LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E2B · default reasoning · chat, Gemma 4 E2B · reasoning on · chat, Qwen3.5-2B · reasoning on · chat and Gemma 4 12B Unified · default reasoning · chat (in the table under Info).
With an “I don't know” option
The same 547 kept questions with a sixth option, “I don't know”, in English and in Turkish, as on the Hallucination tab, asked of the same models (how).
46 of the 67 models with both runs had a higher non-hallucination rate in English than in Turkish; 17 lower; four the same; 12 of the 67 differences in wrong answers are significant after Holm's correction: d1-2B (more wrong answers in Turkish), GLiDE 0.4B (more wrong answers in English), Laya Multilingual (more wrong answers in Turkish), OneJev1.6-0.8B (more wrong answers in Turkish), GLiDE 2B (more wrong answers in Turkish), GLiDE 0.8B (more wrong answers in Turkish), GLiClass v4 Edge (more wrong answers in Turkish), StartLux-Decision-v1.1-2B-Instant (more wrong answers in Turkish), GLiNER2.5-multi-Decide (more wrong answers in Turkish), KnowLine-0.8B-Gen1 (more wrong answers in Turkish), StartLux-Decision-v1.1-0.8B-Instant (more wrong answers in Turkish) and Metask-Jev-4B (more wrong answers in Turkish) (Paired test). celeris-1.1-decision has an English rate of 74.8% (98 “I don't know” of 131 not right) with an attempt rate of 82.1% (449 answered) but no Turkish run with the option on these questions, so no comparison or test. celeris-1-decision has an English rate of 71.3% (92 “I don't know” of 129 not right) with an attempt rate of 83.2% (455 answered) but no Turkish run with the option on these questions, so no comparison or test. KnowLine-2B-Gen1 has an English rate of 22.0% (38 “I don't know” of 173 not right) with an attempt rate of 93.1% (509 answered) but no Turkish run with the option on these questions, so no comparison or test. Read each rate with its attempt rate: a model that always chose the option would reach 100%.
With the option written “I don't know” in Turkish too, so that only the language differs, the decision models' comparison is in the next section.
Non-hallucination rate in Turkish and in English
The option among the answers that were not right, on the 547 kept questions: each model's first run in each language, ordered by the English rate · higher is better, read with the attempt rate (under the chart and in each bar's tooltip)
Turkish option, the upper bar English option, the lower bar, striped Numbers on the right: Turkish · English. Liquid Fastino Drex Mammoth Laya OneJev Elephant SubQ GLiClass Open-Jev StartLux Perplexity Winnow Inception Cygnet KnowLine GPT Gemini Turion Jev vLLM-SR vera REAX Clef Logits Jev Microsoft Chinese Decision Pointer Cerno Sieve Verdin Metask Valen Dexy Strands Adjudex Copernicus Jeffan Karpo Attempt rate, Turkish · English: d1.2-ultra 88.8% · 80.1%; d1.2 90.3% · 89.2%; d1-2B 70.6% · 72.4%; d1-ultra 94.5% · 90.7%; GLiDE 0.4B 13.2% · 72.0%; Drex Max v1.1 98.4% · 92.1%; Mammoth-1 95.6% · 93.2%; GLiDE · v1.2 98.9% · 94.5%; Laya Multilingual 77.3% · 62.0%; GLiDE 27B 98.4% · 96.0%; OneJev1.6-0.8B 79.5% · 86.5%; GLiDE 2B 87.9% · 91.4%; Elephant-1 97.3% · 95.8%; SubQ Snap 1.1 96.5% · 96.2%; GLiDE 0.8B 86.5% · 91.0%; GLiClass v4 Edge 89.4% · 90.5%; Laya 90.3% · 84.8%; Open-Jev-27B-v1.1 98.9% · 97.1%; StartLux-Decision-v1.1-2B-Instant 94.0% · 94.1%; GLiDE 9B 98.4% · 97.4%; GLiNER2.5-multi-Decide 95.6% · 87.2%; Perplexity Decider 27B v1.2 · API · with web 99.5% · 98.2%; Winnow-12B 98.5% · 97.3%; Mercury Decide 98.5% · 98.0%; Cygnet 99.1% · 98.2%; KnowLine-0.8B-Gen1 97.3% · 95.2%; GPT-6 Luna · Decisions API 98.9% · 98.2%; Gemini Decision 1 · 20261111 99.1% · 99.1%; Perplexity Decider 27B v1.3 100.0% · 99.1%; Turion Jev 31B 99.6% · 99.3%; d3-ultra 99.8% · 99.1%; vera-medium-omni 99.1% · 98.7%; Perplexity Decider Flash v1.3 98.4% · 98.7%; REAX 1.3 99.8% · 99.5%; Clef-Flash 97.8% · 99.1%; StartLux-Decision-v1.1-0.8B-Instant 96.0% · 97.8%; Perplexity Decider Flash v1.2 98.9% · 99.1%; Perplexity Decider 27B v1.2 · API 100.0% · 99.5%; Logits Sigma-1 Duo 100.0% · 99.6%; Jev 1.14 99.5% · 99.5%; OneJev-27B 99.8% · 99.5%; Microsoft-Decision-1 99.6% · 99.5%; Chinese Decision Pointer 4B 99.3% · 99.3%; Drex Max 99.6% · 99.6%; Perplexity Decider 27B v1.2 · DGX Spark 100.0% · 99.6%; d1 99.8% · 99.6%; StartLux-Decision-v1.1-9B-Instant 99.8% · 99.6%; Cerno 9B v1.2 100.0% · 99.6%; Sieve 31B v3 100.0% · 99.8%; Verdin-4B-v1.1 99.8% · 99.6%; d3 100.0% · 99.8%; Metask-Jev-4B 99.6% · 99.6%; Clef 99.8% · 99.8%; KnowLine-12B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen2 100.0% · 99.8%; Valen-Decision-1.5-9B 100.0% · 99.8%; Dexy-Decision-9B-V2 99.8% · 99.8%; KnowLine 4B Gen5.5 99.5% · 99.8%; Strands Decider 2B 99.1% · 99.6%; Adjudex 9B v5 100.0% · 100.0%; Copernicus-31B 99.8% · 100.0%; Jeffan-9B-1211 99.5% · 100.0%; Karpo Decision 31B 99.8% · 100.0%; KnowLine-31B-Gen1.5 100.0% · 100.0%; KnowLine-31B-Gen2 100.0% · 100.0%; StartLux-Decision-v1.1-4B-Instant 99.6% · 100.0%. A model that always chose the option would reach 100% with no right answers. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision and KnowLine-2B-Gen1 (in the table under Info).
Wrong only in Turkish, or only in English?
Each model's first run in each language, both with the option, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the decision models · the shorter bar is the language with fewer wrong answers
Wrong only in Turkish, the upper bar Wrong only in English, the lower bar, striped Numbers on the right: Turkish · English. GLiClass Liquid KnowLine StartLux Fastino Laya Metask OneJev Perplexity Verdin vera Clef Valen Chinese Decision Pointer Strands GPT Sieve Karpo Elephant Turion Jev Gemini Cerno SubQ Microsoft Drex Adjudex vLLM-SR Logits REAX Copernicus Mammoth Jev Open-Jev Cygnet Dexy Jeffan Inception Winnow * A significant difference (Holm-adjusted p below 0.05): GLiClass v4 Edge, d1-2B, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, StartLux-Decision-v1.1-2B-Instant, Laya Multilingual, GLiDE 2B, Metask-Jev-4B, OneJev1.6-0.8B, GLiNER2.5-multi-Decide and GLiDE 0.8B (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p = 0.0026 and p = 0.0079). The other 55: p = 0.091, p = 0.099, p = 0.23, p = 0.28, p = 0.33, p = 0.4, p = 0.4, p = 0.71, p = 0.88, p = 0.88, p = 0.9, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision and KnowLine-2B-Gen1 (in the table under Info).
Only the language differs: “I don't know” in both
The same 547 kept questions with the sixth option written “I don't know” in English and in Turkish too, with the same sentence added to the instructions: each model's first English run with the option, as in the section above, against its Turkish run from the option's word check of the Hallucination tab, counted on these questions. Only the language differs: of the question and its options, of the word in the instructions that names it and of the options heading (how).
Twelve of the 75 decision models have both runs. Among them, wrong of answers was lower in English for five and higher for seven; the non-hallucination rate was lower in English for one and higher for 11. None of the 12 differences in wrong answers is significant after Holm's correction over the 12, so each could be chance; the smallest adjusted p is SubQ Snap 1.1's, p = 0.36 (Paired test). English only, with no Turkish run with the option written “I don't know”: KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1.2 · API, Jeffan-9B-1211, d1, StartLux-Decision-v1.1-9B-Instant, Clef, KnowLine-12B-Gen1, GLiDE 9B, OneJev-27B, Jev 1.14, Cerno 9B v1.2, KnowLine-9B-Gen1, Adjudex 9B v5, Mercury Decide, Microsoft-Decision-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2, celeris-1-decision, GPT-6 Luna · Decisions API, Cygnet, Winnow-12B, Perplexity Decider Flash v1.3, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Clef-Flash, Verdin-4B-v1.1, Perplexity Decider Flash v1.2, KnowLine 4B Gen5.5, Chinese Decision Pointer 4B, Metask-Jev-4B, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, d1-2B, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1, GLiClass v4 Edge, Strands Decider 2B, Laya, GLiNER2.5-multi-Decide and Laya Multilingual (in the main table and the table under Info, with no test).
Wrong of answers in Turkish and in English
Wrong / (right + wrong), with the option written “I don't know” in both languages, on the 547 kept questions: each model's first run in each language, ordered by the Turkish share, lowest first · lower is better, read with the answered share (under the chart and in each bar's tooltip)
Turkish, with “I don't know”, the upper bar English, with “I don't know”, the lower bar, striped Numbers on the right: Turkish · English. Liquid Mammoth SubQ Fastino Drex Perplexity vLLM-SR Elephant Answered, Turkish · English: d1.2-ultra 89.0% · 80.1%; d1-ultra 95.1% · 90.7%; Mammoth-1 95.4% · 93.2%; d1.2 91.6% · 89.2%; SubQ Snap 1.1 96.5% · 96.2%; GLiDE · v1.2 98.4% · 94.5%; Drex Max 99.8% · 99.6%; Drex Max v1.1 96.7% · 92.1%; Perplexity Decider 27B v1.2 · DGX Spark 99.8% · 99.6%; d3-ultra 99.6% · 99.1%; d3 100.0% · 99.8%; Elephant-1 95.6% · 95.8%. A model that answered fewer questions can have fewer wrong answers among them. Not charted, with one side only: KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1, Perplexity Decider 27B v1.2 · API, d1.1, Jeffan-9B-1211, d1, StartLux-Decision-v1.1-9B-Instant, Clef, KnowLine-12B-Gen1, GLiDE 9B, OneJev-27B, Jev 1.14, Cerno 9B v1.2, Drex 2.0, Jev 1.13, KnowLine-9B-Gen1, SubQ Snap Preview, Adjudex 9B v5, Mercury Decide, Microsoft-Decision-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2, celeris-1-decision, GPT-6 Luna · Decisions API, Cygnet, Winnow-12B, Perplexity Decider Flash v1.3, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Clef-Flash, Verdin-4B-v1.1, Perplexity Decider Flash v1.2, KnowLine 4B Gen5.5, Chinese Decision Pointer 4B, Metask-Jev-4B, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, d1-2B, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1, GLiClass v4 Edge, Decider 1, Strands Decider 2B, Laya, GLiNER2.5-multi-Decide and Laya Multilingual (in the table under Info).
Wrong only in Turkish, or only in English?
Each model's first run in each language, both with the option written “I don't know”, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 12 models with both runs · the shorter bar is the language with fewer wrong answers
Wrong only in Turkish, the upper bar Wrong only in English, the lower bar, striped Numbers on the right: Turkish · English. SubQ Liquid Perplexity Elephant Fastino Drex vLLM-SR Mammoth No difference is significant (Holm-adjusted p below 0.05). Adjusted: p = 0.36, p = 0.48, p = 0.72, p = 0.99, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with one side only: KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1, Perplexity Decider 27B v1.2 · API, d1.1, Jeffan-9B-1211, d1, StartLux-Decision-v1.1-9B-Instant, Clef, KnowLine-12B-Gen1, GLiDE 9B, OneJev-27B, Jev 1.14, Cerno 9B v1.2, Drex 2.0, Jev 1.13, KnowLine-9B-Gen1, SubQ Snap Preview, Adjudex 9B v5, Mercury Decide, Microsoft-Decision-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2, celeris-1-decision, GPT-6 Luna · Decisions API, Cygnet, Winnow-12B, Perplexity Decider Flash v1.3, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Clef-Flash, Verdin-4B-v1.1, Perplexity Decider Flash v1.2, KnowLine 4B Gen5.5, Chinese Decision Pointer 4B, Metask-Jev-4B, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, d1-2B, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1, GLiClass v4 Edge, Decider 1, Strands Decider 2B, Laya, GLiNER2.5-multi-Decide and Laya Multilingual (in the table under Info).
The option's word: “I don't know” instead of “Bilmiyorum”
The Hallucination tab's 2,198 Turkish questions, with the sixth option “Bilmiyorum” and again with “I don't know”, and only the sentence that names it changed: a model that skipped “Bilmiyorum” because it did not recognise the Turkish word should choose “I don't know” more often. Each of the 13 models' first run with each word, in the Hallucination tab's order; the full table and method are on the Hallucination tab.
| Model | Answered with “Bilmiyorum”(right + wrong) / questions; higher is better | Answered with “I don't know”(right + wrong) / questions; higher is better | Wrong of answers with “Bilmiyorum”wrong / (right + wrong); lower is better | Wrong of answers with “I don't know”wrong / (right + wrong); lower is better | Non-hallucination with “Bilmiyorum”“Bilmiyorum” / (“Bilmiyorum” + wrong); higher is better | Non-hallucination with “I don't know”“I don't know” / (“I don't know” + wrong); higher is better | Difference“I don't know” minus “Bilmiyorum” in non-hallucination, in points; positive = higher with the English word | Holm-adjusted pexact McNemar test of wrong answers |
|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Efficient | 54.7%1,202 of 2,198 answered | 53.7%1,180 of 2,198 answered | 12.1%145 of 1,202 | 11.6%137 of 1,180 | 87.3%1,057 right, 145 wrong, 996 “Bilmiyorum” | 88.1%1,043 right, 137 wrong, 1,018 “I don't know” | +0.8 | p = 1 |
| d1.2 Closed · Efficient | 55.7%1,225 of 2,198 answered | 59.0%1,297 of 2,198 answered | 20.7%253 of 1,225 | 21.5%279 of 1,297 | 79.4%972 right, 253 wrong, 973 “Bilmiyorum” | 76.4%1,018 right, 279 wrong, 901 “I don't know” | -3.0 | p = 0.4 |
| d1-ultra Closed · Efficient | 71.9%1,581 of 2,198 answered | 72.0%1,582 of 2,198 answered | 13.0%205 of 1,581 | 13.7%216 of 1,582 | 75.1%1,376 right, 205 wrong, 617 “Bilmiyorum” | 74.0%1,366 right, 216 wrong, 616 “I don't know” | -1.0 | p = 1 |
| Mammoth-1 Open · 27B | 68.6%1,508 of 2,198 answered | 65.7%1,445 of 2,198 answered | 20.8%314 of 1,508 | 19.1%276 of 1,445 | 68.7%1,194 right, 314 wrong, 690 “Bilmiyorum” | 73.2%1,169 right, 276 wrong, 753 “I don't know” | +4.5 * | p < 0.0001 |
| GLiDE API Closed · Efficient | 76.3%1,677 of 2,198 answered | 73.6%1,617 of 2,198 answered | 21.2%356 of 1,677 | 20.7%335 of 1,617 | 59.4%1,321 right, 356 wrong, 521 “Bilmiyorum” | 63.4%1,282 right, 335 wrong, 581 “I don't know” | +4.0 | p = 0.51 |
| SubQ Snap 1.1 Closed · Efficient | 80.1%1,761 of 2,198 answered | 77.3%1,700 of 2,198 answered | 27.8%490 of 1,761 | 26.7%454 of 1,700 | 47.1%1,271 right, 490 wrong, 437 “Bilmiyorum” | 52.3%1,246 right, 454 wrong, 498 “I don't know” | +5.2 | p = 0.25 |
| GLiDE v1.2 Open · 27B | 85.4%1,877 of 2,198 answered | 80.6%1,772 of 2,198 answered | 26.2%491 of 1,877 | 24.1%427 of 1,772 | 39.5%1,386 right, 491 wrong, 321 “Bilmiyorum” | 49.9%1,345 right, 427 wrong, 426 “I don't know” | +10.4 * | p < 0.0001 |
| Drex Max v1.1 Closed · Efficient (provisional) | 85.4%1,878 of 2,198 answered | 77.9%1,712 of 2,198 answered | 27.5%517 of 1,878 | 24.1%413 of 1,712 | 38.2%1,361 right, 517 wrong, 320 “Bilmiyorum” | 54.1%1,299 right, 413 wrong, 486 “I don't know” | +15.8 * | p < 0.0001 |
| Elephant-1 Open · 9.0B | 81.6%1,793 of 2,198 answered | 72.5%1,594 of 2,198 answered | 42.3%758 of 1,793 | 39.0%622 of 1,594 | 34.8%1,035 right, 758 wrong, 405 “Bilmiyorum” | 49.3%972 right, 622 wrong, 604 “I don't know” | +14.4 * | p < 0.0001 |
| d3-ultra Open · 26B | 99.5%2,188 of 2,198 answered | 99.6%2,189 of 2,198 answered | 32.2%704 of 2,188 | 32.2%704 of 2,189 | 1.4%1,484 right, 704 wrong, 10 “Bilmiyorum” | 1.3%1,485 right, 704 wrong, 9 “I don't know” | -0.1 | p = 1 |
| Drex Max Open · 27B | 99.9%2,196 of 2,198 answered | 99.9%2,196 of 2,198 answered | 32.4%711 of 2,196 | 32.4%712 of 2,196 | 0.3%1,485 right, 711 wrong, 2 “Bilmiyorum” | 0.3%1,484 right, 712 wrong, 2 “I don't know” | -0.0 | p = 1 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 99.9%2,196 of 2,198 answered | 99.8%2,194 of 2,198 answered | 32.7%717 of 2,196 | 33.2%728 of 2,194 | 0.3%1,479 right, 717 wrong, 2 “Bilmiyorum” | 0.5%1,466 right, 728 wrong, 4 “I don't know” | +0.3 | p = 1 |
| d3 Open · 26B | 100.0%2,198 of 2,198 answered | 99.9%2,196 of 2,198 answered | 32.5%715 of 2,198 | 32.5%714 of 2,196 | 0.0%1,483 right, 715 wrong, 0 “Bilmiyorum” | 0.3%1,482 right, 714 wrong, 2 “I don't know” | +0.3 | p = 1 |
Answered: (right + wrong) / 2,198 questions. Wrong of answers: wrong / (right + wrong). Both are worked out from the published counts; under each non-hallucination rate, of the 2,198 questions, the right answers, the wrong ones and the option. * marks a difference whose paired test (the exact McNemar test on the questions only one of the two runs got wrong, with p adjusted by Holm's method over the models) has a Holm-adjusted p below 0.05.
Cost per 1,000 questionsper 1M questions in Turkish and in English
Estimated dollars per 1,000 questionsper 1M questions: each of the 133 setups' Text tab run and its first English run, ordered by English cost · lower is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Perplexity Liquid Fastino Inception Laya Microsoft Elephant meraGPT SubQ Celeris Clef Strands Drex Adjudex Dexy Jev Chinese Decision Pointer Qwen Verdin vera KnowLine Gemini Mammoth Valen GPT Cygnet Jeffan GLM Winnow Gemma vLLM-SR Metask DeepSeek OneJev Erk Copernicus Karpo REAX Logits MiniMax Muse MiMo Open-Jev Claude Inkling MiniCPM Kimi Grok Cerno GLiClass Sieve StartLux Turion Jev Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. No list price: on our own GPUs, with no estimate. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. Log scale. No list price in Turkish: GLiClass v4 Edge, GLiDE 0.4B, GLiDE 0.8B, GLiDE 27B, GLiDE 2B, GLiDE 9B, KnowLine 4B Gen5.5, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, KnowLine-31B-Gen1.5, OneJev1.6-0.8B, Sieve 31B v3, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, StartLux-Decision-v1.1-9B-Instant and Turion Jev 31B, on our own hardware; the maker lists no price for the same model. No list price in English: Cerno 9B v1.2, GLiClass v4 Edge, GLiDE 0.4B, GLiDE 0.8B, GLiDE 27B, GLiDE 2B, GLiDE 9B, KnowLine 4B Gen5.5, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, KnowLine-31B-Gen1.5, OneJev1.6-0.8B, Sieve 31B v3, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, StartLux-Decision-v1.1-9B-Instant and Turion Jev 31B, on our own hardware; the maker lists no price for the same model.
Cost per 1,000 questionsper 1M questions in Turkish and in English
Estimated dollars per 1,000 questionsper 1M questions: each of the 58 chat model setups' Text tab run and its first English run, ordered by English cost · lower is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Fastino Qwen GLM Gemma GPT DeepSeek Erk MiniMax Muse Gemini MiMo Claude Liquid Inkling MiniCPM Kimi Grok Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. Log scale.
Cost per 1,000 questionsper 1M questions in Turkish and in English
Estimated dollars per 1,000 questionsper 1M questions: each of the 75 decision models' Text tab run and its first English run, ordered by English cost · lower is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Perplexity Liquid Fastino Inception Laya Microsoft Elephant meraGPT SubQ Celeris Clef Strands Drex Adjudex Dexy Jev Chinese Decision Pointer Verdin vera KnowLine Gemini Mammoth Valen GPT Cygnet Jeffan Winnow vLLM-SR Metask OneJev Copernicus Karpo REAX Logits Open-Jev Cerno GLiClass Sieve StartLux Turion Jev Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. No list price: on our own GPUs, with no estimate. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. Log scale. No list price in Turkish: GLiClass v4 Edge, GLiDE 0.4B, GLiDE 0.8B, GLiDE 27B, GLiDE 2B, GLiDE 9B, KnowLine 4B Gen5.5, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, KnowLine-31B-Gen1.5, OneJev1.6-0.8B, Sieve 31B v3, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, StartLux-Decision-v1.1-9B-Instant and Turion Jev 31B, on our own hardware; the maker lists no price for the same model. No list price in English: Cerno 9B v1.2, GLiClass v4 Edge, GLiDE 0.4B, GLiDE 0.8B, GLiDE 27B, GLiDE 2B, GLiDE 9B, KnowLine 4B Gen5.5, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, KnowLine-31B-Gen1.5, OneJev1.6-0.8B, Sieve 31B v3, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, StartLux-Decision-v1.1-9B-Instant and Turion Jev 31B, on our own hardware; the maker lists no price for the same model.
Median latency per request in Turkish and in English
Median time per request: each of the 133 setups' Text tab run and its first English run, ordered by English latency · lower is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Fastino Laya Liquid GLiClass OneJev Strands StartLux Perplexity KnowLine Adjudex Dexy Chinese Decision Pointer Cygnet Verdin vera Valen Jeffan Elephant Winnow vLLM-SR Drex Metask Clef Jev Copernicus Mammoth Karpo Cerno REAX Gemini Logits Gemma Inception Microsoft GPT SubQ Celeris Qwen Turion Jev Open-Jev meraGPT Sieve Erk Claude GLM Kimi DeepSeek MiMo MiniCPM Muse Inkling MiniMax Grok Time per request as measured from our computer during each run; API times include the network. Text tab runs: 1 question per request for 95 runs and 16 for 38; 1 request in flight for 107 runs, 4 for 14, 32 for 10 and 8 for 2. English runs: 1 question per request for 95 runs and 16 for 38; 1 request in flight for 107 runs, 4 for 14, 32 for 9 and 8 for 3. Of the 77 self-hosted setups, KnowLine-31B-Gen1.5, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, StartLux-Decision-v1.1-9B-Instant, GLiDE 9B, Cerno 9B v1.2, StartLux-Decision-v1.1-4B-Instant, KnowLine 4B Gen5.5, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1 and GLiClass v4 Edge ran their English runs on an NVIDIA DGX Spark, the others on our RTX PRO 6000 GPUs; their Text tab runs ran on the same machine, except those of GLiDE · v1.2, Perplexity Decider 27B v1.2 · DGX Spark and Perplexity Decider Flash v1.2, which ran on an NVIDIA DGX Spark; those of Cerno 9B v1.2, which ran on our RTX PRO 6000 GPUs (Our method). Clef and Clef-Flash ran their English runs at the same time on one GPU. * Measured on a sample (200 questions): the Text tab setup's repeats on a fixed sample, as on the home page. Log scale.
Median latency per request in Turkish and in English
Median time per request: each of the 58 chat model setups' Text tab run and its first English run, ordered by English latency · lower is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Fastino Gemma Qwen Gemini Erk Claude GLM Liquid GPT Kimi DeepSeek MiMo MiniCPM Muse Inkling MiniMax Grok Time per request as measured from our computer during each run; API times include the network. Text tab runs: 16 questions per request for 38 runs and 1 for 20; 1 request in flight for 33 runs, 4 for 13, 32 for 10 and 8 for 2. English runs: 16 questions per request for 38 runs and 1 for 20; 1 request in flight for 33 runs, 4 for 13, 32 for 9 and 8 for 3. The English runs of the 22 self-hosted setups ran on our RTX PRO 6000 GPUs; their Text tab runs ran on the same machine (Our method). * Measured on a sample (200 questions): the Text tab setup's repeats on a fixed sample, as on the home page. Log scale.
Median latency per request in Turkish and in English
Median time per request: each of the 75 decision models' Text tab run and its first English run, ordered by English latency · lower is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Fastino Laya Liquid GLiClass OneJev Strands StartLux Perplexity KnowLine Adjudex Dexy Chinese Decision Pointer Cygnet Verdin vera Valen Jeffan Elephant Winnow vLLM-SR Drex Metask Clef Jev Copernicus Mammoth Karpo Cerno REAX Gemini Logits Inception Microsoft GPT SubQ Celeris Turion Jev Open-Jev meraGPT Sieve Time per request as measured from our computer during each run; API times include the network. Text tab runs: 1 question per request for 75 runs; 1 request in flight for 74 runs and 4 for 1. English runs: 1 question per request for 75 runs; 1 request in flight for 74 runs and 4 for 1. Of the 55 self-hosted setups, KnowLine-31B-Gen1.5, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, StartLux-Decision-v1.1-9B-Instant, GLiDE 9B, Cerno 9B v1.2, StartLux-Decision-v1.1-4B-Instant, KnowLine 4B Gen5.5, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1 and GLiClass v4 Edge ran their English runs on an NVIDIA DGX Spark, the others on our RTX PRO 6000 GPUs; their Text tab runs ran on the same machine, except those of GLiDE · v1.2, Perplexity Decider 27B v1.2 · DGX Spark and Perplexity Decider Flash v1.2, which ran on an NVIDIA DGX Spark; those of Cerno 9B v1.2, which ran on our RTX PRO 6000 GPUs (Our method). Clef and Clef-Flash ran their English runs at the same time on one GPU. Log scale.
Risk and coverage
How often each setup answers and how often its answers are wrong, on the same 547 kept questions in both languages: without the option, by holding back the answers below a confidence score, and with the option, as the model chose it.
Method
Answered = right + wrong (an invalid answer or one without a score is never answered). Coverage = answered / 547. Error rate among answered (risk) = wrong / answered. Abstained: for a run with the option, the times it chose the option; for a confidence point, 547 minus answered minus invalid. Each confidence curve has one point per distinct score of the run on these questions (answer when the score is at least that score; the score is the recorded probability of the chosen option for a decision model and the confidence a chat model stated with its answer): answers with equal scores enter together, so coverage moves in jumps and every point is one the run achieved; a setup that reports its scores with few decimals has many equal scores, so its curve has fewer points, moves in larger steps and can start well above 0% coverage (each curve is drawn from its first point). Every series counts these questions only (a run on the full bank is restricted to them) and is each setup's first run. Turkish curves: the Text tab run; English curves: the first English run. Turkish with “Bilmiyorum” is the Turkish side of the English “I don't know” comparison or, for a model outside it, its Hallucination tab run, restricted to these; Turkish with “I don't know” is the option's word control on the Hallucination tab's questions, restricted to these, for decision models only. A run not yet in reads Not run for a decision model and Running for a chat model. A setup asked for a plain answer line (one question per request, no stated confidence and no option probabilities) has no confidence curve: not measured, only its runs with the option.
All series are counted on the same kept reading questions; a full-bank run is restricted to them; first runs only. Answered = right + wrong (an invalid answer or one without a score is never answered). Coverage = answered / kept questions. Risk = error rate among answered = wrong / answered. Abstained: for a native point, the times the model chose the option; for a curve point, kept questions - answered - invalid. A native point is the run with its abstention option (Turkish "Bilmiyorum", Turkish "I don't know" as the wording control, English "I don't know"); Turkish "Bilmiyorum" is the Turkish side of the English "I don't know" comparison or, for a model outside it, its Hallucination tab run; a missing one reads Not run (Running while a chat model's run is still going). A curve is one five-option run held back by its recorded score (the probability of the chosen option): one point per distinct score s, answering when the score is at least s; answers with equal scores enter together, so coverage moves in jumps and every point is an achieved one; all distinct points kept, ordered by coverage.
The runs with the option
| Model | Turkish with “Bilmiyorum”coverage · error rate among answered | Turkish with “I don't know”coverage · error rate among answered | English with “I don't know”coverage · error rate among answered |
|---|---|---|---|
| Claude Fable 5.2 max · chat Closed · Flagship | Queuedits run with the option has not started yet | Not rundecision models only | 99.3% · 1.8%533 right, 10 wrong, 4 abstained · run 1032 |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 99.3% · 1.8%533 right, 10 wrong, 4 abstained · run 1117, full bank restricted | Not rundecision models only | 99.3% · 2.9%527 right, 16 wrong, 4 abstained · run 1109 |
| GPT-6 Astra low · chat Closed · Flagship | 99.3% · 2.6%529 right, 14 wrong, 4 abstained · run 816, full bank restricted | Not rundecision models only | 99.3% · 3.3%525 right, 18 wrong, 4 abstained · run 647 |
| GPT-6 Astra high · chat Closed · Flagship | 99.3% · 3.1%526 right, 17 wrong, 4 abstained · run 818, full bank restricted | Not rundecision models only | 99.3% · 3.1%526 right, 17 wrong, 4 abstained · run 814 |
| GPT-6.1 Terra high · chat Closed · Workhorse | 99.1% · 3.5%523 right, 19 wrong, 5 abstained · run 807, full bank restricted | Not rundecision models only | 99.3% · 3.7%523 right, 20 wrong, 4 abstained · run 766 |
| GPT-6.1 Terra low · chat Closed · Workhorse | 99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 801, full bank restricted | Not rundecision models only | 99.3% · 4.1%521 right, 22 wrong, 4 abstained · run 751 |
| Claude Fable 5.1 high · chat Closed · Flagship | 99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 695, full bank restricted | Not rundecision models only | 99.6% · 4.6%520 right, 25 wrong, 2 abstained · run 655 |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 99.6% · 2.4%532 right, 13 wrong, 2 abstained · run 681, full bank restricted | Not rundecision models only | 99.6% · 3.3%527 right, 18 wrong, 2 abstained · run 657 |
| Gemini 4 Argon low · chat Closed · Flagship | 99.3% · 2.4%530 right, 13 wrong, 4 abstained · run 679, full bank restricted | Not rundecision models only | 99.1% · 4.1%520 right, 22 wrong, 5 abstained · run 652 |
| GPT-6 Astra medium · chat Closed · Flagship | 99.3% · 2.9%527 right, 16 wrong, 4 abstained · run 817, full bank restricted | Not rundecision models only | 99.3% · 3.7%523 right, 20 wrong, 4 abstained · run 813 |
| GPT-6.1 Sol low · chat Closed · Workhorse | 99.3% · 3.3%525 right, 18 wrong, 4 abstained · run 799, full bank restricted | Not rundecision models only | 99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 744 |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 99.3% · 3.5%524 right, 19 wrong, 4 abstained · run 819, full bank restricted | Not rundecision models only | 99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 815 |
| GPT-6.1 Sol high · chat Closed · Workhorse | 99.3% · 2.9%527 right, 16 wrong, 4 abstained · run 802, full bank restricted | Not rundecision models only | 99.3% · 3.7%523 right, 20 wrong, 4 abstained · run 758 |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 99.5% · 3.3%526 right, 18 wrong, 3 abstained · run 691, full bank restricted | Not rundecision models only | 99.6% · 4.4%521 right, 24 wrong, 2 abstained · run 669 |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 99.3% · 3.5%524 right, 19 wrong, 4 abstained · run 810, full bank restricted | Not rundecision models only | 99.3% · 5.0%516 right, 27 wrong, 4 abstained · run 769 |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 99.8% · 4.6%521 right, 25 wrong, 1 abstained · run 803, full bank restricted | Not rundecision models only | 100.0% · 5.7%516 right, 31 wrong, 0 abstained · run 670 |
| Claude Opus 5.5 high · chat Closed · Flagship | 99.5% · 4.2%521 right, 23 wrong, 3 abstained · run 773, full bank restricted | Not rundecision models only | 99.6% · 5.3%516 right, 29 wrong, 2 abstained · run 665 |
| Qwen4-Max high · chat Closed · Workhorse | 99.8% · 5.1%518 right, 28 wrong, 1 abstained · run 1339, full bank restricted | Not rundecision models only | 100.0% · 5.3%518 right, 29 wrong, 0 abstained · run 1338 |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 99.5% · 4.4%520 right, 24 wrong, 3 abstained · run 676, full bank restricted | Not rundecision models only | 99.8% · 5.1%518 right, 28 wrong, 1 abstained · run 645 |
| Gemini 4 Carbon high · chat Closed · Flagship | 99.3% · 2.0%532 right, 11 wrong, 4 abstained · run 1121, full bank restricted | Not rundecision models only | 99.3% · 2.2%531 right, 12 wrong, 4 abstained · run 1067 |
| KnowLine-31B-Gen1.5 Open · 31B | 100.0% · 6.2%513 right, 34 wrong, 0 abstained · run 959, full bank restricted | Not run | 100.0% · 6.8%510 right, 37 wrong, 0 abstained · run 955 |
| Grok 4.7 high · chat Closed · Workhorse | 98.5% · 3.9%518 right, 21 wrong, 8 abstained · run 675, full bank restricted | Not rundecision models only | 97.8% · 5.8%504 right, 31 wrong, 12 abstained · run 648 |
| Claude Opus 5.5 low · chat Closed · Flagship | 99.6% · 5.3%516 right, 29 wrong, 2 abstained · run 683, full bank restricted | Not rundecision models only | 99.6% · 7.5%504 right, 41 wrong, 2 abstained · run 649 |
| Copernicus-31B Open · 31B | 99.8% · 6.0%513 right, 33 wrong, 1 abstained · run 1172, full bank restricted | Not run | 100.0% · 7.1%508 right, 39 wrong, 0 abstained · run 1170 |
| Claude Haiku 5.5 max · chat Closed · Efficient | 96.0% · 5.5%496 right, 29 wrong, 6 abstained · run 950, full bank restricted | Not rundecision models only | 96.3% · 8.9%480 right, 47 wrong, 4 abstained · run 951 |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 99.8% · 7.0%508 right, 38 wrong, 1 abstained · run 811, full bank restricted | Not rundecision models only | 99.5% · 7.2%505 right, 39 wrong, 3 abstained · run 809 |
| REAX 1.3 Open · 57B | 99.8% · 6.4%511 right, 35 wrong, 1 abstained · run 833, full bank restricted | Not run | 99.5% · 7.7%502 right, 42 wrong, 3 abstained · run 831 |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 99.5% · 6.6%508 right, 36 wrong, 3 abstained · run 686, full bank restricted | Not rundecision models only | 99.5% · 7.0%506 right, 38 wrong, 3 abstained · run 651 |
| KnowLine-31B-Gen2 Open · 31B | 100.0% · 5.3%518 right, 29 wrong, 0 abstained · run 1331, full bank restricted | Not run | 100.0% · 7.3%507 right, 40 wrong, 0 abstained · run 1329 |
| Karpo Decision 31B Open · 31B | 99.8% · 5.5%516 right, 30 wrong, 1 abstained · run 1163, full bank restricted | Not run | 100.0% · 7.7%505 right, 42 wrong, 0 abstained · run 1161 |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 99.8% · 6.6%510 right, 36 wrong, 1 abstained · run 812, full bank restricted | Not rundecision models only | 100.0% · 6.9%509 right, 38 wrong, 0 abstained · run 677 |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 99.5% · 7.2%505 right, 39 wrong, 3 abstained · run 653, full bank restricted | Not rundecision models only | 99.3% · 9.0%494 right, 49 wrong, 4 abstained · run 646 |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 99.3% · 7.9%500 right, 43 wrong, 4 abstained · run 872, full bank restricted | Not rundecision models only | 99.5% · 8.3%499 right, 45 wrong, 3 abstained · run 873 |
| Gemini Decision 1 20261111 Closed · Efficient | 99.1% · 7.6%501 right, 41 wrong, 5 abstained · run 1023, full bank restricted | Not run | 99.1% · 9.2%492 right, 50 wrong, 5 abstained · run 1019 |
| Logits Sigma-1 Duo Open · 57B | 100.0% · 6.6%511 right, 36 wrong, 0 abstained · run 1233, full bank restricted | Not run | 99.6% · 8.1%501 right, 44 wrong, 2 abstained · run 1231 |
| GLiDE 27B Open · 27B | 98.4% · 7.2%499 right, 39 wrong, 9 abstained · run 905, full bank restricted | Not run | 96.0% · 6.9%489 right, 36 wrong, 22 abstained · run 903 |
| d1-ultra Closed · Efficient | 94.5% · 5.0%491 right, 26 wrong, 30 abstained · run 350, full bank restricted | 95.1% · 5.4%492 right, 28 wrong, 27 abstained · run 511, full bank restricted | 90.7% · 7.9%457 right, 39 wrong, 51 abstained · run 449 |
| Sieve 31B v3 Open · 31B | 100.0% · 6.0%514 right, 33 wrong, 0 abstained · run 1253, full bank restricted | Not run | 99.8% · 8.2%501 right, 45 wrong, 1 abstained · run 1066 |
| Turion Jev 31B Open · 31B | 99.6% · 6.1%512 right, 33 wrong, 2 abstained · run 1285, full bank restricted | Not run | 99.3% · 7.9%500 right, 43 wrong, 4 abstained · run 1077 |
| Gemma 4 31B reasoning off · chat Open · 31B | 99.6% · 7.7%503 right, 42 wrong, 2 abstained · run 776, full bank restricted | Not rundecision models only | 99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 775 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 99.5% · 9.4%493 right, 51 wrong, 3 abstained · run 642, full bank restricted | Not run | 98.2% · 10.4%481 right, 56 wrong, 10 abstained · run 644 |
| GLiDE v1.2 Open · 27B | 98.9% · 8.7%494 right, 47 wrong, 6 abstained · run 369, full bank restricted | 98.4% · 8.7%491 right, 47 wrong, 9 abstained · run 525, full bank restricted | 94.5% · 7.0%481 right, 36 wrong, 30 abstained · run 523 |
| d1.2-ultra Closed · Efficient | 88.8% · 3.9%467 right, 19 wrong, 61 abstained · run 1367, full bank restricted | 89.0% · 3.7%469 right, 18 wrong, 60 abstained · run 1378, full bank restricted | 80.1% · 3.0%425 right, 13 wrong, 109 abstained · run 1363 |
| Mammoth-1 Open · 27B | 95.6% · 6.3%490 right, 33 wrong, 24 abstained · run 386, full bank restricted | 95.4% · 6.1%490 right, 32 wrong, 25 abstained · run 519, full bank restricted | 93.2% · 7.1%474 right, 36 wrong, 37 abstained · run 505 |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 99.8% · 9.5%494 right, 52 wrong, 1 abstained · run 789, full bank restricted | Not rundecision models only | 100.0% · 10.8%488 right, 59 wrong, 0 abstained · run 786 |
| d3 Open · 26B | 100.0% · 9.7%494 right, 53 wrong, 0 abstained · run 358, full bank restricted | 100.0% · 9.5%495 right, 52 wrong, 0 abstained · run 544, full bank restricted | 99.8% · 10.6%488 right, 58 wrong, 1 abstained · run 793 |
| Drex Max Open · 27B | 99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 367, full bank restricted | 99.8% · 9.0%497 right, 49 wrong, 1 abstained · run 542, full bank restricted | 99.6% · 11.0%485 right, 60 wrong, 2 abstained · run 798 |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 99.6% · 10.1%490 right, 55 wrong, 2 abstained · run 825, full bank restricted | Not rundecision models only | 99.8% · 11.2%485 right, 61 wrong, 1 abstained · run 824 |
| Muse Ember max · chat Closed · Efficient | 99.3% · 8.7%496 right, 47 wrong, 4 abstained · run 940, full bank restricted | Not rundecision models only | 99.5% · 9.9%490 right, 54 wrong, 3 abstained · run 943 |
| Perplexity Decider 27B v1.3 Open · 27B | 100.0% · 9.5%495 right, 52 wrong, 0 abstained · run 1141, full bank restricted | Not run | 99.1% · 9.6%490 right, 52 wrong, 5 abstained · run 1139 |
| d3-ultra Open · 26B | 99.8% · 8.4%500 right, 46 wrong, 1 abstained · run 363, full bank restricted | 99.6% · 9.4%494 right, 51 wrong, 2 abstained · run 540, full bank restricted | 99.1% · 10.3%486 right, 56 wrong, 5 abstained · run 795 |
| d1.2 Closed · Efficient | 90.3% · 7.1%459 right, 35 wrong, 53 abstained · run 1365, full bank restricted | 91.6% · 8.4%459 right, 42 wrong, 46 abstained · run 1377, full bank restricted | 89.2% · 7.6%451 right, 37 wrong, 59 abstained · run 1361 |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 99.5% · 12.7%475 right, 69 wrong, 3 abstained · run 879, full bank restricted | Not rundecision models only | 99.8% · 10.6%488 right, 58 wrong, 1 abstained · run 880 |
| Qwen3.8-27B low · chat Open · 27B | 99.8% · 8.8%498 right, 48 wrong, 1 abstained · run 666, full bank restricted | Not rundecision models only | 99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 671 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 100.0% · 9.1%497 right, 50 wrong, 0 abstained · run 372, full bank restricted | 99.8% · 9.3%495 right, 51 wrong, 1 abstained · run 531, full bank restricted | 99.6% · 11.7%481 right, 64 wrong, 2 abstained · run 529 |
| Drex Max v1.1 Closed · Efficient (provisional) | 98.4% · 9.5%487 right, 51 wrong, 9 abstained · run 352, full bank restricted | 96.7% · 9.1%481 right, 48 wrong, 18 abstained · run 513, full bank restricted | 92.1% · 8.1%463 right, 41 wrong, 43 abstained · run 447 |
| Perplexity Decider 27B v1 Open · 27B | Not run | Not run | Not run |
| Perplexity Decider 27B v1.2 API Open · 27B | 100.0% · 9.3%496 right, 51 wrong, 0 abstained · run 641, full bank restricted | Not run | 99.5% · 11.6%481 right, 63 wrong, 3 abstained · run 643 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 100.0% · 11.3%485 right, 62 wrong, 0 abstained · run 868, full bank restricted | Not rundecision models only | 100.0% · 11.0%487 right, 60 wrong, 0 abstained · run 869 |
| d1.1 Closed · Efficient | Not run | Not run | Not run |
| GLM-5.3 low · chat Open · Workhorse | 99.5% · 11.2%483 right, 61 wrong, 3 abstained · run 656, full bank restricted | Not rundecision models only | 99.6% · 10.3%489 right, 56 wrong, 2 abstained · run 650 |
| Jeffan-9B-1211 Open · 9.0B | 99.5% · 13.1%473 right, 71 wrong, 3 abstained · run 1199, full bank restricted | Not run | 100.0% · 13.0%476 right, 71 wrong, 0 abstained · run 1197 |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 99.5% · 15.1%462 right, 82 wrong, 3 abstained · run 822, full bank restricted | Not rundecision models only | 99.3% · 12.3%476 right, 67 wrong, 4 abstained · run 821 |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 100.0% · 10.4%490 right, 57 wrong, 0 abstained · run 806, full bank restricted | Not rundecision models only | 99.3% · 12.9%473 right, 70 wrong, 4 abstained · run 805 |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 99.1% · 16.2%454 right, 88 wrong, 2 abstained · run 1150, full bank restricted | Not rundecision models only | 99.6% · 15.2%462 right, 83 wrong, 2 abstained · run 1073 |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 99.5% · 10.1%489 right, 55 wrong, 3 abstained · run 987, full bank restricted | Not rundecision models only | 99.3% · 12.3%476 right, 67 wrong, 4 abstained · run 988 |
| d1 Closed · Efficient | 99.8% · 16.3%457 right, 89 wrong, 1 abstained · run 472 | Not run | 99.6% · 12.3%478 right, 67 wrong, 2 abstained · run 445 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 99.8% · 17.4%451 right, 95 wrong, 1 abstained · run 1297, full bank restricted | Not run | 99.6% · 13.6%471 right, 74 wrong, 2 abstained · run 1295 |
| Clef Open · 27B | 99.8% · 12.3%479 right, 67 wrong, 1 abstained · run 788, full bank restricted | Not run | 99.8% · 13.7%471 right, 75 wrong, 1 abstained · run 785 |
| KnowLine-12B-Gen1 Open · 12B | 100.0% · 12.2%480 right, 67 wrong, 0 abstained · run 1318, full bank restricted | Not run | 99.8% · 13.9%470 right, 76 wrong, 1 abstained · run 1316 |
| GLiDE 9B Open · 7.9B | 98.4% · 15.4%455 right, 83 wrong, 9 abstained · run 913, full bank restricted | Not run | 97.4% · 13.1%463 right, 70 wrong, 14 abstained · run 911 |
| GPT-6 Luna low · chat Closed · Efficient | 99.6% · 11.9%480 right, 65 wrong, 2 abstained · run 796, full bank restricted | Not rundecision models only | 99.1% · 12.7%473 right, 69 wrong, 5 abstained · run 735 |
| OneJev-27B Open · 27B | 99.8% · 11.5%483 right, 63 wrong, 1 abstained · run 782, full bank restricted | Not run | 99.5% · 13.2%472 right, 72 wrong, 3 abstained · run 780 |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 800, full bank restricted | Not rundecision models only | 99.5% · 10.8%485 right, 59 wrong, 3 abstained · run 668 |
| Jev 1.14 Closed · Efficient | 99.5% · 12.7%475 right, 69 wrong, 3 abstained · run 470 | Not run | 99.5% · 13.2%472 right, 72 wrong, 3 abstained · run 446 |
| Cerno 9B v1.2 Open · 9.0B | 100.0% · 16.6%456 right, 91 wrong, 0 abstained · run 1204, full bank restricted | Not run | 99.6% · 14.1%468 right, 77 wrong, 2 abstained · run 1087 |
| Drex 2.0 Open · 35B MoE · 3.0B active | Not run | Not run | Not run |
| Jev 1.13 Closed · Efficient | Not run | Not run | Not run |
| KnowLine-9B-Gen1 Open · 9.0B | 100.0% · 17.9%449 right, 98 wrong, 0 abstained · run 1181, full bank restricted | Not run | 99.8% · 13.9%470 right, 76 wrong, 1 abstained · run 1179 |
| SubQ Snap 1.1 Closed · Efficient | 96.5% · 9.7%477 right, 51 wrong, 19 abstained · run 392, full bank restricted | 96.5% · 8.5%483 right, 45 wrong, 19 abstained · run 512, full bank restricted | 96.2% · 12.0%463 right, 63 wrong, 21 abstained · run 448 |
| SubQ Snap Preview Closed · Efficient | Not run | Not run | Not run |
| Adjudex 9B v5 Open · 9.0B | 100.0% · 16.3%458 right, 89 wrong, 0 abstained · run 1190, full bank restricted | Not run | 100.0% · 14.1%470 right, 77 wrong, 0 abstained · run 1188 |
| Mercury Decide Closed · Efficient | 98.5% · 13.2%468 right, 71 wrong, 8 abstained · run 489 | Not run | 98.0% · 13.2%465 right, 71 wrong, 11 abstained · run 450 |
| Microsoft-Decision-1 Closed · Efficient | 99.6% · 16.1%457 right, 88 wrong, 2 abstained · run 487 | Not run | 99.5% · 14.2%467 right, 77 wrong, 3 abstained · run 451 |
| Elephant-1 Open · 9.0B | 97.3% · 15.4%450 right, 82 wrong, 15 abstained · run 382, full bank restricted | 95.6% · 15.7%441 right, 82 wrong, 24 abstained · run 515, full bank restricted | 95.8% · 12.8%457 right, 67 wrong, 23 abstained · run 501 |
| Open-Jev-27B-v1.1 Open · 27B | 98.9% · 14.4%463 right, 78 wrong, 6 abstained · run 768, full bank restricted | Not run | 97.1% · 14.3%455 right, 76 wrong, 16 abstained · run 763 |
| KnowLine-9B-Gen2 Open · 9.0B | 100.0% · 17.4%452 right, 95 wrong, 0 abstained · run 1325, full bank restricted | Not run | 99.8% · 14.3%468 right, 78 wrong, 1 abstained · run 1323 |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 99.3% · 13.3%471 right, 72 wrong, 4 abstained · run 674, full bank restricted | Not rundecision models only | 99.5% · 14.3%466 right, 78 wrong, 3 abstained · run 673 |
| Qwen3.8-27B yes/no scoring Open · 27B | Not runyes/no scoring cannot offer it | Not rundecision models only | Not runyes/no scoring cannot offer it |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 99.6% · 16.5%455 right, 90 wrong, 1 abstained · run 864, full bank restricted | Not rundecision models only | 99.5% · 15.6%459 right, 85 wrong, 3 abstained · run 865 |
| celeris-1-decision Closed · Efficient | Not run | Not run | 83.2% · 8.1%418 right, 37 wrong, 92 abstained · run 548 |
| GPT-6 Luna Decisions API Closed · Efficient | 98.9% · 14.4%463 right, 78 wrong, 6 abstained · run 453 | Not run | 98.2% · 17.9%441 right, 96 wrong, 10 abstained · run 444 |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 99.1% · 15.1%460 right, 82 wrong, 5 abstained · run 690, full bank restricted | Not rundecision models only | 99.3% · 15.8%457 right, 86 wrong, 4 abstained · run 654 |
| Cygnet Open · 12B | 99.1% · 14.9%461 right, 81 wrong, 5 abstained · run 757, full bank restricted | Not run | 98.2% · 14.7%458 right, 79 wrong, 10 abstained · run 755 |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 99.3% · 13.8%468 right, 75 wrong, 4 abstained · run 682, full bank restricted | Not rundecision models only | 99.5% · 15.4%460 right, 84 wrong, 3 abstained · run 680 |
| Winnow-12B Open · 12B | 98.5% · 15.8%454 right, 85 wrong, 8 abstained · run 750, full bank restricted | Not run | 97.3% · 16.0%447 right, 85 wrong, 15 abstained · run 748 |
| Perplexity Decider Flash v1.3 Open · 4.0B | 98.4% · 20.4%428 right, 110 wrong, 9 abstained · run 1130, full bank restricted | Not run | 98.7% · 16.7%450 right, 90 wrong, 7 abstained · run 1128 |
| Valen-Decision-1.5-9B Open · 9.0B | 100.0% · 20.1%437 right, 110 wrong, 0 abstained · run 1282, full bank restricted | Not run | 99.8% · 15.4%462 right, 84 wrong, 1 abstained · run 1280 |
| Dexy-Decision-9B-V2 Open · 9.0B | 99.8% · 16.5%456 right, 90 wrong, 1 abstained · run 1262, full bank restricted | Not run | 99.8% · 16.5%456 right, 90 wrong, 1 abstained · run 1260 |
| vera-medium-omni Open · 4.2B | 99.1% · 20.8%429 right, 113 wrong, 5 abstained · run 1271, full bank restricted | Not run | 98.7% · 16.1%453 right, 87 wrong, 7 abstained · run 1269 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 99.6% · 21.7%427 right, 118 wrong, 2 abstained · run 1239, full bank restricted | Not run | 100.0% · 17.2%453 right, 94 wrong, 0 abstained · run 1083 |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 95.4% · 18.4%426 right, 96 wrong, 25 abstained · run 1049, full bank restricted | Not rundecision models only | 97.8% · 17.2%443 right, 92 wrong, 10 abstained · run 890 |
| Clef-Flash Open · 9.0B | 97.8% · 21.5%420 right, 115 wrong, 12 abstained · run 664, full bank restricted | Not run | 99.1% · 16.6%452 right, 90 wrong, 5 abstained · run 660 |
| Verdin-4B-v1.1 Open · 4.0B | 99.8% · 23.3%419 right, 127 wrong, 1 abstained · run 1251, full bank restricted | Not run | 99.6% · 17.2%451 right, 94 wrong, 2 abstained · run 1249 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 98.9% · 24.6%408 right, 133 wrong, 6 abstained · run 537 | Not run | 99.1% · 18.5%442 right, 100 wrong, 5 abstained · run 535 |
| KnowLine 4B Gen5.5 Open · 4.0B | 99.5% · 21.7%426 right, 118 wrong, 3 abstained · run 967, full bank restricted | Not run | 99.8% · 19.2%441 right, 105 wrong, 1 abstained · run 963 |
| Chinese Decision Pointer 4B Open · 4.0B | 99.3% · 23.8%414 right, 129 wrong, 4 abstained · run 1213, full bank restricted | Not run | 99.3% · 19.2%439 right, 104 wrong, 4 abstained · run 1211 |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 98.4% · 26.0%398 right, 140 wrong, 5 abstained · run 1147, full bank restricted | Not rundecision models only | 99.3% · 19.5%437 right, 106 wrong, 4 abstained · run 1144 |
| Qwen3-14B reasoning off · chat Open · 15B | 99.8% · 22.3%424 right, 122 wrong, 1 abstained · run 772, full bank restricted | Not rundecision models only | 99.6% · 19.8%437 right, 108 wrong, 2 abstained · run 771 |
| Metask-Jev-4B Open · 4.5B | 99.6% · 34.3%358 right, 187 wrong, 2 abstained · run 743, full bank restricted | Not run | 99.6% · 22.6%422 right, 123 wrong, 2 abstained · run 741 |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 90.7% · 46.2%267 right, 229 wrong, 43 abstained · run 1038, full bank restricted | Not rundecision models only | 95.6% · 21.6%410 right, 113 wrong, 21 abstained · run 1035 |
| Erk-14B reasoning off · chat Open · 14B | 99.8% · 23.6%417 right, 129 wrong, 1 abstained · run 765, full bank restricted | Not rundecision models only | 99.5% · 25.9%403 right, 141 wrong, 3 abstained · run 764 |
| LFM3-1.7B reasoning off · chat Open · 1.7B | 56.9% · 55.6%138 right, 173 wrong, 215 abstained · run 1234, full bank restricted | Not rundecision models only | 96.7% · 25.9%392 right, 137 wrong, 18 abstained · run 1225 |
| KnowLine-2B-Gen1 Open · 2.0B | Not run | Not run | 93.1% · 26.5%374 right, 135 wrong, 38 abstained · run 1059 |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 80.6% · 47.6%231 right, 210 wrong, 106 abstained · run 1090, full bank restricted | Not rundecision models only | 95.2% · 27.1%380 right, 141 wrong, 26 abstained · run 1062 |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 52.5% · 67.9%92 right, 195 wrong, 201 abstained · run 1298, full bank restricted | Not rundecision models only | 90.5% · 28.7%353 right, 142 wrong, 51 abstained · run 887 |
| GLiDE 2B Open · 1.9B | 87.9% · 43.5%272 right, 209 wrong, 66 abstained · run 921, full bank restricted | Not run | 91.4% · 26.4%368 right, 132 wrong, 47 abstained · run 919 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 94.0% · 45.7%279 right, 235 wrong, 33 abstained · run 979, full bank restricted | Not run | 94.1% · 30.5%358 right, 157 wrong, 32 abstained · run 975 |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 20.8% · 65.8%39 right, 75 wrong, 22 abstained · run 1304, full bank restricted | Not rundecision models only | 89.2% · 33.6%324 right, 164 wrong, 33 abstained · run 1095 |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 83.4% · 61.2%177 right, 279 wrong, 55 abstained · run 1219, full bank restricted | Not rundecision models only | 96.3% · 29.6%371 right, 156 wrong, 18 abstained · run 1216 |
| d1-2B Open · 1.9B | 70.6% · 53.6%179 right, 207 wrong, 161 abstained · run 712, full bank restricted | Not run | 72.4% · 24.5%299 right, 97 wrong, 151 abstained · run 710 |
| GLiDE 0.8B Open · 0.8B | 86.5% · 48.6%243 right, 230 wrong, 74 abstained · run 929, full bank restricted | Not run | 91.0% · 35.5%321 right, 177 wrong, 49 abstained · run 927 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 96.0% · 61.0%205 right, 320 wrong, 22 abstained · run 1017, full bank restricted | Not run | 97.8% · 40.6%318 right, 217 wrong, 12 abstained · run 1013 |
| GLiDE 0.4B Open · 0.4B | 13.2% · 76.4%17 right, 55 wrong, 475 abstained · run 937, full bank restricted | Not run | 72.0% · 34.3%259 right, 135 wrong, 153 abstained · run 935 |
| OneJev1.6-0.8B Open · 0.8B | 79.5% · 58.2%182 right, 253 wrong, 112 abstained · run 1031, full bank restricted | Not run | 86.5% · 39.5%286 right, 187 wrong, 74 abstained · run 1027 |
| KnowLine-0.8B-Gen1 Open · 0.8B | 97.3% · 61.1%207 right, 325 wrong, 15 abstained · run 1007, full bank restricted | Not run | 95.2% · 41.7%304 right, 217 wrong, 26 abstained · run 1003 |
| GLiClass v4 Edge Open · 0.4B | 89.4% · 72.0%137 right, 352 wrong, 58 abstained · run 1048, full bank restricted | Not run | 90.5% · 39.6%299 right, 196 wrong, 52 abstained · run 1044 |
| Decider 1 Closed · Efficient | Not run | Not run | Not run |
| Strands Decider 2B Open · 2.0B | 99.1% · 60.3%215 right, 327 wrong, 5 abstained · run 706, full bank restricted | Not run | 99.6% · 55.2%244 right, 301 wrong, 2 abstained · run 704 |
| Laya Open · 0.4B | 90.3% · 78.9%104 right, 390 wrong, 53 abstained · run 730, full bank restricted | Not run | 84.8% · 79.1%97 right, 367 wrong, 83 abstained · run 728 |
| GLiNER2.5-multi-Decide Open · 0.3B | 95.6% · 78.8%111 right, 412 wrong, 24 abstained · run 718, full bank restricted | Not run | 87.2% · 76.1%114 right, 363 wrong, 70 abstained · run 716 |
| GLiNER2.5 Multi Open · 0.3B | 94.9% · 82.5%91 right, 428 wrong, 28 abstained · run 724, full bank restricted | Not rundecision models only | 83.2% · 83.1%77 right, 378 wrong, 92 abstained · run 722 |
| Laya Multilingual Open · 0.3B | 77.3% · 84.9%64 right, 359 wrong, 124 abstained · run 737, full bank restricted | Not run | 62.0% · 82.6%59 right, 280 wrong, 208 abstained · run 734 |
| Model | Turkish with “Bilmiyorum”coverage · error rate among answered | Turkish with “I don't know”coverage · error rate among answered | English with “I don't know”coverage · error rate among answered |
|---|---|---|---|
| Claude Fable 5.2 max · chat Closed · Flagship | Queuedits run with the option has not started yet | Not rundecision models only | 99.3% · 1.8%533 right, 10 wrong, 4 abstained · run 1032 |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 99.3% · 1.8%533 right, 10 wrong, 4 abstained · run 1117, full bank restricted | Not rundecision models only | 99.3% · 2.9%527 right, 16 wrong, 4 abstained · run 1109 |
| GPT-6 Astra low · chat Closed · Flagship | 99.3% · 2.6%529 right, 14 wrong, 4 abstained · run 816, full bank restricted | Not rundecision models only | 99.3% · 3.3%525 right, 18 wrong, 4 abstained · run 647 |
| GPT-6 Astra high · chat Closed · Flagship | 99.3% · 3.1%526 right, 17 wrong, 4 abstained · run 818, full bank restricted | Not rundecision models only | 99.3% · 3.1%526 right, 17 wrong, 4 abstained · run 814 |
| GPT-6.1 Terra high · chat Closed · Workhorse | 99.1% · 3.5%523 right, 19 wrong, 5 abstained · run 807, full bank restricted | Not rundecision models only | 99.3% · 3.7%523 right, 20 wrong, 4 abstained · run 766 |
| GPT-6.1 Terra low · chat Closed · Workhorse | 99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 801, full bank restricted | Not rundecision models only | 99.3% · 4.1%521 right, 22 wrong, 4 abstained · run 751 |
| Claude Fable 5.1 high · chat Closed · Flagship | 99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 695, full bank restricted | Not rundecision models only | 99.6% · 4.6%520 right, 25 wrong, 2 abstained · run 655 |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 99.6% · 2.4%532 right, 13 wrong, 2 abstained · run 681, full bank restricted | Not rundecision models only | 99.6% · 3.3%527 right, 18 wrong, 2 abstained · run 657 |
| Gemini 4 Argon low · chat Closed · Flagship | 99.3% · 2.4%530 right, 13 wrong, 4 abstained · run 679, full bank restricted | Not rundecision models only | 99.1% · 4.1%520 right, 22 wrong, 5 abstained · run 652 |
| GPT-6 Astra medium · chat Closed · Flagship | 99.3% · 2.9%527 right, 16 wrong, 4 abstained · run 817, full bank restricted | Not rundecision models only | 99.3% · 3.7%523 right, 20 wrong, 4 abstained · run 813 |
| GPT-6.1 Sol low · chat Closed · Workhorse | 99.3% · 3.3%525 right, 18 wrong, 4 abstained · run 799, full bank restricted | Not rundecision models only | 99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 744 |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 99.3% · 3.5%524 right, 19 wrong, 4 abstained · run 819, full bank restricted | Not rundecision models only | 99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 815 |
| GPT-6.1 Sol high · chat Closed · Workhorse | 99.3% · 2.9%527 right, 16 wrong, 4 abstained · run 802, full bank restricted | Not rundecision models only | 99.3% · 3.7%523 right, 20 wrong, 4 abstained · run 758 |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 99.5% · 3.3%526 right, 18 wrong, 3 abstained · run 691, full bank restricted | Not rundecision models only | 99.6% · 4.4%521 right, 24 wrong, 2 abstained · run 669 |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 99.3% · 3.5%524 right, 19 wrong, 4 abstained · run 810, full bank restricted | Not rundecision models only | 99.3% · 5.0%516 right, 27 wrong, 4 abstained · run 769 |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 99.8% · 4.6%521 right, 25 wrong, 1 abstained · run 803, full bank restricted | Not rundecision models only | 100.0% · 5.7%516 right, 31 wrong, 0 abstained · run 670 |
| Claude Opus 5.5 high · chat Closed · Flagship | 99.5% · 4.2%521 right, 23 wrong, 3 abstained · run 773, full bank restricted | Not rundecision models only | 99.6% · 5.3%516 right, 29 wrong, 2 abstained · run 665 |
| Qwen4-Max high · chat Closed · Workhorse | 99.8% · 5.1%518 right, 28 wrong, 1 abstained · run 1339, full bank restricted | Not rundecision models only | 100.0% · 5.3%518 right, 29 wrong, 0 abstained · run 1338 |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 99.5% · 4.4%520 right, 24 wrong, 3 abstained · run 676, full bank restricted | Not rundecision models only | 99.8% · 5.1%518 right, 28 wrong, 1 abstained · run 645 |
| Gemini 4 Carbon high · chat Closed · Flagship | 99.3% · 2.0%532 right, 11 wrong, 4 abstained · run 1121, full bank restricted | Not rundecision models only | 99.3% · 2.2%531 right, 12 wrong, 4 abstained · run 1067 |
| Grok 4.7 high · chat Closed · Workhorse | 98.5% · 3.9%518 right, 21 wrong, 8 abstained · run 675, full bank restricted | Not rundecision models only | 97.8% · 5.8%504 right, 31 wrong, 12 abstained · run 648 |
| Claude Opus 5.5 low · chat Closed · Flagship | 99.6% · 5.3%516 right, 29 wrong, 2 abstained · run 683, full bank restricted | Not rundecision models only | 99.6% · 7.5%504 right, 41 wrong, 2 abstained · run 649 |
| Claude Haiku 5.5 max · chat Closed · Efficient | 96.0% · 5.5%496 right, 29 wrong, 6 abstained · run 950, full bank restricted | Not rundecision models only | 96.3% · 8.9%480 right, 47 wrong, 4 abstained · run 951 |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 99.8% · 7.0%508 right, 38 wrong, 1 abstained · run 811, full bank restricted | Not rundecision models only | 99.5% · 7.2%505 right, 39 wrong, 3 abstained · run 809 |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 99.5% · 6.6%508 right, 36 wrong, 3 abstained · run 686, full bank restricted | Not rundecision models only | 99.5% · 7.0%506 right, 38 wrong, 3 abstained · run 651 |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 99.8% · 6.6%510 right, 36 wrong, 1 abstained · run 812, full bank restricted | Not rundecision models only | 100.0% · 6.9%509 right, 38 wrong, 0 abstained · run 677 |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 99.5% · 7.2%505 right, 39 wrong, 3 abstained · run 653, full bank restricted | Not rundecision models only | 99.3% · 9.0%494 right, 49 wrong, 4 abstained · run 646 |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 99.3% · 7.9%500 right, 43 wrong, 4 abstained · run 872, full bank restricted | Not rundecision models only | 99.5% · 8.3%499 right, 45 wrong, 3 abstained · run 873 |
| Gemma 4 31B reasoning off · chat Open · 31B | 99.6% · 7.7%503 right, 42 wrong, 2 abstained · run 776, full bank restricted | Not rundecision models only | 99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 775 |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 99.8% · 9.5%494 right, 52 wrong, 1 abstained · run 789, full bank restricted | Not rundecision models only | 100.0% · 10.8%488 right, 59 wrong, 0 abstained · run 786 |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 99.6% · 10.1%490 right, 55 wrong, 2 abstained · run 825, full bank restricted | Not rundecision models only | 99.8% · 11.2%485 right, 61 wrong, 1 abstained · run 824 |
| Muse Ember max · chat Closed · Efficient | 99.3% · 8.7%496 right, 47 wrong, 4 abstained · run 940, full bank restricted | Not rundecision models only | 99.5% · 9.9%490 right, 54 wrong, 3 abstained · run 943 |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 99.5% · 12.7%475 right, 69 wrong, 3 abstained · run 879, full bank restricted | Not rundecision models only | 99.8% · 10.6%488 right, 58 wrong, 1 abstained · run 880 |
| Qwen3.8-27B low · chat Open · 27B | 99.8% · 8.8%498 right, 48 wrong, 1 abstained · run 666, full bank restricted | Not rundecision models only | 99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 671 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 100.0% · 11.3%485 right, 62 wrong, 0 abstained · run 868, full bank restricted | Not rundecision models only | 100.0% · 11.0%487 right, 60 wrong, 0 abstained · run 869 |
| GLM-5.3 low · chat Open · Workhorse | 99.5% · 11.2%483 right, 61 wrong, 3 abstained · run 656, full bank restricted | Not rundecision models only | 99.6% · 10.3%489 right, 56 wrong, 2 abstained · run 650 |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 99.5% · 15.1%462 right, 82 wrong, 3 abstained · run 822, full bank restricted | Not rundecision models only | 99.3% · 12.3%476 right, 67 wrong, 4 abstained · run 821 |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 100.0% · 10.4%490 right, 57 wrong, 0 abstained · run 806, full bank restricted | Not rundecision models only | 99.3% · 12.9%473 right, 70 wrong, 4 abstained · run 805 |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 99.1% · 16.2%454 right, 88 wrong, 2 abstained · run 1150, full bank restricted | Not rundecision models only | 99.6% · 15.2%462 right, 83 wrong, 2 abstained · run 1073 |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 99.5% · 10.1%489 right, 55 wrong, 3 abstained · run 987, full bank restricted | Not rundecision models only | 99.3% · 12.3%476 right, 67 wrong, 4 abstained · run 988 |
| GPT-6 Luna low · chat Closed · Efficient | 99.6% · 11.9%480 right, 65 wrong, 2 abstained · run 796, full bank restricted | Not rundecision models only | 99.1% · 12.7%473 right, 69 wrong, 5 abstained · run 735 |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 800, full bank restricted | Not rundecision models only | 99.5% · 10.8%485 right, 59 wrong, 3 abstained · run 668 |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 99.3% · 13.3%471 right, 72 wrong, 4 abstained · run 674, full bank restricted | Not rundecision models only | 99.5% · 14.3%466 right, 78 wrong, 3 abstained · run 673 |
| Qwen3.8-27B yes/no scoring Open · 27B | Not runyes/no scoring cannot offer it | Not rundecision models only | Not runyes/no scoring cannot offer it |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 99.6% · 16.5%455 right, 90 wrong, 1 abstained · run 864, full bank restricted | Not rundecision models only | 99.5% · 15.6%459 right, 85 wrong, 3 abstained · run 865 |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 99.1% · 15.1%460 right, 82 wrong, 5 abstained · run 690, full bank restricted | Not rundecision models only | 99.3% · 15.8%457 right, 86 wrong, 4 abstained · run 654 |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 99.3% · 13.8%468 right, 75 wrong, 4 abstained · run 682, full bank restricted | Not rundecision models only | 99.5% · 15.4%460 right, 84 wrong, 3 abstained · run 680 |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 95.4% · 18.4%426 right, 96 wrong, 25 abstained · run 1049, full bank restricted | Not rundecision models only | 97.8% · 17.2%443 right, 92 wrong, 10 abstained · run 890 |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 98.4% · 26.0%398 right, 140 wrong, 5 abstained · run 1147, full bank restricted | Not rundecision models only | 99.3% · 19.5%437 right, 106 wrong, 4 abstained · run 1144 |
| Qwen3-14B reasoning off · chat Open · 15B | 99.8% · 22.3%424 right, 122 wrong, 1 abstained · run 772, full bank restricted | Not rundecision models only | 99.6% · 19.8%437 right, 108 wrong, 2 abstained · run 771 |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 90.7% · 46.2%267 right, 229 wrong, 43 abstained · run 1038, full bank restricted | Not rundecision models only | 95.6% · 21.6%410 right, 113 wrong, 21 abstained · run 1035 |
| Erk-14B reasoning off · chat Open · 14B | 99.8% · 23.6%417 right, 129 wrong, 1 abstained · run 765, full bank restricted | Not rundecision models only | 99.5% · 25.9%403 right, 141 wrong, 3 abstained · run 764 |
| LFM3-1.7B reasoning off · chat Open · 1.7B | 56.9% · 55.6%138 right, 173 wrong, 215 abstained · run 1234, full bank restricted | Not rundecision models only | 96.7% · 25.9%392 right, 137 wrong, 18 abstained · run 1225 |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 80.6% · 47.6%231 right, 210 wrong, 106 abstained · run 1090, full bank restricted | Not rundecision models only | 95.2% · 27.1%380 right, 141 wrong, 26 abstained · run 1062 |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 52.5% · 67.9%92 right, 195 wrong, 201 abstained · run 1298, full bank restricted | Not rundecision models only | 90.5% · 28.7%353 right, 142 wrong, 51 abstained · run 887 |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 20.8% · 65.8%39 right, 75 wrong, 22 abstained · run 1304, full bank restricted | Not rundecision models only | 89.2% · 33.6%324 right, 164 wrong, 33 abstained · run 1095 |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 83.4% · 61.2%177 right, 279 wrong, 55 abstained · run 1219, full bank restricted | Not rundecision models only | 96.3% · 29.6%371 right, 156 wrong, 18 abstained · run 1216 |
| GLiNER2.5 Multi Open · 0.3B | 94.9% · 82.5%91 right, 428 wrong, 28 abstained · run 724, full bank restricted | Not rundecision models only | 83.2% · 83.1%77 right, 378 wrong, 92 abstained · run 722 |
| Model | Turkish with “Bilmiyorum”coverage · error rate among answered | Turkish with “I don't know”coverage · error rate among answered | English with “I don't know”coverage · error rate among answered |
|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | 100.0% · 6.2%513 right, 34 wrong, 0 abstained · run 959, full bank restricted | Not run | 100.0% · 6.8%510 right, 37 wrong, 0 abstained · run 955 |
| Copernicus-31B Open · 31B | 99.8% · 6.0%513 right, 33 wrong, 1 abstained · run 1172, full bank restricted | Not run | 100.0% · 7.1%508 right, 39 wrong, 0 abstained · run 1170 |
| REAX 1.3 Open · 57B | 99.8% · 6.4%511 right, 35 wrong, 1 abstained · run 833, full bank restricted | Not run | 99.5% · 7.7%502 right, 42 wrong, 3 abstained · run 831 |
| KnowLine-31B-Gen2 Open · 31B | 100.0% · 5.3%518 right, 29 wrong, 0 abstained · run 1331, full bank restricted | Not run | 100.0% · 7.3%507 right, 40 wrong, 0 abstained · run 1329 |
| Karpo Decision 31B Open · 31B | 99.8% · 5.5%516 right, 30 wrong, 1 abstained · run 1163, full bank restricted | Not run | 100.0% · 7.7%505 right, 42 wrong, 0 abstained · run 1161 |
| Gemini Decision 1 20261111 Closed · Efficient | 99.1% · 7.6%501 right, 41 wrong, 5 abstained · run 1023, full bank restricted | Not run | 99.1% · 9.2%492 right, 50 wrong, 5 abstained · run 1019 |
| Logits Sigma-1 Duo Open · 57B | 100.0% · 6.6%511 right, 36 wrong, 0 abstained · run 1233, full bank restricted | Not run | 99.6% · 8.1%501 right, 44 wrong, 2 abstained · run 1231 |
| GLiDE 27B Open · 27B | 98.4% · 7.2%499 right, 39 wrong, 9 abstained · run 905, full bank restricted | Not run | 96.0% · 6.9%489 right, 36 wrong, 22 abstained · run 903 |
| d1-ultra Closed · Efficient | 94.5% · 5.0%491 right, 26 wrong, 30 abstained · run 350, full bank restricted | 95.1% · 5.4%492 right, 28 wrong, 27 abstained · run 511, full bank restricted | 90.7% · 7.9%457 right, 39 wrong, 51 abstained · run 449 |
| Sieve 31B v3 Open · 31B | 100.0% · 6.0%514 right, 33 wrong, 0 abstained · run 1253, full bank restricted | Not run | 99.8% · 8.2%501 right, 45 wrong, 1 abstained · run 1066 |
| Turion Jev 31B Open · 31B | 99.6% · 6.1%512 right, 33 wrong, 2 abstained · run 1285, full bank restricted | Not run | 99.3% · 7.9%500 right, 43 wrong, 4 abstained · run 1077 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 99.5% · 9.4%493 right, 51 wrong, 3 abstained · run 642, full bank restricted | Not run | 98.2% · 10.4%481 right, 56 wrong, 10 abstained · run 644 |
| GLiDE v1.2 Open · 27B | 98.9% · 8.7%494 right, 47 wrong, 6 abstained · run 369, full bank restricted | 98.4% · 8.7%491 right, 47 wrong, 9 abstained · run 525, full bank restricted | 94.5% · 7.0%481 right, 36 wrong, 30 abstained · run 523 |
| d1.2-ultra Closed · Efficient | 88.8% · 3.9%467 right, 19 wrong, 61 abstained · run 1367, full bank restricted | 89.0% · 3.7%469 right, 18 wrong, 60 abstained · run 1378, full bank restricted | 80.1% · 3.0%425 right, 13 wrong, 109 abstained · run 1363 |
| Mammoth-1 Open · 27B | 95.6% · 6.3%490 right, 33 wrong, 24 abstained · run 386, full bank restricted | 95.4% · 6.1%490 right, 32 wrong, 25 abstained · run 519, full bank restricted | 93.2% · 7.1%474 right, 36 wrong, 37 abstained · run 505 |
| d3 Open · 26B | 100.0% · 9.7%494 right, 53 wrong, 0 abstained · run 358, full bank restricted | 100.0% · 9.5%495 right, 52 wrong, 0 abstained · run 544, full bank restricted | 99.8% · 10.6%488 right, 58 wrong, 1 abstained · run 793 |
| Drex Max Open · 27B | 99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 367, full bank restricted | 99.8% · 9.0%497 right, 49 wrong, 1 abstained · run 542, full bank restricted | 99.6% · 11.0%485 right, 60 wrong, 2 abstained · run 798 |
| Perplexity Decider 27B v1.3 Open · 27B | 100.0% · 9.5%495 right, 52 wrong, 0 abstained · run 1141, full bank restricted | Not run | 99.1% · 9.6%490 right, 52 wrong, 5 abstained · run 1139 |
| d3-ultra Open · 26B | 99.8% · 8.4%500 right, 46 wrong, 1 abstained · run 363, full bank restricted | 99.6% · 9.4%494 right, 51 wrong, 2 abstained · run 540, full bank restricted | 99.1% · 10.3%486 right, 56 wrong, 5 abstained · run 795 |
| d1.2 Closed · Efficient | 90.3% · 7.1%459 right, 35 wrong, 53 abstained · run 1365, full bank restricted | 91.6% · 8.4%459 right, 42 wrong, 46 abstained · run 1377, full bank restricted | 89.2% · 7.6%451 right, 37 wrong, 59 abstained · run 1361 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 100.0% · 9.1%497 right, 50 wrong, 0 abstained · run 372, full bank restricted | 99.8% · 9.3%495 right, 51 wrong, 1 abstained · run 531, full bank restricted | 99.6% · 11.7%481 right, 64 wrong, 2 abstained · run 529 |
| Drex Max v1.1 Closed · Efficient (provisional) | 98.4% · 9.5%487 right, 51 wrong, 9 abstained · run 352, full bank restricted | 96.7% · 9.1%481 right, 48 wrong, 18 abstained · run 513, full bank restricted | 92.1% · 8.1%463 right, 41 wrong, 43 abstained · run 447 |
| Perplexity Decider 27B v1 Open · 27B | Not run | Not run | Not run |
| Perplexity Decider 27B v1.2 API Open · 27B | 100.0% · 9.3%496 right, 51 wrong, 0 abstained · run 641, full bank restricted | Not run | 99.5% · 11.6%481 right, 63 wrong, 3 abstained · run 643 |
| d1.1 Closed · Efficient | Not run | Not run | Not run |
| Jeffan-9B-1211 Open · 9.0B | 99.5% · 13.1%473 right, 71 wrong, 3 abstained · run 1199, full bank restricted | Not run | 100.0% · 13.0%476 right, 71 wrong, 0 abstained · run 1197 |
| d1 Closed · Efficient | 99.8% · 16.3%457 right, 89 wrong, 1 abstained · run 472 | Not run | 99.6% · 12.3%478 right, 67 wrong, 2 abstained · run 445 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 99.8% · 17.4%451 right, 95 wrong, 1 abstained · run 1297, full bank restricted | Not run | 99.6% · 13.6%471 right, 74 wrong, 2 abstained · run 1295 |
| Clef Open · 27B | 99.8% · 12.3%479 right, 67 wrong, 1 abstained · run 788, full bank restricted | Not run | 99.8% · 13.7%471 right, 75 wrong, 1 abstained · run 785 |
| KnowLine-12B-Gen1 Open · 12B | 100.0% · 12.2%480 right, 67 wrong, 0 abstained · run 1318, full bank restricted | Not run | 99.8% · 13.9%470 right, 76 wrong, 1 abstained · run 1316 |
| GLiDE 9B Open · 7.9B | 98.4% · 15.4%455 right, 83 wrong, 9 abstained · run 913, full bank restricted | Not run | 97.4% · 13.1%463 right, 70 wrong, 14 abstained · run 911 |
| OneJev-27B Open · 27B | 99.8% · 11.5%483 right, 63 wrong, 1 abstained · run 782, full bank restricted | Not run | 99.5% · 13.2%472 right, 72 wrong, 3 abstained · run 780 |
| Jev 1.14 Closed · Efficient | 99.5% · 12.7%475 right, 69 wrong, 3 abstained · run 470 | Not run | 99.5% · 13.2%472 right, 72 wrong, 3 abstained · run 446 |
| Cerno 9B v1.2 Open · 9.0B | 100.0% · 16.6%456 right, 91 wrong, 0 abstained · run 1204, full bank restricted | Not run | 99.6% · 14.1%468 right, 77 wrong, 2 abstained · run 1087 |
| Drex 2.0 Open · 35B MoE · 3.0B active | Not run | Not run | Not run |
| Jev 1.13 Closed · Efficient | Not run | Not run | Not run |
| KnowLine-9B-Gen1 Open · 9.0B | 100.0% · 17.9%449 right, 98 wrong, 0 abstained · run 1181, full bank restricted | Not run | 99.8% · 13.9%470 right, 76 wrong, 1 abstained · run 1179 |
| SubQ Snap 1.1 Closed · Efficient | 96.5% · 9.7%477 right, 51 wrong, 19 abstained · run 392, full bank restricted | 96.5% · 8.5%483 right, 45 wrong, 19 abstained · run 512, full bank restricted | 96.2% · 12.0%463 right, 63 wrong, 21 abstained · run 448 |
| SubQ Snap Preview Closed · Efficient | Not run | Not run | Not run |
| Adjudex 9B v5 Open · 9.0B | 100.0% · 16.3%458 right, 89 wrong, 0 abstained · run 1190, full bank restricted | Not run | 100.0% · 14.1%470 right, 77 wrong, 0 abstained · run 1188 |
| Mercury Decide Closed · Efficient | 98.5% · 13.2%468 right, 71 wrong, 8 abstained · run 489 | Not run | 98.0% · 13.2%465 right, 71 wrong, 11 abstained · run 450 |
| Microsoft-Decision-1 Closed · Efficient | 99.6% · 16.1%457 right, 88 wrong, 2 abstained · run 487 | Not run | 99.5% · 14.2%467 right, 77 wrong, 3 abstained · run 451 |
| Elephant-1 Open · 9.0B | 97.3% · 15.4%450 right, 82 wrong, 15 abstained · run 382, full bank restricted | 95.6% · 15.7%441 right, 82 wrong, 24 abstained · run 515, full bank restricted | 95.8% · 12.8%457 right, 67 wrong, 23 abstained · run 501 |
| Open-Jev-27B-v1.1 Open · 27B | 98.9% · 14.4%463 right, 78 wrong, 6 abstained · run 768, full bank restricted | Not run | 97.1% · 14.3%455 right, 76 wrong, 16 abstained · run 763 |
| KnowLine-9B-Gen2 Open · 9.0B | 100.0% · 17.4%452 right, 95 wrong, 0 abstained · run 1325, full bank restricted | Not run | 99.8% · 14.3%468 right, 78 wrong, 1 abstained · run 1323 |
| celeris-1-decision Closed · Efficient | Not run | Not run | 83.2% · 8.1%418 right, 37 wrong, 92 abstained · run 548 |
| GPT-6 Luna Decisions API Closed · Efficient | 98.9% · 14.4%463 right, 78 wrong, 6 abstained · run 453 | Not run | 98.2% · 17.9%441 right, 96 wrong, 10 abstained · run 444 |
| Cygnet Open · 12B | 99.1% · 14.9%461 right, 81 wrong, 5 abstained · run 757, full bank restricted | Not run | 98.2% · 14.7%458 right, 79 wrong, 10 abstained · run 755 |
| Winnow-12B Open · 12B | 98.5% · 15.8%454 right, 85 wrong, 8 abstained · run 750, full bank restricted | Not run | 97.3% · 16.0%447 right, 85 wrong, 15 abstained · run 748 |
| Perplexity Decider Flash v1.3 Open · 4.0B | 98.4% · 20.4%428 right, 110 wrong, 9 abstained · run 1130, full bank restricted | Not run | 98.7% · 16.7%450 right, 90 wrong, 7 abstained · run 1128 |
| Valen-Decision-1.5-9B Open · 9.0B | 100.0% · 20.1%437 right, 110 wrong, 0 abstained · run 1282, full bank restricted | Not run | 99.8% · 15.4%462 right, 84 wrong, 1 abstained · run 1280 |
| Dexy-Decision-9B-V2 Open · 9.0B | 99.8% · 16.5%456 right, 90 wrong, 1 abstained · run 1262, full bank restricted | Not run | 99.8% · 16.5%456 right, 90 wrong, 1 abstained · run 1260 |
| vera-medium-omni Open · 4.2B | 99.1% · 20.8%429 right, 113 wrong, 5 abstained · run 1271, full bank restricted | Not run | 98.7% · 16.1%453 right, 87 wrong, 7 abstained · run 1269 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 99.6% · 21.7%427 right, 118 wrong, 2 abstained · run 1239, full bank restricted | Not run | 100.0% · 17.2%453 right, 94 wrong, 0 abstained · run 1083 |
| Clef-Flash Open · 9.0B | 97.8% · 21.5%420 right, 115 wrong, 12 abstained · run 664, full bank restricted | Not run | 99.1% · 16.6%452 right, 90 wrong, 5 abstained · run 660 |
| Verdin-4B-v1.1 Open · 4.0B | 99.8% · 23.3%419 right, 127 wrong, 1 abstained · run 1251, full bank restricted | Not run | 99.6% · 17.2%451 right, 94 wrong, 2 abstained · run 1249 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 98.9% · 24.6%408 right, 133 wrong, 6 abstained · run 537 | Not run | 99.1% · 18.5%442 right, 100 wrong, 5 abstained · run 535 |
| KnowLine 4B Gen5.5 Open · 4.0B | 99.5% · 21.7%426 right, 118 wrong, 3 abstained · run 967, full bank restricted | Not run | 99.8% · 19.2%441 right, 105 wrong, 1 abstained · run 963 |
| Chinese Decision Pointer 4B Open · 4.0B | 99.3% · 23.8%414 right, 129 wrong, 4 abstained · run 1213, full bank restricted | Not run | 99.3% · 19.2%439 right, 104 wrong, 4 abstained · run 1211 |
| Metask-Jev-4B Open · 4.5B | 99.6% · 34.3%358 right, 187 wrong, 2 abstained · run 743, full bank restricted | Not run | 99.6% · 22.6%422 right, 123 wrong, 2 abstained · run 741 |
| KnowLine-2B-Gen1 Open · 2.0B | Not run | Not run | 93.1% · 26.5%374 right, 135 wrong, 38 abstained · run 1059 |
| GLiDE 2B Open · 1.9B | 87.9% · 43.5%272 right, 209 wrong, 66 abstained · run 921, full bank restricted | Not run | 91.4% · 26.4%368 right, 132 wrong, 47 abstained · run 919 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 94.0% · 45.7%279 right, 235 wrong, 33 abstained · run 979, full bank restricted | Not run | 94.1% · 30.5%358 right, 157 wrong, 32 abstained · run 975 |
| d1-2B Open · 1.9B | 70.6% · 53.6%179 right, 207 wrong, 161 abstained · run 712, full bank restricted | Not run | 72.4% · 24.5%299 right, 97 wrong, 151 abstained · run 710 |
| GLiDE 0.8B Open · 0.8B | 86.5% · 48.6%243 right, 230 wrong, 74 abstained · run 929, full bank restricted | Not run | 91.0% · 35.5%321 right, 177 wrong, 49 abstained · run 927 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 96.0% · 61.0%205 right, 320 wrong, 22 abstained · run 1017, full bank restricted | Not run | 97.8% · 40.6%318 right, 217 wrong, 12 abstained · run 1013 |
| GLiDE 0.4B Open · 0.4B | 13.2% · 76.4%17 right, 55 wrong, 475 abstained · run 937, full bank restricted | Not run | 72.0% · 34.3%259 right, 135 wrong, 153 abstained · run 935 |
| OneJev1.6-0.8B Open · 0.8B | 79.5% · 58.2%182 right, 253 wrong, 112 abstained · run 1031, full bank restricted | Not run | 86.5% · 39.5%286 right, 187 wrong, 74 abstained · run 1027 |
| KnowLine-0.8B-Gen1 Open · 0.8B | 97.3% · 61.1%207 right, 325 wrong, 15 abstained · run 1007, full bank restricted | Not run | 95.2% · 41.7%304 right, 217 wrong, 26 abstained · run 1003 |
| GLiClass v4 Edge Open · 0.4B | 89.4% · 72.0%137 right, 352 wrong, 58 abstained · run 1048, full bank restricted | Not run | 90.5% · 39.6%299 right, 196 wrong, 52 abstained · run 1044 |
| Decider 1 Closed · Efficient | Not run | Not run | Not run |
| Strands Decider 2B Open · 2.0B | 99.1% · 60.3%215 right, 327 wrong, 5 abstained · run 706, full bank restricted | Not run | 99.6% · 55.2%244 right, 301 wrong, 2 abstained · run 704 |
| Laya Open · 0.4B | 90.3% · 78.9%104 right, 390 wrong, 53 abstained · run 730, full bank restricted | Not run | 84.8% · 79.1%97 right, 367 wrong, 83 abstained · run 728 |
| GLiNER2.5-multi-Decide Open · 0.3B | 95.6% · 78.8%111 right, 412 wrong, 24 abstained · run 718, full bank restricted | Not run | 87.2% · 76.1%114 right, 363 wrong, 70 abstained · run 716 |
| Laya Multilingual Open · 0.3B | 77.3% · 84.9%64 right, 359 wrong, 124 abstained · run 737, full bank restricted | Not run | 62.0% · 82.6%59 right, 280 wrong, 208 abstained · run 734 |
Each cell: coverage · error rate among answered, then the counts and the run. Not run: no such run for a decision model, or the Turkish “I don't know” run, which is for decision models only; Running: a chat model's run still running.
Risk and coverage in Turkish
The 547 kept questions in Turkish: each setup's five-option run answering only at or above a confidence score (the line), and a run with the option (the marker: “Bilmiyorum” “I don't know”) · lower is better at the same coverage
- Claude Fable 5.2 · max · chat: confidence, run 980
- Gemini 4 Hydrogen · high · chat: confidence, run 1114 · “Bilmiyorum”, run 1117
- GPT-6 Astra · low · chat: confidence, run 48 · “Bilmiyorum”, run 816
- GPT-6 Astra · high · chat: confidence, run 277 · “Bilmiyorum”, run 818
- GPT-6.1 Terra · high · chat: confidence, run 275 · “Bilmiyorum”, run 807
- GPT-6.1 Terra · low · chat: confidence, run 274 · “Bilmiyorum”, run 801
- Claude Fable 5.1 · high · chat: confidence, run 310 · “Bilmiyorum”, run 695
- Gemini 3.8 Flash · high · chat: confidence, run 140 · “Bilmiyorum”, run 681
- Gemini 4 Argon · low · chat: confidence, run 557 · “Bilmiyorum”, run 679
- GPT-6 Astra · medium · chat: confidence, run 610 · “Bilmiyorum”, run 817
- GPT-6.1 Sol · low · chat: confidence, run 124 · “Bilmiyorum”, run 799
- GPT-6 Astra · low · chat · 1 per request: confidence, run 191 · “Bilmiyorum”, run 819
- GPT-6.1 Sol · high · chat: confidence, run 276 · “Bilmiyorum”, run 802
- Gemini 3.8 Flash · low · chat · 1 per request: confidence, run 188 · “Bilmiyorum”, run 691
- GPT-6.1 Sol · low · chat · 1 per request: confidence, run 190 · “Bilmiyorum”, run 810
- Claude Opus 5.5 · low · chat · 1 per request: confidence, run 192 · “Bilmiyorum”, run 803
- Claude Opus 5.5 · high · chat: confidence, run 278 · “Bilmiyorum”, run 773
- Qwen4-Max · high · chat: confidence, run 1336 · “Bilmiyorum”, run 1339
- Gemini 3.8 Flash · low · chat: confidence, run 123 · “Bilmiyorum”, run 676
- Gemini 4 Carbon · high · chat: confidence, run 1108 · “Bilmiyorum”, run 1121
- KnowLine-31B-Gen1.5: confidence, run 957 · “Bilmiyorum”, run 959
- Grok 4.7 · high · chat: confidence, run 311 · “Bilmiyorum”, run 675
- Claude Opus 5.5 · low · chat: confidence, run 47 · “Bilmiyorum”, run 683
- Copernicus-31B: confidence, run 1165 · “Bilmiyorum”, run 1172
- Claude Haiku 5.5 · max · chat: confidence, run 948 · “Bilmiyorum”, run 950
- DeepSeek-V4.1-Flash · reasoning on · chat · 1017: confidence, run 585 · “Bilmiyorum”, run 811
- REAX 1.3: confidence, run 827 · “Bilmiyorum”, run 833
- Kimi K3.1 Pro · low · chat: confidence, run 320 · “Bilmiyorum”, run 686
- KnowLine-31B-Gen2: confidence, run 1314 · “Bilmiyorum”, run 1331
- Karpo Decision 31B: confidence, run 1156 · “Bilmiyorum”, run 1163
- Claude Sonnet 5.5 · high · chat · 1 per request: confidence, run 193 · “Bilmiyorum”, run 812
- Kimi K3.1 Pro · no reasoning · chat: confidence, run 315 · “Bilmiyorum”, run 653
- Qwen3.5-397B-A17B · high · chat: confidence, run 870 · “Bilmiyorum”, run 872
- Gemini Decision 1 · 20261111: confidence, run 1021 · “Bilmiyorum”, run 1023
- Logits Sigma-1 Duo: confidence, run 1221 · “Bilmiyorum”, run 1233
- GLiDE 27B: confidence, run 899 · “Bilmiyorum”, run 905
- d1-ultra: confidence, run 199 · “Bilmiyorum”, run 350 · “I don't know”, run 511
- Sieve 31B v3: confidence, run 1241 · “Bilmiyorum”, run 1253
- Turion Jev 31B: confidence, run 1273 · “Bilmiyorum”, run 1285
- Gemma 4 31B · reasoning off · chat: confidence, run 119 · “Bilmiyorum”, run 776
- Perplexity Decider 27B v1.2 · API · with web: confidence, run 640 · “Bilmiyorum”, run 642
- GLiDE · v1.2: confidence, run 248 · “Bilmiyorum”, run 369 · “I don't know”, run 525
- d1.2-ultra: confidence, run 1347 · “Bilmiyorum”, run 1367 · “I don't know”, run 1378
- Mammoth-1: confidence, run 374 · “Bilmiyorum”, run 386 · “I don't know”, run 519
- Gemma 4 31B · reasoning off · chat · 1 per request: confidence, run 189 · “Bilmiyorum”, run 789
- d3: confidence, run 259 · “Bilmiyorum”, run 358 · “I don't know”, run 544
- Drex Max: confidence, run 322 · “Bilmiyorum”, run 367 · “I don't know”, run 542
- GLM-5.3-Flash · reasoning on · chat: confidence, run 160 · “Bilmiyorum”, run 825
- Muse Ember · max · chat: confidence, run 939 · “Bilmiyorum”, run 940
- Perplexity Decider 27B v1.3: confidence, run 1134 · “Bilmiyorum”, run 1141
- d3-ultra: confidence, run 269 · “Bilmiyorum”, run 363 · “I don't know”, run 540
- d1.2: confidence, run 1346 · “Bilmiyorum”, run 1365 · “I don't know”, run 1377
- MiMo-V2.6-Flash · high · chat: confidence, run 877 · “Bilmiyorum”, run 879
- Qwen3.8-27B · low · chat: confidence, run 318 · “Bilmiyorum”, run 666
- Perplexity Decider 27B v1.2 · DGX Spark: confidence, run 217 · “Bilmiyorum”, run 372 · “I don't know”, run 531
- Drex Max v1.1: confidence, run 326 · “Bilmiyorum”, run 352 · “I don't know”, run 513
- Perplexity Decider 27B v1: confidence, run 44
- Perplexity Decider 27B v1.2 · API: confidence, run 633 · “Bilmiyorum”, run 641
- Inkling-Small · high · chat · Jul 2026: confidence, run 866 · “Bilmiyorum”, run 868
- d1.1: confidence, run 254
- GLM-5.3 · low · chat: confidence, run 314 · “Bilmiyorum”, run 656
- Jeffan-9B-1211: confidence, run 1192 · “Bilmiyorum”, run 1199
- GLM-5.3-Flash · reasoning off · chat: confidence, run 159 · “Bilmiyorum”, run 822
- DeepSeek-V4.1-Flash · reasoning off · chat · 1017: confidence, run 558 · “Bilmiyorum”, run 806
- Qwen3.5-4B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1150
- Qwen3.8 Flash · no reasoning · chat: confidence, run 985 · “Bilmiyorum”, run 987
- d1: confidence, run 52 · “Bilmiyorum”, run 472
- StartLux-Decision-v1.1-9B-Instant: confidence, run 1290 · “Bilmiyorum”, run 1297
- Clef: confidence, run 125 · “Bilmiyorum”, run 788
- KnowLine-12B-Gen1: confidence, run 1309 · “Bilmiyorum”, run 1318
- GLiDE 9B: confidence, run 907 · “Bilmiyorum”, run 913
- GPT-6 Luna · low · chat: confidence, run 133 · “Bilmiyorum”, run 796
- OneJev-27B: confidence, run 264 · “Bilmiyorum”, run 782
- Claude Sonnet 5.5 · high · chat: confidence, run 55 · “Bilmiyorum”, run 800
- Jev 1.14: confidence, run 194 · “Bilmiyorum”, run 470
- Cerno 9B v1.2: confidence, run 1201 · “Bilmiyorum”, run 1204
- Drex 2.0: confidence, run 334
- Jev 1.13: confidence, run 49
- KnowLine-9B-Gen1: confidence, run 1174 · “Bilmiyorum”, run 1181
- SubQ Snap 1.1: confidence, run 346 · “Bilmiyorum”, run 392 · “I don't know”, run 512
- SubQ Snap Preview: confidence, run 177
- Adjudex 9B v5: confidence, run 1183 · “Bilmiyorum”, run 1190
- Mercury Decide: confidence, run 204 · “Bilmiyorum”, run 489
- Microsoft-Decision-1: confidence, run 205 · “Bilmiyorum”, run 487
- Elephant-1: confidence, run 373 · “Bilmiyorum”, run 382 · “I don't know”, run 515
- Open-Jev-27B-v1.1: confidence, run 45 · “Bilmiyorum”, run 768
- KnowLine-9B-Gen2: confidence, run 1306 · “Bilmiyorum”, run 1325
- Qwen3.8-27B · stated confidence · reasoning off · chat: confidence, run 53 · “Bilmiyorum”, run 674
- Qwen3.8-27B · yes/no scoring: confidence, run 50
- MiniMax-M3.1-Flash · high · chat: confidence, run 862 · “Bilmiyorum”, run 864
- celeris-1-decision: confidence, run 150
- GPT-6 Luna · Decisions API: confidence, run 143 · “Bilmiyorum”, run 453
- Claude Sonnet 5.5 · low · chat: confidence, run 54 · “Bilmiyorum”, run 690
- Cygnet: confidence, run 127 · “Bilmiyorum”, run 757
- Qwen3.8-27B · vote share · reasoning off · chat: confidence, run 83 · “Bilmiyorum”, run 682
- Winnow-12B: confidence, run 128 · “Bilmiyorum”, run 750
- Perplexity Decider Flash v1.3: confidence, run 1123 · “Bilmiyorum”, run 1130
- Valen-Decision-1.5-9B: confidence, run 1275 · “Bilmiyorum”, run 1282
- Dexy-Decision-9B-V2: confidence, run 1255 · “Bilmiyorum”, run 1262
- vera-medium-omni: confidence, run 1264 · “Bilmiyorum”, run 1271
- StartLux-Decision-v1.1-4B-Instant: confidence, run 1236 · “Bilmiyorum”, run 1239
- Gemma 4 E4B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1049
- Clef-Flash: confidence, run 139 · “Bilmiyorum”, run 664
- Verdin-4B-v1.1: confidence, run 1244 · “Bilmiyorum”, run 1251
- Perplexity Decider Flash v1.2: confidence, run 223 · “Bilmiyorum”, run 537
- KnowLine 4B Gen5.5: confidence, run 965 · “Bilmiyorum”, run 967
- Chinese Decision Pointer 4B: confidence, run 1206 · “Bilmiyorum”, run 1213
- Qwen3.5-4B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1147
- Qwen3-14B · reasoning off · chat: confidence, run 131 · “Bilmiyorum”, run 772
- Metask-Jev-4B: confidence, run 126 · “Bilmiyorum”, run 743
- LFM2.5-2.6B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1038
- Erk-14B · reasoning off · chat: confidence, run 132 · “Bilmiyorum”, run 765
- LFM3-1.7B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1234
- KnowLine-2B-Gen1: confidence, run 1079
- LFM3-1.7B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1090
- LFM2.5-8B-A1B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1298
- GLiDE 2B: confidence, run 915 · “Bilmiyorum”, run 921
- StartLux-Decision-v1.1-2B-Instant: confidence, run 977 · “Bilmiyorum”, run 979
- MiniCPM5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1304
- Qwen3.5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1219
- d1-2B: confidence, run 340 · “Bilmiyorum”, run 712
- GLiDE 0.8B: confidence, run 923 · “Bilmiyorum”, run 929
- StartLux-Decision-v1.1-0.8B-Instant: confidence, run 1015 · “Bilmiyorum”, run 1017
- GLiDE 0.4B: confidence, run 931 · “Bilmiyorum”, run 937
- OneJev1.6-0.8B: confidence, run 1029 · “Bilmiyorum”, run 1031
- KnowLine-0.8B-Gen1: confidence, run 1005 · “Bilmiyorum”, run 1007
- GLiClass v4 Edge: confidence, run 1046 · “Bilmiyorum”, run 1048
- Decider 1: confidence, run 243
- Strands Decider 2B: confidence, run 141 · “Bilmiyorum”, run 706
- Laya: confidence, run 51 · “Bilmiyorum”, run 730
- GLiNER2.5-multi-Decide: confidence, run 135 · “Bilmiyorum”, run 718
- GLiNER2.5 Multi: confidence, run 122 · “Bilmiyorum”, run 724
- Laya Multilingual: confidence, run 46 · “Bilmiyorum”, run 737
Risk and coverage in English
The 547 kept questions in English: each setup's five-option run answering only at or above a confidence score (the line), and a run with the option (the marker: “I don't know”) · lower is better at the same coverage
- Claude Fable 5.2 · max · chat: confidence, run 971 · “I don't know”, run 1032
- Gemini 4 Hydrogen · high · chat: confidence, run 1107 · “I don't know”, run 1109
- GPT-6 Astra · low · chat: confidence, run 616 · “I don't know”, run 647
- GPT-6 Astra · high · chat: confidence, run 615 · “I don't know”, run 814
- GPT-6.1 Terra · high · chat: confidence, run 620 · “I don't know”, run 766
- GPT-6.1 Terra · low · chat: confidence, run 621 · “I don't know”, run 751
- Claude Fable 5.1 · high · chat: confidence, run 580 · “I don't know”, run 655
- Gemini 3.8 Flash · high · chat: confidence, run 583 · “I don't know”, run 657
- Gemini 4 Argon · low · chat: confidence, run 593 · “I don't know”, run 652
- GPT-6 Astra · medium · chat: confidence, run 624 · “I don't know”, run 813
- GPT-6.1 Sol · low · chat: confidence, run 619 · “I don't know”, run 744
- GPT-6 Astra · low · chat · 1 per request: confidence, run 622 · “I don't know”, run 815
- GPT-6.1 Sol · high · chat: confidence, run 618 · “I don't know”, run 758
- Gemini 3.8 Flash · low · chat · 1 per request: confidence, run 589 · “I don't know”, run 669
- GPT-6.1 Sol · low · chat · 1 per request: confidence, run 623 · “I don't know”, run 769
- Claude Opus 5.5 · low · chat · 1 per request: confidence, run 592 · “I don't know”, run 670
- Claude Opus 5.5 · high · chat: confidence, run 587 · “I don't know”, run 665
- Qwen4-Max · high · chat: confidence, run 1334 · “I don't know”, run 1338
- Gemini 3.8 Flash · low · chat: confidence, run 578 · “I don't know”, run 645
- Gemini 4 Carbon · high · chat: confidence, run 1055 · “I don't know”, run 1067
- KnowLine-31B-Gen1.5: confidence, run 953 · “I don't know”, run 955
- Grok 4.7 · high · chat: confidence, run 579 · “I don't know”, run 648
- Claude Opus 5.5 · low · chat: confidence, run 588 · “I don't know”, run 649
- Copernicus-31B: confidence, run 1168 · “I don't know”, run 1170
- Claude Haiku 5.5 · max · chat: confidence, run 949 · “I don't know”, run 951
- DeepSeek-V4.1-Flash · reasoning on · chat · 1017: confidence, run 808 · “I don't know”, run 809
- REAX 1.3: confidence, run 829 · “I don't know”, run 831
- Kimi K3.1 Pro · low · chat: confidence, run 584 · “I don't know”, run 651
- KnowLine-31B-Gen2: confidence, run 1327 · “I don't know”, run 1329
- Karpo Decision 31B: confidence, run 1159 · “I don't know”, run 1161
- Claude Sonnet 5.5 · high · chat · 1 per request: confidence, run 594 · “I don't know”, run 677
- Kimi K3.1 Pro · no reasoning · chat: confidence, run 582 · “I don't know”, run 646
- Qwen3.5-397B-A17B · high · chat: confidence, run 871 · “I don't know”, run 873
- Gemini Decision 1 · 20261111: confidence, run 1009 · “I don't know”, run 1019
- Logits Sigma-1 Duo: confidence, run 1229 · “I don't know”, run 1231
- GLiDE 27B: confidence, run 901 · “I don't know”, run 903
- d1-ultra: confidence, run 411 · “I don't know”, run 449
- Sieve 31B v3: confidence, run 1064 · “I don't know”, run 1066
- Turion Jev 31B: confidence, run 1075 · “I don't know”, run 1077
- Gemma 4 31B · reasoning off · chat: confidence, run 774 · “I don't know”, run 775
- Perplexity Decider 27B v1.2 · API · with web: confidence, run 639 · “I don't know”, run 644
- GLiDE · v1.2: confidence, run 521 · “I don't know”, run 523
- d1.2-ultra: confidence, run 1359 · “I don't know”, run 1363
- Mammoth-1: confidence, run 503 · “I don't know”, run 505
- Gemma 4 31B · reasoning off · chat · 1 per request: confidence, run 783 · “I don't know”, run 786
- d3: confidence, run 554 · “I don't know”, run 793
- Drex Max: confidence, run 552 · “I don't know”, run 798
- GLM-5.3-Flash · reasoning on · chat: confidence, run 823 · “I don't know”, run 824
- Muse Ember · max · chat: confidence, run 942 · “I don't know”, run 943
- Perplexity Decider 27B v1.3: confidence, run 1137 · “I don't know”, run 1139
- d3-ultra: confidence, run 550 · “I don't know”, run 795
- d1.2: confidence, run 1357 · “I don't know”, run 1361
- MiMo-V2.6-Flash · high · chat: confidence, run 878 · “I don't know”, run 880
- Qwen3.8-27B · low · chat: confidence, run 667 · “I don't know”, run 671
- Perplexity Decider 27B v1.2 · DGX Spark: confidence, run 527 · “I don't know”, run 529
- Drex Max v1.1: confidence, run 399 · “I don't know”, run 447
- Perplexity Decider 27B v1: confidence, run 626
- Perplexity Decider 27B v1.2 · API: confidence, run 637 · “I don't know”, run 643
- Inkling-Small · high · chat · Jul 2026: confidence, run 867 · “I don't know”, run 869
- d1.1: confidence, run 565
- GLM-5.3 · low · chat: confidence, run 577 · “I don't know”, run 650
- Jeffan-9B-1211: confidence, run 1195 · “I don't know”, run 1197
- GLM-5.3-Flash · reasoning off · chat: confidence, run 820 · “I don't know”, run 821
- DeepSeek-V4.1-Flash · reasoning off · chat · 1017: confidence, run 804 · “I don't know”, run 805
- Qwen3.5-4B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1073
- Qwen3.8 Flash · no reasoning · chat: confidence, run 986 · “I don't know”, run 988
- d1: confidence, run 430 · “I don't know”, run 445
- StartLux-Decision-v1.1-9B-Instant: confidence, run 1293 · “I don't know”, run 1295
- Clef: confidence, run 613 · “I don't know”, run 785
- KnowLine-12B-Gen1: confidence, run 1313 · “I don't know”, run 1316
- GLiDE 9B: confidence, run 909 · “I don't know”, run 911
- GPT-6 Luna · low · chat: confidence, run 617 · “I don't know”, run 735
- OneJev-27B: confidence, run 778 · “I don't know”, run 780
- Claude Sonnet 5.5 · high · chat: confidence, run 590 · “I don't know”, run 668
- Jev 1.14: confidence, run 410 · “I don't know”, run 446
- Cerno 9B v1.2: confidence, run 1085 · “I don't know”, run 1087
- Drex 2.0: confidence, run 566
- Jev 1.13: confidence, run 564
- KnowLine-9B-Gen1: confidence, run 1177 · “I don't know”, run 1179
- SubQ Snap 1.1: confidence, run 413 · “I don't know”, run 448
- SubQ Snap Preview: confidence, run 595
- Adjudex 9B v5: confidence, run 1186 · “I don't know”, run 1188
- Mercury Decide: confidence, run 398 · “I don't know”, run 450
- Microsoft-Decision-1: confidence, run 412 · “I don't know”, run 451
- Elephant-1: confidence, run 499 · “I don't know”, run 501
- Open-Jev-27B-v1.1: confidence, run 760 · “I don't know”, run 763
- KnowLine-9B-Gen2: confidence, run 1321 · “I don't know”, run 1323
- Qwen3.8-27B · stated confidence · reasoning off · chat: confidence, run 672 · “I don't know”, run 673
- Qwen3.8-27B · yes/no scoring: confidence, run 696
- MiniMax-M3.1-Flash · high · chat: confidence, run 863 · “I don't know”, run 865
- celeris-1-decision: confidence, run 546 · “I don't know”, run 548
- GPT-6 Luna · Decisions API: confidence, run 397 · “I don't know”, run 444
- Claude Sonnet 5.5 · low · chat: confidence, run 591 · “I don't know”, run 654
- Cygnet: confidence, run 753 · “I don't know”, run 755
- Qwen3.8-27B · vote share · reasoning off · chat: confidence, run 678 · “I don't know”, run 680
- Winnow-12B: confidence, run 746 · “I don't know”, run 748
- Perplexity Decider Flash v1.3: confidence, run 1126 · “I don't know”, run 1128
- Valen-Decision-1.5-9B: confidence, run 1278 · “I don't know”, run 1280
- Dexy-Decision-9B-V2: confidence, run 1258 · “I don't know”, run 1260
- vera-medium-omni: confidence, run 1267 · “I don't know”, run 1269
- StartLux-Decision-v1.1-4B-Instant: confidence, run 1081 · “I don't know”, run 1083
- Gemma 4 E4B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 890
- Clef-Flash: confidence, run 614 · “I don't know”, run 660
- Verdin-4B-v1.1: confidence, run 1247 · “I don't know”, run 1249
- Perplexity Decider Flash v1.2: confidence, run 533 · “I don't know”, run 535
- KnowLine 4B Gen5.5: confidence, run 961 · “I don't know”, run 963
- Chinese Decision Pointer 4B: confidence, run 1209 · “I don't know”, run 1211
- Qwen3.5-4B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1144
- Qwen3-14B · reasoning off · chat: confidence, run 770 · “I don't know”, run 771
- Metask-Jev-4B: confidence, run 739 · “I don't know”, run 741
- LFM2.5-2.6B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1035
- Erk-14B · reasoning off · chat: confidence, run 762 · “I don't know”, run 764
- LFM3-1.7B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1225
- KnowLine-2B-Gen1: confidence, run 1057 · “I don't know”, run 1059
- LFM3-1.7B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1062
- LFM2.5-8B-A1B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 887
- GLiDE 2B: confidence, run 917 · “I don't know”, run 919
- StartLux-Decision-v1.1-2B-Instant: confidence, run 973 · “I don't know”, run 975
- MiniCPM5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1095
- Qwen3.5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1216
- d1-2B: confidence, run 708 · “I don't know”, run 710
- GLiDE 0.8B: confidence, run 925 · “I don't know”, run 927
- StartLux-Decision-v1.1-0.8B-Instant: confidence, run 1011 · “I don't know”, run 1013
- GLiDE 0.4B: confidence, run 933 · “I don't know”, run 935
- OneJev1.6-0.8B: confidence, run 1025 · “I don't know”, run 1027
- KnowLine-0.8B-Gen1: confidence, run 1001 · “I don't know”, run 1003
- GLiClass v4 Edge: confidence, run 1042 · “I don't know”, run 1044
- Decider 1: confidence, run 596
- Strands Decider 2B: confidence, run 702 · “I don't know”, run 704
- Laya: confidence, run 726 · “I don't know”, run 728
- GLiNER2.5-multi-Decide: confidence, run 714 · “I don't know”, run 716
- GLiNER2.5 Multi: confidence, run 720 · “I don't know”, run 722
- Laya Multilingual: confidence, run 732 · “I don't know”, run 734
Risk and coverage in Turkish
The 547 kept questions in Turkish: each setup's five-option run answering only at or above a confidence score (the line), and a run with the option (the marker: “Bilmiyorum” “I don't know”) · lower is better at the same coverage
- Claude Fable 5.2 · max · chat: confidence, run 980
- Gemini 4 Hydrogen · high · chat: confidence, run 1114 · “Bilmiyorum”, run 1117
- GPT-6 Astra · low · chat: confidence, run 48 · “Bilmiyorum”, run 816
- GPT-6 Astra · high · chat: confidence, run 277 · “Bilmiyorum”, run 818
- GPT-6.1 Terra · high · chat: confidence, run 275 · “Bilmiyorum”, run 807
- GPT-6.1 Terra · low · chat: confidence, run 274 · “Bilmiyorum”, run 801
- Claude Fable 5.1 · high · chat: confidence, run 310 · “Bilmiyorum”, run 695
- Gemini 3.8 Flash · high · chat: confidence, run 140 · “Bilmiyorum”, run 681
- Gemini 4 Argon · low · chat: confidence, run 557 · “Bilmiyorum”, run 679
- GPT-6 Astra · medium · chat: confidence, run 610 · “Bilmiyorum”, run 817
- GPT-6.1 Sol · low · chat: confidence, run 124 · “Bilmiyorum”, run 799
- GPT-6 Astra · low · chat · 1 per request: confidence, run 191 · “Bilmiyorum”, run 819
- GPT-6.1 Sol · high · chat: confidence, run 276 · “Bilmiyorum”, run 802
- Gemini 3.8 Flash · low · chat · 1 per request: confidence, run 188 · “Bilmiyorum”, run 691
- GPT-6.1 Sol · low · chat · 1 per request: confidence, run 190 · “Bilmiyorum”, run 810
- Claude Opus 5.5 · low · chat · 1 per request: confidence, run 192 · “Bilmiyorum”, run 803
- Claude Opus 5.5 · high · chat: confidence, run 278 · “Bilmiyorum”, run 773
- Qwen4-Max · high · chat: confidence, run 1336 · “Bilmiyorum”, run 1339
- Gemini 3.8 Flash · low · chat: confidence, run 123 · “Bilmiyorum”, run 676
- Gemini 4 Carbon · high · chat: confidence, run 1108 · “Bilmiyorum”, run 1121
- Grok 4.7 · high · chat: confidence, run 311 · “Bilmiyorum”, run 675
- Claude Opus 5.5 · low · chat: confidence, run 47 · “Bilmiyorum”, run 683
- Claude Haiku 5.5 · max · chat: confidence, run 948 · “Bilmiyorum”, run 950
- DeepSeek-V4.1-Flash · reasoning on · chat · 1017: confidence, run 585 · “Bilmiyorum”, run 811
- Kimi K3.1 Pro · low · chat: confidence, run 320 · “Bilmiyorum”, run 686
- Claude Sonnet 5.5 · high · chat · 1 per request: confidence, run 193 · “Bilmiyorum”, run 812
- Kimi K3.1 Pro · no reasoning · chat: confidence, run 315 · “Bilmiyorum”, run 653
- Qwen3.5-397B-A17B · high · chat: confidence, run 870 · “Bilmiyorum”, run 872
- Gemma 4 31B · reasoning off · chat: confidence, run 119 · “Bilmiyorum”, run 776
- Gemma 4 31B · reasoning off · chat · 1 per request: confidence, run 189 · “Bilmiyorum”, run 789
- GLM-5.3-Flash · reasoning on · chat: confidence, run 160 · “Bilmiyorum”, run 825
- Muse Ember · max · chat: confidence, run 939 · “Bilmiyorum”, run 940
- MiMo-V2.6-Flash · high · chat: confidence, run 877 · “Bilmiyorum”, run 879
- Qwen3.8-27B · low · chat: confidence, run 318 · “Bilmiyorum”, run 666
- Inkling-Small · high · chat · Jul 2026: confidence, run 866 · “Bilmiyorum”, run 868
- GLM-5.3 · low · chat: confidence, run 314 · “Bilmiyorum”, run 656
- GLM-5.3-Flash · reasoning off · chat: confidence, run 159 · “Bilmiyorum”, run 822
- DeepSeek-V4.1-Flash · reasoning off · chat · 1017: confidence, run 558 · “Bilmiyorum”, run 806
- Qwen3.5-4B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1150
- Qwen3.8 Flash · no reasoning · chat: confidence, run 985 · “Bilmiyorum”, run 987
- GPT-6 Luna · low · chat: confidence, run 133 · “Bilmiyorum”, run 796
- Claude Sonnet 5.5 · high · chat: confidence, run 55 · “Bilmiyorum”, run 800
- Qwen3.8-27B · stated confidence · reasoning off · chat: confidence, run 53 · “Bilmiyorum”, run 674
- Qwen3.8-27B · yes/no scoring: confidence, run 50
- MiniMax-M3.1-Flash · high · chat: confidence, run 862 · “Bilmiyorum”, run 864
- Claude Sonnet 5.5 · low · chat: confidence, run 54 · “Bilmiyorum”, run 690
- Qwen3.8-27B · vote share · reasoning off · chat: confidence, run 83 · “Bilmiyorum”, run 682
- Gemma 4 E4B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1049
- Qwen3.5-4B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1147
- Qwen3-14B · reasoning off · chat: confidence, run 131 · “Bilmiyorum”, run 772
- LFM2.5-2.6B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1038
- Erk-14B · reasoning off · chat: confidence, run 132 · “Bilmiyorum”, run 765
- LFM3-1.7B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1234
- LFM3-1.7B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1090
- LFM2.5-8B-A1B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1298
- MiniCPM5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1304
- Qwen3.5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1219
- GLiNER2.5 Multi: confidence, run 122 · “Bilmiyorum”, run 724
Risk and coverage in English
The 547 kept questions in English: each setup's five-option run answering only at or above a confidence score (the line), and a run with the option (the marker: “I don't know”) · lower is better at the same coverage
- Claude Fable 5.2 · max · chat: confidence, run 971 · “I don't know”, run 1032
- Gemini 4 Hydrogen · high · chat: confidence, run 1107 · “I don't know”, run 1109
- GPT-6 Astra · low · chat: confidence, run 616 · “I don't know”, run 647
- GPT-6 Astra · high · chat: confidence, run 615 · “I don't know”, run 814
- GPT-6.1 Terra · high · chat: confidence, run 620 · “I don't know”, run 766
- GPT-6.1 Terra · low · chat: confidence, run 621 · “I don't know”, run 751
- Claude Fable 5.1 · high · chat: confidence, run 580 · “I don't know”, run 655
- Gemini 3.8 Flash · high · chat: confidence, run 583 · “I don't know”, run 657
- Gemini 4 Argon · low · chat: confidence, run 593 · “I don't know”, run 652
- GPT-6 Astra · medium · chat: confidence, run 624 · “I don't know”, run 813
- GPT-6.1 Sol · low · chat: confidence, run 619 · “I don't know”, run 744
- GPT-6 Astra · low · chat · 1 per request: confidence, run 622 · “I don't know”, run 815
- GPT-6.1 Sol · high · chat: confidence, run 618 · “I don't know”, run 758
- Gemini 3.8 Flash · low · chat · 1 per request: confidence, run 589 · “I don't know”, run 669
- GPT-6.1 Sol · low · chat · 1 per request: confidence, run 623 · “I don't know”, run 769
- Claude Opus 5.5 · low · chat · 1 per request: confidence, run 592 · “I don't know”, run 670
- Claude Opus 5.5 · high · chat: confidence, run 587 · “I don't know”, run 665
- Qwen4-Max · high · chat: confidence, run 1334 · “I don't know”, run 1338
- Gemini 3.8 Flash · low · chat: confidence, run 578 · “I don't know”, run 645
- Gemini 4 Carbon · high · chat: confidence, run 1055 · “I don't know”, run 1067
- Grok 4.7 · high · chat: confidence, run 579 · “I don't know”, run 648
- Claude Opus 5.5 · low · chat: confidence, run 588 · “I don't know”, run 649
- Claude Haiku 5.5 · max · chat: confidence, run 949 · “I don't know”, run 951
- DeepSeek-V4.1-Flash · reasoning on · chat · 1017: confidence, run 808 · “I don't know”, run 809
- Kimi K3.1 Pro · low · chat: confidence, run 584 · “I don't know”, run 651
- Claude Sonnet 5.5 · high · chat · 1 per request: confidence, run 594 · “I don't know”, run 677
- Kimi K3.1 Pro · no reasoning · chat: confidence, run 582 · “I don't know”, run 646
- Qwen3.5-397B-A17B · high · chat: confidence, run 871 · “I don't know”, run 873
- Gemma 4 31B · reasoning off · chat: confidence, run 774 · “I don't know”, run 775
- Gemma 4 31B · reasoning off · chat · 1 per request: confidence, run 783 · “I don't know”, run 786
- GLM-5.3-Flash · reasoning on · chat: confidence, run 823 · “I don't know”, run 824
- Muse Ember · max · chat: confidence, run 942 · “I don't know”, run 943
- MiMo-V2.6-Flash · high · chat: confidence, run 878 · “I don't know”, run 880
- Qwen3.8-27B · low · chat: confidence, run 667 · “I don't know”, run 671
- Inkling-Small · high · chat · Jul 2026: confidence, run 867 · “I don't know”, run 869
- GLM-5.3 · low · chat: confidence, run 577 · “I don't know”, run 650
- GLM-5.3-Flash · reasoning off · chat: confidence, run 820 · “I don't know”, run 821
- DeepSeek-V4.1-Flash · reasoning off · chat · 1017: confidence, run 804 · “I don't know”, run 805
- Qwen3.5-4B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1073
- Qwen3.8 Flash · no reasoning · chat: confidence, run 986 · “I don't know”, run 988
- GPT-6 Luna · low · chat: confidence, run 617 · “I don't know”, run 735
- Claude Sonnet 5.5 · high · chat: confidence, run 590 · “I don't know”, run 668
- Qwen3.8-27B · stated confidence · reasoning off · chat: confidence, run 672 · “I don't know”, run 673
- Qwen3.8-27B · yes/no scoring: confidence, run 696
- MiniMax-M3.1-Flash · high · chat: confidence, run 863 · “I don't know”, run 865
- Claude Sonnet 5.5 · low · chat: confidence, run 591 · “I don't know”, run 654
- Qwen3.8-27B · vote share · reasoning off · chat: confidence, run 678 · “I don't know”, run 680
- Gemma 4 E4B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 890
- Qwen3.5-4B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1144
- Qwen3-14B · reasoning off · chat: confidence, run 770 · “I don't know”, run 771
- LFM2.5-2.6B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1035
- Erk-14B · reasoning off · chat: confidence, run 762 · “I don't know”, run 764
- LFM3-1.7B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1225
- LFM3-1.7B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1062
- LFM2.5-8B-A1B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 887
- MiniCPM5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1095
- Qwen3.5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1216
- GLiNER2.5 Multi: confidence, run 720 · “I don't know”, run 722
Risk and coverage in Turkish
The 547 kept questions in Turkish: each setup's five-option run answering only at or above a confidence score (the line), and a run with the option (the marker: “Bilmiyorum” “I don't know”) · lower is better at the same coverage
- KnowLine-31B-Gen1.5: confidence, run 957 · “Bilmiyorum”, run 959
- Copernicus-31B: confidence, run 1165 · “Bilmiyorum”, run 1172
- REAX 1.3: confidence, run 827 · “Bilmiyorum”, run 833
- KnowLine-31B-Gen2: confidence, run 1314 · “Bilmiyorum”, run 1331
- Karpo Decision 31B: confidence, run 1156 · “Bilmiyorum”, run 1163
- Gemini Decision 1 · 20261111: confidence, run 1021 · “Bilmiyorum”, run 1023
- Logits Sigma-1 Duo: confidence, run 1221 · “Bilmiyorum”, run 1233
- GLiDE 27B: confidence, run 899 · “Bilmiyorum”, run 905
- d1-ultra: confidence, run 199 · “Bilmiyorum”, run 350 · “I don't know”, run 511
- Sieve 31B v3: confidence, run 1241 · “Bilmiyorum”, run 1253
- Turion Jev 31B: confidence, run 1273 · “Bilmiyorum”, run 1285
- Perplexity Decider 27B v1.2 · API · with web: confidence, run 640 · “Bilmiyorum”, run 642
- GLiDE · v1.2: confidence, run 248 · “Bilmiyorum”, run 369 · “I don't know”, run 525
- d1.2-ultra: confidence, run 1347 · “Bilmiyorum”, run 1367 · “I don't know”, run 1378
- Mammoth-1: confidence, run 374 · “Bilmiyorum”, run 386 · “I don't know”, run 519
- d3: confidence, run 259 · “Bilmiyorum”, run 358 · “I don't know”, run 544
- Drex Max: confidence, run 322 · “Bilmiyorum”, run 367 · “I don't know”, run 542
- Perplexity Decider 27B v1.3: confidence, run 1134 · “Bilmiyorum”, run 1141
- d3-ultra: confidence, run 269 · “Bilmiyorum”, run 363 · “I don't know”, run 540
- d1.2: confidence, run 1346 · “Bilmiyorum”, run 1365 · “I don't know”, run 1377
- Perplexity Decider 27B v1.2 · DGX Spark: confidence, run 217 · “Bilmiyorum”, run 372 · “I don't know”, run 531
- Drex Max v1.1: confidence, run 326 · “Bilmiyorum”, run 352 · “I don't know”, run 513
- Perplexity Decider 27B v1: confidence, run 44
- Perplexity Decider 27B v1.2 · API: confidence, run 633 · “Bilmiyorum”, run 641
- d1.1: confidence, run 254
- Jeffan-9B-1211: confidence, run 1192 · “Bilmiyorum”, run 1199
- d1: confidence, run 52 · “Bilmiyorum”, run 472
- StartLux-Decision-v1.1-9B-Instant: confidence, run 1290 · “Bilmiyorum”, run 1297
- Clef: confidence, run 125 · “Bilmiyorum”, run 788
- KnowLine-12B-Gen1: confidence, run 1309 · “Bilmiyorum”, run 1318
- GLiDE 9B: confidence, run 907 · “Bilmiyorum”, run 913
- OneJev-27B: confidence, run 264 · “Bilmiyorum”, run 782
- Jev 1.14: confidence, run 194 · “Bilmiyorum”, run 470
- Cerno 9B v1.2: confidence, run 1201 · “Bilmiyorum”, run 1204
- Drex 2.0: confidence, run 334
- Jev 1.13: confidence, run 49
- KnowLine-9B-Gen1: confidence, run 1174 · “Bilmiyorum”, run 1181
- SubQ Snap 1.1: confidence, run 346 · “Bilmiyorum”, run 392 · “I don't know”, run 512
- SubQ Snap Preview: confidence, run 177
- Adjudex 9B v5: confidence, run 1183 · “Bilmiyorum”, run 1190
- Mercury Decide: confidence, run 204 · “Bilmiyorum”, run 489
- Microsoft-Decision-1: confidence, run 205 · “Bilmiyorum”, run 487
- Elephant-1: confidence, run 373 · “Bilmiyorum”, run 382 · “I don't know”, run 515
- Open-Jev-27B-v1.1: confidence, run 45 · “Bilmiyorum”, run 768
- KnowLine-9B-Gen2: confidence, run 1306 · “Bilmiyorum”, run 1325
- celeris-1-decision: confidence, run 150
- GPT-6 Luna · Decisions API: confidence, run 143 · “Bilmiyorum”, run 453
- Cygnet: confidence, run 127 · “Bilmiyorum”, run 757
- Winnow-12B: confidence, run 128 · “Bilmiyorum”, run 750
- Perplexity Decider Flash v1.3: confidence, run 1123 · “Bilmiyorum”, run 1130
- Valen-Decision-1.5-9B: confidence, run 1275 · “Bilmiyorum”, run 1282
- Dexy-Decision-9B-V2: confidence, run 1255 · “Bilmiyorum”, run 1262
- vera-medium-omni: confidence, run 1264 · “Bilmiyorum”, run 1271
- StartLux-Decision-v1.1-4B-Instant: confidence, run 1236 · “Bilmiyorum”, run 1239
- Clef-Flash: confidence, run 139 · “Bilmiyorum”, run 664
- Verdin-4B-v1.1: confidence, run 1244 · “Bilmiyorum”, run 1251
- Perplexity Decider Flash v1.2: confidence, run 223 · “Bilmiyorum”, run 537
- KnowLine 4B Gen5.5: confidence, run 965 · “Bilmiyorum”, run 967
- Chinese Decision Pointer 4B: confidence, run 1206 · “Bilmiyorum”, run 1213
- Metask-Jev-4B: confidence, run 126 · “Bilmiyorum”, run 743
- KnowLine-2B-Gen1: confidence, run 1079
- GLiDE 2B: confidence, run 915 · “Bilmiyorum”, run 921
- StartLux-Decision-v1.1-2B-Instant: confidence, run 977 · “Bilmiyorum”, run 979
- d1-2B: confidence, run 340 · “Bilmiyorum”, run 712
- GLiDE 0.8B: confidence, run 923 · “Bilmiyorum”, run 929
- StartLux-Decision-v1.1-0.8B-Instant: confidence, run 1015 · “Bilmiyorum”, run 1017
- GLiDE 0.4B: confidence, run 931 · “Bilmiyorum”, run 937
- OneJev1.6-0.8B: confidence, run 1029 · “Bilmiyorum”, run 1031
- KnowLine-0.8B-Gen1: confidence, run 1005 · “Bilmiyorum”, run 1007
- GLiClass v4 Edge: confidence, run 1046 · “Bilmiyorum”, run 1048
- Decider 1: confidence, run 243
- Strands Decider 2B: confidence, run 141 · “Bilmiyorum”, run 706
- Laya: confidence, run 51 · “Bilmiyorum”, run 730
- GLiNER2.5-multi-Decide: confidence, run 135 · “Bilmiyorum”, run 718
- Laya Multilingual: confidence, run 46 · “Bilmiyorum”, run 737
Risk and coverage in English
The 547 kept questions in English: each setup's five-option run answering only at or above a confidence score (the line), and a run with the option (the marker: “I don't know”) · lower is better at the same coverage
- KnowLine-31B-Gen1.5: confidence, run 953 · “I don't know”, run 955
- Copernicus-31B: confidence, run 1168 · “I don't know”, run 1170
- REAX 1.3: confidence, run 829 · “I don't know”, run 831
- KnowLine-31B-Gen2: confidence, run 1327 · “I don't know”, run 1329
- Karpo Decision 31B: confidence, run 1159 · “I don't know”, run 1161
- Gemini Decision 1 · 20261111: confidence, run 1009 · “I don't know”, run 1019
- Logits Sigma-1 Duo: confidence, run 1229 · “I don't know”, run 1231
- GLiDE 27B: confidence, run 901 · “I don't know”, run 903
- d1-ultra: confidence, run 411 · “I don't know”, run 449
- Sieve 31B v3: confidence, run 1064 · “I don't know”, run 1066
- Turion Jev 31B: confidence, run 1075 · “I don't know”, run 1077
- Perplexity Decider 27B v1.2 · API · with web: confidence, run 639 · “I don't know”, run 644
- GLiDE · v1.2: confidence, run 521 · “I don't know”, run 523
- d1.2-ultra: confidence, run 1359 · “I don't know”, run 1363
- Mammoth-1: confidence, run 503 · “I don't know”, run 505
- d3: confidence, run 554 · “I don't know”, run 793
- Drex Max: confidence, run 552 · “I don't know”, run 798
- Perplexity Decider 27B v1.3: confidence, run 1137 · “I don't know”, run 1139
- d3-ultra: confidence, run 550 · “I don't know”, run 795
- d1.2: confidence, run 1357 · “I don't know”, run 1361
- Perplexity Decider 27B v1.2 · DGX Spark: confidence, run 527 · “I don't know”, run 529
- Drex Max v1.1: confidence, run 399 · “I don't know”, run 447
- Perplexity Decider 27B v1: confidence, run 626
- Perplexity Decider 27B v1.2 · API: confidence, run 637 · “I don't know”, run 643
- d1.1: confidence, run 565
- Jeffan-9B-1211: confidence, run 1195 · “I don't know”, run 1197
- d1: confidence, run 430 · “I don't know”, run 445
- StartLux-Decision-v1.1-9B-Instant: confidence, run 1293 · “I don't know”, run 1295
- Clef: confidence, run 613 · “I don't know”, run 785
- KnowLine-12B-Gen1: confidence, run 1313 · “I don't know”, run 1316
- GLiDE 9B: confidence, run 909 · “I don't know”, run 911
- OneJev-27B: confidence, run 778 · “I don't know”, run 780
- Jev 1.14: confidence, run 410 · “I don't know”, run 446
- Cerno 9B v1.2: confidence, run 1085 · “I don't know”, run 1087
- Drex 2.0: confidence, run 566
- Jev 1.13: confidence, run 564
- KnowLine-9B-Gen1: confidence, run 1177 · “I don't know”, run 1179
- SubQ Snap 1.1: confidence, run 413 · “I don't know”, run 448
- SubQ Snap Preview: confidence, run 595
- Adjudex 9B v5: confidence, run 1186 · “I don't know”, run 1188
- Mercury Decide: confidence, run 398 · “I don't know”, run 450
- Microsoft-Decision-1: confidence, run 412 · “I don't know”, run 451
- Elephant-1: confidence, run 499 · “I don't know”, run 501
- Open-Jev-27B-v1.1: confidence, run 760 · “I don't know”, run 763
- KnowLine-9B-Gen2: confidence, run 1321 · “I don't know”, run 1323
- celeris-1-decision: confidence, run 546 · “I don't know”, run 548
- GPT-6 Luna · Decisions API: confidence, run 397 · “I don't know”, run 444
- Cygnet: confidence, run 753 · “I don't know”, run 755
- Winnow-12B: confidence, run 746 · “I don't know”, run 748
- Perplexity Decider Flash v1.3: confidence, run 1126 · “I don't know”, run 1128
- Valen-Decision-1.5-9B: confidence, run 1278 · “I don't know”, run 1280
- Dexy-Decision-9B-V2: confidence, run 1258 · “I don't know”, run 1260
- vera-medium-omni: confidence, run 1267 · “I don't know”, run 1269
- StartLux-Decision-v1.1-4B-Instant: confidence, run 1081 · “I don't know”, run 1083
- Clef-Flash: confidence, run 614 · “I don't know”, run 660
- Verdin-4B-v1.1: confidence, run 1247 · “I don't know”, run 1249
- Perplexity Decider Flash v1.2: confidence, run 533 · “I don't know”, run 535
- KnowLine 4B Gen5.5: confidence, run 961 · “I don't know”, run 963
- Chinese Decision Pointer 4B: confidence, run 1209 · “I don't know”, run 1211
- Metask-Jev-4B: confidence, run 739 · “I don't know”, run 741
- KnowLine-2B-Gen1: confidence, run 1057 · “I don't know”, run 1059
- GLiDE 2B: confidence, run 917 · “I don't know”, run 919
- StartLux-Decision-v1.1-2B-Instant: confidence, run 973 · “I don't know”, run 975
- d1-2B: confidence, run 708 · “I don't know”, run 710
- GLiDE 0.8B: confidence, run 925 · “I don't know”, run 927
- StartLux-Decision-v1.1-0.8B-Instant: confidence, run 1011 · “I don't know”, run 1013
- GLiDE 0.4B: confidence, run 933 · “I don't know”, run 935
- OneJev1.6-0.8B: confidence, run 1025 · “I don't know”, run 1027
- KnowLine-0.8B-Gen1: confidence, run 1001 · “I don't know”, run 1003
- GLiClass v4 Edge: confidence, run 1042 · “I don't know”, run 1044
- Decider 1: confidence, run 596
- Strands Decider 2B: confidence, run 702 · “I don't know”, run 704
- Laya: confidence, run 726 · “I don't know”, run 728
- GLiNER2.5-multi-Decide: confidence, run 714 · “I don't know”, run 716
- Laya Multilingual: confidence, run 732 · “I don't know”, run 734
Info The table, the setups without an English run and our method, then the same with “I don't know”
About this test
Since 5 November 2026 every setup of the Text tab also runs in English, decision models and chat models alike, each with its Text tab run's settings and compared with that run. 133 of the Text tab's 137 setups have an English run so far, 75 decision models and 58 chat model setups; the other four are listed under Results with the reason each has none yet. This tab is separate from the leaderboard; it ranks nothing on the Text tab and changes none of its numbers.
The Text tab's reading questions (units 1 to 6; the grammar questions are left out) were translated into English by Claude Opus 5.5. The same model flagged the questions whose answer depends on Turkish wording; the 109 flagged questions are left out and the other 547 kept. Each of the 133 English setups answered the kept questions in English and is compared with its own Text tab run on the same 547 questions.
Download: results.json (the comparison's numbers, under english_reading), answers-english.json (each English run's answers, under english_runs), english-reading.json (the translator, its prompts, the flags, the question units and each question's hash of its English text; no question text) and kept-question-ids.txt (the 547 kept questions' IDs, one per line).
Results
The setups of the model group in view, ordered by accuracy in English, highest first. Each row compares the setup's first English run with its Text tab run, out of the 547 kept questions; an invalid answer counts as not right. Where a side has more than one run, its accuracy shows the range over its runs and its runs show how many answers changed between them. The paired counts, p and the Holm-adjusted p (over all 133 English setups, whatever the group shown) are those of the exact McNemar test; * marks a Holm-adjusted p below 0.05. LFM3-1.7B, LFM3-1.7B and MiniCPM5-2B are run with 3 sampling seeds: until all are in, the number is the first English run's, named with the seeds in so far; then it is the mean of the runs' accuracies, ranked by it, with their lowest and highest (min–max, a range, not a confidence interval) under it. The paired comparison with Turkish and the “I don't know” runs stay on their own runs.
| Model | English runsthe first is compared; answers changed between runs where more than one | Turkish runsthe Text tab run, then its repeats; answers changed between them | Turkish accuracythe Text tab run on the kept questions; range over its runs; higher is better | English accuracythe first English run; range over its runs; higher is better | Differencepoints higher in English than in Turkish | Right only in Englishquestions the English run got right and the Turkish run did not | Right only in Turkishquestions the Turkish run got right and the English run did not | pexact McNemar | Holm padjusted over all 133 English setups |
|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.2 max · chat Closed · Flagship | run 971 | run 980 | 97.8%535 of 547 right | 98.0%536 of 547 right | +0.2 | 5 | 4 | 1 | 1 |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | run 1107 | run 1114 | 97.1%531 of 547 right | 96.7%529 of 547 right | -0.4 | 3 | 5 | 0.73 | 1 |
| GPT-6 Astra low · chat Closed · Flagship | run 616 | run 48 | 96.7%529 of 547 right | 96.3%527 of 547 right | -0.4 | 4 | 6 | 0.75 | 1 |
| GPT-6 Astra high · chat Closed · Flagship | run 615 | run 277 | 96.9%530 of 547 right | 96.2%526 of 547 right | -0.7 | 5 | 9 | 0.42 | 1 |
| GPT-6.1 Terra high · chat Closed · Workhorse | run 620 | run 275 | 96.3%527 of 547 right | 96.2%526 of 547 right | -0.2 | 9 | 10 | 1 | 1 |
| GPT-6.1 Terra low · chat Closed · Workhorse | run 621 | run 274 | 95.6%523 of 547 right | 96.2%526 of 547 right | +0.5 | 13 | 10 | 0.68 | 1 |
| Claude Fable 5.1 high · chat Closed · Flagship | run 580 | run 310 | 95.4%522 of 547 right | 96.0%525 of 547 right | +0.5 | 12 | 9 | 0.66 | 1 |
| Gemini 3.8 Flash high · chat Closed · Workhorse | run 583 | run 140 | 96.9%530 of 547 right | 96.0%525 of 547 right | -0.9 | 3 | 8 | 0.23 | 1 |
| Gemini 4 Argon low · chat Closed · Flagship | run 593 | run 557 | 96.9%530 of 547 right | 96.0%525 of 547 right | -0.9 | 7 | 12 | 0.36 | 1 |
| GPT-6 Astra medium · chat Closed · Flagship | run 624 | run 610 | 96.7%529 of 547 right | 96.0%525 of 547 right | -0.7 | 4 | 8 | 0.39 | 1 |
| GPT-6.1 Sol low · chat Closed · Workhorse | run 619 | run 124 | 95.8%524 of 547 right | 95.8%524 of 547 right | +0.0 | 6 | 6 | 1 | 1 |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | run 622 | run 191 | 97.1%531 of 547 right | 95.6%523 of 547 right | -1.5 | 0 | 8 | 0.0078 | 0.83 |
| GPT-6.1 Sol high · chat Closed · Workhorse | run 618 | run 276 | 96.2%526 of 547 right | 95.6%523 of 547 right | -0.5 | 5 | 8 | 0.58 | 1 |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | run 589 | run 188 | 96.2%526 of 547 right | 95.2%521 of 547 right | -0.9 | 7 | 12 | 0.36 | 1 |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | run 623 | run 190 | 95.6%523 of 547 right | 95.2%521 of 547 right | -0.4 | 6 | 8 | 0.79 | 1 |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | run 592 | run 192 | 95.4%522 of 547 right | 94.7%518 of 547 right | -0.7 | 11 | 15 | 0.56 | 1 |
| Claude Opus 5.5 high · chat Closed · Flagship | run 587 | run 278 | 95.6%523 of 547 right | 94.5%517 of 547 right | -1.1 | 6 | 12 | 0.24 | 1 |
| Qwen4-Max high · chat Closed · Workhorse | run 1334 | run 1336 | 94.7%518 of 547 right | 94.5%517 of 547 right | -0.2 | 13 | 14 | 1 | 1 |
| Gemini 3.8 Flash low · chat Closed · Workhorse | run 578 | run 123 | 95.2%521 of 547 right | 94.3%516 of 547 right | -0.9 | 10 | 15 | 0.42 | 1 |
| Gemini 4 Carbon high · chat Closed · Flagship | run 1055 | run 1108 | 97.6%534 of 547 right | 94.3%516 of 547 right, 16 invalid | -3.3 | 3 | 21 | 0.00028 | 0.031 * |
| KnowLine-31B-Gen1.5 Open · 31B | run 953 | run 957 | 94.3%516 of 547 right | 93.6%512 of 547 right | -0.7 | 11 | 15 | 0.56 | 1 |
| Grok 4.7 high · chat Closed · Workhorse | run 579 | run 311 | 95.1%520 of 547 right | 93.4%511 of 547 right | -1.6 | 4 | 13 | 0.049 | 1 |
| Claude Opus 5.5 low · chat Closed · Flagship | run 588 | run 47 | 95.1%520 of 547 right | 93.1%509 of 547 right | -2.0 | 12 | 23 | 0.09 | 1 |
| Copernicus-31B Open · 31B | run 1168 | run 1165 | 94.5%517 of 547 right | 92.9%508 of 547 right | -1.6 | 8 | 17 | 0.11 | 1 |
| Claude Haiku 5.5 max · chat Closed · Efficient | run 949 | run 948 | 89.4%489 of 547 right, 18 invalid | 92.7%507 of 547 right | +3.3 | 30 | 12 | 0.0079 | 0.83 |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | run 808 | run 585 | 94.5%517 of 547 right | 92.5%506 of 547 right | -2.0 | 10 | 21 | 0.071 | 1 |
| REAX 1.3 Open · 57B | run 829 | run 827 | 94.0%514 of 547 right | 92.5%506 of 547 right | -1.5 | 12 | 20 | 0.22 | 1 |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | run 584 | run 320 | 95.2%521 of 547 right | 92.3%505 of 547 right | -2.9 | 5 | 21 | 0.0025 | 0.28 |
| KnowLine-31B-Gen2 Open · 31B | run 1327 | run 1314 | 94.5%517 of 547 right | 92.3%505 of 547 right | -2.2 | 8 | 20 | 0.036 | 1 |
| Karpo Decision 31B Open · 31B | run 1159 | run 1156 | 94.9%519 of 547 right | 92.1%504 of 547 right | -2.7 | 12 | 27 | 0.024 | 1 |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | run 594 | run 193 | 94.3%516 of 547 right | 91.6%501 of 547 right | -2.7 | 7 | 22 | 0.0081 | 0.85 |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | run 582 | run 315 | 94.0%514 of 547 right | 91.4%500 of 547 right | -2.6 | 7 | 21 | 0.013 | 1 |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | run 871 | run 870 | 91.4%500 of 547 right | 91.4%500 of 547 right | +0.0 | 13 | 13 | 1 | 1 |
| Gemini Decision 1 20261111 Closed · Efficient | run 1009 | run 1021 | 92.9%508 of 547 right | 91.0%498 of 547 right | -1.8 | 10 | 20 | 0.099 | 1 |
| Logits Sigma-1 Duo Open · 57B | run 1229 | run 1221 | 93.4%511 of 547 right | 91.0%498 of 547 right | -2.4 | 17 | 30 | 0.079 | 1 |
| GLiDE 27B Open · 27B | run 901 | run 899 | 91.2%499 of 547 right | 90.9%497 of 547 right | -0.4 | 19 | 21 | 0.87 | 1 |
| d1-ultra Closed · Efficient | runs 411, 418, 422, 426, 42828 answers changed between runs | runs 199 to 20318 answers changed between runs | 93.6%512 of 547 rightrange 92.7% to 93.6% | 90.7%496 of 547 rightrange 90.1% to 91.0% | -2.9 | 8 | 24 | 0.007 | 0.76 |
| Sieve 31B v3 Open · 31B | run 1064 | run 1241 | 93.4%511 of 547 right | 90.7%496 of 547 right | -2.7 | 10 | 25 | 0.017 | 1 |
| Turion Jev 31B Open · 31B | run 1075 | run 1273 | 93.6%512 of 547 right | 90.7%496 of 547 right | -2.9 | 10 | 26 | 0.011 | 1 |
| Gemma 4 31B reasoning off · chat Open · 31B | run 774 | run 119 | 92.5%506 of 547 right | 90.5%495 of 547 right | -2.0 | 18 | 29 | 0.14 | 1 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | run 639 | run 640 | 90.7%496 of 547 right | 90.5%495 of 547 right | -0.2 | 20 | 21 | 1 | 1 |
| GLiDE v1.2 Open · 27B | run 521 | runs 248, 250 to 2530 answers changed between runs | 90.5%495 of 547 rightthe same in every run | 90.3%494 of 547 right | -0.2 | 21 | 22 | 1 | 1 |
| d1.2-ultra Closed · Efficient | run 1359 | runs 1347, 1353, 1368, 1371, 13730 answers changed between runs | 90.9%497 of 547 rightthe same in every run | 90.3%494 of 547 right | -0.5 | 22 | 25 | 0.77 | 1 |
| Mammoth-1 Open · 27B | run 503 | runs 374, 377, 380, 383 and 3840 answers changed between runs | 92.9%508 of 547 rightthe same in every run | 90.1%493 of 547 right | -2.7 | 15 | 30 | 0.036 | 1 |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | run 783 | run 189 | 90.3%494 of 547 right | 89.9%492 of 547 right | -0.4 | 24 | 26 | 0.89 | 1 |
| d3 Open · 26B | run 554 | runs 259 to 2630 answers changed between runs | 91.0%498 of 547 rightthe same in every run | 89.8%491 of 547 right | -1.3 | 15 | 22 | 0.32 | 1 |
| Drex Max Open · 27B | run 552 | runs 322 to 324, 327 and 3280 answers changed between runs | 91.4%500 of 547 rightthe same in every run | 89.6%490 of 547 right | -1.8 | 19 | 29 | 0.19 | 1 |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | run 823 | run 160 | 90.1%493 of 547 right | 89.6%490 of 547 right | -0.5 | 26 | 29 | 0.79 | 1 |
| Muse Ember max · chat Closed · Efficient | run 942 | run 939 | 90.9%497 of 547 right | 89.6%490 of 547 right | -1.3 | 17 | 24 | 0.35 | 1 |
| Perplexity Decider 27B v1.3 Open · 27B | run 1137 | run 1134 | 90.7%496 of 547 right | 89.6%490 of 547 right | -1.1 | 19 | 25 | 0.45 | 1 |
| d3-ultra Open · 26B | run 550 | runs 269 to 2730 answers changed between runs | 91.4%500 of 547 rightthe same in every run | 89.4%489 of 547 right | -2.0 | 22 | 33 | 0.18 | 1 |
| d1.2 Closed · Efficient | run 1357 | runs 1346, 1352, 1369 and 1370, 137257 answers changed between runs | 88.8%486 of 547 rightrange 87.6% to 88.8% | 89.0%487 of 547 right | +0.2 | 31 | 30 | 1 | 1 |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | run 878 | run 877 | 86.8%475 of 547 right | 89.0%487 of 547 right | +2.2 | 37 | 25 | 0.16 | 1 |
| Qwen3.8-27B low · chat Open · 27B | run 667 | run 318 | 90.7%496 of 547 right | 89.0%487 of 547 right | -1.6 | 16 | 25 | 0.21 | 1 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | run 527 | runs 217 to 2210 answers changed between runs | 91.4%500 of 547 rightthe same in every run | 88.7%485 of 547 right | -2.7 | 15 | 30 | 0.036 | 1 |
| Drex Max v1.1 Closed · Efficient (provisional) | run 399 | runs 326, 329 to 3320 answers changed between runs | 90.1%493 of 547 rightthe same in every run | 88.5%484 of 547 right | -1.6 | 18 | 27 | 0.23 | 1 |
| Perplexity Decider 27B v1 Open · 27B | runs 626 to 6303 answers changed between runs | runs 44, 64, 84, 96, 1000 answers changed between runs | 89.0%487 of 547 rightthe same in every run | 88.5%484 of 547 rightrange 88.5% to 88.7% | -0.5 | 26 | 29 | 0.79 | 1 |
| Perplexity Decider 27B v1.2 API Open · 27B | run 637 | run 633 | 91.0%498 of 547 right | 88.5%484 of 547 right | -2.6 | 17 | 31 | 0.059 | 1 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | run 867 | run 866 | 89.4%489 of 547 right | 88.3%483 of 547 right | -1.1 | 24 | 30 | 0.5 | 1 |
| d1.1 Closed · Efficient | runs 565, 568, 570, 573, 57543 answers changed between runs | runs 254 to 25862 answers changed between runs | 88.3%483 of 547 rightrange 87.2% to 89.2% | 88.3%483 of 547 rightrange 87.8% to 88.8% | +0.0 | 26 | 26 | 1 | 1 |
| GLM-5.3 low · chat Open · Workhorse | run 577 | run 314 | 89.2%488 of 547 right | 87.8%480 of 547 right | -1.5 | 21 | 29 | 0.32 | 1 |
| Jeffan-9B-1211 Open · 9.0B | run 1195 | run 1192 | 86.5%473 of 547 right | 87.6%479 of 547 right | +1.1 | 33 | 27 | 0.52 | 1 |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | run 820 | run 159 | 84.3%461 of 547 right | 87.2%477 of 547 right | +2.9 | 39 | 23 | 0.056 | 1 |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | run 804 | run 558 | 89.9%492 of 547 right | 87.0%476 of 547 right | -2.9 | 17 | 33 | 0.033 | 1 |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | run 1072 | run 1149 | 82.8%453 of 547 right, 2 invalid | 87.0%476 of 547 right | +4.2 | 49 | 26 | 0.011 | 1 |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | run 986 | run 985 | 88.8%486 of 547 right | 87.0%476 of 547 right | -1.8 | 22 | 32 | 0.22 | 1 |
| d1 Closed · Efficient | runs 430, 432 to 43550 answers changed between runs | runs 52, 66, 94, 105, 10819 answers changed between runs | 85.0%465 of 547 rightrange 85.0% to 85.7% | 86.8%475 of 547 rightrange 85.7% to 87.9% | +1.8 | 36 | 26 | 0.25 | 1 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | run 1293 | run 1290 | 83.0%454 of 547 right | 86.8%475 of 547 right | +3.8 | 47 | 26 | 0.019 | 1 |
| Clef Open · 27B | run 613 | run 125 | 88.1%482 of 547 right | 86.7%474 of 547 right | -1.5 | 18 | 26 | 0.29 | 1 |
| KnowLine-12B-Gen1 Open · 12B | run 1313 | run 1309 | 87.6%479 of 547 right | 86.7%474 of 547 right | -0.9 | 20 | 25 | 0.55 | 1 |
| GLiDE 9B Open · 7.9B | run 909 | run 907 | 84.3%461 of 547 right | 86.3%472 of 547 right | +2.0 | 37 | 26 | 0.21 | 1 |
| GPT-6 Luna low · chat Closed · Efficient | run 617 | run 133 | 88.7%485 of 547 right | 86.3%472 of 547 right | -2.4 | 26 | 39 | 0.14 | 1 |
| OneJev-27B Open · 27B | run 778 | runs 264 to 2680 answers changed between runs | 87.8%480 of 547 rightthe same in every run | 86.3%472 of 547 right | -1.5 | 23 | 31 | 0.34 | 1 |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | run 590 | run 55 | 87.6%479 of 547 right | 86.1%471 of 547 right | -1.5 | 32 | 40 | 0.41 | 1 |
| Jev 1.14 Closed · Efficient | runs 410, 416, 421, 423, 42514 answers changed between runs | runs 194 to 19817 answers changed between runs | 87.2%477 of 547 rightrange 86.5% to 87.2% | 86.1%471 of 547 rightrange 86.1% to 86.5% | -1.1 | 17 | 23 | 0.43 | 1 |
| Cerno 9B v1.2 Open · 9.0B | run 1085 | run 1201 | 83.5%457 of 547 right | 85.9%470 of 547 right | +2.4 | 46 | 33 | 0.18 | 1 |
| Drex 2.0 Open · 35B MoE · 3.0B active | runs 566, 571, 576, 581, 5860 answers changed between runs | runs 334 to 3383 answers changed between runs | 84.8%464 of 547 rightrange 84.6% to 84.8% | 85.9%470 of 547 rightthe same in every run | +1.1 | 33 | 27 | 0.52 | 1 |
| Jev 1.13 Closed · Efficient | runs 564, 567, 569, 572, 57420 answers changed between runs | runs 49, 65, 85, 97, 10118 answers changed between runs | 86.1%471 of 547 rightrange 86.1% to 87.2% | 85.9%470 of 547 rightrange 85.9% to 87.6% | -0.2 | 18 | 19 | 1 | 1 |
| KnowLine-9B-Gen1 Open · 9.0B | run 1177 | run 1174 | 82.4%451 of 547 right | 85.9%470 of 547 right | +3.5 | 42 | 23 | 0.025 | 1 |
| SubQ Snap 1.1 Closed · Efficient | run 413 | runs 346, 387 to 3900 answers changed between runs | 89.2%488 of 547 rightthe same in every run | 85.9%470 of 547 right | -3.3 | 23 | 41 | 0.033 | 1 |
| SubQ Snap Preview Closed · Efficient | runs 595, 597 to 6000 answers changed between runs | runs 177 to 1812 answers changed between runs | 89.9%492 of 547 rightrange 89.9% to 90.3% | 85.9%470 of 547 rightthe same in every run | -4.0 | 20 | 42 | 0.0071 | 0.76 |
| Adjudex 9B v5 Open · 9.0B | run 1186 | run 1183 | 83.9%459 of 547 right | 85.7%469 of 547 right | +1.8 | 41 | 31 | 0.29 | 1 |
| Mercury Decide Closed · Efficient | runs 398, 400 to 40320 answers changed between runs | runs 204, 210, 212, 214, 21619 answers changed between runs | 87.0%476 of 547 rightrange 86.7% to 87.4% | 85.7%469 of 547 rightrange 85.0% to 85.9% | -1.3 | 28 | 35 | 0.45 | 1 |
| Microsoft-Decision-1 Closed · Efficient | runs 412, 420, 424, 427, 43119 answers changed between runs | runs 205, 209, 211, 213, 21536 answers changed between runs | 76.4%418 of 547 rightrange 76.2% to 77.1% | 85.7%469 of 547 rightrange 85.0% to 86.1% | +9.3 | 74 | 23 | < 0.0001 | < 0.0001 * |
| Elephant-1 Open · 9.0B | run 499 | runs 373, 375 and 376, 378 and 3790 answers changed between runs | 83.5%457 of 547 rightthe same in every run | 85.4%467 of 547 right | +1.8 | 39 | 29 | 0.27 | 1 |
| Open-Jev-27B-v1.1 Open · 27B | run 760 | runs 45, 60, 92, 102, 1060 answers changed between runs | 86.5%473 of 547 rightthe same in every run | 85.4%467 of 547 right | -1.1 | 18 | 24 | 0.44 | 1 |
| KnowLine-9B-Gen2 Open · 9.0B | run 1321 | run 1306 | 82.8%453 of 547 right | 85.2%466 of 547 right | +2.4 | 38 | 25 | 0.13 | 1 |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | run 672 | runs 53, 74, 95, 103, 1075 answers changed between runs | 86.3%472 of 547 rightrange 86.1% to 86.3% | 84.8%464 of 547 right | -1.5 | 26 | 34 | 0.37 | 1 |
| Qwen3.8-27B yes/no scoring Open · 27B | runs 696 to 7001 answer changed between runs | runs 50, 75, 89, 99, 1049 answers changed between runs | 84.6%463 of 547 rightrange 84.6% to 84.8% | 84.8%464 of 547 rightthe same in every run | +0.2 | 23 | 22 | 1 | 1 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | run 863 | run 862 | 86.3%472 of 547 right | 84.5%462 of 547 right, 1 invalid | -1.8 | 32 | 42 | 0.3 | 1 |
| celeris-1-decision Closed · Efficient | run 546 | runs 150 and 151, 156 to 15819 answers changed between runs | 87.4%478 of 547 rightrange 86.7% to 87.4% | 84.1%460 of 547 right | -3.3 | 24 | 42 | 0.036 | 1 |
| GPT-6 Luna Decisions API Closed · Efficient | run 397 | run 143 | 86.7%474 of 547 right, 1 invalid | 84.1%460 of 547 right, 2 invalid | -2.6 | 28 | 42 | 0.12 | 1 |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | run 591 | run 54 | 84.8%464 of 547 right | 83.9%459 of 547 right | -0.9 | 27 | 32 | 0.6 | 1 |
| Cygnet Open · 12B | run 753 | run 127 | 85.6%468 of 547 right | 83.9%459 of 547 right | -1.6 | 26 | 35 | 0.31 | 1 |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | run 678 | run 83 | 85.9%470 of 547 right | 83.9%459 of 547 right | -2.0 | 28 | 39 | 0.22 | 1 |
| Winnow-12B Open · 12B | run 746 | run 128 | 85.0%465 of 547 right | 83.9%459 of 547 right | -1.1 | 34 | 40 | 0.56 | 1 |
| Perplexity Decider Flash v1.3 Open · 4.0B | run 1126 | run 1123 | 78.8%431 of 547 right | 83.7%458 of 547 right | +4.9 | 60 | 33 | 0.0067 | 0.73 |
| Valen-Decision-1.5-9B Open · 9.0B | run 1278 | run 1275 | 79.0%432 of 547 right | 83.7%458 of 547 right | +4.8 | 61 | 35 | 0.01 | 1 |
| Dexy-Decision-9B-V2 Open · 9.0B | run 1258 | run 1255 | 83.4%456 of 547 right | 83.5%457 of 547 right | +0.2 | 37 | 36 | 1 | 1 |
| vera-medium-omni Open · 4.2B | run 1267 | run 1264 | 79.5%435 of 547 right | 83.5%457 of 547 right | +4.0 | 52 | 30 | 0.02 | 1 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | run 1081 | run 1236 | 78.4%429 of 547 right | 82.8%453 of 547 right | +4.4 | 57 | 33 | 0.015 | 1 |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | run 889 | run 1040 | 80.6%441 of 547 right, 2 invalid | 82.4%451 of 547 right, 2 invalid | +1.8 | 49 | 39 | 0.34 | 1 |
| Clef-Flash Open · 9.0B | run 614 | run 139 | 77.7%425 of 547 right | 82.3%450 of 547 right | +4.6 | 55 | 30 | 0.0088 | 0.91 |
| Verdin-4B-v1.1 Open · 4.0B | run 1247 | run 1244 | 77.5%424 of 547 right | 82.3%450 of 547 right | +4.8 | 60 | 34 | 0.0095 | 0.97 |
| Perplexity Decider Flash v1.2 Open · 4.0B | run 533 | runs 223 to 2270 answers changed between runs | 76.6%419 of 547 rightthe same in every run | 81.9%448 of 547 right | +5.3 | 63 | 34 | 0.0042 | 0.46 |
| KnowLine 4B Gen5.5 Open · 4.0B | run 961 | run 965 | 78.6%430 of 547 right | 80.8%442 of 547 right | +2.2 | 51 | 39 | 0.25 | 1 |
| Chinese Decision Pointer 4B Open · 4.0B | run 1209 | run 1206 | 75.7%414 of 547 right | 80.4%440 of 547 right | +4.8 | 63 | 37 | 0.012 | 1 |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | run 1143 | run 1146 | 72.0%394 of 547 right, 4 invalid | 80.4%440 of 547 right, 1 invalid | +8.4 | 95 | 49 | 0.00016 | 0.018 * |
| Qwen3-14B reasoning off · chat Open · 15B | run 770 | run 131 | 76.8%420 of 547 right | 79.7%436 of 547 right | +2.9 | 56 | 40 | 0.13 | 1 |
| Metask-Jev-4B Open · 4.5B | run 739 | run 126 | 67.1%367 of 547 right | 77.1%422 of 547 right | +10.1 | 90 | 35 | < 0.0001 | 0.00011 * |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | run 1034 | run 1037 | 47.9%262 of 547 right, 14 invalid | 73.9%404 of 547 right | +26.0 | 175 | 33 | < 0.0001 | < 0.0001 * |
| Erk-14B reasoning off · chat Open · 14B | run 762 | run 132 | 75.3%412 of 547 right | 72.0%394 of 547 right | -3.3 | 48 | 66 | 0.11 | 1 |
| LFM3-1.7B reasoning off · chat Open · 1.7B | run 1223 | run 1227 | 40.2%220 of 547 right, 1 invalid | 72.9%399 of 547 righthistorical paired baseline, run 1223 | +32.7 | 221 | 42 | < 0.0001 | < 0.0001 * |
| KnowLine-2B-Gen1 Open · 2.0B | run 1057 | run 1079 | 58.0%317 of 547 right | 71.3%390 of 547 right | +13.3 | 112 | 39 | < 0.0001 | < 0.0001 * |
| LFM3-1.7B reasoning on · chat Open · 1.7B | run 1061 | run 1089 | 48.1%263 of 547 right | 71.1%389 of 547 righthistorical paired baseline, run 1061 | +23.0 | 166 | 40 | < 0.0001 | < 0.0001 * |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | run 886 | run 1288 | 27.1%148 of 547 right, 87 invalid | 70.4%385 of 547 right, 3 invalid | +43.3 | 264 | 27 | < 0.0001 | < 0.0001 * |
| GLiDE 2B Open · 1.9B | run 917 | run 915 | 54.3%297 of 547 right | 70.0%383 of 547 right | +15.7 | 135 | 49 | < 0.0001 | < 0.0001 * |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | run 973 | run 977 | 55.8%305 of 547 right | 69.7%381 of 547 right | +13.9 | 129 | 53 | < 0.0001 | < 0.0001 * |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | run 1094 | run 1299 | 7.1%39 of 547 right, 417 invalid | 67.6%370 of 547 right, 8 invalidhistorical paired baseline, run 1094 | +60.5 | 337 | 6 | < 0.0001 | < 0.0001 * |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | run 1215 | run 1218 | 36.7%201 of 547 right, 34 invalid | 67.3%368 of 547 right, 2 invalid | +30.5 | 207 | 40 | < 0.0001 | < 0.0001 * |
| d1-2B Open · 1.9B | run 708 | runs 340 to 3440 answers changed between runs | 41.9%229 of 547 rightthe same in every run | 65.6%359 of 547 right | +23.8 | 171 | 41 | < 0.0001 | < 0.0001 * |
| GLiDE 0.8B Open · 0.8B | run 925 | run 923 | 46.1%252 of 547 right | 60.7%332 of 547 right | +14.6 | 141 | 61 | < 0.0001 | < 0.0001 * |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | run 1011 | run 1015 | 37.3%204 of 547 right | 59.6%326 of 547 right | +22.3 | 161 | 39 | < 0.0001 | < 0.0001 * |
| GLiDE 0.4B Open · 0.4B | run 933 | run 931 | 25.8%141 of 547 right | 59.4%325 of 547 right | +33.6 | 222 | 38 | < 0.0001 | < 0.0001 * |
| OneJev1.6-0.8B Open · 0.8B | run 1025 | run 1029 | 42.8%234 of 547 right | 59.4%325 of 547 right | +16.6 | 148 | 57 | < 0.0001 | < 0.0001 * |
| KnowLine-0.8B-Gen1 Open · 0.8B | run 1001 | run 1005 | 40.2%220 of 547 right | 58.9%322 of 547 right | +18.6 | 150 | 48 | < 0.0001 | < 0.0001 * |
| GLiClass v4 Edge Open · 0.4B | run 1042 | run 1046 | 26.9%147 of 547 right | 58.3%319 of 547 right | +31.4 | 206 | 34 | < 0.0001 | < 0.0001 * |
| Decider 1 Closed · Efficient | run 596 | runs 243 to 2470 answers changed between runs | 36.6%200 of 547 rightthe same in every run | 55.8%305 of 547 right | +19.2 | 152 | 47 | < 0.0001 | < 0.0001 * |
| Strands Decider 2B Open · 2.0B | run 702 | run 141 | 39.1%214 of 547 right | 45.5%249 of 547 right | +6.4 | 74 | 39 | 0.0013 | 0.14 |
| Laya Open · 0.4B | run 726 | runs 51, 234 to 2370 answers changed between runs | 20.5%112 of 547 rightthe same in every run | 22.9%125 of 547 right | +2.4 | 86 | 73 | 0.34 | 1 |
| GLiNER2.5-multi-Decide Open · 0.3B | run 714 | run 135 | 20.7%113 of 547 right | 21.2%116 of 547 right | +0.5 | 42 | 39 | 0.82 | 1 |
| GLiNER2.5 Multi Open · 0.3B | run 720 | run 122 | 16.8%92 of 547 right | 16.3%89 of 547 right | -0.5 | 30 | 33 | 0.8 | 1 |
| Laya Multilingual Open · 0.3B | run 732 | runs 46, 238 to 2410 answers changed between runs | 17.0%93 of 547 rightthe same in every run | 15.4%84 of 547 right | -1.6 | 40 | 49 | 0.4 | 1 |
| Model | English runsthe first is compared; answers changed between runs where more than one | Turkish runsthe Text tab run, then its repeats; answers changed between them | Turkish accuracythe Text tab run on the kept questions; range over its runs; higher is better | English accuracythe first English run; range over its runs; higher is better | Differencepoints higher in English than in Turkish | Right only in Englishquestions the English run got right and the Turkish run did not | Right only in Turkishquestions the Turkish run got right and the English run did not | pexact McNemar | Holm padjusted over all 133 English setups |
|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.2 max · chat Closed · Flagship | run 971 | run 980 | 97.8%535 of 547 right | 98.0%536 of 547 right | +0.2 | 5 | 4 | 1 | 1 |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | run 1107 | run 1114 | 97.1%531 of 547 right | 96.7%529 of 547 right | -0.4 | 3 | 5 | 0.73 | 1 |
| GPT-6 Astra low · chat Closed · Flagship | run 616 | run 48 | 96.7%529 of 547 right | 96.3%527 of 547 right | -0.4 | 4 | 6 | 0.75 | 1 |
| GPT-6 Astra high · chat Closed · Flagship | run 615 | run 277 | 96.9%530 of 547 right | 96.2%526 of 547 right | -0.7 | 5 | 9 | 0.42 | 1 |
| GPT-6.1 Terra high · chat Closed · Workhorse | run 620 | run 275 | 96.3%527 of 547 right | 96.2%526 of 547 right | -0.2 | 9 | 10 | 1 | 1 |
| GPT-6.1 Terra low · chat Closed · Workhorse | run 621 | run 274 | 95.6%523 of 547 right | 96.2%526 of 547 right | +0.5 | 13 | 10 | 0.68 | 1 |
| Claude Fable 5.1 high · chat Closed · Flagship | run 580 | run 310 | 95.4%522 of 547 right | 96.0%525 of 547 right | +0.5 | 12 | 9 | 0.66 | 1 |
| Gemini 3.8 Flash high · chat Closed · Workhorse | run 583 | run 140 | 96.9%530 of 547 right | 96.0%525 of 547 right | -0.9 | 3 | 8 | 0.23 | 1 |
| Gemini 4 Argon low · chat Closed · Flagship | run 593 | run 557 | 96.9%530 of 547 right | 96.0%525 of 547 right | -0.9 | 7 | 12 | 0.36 | 1 |
| GPT-6 Astra medium · chat Closed · Flagship | run 624 | run 610 | 96.7%529 of 547 right | 96.0%525 of 547 right | -0.7 | 4 | 8 | 0.39 | 1 |
| GPT-6.1 Sol low · chat Closed · Workhorse | run 619 | run 124 | 95.8%524 of 547 right | 95.8%524 of 547 right | +0.0 | 6 | 6 | 1 | 1 |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | run 622 | run 191 | 97.1%531 of 547 right | 95.6%523 of 547 right | -1.5 | 0 | 8 | 0.0078 | 0.83 |
| GPT-6.1 Sol high · chat Closed · Workhorse | run 618 | run 276 | 96.2%526 of 547 right | 95.6%523 of 547 right | -0.5 | 5 | 8 | 0.58 | 1 |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | run 589 | run 188 | 96.2%526 of 547 right | 95.2%521 of 547 right | -0.9 | 7 | 12 | 0.36 | 1 |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | run 623 | run 190 | 95.6%523 of 547 right | 95.2%521 of 547 right | -0.4 | 6 | 8 | 0.79 | 1 |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | run 592 | run 192 | 95.4%522 of 547 right | 94.7%518 of 547 right | -0.7 | 11 | 15 | 0.56 | 1 |
| Claude Opus 5.5 high · chat Closed · Flagship | run 587 | run 278 | 95.6%523 of 547 right | 94.5%517 of 547 right | -1.1 | 6 | 12 | 0.24 | 1 |
| Qwen4-Max high · chat Closed · Workhorse | run 1334 | run 1336 | 94.7%518 of 547 right | 94.5%517 of 547 right | -0.2 | 13 | 14 | 1 | 1 |
| Gemini 3.8 Flash low · chat Closed · Workhorse | run 578 | run 123 | 95.2%521 of 547 right | 94.3%516 of 547 right | -0.9 | 10 | 15 | 0.42 | 1 |
| Gemini 4 Carbon high · chat Closed · Flagship | run 1055 | run 1108 | 97.6%534 of 547 right | 94.3%516 of 547 right, 16 invalid | -3.3 | 3 | 21 | 0.00028 | 0.031 * |
| Grok 4.7 high · chat Closed · Workhorse | run 579 | run 311 | 95.1%520 of 547 right | 93.4%511 of 547 right | -1.6 | 4 | 13 | 0.049 | 1 |
| Claude Opus 5.5 low · chat Closed · Flagship | run 588 | run 47 | 95.1%520 of 547 right | 93.1%509 of 547 right | -2.0 | 12 | 23 | 0.09 | 1 |
| Claude Haiku 5.5 max · chat Closed · Efficient | run 949 | run 948 | 89.4%489 of 547 right, 18 invalid | 92.7%507 of 547 right | +3.3 | 30 | 12 | 0.0079 | 0.83 |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | run 808 | run 585 | 94.5%517 of 547 right | 92.5%506 of 547 right | -2.0 | 10 | 21 | 0.071 | 1 |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | run 584 | run 320 | 95.2%521 of 547 right | 92.3%505 of 547 right | -2.9 | 5 | 21 | 0.0025 | 0.28 |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | run 594 | run 193 | 94.3%516 of 547 right | 91.6%501 of 547 right | -2.7 | 7 | 22 | 0.0081 | 0.85 |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | run 582 | run 315 | 94.0%514 of 547 right | 91.4%500 of 547 right | -2.6 | 7 | 21 | 0.013 | 1 |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | run 871 | run 870 | 91.4%500 of 547 right | 91.4%500 of 547 right | +0.0 | 13 | 13 | 1 | 1 |
| Gemma 4 31B reasoning off · chat Open · 31B | run 774 | run 119 | 92.5%506 of 547 right | 90.5%495 of 547 right | -2.0 | 18 | 29 | 0.14 | 1 |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | run 783 | run 189 | 90.3%494 of 547 right | 89.9%492 of 547 right | -0.4 | 24 | 26 | 0.89 | 1 |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | run 823 | run 160 | 90.1%493 of 547 right | 89.6%490 of 547 right | -0.5 | 26 | 29 | 0.79 | 1 |
| Muse Ember max · chat Closed · Efficient | run 942 | run 939 | 90.9%497 of 547 right | 89.6%490 of 547 right | -1.3 | 17 | 24 | 0.35 | 1 |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | run 878 | run 877 | 86.8%475 of 547 right | 89.0%487 of 547 right | +2.2 | 37 | 25 | 0.16 | 1 |
| Qwen3.8-27B low · chat Open · 27B | run 667 | run 318 | 90.7%496 of 547 right | 89.0%487 of 547 right | -1.6 | 16 | 25 | 0.21 | 1 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | run 867 | run 866 | 89.4%489 of 547 right | 88.3%483 of 547 right | -1.1 | 24 | 30 | 0.5 | 1 |
| GLM-5.3 low · chat Open · Workhorse | run 577 | run 314 | 89.2%488 of 547 right | 87.8%480 of 547 right | -1.5 | 21 | 29 | 0.32 | 1 |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | run 820 | run 159 | 84.3%461 of 547 right | 87.2%477 of 547 right | +2.9 | 39 | 23 | 0.056 | 1 |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | run 804 | run 558 | 89.9%492 of 547 right | 87.0%476 of 547 right | -2.9 | 17 | 33 | 0.033 | 1 |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | run 1072 | run 1149 | 82.8%453 of 547 right, 2 invalid | 87.0%476 of 547 right | +4.2 | 49 | 26 | 0.011 | 1 |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | run 986 | run 985 | 88.8%486 of 547 right | 87.0%476 of 547 right | -1.8 | 22 | 32 | 0.22 | 1 |
| GPT-6 Luna low · chat Closed · Efficient | run 617 | run 133 | 88.7%485 of 547 right | 86.3%472 of 547 right | -2.4 | 26 | 39 | 0.14 | 1 |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | run 590 | run 55 | 87.6%479 of 547 right | 86.1%471 of 547 right | -1.5 | 32 | 40 | 0.41 | 1 |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | run 672 | runs 53, 74, 95, 103, 1075 answers changed between runs | 86.3%472 of 547 rightrange 86.1% to 86.3% | 84.8%464 of 547 right | -1.5 | 26 | 34 | 0.37 | 1 |
| Qwen3.8-27B yes/no scoring Open · 27B | runs 696 to 7001 answer changed between runs | runs 50, 75, 89, 99, 1049 answers changed between runs | 84.6%463 of 547 rightrange 84.6% to 84.8% | 84.8%464 of 547 rightthe same in every run | +0.2 | 23 | 22 | 1 | 1 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | run 863 | run 862 | 86.3%472 of 547 right | 84.5%462 of 547 right, 1 invalid | -1.8 | 32 | 42 | 0.3 | 1 |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | run 591 | run 54 | 84.8%464 of 547 right | 83.9%459 of 547 right | -0.9 | 27 | 32 | 0.6 | 1 |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | run 678 | run 83 | 85.9%470 of 547 right | 83.9%459 of 547 right | -2.0 | 28 | 39 | 0.22 | 1 |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | run 889 | run 1040 | 80.6%441 of 547 right, 2 invalid | 82.4%451 of 547 right, 2 invalid | +1.8 | 49 | 39 | 0.34 | 1 |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | run 1143 | run 1146 | 72.0%394 of 547 right, 4 invalid | 80.4%440 of 547 right, 1 invalid | +8.4 | 95 | 49 | 0.00016 | 0.018 * |
| Qwen3-14B reasoning off · chat Open · 15B | run 770 | run 131 | 76.8%420 of 547 right | 79.7%436 of 547 right | +2.9 | 56 | 40 | 0.13 | 1 |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | run 1034 | run 1037 | 47.9%262 of 547 right, 14 invalid | 73.9%404 of 547 right | +26.0 | 175 | 33 | < 0.0001 | < 0.0001 * |
| Erk-14B reasoning off · chat Open · 14B | run 762 | run 132 | 75.3%412 of 547 right | 72.0%394 of 547 right | -3.3 | 48 | 66 | 0.11 | 1 |
| LFM3-1.7B reasoning off · chat Open · 1.7B | run 1223 | run 1227 | 40.2%220 of 547 right, 1 invalid | 72.9%399 of 547 righthistorical paired baseline, run 1223 | +32.7 | 221 | 42 | < 0.0001 | < 0.0001 * |
| LFM3-1.7B reasoning on · chat Open · 1.7B | run 1061 | run 1089 | 48.1%263 of 547 right | 71.1%389 of 547 righthistorical paired baseline, run 1061 | +23.0 | 166 | 40 | < 0.0001 | < 0.0001 * |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | run 886 | run 1288 | 27.1%148 of 547 right, 87 invalid | 70.4%385 of 547 right, 3 invalid | +43.3 | 264 | 27 | < 0.0001 | < 0.0001 * |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | run 1094 | run 1299 | 7.1%39 of 547 right, 417 invalid | 67.6%370 of 547 right, 8 invalidhistorical paired baseline, run 1094 | +60.5 | 337 | 6 | < 0.0001 | < 0.0001 * |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | run 1215 | run 1218 | 36.7%201 of 547 right, 34 invalid | 67.3%368 of 547 right, 2 invalid | +30.5 | 207 | 40 | < 0.0001 | < 0.0001 * |
| GLiNER2.5 Multi Open · 0.3B | run 720 | run 122 | 16.8%92 of 547 right | 16.3%89 of 547 right | -0.5 | 30 | 33 | 0.8 | 1 |
| Model | English runsthe first is compared; answers changed between runs where more than one | Turkish runsthe Text tab run, then its repeats; answers changed between them | Turkish accuracythe Text tab run on the kept questions; range over its runs; higher is better | English accuracythe first English run; range over its runs; higher is better | Differencepoints higher in English than in Turkish | Right only in Englishquestions the English run got right and the Turkish run did not | Right only in Turkishquestions the Turkish run got right and the English run did not | pexact McNemar | Holm padjusted over all 133 English setups |
|---|---|---|---|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | run 953 | run 957 | 94.3%516 of 547 right | 93.6%512 of 547 right | -0.7 | 11 | 15 | 0.56 | 1 |
| Copernicus-31B Open · 31B | run 1168 | run 1165 | 94.5%517 of 547 right | 92.9%508 of 547 right | -1.6 | 8 | 17 | 0.11 | 1 |
| REAX 1.3 Open · 57B | run 829 | run 827 | 94.0%514 of 547 right | 92.5%506 of 547 right | -1.5 | 12 | 20 | 0.22 | 1 |
| KnowLine-31B-Gen2 Open · 31B | run 1327 | run 1314 | 94.5%517 of 547 right | 92.3%505 of 547 right | -2.2 | 8 | 20 | 0.036 | 1 |
| Karpo Decision 31B Open · 31B | run 1159 | run 1156 | 94.9%519 of 547 right | 92.1%504 of 547 right | -2.7 | 12 | 27 | 0.024 | 1 |
| Gemini Decision 1 20261111 Closed · Efficient | run 1009 | run 1021 | 92.9%508 of 547 right | 91.0%498 of 547 right | -1.8 | 10 | 20 | 0.099 | 1 |
| Logits Sigma-1 Duo Open · 57B | run 1229 | run 1221 | 93.4%511 of 547 right | 91.0%498 of 547 right | -2.4 | 17 | 30 | 0.079 | 1 |
| GLiDE 27B Open · 27B | run 901 | run 899 | 91.2%499 of 547 right | 90.9%497 of 547 right | -0.4 | 19 | 21 | 0.87 | 1 |
| d1-ultra Closed · Efficient | runs 411, 418, 422, 426, 42828 answers changed between runs | runs 199 to 20318 answers changed between runs | 93.6%512 of 547 rightrange 92.7% to 93.6% | 90.7%496 of 547 rightrange 90.1% to 91.0% | -2.9 | 8 | 24 | 0.007 | 0.76 |
| Sieve 31B v3 Open · 31B | run 1064 | run 1241 | 93.4%511 of 547 right | 90.7%496 of 547 right | -2.7 | 10 | 25 | 0.017 | 1 |
| Turion Jev 31B Open · 31B | run 1075 | run 1273 | 93.6%512 of 547 right | 90.7%496 of 547 right | -2.9 | 10 | 26 | 0.011 | 1 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | run 639 | run 640 | 90.7%496 of 547 right | 90.5%495 of 547 right | -0.2 | 20 | 21 | 1 | 1 |
| GLiDE v1.2 Open · 27B | run 521 | runs 248, 250 to 2530 answers changed between runs | 90.5%495 of 547 rightthe same in every run | 90.3%494 of 547 right | -0.2 | 21 | 22 | 1 | 1 |
| d1.2-ultra Closed · Efficient | run 1359 | runs 1347, 1353, 1368, 1371, 13730 answers changed between runs | 90.9%497 of 547 rightthe same in every run | 90.3%494 of 547 right | -0.5 | 22 | 25 | 0.77 | 1 |
| Mammoth-1 Open · 27B | run 503 | runs 374, 377, 380, 383 and 3840 answers changed between runs | 92.9%508 of 547 rightthe same in every run | 90.1%493 of 547 right | -2.7 | 15 | 30 | 0.036 | 1 |
| d3 Open · 26B | run 554 | runs 259 to 2630 answers changed between runs | 91.0%498 of 547 rightthe same in every run | 89.8%491 of 547 right | -1.3 | 15 | 22 | 0.32 | 1 |
| Drex Max Open · 27B | run 552 | runs 322 to 324, 327 and 3280 answers changed between runs | 91.4%500 of 547 rightthe same in every run | 89.6%490 of 547 right | -1.8 | 19 | 29 | 0.19 | 1 |
| Perplexity Decider 27B v1.3 Open · 27B | run 1137 | run 1134 | 90.7%496 of 547 right | 89.6%490 of 547 right | -1.1 | 19 | 25 | 0.45 | 1 |
| d3-ultra Open · 26B | run 550 | runs 269 to 2730 answers changed between runs | 91.4%500 of 547 rightthe same in every run | 89.4%489 of 547 right | -2.0 | 22 | 33 | 0.18 | 1 |
| d1.2 Closed · Efficient | run 1357 | runs 1346, 1352, 1369 and 1370, 137257 answers changed between runs | 88.8%486 of 547 rightrange 87.6% to 88.8% | 89.0%487 of 547 right | +0.2 | 31 | 30 | 1 | 1 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | run 527 | runs 217 to 2210 answers changed between runs | 91.4%500 of 547 rightthe same in every run | 88.7%485 of 547 right | -2.7 | 15 | 30 | 0.036 | 1 |
| Drex Max v1.1 Closed · Efficient (provisional) | run 399 | runs 326, 329 to 3320 answers changed between runs | 90.1%493 of 547 rightthe same in every run | 88.5%484 of 547 right | -1.6 | 18 | 27 | 0.23 | 1 |
| Perplexity Decider 27B v1 Open · 27B | runs 626 to 6303 answers changed between runs | runs 44, 64, 84, 96, 1000 answers changed between runs | 89.0%487 of 547 rightthe same in every run | 88.5%484 of 547 rightrange 88.5% to 88.7% | -0.5 | 26 | 29 | 0.79 | 1 |
| Perplexity Decider 27B v1.2 API Open · 27B | run 637 | run 633 | 91.0%498 of 547 right | 88.5%484 of 547 right | -2.6 | 17 | 31 | 0.059 | 1 |
| d1.1 Closed · Efficient | runs 565, 568, 570, 573, 57543 answers changed between runs | runs 254 to 25862 answers changed between runs | 88.3%483 of 547 rightrange 87.2% to 89.2% | 88.3%483 of 547 rightrange 87.8% to 88.8% | +0.0 | 26 | 26 | 1 | 1 |
| Jeffan-9B-1211 Open · 9.0B | run 1195 | run 1192 | 86.5%473 of 547 right | 87.6%479 of 547 right | +1.1 | 33 | 27 | 0.52 | 1 |
| d1 Closed · Efficient | runs 430, 432 to 43550 answers changed between runs | runs 52, 66, 94, 105, 10819 answers changed between runs | 85.0%465 of 547 rightrange 85.0% to 85.7% | 86.8%475 of 547 rightrange 85.7% to 87.9% | +1.8 | 36 | 26 | 0.25 | 1 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | run 1293 | run 1290 | 83.0%454 of 547 right | 86.8%475 of 547 right | +3.8 | 47 | 26 | 0.019 | 1 |
| Clef Open · 27B | run 613 | run 125 | 88.1%482 of 547 right | 86.7%474 of 547 right | -1.5 | 18 | 26 | 0.29 | 1 |
| KnowLine-12B-Gen1 Open · 12B | run 1313 | run 1309 | 87.6%479 of 547 right | 86.7%474 of 547 right | -0.9 | 20 | 25 | 0.55 | 1 |
| GLiDE 9B Open · 7.9B | run 909 | run 907 | 84.3%461 of 547 right | 86.3%472 of 547 right | +2.0 | 37 | 26 | 0.21 | 1 |
| OneJev-27B Open · 27B | run 778 | runs 264 to 2680 answers changed between runs | 87.8%480 of 547 rightthe same in every run | 86.3%472 of 547 right | -1.5 | 23 | 31 | 0.34 | 1 |
| Jev 1.14 Closed · Efficient | runs 410, 416, 421, 423, 42514 answers changed between runs | runs 194 to 19817 answers changed between runs | 87.2%477 of 547 rightrange 86.5% to 87.2% | 86.1%471 of 547 rightrange 86.1% to 86.5% | -1.1 | 17 | 23 | 0.43 | 1 |
| Cerno 9B v1.2 Open · 9.0B | run 1085 | run 1201 | 83.5%457 of 547 right | 85.9%470 of 547 right | +2.4 | 46 | 33 | 0.18 | 1 |
| Drex 2.0 Open · 35B MoE · 3.0B active | runs 566, 571, 576, 581, 5860 answers changed between runs | runs 334 to 3383 answers changed between runs | 84.8%464 of 547 rightrange 84.6% to 84.8% | 85.9%470 of 547 rightthe same in every run | +1.1 | 33 | 27 | 0.52 | 1 |
| Jev 1.13 Closed · Efficient | runs 564, 567, 569, 572, 57420 answers changed between runs | runs 49, 65, 85, 97, 10118 answers changed between runs | 86.1%471 of 547 rightrange 86.1% to 87.2% | 85.9%470 of 547 rightrange 85.9% to 87.6% | -0.2 | 18 | 19 | 1 | 1 |
| KnowLine-9B-Gen1 Open · 9.0B | run 1177 | run 1174 | 82.4%451 of 547 right | 85.9%470 of 547 right | +3.5 | 42 | 23 | 0.025 | 1 |
| SubQ Snap 1.1 Closed · Efficient | run 413 | runs 346, 387 to 3900 answers changed between runs | 89.2%488 of 547 rightthe same in every run | 85.9%470 of 547 right | -3.3 | 23 | 41 | 0.033 | 1 |
| SubQ Snap Preview Closed · Efficient | runs 595, 597 to 6000 answers changed between runs | runs 177 to 1812 answers changed between runs | 89.9%492 of 547 rightrange 89.9% to 90.3% | 85.9%470 of 547 rightthe same in every run | -4.0 | 20 | 42 | 0.0071 | 0.76 |
| Adjudex 9B v5 Open · 9.0B | run 1186 | run 1183 | 83.9%459 of 547 right | 85.7%469 of 547 right | +1.8 | 41 | 31 | 0.29 | 1 |
| Mercury Decide Closed · Efficient | runs 398, 400 to 40320 answers changed between runs | runs 204, 210, 212, 214, 21619 answers changed between runs | 87.0%476 of 547 rightrange 86.7% to 87.4% | 85.7%469 of 547 rightrange 85.0% to 85.9% | -1.3 | 28 | 35 | 0.45 | 1 |
| Microsoft-Decision-1 Closed · Efficient | runs 412, 420, 424, 427, 43119 answers changed between runs | runs 205, 209, 211, 213, 21536 answers changed between runs | 76.4%418 of 547 rightrange 76.2% to 77.1% | 85.7%469 of 547 rightrange 85.0% to 86.1% | +9.3 | 74 | 23 | < 0.0001 | < 0.0001 * |
| Elephant-1 Open · 9.0B | run 499 | runs 373, 375 and 376, 378 and 3790 answers changed between runs | 83.5%457 of 547 rightthe same in every run | 85.4%467 of 547 right | +1.8 | 39 | 29 | 0.27 | 1 |
| Open-Jev-27B-v1.1 Open · 27B | run 760 | runs 45, 60, 92, 102, 1060 answers changed between runs | 86.5%473 of 547 rightthe same in every run | 85.4%467 of 547 right | -1.1 | 18 | 24 | 0.44 | 1 |
| KnowLine-9B-Gen2 Open · 9.0B | run 1321 | run 1306 | 82.8%453 of 547 right | 85.2%466 of 547 right | +2.4 | 38 | 25 | 0.13 | 1 |
| celeris-1-decision Closed · Efficient | run 546 | runs 150 and 151, 156 to 15819 answers changed between runs | 87.4%478 of 547 rightrange 86.7% to 87.4% | 84.1%460 of 547 right | -3.3 | 24 | 42 | 0.036 | 1 |
| GPT-6 Luna Decisions API Closed · Efficient | run 397 | run 143 | 86.7%474 of 547 right, 1 invalid | 84.1%460 of 547 right, 2 invalid | -2.6 | 28 | 42 | 0.12 | 1 |
| Cygnet Open · 12B | run 753 | run 127 | 85.6%468 of 547 right | 83.9%459 of 547 right | -1.6 | 26 | 35 | 0.31 | 1 |
| Winnow-12B Open · 12B | run 746 | run 128 | 85.0%465 of 547 right | 83.9%459 of 547 right | -1.1 | 34 | 40 | 0.56 | 1 |
| Perplexity Decider Flash v1.3 Open · 4.0B | run 1126 | run 1123 | 78.8%431 of 547 right | 83.7%458 of 547 right | +4.9 | 60 | 33 | 0.0067 | 0.73 |
| Valen-Decision-1.5-9B Open · 9.0B | run 1278 | run 1275 | 79.0%432 of 547 right | 83.7%458 of 547 right | +4.8 | 61 | 35 | 0.01 | 1 |
| Dexy-Decision-9B-V2 Open · 9.0B | run 1258 | run 1255 | 83.4%456 of 547 right | 83.5%457 of 547 right | +0.2 | 37 | 36 | 1 | 1 |
| vera-medium-omni Open · 4.2B | run 1267 | run 1264 | 79.5%435 of 547 right | 83.5%457 of 547 right | +4.0 | 52 | 30 | 0.02 | 1 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | run 1081 | run 1236 | 78.4%429 of 547 right | 82.8%453 of 547 right | +4.4 | 57 | 33 | 0.015 | 1 |
| Clef-Flash Open · 9.0B | run 614 | run 139 | 77.7%425 of 547 right | 82.3%450 of 547 right | +4.6 | 55 | 30 | 0.0088 | 0.91 |
| Verdin-4B-v1.1 Open · 4.0B | run 1247 | run 1244 | 77.5%424 of 547 right | 82.3%450 of 547 right | +4.8 | 60 | 34 | 0.0095 | 0.97 |
| Perplexity Decider Flash v1.2 Open · 4.0B | run 533 | runs 223 to 2270 answers changed between runs | 76.6%419 of 547 rightthe same in every run | 81.9%448 of 547 right | +5.3 | 63 | 34 | 0.0042 | 0.46 |
| KnowLine 4B Gen5.5 Open · 4.0B | run 961 | run 965 | 78.6%430 of 547 right | 80.8%442 of 547 right | +2.2 | 51 | 39 | 0.25 | 1 |
| Chinese Decision Pointer 4B Open · 4.0B | run 1209 | run 1206 | 75.7%414 of 547 right | 80.4%440 of 547 right | +4.8 | 63 | 37 | 0.012 | 1 |
| Metask-Jev-4B Open · 4.5B | run 739 | run 126 | 67.1%367 of 547 right | 77.1%422 of 547 right | +10.1 | 90 | 35 | < 0.0001 | 0.00011 * |
| KnowLine-2B-Gen1 Open · 2.0B | run 1057 | run 1079 | 58.0%317 of 547 right | 71.3%390 of 547 right | +13.3 | 112 | 39 | < 0.0001 | < 0.0001 * |
| GLiDE 2B Open · 1.9B | run 917 | run 915 | 54.3%297 of 547 right | 70.0%383 of 547 right | +15.7 | 135 | 49 | < 0.0001 | < 0.0001 * |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | run 973 | run 977 | 55.8%305 of 547 right | 69.7%381 of 547 right | +13.9 | 129 | 53 | < 0.0001 | < 0.0001 * |
| d1-2B Open · 1.9B | run 708 | runs 340 to 3440 answers changed between runs | 41.9%229 of 547 rightthe same in every run | 65.6%359 of 547 right | +23.8 | 171 | 41 | < 0.0001 | < 0.0001 * |
| GLiDE 0.8B Open · 0.8B | run 925 | run 923 | 46.1%252 of 547 right | 60.7%332 of 547 right | +14.6 | 141 | 61 | < 0.0001 | < 0.0001 * |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | run 1011 | run 1015 | 37.3%204 of 547 right | 59.6%326 of 547 right | +22.3 | 161 | 39 | < 0.0001 | < 0.0001 * |
| GLiDE 0.4B Open · 0.4B | run 933 | run 931 | 25.8%141 of 547 right | 59.4%325 of 547 right | +33.6 | 222 | 38 | < 0.0001 | < 0.0001 * |
| OneJev1.6-0.8B Open · 0.8B | run 1025 | run 1029 | 42.8%234 of 547 right | 59.4%325 of 547 right | +16.6 | 148 | 57 | < 0.0001 | < 0.0001 * |
| KnowLine-0.8B-Gen1 Open · 0.8B | run 1001 | run 1005 | 40.2%220 of 547 right | 58.9%322 of 547 right | +18.6 | 150 | 48 | < 0.0001 | < 0.0001 * |
| GLiClass v4 Edge Open · 0.4B | run 1042 | run 1046 | 26.9%147 of 547 right | 58.3%319 of 547 right | +31.4 | 206 | 34 | < 0.0001 | < 0.0001 * |
| Decider 1 Closed · Efficient | run 596 | runs 243 to 2470 answers changed between runs | 36.6%200 of 547 rightthe same in every run | 55.8%305 of 547 right | +19.2 | 152 | 47 | < 0.0001 | < 0.0001 * |
| Strands Decider 2B Open · 2.0B | run 702 | run 141 | 39.1%214 of 547 right | 45.5%249 of 547 right | +6.4 | 74 | 39 | 0.0013 | 0.14 |
| Laya Open · 0.4B | run 726 | runs 51, 234 to 2370 answers changed between runs | 20.5%112 of 547 rightthe same in every run | 22.9%125 of 547 right | +2.4 | 86 | 73 | 0.34 | 1 |
| GLiNER2.5-multi-Decide Open · 0.3B | run 714 | run 135 | 20.7%113 of 547 right | 21.2%116 of 547 right | +0.5 | 42 | 39 | 0.82 | 1 |
| Laya Multilingual Open · 0.3B | run 732 | runs 46, 238 to 2410 answers changed between runs | 17.0%93 of 547 rightthe same in every run | 15.4%84 of 547 right | -1.6 | 40 | 49 | 0.4 | 1 |
Without an English run
Four of the Text tab's 137 setups have no English run yet, each with its reason:
- Perplexity Decider 27B v1.1: Perplexity's API now answers requests for v1.1 with v1.2, so a new run could not measure v1.1.
- GLiDE API: since 10 October, our requests for fastino/GLiDE have been answered by Fastino's glide_v2 engine, not the glide engine of its Turkish run (6 October), so an English run would not measure the same model.
- DeepSeek-V4.1-Flash reasoning on · chat · first release and DeepSeek-V4.1-Flash reasoning off · chat · first release: their first-release weights are no longer on our disk (the server folder now holds the 1017 weights), so a run needs them downloaded again first.
Without an English run
Two of the Text tab's 60 chat model setups have no English run yet, each with its reason:
- DeepSeek-V4.1-Flash reasoning on · chat · first release and DeepSeek-V4.1-Flash reasoning off · chat · first release: their first-release weights are no longer on our disk (the server folder now holds the 1017 weights), so a run needs them downloaded again first.
Without an English run
Two of the Text tab's 77 decision models have no English run yet, each with its reason:
- Perplexity Decider 27B v1.1: Perplexity's API now answers requests for v1.1 with v1.2, so a new run could not measure v1.1.
- GLiDE API: since 10 October, our requests for fastino/GLiDE have been answered by Fastino's glide_v2 engine, not the glide engine of its Turkish run (6 October), so an English run would not measure the same model.
Our method
The translation
The translator is Claude Opus 5.5 (claude-opus-5-5) at effort high; route: Claude Code CLI, Claude subscription, Claude Code version 2.1.312. Its maker has no decision model on our board. Of its maker's chat models, Claude Fable 5.2, Claude Fable 5.1, Claude Opus 5.5 and Claude Sonnet 5.5 answered the English questions too, the translator's own model among them. It was told to translate faithfully, keep the option letters and their order, not solve the question and not add or remove information (translation prompt version 2976d0f96197).
The flags
In a separate request, the same model was shown each Turkish question, its English translation and the keyed option, and asked whether answering it correctly depends on Turkish wording, with the reasons (flag prompt version 08a5c70222aa). It flagged 109 of the 656 questions; they are left out, and 547 are kept. By reason, as a flagged question can have several:
- word or phrase meaning: 76
- idiom: 44
- sound: 5
- marked words: 55
- other: 13
No person checked the translations or the flags, and this is a limit of the method: a translation error, or a question whose answer still depends on Turkish wording that the model did not flag, would stay among the 547 kept questions and could move a setup's accuracy in English. Both prompts in full are in english-reading.json and runner/translate_reading.py.
The structural check
A script (runner/translate_reading.py, its check step) then compared each translation with its Turkish original: a problem is a missing translation, option letters other than the original's, an empty text or a different count of underline markup; a warning is a change in the numbering or the numbers, Turkish letters in the English, or an English text under half or over 2.5 times the Turkish length. It found problems in 1 question (3043), already left out by the flags, so it left out no further question; its 159 warnings leave out nothing. In all, the checks left out 109 of the 656 questions and kept 547: their IDs are in kept-question-ids.txt, one per line, and in english-reading.json.
Why the grammar questions are not translated
The Text tab's 1,542 grammar questions test Turkish grammar itself, so a translation would test something else, not the same question in another language. Only the 656 reading questions were translated.
Asking the decision models
Each decision model is asked exactly as on the Text tab, with its own settings, one question per request, and with its option probabilities, except for one word of the System One instructions, the options heading and the question itself. The Text tab's instructions:
Which option is the correct answer to this Turkish multiple-choice question?
The English runs' instructions:
Which option is the correct answer to this English multiple-choice question?
The options follow the heading “Options:” instead of “Seçenekler:”, and the question and its options are the English translation. Prompt version 47f5776e71db; every English run used the translations with SHA-256 bf0d47cd385a93050c65906977e199ef7e45d5848dbbdef2bc8d124f4d5abf68. d1 was asked as d1, while its Turkish runs used d1:free, because Liquid's API no longer serves that name; its model list still names d1, released 29 September 2026. Perplexity Decider 27B v1.2 (its setup tagged DGX Spark), Perplexity Decider Flash v1.2 and GLiDE · v1.2 made their Text tab runs on an NVIDIA DGX Spark and their English runs on our RTX PRO 6000 GPU, with the same checkpoint, server code and settings. celeris-1-decision was added on 3 November 2026 with one English run, although its repeated Turkish runs changed answers, for which the rule under Runs gives five. SubQ Snap Preview's English runs (5 November 2026) returned the same choices and probabilities as SubQ Snap 1.1's English run (2 November 2026) on all 547 questions, and every response named the Preview model. Its Turkish run (13 October 2026) and SubQ Snap 1.1's (1 November 2026) differ on 519 of 2,198 questions. The model behind each name was not verified; read SubQ Snap Preview's Turkish against English comparison with this in mind. LFM2.5-2.6B, LFM2.5-8B-A1B and LFM2.5-24B-A2B (at the revisions served, 7141b56, dd83f5c and 1769516): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored (Zeki #8202).
Asking the chat models
Each chat model setup not asked for a plain answer line (below) is asked as its Text tab run was, with the same settings (reasoning, and 16 questions per request for 38 and 1 question per request for 11), and with the Text tab's chat prompts in English: the system prompt and the instruction that ask for each answer's letter with a confidence score between 0 and 1, the question header and the options heading “Options:”. A chat model gives no option probabilities, so its confidence score is the one it stated. The English runs' system prompt:
You are taking an English multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.
Their instruction:
Answer each of the following multiple-choice questions. Without explanation, return for each question the letter of the correct option together with a confidence score between 0 and 1 showing how sure you are that your answer is correct, in the specified JSON format.
Chat prompt version 5c3a4849752e; the Text tab's Turkish prompts are under The instructions of each kind of run. The chat models' runs with an “I don't know” option are under Our method with an “I don't know” option.
Asking the chat models for a plain answer line
Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat and Qwen3.5-2B · reasoning off · chat are asked in English as their Text tab runs were in Turkish: one question per request on logic's server with only the request settings of logic's own runs of the same model (How we called it, on the model page), a user message with one instruction, word for word that of logic's runs, a blank line and the question as the other runs in its language show it; no system prompt and no answer schema. The answer is the letter on the reply's last “ANSWER: X” line in English, “CEVAP: X” in Turkish. They state no confidence and report no option probabilities, so their confidence is not measured. The instruction in English ({letters} is the question's own letters, as “A, B, C, D or E”):
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = {letters}).
In Turkish:
Aşağıdaki çoktan seçmeli soruyu çöz. Yanıtının son satırına yalnızca 'CEVAP: X' yaz (X = {letters}).
Plain prompt versions ad6f4cc346df (English) and ba455bb00e5c (Turkish).
Runs
One English run per setup; five where the setup's whole-bank Turkish repeats on the Text tab changed answers. Here: five runs for d1-ultra, Perplexity Decider 27B v1, d1.1, d1, Jev 1.14, Drex 2.0, Jev 1.13, SubQ Snap Preview, Mercury Decide, Microsoft-Decision-1 and Qwen3.8-27B · yes/no scoring; one for the other 122. The English runs are kept out of the leaderboard and its repeats.
The comparison
On the 547 kept questions only: accuracy is the right answers over 547 for the first English run and for the Text tab run (an invalid answer counts as not right), and the difference is how many points higher the English run scored than the Turkish one (negative where Turkish is higher). The paired test is the exact McNemar test on the questions only one of the two runs got right, with p adjusted by Holm's method over all 133 English setups, decision and chat alike, whatever the group shown. Where a side has more than one run, its range is the lowest and highest accuracy over its runs, and its answers changed are the kept questions whose answer differs between them.
Why the translations are not published
The site never publishes question text, Turkish or English: the publisher owns the questions. english-reading.json holds instead each reading question's ID, unit, whether it was kept, its flag reasons and the SHA-256 of its English text, with the translator and both prompts, so that a copy of these translations can be checked against it. Translating again with runner/translate_reading.py need not give the same text word for word.
The instructions of each kind of run
Each decision model run's System One instructions in full, and each chat model run's system prompt and instruction (a run asked for a plain answer line has one instruction and no system prompt; its {letters} is the question's own letters), as results.json and answers.json give them; a chat model's run with the option keeps the system prompt of its five-option run. The question and its options follow in the run's language; their text is not published.
Decision models, five options, in Turkish (the Text tab)
Which option is the correct answer to this Turkish multiple-choice question?
Decision models, five options, in English
Which option is the correct answer to this English multiple-choice question?
Chat models, in Turkish (the Text tab): the system prompt
You are taking a Turkish multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.
Chat models, in Turkish (the Text tab): the instruction
Aşağıdaki çoktan seçmeli soruların her birini yanıtlayın. Açıklama yapmadan, her soru için doğru seçeneğin harfini ve cevabınızın doğru olduğundan ne kadar emin olduğunuzu gösteren 0 ile 1 arasında bir güven puanını belirtilen JSON biçiminde birlikte döndürün.
Chat models, in English (prompt version 5c3a4849752e): the system prompt
You are taking an English multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.
Chat models, in English: the instruction
Answer each of the following multiple-choice questions. Without explanation, return for each question the letter of the correct option together with a confidence score between 0 and 1 showing how sure you are that your answer is correct, in the specified JSON format.
Chat models asked for a plain answer line, in Turkish (the Text tab) (prompt version ba455bb00e5c): the instruction
Aşağıdaki çoktan seçmeli soruyu çöz. Yanıtının son satırına yalnızca 'CEVAP: X' yaz (X = {letters}).
Chat models asked for a plain answer line, in English (prompt version ad6f4cc346df): the instruction
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = {letters}).
With the option, in Turkish (“Bilmiyorum”, the Hallucination tab)
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose Bilmiyorum.
With the option, in English (“I don't know”)
Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.
The option's word: the Turkish questions with “I don't know”
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose I don't know.
Chat models with the option, in Turkish (prompt version d3c64e8c862e): the instruction
Aşağıdaki çoktan seçmeli soruların her birini yanıtlayın. Açıklama yapmadan, her soru için doğru seçeneğin harfini ve cevabınızın doğru olduğundan ne kadar emin olduğunuzu gösteren 0 ile 1 arasında bir güven puanını belirtilen JSON biçiminde birlikte döndürün. Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.
Chat models with the option, in English (prompt version e53d4aa27643): the instruction
Answer each of the following multiple-choice questions. Without explanation, return for each question the letter of the correct option together with a confidence score between 0 and 1 showing how sure you are that your answer is correct, in the specified JSON format. If you do not know the answer, choose I don't know.
Chat models asked for a plain answer line with the option, in Turkish (prompt version 711479895d19): the instruction
Aşağıdaki çoktan seçmeli soruyu çöz. Yanıtının son satırına yalnızca 'CEVAP: X' yaz (X = {letters}). Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.
Chat models asked for a plain answer line with the option, in English (prompt version 4cd69035a45e): the instruction
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = {letters}). If you do not know the answer, choose I don't know.
Results with an “I don't know” option
Ordered by the non-hallucination rate in English, highest first. Each row compares the model's first run with the English option with its first run with the Turkish option, out of the 547 kept questions. Where a side has more than one run, its rates show the range over its runs. The paired counts, p and the Holm-adjusted p (over the 67 decision models and the 56 chat models, each group a family of its own) are those of the exact McNemar test on wrong answers; * marks a Holm-adjusted p below 0.05.
| Model | English non-hallucinationof answers not right, % “I don't know” · higher is better, read with the answered share | English answerednot “I don't know”: (right + wrong) / 547 | English error rateof its answers: wrong / (right + wrong) · lower is better | Turkish non-hallucinationof answers not right, % the Turkish option · higher is better, read with the answered share | Turkish answerednot the Turkish option: (right + wrong) / 547 | Turkish error rateof its answers: wrong / (right + wrong) · lower is better | English runswith “I don't know”; the first is compared | Turkish runswith the Turkish option; the first is compared | English accuracyhigher is better | Turkish accuracyhigher is better | English mean probabilityof “I don't know” | Turkish mean probabilityof the Turkish option | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 67 decision models; over the 56 chat models |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Efficient | 89.3%109 “I don't know” of 122 not right | 80.1%438 of 547 answered, 109 “I don't know” | 3.0%13 of 438 | 76.2%61 “I don't know” of 80 not right | 88.8%486 of 547 answered, 61 “I don't know” | 3.9%19 of 486 | run 1363a new run | run 1367the Hallucination tab's full runs, counted on the kept questions | 77.7%425 of 547 right | 85.4%467 of 547 right | 26.0% | 17.6% | 6 | 12 | 0.24 | 1 |
| celeris-1.1-decision Closed · Efficient | 74.8%98 “I don't know” of 131 not right | 82.1%449 of 547 answered, 98 “I don't know” | 7.3%33 of 449 | Not run | Not run | Not run | run 884a new run | Not run | 76.1%416 of 547 right | Not run | 13.3% | Not run | Not run | Not run | Not run | Not run |
| celeris-1-decision Closed · Efficient | 71.3%92 “I don't know” of 129 not right | 83.2%455 of 547 answered, 92 “I don't know” | 8.1%37 of 455 | Not run | Not run | Not run | run 548a new run | Not run | 76.4%418 of 547 right | Not run | 12.7% | Not run | Not run | Not run | Not run | Not run |
| d1.2 Closed · Efficient | 61.5%59 “I don't know” of 96 not right | 89.2%488 of 547 answered, 59 “I don't know” | 7.6%37 of 488 | 60.2%53 “I don't know” of 88 not right | 90.3%494 of 547 answered, 53 “I don't know” | 7.1%35 of 494 | run 1361a new run | run 1365the Hallucination tab's full runs, counted on the kept questions | 82.4%451 of 547 right | 83.9%459 of 547 right | 19.4% | 17.1% | 21 | 19 | 0.87 | 1 |
| d1-2B Open · 1.9B | 60.9%151 “I don't know” of 248 not right | 72.4%396 of 547 answered, 151 “I don't know” | 24.5%97 of 396 | 43.8%161 “I don't know” of 368 not right | 70.6%386 of 547 answered, 161 “I don't know” | 53.6%207 of 386 | run 710a new run | run 712the Hallucination tab's full runs, counted on the kept questions | 54.7%299 of 547 right | 32.7%179 of 547 right | 26.9% | 23.7% | 38 | 148 | < 0.0001 | < 0.0001 * |
| d1-ultra Closed · Efficient | 56.7%51 “I don't know” of 90 not rightrange 55.3% to 61.3% | 90.7%496 of 547 answered, 51 “I don't know”range 89.6% to 91.4% | 7.9%39 of 496 | 53.6%30 “I don't know” of 56 not rightrange 44.4% to 59.3% | 94.5%517 of 547 answered, 30 “I don't know”range 94.1% to 95.6% | 5.0%26 of 517 | runs 449, 456, 462, 466, 474new runs | runs 350, 354, 360, 365, 370the Hallucination tab's full runs, counted on the kept questions | 83.5%457 of 547 rightrange 82.6% to 84.5% | 89.8%491 of 547 rightrange 89.2% to 90.1% | 25.3%range 25.3% to 25.6% | 22.8%range 22.4% to 23.0% | 20 | 7 | 0.019 | 0.88 |
| GLiDE 0.4B Open · 0.4B | 53.1%153 “I don't know” of 288 not right | 72.0%394 of 547 answered, 153 “I don't know” | 34.3%135 of 394 | 89.6%475 “I don't know” of 530 not right | 13.2%72 of 547 answered, 475 “I don't know” | 76.4%55 of 72 | run 935a new run | run 937the Hallucination tab's full runs, counted on the kept questions | 47.3%259 of 547 right | 3.1%17 of 547 right | 23.3% | 35.2% | 125 | 45 | < 0.0001 | < 0.0001 * |
| Drex Max v1.1 Closed · Efficient (provisional) | 51.2%43 “I don't know” of 84 not right | 92.1%504 of 547 answered, 43 “I don't know” | 8.1%41 of 504 | 15.0%9 “I don't know” of 60 not right | 98.4%538 of 547 answered, 9 “I don't know” | 9.5%51 of 538 | run 447a new run | run 352the Hallucination tab's full runs, counted on the kept questions | 84.6%463 of 547 right | 89.0%487 of 547 right | 14.3% | 8.4% | 20 | 30 | 0.2 | 1 |
| Mammoth-1 Open · 27B | 50.7%37 “I don't know” of 73 not right | 93.2%510 of 547 answered, 37 “I don't know” | 7.1%36 of 510 | 42.1%24 “I don't know” of 57 not right | 95.6%523 of 547 answered, 24 “I don't know” | 6.3%33 of 523 | run 505a new run | run 386the Hallucination tab's full runs, counted on the kept questions | 86.7%474 of 547 right | 89.6%490 of 547 right | 9.8% | 7.4% | 18 | 15 | 0.73 | 1 |
| GLiDE v1.2 Open · 27B | 45.5%30 “I don't know” of 66 not right | 94.5%517 of 547 answered, 30 “I don't know” | 7.0%36 of 517 | 11.3%6 “I don't know” of 53 not right | 98.9%541 of 547 answered, 6 “I don't know” | 8.7%47 of 541 | run 523a new run | run 369the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.3%494 of 547 right | 12.5% | 8.7% | 16 | 27 | 0.13 | 1 |
| Laya Multilingual Open · 0.3B | 42.6%208 “I don't know” of 488 not right | 62.0%339 of 547 answered, 208 “I don't know” | 82.6%280 of 339 | 25.7%124 “I don't know” of 483 not right | 77.3%423 of 547 answered, 124 “I don't know” | 84.9%359 of 423 | run 734a new run | run 737the Hallucination tab's full runs, counted on the kept questions | 10.8%59 of 547 right | 11.7%64 of 547 right | 34.7% | 21.6% | 51 | 130 | < 0.0001 | < 0.0001 * |
| GLiDE 27B Open · 27B | 37.9%22 “I don't know” of 58 not right | 96.0%525 of 547 answered, 22 “I don't know” | 6.9%36 of 525 | 18.8%9 “I don't know” of 48 not right | 98.4%538 of 547 answered, 9 “I don't know” | 7.2%39 of 538 | run 903a new run | run 905the Hallucination tab's full runs, counted on the kept questions | 89.4%489 of 547 right | 91.2%499 of 547 right | 9.3% | 6.5% | 15 | 18 | 0.73 | 1 |
| Claude Fable 5.2 max · chat Closed · Flagship | 28.6%4 “I don't know” of 14 not right | 99.3%543 of 547 answered, 4 “I don't know” | 1.8%10 of 543 | Running | Running | Running | run 1032a new run | Running | 97.4%533 of 547 right | Running | No probabilitiesa chat model states a confidence | Running | Running | Running | Running | Running |
| OneJev1.6-0.8B Open · 0.8B | 28.4%74 “I don't know” of 261 not right | 86.5%473 of 547 answered, 74 “I don't know” | 39.5%187 of 473 | 30.7%112 “I don't know” of 365 not right | 79.5%435 of 547 answered, 112 “I don't know” | 58.2%253 of 435 | run 1027a new run | run 1031the Hallucination tab's full runs, counted on the kept questions | 52.3%286 of 547 right | 33.3%182 of 547 right | 10.7% | 15.6% | 55 | 121 | < 0.0001 | < 0.0001 * |
| Grok 4.7 high · chat Closed · Workhorse | 27.9%12 “I don't know” of 43 not right | 97.8%535 of 547 answered, 12 “I don't know” | 5.8%31 of 535 | 27.6%8 “I don't know” of 29 not right | 98.5%539 of 547 answered, 8 “I don't know” | 3.9%21 of 539 | run 648a new run | run 675the Hallucination tab's full runs, counted on the kept questions | 92.1%504 of 547 right | 94.7%518 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 18 | 8 | 0.076 | 1 |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 26.4%51 “I don't know” of 193 not right | 90.5%495 of 547 answered, 51 “I don't know”, 1 invalid | 28.7%142 of 495 | 50.8%201 “I don't know” of 396 not right | 52.5%287 of 547 answered, 201 “I don't know”, 59 invalid | 67.9%195 of 287 | run 887a new run | run 1298the Hallucination tab's full runs, counted on the kept questions | 64.5%353 of 547 right | 16.8%92 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 80 | 133 | 0.00034 | 0.018 * |
| GLiDE 2B Open · 1.9B | 26.3%47 “I don't know” of 179 not right | 91.4%500 of 547 answered, 47 “I don't know” | 26.4%132 of 500 | 24.0%66 “I don't know” of 275 not right | 87.9%481 of 547 answered, 66 “I don't know” | 43.5%209 of 481 | run 919a new run | run 921the Hallucination tab's full runs, counted on the kept questions | 67.3%368 of 547 right | 49.7%272 of 547 right | 12.5% | 15.4% | 48 | 125 | < 0.0001 | < 0.0001 * |
| Raon-1 nano reasoning off · chat Open · 3.1B | 26.2%71 “I don't know” of 271 not right | 86.8%475 of 547 answered, 71 “I don't know”, 1 invalid | 42.1%200 of 475 | Not run | Not run | Not run | run 1153a new run | Not run | 50.3%275 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| Elephant-1 Open · 9.0B | 25.6%23 “I don't know” of 90 not right | 95.8%524 of 547 answered, 23 “I don't know” | 12.8%67 of 524 | 15.5%15 “I don't know” of 97 not right | 97.3%532 of 547 answered, 15 “I don't know” | 15.4%82 of 532 | run 501a new run | run 382the Hallucination tab's full runs, counted on the kept questions | 83.5%457 of 547 right | 82.3%450 of 547 right | 6.8% | 5.7% | 31 | 46 | 0.11 | 1 |
| Gemini 4 Carbon high · chat Closed · Flagship | 25.0%4 “I don't know” of 16 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.2%12 of 543 | 26.7%4 “I don't know” of 15 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.0%11 of 543 | run 1067a new run | run 1121the Hallucination tab's full runs, counted on the kept questions | 97.1%531 of 547 right | 97.3%532 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 3 | 2 | 1 | 1 |
| SubQ Snap 1.1 Closed · Efficient | 25.0%21 “I don't know” of 84 not right | 96.2%526 of 547 answered, 21 “I don't know” | 12.0%63 of 526 | 27.1%19 “I don't know” of 70 not right | 96.5%528 of 547 answered, 19 “I don't know” | 9.7%51 of 528 | run 448a new run | run 392the Hallucination tab's full runs, counted on the kept questions | 84.6%463 of 547 right | 87.2%477 of 547 right | 11.7% | 8.4% | 39 | 27 | 0.18 | 1 |
| KnowLine-2B-Gen1 Open · 2.0B | 22.0%38 “I don't know” of 173 not right | 93.1%509 of 547 answered, 38 “I don't know” | 26.5%135 of 509 | Not run | Not run | Not run | run 1059a new run | Not run | 68.4%374 of 547 right | Not run | 7.3% | Not run | Not run | Not run | Not run | Not run |
| GLiDE 0.8B Open · 0.8B | 21.7%49 “I don't know” of 226 not right | 91.0%498 of 547 answered, 49 “I don't know” | 35.5%177 of 498 | 24.3%74 “I don't know” of 304 not right | 86.5%473 of 547 answered, 74 “I don't know” | 48.6%230 of 473 | run 927a new run | run 929the Hallucination tab's full runs, counted on the kept questions | 58.7%321 of 547 right | 44.4%243 of 547 right | 14.8% | 15.4% | 68 | 121 | 0.00014 | 0.0079 * |
| GLiClass v4 Edge Open · 0.4B | 21.0%52 “I don't know” of 248 not right | 90.5%495 of 547 answered, 52 “I don't know” | 39.6%196 of 495 | 14.1%58 “I don't know” of 410 not right | 89.4%489 of 547 answered, 58 “I don't know” | 72.0%352 of 489 | run 1044a new run | run 1048the Hallucination tab's full runs, counted on the kept questions | 54.7%299 of 547 right | 25.0%137 of 547 right | 11.8% | 12.0% | 48 | 204 | < 0.0001 | < 0.0001 * |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 20.0%4 “I don't know” of 20 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.9%16 of 543 | 28.6%4 “I don't know” of 14 not right | 99.3%543 of 547 answered, 4 “I don't know” | 1.8%10 of 543 | run 1109a new run | run 1117the Hallucination tab's full runs, counted on the kept questions | 96.3%527 of 547 right | 97.4%533 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 7 | 1 | 0.07 | 1 |
| MiniCPM5-2B reasoning on · chat Open · 2.5B | 19.7%28 “I don't know” of 142 not right | 94.1%515 of 547 answered, 28 “I don't know”, 4 invalid | 22.1%114 of 515 | Not run | Not run | Not run | run 1052a new run | Not run | 73.3%401 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| GLiNER2.5 Multi Open · 0.3B | 19.6%92 “I don't know” of 470 not right | 83.2%455 of 547 answered, 92 “I don't know” | 83.1%378 of 455 | 6.1%28 “I don't know” of 456 not right | 94.9%519 of 547 answered, 28 “I don't know” | 82.5%428 of 519 | run 722a new run | run 724the Hallucination tab's full runs, counted on the kept questions | 14.1%77 of 547 right | 16.6%91 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 45 | 95 | < 0.0001 | 0.0015 * |
| GPT-6 Astra high · chat Closed · Flagship | 19.0%4 “I don't know” of 21 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.1%17 of 543 | 19.0%4 “I don't know” of 21 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.1%17 of 543 | run 814a new run | run 818the Hallucination tab's full runs, counted on the kept questions | 96.2%526 of 547 right | 96.2%526 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 6 | 6 | 1 | 1 |
| Gemini 4 Argon low · chat Closed · Flagship | 18.5%5 “I don't know” of 27 not right | 99.1%542 of 547 answered, 5 “I don't know” | 4.1%22 of 542 | 23.5%4 “I don't know” of 17 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.4%13 of 543 | run 652a new run | run 679the Hallucination tab's full runs, counted on the kept questions | 95.1%520 of 547 right | 96.9%530 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 15 | 6 | 0.078 | 1 |
| Laya Open · 0.4B | 18.4%83 “I don't know” of 450 not right | 84.8%464 of 547 answered, 83 “I don't know” | 79.1%367 of 464 | 12.0%53 “I don't know” of 443 not right | 90.3%494 of 547 answered, 53 “I don't know” | 78.9%390 of 494 | run 728a new run | run 730the Hallucination tab's full runs, counted on the kept questions | 17.7%97 of 547 right | 19.0%104 of 547 right | 14.2% | 11.1% | 97 | 120 | 0.14 | 1 |
| GPT-6 Astra low · chat Closed · Flagship | 18.2%4 “I don't know” of 22 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.3%18 of 543 | 22.2%4 “I don't know” of 18 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.6%14 of 543 | run 647a new run | run 816the Hallucination tab's full runs, counted on the kept questions | 96.0%525 of 547 right | 96.7%529 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 8 | 4 | 0.39 | 1 |
| Open-Jev-27B-v1.1 Open · 27B | 17.4%16 “I don't know” of 92 not right | 97.1%531 of 547 answered, 16 “I don't know” | 14.3%76 of 531 | 7.1%6 “I don't know” of 84 not right | 98.9%541 of 547 answered, 6 “I don't know” | 14.4%78 of 541 | run 763a new run | run 768the Hallucination tab's full runs, counted on the kept questions | 83.2%455 of 547 right | 84.6%463 of 547 right | 15.2% | 13.2% | 26 | 28 | 0.89 | 1 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 16.9%32 “I don't know” of 189 not right | 94.1%515 of 547 answered, 32 “I don't know” | 30.5%157 of 515 | 12.3%33 “I don't know” of 268 not right | 94.0%514 of 547 answered, 33 “I don't know” | 45.7%235 of 514 | run 975a new run | run 979the Hallucination tab's full runs, counted on the kept questions | 65.4%358 of 547 right | 51.0%279 of 547 right | 6.7% | 7.6% | 47 | 125 | < 0.0001 | < 0.0001 * |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 16.8%33 “I don't know” of 197 not right | 89.2%488 of 547 answered, 33 “I don't know”, 26 invalid | 33.6%164 of 488 | 22.7%22 “I don't know” of 97 not right | 20.8%114 of 547 answered, 22 “I don't know”, 411 invalid | 65.8%75 of 114 | run 1095a new run | run 1304the Hallucination tab's full runs, counted on the kept questions | 59.2%324 of 547 right | 7.1%39 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 126 | 37 | < 0.0001 | < 0.0001 * |
| GLiDE 9B Open · 7.9B | 16.7%14 “I don't know” of 84 not right | 97.4%533 of 547 answered, 14 “I don't know” | 13.1%70 of 533 | 9.8%9 “I don't know” of 92 not right | 98.4%538 of 547 answered, 9 “I don't know” | 15.4%83 of 538 | run 911a new run | run 913the Hallucination tab's full runs, counted on the kept questions | 84.6%463 of 547 right | 83.2%455 of 547 right | 7.8% | 6.9% | 25 | 38 | 0.13 | 1 |
| GPT-6 Astra medium · chat Closed · Flagship | 16.7%4 “I don't know” of 24 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.7%20 of 543 | 20.0%4 “I don't know” of 20 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.9%16 of 543 | run 813a new run | run 817the Hallucination tab's full runs, counted on the kept questions | 95.6%523 of 547 right | 96.3%527 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 8 | 4 | 0.39 | 1 |
| GPT-6.1 Sol high · chat Closed · Workhorse | 16.7%4 “I don't know” of 24 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.7%20 of 543 | 20.0%4 “I don't know” of 20 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.9%16 of 543 | run 758a new run | run 802the Hallucination tab's full runs, counted on the kept questions | 95.6%523 of 547 right | 96.3%527 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 9 | 5 | 0.42 | 1 |
| GPT-6.1 Terra high · chat Closed · Workhorse | 16.7%4 “I don't know” of 24 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.7%20 of 543 | 20.8%5 “I don't know” of 24 not right | 99.1%542 of 547 answered, 5 “I don't know” | 3.5%19 of 542 | run 766a new run | run 807the Hallucination tab's full runs, counted on the kept questions | 95.6%523 of 547 right | 95.6%523 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 8 | 7 | 1 | 1 |
| GLiNER2.5-multi-Decide Open · 0.3B | 16.2%70 “I don't know” of 433 not right | 87.2%477 of 547 answered, 70 “I don't know” | 76.1%363 of 477 | 5.5%24 “I don't know” of 436 not right | 95.6%523 of 547 answered, 24 “I don't know” | 78.8%412 of 523 | run 716a new run | run 718the Hallucination tab's full runs, counted on the kept questions | 20.8%114 of 547 right | 20.3%111 of 547 right | 13.2% | 10.4% | 46 | 95 | < 0.0001 | 0.0026 * |
| Granite-4.2-3B reasoning on · chat Open · 3.0B | 16.1%23 “I don't know” of 143 not right | 95.8%524 of 547 answered, 23 “I don't know” | 22.9%120 of 524 | Not run | Not run | Not run | run 1110a new run | Not run | 73.9%404 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 15.7%21 “I don't know” of 134 not right | 95.6%523 of 547 answered, 21 “I don't know”, 3 invalid | 21.6%113 of 523 | 15.8%43 “I don't know” of 272 not right | 90.7%496 of 547 answered, 43 “I don't know”, 8 invalid | 46.2%229 of 496 | run 1035a new run | run 1038the Hallucination tab's full runs, counted on the kept questions | 75.0%410 of 547 right | 48.8%267 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 44 | 160 | < 0.0001 | < 0.0001 * |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 15.6%26 “I don't know” of 167 not right | 95.2%521 of 547 answered, 26 “I don't know” | 27.1%141 of 521 | 33.5%106 “I don't know” of 316 not right | 80.6%441 of 547 answered, 106 “I don't know” | 47.6%210 of 441 | run 1062a new run | run 1090the Hallucination tab's full runs, counted on the kept questions | 69.5%380 of 547 right | 42.2%231 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 68 | 137 | < 0.0001 | < 0.0001 * |
| GPT-6.1 Terra low · chat Closed · Workhorse | 15.4%4 “I don't know” of 26 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.1%22 of 543 | 14.8%4 “I don't know” of 27 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.2%23 of 543 | run 751a new run | run 801the Hallucination tab's full runs, counted on the kept questions | 95.2%521 of 547 right | 95.1%520 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 11 | 12 | 1 | 1 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 15.2%10 “I don't know” of 66 not right | 98.2%537 of 547 answered, 10 “I don't know” | 10.4%56 of 537 | 5.6%3 “I don't know” of 54 not right | 99.5%544 of 547 answered, 3 “I don't know” | 9.4%51 of 544 | run 644a new run | run 642the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.1%493 of 547 right | 4.6% | 2.2% | 26 | 21 | 0.56 | 1 |
| Winnow-12B Open · 12B | 15.0%15 “I don't know” of 100 not right | 97.3%532 of 547 answered, 15 “I don't know” | 16.0%85 of 532 | 8.6%8 “I don't know” of 93 not right | 98.5%539 of 547 answered, 8 “I don't know” | 15.8%85 of 539 | run 748a new run | run 750the Hallucination tab's full runs, counted on the kept questions | 81.7%447 of 547 right | 83.0%454 of 547 right | 3.1% | 1.8% | 32 | 32 | 1 | 1 |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 14.8%4 “I don't know” of 27 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.2%23 of 543 | 17.4%4 “I don't know” of 23 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.5%19 of 543 | run 815a new run | run 819the Hallucination tab's full runs, counted on the kept questions | 95.1%520 of 547 right | 95.8%524 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 10 | 6 | 0.45 | 1 |
| GPT-6.1 Sol low · chat Closed · Workhorse | 14.8%4 “I don't know” of 27 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.2%23 of 543 | 18.2%4 “I don't know” of 22 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.3%18 of 543 | run 744a new run | run 799the Hallucination tab's full runs, counted on the kept questions | 95.1%520 of 547 right | 96.0%525 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 9 | 4 | 0.27 | 1 |
| Mercury Decide Closed · Efficient | 13.4%11 “I don't know” of 82 not rightrange 11.1% to 13.4% | 98.0%536 of 547 answered, 11 “I don't know”range 98.0% to 98.4% | 13.2%71 of 536 | 10.1%8 “I don't know” of 79 not rightrange 10.1% to 10.5% | 98.5%539 of 547 answered, 8 “I don't know”the same in every run | 13.2%71 of 539 | runs 450, 460, 469, 478, 484new runs | runs 489, 491, 493, 495, 497new runs on the kept questions | 85.0%465 of 547 rightrange 85.0% to 85.4% | 85.6%468 of 547 rightrange 85.6% to 86.1% | 3.2%range 3.1% to 3.2% | 1.9%range 1.9% to 2.0% | 29 | 29 | 1 | 1 |
| LFM2.5-24B-A2B default reasoning · chat Open · 24B MoE · 2.5B active | 13.2%14 “I don't know” of 106 not right | 97.4%533 of 547 answered, 14 “I don't know” | 17.3%92 of 533 | Not run | Not run | Not run | run 1070a new run | Not run | 80.6%441 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 12.9%4 “I don't know” of 31 not right | 99.3%543 of 547 answered, 4 “I don't know” | 5.0%27 of 543 | 17.4%4 “I don't know” of 23 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.5%19 of 543 | run 769a new run | run 810the Hallucination tab's full runs, counted on the kept questions | 94.3%516 of 547 right | 95.8%524 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 9 | 1 | 0.021 | 1 |
| LFM3-1.7B reasoning off · chat Open · 1.7B | 11.6%18 “I don't know” of 155 not right | 96.7%529 of 547 answered, 18 “I don't know” | 25.9%137 of 529 | 55.4%215 “I don't know” of 388 not right | 56.9%311 of 547 answered, 215 “I don't know”, 21 invalid | 55.6%173 of 311 | run 1225a new run | run 1234the Hallucination tab's full runs, counted on the kept questions | 71.7%392 of 547 right | 25.2%138 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 79 | 115 | 0.012 | 0.57 |
| Gemma 4 E2B default reasoning · chat Open · 5.1B | 11.5%17 “I don't know” of 148 not right | 96.5%528 of 547 answered, 17 “I don't know”, 2 invalid | 24.8%131 of 528 | Not run | Not run | Not run | run 893a new run | Not run | 72.6%397 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| Gemma 4 E2B reasoning on · chat Open · 5.1B | 11.3%17 “I don't know” of 150 not right | 96.3%527 of 547 answered, 17 “I don't know”, 3 invalid | 25.2%133 of 527 | Not run | Not run | Not run | run 1350a new run | Not run | 72.0%394 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| Cygnet Open · 12B | 11.2%10 “I don't know” of 89 not right | 98.2%537 of 547 answered, 10 “I don't know” | 14.7%79 of 537 | 5.8%5 “I don't know” of 86 not right | 99.1%542 of 547 answered, 5 “I don't know” | 14.9%81 of 542 | run 755a new run | run 757the Hallucination tab's full runs, counted on the kept questions | 83.7%458 of 547 right | 84.3%461 of 547 right | 3.5% | 2.8% | 28 | 30 | 0.9 | 1 |
| KnowLine-0.8B-Gen1 Open · 0.8B | 10.7%26 “I don't know” of 243 not right | 95.2%521 of 547 answered, 26 “I don't know” | 41.7%217 of 521 | 4.4%15 “I don't know” of 340 not right | 97.3%532 of 547 answered, 15 “I don't know” | 61.1%325 of 532 | run 1003a new run | run 1007the Hallucination tab's full runs, counted on the kept questions | 55.6%304 of 547 right | 37.8%207 of 547 right | 5.8% | 6.0% | 49 | 157 | < 0.0001 | < 0.0001 * |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 10.3%18 “I don't know” of 174 not right | 96.3%527 of 547 answered, 18 “I don't know”, 2 invalid | 29.6%156 of 527 | 16.5%55 “I don't know” of 334 not right | 83.4%456 of 547 answered, 55 “I don't know”, 36 invalid | 61.2%279 of 456 | run 1216a new run | run 1219the Hallucination tab's full runs, counted on the kept questions | 67.8%371 of 547 right | 32.4%177 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 50 | 173 | < 0.0001 | < 0.0001 * |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 10.0%2 “I don't know” of 20 not right | 99.6%545 of 547 answered, 2 “I don't know” | 3.3%18 of 545 | 13.3%2 “I don't know” of 15 not right | 99.6%545 of 547 answered, 2 “I don't know” | 2.4%13 of 545 | run 657a new run | run 681the Hallucination tab's full runs, counted on the kept questions | 96.3%527 of 547 right | 97.3%532 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 8 | 3 | 0.23 | 1 |
| Qwen3.5-2B reasoning on · chat Open · 2.0B | 9.9%14 “I don't know” of 142 not right | 97.1%531 of 547 answered, 14 “I don't know”, 2 invalid | 24.1%128 of 531 | Not run | Not run | Not run | run 1344a new run | Not run | 73.7%403 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 9.8%10 “I don't know” of 102 not right | 97.8%535 of 547 answered, 10 “I don't know”, 2 invalid | 17.2%92 of 535 | 20.7%25 “I don't know” of 121 not right | 95.4%522 of 547 answered, 25 “I don't know” | 18.4%96 of 522 | run 890a new run | run 1049the Hallucination tab's full runs, counted on the kept questions | 81.0%443 of 547 right | 77.9%426 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 44 | 48 | 0.75 | 1 |
| GPT-6 Luna Decisions API Closed · Efficient | 9.4%10 “I don't know” of 106 not right | 98.2%537 of 547 answered, 10 “I don't know” | 17.9%96 of 537 | 7.1%6 “I don't know” of 84 not right | 98.9%541 of 547 answered, 6 “I don't know” | 14.4%78 of 541 | run 444a new run | run 453a new run on the kept questions | 80.6%441 of 547 right | 84.6%463 of 547 right | 3.0% | 2.7% | 47 | 29 | 0.05 | 1 |
| Gemini Decision 1 20261111 Closed · Efficient | 9.1%5 “I don't know” of 55 not right | 99.1%542 of 547 answered, 5 “I don't know” | 9.2%50 of 542 | 10.9%5 “I don't know” of 46 not right | 99.1%542 of 547 answered, 5 “I don't know” | 7.6%41 of 542 | run 1019a new run | run 1023the Hallucination tab's full runs, counted on the kept questions | 89.9%492 of 547 right | 91.6%501 of 547 right | 4.6% | 4.2% | 20 | 11 | 0.15 | 1 |
| Perplexity Decider 27B v1.3 Open · 27B | 8.8%5 “I don't know” of 57 not right | 99.1%542 of 547 answered, 5 “I don't know” | 9.6%52 of 542 | 0.0%0 “I don't know” of 52 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.5%52 of 547 | run 1139a new run | run 1141the Hallucination tab's full runs, counted on the kept questions | 89.6%490 of 547 right | 90.5%495 of 547 right | 3.8% | 1.7% | 24 | 24 | 1 | 1 |
| Turion Jev 31B Open · 31B | 8.5%4 “I don't know” of 47 not right | 99.3%543 of 547 answered, 4 “I don't know” | 7.9%43 of 543 | 5.7%2 “I don't know” of 35 not right | 99.6%545 of 547 answered, 2 “I don't know” | 6.1%33 of 545 | run 1077a new run | run 1285the Hallucination tab's full runs, counted on the kept questions | 91.4%500 of 547 right | 93.6%512 of 547 right | 2.3% | 1.6% | 21 | 11 | 0.11 | 1 |
| d3-ultra Open · 26B | 8.2%5 “I don't know” of 61 not right | 99.1%542 of 547 answered, 5 “I don't know” | 10.3%56 of 542 | 2.1%1 “I don't know” of 47 not right | 99.8%546 of 547 answered, 1 “I don't know” | 8.4%46 of 546 | run 795a new run | run 363the Hallucination tab's full runs, counted on the kept questions | 88.8%486 of 547 right | 91.4%500 of 547 right | 2.8% | 1.0% | 33 | 23 | 0.23 | 1 |
| Claude Haiku 5.5 max · chat Closed · Efficient | 7.8%4 “I don't know” of 51 not right | 96.3%527 of 547 answered, 4 “I don't know”, 16 invalid | 8.9%47 of 527 | 17.1%6 “I don't know” of 35 not right | 96.0%525 of 547 answered, 6 “I don't know”, 16 invalid | 5.5%29 of 525 | run 951a new run | run 950the Hallucination tab's full runs, counted on the kept questions | 87.8%480 of 547 right | 90.7%496 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 25 | 7 | 0.0021 | 0.11 |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 7.7%2 “I don't know” of 26 not right | 99.6%545 of 547 answered, 2 “I don't know” | 4.4%24 of 545 | 14.3%3 “I don't know” of 21 not right | 99.5%544 of 547 answered, 3 “I don't know” | 3.3%18 of 544 | run 669a new run | run 691the Hallucination tab's full runs, counted on the kept questions | 95.2%521 of 547 right | 96.2%526 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 12 | 6 | 0.24 | 1 |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 7.5%4 “I don't know” of 53 not right | 99.3%543 of 547 answered, 4 “I don't know” | 9.0%49 of 543 | 7.1%3 “I don't know” of 42 not right | 99.5%544 of 547 answered, 3 “I don't know” | 7.2%39 of 544 | run 646a new run | run 653the Hallucination tab's full runs, counted on the kept questions | 90.3%494 of 547 right | 92.3%505 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 25 | 15 | 0.15 | 1 |
| vera-medium-omni Open · 4.2B | 7.4%7 “I don't know” of 94 not right | 98.7%540 of 547 answered, 7 “I don't know” | 16.1%87 of 540 | 4.2%5 “I don't know” of 118 not right | 99.1%542 of 547 answered, 5 “I don't know” | 20.8%113 of 542 | run 1269a new run | run 1271the Hallucination tab's full runs, counted on the kept questions | 82.8%453 of 547 right | 78.4%429 of 547 right | 5.7% | 5.3% | 26 | 52 | 0.0043 | 0.23 |
| Claude Fable 5.1 high · chat Closed · Flagship | 7.4%2 “I don't know” of 27 not right | 99.6%545 of 547 answered, 2 “I don't know” | 4.6%25 of 545 | 14.8%4 “I don't know” of 27 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.2%23 of 543 | run 655a new run | run 695the Hallucination tab's full runs, counted on the kept questions | 95.1%520 of 547 right | 95.1%520 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 11 | 9 | 0.82 | 1 |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 7.3%3 “I don't know” of 41 not right | 99.5%544 of 547 answered, 3 “I don't know” | 7.0%38 of 544 | 7.7%3 “I don't know” of 39 not right | 99.5%544 of 547 answered, 3 “I don't know” | 6.6%36 of 544 | run 651a new run | run 686the Hallucination tab's full runs, counted on the kept questions | 92.5%506 of 547 right | 92.9%508 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 16 | 14 | 0.86 | 1 |
| Perplexity Decider Flash v1.3 Open · 4.0B | 7.2%7 “I don't know” of 97 not right | 98.7%540 of 547 answered, 7 “I don't know” | 16.7%90 of 540 | 7.6%9 “I don't know” of 119 not right | 98.4%538 of 547 answered, 9 “I don't know” | 20.4%110 of 538 | run 1128a new run | run 1130the Hallucination tab's full runs, counted on the kept questions | 82.3%450 of 547 right | 78.2%428 of 547 right | 5.1% | 5.0% | 37 | 57 | 0.049 | 1 |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 7.1%3 “I don't know” of 42 not right | 99.5%544 of 547 answered, 3 “I don't know” | 7.2%39 of 544 | 2.6%1 “I don't know” of 39 not right | 99.8%546 of 547 answered, 1 “I don't know” | 7.0%38 of 546 | run 809a new run | run 811the Hallucination tab's full runs, counted on the kept questions | 92.3%505 of 547 right | 92.9%508 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 19 | 18 | 1 | 1 |
| GPT-6 Luna low · chat Closed · Efficient | 6.8%5 “I don't know” of 74 not right | 99.1%542 of 547 answered, 5 “I don't know” | 12.7%69 of 542 | 3.0%2 “I don't know” of 67 not right | 99.6%545 of 547 answered, 2 “I don't know” | 11.9%65 of 545 | run 735a new run | run 796the Hallucination tab's full runs, counted on the kept questions | 86.5%473 of 547 right | 87.8%480 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 31 | 27 | 0.69 | 1 |
| REAX 1.3 Open · 57B | 6.7%3 “I don't know” of 45 not right | 99.5%544 of 547 answered, 3 “I don't know” | 7.7%42 of 544 | 2.8%1 “I don't know” of 36 not right | 99.8%546 of 547 answered, 1 “I don't know” | 6.4%35 of 546 | run 831a new run | run 833the Hallucination tab's full runs, counted on the kept questions | 91.8%502 of 547 right | 93.4%511 of 547 right | 1.9% | 0.7% | 20 | 13 | 0.3 | 1 |
| Claude Opus 5.5 high · chat Closed · Flagship | 6.5%2 “I don't know” of 31 not right | 99.6%545 of 547 answered, 2 “I don't know” | 5.3%29 of 545 | 11.5%3 “I don't know” of 26 not right | 99.5%544 of 547 answered, 3 “I don't know” | 4.2%23 of 544 | run 665a new run | run 773the Hallucination tab's full runs, counted on the kept questions | 94.3%516 of 547 right | 95.2%521 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 13 | 7 | 0.26 | 1 |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 6.2%3 “I don't know” of 48 not right | 99.5%544 of 547 answered, 3 “I don't know” | 8.3%45 of 544 | 8.5%4 “I don't know” of 47 not right | 99.3%543 of 547 answered, 4 “I don't know” | 7.9%43 of 543 | run 873a new run | run 872the Hallucination tab's full runs, counted on the kept questions | 91.2%499 of 547 right | 91.4%500 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 18 | 16 | 0.86 | 1 |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 5.6%4 “I don't know” of 71 not right | 99.3%543 of 547 answered, 4 “I don't know” | 12.3%67 of 543 | 3.5%3 “I don't know” of 85 not right | 99.5%544 of 547 answered, 3 “I don't know” | 15.1%82 of 544 | run 821a new run | run 822the Hallucination tab's full runs, counted on the kept questions | 87.0%476 of 547 right | 84.5%462 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 26 | 41 | 0.086 | 1 |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 5.6%4 “I don't know” of 71 not right | 99.3%543 of 547 answered, 4 “I don't know” | 12.3%67 of 543 | 5.2%3 “I don't know” of 58 not right | 99.5%544 of 547 answered, 3 “I don't know” | 10.1%55 of 544 | run 988a new run | run 987the Hallucination tab's full runs, counted on the kept questions | 87.0%476 of 547 right | 89.4%489 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 31 | 19 | 0.12 | 1 |
| Gemma 4 12B Unified default reasoning · chat Open · 12B | 5.5%4 “I don't know” of 73 not right | 98.9%541 of 547 answered, 4 “I don't know”, 2 invalid | 12.8%69 of 541 | Not run | Not run | Not run | run 896a new run | Not run | 86.3%472 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 5.4%4 “I don't know” of 74 not right | 99.3%543 of 547 answered, 4 “I don't know” | 12.9%70 of 543 | 0.0%0 “I don't know” of 57 not right | 100.0%547 of 547 answered, 0 “I don't know” | 10.4%57 of 547 | run 805a new run | run 806the Hallucination tab's full runs, counted on the kept questions | 86.5%473 of 547 right | 89.6%490 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 33 | 20 | 0.098 | 1 |
| Clef-Flash Open · 9.0B | 5.3%5 “I don't know” of 95 not right | 99.1%542 of 547 answered, 5 “I don't know” | 16.6%90 of 542 | 9.4%12 “I don't know” of 127 not right | 97.8%535 of 547 answered, 12 “I don't know” | 21.5%115 of 535 | run 660a new run | run 664the Hallucination tab's full runs, counted on the kept questions | 82.6%452 of 547 right | 76.8%420 of 547 right | 3.1% | 4.8% | 27 | 52 | 0.0066 | 0.33 |
| Muse Ember max · chat Closed · Efficient | 5.3%3 “I don't know” of 57 not right | 99.5%544 of 547 answered, 3 “I don't know” | 9.9%54 of 544 | 7.8%4 “I don't know” of 51 not right | 99.3%543 of 547 answered, 4 “I don't know” | 8.7%47 of 543 | run 943a new run | run 940the Hallucination tab's full runs, counted on the kept questions | 89.6%490 of 547 right | 90.7%496 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 25 | 18 | 0.36 | 1 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 5.2%12 “I don't know” of 229 not right | 97.8%535 of 547 answered, 12 “I don't know” | 40.6%217 of 535 | 6.4%22 “I don't know” of 342 not right | 96.0%525 of 547 answered, 22 “I don't know” | 61.0%320 of 525 | run 1013a new run | run 1017the Hallucination tab's full runs, counted on the kept questions | 58.1%318 of 547 right | 37.5%205 of 547 right | 6.6% | 7.3% | 48 | 151 | < 0.0001 | < 0.0001 * |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 4.8%3 “I don't know” of 62 not right | 99.5%544 of 547 answered, 3 “I don't know” | 10.8%59 of 544 | 3.6%2 “I don't know” of 55 not right | 99.6%545 of 547 answered, 2 “I don't know” | 9.7%53 of 545 | run 668a new run | run 800the Hallucination tab's full runs, counted on the kept questions | 88.7%485 of 547 right | 89.9%492 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 31 | 25 | 0.5 | 1 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 4.8%5 “I don't know” of 105 not right | 99.1%542 of 547 answered, 5 “I don't know” | 18.5%100 of 542 | 4.3%6 “I don't know” of 139 not right | 98.9%541 of 547 answered, 6 “I don't know” | 24.6%133 of 541 | run 535a new run | run 537a new run on the kept questions | 80.8%442 of 547 right | 74.6%408 of 547 right | 5.5% | 6.3% | 36 | 69 | 0.0017 | 0.091 |
| Claude Opus 5.5 low · chat Closed · Flagship | 4.7%2 “I don't know” of 43 not right | 99.6%545 of 547 answered, 2 “I don't know” | 7.5%41 of 545 | 6.5%2 “I don't know” of 31 not right | 99.6%545 of 547 answered, 2 “I don't know” | 5.3%29 of 545 | run 649a new run | run 683the Hallucination tab's full runs, counted on the kept questions | 92.1%504 of 547 right | 94.3%516 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 25 | 13 | 0.073 | 1 |
| Perplexity Decider 27B v1.2 API Open · 27B | 4.5%3 “I don't know” of 66 not right | 99.5%544 of 547 answered, 3 “I don't know” | 11.6%63 of 544 | 0.0%0 “I don't know” of 51 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.3%51 of 547 | run 643a new run | run 641the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.7%496 of 547 right | 3.1% | 1.1% | 27 | 15 | 0.088 | 1 |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 4.4%4 “I don't know” of 90 not right | 99.3%543 of 547 answered, 4 “I don't know” | 15.8%86 of 543 | 5.7%5 “I don't know” of 87 not right | 99.1%542 of 547 answered, 5 “I don't know” | 15.1%82 of 542 | run 654a new run | run 690the Hallucination tab's full runs, counted on the kept questions | 83.5%457 of 547 right | 84.1%460 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 36 | 32 | 0.72 | 1 |
| Logits Sigma-1 Duo Open · 57B | 4.3%2 “I don't know” of 46 not right | 99.6%545 of 547 answered, 2 “I don't know” | 8.1%44 of 545 | 0.0%0 “I don't know” of 36 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.6%36 of 547 | run 1231a new run | run 1233the Hallucination tab's full runs, counted on the kept questions | 91.6%501 of 547 right | 93.4%511 of 547 right | 1.8% | 0.8% | 25 | 17 | 0.28 | 1 |
| Jev 1.14 Closed · Efficient | 4.0%3 “I don't know” of 75 not rightrange 4.0% to 5.2% | 99.5%544 of 547 answered, 3 “I don't know”range 99.3% to 99.5% | 13.2%72 of 544 | 4.2%3 “I don't know” of 72 not rightrange 4.0% to 4.5% | 99.5%544 of 547 answered, 3 “I don't know”the same in every run | 12.7%69 of 544 | runs 446, 454, 457, 461, 464new runs | runs 470, 473, 476, 480, 482new runs on the kept questions | 86.3%472 of 547 rightrange 85.9% to 86.7% | 86.8%475 of 547 rightrange 86.3% to 87.8% | 1.6%range 1.6% to 1.6% | 1.7%range 1.6% to 1.7% | 20 | 17 | 0.74 | 1 |
| OneJev-27B Open · 27B | 4.0%3 “I don't know” of 75 not right | 99.5%544 of 547 answered, 3 “I don't know” | 13.2%72 of 544 | 1.6%1 “I don't know” of 64 not right | 99.8%546 of 547 answered, 1 “I don't know” | 11.5%63 of 546 | run 780a new run | run 782the Hallucination tab's full runs, counted on the kept questions | 86.3%472 of 547 right | 88.3%483 of 547 right | 2.3% | 1.3% | 34 | 25 | 0.3 | 1 |
| Microsoft-Decision-1 Closed · Efficient | 3.8%3 “I don't know” of 80 not rightrange 3.8% to 4.0% | 99.5%544 of 547 answered, 3 “I don't know”the same in every run | 14.2%77 of 544 | 2.2%2 “I don't know” of 90 not rightrange 2.2% to 4.3% | 99.6%545 of 547 answered, 2 “I don't know”range 99.3% to 99.6% | 16.1%88 of 545 | runs 451, 459, 467, 477, 483new runs | runs 487, 490, 492, 494, 496new runs on the kept questions | 85.4%467 of 547 rightrange 85.4% to 86.3% | 83.5%457 of 547 rightrange 82.8% to 83.5% | 3.6%range 3.6% to 3.6% | 2.3%range 2.3% to 2.3% | 24 | 35 | 0.19 | 1 |
| Chinese Decision Pointer 4B Open · 4.0B | 3.7%4 “I don't know” of 108 not right | 99.3%543 of 547 answered, 4 “I don't know” | 19.2%104 of 543 | 3.0%4 “I don't know” of 133 not right | 99.3%543 of 547 answered, 4 “I don't know” | 23.8%129 of 543 | run 1211a new run | run 1213the Hallucination tab's full runs, counted on the kept questions | 80.3%439 of 547 right | 75.7%414 of 547 right | 2.1% | 1.9% | 41 | 66 | 0.02 | 0.9 |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 3.7%3 “I don't know” of 81 not right | 99.5%544 of 547 answered, 3 “I don't know” | 14.3%78 of 544 | 5.3%4 “I don't know” of 76 not right | 99.3%543 of 547 answered, 4 “I don't know” | 13.3%72 of 543 | run 673a new run | run 674the Hallucination tab's full runs, counted on the kept questions | 85.2%466 of 547 right | 86.1%471 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 33 | 27 | 0.52 | 1 |
| Gemma 4 31B reasoning off · chat Open · 31B | 3.6%2 “I don't know” of 55 not right | 99.6%545 of 547 answered, 2 “I don't know” | 9.7%53 of 545 | 4.5%2 “I don't know” of 44 not right | 99.6%545 of 547 answered, 2 “I don't know” | 7.7%42 of 545 | run 775a new run | run 776the Hallucination tab's full runs, counted on the kept questions | 89.9%492 of 547 right | 92.0%503 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 30 | 19 | 0.15 | 1 |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 3.6%4 “I don't know” of 110 not right | 99.3%543 of 547 answered, 4 “I don't know” | 19.5%106 of 543 | 3.4%5 “I don't know” of 145 not right | 98.4%538 of 547 answered, 5 “I don't know”, 4 invalid | 26.0%140 of 538 | run 1144a new run | run 1147the Hallucination tab's full runs, counted on the kept questions | 79.9%437 of 547 right | 72.8%398 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 52 | 86 | 0.0048 | 0.23 |
| Qwen3.8-27B low · chat Open · 27B | 3.6%2 “I don't know” of 55 not right | 99.6%545 of 547 answered, 2 “I don't know” | 9.7%53 of 545 | 2.0%1 “I don't know” of 49 not right | 99.8%546 of 547 answered, 1 “I don't know” | 8.8%48 of 546 | run 671a new run | run 666the Hallucination tab's full runs, counted on the kept questions | 89.9%492 of 547 right | 91.0%498 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 23 | 18 | 0.53 | 1 |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 3.4%1 “I don't know” of 29 not right | 99.8%546 of 547 answered, 1 “I don't know” | 5.1%28 of 546 | 11.1%3 “I don't know” of 27 not right | 99.5%544 of 547 answered, 3 “I don't know” | 4.4%24 of 544 | run 645a new run | run 676the Hallucination tab's full runs, counted on the kept questions | 94.7%518 of 547 right | 95.1%520 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 14 | 10 | 0.54 | 1 |
| GLM-5.3 low · chat Open · Workhorse | 3.4%2 “I don't know” of 58 not right | 99.6%545 of 547 answered, 2 “I don't know” | 10.3%56 of 545 | 4.7%3 “I don't know” of 64 not right | 99.5%544 of 547 answered, 3 “I don't know” | 11.2%61 of 544 | run 650a new run | run 656the Hallucination tab's full runs, counted on the kept questions | 89.4%489 of 547 right | 88.3%483 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 23 | 28 | 0.58 | 1 |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 3.4%3 “I don't know” of 87 not right | 99.5%544 of 547 answered, 3 “I don't know” | 15.4%84 of 544 | 5.1%4 “I don't know” of 79 not right | 99.3%543 of 547 answered, 4 “I don't know” | 13.8%75 of 543 | run 680a new run | run 682the Hallucination tab's full runs, counted on the kept questions | 84.1%460 of 547 right | 85.6%468 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 35 | 26 | 0.31 | 1 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 3.4%3 “I don't know” of 88 not right | 99.5%544 of 547 answered, 3 “I don't know” | 15.6%85 of 544 | 1.1%1 “I don't know” of 91 not right | 99.6%545 of 547 answered, 1 “I don't know”, 1 invalid | 16.5%90 of 545 | run 865a new run | run 864the Hallucination tab's full runs, counted on the kept questions | 83.9%459 of 547 right | 83.2%455 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 38 | 43 | 0.66 | 1 |
| Drex Max Open · 27B | 3.2%2 “I don't know” of 62 not right | 99.6%545 of 547 answered, 2 “I don't know” | 11.0%60 of 545 | 3.6%2 “I don't know” of 55 not right | 99.6%545 of 547 answered, 2 “I don't know” | 9.7%53 of 545 | run 798a new run | run 367the Hallucination tab's full runs, counted on the kept questions | 88.7%485 of 547 right | 89.9%492 of 547 right | 1.7% | 0.6% | 28 | 21 | 0.39 | 1 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 3.0%2 “I don't know” of 66 not right | 99.6%545 of 547 answered, 2 “I don't know” | 11.7%64 of 545 | 0.0%0 “I don't know” of 50 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.1%50 of 547 | run 529a new run | run 372the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.9%497 of 547 right | 2.8% | 0.9% | 29 | 15 | 0.049 | 1 |
| d1 Closed · Efficient | 2.9%2 “I don't know” of 69 not rightrange 2.6% to 2.9% | 99.6%545 of 547 answered, 2 “I don't know”the same in every run | 12.3%67 of 545 | 1.1%1 “I don't know” of 90 not rightrange 0.0% to 1.1% | 99.8%546 of 547 answered, 1 “I don't know”range 99.8% to 100.0% | 16.3%89 of 546 | runs 445, 455, 458, 463, 465new runs | runs 472, 475, 479, 481, 485new runs on the kept questions | 87.4%478 of 547 rightrange 85.9% to 87.4% | 83.5%457 of 547 rightrange 83.0% to 85.0% | 0.9%range 0.9% to 1.0% | 1.3%range 1.1% to 1.3% | 21 | 43 | 0.0081 | 0.4 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 2.6%2 “I don't know” of 76 not right | 99.6%545 of 547 answered, 2 “I don't know” | 13.6%74 of 545 | 1.0%1 “I don't know” of 96 not right | 99.8%546 of 547 answered, 1 “I don't know” | 17.4%95 of 546 | run 1295a new run | run 1297the Hallucination tab's full runs, counted on the kept questions | 86.1%471 of 547 right | 82.4%451 of 547 right | 1.8% | 1.4% | 26 | 47 | 0.019 | 0.88 |
| Cerno 9B v1.2 Open · 9.0B | 2.5%2 “I don't know” of 79 not right | 99.6%545 of 547 answered, 2 “I don't know” | 14.1%77 of 545 | 0.0%0 “I don't know” of 91 not right | 100.0%547 of 547 answered, 0 “I don't know” | 16.6%91 of 547 | run 1087a new run | run 1204the Hallucination tab's full runs, counted on the kept questions | 85.6%468 of 547 right | 83.4%456 of 547 right | 2.2% | 1.6% | 34 | 48 | 0.15 | 1 |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 2.4%2 “I don't know” of 85 not right | 99.6%545 of 547 answered, 2 “I don't know” | 15.2%83 of 545 | 2.2%2 “I don't know” of 90 not right | 99.1%542 of 547 answered, 2 “I don't know”, 3 invalid | 16.2%88 of 542 | run 1073a new run | run 1150the Hallucination tab's full runs, counted on the kept questions | 84.5%462 of 547 right | 83.0%454 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 44 | 49 | 0.68 | 1 |
| Sieve 31B v3 Open · 31B | 2.2%1 “I don't know” of 46 not right | 99.8%546 of 547 answered, 1 “I don't know” | 8.2%45 of 546 | 0.0%0 “I don't know” of 33 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.0%33 of 547 | run 1066a new run | run 1253the Hallucination tab's full runs, counted on the kept questions | 91.6%501 of 547 right | 94.0%514 of 547 right | 0.8% | 0.3% | 23 | 11 | 0.058 | 1 |
| Erk-14B reasoning off · chat Open · 14B | 2.1%3 “I don't know” of 144 not right | 99.5%544 of 547 answered, 3 “I don't know” | 25.9%141 of 544 | 0.8%1 “I don't know” of 130 not right | 99.8%546 of 547 answered, 1 “I don't know” | 23.6%129 of 546 | run 764a new run | run 765the Hallucination tab's full runs, counted on the kept questions | 73.7%403 of 547 right | 76.2%417 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 68 | 56 | 0.32 | 1 |
| Verdin-4B-v1.1 Open · 4.0B | 2.1%2 “I don't know” of 96 not right | 99.6%545 of 547 answered, 2 “I don't know” | 17.2%94 of 545 | 0.8%1 “I don't know” of 128 not right | 99.8%546 of 547 answered, 1 “I don't know” | 23.3%127 of 546 | run 1249a new run | run 1251the Hallucination tab's full runs, counted on the kept questions | 82.4%451 of 547 right | 76.6%419 of 547 right | 3.5% | 2.2% | 37 | 70 | 0.0018 | 0.099 |
| Qwen3-14B reasoning off · chat Open · 15B | 1.8%2 “I don't know” of 110 not right | 99.6%545 of 547 answered, 2 “I don't know” | 19.8%108 of 545 | 0.8%1 “I don't know” of 123 not right | 99.8%546 of 547 answered, 1 “I don't know” | 22.3%122 of 546 | run 771a new run | run 772the Hallucination tab's full runs, counted on the kept questions | 79.9%437 of 547 right | 77.5%424 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 40 | 54 | 0.18 | 1 |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 1.7%1 “I don't know” of 59 not right | 99.8%546 of 547 answered, 1 “I don't know” | 10.6%58 of 546 | 4.2%3 “I don't know” of 72 not right | 99.5%544 of 547 answered, 3 “I don't know” | 12.7%69 of 544 | run 880a new run | run 879the Hallucination tab's full runs, counted on the kept questions | 89.2%488 of 547 right | 86.8%475 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 20 | 31 | 0.16 | 1 |
| d3 Open · 26B | 1.7%1 “I don't know” of 59 not right | 99.8%546 of 547 answered, 1 “I don't know” | 10.6%58 of 546 | 0.0%0 “I don't know” of 53 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.7%53 of 547 | run 793a new run | run 358the Hallucination tab's full runs, counted on the kept questions | 89.2%488 of 547 right | 90.3%494 of 547 right | 2.9% | 1.1% | 24 | 19 | 0.54 | 1 |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 1.6%1 “I don't know” of 62 not right | 99.8%546 of 547 answered, 1 “I don't know” | 11.2%61 of 546 | 3.5%2 “I don't know” of 57 not right | 99.6%545 of 547 answered, 2 “I don't know” | 10.1%55 of 545 | run 824a new run | run 825the Hallucination tab's full runs, counted on the kept questions | 88.7%485 of 547 right | 89.6%490 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 28 | 22 | 0.48 | 1 |
| Metask-Jev-4B Open · 4.5B | 1.6%2 “I don't know” of 125 not right | 99.6%545 of 547 answered, 2 “I don't know” | 22.6%123 of 545 | 1.1%2 “I don't know” of 189 not right | 99.6%545 of 547 answered, 2 “I don't know” | 34.3%187 of 545 | run 741a new run | run 743the Hallucination tab's full runs, counted on the kept questions | 77.1%422 of 547 right | 65.4%358 of 547 right | 3.8% | 4.1% | 34 | 98 | < 0.0001 | < 0.0001 * |
| Clef Open · 27B | 1.3%1 “I don't know” of 76 not right | 99.8%546 of 547 answered, 1 “I don't know” | 13.7%75 of 546 | 1.5%1 “I don't know” of 68 not right | 99.8%546 of 547 answered, 1 “I don't know” | 12.3%67 of 546 | run 785a new run | run 788the Hallucination tab's full runs, counted on the kept questions | 86.1%471 of 547 right | 87.6%479 of 547 right | 2.4% | 3.6% | 26 | 18 | 0.29 | 1 |
| KnowLine-12B-Gen1 Open · 12B | 1.3%1 “I don't know” of 77 not right | 99.8%546 of 547 answered, 1 “I don't know” | 13.9%76 of 546 | 0.0%0 “I don't know” of 67 not right | 100.0%547 of 547 answered, 0 “I don't know” | 12.2%67 of 547 | run 1316a new run | run 1318the Hallucination tab's full runs, counted on the kept questions | 85.9%470 of 547 right | 87.8%480 of 547 right | 1.3% | 0.6% | 26 | 17 | 0.22 | 1 |
| KnowLine-9B-Gen1 Open · 9.0B | 1.3%1 “I don't know” of 77 not right | 99.8%546 of 547 answered, 1 “I don't know” | 13.9%76 of 546 | 0.0%0 “I don't know” of 98 not right | 100.0%547 of 547 answered, 0 “I don't know” | 17.9%98 of 547 | run 1179a new run | run 1181the Hallucination tab's full runs, counted on the kept questions | 85.9%470 of 547 right | 82.1%449 of 547 right | 1.6% | 0.7% | 18 | 40 | 0.0054 | 0.28 |
| KnowLine-9B-Gen2 Open · 9.0B | 1.3%1 “I don't know” of 79 not right | 99.8%546 of 547 answered, 1 “I don't know” | 14.3%78 of 546 | 0.0%0 “I don't know” of 95 not right | 100.0%547 of 547 answered, 0 “I don't know” | 17.4%95 of 547 | run 1323a new run | run 1325the Hallucination tab's full runs, counted on the kept questions | 85.6%468 of 547 right | 82.6%452 of 547 right | 3.8% | 3.5% | 23 | 40 | 0.043 | 1 |
| Valen-Decision-1.5-9B Open · 9.0B | 1.2%1 “I don't know” of 85 not right | 99.8%546 of 547 answered, 1 “I don't know” | 15.4%84 of 546 | 0.0%0 “I don't know” of 110 not right | 100.0%547 of 547 answered, 0 “I don't know” | 20.1%110 of 547 | run 1280a new run | run 1282the Hallucination tab's full runs, counted on the kept questions | 84.5%462 of 547 right | 79.9%437 of 547 right | 3.0% | 2.7% | 32 | 58 | 0.008 | 0.4 |
| Dexy-Decision-9B-V2 Open · 9.0B | 1.1%1 “I don't know” of 91 not right | 99.8%546 of 547 answered, 1 “I don't know” | 16.5%90 of 546 | 1.1%1 “I don't know” of 91 not right | 99.8%546 of 547 answered, 1 “I don't know” | 16.5%90 of 546 | run 1260a new run | run 1262the Hallucination tab's full runs, counted on the kept questions | 83.4%456 of 547 right | 83.4%456 of 547 right | 0.7% | 0.6% | 38 | 38 | 1 | 1 |
| KnowLine 4B Gen5.5 Open · 4.0B | 0.9%1 “I don't know” of 106 not right | 99.8%546 of 547 answered, 1 “I don't know” | 19.2%105 of 546 | 2.5%3 “I don't know” of 121 not right | 99.5%544 of 547 answered, 3 “I don't know” | 21.7%118 of 544 | run 963a new run | run 967the Hallucination tab's full runs, counted on the kept questions | 80.6%441 of 547 right | 77.9%426 of 547 right | 1.3% | 1.0% | 40 | 53 | 0.21 | 1 |
| Strands Decider 2B Open · 2.0B | 0.7%2 “I don't know” of 303 not right | 99.6%545 of 547 answered, 2 “I don't know” | 55.2%301 of 545 | 1.5%5 “I don't know” of 332 not right | 99.1%542 of 547 answered, 5 “I don't know” | 60.3%327 of 542 | run 704a new run | run 706the Hallucination tab's full runs, counted on the kept questions | 44.6%244 of 547 right | 39.3%215 of 547 right | 3.8% | 4.9% | 49 | 75 | 0.024 | 1 |
| Adjudex 9B v5 Open · 9.0B | 0.0%0 “I don't know” of 77 not right | 100.0%547 of 547 answered, 0 “I don't know” | 14.1%77 of 547 | 0.0%0 “I don't know” of 89 not right | 100.0%547 of 547 answered, 0 “I don't know” | 16.3%89 of 547 | run 1188a new run | run 1190the Hallucination tab's full runs, counted on the kept questions | 85.9%470 of 547 right | 83.7%458 of 547 right | 0.6% | 0.3% | 32 | 44 | 0.21 | 1 |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 0.0%0 “I don't know” of 31 not right | 100.0%547 of 547 answered, 0 “I don't know” | 5.7%31 of 547 | 3.8%1 “I don't know” of 26 not right | 99.8%546 of 547 answered, 1 “I don't know” | 4.6%25 of 546 | run 670a new run | run 803the Hallucination tab's full runs, counted on the kept questions | 94.3%516 of 547 right | 95.2%521 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 13 | 7 | 0.26 | 1 |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 0.0%0 “I don't know” of 38 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.9%38 of 547 | 2.7%1 “I don't know” of 37 not right | 99.8%546 of 547 answered, 1 “I don't know” | 6.6%36 of 546 | run 677a new run | run 812the Hallucination tab's full runs, counted on the kept questions | 93.1%509 of 547 right | 93.2%510 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 16 | 14 | 0.86 | 1 |
| Copernicus-31B Open · 31B | 0.0%0 “I don't know” of 39 not right | 100.0%547 of 547 answered, 0 “I don't know” | 7.1%39 of 547 | 2.9%1 “I don't know” of 34 not right | 99.8%546 of 547 answered, 1 “I don't know” | 6.0%33 of 546 | run 1170a new run | run 1172the Hallucination tab's full runs, counted on the kept questions | 92.9%508 of 547 right | 93.8%513 of 547 right | 1.0% | 0.7% | 19 | 13 | 0.38 | 1 |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 0.0%0 “I don't know” of 59 not right | 100.0%547 of 547 answered, 0 “I don't know” | 10.8%59 of 547 | 1.9%1 “I don't know” of 53 not right | 99.8%546 of 547 answered, 1 “I don't know” | 9.5%52 of 546 | run 786a new run | run 789the Hallucination tab's full runs, counted on the kept questions | 89.2%488 of 547 right | 90.3%494 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 31 | 24 | 0.42 | 1 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 0.0%0 “I don't know” of 60 not right | 100.0%547 of 547 answered, 0 “I don't know” | 11.0%60 of 547 | 0.0%0 “I don't know” of 62 not right | 100.0%547 of 547 answered, 0 “I don't know” | 11.3%62 of 547 | run 869a new run | run 868the Hallucination tab's full runs, counted on the kept questions | 89.0%487 of 547 right | 88.7%485 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 30 | 32 | 0.9 | 1 |
| Jeffan-9B-1211 Open · 9.0B | 0.0%0 “I don't know” of 71 not right | 100.0%547 of 547 answered, 0 “I don't know” | 13.0%71 of 547 | 4.1%3 “I don't know” of 74 not right | 99.5%544 of 547 answered, 3 “I don't know” | 13.1%71 of 544 | run 1197a new run | run 1199the Hallucination tab's full runs, counted on the kept questions | 87.0%476 of 547 right | 86.5%473 of 547 right | 3.7% | 3.7% | 31 | 31 | 1 | 1 |
| Karpo Decision 31B Open · 31B | 0.0%0 “I don't know” of 42 not right | 100.0%547 of 547 answered, 0 “I don't know” | 7.7%42 of 547 | 3.2%1 “I don't know” of 31 not right | 99.8%546 of 547 answered, 1 “I don't know” | 5.5%30 of 546 | run 1161a new run | run 1163the Hallucination tab's full runs, counted on the kept questions | 92.3%505 of 547 right | 94.3%516 of 547 right | 0.6% | 0.3% | 25 | 13 | 0.073 | 1 |
| KnowLine-31B-Gen1.5 Open · 31B | 0.0%0 “I don't know” of 37 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.8%37 of 547 | 0.0%0 “I don't know” of 34 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.2%34 of 547 | run 955a new run | run 959the Hallucination tab's full runs, counted on the kept questions | 93.2%510 of 547 right | 93.8%513 of 547 right | 0.2% | 0.2% | 16 | 13 | 0.71 | 1 |
| KnowLine-31B-Gen2 Open · 31B | 0.0%0 “I don't know” of 40 not right | 100.0%547 of 547 answered, 0 “I don't know” | 7.3%40 of 547 | 0.0%0 “I don't know” of 29 not right | 100.0%547 of 547 answered, 0 “I don't know” | 5.3%29 of 547 | run 1329a new run | run 1331the Hallucination tab's full runs, counted on the kept questions | 92.7%507 of 547 right | 94.7%518 of 547 right | 0.2% | 0.1% | 20 | 9 | 0.061 | 1 |
| Qwen4-Max high · chat Closed · Workhorse | 0.0%0 “I don't know” of 29 not right | 100.0%547 of 547 answered, 0 “I don't know” | 5.3%29 of 547 | 3.4%1 “I don't know” of 29 not right | 99.8%546 of 547 answered, 1 “I don't know” | 5.1%28 of 546 | run 1338a new run | run 1339the Hallucination tab's full runs, counted on the kept questions | 94.7%518 of 547 right | 94.7%518 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 12 | 11 | 1 | 1 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 0.0%0 “I don't know” of 94 not right | 100.0%547 of 547 answered, 0 “I don't know” | 17.2%94 of 547 | 1.7%2 “I don't know” of 120 not right | 99.6%545 of 547 answered, 2 “I don't know” | 21.7%118 of 545 | run 1083a new run | run 1239the Hallucination tab's full runs, counted on the kept questions | 82.8%453 of 547 right | 78.1%427 of 547 right | 3.2% | 2.9% | 33 | 57 | 0.015 | 0.71 |
| Model | English non-hallucinationof answers not right, % “I don't know” · higher is better, read with the answered share | English answerednot “I don't know”: (right + wrong) / 547 | English error rateof its answers: wrong / (right + wrong) · lower is better | Turkish non-hallucinationof answers not right, % the Turkish option · higher is better, read with the answered share | Turkish answerednot the Turkish option: (right + wrong) / 547 | Turkish error rateof its answers: wrong / (right + wrong) · lower is better | English runswith “I don't know”; the first is compared | Turkish runswith the Turkish option; the first is compared | English accuracyhigher is better | Turkish accuracyhigher is better | English mean probabilityof “I don't know” | Turkish mean probabilityof the Turkish option | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 56 chat models |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.2 max · chat Closed · Flagship | 28.6%4 “I don't know” of 14 not right | 99.3%543 of 547 answered, 4 “I don't know” | 1.8%10 of 543 | Running | Running | Running | run 1032a new run | Running | 97.4%533 of 547 right | Running | No probabilitiesa chat model states a confidence | Running | Running | Running | Running | Running |
| Grok 4.7 high · chat Closed · Workhorse | 27.9%12 “I don't know” of 43 not right | 97.8%535 of 547 answered, 12 “I don't know” | 5.8%31 of 535 | 27.6%8 “I don't know” of 29 not right | 98.5%539 of 547 answered, 8 “I don't know” | 3.9%21 of 539 | run 648a new run | run 675the Hallucination tab's full runs, counted on the kept questions | 92.1%504 of 547 right | 94.7%518 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 18 | 8 | 0.076 | 1 |
| LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active | 26.4%51 “I don't know” of 193 not right | 90.5%495 of 547 answered, 51 “I don't know”, 1 invalid | 28.7%142 of 495 | 50.8%201 “I don't know” of 396 not right | 52.5%287 of 547 answered, 201 “I don't know”, 59 invalid | 67.9%195 of 287 | run 887a new run | run 1298the Hallucination tab's full runs, counted on the kept questions | 64.5%353 of 547 right | 16.8%92 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 80 | 133 | 0.00034 | 0.018 * |
| Raon-1 nano reasoning off · chat Open · 3.1B | 26.2%71 “I don't know” of 271 not right | 86.8%475 of 547 answered, 71 “I don't know”, 1 invalid | 42.1%200 of 475 | Not run | Not run | Not run | run 1153a new run | Not run | 50.3%275 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| Gemini 4 Carbon high · chat Closed · Flagship | 25.0%4 “I don't know” of 16 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.2%12 of 543 | 26.7%4 “I don't know” of 15 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.0%11 of 543 | run 1067a new run | run 1121the Hallucination tab's full runs, counted on the kept questions | 97.1%531 of 547 right | 97.3%532 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 3 | 2 | 1 | 1 |
| Gemini 4 Hydrogen high · chat Closed · Workhorse | 20.0%4 “I don't know” of 20 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.9%16 of 543 | 28.6%4 “I don't know” of 14 not right | 99.3%543 of 547 answered, 4 “I don't know” | 1.8%10 of 543 | run 1109a new run | run 1117the Hallucination tab's full runs, counted on the kept questions | 96.3%527 of 547 right | 97.4%533 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 7 | 1 | 0.07 | 1 |
| MiniCPM5-2B reasoning on · chat Open · 2.5B | 19.7%28 “I don't know” of 142 not right | 94.1%515 of 547 answered, 28 “I don't know”, 4 invalid | 22.1%114 of 515 | Not run | Not run | Not run | run 1052a new run | Not run | 73.3%401 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| GLiNER2.5 Multi Open · 0.3B | 19.6%92 “I don't know” of 470 not right | 83.2%455 of 547 answered, 92 “I don't know” | 83.1%378 of 455 | 6.1%28 “I don't know” of 456 not right | 94.9%519 of 547 answered, 28 “I don't know” | 82.5%428 of 519 | run 722a new run | run 724the Hallucination tab's full runs, counted on the kept questions | 14.1%77 of 547 right | 16.6%91 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 45 | 95 | < 0.0001 | 0.0015 * |
| GPT-6 Astra high · chat Closed · Flagship | 19.0%4 “I don't know” of 21 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.1%17 of 543 | 19.0%4 “I don't know” of 21 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.1%17 of 543 | run 814a new run | run 818the Hallucination tab's full runs, counted on the kept questions | 96.2%526 of 547 right | 96.2%526 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 6 | 6 | 1 | 1 |
| Gemini 4 Argon low · chat Closed · Flagship | 18.5%5 “I don't know” of 27 not right | 99.1%542 of 547 answered, 5 “I don't know” | 4.1%22 of 542 | 23.5%4 “I don't know” of 17 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.4%13 of 543 | run 652a new run | run 679the Hallucination tab's full runs, counted on the kept questions | 95.1%520 of 547 right | 96.9%530 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 15 | 6 | 0.078 | 1 |
| GPT-6 Astra low · chat Closed · Flagship | 18.2%4 “I don't know” of 22 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.3%18 of 543 | 22.2%4 “I don't know” of 18 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.6%14 of 543 | run 647a new run | run 816the Hallucination tab's full runs, counted on the kept questions | 96.0%525 of 547 right | 96.7%529 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 8 | 4 | 0.39 | 1 |
| MiniCPM5-2B reasoning off · chat Open · 2.5B | 16.8%33 “I don't know” of 197 not right | 89.2%488 of 547 answered, 33 “I don't know”, 26 invalid | 33.6%164 of 488 | 22.7%22 “I don't know” of 97 not right | 20.8%114 of 547 answered, 22 “I don't know”, 411 invalid | 65.8%75 of 114 | run 1095a new run | run 1304the Hallucination tab's full runs, counted on the kept questions | 59.2%324 of 547 right | 7.1%39 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 126 | 37 | < 0.0001 | < 0.0001 * |
| GPT-6 Astra medium · chat Closed · Flagship | 16.7%4 “I don't know” of 24 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.7%20 of 543 | 20.0%4 “I don't know” of 20 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.9%16 of 543 | run 813a new run | run 817the Hallucination tab's full runs, counted on the kept questions | 95.6%523 of 547 right | 96.3%527 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 8 | 4 | 0.39 | 1 |
| GPT-6.1 Sol high · chat Closed · Workhorse | 16.7%4 “I don't know” of 24 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.7%20 of 543 | 20.0%4 “I don't know” of 20 not right | 99.3%543 of 547 answered, 4 “I don't know” | 2.9%16 of 543 | run 758a new run | run 802the Hallucination tab's full runs, counted on the kept questions | 95.6%523 of 547 right | 96.3%527 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 9 | 5 | 0.42 | 1 |
| GPT-6.1 Terra high · chat Closed · Workhorse | 16.7%4 “I don't know” of 24 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.7%20 of 543 | 20.8%5 “I don't know” of 24 not right | 99.1%542 of 547 answered, 5 “I don't know” | 3.5%19 of 542 | run 766a new run | run 807the Hallucination tab's full runs, counted on the kept questions | 95.6%523 of 547 right | 95.6%523 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 8 | 7 | 1 | 1 |
| Granite-4.2-3B reasoning on · chat Open · 3.0B | 16.1%23 “I don't know” of 143 not right | 95.8%524 of 547 answered, 23 “I don't know” | 22.9%120 of 524 | Not run | Not run | Not run | run 1110a new run | Not run | 73.9%404 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| LFM2.5-2.6B default reasoning · chat Open · 2.7B | 15.7%21 “I don't know” of 134 not right | 95.6%523 of 547 answered, 21 “I don't know”, 3 invalid | 21.6%113 of 523 | 15.8%43 “I don't know” of 272 not right | 90.7%496 of 547 answered, 43 “I don't know”, 8 invalid | 46.2%229 of 496 | run 1035a new run | run 1038the Hallucination tab's full runs, counted on the kept questions | 75.0%410 of 547 right | 48.8%267 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 44 | 160 | < 0.0001 | < 0.0001 * |
| LFM3-1.7B reasoning on · chat Open · 1.7B | 15.6%26 “I don't know” of 167 not right | 95.2%521 of 547 answered, 26 “I don't know” | 27.1%141 of 521 | 33.5%106 “I don't know” of 316 not right | 80.6%441 of 547 answered, 106 “I don't know” | 47.6%210 of 441 | run 1062a new run | run 1090the Hallucination tab's full runs, counted on the kept questions | 69.5%380 of 547 right | 42.2%231 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 68 | 137 | < 0.0001 | < 0.0001 * |
| GPT-6.1 Terra low · chat Closed · Workhorse | 15.4%4 “I don't know” of 26 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.1%22 of 543 | 14.8%4 “I don't know” of 27 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.2%23 of 543 | run 751a new run | run 801the Hallucination tab's full runs, counted on the kept questions | 95.2%521 of 547 right | 95.1%520 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 11 | 12 | 1 | 1 |
| GPT-6 Astra low · chat · 1 per request Closed · Flagship | 14.8%4 “I don't know” of 27 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.2%23 of 543 | 17.4%4 “I don't know” of 23 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.5%19 of 543 | run 815a new run | run 819the Hallucination tab's full runs, counted on the kept questions | 95.1%520 of 547 right | 95.8%524 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 10 | 6 | 0.45 | 1 |
| GPT-6.1 Sol low · chat Closed · Workhorse | 14.8%4 “I don't know” of 27 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.2%23 of 543 | 18.2%4 “I don't know” of 22 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.3%18 of 543 | run 744a new run | run 799the Hallucination tab's full runs, counted on the kept questions | 95.1%520 of 547 right | 96.0%525 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 9 | 4 | 0.27 | 1 |
| LFM2.5-24B-A2B default reasoning · chat Open · 24B MoE · 2.5B active | 13.2%14 “I don't know” of 106 not right | 97.4%533 of 547 answered, 14 “I don't know” | 17.3%92 of 533 | Not run | Not run | Not run | run 1070a new run | Not run | 80.6%441 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse | 12.9%4 “I don't know” of 31 not right | 99.3%543 of 547 answered, 4 “I don't know” | 5.0%27 of 543 | 17.4%4 “I don't know” of 23 not right | 99.3%543 of 547 answered, 4 “I don't know” | 3.5%19 of 543 | run 769a new run | run 810the Hallucination tab's full runs, counted on the kept questions | 94.3%516 of 547 right | 95.8%524 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 9 | 1 | 0.021 | 1 |
| LFM3-1.7B reasoning off · chat Open · 1.7B | 11.6%18 “I don't know” of 155 not right | 96.7%529 of 547 answered, 18 “I don't know” | 25.9%137 of 529 | 55.4%215 “I don't know” of 388 not right | 56.9%311 of 547 answered, 215 “I don't know”, 21 invalid | 55.6%173 of 311 | run 1225a new run | run 1234the Hallucination tab's full runs, counted on the kept questions | 71.7%392 of 547 right | 25.2%138 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 79 | 115 | 0.012 | 0.57 |
| Gemma 4 E2B default reasoning · chat Open · 5.1B | 11.5%17 “I don't know” of 148 not right | 96.5%528 of 547 answered, 17 “I don't know”, 2 invalid | 24.8%131 of 528 | Not run | Not run | Not run | run 893a new run | Not run | 72.6%397 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| Gemma 4 E2B reasoning on · chat Open · 5.1B | 11.3%17 “I don't know” of 150 not right | 96.3%527 of 547 answered, 17 “I don't know”, 3 invalid | 25.2%133 of 527 | Not run | Not run | Not run | run 1350a new run | Not run | 72.0%394 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| Qwen3.5-2B reasoning off · chat Open · 2.0B | 10.3%18 “I don't know” of 174 not right | 96.3%527 of 547 answered, 18 “I don't know”, 2 invalid | 29.6%156 of 527 | 16.5%55 “I don't know” of 334 not right | 83.4%456 of 547 answered, 55 “I don't know”, 36 invalid | 61.2%279 of 456 | run 1216a new run | run 1219the Hallucination tab's full runs, counted on the kept questions | 67.8%371 of 547 right | 32.4%177 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 50 | 173 | < 0.0001 | < 0.0001 * |
| Gemini 3.8 Flash high · chat Closed · Workhorse | 10.0%2 “I don't know” of 20 not right | 99.6%545 of 547 answered, 2 “I don't know” | 3.3%18 of 545 | 13.3%2 “I don't know” of 15 not right | 99.6%545 of 547 answered, 2 “I don't know” | 2.4%13 of 545 | run 657a new run | run 681the Hallucination tab's full runs, counted on the kept questions | 96.3%527 of 547 right | 97.3%532 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 8 | 3 | 0.23 | 1 |
| Qwen3.5-2B reasoning on · chat Open · 2.0B | 9.9%14 “I don't know” of 142 not right | 97.1%531 of 547 answered, 14 “I don't know”, 2 invalid | 24.1%128 of 531 | Not run | Not run | Not run | run 1344a new run | Not run | 73.7%403 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 9.8%10 “I don't know” of 102 not right | 97.8%535 of 547 answered, 10 “I don't know”, 2 invalid | 17.2%92 of 535 | 20.7%25 “I don't know” of 121 not right | 95.4%522 of 547 answered, 25 “I don't know” | 18.4%96 of 522 | run 890a new run | run 1049the Hallucination tab's full runs, counted on the kept questions | 81.0%443 of 547 right | 77.9%426 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 44 | 48 | 0.75 | 1 |
| Claude Haiku 5.5 max · chat Closed · Efficient | 7.8%4 “I don't know” of 51 not right | 96.3%527 of 547 answered, 4 “I don't know”, 16 invalid | 8.9%47 of 527 | 17.1%6 “I don't know” of 35 not right | 96.0%525 of 547 answered, 6 “I don't know”, 16 invalid | 5.5%29 of 525 | run 951a new run | run 950the Hallucination tab's full runs, counted on the kept questions | 87.8%480 of 547 right | 90.7%496 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 25 | 7 | 0.0021 | 0.11 |
| Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse | 7.7%2 “I don't know” of 26 not right | 99.6%545 of 547 answered, 2 “I don't know” | 4.4%24 of 545 | 14.3%3 “I don't know” of 21 not right | 99.5%544 of 547 answered, 3 “I don't know” | 3.3%18 of 544 | run 669a new run | run 691the Hallucination tab's full runs, counted on the kept questions | 95.2%521 of 547 right | 96.2%526 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 12 | 6 | 0.24 | 1 |
| Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active | 7.5%4 “I don't know” of 53 not right | 99.3%543 of 547 answered, 4 “I don't know” | 9.0%49 of 543 | 7.1%3 “I don't know” of 42 not right | 99.5%544 of 547 answered, 3 “I don't know” | 7.2%39 of 544 | run 646a new run | run 653the Hallucination tab's full runs, counted on the kept questions | 90.3%494 of 547 right | 92.3%505 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 25 | 15 | 0.15 | 1 |
| Claude Fable 5.1 high · chat Closed · Flagship | 7.4%2 “I don't know” of 27 not right | 99.6%545 of 547 answered, 2 “I don't know” | 4.6%25 of 545 | 14.8%4 “I don't know” of 27 not right | 99.3%543 of 547 answered, 4 “I don't know” | 4.2%23 of 543 | run 655a new run | run 695the Hallucination tab's full runs, counted on the kept questions | 95.1%520 of 547 right | 95.1%520 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 11 | 9 | 0.82 | 1 |
| Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active | 7.3%3 “I don't know” of 41 not right | 99.5%544 of 547 answered, 3 “I don't know” | 7.0%38 of 544 | 7.7%3 “I don't know” of 39 not right | 99.5%544 of 547 answered, 3 “I don't know” | 6.6%36 of 544 | run 651a new run | run 686the Hallucination tab's full runs, counted on the kept questions | 92.5%506 of 547 right | 92.9%508 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 16 | 14 | 0.86 | 1 |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active | 7.1%3 “I don't know” of 42 not right | 99.5%544 of 547 answered, 3 “I don't know” | 7.2%39 of 544 | 2.6%1 “I don't know” of 39 not right | 99.8%546 of 547 answered, 1 “I don't know” | 7.0%38 of 546 | run 809a new run | run 811the Hallucination tab's full runs, counted on the kept questions | 92.3%505 of 547 right | 92.9%508 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 19 | 18 | 1 | 1 |
| GPT-6 Luna low · chat Closed · Efficient | 6.8%5 “I don't know” of 74 not right | 99.1%542 of 547 answered, 5 “I don't know” | 12.7%69 of 542 | 3.0%2 “I don't know” of 67 not right | 99.6%545 of 547 answered, 2 “I don't know” | 11.9%65 of 545 | run 735a new run | run 796the Hallucination tab's full runs, counted on the kept questions | 86.5%473 of 547 right | 87.8%480 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 31 | 27 | 0.69 | 1 |
| Claude Opus 5.5 high · chat Closed · Flagship | 6.5%2 “I don't know” of 31 not right | 99.6%545 of 547 answered, 2 “I don't know” | 5.3%29 of 545 | 11.5%3 “I don't know” of 26 not right | 99.5%544 of 547 answered, 3 “I don't know” | 4.2%23 of 544 | run 665a new run | run 773the Hallucination tab's full runs, counted on the kept questions | 94.3%516 of 547 right | 95.2%521 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 13 | 7 | 0.26 | 1 |
| Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active | 6.2%3 “I don't know” of 48 not right | 99.5%544 of 547 answered, 3 “I don't know” | 8.3%45 of 544 | 8.5%4 “I don't know” of 47 not right | 99.3%543 of 547 answered, 4 “I don't know” | 7.9%43 of 543 | run 873a new run | run 872the Hallucination tab's full runs, counted on the kept questions | 91.2%499 of 547 right | 91.4%500 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 18 | 16 | 0.86 | 1 |
| GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active | 5.6%4 “I don't know” of 71 not right | 99.3%543 of 547 answered, 4 “I don't know” | 12.3%67 of 543 | 3.5%3 “I don't know” of 85 not right | 99.5%544 of 547 answered, 3 “I don't know” | 15.1%82 of 544 | run 821a new run | run 822the Hallucination tab's full runs, counted on the kept questions | 87.0%476 of 547 right | 84.5%462 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 26 | 41 | 0.086 | 1 |
| Qwen3.8 Flash no reasoning · chat Closed · Efficient | 5.6%4 “I don't know” of 71 not right | 99.3%543 of 547 answered, 4 “I don't know” | 12.3%67 of 543 | 5.2%3 “I don't know” of 58 not right | 99.5%544 of 547 answered, 3 “I don't know” | 10.1%55 of 544 | run 988a new run | run 987the Hallucination tab's full runs, counted on the kept questions | 87.0%476 of 547 right | 89.4%489 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 31 | 19 | 0.12 | 1 |
| Gemma 4 12B Unified default reasoning · chat Open · 12B | 5.5%4 “I don't know” of 73 not right | 98.9%541 of 547 answered, 4 “I don't know”, 2 invalid | 12.8%69 of 541 | Not run | Not run | Not run | run 896a new run | Not run | 86.3%472 of 547 right | Not run | Not measureda plain answer line, no confidence | Not run | Not run | Not run | Not run | Not run |
| DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active | 5.4%4 “I don't know” of 74 not right | 99.3%543 of 547 answered, 4 “I don't know” | 12.9%70 of 543 | 0.0%0 “I don't know” of 57 not right | 100.0%547 of 547 answered, 0 “I don't know” | 10.4%57 of 547 | run 805a new run | run 806the Hallucination tab's full runs, counted on the kept questions | 86.5%473 of 547 right | 89.6%490 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 33 | 20 | 0.098 | 1 |
| Muse Ember max · chat Closed · Efficient | 5.3%3 “I don't know” of 57 not right | 99.5%544 of 547 answered, 3 “I don't know” | 9.9%54 of 544 | 7.8%4 “I don't know” of 51 not right | 99.3%543 of 547 answered, 4 “I don't know” | 8.7%47 of 543 | run 943a new run | run 940the Hallucination tab's full runs, counted on the kept questions | 89.6%490 of 547 right | 90.7%496 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 25 | 18 | 0.36 | 1 |
| Claude Sonnet 5.5 high · chat Closed · Workhorse | 4.8%3 “I don't know” of 62 not right | 99.5%544 of 547 answered, 3 “I don't know” | 10.8%59 of 544 | 3.6%2 “I don't know” of 55 not right | 99.6%545 of 547 answered, 2 “I don't know” | 9.7%53 of 545 | run 668a new run | run 800the Hallucination tab's full runs, counted on the kept questions | 88.7%485 of 547 right | 89.9%492 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 31 | 25 | 0.5 | 1 |
| Claude Opus 5.5 low · chat Closed · Flagship | 4.7%2 “I don't know” of 43 not right | 99.6%545 of 547 answered, 2 “I don't know” | 7.5%41 of 545 | 6.5%2 “I don't know” of 31 not right | 99.6%545 of 547 answered, 2 “I don't know” | 5.3%29 of 545 | run 649a new run | run 683the Hallucination tab's full runs, counted on the kept questions | 92.1%504 of 547 right | 94.3%516 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 25 | 13 | 0.073 | 1 |
| Claude Sonnet 5.5 low · chat Closed · Workhorse | 4.4%4 “I don't know” of 90 not right | 99.3%543 of 547 answered, 4 “I don't know” | 15.8%86 of 543 | 5.7%5 “I don't know” of 87 not right | 99.1%542 of 547 answered, 5 “I don't know” | 15.1%82 of 542 | run 654a new run | run 690the Hallucination tab's full runs, counted on the kept questions | 83.5%457 of 547 right | 84.1%460 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 36 | 32 | 0.72 | 1 |
| Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B | 3.7%3 “I don't know” of 81 not right | 99.5%544 of 547 answered, 3 “I don't know” | 14.3%78 of 544 | 5.3%4 “I don't know” of 76 not right | 99.3%543 of 547 answered, 4 “I don't know” | 13.3%72 of 543 | run 673a new run | run 674the Hallucination tab's full runs, counted on the kept questions | 85.2%466 of 547 right | 86.1%471 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 33 | 27 | 0.52 | 1 |
| Gemma 4 31B reasoning off · chat Open · 31B | 3.6%2 “I don't know” of 55 not right | 99.6%545 of 547 answered, 2 “I don't know” | 9.7%53 of 545 | 4.5%2 “I don't know” of 44 not right | 99.6%545 of 547 answered, 2 “I don't know” | 7.7%42 of 545 | run 775a new run | run 776the Hallucination tab's full runs, counted on the kept questions | 89.9%492 of 547 right | 92.0%503 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 30 | 19 | 0.15 | 1 |
| Qwen3.5-4B reasoning off · chat Open · 4.0B | 3.6%4 “I don't know” of 110 not right | 99.3%543 of 547 answered, 4 “I don't know” | 19.5%106 of 543 | 3.4%5 “I don't know” of 145 not right | 98.4%538 of 547 answered, 5 “I don't know”, 4 invalid | 26.0%140 of 538 | run 1144a new run | run 1147the Hallucination tab's full runs, counted on the kept questions | 79.9%437 of 547 right | 72.8%398 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 52 | 86 | 0.0048 | 0.23 |
| Qwen3.8-27B low · chat Open · 27B | 3.6%2 “I don't know” of 55 not right | 99.6%545 of 547 answered, 2 “I don't know” | 9.7%53 of 545 | 2.0%1 “I don't know” of 49 not right | 99.8%546 of 547 answered, 1 “I don't know” | 8.8%48 of 546 | run 671a new run | run 666the Hallucination tab's full runs, counted on the kept questions | 89.9%492 of 547 right | 91.0%498 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 23 | 18 | 0.53 | 1 |
| Gemini 3.8 Flash low · chat Closed · Workhorse | 3.4%1 “I don't know” of 29 not right | 99.8%546 of 547 answered, 1 “I don't know” | 5.1%28 of 546 | 11.1%3 “I don't know” of 27 not right | 99.5%544 of 547 answered, 3 “I don't know” | 4.4%24 of 544 | run 645a new run | run 676the Hallucination tab's full runs, counted on the kept questions | 94.7%518 of 547 right | 95.1%520 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 14 | 10 | 0.54 | 1 |
| GLM-5.3 low · chat Open · Workhorse | 3.4%2 “I don't know” of 58 not right | 99.6%545 of 547 answered, 2 “I don't know” | 10.3%56 of 545 | 4.7%3 “I don't know” of 64 not right | 99.5%544 of 547 answered, 3 “I don't know” | 11.2%61 of 544 | run 650a new run | run 656the Hallucination tab's full runs, counted on the kept questions | 89.4%489 of 547 right | 88.3%483 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 23 | 28 | 0.58 | 1 |
| Qwen3.8-27B vote share · reasoning off · chat Open · 27B | 3.4%3 “I don't know” of 87 not right | 99.5%544 of 547 answered, 3 “I don't know” | 15.4%84 of 544 | 5.1%4 “I don't know” of 79 not right | 99.3%543 of 547 answered, 4 “I don't know” | 13.8%75 of 543 | run 680a new run | run 682the Hallucination tab's full runs, counted on the kept questions | 84.1%460 of 547 right | 85.6%468 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 35 | 26 | 0.31 | 1 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | 3.4%3 “I don't know” of 88 not right | 99.5%544 of 547 answered, 3 “I don't know” | 15.6%85 of 544 | 1.1%1 “I don't know” of 91 not right | 99.6%545 of 547 answered, 1 “I don't know”, 1 invalid | 16.5%90 of 545 | run 865a new run | run 864the Hallucination tab's full runs, counted on the kept questions | 83.9%459 of 547 right | 83.2%455 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 38 | 43 | 0.66 | 1 |
| Qwen3.5-4B reasoning on · chat Open · 4.0B | 2.4%2 “I don't know” of 85 not right | 99.6%545 of 547 answered, 2 “I don't know” | 15.2%83 of 545 | 2.2%2 “I don't know” of 90 not right | 99.1%542 of 547 answered, 2 “I don't know”, 3 invalid | 16.2%88 of 542 | run 1073a new run | run 1150the Hallucination tab's full runs, counted on the kept questions | 84.5%462 of 547 right | 83.0%454 of 547 right | Not measureda plain answer line, no confidence | Not measureda plain answer line, no confidence | 44 | 49 | 0.68 | 1 |
| Erk-14B reasoning off · chat Open · 14B | 2.1%3 “I don't know” of 144 not right | 99.5%544 of 547 answered, 3 “I don't know” | 25.9%141 of 544 | 0.8%1 “I don't know” of 130 not right | 99.8%546 of 547 answered, 1 “I don't know” | 23.6%129 of 546 | run 764a new run | run 765the Hallucination tab's full runs, counted on the kept questions | 73.7%403 of 547 right | 76.2%417 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 68 | 56 | 0.32 | 1 |
| Qwen3-14B reasoning off · chat Open · 15B | 1.8%2 “I don't know” of 110 not right | 99.6%545 of 547 answered, 2 “I don't know” | 19.8%108 of 545 | 0.8%1 “I don't know” of 123 not right | 99.8%546 of 547 answered, 1 “I don't know” | 22.3%122 of 546 | run 771a new run | run 772the Hallucination tab's full runs, counted on the kept questions | 79.9%437 of 547 right | 77.5%424 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 40 | 54 | 0.18 | 1 |
| MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active | 1.7%1 “I don't know” of 59 not right | 99.8%546 of 547 answered, 1 “I don't know” | 10.6%58 of 546 | 4.2%3 “I don't know” of 72 not right | 99.5%544 of 547 answered, 3 “I don't know” | 12.7%69 of 544 | run 880a new run | run 879the Hallucination tab's full runs, counted on the kept questions | 89.2%488 of 547 right | 86.8%475 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 20 | 31 | 0.16 | 1 |
| GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active | 1.6%1 “I don't know” of 62 not right | 99.8%546 of 547 answered, 1 “I don't know” | 11.2%61 of 546 | 3.5%2 “I don't know” of 57 not right | 99.6%545 of 547 answered, 2 “I don't know” | 10.1%55 of 545 | run 824a new run | run 825the Hallucination tab's full runs, counted on the kept questions | 88.7%485 of 547 right | 89.6%490 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 28 | 22 | 0.48 | 1 |
| Claude Opus 5.5 low · chat · 1 per request Closed · Flagship | 0.0%0 “I don't know” of 31 not right | 100.0%547 of 547 answered, 0 “I don't know” | 5.7%31 of 547 | 3.8%1 “I don't know” of 26 not right | 99.8%546 of 547 answered, 1 “I don't know” | 4.6%25 of 546 | run 670a new run | run 803the Hallucination tab's full runs, counted on the kept questions | 94.3%516 of 547 right | 95.2%521 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 13 | 7 | 0.26 | 1 |
| Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse | 0.0%0 “I don't know” of 38 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.9%38 of 547 | 2.7%1 “I don't know” of 37 not right | 99.8%546 of 547 answered, 1 “I don't know” | 6.6%36 of 546 | run 677a new run | run 812the Hallucination tab's full runs, counted on the kept questions | 93.1%509 of 547 right | 93.2%510 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 16 | 14 | 0.86 | 1 |
| Gemma 4 31B reasoning off · chat · 1 per request Open · 31B | 0.0%0 “I don't know” of 59 not right | 100.0%547 of 547 answered, 0 “I don't know” | 10.8%59 of 547 | 1.9%1 “I don't know” of 53 not right | 99.8%546 of 547 answered, 1 “I don't know” | 9.5%52 of 546 | run 786a new run | run 789the Hallucination tab's full runs, counted on the kept questions | 89.2%488 of 547 right | 90.3%494 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 31 | 24 | 0.42 | 1 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | 0.0%0 “I don't know” of 60 not right | 100.0%547 of 547 answered, 0 “I don't know” | 11.0%60 of 547 | 0.0%0 “I don't know” of 62 not right | 100.0%547 of 547 answered, 0 “I don't know” | 11.3%62 of 547 | run 869a new run | run 868the Hallucination tab's full runs, counted on the kept questions | 89.0%487 of 547 right | 88.7%485 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 30 | 32 | 0.9 | 1 |
| Qwen4-Max high · chat Closed · Workhorse | 0.0%0 “I don't know” of 29 not right | 100.0%547 of 547 answered, 0 “I don't know” | 5.3%29 of 547 | 3.4%1 “I don't know” of 29 not right | 99.8%546 of 547 answered, 1 “I don't know” | 5.1%28 of 546 | run 1338a new run | run 1339the Hallucination tab's full runs, counted on the kept questions | 94.7%518 of 547 right | 94.7%518 of 547 right | No probabilitiesa chat model states a confidence | No probabilitiesa chat model states a confidence | 12 | 11 | 1 | 1 |
| Model | English non-hallucinationof answers not right, % “I don't know” · higher is better, read with the answered share | English answerednot “I don't know”: (right + wrong) / 547 | English error rateof its answers: wrong / (right + wrong) · lower is better | Turkish non-hallucinationof answers not right, % the Turkish option · higher is better, read with the answered share | Turkish answerednot the Turkish option: (right + wrong) / 547 | Turkish error rateof its answers: wrong / (right + wrong) · lower is better | English runswith “I don't know”; the first is compared | Turkish runswith the Turkish option; the first is compared | English accuracyhigher is better | Turkish accuracyhigher is better | English mean probabilityof “I don't know” | Turkish mean probabilityof the Turkish option | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 67 decision models |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Efficient | 89.3%109 “I don't know” of 122 not right | 80.1%438 of 547 answered, 109 “I don't know” | 3.0%13 of 438 | 76.2%61 “I don't know” of 80 not right | 88.8%486 of 547 answered, 61 “I don't know” | 3.9%19 of 486 | run 1363a new run | run 1367the Hallucination tab's full runs, counted on the kept questions | 77.7%425 of 547 right | 85.4%467 of 547 right | 26.0% | 17.6% | 6 | 12 | 0.24 | 1 |
| celeris-1.1-decision Closed · Efficient | 74.8%98 “I don't know” of 131 not right | 82.1%449 of 547 answered, 98 “I don't know” | 7.3%33 of 449 | Not run | Not run | Not run | run 884a new run | Not run | 76.1%416 of 547 right | Not run | 13.3% | Not run | Not run | Not run | Not run | Not run |
| celeris-1-decision Closed · Efficient | 71.3%92 “I don't know” of 129 not right | 83.2%455 of 547 answered, 92 “I don't know” | 8.1%37 of 455 | Not run | Not run | Not run | run 548a new run | Not run | 76.4%418 of 547 right | Not run | 12.7% | Not run | Not run | Not run | Not run | Not run |
| d1.2 Closed · Efficient | 61.5%59 “I don't know” of 96 not right | 89.2%488 of 547 answered, 59 “I don't know” | 7.6%37 of 488 | 60.2%53 “I don't know” of 88 not right | 90.3%494 of 547 answered, 53 “I don't know” | 7.1%35 of 494 | run 1361a new run | run 1365the Hallucination tab's full runs, counted on the kept questions | 82.4%451 of 547 right | 83.9%459 of 547 right | 19.4% | 17.1% | 21 | 19 | 0.87 | 1 |
| d1-2B Open · 1.9B | 60.9%151 “I don't know” of 248 not right | 72.4%396 of 547 answered, 151 “I don't know” | 24.5%97 of 396 | 43.8%161 “I don't know” of 368 not right | 70.6%386 of 547 answered, 161 “I don't know” | 53.6%207 of 386 | run 710a new run | run 712the Hallucination tab's full runs, counted on the kept questions | 54.7%299 of 547 right | 32.7%179 of 547 right | 26.9% | 23.7% | 38 | 148 | < 0.0001 | < 0.0001 * |
| d1-ultra Closed · Efficient | 56.7%51 “I don't know” of 90 not rightrange 55.3% to 61.3% | 90.7%496 of 547 answered, 51 “I don't know”range 89.6% to 91.4% | 7.9%39 of 496 | 53.6%30 “I don't know” of 56 not rightrange 44.4% to 59.3% | 94.5%517 of 547 answered, 30 “I don't know”range 94.1% to 95.6% | 5.0%26 of 517 | runs 449, 456, 462, 466, 474new runs | runs 350, 354, 360, 365, 370the Hallucination tab's full runs, counted on the kept questions | 83.5%457 of 547 rightrange 82.6% to 84.5% | 89.8%491 of 547 rightrange 89.2% to 90.1% | 25.3%range 25.3% to 25.6% | 22.8%range 22.4% to 23.0% | 20 | 7 | 0.019 | 0.88 |
| GLiDE 0.4B Open · 0.4B | 53.1%153 “I don't know” of 288 not right | 72.0%394 of 547 answered, 153 “I don't know” | 34.3%135 of 394 | 89.6%475 “I don't know” of 530 not right | 13.2%72 of 547 answered, 475 “I don't know” | 76.4%55 of 72 | run 935a new run | run 937the Hallucination tab's full runs, counted on the kept questions | 47.3%259 of 547 right | 3.1%17 of 547 right | 23.3% | 35.2% | 125 | 45 | < 0.0001 | < 0.0001 * |
| Drex Max v1.1 Closed · Efficient (provisional) | 51.2%43 “I don't know” of 84 not right | 92.1%504 of 547 answered, 43 “I don't know” | 8.1%41 of 504 | 15.0%9 “I don't know” of 60 not right | 98.4%538 of 547 answered, 9 “I don't know” | 9.5%51 of 538 | run 447a new run | run 352the Hallucination tab's full runs, counted on the kept questions | 84.6%463 of 547 right | 89.0%487 of 547 right | 14.3% | 8.4% | 20 | 30 | 0.2 | 1 |
| Mammoth-1 Open · 27B | 50.7%37 “I don't know” of 73 not right | 93.2%510 of 547 answered, 37 “I don't know” | 7.1%36 of 510 | 42.1%24 “I don't know” of 57 not right | 95.6%523 of 547 answered, 24 “I don't know” | 6.3%33 of 523 | run 505a new run | run 386the Hallucination tab's full runs, counted on the kept questions | 86.7%474 of 547 right | 89.6%490 of 547 right | 9.8% | 7.4% | 18 | 15 | 0.73 | 1 |
| GLiDE v1.2 Open · 27B | 45.5%30 “I don't know” of 66 not right | 94.5%517 of 547 answered, 30 “I don't know” | 7.0%36 of 517 | 11.3%6 “I don't know” of 53 not right | 98.9%541 of 547 answered, 6 “I don't know” | 8.7%47 of 541 | run 523a new run | run 369the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.3%494 of 547 right | 12.5% | 8.7% | 16 | 27 | 0.13 | 1 |
| Laya Multilingual Open · 0.3B | 42.6%208 “I don't know” of 488 not right | 62.0%339 of 547 answered, 208 “I don't know” | 82.6%280 of 339 | 25.7%124 “I don't know” of 483 not right | 77.3%423 of 547 answered, 124 “I don't know” | 84.9%359 of 423 | run 734a new run | run 737the Hallucination tab's full runs, counted on the kept questions | 10.8%59 of 547 right | 11.7%64 of 547 right | 34.7% | 21.6% | 51 | 130 | < 0.0001 | < 0.0001 * |
| GLiDE 27B Open · 27B | 37.9%22 “I don't know” of 58 not right | 96.0%525 of 547 answered, 22 “I don't know” | 6.9%36 of 525 | 18.8%9 “I don't know” of 48 not right | 98.4%538 of 547 answered, 9 “I don't know” | 7.2%39 of 538 | run 903a new run | run 905the Hallucination tab's full runs, counted on the kept questions | 89.4%489 of 547 right | 91.2%499 of 547 right | 9.3% | 6.5% | 15 | 18 | 0.73 | 1 |
| OneJev1.6-0.8B Open · 0.8B | 28.4%74 “I don't know” of 261 not right | 86.5%473 of 547 answered, 74 “I don't know” | 39.5%187 of 473 | 30.7%112 “I don't know” of 365 not right | 79.5%435 of 547 answered, 112 “I don't know” | 58.2%253 of 435 | run 1027a new run | run 1031the Hallucination tab's full runs, counted on the kept questions | 52.3%286 of 547 right | 33.3%182 of 547 right | 10.7% | 15.6% | 55 | 121 | < 0.0001 | < 0.0001 * |
| GLiDE 2B Open · 1.9B | 26.3%47 “I don't know” of 179 not right | 91.4%500 of 547 answered, 47 “I don't know” | 26.4%132 of 500 | 24.0%66 “I don't know” of 275 not right | 87.9%481 of 547 answered, 66 “I don't know” | 43.5%209 of 481 | run 919a new run | run 921the Hallucination tab's full runs, counted on the kept questions | 67.3%368 of 547 right | 49.7%272 of 547 right | 12.5% | 15.4% | 48 | 125 | < 0.0001 | < 0.0001 * |
| Elephant-1 Open · 9.0B | 25.6%23 “I don't know” of 90 not right | 95.8%524 of 547 answered, 23 “I don't know” | 12.8%67 of 524 | 15.5%15 “I don't know” of 97 not right | 97.3%532 of 547 answered, 15 “I don't know” | 15.4%82 of 532 | run 501a new run | run 382the Hallucination tab's full runs, counted on the kept questions | 83.5%457 of 547 right | 82.3%450 of 547 right | 6.8% | 5.7% | 31 | 46 | 0.11 | 1 |
| SubQ Snap 1.1 Closed · Efficient | 25.0%21 “I don't know” of 84 not right | 96.2%526 of 547 answered, 21 “I don't know” | 12.0%63 of 526 | 27.1%19 “I don't know” of 70 not right | 96.5%528 of 547 answered, 19 “I don't know” | 9.7%51 of 528 | run 448a new run | run 392the Hallucination tab's full runs, counted on the kept questions | 84.6%463 of 547 right | 87.2%477 of 547 right | 11.7% | 8.4% | 39 | 27 | 0.18 | 1 |
| KnowLine-2B-Gen1 Open · 2.0B | 22.0%38 “I don't know” of 173 not right | 93.1%509 of 547 answered, 38 “I don't know” | 26.5%135 of 509 | Not run | Not run | Not run | run 1059a new run | Not run | 68.4%374 of 547 right | Not run | 7.3% | Not run | Not run | Not run | Not run | Not run |
| GLiDE 0.8B Open · 0.8B | 21.7%49 “I don't know” of 226 not right | 91.0%498 of 547 answered, 49 “I don't know” | 35.5%177 of 498 | 24.3%74 “I don't know” of 304 not right | 86.5%473 of 547 answered, 74 “I don't know” | 48.6%230 of 473 | run 927a new run | run 929the Hallucination tab's full runs, counted on the kept questions | 58.7%321 of 547 right | 44.4%243 of 547 right | 14.8% | 15.4% | 68 | 121 | 0.00014 | 0.0079 * |
| GLiClass v4 Edge Open · 0.4B | 21.0%52 “I don't know” of 248 not right | 90.5%495 of 547 answered, 52 “I don't know” | 39.6%196 of 495 | 14.1%58 “I don't know” of 410 not right | 89.4%489 of 547 answered, 58 “I don't know” | 72.0%352 of 489 | run 1044a new run | run 1048the Hallucination tab's full runs, counted on the kept questions | 54.7%299 of 547 right | 25.0%137 of 547 right | 11.8% | 12.0% | 48 | 204 | < 0.0001 | < 0.0001 * |
| Laya Open · 0.4B | 18.4%83 “I don't know” of 450 not right | 84.8%464 of 547 answered, 83 “I don't know” | 79.1%367 of 464 | 12.0%53 “I don't know” of 443 not right | 90.3%494 of 547 answered, 53 “I don't know” | 78.9%390 of 494 | run 728a new run | run 730the Hallucination tab's full runs, counted on the kept questions | 17.7%97 of 547 right | 19.0%104 of 547 right | 14.2% | 11.1% | 97 | 120 | 0.14 | 1 |
| Open-Jev-27B-v1.1 Open · 27B | 17.4%16 “I don't know” of 92 not right | 97.1%531 of 547 answered, 16 “I don't know” | 14.3%76 of 531 | 7.1%6 “I don't know” of 84 not right | 98.9%541 of 547 answered, 6 “I don't know” | 14.4%78 of 541 | run 763a new run | run 768the Hallucination tab's full runs, counted on the kept questions | 83.2%455 of 547 right | 84.6%463 of 547 right | 15.2% | 13.2% | 26 | 28 | 0.89 | 1 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | 16.9%32 “I don't know” of 189 not right | 94.1%515 of 547 answered, 32 “I don't know” | 30.5%157 of 515 | 12.3%33 “I don't know” of 268 not right | 94.0%514 of 547 answered, 33 “I don't know” | 45.7%235 of 514 | run 975a new run | run 979the Hallucination tab's full runs, counted on the kept questions | 65.4%358 of 547 right | 51.0%279 of 547 right | 6.7% | 7.6% | 47 | 125 | < 0.0001 | < 0.0001 * |
| GLiDE 9B Open · 7.9B | 16.7%14 “I don't know” of 84 not right | 97.4%533 of 547 answered, 14 “I don't know” | 13.1%70 of 533 | 9.8%9 “I don't know” of 92 not right | 98.4%538 of 547 answered, 9 “I don't know” | 15.4%83 of 538 | run 911a new run | run 913the Hallucination tab's full runs, counted on the kept questions | 84.6%463 of 547 right | 83.2%455 of 547 right | 7.8% | 6.9% | 25 | 38 | 0.13 | 1 |
| GLiNER2.5-multi-Decide Open · 0.3B | 16.2%70 “I don't know” of 433 not right | 87.2%477 of 547 answered, 70 “I don't know” | 76.1%363 of 477 | 5.5%24 “I don't know” of 436 not right | 95.6%523 of 547 answered, 24 “I don't know” | 78.8%412 of 523 | run 716a new run | run 718the Hallucination tab's full runs, counted on the kept questions | 20.8%114 of 547 right | 20.3%111 of 547 right | 13.2% | 10.4% | 46 | 95 | < 0.0001 | 0.0026 * |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 15.2%10 “I don't know” of 66 not right | 98.2%537 of 547 answered, 10 “I don't know” | 10.4%56 of 537 | 5.6%3 “I don't know” of 54 not right | 99.5%544 of 547 answered, 3 “I don't know” | 9.4%51 of 544 | run 644a new run | run 642the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.1%493 of 547 right | 4.6% | 2.2% | 26 | 21 | 0.56 | 1 |
| Winnow-12B Open · 12B | 15.0%15 “I don't know” of 100 not right | 97.3%532 of 547 answered, 15 “I don't know” | 16.0%85 of 532 | 8.6%8 “I don't know” of 93 not right | 98.5%539 of 547 answered, 8 “I don't know” | 15.8%85 of 539 | run 748a new run | run 750the Hallucination tab's full runs, counted on the kept questions | 81.7%447 of 547 right | 83.0%454 of 547 right | 3.1% | 1.8% | 32 | 32 | 1 | 1 |
| Mercury Decide Closed · Efficient | 13.4%11 “I don't know” of 82 not rightrange 11.1% to 13.4% | 98.0%536 of 547 answered, 11 “I don't know”range 98.0% to 98.4% | 13.2%71 of 536 | 10.1%8 “I don't know” of 79 not rightrange 10.1% to 10.5% | 98.5%539 of 547 answered, 8 “I don't know”the same in every run | 13.2%71 of 539 | runs 450, 460, 469, 478, 484new runs | runs 489, 491, 493, 495, 497new runs on the kept questions | 85.0%465 of 547 rightrange 85.0% to 85.4% | 85.6%468 of 547 rightrange 85.6% to 86.1% | 3.2%range 3.1% to 3.2% | 1.9%range 1.9% to 2.0% | 29 | 29 | 1 | 1 |
| Cygnet Open · 12B | 11.2%10 “I don't know” of 89 not right | 98.2%537 of 547 answered, 10 “I don't know” | 14.7%79 of 537 | 5.8%5 “I don't know” of 86 not right | 99.1%542 of 547 answered, 5 “I don't know” | 14.9%81 of 542 | run 755a new run | run 757the Hallucination tab's full runs, counted on the kept questions | 83.7%458 of 547 right | 84.3%461 of 547 right | 3.5% | 2.8% | 28 | 30 | 0.9 | 1 |
| KnowLine-0.8B-Gen1 Open · 0.8B | 10.7%26 “I don't know” of 243 not right | 95.2%521 of 547 answered, 26 “I don't know” | 41.7%217 of 521 | 4.4%15 “I don't know” of 340 not right | 97.3%532 of 547 answered, 15 “I don't know” | 61.1%325 of 532 | run 1003a new run | run 1007the Hallucination tab's full runs, counted on the kept questions | 55.6%304 of 547 right | 37.8%207 of 547 right | 5.8% | 6.0% | 49 | 157 | < 0.0001 | < 0.0001 * |
| GPT-6 Luna Decisions API Closed · Efficient | 9.4%10 “I don't know” of 106 not right | 98.2%537 of 547 answered, 10 “I don't know” | 17.9%96 of 537 | 7.1%6 “I don't know” of 84 not right | 98.9%541 of 547 answered, 6 “I don't know” | 14.4%78 of 541 | run 444a new run | run 453a new run on the kept questions | 80.6%441 of 547 right | 84.6%463 of 547 right | 3.0% | 2.7% | 47 | 29 | 0.05 | 1 |
| Gemini Decision 1 20261111 Closed · Efficient | 9.1%5 “I don't know” of 55 not right | 99.1%542 of 547 answered, 5 “I don't know” | 9.2%50 of 542 | 10.9%5 “I don't know” of 46 not right | 99.1%542 of 547 answered, 5 “I don't know” | 7.6%41 of 542 | run 1019a new run | run 1023the Hallucination tab's full runs, counted on the kept questions | 89.9%492 of 547 right | 91.6%501 of 547 right | 4.6% | 4.2% | 20 | 11 | 0.15 | 1 |
| Perplexity Decider 27B v1.3 Open · 27B | 8.8%5 “I don't know” of 57 not right | 99.1%542 of 547 answered, 5 “I don't know” | 9.6%52 of 542 | 0.0%0 “I don't know” of 52 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.5%52 of 547 | run 1139a new run | run 1141the Hallucination tab's full runs, counted on the kept questions | 89.6%490 of 547 right | 90.5%495 of 547 right | 3.8% | 1.7% | 24 | 24 | 1 | 1 |
| Turion Jev 31B Open · 31B | 8.5%4 “I don't know” of 47 not right | 99.3%543 of 547 answered, 4 “I don't know” | 7.9%43 of 543 | 5.7%2 “I don't know” of 35 not right | 99.6%545 of 547 answered, 2 “I don't know” | 6.1%33 of 545 | run 1077a new run | run 1285the Hallucination tab's full runs, counted on the kept questions | 91.4%500 of 547 right | 93.6%512 of 547 right | 2.3% | 1.6% | 21 | 11 | 0.11 | 1 |
| d3-ultra Open · 26B | 8.2%5 “I don't know” of 61 not right | 99.1%542 of 547 answered, 5 “I don't know” | 10.3%56 of 542 | 2.1%1 “I don't know” of 47 not right | 99.8%546 of 547 answered, 1 “I don't know” | 8.4%46 of 546 | run 795a new run | run 363the Hallucination tab's full runs, counted on the kept questions | 88.8%486 of 547 right | 91.4%500 of 547 right | 2.8% | 1.0% | 33 | 23 | 0.23 | 1 |
| vera-medium-omni Open · 4.2B | 7.4%7 “I don't know” of 94 not right | 98.7%540 of 547 answered, 7 “I don't know” | 16.1%87 of 540 | 4.2%5 “I don't know” of 118 not right | 99.1%542 of 547 answered, 5 “I don't know” | 20.8%113 of 542 | run 1269a new run | run 1271the Hallucination tab's full runs, counted on the kept questions | 82.8%453 of 547 right | 78.4%429 of 547 right | 5.7% | 5.3% | 26 | 52 | 0.0043 | 0.23 |
| Perplexity Decider Flash v1.3 Open · 4.0B | 7.2%7 “I don't know” of 97 not right | 98.7%540 of 547 answered, 7 “I don't know” | 16.7%90 of 540 | 7.6%9 “I don't know” of 119 not right | 98.4%538 of 547 answered, 9 “I don't know” | 20.4%110 of 538 | run 1128a new run | run 1130the Hallucination tab's full runs, counted on the kept questions | 82.3%450 of 547 right | 78.2%428 of 547 right | 5.1% | 5.0% | 37 | 57 | 0.049 | 1 |
| REAX 1.3 Open · 57B | 6.7%3 “I don't know” of 45 not right | 99.5%544 of 547 answered, 3 “I don't know” | 7.7%42 of 544 | 2.8%1 “I don't know” of 36 not right | 99.8%546 of 547 answered, 1 “I don't know” | 6.4%35 of 546 | run 831a new run | run 833the Hallucination tab's full runs, counted on the kept questions | 91.8%502 of 547 right | 93.4%511 of 547 right | 1.9% | 0.7% | 20 | 13 | 0.3 | 1 |
| Clef-Flash Open · 9.0B | 5.3%5 “I don't know” of 95 not right | 99.1%542 of 547 answered, 5 “I don't know” | 16.6%90 of 542 | 9.4%12 “I don't know” of 127 not right | 97.8%535 of 547 answered, 12 “I don't know” | 21.5%115 of 535 | run 660a new run | run 664the Hallucination tab's full runs, counted on the kept questions | 82.6%452 of 547 right | 76.8%420 of 547 right | 3.1% | 4.8% | 27 | 52 | 0.0066 | 0.33 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | 5.2%12 “I don't know” of 229 not right | 97.8%535 of 547 answered, 12 “I don't know” | 40.6%217 of 535 | 6.4%22 “I don't know” of 342 not right | 96.0%525 of 547 answered, 22 “I don't know” | 61.0%320 of 525 | run 1013a new run | run 1017the Hallucination tab's full runs, counted on the kept questions | 58.1%318 of 547 right | 37.5%205 of 547 right | 6.6% | 7.3% | 48 | 151 | < 0.0001 | < 0.0001 * |
| Perplexity Decider Flash v1.2 Open · 4.0B | 4.8%5 “I don't know” of 105 not right | 99.1%542 of 547 answered, 5 “I don't know” | 18.5%100 of 542 | 4.3%6 “I don't know” of 139 not right | 98.9%541 of 547 answered, 6 “I don't know” | 24.6%133 of 541 | run 535a new run | run 537a new run on the kept questions | 80.8%442 of 547 right | 74.6%408 of 547 right | 5.5% | 6.3% | 36 | 69 | 0.0017 | 0.091 |
| Perplexity Decider 27B v1.2 API Open · 27B | 4.5%3 “I don't know” of 66 not right | 99.5%544 of 547 answered, 3 “I don't know” | 11.6%63 of 544 | 0.0%0 “I don't know” of 51 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.3%51 of 547 | run 643a new run | run 641the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.7%496 of 547 right | 3.1% | 1.1% | 27 | 15 | 0.088 | 1 |
| Logits Sigma-1 Duo Open · 57B | 4.3%2 “I don't know” of 46 not right | 99.6%545 of 547 answered, 2 “I don't know” | 8.1%44 of 545 | 0.0%0 “I don't know” of 36 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.6%36 of 547 | run 1231a new run | run 1233the Hallucination tab's full runs, counted on the kept questions | 91.6%501 of 547 right | 93.4%511 of 547 right | 1.8% | 0.8% | 25 | 17 | 0.28 | 1 |
| Jev 1.14 Closed · Efficient | 4.0%3 “I don't know” of 75 not rightrange 4.0% to 5.2% | 99.5%544 of 547 answered, 3 “I don't know”range 99.3% to 99.5% | 13.2%72 of 544 | 4.2%3 “I don't know” of 72 not rightrange 4.0% to 4.5% | 99.5%544 of 547 answered, 3 “I don't know”the same in every run | 12.7%69 of 544 | runs 446, 454, 457, 461, 464new runs | runs 470, 473, 476, 480, 482new runs on the kept questions | 86.3%472 of 547 rightrange 85.9% to 86.7% | 86.8%475 of 547 rightrange 86.3% to 87.8% | 1.6%range 1.6% to 1.6% | 1.7%range 1.6% to 1.7% | 20 | 17 | 0.74 | 1 |
| OneJev-27B Open · 27B | 4.0%3 “I don't know” of 75 not right | 99.5%544 of 547 answered, 3 “I don't know” | 13.2%72 of 544 | 1.6%1 “I don't know” of 64 not right | 99.8%546 of 547 answered, 1 “I don't know” | 11.5%63 of 546 | run 780a new run | run 782the Hallucination tab's full runs, counted on the kept questions | 86.3%472 of 547 right | 88.3%483 of 547 right | 2.3% | 1.3% | 34 | 25 | 0.3 | 1 |
| Microsoft-Decision-1 Closed · Efficient | 3.8%3 “I don't know” of 80 not rightrange 3.8% to 4.0% | 99.5%544 of 547 answered, 3 “I don't know”the same in every run | 14.2%77 of 544 | 2.2%2 “I don't know” of 90 not rightrange 2.2% to 4.3% | 99.6%545 of 547 answered, 2 “I don't know”range 99.3% to 99.6% | 16.1%88 of 545 | runs 451, 459, 467, 477, 483new runs | runs 487, 490, 492, 494, 496new runs on the kept questions | 85.4%467 of 547 rightrange 85.4% to 86.3% | 83.5%457 of 547 rightrange 82.8% to 83.5% | 3.6%range 3.6% to 3.6% | 2.3%range 2.3% to 2.3% | 24 | 35 | 0.19 | 1 |
| Chinese Decision Pointer 4B Open · 4.0B | 3.7%4 “I don't know” of 108 not right | 99.3%543 of 547 answered, 4 “I don't know” | 19.2%104 of 543 | 3.0%4 “I don't know” of 133 not right | 99.3%543 of 547 answered, 4 “I don't know” | 23.8%129 of 543 | run 1211a new run | run 1213the Hallucination tab's full runs, counted on the kept questions | 80.3%439 of 547 right | 75.7%414 of 547 right | 2.1% | 1.9% | 41 | 66 | 0.02 | 0.9 |
| Drex Max Open · 27B | 3.2%2 “I don't know” of 62 not right | 99.6%545 of 547 answered, 2 “I don't know” | 11.0%60 of 545 | 3.6%2 “I don't know” of 55 not right | 99.6%545 of 547 answered, 2 “I don't know” | 9.7%53 of 545 | run 798a new run | run 367the Hallucination tab's full runs, counted on the kept questions | 88.7%485 of 547 right | 89.9%492 of 547 right | 1.7% | 0.6% | 28 | 21 | 0.39 | 1 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 3.0%2 “I don't know” of 66 not right | 99.6%545 of 547 answered, 2 “I don't know” | 11.7%64 of 545 | 0.0%0 “I don't know” of 50 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.1%50 of 547 | run 529a new run | run 372the Hallucination tab's full runs, counted on the kept questions | 87.9%481 of 547 right | 90.9%497 of 547 right | 2.8% | 0.9% | 29 | 15 | 0.049 | 1 |
| d1 Closed · Efficient | 2.9%2 “I don't know” of 69 not rightrange 2.6% to 2.9% | 99.6%545 of 547 answered, 2 “I don't know”the same in every run | 12.3%67 of 545 | 1.1%1 “I don't know” of 90 not rightrange 0.0% to 1.1% | 99.8%546 of 547 answered, 1 “I don't know”range 99.8% to 100.0% | 16.3%89 of 546 | runs 445, 455, 458, 463, 465new runs | runs 472, 475, 479, 481, 485new runs on the kept questions | 87.4%478 of 547 rightrange 85.9% to 87.4% | 83.5%457 of 547 rightrange 83.0% to 85.0% | 0.9%range 0.9% to 1.0% | 1.3%range 1.1% to 1.3% | 21 | 43 | 0.0081 | 0.4 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | 2.6%2 “I don't know” of 76 not right | 99.6%545 of 547 answered, 2 “I don't know” | 13.6%74 of 545 | 1.0%1 “I don't know” of 96 not right | 99.8%546 of 547 answered, 1 “I don't know” | 17.4%95 of 546 | run 1295a new run | run 1297the Hallucination tab's full runs, counted on the kept questions | 86.1%471 of 547 right | 82.4%451 of 547 right | 1.8% | 1.4% | 26 | 47 | 0.019 | 0.88 |
| Cerno 9B v1.2 Open · 9.0B | 2.5%2 “I don't know” of 79 not right | 99.6%545 of 547 answered, 2 “I don't know” | 14.1%77 of 545 | 0.0%0 “I don't know” of 91 not right | 100.0%547 of 547 answered, 0 “I don't know” | 16.6%91 of 547 | run 1087a new run | run 1204the Hallucination tab's full runs, counted on the kept questions | 85.6%468 of 547 right | 83.4%456 of 547 right | 2.2% | 1.6% | 34 | 48 | 0.15 | 1 |
| Sieve 31B v3 Open · 31B | 2.2%1 “I don't know” of 46 not right | 99.8%546 of 547 answered, 1 “I don't know” | 8.2%45 of 546 | 0.0%0 “I don't know” of 33 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.0%33 of 547 | run 1066a new run | run 1253the Hallucination tab's full runs, counted on the kept questions | 91.6%501 of 547 right | 94.0%514 of 547 right | 0.8% | 0.3% | 23 | 11 | 0.058 | 1 |
| Verdin-4B-v1.1 Open · 4.0B | 2.1%2 “I don't know” of 96 not right | 99.6%545 of 547 answered, 2 “I don't know” | 17.2%94 of 545 | 0.8%1 “I don't know” of 128 not right | 99.8%546 of 547 answered, 1 “I don't know” | 23.3%127 of 546 | run 1249a new run | run 1251the Hallucination tab's full runs, counted on the kept questions | 82.4%451 of 547 right | 76.6%419 of 547 right | 3.5% | 2.2% | 37 | 70 | 0.0018 | 0.099 |
| d3 Open · 26B | 1.7%1 “I don't know” of 59 not right | 99.8%546 of 547 answered, 1 “I don't know” | 10.6%58 of 546 | 0.0%0 “I don't know” of 53 not right | 100.0%547 of 547 answered, 0 “I don't know” | 9.7%53 of 547 | run 793a new run | run 358the Hallucination tab's full runs, counted on the kept questions | 89.2%488 of 547 right | 90.3%494 of 547 right | 2.9% | 1.1% | 24 | 19 | 0.54 | 1 |
| Metask-Jev-4B Open · 4.5B | 1.6%2 “I don't know” of 125 not right | 99.6%545 of 547 answered, 2 “I don't know” | 22.6%123 of 545 | 1.1%2 “I don't know” of 189 not right | 99.6%545 of 547 answered, 2 “I don't know” | 34.3%187 of 545 | run 741a new run | run 743the Hallucination tab's full runs, counted on the kept questions | 77.1%422 of 547 right | 65.4%358 of 547 right | 3.8% | 4.1% | 34 | 98 | < 0.0001 | < 0.0001 * |
| Clef Open · 27B | 1.3%1 “I don't know” of 76 not right | 99.8%546 of 547 answered, 1 “I don't know” | 13.7%75 of 546 | 1.5%1 “I don't know” of 68 not right | 99.8%546 of 547 answered, 1 “I don't know” | 12.3%67 of 546 | run 785a new run | run 788the Hallucination tab's full runs, counted on the kept questions | 86.1%471 of 547 right | 87.6%479 of 547 right | 2.4% | 3.6% | 26 | 18 | 0.29 | 1 |
| KnowLine-12B-Gen1 Open · 12B | 1.3%1 “I don't know” of 77 not right | 99.8%546 of 547 answered, 1 “I don't know” | 13.9%76 of 546 | 0.0%0 “I don't know” of 67 not right | 100.0%547 of 547 answered, 0 “I don't know” | 12.2%67 of 547 | run 1316a new run | run 1318the Hallucination tab's full runs, counted on the kept questions | 85.9%470 of 547 right | 87.8%480 of 547 right | 1.3% | 0.6% | 26 | 17 | 0.22 | 1 |
| KnowLine-9B-Gen1 Open · 9.0B | 1.3%1 “I don't know” of 77 not right | 99.8%546 of 547 answered, 1 “I don't know” | 13.9%76 of 546 | 0.0%0 “I don't know” of 98 not right | 100.0%547 of 547 answered, 0 “I don't know” | 17.9%98 of 547 | run 1179a new run | run 1181the Hallucination tab's full runs, counted on the kept questions | 85.9%470 of 547 right | 82.1%449 of 547 right | 1.6% | 0.7% | 18 | 40 | 0.0054 | 0.28 |
| KnowLine-9B-Gen2 Open · 9.0B | 1.3%1 “I don't know” of 79 not right | 99.8%546 of 547 answered, 1 “I don't know” | 14.3%78 of 546 | 0.0%0 “I don't know” of 95 not right | 100.0%547 of 547 answered, 0 “I don't know” | 17.4%95 of 547 | run 1323a new run | run 1325the Hallucination tab's full runs, counted on the kept questions | 85.6%468 of 547 right | 82.6%452 of 547 right | 3.8% | 3.5% | 23 | 40 | 0.043 | 1 |
| Valen-Decision-1.5-9B Open · 9.0B | 1.2%1 “I don't know” of 85 not right | 99.8%546 of 547 answered, 1 “I don't know” | 15.4%84 of 546 | 0.0%0 “I don't know” of 110 not right | 100.0%547 of 547 answered, 0 “I don't know” | 20.1%110 of 547 | run 1280a new run | run 1282the Hallucination tab's full runs, counted on the kept questions | 84.5%462 of 547 right | 79.9%437 of 547 right | 3.0% | 2.7% | 32 | 58 | 0.008 | 0.4 |
| Dexy-Decision-9B-V2 Open · 9.0B | 1.1%1 “I don't know” of 91 not right | 99.8%546 of 547 answered, 1 “I don't know” | 16.5%90 of 546 | 1.1%1 “I don't know” of 91 not right | 99.8%546 of 547 answered, 1 “I don't know” | 16.5%90 of 546 | run 1260a new run | run 1262the Hallucination tab's full runs, counted on the kept questions | 83.4%456 of 547 right | 83.4%456 of 547 right | 0.7% | 0.6% | 38 | 38 | 1 | 1 |
| KnowLine 4B Gen5.5 Open · 4.0B | 0.9%1 “I don't know” of 106 not right | 99.8%546 of 547 answered, 1 “I don't know” | 19.2%105 of 546 | 2.5%3 “I don't know” of 121 not right | 99.5%544 of 547 answered, 3 “I don't know” | 21.7%118 of 544 | run 963a new run | run 967the Hallucination tab's full runs, counted on the kept questions | 80.6%441 of 547 right | 77.9%426 of 547 right | 1.3% | 1.0% | 40 | 53 | 0.21 | 1 |
| Strands Decider 2B Open · 2.0B | 0.7%2 “I don't know” of 303 not right | 99.6%545 of 547 answered, 2 “I don't know” | 55.2%301 of 545 | 1.5%5 “I don't know” of 332 not right | 99.1%542 of 547 answered, 5 “I don't know” | 60.3%327 of 542 | run 704a new run | run 706the Hallucination tab's full runs, counted on the kept questions | 44.6%244 of 547 right | 39.3%215 of 547 right | 3.8% | 4.9% | 49 | 75 | 0.024 | 1 |
| Adjudex 9B v5 Open · 9.0B | 0.0%0 “I don't know” of 77 not right | 100.0%547 of 547 answered, 0 “I don't know” | 14.1%77 of 547 | 0.0%0 “I don't know” of 89 not right | 100.0%547 of 547 answered, 0 “I don't know” | 16.3%89 of 547 | run 1188a new run | run 1190the Hallucination tab's full runs, counted on the kept questions | 85.9%470 of 547 right | 83.7%458 of 547 right | 0.6% | 0.3% | 32 | 44 | 0.21 | 1 |
| Copernicus-31B Open · 31B | 0.0%0 “I don't know” of 39 not right | 100.0%547 of 547 answered, 0 “I don't know” | 7.1%39 of 547 | 2.9%1 “I don't know” of 34 not right | 99.8%546 of 547 answered, 1 “I don't know” | 6.0%33 of 546 | run 1170a new run | run 1172the Hallucination tab's full runs, counted on the kept questions | 92.9%508 of 547 right | 93.8%513 of 547 right | 1.0% | 0.7% | 19 | 13 | 0.38 | 1 |
| Jeffan-9B-1211 Open · 9.0B | 0.0%0 “I don't know” of 71 not right | 100.0%547 of 547 answered, 0 “I don't know” | 13.0%71 of 547 | 4.1%3 “I don't know” of 74 not right | 99.5%544 of 547 answered, 3 “I don't know” | 13.1%71 of 544 | run 1197a new run | run 1199the Hallucination tab's full runs, counted on the kept questions | 87.0%476 of 547 right | 86.5%473 of 547 right | 3.7% | 3.7% | 31 | 31 | 1 | 1 |
| Karpo Decision 31B Open · 31B | 0.0%0 “I don't know” of 42 not right | 100.0%547 of 547 answered, 0 “I don't know” | 7.7%42 of 547 | 3.2%1 “I don't know” of 31 not right | 99.8%546 of 547 answered, 1 “I don't know” | 5.5%30 of 546 | run 1161a new run | run 1163the Hallucination tab's full runs, counted on the kept questions | 92.3%505 of 547 right | 94.3%516 of 547 right | 0.6% | 0.3% | 25 | 13 | 0.073 | 1 |
| KnowLine-31B-Gen1.5 Open · 31B | 0.0%0 “I don't know” of 37 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.8%37 of 547 | 0.0%0 “I don't know” of 34 not right | 100.0%547 of 547 answered, 0 “I don't know” | 6.2%34 of 547 | run 955a new run | run 959the Hallucination tab's full runs, counted on the kept questions | 93.2%510 of 547 right | 93.8%513 of 547 right | 0.2% | 0.2% | 16 | 13 | 0.71 | 1 |
| KnowLine-31B-Gen2 Open · 31B | 0.0%0 “I don't know” of 40 not right | 100.0%547 of 547 answered, 0 “I don't know” | 7.3%40 of 547 | 0.0%0 “I don't know” of 29 not right | 100.0%547 of 547 answered, 0 “I don't know” | 5.3%29 of 547 | run 1329a new run | run 1331the Hallucination tab's full runs, counted on the kept questions | 92.7%507 of 547 right | 94.7%518 of 547 right | 0.2% | 0.1% | 20 | 9 | 0.061 | 1 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | 0.0%0 “I don't know” of 94 not right | 100.0%547 of 547 answered, 0 “I don't know” | 17.2%94 of 547 | 1.7%2 “I don't know” of 120 not right | 99.6%545 of 547 answered, 2 “I don't know” | 21.7%118 of 545 | run 1083a new run | run 1239the Hallucination tab's full runs, counted on the kept questions | 82.8%453 of 547 right | 78.1%427 of 547 right | 3.2% | 2.9% | 33 | 57 | 0.015 | 0.71 |
Our method with an “I don't know” option
The option
As on the Hallucination tab, each question gets a sixth option after its own five: F, “I don't know” in English, “Bilmiyorum” in Turkish. It is always the last option, never shuffled and never correct. The English reading instructions of a decision model get one added sentence: “If you do not know the answer, choose I don't know.” The full instructions:
Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.
Prompt version 413d6775fc58. The question and its options are the English translation, as in the accuracy comparison above; no person checked the translations. The Turkish side uses the Hallucination tab's prompts unchanged.
Any other chat model is asked with the chat models' English prompts, the option after each question's own and one sentence added at the end of each instruction: “If you do not know the answer, choose I don't know.” Its system prompt and instruction when it states a confidence:
You are taking an English multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.
Answer each of the following multiple-choice questions. Without explanation, return for each question the letter of the correct option together with a confidence score between 0 and 1 showing how sure you are that your answer is correct, in the specified JSON format. If you do not know the answer, choose I don't know.
Chat prompt version e53d4aa27643. Its Turkish side uses the Hallucination tab's chat prompts unchanged.
A chat model asked for a plain answer line (LFM2.5-8B-A1B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning on · chat, LFM3-1.7B · reasoning off · chat, Qwen3.5-2B · reasoning off · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, Qwen3.5-4B · reasoning on · chat, Raon-1 nano, MiniCPM5-2B, Granite-4.2-3B, LFM2.5-24B-A2B, Gemma 4 E2B, Gemma 4 E2B, Qwen3.5-2B and Gemma 4 12B Unified) is asked with its plain answer line's English instruction, the option's letter among its letters and one sentence added at its end: “If you do not know the answer, choose I don't know.” Plain prompt version 4cd69035a45e. Its Turkish side uses the Hallucination tab's plain prompts with “Bilmiyorum” unchanged. It states no confidence and reports no option probabilities, so its mean probability of the option is not measured.
The Turkish side
The Turkish side: the Hallucination tab's published full runs, counted on the kept questions only, for d1.2-ultra (run 1367), d1.2 (run 1365), d1-2B (run 712), d1-ultra (runs 350, 354, 360, 365, 370), GLiDE 0.4B (run 937), Drex Max v1.1 (run 352), Mammoth-1 (run 386), GLiDE · v1.2 (run 369), Laya Multilingual (run 737), GLiDE 27B (run 905), OneJev1.6-0.8B (run 1031), Grok 4.7 · high · chat (run 675), LFM2.5-8B-A1B · default reasoning · chat (run 1298), GLiDE 2B (run 921), Elephant-1 (run 382), Gemini 4 Carbon · high · chat (run 1121), SubQ Snap 1.1 (run 392), GLiDE 0.8B (run 929), GLiClass v4 Edge (run 1048), Gemini 4 Hydrogen · high · chat (run 1117), GLiNER2.5 Multi (run 724), GPT-6 Astra · high · chat (run 818), Gemini 4 Argon · low · chat (run 679), Laya (run 730), GPT-6 Astra · low · chat (run 816), Open-Jev-27B-v1.1 (run 768), StartLux-Decision-v1.1-2B-Instant (run 979), MiniCPM5-2B · reasoning off · chat (run 1304), GLiDE 9B (run 913), GPT-6 Astra · medium · chat (run 817), GPT-6.1 Sol · high · chat (run 802), GPT-6.1 Terra · high · chat (run 807), GLiNER2.5-multi-Decide (run 718), LFM2.5-2.6B · default reasoning · chat (run 1038), LFM3-1.7B · reasoning on · chat (run 1090), GPT-6.1 Terra · low · chat (run 801), Perplexity Decider 27B v1.2 · API · with web (run 642), Winnow-12B (run 750), GPT-6 Astra · low · chat · 1 per request (run 819), GPT-6.1 Sol · low · chat (run 799), GPT-6.1 Sol · low · chat · 1 per request (run 810), LFM3-1.7B · reasoning off · chat (run 1234), Cygnet (run 757), KnowLine-0.8B-Gen1 (run 1007), Qwen3.5-2B · reasoning off · chat (run 1219), Gemini 3.8 Flash · high · chat (run 681), Gemma 4 E4B · default reasoning · chat (run 1049), Gemini Decision 1 · 20261111 (run 1023), Perplexity Decider 27B v1.3 (run 1141), Turion Jev 31B (run 1285), d3-ultra (run 363), Claude Haiku 5.5 · max · chat (run 950), Gemini 3.8 Flash · low · chat · 1 per request (run 691), Kimi K3.1 Pro · no reasoning · chat (run 653), vera-medium-omni (run 1271), Claude Fable 5.1 · high · chat (run 695), Kimi K3.1 Pro · low · chat (run 686), Perplexity Decider Flash v1.3 (run 1130), DeepSeek-V4.1-Flash · reasoning on · chat · 1017 (run 811), GPT-6 Luna · low · chat (run 796), REAX 1.3 (run 833), Claude Opus 5.5 · high · chat (run 773), Qwen3.5-397B-A17B · high · chat (run 872), GLM-5.3-Flash · reasoning off · chat (run 822), Qwen3.8 Flash · no reasoning · chat (run 987), DeepSeek-V4.1-Flash · reasoning off · chat · 1017 (run 806), Clef-Flash (run 664), Muse Ember · max · chat (run 940), StartLux-Decision-v1.1-0.8B-Instant (run 1017), Claude Sonnet 5.5 · high · chat (run 800), Claude Opus 5.5 · low · chat (run 683), Perplexity Decider 27B v1.2 · API (run 641), Claude Sonnet 5.5 · low · chat (run 690), Logits Sigma-1 Duo (run 1233), OneJev-27B (run 782), Chinese Decision Pointer 4B (run 1213), Qwen3.8-27B · stated confidence · reasoning off · chat (run 674), Gemma 4 31B · reasoning off · chat (run 776), Qwen3.5-4B · reasoning off · chat (run 1147), Qwen3.8-27B · low · chat (run 666), Gemini 3.8 Flash · low · chat (run 676), GLM-5.3 · low · chat (run 656), Qwen3.8-27B · vote share · reasoning off · chat (run 682), MiniMax-M3.1-Flash · high · chat (run 864), Drex Max (run 367), Perplexity Decider 27B v1.2 · DGX Spark (run 372), StartLux-Decision-v1.1-9B-Instant (run 1297), Cerno 9B v1.2 (run 1204), Qwen3.5-4B · reasoning on · chat (run 1150), Sieve 31B v3 (run 1253), Erk-14B · reasoning off · chat (run 765), Verdin-4B-v1.1 (run 1251), Qwen3-14B · reasoning off · chat (run 772), MiMo-V2.6-Flash · high · chat (run 879), d3 (run 358), GLM-5.3-Flash · reasoning on · chat (run 825), Metask-Jev-4B (run 743), Clef (run 788), KnowLine-12B-Gen1 (run 1318), KnowLine-9B-Gen1 (run 1181), KnowLine-9B-Gen2 (run 1325), Valen-Decision-1.5-9B (run 1282), Dexy-Decision-9B-V2 (run 1262), KnowLine 4B Gen5.5 (run 967), Strands Decider 2B (run 706), Adjudex 9B v5 (run 1190), Claude Opus 5.5 · low · chat · 1 per request (run 803), Claude Sonnet 5.5 · high · chat · 1 per request (run 812), Copernicus-31B (run 1172), Gemma 4 31B · reasoning off · chat · 1 per request (run 789), Inkling-Small · high · chat · Jul 2026 (run 868), Jeffan-9B-1211 (run 1199), Karpo Decision 31B (run 1163), KnowLine-31B-Gen1.5 (run 959), KnowLine-31B-Gen2 (run 1331), Qwen4-Max · high · chat (run 1339) and StartLux-Decision-v1.1-4B-Instant (run 1239); new runs on the kept questions only, with the Hallucination tab's prompts, for Mercury Decide (runs 489, 491, 493, 495, 497), GPT-6 Luna · Decisions API (run 453), Perplexity Decider Flash v1.2 (run 537), Jev 1.14 (runs 470, 473, 476, 480, 482), Microsoft-Decision-1 (runs 487, 490, 492, 494, 496) and d1 (runs 472, 475, 479, 481, 485). The new Turkish runs are not on the Hallucination tab, which uses full runs only.
d1 was asked as d1 in both languages, while its Text tab run used d1:free, because Liquid's API no longer serves that name; its model list still names d1, released 29 September 2026.
celeris-1.1-decision is shown in English only (run 884): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
celeris-1-decision is shown in English only (run 548): no Turkish run with “Bilmiyorum” on these questions; new Turkish runs were outside the request of 3 November 2026. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
Raon-1 nano · reasoning off · chat is shown in English only (run 1153): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
KnowLine-2B-Gen1 is shown in English only (run 1059): no Turkish run with “Bilmiyorum”: its Turkish Text run alone was asked (Ümit #6224, 13 November 2026). Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
MiniCPM5-2B · reasoning on · chat is shown in English only (run 1052): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
Granite-4.2-3B · reasoning on · chat is shown in English only (run 1110): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
LFM2.5-24B-A2B · default reasoning · chat is shown in English only (run 1070): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
Gemma 4 E2B · default reasoning · chat is shown in English only (run 893): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
Gemma 4 E2B · reasoning on · chat is shown in English only (run 1350): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
Qwen3.5-2B · reasoning on · chat is shown in English only (run 1344): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
Gemma 4 12B Unified · default reasoning · chat is shown in English only (run 896): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.
Running: a chat model's runs with the option go English first, then Turkish, and each is added as it finishes. Until both runs of a chat model are in, it is shown with the side it has and the other reads Running, in no paired test and no Holm adjustment: Claude Fable 5.2 · max · chat.
Runs
five runs in English and five runs in Turkish for d1-ultra, Mercury Decide, Jev 1.14, Microsoft-Decision-1 and d1; one run in English and one run in Turkish for d1.2-ultra, d1.2, d1-2B, GLiDE 0.4B, Drex Max v1.1, Mammoth-1, GLiDE · v1.2, Laya Multilingual, GLiDE 27B, OneJev1.6-0.8B, Grok 4.7 · high · chat, LFM2.5-8B-A1B · default reasoning · chat, GLiDE 2B, Elephant-1, Gemini 4 Carbon · high · chat, SubQ Snap 1.1, GLiDE 0.8B, GLiClass v4 Edge, Gemini 4 Hydrogen · high · chat, GLiNER2.5 Multi, GPT-6 Astra · high · chat, Gemini 4 Argon · low · chat, Laya, GPT-6 Astra · low · chat, Open-Jev-27B-v1.1, StartLux-Decision-v1.1-2B-Instant, MiniCPM5-2B · reasoning off · chat, GLiDE 9B, GPT-6 Astra · medium · chat, GPT-6.1 Sol · high · chat, GPT-6.1 Terra · high · chat, GLiNER2.5-multi-Decide, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning on · chat, GPT-6.1 Terra · low · chat, Perplexity Decider 27B v1.2 · API · with web, Winnow-12B, GPT-6 Astra · low · chat · 1 per request, GPT-6.1 Sol · low · chat, GPT-6.1 Sol · low · chat · 1 per request, LFM3-1.7B · reasoning off · chat, Cygnet, KnowLine-0.8B-Gen1, Qwen3.5-2B · reasoning off · chat, Gemini 3.8 Flash · high · chat, Gemma 4 E4B · default reasoning · chat, GPT-6 Luna · Decisions API, Gemini Decision 1 · 20261111, Perplexity Decider 27B v1.3, Turion Jev 31B, d3-ultra, Claude Haiku 5.5 · max · chat, Gemini 3.8 Flash · low · chat · 1 per request, Kimi K3.1 Pro · no reasoning · chat, vera-medium-omni, Claude Fable 5.1 · high · chat, Kimi K3.1 Pro · low · chat, Perplexity Decider Flash v1.3, DeepSeek-V4.1-Flash · reasoning on · chat · 1017, GPT-6 Luna · low · chat, REAX 1.3, Claude Opus 5.5 · high · chat, Qwen3.5-397B-A17B · high · chat, GLM-5.3-Flash · reasoning off · chat, Qwen3.8 Flash · no reasoning · chat, DeepSeek-V4.1-Flash · reasoning off · chat · 1017, Clef-Flash, Muse Ember · max · chat, StartLux-Decision-v1.1-0.8B-Instant, Claude Sonnet 5.5 · high · chat, Perplexity Decider Flash v1.2, Claude Opus 5.5 · low · chat, Perplexity Decider 27B v1.2 · API, Claude Sonnet 5.5 · low · chat, Logits Sigma-1 Duo, OneJev-27B, Chinese Decision Pointer 4B, Qwen3.8-27B · stated confidence · reasoning off · chat, Gemma 4 31B · reasoning off · chat, Qwen3.5-4B · reasoning off · chat, Qwen3.8-27B · low · chat, Gemini 3.8 Flash · low · chat, GLM-5.3 · low · chat, Qwen3.8-27B · vote share · reasoning off · chat, MiniMax-M3.1-Flash · high · chat, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark, StartLux-Decision-v1.1-9B-Instant, Cerno 9B v1.2, Qwen3.5-4B · reasoning on · chat, Sieve 31B v3, Erk-14B · reasoning off · chat, Verdin-4B-v1.1, Qwen3-14B · reasoning off · chat, MiMo-V2.6-Flash · high · chat, d3, GLM-5.3-Flash · reasoning on · chat, Metask-Jev-4B, Clef, KnowLine-12B-Gen1, KnowLine-9B-Gen1, KnowLine-9B-Gen2, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, KnowLine 4B Gen5.5, Strands Decider 2B, Adjudex 9B v5, Claude Opus 5.5 · low · chat · 1 per request, Claude Sonnet 5.5 · high · chat · 1 per request, Copernicus-31B, Gemma 4 31B · reasoning off · chat · 1 per request, Inkling-Small · high · chat · Jul 2026, Jeffan-9B-1211, Karpo Decision 31B, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2, Qwen4-Max · high · chat and StartLux-Decision-v1.1-4B-Instant. These runs are kept out of the leaderboard, its repeats, the Hallucination tab and the accuracy comparison above.
The rates and how to read them
Each answer is right, wrong, the option or invalid. Out of the 547 kept questions, accuracy is right / 547; Answered is (right + wrong) / 547, the share of questions the model answered (the attempt rate), higher is better; Wrong of answers is wrong / (right + wrong), the share of its answers that were wrong, lower is better, and Undefined when it answered none. These two lead the table, with their counts under them. The non-hallucination rate, after them, is the option / (the option + wrong): among the answers that were not right, the share where the model said it did not know. As the Hallucination tab says, it is not a direct measure of what the model does not know: a model that always chose the option would reach 100% with no right answers, so read it with the attempt rate. With neither a wrong answer nor the option, the rate is undefined (No rate). The mean probability is the probability the model gave the option, averaged over its answers that report option probabilities; a chat model reports none. The paired test is the exact McNemar test on the questions only one of the two first runs got wrong, with p adjusted by Holm's method over the 67 decision models and the 56 chat models, each group a family of its own. No new combined score is made from these rates.
Results with “I don't know” in both languages
In the order of the Results table above. Each row compares the model's first English run with “I don't know” with its Turkish run with the option also written “I don't know”, out of the 547 kept questions. The paired counts, p and the Holm-adjusted p (over the 12 models with both runs) are those of the exact McNemar test on wrong answers; * marks a Holm-adjusted p below 0.05. Not run: no such run; Undefined: a rate with nothing to count.
| Model | Turkish answered(right + wrong) / 547; higher is better | English answered(right + wrong) / 547; higher is better | Turkish wrong of answerswrong / (right + wrong); lower is better | English wrong of answerswrong / (right + wrong); lower is better | Turkish non-hallucination“I don't know” / (“I don't know” + wrong); higher is better, read with the answered share | English non-hallucination“I don't know” / (“I don't know” + wrong); higher is better, read with the answered share | Turkish accuracyhigher is better | English accuracyhigher is better | Turkish mean probabilityof “I don't know” | English mean probabilityof “I don't know” | Turkish runthe option-word run, counted on the kept questions | English runthe first English run with “I don't know” | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 12 models with both runs |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | Not run | 100.0%547 of 547 answered, 0 “I don't know” | Not run | 6.8%37 of 547 | Not run | 0.0%0 “I don't know” of 37 not right | Not run | 93.2%510 of 547 right | Not run | 0.2% | Not run | run 955 | Not run | Not run | Not run | Not run |
| Copernicus-31B Open · 31B | Not run | 100.0%547 of 547 answered, 0 “I don't know” | Not run | 7.1%39 of 547 | Not run | 0.0%0 “I don't know” of 39 not right | Not run | 92.9%508 of 547 right | Not run | 1.0% | Not run | run 1170 | Not run | Not run | Not run | Not run |
| REAX 1.3 Open · 57B | Not run | 99.5%544 of 547 answered, 3 “I don't know” | Not run | 7.7%42 of 544 | Not run | 6.7%3 “I don't know” of 45 not right | Not run | 91.8%502 of 547 right | Not run | 1.9% | Not run | run 831 | Not run | Not run | Not run | Not run |
| KnowLine-31B-Gen2 Open · 31B | Not run | 100.0%547 of 547 answered, 0 “I don't know” | Not run | 7.3%40 of 547 | Not run | 0.0%0 “I don't know” of 40 not right | Not run | 92.7%507 of 547 right | Not run | 0.2% | Not run | run 1329 | Not run | Not run | Not run | Not run |
| Karpo Decision 31B Open · 31B | Not run | 100.0%547 of 547 answered, 0 “I don't know” | Not run | 7.7%42 of 547 | Not run | 0.0%0 “I don't know” of 42 not right | Not run | 92.3%505 of 547 right | Not run | 0.6% | Not run | run 1161 | Not run | Not run | Not run | Not run |
| Gemini Decision 1 20261111 Closed · Efficient | Not run | 99.1%542 of 547 answered, 5 “I don't know” | Not run | 9.2%50 of 542 | Not run | 9.1%5 “I don't know” of 55 not right | Not run | 89.9%492 of 547 right | Not run | 4.6% | Not run | run 1019 | Not run | Not run | Not run | Not run |
| Logits Sigma-1 Duo Open · 57B | Not run | 99.6%545 of 547 answered, 2 “I don't know” | Not run | 8.1%44 of 545 | Not run | 4.3%2 “I don't know” of 46 not right | Not run | 91.6%501 of 547 right | Not run | 1.8% | Not run | run 1231 | Not run | Not run | Not run | Not run |
| GLiDE 27B Open · 27B | Not run | 96.0%525 of 547 answered, 22 “I don't know” | Not run | 6.9%36 of 525 | Not run | 37.9%22 “I don't know” of 58 not right | Not run | 89.4%489 of 547 right | Not run | 9.3% | Not run | run 903 | Not run | Not run | Not run | Not run |
| d1-ultra Closed · Efficient | 95.1%520 of 547 answered, 27 “I don't know” | 90.7%496 of 547 answered, 51 “I don't know” | 5.4%28 of 520 | 7.9%39 of 496 | 49.1%27 “I don't know” of 55 not right | 56.7%51 “I don't know” of 90 not right | 89.9%492 of 547 right | 83.5%457 of 547 right | 23.2% | 25.3% | run 511 | run 449 | 18 | 7 | 0.043 | 0.48 |
| Sieve 31B v3 Open · 31B | Not run | 99.8%546 of 547 answered, 1 “I don't know” | Not run | 8.2%45 of 546 | Not run | 2.2%1 “I don't know” of 46 not right | Not run | 91.6%501 of 547 right | Not run | 0.8% | Not run | run 1066 | Not run | Not run | Not run | Not run |
| Turion Jev 31B Open · 31B | Not run | 99.3%543 of 547 answered, 4 “I don't know” | Not run | 7.9%43 of 543 | Not run | 8.5%4 “I don't know” of 47 not right | Not run | 91.4%500 of 547 right | Not run | 2.3% | Not run | run 1077 | Not run | Not run | Not run | Not run |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | Not run | 98.2%537 of 547 answered, 10 “I don't know” | Not run | 10.4%56 of 537 | Not run | 15.2%10 “I don't know” of 66 not right | Not run | 87.9%481 of 547 right | Not run | 4.6% | Not run | run 644 | Not run | Not run | Not run | Not run |
| GLiDE v1.2 Open · 27B | 98.4%538 of 547 answered, 9 “I don't know” | 94.5%517 of 547 answered, 30 “I don't know” | 8.7%47 of 538 | 7.0%36 of 517 | 16.1%9 “I don't know” of 56 not right | 45.5%30 “I don't know” of 66 not right | 89.8%491 of 547 right | 87.9%481 of 547 right | 9.8% | 12.5% | run 525 | run 523 | 16 | 27 | 0.13 | 1 |
| d1.2-ultra Closed · Efficient | 89.0%487 of 547 answered, 60 “I don't know” | 80.1%438 of 547 answered, 109 “I don't know” | 3.7%18 of 487 | 3.0%13 of 438 | 76.9%60 “I don't know” of 78 not right | 89.3%109 “I don't know” of 122 not right | 85.7%469 of 547 right | 77.7%425 of 547 right | 17.7% | 26.0% | run 1378 | run 1363 | 7 | 12 | 0.36 | 1 |
| Mammoth-1 Open · 27B | 95.4%522 of 547 answered, 25 “I don't know” | 93.2%510 of 547 answered, 37 “I don't know” | 6.1%32 of 522 | 7.1%36 of 510 | 43.9%25 “I don't know” of 57 not right | 50.7%37 “I don't know” of 73 not right | 89.6%490 of 547 right | 86.7%474 of 547 right | 7.4% | 9.8% | run 519 | run 505 | 19 | 15 | 0.61 | 1 |
| d3 Open · 26B | 100.0%547 of 547 answered, 0 “I don't know” | 99.8%546 of 547 answered, 1 “I don't know” | 9.5%52 of 547 | 10.6%58 of 546 | 0.0%0 “I don't know” of 52 not right | 1.7%1 “I don't know” of 59 not right | 90.5%495 of 547 right | 89.2%488 of 547 right | 1.4% | 2.9% | run 544 | run 793 | 24 | 18 | 0.44 | 1 |
| Drex Max Open · 27B | 99.8%546 of 547 answered, 1 “I don't know” | 99.6%545 of 547 answered, 2 “I don't know” | 9.0%49 of 546 | 11.0%60 of 545 | 2.0%1 “I don't know” of 50 not right | 3.2%2 “I don't know” of 62 not right | 90.9%497 of 547 right | 88.7%485 of 547 right | 0.9% | 1.7% | run 542 | run 798 | 28 | 17 | 0.14 | 1 |
| Perplexity Decider 27B v1.3 Open · 27B | Not run | 99.1%542 of 547 answered, 5 “I don't know” | Not run | 9.6%52 of 542 | Not run | 8.8%5 “I don't know” of 57 not right | Not run | 89.6%490 of 547 right | Not run | 3.8% | Not run | run 1139 | Not run | Not run | Not run | Not run |
| d3-ultra Open · 26B | 99.6%545 of 547 answered, 2 “I don't know” | 99.1%542 of 547 answered, 5 “I don't know” | 9.4%51 of 545 | 10.3%56 of 542 | 3.8%2 “I don't know” of 53 not right | 8.2%5 “I don't know” of 61 not right | 90.3%494 of 547 right | 88.8%486 of 547 right | 1.3% | 2.8% | run 540 | run 795 | 31 | 26 | 0.6 | 1 |
| d1.2 Closed · Efficient | 91.6%501 of 547 answered, 46 “I don't know” | 89.2%488 of 547 answered, 59 “I don't know” | 8.4%42 of 501 | 7.6%37 of 488 | 52.3%46 “I don't know” of 88 not right | 61.5%59 “I don't know” of 96 not right | 83.9%459 of 547 right | 82.4%451 of 547 right | 17.0% | 19.4% | run 1377 | run 1361 | 19 | 24 | 0.54 | 1 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 99.8%546 of 547 answered, 1 “I don't know” | 99.6%545 of 547 answered, 2 “I don't know” | 9.3%51 of 546 | 11.7%64 of 545 | 1.9%1 “I don't know” of 52 not right | 3.0%2 “I don't know” of 66 not right | 90.5%495 of 547 right | 87.9%481 of 547 right | 1.4% | 2.8% | run 531 | run 529 | 29 | 16 | 0.072 | 0.72 |
| Drex Max v1.1 Closed · Efficient (provisional) | 96.7%529 of 547 answered, 18 “I don't know” | 92.1%504 of 547 answered, 43 “I don't know” | 9.1%48 of 529 | 8.1%41 of 504 | 27.3%18 “I don't know” of 66 not right | 51.2%43 “I don't know” of 84 not right | 87.9%481 of 547 right | 84.6%463 of 547 right | 10.2% | 14.3% | run 513 | run 447 | 20 | 27 | 0.38 | 1 |
| Perplexity Decider 27B v1.2 API Open · 27B | Not run | 99.5%544 of 547 answered, 3 “I don't know” | Not run | 11.6%63 of 544 | Not run | 4.5%3 “I don't know” of 66 not right | Not run | 87.9%481 of 547 right | Not run | 3.1% | Not run | run 643 | Not run | Not run | Not run | Not run |
| Jeffan-9B-1211 Open · 9.0B | Not run | 100.0%547 of 547 answered, 0 “I don't know” | Not run | 13.0%71 of 547 | Not run | 0.0%0 “I don't know” of 71 not right | Not run | 87.0%476 of 547 right | Not run | 3.7% | Not run | run 1197 | Not run | Not run | Not run | Not run |
| d1 Closed · Efficient | Not run | 99.6%545 of 547 answered, 2 “I don't know” | Not run | 12.3%67 of 545 | Not run | 2.9%2 “I don't know” of 69 not right | Not run | 87.4%478 of 547 right | Not run | 0.9% | Not run | run 445 | Not run | Not run | Not run | Not run |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | Not run | 99.6%545 of 547 answered, 2 “I don't know” | Not run | 13.6%74 of 545 | Not run | 2.6%2 “I don't know” of 76 not right | Not run | 86.1%471 of 547 right | Not run | 1.8% | Not run | run 1295 | Not run | Not run | Not run | Not run |
| Clef Open · 27B | Not run | 99.8%546 of 547 answered, 1 “I don't know” | Not run | 13.7%75 of 546 | Not run | 1.3%1 “I don't know” of 76 not right | Not run | 86.1%471 of 547 right | Not run | 2.4% | Not run | run 785 | Not run | Not run | Not run | Not run |
| KnowLine-12B-Gen1 Open · 12B | Not run | 99.8%546 of 547 answered, 1 “I don't know” | Not run | 13.9%76 of 546 | Not run | 1.3%1 “I don't know” of 77 not right | Not run | 85.9%470 of 547 right | Not run | 1.3% | Not run | run 1316 | Not run | Not run | Not run | Not run |
| GLiDE 9B Open · 7.9B | Not run | 97.4%533 of 547 answered, 14 “I don't know” | Not run | 13.1%70 of 533 | Not run | 16.7%14 “I don't know” of 84 not right | Not run | 84.6%463 of 547 right | Not run | 7.8% | Not run | run 911 | Not run | Not run | Not run | Not run |
| OneJev-27B Open · 27B | Not run | 99.5%544 of 547 answered, 3 “I don't know” | Not run | 13.2%72 of 544 | Not run | 4.0%3 “I don't know” of 75 not right | Not run | 86.3%472 of 547 right | Not run | 2.3% | Not run | run 780 | Not run | Not run | Not run | Not run |
| Jev 1.14 Closed · Efficient | Not run | 99.5%544 of 547 answered, 3 “I don't know” | Not run | 13.2%72 of 544 | Not run | 4.0%3 “I don't know” of 75 not right | Not run | 86.3%472 of 547 right | Not run | 1.6% | Not run | run 446 | Not run | Not run | Not run | Not run |
| Cerno 9B v1.2 Open · 9.0B | Not run | 99.6%545 of 547 answered, 2 “I don't know” | Not run | 14.1%77 of 545 | Not run | 2.5%2 “I don't know” of 79 not right | Not run | 85.6%468 of 547 right | Not run | 2.2% | Not run | run 1087 | Not run | Not run | Not run | Not run |
| KnowLine-9B-Gen1 Open · 9.0B | Not run | 99.8%546 of 547 answered, 1 “I don't know” | Not run | 13.9%76 of 546 | Not run | 1.3%1 “I don't know” of 77 not right | Not run | 85.9%470 of 547 right | Not run | 1.6% | Not run | run 1179 | Not run | Not run | Not run | Not run |
| SubQ Snap 1.1 Closed · Efficient | 96.5%528 of 547 answered, 19 “I don't know” | 96.2%526 of 547 answered, 21 “I don't know” | 8.5%45 of 528 | 12.0%63 of 526 | 29.7%19 “I don't know” of 64 not right | 25.0%21 “I don't know” of 84 not right | 88.3%483 of 547 right | 84.6%463 of 547 right | 8.9% | 11.7% | run 512 | run 448 | 40 | 22 | 0.03 | 0.36 |
| Adjudex 9B v5 Open · 9.0B | Not run | 100.0%547 of 547 answered, 0 “I don't know” | Not run | 14.1%77 of 547 | Not run | 0.0%0 “I don't know” of 77 not right | Not run | 85.9%470 of 547 right | Not run | 0.6% | Not run | run 1188 | Not run | Not run | Not run | Not run |
| Mercury Decide Closed · Efficient | Not run | 98.0%536 of 547 answered, 11 “I don't know” | Not run | 13.2%71 of 536 | Not run | 13.4%11 “I don't know” of 82 not right | Not run | 85.0%465 of 547 right | Not run | 3.2% | Not run | run 450 | Not run | Not run | Not run | Not run |
| Microsoft-Decision-1 Closed · Efficient | Not run | 99.5%544 of 547 answered, 3 “I don't know” | Not run | 14.2%77 of 544 | Not run | 3.8%3 “I don't know” of 80 not right | Not run | 85.4%467 of 547 right | Not run | 3.6% | Not run | run 451 | Not run | Not run | Not run | Not run |
| Elephant-1 Open · 9.0B | 95.6%523 of 547 answered, 24 “I don't know” | 95.8%524 of 547 answered, 23 “I don't know” | 15.7%82 of 523 | 12.8%67 of 524 | 22.6%24 “I don't know” of 106 not right | 25.6%23 “I don't know” of 90 not right | 80.6%441 of 547 right | 83.5%457 of 547 right | 7.3% | 6.8% | run 515 | run 501 | 31 | 46 | 0.11 | 0.99 |
| Open-Jev-27B-v1.1 Open · 27B | Not run | 97.1%531 of 547 answered, 16 “I don't know” | Not run | 14.3%76 of 531 | Not run | 17.4%16 “I don't know” of 92 not right | Not run | 83.2%455 of 547 right | Not run | 15.2% | Not run | run 763 | Not run | Not run | Not run | Not run |
| KnowLine-9B-Gen2 Open · 9.0B | Not run | 99.8%546 of 547 answered, 1 “I don't know” | Not run | 14.3%78 of 546 | Not run | 1.3%1 “I don't know” of 79 not right | Not run | 85.6%468 of 547 right | Not run | 3.8% | Not run | run 1323 | Not run | Not run | Not run | Not run |
| celeris-1-decision Closed · Efficient | Not run | 83.2%455 of 547 answered, 92 “I don't know” | Not run | 8.1%37 of 455 | Not run | 71.3%92 “I don't know” of 129 not right | Not run | 76.4%418 of 547 right | Not run | 12.7% | Not run | run 548 | Not run | Not run | Not run | Not run |
| GPT-6 Luna Decisions API Closed · Efficient | Not run | 98.2%537 of 547 answered, 10 “I don't know” | Not run | 17.9%96 of 537 | Not run | 9.4%10 “I don't know” of 106 not right | Not run | 80.6%441 of 547 right | Not run | 3.0% | Not run | run 444 | Not run | Not run | Not run | Not run |
| Cygnet Open · 12B | Not run | 98.2%537 of 547 answered, 10 “I don't know” | Not run | 14.7%79 of 537 | Not run | 11.2%10 “I don't know” of 89 not right | Not run | 83.7%458 of 547 right | Not run | 3.5% | Not run | run 755 | Not run | Not run | Not run | Not run |
| Winnow-12B Open · 12B | Not run | 97.3%532 of 547 answered, 15 “I don't know” | Not run | 16.0%85 of 532 | Not run | 15.0%15 “I don't know” of 100 not right | Not run | 81.7%447 of 547 right | Not run | 3.1% | Not run | run 748 | Not run | Not run | Not run | Not run |
| Perplexity Decider Flash v1.3 Open · 4.0B | Not run | 98.7%540 of 547 answered, 7 “I don't know” | Not run | 16.7%90 of 540 | Not run | 7.2%7 “I don't know” of 97 not right | Not run | 82.3%450 of 547 right | Not run | 5.1% | Not run | run 1128 | Not run | Not run | Not run | Not run |
| Valen-Decision-1.5-9B Open · 9.0B | Not run | 99.8%546 of 547 answered, 1 “I don't know” | Not run | 15.4%84 of 546 | Not run | 1.2%1 “I don't know” of 85 not right | Not run | 84.5%462 of 547 right | Not run | 3.0% | Not run | run 1280 | Not run | Not run | Not run | Not run |
| Dexy-Decision-9B-V2 Open · 9.0B | Not run | 99.8%546 of 547 answered, 1 “I don't know” | Not run | 16.5%90 of 546 | Not run | 1.1%1 “I don't know” of 91 not right | Not run | 83.4%456 of 547 right | Not run | 0.7% | Not run | run 1260 | Not run | Not run | Not run | Not run |
| vera-medium-omni Open · 4.2B | Not run | 98.7%540 of 547 answered, 7 “I don't know” | Not run | 16.1%87 of 540 | Not run | 7.4%7 “I don't know” of 94 not right | Not run | 82.8%453 of 547 right | Not run | 5.7% | Not run | run 1269 | Not run | Not run | Not run | Not run |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | Not run | 100.0%547 of 547 answered, 0 “I don't know” | Not run | 17.2%94 of 547 | Not run | 0.0%0 “I don't know” of 94 not right | Not run | 82.8%453 of 547 right | Not run | 3.2% | Not run | run 1083 | Not run | Not run | Not run | Not run |
| Clef-Flash Open · 9.0B | Not run | 99.1%542 of 547 answered, 5 “I don't know” | Not run | 16.6%90 of 542 | Not run | 5.3%5 “I don't know” of 95 not right | Not run | 82.6%452 of 547 right | Not run | 3.1% | Not run | run 660 | Not run | Not run | Not run | Not run |
| Verdin-4B-v1.1 Open · 4.0B | Not run | 99.6%545 of 547 answered, 2 “I don't know” | Not run | 17.2%94 of 545 | Not run | 2.1%2 “I don't know” of 96 not right | Not run | 82.4%451 of 547 right | Not run | 3.5% | Not run | run 1249 | Not run | Not run | Not run | Not run |
| Perplexity Decider Flash v1.2 Open · 4.0B | Not run | 99.1%542 of 547 answered, 5 “I don't know” | Not run | 18.5%100 of 542 | Not run | 4.8%5 “I don't know” of 105 not right | Not run | 80.8%442 of 547 right | Not run | 5.5% | Not run | run 535 | Not run | Not run | Not run | Not run |
| KnowLine 4B Gen5.5 Open · 4.0B | Not run | 99.8%546 of 547 answered, 1 “I don't know” | Not run | 19.2%105 of 546 | Not run | 0.9%1 “I don't know” of 106 not right | Not run | 80.6%441 of 547 right | Not run | 1.3% | Not run | run 963 | Not run | Not run | Not run | Not run |
| Chinese Decision Pointer 4B Open · 4.0B | Not run | 99.3%543 of 547 answered, 4 “I don't know” | Not run | 19.2%104 of 543 | Not run | 3.7%4 “I don't know” of 108 not right | Not run | 80.3%439 of 547 right | Not run | 2.1% | Not run | run 1211 | Not run | Not run | Not run | Not run |
| Metask-Jev-4B Open · 4.5B | Not run | 99.6%545 of 547 answered, 2 “I don't know” | Not run | 22.6%123 of 545 | Not run | 1.6%2 “I don't know” of 125 not right | Not run | 77.1%422 of 547 right | Not run | 3.8% | Not run | run 741 | Not run | Not run | Not run | Not run |
| KnowLine-2B-Gen1 Open · 2.0B | Not run | 93.1%509 of 547 answered, 38 “I don't know” | Not run | 26.5%135 of 509 | Not run | 22.0%38 “I don't know” of 173 not right | Not run | 68.4%374 of 547 right | Not run | 7.3% | Not run | run 1059 | Not run | Not run | Not run | Not run |
| GLiDE 2B Open · 1.9B | Not run | 91.4%500 of 547 answered, 47 “I don't know” | Not run | 26.4%132 of 500 | Not run | 26.3%47 “I don't know” of 179 not right | Not run | 67.3%368 of 547 right | Not run | 12.5% | Not run | run 919 | Not run | Not run | Not run | Not run |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | Not run | 94.1%515 of 547 answered, 32 “I don't know” | Not run | 30.5%157 of 515 | Not run | 16.9%32 “I don't know” of 189 not right | Not run | 65.4%358 of 547 right | Not run | 6.7% | Not run | run 975 | Not run | Not run | Not run | Not run |
| d1-2B Open · 1.9B | Not run | 72.4%396 of 547 answered, 151 “I don't know” | Not run | 24.5%97 of 396 | Not run | 60.9%151 “I don't know” of 248 not right | Not run | 54.7%299 of 547 right | Not run | 26.9% | Not run | run 710 | Not run | Not run | Not run | Not run |
| GLiDE 0.8B Open · 0.8B | Not run | 91.0%498 of 547 answered, 49 “I don't know” | Not run | 35.5%177 of 498 | Not run | 21.7%49 “I don't know” of 226 not right | Not run | 58.7%321 of 547 right | Not run | 14.8% | Not run | run 927 | Not run | Not run | Not run | Not run |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | Not run | 97.8%535 of 547 answered, 12 “I don't know” | Not run | 40.6%217 of 535 | Not run | 5.2%12 “I don't know” of 229 not right | Not run | 58.1%318 of 547 right | Not run | 6.6% | Not run | run 1013 | Not run | Not run | Not run | Not run |
| GLiDE 0.4B Open · 0.4B | Not run | 72.0%394 of 547 answered, 153 “I don't know” | Not run | 34.3%135 of 394 | Not run | 53.1%153 “I don't know” of 288 not right | Not run | 47.3%259 of 547 right | Not run | 23.3% | Not run | run 935 | Not run | Not run | Not run | Not run |
| OneJev1.6-0.8B Open · 0.8B | Not run | 86.5%473 of 547 answered, 74 “I don't know” | Not run | 39.5%187 of 473 | Not run | 28.4%74 “I don't know” of 261 not right | Not run | 52.3%286 of 547 right | Not run | 10.7% | Not run | run 1027 | Not run | Not run | Not run | Not run |
| KnowLine-0.8B-Gen1 Open · 0.8B | Not run | 95.2%521 of 547 answered, 26 “I don't know” | Not run | 41.7%217 of 521 | Not run | 10.7%26 “I don't know” of 243 not right | Not run | 55.6%304 of 547 right | Not run | 5.8% | Not run | run 1003 | Not run | Not run | Not run | Not run |
| GLiClass v4 Edge Open · 0.4B | Not run | 90.5%495 of 547 answered, 52 “I don't know” | Not run | 39.6%196 of 495 | Not run | 21.0%52 “I don't know” of 248 not right | Not run | 54.7%299 of 547 right | Not run | 11.8% | Not run | run 1044 | Not run | Not run | Not run | Not run |
| Strands Decider 2B Open · 2.0B | Not run | 99.6%545 of 547 answered, 2 “I don't know” | Not run | 55.2%301 of 545 | Not run | 0.7%2 “I don't know” of 303 not right | Not run | 44.6%244 of 547 right | Not run | 3.8% | Not run | run 704 | Not run | Not run | Not run | Not run |
| Laya Open · 0.4B | Not run | 84.8%464 of 547 answered, 83 “I don't know” | Not run | 79.1%367 of 464 | Not run | 18.4%83 “I don't know” of 450 not right | Not run | 17.7%97 of 547 right | Not run | 14.2% | Not run | run 728 | Not run | Not run | Not run | Not run |
| GLiNER2.5-multi-Decide Open · 0.3B | Not run | 87.2%477 of 547 answered, 70 “I don't know” | Not run | 76.1%363 of 477 | Not run | 16.2%70 “I don't know” of 433 not right | Not run | 20.8%114 of 547 right | Not run | 13.2% | Not run | run 716 | Not run | Not run | Not run | Not run |
| Laya Multilingual Open · 0.3B | Not run | 62.0%339 of 547 answered, 208 “I don't know” | Not run | 82.6%280 of 339 | Not run | 42.6%208 “I don't know” of 488 not right | Not run | 10.8%59 of 547 right | Not run | 34.7% | Not run | run 734 | Not run | Not run | Not run | Not run |
Our method with “I don't know” in both languages
What is the same
In both languages the sixth option, F, is written “I don't know”, always the last option, never shuffled and never correct, and the instructions get the same added sentence: “If you do not know the answer, choose I don't know.” Both sides are counted on the same 547 kept questions.
What differs
The language of the question and its options (the English translation, as in the accuracy comparison above, against the Turkish original); the one word of the instructions that names the language of the question; and the options heading, “Seçenekler:” against “Options:”. The instructions of the Turkish runs:
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose I don't know.
The instructions of the English runs:
Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.
The runs
The Turkish side is the Hallucination tab's option-word run (the option's word) on all 2,198 Turkish questions, counted on the 547 kept questions only: d1-ultra (run 511), GLiDE · v1.2 (run 525), d1.2-ultra (run 1378), Mammoth-1 (run 519), d3 (run 544), Drex Max (run 542), d3-ultra (run 540), d1.2 (run 1377), Perplexity Decider 27B v1.2 · DGX Spark (run 531), Drex Max v1.1 (run 513), SubQ Snap 1.1 (run 512) and Elephant-1 (run 515). The English side is the first English run with “I don't know” of the section above: KnowLine-31B-Gen1.5 (run 955), Copernicus-31B (run 1170), REAX 1.3 (run 831), KnowLine-31B-Gen2 (run 1329), Karpo Decision 31B (run 1161), Gemini Decision 1 · 20261111 (run 1019), Logits Sigma-1 Duo (run 1231), GLiDE 27B (run 903), d1-ultra (run 449), Sieve 31B v3 (run 1066), Turion Jev 31B (run 1077), Perplexity Decider 27B v1.2 · API · with web (run 644), GLiDE · v1.2 (run 523), d1.2-ultra (run 1363), Mammoth-1 (run 505), d3 (run 793), Drex Max (run 798), Perplexity Decider 27B v1.3 (run 1139), d3-ultra (run 795), d1.2 (run 1361), Perplexity Decider 27B v1.2 · DGX Spark (run 529), Drex Max v1.1 (run 447), Perplexity Decider 27B v1.2 · API (run 643), Jeffan-9B-1211 (run 1197), d1 (run 445), StartLux-Decision-v1.1-9B-Instant (run 1295), Clef (run 785), KnowLine-12B-Gen1 (run 1316), GLiDE 9B (run 911), OneJev-27B (run 780), Jev 1.14 (run 446), Cerno 9B v1.2 (run 1087), KnowLine-9B-Gen1 (run 1179), SubQ Snap 1.1 (run 448), Adjudex 9B v5 (run 1188), Mercury Decide (run 450), Microsoft-Decision-1 (run 451), Elephant-1 (run 501), Open-Jev-27B-v1.1 (run 763), KnowLine-9B-Gen2 (run 1323), celeris-1-decision (run 548), GPT-6 Luna · Decisions API (run 444), Cygnet (run 755), Winnow-12B (run 748), Perplexity Decider Flash v1.3 (run 1128), Valen-Decision-1.5-9B (run 1280), Dexy-Decision-9B-V2 (run 1260), vera-medium-omni (run 1269), StartLux-Decision-v1.1-4B-Instant (run 1083), Clef-Flash (run 660), Verdin-4B-v1.1 (run 1249), Perplexity Decider Flash v1.2 (run 535), KnowLine 4B Gen5.5 (run 963), Chinese Decision Pointer 4B (run 1211), Metask-Jev-4B (run 741), KnowLine-2B-Gen1 (run 1059), GLiDE 2B (run 919), StartLux-Decision-v1.1-2B-Instant (run 975), d1-2B (run 710), GLiDE 0.8B (run 927), StartLux-Decision-v1.1-0.8B-Instant (run 1013), GLiDE 0.4B (run 935), OneJev1.6-0.8B (run 1027), KnowLine-0.8B-Gen1 (run 1003), GLiClass v4 Edge (run 1044), Strands Decider 2B (run 704), Laya (run 728), GLiNER2.5-multi-Decide (run 716) and Laya Multilingual (run 734). First runs only: where a model has more runs, the others are not counted here. No run was made for this comparison.
English only, with no Turkish run with the option written “I don't know”: KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1.2 · API, Jeffan-9B-1211, d1, StartLux-Decision-v1.1-9B-Instant, Clef, KnowLine-12B-Gen1, GLiDE 9B, OneJev-27B, Jev 1.14, Cerno 9B v1.2, KnowLine-9B-Gen1, Adjudex 9B v5, Mercury Decide, Microsoft-Decision-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2, celeris-1-decision, GPT-6 Luna · Decisions API, Cygnet, Winnow-12B, Perplexity Decider Flash v1.3, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Clef-Flash, Verdin-4B-v1.1, Perplexity Decider Flash v1.2, KnowLine 4B Gen5.5, Chinese Decision Pointer 4B, Metask-Jev-4B, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, d1-2B, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1, GLiClass v4 Edge, Strands Decider 2B, Laya, GLiNER2.5-multi-Decide and Laya Multilingual. Their other side and paired columns read Not run; they are in no paired test and no Holm adjustment.
The test
A wrong answer is a valid choice other than “I don't know” that is not right. The paired test is the exact McNemar test on the questions only one of the two runs got wrong, with p adjusted by Holm's method over the 12 models with both runs: a family of its own, separate from every other family on the site. The comparison with “Bilmiyorum” in Turkish stays as published in the section above (its table).
Languages the model cards list
Of the 31 models whose cards or docs we checked, only Elephant-1, Mammoth-1, REAX 1.3, Copernicus-31B and vera-medium-omni name a language: English, by the card's language: en tag. Jev's docs call English its primary language. No card names Turkish. The other 103 English setups had their cards checked only for Turkish, on 16 November 2026, for the Turkish leaderboards' ‡ marks: Metask-Jev-4B and Erk-14B · reasoning off · chat list it.
Each model's card at the revision we ran (local weights) or its maker's model page and docs (APIs), searched for language statements and the Hugging Face language: field, on 2026-11-02. “not stated”: neither names a language.
| Model | Languages listed | Source | Checked |
|---|---|---|---|
| d1-ultra Closed · Efficient | not stated | docs.liquid.ai | 2026-11-02 |
| d1 Closed · Efficient | not stated | docs.liquid.ai | 2026-11-02 |
| SubQ Snap 1.1 Closed · Efficient | not stated | docs.subq.ai | 2026-11-02 |
| Elephant-1 Open · 9.0B | English (the card's language: en tag; its text names no language) | huggingface.co | 2026-11-02 |
| Mammoth-1 Open · 27B | English (the card's language: en tag; its text names no language) | huggingface.co | 2026-11-02 |
| REAX 1.3 Open · 57B | English (the card's language: en tag; its text names no language) | huggingface.co | 2026-11-02 |
| GLiDE v1.2 Open · 27B | not stated | huggingface.co | 2026-11-02 |
| GLiDE API Closed · Efficient | not stated | docs.fastino.ai | 2026-11-02 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | not stated | huggingface.co | 2026-11-02 |
| Perplexity Decider Flash v1.2 Open · 4.0B | not stated | huggingface.co | 2026-11-02 |
| Perplexity Decider Flash v1.3 Open · 4.0B | not stated | huggingface.co | 2026-11-02 |
| Perplexity Decider 27B v1.3 Open · 27B | not stated | huggingface.co | 2026-11-02 |
| Karpo Decision 31B Open · 31B | not stated | huggingface.co | 2026-11-02 |
| Copernicus-31B Open · 31B | English (the card's language: en tag; its text names no language) | huggingface.co | 2026-11-02 |
| Adjudex 9B v5 Open · 9.0B | not stated | huggingface.co | 2026-11-02 |
| Jeffan-9B-1211 Open · 9.0B | not stated | huggingface.co | 2026-11-02 |
| Logits Sigma-1 Duo Open · 57B | not stated | huggingface.co | 2026-11-02 |
| vera-medium-omni Open · 4.2B | English (the card's language: en tag; its text names no language) | huggingface.co | 2026-11-02 |
| Dexy-Decision-9B-V2 Open · 9.0B | not stated | huggingface.co | 2026-11-02 |
| Valen-Decision-1.5-9B Open · 9.0B | not stated | huggingface.co | 2026-11-02 |
| Cerno 9B v1.2 Open · 9.0B | not stated | huggingface.co | 2026-11-02 |
| Jev 1.14 Closed · Efficient | "English is the primary training language and where accuracy is currently best. Other languages, including CJK scripts, are handled but not equally well" | docs.typesafe.ai | 2026-11-02 |
| Microsoft-Decision-1 Closed · Efficient | not stated | openrouter.ai | 2026-11-02 |
| Mercury Decide Closed · Efficient | not stated | openrouter.ai | 2026-11-02 |
| celeris-1-decision Closed · Efficient | not stated | docs.celeris.ai | 2026-11-02 |
| Clef Open · 27B | not stated | huggingface.co | 2026-11-02 |
| GPT-6 Luna Decisions API Closed · Efficient | not stated | developers.openai.com | 2026-11-02 |
| Drex Max v1.1 Closed · Efficient (provisional) | not stated | console.nace.ai | 2026-11-02 |
| Drex Max Open · 27B | not stated | huggingface.co | 2026-11-02 |
| d3 Open · 26B | not stated | huggingface.co | 2026-11-02 |
| d3-ultra Open · 26B | not stated | huggingface.co | 2026-11-02 |