Turkish vs English

The Text tab's reading questions, in Turkish and in English

Accuracy in Turkish and in English · 547 reading questions, the English machine-translated

Share of the 547 kept reading questions answered right: each of the 133 setups' Text tab run and its first English run, ordered by English accuracy · higher is better

  1. Claude Fable 5.2 max · chat97.8% · 98.0%
  2. Gemini 4 Hydrogen high · chat97.1% · 96.7%
  3. GPT-6 Astra low · chat96.7% · 96.3%
  4. GPT-6 Astra high · chat96.9% · 96.2%
  5. GPT-6.1 Terra high · chat96.3% · 96.2%
  6. GPT-6.1 Terra low · chat95.6% · 96.2%
  7. Claude Fable 5.1 high · chat95.4% · 96.0%
  8. Gemini 3.8 Flash high · chat96.9% · 96.0%
  9. Gemini 4 Argon low · chat96.9% · 96.0%
  10. GPT-6 Astra medium · chat96.7% · 96.0%
  11. GPT-6.1 Sol low · chat95.8% · 95.8%
  12. GPT-6 Astra low · chat · 1 per request97.1% · 95.6%
  13. GPT-6.1 Sol high · chat96.2% · 95.6%
  14. Gemini 3.8 Flash low · chat · 1 per request96.2% · 95.2%
  15. GPT-6.1 Sol low · chat · 1 per request95.6% · 95.2%
  16. Claude Opus 5.5 low · chat · 1 per request95.4% · 94.7%
  17. Claude Opus 5.5 high · chat95.6% · 94.5%
  18. Qwen4-Max high · chat94.7% · 94.5%
  19. Gemini 3.8 Flash low · chat95.2% · 94.3%
  20. Gemini 4 Carbon high · chat97.6% · 94.3%
  21. KnowLine-31B-Gen1.594.3% · 93.6%
  22. Grok 4.7 high · chat95.1% · 93.4%
  23. Claude Opus 5.5 low · chat95.1% · 93.1%
  24. Copernicus-31B94.5% · 92.9%
  25. Claude Haiku 5.5 max · chat89.4% · 92.7%
  26. DeepSeek-V4.1-Flash reasoning on · chat · 101794.5% · 92.5%
  27. REAX 1.394.0% · 92.5%
  28. Kimi K3.1 Pro low · chat95.2% · 92.3%
  29. KnowLine-31B-Gen294.5% · 92.3%
  30. Karpo Decision 31B94.9% · 92.1%
  31. Claude Sonnet 5.5 high · chat · 1 per request94.3% · 91.6%
  32. Kimi K3.1 Pro no reasoning · chat94.0% · 91.4%
  33. Qwen3.5-397B-A17B high · chat91.4% · 91.4%
  34. Gemini Decision 1 2026111192.9% · 91.0%
  35. Logits Sigma-1 Duo93.4% · 91.0%
  36. GLiDE 27B91.2% · 90.9%
  37. d1-ultra93.6% · 90.7%
  38. Sieve 31B v393.4% · 90.7%
  39. Turion Jev 31B93.6% · 90.7%
  40. Gemma 4 31B reasoning off · chat92.5% · 90.5%
  41. Perplexity Decider 27B v1.2 API · with web90.7% · 90.5%
  42. GLiDE v1.290.5% · 90.3%
  43. d1.2-ultra90.9% · 90.3%
  44. Mammoth-192.9% · 90.1%
  45. Gemma 4 31B reasoning off · chat · 1 per request90.3% · 89.9%
  46. d391.0% · 89.8%
  47. Drex Max91.4% · 89.6%
  48. GLM-5.3-Flash reasoning on · chat90.1% · 89.6%
  49. Muse Ember max · chat90.9% · 89.6%
  50. Perplexity Decider 27B v1.390.7% · 89.6%
  51. d3-ultra91.4% · 89.4%
  52. d1.288.8% · 89.0%
  53. MiMo-V2.6-Flash high · chat86.8% · 89.0%
  54. Qwen3.8-27B low · chat90.7% · 89.0%
  55. Perplexity Decider 27B v1.2 DGX Spark91.4% · 88.7%
  56. Drex Max v1.190.1% · 88.5%
  57. Perplexity Decider 27B v189.0% · 88.5%
  58. Perplexity Decider 27B v1.2 API91.0% · 88.5%
  59. Inkling-Small high · chat · Jul 202689.4% · 88.3%
  60. d1.188.3% · 88.3%
  61. GLM-5.3 low · chat89.2% · 87.8%
  62. Jeffan-9B-121186.5% · 87.6%
  63. GLM-5.3-Flash reasoning off · chat84.3% · 87.2%
  64. DeepSeek-V4.1-Flash reasoning off · chat · 101789.9% · 87.0%
  65. Qwen3.5-4B reasoning on · chat82.8% · 87.0%
  66. Qwen3.8 Flash no reasoning · chat88.8% · 87.0%
  67. d185.0% · 86.8%
  68. StartLux-Decision-v1.1-9B-Instant83.0% · 86.8%
  69. Clef88.1% · 86.7%
  70. KnowLine-12B-Gen187.6% · 86.7%
  71. GLiDE 9B84.3% · 86.3%
  72. GPT-6 Luna low · chat88.7% · 86.3%
  73. OneJev-27B87.8% · 86.3%
  74. Claude Sonnet 5.5 high · chat87.6% · 86.1%
  75. Jev 1.1487.2% · 86.1%
  76. Cerno 9B v1.283.5% · 85.9%
  77. Drex 2.084.8% · 85.9%
  78. Jev 1.1386.1% · 85.9%
  79. KnowLine-9B-Gen182.4% · 85.9%
  80. SubQ Snap 1.189.2% · 85.9%
  81. SubQ Snap Preview89.9% · 85.9%
  82. Adjudex 9B v583.9% · 85.7%
  83. Mercury Decide87.0% · 85.7%
  84. Microsoft-Decision-176.4% · 85.7%
  85. Elephant-183.5% · 85.4%
  86. Open-Jev-27B-v1.186.5% · 85.4%
  87. KnowLine-9B-Gen282.8% · 85.2%
  88. Qwen3.8-27B stated confidence · reasoning off · chat86.3% · 84.8%
  89. Qwen3.8-27B yes/no scoring84.6% · 84.8%
  90. MiniMax-M3.1-Flash high · chat86.3% · 84.5%
  91. celeris-1-decision87.4% · 84.1%
  92. GPT-6 Luna Decisions API86.7% · 84.1%
  93. Claude Sonnet 5.5 low · chat84.8% · 83.9%
  94. Cygnet85.6% · 83.9%
  95. Qwen3.8-27B vote share · reasoning off · chat85.9% · 83.9%
  96. Winnow-12B85.0% · 83.9%
  97. Perplexity Decider Flash v1.378.8% · 83.7%
  98. Valen-Decision-1.5-9B79.0% · 83.7%
  99. Dexy-Decision-9B-V283.4% · 83.5%
  100. vera-medium-omni79.5% · 83.5%
  101. StartLux-Decision-v1.1-4B-Instant78.4% · 82.8%
  102. Gemma 4 E4B default reasoning · chat80.6% · 82.4%
  103. Clef-Flash77.7% · 82.3%
  104. Verdin-4B-v1.177.5% · 82.3%
  105. Perplexity Decider Flash v1.276.6% · 81.9%
  106. KnowLine 4B Gen5.578.6% · 80.8%
  107. Chinese Decision Pointer 4B75.7% · 80.4%
  108. Qwen3.5-4B reasoning off · chat72.0% · 80.4%
  109. Qwen3-14B reasoning off · chat76.8% · 79.7%
  110. Metask-Jev-4B67.1% · 77.1%
  111. LFM2.5-2.6B default reasoning · chat47.9% · 73.9%
  112. Erk-14B reasoning off · chat75.3% · 72.0%
  113. LFM3-1.7B reasoning off · chat40.2% · 72.9%
  114. KnowLine-2B-Gen158.0% · 71.3%
  115. LFM3-1.7B reasoning on · chat48.1% · 71.1%
  116. LFM2.5-8B-A1B default reasoning · chat27.1% · 70.4%
  117. GLiDE 2B54.3% · 70.0%
  118. StartLux-Decision-v1.1-2B-Instant55.8% · 69.7%
  119. MiniCPM5-2B reasoning off · chat7.1% · 67.6%
  120. Qwen3.5-2B reasoning off · chat36.7% · 67.3%
  121. d1-2B41.9% · 65.6%
  122. GLiDE 0.8B46.1% · 60.7%
  123. StartLux-Decision-v1.1-0.8B-Instant37.3% · 59.6%
  124. GLiDE 0.4B25.8% · 59.4%
  125. OneJev1.6-0.8B42.8% · 59.4%
  126. KnowLine-0.8B-Gen140.2% · 58.9%
  127. GLiClass v4 Edge26.9% · 58.3%
  128. Decider 136.6% · 55.8%
  129. Strands Decider 2B39.1% · 45.5%
  130. Laya20.5% · 22.9%
  131. GLiNER2.5-multi-Decide20.7% · 21.2%
  132. GLiNER2.5 Multi16.8% · 16.3%
  133. Laya Multilingual17.0% · 15.4%

Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Claude Gemini GPT Qwen KnowLine Grok Copernicus DeepSeek REAX Kimi Karpo Logits Fastino Liquid Sieve Turion Jev Gemma Perplexity Mammoth vLLM-SR Drex GLM Muse MiMo Inkling Jeffan StartLux Clef OneJev Jev Cerno SubQ Adjudex Inception Microsoft Elephant Open-Jev MiniMax Celeris Cygnet Winnow Valen Dexy vera Verdin Chinese Decision Pointer Metask Erk MiniCPM GLiClass meraGPT Strands Laya

Right only in Turkish, or only in English?

Each setup's first English run against its Text tab run on the 547 kept questions: the questions only one of the two got right, by the exact McNemar test with p adjusted by Holm's method over all 133 English setups · the longer bar is the language with more right answers

  1. MiniCPM5-2B reasoning off · chat6 · 337 *
  2. LFM2.5-8B-A1B default reasoning · chat27 · 264 *
  3. GLiDE 0.4B38 · 222 *
  4. GLiClass v4 Edge34 · 206 *
  5. LFM3-1.7B reasoning off · chat42 · 221 *
  6. Qwen3.5-2B reasoning off · chat40 · 207 *
  7. LFM2.5-2.6B default reasoning · chat33 · 175 *
  8. d1-2B41 · 171 *
  9. LFM3-1.7B reasoning on · chat40 · 166 *
  10. StartLux-Decision-v1.1-0.8B-Instant39 · 161 *
  11. Decider 147 · 152 *
  12. KnowLine-0.8B-Gen148 · 150 *
  13. OneJev1.6-0.8B57 · 148 *
  14. GLiDE 2B49 · 135 *
  15. KnowLine-2B-Gen139 · 112 *
  16. StartLux-Decision-v1.1-2B-Instant53 · 129 *
  17. GLiDE 0.8B61 · 141 *
  18. Microsoft-Decision-123 · 74 *
  19. Metask-Jev-4B35 · 90 *
  20. Qwen3.5-4B reasoning off · chat49 · 95 *
  21. Gemini 4 Carbon high · chat21 · 3 *
  22. Strands Decider 2B39 · 74
  23. Kimi K3.1 Pro low · chat21 · 5
  24. Perplexity Decider Flash v1.234 · 63
  25. Perplexity Decider Flash v1.333 · 60
  26. d1-ultra24 · 8
  27. SubQ Snap Preview42 · 20
  28. GPT-6 Astra low · chat · 1 per request8 · 0
  29. Claude Haiku 5.5 max · chat12 · 30
  30. Claude Sonnet 5.5 high · chat · 1 per request22 · 7
  31. Clef-Flash30 · 55
  32. Verdin-4B-v1.134 · 60
  33. Valen-Decision-1.5-9B35 · 61
  34. Qwen3.5-4B reasoning on · chat26 · 49
  35. Turion Jev 31B26 · 10
  36. Chinese Decision Pointer 4B37 · 63
  37. Kimi K3.1 Pro no reasoning · chat21 · 7
  38. StartLux-Decision-v1.1-4B-Instant33 · 57
  39. Sieve 31B v325 · 10
  40. StartLux-Decision-v1.1-9B-Instant26 · 47
  41. vera-medium-omni30 · 52
  42. Karpo Decision 31B27 · 12
  43. KnowLine-9B-Gen123 · 42
  44. SubQ Snap 1.141 · 23
  45. DeepSeek-V4.1-Flash reasoning off · chat · 101733 · 17
  46. celeris-1-decision42 · 24
  47. Mammoth-130 · 15
  48. Perplexity Decider 27B v1.2 DGX Spark30 · 15
  49. KnowLine-31B-Gen220 · 8
  50. Grok 4.7 high · chat13 · 4
  51. GLM-5.3-Flash reasoning off · chat23 · 39
  52. Perplexity Decider 27B v1.2 API31 · 17
  53. DeepSeek-V4.1-Flash reasoning on · chat · 101721 · 10
  54. Logits Sigma-1 Duo30 · 17
  55. Claude Opus 5.5 low · chat23 · 12
  56. Gemini Decision 1 2026111120 · 10
  57. Copernicus-31B17 · 8
  58. Erk-14B reasoning off · chat66 · 48
  59. GPT-6 Luna Decisions API42 · 28
  60. Qwen3-14B reasoning off · chat40 · 56
  61. KnowLine-9B-Gen225 · 38
  62. GPT-6 Luna low · chat39 · 26
  63. Gemma 4 31B reasoning off · chat29 · 18
  64. MiMo-V2.6-Flash high · chat25 · 37
  65. Cerno 9B v1.233 · 46
  66. d3-ultra33 · 22
  67. Drex Max29 · 19
  68. GLiDE 9B26 · 37
  69. Qwen3.8-27B low · chat25 · 16
  70. REAX 1.320 · 12
  71. Qwen3.8 Flash no reasoning · chat32 · 22
  72. Qwen3.8-27B vote share · reasoning off · chat39 · 28
  73. Gemini 3.8 Flash high · chat8 · 3
  74. Drex Max v1.127 · 18
  75. Claude Opus 5.5 high · chat12 · 6
  76. KnowLine 4B Gen5.539 · 51
  77. d126 · 36
  78. Elephant-129 · 39
  79. Adjudex 9B v531 · 41
  80. Clef26 · 18
  81. MiniMax-M3.1-Flash high · chat42 · 32
  82. Cygnet35 · 26
  83. GLM-5.3 low · chat29 · 21
  84. d322 · 15
  85. Gemma 4 E4B default reasoning · chat39 · 49
  86. OneJev-27B31 · 23
  87. Laya73 · 86
  88. Muse Ember max · chat24 · 17
  89. Gemini 3.8 Flash low · chat · 1 per request12 · 7
  90. Gemini 4 Argon low · chat12 · 7
  91. Qwen3.8-27B stated confidence · reasoning off · chat34 · 26
  92. GPT-6 Astra medium · chat8 · 4
  93. Laya Multilingual49 · 40
  94. Claude Sonnet 5.5 high · chat40 · 32
  95. GPT-6 Astra high · chat9 · 5
  96. Gemini 3.8 Flash low · chat15 · 10
  97. Jev 1.1423 · 17
  98. Open-Jev-27B-v1.124 · 18
  99. Mercury Decide35 · 28
  100. Perplexity Decider 27B v1.325 · 19
  101. Inkling-Small high · chat · Jul 202630 · 24
  102. Drex 2.027 · 33
  103. Jeffan-9B-121127 · 33
  104. KnowLine-12B-Gen125 · 20
  105. Claude Opus 5.5 low · chat · 1 per request15 · 11
  106. KnowLine-31B-Gen1.515 · 11
  107. Winnow-12B40 · 34
  108. GPT-6.1 Sol high · chat8 · 5
  109. Claude Sonnet 5.5 low · chat32 · 27
  110. Claude Fable 5.1 high · chat9 · 12
  111. GPT-6.1 Terra low · chat10 · 13
  112. Gemini 4 Hydrogen high · chat5 · 3
  113. GPT-6 Astra low · chat6 · 4
  114. d1.2-ultra25 · 22
  115. GLM-5.3-Flash reasoning on · chat29 · 26
  116. Perplexity Decider 27B v129 · 26
  117. GPT-6.1 Sol low · chat · 1 per request8 · 6
  118. GLiNER2.5 Multi33 · 30
  119. GLiNER2.5-multi-Decide39 · 42
  120. GLiDE 27B21 · 19
  121. Gemma 4 31B reasoning off · chat · 1 per request26 · 24
  122. Claude Fable 5.2 max · chat4 · 5
  123. d1.126 · 26
  124. d1.230 · 31
  125. Dexy-Decision-9B-V236 · 37
  126. GLiDE v1.222 · 21
  127. GPT-6.1 Sol low · chat6 · 6
  128. GPT-6.1 Terra high · chat10 · 9
  129. Jev 1.1319 · 18
  130. Perplexity Decider 27B v1.2 API · with web21 · 20
  131. Qwen3.5-397B-A17B high · chat13 · 13
  132. Qwen3.8-27B yes/no scoring22 · 23
  133. Qwen4-Max high · chat14 · 13

Right only in Turkish, the upper bar Right only in English, the lower bar, striped Numbers on the right: Turkish · English. MiniCPM Liquid Fastino GLiClass Qwen StartLux meraGPT KnowLine OneJev Microsoft Metask Gemini Strands Kimi Perplexity SubQ GPT Claude Clef Verdin Valen Turion Jev Chinese Decision Pointer Sieve vera Karpo DeepSeek Celeris Mammoth Grok GLM Logits Copernicus Erk Gemma MiMo Cerno vLLM-SR Drex REAX Elephant Adjudex MiniMax Cygnet Laya Muse Jev Open-Jev Inception Inkling Jeffan Winnow Dexy * A significant difference (Holm-adjusted p below 0.05): MiniCPM5-2B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat, GLiDE 0.4B, GLiClass v4 Edge, LFM3-1.7B · reasoning off · chat, Qwen3.5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, d1-2B, LFM3-1.7B · reasoning on · chat, StartLux-Decision-v1.1-0.8B-Instant, Decider 1, KnowLine-0.8B-Gen1, OneJev1.6-0.8B, GLiDE 2B, KnowLine-2B-Gen1, StartLux-Decision-v1.1-2B-Instant, GLiDE 0.8B, Microsoft-Decision-1, Metask-Jev-4B, Qwen3.5-4B · reasoning off · chat and Gemini 4 Carbon · high · chat (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p = 0.00011, p = 0.018 and p = 0.031). The other 112: p = 0.14, p = 0.28, p = 0.46, p = 0.73, p = 0.76, p = 0.76, p = 0.83, p = 0.83, p = 0.85, p = 0.91, p = 0.97, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1.

KnowLine-31B-Gen1.5 against each decision model in English

In English, against KnowLine-31B-Gen1.5, 11 are too close to call and 63 are behind it (of the 74 other decision models, after Holm's correction).

ModelRunfirst English runOnly KnowLine-31B-Gen1.5 rightthe other not rightOnly the other rightKnowLine-31B-Gen1.5 not rightpexact McNemarHolm pover the 74Verdictby Holm p, 0.05
Copernicus-31B Open · 31Brun 1168730.340.57too close to call
REAX 1.3 Open · 57Brun 8291480.290.57too close to call
Karpo Decision 31B Open · 31Brun 11591680.150.45too close to call
KnowLine-31B-Gen2 Open · 31Brun 1327810.0390.17too close to call
Gemini Decision 1 20261111 Closed · Efficientrun 100924100.0240.17too close to call
Logits Sigma-1 Duo Open · 57Brun 122924100.0240.17too close to call
GLiDE 27B Open · 27Brun 90127120.0240.17too close to call
d1-ultra Closed · Efficientrun 41126100.0110.091too close to call
Perplexity Decider 27B v1.2 API · with web Open · 27Brun 63928110.00950.085too close to call
GLiDE v1.2 Open · 27Brun 52130120.00790.079too close to call
d1.2-ultra Closed · Efficientrun 135928100.00510.056too close to call
Mammoth-1 Open · 27Brun 5032890.00260.032behind KnowLine-31B-Gen1.5
Sieve 31B v3 Open · 31Brun 10642150.00250.032behind KnowLine-31B-Gen1.5
Turion Jev 31B Open · 31Brun 10752040.00150.022behind KnowLine-31B-Gen1.5
d3-ultra Open · 26Brun 55034110.000820.012behind KnowLine-31B-Gen1.5
d3 Open · 26Brun 5542980.000750.012behind KnowLine-31B-Gen1.5
Drex Max Open · 27Brun 5523190.000680.012behind KnowLine-31B-Gen1.5
Perplexity Decider 27B v1.3 Open · 27Brun 11373080.000470.0085behind KnowLine-31B-Gen1.5
d1.2 Closed · Efficientrun 135735100.000250.0047behind KnowLine-31B-Gen1.5
Drex Max v1.1 Closed · Efficient (provisional)run 3993810< 0.00010.0012behind KnowLine-31B-Gen1.5
Jeffan-9B-1211 Open · 9.0Brun 11954714< 0.00010.00057behind KnowLine-31B-Gen1.5
d1.1 Closed · Efficientrun 565389< 0.00010.00055behind KnowLine-31B-Gen1.5
Perplexity Decider 27B v1.2 DGX Spark Open · 27Brun 527336< 0.00010.00033behind KnowLine-31B-Gen1.5
Perplexity Decider 27B v1 Open · 27Brun 626346< 0.00010.00021behind KnowLine-31B-Gen1.5
Perplexity Decider 27B v1.2 API Open · 27Brun 637346< 0.00010.00021behind KnowLine-31B-Gen1.5
StartLux-Decision-v1.1-9B-Instant Open · 9.0Brun 12934912< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Cerno 9B v1.2 Open · 9.0Brun 10855816< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
GLiDE 9B Open · 7.9Brun 9095212< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
KnowLine-12B-Gen1 Open · 12Brun 13134810< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
d1 Closed · Efficientrun 430458< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
KnowLine-9B-Gen1 Open · 9.0Brun 11775412< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Adjudex 9B v5 Open · 9.0Brun 11865613< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Clef Open · 27Brun 613457< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
OneJev-27B Open · 27Brun 778477< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
KnowLine-9B-Gen2 Open · 9.0Brun 13215812< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Microsoft-Decision-1 Closed · Efficientrun 412529< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Drex 2.0 Open · 35B MoE · 3.0B activerun 566508< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Jev 1.14 Closed · Efficientrun 410487< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
SubQ Snap 1.1 Closed · Efficientrun 413497< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
SubQ Snap Preview Closed · Efficientrun 595497< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Elephant-1 Open · 9.0Brun 499549< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Mercury Decide Closed · Efficientrun 398507< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Jev 1.13 Closed · Efficientrun 564486< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Open-Jev-27B-v1.1 Open · 27Brun 760527< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Perplexity Decider Flash v1.3 Open · 4.0Brun 11266713< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Winnow-12B Open · 12Brun 7466512< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Valen-Decision-1.5-9B Open · 9.0Brun 12786612< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Cygnet Open · 12Brun 7536411< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
celeris-1-decision Closed · Efficientrun 5466210< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Dexy-Decision-9B-V2 Open · 9.0Brun 12586712< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
StartLux-Decision-v1.1-4B-Instant Open · 4.0Brun 10817415< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
GPT-6 Luna Decisions API Closed · Efficientrun 397619< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
vera-medium-omni Open · 4.2Brun 12676611< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Perplexity Decider Flash v1.2 Open · 4.0Brun 5337612< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Verdin-4B-v1.1 Open · 4.0Brun 1247719< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Clef-Flash Open · 9.0Brun 614697< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
KnowLine 4B Gen5.5 Open · 4.0Brun 9618111< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Chinese Decision Pointer 4B Open · 4.0Brun 12098311< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Metask-Jev-4B Open · 4.5Brun 739977< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
KnowLine-2B-Gen1 Open · 2.0Brun 105713210< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
GLiDE 2B Open · 1.9Brun 9171389< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
StartLux-Decision-v1.1-2B-Instant Open · 2.0Brun 9731409< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
d1-2B Open · 1.9Brun 7081596< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
GLiDE 0.8B Open · 0.8Brun 9251866< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8Brun 10111948< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
GLiDE 0.4B Open · 0.4Brun 9331958< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
GLiClass v4 Edge Open · 0.4Brun 10422029< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
KnowLine-0.8B-Gen1 Open · 0.8Brun 10011977< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
OneJev1.6-0.8B Open · 0.8Brun 10251914< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Decider 1 Closed · Efficientrun 5962169< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Strands Decider 2B Open · 2.0Brun 7022707< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Laya Open · 0.4Brun 72639811< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
GLiNER2.5-multi-Decide Open · 0.3Brun 7144026< 0.0001< 0.0001behind KnowLine-31B-Gen1.5
Laya Multilingual Open · 0.3Brun 7324324< 0.0001< 0.0001behind KnowLine-31B-Gen1.5

Ordered by p, highest first. On the 547 kept reading questions, KnowLine-31B-Gen1.5's first English run (run 953) against each other decision model's first English run: the exact McNemar test on the questions only one of the two got right, with p adjusted by Holm's method over the 74 decision models. Verdict: too close to call when the Holm-adjusted p is 0.05 or more; otherwise behind KnowLine-31B-Gen1.5 when KnowLine-31B-Gen1.5 has more questions only it got right, ahead of it when the other model has more. Each model's name links its model page's English section, as in the tables above; Paired test compares each model with itself in Turkish.

Limit: a model translated the questions and no person checked them (Our method).

More about this page

Turkish vs English

ModelAccuracyAnswered with “I don't know”Wrong of answers with “I don't know”Non-hallucination with “I don't know”Turkish with “Bilmiyorum”wrong of answers, lower is better; answered and non-hallucination under itCostLatencyRank by accuracyLanguages the model card listsand its source
Turkishhigher is betterEnglishhigher is betterDifferencepoints higher in English than in Turkish (negative: higher in Turkish)Turkish(right + wrong) / 547; higher is betterEnglish(right + wrong) / 547; higher is betterTurkishwrong / (right + wrong); lower is betterEnglishwrong / (right + wrong); lower is betterTurkish“I don't know” / (“I don't know” + wrong); higher is betterEnglish“I don't know” / (“I don't know” + wrong); higher is betterDifferencenon-hallucination points higher in English than in Turkish (negative: higher in Turkish)Turkish$ per 1,000 questionsper 1M questions; lower is betterEnglish$ per 1,000 questionsper 1M questions; lower is betterTurkishmedian seconds per request; lower is betterEnglishmedian seconds per request; lower is betterTurkishamong the 133; lower is betterEnglishamong the 133; lower is better
Claude Fable 5.2 max · chat Closed · Flagship97.8%98.0%+0.2Not rundecision models only99.3%543 of 547 answeredNot rundecision models only1.8%10 of 543Not rundecision models only28.6%4 “I don't know” of 14 not rightNot rundecision models onlyQueuedits run with the option has not started yet53.38153,381subscription, priced as the API52.94252,942subscription, priced as the API81.116 questions per request, 1 in flight74.016 questions per request, 1 in flight11other languages, not Turkishplatform.claude.com
Gemini 4 Carbon high · chat Closed · Flagship97.6%94.3%-3.3 *Not rundecision models only99.3%543 of 547 answeredNot rundecision models only2.2%12 of 543Not rundecision models only25.0%4 “I don't know” of 16 not rightNot rundecision models only2.0%11 of 543answered 99.3%, non-hallucination 26.7%31.30931,309subscription, priced as the API38.05338,053subscription, priced as the API217.716 questions per request, 1 in flight287.816 questions per request, 1 in flight219not stateddeepmind.google
Gemini 4 Hydrogen high · chat Closed · Workhorse97.1%96.7%-0.4Not rundecision models only99.3%543 of 547 answeredNot rundecision models only2.9%16 of 543Not rundecision models only20.0%4 “I don't know” of 20 not rightNot rundecision models only1.8%10 of 543answered 99.3%, non-hallucination 28.6%4.8064,806subscription, priced as the API4.9274,927subscription, priced as the API31.716 questions per request, 1 in flight28.416 questions per request, 1 in flight32not stateddeepmind.google
GPT-6 Astra low · chat · 1 per request Closed · Flagship97.1%95.6%-1.5Not rundecision models only99.3%543 of 547 answeredNot rundecision models only4.2%23 of 543Not rundecision models only14.8%4 “I don't know” of 27 not rightNot rundecision models only3.5%19 of 543answered 99.3%, non-hallucination 17.4%26.73826,738subscription, priced as the API28.65328,653subscription, priced as the API6.411 question per request, 1 in flight6.431 question per request, 1 in flight412not stateddevelopers.openai.com
Gemini 3.8 Flash high · chat Closed · Workhorse96.9%96.0%-0.9Not rundecision models only99.6%545 of 547 answeredNot rundecision models only3.3%18 of 545Not rundecision models only10.0%2 “I don't know” of 20 not rightNot rundecision models only2.4%13 of 545answered 99.6%, non-hallucination 13.3%2.4192,419at list prices3.6953,695at list prices53.716 questions per request, 4 in flight104.616 questions per request, 4 in flight57not stateddeepmind.google
Gemini 4 Argon low · chat Closed · Flagship96.9%96.0%-0.9Not rundecision models only99.1%542 of 547 answeredNot rundecision models only4.1%22 of 542Not rundecision models only18.5%5 “I don't know” of 27 not rightNot rundecision models only2.4%13 of 543answered 99.3%, non-hallucination 23.5%1.7861,786at list prices1.9001,900at list prices5.4616 questions per request, 1 in flight6.0316 questions per request, 1 in flight68not stateddeepmind.google
GPT-6 Astra high · chat Closed · Flagship96.9%96.2%-0.7Not rundecision models only99.3%543 of 547 answeredNot rundecision models only3.1%17 of 543Not rundecision models only19.0%4 “I don't know” of 21 not rightNot rundecision models only3.1%17 of 543answered 99.3%, non-hallucination 19.0%6.0666,066subscription, priced as the API6.3346,334subscription, priced as the API21.716 questions per request, 1 in flight15.616 questions per request, 1 in flight74not stateddevelopers.openai.com
GPT-6 Astra low · chat Closed · Flagship96.7%96.3%-0.4Not rundecision models only99.3%543 of 547 answeredNot rundecision models only3.3%18 of 543Not rundecision models only18.2%4 “I don't know” of 22 not rightNot rundecision models only2.6%14 of 543answered 99.3%, non-hallucination 22.2%5.3785,378subscription, priced as the API5.9185,918subscription, priced as the API13.016 questions per request, 1 in flight, measured on a sample (200 questions)11.516 questions per request, 1 in flight83not stateddevelopers.openai.com
GPT-6 Astra medium · chat Closed · Flagship96.7%96.0%-0.7Not rundecision models only99.3%543 of 547 answeredNot rundecision models only3.7%20 of 543Not rundecision models only16.7%4 “I don't know” of 24 not rightNot rundecision models only2.9%16 of 543answered 99.3%, non-hallucination 20.0%5.4475,447subscription, priced as the API4.6964,696subscription, priced as the API15.316 questions per request, 1 in flight10.716 questions per request, 1 in flight99not stateddevelopers.openai.com
GPT-6.1 Terra high · chat Closed · Workhorse96.3%96.2%-0.2Not rundecision models only99.3%543 of 547 answeredNot rundecision models only3.7%20 of 543Not rundecision models only16.7%4 “I don't know” of 24 not rightNot rundecision models only3.5%19 of 542answered 99.1%, non-hallucination 20.8%0.515515subscription, priced as the API0.509509subscription, priced as the API7.1016 questions per request, 1 in flight7.9416 questions per request, 1 in flight105not stateddevelopers.openai.com
Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse96.2%95.2%-0.9Not rundecision models only99.6%545 of 547 answeredNot rundecision models only4.4%24 of 545Not rundecision models only7.7%2 “I don't know” of 26 not rightNot rundecision models only3.3%18 of 544answered 99.5%, non-hallucination 14.3%0.963963at list prices0.858858at list prices2.721 question per request, 1 in flight2.851 question per request, 1 in flight1114not stateddeepmind.google
GPT-6.1 Sol high · chat Closed · Workhorse96.2%95.6%-0.5Not rundecision models only99.3%543 of 547 answeredNot rundecision models only3.7%20 of 543Not rundecision models only16.7%4 “I don't know” of 24 not rightNot rundecision models only2.9%16 of 543answered 99.3%, non-hallucination 20.0%1.6221,622subscription, priced as the API1.4271,427subscription, priced as the API24.716 questions per request, 1 in flight16.916 questions per request, 1 in flight1213not stateddevelopers.openai.com
GPT-6.1 Sol low · chat Closed · Workhorse95.8%95.8%+0.0Not rundecision models only99.3%543 of 547 answeredNot rundecision models only4.2%23 of 543Not rundecision models only14.8%4 “I don't know” of 27 not rightNot rundecision models only3.3%18 of 543answered 99.3%, non-hallucination 18.2%1.1301,130subscription, priced as the API1.2191,219subscription, priced as the API13.316 questions per request, 1 in flight10.316 questions per request, 1 in flight1311not stateddevelopers.openai.com
Claude Opus 5.5 high · chat Closed · Flagship95.6%94.5%-1.1Not rundecision models only99.6%545 of 547 answeredNot rundecision models only5.3%29 of 545Not rundecision models only6.5%2 “I don't know” of 31 not rightNot rundecision models only4.2%23 of 544answered 99.5%, non-hallucination 11.5%6.5096,509subscription, priced as the API4.5194,519subscription, priced as the API22.116 questions per request, 1 in flight11.316 questions per request, 1 in flight1417other languages, not Turkishplatform.claude.com
GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse95.6%95.2%-0.4Not rundecision models only99.3%543 of 547 answeredNot rundecision models only5.0%27 of 543Not rundecision models only12.9%4 “I don't know” of 31 not rightNot rundecision models only3.5%19 of 543answered 99.3%, non-hallucination 17.4%5.5695,569subscription, priced as the API6.1136,113subscription, priced as the API6.611 question per request, 1 in flight5.591 question per request, 1 in flight1515not stateddevelopers.openai.com
GPT-6.1 Terra low · chat Closed · Workhorse95.6%96.2%+0.5Not rundecision models only99.3%543 of 547 answeredNot rundecision models only4.1%22 of 543Not rundecision models only15.4%4 “I don't know” of 26 not rightNot rundecision models only4.2%23 of 543answered 99.3%, non-hallucination 14.8%0.426426subscription, priced as the API0.491491subscription, priced as the API4.2416 questions per request, 1 in flight7.2916 questions per request, 1 in flight166not stateddevelopers.openai.com
Claude Fable 5.1 high · chat Closed · Flagship95.4%96.0%+0.5Not rundecision models only99.6%545 of 547 answeredNot rundecision models only4.6%25 of 545Not rundecision models only7.4%2 “I don't know” of 27 not rightNot rundecision models only4.2%23 of 543answered 99.3%, non-hallucination 14.8%17.04217,042subscription, priced as the API11.53111,531subscription, priced as the API30.316 questions per request, 1 in flight15.616 questions per request, 1 in flight1710not statedplatform.claude.com
Claude Opus 5.5 low · chat · 1 per request Closed · Flagship95.4%94.7%-0.7Not rundecision models only100.0%547 of 547 answeredNot rundecision models only5.7%31 of 547Not rundecision models only0.0%0 “I don't know” of 31 not rightNot rundecision models only4.6%25 of 546answered 99.8%, non-hallucination 3.8%11.61011,610subscription, priced as the API10.09410,094subscription, priced as the API3.711 question per request, 1 in flight3.501 question per request, 1 in flight1816other languages, not Turkishplatform.claude.com
Gemini 3.8 Flash low · chat Closed · Workhorse95.2%94.3%-0.9Not rundecision models only99.8%546 of 547 answeredNot rundecision models only5.1%28 of 546Not rundecision models only3.4%1 “I don't know” of 29 not rightNot rundecision models only4.4%24 of 544answered 99.5%, non-hallucination 11.1%0.269269at list prices0.288288at list prices2.1916 questions per request, 1 in flight4.1916 questions per request, 1 in flight1920not stateddeepmind.google
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active95.2%92.3%-2.9Not rundecision models only99.5%544 of 547 answeredNot rundecision models only7.0%38 of 544Not rundecision models only7.3%3 “I don't know” of 41 not rightNot rundecision models only6.6%36 of 544answered 99.5%, non-hallucination 7.7%2.0482,048at list prices0.984984at list prices44.916 questions per request, 4 in flight16.616 questions per request, 4 in flight2028not statedhuggingface.co
Claude Opus 5.5 low · chat Closed · Flagship95.1%93.1%-2.0Not rundecision models only99.6%545 of 547 answeredNot rundecision models only7.5%41 of 545Not rundecision models only4.7%2 “I don't know” of 43 not rightNot rundecision models only5.3%29 of 545answered 99.6%, non-hallucination 6.5%4.9314,931subscription, priced as the API1.0871,087subscription, priced as the API13.416 questions per request, 1 in flight, measured on a sample (200 questions)7.6416 questions per request, 1 in flight2123other languages, not Turkishplatform.claude.com
Grok 4.7 high · chat Closed · Workhorse95.1%93.4%-1.6Not rundecision models only97.8%535 of 547 answeredNot rundecision models only5.8%31 of 535Not rundecision models only27.9%12 “I don't know” of 43 not rightNot rundecision models only3.9%21 of 539answered 98.5%, non-hallucination 27.6%14.29214,292subscription, priced as the API6.8516,851subscription, priced as the API401.916 questions per request, 4 in flight197.216 questions per request, 4 in flight2222not statedmedia.x.ai
Karpo Decision 31B Open · 31B94.9%92.1%-2.7Not rundecision models only100.0%547 of 547 answeredNot rundecision models only7.7%42 of 547Not rundecision models only0.0%0 “I don't know” of 42 not rightNot rundecision models only5.5%30 of 546answered 99.8%, non-hallucination 3.2%0.105105from GPU time0.145145from GPU time0.1591 question per request, 1 in flight0.2071 question per request, 1 in flight2330not statedhuggingface.co
Qwen4-Max high · chat Closed · Workhorse94.7%94.5%-0.2Not rundecision models only100.0%547 of 547 answeredNot rundecision models only5.3%29 of 547Not rundecision models only0.0%0 “I don't know” of 29 not rightNot rundecision models only5.1%28 of 546answered 99.8%, non-hallucination 3.4%2.5332,533at list prices1.1191,119at list prices91.516 questions per request, 4 in flight29.516 questions per request, 4 in flight2418not statedopenrouter.ai
Copernicus-31B Open · 31B94.5%92.9%-1.6Not rundecision models only100.0%547 of 547 answeredNot rundecision models only7.1%39 of 547Not rundecision models only0.0%0 “I don't know” of 39 not rightNot rundecision models only6.0%33 of 546answered 99.8%, non-hallucination 2.9%0.134134from GPU time0.142142from GPU time0.1841 question per request, 1 in flight0.1881 question per request, 1 in flight2524English (the card's language: en tag; its text names no language)huggingface.co
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active94.5%92.5%-2.0Not rundecision models only99.5%544 of 547 answeredNot rundecision models only7.2%39 of 544Not rundecision models only7.1%3 “I don't know” of 42 not rightNot rundecision models only7.0%38 of 546answered 99.8%, non-hallucination 2.6%1.2781,278at the maker's list price0.421421at the maker's list price486.916 questions per request, 8 in flight189.416 questions per request, 8 in flight2626not statedhuggingface.co
KnowLine-31B-Gen2 Open · 31B94.5%92.3%-2.2Not rundecision models only100.0%547 of 547 answeredNot rundecision models only7.3%40 of 547Not rundecision models only0.0%0 “I don't know” of 40 not rightNot rundecision models only5.3%29 of 547answered 100.0%, non-hallucination 0.0%0.120120from GPU time0.127127from GPU time0.1681 question per request, 1 in flight0.1711 question per request, 1 in flight2729other languages, not Turkishhuggingface.co
Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse94.3%91.6%-2.7Not rundecision models only100.0%547 of 547 answeredNot rundecision models only6.9%38 of 547Not rundecision models only0.0%0 “I don't know” of 38 not rightNot rundecision models only6.6%36 of 546answered 99.8%, non-hallucination 2.7%7.7487,748subscription, priced as the API5.5135,513subscription, priced as the API4.641 question per request, 1 in flight3.021 question per request, 1 in flight2831other languages, not Turkishplatform.claude.com
KnowLine-31B-Gen1.5 Open · 31B94.3%93.6%-0.7Not rundecision models only100.0%547 of 547 answeredNot rundecision models only6.8%37 of 547Not rundecision models only0.0%0 “I don't know” of 37 not rightNot rundecision models only6.2%34 of 547answered 100.0%, non-hallucination 0.0%No list priceNo list price1.051 question per request, 1 in flight1.121 question per request, 1 in flight2921other languages, not Turkishhuggingface.co
Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active94.0%91.4%-2.6Not rundecision models only99.3%543 of 547 answeredNot rundecision models only9.0%49 of 543Not rundecision models only7.5%4 “I don't know” of 53 not rightNot rundecision models only7.2%39 of 544answered 99.5%, non-hallucination 7.1%0.853853at list prices0.735735at list prices5.7516 questions per request, 4 in flight6.3116 questions per request, 4 in flight3032not statedhuggingface.co
REAX 1.3 Open · 57B94.0%92.5%-1.5Not rundecision models only99.5%544 of 547 answeredNot rundecision models only7.7%42 of 544Not rundecision models only6.7%3 “I don't know” of 45 not rightNot rundecision models only6.4%35 of 546answered 99.8%, non-hallucination 2.8%0.185185from GPU time0.169169from GPU time0.2501 question per request, 1 in flight0.2291 question per request, 1 in flight3127English (the card's language: en tag; its text names no language)huggingface.co
d1-ultra Closed · Efficient93.6%90.7%-2.995.1%520 of 547 answered90.7%496 of 547 answered5.4%28 of 5207.9%39 of 49649.1%27 “I don't know” of 55 not right56.7%51 “I don't know” of 90 not right+7.65.0%26 of 517answered 94.5%, non-hallucination 53.6%0.01414.34at list prices0.01413.98at list prices0.2231 question per request, 1 in flight0.2201 question per request, 1 in flight3237not stateddocs.liquid.ai
Turion Jev 31B Open · 31B93.6%90.7%-2.9Not rundecision models only99.3%543 of 547 answeredNot rundecision models only7.9%43 of 543Not rundecision models only8.5%4 “I don't know” of 47 not rightNot rundecision models only6.1%33 of 545answered 99.6%, non-hallucination 5.7%No list priceNo list price0.4051 question per request, 1 in flight0.4451 question per request, 1 in flight3338not statedhuggingface.co
Logits Sigma-1 Duo Open · 57B93.4%91.0%-2.4Not rundecision models only99.6%545 of 547 answeredNot rundecision models only8.1%44 of 545Not rundecision models only4.3%2 “I don't know” of 46 not rightNot rundecision models only6.6%36 of 547answered 100.0%, non-hallucination 0.0%0.181181from GPU time0.191191from GPU time0.2501 question per request, 1 in flight0.2631 question per request, 1 in flight3434not statedhuggingface.co
Sieve 31B v3 Open · 31B93.4%90.7%-2.7Not rundecision models only99.8%546 of 547 answeredNot rundecision models only8.2%45 of 546Not rundecision models only2.2%1 “I don't know” of 46 not rightNot rundecision models only6.0%33 of 547answered 100.0%, non-hallucination 0.0%No list priceNo list price1.001 question per request, 1 in flight1.061 question per request, 1 in flight3539other languages, not Turkishhuggingface.co
Gemini Decision 1 20261111 Closed · Efficient92.9%91.0%-1.8Not rundecision models only99.1%542 of 547 answeredNot rundecision models only9.2%50 of 542Not rundecision models only9.1%5 “I don't know” of 55 not rightNot rundecision models only7.6%41 of 542answered 99.1%, non-hallucination 10.9%0.03737.19at list prices0.04039.51at list prices0.2271 question per request, 1 in flight0.2351 question per request, 1 in flight3635not statedaistudio.google.com
Mammoth-1 Open · 27B92.9%90.1%-2.795.4%522 of 547 answered93.2%510 of 547 answered6.1%32 of 5227.1%36 of 51043.9%25 “I don't know” of 57 not right50.7%37 “I don't know” of 73 not right+6.86.3%33 of 523answered 95.6%, non-hallucination 42.1%0.04039.62at the maker's list price0.04040.07at the maker's list price0.1691 question per request, 1 in flight0.1891 question per request, 1 in flight3744English (the card's language: en tag; its text names no language)huggingface.co
Gemma 4 31B reasoning off · chat Open · 31B92.5%90.5%-2.0Not rundecision models only99.6%545 of 547 answeredNot rundecision models only9.7%53 of 545Not rundecision models only3.6%2 “I don't know” of 55 not rightNot rundecision models only7.7%42 of 545answered 99.6%, non-hallucination 4.5%0.04848.22at a host's list price for the same weights0.05352.81at a host's list price for the same weights3.9716 questions per request, 1 in flight3.1916 questions per request, 1 in flight3840not statedhuggingface.co
d3-ultra Open · 26B91.4%89.4%-2.099.6%545 of 547 answered99.1%542 of 547 answered9.4%51 of 54510.3%56 of 5423.8%2 “I don't know” of 53 not right8.2%5 “I don't know” of 61 not right+4.48.4%46 of 546answered 99.8%, non-hallucination 2.1%0.216216from GPU time0.106106from GPU time0.3331 question per request, 1 in flight0.1491 question per request, 1 in flight3951not statedhuggingface.co
Drex Max Open · 27B91.4%89.6%-1.899.8%546 of 547 answered99.6%545 of 547 answered9.0%49 of 54611.0%60 of 5452.0%1 “I don't know” of 50 not right3.2%2 “I don't know” of 62 not right+1.29.7%53 of 545answered 99.6%, non-hallucination 3.6%0.02323.29at the maker's list price0.02423.56at the maker's list price0.0821 question per request, 1 in flight0.0811 question per request, 1 in flight4047not statedhuggingface.co
Perplexity Decider 27B v1.2 DGX Spark Open · 27B91.4%88.7%-2.799.8%546 of 547 answered99.6%545 of 547 answered9.3%51 of 54611.7%64 of 5451.9%1 “I don't know” of 52 not right3.0%2 “I don't know” of 66 not right+1.19.1%50 of 547answered 100.0%, non-hallucination 0.0%0.0087.76at the maker's list price0.0087.85at the maker's list price0.5801 question per request, 1 in flight0.1261 question per request, 1 in flight4155not statedhuggingface.co
Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active91.4%91.4%+0.0Not rundecision models only99.5%544 of 547 answeredNot rundecision models only8.3%45 of 544Not rundecision models only6.2%3 “I don't know” of 48 not rightNot rundecision models only7.9%43 of 543answered 99.3%, non-hallucination 8.5%2.4112,411at list prices1.7421,742at list prices251.316 questions per request, 4 in flight162.916 questions per request, 4 in flight4233not statedhuggingface.co
GLiDE 27B Open · 27B91.2%90.9%-0.4Not rundecision models only96.0%525 of 547 answeredNot rundecision models only6.9%36 of 525Not rundecision models only37.9%22 “I don't know” of 58 not rightNot rundecision models only7.2%39 of 538answered 98.4%, non-hallucination 18.8%No list priceNo list price0.5421 question per request, 1 in flight0.5451 question per request, 1 in flight4336not statedhuggingface.co
d3 Open · 26B91.0%89.8%-1.3100.0%547 of 547 answered99.8%546 of 547 answered9.5%52 of 54710.6%58 of 5460.0%0 “I don't know” of 52 not right1.7%1 “I don't know” of 59 not right+1.79.7%53 of 547answered 100.0%, non-hallucination 0.0%0.07070.21from GPU time0.05555.35from GPU time0.0921 question per request, 1 in flight0.0741 question per request, 1 in flight4446not statedhuggingface.co
Perplexity Decider 27B v1.2 API Open · 27B91.0%88.5%-2.6Not rundecision models only99.5%544 of 547 answeredNot rundecision models only11.6%63 of 544Not rundecision models only4.5%3 “I don't know” of 66 not rightNot rundecision models only9.3%51 of 547answered 100.0%, non-hallucination 0.0%0.0087.70at list prices0.0087.79at list prices0.2541 question per request, 1 in flight0.2561 question per request, 1 in flight4556not statedhuggingface.co
d1.2-ultra Closed · Efficient90.9%90.3%-0.589.0%487 of 547 answered80.1%438 of 547 answered3.7%18 of 4873.0%13 of 43876.9%60 “I don't know” of 78 not right89.3%109 “I don't know” of 122 not right+12.43.9%19 of 486answered 88.8%, non-hallucination 76.2%0.01414.34at list prices0.01413.98at list prices0.2201 question per request, 1 in flight0.2211 question per request, 1 in flight4642not statedwww.liquid.ai
Muse Ember max · chat Closed · Efficient90.9%89.6%-1.3Not rundecision models only99.5%544 of 547 answeredNot rundecision models only9.9%54 of 544Not rundecision models only5.3%3 “I don't know” of 57 not rightNot rundecision models only8.7%47 of 543answered 99.3%, non-hallucination 7.8%0.650650at list prices0.245245at list prices30.916 questions per request, 4 in flight11.616 questions per request, 4 in flight4748not stateddev.meta.ai
Perplexity Decider 27B v1.2 API · with web Open · 27B90.7%90.5%-0.2Not rundecision models only98.2%537 of 547 answeredNot rundecision models only10.4%56 of 537Not rundecision models only15.2%10 “I don't know” of 66 not rightNot rundecision models only9.4%51 of 544answered 99.5%, non-hallucination 5.6%≤ 1.034≤ 1,034at list prices≤ 1.036≤ 1,036at list prices0.4821 question per request, 1 in flight0.4841 question per request, 1 in flight4841not statedhuggingface.co
Perplexity Decider 27B v1.3 Open · 27B90.7%89.6%-1.1Not rundecision models only99.1%542 of 547 answeredNot rundecision models only9.6%52 of 542Not rundecision models only8.8%5 “I don't know” of 57 not rightNot rundecision models only9.5%52 of 547answered 100.0%, non-hallucination 0.0%0.08181.18from GPU time0.08382.63from GPU time0.1101 question per request, 1 in flight0.1151 question per request, 1 in flight4949not statedhuggingface.co
Qwen3.8-27B low · chat Open · 27B90.7%89.0%-1.6Not rundecision models only99.6%545 of 547 answeredNot rundecision models only9.7%53 of 545Not rundecision models only3.6%2 “I don't know” of 55 not rightNot rundecision models only8.8%48 of 546answered 99.8%, non-hallucination 2.0%3.0953,095at the maker's list price0.853853at the maker's list price648.416 questions per request, 4 in flight71.716 questions per request, 4 in flight5052not statedhuggingface.co
GLiDE v1.2 Open · 27B90.5%90.3%-0.298.4%538 of 547 answered94.5%517 of 547 answered8.7%47 of 5387.0%36 of 51716.1%9 “I don't know” of 56 not right45.5%30 “I don't know” of 66 not right+29.48.7%47 of 541answered 98.9%, non-hallucination 11.3%0.01414.37at the maker's list price0.01514.55at the maker's list price0.4131 question per request, 1 in flight0.1041 question per request, 1 in flight5143not statedhuggingface.co
Gemma 4 31B reasoning off · chat · 1 per request Open · 31B90.3%89.9%-0.4Not rundecision models only100.0%547 of 547 answeredNot rundecision models only10.8%59 of 547Not rundecision models only0.0%0 “I don't know” of 59 not rightNot rundecision models only9.5%52 of 546answered 99.8%, non-hallucination 1.9%0.06868.19at a host's list price for the same weights0.07070.17at a host's list price for the same weights0.2981 question per request, 1 in flight0.2991 question per request, 1 in flight5245not statedhuggingface.co
Drex Max v1.1 Closed · Efficient (provisional)90.1%88.5%-1.696.7%529 of 547 answered92.1%504 of 547 answered9.1%48 of 5298.1%41 of 50427.3%18 “I don't know” of 66 not right51.2%43 “I don't know” of 84 not right+23.99.5%51 of 538answered 98.4%, non-hallucination 15.0%0.02323.11at list prices0.02323.38at list prices0.3131 question per request, 1 in flight0.3351 question per request, 1 in flight5357not statedconsole.nace.ai
GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active90.1%89.6%-0.5Not rundecision models only99.8%546 of 547 answeredNot rundecision models only11.2%61 of 546Not rundecision models only1.6%1 “I don't know” of 62 not rightNot rundecision models only10.1%55 of 545answered 99.6%, non-hallucination 3.5%0.543543at the maker's list price0.08181.38at the maker's list price102.616 questions per request, 8 in flight33.116 questions per request, 8 in flight5450other languages, not Turkishhuggingface.co
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active89.9%87.0%-2.9Not rundecision models only99.3%543 of 547 answeredNot rundecision models only12.9%70 of 543Not rundecision models only5.4%4 “I don't know” of 74 not rightNot rundecision models only10.4%57 of 547answered 100.0%, non-hallucination 0.0%0.09898.47at the maker's list price0.07979.31at the maker's list price5.0816 questions per request, 1 in flight6.5916 questions per request, 1 in flight5564not statedhuggingface.co
SubQ Snap Preview Closed · Efficient89.9%85.9%-4.0Not rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot run0.01919.41at list prices0.01919.48at list prices0.3891 question per request, 1 in flight0.3811 question per request, 1 in flight5676not stateddocs.subq.ai
Claude Haiku 5.5 max · chat Closed · Efficient89.4%92.7%+3.3Not rundecision models only96.3%527 of 547 answeredNot rundecision models only8.9%47 of 527Not rundecision models only7.8%4 “I don't know” of 51 not rightNot rundecision models only5.5%29 of 525answered 96.0%, non-hallucination 17.1%2.9652,965at list prices1.2611,261at list prices417.016 questions per request, 4 in flight148.916 questions per request, 4 in flight5725other languages, not Turkishplatform.claude.com
Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active89.4%88.3%-1.1Not rundecision models only100.0%547 of 547 answeredNot rundecision models only11.0%60 of 547Not rundecision models only0.0%0 “I don't know” of 60 not rightNot rundecision models only11.3%62 of 547answered 100.0%, non-hallucination 0.0%1.1371,137at list prices0.425425at list prices97.716 questions per request, 4 in flight25.316 questions per request, 4 in flight5859other languages, not Turkishhuggingface.co
GLM-5.3 low · chat Open · Workhorse89.2%87.8%-1.5Not rundecision models only99.6%545 of 547 answeredNot rundecision models only10.3%56 of 545Not rundecision models only3.4%2 “I don't know” of 58 not rightNot rundecision models only11.2%61 of 544answered 99.5%, non-hallucination 4.7%0.814814at list prices0.408408at list prices17.116 questions per request, 4 in flight6.8416 questions per request, 4 in flight5961other languages, not Turkishhuggingface.co
SubQ Snap 1.1 Closed · Efficient89.2%85.9%-3.396.5%528 of 547 answered96.2%526 of 547 answered8.5%45 of 52812.0%63 of 52629.7%19 “I don't know” of 64 not right25.0%21 “I don't know” of 84 not right-4.79.7%51 of 528answered 96.5%, non-hallucination 27.1%0.01515.41at list prices0.01615.59at list prices0.3781 question per request, 1 in flight0.3811 question per request, 1 in flight6077not stateddocs.subq.ai
Perplexity Decider 27B v1 Open · 27B89.0%88.5%-0.5Not rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot run0.01615.54at list prices0.01615.59at list prices0.2161 question per request, 1 in flight0.2471 question per request, 1 in flight6158not statedhuggingface.co
d1.2 Closed · Efficient88.8%89.0%+0.291.6%501 of 547 answered89.2%488 of 547 answered8.4%42 of 5017.6%37 of 48852.3%46 “I don't know” of 88 not right61.5%59 “I don't know” of 96 not right+9.27.1%35 of 494answered 90.3%, non-hallucination 60.2%0.0076.83at list prices0.0076.66at list prices0.2131 question per request, 1 in flight0.2191 question per request, 1 in flight6253not statedwww.liquid.ai
Qwen3.8 Flash no reasoning · chat Closed · Efficient88.8%87.0%-1.8Not rundecision models only99.3%543 of 547 answeredNot rundecision models only12.3%67 of 543Not rundecision models only5.6%4 “I don't know” of 71 not rightNot rundecision models only10.1%55 of 544answered 99.5%, non-hallucination 5.2%0.04747.13at list prices0.05049.69at list prices10.016 questions per request, 4 in flight10.116 questions per request, 4 in flight6365not statedhuggingface.co
GPT-6 Luna low · chat Closed · Efficient88.7%86.3%-2.4Not rundecision models only99.1%542 of 547 answeredNot rundecision models only12.7%69 of 542Not rundecision models only6.8%5 “I don't know” of 74 not rightNot rundecision models only11.9%65 of 545answered 99.6%, non-hallucination 3.0%0.04848.01subscription, priced as the API0.05857.70subscription, priced as the API7.7116 questions per request, 1 in flight7.7216 questions per request, 1 in flight6471not stateddevelopers.openai.com
d1.1 Closed · Efficient88.3%88.3%+0.0Not rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot run0.0076.83at list prices0.0076.66at list prices0.1951 question per request, 1 in flight0.1731 question per request, 1 in flight6560not stateddocs.liquid.ai
Clef Open · 27B88.1%86.7%-1.5Not rundecision models only99.8%546 of 547 answeredNot rundecision models only13.7%75 of 546Not rundecision models only1.3%1 “I don't know” of 76 not rightNot rundecision models only12.3%67 of 546answered 99.8%, non-hallucination 1.5%0.110110at the maker's list price0.111111at the maker's list price0.2051 question per request, 1 in flight0.2201 question per request, 1 in flight6669not statedhuggingface.co
OneJev-27B Open · 27B87.8%86.3%-1.5Not rundecision models only99.5%544 of 547 answeredNot rundecision models only13.2%72 of 544Not rundecision models only4.0%3 “I don't know” of 75 not rightNot rundecision models only11.5%63 of 546answered 99.8%, non-hallucination 1.6%0.09999.09from GPU time0.106106from GPU time0.1251 question per request, 1 in flight0.1301 question per request, 1 in flight6772not statedhuggingface.co
Claude Sonnet 5.5 high · chat Closed · Workhorse87.6%86.1%-1.5Not rundecision models only99.5%544 of 547 answeredNot rundecision models only10.8%59 of 544Not rundecision models only4.8%3 “I don't know” of 62 not rightNot rundecision models only9.7%53 of 545answered 99.6%, non-hallucination 3.6%3.6543,654subscription, priced as the API1.9981,998subscription, priced as the API20.516 questions per request, 1 in flight, measured on a sample (200 questions)3.8516 questions per request, 1 in flight6874other languages, not Turkishplatform.claude.com
KnowLine-12B-Gen1 Open · 12B87.6%86.7%-0.9Not rundecision models only99.8%546 of 547 answeredNot rundecision models only13.9%76 of 546Not rundecision models only1.3%1 “I don't know” of 77 not rightNot rundecision models only12.2%67 of 547answered 100.0%, non-hallucination 0.0%0.05252.28from GPU time0.05756.86from GPU time0.0711 question per request, 1 in flight0.0781 question per request, 1 in flight6970other languages, not Turkishhuggingface.co
celeris-1-decision Closed · Efficient87.4%84.1%-3.3Not rundecision models only83.2%455 of 547 answeredNot rundecision models only8.1%37 of 455Not rundecision models only71.3%92 “I don't know” of 129 not rightNot rundecision models onlyNot run0.01515.28at list prices0.01616.09at list prices0.4011 question per request, 1 in flight0.3901 question per request, 1 in flight7091not stateddocs.celeris.ai
Jev 1.14 Closed · Efficient87.2%86.1%-1.1Not rundecision models only99.5%544 of 547 answeredNot rundecision models only13.2%72 of 544Not rundecision models only4.0%3 “I don't know” of 75 not rightNot rundecision models only12.7%69 of 544answered 99.5%, non-hallucination 4.2%0.02928.84at list prices0.02726.64at list prices0.2111 question per request, 1 in flight0.1811 question per request, 1 in flight7175"English is the primary training language and where accuracy is currently best. Other languages, including CJK scripts, are handled but not equally well"docs.typesafe.ai
Mercury Decide Closed · Efficient87.0%85.7%-1.3Not rundecision models only98.0%536 of 547 answeredNot rundecision models only13.2%71 of 536Not rundecision models only13.4%11 “I don't know” of 82 not rightNot rundecision models only13.2%71 of 539answered 98.5%, non-hallucination 10.1%0.0087.71at list prices0.0087.79at list prices0.3131 question per request, 1 in flight0.3101 question per request, 1 in flight7282not statedopenrouter.ai
MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active86.8%89.0%+2.2Not rundecision models only99.8%546 of 547 answeredNot rundecision models only10.6%58 of 546Not rundecision models only1.7%1 “I don't know” of 59 not rightNot rundecision models only12.7%69 of 544answered 99.5%, non-hallucination 4.2%2.2192,219at list prices0.298298at list prices21.41 question per request, 32 in flight6.671 question per request, 8 in flight7354other languages, not Turkishhuggingface.co
GPT-6 Luna Decisions API Closed · Efficient86.7%84.1%-2.6Not rundecision models only98.2%537 of 547 answeredNot rundecision models only17.9%96 of 537Not rundecision models only9.4%10 “I don't know” of 106 not rightNot rundecision models only14.4%78 of 541answered 98.9%, non-hallucination 7.1%0.04040.44at list prices0.04141.09at list prices0.1911 question per request, 4 in flight0.3231 question per request, 4 in flight7492not stateddevelopers.openai.com
Jeffan-9B-1211 Open · 9.0B86.5%87.6%+1.1Not rundecision models only100.0%547 of 547 answeredNot rundecision models only13.0%71 of 547Not rundecision models only0.0%0 “I don't know” of 71 not rightNot rundecision models only13.1%71 of 544answered 99.5%, non-hallucination 4.1%0.04444.29from GPU time0.04645.85from GPU time0.0641 question per request, 1 in flight0.0671 question per request, 1 in flight7562not statedhuggingface.co
Open-Jev-27B-v1.1 Open · 27B86.5%85.4%-1.1Not rundecision models only97.1%531 of 547 answeredNot rundecision models only14.3%76 of 531Not rundecision models only17.4%16 “I don't know” of 92 not rightNot rundecision models only14.4%78 of 541answered 98.9%, non-hallucination 7.1%0.271271from GPU time0.351351from GPU time0.3861 question per request, 1 in flight0.4951 question per request, 1 in flight7685not statedhuggingface.co
MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active86.3%84.5%-1.8Not rundecision models only99.5%544 of 547 answeredNot rundecision models only15.6%85 of 544Not rundecision models only3.4%3 “I don't know” of 88 not rightNot rundecision models only16.5%90 of 545answered 99.6%, non-hallucination 1.1%0.814814at list prices0.193193at list prices112.616 questions per request, 4 in flight26.716 questions per request, 4 in flight7790not statedhuggingface.co
Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B86.3%84.8%-1.5Not rundecision models only99.5%544 of 547 answeredNot rundecision models only14.3%78 of 544Not rundecision models only3.7%3 “I don't know” of 81 not rightNot rundecision models only13.3%72 of 543answered 99.3%, non-hallucination 5.3%0.234234at the maker's list price0.212212at the maker's list price0.4331 question per request, 1 in flight0.3901 question per request, 1 in flight7888not statedhuggingface.co
Jev 1.13 Closed · Efficient86.1%85.9%-0.2Not rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot run0.02928.85at list prices0.02726.64at list prices0.2431 question per request, 1 in flight0.1731 question per request, 1 in flight7978not stateddocs.typesafe.ai
Qwen3.8-27B vote share · reasoning off · chat Open · 27B85.9%83.9%-2.0Not rundecision models only99.5%544 of 547 answeredNot rundecision models only15.4%84 of 544Not rundecision models only3.4%3 “I don't know” of 87 not rightNot rundecision models only13.8%75 of 543answered 99.3%, non-hallucination 5.1%0.825825at the maker's list price0.645645at the maker's list price1.621 question per request, 1 in flight1.701 question per request, 1 in flight8093not statedhuggingface.co
Cygnet Open · 12B85.6%83.9%-1.6Not rundecision models only98.2%537 of 547 answeredNot rundecision models only14.7%79 of 537Not rundecision models only11.2%10 “I don't know” of 89 not rightNot rundecision models only14.9%81 of 542answered 99.1%, non-hallucination 5.8%0.03939.39at a host's list price for the same weights0.04141.42at a host's list price for the same weights0.0361 question per request, 1 in flight0.0431 question per request, 1 in flight8194not statedhuggingface.co
d1 Closed · Efficient85.0%86.8%+1.8Not rundecision models only99.6%545 of 547 answeredNot rundecision models only12.3%67 of 545Not rundecision models only2.9%2 “I don't know” of 69 not rightNot rundecision models only16.3%89 of 546answered 99.8%, non-hallucination 1.1%0.0077.43free tier, at the list price0.0076.66at list prices0.3051 question per request, 1 in flight0.1801 question per request, 1 in flight8267not stateddocs.liquid.ai
Winnow-12B Open · 12B85.0%83.9%-1.1Not rundecision models only97.3%532 of 547 answeredNot rundecision models only16.0%85 of 532Not rundecision models only15.0%15 “I don't know” of 100 not rightNot rundecision models only15.8%85 of 539answered 98.5%, non-hallucination 8.6%0.04141.40from GPU time0.05049.59from GPU time0.0591 question per request, 1 in flight0.0701 question per request, 1 in flight8395not statedhuggingface.co
Claude Sonnet 5.5 low · chat Closed · Workhorse84.8%83.9%-0.9Not rundecision models only99.3%543 of 547 answeredNot rundecision models only15.8%86 of 543Not rundecision models only4.4%4 “I don't know” of 90 not rightNot rundecision models only15.1%82 of 542answered 99.1%, non-hallucination 5.7%1.8961,896subscription, priced as the API0.354354subscription, priced as the API3.8716 questions per request, 1 in flight, measured on a sample (200 questions)3.6816 questions per request, 1 in flight8496other languages, not Turkishplatform.claude.com
Drex 2.0 Open · 35B MoE · 3.0B active84.8%85.9%+1.1Not rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot run0.01919.15at list prices0.01919.42at list prices0.3251 question per request, 1 in flight0.2831 question per request, 1 in flight8579not statedhuggingface.co
Qwen3.8-27B yes/no scoring Open · 27B84.6%84.8%+0.2Not rundecision models onlyNot runyes/no scoring cannot offer itNot rundecision models onlyNot runyes/no scoring cannot offer itNot rundecision models onlyNot runyes/no scoring cannot offer itNot rundecision models onlyNot runyes/no scoring cannot offer it0.704704at the maker's list price0.741741at the maker's list price0.2801 question per request, 1 in flight0.4131 question per request, 1 in flight8689not statedhuggingface.co
GLiDE 9B Open · 7.9B84.3%86.3%+2.0Not rundecision models only97.4%533 of 547 answeredNot rundecision models only13.1%70 of 533Not rundecision models only16.7%14 “I don't know” of 84 not rightNot rundecision models only15.4%83 of 538answered 98.4%, non-hallucination 9.8%No list priceNo list price0.2071 question per request, 1 in flight0.2071 question per request, 1 in flight8773not statedhuggingface.co
GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active84.3%87.2%+2.9Not rundecision models only99.3%543 of 547 answeredNot rundecision models only12.3%67 of 543Not rundecision models only5.6%4 “I don't know” of 71 not rightNot rundecision models only15.1%82 of 544answered 99.5%, non-hallucination 3.5%0.04747.28at the maker's list price0.04646.07at the maker's list price1.0716 questions per request, 1 in flight3.3716 questions per request, 1 in flight8863other languages, not Turkishhuggingface.co
Adjudex 9B v5 Open · 9.0B83.9%85.7%+1.8Not rundecision models only100.0%547 of 547 answeredNot rundecision models only14.1%77 of 547Not rundecision models only0.0%0 “I don't know” of 77 not rightNot rundecision models only16.3%89 of 547answered 100.0%, non-hallucination 0.0%0.02626.30from GPU time0.02524.93from GPU time0.0381 question per request, 1 in flight0.0341 question per request, 1 in flight8983not statedhuggingface.co
Cerno 9B v1.2 Open · 9.0B83.5%85.9%+2.4Not rundecision models only99.6%545 of 547 answeredNot rundecision models only14.1%77 of 545Not rundecision models only2.5%2 “I don't know” of 79 not rightNot rundecision models only16.6%91 of 547answered 100.0%, non-hallucination 0.0%0.03636.02from GPU timeNo list price0.0511 question per request, 1 in flight0.2111 question per request, 1 in flight9080not statedhuggingface.co
Elephant-1 Open · 9.0B83.5%85.4%+1.895.6%523 of 547 answered95.8%524 of 547 answered15.7%82 of 52312.8%67 of 52422.6%24 “I don't know” of 106 not right25.6%23 “I don't know” of 90 not right+2.915.4%82 of 532answered 97.3%, non-hallucination 15.5%0.01413.87at the maker's list price0.01414.02at the maker's list price0.0741 question per request, 1 in flight0.0701 question per request, 1 in flight9186English (the card's language: en tag; its text names no language)huggingface.co
Dexy-Decision-9B-V2 Open · 9.0B83.4%83.5%+0.2Not rundecision models only99.8%546 of 547 answeredNot rundecision models only16.5%90 of 546Not rundecision models only1.1%1 “I don't know” of 91 not rightNot rundecision models only16.5%90 of 546answered 99.8%, non-hallucination 1.1%0.02323.03from GPU time0.02525.41from GPU time0.0311 question per request, 1 in flight0.0361 question per request, 1 in flight9299not statedhuggingface.co
StartLux-Decision-v1.1-9B-Instant Open · 9.0B83.0%86.8%+3.8Not rundecision models only99.6%545 of 547 answeredNot rundecision models only13.6%74 of 545Not rundecision models only2.6%2 “I don't know” of 76 not rightNot rundecision models only17.4%95 of 546answered 99.8%, non-hallucination 1.0%No list priceNo list price0.1541 question per request, 1 in flight0.1561 question per request, 1 in flight9368other languages, not Turkishhuggingface.co
KnowLine-9B-Gen2 Open · 9.0B82.8%85.2%+2.4Not rundecision models only99.8%546 of 547 answeredNot rundecision models only14.3%78 of 546Not rundecision models only1.3%1 “I don't know” of 79 not rightNot rundecision models only17.4%95 of 547answered 100.0%, non-hallucination 0.0%0.03535.40from GPU time0.03939.48from GPU time0.0481 question per request, 1 in flight0.0561 question per request, 1 in flight9487other languages, not Turkishhuggingface.co
Qwen3.5-4B reasoning on · chat Open · 4.0B82.8%87.0%+4.2Not rundecision models only99.6%545 of 547 answeredNot rundecision models only15.2%83 of 545Not rundecision models only2.4%2 “I don't know” of 85 not rightNot rundecision models only16.2%88 of 542answered 99.1%, non-hallucination 2.2%2.9982,998from GPU time1.3901,390from GPU time140.31 question per request, 32 in flight58.31 question per request, 32 in flight9566not statedhuggingface.co
KnowLine-9B-Gen1 Open · 9.0B82.4%85.9%+3.5Not rundecision models only99.8%546 of 547 answeredNot rundecision models only13.9%76 of 546Not rundecision models only1.3%1 “I don't know” of 77 not rightNot rundecision models only17.9%98 of 547answered 100.0%, non-hallucination 0.0%0.04140.69from GPU time0.04241.55from GPU time0.0571 question per request, 1 in flight0.0591 question per request, 1 in flight9681other languages, not Turkishhuggingface.co
Gemma 4 E4B default reasoning · chat Open · 8.0B80.6%82.4%+1.8Not rundecision models only97.8%535 of 547 answeredNot rundecision models only17.2%92 of 535Not rundecision models only9.8%10 “I don't know” of 102 not rightNot rundecision models only18.4%96 of 522answered 95.4%, non-hallucination 20.7%0.307307from GPU time0.199199from GPU time13.41 question per request, 32 in flight8.391 question per request, 32 in flight97102not statedhuggingface.co
vera-medium-omni Open · 4.2B79.5%83.5%+4.0Not rundecision models only98.7%540 of 547 answeredNot rundecision models only16.1%87 of 540Not rundecision models only7.4%7 “I don't know” of 94 not rightNot rundecision models only20.8%113 of 542answered 99.1%, non-hallucination 4.2%0.02120.72from GPU time0.03736.63from GPU time0.0291 question per request, 1 in flight0.0541 question per request, 1 in flight98100English (the card's language: en tag; its text names no language)huggingface.co
Valen-Decision-1.5-9B Open · 9.0B79.0%83.7%+4.8Not rundecision models only99.8%546 of 547 answeredNot rundecision models only15.4%84 of 546Not rundecision models only1.2%1 “I don't know” of 85 not rightNot rundecision models only20.1%110 of 547answered 100.0%, non-hallucination 0.0%0.04040.49from GPU time0.04140.55from GPU time0.0591 question per request, 1 in flight0.0591 question per request, 1 in flight9997not statedhuggingface.co
Perplexity Decider Flash v1.3 Open · 4.0B78.8%83.7%+4.9Not rundecision models only98.7%540 of 547 answeredNot rundecision models only16.7%90 of 540Not rundecision models only7.2%7 “I don't know” of 97 not rightNot rundecision models only20.4%110 of 538answered 98.4%, non-hallucination 7.6%0.02524.69from GPU time0.02323.17from GPU time0.0331 question per request, 1 in flight0.0331 question per request, 1 in flight10098not statedhuggingface.co
KnowLine 4B Gen5.5 Open · 4.0B78.6%80.8%+2.2Not rundecision models only99.8%546 of 547 answeredNot rundecision models only19.2%105 of 546Not rundecision models only0.9%1 “I don't know” of 106 not rightNot rundecision models only21.7%118 of 544answered 99.5%, non-hallucination 2.5%No list priceNo list price0.0991 question per request, 1 in flight0.0991 question per request, 1 in flight101106other languages, not Turkishhuggingface.co
StartLux-Decision-v1.1-4B-Instant Open · 4.0B78.4%82.8%+4.4Not rundecision models only100.0%547 of 547 answeredNot rundecision models only17.2%94 of 547Not rundecision models only0.0%0 “I don't know” of 94 not rightNot rundecision models only21.7%118 of 545answered 99.6%, non-hallucination 1.7%No list priceNo list price0.0741 question per request, 1 in flight0.0751 question per request, 1 in flight102101other languages, not Turkishhuggingface.co
Clef-Flash Open · 9.0B77.7%82.3%+4.6Not rundecision models only99.1%542 of 547 answeredNot rundecision models only16.6%90 of 542Not rundecision models only5.3%5 “I don't know” of 95 not rightNot rundecision models only21.5%115 of 535answered 97.8%, non-hallucination 9.4%0.01717.37at the maker's list price0.01817.53at the maker's list price0.0751 question per request, 1 in flight0.1401 question per request, 1 in flight103103not statedhuggingface.co
Verdin-4B-v1.1 Open · 4.0B77.5%82.3%+4.8Not rundecision models only99.6%545 of 547 answeredNot rundecision models only17.2%94 of 545Not rundecision models only2.1%2 “I don't know” of 96 not rightNot rundecision models only23.3%127 of 546answered 99.8%, non-hallucination 0.8%0.03434.21from GPU time0.03434.28from GPU time0.0491 question per request, 1 in flight0.0521 question per request, 1 in flight104104other languages, not Turkishhuggingface.co
Qwen3-14B reasoning off · chat Open · 15B76.8%79.7%+2.9Not rundecision models only99.6%545 of 547 answeredNot rundecision models only19.8%108 of 545Not rundecision models only1.8%2 “I don't know” of 110 not rightNot rundecision models only22.3%122 of 546answered 99.8%, non-hallucination 0.8%0.03232.31at a host's list price for the same weights0.03131.17at a host's list price for the same weights2.8516 questions per request, 1 in flight2.8616 questions per request, 1 in flight105109not statedhuggingface.co
Perplexity Decider Flash v1.2 Open · 4.0B76.6%81.9%+5.3Not rundecision models only99.1%542 of 547 answeredNot rundecision models only18.5%100 of 542Not rundecision models only4.8%5 “I don't know” of 105 not rightNot rundecision models only24.6%133 of 541answered 98.9%, non-hallucination 4.3%0.0043.88at the maker's list price0.0043.93at the maker's list price0.1321 question per request, 1 in flight0.0371 question per request, 1 in flight106105not statedhuggingface.co
Microsoft-Decision-1 Closed · Efficient76.4%85.7%+9.3 *Not rundecision models only99.5%544 of 547 answeredNot rundecision models only14.2%77 of 544Not rundecision models only3.8%3 “I don't know” of 80 not rightNot rundecision models only16.1%88 of 545answered 99.6%, non-hallucination 2.2%0.01413.82at list prices0.01414.01at list prices0.2861 question per request, 1 in flight0.3151 question per request, 1 in flight10784not statedopenrouter.ai
Chinese Decision Pointer 4B Open · 4.0B75.7%80.4%+4.8Not rundecision models only99.3%543 of 547 answeredNot rundecision models only19.2%104 of 543Not rundecision models only3.7%4 “I don't know” of 108 not rightNot rundecision models only23.8%129 of 543answered 99.3%, non-hallucination 3.0%0.02524.58from GPU time0.02727.21from GPU time0.0351 question per request, 1 in flight0.0381 question per request, 1 in flight108107other languages, not Turkishhuggingface.co
Erk-14B reasoning off · chat Open · 14B75.3%72.0%-3.3Not rundecision models only99.5%544 of 547 answeredNot rundecision models only25.9%141 of 544Not rundecision models only2.1%3 “I don't know” of 144 not rightNot rundecision models only23.6%129 of 546answered 99.8%, non-hallucination 0.8%0.106106from GPU time0.123123from GPU time2.3916 questions per request, 1 in flight2.8616 questions per request, 1 in flight109112Turkish listedhuggingface.co
Qwen3.5-4B reasoning off · chat Open · 4.0B72.0%80.4%+8.4 *Not rundecision models only99.3%543 of 547 answeredNot rundecision models only19.5%106 of 543Not rundecision models only3.6%4 “I don't know” of 110 not rightNot rundecision models only26.0%140 of 538answered 98.4%, non-hallucination 3.4%0.894894from GPU time0.309309from GPU time32.21 question per request, 32 in flight9.741 question per request, 32 in flight110108not statedhuggingface.co
Metask-Jev-4B Open · 4.5B67.1%77.1%+10.1 *Not rundecision models only99.6%545 of 547 answeredNot rundecision models only22.6%123 of 545Not rundecision models only1.6%2 “I don't know” of 125 not rightNot rundecision models only34.3%187 of 545answered 99.6%, non-hallucination 1.1%0.06362.82from GPU time0.05959.11from GPU time0.0901 question per request, 1 in flight0.0851 question per request, 1 in flight111110Turkish listedhuggingface.co
KnowLine-2B-Gen1 Open · 2.0B58.0%71.3%+13.3 *Not rundecision models only93.1%509 of 547 answeredNot rundecision models only26.5%135 of 509Not rundecision models only22.0%38 “I don't know” of 173 not rightNot rundecision models onlyNot runNo list priceNo list price0.0621 question per request, 1 in flight0.0621 question per request, 1 in flight112114other languages, not Turkishhuggingface.co
StartLux-Decision-v1.1-2B-Instant Open · 2.0B55.8%69.7%+13.9 *Not rundecision models only94.1%515 of 547 answeredNot rundecision models only30.5%157 of 515Not rundecision models only16.9%32 “I don't know” of 189 not rightNot rundecision models only45.7%235 of 514answered 94.0%, non-hallucination 12.3%No list priceNo list price0.0401 question per request, 1 in flight0.0401 question per request, 1 in flight113118other languages, not Turkishhuggingface.co
GLiDE 2B Open · 1.9B54.3%70.0%+15.7 *Not rundecision models only91.4%500 of 547 answeredNot rundecision models only26.4%132 of 500Not rundecision models only26.3%47 “I don't know” of 179 not rightNot rundecision models only43.5%209 of 481answered 87.9%, non-hallucination 24.0%No list priceNo list price0.0341 question per request, 1 in flight0.0351 question per request, 1 in flight114117not statedhuggingface.co
LFM3-1.7B reasoning on · chat Open · 1.7B48.1%71.1%historical paired baseline, run 1061+23.0 *Not rundecision models only95.2%521 of 547 answeredNot rundecision models only27.1%141 of 521Not rundecision models only15.6%26 “I don't know” of 167 not rightNot rundecision models only47.6%210 of 441answered 80.6%, non-hallucination 33.5%1.2771,277at the maker's list price0.387387at the maker's list price24.21 question per request, 32 in flight5.241 question per request, 32 in flight115115other languages, not Turkishhuggingface.co
LFM2.5-2.6B default reasoning · chat Open · 2.7B47.9%73.9%+26.0 *Not rundecision models only95.6%523 of 547 answeredNot rundecision models only21.6%113 of 523Not rundecision models only15.7%21 “I don't know” of 134 not rightNot rundecision models only46.2%229 of 496answered 90.7%, non-hallucination 15.8%2.7612,761at the maker's list price0.964964at the maker's list price51.21 question per request, 32 in flight15.01 question per request, 32 in flight116111other languages, not Turkishhuggingface.co
GLiDE 0.8B Open · 0.8B46.1%60.7%+14.6 *Not rundecision models only91.0%498 of 547 answeredNot rundecision models only35.5%177 of 498Not rundecision models only21.7%49 “I don't know” of 226 not rightNot rundecision models only48.6%230 of 473answered 86.5%, non-hallucination 24.3%No list priceNo list price0.0221 question per request, 1 in flight0.0231 question per request, 1 in flight117122not statedhuggingface.co
OneJev1.6-0.8B Open · 0.8B42.8%59.4%+16.6 *Not rundecision models only86.5%473 of 547 answeredNot rundecision models only39.5%187 of 473Not rundecision models only28.4%74 “I don't know” of 261 not rightNot rundecision models only58.2%253 of 435answered 79.5%, non-hallucination 30.7%No list priceNo list price0.0171 question per request, 1 in flight0.0181 question per request, 1 in flight118124not statedhuggingface.co
d1-2B Open · 1.9B41.9%65.6%+23.8 *Not rundecision models only72.4%396 of 547 answeredNot rundecision models only24.5%97 of 396Not rundecision models only60.9%151 “I don't know” of 248 not rightNot rundecision models only53.6%207 of 386answered 70.6%, non-hallucination 43.8%0.01313.17from GPU time0.0088.08from GPU time0.0191 question per request, 1 in flight0.0111 question per request, 1 in flight119121other languages, not Turkishhuggingface.co
KnowLine-0.8B-Gen1 Open · 0.8B40.2%58.9%+18.6 *Not rundecision models only95.2%521 of 547 answeredNot rundecision models only41.7%217 of 521Not rundecision models only10.7%26 “I don't know” of 243 not rightNot rundecision models only61.1%325 of 532answered 97.3%, non-hallucination 4.4%No list priceNo list price0.0331 question per request, 1 in flight0.0331 question per request, 1 in flight120126other languages, not Turkishhuggingface.co
LFM3-1.7B reasoning off · chat Open · 1.7B40.2%72.9%historical paired baseline, run 1223+32.7 *Not rundecision models only96.7%529 of 547 answeredNot rundecision models only25.9%137 of 529Not rundecision models only11.6%18 “I don't know” of 155 not rightNot rundecision models only55.6%173 of 311answered 56.9%, non-hallucination 55.4%0.646646at the maker's list price0.355355at the maker's list price4.901 question per request, 32 in flight4.691 question per request, 32 in flight121113other languages, not Turkishhuggingface.co
Strands Decider 2B Open · 2.0B39.1%45.5%+6.4Not rundecision models only99.6%545 of 547 answeredNot rundecision models only55.2%301 of 545Not rundecision models only0.7%2 “I don't know” of 303 not rightNot rundecision models only60.3%327 of 542answered 99.1%, non-hallucination 1.5%0.02020.12from GPU time0.01818.15from GPU time0.0261 question per request, 1 in flight0.0261 question per request, 1 in flight122129not statedhuggingface.co
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B37.3%59.6%+22.3 *Not rundecision models only97.8%535 of 547 answeredNot rundecision models only40.6%217 of 535Not rundecision models only5.2%12 “I don't know” of 229 not rightNot rundecision models only61.0%320 of 525answered 96.0%, non-hallucination 6.4%No list priceNo list price0.0281 question per request, 1 in flight0.0281 question per request, 1 in flight123123other languages, not Turkishhuggingface.co
Qwen3.5-2B reasoning off · chat Open · 2.0B36.7%67.3%+30.5 *Not rundecision models only96.3%527 of 547 answeredNot rundecision models only29.6%156 of 527Not rundecision models only10.3%18 “I don't know” of 174 not rightNot rundecision models only61.2%279 of 456answered 83.4%, non-hallucination 16.5%0.268268from GPU time0.280280from GPU time8.841 question per request, 32 in flight9.271 question per request, 32 in flight124120not statedhuggingface.co
Decider 1 Closed · Efficient36.6%55.8%+19.2 *Not rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot runNot rundecision models onlyNot run0.01514.76at list prices0.01514.95at list prices0.3821 question per request, 1 in flight0.6991 question per request, 1 in flight125128not statedmeragpt.com
LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active27.1%70.4%+43.3 *Not rundecision models only90.5%495 of 547 answeredNot rundecision models only28.7%142 of 495Not rundecision models only26.4%51 “I don't know” of 193 not rightNot rundecision models only67.9%195 of 287answered 52.5%, non-hallucination 50.8%1.0501,050from GPU time0.517517from GPU time39.21 question per request, 32 in flight19.41 question per request, 32 in flight126116other languages, not Turkishhuggingface.co
GLiClass v4 Edge Open · 0.4B26.9%58.3%+31.4 *Not rundecision models only90.5%495 of 547 answeredNot rundecision models only39.6%196 of 495Not rundecision models only21.0%52 “I don't know” of 248 not rightNot rundecision models only72.0%352 of 489answered 89.4%, non-hallucination 14.1%No list priceNo list price0.0251 question per request, 1 in flight0.0171 question per request, 1 in flight127127other languages, not Turkishhuggingface.co
GLiDE 0.4B Open · 0.4B25.8%59.4%+33.6 *Not rundecision models only72.0%394 of 547 answeredNot rundecision models only34.3%135 of 394Not rundecision models only53.1%153 “I don't know” of 288 not rightNot rundecision models only76.4%55 of 72answered 13.2%, non-hallucination 89.6%No list priceNo list price0.0161 question per request, 1 in flight0.0141 question per request, 1 in flight128125not statedhuggingface.co
GLiNER2.5-multi-Decide Open · 0.3B20.7%21.2%+0.5Not rundecision models only87.2%477 of 547 answeredNot rundecision models only76.1%363 of 477Not rundecision models only16.2%70 “I don't know” of 433 not rightNot rundecision models only78.8%412 of 523answered 95.6%, non-hallucination 5.5%0.0066.27from GPU time0.0076.73from GPU time0.0091 question per request, 1 in flight0.0091 question per request, 1 in flight129131other languages, not Turkishhuggingface.co
Laya Open · 0.4B20.5%22.9%+2.4Not rundecision models only84.8%464 of 547 answeredNot rundecision models only79.1%367 of 464Not rundecision models only18.4%83 “I don't know” of 450 not rightNot rundecision models only78.9%390 of 494answered 90.3%, non-hallucination 12.0%0.0099.00from GPU time0.0087.84from GPU time0.0111 question per request, 1 in flight0.0111 question per request, 1 in flight130130other languages, not Turkishhuggingface.co
Laya Multilingual Open · 0.3B17.0%15.4%-1.6Not rundecision models only62.0%339 of 547 answeredNot rundecision models only82.6%280 of 339Not rundecision models only42.6%208 “I don't know” of 488 not rightNot rundecision models only84.9%359 of 423answered 77.3%, non-hallucination 25.7%0.0087.74from GPU time0.01312.62from GPU time0.0091 question per request, 1 in flight0.0181 question per request, 1 in flight131133not statedhuggingface.co
GLiNER2.5 Multi Open · 0.3B16.8%16.3%-0.5Not rundecision models only83.2%455 of 547 answeredNot rundecision models only83.1%378 of 455Not rundecision models only19.6%92 “I don't know” of 470 not rightNot rundecision models only82.5%428 of 519answered 94.9%, non-hallucination 6.1%0.0065.88at the maker's list price0.0088.39at the maker's list price0.0321 question per request, 1 in flight0.0091 question per request, 1 in flight132132other languages, not Turkishhuggingface.co
MiniCPM5-2B reasoning off · chat Open · 2.5B7.1%67.6%historical paired baseline, run 1094+60.5 *Not rundecision models only89.2%488 of 547 answeredNot rundecision models only33.6%164 of 488Not rundecision models only16.8%33 “I don't know” of 197 not rightNot rundecision models only65.8%75 of 114answered 20.8%, non-hallucination 22.7%5.4645,464from GPU time0.429429from GPU time172.01 question per request, 32 in flight7.291 question per request, 32 in flight133119other languages, not Turkishhuggingface.co

Ordered by Turkish accuracy, highest first. * marks a difference that is significant after Holm's correction: its paired test (the exact McNemar test, with p adjusted by Holm's method over all 133 English setups) has a Holm-adjusted p below 0.05. Accuracy: right answers over the 547 kept reading questions, the first English run against the Text tab run (Accuracy, Paired test). The “I don't know” columns: each model's first English run with “I don't know” against its Turkish run with the option also written “I don't know” (the Hallucination tab's option-word run, counted on these questions), so only the language differs (Only the language differs). Answered: (right + wrong) / 547; Wrong of answers: wrong / (right + wrong); Non-hallucination: “I don't know” / (“I don't know” + wrong); the difference's paired test is of wrong answers, with p adjusted by Holm's method over the 12 models with both runs. Turkish with “Bilmiyorum”: the Turkish side of the published comparison with “Bilmiyorum” (With “I don't know”) or, for a model outside it, its Hallucination tab run counted on these questions; its wrong of answers with its answered share and non-hallucination rate under it. Not run: no such run for a decision model (the Turkish “I don't know” run is for decision models only); Running: a chat model's run still running. Cost and latency: the Text tab run and the first English run, cost in $ per 1,000 questionsper 1M questions and the median time per request with the request size and the requests in flight under it (Cost, Latency). No list price: a setup on our own hardware whose maker lists no price for the same model. Rank: by accuracy among the 133 setups in each language, 1 the most accurate, tied setups sharing a rank. LFM3-1.7B, LFM3-1.7B and MiniCPM5-2B are run with 3 sampling seeds: until all are in, the number is the first English run's, named with the seeds in so far; then it is the mean of the runs' accuracies, ranked by it, with their lowest and highest (min–max, a range, not a confidence interval) under it. The paired comparison with Turkish and the “I don't know” runs stay on their own runs. Languages: what each model's card or its maker's docs list, with the source (Languages the model cards list); for a setup outside that table, whether its card lists Turkish, as the Turkish leaderboards' ‡ marks read it; Not checked: no card checked yet. The full tables are under Info.

Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. No list price: on our own GPUs, with no estimate. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches.

English-only setups (no Turkish run, so in no paired test and no Holm adjustment): Gemma 4 12B Unified · default reasoning · chat: English 86.8% (475 of 547 right), Turkish queued (in logic's GPU 1 queue (Ümit #6527, logic #6955)), celeris-1.1-decision: English 83.7% (458 of 547 right), Turkish Not run, LFM2.5-24B-A2B · default reasoning · chat: English 82.6% (452 of 547 right), Turkish queued (not scheduled yet: needs the model served again (Ümit #6527)), MiniCPM5-2B · reasoning on · chat: English 78.1% (the mean of 3 seeds, 76.6%–79.0% min–max of 3 seeds), Turkish Not run, Gemma 4 E2B · reasoning on · chat: English 76.2% (417 of 547 right), Turkish Not run, Qwen3.5-2B · reasoning on · chat: English 75.9% (415 of 547 right), Turkish Not run, Gemma 4 E2B · default reasoning · chat: English 74.6% (408 of 547 right), Turkish queued (in logic's GPU 1 queue (Ümit #6527, logic #6955)), Granite-4.2-3B · reasoning on · chat: English 74.4% (407 of 547 right), Turkish paused (its Text run stopped at 741 of 2,198 so that other setups go first on logic's GPU 1; it resumes when logic loads Granite again (logic #7481, #7599)) and Raon-1 nano · reasoning off · chat: English 53.7% (294 of 547 right), Turkish queued (asked on its next load in logic's GPU 1 queue, whichever way its smoke came out (run 1154: 3 of 20 replies at the 32,768-token limit, 5 without a letter; Ümit #6515 via Zeki #8211)).

With an “I don't know” option

The same 547 kept questions with a sixth option, “I don't know”, in English and in Turkish, as on the Hallucination tab, asked of the same models (how).

62 of the 123 models with both runs had a higher non-hallucination rate in English than in Turkish; 55 lower; six the same; 18 of the 123 differences in wrong answers are significant after Holm's correction within each group (decision models, chat models): d1-2B (more wrong answers in Turkish), GLiDE 0.4B (more wrong answers in English), Laya Multilingual (more wrong answers in Turkish), OneJev1.6-0.8B (more wrong answers in Turkish), LFM2.5-8B-A1B · default reasoning · chat (more wrong answers in Turkish), GLiDE 2B (more wrong answers in Turkish), GLiDE 0.8B (more wrong answers in Turkish), GLiClass v4 Edge (more wrong answers in Turkish), GLiNER2.5 Multi (more wrong answers in Turkish), StartLux-Decision-v1.1-2B-Instant (more wrong answers in Turkish), MiniCPM5-2B · reasoning off · chat (more wrong answers in English), GLiNER2.5-multi-Decide (more wrong answers in Turkish), LFM2.5-2.6B · default reasoning · chat (more wrong answers in Turkish), LFM3-1.7B · reasoning on · chat (more wrong answers in Turkish), KnowLine-0.8B-Gen1 (more wrong answers in Turkish), Qwen3.5-2B · reasoning off · chat (more wrong answers in Turkish), StartLux-Decision-v1.1-0.8B-Instant (more wrong answers in Turkish) and Metask-Jev-4B (more wrong answers in Turkish) (Paired test). celeris-1.1-decision has an English rate of 74.8% (98 “I don't know” of 131 not right) with an attempt rate of 82.1% (449 answered) but no Turkish run with the option on these questions, so no comparison or test. celeris-1-decision has an English rate of 71.3% (92 “I don't know” of 129 not right) with an attempt rate of 83.2% (455 answered) but no Turkish run with the option on these questions, so no comparison or test. Raon-1 nano · reasoning off · chat has an English rate of 26.2% (71 “I don't know” of 271 not right) with an attempt rate of 86.8% (475 answered) but no Turkish run with the option on these questions, so no comparison or test. KnowLine-2B-Gen1 has an English rate of 22.0% (38 “I don't know” of 173 not right) with an attempt rate of 93.1% (509 answered) but no Turkish run with the option on these questions, so no comparison or test. MiniCPM5-2B · reasoning on · chat has an English rate of 19.7% (28 “I don't know” of 142 not right) with an attempt rate of 94.1% (515 answered) but no Turkish run with the option on these questions, so no comparison or test. Granite-4.2-3B · reasoning on · chat has an English rate of 16.1% (23 “I don't know” of 143 not right) with an attempt rate of 95.8% (524 answered) but no Turkish run with the option on these questions, so no comparison or test. LFM2.5-24B-A2B · default reasoning · chat has an English rate of 13.2% (14 “I don't know” of 106 not right) with an attempt rate of 97.4% (533 answered) but no Turkish run with the option on these questions, so no comparison or test. Gemma 4 E2B · default reasoning · chat has an English rate of 11.5% (17 “I don't know” of 148 not right) with an attempt rate of 96.5% (528 answered) but no Turkish run with the option on these questions, so no comparison or test. Gemma 4 E2B · reasoning on · chat has an English rate of 11.3% (17 “I don't know” of 150 not right) with an attempt rate of 96.3% (527 answered) but no Turkish run with the option on these questions, so no comparison or test. Qwen3.5-2B · reasoning on · chat has an English rate of 9.9% (14 “I don't know” of 142 not right) with an attempt rate of 97.1% (531 answered) but no Turkish run with the option on these questions, so no comparison or test. Gemma 4 12B Unified · default reasoning · chat has an English rate of 5.5% (4 “I don't know” of 73 not right) with an attempt rate of 98.9% (541 answered) but no Turkish run with the option on these questions, so no comparison or test. Still running in Turkish, so charted with one side and in no test: Claude Fable 5.2 · max · chat. Read each rate with its attempt rate: a model that always chose the option would reach 100%.

With the option written “I don't know” in Turkish too, so that only the language differs, the decision models' comparison is in the next section.

Non-hallucination rate in Turkish and in English

The option among the answers that were not right, on the 547 kept questions: each model's first run in each language, ordered by the English rate · higher is better, read with the attempt rate (under the chart and in each bar's tooltip)

  1. d1.2-ultra76.2% · 89.3%
  2. d1.260.2% · 61.5%
  3. d1-2B43.8% · 60.9%
  4. d1-ultra53.6% · 56.7%
  5. GLiDE 0.4B89.6% · 53.1%
  6. Drex Max v1.115.0% · 51.2%
  7. Mammoth-142.1% · 50.7%
  8. GLiDE v1.211.3% · 45.5%
  9. Laya Multilingual25.7% · 42.6%
  10. GLiDE 27B18.8% · 37.9%
  11. Claude Fable 5.2 max · chatRunning · 28.6%
  12. OneJev1.6-0.8B30.7% · 28.4%
  13. Grok 4.7 high · chat27.6% · 27.9%
  14. LFM2.5-8B-A1B default reasoning · chat50.8% · 26.4%
  15. GLiDE 2B24.0% · 26.3%
  16. Elephant-115.5% · 25.6%
  17. Gemini 4 Carbon high · chat26.7% · 25.0%
  18. SubQ Snap 1.127.1% · 25.0%
  19. GLiDE 0.8B24.3% · 21.7%
  20. GLiClass v4 Edge14.1% · 21.0%
  21. Gemini 4 Hydrogen high · chat28.6% · 20.0%
  22. GLiNER2.5 Multi6.1% · 19.6%
  23. GPT-6 Astra high · chat19.0% · 19.0%
  24. Gemini 4 Argon low · chat23.5% · 18.5%
  25. Laya12.0% · 18.4%
  26. GPT-6 Astra low · chat22.2% · 18.2%
  27. Open-Jev-27B-v1.17.1% · 17.4%
  28. StartLux-Decision-v1.1-2B-Instant12.3% · 16.9%
  29. MiniCPM5-2B reasoning off · chat22.7% · 16.8%
  30. GLiDE 9B9.8% · 16.7%
  31. GPT-6 Astra medium · chat20.0% · 16.7%
  32. GPT-6.1 Sol high · chat20.0% · 16.7%
  33. GPT-6.1 Terra high · chat20.8% · 16.7%
  34. GLiNER2.5-multi-Decide5.5% · 16.2%
  35. LFM2.5-2.6B default reasoning · chat15.8% · 15.7%
  36. LFM3-1.7B reasoning on · chat33.5% · 15.6%
  37. GPT-6.1 Terra low · chat14.8% · 15.4%
  38. Perplexity Decider 27B v1.2 API · with web5.6% · 15.2%
  39. Winnow-12B8.6% · 15.0%
  40. GPT-6 Astra low · chat · 1 per request17.4% · 14.8%
  41. GPT-6.1 Sol low · chat18.2% · 14.8%
  42. Mercury Decide10.1% · 13.4%
  43. GPT-6.1 Sol low · chat · 1 per request17.4% · 12.9%
  44. LFM3-1.7B reasoning off · chat55.4% · 11.6%
  45. Cygnet5.8% · 11.2%
  46. KnowLine-0.8B-Gen14.4% · 10.7%
  47. Qwen3.5-2B reasoning off · chat16.5% · 10.3%
  48. Gemini 3.8 Flash high · chat13.3% · 10.0%
  49. Gemma 4 E4B default reasoning · chat20.7% · 9.8%
  50. GPT-6 Luna Decisions API7.1% · 9.4%
  51. Gemini Decision 1 2026111110.9% · 9.1%
  52. Perplexity Decider 27B v1.30.0% · 8.8%
  53. Turion Jev 31B5.7% · 8.5%
  54. d3-ultra2.1% · 8.2%
  55. Claude Haiku 5.5 max · chat17.1% · 7.8%
  56. Gemini 3.8 Flash low · chat · 1 per request14.3% · 7.7%
  57. Kimi K3.1 Pro no reasoning · chat7.1% · 7.5%
  58. vera-medium-omni4.2% · 7.4%
  59. Claude Fable 5.1 high · chat14.8% · 7.4%
  60. Kimi K3.1 Pro low · chat7.7% · 7.3%
  61. Perplexity Decider Flash v1.37.6% · 7.2%
  62. DeepSeek-V4.1-Flash reasoning on · chat · 10172.6% · 7.1%
  63. GPT-6 Luna low · chat3.0% · 6.8%
  64. REAX 1.32.8% · 6.7%
  65. Claude Opus 5.5 high · chat11.5% · 6.5%
  66. Qwen3.5-397B-A17B high · chat8.5% · 6.2%
  67. GLM-5.3-Flash reasoning off · chat3.5% · 5.6%
  68. Qwen3.8 Flash no reasoning · chat5.2% · 5.6%
  69. DeepSeek-V4.1-Flash reasoning off · chat · 10170.0% · 5.4%
  70. Clef-Flash9.4% · 5.3%
  71. Muse Ember max · chat7.8% · 5.3%
  72. StartLux-Decision-v1.1-0.8B-Instant6.4% · 5.2%
  73. Claude Sonnet 5.5 high · chat3.6% · 4.8%
  74. Perplexity Decider Flash v1.24.3% · 4.8%
  75. Claude Opus 5.5 low · chat6.5% · 4.7%
  76. Perplexity Decider 27B v1.2 API0.0% · 4.5%
  77. Claude Sonnet 5.5 low · chat5.7% · 4.4%
  78. Logits Sigma-1 Duo0.0% · 4.3%
  79. Jev 1.144.2% · 4.0%
  80. OneJev-27B1.6% · 4.0%
  81. Microsoft-Decision-12.2% · 3.8%
  82. Chinese Decision Pointer 4B3.0% · 3.7%
  83. Qwen3.8-27B stated confidence · reasoning off · chat5.3% · 3.7%
  84. Gemma 4 31B reasoning off · chat4.5% · 3.6%
  85. Qwen3.5-4B reasoning off · chat3.4% · 3.6%
  86. Qwen3.8-27B low · chat2.0% · 3.6%
  87. Gemini 3.8 Flash low · chat11.1% · 3.4%
  88. GLM-5.3 low · chat4.7% · 3.4%
  89. Qwen3.8-27B vote share · reasoning off · chat5.1% · 3.4%
  90. MiniMax-M3.1-Flash high · chat1.1% · 3.4%
  91. Drex Max3.6% · 3.2%
  92. Perplexity Decider 27B v1.2 DGX Spark0.0% · 3.0%
  93. d11.1% · 2.9%
  94. StartLux-Decision-v1.1-9B-Instant1.0% · 2.6%
  95. Cerno 9B v1.20.0% · 2.5%
  96. Qwen3.5-4B reasoning on · chat2.2% · 2.4%
  97. Sieve 31B v30.0% · 2.2%
  98. Erk-14B reasoning off · chat0.8% · 2.1%
  99. Verdin-4B-v1.10.8% · 2.1%
  100. Qwen3-14B reasoning off · chat0.8% · 1.8%
  101. d30.0% · 1.7%
  102. MiMo-V2.6-Flash high · chat4.2% · 1.7%
  103. GLM-5.3-Flash reasoning on · chat3.5% · 1.6%
  104. Metask-Jev-4B1.1% · 1.6%
  105. Clef1.5% · 1.3%
  106. KnowLine-12B-Gen10.0% · 1.3%
  107. KnowLine-9B-Gen10.0% · 1.3%
  108. KnowLine-9B-Gen20.0% · 1.3%
  109. Valen-Decision-1.5-9B0.0% · 1.2%
  110. Dexy-Decision-9B-V21.1% · 1.1%
  111. KnowLine 4B Gen5.52.5% · 0.9%
  112. Strands Decider 2B1.5% · 0.7%
  113. Adjudex 9B v50.0% · 0.0%
  114. Claude Opus 5.5 low · chat · 1 per request3.8% · 0.0%
  115. Claude Sonnet 5.5 high · chat · 1 per request2.7% · 0.0%
  116. Copernicus-31B2.9% · 0.0%
  117. Gemma 4 31B reasoning off · chat · 1 per request1.9% · 0.0%
  118. Inkling-Small high · chat · Jul 20260.0% · 0.0%
  119. Jeffan-9B-12114.1% · 0.0%
  120. Karpo Decision 31B3.2% · 0.0%
  121. KnowLine-31B-Gen1.50.0% · 0.0%
  122. KnowLine-31B-Gen20.0% · 0.0%
  123. Qwen4-Max high · chat3.4% · 0.0%
  124. StartLux-Decision-v1.1-4B-Instant1.7% · 0.0%

Turkish option, the upper bar English option, the lower bar, striped Numbers on the right: Turkish · English. Liquid Fastino Drex Mammoth Laya Claude OneJev Grok Elephant Gemini SubQ GLiClass GPT Open-Jev StartLux MiniCPM Perplexity Winnow Inception Cygnet KnowLine Qwen Gemma Turion Jev vLLM-SR Kimi vera DeepSeek REAX GLM Clef Muse Logits Jev Microsoft Chinese Decision Pointer MiniMax Cerno Sieve Erk Verdin MiMo Metask Valen Dexy Strands Adjudex Copernicus Inkling Jeffan Karpo Attempt rate, Turkish · English: d1.2-ultra 88.8% · 80.1%; d1.2 90.3% · 89.2%; d1-2B 70.6% · 72.4%; d1-ultra 94.5% · 90.7%; GLiDE 0.4B 13.2% · 72.0%; Drex Max v1.1 98.4% · 92.1%; Mammoth-1 95.6% · 93.2%; GLiDE · v1.2 98.9% · 94.5%; Laya Multilingual 77.3% · 62.0%; GLiDE 27B 98.4% · 96.0%; Claude Fable 5.2 · max · chat running · 99.3%; OneJev1.6-0.8B 79.5% · 86.5%; Grok 4.7 · high · chat 98.5% · 97.8%; LFM2.5-8B-A1B · default reasoning · chat 52.5% · 90.5%; GLiDE 2B 87.9% · 91.4%; Elephant-1 97.3% · 95.8%; Gemini 4 Carbon · high · chat 99.3% · 99.3%; SubQ Snap 1.1 96.5% · 96.2%; GLiDE 0.8B 86.5% · 91.0%; GLiClass v4 Edge 89.4% · 90.5%; Gemini 4 Hydrogen · high · chat 99.3% · 99.3%; GLiNER2.5 Multi 94.9% · 83.2%; GPT-6 Astra · high · chat 99.3% · 99.3%; Gemini 4 Argon · low · chat 99.3% · 99.1%; Laya 90.3% · 84.8%; GPT-6 Astra · low · chat 99.3% · 99.3%; Open-Jev-27B-v1.1 98.9% · 97.1%; StartLux-Decision-v1.1-2B-Instant 94.0% · 94.1%; MiniCPM5-2B · reasoning off · chat 20.8% · 89.2%; GLiDE 9B 98.4% · 97.4%; GPT-6 Astra · medium · chat 99.3% · 99.3%; GPT-6.1 Sol · high · chat 99.3% · 99.3%; GPT-6.1 Terra · high · chat 99.1% · 99.3%; GLiNER2.5-multi-Decide 95.6% · 87.2%; LFM2.5-2.6B · default reasoning · chat 90.7% · 95.6%; LFM3-1.7B · reasoning on · chat 80.6% · 95.2%; GPT-6.1 Terra · low · chat 99.3% · 99.3%; Perplexity Decider 27B v1.2 · API · with web 99.5% · 98.2%; Winnow-12B 98.5% · 97.3%; GPT-6 Astra · low · chat · 1 per request 99.3% · 99.3%; GPT-6.1 Sol · low · chat 99.3% · 99.3%; Mercury Decide 98.5% · 98.0%; GPT-6.1 Sol · low · chat · 1 per request 99.3% · 99.3%; LFM3-1.7B · reasoning off · chat 56.9% · 96.7%; Cygnet 99.1% · 98.2%; KnowLine-0.8B-Gen1 97.3% · 95.2%; Qwen3.5-2B · reasoning off · chat 83.4% · 96.3%; Gemini 3.8 Flash · high · chat 99.6% · 99.6%; Gemma 4 E4B · default reasoning · chat 95.4% · 97.8%; GPT-6 Luna · Decisions API 98.9% · 98.2%; Gemini Decision 1 · 20261111 99.1% · 99.1%; Perplexity Decider 27B v1.3 100.0% · 99.1%; Turion Jev 31B 99.6% · 99.3%; d3-ultra 99.8% · 99.1%; Claude Haiku 5.5 · max · chat 96.0% · 96.3%; Gemini 3.8 Flash · low · chat · 1 per request 99.5% · 99.6%; Kimi K3.1 Pro · no reasoning · chat 99.5% · 99.3%; vera-medium-omni 99.1% · 98.7%; Claude Fable 5.1 · high · chat 99.3% · 99.6%; Kimi K3.1 Pro · low · chat 99.5% · 99.5%; Perplexity Decider Flash v1.3 98.4% · 98.7%; DeepSeek-V4.1-Flash · reasoning on · chat · 1017 99.8% · 99.5%; GPT-6 Luna · low · chat 99.6% · 99.1%; REAX 1.3 99.8% · 99.5%; Claude Opus 5.5 · high · chat 99.5% · 99.6%; Qwen3.5-397B-A17B · high · chat 99.3% · 99.5%; GLM-5.3-Flash · reasoning off · chat 99.5% · 99.3%; Qwen3.8 Flash · no reasoning · chat 99.5% · 99.3%; DeepSeek-V4.1-Flash · reasoning off · chat · 1017 100.0% · 99.3%; Clef-Flash 97.8% · 99.1%; Muse Ember · max · chat 99.3% · 99.5%; StartLux-Decision-v1.1-0.8B-Instant 96.0% · 97.8%; Claude Sonnet 5.5 · high · chat 99.6% · 99.5%; Perplexity Decider Flash v1.2 98.9% · 99.1%; Claude Opus 5.5 · low · chat 99.6% · 99.6%; Perplexity Decider 27B v1.2 · API 100.0% · 99.5%; Claude Sonnet 5.5 · low · chat 99.1% · 99.3%; Logits Sigma-1 Duo 100.0% · 99.6%; Jev 1.14 99.5% · 99.5%; OneJev-27B 99.8% · 99.5%; Microsoft-Decision-1 99.6% · 99.5%; Chinese Decision Pointer 4B 99.3% · 99.3%; Qwen3.8-27B · stated confidence · reasoning off · chat 99.3% · 99.5%; Gemma 4 31B · reasoning off · chat 99.6% · 99.6%; Qwen3.5-4B · reasoning off · chat 98.4% · 99.3%; Qwen3.8-27B · low · chat 99.8% · 99.6%; Gemini 3.8 Flash · low · chat 99.5% · 99.8%; GLM-5.3 · low · chat 99.5% · 99.6%; Qwen3.8-27B · vote share · reasoning off · chat 99.3% · 99.5%; MiniMax-M3.1-Flash · high · chat 99.6% · 99.5%; Drex Max 99.6% · 99.6%; Perplexity Decider 27B v1.2 · DGX Spark 100.0% · 99.6%; d1 99.8% · 99.6%; StartLux-Decision-v1.1-9B-Instant 99.8% · 99.6%; Cerno 9B v1.2 100.0% · 99.6%; Qwen3.5-4B · reasoning on · chat 99.1% · 99.6%; Sieve 31B v3 100.0% · 99.8%; Erk-14B · reasoning off · chat 99.8% · 99.5%; Verdin-4B-v1.1 99.8% · 99.6%; Qwen3-14B · reasoning off · chat 99.8% · 99.6%; d3 100.0% · 99.8%; MiMo-V2.6-Flash · high · chat 99.5% · 99.8%; GLM-5.3-Flash · reasoning on · chat 99.6% · 99.8%; Metask-Jev-4B 99.6% · 99.6%; Clef 99.8% · 99.8%; KnowLine-12B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen2 100.0% · 99.8%; Valen-Decision-1.5-9B 100.0% · 99.8%; Dexy-Decision-9B-V2 99.8% · 99.8%; KnowLine 4B Gen5.5 99.5% · 99.8%; Strands Decider 2B 99.1% · 99.6%; Adjudex 9B v5 100.0% · 100.0%; Claude Opus 5.5 · low · chat · 1 per request 99.8% · 100.0%; Claude Sonnet 5.5 · high · chat · 1 per request 99.8% · 100.0%; Copernicus-31B 99.8% · 100.0%; Gemma 4 31B · reasoning off · chat · 1 per request 99.8% · 100.0%; Inkling-Small · high · chat · Jul 2026 100.0% · 100.0%; Jeffan-9B-1211 99.5% · 100.0%; Karpo Decision 31B 99.8% · 100.0%; KnowLine-31B-Gen1.5 100.0% · 100.0%; KnowLine-31B-Gen2 100.0% · 100.0%; Qwen4-Max · high · chat 99.8% · 100.0%; StartLux-Decision-v1.1-4B-Instant 99.6% · 100.0%. A model that always chose the option would reach 100% with no right answers. Running: a chat model's runs with the option go English first, then Turkish, and each is added as it finishes. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision, Raon-1 nano · reasoning off · chat, KnowLine-2B-Gen1, MiniCPM5-2B · reasoning on · chat, Granite-4.2-3B · reasoning on · chat, LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E2B · default reasoning · chat, Gemma 4 E2B · reasoning on · chat, Qwen3.5-2B · reasoning on · chat and Gemma 4 12B Unified · default reasoning · chat (in the table under Info).

Wrong only in Turkish, or only in English?

Each model's first run in each language, both with the option, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method within each group (decision models, chat models) · the shorter bar is the language with fewer wrong answers

  1. GLiClass v4 Edge204 · 48 *
  2. Qwen3.5-2B reasoning off · chat173 · 50 *
  3. LFM2.5-2.6B default reasoning · chat160 · 44 *
  4. d1-2B148 · 38 *
  5. KnowLine-0.8B-Gen1157 · 49 *
  6. StartLux-Decision-v1.1-0.8B-Instant151 · 48 *
  7. MiniCPM5-2B reasoning off · chat37 · 126 *
  8. GLiDE 0.4B45 · 125 *
  9. StartLux-Decision-v1.1-2B-Instant125 · 47 *
  10. Laya Multilingual130 · 51 *
  11. GLiDE 2B125 · 48 *
  12. Metask-Jev-4B98 · 34 *
  13. OneJev1.6-0.8B121 · 55 *
  14. LFM3-1.7B reasoning on · chat137 · 68 *
  15. GLiNER2.5 Multi95 · 45 *
  16. GLiNER2.5-multi-Decide95 · 46 *
  17. GLiDE 0.8B121 · 68 *
  18. LFM2.5-8B-A1B default reasoning · chat133 · 80 *
  19. Perplexity Decider Flash v1.269 · 36
  20. Verdin-4B-v1.170 · 37
  21. Claude Haiku 5.5 max · chat7 · 25
  22. vera-medium-omni52 · 26
  23. Qwen3.5-4B reasoning off · chat86 · 52
  24. KnowLine-9B-Gen140 · 18
  25. Clef-Flash52 · 27
  26. Valen-Decision-1.5-9B58 · 32
  27. d143 · 21
  28. LFM3-1.7B reasoning off · chat115 · 79
  29. StartLux-Decision-v1.1-4B-Instant57 · 33
  30. StartLux-Decision-v1.1-9B-Instant47 · 26
  31. d1-ultra7 · 20
  32. Chinese Decision Pointer 4B66 · 41
  33. GPT-6.1 Sol low · chat · 1 per request1 · 9
  34. Strands Decider 2B75 · 49
  35. KnowLine-9B-Gen240 · 23
  36. Perplexity Decider 27B v1.2 DGX Spark15 · 29
  37. Perplexity Decider Flash v1.357 · 37
  38. GPT-6 Luna Decisions API29 · 47
  39. Sieve 31B v311 · 23
  40. KnowLine-31B-Gen29 · 20
  41. Gemini 4 Hydrogen high · chat1 · 7
  42. Claude Opus 5.5 low · chat13 · 25
  43. Karpo Decision 31B13 · 25
  44. Grok 4.7 high · chat8 · 18
  45. Gemini 4 Argon low · chat6 · 15
  46. GLM-5.3-Flash reasoning off · chat41 · 26
  47. Perplexity Decider 27B v1.2 API15 · 27
  48. DeepSeek-V4.1-Flash reasoning off · chat · 101720 · 33
  49. Elephant-146 · 31
  50. Turion Jev 31B11 · 21
  51. Qwen3.8 Flash no reasoning · chat19 · 31
  52. GLiDE v1.227 · 16
  53. GLiDE 9B38 · 25
  54. Laya120 · 97
  55. Gemini Decision 1 2026111111 · 20
  56. Cerno 9B v1.248 · 34
  57. Gemma 4 31B reasoning off · chat19 · 30
  58. Kimi K3.1 Pro no reasoning · chat15 · 25
  59. MiMo-V2.6-Flash high · chat31 · 20
  60. SubQ Snap 1.127 · 39
  61. Qwen3-14B reasoning off · chat54 · 40
  62. Microsoft-Decision-135 · 24
  63. Drex Max v1.130 · 20
  64. Adjudex 9B v544 · 32
  65. KnowLine 4B Gen5.553 · 40
  66. KnowLine-12B-Gen117 · 26
  67. Gemini 3.8 Flash high · chat3 · 8
  68. d3-ultra23 · 33
  69. d1.2-ultra12 · 6
  70. Gemini 3.8 Flash low · chat · 1 per request6 · 12
  71. Claude Opus 5.5 high · chat7 · 13
  72. Claude Opus 5.5 low · chat · 1 per request7 · 13
  73. GPT-6.1 Sol low · chat4 · 9
  74. Logits Sigma-1 Duo17 · 25
  75. Clef18 · 26
  76. REAX 1.313 · 20
  77. OneJev-27B25 · 34
  78. Qwen3.8-27B vote share · reasoning off · chat26 · 35
  79. Erk-14B reasoning off · chat56 · 68
  80. Muse Ember max · chat18 · 25
  81. Copernicus-31B13 · 19
  82. GPT-6 Astra low · chat4 · 8
  83. GPT-6 Astra medium · chat4 · 8
  84. Drex Max21 · 28
  85. Gemma 4 31B reasoning off · chat · 1 per request24 · 31
  86. GPT-6.1 Sol high · chat5 · 9
  87. GPT-6 Astra low · chat · 1 per request6 · 10
  88. GLM-5.3-Flash reasoning on · chat22 · 28
  89. Claude Sonnet 5.5 high · chat25 · 31
  90. Qwen3.8-27B stated confidence · reasoning off · chat27 · 33
  91. Qwen3.8-27B low · chat18 · 23
  92. Gemini 3.8 Flash low · chat10 · 14
  93. d319 · 24
  94. Perplexity Decider 27B v1.2 API · with web21 · 26
  95. GLM-5.3 low · chat28 · 23
  96. MiniMax-M3.1-Flash high · chat43 · 38
  97. Qwen3.5-4B reasoning on · chat49 · 44
  98. GPT-6 Luna low · chat27 · 31
  99. KnowLine-31B-Gen1.513 · 16
  100. Claude Sonnet 5.5 low · chat32 · 36
  101. GLiDE 27B18 · 15
  102. Mammoth-115 · 18
  103. Jev 1.1417 · 20
  104. Gemma 4 E4B default reasoning · chat48 · 44
  105. Claude Fable 5.1 high · chat9 · 11
  106. Claude Sonnet 5.5 high · chat · 1 per request14 · 16
  107. Kimi K3.1 Pro low · chat14 · 16
  108. Qwen3.5-397B-A17B high · chat16 · 18
  109. d1.219 · 21
  110. Open-Jev-27B-v1.128 · 26
  111. Cygnet30 · 28
  112. Inkling-Small high · chat · Jul 202632 · 30
  113. DeepSeek-V4.1-Flash reasoning on · chat · 101718 · 19
  114. Dexy-Decision-9B-V238 · 38
  115. Gemini 4 Carbon high · chat2 · 3
  116. GPT-6 Astra high · chat6 · 6
  117. GPT-6.1 Terra high · chat7 · 8
  118. GPT-6.1 Terra low · chat12 · 11
  119. Jeffan-9B-121131 · 31
  120. Mercury Decide29 · 29
  121. Perplexity Decider 27B v1.324 · 24
  122. Qwen4-Max high · chat11 · 12
  123. Winnow-12B32 · 32

Wrong only in Turkish, the upper bar Wrong only in English, the lower bar, striped Numbers on the right: Turkish · English. GLiClass Qwen Liquid KnowLine StartLux MiniCPM Fastino Laya Metask OneJev Perplexity Verdin Claude vera Clef Valen Chinese Decision Pointer GPT Strands Sieve Gemini Karpo Grok GLM DeepSeek Elephant Turion Jev Cerno Gemma Kimi MiMo SubQ Microsoft Drex Adjudex vLLM-SR Logits REAX Erk Muse Copernicus MiniMax Mammoth Jev Open-Jev Cygnet Inkling Dexy Jeffan Inception Winnow * A significant difference (Holm-adjusted p below 0.05): GLiClass v4 Edge, Qwen3.5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, d1-2B, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-0.8B-Instant, MiniCPM5-2B · reasoning off · chat, GLiDE 0.4B, StartLux-Decision-v1.1-2B-Instant, Laya Multilingual, GLiDE 2B, Metask-Jev-4B, OneJev1.6-0.8B, LFM3-1.7B · reasoning on · chat, GLiNER2.5 Multi, GLiNER2.5-multi-Decide, GLiDE 0.8B and LFM2.5-8B-A1B · default reasoning · chat (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p = 0.0015, p = 0.0026, p = 0.0079 and p = 0.018). The other 105: p = 0.091, p = 0.099, p = 0.11, p = 0.23, p = 0.23, p = 0.28, p = 0.33, p = 0.4, p = 0.4, p = 0.57, p = 0.71, p = 0.88, p = 0.88, p = 0.9, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision, Raon-1 nano · reasoning off · chat, KnowLine-2B-Gen1, MiniCPM5-2B · reasoning on · chat, Granite-4.2-3B · reasoning on · chat, LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E2B · default reasoning · chat, Gemma 4 E2B · reasoning on · chat, Qwen3.5-2B · reasoning on · chat and Gemma 4 12B Unified · default reasoning · chat (in the table under Info).

Only the language differs: “I don't know” in both

The same 547 kept questions with the sixth option written “I don't know” in English and in Turkish too, with the same sentence added to the instructions: each model's first English run with the option, as in the section above, against its Turkish run from the option's word check of the Hallucination tab, counted on these questions. Only the language differs: of the question and its options, of the word in the instructions that names it and of the options heading (how).

Twelve of the 75 decision models have both runs. Among them, wrong of answers was lower in English for five and higher for seven; the non-hallucination rate was lower in English for one and higher for 11. None of the 12 differences in wrong answers is significant after Holm's correction over the 12, so each could be chance; the smallest adjusted p is SubQ Snap 1.1's, p = 0.36 (Paired test). English only, with no Turkish run with the option written “I don't know”: KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1.2 · API, Jeffan-9B-1211, d1, StartLux-Decision-v1.1-9B-Instant, Clef, KnowLine-12B-Gen1, GLiDE 9B, OneJev-27B, Jev 1.14, Cerno 9B v1.2, KnowLine-9B-Gen1, Adjudex 9B v5, Mercury Decide, Microsoft-Decision-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2, celeris-1-decision, GPT-6 Luna · Decisions API, Cygnet, Winnow-12B, Perplexity Decider Flash v1.3, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Clef-Flash, Verdin-4B-v1.1, Perplexity Decider Flash v1.2, KnowLine 4B Gen5.5, Chinese Decision Pointer 4B, Metask-Jev-4B, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, d1-2B, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1, GLiClass v4 Edge, Strands Decider 2B, Laya, GLiNER2.5-multi-Decide and Laya Multilingual (in the main table and the table under Info, with no test).

Wrong of answers in Turkish and in English

Wrong / (right + wrong), with the option written “I don't know” in both languages, on the 547 kept questions: each model's first run in each language, ordered by the Turkish share, lowest first · lower is better, read with the answered share (under the chart and in each bar's tooltip)

  1. d1.2-ultra3.7% · 3.0%
  2. d1-ultra5.4% · 7.9%
  3. Mammoth-16.1% · 7.1%
  4. d1.28.4% · 7.6%
  5. SubQ Snap 1.18.5% · 12.0%
  6. GLiDE v1.28.7% · 7.0%
  7. Drex Max9.0% · 11.0%
  8. Drex Max v1.19.1% · 8.1%
  9. Perplexity Decider 27B v1.2 DGX Spark9.3% · 11.7%
  10. d3-ultra9.4% · 10.3%
  11. d39.5% · 10.6%
  12. Elephant-115.7% · 12.8%

Turkish, with “I don't know”, the upper bar English, with “I don't know”, the lower bar, striped Numbers on the right: Turkish · English. Liquid Mammoth SubQ Fastino Drex Perplexity vLLM-SR Elephant Answered, Turkish · English: d1.2-ultra 89.0% · 80.1%; d1-ultra 95.1% · 90.7%; Mammoth-1 95.4% · 93.2%; d1.2 91.6% · 89.2%; SubQ Snap 1.1 96.5% · 96.2%; GLiDE · v1.2 98.4% · 94.5%; Drex Max 99.8% · 99.6%; Drex Max v1.1 96.7% · 92.1%; Perplexity Decider 27B v1.2 · DGX Spark 99.8% · 99.6%; d3-ultra 99.6% · 99.1%; d3 100.0% · 99.8%; Elephant-1 95.6% · 95.8%. A model that answered fewer questions can have fewer wrong answers among them. Not charted, with one side only: KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1, Perplexity Decider 27B v1.2 · API, d1.1, Jeffan-9B-1211, d1, StartLux-Decision-v1.1-9B-Instant, Clef, KnowLine-12B-Gen1, GLiDE 9B, OneJev-27B, Jev 1.14, Cerno 9B v1.2, Drex 2.0, Jev 1.13, KnowLine-9B-Gen1, SubQ Snap Preview, Adjudex 9B v5, Mercury Decide, Microsoft-Decision-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2, celeris-1-decision, GPT-6 Luna · Decisions API, Cygnet, Winnow-12B, Perplexity Decider Flash v1.3, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Clef-Flash, Verdin-4B-v1.1, Perplexity Decider Flash v1.2, KnowLine 4B Gen5.5, Chinese Decision Pointer 4B, Metask-Jev-4B, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, d1-2B, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1, GLiClass v4 Edge, Decider 1, Strands Decider 2B, Laya, GLiNER2.5-multi-Decide and Laya Multilingual (in the table under Info).

Wrong only in Turkish, or only in English?

Each model's first run in each language, both with the option written “I don't know”, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 12 models with both runs · the shorter bar is the language with fewer wrong answers

  1. SubQ Snap 1.122 · 40
  2. d1-ultra7 · 18
  3. Perplexity Decider 27B v1.2 DGX Spark16 · 29
  4. Elephant-146 · 31
  5. GLiDE v1.227 · 16
  6. Drex Max17 · 28
  7. d1.2-ultra12 · 7
  8. Drex Max v1.127 · 20
  9. d318 · 24
  10. d1.224 · 19
  11. d3-ultra26 · 31
  12. Mammoth-115 · 19

Wrong only in Turkish, the upper bar Wrong only in English, the lower bar, striped Numbers on the right: Turkish · English. SubQ Liquid Perplexity Elephant Fastino Drex vLLM-SR Mammoth No difference is significant (Holm-adjusted p below 0.05). Adjusted: p = 0.36, p = 0.48, p = 0.72, p = 0.99, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with one side only: KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1, Perplexity Decider 27B v1.2 · API, d1.1, Jeffan-9B-1211, d1, StartLux-Decision-v1.1-9B-Instant, Clef, KnowLine-12B-Gen1, GLiDE 9B, OneJev-27B, Jev 1.14, Cerno 9B v1.2, Drex 2.0, Jev 1.13, KnowLine-9B-Gen1, SubQ Snap Preview, Adjudex 9B v5, Mercury Decide, Microsoft-Decision-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2, celeris-1-decision, GPT-6 Luna · Decisions API, Cygnet, Winnow-12B, Perplexity Decider Flash v1.3, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Clef-Flash, Verdin-4B-v1.1, Perplexity Decider Flash v1.2, KnowLine 4B Gen5.5, Chinese Decision Pointer 4B, Metask-Jev-4B, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, d1-2B, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1, GLiClass v4 Edge, Decider 1, Strands Decider 2B, Laya, GLiNER2.5-multi-Decide and Laya Multilingual (in the table under Info).

The option's word: “I don't know” instead of “Bilmiyorum”

The Hallucination tab's 2,198 Turkish questions, with the sixth option “Bilmiyorum” and again with “I don't know”, and only the sentence that names it changed: a model that skipped “Bilmiyorum” because it did not recognise the Turkish word should choose “I don't know” more often. Each of the 13 models' first run with each word, in the Hallucination tab's order; the full table and method are on the Hallucination tab.

ModelAnswered with “Bilmiyorum”(right + wrong) / questions; higher is betterAnswered with “I don't know”(right + wrong) / questions; higher is betterWrong of answers with “Bilmiyorum”wrong / (right + wrong); lower is betterWrong of answers with “I don't know”wrong / (right + wrong); lower is betterNon-hallucination with “Bilmiyorum”“Bilmiyorum” / (“Bilmiyorum” + wrong); higher is betterNon-hallucination with “I don't know”“I don't know” / (“I don't know” + wrong); higher is betterDifference“I don't know” minus “Bilmiyorum” in non-hallucination, in points; positive = higher with the English wordHolm-adjusted pexact McNemar test of wrong answers
d1.2-ultra Closed · Efficient54.7%1,202 of 2,198 answered53.7%1,180 of 2,198 answered12.1%145 of 1,20211.6%137 of 1,18087.3%1,057 right, 145 wrong, 996 “Bilmiyorum”88.1%1,043 right, 137 wrong, 1,018 “I don't know”+0.8p = 1
d1.2 Closed · Efficient55.7%1,225 of 2,198 answered59.0%1,297 of 2,198 answered20.7%253 of 1,22521.5%279 of 1,29779.4%972 right, 253 wrong, 973 “Bilmiyorum”76.4%1,018 right, 279 wrong, 901 “I don't know”-3.0p = 0.4
d1-ultra Closed · Efficient71.9%1,581 of 2,198 answered72.0%1,582 of 2,198 answered13.0%205 of 1,58113.7%216 of 1,58275.1%1,376 right, 205 wrong, 617 “Bilmiyorum”74.0%1,366 right, 216 wrong, 616 “I don't know”-1.0p = 1
Mammoth-1 Open · 27B68.6%1,508 of 2,198 answered65.7%1,445 of 2,198 answered20.8%314 of 1,50819.1%276 of 1,44568.7%1,194 right, 314 wrong, 690 “Bilmiyorum”73.2%1,169 right, 276 wrong, 753 “I don't know”+4.5 *p < 0.0001
GLiDE API Closed · Efficient76.3%1,677 of 2,198 answered73.6%1,617 of 2,198 answered21.2%356 of 1,67720.7%335 of 1,61759.4%1,321 right, 356 wrong, 521 “Bilmiyorum”63.4%1,282 right, 335 wrong, 581 “I don't know”+4.0p = 0.51
SubQ Snap 1.1 Closed · Efficient80.1%1,761 of 2,198 answered77.3%1,700 of 2,198 answered27.8%490 of 1,76126.7%454 of 1,70047.1%1,271 right, 490 wrong, 437 “Bilmiyorum”52.3%1,246 right, 454 wrong, 498 “I don't know”+5.2p = 0.25
GLiDE v1.2 Open · 27B85.4%1,877 of 2,198 answered80.6%1,772 of 2,198 answered26.2%491 of 1,87724.1%427 of 1,77239.5%1,386 right, 491 wrong, 321 “Bilmiyorum”49.9%1,345 right, 427 wrong, 426 “I don't know”+10.4 *p < 0.0001
Drex Max v1.1 Closed · Efficient (provisional)85.4%1,878 of 2,198 answered77.9%1,712 of 2,198 answered27.5%517 of 1,87824.1%413 of 1,71238.2%1,361 right, 517 wrong, 320 “Bilmiyorum”54.1%1,299 right, 413 wrong, 486 “I don't know”+15.8 *p < 0.0001
Elephant-1 Open · 9.0B81.6%1,793 of 2,198 answered72.5%1,594 of 2,198 answered42.3%758 of 1,79339.0%622 of 1,59434.8%1,035 right, 758 wrong, 405 “Bilmiyorum”49.3%972 right, 622 wrong, 604 “I don't know”+14.4 *p < 0.0001
d3-ultra Open · 26B99.5%2,188 of 2,198 answered99.6%2,189 of 2,198 answered32.2%704 of 2,18832.2%704 of 2,1891.4%1,484 right, 704 wrong, 10 “Bilmiyorum”1.3%1,485 right, 704 wrong, 9 “I don't know”-0.1p = 1
Drex Max Open · 27B99.9%2,196 of 2,198 answered99.9%2,196 of 2,198 answered32.4%711 of 2,19632.4%712 of 2,1960.3%1,485 right, 711 wrong, 2 “Bilmiyorum”0.3%1,484 right, 712 wrong, 2 “I don't know”-0.0p = 1
Perplexity Decider 27B v1.2 DGX Spark Open · 27B99.9%2,196 of 2,198 answered99.8%2,194 of 2,198 answered32.7%717 of 2,19633.2%728 of 2,1940.3%1,479 right, 717 wrong, 2 “Bilmiyorum”0.5%1,466 right, 728 wrong, 4 “I don't know”+0.3p = 1
d3 Open · 26B100.0%2,198 of 2,198 answered99.9%2,196 of 2,198 answered32.5%715 of 2,19832.5%714 of 2,1960.0%1,483 right, 715 wrong, 0 “Bilmiyorum”0.3%1,482 right, 714 wrong, 2 “I don't know”+0.3p = 1

Answered: (right + wrong) / 2,198 questions. Wrong of answers: wrong / (right + wrong). Both are worked out from the published counts; under each non-hallucination rate, of the 2,198 questions, the right answers, the wrong ones and the option. * marks a difference whose paired test (the exact McNemar test on the questions only one of the two runs got wrong, with p adjusted by Holm's method over the models) has a Holm-adjusted p below 0.05.

Cost per 1,000 questionsper 1M questions in Turkish and in English

Estimated dollars per 1,000 questionsper 1M questions: each of the 133 setups' Text tab run and its first English run, ordered by English cost · lower is better

  1. Perplexity Decider Flash v1.20.0043.88 · 0.0043.93
  2. d1.10.0076.83 · 0.0076.66
  3. d1.20.0076.83 · 0.0076.66
  4. d10.0077.43 · 0.0076.66
  5. GLiNER2.5-multi-Decide0.0066.27 · 0.0076.73
  6. Perplexity Decider 27B v1.2 API0.0087.70 · 0.0087.79
  7. Mercury Decide0.0087.71 · 0.0087.79
  8. Laya0.0099.00 · 0.0087.84
  9. Perplexity Decider 27B v1.2 DGX Spark0.0087.76 · 0.0087.85
  10. d1-2B0.01313.17 · 0.0088.08
  11. GLiNER2.5 Multi0.0065.88 · 0.0088.39
  12. Laya Multilingual0.0087.74 · 0.01312.62
  13. d1-ultra0.01414.34 · 0.01413.98
  14. d1.2-ultra0.01414.34 · 0.01413.98
  15. Microsoft-Decision-10.01413.82 · 0.01414.01
  16. Elephant-10.01413.87 · 0.01414.02
  17. GLiDE v1.20.01414.37 · 0.01514.55
  18. Decider 10.01514.76 · 0.01514.95
  19. SubQ Snap 1.10.01515.41 · 0.01615.59
  20. Perplexity Decider 27B v10.01615.54 · 0.01615.59
  21. celeris-1-decision0.01515.28 · 0.01616.09
  22. Clef-Flash0.01717.37 · 0.01817.53
  23. Strands Decider 2B0.02020.12 · 0.01818.15
  24. Drex 2.00.01919.15 · 0.01919.42
  25. SubQ Snap Preview0.01919.41 · 0.01919.48
  26. Perplexity Decider Flash v1.30.02524.69 · 0.02323.17
  27. Drex Max v1.10.02323.11 · 0.02323.38
  28. Drex Max0.02323.29 · 0.02423.56
  29. Adjudex 9B v50.02626.30 · 0.02524.93
  30. Dexy-Decision-9B-V20.02323.03 · 0.02525.41
  31. Jev 1.140.02928.84 · 0.02726.64
  32. Jev 1.130.02928.85 · 0.02726.64
  33. Chinese Decision Pointer 4B0.02524.58 · 0.02727.21
  34. Qwen3-14B reasoning off · chat0.03232.31 · 0.03131.17
  35. Verdin-4B-v1.10.03434.21 · 0.03434.28
  36. vera-medium-omni0.02120.72 · 0.03736.63
  37. KnowLine-9B-Gen20.03535.40 · 0.03939.48
  38. Gemini Decision 1 202611110.03737.19 · 0.04039.51
  39. Mammoth-10.04039.62 · 0.04040.07
  40. Valen-Decision-1.5-9B0.04040.49 · 0.04140.55
  41. GPT-6 Luna Decisions API0.04040.44 · 0.04141.09
  42. Cygnet0.03939.39 · 0.04141.42
  43. KnowLine-9B-Gen10.04140.69 · 0.04241.55
  44. Jeffan-9B-12110.04444.29 · 0.04645.85
  45. GLM-5.3-Flash reasoning off · chat0.04747.28 · 0.04646.07
  46. Winnow-12B0.04141.40 · 0.05049.59
  47. Qwen3.8 Flash no reasoning · chat0.04747.13 · 0.05049.69
  48. Gemma 4 31B reasoning off · chat0.04848.22 · 0.05352.81
  49. d30.07070.21 · 0.05555.35
  50. KnowLine-12B-Gen10.05252.28 · 0.05756.86
  51. GPT-6 Luna low · chat0.04848.01 · 0.05857.70
  52. Metask-Jev-4B0.06362.82 · 0.05959.11
  53. Gemma 4 31B reasoning off · chat · 1 per request0.06868.19 · 0.07070.17
  54. DeepSeek-V4.1-Flash reasoning off · chat · 10170.09898.47 · 0.07979.31
  55. GLM-5.3-Flash reasoning on · chat0.543543 · 0.08181.38
  56. Perplexity Decider 27B v1.30.08181.18 · 0.08382.63
  57. OneJev-27B0.09999.09 · 0.106106
  58. d3-ultra0.216216 · 0.106106
  59. Clef0.110110 · 0.111111
  60. Erk-14B reasoning off · chat0.106106 · 0.123123
  61. KnowLine-31B-Gen20.120120 · 0.127127
  62. Copernicus-31B0.134134 · 0.142142
  63. Karpo Decision 31B0.105105 · 0.145145
  64. REAX 1.30.185185 · 0.169169
  65. Logits Sigma-1 Duo0.181181 · 0.191191
  66. MiniMax-M3.1-Flash high · chat0.814814 · 0.193193
  67. Gemma 4 E4B default reasoning · chat0.307307 · 0.199199
  68. Qwen3.8-27B stated confidence · reasoning off · chat0.234234 · 0.212212
  69. Muse Ember max · chat0.650650 · 0.245245
  70. Qwen3.5-2B reasoning off · chat0.268268 · 0.280280
  71. Gemini 3.8 Flash low · chat0.269269 · 0.288288
  72. MiMo-V2.6-Flash high · chat2.2192,219 · 0.298298
  73. Qwen3.5-4B reasoning off · chat0.894894 · 0.309309
  74. Open-Jev-27B-v1.10.271271 · 0.351351
  75. Claude Sonnet 5.5 low · chat1.8961,896 · 0.354354
  76. LFM3-1.7B reasoning off · chat0.646646 · 0.355355
  77. LFM3-1.7B reasoning on · chat1.2771,277 · 0.387387
  78. GLM-5.3 low · chat0.814814 · 0.408408
  79. DeepSeek-V4.1-Flash reasoning on · chat · 10171.2781,278 · 0.421421
  80. Inkling-Small high · chat · Jul 20261.1371,137 · 0.425425
  81. MiniCPM5-2B reasoning off · chat5.4645,464 · 0.429429
  82. GPT-6.1 Terra low · chat0.426426 · 0.491491
  83. GPT-6.1 Terra high · chat0.515515 · 0.509509
  84. LFM2.5-8B-A1B default reasoning · chat1.0501,050 · 0.517517
  85. Qwen3.8-27B vote share · reasoning off · chat0.825825 · 0.645645
  86. Kimi K3.1 Pro no reasoning · chat0.853853 · 0.735735
  87. Qwen3.8-27B yes/no scoring0.704704 · 0.741741
  88. Qwen3.8-27B low · chat3.0953,095 · 0.853853
  89. Gemini 3.8 Flash low · chat · 1 per request0.963963 · 0.858858
  90. LFM2.5-2.6B default reasoning · chat2.7612,761 · 0.964964
  91. Kimi K3.1 Pro low · chat2.0482,048 · 0.984984
  92. Perplexity Decider 27B v1.2 API · with web≤ 1.034≤ 1,034 · ≤ 1.036≤ 1,036
  93. Claude Opus 5.5 low · chat4.9314,931 · 1.0871,087
  94. Qwen4-Max high · chat2.5332,533 · 1.1191,119
  95. GPT-6.1 Sol low · chat1.1301,130 · 1.2191,219
  96. Claude Haiku 5.5 max · chat2.9652,965 · 1.2611,261
  97. Qwen3.5-4B reasoning on · chat2.9982,998 · 1.3901,390
  98. GPT-6.1 Sol high · chat1.6221,622 · 1.4271,427
  99. Qwen3.5-397B-A17B high · chat2.4112,411 · 1.7421,742
  100. Gemini 4 Argon low · chat1.7861,786 · 1.9001,900
  101. Claude Sonnet 5.5 high · chat3.6543,654 · 1.9981,998
  102. Gemini 3.8 Flash high · chat2.4192,419 · 3.6953,695
  103. Claude Opus 5.5 high · chat6.5096,509 · 4.5194,519
  104. GPT-6 Astra medium · chat5.4475,447 · 4.6964,696
  105. Gemini 4 Hydrogen high · chat4.8064,806 · 4.9274,927
  106. Claude Sonnet 5.5 high · chat · 1 per request7.7487,748 · 5.5135,513
  107. GPT-6 Astra low · chat5.3785,378 · 5.9185,918
  108. GPT-6.1 Sol low · chat · 1 per request5.5695,569 · 6.1136,113
  109. GPT-6 Astra high · chat6.0666,066 · 6.3346,334
  110. Grok 4.7 high · chat14.29214,292 · 6.8516,851
  111. Claude Opus 5.5 low · chat · 1 per request11.61011,610 · 10.09410,094
  112. Claude Fable 5.1 high · chat17.04217,042 · 11.53111,531
  113. GPT-6 Astra low · chat · 1 per request26.73826,738 · 28.65328,653
  114. Gemini 4 Carbon high · chat31.30931,309 · 38.05338,053
  115. Claude Fable 5.2 max · chat53.38153,381 · 52.94252,942
  116. Cerno 9B v1.20.03636.02 · no list price
  117. GLiClass v4 Edgeno list price · no list price
  118. GLiDE 0.4Bno list price · no list price
  119. GLiDE 0.8Bno list price · no list price
  120. GLiDE 27Bno list price · no list price
  121. GLiDE 2Bno list price · no list price
  122. GLiDE 9Bno list price · no list price
  123. KnowLine 4B Gen5.5no list price · no list price
  124. KnowLine-0.8B-Gen1no list price · no list price
  125. KnowLine-2B-Gen1no list price · no list price
  126. KnowLine-31B-Gen1.5no list price · no list price
  127. OneJev1.6-0.8Bno list price · no list price
  128. Sieve 31B v3no list price · no list price
  129. StartLux-Decision-v1.1-0.8B-Instantno list price · no list price
  130. StartLux-Decision-v1.1-2B-Instantno list price · no list price
  131. StartLux-Decision-v1.1-4B-Instantno list price · no list price
  132. StartLux-Decision-v1.1-9B-Instantno list price · no list price
  133. Turion Jev 31Bno list price · no list price

Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Perplexity Liquid Fastino Inception Laya Microsoft Elephant meraGPT SubQ Celeris Clef Strands Drex Adjudex Dexy Jev Chinese Decision Pointer Qwen Verdin vera KnowLine Gemini Mammoth Valen GPT Cygnet Jeffan GLM Winnow Gemma vLLM-SR Metask DeepSeek OneJev Erk Copernicus Karpo REAX Logits MiniMax Muse MiMo Open-Jev Claude Inkling MiniCPM Kimi Grok Cerno GLiClass Sieve StartLux Turion Jev Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. No list price: on our own GPUs, with no estimate. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. Log scale. No list price in Turkish: GLiClass v4 Edge, GLiDE 0.4B, GLiDE 0.8B, GLiDE 27B, GLiDE 2B, GLiDE 9B, KnowLine 4B Gen5.5, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, KnowLine-31B-Gen1.5, OneJev1.6-0.8B, Sieve 31B v3, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, StartLux-Decision-v1.1-9B-Instant and Turion Jev 31B, on our own hardware; the maker lists no price for the same model. No list price in English: Cerno 9B v1.2, GLiClass v4 Edge, GLiDE 0.4B, GLiDE 0.8B, GLiDE 27B, GLiDE 2B, GLiDE 9B, KnowLine 4B Gen5.5, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, KnowLine-31B-Gen1.5, OneJev1.6-0.8B, Sieve 31B v3, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, StartLux-Decision-v1.1-9B-Instant and Turion Jev 31B, on our own hardware; the maker lists no price for the same model.

Median latency per request in Turkish and in English

Median time per request: each of the 133 setups' Text tab run and its first English run, ordered by English latency · lower is better

  1. GLiNER2.5-multi-Decide0.009 · 0.009
  2. GLiNER2.5 Multi0.032 · 0.009
  3. Laya0.011 · 0.011
  4. d1-2B0.019 · 0.011
  5. GLiDE 0.4B0.016 · 0.014
  6. GLiClass v4 Edge0.025 · 0.017
  7. OneJev1.6-0.8B0.017 · 0.018
  8. Laya Multilingual0.009 · 0.018
  9. GLiDE 0.8B0.022 · 0.023
  10. Strands Decider 2B0.026 · 0.026
  11. StartLux-Decision-v1.1-0.8B-Instant0.028 · 0.028
  12. Perplexity Decider Flash v1.30.033 · 0.033
  13. KnowLine-0.8B-Gen10.033 · 0.033
  14. Adjudex 9B v50.038 · 0.034
  15. GLiDE 2B0.034 · 0.035
  16. Dexy-Decision-9B-V20.031 · 0.036
  17. Perplexity Decider Flash v1.20.132 · 0.037
  18. Chinese Decision Pointer 4B0.035 · 0.038
  19. StartLux-Decision-v1.1-2B-Instant0.040 · 0.040
  20. Cygnet0.036 · 0.043
  21. Verdin-4B-v1.10.049 · 0.052
  22. vera-medium-omni0.029 · 0.054
  23. KnowLine-9B-Gen20.048 · 0.056
  24. KnowLine-9B-Gen10.057 · 0.059
  25. Valen-Decision-1.5-9B0.059 · 0.059
  26. KnowLine-2B-Gen10.062 · 0.062
  27. Jeffan-9B-12110.064 · 0.067
  28. Elephant-10.074 · 0.070
  29. Winnow-12B0.059 · 0.070
  30. d30.092 · 0.074
  31. StartLux-Decision-v1.1-4B-Instant0.074 · 0.075
  32. KnowLine-12B-Gen10.071 · 0.078
  33. Drex Max0.082 · 0.081
  34. Metask-Jev-4B0.090 · 0.085
  35. KnowLine 4B Gen5.50.099 · 0.099
  36. GLiDE v1.20.413 · 0.104
  37. Perplexity Decider 27B v1.30.110 · 0.115
  38. Perplexity Decider 27B v1.2 DGX Spark0.580 · 0.126
  39. OneJev-27B0.125 · 0.130
  40. Clef-Flash0.075 · 0.140
  41. d3-ultra0.333 · 0.149
  42. StartLux-Decision-v1.1-9B-Instant0.154 · 0.156
  43. KnowLine-31B-Gen20.168 · 0.171
  44. Jev 1.130.243 · 0.173
  45. d1.10.195 · 0.173
  46. d10.305 · 0.180
  47. Jev 1.140.211 · 0.181
  48. Copernicus-31B0.184 · 0.188
  49. Mammoth-10.169 · 0.189
  50. GLiDE 9B0.207 · 0.207
  51. Karpo Decision 31B0.159 · 0.207
  52. Cerno 9B v1.20.051 · 0.211
  53. d1.20.213 · 0.219
  54. d1-ultra0.223 · 0.220
  55. Clef0.205 · 0.220
  56. d1.2-ultra0.220 · 0.221
  57. REAX 1.30.250 · 0.229
  58. Gemini Decision 1 202611110.227 · 0.235
  59. Perplexity Decider 27B v10.216 · 0.247
  60. Perplexity Decider 27B v1.2 API0.254 · 0.256
  61. Logits Sigma-1 Duo0.250 · 0.263
  62. Drex 2.00.325 · 0.283
  63. Gemma 4 31B reasoning off · chat · 1 per request0.298 · 0.299
  64. Mercury Decide0.313 · 0.310
  65. Microsoft-Decision-10.286 · 0.315
  66. GPT-6 Luna Decisions API0.191 · 0.323
  67. Drex Max v1.10.313 · 0.335
  68. SubQ Snap 1.10.378 · 0.381
  69. SubQ Snap Preview0.389 · 0.381
  70. celeris-1-decision0.401 · 0.390
  71. Qwen3.8-27B stated confidence · reasoning off · chat0.433 · 0.390
  72. Qwen3.8-27B yes/no scoring0.280 · 0.413
  73. Turion Jev 31B0.405 · 0.445
  74. Perplexity Decider 27B v1.2 API · with web0.482 · 0.484
  75. Open-Jev-27B-v1.10.386 · 0.495
  76. GLiDE 27B0.542 · 0.545
  77. Decider 10.382 · 0.699
  78. Sieve 31B v31.00 · 1.06
  79. KnowLine-31B-Gen1.51.05 · 1.12
  80. Qwen3.8-27B vote share · reasoning off · chat1.62 · 1.70
  81. Gemini 3.8 Flash low · chat · 1 per request2.72 · 2.85
  82. Erk-14B reasoning off · chat2.39 · 2.86
  83. Qwen3-14B reasoning off · chat2.85 · 2.86
  84. Claude Sonnet 5.5 high · chat · 1 per request4.64 · 3.02
  85. Gemma 4 31B reasoning off · chat3.97 · 3.19
  86. GLM-5.3-Flash reasoning off · chat1.07 · 3.37
  87. Claude Opus 5.5 low · chat · 1 per request3.71 · 3.50
  88. Claude Sonnet 5.5 low · chat3.87* · 3.68
  89. Claude Sonnet 5.5 high · chat20.5* · 3.85
  90. Gemini 3.8 Flash low · chat2.19 · 4.19
  91. LFM3-1.7B reasoning off · chat4.90 · 4.69
  92. LFM3-1.7B reasoning on · chat24.2 · 5.24
  93. GPT-6.1 Sol low · chat · 1 per request6.61 · 5.59
  94. Gemini 4 Argon low · chat5.46 · 6.03
  95. Kimi K3.1 Pro no reasoning · chat5.75 · 6.31
  96. GPT-6 Astra low · chat · 1 per request6.41 · 6.43
  97. DeepSeek-V4.1-Flash reasoning off · chat · 10175.08 · 6.59
  98. MiMo-V2.6-Flash high · chat21.4 · 6.67
  99. GLM-5.3 low · chat17.1 · 6.84
  100. MiniCPM5-2B reasoning off · chat172.0 · 7.29
  101. GPT-6.1 Terra low · chat4.24 · 7.29
  102. Claude Opus 5.5 low · chat13.4* · 7.64
  103. GPT-6 Luna low · chat7.71 · 7.72
  104. GPT-6.1 Terra high · chat7.10 · 7.94
  105. Gemma 4 E4B default reasoning · chat13.4 · 8.39
  106. Qwen3.5-2B reasoning off · chat8.84 · 9.27
  107. Qwen3.5-4B reasoning off · chat32.2 · 9.74
  108. Qwen3.8 Flash no reasoning · chat10.0 · 10.1
  109. GPT-6.1 Sol low · chat13.3 · 10.3
  110. GPT-6 Astra medium · chat15.3 · 10.7
  111. Claude Opus 5.5 high · chat22.1 · 11.3
  112. GPT-6 Astra low · chat13.0* · 11.5
  113. Muse Ember max · chat30.9 · 11.6
  114. LFM2.5-2.6B default reasoning · chat51.2 · 15.0
  115. GPT-6 Astra high · chat21.7 · 15.6
  116. Claude Fable 5.1 high · chat30.3 · 15.6
  117. Kimi K3.1 Pro low · chat44.9 · 16.6
  118. GPT-6.1 Sol high · chat24.7 · 16.9
  119. LFM2.5-8B-A1B default reasoning · chat39.2 · 19.4
  120. Inkling-Small high · chat · Jul 202697.7 · 25.3
  121. MiniMax-M3.1-Flash high · chat112.6 · 26.7
  122. Gemini 4 Hydrogen high · chat31.7 · 28.4
  123. Qwen4-Max high · chat91.5 · 29.5
  124. GLM-5.3-Flash reasoning on · chat102.6 · 33.1
  125. Qwen3.5-4B reasoning on · chat140.3 · 58.3
  126. Qwen3.8-27B low · chat648.4 · 71.7
  127. Claude Fable 5.2 max · chat81.1 · 74.0
  128. Gemini 3.8 Flash high · chat53.7 · 104.6
  129. Claude Haiku 5.5 max · chat417.0 · 148.9
  130. Qwen3.5-397B-A17B high · chat251.3 · 162.9
  131. DeepSeek-V4.1-Flash reasoning on · chat · 1017486.9 · 189.4
  132. Grok 4.7 high · chat401.9 · 197.2
  133. Gemini 4 Carbon high · chat217.7 · 287.8

Turkish (the Text tab run), the upper bar English (first run), the lower bar, striped Numbers on the right: Turkish · English. Fastino Laya Liquid GLiClass OneJev Strands StartLux Perplexity KnowLine Adjudex Dexy Chinese Decision Pointer Cygnet Verdin vera Valen Jeffan Elephant Winnow vLLM-SR Drex Metask Clef Jev Copernicus Mammoth Karpo Cerno REAX Gemini Logits Gemma Inception Microsoft GPT SubQ Celeris Qwen Turion Jev Open-Jev meraGPT Sieve Erk Claude GLM Kimi DeepSeek MiMo MiniCPM Muse Inkling MiniMax Grok Time per request as measured from our computer during each run; API times include the network. Text tab runs: 1 question per request for 95 runs and 16 for 38; 1 request in flight for 107 runs, 4 for 14, 32 for 10 and 8 for 2. English runs: 1 question per request for 95 runs and 16 for 38; 1 request in flight for 107 runs, 4 for 14, 32 for 9 and 8 for 3. Of the 77 self-hosted setups, KnowLine-31B-Gen1.5, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, StartLux-Decision-v1.1-9B-Instant, GLiDE 9B, Cerno 9B v1.2, StartLux-Decision-v1.1-4B-Instant, KnowLine 4B Gen5.5, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1 and GLiClass v4 Edge ran their English runs on an NVIDIA DGX Spark, the others on our RTX PRO 6000 GPUs; their Text tab runs ran on the same machine, except those of GLiDE · v1.2, Perplexity Decider 27B v1.2 · DGX Spark and Perplexity Decider Flash v1.2, which ran on an NVIDIA DGX Spark; those of Cerno 9B v1.2, which ran on our RTX PRO 6000 GPUs (Our method). Clef and Clef-Flash ran their English runs at the same time on one GPU. * Measured on a sample (200 questions): the Text tab setup's repeats on a fixed sample, as on the home page. Log scale.

Risk and coverage

How often each setup answers and how often its answers are wrong, on the same 547 kept questions in both languages: without the option, by holding back the answers below a confidence score, and with the option, as the model chose it.

Method

Answered = right + wrong (an invalid answer or one without a score is never answered). Coverage = answered / 547. Error rate among answered (risk) = wrong / answered. Abstained: for a run with the option, the times it chose the option; for a confidence point, 547 minus answered minus invalid. Each confidence curve has one point per distinct score of the run on these questions (answer when the score is at least that score; the score is the recorded probability of the chosen option for a decision model and the confidence a chat model stated with its answer): answers with equal scores enter together, so coverage moves in jumps and every point is one the run achieved; a setup that reports its scores with few decimals has many equal scores, so its curve has fewer points, moves in larger steps and can start well above 0% coverage (each curve is drawn from its first point). Every series counts these questions only (a run on the full bank is restricted to them) and is each setup's first run. Turkish curves: the Text tab run; English curves: the first English run. Turkish with “Bilmiyorum” is the Turkish side of the English “I don't know” comparison or, for a model outside it, its Hallucination tab run, restricted to these; Turkish with “I don't know” is the option's word control on the Hallucination tab's questions, restricted to these, for decision models only. A run not yet in reads Not run for a decision model and Running for a chat model. A setup asked for a plain answer line (one question per request, no stated confidence and no option probabilities) has no confidence curve: not measured, only its runs with the option.

All series are counted on the same kept reading questions; a full-bank run is restricted to them; first runs only. Answered = right + wrong (an invalid answer or one without a score is never answered). Coverage = answered / kept questions. Risk = error rate among answered = wrong / answered. Abstained: for a native point, the times the model chose the option; for a curve point, kept questions - answered - invalid. A native point is the run with its abstention option (Turkish "Bilmiyorum", Turkish "I don't know" as the wording control, English "I don't know"); Turkish "Bilmiyorum" is the Turkish side of the English "I don't know" comparison or, for a model outside it, its Hallucination tab run; a missing one reads Not run (Running while a chat model's run is still going). A curve is one five-option run held back by its recorded score (the probability of the chosen option): one point per distinct score s, answering when the score is at least s; answers with equal scores enter together, so coverage moves in jumps and every point is an achieved one; all distinct points kept, ordered by coverage.

The runs with the option

ModelTurkish with “Bilmiyorum”coverage · error rate among answeredTurkish with “I don't know”coverage · error rate among answeredEnglish with “I don't know”coverage · error rate among answered
Claude Fable 5.2 max · chat Closed · FlagshipQueuedits run with the option has not started yetNot rundecision models only99.3% · 1.8%533 right, 10 wrong, 4 abstained · run 1032
Gemini 4 Hydrogen high · chat Closed · Workhorse99.3% · 1.8%533 right, 10 wrong, 4 abstained · run 1117, full bank restrictedNot rundecision models only99.3% · 2.9%527 right, 16 wrong, 4 abstained · run 1109
GPT-6 Astra low · chat Closed · Flagship99.3% · 2.6%529 right, 14 wrong, 4 abstained · run 816, full bank restrictedNot rundecision models only99.3% · 3.3%525 right, 18 wrong, 4 abstained · run 647
GPT-6 Astra high · chat Closed · Flagship99.3% · 3.1%526 right, 17 wrong, 4 abstained · run 818, full bank restrictedNot rundecision models only99.3% · 3.1%526 right, 17 wrong, 4 abstained · run 814
GPT-6.1 Terra high · chat Closed · Workhorse99.1% · 3.5%523 right, 19 wrong, 5 abstained · run 807, full bank restrictedNot rundecision models only99.3% · 3.7%523 right, 20 wrong, 4 abstained · run 766
GPT-6.1 Terra low · chat Closed · Workhorse99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 801, full bank restrictedNot rundecision models only99.3% · 4.1%521 right, 22 wrong, 4 abstained · run 751
Claude Fable 5.1 high · chat Closed · Flagship99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 695, full bank restrictedNot rundecision models only99.6% · 4.6%520 right, 25 wrong, 2 abstained · run 655
Gemini 3.8 Flash high · chat Closed · Workhorse99.6% · 2.4%532 right, 13 wrong, 2 abstained · run 681, full bank restrictedNot rundecision models only99.6% · 3.3%527 right, 18 wrong, 2 abstained · run 657
Gemini 4 Argon low · chat Closed · Flagship99.3% · 2.4%530 right, 13 wrong, 4 abstained · run 679, full bank restrictedNot rundecision models only99.1% · 4.1%520 right, 22 wrong, 5 abstained · run 652
GPT-6 Astra medium · chat Closed · Flagship99.3% · 2.9%527 right, 16 wrong, 4 abstained · run 817, full bank restrictedNot rundecision models only99.3% · 3.7%523 right, 20 wrong, 4 abstained · run 813
GPT-6.1 Sol low · chat Closed · Workhorse99.3% · 3.3%525 right, 18 wrong, 4 abstained · run 799, full bank restrictedNot rundecision models only99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 744
GPT-6 Astra low · chat · 1 per request Closed · Flagship99.3% · 3.5%524 right, 19 wrong, 4 abstained · run 819, full bank restrictedNot rundecision models only99.3% · 4.2%520 right, 23 wrong, 4 abstained · run 815
GPT-6.1 Sol high · chat Closed · Workhorse99.3% · 2.9%527 right, 16 wrong, 4 abstained · run 802, full bank restrictedNot rundecision models only99.3% · 3.7%523 right, 20 wrong, 4 abstained · run 758
Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse99.5% · 3.3%526 right, 18 wrong, 3 abstained · run 691, full bank restrictedNot rundecision models only99.6% · 4.4%521 right, 24 wrong, 2 abstained · run 669
GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse99.3% · 3.5%524 right, 19 wrong, 4 abstained · run 810, full bank restrictedNot rundecision models only99.3% · 5.0%516 right, 27 wrong, 4 abstained · run 769
Claude Opus 5.5 low · chat · 1 per request Closed · Flagship99.8% · 4.6%521 right, 25 wrong, 1 abstained · run 803, full bank restrictedNot rundecision models only100.0% · 5.7%516 right, 31 wrong, 0 abstained · run 670
Claude Opus 5.5 high · chat Closed · Flagship99.5% · 4.2%521 right, 23 wrong, 3 abstained · run 773, full bank restrictedNot rundecision models only99.6% · 5.3%516 right, 29 wrong, 2 abstained · run 665
Qwen4-Max high · chat Closed · Workhorse99.8% · 5.1%518 right, 28 wrong, 1 abstained · run 1339, full bank restrictedNot rundecision models only100.0% · 5.3%518 right, 29 wrong, 0 abstained · run 1338
Gemini 3.8 Flash low · chat Closed · Workhorse99.5% · 4.4%520 right, 24 wrong, 3 abstained · run 676, full bank restrictedNot rundecision models only99.8% · 5.1%518 right, 28 wrong, 1 abstained · run 645
Gemini 4 Carbon high · chat Closed · Flagship99.3% · 2.0%532 right, 11 wrong, 4 abstained · run 1121, full bank restrictedNot rundecision models only99.3% · 2.2%531 right, 12 wrong, 4 abstained · run 1067
KnowLine-31B-Gen1.5 Open · 31B100.0% · 6.2%513 right, 34 wrong, 0 abstained · run 959, full bank restrictedNot run100.0% · 6.8%510 right, 37 wrong, 0 abstained · run 955
Grok 4.7 high · chat Closed · Workhorse98.5% · 3.9%518 right, 21 wrong, 8 abstained · run 675, full bank restrictedNot rundecision models only97.8% · 5.8%504 right, 31 wrong, 12 abstained · run 648
Claude Opus 5.5 low · chat Closed · Flagship99.6% · 5.3%516 right, 29 wrong, 2 abstained · run 683, full bank restrictedNot rundecision models only99.6% · 7.5%504 right, 41 wrong, 2 abstained · run 649
Copernicus-31B Open · 31B99.8% · 6.0%513 right, 33 wrong, 1 abstained · run 1172, full bank restrictedNot run100.0% · 7.1%508 right, 39 wrong, 0 abstained · run 1170
Claude Haiku 5.5 max · chat Closed · Efficient96.0% · 5.5%496 right, 29 wrong, 6 abstained · run 950, full bank restrictedNot rundecision models only96.3% · 8.9%480 right, 47 wrong, 4 abstained · run 951
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active99.8% · 7.0%508 right, 38 wrong, 1 abstained · run 811, full bank restrictedNot rundecision models only99.5% · 7.2%505 right, 39 wrong, 3 abstained · run 809
REAX 1.3 Open · 57B99.8% · 6.4%511 right, 35 wrong, 1 abstained · run 833, full bank restrictedNot run99.5% · 7.7%502 right, 42 wrong, 3 abstained · run 831
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active99.5% · 6.6%508 right, 36 wrong, 3 abstained · run 686, full bank restrictedNot rundecision models only99.5% · 7.0%506 right, 38 wrong, 3 abstained · run 651
KnowLine-31B-Gen2 Open · 31B100.0% · 5.3%518 right, 29 wrong, 0 abstained · run 1331, full bank restrictedNot run100.0% · 7.3%507 right, 40 wrong, 0 abstained · run 1329
Karpo Decision 31B Open · 31B99.8% · 5.5%516 right, 30 wrong, 1 abstained · run 1163, full bank restrictedNot run100.0% · 7.7%505 right, 42 wrong, 0 abstained · run 1161
Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse99.8% · 6.6%510 right, 36 wrong, 1 abstained · run 812, full bank restrictedNot rundecision models only100.0% · 6.9%509 right, 38 wrong, 0 abstained · run 677
Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active99.5% · 7.2%505 right, 39 wrong, 3 abstained · run 653, full bank restrictedNot rundecision models only99.3% · 9.0%494 right, 49 wrong, 4 abstained · run 646
Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active99.3% · 7.9%500 right, 43 wrong, 4 abstained · run 872, full bank restrictedNot rundecision models only99.5% · 8.3%499 right, 45 wrong, 3 abstained · run 873
Gemini Decision 1 20261111 Closed · Efficient99.1% · 7.6%501 right, 41 wrong, 5 abstained · run 1023, full bank restrictedNot run99.1% · 9.2%492 right, 50 wrong, 5 abstained · run 1019
Logits Sigma-1 Duo Open · 57B100.0% · 6.6%511 right, 36 wrong, 0 abstained · run 1233, full bank restrictedNot run99.6% · 8.1%501 right, 44 wrong, 2 abstained · run 1231
GLiDE 27B Open · 27B98.4% · 7.2%499 right, 39 wrong, 9 abstained · run 905, full bank restrictedNot run96.0% · 6.9%489 right, 36 wrong, 22 abstained · run 903
d1-ultra Closed · Efficient94.5% · 5.0%491 right, 26 wrong, 30 abstained · run 350, full bank restricted95.1% · 5.4%492 right, 28 wrong, 27 abstained · run 511, full bank restricted90.7% · 7.9%457 right, 39 wrong, 51 abstained · run 449
Sieve 31B v3 Open · 31B100.0% · 6.0%514 right, 33 wrong, 0 abstained · run 1253, full bank restrictedNot run99.8% · 8.2%501 right, 45 wrong, 1 abstained · run 1066
Turion Jev 31B Open · 31B99.6% · 6.1%512 right, 33 wrong, 2 abstained · run 1285, full bank restrictedNot run99.3% · 7.9%500 right, 43 wrong, 4 abstained · run 1077
Gemma 4 31B reasoning off · chat Open · 31B99.6% · 7.7%503 right, 42 wrong, 2 abstained · run 776, full bank restrictedNot rundecision models only99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 775
Perplexity Decider 27B v1.2 API · with web Open · 27B99.5% · 9.4%493 right, 51 wrong, 3 abstained · run 642, full bank restrictedNot run98.2% · 10.4%481 right, 56 wrong, 10 abstained · run 644
GLiDE v1.2 Open · 27B98.9% · 8.7%494 right, 47 wrong, 6 abstained · run 369, full bank restricted98.4% · 8.7%491 right, 47 wrong, 9 abstained · run 525, full bank restricted94.5% · 7.0%481 right, 36 wrong, 30 abstained · run 523
d1.2-ultra Closed · Efficient88.8% · 3.9%467 right, 19 wrong, 61 abstained · run 1367, full bank restricted89.0% · 3.7%469 right, 18 wrong, 60 abstained · run 1378, full bank restricted80.1% · 3.0%425 right, 13 wrong, 109 abstained · run 1363
Mammoth-1 Open · 27B95.6% · 6.3%490 right, 33 wrong, 24 abstained · run 386, full bank restricted95.4% · 6.1%490 right, 32 wrong, 25 abstained · run 519, full bank restricted93.2% · 7.1%474 right, 36 wrong, 37 abstained · run 505
Gemma 4 31B reasoning off · chat · 1 per request Open · 31B99.8% · 9.5%494 right, 52 wrong, 1 abstained · run 789, full bank restrictedNot rundecision models only100.0% · 10.8%488 right, 59 wrong, 0 abstained · run 786
d3 Open · 26B100.0% · 9.7%494 right, 53 wrong, 0 abstained · run 358, full bank restricted100.0% · 9.5%495 right, 52 wrong, 0 abstained · run 544, full bank restricted99.8% · 10.6%488 right, 58 wrong, 1 abstained · run 793
Drex Max Open · 27B99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 367, full bank restricted99.8% · 9.0%497 right, 49 wrong, 1 abstained · run 542, full bank restricted99.6% · 11.0%485 right, 60 wrong, 2 abstained · run 798
GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active99.6% · 10.1%490 right, 55 wrong, 2 abstained · run 825, full bank restrictedNot rundecision models only99.8% · 11.2%485 right, 61 wrong, 1 abstained · run 824
Muse Ember max · chat Closed · Efficient99.3% · 8.7%496 right, 47 wrong, 4 abstained · run 940, full bank restrictedNot rundecision models only99.5% · 9.9%490 right, 54 wrong, 3 abstained · run 943
Perplexity Decider 27B v1.3 Open · 27B100.0% · 9.5%495 right, 52 wrong, 0 abstained · run 1141, full bank restrictedNot run99.1% · 9.6%490 right, 52 wrong, 5 abstained · run 1139
d3-ultra Open · 26B99.8% · 8.4%500 right, 46 wrong, 1 abstained · run 363, full bank restricted99.6% · 9.4%494 right, 51 wrong, 2 abstained · run 540, full bank restricted99.1% · 10.3%486 right, 56 wrong, 5 abstained · run 795
d1.2 Closed · Efficient90.3% · 7.1%459 right, 35 wrong, 53 abstained · run 1365, full bank restricted91.6% · 8.4%459 right, 42 wrong, 46 abstained · run 1377, full bank restricted89.2% · 7.6%451 right, 37 wrong, 59 abstained · run 1361
MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active99.5% · 12.7%475 right, 69 wrong, 3 abstained · run 879, full bank restrictedNot rundecision models only99.8% · 10.6%488 right, 58 wrong, 1 abstained · run 880
Qwen3.8-27B low · chat Open · 27B99.8% · 8.8%498 right, 48 wrong, 1 abstained · run 666, full bank restrictedNot rundecision models only99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 671
Perplexity Decider 27B v1.2 DGX Spark Open · 27B100.0% · 9.1%497 right, 50 wrong, 0 abstained · run 372, full bank restricted99.8% · 9.3%495 right, 51 wrong, 1 abstained · run 531, full bank restricted99.6% · 11.7%481 right, 64 wrong, 2 abstained · run 529
Drex Max v1.1 Closed · Efficient (provisional)98.4% · 9.5%487 right, 51 wrong, 9 abstained · run 352, full bank restricted96.7% · 9.1%481 right, 48 wrong, 18 abstained · run 513, full bank restricted92.1% · 8.1%463 right, 41 wrong, 43 abstained · run 447
Perplexity Decider 27B v1 Open · 27BNot runNot runNot run
Perplexity Decider 27B v1.2 API Open · 27B100.0% · 9.3%496 right, 51 wrong, 0 abstained · run 641, full bank restrictedNot run99.5% · 11.6%481 right, 63 wrong, 3 abstained · run 643
Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active100.0% · 11.3%485 right, 62 wrong, 0 abstained · run 868, full bank restrictedNot rundecision models only100.0% · 11.0%487 right, 60 wrong, 0 abstained · run 869
d1.1 Closed · EfficientNot runNot runNot run
GLM-5.3 low · chat Open · Workhorse99.5% · 11.2%483 right, 61 wrong, 3 abstained · run 656, full bank restrictedNot rundecision models only99.6% · 10.3%489 right, 56 wrong, 2 abstained · run 650
Jeffan-9B-1211 Open · 9.0B99.5% · 13.1%473 right, 71 wrong, 3 abstained · run 1199, full bank restrictedNot run100.0% · 13.0%476 right, 71 wrong, 0 abstained · run 1197
GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active99.5% · 15.1%462 right, 82 wrong, 3 abstained · run 822, full bank restrictedNot rundecision models only99.3% · 12.3%476 right, 67 wrong, 4 abstained · run 821
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active100.0% · 10.4%490 right, 57 wrong, 0 abstained · run 806, full bank restrictedNot rundecision models only99.3% · 12.9%473 right, 70 wrong, 4 abstained · run 805
Qwen3.5-4B reasoning on · chat Open · 4.0B99.1% · 16.2%454 right, 88 wrong, 2 abstained · run 1150, full bank restrictedNot rundecision models only99.6% · 15.2%462 right, 83 wrong, 2 abstained · run 1073
Qwen3.8 Flash no reasoning · chat Closed · Efficient99.5% · 10.1%489 right, 55 wrong, 3 abstained · run 987, full bank restrictedNot rundecision models only99.3% · 12.3%476 right, 67 wrong, 4 abstained · run 988
d1 Closed · Efficient99.8% · 16.3%457 right, 89 wrong, 1 abstained · run 472Not run99.6% · 12.3%478 right, 67 wrong, 2 abstained · run 445
StartLux-Decision-v1.1-9B-Instant Open · 9.0B99.8% · 17.4%451 right, 95 wrong, 1 abstained · run 1297, full bank restrictedNot run99.6% · 13.6%471 right, 74 wrong, 2 abstained · run 1295
Clef Open · 27B99.8% · 12.3%479 right, 67 wrong, 1 abstained · run 788, full bank restrictedNot run99.8% · 13.7%471 right, 75 wrong, 1 abstained · run 785
KnowLine-12B-Gen1 Open · 12B100.0% · 12.2%480 right, 67 wrong, 0 abstained · run 1318, full bank restrictedNot run99.8% · 13.9%470 right, 76 wrong, 1 abstained · run 1316
GLiDE 9B Open · 7.9B98.4% · 15.4%455 right, 83 wrong, 9 abstained · run 913, full bank restrictedNot run97.4% · 13.1%463 right, 70 wrong, 14 abstained · run 911
GPT-6 Luna low · chat Closed · Efficient99.6% · 11.9%480 right, 65 wrong, 2 abstained · run 796, full bank restrictedNot rundecision models only99.1% · 12.7%473 right, 69 wrong, 5 abstained · run 735
OneJev-27B Open · 27B99.8% · 11.5%483 right, 63 wrong, 1 abstained · run 782, full bank restrictedNot run99.5% · 13.2%472 right, 72 wrong, 3 abstained · run 780
Claude Sonnet 5.5 high · chat Closed · Workhorse99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 800, full bank restrictedNot rundecision models only99.5% · 10.8%485 right, 59 wrong, 3 abstained · run 668
Jev 1.14 Closed · Efficient99.5% · 12.7%475 right, 69 wrong, 3 abstained · run 470Not run99.5% · 13.2%472 right, 72 wrong, 3 abstained · run 446
Cerno 9B v1.2 Open · 9.0B100.0% · 16.6%456 right, 91 wrong, 0 abstained · run 1204, full bank restrictedNot run99.6% · 14.1%468 right, 77 wrong, 2 abstained · run 1087
Drex 2.0 Open · 35B MoE · 3.0B activeNot runNot runNot run
Jev 1.13 Closed · EfficientNot runNot runNot run
KnowLine-9B-Gen1 Open · 9.0B100.0% · 17.9%449 right, 98 wrong, 0 abstained · run 1181, full bank restrictedNot run99.8% · 13.9%470 right, 76 wrong, 1 abstained · run 1179
SubQ Snap 1.1 Closed · Efficient96.5% · 9.7%477 right, 51 wrong, 19 abstained · run 392, full bank restricted96.5% · 8.5%483 right, 45 wrong, 19 abstained · run 512, full bank restricted96.2% · 12.0%463 right, 63 wrong, 21 abstained · run 448
SubQ Snap Preview Closed · EfficientNot runNot runNot run
Adjudex 9B v5 Open · 9.0B100.0% · 16.3%458 right, 89 wrong, 0 abstained · run 1190, full bank restrictedNot run100.0% · 14.1%470 right, 77 wrong, 0 abstained · run 1188
Mercury Decide Closed · Efficient98.5% · 13.2%468 right, 71 wrong, 8 abstained · run 489Not run98.0% · 13.2%465 right, 71 wrong, 11 abstained · run 450
Microsoft-Decision-1 Closed · Efficient99.6% · 16.1%457 right, 88 wrong, 2 abstained · run 487Not run99.5% · 14.2%467 right, 77 wrong, 3 abstained · run 451
Elephant-1 Open · 9.0B97.3% · 15.4%450 right, 82 wrong, 15 abstained · run 382, full bank restricted95.6% · 15.7%441 right, 82 wrong, 24 abstained · run 515, full bank restricted95.8% · 12.8%457 right, 67 wrong, 23 abstained · run 501
Open-Jev-27B-v1.1 Open · 27B98.9% · 14.4%463 right, 78 wrong, 6 abstained · run 768, full bank restrictedNot run97.1% · 14.3%455 right, 76 wrong, 16 abstained · run 763
KnowLine-9B-Gen2 Open · 9.0B100.0% · 17.4%452 right, 95 wrong, 0 abstained · run 1325, full bank restrictedNot run99.8% · 14.3%468 right, 78 wrong, 1 abstained · run 1323
Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B99.3% · 13.3%471 right, 72 wrong, 4 abstained · run 674, full bank restrictedNot rundecision models only99.5% · 14.3%466 right, 78 wrong, 3 abstained · run 673
Qwen3.8-27B yes/no scoring Open · 27BNot runyes/no scoring cannot offer itNot rundecision models onlyNot runyes/no scoring cannot offer it
MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active99.6% · 16.5%455 right, 90 wrong, 1 abstained · run 864, full bank restrictedNot rundecision models only99.5% · 15.6%459 right, 85 wrong, 3 abstained · run 865
celeris-1-decision Closed · EfficientNot runNot run83.2% · 8.1%418 right, 37 wrong, 92 abstained · run 548
GPT-6 Luna Decisions API Closed · Efficient98.9% · 14.4%463 right, 78 wrong, 6 abstained · run 453Not run98.2% · 17.9%441 right, 96 wrong, 10 abstained · run 444
Claude Sonnet 5.5 low · chat Closed · Workhorse99.1% · 15.1%460 right, 82 wrong, 5 abstained · run 690, full bank restrictedNot rundecision models only99.3% · 15.8%457 right, 86 wrong, 4 abstained · run 654
Cygnet Open · 12B99.1% · 14.9%461 right, 81 wrong, 5 abstained · run 757, full bank restrictedNot run98.2% · 14.7%458 right, 79 wrong, 10 abstained · run 755
Qwen3.8-27B vote share · reasoning off · chat Open · 27B99.3% · 13.8%468 right, 75 wrong, 4 abstained · run 682, full bank restrictedNot rundecision models only99.5% · 15.4%460 right, 84 wrong, 3 abstained · run 680
Winnow-12B Open · 12B98.5% · 15.8%454 right, 85 wrong, 8 abstained · run 750, full bank restrictedNot run97.3% · 16.0%447 right, 85 wrong, 15 abstained · run 748
Perplexity Decider Flash v1.3 Open · 4.0B98.4% · 20.4%428 right, 110 wrong, 9 abstained · run 1130, full bank restrictedNot run98.7% · 16.7%450 right, 90 wrong, 7 abstained · run 1128
Valen-Decision-1.5-9B Open · 9.0B100.0% · 20.1%437 right, 110 wrong, 0 abstained · run 1282, full bank restrictedNot run99.8% · 15.4%462 right, 84 wrong, 1 abstained · run 1280
Dexy-Decision-9B-V2 Open · 9.0B99.8% · 16.5%456 right, 90 wrong, 1 abstained · run 1262, full bank restrictedNot run99.8% · 16.5%456 right, 90 wrong, 1 abstained · run 1260
vera-medium-omni Open · 4.2B99.1% · 20.8%429 right, 113 wrong, 5 abstained · run 1271, full bank restrictedNot run98.7% · 16.1%453 right, 87 wrong, 7 abstained · run 1269
StartLux-Decision-v1.1-4B-Instant Open · 4.0B99.6% · 21.7%427 right, 118 wrong, 2 abstained · run 1239, full bank restrictedNot run100.0% · 17.2%453 right, 94 wrong, 0 abstained · run 1083
Gemma 4 E4B default reasoning · chat Open · 8.0B95.4% · 18.4%426 right, 96 wrong, 25 abstained · run 1049, full bank restrictedNot rundecision models only97.8% · 17.2%443 right, 92 wrong, 10 abstained · run 890
Clef-Flash Open · 9.0B97.8% · 21.5%420 right, 115 wrong, 12 abstained · run 664, full bank restrictedNot run99.1% · 16.6%452 right, 90 wrong, 5 abstained · run 660
Verdin-4B-v1.1 Open · 4.0B99.8% · 23.3%419 right, 127 wrong, 1 abstained · run 1251, full bank restrictedNot run99.6% · 17.2%451 right, 94 wrong, 2 abstained · run 1249
Perplexity Decider Flash v1.2 Open · 4.0B98.9% · 24.6%408 right, 133 wrong, 6 abstained · run 537Not run99.1% · 18.5%442 right, 100 wrong, 5 abstained · run 535
KnowLine 4B Gen5.5 Open · 4.0B99.5% · 21.7%426 right, 118 wrong, 3 abstained · run 967, full bank restrictedNot run99.8% · 19.2%441 right, 105 wrong, 1 abstained · run 963
Chinese Decision Pointer 4B Open · 4.0B99.3% · 23.8%414 right, 129 wrong, 4 abstained · run 1213, full bank restrictedNot run99.3% · 19.2%439 right, 104 wrong, 4 abstained · run 1211
Qwen3.5-4B reasoning off · chat Open · 4.0B98.4% · 26.0%398 right, 140 wrong, 5 abstained · run 1147, full bank restrictedNot rundecision models only99.3% · 19.5%437 right, 106 wrong, 4 abstained · run 1144
Qwen3-14B reasoning off · chat Open · 15B99.8% · 22.3%424 right, 122 wrong, 1 abstained · run 772, full bank restrictedNot rundecision models only99.6% · 19.8%437 right, 108 wrong, 2 abstained · run 771
Metask-Jev-4B Open · 4.5B99.6% · 34.3%358 right, 187 wrong, 2 abstained · run 743, full bank restrictedNot run99.6% · 22.6%422 right, 123 wrong, 2 abstained · run 741
LFM2.5-2.6B default reasoning · chat Open · 2.7B90.7% · 46.2%267 right, 229 wrong, 43 abstained · run 1038, full bank restrictedNot rundecision models only95.6% · 21.6%410 right, 113 wrong, 21 abstained · run 1035
Erk-14B reasoning off · chat Open · 14B99.8% · 23.6%417 right, 129 wrong, 1 abstained · run 765, full bank restrictedNot rundecision models only99.5% · 25.9%403 right, 141 wrong, 3 abstained · run 764
LFM3-1.7B reasoning off · chat Open · 1.7B56.9% · 55.6%138 right, 173 wrong, 215 abstained · run 1234, full bank restrictedNot rundecision models only96.7% · 25.9%392 right, 137 wrong, 18 abstained · run 1225
KnowLine-2B-Gen1 Open · 2.0BNot runNot run93.1% · 26.5%374 right, 135 wrong, 38 abstained · run 1059
LFM3-1.7B reasoning on · chat Open · 1.7B80.6% · 47.6%231 right, 210 wrong, 106 abstained · run 1090, full bank restrictedNot rundecision models only95.2% · 27.1%380 right, 141 wrong, 26 abstained · run 1062
LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active52.5% · 67.9%92 right, 195 wrong, 201 abstained · run 1298, full bank restrictedNot rundecision models only90.5% · 28.7%353 right, 142 wrong, 51 abstained · run 887
GLiDE 2B Open · 1.9B87.9% · 43.5%272 right, 209 wrong, 66 abstained · run 921, full bank restrictedNot run91.4% · 26.4%368 right, 132 wrong, 47 abstained · run 919
StartLux-Decision-v1.1-2B-Instant Open · 2.0B94.0% · 45.7%279 right, 235 wrong, 33 abstained · run 979, full bank restrictedNot run94.1% · 30.5%358 right, 157 wrong, 32 abstained · run 975
MiniCPM5-2B reasoning off · chat Open · 2.5B20.8% · 65.8%39 right, 75 wrong, 22 abstained · run 1304, full bank restrictedNot rundecision models only89.2% · 33.6%324 right, 164 wrong, 33 abstained · run 1095
Qwen3.5-2B reasoning off · chat Open · 2.0B83.4% · 61.2%177 right, 279 wrong, 55 abstained · run 1219, full bank restrictedNot rundecision models only96.3% · 29.6%371 right, 156 wrong, 18 abstained · run 1216
d1-2B Open · 1.9B70.6% · 53.6%179 right, 207 wrong, 161 abstained · run 712, full bank restrictedNot run72.4% · 24.5%299 right, 97 wrong, 151 abstained · run 710
GLiDE 0.8B Open · 0.8B86.5% · 48.6%243 right, 230 wrong, 74 abstained · run 929, full bank restrictedNot run91.0% · 35.5%321 right, 177 wrong, 49 abstained · run 927
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B96.0% · 61.0%205 right, 320 wrong, 22 abstained · run 1017, full bank restrictedNot run97.8% · 40.6%318 right, 217 wrong, 12 abstained · run 1013
GLiDE 0.4B Open · 0.4B13.2% · 76.4%17 right, 55 wrong, 475 abstained · run 937, full bank restrictedNot run72.0% · 34.3%259 right, 135 wrong, 153 abstained · run 935
OneJev1.6-0.8B Open · 0.8B79.5% · 58.2%182 right, 253 wrong, 112 abstained · run 1031, full bank restrictedNot run86.5% · 39.5%286 right, 187 wrong, 74 abstained · run 1027
KnowLine-0.8B-Gen1 Open · 0.8B97.3% · 61.1%207 right, 325 wrong, 15 abstained · run 1007, full bank restrictedNot run95.2% · 41.7%304 right, 217 wrong, 26 abstained · run 1003
GLiClass v4 Edge Open · 0.4B89.4% · 72.0%137 right, 352 wrong, 58 abstained · run 1048, full bank restrictedNot run90.5% · 39.6%299 right, 196 wrong, 52 abstained · run 1044
Decider 1 Closed · EfficientNot runNot runNot run
Strands Decider 2B Open · 2.0B99.1% · 60.3%215 right, 327 wrong, 5 abstained · run 706, full bank restrictedNot run99.6% · 55.2%244 right, 301 wrong, 2 abstained · run 704
Laya Open · 0.4B90.3% · 78.9%104 right, 390 wrong, 53 abstained · run 730, full bank restrictedNot run84.8% · 79.1%97 right, 367 wrong, 83 abstained · run 728
GLiNER2.5-multi-Decide Open · 0.3B95.6% · 78.8%111 right, 412 wrong, 24 abstained · run 718, full bank restrictedNot run87.2% · 76.1%114 right, 363 wrong, 70 abstained · run 716
GLiNER2.5 Multi Open · 0.3B94.9% · 82.5%91 right, 428 wrong, 28 abstained · run 724, full bank restrictedNot rundecision models only83.2% · 83.1%77 right, 378 wrong, 92 abstained · run 722
Laya Multilingual Open · 0.3B77.3% · 84.9%64 right, 359 wrong, 124 abstained · run 737, full bank restrictedNot run62.0% · 82.6%59 right, 280 wrong, 208 abstained · run 734

Each cell: coverage · error rate among answered, then the counts and the run. Not run: no such run for a decision model, or the Turkish “I don't know” run, which is for decision models only; Running: a chat model's run still running.

Risk and coverage in Turkish

The 547 kept questions in Turkish: each setup's five-option run answering only at or above a confidence score (the line), and a run with the option (the marker: “Bilmiyorum” “I don't know”) · lower is better at the same coverage

0%20%40%60%80%100%0%25%50%75%100%Coverage (answered / 547)Error rate among answeredClaude Fable 5.2 · max · chat: Turkish, five options, held back by confidence, run 980 (34 points)Gemini 4 Hydrogen · high · chat: Turkish, five options, held back by confidence, run 1114 (19 points)GPT-6 Astra · low · chat: Turkish, five options, held back by confidence, run 48 (28 points)GPT-6 Astra · high · chat: Turkish, five options, held back by confidence, run 277 (26 points)GPT-6.1 Terra · high · chat: Turkish, five options, held back by confidence, run 275 (28 points)GPT-6.1 Terra · low · chat: Turkish, five options, held back by confidence, run 274 (24 points)Claude Fable 5.1 · high · chat: Turkish, five options, held back by confidence, run 310 (19 points)Gemini 3.8 Flash · high · chat: Turkish, five options, held back by confidence, run 140 (10 points)Gemini 4 Argon · low · chat: Turkish, five options, held back by confidence, run 557 (12 points)GPT-6 Astra · medium · chat: Turkish, five options, held back by confidence, run 610 (26 points)GPT-6.1 Sol · low · chat: Turkish, five options, held back by confidence, run 124 (22 points)GPT-6 Astra · low · chat · 1 per request: Turkish, five options, held back by confidence, run 191 (21 points)GPT-6.1 Sol · high · chat: Turkish, five options, held back by confidence, run 276 (24 points)Gemini 3.8 Flash · low · chat · 1 per request: Turkish, five options, held back by confidence, run 188 (10 points)GPT-6.1 Sol · low · chat · 1 per request: Turkish, five options, held back by confidence, run 190 (16 points)Claude Opus 5.5 · low · chat · 1 per request: Turkish, five options, held back by confidence, run 192 (22 points)Claude Opus 5.5 · high · chat: Turkish, five options, held back by confidence, run 278 (21 points)Qwen4-Max · high · chat: Turkish, five options, held back by confidence, run 1336 (31 points)Gemini 3.8 Flash · low · chat: Turkish, five options, held back by confidence, run 123 (14 points)Gemini 4 Carbon · high · chat: Turkish, five options, held back by confidence, run 1108 (20 points)KnowLine-31B-Gen1.5: Turkish, five options, held back by confidence, run 957 (547 points)Grok 4.7 · high · chat: Turkish, five options, held back by confidence, run 311 (47 points)Claude Opus 5.5 · low · chat: Turkish, five options, held back by confidence, run 47 (17 points)Copernicus-31B: Turkish, five options, held back by confidence, run 1165 (547 points)Claude Haiku 5.5 · max · chat: Turkish, five options, held back by confidence, run 948 (23 points)DeepSeek-V4.1-Flash · reasoning on · chat · 1017: Turkish, five options, held back by confidence, run 585 (547 points)REAX 1.3: Turkish, five options, held back by confidence, run 827 (547 points)Kimi K3.1 Pro · low · chat: Turkish, five options, held back by confidence, run 320 (18 points)KnowLine-31B-Gen2: Turkish, five options, held back by confidence, run 1314 (547 points)Karpo Decision 31B: Turkish, five options, held back by confidence, run 1156 (547 points)Claude Sonnet 5.5 · high · chat · 1 per request: Turkish, five options, held back by confidence, run 193 (22 points)Kimi K3.1 Pro · no reasoning · chat: Turkish, five options, held back by confidence, run 315 (19 points)Qwen3.5-397B-A17B · high · chat: Turkish, five options, held back by confidence, run 870 (14 points)Gemini Decision 1 · 20261111: Turkish, five options, held back by confidence, run 1021 (547 points)Logits Sigma-1 Duo: Turkish, five options, held back by confidence, run 1221 (547 points)GLiDE 27B: Turkish, five options, held back by confidence, run 899 (547 points)d1-ultra: Turkish, five options, held back by confidence, run 199 (547 points)Sieve 31B v3: Turkish, five options, held back by confidence, run 1241 (386 points)Turion Jev 31B: Turkish, five options, held back by confidence, run 1273 (546 points)Gemma 4 31B · reasoning off · chat: Turkish, five options, held back by confidence, run 119 (547 points)Perplexity Decider 27B v1.2 · API · with web: Turkish, five options, held back by confidence, run 640 (547 points)GLiDE · v1.2: Turkish, five options, held back by confidence, run 248 (547 points)d1.2-ultra: Turkish, five options, held back by confidence, run 1347 (547 points)Mammoth-1: Turkish, five options, held back by confidence, run 374 (541 points)Gemma 4 31B · reasoning off · chat · 1 per request: Turkish, five options, held back by confidence, run 189 (547 points)d3: Turkish, five options, held back by confidence, run 259 (547 points)Drex Max: Turkish, five options, held back by confidence, run 322 (547 points)GLM-5.3-Flash · reasoning on · chat: Turkish, five options, held back by confidence, run 160 (16 points)Muse Ember · max · chat: Turkish, five options, held back by confidence, run 939 (34 points)Perplexity Decider 27B v1.3: Turkish, five options, held back by confidence, run 1134 (547 points)d3-ultra: Turkish, five options, held back by confidence, run 269 (547 points)d1.2: Turkish, five options, held back by confidence, run 1346 (547 points)MiMo-V2.6-Flash · high · chat: Turkish, five options, held back by confidence, run 877 (27 points)Qwen3.8-27B · low · chat: Turkish, five options, held back by confidence, run 318 (547 points)Perplexity Decider 27B v1.2 · DGX Spark: Turkish, five options, held back by confidence, run 217 (547 points)Drex Max v1.1: Turkish, five options, held back by confidence, run 326 (547 points)Perplexity Decider 27B v1: Turkish, five options, held back by confidence, run 44 (547 points)Perplexity Decider 27B v1.2 · API: Turkish, five options, held back by confidence, run 633 (547 points)Inkling-Small · high · chat · Jul 2026: Turkish, five options, held back by confidence, run 866 (25 points)d1.1: Turkish, five options, held back by confidence, run 254 (547 points)GLM-5.3 · low · chat: Turkish, five options, held back by confidence, run 314 (16 points)Jeffan-9B-1211: Turkish, five options, held back by confidence, run 1192 (542 points)GLM-5.3-Flash · reasoning off · chat: Turkish, five options, held back by confidence, run 159 (12 points)DeepSeek-V4.1-Flash · reasoning off · chat · 1017: Turkish, five options, held back by confidence, run 558 (547 points)Qwen3.8 Flash · no reasoning · chat: Turkish, five options, held back by confidence, run 985 (17 points)d1: Turkish, five options, held back by confidence, run 52 (547 points)StartLux-Decision-v1.1-9B-Instant: Turkish, five options, held back by confidence, run 1290 (547 points)Clef: Turkish, five options, held back by confidence, run 125 (454 points)KnowLine-12B-Gen1: Turkish, five options, held back by confidence, run 1309 (547 points)GLiDE 9B: Turkish, five options, held back by confidence, run 907 (547 points)GPT-6 Luna · low · chat: Turkish, five options, held back by confidence, run 133 (30 points)OneJev-27B: Turkish, five options, held back by confidence, run 264 (547 points)Claude Sonnet 5.5 · high · chat: Turkish, five options, held back by confidence, run 55 (20 points)Jev 1.14: Turkish, five options, held back by confidence, run 194 (72 points)Cerno 9B v1.2: Turkish, five options, held back by confidence, run 1201 (495 points)Drex 2.0: Turkish, five options, held back by confidence, run 334 (513 points)Jev 1.13: Turkish, five options, held back by confidence, run 49 (68 points)KnowLine-9B-Gen1: Turkish, five options, held back by confidence, run 1174 (547 points)SubQ Snap 1.1: Turkish, five options, held back by confidence, run 346 (546 points)SubQ Snap Preview: Turkish, five options, held back by confidence, run 177 (547 points)Adjudex 9B v5: Turkish, five options, held back by confidence, run 1183 (547 points)Mercury Decide: Turkish, five options, held back by confidence, run 204 (547 points)Microsoft-Decision-1: Turkish, five options, held back by confidence, run 205 (547 points)Elephant-1: Turkish, five options, held back by confidence, run 373 (545 points)Open-Jev-27B-v1.1: Turkish, five options, held back by confidence, run 45 (547 points)KnowLine-9B-Gen2: Turkish, five options, held back by confidence, run 1306 (547 points)Qwen3.8-27B · stated confidence · reasoning off · chat: Turkish, five options, held back by confidence, run 53 (5 points)Qwen3.8-27B · yes/no scoring: Turkish, five options, held back by confidence, run 50 (547 points)MiniMax-M3.1-Flash · high · chat: Turkish, five options, held back by confidence, run 862 (37 points)celeris-1-decision: Turkish, five options, held back by confidence, run 150 (70 points)GPT-6 Luna · Decisions API: Turkish, five options, held back by confidence, run 143 (64 points)Claude Sonnet 5.5 · low · chat: Turkish, five options, held back by confidence, run 54 (20 points)Cygnet: Turkish, five options, held back by confidence, run 127 (547 points)Qwen3.8-27B · vote share · reasoning off · chat: Turkish, five options, held back by confidence, run 83 (7 points)Winnow-12B: Turkish, five options, held back by confidence, run 128 (547 points)Perplexity Decider Flash v1.3: Turkish, five options, held back by confidence, run 1123 (547 points)Valen-Decision-1.5-9B: Turkish, five options, held back by confidence, run 1275 (547 points)Dexy-Decision-9B-V2: Turkish, five options, held back by confidence, run 1255 (547 points)vera-medium-omni: Turkish, five options, held back by confidence, run 1264 (526 points)StartLux-Decision-v1.1-4B-Instant: Turkish, five options, held back by confidence, run 1236 (547 points)Clef-Flash: Turkish, five options, held back by confidence, run 139 (504 points)Verdin-4B-v1.1: Turkish, five options, held back by confidence, run 1244 (547 points)Perplexity Decider Flash v1.2: Turkish, five options, held back by confidence, run 223 (546 points)KnowLine 4B Gen5.5: Turkish, five options, held back by confidence, run 965 (547 points)Chinese Decision Pointer 4B: Turkish, five options, held back by confidence, run 1206 (547 points)Qwen3-14B · reasoning off · chat: Turkish, five options, held back by confidence, run 131 (546 points)Metask-Jev-4B: Turkish, five options, held back by confidence, run 126 (546 points)Erk-14B · reasoning off · chat: Turkish, five options, held back by confidence, run 132 (544 points)KnowLine-2B-Gen1: Turkish, five options, held back by confidence, run 1079 (547 points)GLiDE 2B: Turkish, five options, held back by confidence, run 915 (547 points)StartLux-Decision-v1.1-2B-Instant: Turkish, five options, held back by confidence, run 977 (547 points)d1-2B: Turkish, five options, held back by confidence, run 340 (547 points)GLiDE 0.8B: Turkish, five options, held back by confidence, run 923 (547 points)StartLux-Decision-v1.1-0.8B-Instant: Turkish, five options, held back by confidence, run 1015 (546 points)GLiDE 0.4B: Turkish, five options, held back by confidence, run 931 (547 points)OneJev1.6-0.8B: Turkish, five options, held back by confidence, run 1029 (547 points)KnowLine-0.8B-Gen1: Turkish, five options, held back by confidence, run 1005 (547 points)GLiClass v4 Edge: Turkish, five options, held back by confidence, run 1046 (547 points)Decider 1: Turkish, five options, held back by confidence, run 243 (515 points)Strands Decider 2B: Turkish, five options, held back by confidence, run 141 (514 points)Laya: Turkish, five options, held back by confidence, run 51 (480 points)GLiNER2.5-multi-Decide: Turkish, five options, held back by confidence, run 135 (547 points)GLiNER2.5 Multi: Turkish, five options, held back by confidence, run 122 (547 points)Laya Multilingual: Turkish, five options, held back by confidence, run 46 (514 points)Gemini 4 Hydrogen · high · chat: Turkish with “Bilmiyorum”, run 1117 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 1.8%; 533 right, 10 wrong, 4 abstained, 0 invalidGPT-6 Astra · low · chat: Turkish with “Bilmiyorum”, run 816 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 2.6%; 529 right, 14 wrong, 4 abstained, 0 invalidGPT-6 Astra · high · chat: Turkish with “Bilmiyorum”, run 818 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 3.1%; 526 right, 17 wrong, 4 abstained, 0 invalidGPT-6.1 Terra · high · chat: Turkish with “Bilmiyorum”, run 807 (its full-bank run, restricted to these questions): coverage 99.1%, error rate among answered 3.5%; 523 right, 19 wrong, 5 abstained, 0 invalidGPT-6.1 Terra · low · chat: Turkish with “Bilmiyorum”, run 801 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 4.2%; 520 right, 23 wrong, 4 abstained, 0 invalidClaude Fable 5.1 · high · chat: Turkish with “Bilmiyorum”, run 695 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 4.2%; 520 right, 23 wrong, 4 abstained, 0 invalidGemini 3.8 Flash · high · chat: Turkish with “Bilmiyorum”, run 681 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 2.4%; 532 right, 13 wrong, 2 abstained, 0 invalidGemini 4 Argon · low · chat: Turkish with “Bilmiyorum”, run 679 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 2.4%; 530 right, 13 wrong, 4 abstained, 0 invalidGPT-6 Astra · medium · chat: Turkish with “Bilmiyorum”, run 817 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 2.9%; 527 right, 16 wrong, 4 abstained, 0 invalidGPT-6.1 Sol · low · chat: Turkish with “Bilmiyorum”, run 799 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 3.3%; 525 right, 18 wrong, 4 abstained, 0 invalidGPT-6 Astra · low · chat · 1 per request: Turkish with “Bilmiyorum”, run 819 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 3.5%; 524 right, 19 wrong, 4 abstained, 0 invalidGPT-6.1 Sol · high · chat: Turkish with “Bilmiyorum”, run 802 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 2.9%; 527 right, 16 wrong, 4 abstained, 0 invalidGemini 3.8 Flash · low · chat · 1 per request: Turkish with “Bilmiyorum”, run 691 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 3.3%; 526 right, 18 wrong, 3 abstained, 0 invalidGPT-6.1 Sol · low · chat · 1 per request: Turkish with “Bilmiyorum”, run 810 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 3.5%; 524 right, 19 wrong, 4 abstained, 0 invalidClaude Opus 5.5 · low · chat · 1 per request: Turkish with “Bilmiyorum”, run 803 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 4.6%; 521 right, 25 wrong, 1 abstained, 0 invalidClaude Opus 5.5 · high · chat: Turkish with “Bilmiyorum”, run 773 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 4.2%; 521 right, 23 wrong, 3 abstained, 0 invalidQwen4-Max · high · chat: Turkish with “Bilmiyorum”, run 1339 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 5.1%; 518 right, 28 wrong, 1 abstained, 0 invalidGemini 3.8 Flash · low · chat: Turkish with “Bilmiyorum”, run 676 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 4.4%; 520 right, 24 wrong, 3 abstained, 0 invalidGemini 4 Carbon · high · chat: Turkish with “Bilmiyorum”, run 1121 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 2.0%; 532 right, 11 wrong, 4 abstained, 0 invalidKnowLine-31B-Gen1.5: Turkish with “Bilmiyorum”, run 959 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 6.2%; 513 right, 34 wrong, 0 abstained, 0 invalidGrok 4.7 · high · chat: Turkish with “Bilmiyorum”, run 675 (its full-bank run, restricted to these questions): coverage 98.5%, error rate among answered 3.9%; 518 right, 21 wrong, 8 abstained, 0 invalidClaude Opus 5.5 · low · chat: Turkish with “Bilmiyorum”, run 683 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 5.3%; 516 right, 29 wrong, 2 abstained, 0 invalidCopernicus-31B: Turkish with “Bilmiyorum”, run 1172 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 6.0%; 513 right, 33 wrong, 1 abstained, 0 invalidClaude Haiku 5.5 · max · chat: Turkish with “Bilmiyorum”, run 950 (its full-bank run, restricted to these questions): coverage 96.0%, error rate among answered 5.5%; 496 right, 29 wrong, 6 abstained, 16 invalidDeepSeek-V4.1-Flash · reasoning on · chat · 1017: Turkish with “Bilmiyorum”, run 811 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 7.0%; 508 right, 38 wrong, 1 abstained, 0 invalidREAX 1.3: Turkish with “Bilmiyorum”, run 833 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 6.4%; 511 right, 35 wrong, 1 abstained, 0 invalidKimi K3.1 Pro · low · chat: Turkish with “Bilmiyorum”, run 686 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 6.6%; 508 right, 36 wrong, 3 abstained, 0 invalidKnowLine-31B-Gen2: Turkish with “Bilmiyorum”, run 1331 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 5.3%; 518 right, 29 wrong, 0 abstained, 0 invalidKarpo Decision 31B: Turkish with “Bilmiyorum”, run 1163 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 5.5%; 516 right, 30 wrong, 1 abstained, 0 invalidClaude Sonnet 5.5 · high · chat · 1 per request: Turkish with “Bilmiyorum”, run 812 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 6.6%; 510 right, 36 wrong, 1 abstained, 0 invalidKimi K3.1 Pro · no reasoning · chat: Turkish with “Bilmiyorum”, run 653 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 7.2%; 505 right, 39 wrong, 3 abstained, 0 invalidQwen3.5-397B-A17B · high · chat: Turkish with “Bilmiyorum”, run 872 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 7.9%; 500 right, 43 wrong, 4 abstained, 0 invalidGemini Decision 1 · 20261111: Turkish with “Bilmiyorum”, run 1023 (its full-bank run, restricted to these questions): coverage 99.1%, error rate among answered 7.6%; 501 right, 41 wrong, 5 abstained, 0 invalidLogits Sigma-1 Duo: Turkish with “Bilmiyorum”, run 1233 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 6.6%; 511 right, 36 wrong, 0 abstained, 0 invalidGLiDE 27B: Turkish with “Bilmiyorum”, run 905 (its full-bank run, restricted to these questions): coverage 98.4%, error rate among answered 7.2%; 499 right, 39 wrong, 9 abstained, 0 invalidd1-ultra: Turkish with “Bilmiyorum”, run 350 (its full-bank run, restricted to these questions): coverage 94.5%, error rate among answered 5.0%; 491 right, 26 wrong, 30 abstained, 0 invalidd1-ultra: Turkish with “I don't know”, run 511 (its full-bank run, restricted to these questions): coverage 95.1%, error rate among answered 5.4%; 492 right, 28 wrong, 27 abstained, 0 invalidSieve 31B v3: Turkish with “Bilmiyorum”, run 1253 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 6.0%; 514 right, 33 wrong, 0 abstained, 0 invalidTurion Jev 31B: Turkish with “Bilmiyorum”, run 1285 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 6.1%; 512 right, 33 wrong, 2 abstained, 0 invalidGemma 4 31B · reasoning off · chat: Turkish with “Bilmiyorum”, run 776 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 7.7%; 503 right, 42 wrong, 2 abstained, 0 invalidPerplexity Decider 27B v1.2 · API · with web: Turkish with “Bilmiyorum”, run 642 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 9.4%; 493 right, 51 wrong, 3 abstained, 0 invalidGLiDE · v1.2: Turkish with “Bilmiyorum”, run 369 (its full-bank run, restricted to these questions): coverage 98.9%, error rate among answered 8.7%; 494 right, 47 wrong, 6 abstained, 0 invalidGLiDE · v1.2: Turkish with “I don't know”, run 525 (its full-bank run, restricted to these questions): coverage 98.4%, error rate among answered 8.7%; 491 right, 47 wrong, 9 abstained, 0 invalidd1.2-ultra: Turkish with “Bilmiyorum”, run 1367 (its full-bank run, restricted to these questions): coverage 88.8%, error rate among answered 3.9%; 467 right, 19 wrong, 61 abstained, 0 invalidd1.2-ultra: Turkish with “I don't know”, run 1378 (its full-bank run, restricted to these questions): coverage 89.0%, error rate among answered 3.7%; 469 right, 18 wrong, 60 abstained, 0 invalidMammoth-1: Turkish with “Bilmiyorum”, run 386 (its full-bank run, restricted to these questions): coverage 95.6%, error rate among answered 6.3%; 490 right, 33 wrong, 24 abstained, 0 invalidMammoth-1: Turkish with “I don't know”, run 519 (its full-bank run, restricted to these questions): coverage 95.4%, error rate among answered 6.1%; 490 right, 32 wrong, 25 abstained, 0 invalidGemma 4 31B · reasoning off · chat · 1 per request: Turkish with “Bilmiyorum”, run 789 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 9.5%; 494 right, 52 wrong, 1 abstained, 0 invalidd3: Turkish with “Bilmiyorum”, run 358 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 9.7%; 494 right, 53 wrong, 0 abstained, 0 invalidd3: Turkish with “I don't know”, run 544 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 9.5%; 495 right, 52 wrong, 0 abstained, 0 invalidDrex Max: Turkish with “Bilmiyorum”, run 367 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 9.7%; 492 right, 53 wrong, 2 abstained, 0 invalidDrex Max: Turkish with “I don't know”, run 542 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 9.0%; 497 right, 49 wrong, 1 abstained, 0 invalidGLM-5.3-Flash · reasoning on · chat: Turkish with “Bilmiyorum”, run 825 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 10.1%; 490 right, 55 wrong, 2 abstained, 0 invalidMuse Ember · max · chat: Turkish with “Bilmiyorum”, run 940 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 8.7%; 496 right, 47 wrong, 4 abstained, 0 invalidPerplexity Decider 27B v1.3: Turkish with “Bilmiyorum”, run 1141 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 9.5%; 495 right, 52 wrong, 0 abstained, 0 invalidd3-ultra: Turkish with “Bilmiyorum”, run 363 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 8.4%; 500 right, 46 wrong, 1 abstained, 0 invalidd3-ultra: Turkish with “I don't know”, run 540 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 9.4%; 494 right, 51 wrong, 2 abstained, 0 invalidd1.2: Turkish with “Bilmiyorum”, run 1365 (its full-bank run, restricted to these questions): coverage 90.3%, error rate among answered 7.1%; 459 right, 35 wrong, 53 abstained, 0 invalidd1.2: Turkish with “I don't know”, run 1377 (its full-bank run, restricted to these questions): coverage 91.6%, error rate among answered 8.4%; 459 right, 42 wrong, 46 abstained, 0 invalidMiMo-V2.6-Flash · high · chat: Turkish with “Bilmiyorum”, run 879 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 12.7%; 475 right, 69 wrong, 3 abstained, 0 invalidQwen3.8-27B · low · chat: Turkish with “Bilmiyorum”, run 666 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 8.8%; 498 right, 48 wrong, 1 abstained, 0 invalidPerplexity Decider 27B v1.2 · DGX Spark: Turkish with “Bilmiyorum”, run 372 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 9.1%; 497 right, 50 wrong, 0 abstained, 0 invalidPerplexity Decider 27B v1.2 · DGX Spark: Turkish with “I don't know”, run 531 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 9.3%; 495 right, 51 wrong, 1 abstained, 0 invalidDrex Max v1.1: Turkish with “Bilmiyorum”, run 352 (its full-bank run, restricted to these questions): coverage 98.4%, error rate among answered 9.5%; 487 right, 51 wrong, 9 abstained, 0 invalidDrex Max v1.1: Turkish with “I don't know”, run 513 (its full-bank run, restricted to these questions): coverage 96.7%, error rate among answered 9.1%; 481 right, 48 wrong, 18 abstained, 0 invalidPerplexity Decider 27B v1.2 · API: Turkish with “Bilmiyorum”, run 641 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 9.3%; 496 right, 51 wrong, 0 abstained, 0 invalidInkling-Small · high · chat · Jul 2026: Turkish with “Bilmiyorum”, run 868 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 11.3%; 485 right, 62 wrong, 0 abstained, 0 invalidGLM-5.3 · low · chat: Turkish with “Bilmiyorum”, run 656 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 11.2%; 483 right, 61 wrong, 3 abstained, 0 invalidJeffan-9B-1211: Turkish with “Bilmiyorum”, run 1199 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 13.1%; 473 right, 71 wrong, 3 abstained, 0 invalidGLM-5.3-Flash · reasoning off · chat: Turkish with “Bilmiyorum”, run 822 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 15.1%; 462 right, 82 wrong, 3 abstained, 0 invalidDeepSeek-V4.1-Flash · reasoning off · chat · 1017: Turkish with “Bilmiyorum”, run 806 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 10.4%; 490 right, 57 wrong, 0 abstained, 0 invalidQwen3.5-4B · reasoning on · chat: Turkish with “Bilmiyorum”, run 1150 (its full-bank run, restricted to these questions): coverage 99.1%, error rate among answered 16.2%; 454 right, 88 wrong, 2 abstained, 3 invalidQwen3.8 Flash · no reasoning · chat: Turkish with “Bilmiyorum”, run 987 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 10.1%; 489 right, 55 wrong, 3 abstained, 0 invalidd1: Turkish with “Bilmiyorum”, run 472: coverage 99.8%, error rate among answered 16.3%; 457 right, 89 wrong, 1 abstained, 0 invalidStartLux-Decision-v1.1-9B-Instant: Turkish with “Bilmiyorum”, run 1297 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 17.4%; 451 right, 95 wrong, 1 abstained, 0 invalidClef: Turkish with “Bilmiyorum”, run 788 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 12.3%; 479 right, 67 wrong, 1 abstained, 0 invalidKnowLine-12B-Gen1: Turkish with “Bilmiyorum”, run 1318 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 12.2%; 480 right, 67 wrong, 0 abstained, 0 invalidGLiDE 9B: Turkish with “Bilmiyorum”, run 913 (its full-bank run, restricted to these questions): coverage 98.4%, error rate among answered 15.4%; 455 right, 83 wrong, 9 abstained, 0 invalidGPT-6 Luna · low · chat: Turkish with “Bilmiyorum”, run 796 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 11.9%; 480 right, 65 wrong, 2 abstained, 0 invalidOneJev-27B: Turkish with “Bilmiyorum”, run 782 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 11.5%; 483 right, 63 wrong, 1 abstained, 0 invalidClaude Sonnet 5.5 · high · chat: Turkish with “Bilmiyorum”, run 800 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 9.7%; 492 right, 53 wrong, 2 abstained, 0 invalidJev 1.14: Turkish with “Bilmiyorum”, run 470: coverage 99.5%, error rate among answered 12.7%; 475 right, 69 wrong, 3 abstained, 0 invalidCerno 9B v1.2: Turkish with “Bilmiyorum”, run 1204 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 16.6%; 456 right, 91 wrong, 0 abstained, 0 invalidKnowLine-9B-Gen1: Turkish with “Bilmiyorum”, run 1181 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 17.9%; 449 right, 98 wrong, 0 abstained, 0 invalidSubQ Snap 1.1: Turkish with “Bilmiyorum”, run 392 (its full-bank run, restricted to these questions): coverage 96.5%, error rate among answered 9.7%; 477 right, 51 wrong, 19 abstained, 0 invalidSubQ Snap 1.1: Turkish with “I don't know”, run 512 (its full-bank run, restricted to these questions): coverage 96.5%, error rate among answered 8.5%; 483 right, 45 wrong, 19 abstained, 0 invalidAdjudex 9B v5: Turkish with “Bilmiyorum”, run 1190 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 16.3%; 458 right, 89 wrong, 0 abstained, 0 invalidMercury Decide: Turkish with “Bilmiyorum”, run 489: coverage 98.5%, error rate among answered 13.2%; 468 right, 71 wrong, 8 abstained, 0 invalidMicrosoft-Decision-1: Turkish with “Bilmiyorum”, run 487: coverage 99.6%, error rate among answered 16.1%; 457 right, 88 wrong, 2 abstained, 0 invalidElephant-1: Turkish with “Bilmiyorum”, run 382 (its full-bank run, restricted to these questions): coverage 97.3%, error rate among answered 15.4%; 450 right, 82 wrong, 15 abstained, 0 invalidElephant-1: Turkish with “I don't know”, run 515 (its full-bank run, restricted to these questions): coverage 95.6%, error rate among answered 15.7%; 441 right, 82 wrong, 24 abstained, 0 invalidOpen-Jev-27B-v1.1: Turkish with “Bilmiyorum”, run 768 (its full-bank run, restricted to these questions): coverage 98.9%, error rate among answered 14.4%; 463 right, 78 wrong, 6 abstained, 0 invalidKnowLine-9B-Gen2: Turkish with “Bilmiyorum”, run 1325 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 17.4%; 452 right, 95 wrong, 0 abstained, 0 invalidQwen3.8-27B · stated confidence · reasoning off · chat: Turkish with “Bilmiyorum”, run 674 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 13.3%; 471 right, 72 wrong, 4 abstained, 0 invalidMiniMax-M3.1-Flash · high · chat: Turkish with “Bilmiyorum”, run 864 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 16.5%; 455 right, 90 wrong, 1 abstained, 1 invalidGPT-6 Luna · Decisions API: Turkish with “Bilmiyorum”, run 453: coverage 98.9%, error rate among answered 14.4%; 463 right, 78 wrong, 6 abstained, 0 invalidClaude Sonnet 5.5 · low · chat: Turkish with “Bilmiyorum”, run 690 (its full-bank run, restricted to these questions): coverage 99.1%, error rate among answered 15.1%; 460 right, 82 wrong, 5 abstained, 0 invalidCygnet: Turkish with “Bilmiyorum”, run 757 (its full-bank run, restricted to these questions): coverage 99.1%, error rate among answered 14.9%; 461 right, 81 wrong, 5 abstained, 0 invalidQwen3.8-27B · vote share · reasoning off · chat: Turkish with “Bilmiyorum”, run 682 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 13.8%; 468 right, 75 wrong, 4 abstained, 0 invalidWinnow-12B: Turkish with “Bilmiyorum”, run 750 (its full-bank run, restricted to these questions): coverage 98.5%, error rate among answered 15.8%; 454 right, 85 wrong, 8 abstained, 0 invalidPerplexity Decider Flash v1.3: Turkish with “Bilmiyorum”, run 1130 (its full-bank run, restricted to these questions): coverage 98.4%, error rate among answered 20.4%; 428 right, 110 wrong, 9 abstained, 0 invalidValen-Decision-1.5-9B: Turkish with “Bilmiyorum”, run 1282 (its full-bank run, restricted to these questions): coverage 100.0%, error rate among answered 20.1%; 437 right, 110 wrong, 0 abstained, 0 invalidDexy-Decision-9B-V2: Turkish with “Bilmiyorum”, run 1262 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 16.5%; 456 right, 90 wrong, 1 abstained, 0 invalidvera-medium-omni: Turkish with “Bilmiyorum”, run 1271 (its full-bank run, restricted to these questions): coverage 99.1%, error rate among answered 20.8%; 429 right, 113 wrong, 5 abstained, 0 invalidStartLux-Decision-v1.1-4B-Instant: Turkish with “Bilmiyorum”, run 1239 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 21.7%; 427 right, 118 wrong, 2 abstained, 0 invalidGemma 4 E4B · default reasoning · chat: Turkish with “Bilmiyorum”, run 1049 (its full-bank run, restricted to these questions): coverage 95.4%, error rate among answered 18.4%; 426 right, 96 wrong, 25 abstained, 0 invalidClef-Flash: Turkish with “Bilmiyorum”, run 664 (its full-bank run, restricted to these questions): coverage 97.8%, error rate among answered 21.5%; 420 right, 115 wrong, 12 abstained, 0 invalidVerdin-4B-v1.1: Turkish with “Bilmiyorum”, run 1251 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 23.3%; 419 right, 127 wrong, 1 abstained, 0 invalidPerplexity Decider Flash v1.2: Turkish with “Bilmiyorum”, run 537: coverage 98.9%, error rate among answered 24.6%; 408 right, 133 wrong, 6 abstained, 0 invalidKnowLine 4B Gen5.5: Turkish with “Bilmiyorum”, run 967 (its full-bank run, restricted to these questions): coverage 99.5%, error rate among answered 21.7%; 426 right, 118 wrong, 3 abstained, 0 invalidChinese Decision Pointer 4B: Turkish with “Bilmiyorum”, run 1213 (its full-bank run, restricted to these questions): coverage 99.3%, error rate among answered 23.8%; 414 right, 129 wrong, 4 abstained, 0 invalidQwen3.5-4B · reasoning off · chat: Turkish with “Bilmiyorum”, run 1147 (its full-bank run, restricted to these questions): coverage 98.4%, error rate among answered 26.0%; 398 right, 140 wrong, 5 abstained, 4 invalidQwen3-14B · reasoning off · chat: Turkish with “Bilmiyorum”, run 772 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 22.3%; 424 right, 122 wrong, 1 abstained, 0 invalidMetask-Jev-4B: Turkish with “Bilmiyorum”, run 743 (its full-bank run, restricted to these questions): coverage 99.6%, error rate among answered 34.3%; 358 right, 187 wrong, 2 abstained, 0 invalidLFM2.5-2.6B · default reasoning · chat: Turkish with “Bilmiyorum”, run 1038 (its full-bank run, restricted to these questions): coverage 90.7%, error rate among answered 46.2%; 267 right, 229 wrong, 43 abstained, 8 invalidErk-14B · reasoning off · chat: Turkish with “Bilmiyorum”, run 765 (its full-bank run, restricted to these questions): coverage 99.8%, error rate among answered 23.6%; 417 right, 129 wrong, 1 abstained, 0 invalidLFM3-1.7B · reasoning off · chat: Turkish with “Bilmiyorum”, run 1234 (its full-bank run, restricted to these questions): coverage 56.9%, error rate among answered 55.6%; 138 right, 173 wrong, 215 abstained, 21 invalidLFM3-1.7B · reasoning on · chat: Turkish with “Bilmiyorum”, run 1090 (its full-bank run, restricted to these questions): coverage 80.6%, error rate among answered 47.6%; 231 right, 210 wrong, 106 abstained, 0 invalidLFM2.5-8B-A1B · default reasoning · chat: Turkish with “Bilmiyorum”, run 1298 (its full-bank run, restricted to these questions): coverage 52.5%, error rate among answered 67.9%; 92 right, 195 wrong, 201 abstained, 59 invalidGLiDE 2B: Turkish with “Bilmiyorum”, run 921 (its full-bank run, restricted to these questions): coverage 87.9%, error rate among answered 43.5%; 272 right, 209 wrong, 66 abstained, 0 invalidStartLux-Decision-v1.1-2B-Instant: Turkish with “Bilmiyorum”, run 979 (its full-bank run, restricted to these questions): coverage 94.0%, error rate among answered 45.7%; 279 right, 235 wrong, 33 abstained, 0 invalidMiniCPM5-2B · reasoning off · chat: Turkish with “Bilmiyorum”, run 1304 (its full-bank run, restricted to these questions): coverage 20.8%, error rate among answered 65.8%; 39 right, 75 wrong, 22 abstained, 411 invalidQwen3.5-2B · reasoning off · chat: Turkish with “Bilmiyorum”, run 1219 (its full-bank run, restricted to these questions): coverage 83.4%, error rate among answered 61.2%; 177 right, 279 wrong, 55 abstained, 36 invalidd1-2B: Turkish with “Bilmiyorum”, run 712 (its full-bank run, restricted to these questions): coverage 70.6%, error rate among answered 53.6%; 179 right, 207 wrong, 161 abstained, 0 invalidGLiDE 0.8B: Turkish with “Bilmiyorum”, run 929 (its full-bank run, restricted to these questions): coverage 86.5%, error rate among answered 48.6%; 243 right, 230 wrong, 74 abstained, 0 invalidStartLux-Decision-v1.1-0.8B-Instant: Turkish with “Bilmiyorum”, run 1017 (its full-bank run, restricted to these questions): coverage 96.0%, error rate among answered 61.0%; 205 right, 320 wrong, 22 abstained, 0 invalidGLiDE 0.4B: Turkish with “Bilmiyorum”, run 937 (its full-bank run, restricted to these questions): coverage 13.2%, error rate among answered 76.4%; 17 right, 55 wrong, 475 abstained, 0 invalidOneJev1.6-0.8B: Turkish with “Bilmiyorum”, run 1031 (its full-bank run, restricted to these questions): coverage 79.5%, error rate among answered 58.2%; 182 right, 253 wrong, 112 abstained, 0 invalidKnowLine-0.8B-Gen1: Turkish with “Bilmiyorum”, run 1007 (its full-bank run, restricted to these questions): coverage 97.3%, error rate among answered 61.1%; 207 right, 325 wrong, 15 abstained, 0 invalidGLiClass v4 Edge: Turkish with “Bilmiyorum”, run 1048 (its full-bank run, restricted to these questions): coverage 89.4%, error rate among answered 72.0%; 137 right, 352 wrong, 58 abstained, 0 invalidStrands Decider 2B: Turkish with “Bilmiyorum”, run 706 (its full-bank run, restricted to these questions): coverage 99.1%, error rate among answered 60.3%; 215 right, 327 wrong, 5 abstained, 0 invalidLaya: Turkish with “Bilmiyorum”, run 730 (its full-bank run, restricted to these questions): coverage 90.3%, error rate among answered 78.9%; 104 right, 390 wrong, 53 abstained, 0 invalidGLiNER2.5-multi-Decide: Turkish with “Bilmiyorum”, run 718 (its full-bank run, restricted to these questions): coverage 95.6%, error rate among answered 78.8%; 111 right, 412 wrong, 24 abstained, 0 invalidGLiNER2.5 Multi: Turkish with “Bilmiyorum”, run 724 (its full-bank run, restricted to these questions): coverage 94.9%, error rate among answered 82.5%; 91 right, 428 wrong, 28 abstained, 0 invalidLaya Multilingual: Turkish with “Bilmiyorum”, run 737 (its full-bank run, restricted to these questions): coverage 77.3%, error rate among answered 84.9%; 64 right, 359 wrong, 124 abstained, 0 invalid
  • Claude Fable 5.2 · max · chat: confidence, run 980
  • Gemini 4 Hydrogen · high · chat: confidence, run 1114 · “Bilmiyorum”, run 1117
  • GPT-6 Astra · low · chat: confidence, run 48 · “Bilmiyorum”, run 816
  • GPT-6 Astra · high · chat: confidence, run 277 · “Bilmiyorum”, run 818
  • GPT-6.1 Terra · high · chat: confidence, run 275 · “Bilmiyorum”, run 807
  • GPT-6.1 Terra · low · chat: confidence, run 274 · “Bilmiyorum”, run 801
  • Claude Fable 5.1 · high · chat: confidence, run 310 · “Bilmiyorum”, run 695
  • Gemini 3.8 Flash · high · chat: confidence, run 140 · “Bilmiyorum”, run 681
  • Gemini 4 Argon · low · chat: confidence, run 557 · “Bilmiyorum”, run 679
  • GPT-6 Astra · medium · chat: confidence, run 610 · “Bilmiyorum”, run 817
  • GPT-6.1 Sol · low · chat: confidence, run 124 · “Bilmiyorum”, run 799
  • GPT-6 Astra · low · chat · 1 per request: confidence, run 191 · “Bilmiyorum”, run 819
  • GPT-6.1 Sol · high · chat: confidence, run 276 · “Bilmiyorum”, run 802
  • Gemini 3.8 Flash · low · chat · 1 per request: confidence, run 188 · “Bilmiyorum”, run 691
  • GPT-6.1 Sol · low · chat · 1 per request: confidence, run 190 · “Bilmiyorum”, run 810
  • Claude Opus 5.5 · low · chat · 1 per request: confidence, run 192 · “Bilmiyorum”, run 803
  • Claude Opus 5.5 · high · chat: confidence, run 278 · “Bilmiyorum”, run 773
  • Qwen4-Max · high · chat: confidence, run 1336 · “Bilmiyorum”, run 1339
  • Gemini 3.8 Flash · low · chat: confidence, run 123 · “Bilmiyorum”, run 676
  • Gemini 4 Carbon · high · chat: confidence, run 1108 · “Bilmiyorum”, run 1121
  • KnowLine-31B-Gen1.5: confidence, run 957 · “Bilmiyorum”, run 959
  • Grok 4.7 · high · chat: confidence, run 311 · “Bilmiyorum”, run 675
  • Claude Opus 5.5 · low · chat: confidence, run 47 · “Bilmiyorum”, run 683
  • Copernicus-31B: confidence, run 1165 · “Bilmiyorum”, run 1172
  • Claude Haiku 5.5 · max · chat: confidence, run 948 · “Bilmiyorum”, run 950
  • DeepSeek-V4.1-Flash · reasoning on · chat · 1017: confidence, run 585 · “Bilmiyorum”, run 811
  • REAX 1.3: confidence, run 827 · “Bilmiyorum”, run 833
  • Kimi K3.1 Pro · low · chat: confidence, run 320 · “Bilmiyorum”, run 686
  • KnowLine-31B-Gen2: confidence, run 1314 · “Bilmiyorum”, run 1331
  • Karpo Decision 31B: confidence, run 1156 · “Bilmiyorum”, run 1163
  • Claude Sonnet 5.5 · high · chat · 1 per request: confidence, run 193 · “Bilmiyorum”, run 812
  • Kimi K3.1 Pro · no reasoning · chat: confidence, run 315 · “Bilmiyorum”, run 653
  • Qwen3.5-397B-A17B · high · chat: confidence, run 870 · “Bilmiyorum”, run 872
  • Gemini Decision 1 · 20261111: confidence, run 1021 · “Bilmiyorum”, run 1023
  • Logits Sigma-1 Duo: confidence, run 1221 · “Bilmiyorum”, run 1233
  • GLiDE 27B: confidence, run 899 · “Bilmiyorum”, run 905
  • d1-ultra: confidence, run 199 · “Bilmiyorum”, run 350 · “I don't know”, run 511
  • Sieve 31B v3: confidence, run 1241 · “Bilmiyorum”, run 1253
  • Turion Jev 31B: confidence, run 1273 · “Bilmiyorum”, run 1285
  • Gemma 4 31B · reasoning off · chat: confidence, run 119 · “Bilmiyorum”, run 776
  • Perplexity Decider 27B v1.2 · API · with web: confidence, run 640 · “Bilmiyorum”, run 642
  • GLiDE · v1.2: confidence, run 248 · “Bilmiyorum”, run 369 · “I don't know”, run 525
  • d1.2-ultra: confidence, run 1347 · “Bilmiyorum”, run 1367 · “I don't know”, run 1378
  • Mammoth-1: confidence, run 374 · “Bilmiyorum”, run 386 · “I don't know”, run 519
  • Gemma 4 31B · reasoning off · chat · 1 per request: confidence, run 189 · “Bilmiyorum”, run 789
  • d3: confidence, run 259 · “Bilmiyorum”, run 358 · “I don't know”, run 544
  • Drex Max: confidence, run 322 · “Bilmiyorum”, run 367 · “I don't know”, run 542
  • GLM-5.3-Flash · reasoning on · chat: confidence, run 160 · “Bilmiyorum”, run 825
  • Muse Ember · max · chat: confidence, run 939 · “Bilmiyorum”, run 940
  • Perplexity Decider 27B v1.3: confidence, run 1134 · “Bilmiyorum”, run 1141
  • d3-ultra: confidence, run 269 · “Bilmiyorum”, run 363 · “I don't know”, run 540
  • d1.2: confidence, run 1346 · “Bilmiyorum”, run 1365 · “I don't know”, run 1377
  • MiMo-V2.6-Flash · high · chat: confidence, run 877 · “Bilmiyorum”, run 879
  • Qwen3.8-27B · low · chat: confidence, run 318 · “Bilmiyorum”, run 666
  • Perplexity Decider 27B v1.2 · DGX Spark: confidence, run 217 · “Bilmiyorum”, run 372 · “I don't know”, run 531
  • Drex Max v1.1: confidence, run 326 · “Bilmiyorum”, run 352 · “I don't know”, run 513
  • Perplexity Decider 27B v1: confidence, run 44
  • Perplexity Decider 27B v1.2 · API: confidence, run 633 · “Bilmiyorum”, run 641
  • Inkling-Small · high · chat · Jul 2026: confidence, run 866 · “Bilmiyorum”, run 868
  • d1.1: confidence, run 254
  • GLM-5.3 · low · chat: confidence, run 314 · “Bilmiyorum”, run 656
  • Jeffan-9B-1211: confidence, run 1192 · “Bilmiyorum”, run 1199
  • GLM-5.3-Flash · reasoning off · chat: confidence, run 159 · “Bilmiyorum”, run 822
  • DeepSeek-V4.1-Flash · reasoning off · chat · 1017: confidence, run 558 · “Bilmiyorum”, run 806
  • Qwen3.5-4B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1150
  • Qwen3.8 Flash · no reasoning · chat: confidence, run 985 · “Bilmiyorum”, run 987
  • d1: confidence, run 52 · “Bilmiyorum”, run 472
  • StartLux-Decision-v1.1-9B-Instant: confidence, run 1290 · “Bilmiyorum”, run 1297
  • Clef: confidence, run 125 · “Bilmiyorum”, run 788
  • KnowLine-12B-Gen1: confidence, run 1309 · “Bilmiyorum”, run 1318
  • GLiDE 9B: confidence, run 907 · “Bilmiyorum”, run 913
  • GPT-6 Luna · low · chat: confidence, run 133 · “Bilmiyorum”, run 796
  • OneJev-27B: confidence, run 264 · “Bilmiyorum”, run 782
  • Claude Sonnet 5.5 · high · chat: confidence, run 55 · “Bilmiyorum”, run 800
  • Jev 1.14: confidence, run 194 · “Bilmiyorum”, run 470
  • Cerno 9B v1.2: confidence, run 1201 · “Bilmiyorum”, run 1204
  • Drex 2.0: confidence, run 334
  • Jev 1.13: confidence, run 49
  • KnowLine-9B-Gen1: confidence, run 1174 · “Bilmiyorum”, run 1181
  • SubQ Snap 1.1: confidence, run 346 · “Bilmiyorum”, run 392 · “I don't know”, run 512
  • SubQ Snap Preview: confidence, run 177
  • Adjudex 9B v5: confidence, run 1183 · “Bilmiyorum”, run 1190
  • Mercury Decide: confidence, run 204 · “Bilmiyorum”, run 489
  • Microsoft-Decision-1: confidence, run 205 · “Bilmiyorum”, run 487
  • Elephant-1: confidence, run 373 · “Bilmiyorum”, run 382 · “I don't know”, run 515
  • Open-Jev-27B-v1.1: confidence, run 45 · “Bilmiyorum”, run 768
  • KnowLine-9B-Gen2: confidence, run 1306 · “Bilmiyorum”, run 1325
  • Qwen3.8-27B · stated confidence · reasoning off · chat: confidence, run 53 · “Bilmiyorum”, run 674
  • Qwen3.8-27B · yes/no scoring: confidence, run 50
  • MiniMax-M3.1-Flash · high · chat: confidence, run 862 · “Bilmiyorum”, run 864
  • celeris-1-decision: confidence, run 150
  • GPT-6 Luna · Decisions API: confidence, run 143 · “Bilmiyorum”, run 453
  • Claude Sonnet 5.5 · low · chat: confidence, run 54 · “Bilmiyorum”, run 690
  • Cygnet: confidence, run 127 · “Bilmiyorum”, run 757
  • Qwen3.8-27B · vote share · reasoning off · chat: confidence, run 83 · “Bilmiyorum”, run 682
  • Winnow-12B: confidence, run 128 · “Bilmiyorum”, run 750
  • Perplexity Decider Flash v1.3: confidence, run 1123 · “Bilmiyorum”, run 1130
  • Valen-Decision-1.5-9B: confidence, run 1275 · “Bilmiyorum”, run 1282
  • Dexy-Decision-9B-V2: confidence, run 1255 · “Bilmiyorum”, run 1262
  • vera-medium-omni: confidence, run 1264 · “Bilmiyorum”, run 1271
  • StartLux-Decision-v1.1-4B-Instant: confidence, run 1236 · “Bilmiyorum”, run 1239
  • Gemma 4 E4B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1049
  • Clef-Flash: confidence, run 139 · “Bilmiyorum”, run 664
  • Verdin-4B-v1.1: confidence, run 1244 · “Bilmiyorum”, run 1251
  • Perplexity Decider Flash v1.2: confidence, run 223 · “Bilmiyorum”, run 537
  • KnowLine 4B Gen5.5: confidence, run 965 · “Bilmiyorum”, run 967
  • Chinese Decision Pointer 4B: confidence, run 1206 · “Bilmiyorum”, run 1213
  • Qwen3.5-4B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1147
  • Qwen3-14B · reasoning off · chat: confidence, run 131 · “Bilmiyorum”, run 772
  • Metask-Jev-4B: confidence, run 126 · “Bilmiyorum”, run 743
  • LFM2.5-2.6B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1038
  • Erk-14B · reasoning off · chat: confidence, run 132 · “Bilmiyorum”, run 765
  • LFM3-1.7B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1234
  • KnowLine-2B-Gen1: confidence, run 1079
  • LFM3-1.7B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1090
  • LFM2.5-8B-A1B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1298
  • GLiDE 2B: confidence, run 915 · “Bilmiyorum”, run 921
  • StartLux-Decision-v1.1-2B-Instant: confidence, run 977 · “Bilmiyorum”, run 979
  • MiniCPM5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1304
  • Qwen3.5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “Bilmiyorum”, run 1219
  • d1-2B: confidence, run 340 · “Bilmiyorum”, run 712
  • GLiDE 0.8B: confidence, run 923 · “Bilmiyorum”, run 929
  • StartLux-Decision-v1.1-0.8B-Instant: confidence, run 1015 · “Bilmiyorum”, run 1017
  • GLiDE 0.4B: confidence, run 931 · “Bilmiyorum”, run 937
  • OneJev1.6-0.8B: confidence, run 1029 · “Bilmiyorum”, run 1031
  • KnowLine-0.8B-Gen1: confidence, run 1005 · “Bilmiyorum”, run 1007
  • GLiClass v4 Edge: confidence, run 1046 · “Bilmiyorum”, run 1048
  • Decider 1: confidence, run 243
  • Strands Decider 2B: confidence, run 141 · “Bilmiyorum”, run 706
  • Laya: confidence, run 51 · “Bilmiyorum”, run 730
  • GLiNER2.5-multi-Decide: confidence, run 135 · “Bilmiyorum”, run 718
  • GLiNER2.5 Multi: confidence, run 122 · “Bilmiyorum”, run 724
  • Laya Multilingual: confidence, run 46 · “Bilmiyorum”, run 737

Risk and coverage in English

The 547 kept questions in English: each setup's five-option run answering only at or above a confidence score (the line), and a run with the option (the marker: “I don't know”) · lower is better at the same coverage

0%20%40%60%80%100%0%25%50%75%100%Coverage (answered / 547)Error rate among answeredClaude Fable 5.2 · max · chat: English, five options, held back by confidence, run 971 (29 points)Gemini 4 Hydrogen · high · chat: English, five options, held back by confidence, run 1107 (21 points)GPT-6 Astra · low · chat: English, five options, held back by confidence, run 616 (25 points)GPT-6 Astra · high · chat: English, five options, held back by confidence, run 615 (26 points)GPT-6.1 Terra · high · chat: English, five options, held back by confidence, run 620 (29 points)GPT-6.1 Terra · low · chat: English, five options, held back by confidence, run 621 (25 points)Claude Fable 5.1 · high · chat: English, five options, held back by confidence, run 580 (18 points)Gemini 3.8 Flash · high · chat: English, five options, held back by confidence, run 583 (12 points)Gemini 4 Argon · low · chat: English, five options, held back by confidence, run 593 (12 points)GPT-6 Astra · medium · chat: English, five options, held back by confidence, run 624 (27 points)GPT-6.1 Sol · low · chat: English, five options, held back by confidence, run 619 (22 points)GPT-6 Astra · low · chat · 1 per request: English, five options, held back by confidence, run 622 (26 points)GPT-6.1 Sol · high · chat: English, five options, held back by confidence, run 618 (25 points)Gemini 3.8 Flash · low · chat · 1 per request: English, five options, held back by confidence, run 589 (9 points)GPT-6.1 Sol · low · chat · 1 per request: English, five options, held back by confidence, run 623 (23 points)Claude Opus 5.5 · low · chat · 1 per request: English, five options, held back by confidence, run 592 (20 points)Claude Opus 5.5 · high · chat: English, five options, held back by confidence, run 587 (21 points)Qwen4-Max · high · chat: English, five options, held back by confidence, run 1334 (29 points)Gemini 3.8 Flash · low · chat: English, five options, held back by confidence, run 578 (9 points)Gemini 4 Carbon · high · chat: English, five options, held back by confidence, run 1055 (20 points)KnowLine-31B-Gen1.5: English, five options, held back by confidence, run 953 (547 points)Grok 4.7 · high · chat: English, five options, held back by confidence, run 579 (45 points)Claude Opus 5.5 · low · chat: English, five options, held back by confidence, run 588 (14 points)Copernicus-31B: English, five options, held back by confidence, run 1168 (547 points)Claude Haiku 5.5 · max · chat: English, five options, held back by confidence, run 949 (19 points)DeepSeek-V4.1-Flash · reasoning on · chat · 1017: English, five options, held back by confidence, run 808 (547 points)REAX 1.3: English, five options, held back by confidence, run 829 (547 points)Kimi K3.1 Pro · low · chat: English, five options, held back by confidence, run 584 (18 points)KnowLine-31B-Gen2: English, five options, held back by confidence, run 1327 (547 points)Karpo Decision 31B: English, five options, held back by confidence, run 1159 (547 points)Claude Sonnet 5.5 · high · chat · 1 per request: English, five options, held back by confidence, run 594 (21 points)Kimi K3.1 Pro · no reasoning · chat: English, five options, held back by confidence, run 582 (25 points)Qwen3.5-397B-A17B · high · chat: English, five options, held back by confidence, run 871 (13 points)Gemini Decision 1 · 20261111: English, five options, held back by confidence, run 1009 (547 points)Logits Sigma-1 Duo: English, five options, held back by confidence, run 1229 (547 points)GLiDE 27B: English, five options, held back by confidence, run 901 (547 points)d1-ultra: English, five options, held back by confidence, run 411 (547 points)Sieve 31B v3: English, five options, held back by confidence, run 1064 (438 points)Turion Jev 31B: English, five options, held back by confidence, run 1075 (546 points)Gemma 4 31B · reasoning off · chat: English, five options, held back by confidence, run 774 (547 points)Perplexity Decider 27B v1.2 · API · with web: English, five options, held back by confidence, run 639 (547 points)GLiDE · v1.2: English, five options, held back by confidence, run 521 (547 points)d1.2-ultra: English, five options, held back by confidence, run 1359 (547 points)Mammoth-1: English, five options, held back by confidence, run 503 (543 points)Gemma 4 31B · reasoning off · chat · 1 per request: English, five options, held back by confidence, run 783 (547 points)d3: English, five options, held back by confidence, run 554 (547 points)Drex Max: English, five options, held back by confidence, run 552 (547 points)GLM-5.3-Flash · reasoning on · chat: English, five options, held back by confidence, run 823 (20 points)Muse Ember · max · chat: English, five options, held back by confidence, run 942 (27 points)Perplexity Decider 27B v1.3: English, five options, held back by confidence, run 1137 (547 points)d3-ultra: English, five options, held back by confidence, run 550 (547 points)d1.2: English, five options, held back by confidence, run 1357 (547 points)MiMo-V2.6-Flash · high · chat: English, five options, held back by confidence, run 878 (28 points)Qwen3.8-27B · low · chat: English, five options, held back by confidence, run 667 (547 points)Perplexity Decider 27B v1.2 · DGX Spark: English, five options, held back by confidence, run 527 (547 points)Drex Max v1.1: English, five options, held back by confidence, run 399 (547 points)Perplexity Decider 27B v1: English, five options, held back by confidence, run 626 (547 points)Perplexity Decider 27B v1.2 · API: English, five options, held back by confidence, run 637 (547 points)Inkling-Small · high · chat · Jul 2026: English, five options, held back by confidence, run 867 (30 points)d1.1: English, five options, held back by confidence, run 565 (547 points)GLM-5.3 · low · chat: English, five options, held back by confidence, run 577 (19 points)Jeffan-9B-1211: English, five options, held back by confidence, run 1195 (547 points)GLM-5.3-Flash · reasoning off · chat: English, five options, held back by confidence, run 820 (13 points)DeepSeek-V4.1-Flash · reasoning off · chat · 1017: English, five options, held back by confidence, run 804 (547 points)Qwen3.8 Flash · no reasoning · chat: English, five options, held back by confidence, run 986 (15 points)d1: English, five options, held back by confidence, run 430 (547 points)StartLux-Decision-v1.1-9B-Instant: English, five options, held back by confidence, run 1293 (547 points)Clef: English, five options, held back by confidence, run 613 (435 points)KnowLine-12B-Gen1: English, five options, held back by confidence, run 1313 (547 points)GLiDE 9B: English, five options, held back by confidence, run 909 (547 points)GPT-6 Luna · low · chat: English, five options, held back by confidence, run 617 (27 points)OneJev-27B: English, five options, held back by confidence, run 778 (546 points)Claude Sonnet 5.5 · high · chat: English, five options, held back by confidence, run 590 (19 points)Jev 1.14: English, five options, held back by confidence, run 410 (66 points)Cerno 9B v1.2: English, five options, held back by confidence, run 1085 (492 points)Drex 2.0: English, five options, held back by confidence, run 566 (499 points)Jev 1.13: English, five options, held back by confidence, run 564 (66 points)KnowLine-9B-Gen1: English, five options, held back by confidence, run 1177 (547 points)SubQ Snap 1.1: English, five options, held back by confidence, run 413 (547 points)SubQ Snap Preview: English, five options, held back by confidence, run 595 (547 points)Adjudex 9B v5: English, five options, held back by confidence, run 1186 (547 points)Mercury Decide: English, five options, held back by confidence, run 398 (547 points)Microsoft-Decision-1: English, five options, held back by confidence, run 412 (547 points)Elephant-1: English, five options, held back by confidence, run 499 (547 points)Open-Jev-27B-v1.1: English, five options, held back by confidence, run 760 (547 points)KnowLine-9B-Gen2: English, five options, held back by confidence, run 1321 (547 points)Qwen3.8-27B · stated confidence · reasoning off · chat: English, five options, held back by confidence, run 672 (5 points)Qwen3.8-27B · yes/no scoring: English, five options, held back by confidence, run 696 (547 points)MiniMax-M3.1-Flash · high · chat: English, five options, held back by confidence, run 863 (33 points)celeris-1-decision: English, five options, held back by confidence, run 546 (65 points)GPT-6 Luna · Decisions API: English, five options, held back by confidence, run 397 (68 points)Claude Sonnet 5.5 · low · chat: English, five options, held back by confidence, run 591 (17 points)Cygnet: English, five options, held back by confidence, run 753 (547 points)Qwen3.8-27B · vote share · reasoning off · chat: English, five options, held back by confidence, run 678 (8 points)Winnow-12B: English, five options, held back by confidence, run 746 (547 points)Perplexity Decider Flash v1.3: English, five options, held back by confidence, run 1126 (547 points)Valen-Decision-1.5-9B: English, five options, held back by confidence, run 1278 (547 points)Dexy-Decision-9B-V2: English, five options, held back by confidence, run 1258 (547 points)vera-medium-omni: English, five options, held back by confidence, run 1267 (523 points)StartLux-Decision-v1.1-4B-Instant: English, five options, held back by confidence, run 1081 (547 points)Clef-Flash: English, five options, held back by confidence, run 614 (492 points)Verdin-4B-v1.1: English, five options, held back by confidence, run 1247 (547 points)Perplexity Decider Flash v1.2: English, five options, held back by confidence, run 533 (547 points)KnowLine 4B Gen5.5: English, five options, held back by confidence, run 961 (547 points)Chinese Decision Pointer 4B: English, five options, held back by confidence, run 1209 (547 points)Qwen3-14B · reasoning off · chat: English, five options, held back by confidence, run 770 (547 points)Metask-Jev-4B: English, five options, held back by confidence, run 739 (547 points)Erk-14B · reasoning off · chat: English, five options, held back by confidence, run 762 (544 points)KnowLine-2B-Gen1: English, five options, held back by confidence, run 1057 (547 points)GLiDE 2B: English, five options, held back by confidence, run 917 (547 points)StartLux-Decision-v1.1-2B-Instant: English, five options, held back by confidence, run 973 (547 points)d1-2B: English, five options, held back by confidence, run 708 (547 points)GLiDE 0.8B: English, five options, held back by confidence, run 925 (547 points)StartLux-Decision-v1.1-0.8B-Instant: English, five options, held back by confidence, run 1011 (547 points)GLiDE 0.4B: English, five options, held back by confidence, run 933 (547 points)OneJev1.6-0.8B: English, five options, held back by confidence, run 1025 (547 points)KnowLine-0.8B-Gen1: English, five options, held back by confidence, run 1001 (547 points)GLiClass v4 Edge: English, five options, held back by confidence, run 1042 (547 points)Decider 1: English, five options, held back by confidence, run 596 (525 points)Strands Decider 2B: English, five options, held back by confidence, run 702 (525 points)Laya: English, five options, held back by confidence, run 726 (496 points)GLiNER2.5-multi-Decide: English, five options, held back by confidence, run 714 (547 points)GLiNER2.5 Multi: English, five options, held back by confidence, run 720 (547 points)Laya Multilingual: English, five options, held back by confidence, run 732 (522 points)Claude Fable 5.2 · max · chat: English with “I don't know”, run 1032: coverage 99.3%, error rate among answered 1.8%; 533 right, 10 wrong, 4 abstained, 0 invalidGemini 4 Hydrogen · high · chat: English with “I don't know”, run 1109: coverage 99.3%, error rate among answered 2.9%; 527 right, 16 wrong, 4 abstained, 0 invalidGPT-6 Astra · low · chat: English with “I don't know”, run 647: coverage 99.3%, error rate among answered 3.3%; 525 right, 18 wrong, 4 abstained, 0 invalidGPT-6 Astra · high · chat: English with “I don't know”, run 814: coverage 99.3%, error rate among answered 3.1%; 526 right, 17 wrong, 4 abstained, 0 invalidGPT-6.1 Terra · high · chat: English with “I don't know”, run 766: coverage 99.3%, error rate among answered 3.7%; 523 right, 20 wrong, 4 abstained, 0 invalidGPT-6.1 Terra · low · chat: English with “I don't know”, run 751: coverage 99.3%, error rate among answered 4.1%; 521 right, 22 wrong, 4 abstained, 0 invalidClaude Fable 5.1 · high · chat: English with “I don't know”, run 655: coverage 99.6%, error rate among answered 4.6%; 520 right, 25 wrong, 2 abstained, 0 invalidGemini 3.8 Flash · high · chat: English with “I don't know”, run 657: coverage 99.6%, error rate among answered 3.3%; 527 right, 18 wrong, 2 abstained, 0 invalidGemini 4 Argon · low · chat: English with “I don't know”, run 652: coverage 99.1%, error rate among answered 4.1%; 520 right, 22 wrong, 5 abstained, 0 invalidGPT-6 Astra · medium · chat: English with “I don't know”, run 813: coverage 99.3%, error rate among answered 3.7%; 523 right, 20 wrong, 4 abstained, 0 invalidGPT-6.1 Sol · low · chat: English with “I don't know”, run 744: coverage 99.3%, error rate among answered 4.2%; 520 right, 23 wrong, 4 abstained, 0 invalidGPT-6 Astra · low · chat · 1 per request: English with “I don't know”, run 815: coverage 99.3%, error rate among answered 4.2%; 520 right, 23 wrong, 4 abstained, 0 invalidGPT-6.1 Sol · high · chat: English with “I don't know”, run 758: coverage 99.3%, error rate among answered 3.7%; 523 right, 20 wrong, 4 abstained, 0 invalidGemini 3.8 Flash · low · chat · 1 per request: English with “I don't know”, run 669: coverage 99.6%, error rate among answered 4.4%; 521 right, 24 wrong, 2 abstained, 0 invalidGPT-6.1 Sol · low · chat · 1 per request: English with “I don't know”, run 769: coverage 99.3%, error rate among answered 5.0%; 516 right, 27 wrong, 4 abstained, 0 invalidClaude Opus 5.5 · low · chat · 1 per request: English with “I don't know”, run 670: coverage 100.0%, error rate among answered 5.7%; 516 right, 31 wrong, 0 abstained, 0 invalidClaude Opus 5.5 · high · chat: English with “I don't know”, run 665: coverage 99.6%, error rate among answered 5.3%; 516 right, 29 wrong, 2 abstained, 0 invalidQwen4-Max · high · chat: English with “I don't know”, run 1338: coverage 100.0%, error rate among answered 5.3%; 518 right, 29 wrong, 0 abstained, 0 invalidGemini 3.8 Flash · low · chat: English with “I don't know”, run 645: coverage 99.8%, error rate among answered 5.1%; 518 right, 28 wrong, 1 abstained, 0 invalidGemini 4 Carbon · high · chat: English with “I don't know”, run 1067: coverage 99.3%, error rate among answered 2.2%; 531 right, 12 wrong, 4 abstained, 0 invalidKnowLine-31B-Gen1.5: English with “I don't know”, run 955: coverage 100.0%, error rate among answered 6.8%; 510 right, 37 wrong, 0 abstained, 0 invalidGrok 4.7 · high · chat: English with “I don't know”, run 648: coverage 97.8%, error rate among answered 5.8%; 504 right, 31 wrong, 12 abstained, 0 invalidClaude Opus 5.5 · low · chat: English with “I don't know”, run 649: coverage 99.6%, error rate among answered 7.5%; 504 right, 41 wrong, 2 abstained, 0 invalidCopernicus-31B: English with “I don't know”, run 1170: coverage 100.0%, error rate among answered 7.1%; 508 right, 39 wrong, 0 abstained, 0 invalidClaude Haiku 5.5 · max · chat: English with “I don't know”, run 951: coverage 96.3%, error rate among answered 8.9%; 480 right, 47 wrong, 4 abstained, 16 invalidDeepSeek-V4.1-Flash · reasoning on · chat · 1017: English with “I don't know”, run 809: coverage 99.5%, error rate among answered 7.2%; 505 right, 39 wrong, 3 abstained, 0 invalidREAX 1.3: English with “I don't know”, run 831: coverage 99.5%, error rate among answered 7.7%; 502 right, 42 wrong, 3 abstained, 0 invalidKimi K3.1 Pro · low · chat: English with “I don't know”, run 651: coverage 99.5%, error rate among answered 7.0%; 506 right, 38 wrong, 3 abstained, 0 invalidKnowLine-31B-Gen2: English with “I don't know”, run 1329: coverage 100.0%, error rate among answered 7.3%; 507 right, 40 wrong, 0 abstained, 0 invalidKarpo Decision 31B: English with “I don't know”, run 1161: coverage 100.0%, error rate among answered 7.7%; 505 right, 42 wrong, 0 abstained, 0 invalidClaude Sonnet 5.5 · high · chat · 1 per request: English with “I don't know”, run 677: coverage 100.0%, error rate among answered 6.9%; 509 right, 38 wrong, 0 abstained, 0 invalidKimi K3.1 Pro · no reasoning · chat: English with “I don't know”, run 646: coverage 99.3%, error rate among answered 9.0%; 494 right, 49 wrong, 4 abstained, 0 invalidQwen3.5-397B-A17B · high · chat: English with “I don't know”, run 873: coverage 99.5%, error rate among answered 8.3%; 499 right, 45 wrong, 3 abstained, 0 invalidGemini Decision 1 · 20261111: English with “I don't know”, run 1019: coverage 99.1%, error rate among answered 9.2%; 492 right, 50 wrong, 5 abstained, 0 invalidLogits Sigma-1 Duo: English with “I don't know”, run 1231: coverage 99.6%, error rate among answered 8.1%; 501 right, 44 wrong, 2 abstained, 0 invalidGLiDE 27B: English with “I don't know”, run 903: coverage 96.0%, error rate among answered 6.9%; 489 right, 36 wrong, 22 abstained, 0 invalidd1-ultra: English with “I don't know”, run 449: coverage 90.7%, error rate among answered 7.9%; 457 right, 39 wrong, 51 abstained, 0 invalidSieve 31B v3: English with “I don't know”, run 1066: coverage 99.8%, error rate among answered 8.2%; 501 right, 45 wrong, 1 abstained, 0 invalidTurion Jev 31B: English with “I don't know”, run 1077: coverage 99.3%, error rate among answered 7.9%; 500 right, 43 wrong, 4 abstained, 0 invalidGemma 4 31B · reasoning off · chat: English with “I don't know”, run 775: coverage 99.6%, error rate among answered 9.7%; 492 right, 53 wrong, 2 abstained, 0 invalidPerplexity Decider 27B v1.2 · API · with web: English with “I don't know”, run 644: coverage 98.2%, error rate among answered 10.4%; 481 right, 56 wrong, 10 abstained, 0 invalidGLiDE · v1.2: English with “I don't know”, run 523: coverage 94.5%, error rate among answered 7.0%; 481 right, 36 wrong, 30 abstained, 0 invalidd1.2-ultra: English with “I don't know”, run 1363: coverage 80.1%, error rate among answered 3.0%; 425 right, 13 wrong, 109 abstained, 0 invalidMammoth-1: English with “I don't know”, run 505: coverage 93.2%, error rate among answered 7.1%; 474 right, 36 wrong, 37 abstained, 0 invalidGemma 4 31B · reasoning off · chat · 1 per request: English with “I don't know”, run 786: coverage 100.0%, error rate among answered 10.8%; 488 right, 59 wrong, 0 abstained, 0 invalidd3: English with “I don't know”, run 793: coverage 99.8%, error rate among answered 10.6%; 488 right, 58 wrong, 1 abstained, 0 invalidDrex Max: English with “I don't know”, run 798: coverage 99.6%, error rate among answered 11.0%; 485 right, 60 wrong, 2 abstained, 0 invalidGLM-5.3-Flash · reasoning on · chat: English with “I don't know”, run 824: coverage 99.8%, error rate among answered 11.2%; 485 right, 61 wrong, 1 abstained, 0 invalidMuse Ember · max · chat: English with “I don't know”, run 943: coverage 99.5%, error rate among answered 9.9%; 490 right, 54 wrong, 3 abstained, 0 invalidPerplexity Decider 27B v1.3: English with “I don't know”, run 1139: coverage 99.1%, error rate among answered 9.6%; 490 right, 52 wrong, 5 abstained, 0 invalidd3-ultra: English with “I don't know”, run 795: coverage 99.1%, error rate among answered 10.3%; 486 right, 56 wrong, 5 abstained, 0 invalidd1.2: English with “I don't know”, run 1361: coverage 89.2%, error rate among answered 7.6%; 451 right, 37 wrong, 59 abstained, 0 invalidMiMo-V2.6-Flash · high · chat: English with “I don't know”, run 880: coverage 99.8%, error rate among answered 10.6%; 488 right, 58 wrong, 1 abstained, 0 invalidQwen3.8-27B · low · chat: English with “I don't know”, run 671: coverage 99.6%, error rate among answered 9.7%; 492 right, 53 wrong, 2 abstained, 0 invalidPerplexity Decider 27B v1.2 · DGX Spark: English with “I don't know”, run 529: coverage 99.6%, error rate among answered 11.7%; 481 right, 64 wrong, 2 abstained, 0 invalidDrex Max v1.1: English with “I don't know”, run 447: coverage 92.1%, error rate among answered 8.1%; 463 right, 41 wrong, 43 abstained, 0 invalidPerplexity Decider 27B v1.2 · API: English with “I don't know”, run 643: coverage 99.5%, error rate among answered 11.6%; 481 right, 63 wrong, 3 abstained, 0 invalidInkling-Small · high · chat · Jul 2026: English with “I don't know”, run 869: coverage 100.0%, error rate among answered 11.0%; 487 right, 60 wrong, 0 abstained, 0 invalidGLM-5.3 · low · chat: English with “I don't know”, run 650: coverage 99.6%, error rate among answered 10.3%; 489 right, 56 wrong, 2 abstained, 0 invalidJeffan-9B-1211: English with “I don't know”, run 1197: coverage 100.0%, error rate among answered 13.0%; 476 right, 71 wrong, 0 abstained, 0 invalidGLM-5.3-Flash · reasoning off · chat: English with “I don't know”, run 821: coverage 99.3%, error rate among answered 12.3%; 476 right, 67 wrong, 4 abstained, 0 invalidDeepSeek-V4.1-Flash · reasoning off · chat · 1017: English with “I don't know”, run 805: coverage 99.3%, error rate among answered 12.9%; 473 right, 70 wrong, 4 abstained, 0 invalidQwen3.5-4B · reasoning on · chat: English with “I don't know”, run 1073: coverage 99.6%, error rate among answered 15.2%; 462 right, 83 wrong, 2 abstained, 0 invalidQwen3.8 Flash · no reasoning · chat: English with “I don't know”, run 988: coverage 99.3%, error rate among answered 12.3%; 476 right, 67 wrong, 4 abstained, 0 invalidd1: English with “I don't know”, run 445: coverage 99.6%, error rate among answered 12.3%; 478 right, 67 wrong, 2 abstained, 0 invalidStartLux-Decision-v1.1-9B-Instant: English with “I don't know”, run 1295: coverage 99.6%, error rate among answered 13.6%; 471 right, 74 wrong, 2 abstained, 0 invalidClef: English with “I don't know”, run 785: coverage 99.8%, error rate among answered 13.7%; 471 right, 75 wrong, 1 abstained, 0 invalidKnowLine-12B-Gen1: English with “I don't know”, run 1316: coverage 99.8%, error rate among answered 13.9%; 470 right, 76 wrong, 1 abstained, 0 invalidGLiDE 9B: English with “I don't know”, run 911: coverage 97.4%, error rate among answered 13.1%; 463 right, 70 wrong, 14 abstained, 0 invalidGPT-6 Luna · low · chat: English with “I don't know”, run 735: coverage 99.1%, error rate among answered 12.7%; 473 right, 69 wrong, 5 abstained, 0 invalidOneJev-27B: English with “I don't know”, run 780: coverage 99.5%, error rate among answered 13.2%; 472 right, 72 wrong, 3 abstained, 0 invalidClaude Sonnet 5.5 · high · chat: English with “I don't know”, run 668: coverage 99.5%, error rate among answered 10.8%; 485 right, 59 wrong, 3 abstained, 0 invalidJev 1.14: English with “I don't know”, run 446: coverage 99.5%, error rate among answered 13.2%; 472 right, 72 wrong, 3 abstained, 0 invalidCerno 9B v1.2: English with “I don't know”, run 1087: coverage 99.6%, error rate among answered 14.1%; 468 right, 77 wrong, 2 abstained, 0 invalidKnowLine-9B-Gen1: English with “I don't know”, run 1179: coverage 99.8%, error rate among answered 13.9%; 470 right, 76 wrong, 1 abstained, 0 invalidSubQ Snap 1.1: English with “I don't know”, run 448: coverage 96.2%, error rate among answered 12.0%; 463 right, 63 wrong, 21 abstained, 0 invalidAdjudex 9B v5: English with “I don't know”, run 1188: coverage 100.0%, error rate among answered 14.1%; 470 right, 77 wrong, 0 abstained, 0 invalidMercury Decide: English with “I don't know”, run 450: coverage 98.0%, error rate among answered 13.2%; 465 right, 71 wrong, 11 abstained, 0 invalidMicrosoft-Decision-1: English with “I don't know”, run 451: coverage 99.5%, error rate among answered 14.2%; 467 right, 77 wrong, 3 abstained, 0 invalidElephant-1: English with “I don't know”, run 501: coverage 95.8%, error rate among answered 12.8%; 457 right, 67 wrong, 23 abstained, 0 invalidOpen-Jev-27B-v1.1: English with “I don't know”, run 763: coverage 97.1%, error rate among answered 14.3%; 455 right, 76 wrong, 16 abstained, 0 invalidKnowLine-9B-Gen2: English with “I don't know”, run 1323: coverage 99.8%, error rate among answered 14.3%; 468 right, 78 wrong, 1 abstained, 0 invalidQwen3.8-27B · stated confidence · reasoning off · chat: English with “I don't know”, run 673: coverage 99.5%, error rate among answered 14.3%; 466 right, 78 wrong, 3 abstained, 0 invalidMiniMax-M3.1-Flash · high · chat: English with “I don't know”, run 865: coverage 99.5%, error rate among answered 15.6%; 459 right, 85 wrong, 3 abstained, 0 invalidceleris-1-decision: English with “I don't know”, run 548: coverage 83.2%, error rate among answered 8.1%; 418 right, 37 wrong, 92 abstained, 0 invalidGPT-6 Luna · Decisions API: English with “I don't know”, run 444: coverage 98.2%, error rate among answered 17.9%; 441 right, 96 wrong, 10 abstained, 0 invalidClaude Sonnet 5.5 · low · chat: English with “I don't know”, run 654: coverage 99.3%, error rate among answered 15.8%; 457 right, 86 wrong, 4 abstained, 0 invalidCygnet: English with “I don't know”, run 755: coverage 98.2%, error rate among answered 14.7%; 458 right, 79 wrong, 10 abstained, 0 invalidQwen3.8-27B · vote share · reasoning off · chat: English with “I don't know”, run 680: coverage 99.5%, error rate among answered 15.4%; 460 right, 84 wrong, 3 abstained, 0 invalidWinnow-12B: English with “I don't know”, run 748: coverage 97.3%, error rate among answered 16.0%; 447 right, 85 wrong, 15 abstained, 0 invalidPerplexity Decider Flash v1.3: English with “I don't know”, run 1128: coverage 98.7%, error rate among answered 16.7%; 450 right, 90 wrong, 7 abstained, 0 invalidValen-Decision-1.5-9B: English with “I don't know”, run 1280: coverage 99.8%, error rate among answered 15.4%; 462 right, 84 wrong, 1 abstained, 0 invalidDexy-Decision-9B-V2: English with “I don't know”, run 1260: coverage 99.8%, error rate among answered 16.5%; 456 right, 90 wrong, 1 abstained, 0 invalidvera-medium-omni: English with “I don't know”, run 1269: coverage 98.7%, error rate among answered 16.1%; 453 right, 87 wrong, 7 abstained, 0 invalidStartLux-Decision-v1.1-4B-Instant: English with “I don't know”, run 1083: coverage 100.0%, error rate among answered 17.2%; 453 right, 94 wrong, 0 abstained, 0 invalidGemma 4 E4B · default reasoning · chat: English with “I don't know”, run 890: coverage 97.8%, error rate among answered 17.2%; 443 right, 92 wrong, 10 abstained, 2 invalidClef-Flash: English with “I don't know”, run 660: coverage 99.1%, error rate among answered 16.6%; 452 right, 90 wrong, 5 abstained, 0 invalidVerdin-4B-v1.1: English with “I don't know”, run 1249: coverage 99.6%, error rate among answered 17.2%; 451 right, 94 wrong, 2 abstained, 0 invalidPerplexity Decider Flash v1.2: English with “I don't know”, run 535: coverage 99.1%, error rate among answered 18.5%; 442 right, 100 wrong, 5 abstained, 0 invalidKnowLine 4B Gen5.5: English with “I don't know”, run 963: coverage 99.8%, error rate among answered 19.2%; 441 right, 105 wrong, 1 abstained, 0 invalidChinese Decision Pointer 4B: English with “I don't know”, run 1211: coverage 99.3%, error rate among answered 19.2%; 439 right, 104 wrong, 4 abstained, 0 invalidQwen3.5-4B · reasoning off · chat: English with “I don't know”, run 1144: coverage 99.3%, error rate among answered 19.5%; 437 right, 106 wrong, 4 abstained, 0 invalidQwen3-14B · reasoning off · chat: English with “I don't know”, run 771: coverage 99.6%, error rate among answered 19.8%; 437 right, 108 wrong, 2 abstained, 0 invalidMetask-Jev-4B: English with “I don't know”, run 741: coverage 99.6%, error rate among answered 22.6%; 422 right, 123 wrong, 2 abstained, 0 invalidLFM2.5-2.6B · default reasoning · chat: English with “I don't know”, run 1035: coverage 95.6%, error rate among answered 21.6%; 410 right, 113 wrong, 21 abstained, 3 invalidErk-14B · reasoning off · chat: English with “I don't know”, run 764: coverage 99.5%, error rate among answered 25.9%; 403 right, 141 wrong, 3 abstained, 0 invalidLFM3-1.7B · reasoning off · chat: English with “I don't know”, run 1225: coverage 96.7%, error rate among answered 25.9%; 392 right, 137 wrong, 18 abstained, 0 invalidKnowLine-2B-Gen1: English with “I don't know”, run 1059: coverage 93.1%, error rate among answered 26.5%; 374 right, 135 wrong, 38 abstained, 0 invalidLFM3-1.7B · reasoning on · chat: English with “I don't know”, run 1062: coverage 95.2%, error rate among answered 27.1%; 380 right, 141 wrong, 26 abstained, 0 invalidLFM2.5-8B-A1B · default reasoning · chat: English with “I don't know”, run 887: coverage 90.5%, error rate among answered 28.7%; 353 right, 142 wrong, 51 abstained, 1 invalidGLiDE 2B: English with “I don't know”, run 919: coverage 91.4%, error rate among answered 26.4%; 368 right, 132 wrong, 47 abstained, 0 invalidStartLux-Decision-v1.1-2B-Instant: English with “I don't know”, run 975: coverage 94.1%, error rate among answered 30.5%; 358 right, 157 wrong, 32 abstained, 0 invalidMiniCPM5-2B · reasoning off · chat: English with “I don't know”, run 1095: coverage 89.2%, error rate among answered 33.6%; 324 right, 164 wrong, 33 abstained, 26 invalidQwen3.5-2B · reasoning off · chat: English with “I don't know”, run 1216: coverage 96.3%, error rate among answered 29.6%; 371 right, 156 wrong, 18 abstained, 2 invalidd1-2B: English with “I don't know”, run 710: coverage 72.4%, error rate among answered 24.5%; 299 right, 97 wrong, 151 abstained, 0 invalidGLiDE 0.8B: English with “I don't know”, run 927: coverage 91.0%, error rate among answered 35.5%; 321 right, 177 wrong, 49 abstained, 0 invalidStartLux-Decision-v1.1-0.8B-Instant: English with “I don't know”, run 1013: coverage 97.8%, error rate among answered 40.6%; 318 right, 217 wrong, 12 abstained, 0 invalidGLiDE 0.4B: English with “I don't know”, run 935: coverage 72.0%, error rate among answered 34.3%; 259 right, 135 wrong, 153 abstained, 0 invalidOneJev1.6-0.8B: English with “I don't know”, run 1027: coverage 86.5%, error rate among answered 39.5%; 286 right, 187 wrong, 74 abstained, 0 invalidKnowLine-0.8B-Gen1: English with “I don't know”, run 1003: coverage 95.2%, error rate among answered 41.7%; 304 right, 217 wrong, 26 abstained, 0 invalidGLiClass v4 Edge: English with “I don't know”, run 1044: coverage 90.5%, error rate among answered 39.6%; 299 right, 196 wrong, 52 abstained, 0 invalidStrands Decider 2B: English with “I don't know”, run 704: coverage 99.6%, error rate among answered 55.2%; 244 right, 301 wrong, 2 abstained, 0 invalidLaya: English with “I don't know”, run 728: coverage 84.8%, error rate among answered 79.1%; 97 right, 367 wrong, 83 abstained, 0 invalidGLiNER2.5-multi-Decide: English with “I don't know”, run 716: coverage 87.2%, error rate among answered 76.1%; 114 right, 363 wrong, 70 abstained, 0 invalidGLiNER2.5 Multi: English with “I don't know”, run 722: coverage 83.2%, error rate among answered 83.1%; 77 right, 378 wrong, 92 abstained, 0 invalidLaya Multilingual: English with “I don't know”, run 734: coverage 62.0%, error rate among answered 82.6%; 59 right, 280 wrong, 208 abstained, 0 invalid
  • Claude Fable 5.2 · max · chat: confidence, run 971 · “I don't know”, run 1032
  • Gemini 4 Hydrogen · high · chat: confidence, run 1107 · “I don't know”, run 1109
  • GPT-6 Astra · low · chat: confidence, run 616 · “I don't know”, run 647
  • GPT-6 Astra · high · chat: confidence, run 615 · “I don't know”, run 814
  • GPT-6.1 Terra · high · chat: confidence, run 620 · “I don't know”, run 766
  • GPT-6.1 Terra · low · chat: confidence, run 621 · “I don't know”, run 751
  • Claude Fable 5.1 · high · chat: confidence, run 580 · “I don't know”, run 655
  • Gemini 3.8 Flash · high · chat: confidence, run 583 · “I don't know”, run 657
  • Gemini 4 Argon · low · chat: confidence, run 593 · “I don't know”, run 652
  • GPT-6 Astra · medium · chat: confidence, run 624 · “I don't know”, run 813
  • GPT-6.1 Sol · low · chat: confidence, run 619 · “I don't know”, run 744
  • GPT-6 Astra · low · chat · 1 per request: confidence, run 622 · “I don't know”, run 815
  • GPT-6.1 Sol · high · chat: confidence, run 618 · “I don't know”, run 758
  • Gemini 3.8 Flash · low · chat · 1 per request: confidence, run 589 · “I don't know”, run 669
  • GPT-6.1 Sol · low · chat · 1 per request: confidence, run 623 · “I don't know”, run 769
  • Claude Opus 5.5 · low · chat · 1 per request: confidence, run 592 · “I don't know”, run 670
  • Claude Opus 5.5 · high · chat: confidence, run 587 · “I don't know”, run 665
  • Qwen4-Max · high · chat: confidence, run 1334 · “I don't know”, run 1338
  • Gemini 3.8 Flash · low · chat: confidence, run 578 · “I don't know”, run 645
  • Gemini 4 Carbon · high · chat: confidence, run 1055 · “I don't know”, run 1067
  • KnowLine-31B-Gen1.5: confidence, run 953 · “I don't know”, run 955
  • Grok 4.7 · high · chat: confidence, run 579 · “I don't know”, run 648
  • Claude Opus 5.5 · low · chat: confidence, run 588 · “I don't know”, run 649
  • Copernicus-31B: confidence, run 1168 · “I don't know”, run 1170
  • Claude Haiku 5.5 · max · chat: confidence, run 949 · “I don't know”, run 951
  • DeepSeek-V4.1-Flash · reasoning on · chat · 1017: confidence, run 808 · “I don't know”, run 809
  • REAX 1.3: confidence, run 829 · “I don't know”, run 831
  • Kimi K3.1 Pro · low · chat: confidence, run 584 · “I don't know”, run 651
  • KnowLine-31B-Gen2: confidence, run 1327 · “I don't know”, run 1329
  • Karpo Decision 31B: confidence, run 1159 · “I don't know”, run 1161
  • Claude Sonnet 5.5 · high · chat · 1 per request: confidence, run 594 · “I don't know”, run 677
  • Kimi K3.1 Pro · no reasoning · chat: confidence, run 582 · “I don't know”, run 646
  • Qwen3.5-397B-A17B · high · chat: confidence, run 871 · “I don't know”, run 873
  • Gemini Decision 1 · 20261111: confidence, run 1009 · “I don't know”, run 1019
  • Logits Sigma-1 Duo: confidence, run 1229 · “I don't know”, run 1231
  • GLiDE 27B: confidence, run 901 · “I don't know”, run 903
  • d1-ultra: confidence, run 411 · “I don't know”, run 449
  • Sieve 31B v3: confidence, run 1064 · “I don't know”, run 1066
  • Turion Jev 31B: confidence, run 1075 · “I don't know”, run 1077
  • Gemma 4 31B · reasoning off · chat: confidence, run 774 · “I don't know”, run 775
  • Perplexity Decider 27B v1.2 · API · with web: confidence, run 639 · “I don't know”, run 644
  • GLiDE · v1.2: confidence, run 521 · “I don't know”, run 523
  • d1.2-ultra: confidence, run 1359 · “I don't know”, run 1363
  • Mammoth-1: confidence, run 503 · “I don't know”, run 505
  • Gemma 4 31B · reasoning off · chat · 1 per request: confidence, run 783 · “I don't know”, run 786
  • d3: confidence, run 554 · “I don't know”, run 793
  • Drex Max: confidence, run 552 · “I don't know”, run 798
  • GLM-5.3-Flash · reasoning on · chat: confidence, run 823 · “I don't know”, run 824
  • Muse Ember · max · chat: confidence, run 942 · “I don't know”, run 943
  • Perplexity Decider 27B v1.3: confidence, run 1137 · “I don't know”, run 1139
  • d3-ultra: confidence, run 550 · “I don't know”, run 795
  • d1.2: confidence, run 1357 · “I don't know”, run 1361
  • MiMo-V2.6-Flash · high · chat: confidence, run 878 · “I don't know”, run 880
  • Qwen3.8-27B · low · chat: confidence, run 667 · “I don't know”, run 671
  • Perplexity Decider 27B v1.2 · DGX Spark: confidence, run 527 · “I don't know”, run 529
  • Drex Max v1.1: confidence, run 399 · “I don't know”, run 447
  • Perplexity Decider 27B v1: confidence, run 626
  • Perplexity Decider 27B v1.2 · API: confidence, run 637 · “I don't know”, run 643
  • Inkling-Small · high · chat · Jul 2026: confidence, run 867 · “I don't know”, run 869
  • d1.1: confidence, run 565
  • GLM-5.3 · low · chat: confidence, run 577 · “I don't know”, run 650
  • Jeffan-9B-1211: confidence, run 1195 · “I don't know”, run 1197
  • GLM-5.3-Flash · reasoning off · chat: confidence, run 820 · “I don't know”, run 821
  • DeepSeek-V4.1-Flash · reasoning off · chat · 1017: confidence, run 804 · “I don't know”, run 805
  • Qwen3.5-4B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1073
  • Qwen3.8 Flash · no reasoning · chat: confidence, run 986 · “I don't know”, run 988
  • d1: confidence, run 430 · “I don't know”, run 445
  • StartLux-Decision-v1.1-9B-Instant: confidence, run 1293 · “I don't know”, run 1295
  • Clef: confidence, run 613 · “I don't know”, run 785
  • KnowLine-12B-Gen1: confidence, run 1313 · “I don't know”, run 1316
  • GLiDE 9B: confidence, run 909 · “I don't know”, run 911
  • GPT-6 Luna · low · chat: confidence, run 617 · “I don't know”, run 735
  • OneJev-27B: confidence, run 778 · “I don't know”, run 780
  • Claude Sonnet 5.5 · high · chat: confidence, run 590 · “I don't know”, run 668
  • Jev 1.14: confidence, run 410 · “I don't know”, run 446
  • Cerno 9B v1.2: confidence, run 1085 · “I don't know”, run 1087
  • Drex 2.0: confidence, run 566
  • Jev 1.13: confidence, run 564
  • KnowLine-9B-Gen1: confidence, run 1177 · “I don't know”, run 1179
  • SubQ Snap 1.1: confidence, run 413 · “I don't know”, run 448
  • SubQ Snap Preview: confidence, run 595
  • Adjudex 9B v5: confidence, run 1186 · “I don't know”, run 1188
  • Mercury Decide: confidence, run 398 · “I don't know”, run 450
  • Microsoft-Decision-1: confidence, run 412 · “I don't know”, run 451
  • Elephant-1: confidence, run 499 · “I don't know”, run 501
  • Open-Jev-27B-v1.1: confidence, run 760 · “I don't know”, run 763
  • KnowLine-9B-Gen2: confidence, run 1321 · “I don't know”, run 1323
  • Qwen3.8-27B · stated confidence · reasoning off · chat: confidence, run 672 · “I don't know”, run 673
  • Qwen3.8-27B · yes/no scoring: confidence, run 696
  • MiniMax-M3.1-Flash · high · chat: confidence, run 863 · “I don't know”, run 865
  • celeris-1-decision: confidence, run 546 · “I don't know”, run 548
  • GPT-6 Luna · Decisions API: confidence, run 397 · “I don't know”, run 444
  • Claude Sonnet 5.5 · low · chat: confidence, run 591 · “I don't know”, run 654
  • Cygnet: confidence, run 753 · “I don't know”, run 755
  • Qwen3.8-27B · vote share · reasoning off · chat: confidence, run 678 · “I don't know”, run 680
  • Winnow-12B: confidence, run 746 · “I don't know”, run 748
  • Perplexity Decider Flash v1.3: confidence, run 1126 · “I don't know”, run 1128
  • Valen-Decision-1.5-9B: confidence, run 1278 · “I don't know”, run 1280
  • Dexy-Decision-9B-V2: confidence, run 1258 · “I don't know”, run 1260
  • vera-medium-omni: confidence, run 1267 · “I don't know”, run 1269
  • StartLux-Decision-v1.1-4B-Instant: confidence, run 1081 · “I don't know”, run 1083
  • Gemma 4 E4B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 890
  • Clef-Flash: confidence, run 614 · “I don't know”, run 660
  • Verdin-4B-v1.1: confidence, run 1247 · “I don't know”, run 1249
  • Perplexity Decider Flash v1.2: confidence, run 533 · “I don't know”, run 535
  • KnowLine 4B Gen5.5: confidence, run 961 · “I don't know”, run 963
  • Chinese Decision Pointer 4B: confidence, run 1209 · “I don't know”, run 1211
  • Qwen3.5-4B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1144
  • Qwen3-14B · reasoning off · chat: confidence, run 770 · “I don't know”, run 771
  • Metask-Jev-4B: confidence, run 739 · “I don't know”, run 741
  • LFM2.5-2.6B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1035
  • Erk-14B · reasoning off · chat: confidence, run 762 · “I don't know”, run 764
  • LFM3-1.7B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1225
  • KnowLine-2B-Gen1: confidence, run 1057 · “I don't know”, run 1059
  • LFM3-1.7B · reasoning on · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1062
  • LFM2.5-8B-A1B · default reasoning · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 887
  • GLiDE 2B: confidence, run 917 · “I don't know”, run 919
  • StartLux-Decision-v1.1-2B-Instant: confidence, run 973 · “I don't know”, run 975
  • MiniCPM5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1095
  • Qwen3.5-2B · reasoning off · chat: confidence not measured (a plain answer line, no confidence) · “I don't know”, run 1216
  • d1-2B: confidence, run 708 · “I don't know”, run 710
  • GLiDE 0.8B: confidence, run 925 · “I don't know”, run 927
  • StartLux-Decision-v1.1-0.8B-Instant: confidence, run 1011 · “I don't know”, run 1013
  • GLiDE 0.4B: confidence, run 933 · “I don't know”, run 935
  • OneJev1.6-0.8B: confidence, run 1025 · “I don't know”, run 1027
  • KnowLine-0.8B-Gen1: confidence, run 1001 · “I don't know”, run 1003
  • GLiClass v4 Edge: confidence, run 1042 · “I don't know”, run 1044
  • Decider 1: confidence, run 596
  • Strands Decider 2B: confidence, run 702 · “I don't know”, run 704
  • Laya: confidence, run 726 · “I don't know”, run 728
  • GLiNER2.5-multi-Decide: confidence, run 714 · “I don't know”, run 716
  • GLiNER2.5 Multi: confidence, run 720 · “I don't know”, run 722
  • Laya Multilingual: confidence, run 732 · “I don't know”, run 734
Info The table, the setups without an English run and our method, then the same with “I don't know”

About this test

Since 5 November 2026 every setup of the Text tab also runs in English, decision models and chat models alike, each with its Text tab run's settings and compared with that run. 133 of the Text tab's 137 setups have an English run so far, 75 decision models and 58 chat model setups; the other four are listed under Results with the reason each has none yet. This tab is separate from the leaderboard; it ranks nothing on the Text tab and changes none of its numbers.

The Text tab's reading questions (units 1 to 6; the grammar questions are left out) were translated into English by Claude Opus 5.5. The same model flagged the questions whose answer depends on Turkish wording; the 109 flagged questions are left out and the other 547 kept. Each of the 133 English setups answered the kept questions in English and is compared with its own Text tab run on the same 547 questions.

Download: results.json (the comparison's numbers, under english_reading), answers-english.json (each English run's answers, under english_runs), english-reading.json (the translator, its prompts, the flags, the question units and each question's hash of its English text; no question text) and kept-question-ids.txt (the 547 kept questions' IDs, one per line).

Results

The setups of the model group in view, ordered by accuracy in English, highest first. Each row compares the setup's first English run with its Text tab run, out of the 547 kept questions; an invalid answer counts as not right. Where a side has more than one run, its accuracy shows the range over its runs and its runs show how many answers changed between them. The paired counts, p and the Holm-adjusted p (over all 133 English setups, whatever the group shown) are those of the exact McNemar test; * marks a Holm-adjusted p below 0.05. LFM3-1.7B, LFM3-1.7B and MiniCPM5-2B are run with 3 sampling seeds: until all are in, the number is the first English run's, named with the seeds in so far; then it is the mean of the runs' accuracies, ranked by it, with their lowest and highest (min–max, a range, not a confidence interval) under it. The paired comparison with Turkish and the “I don't know” runs stay on their own runs.

ModelEnglish runsthe first is compared; answers changed between runs where more than oneTurkish runsthe Text tab run, then its repeats; answers changed between themTurkish accuracythe Text tab run on the kept questions; range over its runs; higher is betterEnglish accuracythe first English run; range over its runs; higher is betterDifferencepoints higher in English than in TurkishRight only in Englishquestions the English run got right and the Turkish run did notRight only in Turkishquestions the Turkish run got right and the English run did notpexact McNemarHolm padjusted over all 133 English setups
Claude Fable 5.2 max · chat Closed · Flagshiprun 971run 98097.8%535 of 547 right98.0%536 of 547 right+0.25411
Gemini 4 Hydrogen high · chat Closed · Workhorserun 1107run 111497.1%531 of 547 right96.7%529 of 547 right-0.4350.731
GPT-6 Astra low · chat Closed · Flagshiprun 616run 4896.7%529 of 547 right96.3%527 of 547 right-0.4460.751
GPT-6 Astra high · chat Closed · Flagshiprun 615run 27796.9%530 of 547 right96.2%526 of 547 right-0.7590.421
GPT-6.1 Terra high · chat Closed · Workhorserun 620run 27596.3%527 of 547 right96.2%526 of 547 right-0.291011
GPT-6.1 Terra low · chat Closed · Workhorserun 621run 27495.6%523 of 547 right96.2%526 of 547 right+0.513100.681
Claude Fable 5.1 high · chat Closed · Flagshiprun 580run 31095.4%522 of 547 right96.0%525 of 547 right+0.51290.661
Gemini 3.8 Flash high · chat Closed · Workhorserun 583run 14096.9%530 of 547 right96.0%525 of 547 right-0.9380.231
Gemini 4 Argon low · chat Closed · Flagshiprun 593run 55796.9%530 of 547 right96.0%525 of 547 right-0.97120.361
GPT-6 Astra medium · chat Closed · Flagshiprun 624run 61096.7%529 of 547 right96.0%525 of 547 right-0.7480.391
GPT-6.1 Sol low · chat Closed · Workhorserun 619run 12495.8%524 of 547 right95.8%524 of 547 right+0.06611
GPT-6 Astra low · chat · 1 per request Closed · Flagshiprun 622run 19197.1%531 of 547 right95.6%523 of 547 right-1.5080.00780.83
GPT-6.1 Sol high · chat Closed · Workhorserun 618run 27696.2%526 of 547 right95.6%523 of 547 right-0.5580.581
Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorserun 589run 18896.2%526 of 547 right95.2%521 of 547 right-0.97120.361
GPT-6.1 Sol low · chat · 1 per request Closed · Workhorserun 623run 19095.6%523 of 547 right95.2%521 of 547 right-0.4680.791
Claude Opus 5.5 low · chat · 1 per request Closed · Flagshiprun 592run 19295.4%522 of 547 right94.7%518 of 547 right-0.711150.561
Claude Opus 5.5 high · chat Closed · Flagshiprun 587run 27895.6%523 of 547 right94.5%517 of 547 right-1.16120.241
Qwen4-Max high · chat Closed · Workhorserun 1334run 133694.7%518 of 547 right94.5%517 of 547 right-0.2131411
Gemini 3.8 Flash low · chat Closed · Workhorserun 578run 12395.2%521 of 547 right94.3%516 of 547 right-0.910150.421
Gemini 4 Carbon high · chat Closed · Flagshiprun 1055run 110897.6%534 of 547 right94.3%516 of 547 right, 16 invalid-3.33210.000280.031 *
KnowLine-31B-Gen1.5 Open · 31Brun 953run 95794.3%516 of 547 right93.6%512 of 547 right-0.711150.561
Grok 4.7 high · chat Closed · Workhorserun 579run 31195.1%520 of 547 right93.4%511 of 547 right-1.64130.0491
Claude Opus 5.5 low · chat Closed · Flagshiprun 588run 4795.1%520 of 547 right93.1%509 of 547 right-2.012230.091
Copernicus-31B Open · 31Brun 1168run 116594.5%517 of 547 right92.9%508 of 547 right-1.68170.111
Claude Haiku 5.5 max · chat Closed · Efficientrun 949run 94889.4%489 of 547 right, 18 invalid92.7%507 of 547 right+3.330120.00790.83
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B activerun 808run 58594.5%517 of 547 right92.5%506 of 547 right-2.010210.0711
REAX 1.3 Open · 57Brun 829run 82794.0%514 of 547 right92.5%506 of 547 right-1.512200.221
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B activerun 584run 32095.2%521 of 547 right92.3%505 of 547 right-2.95210.00250.28
KnowLine-31B-Gen2 Open · 31Brun 1327run 131494.5%517 of 547 right92.3%505 of 547 right-2.28200.0361
Karpo Decision 31B Open · 31Brun 1159run 115694.9%519 of 547 right92.1%504 of 547 right-2.712270.0241
Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorserun 594run 19394.3%516 of 547 right91.6%501 of 547 right-2.77220.00810.85
Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B activerun 582run 31594.0%514 of 547 right91.4%500 of 547 right-2.67210.0131
Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B activerun 871run 87091.4%500 of 547 right91.4%500 of 547 right+0.0131311
Gemini Decision 1 20261111 Closed · Efficientrun 1009run 102192.9%508 of 547 right91.0%498 of 547 right-1.810200.0991
Logits Sigma-1 Duo Open · 57Brun 1229run 122193.4%511 of 547 right91.0%498 of 547 right-2.417300.0791
GLiDE 27B Open · 27Brun 901run 89991.2%499 of 547 right90.9%497 of 547 right-0.419210.871
d1-ultra Closed · Efficientruns 411, 418, 422, 426, 42828 answers changed between runsruns 199 to 20318 answers changed between runs93.6%512 of 547 rightrange 92.7% to 93.6%90.7%496 of 547 rightrange 90.1% to 91.0%-2.98240.0070.76
Sieve 31B v3 Open · 31Brun 1064run 124193.4%511 of 547 right90.7%496 of 547 right-2.710250.0171
Turion Jev 31B Open · 31Brun 1075run 127393.6%512 of 547 right90.7%496 of 547 right-2.910260.0111
Gemma 4 31B reasoning off · chat Open · 31Brun 774run 11992.5%506 of 547 right90.5%495 of 547 right-2.018290.141
Perplexity Decider 27B v1.2 API · with web Open · 27Brun 639run 64090.7%496 of 547 right90.5%495 of 547 right-0.2202111
GLiDE v1.2 Open · 27Brun 521runs 248, 250 to 2530 answers changed between runs90.5%495 of 547 rightthe same in every run90.3%494 of 547 right-0.2212211
d1.2-ultra Closed · Efficientrun 1359runs 1347, 1353, 1368, 1371, 13730 answers changed between runs90.9%497 of 547 rightthe same in every run90.3%494 of 547 right-0.522250.771
Mammoth-1 Open · 27Brun 503runs 374, 377, 380, 383 and 3840 answers changed between runs92.9%508 of 547 rightthe same in every run90.1%493 of 547 right-2.715300.0361
Gemma 4 31B reasoning off · chat · 1 per request Open · 31Brun 783run 18990.3%494 of 547 right89.9%492 of 547 right-0.424260.891
d3 Open · 26Brun 554runs 259 to 2630 answers changed between runs91.0%498 of 547 rightthe same in every run89.8%491 of 547 right-1.315220.321
Drex Max Open · 27Brun 552runs 322 to 324, 327 and 3280 answers changed between runs91.4%500 of 547 rightthe same in every run89.6%490 of 547 right-1.819290.191
GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B activerun 823run 16090.1%493 of 547 right89.6%490 of 547 right-0.526290.791
Muse Ember max · chat Closed · Efficientrun 942run 93990.9%497 of 547 right89.6%490 of 547 right-1.317240.351
Perplexity Decider 27B v1.3 Open · 27Brun 1137run 113490.7%496 of 547 right89.6%490 of 547 right-1.119250.451
d3-ultra Open · 26Brun 550runs 269 to 2730 answers changed between runs91.4%500 of 547 rightthe same in every run89.4%489 of 547 right-2.022330.181
d1.2 Closed · Efficientrun 1357runs 1346, 1352, 1369 and 1370, 137257 answers changed between runs88.8%486 of 547 rightrange 87.6% to 88.8%89.0%487 of 547 right+0.2313011
MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B activerun 878run 87786.8%475 of 547 right89.0%487 of 547 right+2.237250.161
Qwen3.8-27B low · chat Open · 27Brun 667run 31890.7%496 of 547 right89.0%487 of 547 right-1.616250.211
Perplexity Decider 27B v1.2 DGX Spark Open · 27Brun 527runs 217 to 2210 answers changed between runs91.4%500 of 547 rightthe same in every run88.7%485 of 547 right-2.715300.0361
Drex Max v1.1 Closed · Efficient (provisional)run 399runs 326, 329 to 3320 answers changed between runs90.1%493 of 547 rightthe same in every run88.5%484 of 547 right-1.618270.231
Perplexity Decider 27B v1 Open · 27Bruns 626 to 6303 answers changed between runsruns 44, 64, 84, 96, 1000 answers changed between runs89.0%487 of 547 rightthe same in every run88.5%484 of 547 rightrange 88.5% to 88.7%-0.526290.791
Perplexity Decider 27B v1.2 API Open · 27Brun 637run 63391.0%498 of 547 right88.5%484 of 547 right-2.617310.0591
Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B activerun 867run 86689.4%489 of 547 right88.3%483 of 547 right-1.124300.51
d1.1 Closed · Efficientruns 565, 568, 570, 573, 57543 answers changed between runsruns 254 to 25862 answers changed between runs88.3%483 of 547 rightrange 87.2% to 89.2%88.3%483 of 547 rightrange 87.8% to 88.8%+0.0262611
GLM-5.3 low · chat Open · Workhorserun 577run 31489.2%488 of 547 right87.8%480 of 547 right-1.521290.321
Jeffan-9B-1211 Open · 9.0Brun 1195run 119286.5%473 of 547 right87.6%479 of 547 right+1.133270.521
GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B activerun 820run 15984.3%461 of 547 right87.2%477 of 547 right+2.939230.0561
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B activerun 804run 55889.9%492 of 547 right87.0%476 of 547 right-2.917330.0331
Qwen3.5-4B reasoning on · chat Open · 4.0Brun 1072run 114982.8%453 of 547 right, 2 invalid87.0%476 of 547 right+4.249260.0111
Qwen3.8 Flash no reasoning · chat Closed · Efficientrun 986run 98588.8%486 of 547 right87.0%476 of 547 right-1.822320.221
d1 Closed · Efficientruns 430, 432 to 43550 answers changed between runsruns 52, 66, 94, 105, 10819 answers changed between runs85.0%465 of 547 rightrange 85.0% to 85.7%86.8%475 of 547 rightrange 85.7% to 87.9%+1.836260.251
StartLux-Decision-v1.1-9B-Instant Open · 9.0Brun 1293run 129083.0%454 of 547 right86.8%475 of 547 right+3.847260.0191
Clef Open · 27Brun 613run 12588.1%482 of 547 right86.7%474 of 547 right-1.518260.291
KnowLine-12B-Gen1 Open · 12Brun 1313run 130987.6%479 of 547 right86.7%474 of 547 right-0.920250.551
GLiDE 9B Open · 7.9Brun 909run 90784.3%461 of 547 right86.3%472 of 547 right+2.037260.211
GPT-6 Luna low · chat Closed · Efficientrun 617run 13388.7%485 of 547 right86.3%472 of 547 right-2.426390.141
OneJev-27B Open · 27Brun 778runs 264 to 2680 answers changed between runs87.8%480 of 547 rightthe same in every run86.3%472 of 547 right-1.523310.341
Claude Sonnet 5.5 high · chat Closed · Workhorserun 590run 5587.6%479 of 547 right86.1%471 of 547 right-1.532400.411
Jev 1.14 Closed · Efficientruns 410, 416, 421, 423, 42514 answers changed between runsruns 194 to 19817 answers changed between runs87.2%477 of 547 rightrange 86.5% to 87.2%86.1%471 of 547 rightrange 86.1% to 86.5%-1.117230.431
Cerno 9B v1.2 Open · 9.0Brun 1085run 120183.5%457 of 547 right85.9%470 of 547 right+2.446330.181
Drex 2.0 Open · 35B MoE · 3.0B activeruns 566, 571, 576, 581, 5860 answers changed between runsruns 334 to 3383 answers changed between runs84.8%464 of 547 rightrange 84.6% to 84.8%85.9%470 of 547 rightthe same in every run+1.133270.521
Jev 1.13 Closed · Efficientruns 564, 567, 569, 572, 57420 answers changed between runsruns 49, 65, 85, 97, 10118 answers changed between runs86.1%471 of 547 rightrange 86.1% to 87.2%85.9%470 of 547 rightrange 85.9% to 87.6%-0.2181911
KnowLine-9B-Gen1 Open · 9.0Brun 1177run 117482.4%451 of 547 right85.9%470 of 547 right+3.542230.0251
SubQ Snap 1.1 Closed · Efficientrun 413runs 346, 387 to 3900 answers changed between runs89.2%488 of 547 rightthe same in every run85.9%470 of 547 right-3.323410.0331
SubQ Snap Preview Closed · Efficientruns 595, 597 to 6000 answers changed between runsruns 177 to 1812 answers changed between runs89.9%492 of 547 rightrange 89.9% to 90.3%85.9%470 of 547 rightthe same in every run-4.020420.00710.76
Adjudex 9B v5 Open · 9.0Brun 1186run 118383.9%459 of 547 right85.7%469 of 547 right+1.841310.291
Mercury Decide Closed · Efficientruns 398, 400 to 40320 answers changed between runsruns 204, 210, 212, 214, 21619 answers changed between runs87.0%476 of 547 rightrange 86.7% to 87.4%85.7%469 of 547 rightrange 85.0% to 85.9%-1.328350.451
Microsoft-Decision-1 Closed · Efficientruns 412, 420, 424, 427, 43119 answers changed between runsruns 205, 209, 211, 213, 21536 answers changed between runs76.4%418 of 547 rightrange 76.2% to 77.1%85.7%469 of 547 rightrange 85.0% to 86.1%+9.37423< 0.0001< 0.0001 *
Elephant-1 Open · 9.0Brun 499runs 373, 375 and 376, 378 and 3790 answers changed between runs83.5%457 of 547 rightthe same in every run85.4%467 of 547 right+1.839290.271
Open-Jev-27B-v1.1 Open · 27Brun 760runs 45, 60, 92, 102, 1060 answers changed between runs86.5%473 of 547 rightthe same in every run85.4%467 of 547 right-1.118240.441
KnowLine-9B-Gen2 Open · 9.0Brun 1321run 130682.8%453 of 547 right85.2%466 of 547 right+2.438250.131
Qwen3.8-27B stated confidence · reasoning off · chat Open · 27Brun 672runs 53, 74, 95, 103, 1075 answers changed between runs86.3%472 of 547 rightrange 86.1% to 86.3%84.8%464 of 547 right-1.526340.371
Qwen3.8-27B yes/no scoring Open · 27Bruns 696 to 7001 answer changed between runsruns 50, 75, 89, 99, 1049 answers changed between runs84.6%463 of 547 rightrange 84.6% to 84.8%84.8%464 of 547 rightthe same in every run+0.2232211
MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B activerun 863run 86286.3%472 of 547 right84.5%462 of 547 right, 1 invalid-1.832420.31
celeris-1-decision Closed · Efficientrun 546runs 150 and 151, 156 to 15819 answers changed between runs87.4%478 of 547 rightrange 86.7% to 87.4%84.1%460 of 547 right-3.324420.0361
GPT-6 Luna Decisions API Closed · Efficientrun 397run 14386.7%474 of 547 right, 1 invalid84.1%460 of 547 right, 2 invalid-2.628420.121
Claude Sonnet 5.5 low · chat Closed · Workhorserun 591run 5484.8%464 of 547 right83.9%459 of 547 right-0.927320.61
Cygnet Open · 12Brun 753run 12785.6%468 of 547 right83.9%459 of 547 right-1.626350.311
Qwen3.8-27B vote share · reasoning off · chat Open · 27Brun 678run 8385.9%470 of 547 right83.9%459 of 547 right-2.028390.221
Winnow-12B Open · 12Brun 746run 12885.0%465 of 547 right83.9%459 of 547 right-1.134400.561
Perplexity Decider Flash v1.3 Open · 4.0Brun 1126run 112378.8%431 of 547 right83.7%458 of 547 right+4.960330.00670.73
Valen-Decision-1.5-9B Open · 9.0Brun 1278run 127579.0%432 of 547 right83.7%458 of 547 right+4.861350.011
Dexy-Decision-9B-V2 Open · 9.0Brun 1258run 125583.4%456 of 547 right83.5%457 of 547 right+0.2373611
vera-medium-omni Open · 4.2Brun 1267run 126479.5%435 of 547 right83.5%457 of 547 right+4.052300.021
StartLux-Decision-v1.1-4B-Instant Open · 4.0Brun 1081run 123678.4%429 of 547 right82.8%453 of 547 right+4.457330.0151
Gemma 4 E4B default reasoning · chat Open · 8.0Brun 889run 104080.6%441 of 547 right, 2 invalid82.4%451 of 547 right, 2 invalid+1.849390.341
Clef-Flash Open · 9.0Brun 614run 13977.7%425 of 547 right82.3%450 of 547 right+4.655300.00880.91
Verdin-4B-v1.1 Open · 4.0Brun 1247run 124477.5%424 of 547 right82.3%450 of 547 right+4.860340.00950.97
Perplexity Decider Flash v1.2 Open · 4.0Brun 533runs 223 to 2270 answers changed between runs76.6%419 of 547 rightthe same in every run81.9%448 of 547 right+5.363340.00420.46
KnowLine 4B Gen5.5 Open · 4.0Brun 961run 96578.6%430 of 547 right80.8%442 of 547 right+2.251390.251
Chinese Decision Pointer 4B Open · 4.0Brun 1209run 120675.7%414 of 547 right80.4%440 of 547 right+4.863370.0121
Qwen3.5-4B reasoning off · chat Open · 4.0Brun 1143run 114672.0%394 of 547 right, 4 invalid80.4%440 of 547 right, 1 invalid+8.495490.000160.018 *
Qwen3-14B reasoning off · chat Open · 15Brun 770run 13176.8%420 of 547 right79.7%436 of 547 right+2.956400.131
Metask-Jev-4B Open · 4.5Brun 739run 12667.1%367 of 547 right77.1%422 of 547 right+10.19035< 0.00010.00011 *
LFM2.5-2.6B default reasoning · chat Open · 2.7Brun 1034run 103747.9%262 of 547 right, 14 invalid73.9%404 of 547 right+26.017533< 0.0001< 0.0001 *
Erk-14B reasoning off · chat Open · 14Brun 762run 13275.3%412 of 547 right72.0%394 of 547 right-3.348660.111
LFM3-1.7B reasoning off · chat Open · 1.7Brun 1223run 122740.2%220 of 547 right, 1 invalid72.9%399 of 547 righthistorical paired baseline, run 1223+32.722142< 0.0001< 0.0001 *
KnowLine-2B-Gen1 Open · 2.0Brun 1057run 107958.0%317 of 547 right71.3%390 of 547 right+13.311239< 0.0001< 0.0001 *
LFM3-1.7B reasoning on · chat Open · 1.7Brun 1061run 108948.1%263 of 547 right71.1%389 of 547 righthistorical paired baseline, run 1061+23.016640< 0.0001< 0.0001 *
LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B activerun 886run 128827.1%148 of 547 right, 87 invalid70.4%385 of 547 right, 3 invalid+43.326427< 0.0001< 0.0001 *
GLiDE 2B Open · 1.9Brun 917run 91554.3%297 of 547 right70.0%383 of 547 right+15.713549< 0.0001< 0.0001 *
StartLux-Decision-v1.1-2B-Instant Open · 2.0Brun 973run 97755.8%305 of 547 right69.7%381 of 547 right+13.912953< 0.0001< 0.0001 *
MiniCPM5-2B reasoning off · chat Open · 2.5Brun 1094run 12997.1%39 of 547 right, 417 invalid67.6%370 of 547 right, 8 invalidhistorical paired baseline, run 1094+60.53376< 0.0001< 0.0001 *
Qwen3.5-2B reasoning off · chat Open · 2.0Brun 1215run 121836.7%201 of 547 right, 34 invalid67.3%368 of 547 right, 2 invalid+30.520740< 0.0001< 0.0001 *
d1-2B Open · 1.9Brun 708runs 340 to 3440 answers changed between runs41.9%229 of 547 rightthe same in every run65.6%359 of 547 right+23.817141< 0.0001< 0.0001 *
GLiDE 0.8B Open · 0.8Brun 925run 92346.1%252 of 547 right60.7%332 of 547 right+14.614161< 0.0001< 0.0001 *
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8Brun 1011run 101537.3%204 of 547 right59.6%326 of 547 right+22.316139< 0.0001< 0.0001 *
GLiDE 0.4B Open · 0.4Brun 933run 93125.8%141 of 547 right59.4%325 of 547 right+33.622238< 0.0001< 0.0001 *
OneJev1.6-0.8B Open · 0.8Brun 1025run 102942.8%234 of 547 right59.4%325 of 547 right+16.614857< 0.0001< 0.0001 *
KnowLine-0.8B-Gen1 Open · 0.8Brun 1001run 100540.2%220 of 547 right58.9%322 of 547 right+18.615048< 0.0001< 0.0001 *
GLiClass v4 Edge Open · 0.4Brun 1042run 104626.9%147 of 547 right58.3%319 of 547 right+31.420634< 0.0001< 0.0001 *
Decider 1 Closed · Efficientrun 596runs 243 to 2470 answers changed between runs36.6%200 of 547 rightthe same in every run55.8%305 of 547 right+19.215247< 0.0001< 0.0001 *
Strands Decider 2B Open · 2.0Brun 702run 14139.1%214 of 547 right45.5%249 of 547 right+6.474390.00130.14
Laya Open · 0.4Brun 726runs 51, 234 to 2370 answers changed between runs20.5%112 of 547 rightthe same in every run22.9%125 of 547 right+2.486730.341
GLiNER2.5-multi-Decide Open · 0.3Brun 714run 13520.7%113 of 547 right21.2%116 of 547 right+0.542390.821
GLiNER2.5 Multi Open · 0.3Brun 720run 12216.8%92 of 547 right16.3%89 of 547 right-0.530330.81
Laya Multilingual Open · 0.3Brun 732runs 46, 238 to 2410 answers changed between runs17.0%93 of 547 rightthe same in every run15.4%84 of 547 right-1.640490.41

Without an English run

Four of the Text tab's 137 setups have no English run yet, each with its reason:

  • Perplexity Decider 27B v1.1: Perplexity's API now answers requests for v1.1 with v1.2, so a new run could not measure v1.1.
  • GLiDE API: since 10 October, our requests for fastino/GLiDE have been answered by Fastino's glide_v2 engine, not the glide engine of its Turkish run (6 October), so an English run would not measure the same model.
  • DeepSeek-V4.1-Flash reasoning on · chat · first release and DeepSeek-V4.1-Flash reasoning off · chat · first release: their first-release weights are no longer on our disk (the server folder now holds the 1017 weights), so a run needs them downloaded again first.

Our method

The translation

The translator is Claude Opus 5.5 (claude-opus-5-5) at effort high; route: Claude Code CLI, Claude subscription, Claude Code version 2.1.312. Its maker has no decision model on our board. Of its maker's chat models, Claude Fable 5.2, Claude Fable 5.1, Claude Opus 5.5 and Claude Sonnet 5.5 answered the English questions too, the translator's own model among them. It was told to translate faithfully, keep the option letters and their order, not solve the question and not add or remove information (translation prompt version 2976d0f96197).

The flags

In a separate request, the same model was shown each Turkish question, its English translation and the keyed option, and asked whether answering it correctly depends on Turkish wording, with the reasons (flag prompt version 08a5c70222aa). It flagged 109 of the 656 questions; they are left out, and 547 are kept. By reason, as a flagged question can have several:

  • word or phrase meaning: 76
  • idiom: 44
  • sound: 5
  • marked words: 55
  • other: 13

No person checked the translations or the flags, and this is a limit of the method: a translation error, or a question whose answer still depends on Turkish wording that the model did not flag, would stay among the 547 kept questions and could move a setup's accuracy in English. Both prompts in full are in english-reading.json and runner/translate_reading.py.

The structural check

A script (runner/translate_reading.py, its check step) then compared each translation with its Turkish original: a problem is a missing translation, option letters other than the original's, an empty text or a different count of underline markup; a warning is a change in the numbering or the numbers, Turkish letters in the English, or an English text under half or over 2.5 times the Turkish length. It found problems in 1 question (3043), already left out by the flags, so it left out no further question; its 159 warnings leave out nothing. In all, the checks left out 109 of the 656 questions and kept 547: their IDs are in kept-question-ids.txt, one per line, and in english-reading.json.

Why the grammar questions are not translated

The Text tab's 1,542 grammar questions test Turkish grammar itself, so a translation would test something else, not the same question in another language. Only the 656 reading questions were translated.

Asking the decision models

Each decision model is asked exactly as on the Text tab, with its own settings, one question per request, and with its option probabilities, except for one word of the System One instructions, the options heading and the question itself. The Text tab's instructions:

Which option is the correct answer to this Turkish multiple-choice question?

The English runs' instructions:

Which option is the correct answer to this English multiple-choice question?

The options follow the heading “Options:” instead of “Seçenekler:”, and the question and its options are the English translation. Prompt version 47f5776e71db; every English run used the translations with SHA-256 bf0d47cd385a93050c65906977e199ef7e45d5848dbbdef2bc8d124f4d5abf68. d1 was asked as d1, while its Turkish runs used d1:free, because Liquid's API no longer serves that name; its model list still names d1, released 29 September 2026. Perplexity Decider 27B v1.2 (its setup tagged DGX Spark), Perplexity Decider Flash v1.2 and GLiDE · v1.2 made their Text tab runs on an NVIDIA DGX Spark and their English runs on our RTX PRO 6000 GPU, with the same checkpoint, server code and settings. celeris-1-decision was added on 3 November 2026 with one English run, although its repeated Turkish runs changed answers, for which the rule under Runs gives five. SubQ Snap Preview's English runs (5 November 2026) returned the same choices and probabilities as SubQ Snap 1.1's English run (2 November 2026) on all 547 questions, and every response named the Preview model. Its Turkish run (13 October 2026) and SubQ Snap 1.1's (1 November 2026) differ on 519 of 2,198 questions. The model behind each name was not verified; read SubQ Snap Preview's Turkish against English comparison with this in mind. LFM2.5-2.6B, LFM2.5-8B-A1B and LFM2.5-24B-A2B (at the revisions served, 7141b56, dd83f5c and 1769516): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored (Zeki #8202).

Asking the chat models

Each chat model setup not asked for a plain answer line (below) is asked as its Text tab run was, with the same settings (reasoning, and 16 questions per request for 38 and 1 question per request for 11), and with the Text tab's chat prompts in English: the system prompt and the instruction that ask for each answer's letter with a confidence score between 0 and 1, the question header and the options heading “Options:”. A chat model gives no option probabilities, so its confidence score is the one it stated. The English runs' system prompt:

You are taking an English multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.

Their instruction:

Answer each of the following multiple-choice questions. Without explanation, return for each question the letter of the correct option together with a confidence score between 0 and 1 showing how sure you are that your answer is correct, in the specified JSON format.

Chat prompt version 5c3a4849752e; the Text tab's Turkish prompts are under The instructions of each kind of run. The chat models' runs with an “I don't know” option are under Our method with an “I don't know” option.

Asking the chat models for a plain answer line

Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat and Qwen3.5-2B · reasoning off · chat are asked in English as their Text tab runs were in Turkish: one question per request on logic's server with only the request settings of logic's own runs of the same model (How we called it, on the model page), a user message with one instruction, word for word that of logic's runs, a blank line and the question as the other runs in its language show it; no system prompt and no answer schema. The answer is the letter on the reply's last “ANSWER: X” line in English, “CEVAP: X” in Turkish. They state no confidence and report no option probabilities, so their confidence is not measured. The instruction in English ({letters} is the question's own letters, as “A, B, C, D or E”):

Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = {letters}).

In Turkish:

Aşağıdaki çoktan seçmeli soruyu çöz. Yanıtının son satırına yalnızca 'CEVAP: X' yaz (X = {letters}).

Plain prompt versions ad6f4cc346df (English) and ba455bb00e5c (Turkish).

Runs

One English run per setup; five where the setup's whole-bank Turkish repeats on the Text tab changed answers. Here: five runs for d1-ultra, Perplexity Decider 27B v1, d1.1, d1, Jev 1.14, Drex 2.0, Jev 1.13, SubQ Snap Preview, Mercury Decide, Microsoft-Decision-1 and Qwen3.8-27B · yes/no scoring; one for the other 122. The English runs are kept out of the leaderboard and its repeats.

The comparison

On the 547 kept questions only: accuracy is the right answers over 547 for the first English run and for the Text tab run (an invalid answer counts as not right), and the difference is how many points higher the English run scored than the Turkish one (negative where Turkish is higher). The paired test is the exact McNemar test on the questions only one of the two runs got right, with p adjusted by Holm's method over all 133 English setups, decision and chat alike, whatever the group shown. Where a side has more than one run, its range is the lowest and highest accuracy over its runs, and its answers changed are the kept questions whose answer differs between them.

Why the translations are not published

The site never publishes question text, Turkish or English: the publisher owns the questions. english-reading.json holds instead each reading question's ID, unit, whether it was kept, its flag reasons and the SHA-256 of its English text, with the translator and both prompts, so that a copy of these translations can be checked against it. Translating again with runner/translate_reading.py need not give the same text word for word.

The instructions of each kind of run

Each decision model run's System One instructions in full, and each chat model run's system prompt and instruction (a run asked for a plain answer line has one instruction and no system prompt; its {letters} is the question's own letters), as results.json and answers.json give them; a chat model's run with the option keeps the system prompt of its five-option run. The question and its options follow in the run's language; their text is not published.

Decision models, five options, in Turkish (the Text tab)

Which option is the correct answer to this Turkish multiple-choice question?

Decision models, five options, in English

Which option is the correct answer to this English multiple-choice question?

Chat models, in Turkish (the Text tab): the system prompt

You are taking a Turkish multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.

Chat models, in Turkish (the Text tab): the instruction

Aşağıdaki çoktan seçmeli soruların her birini yanıtlayın. Açıklama yapmadan, her soru için doğru seçeneğin harfini ve cevabınızın doğru olduğundan ne kadar emin olduğunuzu gösteren 0 ile 1 arasında bir güven puanını belirtilen JSON biçiminde birlikte döndürün.

Chat models, in English (prompt version 5c3a4849752e): the system prompt

You are taking an English multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.

Chat models, in English: the instruction

Answer each of the following multiple-choice questions. Without explanation, return for each question the letter of the correct option together with a confidence score between 0 and 1 showing how sure you are that your answer is correct, in the specified JSON format.

Chat models asked for a plain answer line, in Turkish (the Text tab) (prompt version ba455bb00e5c): the instruction

Aşağıdaki çoktan seçmeli soruyu çöz. Yanıtının son satırına yalnızca 'CEVAP: X' yaz (X = {letters}).

Chat models asked for a plain answer line, in English (prompt version ad6f4cc346df): the instruction

Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = {letters}).

With the option, in Turkish (“Bilmiyorum”, the Hallucination tab)

Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose Bilmiyorum.

With the option, in English (“I don't know”)

Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.

The option's word: the Turkish questions with “I don't know”

Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose I don't know.

Chat models with the option, in Turkish (prompt version d3c64e8c862e): the instruction

Aşağıdaki çoktan seçmeli soruların her birini yanıtlayın. Açıklama yapmadan, her soru için doğru seçeneğin harfini ve cevabınızın doğru olduğundan ne kadar emin olduğunuzu gösteren 0 ile 1 arasında bir güven puanını belirtilen JSON biçiminde birlikte döndürün. Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.

Chat models with the option, in English (prompt version e53d4aa27643): the instruction

Answer each of the following multiple-choice questions. Without explanation, return for each question the letter of the correct option together with a confidence score between 0 and 1 showing how sure you are that your answer is correct, in the specified JSON format. If you do not know the answer, choose I don't know.

Chat models asked for a plain answer line with the option, in Turkish (prompt version 711479895d19): the instruction

Aşağıdaki çoktan seçmeli soruyu çöz. Yanıtının son satırına yalnızca 'CEVAP: X' yaz (X = {letters}). Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.

Chat models asked for a plain answer line with the option, in English (prompt version 4cd69035a45e): the instruction

Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = {letters}). If you do not know the answer, choose I don't know.

Results with an “I don't know” option

Ordered by the non-hallucination rate in English, highest first. Each row compares the model's first run with the English option with its first run with the Turkish option, out of the 547 kept questions. Where a side has more than one run, its rates show the range over its runs. The paired counts, p and the Holm-adjusted p (over the 67 decision models and the 56 chat models, each group a family of its own) are those of the exact McNemar test on wrong answers; * marks a Holm-adjusted p below 0.05.

ModelEnglish non-hallucinationof answers not right, % “I don't know” · higher is better, read with the answered shareEnglish answerednot “I don't know”: (right + wrong) / 547English error rateof its answers: wrong / (right + wrong) · lower is betterTurkish non-hallucinationof answers not right, % the Turkish option · higher is better, read with the answered shareTurkish answerednot the Turkish option: (right + wrong) / 547Turkish error rateof its answers: wrong / (right + wrong) · lower is betterEnglish runswith “I don't know”; the first is comparedTurkish runswith the Turkish option; the first is comparedEnglish accuracyhigher is betterTurkish accuracyhigher is betterEnglish mean probabilityof “I don't know”Turkish mean probabilityof the Turkish optionWrong only in Englishquestions the English run got wrong and the Turkish run did notWrong only in Turkishquestions the Turkish run got wrong and the English run did notpexact McNemarHolm padjusted over the 67 decision models; over the 56 chat models
d1.2-ultra Closed · Efficient89.3%109 “I don't know” of 122 not right80.1%438 of 547 answered, 109 “I don't know”3.0%13 of 43876.2%61 “I don't know” of 80 not right88.8%486 of 547 answered, 61 “I don't know”3.9%19 of 486run 1363a new runrun 1367the Hallucination tab's full runs, counted on the kept questions77.7%425 of 547 right85.4%467 of 547 right26.0%17.6%6120.241
celeris-1.1-decision Closed · Efficient74.8%98 “I don't know” of 131 not right82.1%449 of 547 answered, 98 “I don't know”7.3%33 of 449Not runNot runNot runrun 884a new runNot run76.1%416 of 547 rightNot run13.3%Not runNot runNot runNot runNot run
celeris-1-decision Closed · Efficient71.3%92 “I don't know” of 129 not right83.2%455 of 547 answered, 92 “I don't know”8.1%37 of 455Not runNot runNot runrun 548a new runNot run76.4%418 of 547 rightNot run12.7%Not runNot runNot runNot runNot run
d1.2 Closed · Efficient61.5%59 “I don't know” of 96 not right89.2%488 of 547 answered, 59 “I don't know”7.6%37 of 48860.2%53 “I don't know” of 88 not right90.3%494 of 547 answered, 53 “I don't know”7.1%35 of 494run 1361a new runrun 1365the Hallucination tab's full runs, counted on the kept questions82.4%451 of 547 right83.9%459 of 547 right19.4%17.1%21190.871
d1-2B Open · 1.9B60.9%151 “I don't know” of 248 not right72.4%396 of 547 answered, 151 “I don't know”24.5%97 of 39643.8%161 “I don't know” of 368 not right70.6%386 of 547 answered, 161 “I don't know”53.6%207 of 386run 710a new runrun 712the Hallucination tab's full runs, counted on the kept questions54.7%299 of 547 right32.7%179 of 547 right26.9%23.7%38148< 0.0001< 0.0001 *
d1-ultra Closed · Efficient56.7%51 “I don't know” of 90 not rightrange 55.3% to 61.3%90.7%496 of 547 answered, 51 “I don't know”range 89.6% to 91.4%7.9%39 of 49653.6%30 “I don't know” of 56 not rightrange 44.4% to 59.3%94.5%517 of 547 answered, 30 “I don't know”range 94.1% to 95.6%5.0%26 of 517runs 449, 456, 462, 466, 474new runsruns 350, 354, 360, 365, 370the Hallucination tab's full runs, counted on the kept questions83.5%457 of 547 rightrange 82.6% to 84.5%89.8%491 of 547 rightrange 89.2% to 90.1%25.3%range 25.3% to 25.6%22.8%range 22.4% to 23.0%2070.0190.88
GLiDE 0.4B Open · 0.4B53.1%153 “I don't know” of 288 not right72.0%394 of 547 answered, 153 “I don't know”34.3%135 of 39489.6%475 “I don't know” of 530 not right13.2%72 of 547 answered, 475 “I don't know”76.4%55 of 72run 935a new runrun 937the Hallucination tab's full runs, counted on the kept questions47.3%259 of 547 right3.1%17 of 547 right23.3%35.2%12545< 0.0001< 0.0001 *
Drex Max v1.1 Closed · Efficient (provisional)51.2%43 “I don't know” of 84 not right92.1%504 of 547 answered, 43 “I don't know”8.1%41 of 50415.0%9 “I don't know” of 60 not right98.4%538 of 547 answered, 9 “I don't know”9.5%51 of 538run 447a new runrun 352the Hallucination tab's full runs, counted on the kept questions84.6%463 of 547 right89.0%487 of 547 right14.3%8.4%20300.21
Mammoth-1 Open · 27B50.7%37 “I don't know” of 73 not right93.2%510 of 547 answered, 37 “I don't know”7.1%36 of 51042.1%24 “I don't know” of 57 not right95.6%523 of 547 answered, 24 “I don't know”6.3%33 of 523run 505a new runrun 386the Hallucination tab's full runs, counted on the kept questions86.7%474 of 547 right89.6%490 of 547 right9.8%7.4%18150.731
GLiDE v1.2 Open · 27B45.5%30 “I don't know” of 66 not right94.5%517 of 547 answered, 30 “I don't know”7.0%36 of 51711.3%6 “I don't know” of 53 not right98.9%541 of 547 answered, 6 “I don't know”8.7%47 of 541run 523a new runrun 369the Hallucination tab's full runs, counted on the kept questions87.9%481 of 547 right90.3%494 of 547 right12.5%8.7%16270.131
Laya Multilingual Open · 0.3B42.6%208 “I don't know” of 488 not right62.0%339 of 547 answered, 208 “I don't know”82.6%280 of 33925.7%124 “I don't know” of 483 not right77.3%423 of 547 answered, 124 “I don't know”84.9%359 of 423run 734a new runrun 737the Hallucination tab's full runs, counted on the kept questions10.8%59 of 547 right11.7%64 of 547 right34.7%21.6%51130< 0.0001< 0.0001 *
GLiDE 27B Open · 27B37.9%22 “I don't know” of 58 not right96.0%525 of 547 answered, 22 “I don't know”6.9%36 of 52518.8%9 “I don't know” of 48 not right98.4%538 of 547 answered, 9 “I don't know”7.2%39 of 538run 903a new runrun 905the Hallucination tab's full runs, counted on the kept questions89.4%489 of 547 right91.2%499 of 547 right9.3%6.5%15180.731
Claude Fable 5.2 max · chat Closed · Flagship28.6%4 “I don't know” of 14 not right99.3%543 of 547 answered, 4 “I don't know”1.8%10 of 543RunningRunningRunningrun 1032a new runRunning97.4%533 of 547 rightRunningNo probabilitiesa chat model states a confidenceRunningRunningRunningRunningRunning
OneJev1.6-0.8B Open · 0.8B28.4%74 “I don't know” of 261 not right86.5%473 of 547 answered, 74 “I don't know”39.5%187 of 47330.7%112 “I don't know” of 365 not right79.5%435 of 547 answered, 112 “I don't know”58.2%253 of 435run 1027a new runrun 1031the Hallucination tab's full runs, counted on the kept questions52.3%286 of 547 right33.3%182 of 547 right10.7%15.6%55121< 0.0001< 0.0001 *
Grok 4.7 high · chat Closed · Workhorse27.9%12 “I don't know” of 43 not right97.8%535 of 547 answered, 12 “I don't know”5.8%31 of 53527.6%8 “I don't know” of 29 not right98.5%539 of 547 answered, 8 “I don't know”3.9%21 of 539run 648a new runrun 675the Hallucination tab's full runs, counted on the kept questions92.1%504 of 547 right94.7%518 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence1880.0761
LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active26.4%51 “I don't know” of 193 not right90.5%495 of 547 answered, 51 “I don't know”, 1 invalid28.7%142 of 49550.8%201 “I don't know” of 396 not right52.5%287 of 547 answered, 201 “I don't know”, 59 invalid67.9%195 of 287run 887a new runrun 1298the Hallucination tab's full runs, counted on the kept questions64.5%353 of 547 right16.8%92 of 547 rightNot measureda plain answer line, no confidenceNot measureda plain answer line, no confidence801330.000340.018 *
GLiDE 2B Open · 1.9B26.3%47 “I don't know” of 179 not right91.4%500 of 547 answered, 47 “I don't know”26.4%132 of 50024.0%66 “I don't know” of 275 not right87.9%481 of 547 answered, 66 “I don't know”43.5%209 of 481run 919a new runrun 921the Hallucination tab's full runs, counted on the kept questions67.3%368 of 547 right49.7%272 of 547 right12.5%15.4%48125< 0.0001< 0.0001 *
Raon-1 nano reasoning off · chat Open · 3.1B26.2%71 “I don't know” of 271 not right86.8%475 of 547 answered, 71 “I don't know”, 1 invalid42.1%200 of 475Not runNot runNot runrun 1153a new runNot run50.3%275 of 547 rightNot runNot measureda plain answer line, no confidenceNot runNot runNot runNot runNot run
Elephant-1 Open · 9.0B25.6%23 “I don't know” of 90 not right95.8%524 of 547 answered, 23 “I don't know”12.8%67 of 52415.5%15 “I don't know” of 97 not right97.3%532 of 547 answered, 15 “I don't know”15.4%82 of 532run 501a new runrun 382the Hallucination tab's full runs, counted on the kept questions83.5%457 of 547 right82.3%450 of 547 right6.8%5.7%31460.111
Gemini 4 Carbon high · chat Closed · Flagship25.0%4 “I don't know” of 16 not right99.3%543 of 547 answered, 4 “I don't know”2.2%12 of 54326.7%4 “I don't know” of 15 not right99.3%543 of 547 answered, 4 “I don't know”2.0%11 of 543run 1067a new runrun 1121the Hallucination tab's full runs, counted on the kept questions97.1%531 of 547 right97.3%532 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence3211
SubQ Snap 1.1 Closed · Efficient25.0%21 “I don't know” of 84 not right96.2%526 of 547 answered, 21 “I don't know”12.0%63 of 52627.1%19 “I don't know” of 70 not right96.5%528 of 547 answered, 19 “I don't know”9.7%51 of 528run 448a new runrun 392the Hallucination tab's full runs, counted on the kept questions84.6%463 of 547 right87.2%477 of 547 right11.7%8.4%39270.181
KnowLine-2B-Gen1 Open · 2.0B22.0%38 “I don't know” of 173 not right93.1%509 of 547 answered, 38 “I don't know”26.5%135 of 509Not runNot runNot runrun 1059a new runNot run68.4%374 of 547 rightNot run7.3%Not runNot runNot runNot runNot run
GLiDE 0.8B Open · 0.8B21.7%49 “I don't know” of 226 not right91.0%498 of 547 answered, 49 “I don't know”35.5%177 of 49824.3%74 “I don't know” of 304 not right86.5%473 of 547 answered, 74 “I don't know”48.6%230 of 473run 927a new runrun 929the Hallucination tab's full runs, counted on the kept questions58.7%321 of 547 right44.4%243 of 547 right14.8%15.4%681210.000140.0079 *
GLiClass v4 Edge Open · 0.4B21.0%52 “I don't know” of 248 not right90.5%495 of 547 answered, 52 “I don't know”39.6%196 of 49514.1%58 “I don't know” of 410 not right89.4%489 of 547 answered, 58 “I don't know”72.0%352 of 489run 1044a new runrun 1048the Hallucination tab's full runs, counted on the kept questions54.7%299 of 547 right25.0%137 of 547 right11.8%12.0%48204< 0.0001< 0.0001 *
Gemini 4 Hydrogen high · chat Closed · Workhorse20.0%4 “I don't know” of 20 not right99.3%543 of 547 answered, 4 “I don't know”2.9%16 of 54328.6%4 “I don't know” of 14 not right99.3%543 of 547 answered, 4 “I don't know”1.8%10 of 543run 1109a new runrun 1117the Hallucination tab's full runs, counted on the kept questions96.3%527 of 547 right97.4%533 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence710.071
MiniCPM5-2B reasoning on · chat Open · 2.5B19.7%28 “I don't know” of 142 not right94.1%515 of 547 answered, 28 “I don't know”, 4 invalid22.1%114 of 515Not runNot runNot runrun 1052a new runNot run73.3%401 of 547 rightNot runNot measureda plain answer line, no confidenceNot runNot runNot runNot runNot run
GLiNER2.5 Multi Open · 0.3B19.6%92 “I don't know” of 470 not right83.2%455 of 547 answered, 92 “I don't know”83.1%378 of 4556.1%28 “I don't know” of 456 not right94.9%519 of 547 answered, 28 “I don't know”82.5%428 of 519run 722a new runrun 724the Hallucination tab's full runs, counted on the kept questions14.1%77 of 547 right16.6%91 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence4595< 0.00010.0015 *
GPT-6 Astra high · chat Closed · Flagship19.0%4 “I don't know” of 21 not right99.3%543 of 547 answered, 4 “I don't know”3.1%17 of 54319.0%4 “I don't know” of 21 not right99.3%543 of 547 answered, 4 “I don't know”3.1%17 of 543run 814a new runrun 818the Hallucination tab's full runs, counted on the kept questions96.2%526 of 547 right96.2%526 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence6611
Gemini 4 Argon low · chat Closed · Flagship18.5%5 “I don't know” of 27 not right99.1%542 of 547 answered, 5 “I don't know”4.1%22 of 54223.5%4 “I don't know” of 17 not right99.3%543 of 547 answered, 4 “I don't know”2.4%13 of 543run 652a new runrun 679the Hallucination tab's full runs, counted on the kept questions95.1%520 of 547 right96.9%530 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence1560.0781
Laya Open · 0.4B18.4%83 “I don't know” of 450 not right84.8%464 of 547 answered, 83 “I don't know”79.1%367 of 46412.0%53 “I don't know” of 443 not right90.3%494 of 547 answered, 53 “I don't know”78.9%390 of 494run 728a new runrun 730the Hallucination tab's full runs, counted on the kept questions17.7%97 of 547 right19.0%104 of 547 right14.2%11.1%971200.141
GPT-6 Astra low · chat Closed · Flagship18.2%4 “I don't know” of 22 not right99.3%543 of 547 answered, 4 “I don't know”3.3%18 of 54322.2%4 “I don't know” of 18 not right99.3%543 of 547 answered, 4 “I don't know”2.6%14 of 543run 647a new runrun 816the Hallucination tab's full runs, counted on the kept questions96.0%525 of 547 right96.7%529 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence840.391
Open-Jev-27B-v1.1 Open · 27B17.4%16 “I don't know” of 92 not right97.1%531 of 547 answered, 16 “I don't know”14.3%76 of 5317.1%6 “I don't know” of 84 not right98.9%541 of 547 answered, 6 “I don't know”14.4%78 of 541run 763a new runrun 768the Hallucination tab's full runs, counted on the kept questions83.2%455 of 547 right84.6%463 of 547 right15.2%13.2%26280.891
StartLux-Decision-v1.1-2B-Instant Open · 2.0B16.9%32 “I don't know” of 189 not right94.1%515 of 547 answered, 32 “I don't know”30.5%157 of 51512.3%33 “I don't know” of 268 not right94.0%514 of 547 answered, 33 “I don't know”45.7%235 of 514run 975a new runrun 979the Hallucination tab's full runs, counted on the kept questions65.4%358 of 547 right51.0%279 of 547 right6.7%7.6%47125< 0.0001< 0.0001 *
MiniCPM5-2B reasoning off · chat Open · 2.5B16.8%33 “I don't know” of 197 not right89.2%488 of 547 answered, 33 “I don't know”, 26 invalid33.6%164 of 48822.7%22 “I don't know” of 97 not right20.8%114 of 547 answered, 22 “I don't know”, 411 invalid65.8%75 of 114run 1095a new runrun 1304the Hallucination tab's full runs, counted on the kept questions59.2%324 of 547 right7.1%39 of 547 rightNot measureda plain answer line, no confidenceNot measureda plain answer line, no confidence12637< 0.0001< 0.0001 *
GLiDE 9B Open · 7.9B16.7%14 “I don't know” of 84 not right97.4%533 of 547 answered, 14 “I don't know”13.1%70 of 5339.8%9 “I don't know” of 92 not right98.4%538 of 547 answered, 9 “I don't know”15.4%83 of 538run 911a new runrun 913the Hallucination tab's full runs, counted on the kept questions84.6%463 of 547 right83.2%455 of 547 right7.8%6.9%25380.131
GPT-6 Astra medium · chat Closed · Flagship16.7%4 “I don't know” of 24 not right99.3%543 of 547 answered, 4 “I don't know”3.7%20 of 54320.0%4 “I don't know” of 20 not right99.3%543 of 547 answered, 4 “I don't know”2.9%16 of 543run 813a new runrun 817the Hallucination tab's full runs, counted on the kept questions95.6%523 of 547 right96.3%527 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence840.391
GPT-6.1 Sol high · chat Closed · Workhorse16.7%4 “I don't know” of 24 not right99.3%543 of 547 answered, 4 “I don't know”3.7%20 of 54320.0%4 “I don't know” of 20 not right99.3%543 of 547 answered, 4 “I don't know”2.9%16 of 543run 758a new runrun 802the Hallucination tab's full runs, counted on the kept questions95.6%523 of 547 right96.3%527 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence950.421
GPT-6.1 Terra high · chat Closed · Workhorse16.7%4 “I don't know” of 24 not right99.3%543 of 547 answered, 4 “I don't know”3.7%20 of 54320.8%5 “I don't know” of 24 not right99.1%542 of 547 answered, 5 “I don't know”3.5%19 of 542run 766a new runrun 807the Hallucination tab's full runs, counted on the kept questions95.6%523 of 547 right95.6%523 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence8711
GLiNER2.5-multi-Decide Open · 0.3B16.2%70 “I don't know” of 433 not right87.2%477 of 547 answered, 70 “I don't know”76.1%363 of 4775.5%24 “I don't know” of 436 not right95.6%523 of 547 answered, 24 “I don't know”78.8%412 of 523run 716a new runrun 718the Hallucination tab's full runs, counted on the kept questions20.8%114 of 547 right20.3%111 of 547 right13.2%10.4%4695< 0.00010.0026 *
Granite-4.2-3B reasoning on · chat Open · 3.0B16.1%23 “I don't know” of 143 not right95.8%524 of 547 answered, 23 “I don't know”22.9%120 of 524Not runNot runNot runrun 1110a new runNot run73.9%404 of 547 rightNot runNot measureda plain answer line, no confidenceNot runNot runNot runNot runNot run
LFM2.5-2.6B default reasoning · chat Open · 2.7B15.7%21 “I don't know” of 134 not right95.6%523 of 547 answered, 21 “I don't know”, 3 invalid21.6%113 of 52315.8%43 “I don't know” of 272 not right90.7%496 of 547 answered, 43 “I don't know”, 8 invalid46.2%229 of 496run 1035a new runrun 1038the Hallucination tab's full runs, counted on the kept questions75.0%410 of 547 right48.8%267 of 547 rightNot measureda plain answer line, no confidenceNot measureda plain answer line, no confidence44160< 0.0001< 0.0001 *
LFM3-1.7B reasoning on · chat Open · 1.7B15.6%26 “I don't know” of 167 not right95.2%521 of 547 answered, 26 “I don't know”27.1%141 of 52133.5%106 “I don't know” of 316 not right80.6%441 of 547 answered, 106 “I don't know”47.6%210 of 441run 1062a new runrun 1090the Hallucination tab's full runs, counted on the kept questions69.5%380 of 547 right42.2%231 of 547 rightNot measureda plain answer line, no confidenceNot measureda plain answer line, no confidence68137< 0.0001< 0.0001 *
GPT-6.1 Terra low · chat Closed · Workhorse15.4%4 “I don't know” of 26 not right99.3%543 of 547 answered, 4 “I don't know”4.1%22 of 54314.8%4 “I don't know” of 27 not right99.3%543 of 547 answered, 4 “I don't know”4.2%23 of 543run 751a new runrun 801the Hallucination tab's full runs, counted on the kept questions95.2%521 of 547 right95.1%520 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence111211
Perplexity Decider 27B v1.2 API · with web Open · 27B15.2%10 “I don't know” of 66 not right98.2%537 of 547 answered, 10 “I don't know”10.4%56 of 5375.6%3 “I don't know” of 54 not right99.5%544 of 547 answered, 3 “I don't know”9.4%51 of 544run 644a new runrun 642the Hallucination tab's full runs, counted on the kept questions87.9%481 of 547 right90.1%493 of 547 right4.6%2.2%26210.561
Winnow-12B Open · 12B15.0%15 “I don't know” of 100 not right97.3%532 of 547 answered, 15 “I don't know”16.0%85 of 5328.6%8 “I don't know” of 93 not right98.5%539 of 547 answered, 8 “I don't know”15.8%85 of 539run 748a new runrun 750the Hallucination tab's full runs, counted on the kept questions81.7%447 of 547 right83.0%454 of 547 right3.1%1.8%323211
GPT-6 Astra low · chat · 1 per request Closed · Flagship14.8%4 “I don't know” of 27 not right99.3%543 of 547 answered, 4 “I don't know”4.2%23 of 54317.4%4 “I don't know” of 23 not right99.3%543 of 547 answered, 4 “I don't know”3.5%19 of 543run 815a new runrun 819the Hallucination tab's full runs, counted on the kept questions95.1%520 of 547 right95.8%524 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence1060.451
GPT-6.1 Sol low · chat Closed · Workhorse14.8%4 “I don't know” of 27 not right99.3%543 of 547 answered, 4 “I don't know”4.2%23 of 54318.2%4 “I don't know” of 22 not right99.3%543 of 547 answered, 4 “I don't know”3.3%18 of 543run 744a new runrun 799the Hallucination tab's full runs, counted on the kept questions95.1%520 of 547 right96.0%525 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence940.271
Mercury Decide Closed · Efficient13.4%11 “I don't know” of 82 not rightrange 11.1% to 13.4%98.0%536 of 547 answered, 11 “I don't know”range 98.0% to 98.4%13.2%71 of 53610.1%8 “I don't know” of 79 not rightrange 10.1% to 10.5%98.5%539 of 547 answered, 8 “I don't know”the same in every run13.2%71 of 539runs 450, 460, 469, 478, 484new runsruns 489, 491, 493, 495, 497new runs on the kept questions85.0%465 of 547 rightrange 85.0% to 85.4%85.6%468 of 547 rightrange 85.6% to 86.1%3.2%range 3.1% to 3.2%1.9%range 1.9% to 2.0%292911
LFM2.5-24B-A2B default reasoning · chat Open · 24B MoE · 2.5B active13.2%14 “I don't know” of 106 not right97.4%533 of 547 answered, 14 “I don't know”17.3%92 of 533Not runNot runNot runrun 1070a new runNot run80.6%441 of 547 rightNot runNot measureda plain answer line, no confidenceNot runNot runNot runNot runNot run
GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse12.9%4 “I don't know” of 31 not right99.3%543 of 547 answered, 4 “I don't know”5.0%27 of 54317.4%4 “I don't know” of 23 not right99.3%543 of 547 answered, 4 “I don't know”3.5%19 of 543run 769a new runrun 810the Hallucination tab's full runs, counted on the kept questions94.3%516 of 547 right95.8%524 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence910.0211
LFM3-1.7B reasoning off · chat Open · 1.7B11.6%18 “I don't know” of 155 not right96.7%529 of 547 answered, 18 “I don't know”25.9%137 of 52955.4%215 “I don't know” of 388 not right56.9%311 of 547 answered, 215 “I don't know”, 21 invalid55.6%173 of 311run 1225a new runrun 1234the Hallucination tab's full runs, counted on the kept questions71.7%392 of 547 right25.2%138 of 547 rightNot measureda plain answer line, no confidenceNot measureda plain answer line, no confidence791150.0120.57
Gemma 4 E2B default reasoning · chat Open · 5.1B11.5%17 “I don't know” of 148 not right96.5%528 of 547 answered, 17 “I don't know”, 2 invalid24.8%131 of 528Not runNot runNot runrun 893a new runNot run72.6%397 of 547 rightNot runNot measureda plain answer line, no confidenceNot runNot runNot runNot runNot run
Gemma 4 E2B reasoning on · chat Open · 5.1B11.3%17 “I don't know” of 150 not right96.3%527 of 547 answered, 17 “I don't know”, 3 invalid25.2%133 of 527Not runNot runNot runrun 1350a new runNot run72.0%394 of 547 rightNot runNot measureda plain answer line, no confidenceNot runNot runNot runNot runNot run
Cygnet Open · 12B11.2%10 “I don't know” of 89 not right98.2%537 of 547 answered, 10 “I don't know”14.7%79 of 5375.8%5 “I don't know” of 86 not right99.1%542 of 547 answered, 5 “I don't know”14.9%81 of 542run 755a new runrun 757the Hallucination tab's full runs, counted on the kept questions83.7%458 of 547 right84.3%461 of 547 right3.5%2.8%28300.91
KnowLine-0.8B-Gen1 Open · 0.8B10.7%26 “I don't know” of 243 not right95.2%521 of 547 answered, 26 “I don't know”41.7%217 of 5214.4%15 “I don't know” of 340 not right97.3%532 of 547 answered, 15 “I don't know”61.1%325 of 532run 1003a new runrun 1007the Hallucination tab's full runs, counted on the kept questions55.6%304 of 547 right37.8%207 of 547 right5.8%6.0%49157< 0.0001< 0.0001 *
Qwen3.5-2B reasoning off · chat Open · 2.0B10.3%18 “I don't know” of 174 not right96.3%527 of 547 answered, 18 “I don't know”, 2 invalid29.6%156 of 52716.5%55 “I don't know” of 334 not right83.4%456 of 547 answered, 55 “I don't know”, 36 invalid61.2%279 of 456run 1216a new runrun 1219the Hallucination tab's full runs, counted on the kept questions67.8%371 of 547 right32.4%177 of 547 rightNot measureda plain answer line, no confidenceNot measureda plain answer line, no confidence50173< 0.0001< 0.0001 *
Gemini 3.8 Flash high · chat Closed · Workhorse10.0%2 “I don't know” of 20 not right99.6%545 of 547 answered, 2 “I don't know”3.3%18 of 54513.3%2 “I don't know” of 15 not right99.6%545 of 547 answered, 2 “I don't know”2.4%13 of 545run 657a new runrun 681the Hallucination tab's full runs, counted on the kept questions96.3%527 of 547 right97.3%532 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence830.231
Qwen3.5-2B reasoning on · chat Open · 2.0B9.9%14 “I don't know” of 142 not right97.1%531 of 547 answered, 14 “I don't know”, 2 invalid24.1%128 of 531Not runNot runNot runrun 1344a new runNot run73.7%403 of 547 rightNot runNot measureda plain answer line, no confidenceNot runNot runNot runNot runNot run
Gemma 4 E4B default reasoning · chat Open · 8.0B9.8%10 “I don't know” of 102 not right97.8%535 of 547 answered, 10 “I don't know”, 2 invalid17.2%92 of 53520.7%25 “I don't know” of 121 not right95.4%522 of 547 answered, 25 “I don't know”18.4%96 of 522run 890a new runrun 1049the Hallucination tab's full runs, counted on the kept questions81.0%443 of 547 right77.9%426 of 547 rightNot measureda plain answer line, no confidenceNot measureda plain answer line, no confidence44480.751
GPT-6 Luna Decisions API Closed · Efficient9.4%10 “I don't know” of 106 not right98.2%537 of 547 answered, 10 “I don't know”17.9%96 of 5377.1%6 “I don't know” of 84 not right98.9%541 of 547 answered, 6 “I don't know”14.4%78 of 541run 444a new runrun 453a new run on the kept questions80.6%441 of 547 right84.6%463 of 547 right3.0%2.7%47290.051
Gemini Decision 1 20261111 Closed · Efficient9.1%5 “I don't know” of 55 not right99.1%542 of 547 answered, 5 “I don't know”9.2%50 of 54210.9%5 “I don't know” of 46 not right99.1%542 of 547 answered, 5 “I don't know”7.6%41 of 542run 1019a new runrun 1023the Hallucination tab's full runs, counted on the kept questions89.9%492 of 547 right91.6%501 of 547 right4.6%4.2%20110.151
Perplexity Decider 27B v1.3 Open · 27B8.8%5 “I don't know” of 57 not right99.1%542 of 547 answered, 5 “I don't know”9.6%52 of 5420.0%0 “I don't know” of 52 not right100.0%547 of 547 answered, 0 “I don't know”9.5%52 of 547run 1139a new runrun 1141the Hallucination tab's full runs, counted on the kept questions89.6%490 of 547 right90.5%495 of 547 right3.8%1.7%242411
Turion Jev 31B Open · 31B8.5%4 “I don't know” of 47 not right99.3%543 of 547 answered, 4 “I don't know”7.9%43 of 5435.7%2 “I don't know” of 35 not right99.6%545 of 547 answered, 2 “I don't know”6.1%33 of 545run 1077a new runrun 1285the Hallucination tab's full runs, counted on the kept questions91.4%500 of 547 right93.6%512 of 547 right2.3%1.6%21110.111
d3-ultra Open · 26B8.2%5 “I don't know” of 61 not right99.1%542 of 547 answered, 5 “I don't know”10.3%56 of 5422.1%1 “I don't know” of 47 not right99.8%546 of 547 answered, 1 “I don't know”8.4%46 of 546run 795a new runrun 363the Hallucination tab's full runs, counted on the kept questions88.8%486 of 547 right91.4%500 of 547 right2.8%1.0%33230.231
Claude Haiku 5.5 max · chat Closed · Efficient7.8%4 “I don't know” of 51 not right96.3%527 of 547 answered, 4 “I don't know”, 16 invalid8.9%47 of 52717.1%6 “I don't know” of 35 not right96.0%525 of 547 answered, 6 “I don't know”, 16 invalid5.5%29 of 525run 951a new runrun 950the Hallucination tab's full runs, counted on the kept questions87.8%480 of 547 right90.7%496 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence2570.00210.11
Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse7.7%2 “I don't know” of 26 not right99.6%545 of 547 answered, 2 “I don't know”4.4%24 of 54514.3%3 “I don't know” of 21 not right99.5%544 of 547 answered, 3 “I don't know”3.3%18 of 544run 669a new runrun 691the Hallucination tab's full runs, counted on the kept questions95.2%521 of 547 right96.2%526 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence1260.241
Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active7.5%4 “I don't know” of 53 not right99.3%543 of 547 answered, 4 “I don't know”9.0%49 of 5437.1%3 “I don't know” of 42 not right99.5%544 of 547 answered, 3 “I don't know”7.2%39 of 544run 646a new runrun 653the Hallucination tab's full runs, counted on the kept questions90.3%494 of 547 right92.3%505 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence25150.151
vera-medium-omni Open · 4.2B7.4%7 “I don't know” of 94 not right98.7%540 of 547 answered, 7 “I don't know”16.1%87 of 5404.2%5 “I don't know” of 118 not right99.1%542 of 547 answered, 5 “I don't know”20.8%113 of 542run 1269a new runrun 1271the Hallucination tab's full runs, counted on the kept questions82.8%453 of 547 right78.4%429 of 547 right5.7%5.3%26520.00430.23
Claude Fable 5.1 high · chat Closed · Flagship7.4%2 “I don't know” of 27 not right99.6%545 of 547 answered, 2 “I don't know”4.6%25 of 54514.8%4 “I don't know” of 27 not right99.3%543 of 547 answered, 4 “I don't know”4.2%23 of 543run 655a new runrun 695the Hallucination tab's full runs, counted on the kept questions95.1%520 of 547 right95.1%520 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence1190.821
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active7.3%3 “I don't know” of 41 not right99.5%544 of 547 answered, 3 “I don't know”7.0%38 of 5447.7%3 “I don't know” of 39 not right99.5%544 of 547 answered, 3 “I don't know”6.6%36 of 544run 651a new runrun 686the Hallucination tab's full runs, counted on the kept questions92.5%506 of 547 right92.9%508 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence16140.861
Perplexity Decider Flash v1.3 Open · 4.0B7.2%7 “I don't know” of 97 not right98.7%540 of 547 answered, 7 “I don't know”16.7%90 of 5407.6%9 “I don't know” of 119 not right98.4%538 of 547 answered, 9 “I don't know”20.4%110 of 538run 1128a new runrun 1130the Hallucination tab's full runs, counted on the kept questions82.3%450 of 547 right78.2%428 of 547 right5.1%5.0%37570.0491
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active7.1%3 “I don't know” of 42 not right99.5%544 of 547 answered, 3 “I don't know”7.2%39 of 5442.6%1 “I don't know” of 39 not right99.8%546 of 547 answered, 1 “I don't know”7.0%38 of 546run 809a new runrun 811the Hallucination tab's full runs, counted on the kept questions92.3%505 of 547 right92.9%508 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence191811
GPT-6 Luna low · chat Closed · Efficient6.8%5 “I don't know” of 74 not right99.1%542 of 547 answered, 5 “I don't know”12.7%69 of 5423.0%2 “I don't know” of 67 not right99.6%545 of 547 answered, 2 “I don't know”11.9%65 of 545run 735a new runrun 796the Hallucination tab's full runs, counted on the kept questions86.5%473 of 547 right87.8%480 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence31270.691
REAX 1.3 Open · 57B6.7%3 “I don't know” of 45 not right99.5%544 of 547 answered, 3 “I don't know”7.7%42 of 5442.8%1 “I don't know” of 36 not right99.8%546 of 547 answered, 1 “I don't know”6.4%35 of 546run 831a new runrun 833the Hallucination tab's full runs, counted on the kept questions91.8%502 of 547 right93.4%511 of 547 right1.9%0.7%20130.31
Claude Opus 5.5 high · chat Closed · Flagship6.5%2 “I don't know” of 31 not right99.6%545 of 547 answered, 2 “I don't know”5.3%29 of 54511.5%3 “I don't know” of 26 not right99.5%544 of 547 answered, 3 “I don't know”4.2%23 of 544run 665a new runrun 773the Hallucination tab's full runs, counted on the kept questions94.3%516 of 547 right95.2%521 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence1370.261
Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active6.2%3 “I don't know” of 48 not right99.5%544 of 547 answered, 3 “I don't know”8.3%45 of 5448.5%4 “I don't know” of 47 not right99.3%543 of 547 answered, 4 “I don't know”7.9%43 of 543run 873a new runrun 872the Hallucination tab's full runs, counted on the kept questions91.2%499 of 547 right91.4%500 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence18160.861
GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active5.6%4 “I don't know” of 71 not right99.3%543 of 547 answered, 4 “I don't know”12.3%67 of 5433.5%3 “I don't know” of 85 not right99.5%544 of 547 answered, 3 “I don't know”15.1%82 of 544run 821a new runrun 822the Hallucination tab's full runs, counted on the kept questions87.0%476 of 547 right84.5%462 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence26410.0861
Qwen3.8 Flash no reasoning · chat Closed · Efficient5.6%4 “I don't know” of 71 not right99.3%543 of 547 answered, 4 “I don't know”12.3%67 of 5435.2%3 “I don't know” of 58 not right99.5%544 of 547 answered, 3 “I don't know”10.1%55 of 544run 988a new runrun 987the Hallucination tab's full runs, counted on the kept questions87.0%476 of 547 right89.4%489 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence31190.121
Gemma 4 12B Unified default reasoning · chat Open · 12B5.5%4 “I don't know” of 73 not right98.9%541 of 547 answered, 4 “I don't know”, 2 invalid12.8%69 of 541Not runNot runNot runrun 896a new runNot run86.3%472 of 547 rightNot runNot measureda plain answer line, no confidenceNot runNot runNot runNot runNot run
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active5.4%4 “I don't know” of 74 not right99.3%543 of 547 answered, 4 “I don't know”12.9%70 of 5430.0%0 “I don't know” of 57 not right100.0%547 of 547 answered, 0 “I don't know”10.4%57 of 547run 805a new runrun 806the Hallucination tab's full runs, counted on the kept questions86.5%473 of 547 right89.6%490 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence33200.0981
Clef-Flash Open · 9.0B5.3%5 “I don't know” of 95 not right99.1%542 of 547 answered, 5 “I don't know”16.6%90 of 5429.4%12 “I don't know” of 127 not right97.8%535 of 547 answered, 12 “I don't know”21.5%115 of 535run 660a new runrun 664the Hallucination tab's full runs, counted on the kept questions82.6%452 of 547 right76.8%420 of 547 right3.1%4.8%27520.00660.33
Muse Ember max · chat Closed · Efficient5.3%3 “I don't know” of 57 not right99.5%544 of 547 answered, 3 “I don't know”9.9%54 of 5447.8%4 “I don't know” of 51 not right99.3%543 of 547 answered, 4 “I don't know”8.7%47 of 543run 943a new runrun 940the Hallucination tab's full runs, counted on the kept questions89.6%490 of 547 right90.7%496 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence25180.361
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B5.2%12 “I don't know” of 229 not right97.8%535 of 547 answered, 12 “I don't know”40.6%217 of 5356.4%22 “I don't know” of 342 not right96.0%525 of 547 answered, 22 “I don't know”61.0%320 of 525run 1013a new runrun 1017the Hallucination tab's full runs, counted on the kept questions58.1%318 of 547 right37.5%205 of 547 right6.6%7.3%48151< 0.0001< 0.0001 *
Claude Sonnet 5.5 high · chat Closed · Workhorse4.8%3 “I don't know” of 62 not right99.5%544 of 547 answered, 3 “I don't know”10.8%59 of 5443.6%2 “I don't know” of 55 not right99.6%545 of 547 answered, 2 “I don't know”9.7%53 of 545run 668a new runrun 800the Hallucination tab's full runs, counted on the kept questions88.7%485 of 547 right89.9%492 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence31250.51
Perplexity Decider Flash v1.2 Open · 4.0B4.8%5 “I don't know” of 105 not right99.1%542 of 547 answered, 5 “I don't know”18.5%100 of 5424.3%6 “I don't know” of 139 not right98.9%541 of 547 answered, 6 “I don't know”24.6%133 of 541run 535a new runrun 537a new run on the kept questions80.8%442 of 547 right74.6%408 of 547 right5.5%6.3%36690.00170.091
Claude Opus 5.5 low · chat Closed · Flagship4.7%2 “I don't know” of 43 not right99.6%545 of 547 answered, 2 “I don't know”7.5%41 of 5456.5%2 “I don't know” of 31 not right99.6%545 of 547 answered, 2 “I don't know”5.3%29 of 545run 649a new runrun 683the Hallucination tab's full runs, counted on the kept questions92.1%504 of 547 right94.3%516 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence25130.0731
Perplexity Decider 27B v1.2 API Open · 27B4.5%3 “I don't know” of 66 not right99.5%544 of 547 answered, 3 “I don't know”11.6%63 of 5440.0%0 “I don't know” of 51 not right100.0%547 of 547 answered, 0 “I don't know”9.3%51 of 547run 643a new runrun 641the Hallucination tab's full runs, counted on the kept questions87.9%481 of 547 right90.7%496 of 547 right3.1%1.1%27150.0881
Claude Sonnet 5.5 low · chat Closed · Workhorse4.4%4 “I don't know” of 90 not right99.3%543 of 547 answered, 4 “I don't know”15.8%86 of 5435.7%5 “I don't know” of 87 not right99.1%542 of 547 answered, 5 “I don't know”15.1%82 of 542run 654a new runrun 690the Hallucination tab's full runs, counted on the kept questions83.5%457 of 547 right84.1%460 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence36320.721
Logits Sigma-1 Duo Open · 57B4.3%2 “I don't know” of 46 not right99.6%545 of 547 answered, 2 “I don't know”8.1%44 of 5450.0%0 “I don't know” of 36 not right100.0%547 of 547 answered, 0 “I don't know”6.6%36 of 547run 1231a new runrun 1233the Hallucination tab's full runs, counted on the kept questions91.6%501 of 547 right93.4%511 of 547 right1.8%0.8%25170.281
Jev 1.14 Closed · Efficient4.0%3 “I don't know” of 75 not rightrange 4.0% to 5.2%99.5%544 of 547 answered, 3 “I don't know”range 99.3% to 99.5%13.2%72 of 5444.2%3 “I don't know” of 72 not rightrange 4.0% to 4.5%99.5%544 of 547 answered, 3 “I don't know”the same in every run12.7%69 of 544runs 446, 454, 457, 461, 464new runsruns 470, 473, 476, 480, 482new runs on the kept questions86.3%472 of 547 rightrange 85.9% to 86.7%86.8%475 of 547 rightrange 86.3% to 87.8%1.6%range 1.6% to 1.6%1.7%range 1.6% to 1.7%20170.741
OneJev-27B Open · 27B4.0%3 “I don't know” of 75 not right99.5%544 of 547 answered, 3 “I don't know”13.2%72 of 5441.6%1 “I don't know” of 64 not right99.8%546 of 547 answered, 1 “I don't know”11.5%63 of 546run 780a new runrun 782the Hallucination tab's full runs, counted on the kept questions86.3%472 of 547 right88.3%483 of 547 right2.3%1.3%34250.31
Microsoft-Decision-1 Closed · Efficient3.8%3 “I don't know” of 80 not rightrange 3.8% to 4.0%99.5%544 of 547 answered, 3 “I don't know”the same in every run14.2%77 of 5442.2%2 “I don't know” of 90 not rightrange 2.2% to 4.3%99.6%545 of 547 answered, 2 “I don't know”range 99.3% to 99.6%16.1%88 of 545runs 451, 459, 467, 477, 483new runsruns 487, 490, 492, 494, 496new runs on the kept questions85.4%467 of 547 rightrange 85.4% to 86.3%83.5%457 of 547 rightrange 82.8% to 83.5%3.6%range 3.6% to 3.6%2.3%range 2.3% to 2.3%24350.191
Chinese Decision Pointer 4B Open · 4.0B3.7%4 “I don't know” of 108 not right99.3%543 of 547 answered, 4 “I don't know”19.2%104 of 5433.0%4 “I don't know” of 133 not right99.3%543 of 547 answered, 4 “I don't know”23.8%129 of 543run 1211a new runrun 1213the Hallucination tab's full runs, counted on the kept questions80.3%439 of 547 right75.7%414 of 547 right2.1%1.9%41660.020.9
Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B3.7%3 “I don't know” of 81 not right99.5%544 of 547 answered, 3 “I don't know”14.3%78 of 5445.3%4 “I don't know” of 76 not right99.3%543 of 547 answered, 4 “I don't know”13.3%72 of 543run 673a new runrun 674the Hallucination tab's full runs, counted on the kept questions85.2%466 of 547 right86.1%471 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence33270.521
Gemma 4 31B reasoning off · chat Open · 31B3.6%2 “I don't know” of 55 not right99.6%545 of 547 answered, 2 “I don't know”9.7%53 of 5454.5%2 “I don't know” of 44 not right99.6%545 of 547 answered, 2 “I don't know”7.7%42 of 545run 775a new runrun 776the Hallucination tab's full runs, counted on the kept questions89.9%492 of 547 right92.0%503 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence30190.151
Qwen3.5-4B reasoning off · chat Open · 4.0B3.6%4 “I don't know” of 110 not right99.3%543 of 547 answered, 4 “I don't know”19.5%106 of 5433.4%5 “I don't know” of 145 not right98.4%538 of 547 answered, 5 “I don't know”, 4 invalid26.0%140 of 538run 1144a new runrun 1147the Hallucination tab's full runs, counted on the kept questions79.9%437 of 547 right72.8%398 of 547 rightNot measureda plain answer line, no confidenceNot measureda plain answer line, no confidence52860.00480.23
Qwen3.8-27B low · chat Open · 27B3.6%2 “I don't know” of 55 not right99.6%545 of 547 answered, 2 “I don't know”9.7%53 of 5452.0%1 “I don't know” of 49 not right99.8%546 of 547 answered, 1 “I don't know”8.8%48 of 546run 671a new runrun 666the Hallucination tab's full runs, counted on the kept questions89.9%492 of 547 right91.0%498 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence23180.531
Gemini 3.8 Flash low · chat Closed · Workhorse3.4%1 “I don't know” of 29 not right99.8%546 of 547 answered, 1 “I don't know”5.1%28 of 54611.1%3 “I don't know” of 27 not right99.5%544 of 547 answered, 3 “I don't know”4.4%24 of 544run 645a new runrun 676the Hallucination tab's full runs, counted on the kept questions94.7%518 of 547 right95.1%520 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence14100.541
GLM-5.3 low · chat Open · Workhorse3.4%2 “I don't know” of 58 not right99.6%545 of 547 answered, 2 “I don't know”10.3%56 of 5454.7%3 “I don't know” of 64 not right99.5%544 of 547 answered, 3 “I don't know”11.2%61 of 544run 650a new runrun 656the Hallucination tab's full runs, counted on the kept questions89.4%489 of 547 right88.3%483 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence23280.581
Qwen3.8-27B vote share · reasoning off · chat Open · 27B3.4%3 “I don't know” of 87 not right99.5%544 of 547 answered, 3 “I don't know”15.4%84 of 5445.1%4 “I don't know” of 79 not right99.3%543 of 547 answered, 4 “I don't know”13.8%75 of 543run 680a new runrun 682the Hallucination tab's full runs, counted on the kept questions84.1%460 of 547 right85.6%468 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence35260.311
MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active3.4%3 “I don't know” of 88 not right99.5%544 of 547 answered, 3 “I don't know”15.6%85 of 5441.1%1 “I don't know” of 91 not right99.6%545 of 547 answered, 1 “I don't know”, 1 invalid16.5%90 of 545run 865a new runrun 864the Hallucination tab's full runs, counted on the kept questions83.9%459 of 547 right83.2%455 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence38430.661
Drex Max Open · 27B3.2%2 “I don't know” of 62 not right99.6%545 of 547 answered, 2 “I don't know”11.0%60 of 5453.6%2 “I don't know” of 55 not right99.6%545 of 547 answered, 2 “I don't know”9.7%53 of 545run 798a new runrun 367the Hallucination tab's full runs, counted on the kept questions88.7%485 of 547 right89.9%492 of 547 right1.7%0.6%28210.391
Perplexity Decider 27B v1.2 DGX Spark Open · 27B3.0%2 “I don't know” of 66 not right99.6%545 of 547 answered, 2 “I don't know”11.7%64 of 5450.0%0 “I don't know” of 50 not right100.0%547 of 547 answered, 0 “I don't know”9.1%50 of 547run 529a new runrun 372the Hallucination tab's full runs, counted on the kept questions87.9%481 of 547 right90.9%497 of 547 right2.8%0.9%29150.0491
d1 Closed · Efficient2.9%2 “I don't know” of 69 not rightrange 2.6% to 2.9%99.6%545 of 547 answered, 2 “I don't know”the same in every run12.3%67 of 5451.1%1 “I don't know” of 90 not rightrange 0.0% to 1.1%99.8%546 of 547 answered, 1 “I don't know”range 99.8% to 100.0%16.3%89 of 546runs 445, 455, 458, 463, 465new runsruns 472, 475, 479, 481, 485new runs on the kept questions87.4%478 of 547 rightrange 85.9% to 87.4%83.5%457 of 547 rightrange 83.0% to 85.0%0.9%range 0.9% to 1.0%1.3%range 1.1% to 1.3%21430.00810.4
StartLux-Decision-v1.1-9B-Instant Open · 9.0B2.6%2 “I don't know” of 76 not right99.6%545 of 547 answered, 2 “I don't know”13.6%74 of 5451.0%1 “I don't know” of 96 not right99.8%546 of 547 answered, 1 “I don't know”17.4%95 of 546run 1295a new runrun 1297the Hallucination tab's full runs, counted on the kept questions86.1%471 of 547 right82.4%451 of 547 right1.8%1.4%26470.0190.88
Cerno 9B v1.2 Open · 9.0B2.5%2 “I don't know” of 79 not right99.6%545 of 547 answered, 2 “I don't know”14.1%77 of 5450.0%0 “I don't know” of 91 not right100.0%547 of 547 answered, 0 “I don't know”16.6%91 of 547run 1087a new runrun 1204the Hallucination tab's full runs, counted on the kept questions85.6%468 of 547 right83.4%456 of 547 right2.2%1.6%34480.151
Qwen3.5-4B reasoning on · chat Open · 4.0B2.4%2 “I don't know” of 85 not right99.6%545 of 547 answered, 2 “I don't know”15.2%83 of 5452.2%2 “I don't know” of 90 not right99.1%542 of 547 answered, 2 “I don't know”, 3 invalid16.2%88 of 542run 1073a new runrun 1150the Hallucination tab's full runs, counted on the kept questions84.5%462 of 547 right83.0%454 of 547 rightNot measureda plain answer line, no confidenceNot measureda plain answer line, no confidence44490.681
Sieve 31B v3 Open · 31B2.2%1 “I don't know” of 46 not right99.8%546 of 547 answered, 1 “I don't know”8.2%45 of 5460.0%0 “I don't know” of 33 not right100.0%547 of 547 answered, 0 “I don't know”6.0%33 of 547run 1066a new runrun 1253the Hallucination tab's full runs, counted on the kept questions91.6%501 of 547 right94.0%514 of 547 right0.8%0.3%23110.0581
Erk-14B reasoning off · chat Open · 14B2.1%3 “I don't know” of 144 not right99.5%544 of 547 answered, 3 “I don't know”25.9%141 of 5440.8%1 “I don't know” of 130 not right99.8%546 of 547 answered, 1 “I don't know”23.6%129 of 546run 764a new runrun 765the Hallucination tab's full runs, counted on the kept questions73.7%403 of 547 right76.2%417 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence68560.321
Verdin-4B-v1.1 Open · 4.0B2.1%2 “I don't know” of 96 not right99.6%545 of 547 answered, 2 “I don't know”17.2%94 of 5450.8%1 “I don't know” of 128 not right99.8%546 of 547 answered, 1 “I don't know”23.3%127 of 546run 1249a new runrun 1251the Hallucination tab's full runs, counted on the kept questions82.4%451 of 547 right76.6%419 of 547 right3.5%2.2%37700.00180.099
Qwen3-14B reasoning off · chat Open · 15B1.8%2 “I don't know” of 110 not right99.6%545 of 547 answered, 2 “I don't know”19.8%108 of 5450.8%1 “I don't know” of 123 not right99.8%546 of 547 answered, 1 “I don't know”22.3%122 of 546run 771a new runrun 772the Hallucination tab's full runs, counted on the kept questions79.9%437 of 547 right77.5%424 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence40540.181
MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active1.7%1 “I don't know” of 59 not right99.8%546 of 547 answered, 1 “I don't know”10.6%58 of 5464.2%3 “I don't know” of 72 not right99.5%544 of 547 answered, 3 “I don't know”12.7%69 of 544run 880a new runrun 879the Hallucination tab's full runs, counted on the kept questions89.2%488 of 547 right86.8%475 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence20310.161
d3 Open · 26B1.7%1 “I don't know” of 59 not right99.8%546 of 547 answered, 1 “I don't know”10.6%58 of 5460.0%0 “I don't know” of 53 not right100.0%547 of 547 answered, 0 “I don't know”9.7%53 of 547run 793a new runrun 358the Hallucination tab's full runs, counted on the kept questions89.2%488 of 547 right90.3%494 of 547 right2.9%1.1%24190.541
GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active1.6%1 “I don't know” of 62 not right99.8%546 of 547 answered, 1 “I don't know”11.2%61 of 5463.5%2 “I don't know” of 57 not right99.6%545 of 547 answered, 2 “I don't know”10.1%55 of 545run 824a new runrun 825the Hallucination tab's full runs, counted on the kept questions88.7%485 of 547 right89.6%490 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence28220.481
Metask-Jev-4B Open · 4.5B1.6%2 “I don't know” of 125 not right99.6%545 of 547 answered, 2 “I don't know”22.6%123 of 5451.1%2 “I don't know” of 189 not right99.6%545 of 547 answered, 2 “I don't know”34.3%187 of 545run 741a new runrun 743the Hallucination tab's full runs, counted on the kept questions77.1%422 of 547 right65.4%358 of 547 right3.8%4.1%3498< 0.0001< 0.0001 *
Clef Open · 27B1.3%1 “I don't know” of 76 not right99.8%546 of 547 answered, 1 “I don't know”13.7%75 of 5461.5%1 “I don't know” of 68 not right99.8%546 of 547 answered, 1 “I don't know”12.3%67 of 546run 785a new runrun 788the Hallucination tab's full runs, counted on the kept questions86.1%471 of 547 right87.6%479 of 547 right2.4%3.6%26180.291
KnowLine-12B-Gen1 Open · 12B1.3%1 “I don't know” of 77 not right99.8%546 of 547 answered, 1 “I don't know”13.9%76 of 5460.0%0 “I don't know” of 67 not right100.0%547 of 547 answered, 0 “I don't know”12.2%67 of 547run 1316a new runrun 1318the Hallucination tab's full runs, counted on the kept questions85.9%470 of 547 right87.8%480 of 547 right1.3%0.6%26170.221
KnowLine-9B-Gen1 Open · 9.0B1.3%1 “I don't know” of 77 not right99.8%546 of 547 answered, 1 “I don't know”13.9%76 of 5460.0%0 “I don't know” of 98 not right100.0%547 of 547 answered, 0 “I don't know”17.9%98 of 547run 1179a new runrun 1181the Hallucination tab's full runs, counted on the kept questions85.9%470 of 547 right82.1%449 of 547 right1.6%0.7%18400.00540.28
KnowLine-9B-Gen2 Open · 9.0B1.3%1 “I don't know” of 79 not right99.8%546 of 547 answered, 1 “I don't know”14.3%78 of 5460.0%0 “I don't know” of 95 not right100.0%547 of 547 answered, 0 “I don't know”17.4%95 of 547run 1323a new runrun 1325the Hallucination tab's full runs, counted on the kept questions85.6%468 of 547 right82.6%452 of 547 right3.8%3.5%23400.0431
Valen-Decision-1.5-9B Open · 9.0B1.2%1 “I don't know” of 85 not right99.8%546 of 547 answered, 1 “I don't know”15.4%84 of 5460.0%0 “I don't know” of 110 not right100.0%547 of 547 answered, 0 “I don't know”20.1%110 of 547run 1280a new runrun 1282the Hallucination tab's full runs, counted on the kept questions84.5%462 of 547 right79.9%437 of 547 right3.0%2.7%32580.0080.4
Dexy-Decision-9B-V2 Open · 9.0B1.1%1 “I don't know” of 91 not right99.8%546 of 547 answered, 1 “I don't know”16.5%90 of 5461.1%1 “I don't know” of 91 not right99.8%546 of 547 answered, 1 “I don't know”16.5%90 of 546run 1260a new runrun 1262the Hallucination tab's full runs, counted on the kept questions83.4%456 of 547 right83.4%456 of 547 right0.7%0.6%383811
KnowLine 4B Gen5.5 Open · 4.0B0.9%1 “I don't know” of 106 not right99.8%546 of 547 answered, 1 “I don't know”19.2%105 of 5462.5%3 “I don't know” of 121 not right99.5%544 of 547 answered, 3 “I don't know”21.7%118 of 544run 963a new runrun 967the Hallucination tab's full runs, counted on the kept questions80.6%441 of 547 right77.9%426 of 547 right1.3%1.0%40530.211
Strands Decider 2B Open · 2.0B0.7%2 “I don't know” of 303 not right99.6%545 of 547 answered, 2 “I don't know”55.2%301 of 5451.5%5 “I don't know” of 332 not right99.1%542 of 547 answered, 5 “I don't know”60.3%327 of 542run 704a new runrun 706the Hallucination tab's full runs, counted on the kept questions44.6%244 of 547 right39.3%215 of 547 right3.8%4.9%49750.0241
Adjudex 9B v5 Open · 9.0B0.0%0 “I don't know” of 77 not right100.0%547 of 547 answered, 0 “I don't know”14.1%77 of 5470.0%0 “I don't know” of 89 not right100.0%547 of 547 answered, 0 “I don't know”16.3%89 of 547run 1188a new runrun 1190the Hallucination tab's full runs, counted on the kept questions85.9%470 of 547 right83.7%458 of 547 right0.6%0.3%32440.211
Claude Opus 5.5 low · chat · 1 per request Closed · Flagship0.0%0 “I don't know” of 31 not right100.0%547 of 547 answered, 0 “I don't know”5.7%31 of 5473.8%1 “I don't know” of 26 not right99.8%546 of 547 answered, 1 “I don't know”4.6%25 of 546run 670a new runrun 803the Hallucination tab's full runs, counted on the kept questions94.3%516 of 547 right95.2%521 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence1370.261
Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse0.0%0 “I don't know” of 38 not right100.0%547 of 547 answered, 0 “I don't know”6.9%38 of 5472.7%1 “I don't know” of 37 not right99.8%546 of 547 answered, 1 “I don't know”6.6%36 of 546run 677a new runrun 812the Hallucination tab's full runs, counted on the kept questions93.1%509 of 547 right93.2%510 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence16140.861
Copernicus-31B Open · 31B0.0%0 “I don't know” of 39 not right100.0%547 of 547 answered, 0 “I don't know”7.1%39 of 5472.9%1 “I don't know” of 34 not right99.8%546 of 547 answered, 1 “I don't know”6.0%33 of 546run 1170a new runrun 1172the Hallucination tab's full runs, counted on the kept questions92.9%508 of 547 right93.8%513 of 547 right1.0%0.7%19130.381
Gemma 4 31B reasoning off · chat · 1 per request Open · 31B0.0%0 “I don't know” of 59 not right100.0%547 of 547 answered, 0 “I don't know”10.8%59 of 5471.9%1 “I don't know” of 53 not right99.8%546 of 547 answered, 1 “I don't know”9.5%52 of 546run 786a new runrun 789the Hallucination tab's full runs, counted on the kept questions89.2%488 of 547 right90.3%494 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence31240.421
Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active0.0%0 “I don't know” of 60 not right100.0%547 of 547 answered, 0 “I don't know”11.0%60 of 5470.0%0 “I don't know” of 62 not right100.0%547 of 547 answered, 0 “I don't know”11.3%62 of 547run 869a new runrun 868the Hallucination tab's full runs, counted on the kept questions89.0%487 of 547 right88.7%485 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence30320.91
Jeffan-9B-1211 Open · 9.0B0.0%0 “I don't know” of 71 not right100.0%547 of 547 answered, 0 “I don't know”13.0%71 of 5474.1%3 “I don't know” of 74 not right99.5%544 of 547 answered, 3 “I don't know”13.1%71 of 544run 1197a new runrun 1199the Hallucination tab's full runs, counted on the kept questions87.0%476 of 547 right86.5%473 of 547 right3.7%3.7%313111
Karpo Decision 31B Open · 31B0.0%0 “I don't know” of 42 not right100.0%547 of 547 answered, 0 “I don't know”7.7%42 of 5473.2%1 “I don't know” of 31 not right99.8%546 of 547 answered, 1 “I don't know”5.5%30 of 546run 1161a new runrun 1163the Hallucination tab's full runs, counted on the kept questions92.3%505 of 547 right94.3%516 of 547 right0.6%0.3%25130.0731
KnowLine-31B-Gen1.5 Open · 31B0.0%0 “I don't know” of 37 not right100.0%547 of 547 answered, 0 “I don't know”6.8%37 of 5470.0%0 “I don't know” of 34 not right100.0%547 of 547 answered, 0 “I don't know”6.2%34 of 547run 955a new runrun 959the Hallucination tab's full runs, counted on the kept questions93.2%510 of 547 right93.8%513 of 547 right0.2%0.2%16130.711
KnowLine-31B-Gen2 Open · 31B0.0%0 “I don't know” of 40 not right100.0%547 of 547 answered, 0 “I don't know”7.3%40 of 5470.0%0 “I don't know” of 29 not right100.0%547 of 547 answered, 0 “I don't know”5.3%29 of 547run 1329a new runrun 1331the Hallucination tab's full runs, counted on the kept questions92.7%507 of 547 right94.7%518 of 547 right0.2%0.1%2090.0611
Qwen4-Max high · chat Closed · Workhorse0.0%0 “I don't know” of 29 not right100.0%547 of 547 answered, 0 “I don't know”5.3%29 of 5473.4%1 “I don't know” of 29 not right99.8%546 of 547 answered, 1 “I don't know”5.1%28 of 546run 1338a new runrun 1339the Hallucination tab's full runs, counted on the kept questions94.7%518 of 547 right94.7%518 of 547 rightNo probabilitiesa chat model states a confidenceNo probabilitiesa chat model states a confidence121111
StartLux-Decision-v1.1-4B-Instant Open · 4.0B0.0%0 “I don't know” of 94 not right100.0%547 of 547 answered, 0 “I don't know”17.2%94 of 5471.7%2 “I don't know” of 120 not right99.6%545 of 547 answered, 2 “I don't know”21.7%118 of 545run 1083a new runrun 1239the Hallucination tab's full runs, counted on the kept questions82.8%453 of 547 right78.1%427 of 547 right3.2%2.9%33570.0150.71

Our method with an “I don't know” option

The option

As on the Hallucination tab, each question gets a sixth option after its own five: F, “I don't know” in English, “Bilmiyorum” in Turkish. It is always the last option, never shuffled and never correct. The English reading instructions of a decision model get one added sentence: “If you do not know the answer, choose I don't know.” The full instructions:

Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.

Prompt version 413d6775fc58. The question and its options are the English translation, as in the accuracy comparison above; no person checked the translations. The Turkish side uses the Hallucination tab's prompts unchanged.

Any other chat model is asked with the chat models' English prompts, the option after each question's own and one sentence added at the end of each instruction: “If you do not know the answer, choose I don't know.” Its system prompt and instruction when it states a confidence:

You are taking an English multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.

Answer each of the following multiple-choice questions. Without explanation, return for each question the letter of the correct option together with a confidence score between 0 and 1 showing how sure you are that your answer is correct, in the specified JSON format. If you do not know the answer, choose I don't know.

Chat prompt version e53d4aa27643. Its Turkish side uses the Hallucination tab's chat prompts unchanged.

A chat model asked for a plain answer line (LFM2.5-8B-A1B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning on · chat, LFM3-1.7B · reasoning off · chat, Qwen3.5-2B · reasoning off · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, Qwen3.5-4B · reasoning on · chat, Raon-1 nano, MiniCPM5-2B, Granite-4.2-3B, LFM2.5-24B-A2B, Gemma 4 E2B, Gemma 4 E2B, Qwen3.5-2B and Gemma 4 12B Unified) is asked with its plain answer line's English instruction, the option's letter among its letters and one sentence added at its end: “If you do not know the answer, choose I don't know.” Plain prompt version 4cd69035a45e. Its Turkish side uses the Hallucination tab's plain prompts with “Bilmiyorum” unchanged. It states no confidence and reports no option probabilities, so its mean probability of the option is not measured.

The Turkish side

The Turkish side: the Hallucination tab's published full runs, counted on the kept questions only, for d1.2-ultra (run 1367), d1.2 (run 1365), d1-2B (run 712), d1-ultra (runs 350, 354, 360, 365, 370), GLiDE 0.4B (run 937), Drex Max v1.1 (run 352), Mammoth-1 (run 386), GLiDE · v1.2 (run 369), Laya Multilingual (run 737), GLiDE 27B (run 905), OneJev1.6-0.8B (run 1031), Grok 4.7 · high · chat (run 675), LFM2.5-8B-A1B · default reasoning · chat (run 1298), GLiDE 2B (run 921), Elephant-1 (run 382), Gemini 4 Carbon · high · chat (run 1121), SubQ Snap 1.1 (run 392), GLiDE 0.8B (run 929), GLiClass v4 Edge (run 1048), Gemini 4 Hydrogen · high · chat (run 1117), GLiNER2.5 Multi (run 724), GPT-6 Astra · high · chat (run 818), Gemini 4 Argon · low · chat (run 679), Laya (run 730), GPT-6 Astra · low · chat (run 816), Open-Jev-27B-v1.1 (run 768), StartLux-Decision-v1.1-2B-Instant (run 979), MiniCPM5-2B · reasoning off · chat (run 1304), GLiDE 9B (run 913), GPT-6 Astra · medium · chat (run 817), GPT-6.1 Sol · high · chat (run 802), GPT-6.1 Terra · high · chat (run 807), GLiNER2.5-multi-Decide (run 718), LFM2.5-2.6B · default reasoning · chat (run 1038), LFM3-1.7B · reasoning on · chat (run 1090), GPT-6.1 Terra · low · chat (run 801), Perplexity Decider 27B v1.2 · API · with web (run 642), Winnow-12B (run 750), GPT-6 Astra · low · chat · 1 per request (run 819), GPT-6.1 Sol · low · chat (run 799), GPT-6.1 Sol · low · chat · 1 per request (run 810), LFM3-1.7B · reasoning off · chat (run 1234), Cygnet (run 757), KnowLine-0.8B-Gen1 (run 1007), Qwen3.5-2B · reasoning off · chat (run 1219), Gemini 3.8 Flash · high · chat (run 681), Gemma 4 E4B · default reasoning · chat (run 1049), Gemini Decision 1 · 20261111 (run 1023), Perplexity Decider 27B v1.3 (run 1141), Turion Jev 31B (run 1285), d3-ultra (run 363), Claude Haiku 5.5 · max · chat (run 950), Gemini 3.8 Flash · low · chat · 1 per request (run 691), Kimi K3.1 Pro · no reasoning · chat (run 653), vera-medium-omni (run 1271), Claude Fable 5.1 · high · chat (run 695), Kimi K3.1 Pro · low · chat (run 686), Perplexity Decider Flash v1.3 (run 1130), DeepSeek-V4.1-Flash · reasoning on · chat · 1017 (run 811), GPT-6 Luna · low · chat (run 796), REAX 1.3 (run 833), Claude Opus 5.5 · high · chat (run 773), Qwen3.5-397B-A17B · high · chat (run 872), GLM-5.3-Flash · reasoning off · chat (run 822), Qwen3.8 Flash · no reasoning · chat (run 987), DeepSeek-V4.1-Flash · reasoning off · chat · 1017 (run 806), Clef-Flash (run 664), Muse Ember · max · chat (run 940), StartLux-Decision-v1.1-0.8B-Instant (run 1017), Claude Sonnet 5.5 · high · chat (run 800), Claude Opus 5.5 · low · chat (run 683), Perplexity Decider 27B v1.2 · API (run 641), Claude Sonnet 5.5 · low · chat (run 690), Logits Sigma-1 Duo (run 1233), OneJev-27B (run 782), Chinese Decision Pointer 4B (run 1213), Qwen3.8-27B · stated confidence · reasoning off · chat (run 674), Gemma 4 31B · reasoning off · chat (run 776), Qwen3.5-4B · reasoning off · chat (run 1147), Qwen3.8-27B · low · chat (run 666), Gemini 3.8 Flash · low · chat (run 676), GLM-5.3 · low · chat (run 656), Qwen3.8-27B · vote share · reasoning off · chat (run 682), MiniMax-M3.1-Flash · high · chat (run 864), Drex Max (run 367), Perplexity Decider 27B v1.2 · DGX Spark (run 372), StartLux-Decision-v1.1-9B-Instant (run 1297), Cerno 9B v1.2 (run 1204), Qwen3.5-4B · reasoning on · chat (run 1150), Sieve 31B v3 (run 1253), Erk-14B · reasoning off · chat (run 765), Verdin-4B-v1.1 (run 1251), Qwen3-14B · reasoning off · chat (run 772), MiMo-V2.6-Flash · high · chat (run 879), d3 (run 358), GLM-5.3-Flash · reasoning on · chat (run 825), Metask-Jev-4B (run 743), Clef (run 788), KnowLine-12B-Gen1 (run 1318), KnowLine-9B-Gen1 (run 1181), KnowLine-9B-Gen2 (run 1325), Valen-Decision-1.5-9B (run 1282), Dexy-Decision-9B-V2 (run 1262), KnowLine 4B Gen5.5 (run 967), Strands Decider 2B (run 706), Adjudex 9B v5 (run 1190), Claude Opus 5.5 · low · chat · 1 per request (run 803), Claude Sonnet 5.5 · high · chat · 1 per request (run 812), Copernicus-31B (run 1172), Gemma 4 31B · reasoning off · chat · 1 per request (run 789), Inkling-Small · high · chat · Jul 2026 (run 868), Jeffan-9B-1211 (run 1199), Karpo Decision 31B (run 1163), KnowLine-31B-Gen1.5 (run 959), KnowLine-31B-Gen2 (run 1331), Qwen4-Max · high · chat (run 1339) and StartLux-Decision-v1.1-4B-Instant (run 1239); new runs on the kept questions only, with the Hallucination tab's prompts, for Mercury Decide (runs 489, 491, 493, 495, 497), GPT-6 Luna · Decisions API (run 453), Perplexity Decider Flash v1.2 (run 537), Jev 1.14 (runs 470, 473, 476, 480, 482), Microsoft-Decision-1 (runs 487, 490, 492, 494, 496) and d1 (runs 472, 475, 479, 481, 485). The new Turkish runs are not on the Hallucination tab, which uses full runs only.

d1 was asked as d1 in both languages, while its Text tab run used d1:free, because Liquid's API no longer serves that name; its model list still names d1, released 29 September 2026.

celeris-1.1-decision is shown in English only (run 884): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

celeris-1-decision is shown in English only (run 548): no Turkish run with “Bilmiyorum” on these questions; new Turkish runs were outside the request of 3 November 2026. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

Raon-1 nano · reasoning off · chat is shown in English only (run 1153): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

KnowLine-2B-Gen1 is shown in English only (run 1059): no Turkish run with “Bilmiyorum”: its Turkish Text run alone was asked (Ümit #6224, 13 November 2026). Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

MiniCPM5-2B · reasoning on · chat is shown in English only (run 1052): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

Granite-4.2-3B · reasoning on · chat is shown in English only (run 1110): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

LFM2.5-24B-A2B · default reasoning · chat is shown in English only (run 1070): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

Gemma 4 E2B · default reasoning · chat is shown in English only (run 893): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

Gemma 4 E2B · reasoning on · chat is shown in English only (run 1350): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

Qwen3.5-2B · reasoning on · chat is shown in English only (run 1344): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

Gemma 4 12B Unified · default reasoning · chat is shown in English only (run 896): no Turkish run: an English-only setup. Its Turkish and paired columns read Not run; it is in no paired test and no Holm adjustment.

Running: a chat model's runs with the option go English first, then Turkish, and each is added as it finishes. Until both runs of a chat model are in, it is shown with the side it has and the other reads Running, in no paired test and no Holm adjustment: Claude Fable 5.2 · max · chat.

Runs

five runs in English and five runs in Turkish for d1-ultra, Mercury Decide, Jev 1.14, Microsoft-Decision-1 and d1; one run in English and one run in Turkish for d1.2-ultra, d1.2, d1-2B, GLiDE 0.4B, Drex Max v1.1, Mammoth-1, GLiDE · v1.2, Laya Multilingual, GLiDE 27B, OneJev1.6-0.8B, Grok 4.7 · high · chat, LFM2.5-8B-A1B · default reasoning · chat, GLiDE 2B, Elephant-1, Gemini 4 Carbon · high · chat, SubQ Snap 1.1, GLiDE 0.8B, GLiClass v4 Edge, Gemini 4 Hydrogen · high · chat, GLiNER2.5 Multi, GPT-6 Astra · high · chat, Gemini 4 Argon · low · chat, Laya, GPT-6 Astra · low · chat, Open-Jev-27B-v1.1, StartLux-Decision-v1.1-2B-Instant, MiniCPM5-2B · reasoning off · chat, GLiDE 9B, GPT-6 Astra · medium · chat, GPT-6.1 Sol · high · chat, GPT-6.1 Terra · high · chat, GLiNER2.5-multi-Decide, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning on · chat, GPT-6.1 Terra · low · chat, Perplexity Decider 27B v1.2 · API · with web, Winnow-12B, GPT-6 Astra · low · chat · 1 per request, GPT-6.1 Sol · low · chat, GPT-6.1 Sol · low · chat · 1 per request, LFM3-1.7B · reasoning off · chat, Cygnet, KnowLine-0.8B-Gen1, Qwen3.5-2B · reasoning off · chat, Gemini 3.8 Flash · high · chat, Gemma 4 E4B · default reasoning · chat, GPT-6 Luna · Decisions API, Gemini Decision 1 · 20261111, Perplexity Decider 27B v1.3, Turion Jev 31B, d3-ultra, Claude Haiku 5.5 · max · chat, Gemini 3.8 Flash · low · chat · 1 per request, Kimi K3.1 Pro · no reasoning · chat, vera-medium-omni, Claude Fable 5.1 · high · chat, Kimi K3.1 Pro · low · chat, Perplexity Decider Flash v1.3, DeepSeek-V4.1-Flash · reasoning on · chat · 1017, GPT-6 Luna · low · chat, REAX 1.3, Claude Opus 5.5 · high · chat, Qwen3.5-397B-A17B · high · chat, GLM-5.3-Flash · reasoning off · chat, Qwen3.8 Flash · no reasoning · chat, DeepSeek-V4.1-Flash · reasoning off · chat · 1017, Clef-Flash, Muse Ember · max · chat, StartLux-Decision-v1.1-0.8B-Instant, Claude Sonnet 5.5 · high · chat, Perplexity Decider Flash v1.2, Claude Opus 5.5 · low · chat, Perplexity Decider 27B v1.2 · API, Claude Sonnet 5.5 · low · chat, Logits Sigma-1 Duo, OneJev-27B, Chinese Decision Pointer 4B, Qwen3.8-27B · stated confidence · reasoning off · chat, Gemma 4 31B · reasoning off · chat, Qwen3.5-4B · reasoning off · chat, Qwen3.8-27B · low · chat, Gemini 3.8 Flash · low · chat, GLM-5.3 · low · chat, Qwen3.8-27B · vote share · reasoning off · chat, MiniMax-M3.1-Flash · high · chat, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark, StartLux-Decision-v1.1-9B-Instant, Cerno 9B v1.2, Qwen3.5-4B · reasoning on · chat, Sieve 31B v3, Erk-14B · reasoning off · chat, Verdin-4B-v1.1, Qwen3-14B · reasoning off · chat, MiMo-V2.6-Flash · high · chat, d3, GLM-5.3-Flash · reasoning on · chat, Metask-Jev-4B, Clef, KnowLine-12B-Gen1, KnowLine-9B-Gen1, KnowLine-9B-Gen2, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, KnowLine 4B Gen5.5, Strands Decider 2B, Adjudex 9B v5, Claude Opus 5.5 · low · chat · 1 per request, Claude Sonnet 5.5 · high · chat · 1 per request, Copernicus-31B, Gemma 4 31B · reasoning off · chat · 1 per request, Inkling-Small · high · chat · Jul 2026, Jeffan-9B-1211, Karpo Decision 31B, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2, Qwen4-Max · high · chat and StartLux-Decision-v1.1-4B-Instant. These runs are kept out of the leaderboard, its repeats, the Hallucination tab and the accuracy comparison above.

The rates and how to read them

Each answer is right, wrong, the option or invalid. Out of the 547 kept questions, accuracy is right / 547; Answered is (right + wrong) / 547, the share of questions the model answered (the attempt rate), higher is better; Wrong of answers is wrong / (right + wrong), the share of its answers that were wrong, lower is better, and Undefined when it answered none. These two lead the table, with their counts under them. The non-hallucination rate, after them, is the option / (the option + wrong): among the answers that were not right, the share where the model said it did not know. As the Hallucination tab says, it is not a direct measure of what the model does not know: a model that always chose the option would reach 100% with no right answers, so read it with the attempt rate. With neither a wrong answer nor the option, the rate is undefined (No rate). The mean probability is the probability the model gave the option, averaged over its answers that report option probabilities; a chat model reports none. The paired test is the exact McNemar test on the questions only one of the two first runs got wrong, with p adjusted by Holm's method over the 67 decision models and the 56 chat models, each group a family of its own. No new combined score is made from these rates.

Results with “I don't know” in both languages

In the order of the Results table above. Each row compares the model's first English run with “I don't know” with its Turkish run with the option also written “I don't know”, out of the 547 kept questions. The paired counts, p and the Holm-adjusted p (over the 12 models with both runs) are those of the exact McNemar test on wrong answers; * marks a Holm-adjusted p below 0.05. Not run: no such run; Undefined: a rate with nothing to count.

ModelTurkish answered(right + wrong) / 547; higher is betterEnglish answered(right + wrong) / 547; higher is betterTurkish wrong of answerswrong / (right + wrong); lower is betterEnglish wrong of answerswrong / (right + wrong); lower is betterTurkish non-hallucination“I don't know” / (“I don't know” + wrong); higher is better, read with the answered shareEnglish non-hallucination“I don't know” / (“I don't know” + wrong); higher is better, read with the answered shareTurkish accuracyhigher is betterEnglish accuracyhigher is betterTurkish mean probabilityof “I don't know”English mean probabilityof “I don't know”Turkish runthe option-word run, counted on the kept questionsEnglish runthe first English run with “I don't know”Wrong only in Englishquestions the English run got wrong and the Turkish run did notWrong only in Turkishquestions the Turkish run got wrong and the English run did notpexact McNemarHolm padjusted over the 12 models with both runs
KnowLine-31B-Gen1.5 Open · 31BNot run100.0%547 of 547 answered, 0 “I don't know”Not run6.8%37 of 547Not run0.0%0 “I don't know” of 37 not rightNot run93.2%510 of 547 rightNot run0.2%Not runrun 955Not runNot runNot runNot run
Copernicus-31B Open · 31BNot run100.0%547 of 547 answered, 0 “I don't know”Not run7.1%39 of 547Not run0.0%0 “I don't know” of 39 not rightNot run92.9%508 of 547 rightNot run1.0%Not runrun 1170Not runNot runNot runNot run
REAX 1.3 Open · 57BNot run99.5%544 of 547 answered, 3 “I don't know”Not run7.7%42 of 544Not run6.7%3 “I don't know” of 45 not rightNot run91.8%502 of 547 rightNot run1.9%Not runrun 831Not runNot runNot runNot run
KnowLine-31B-Gen2 Open · 31BNot run100.0%547 of 547 answered, 0 “I don't know”Not run7.3%40 of 547Not run0.0%0 “I don't know” of 40 not rightNot run92.7%507 of 547 rightNot run0.2%Not runrun 1329Not runNot runNot runNot run
Karpo Decision 31B Open · 31BNot run100.0%547 of 547 answered, 0 “I don't know”Not run7.7%42 of 547Not run0.0%0 “I don't know” of 42 not rightNot run92.3%505 of 547 rightNot run0.6%Not runrun 1161Not runNot runNot runNot run
Gemini Decision 1 20261111 Closed · EfficientNot run99.1%542 of 547 answered, 5 “I don't know”Not run9.2%50 of 542Not run9.1%5 “I don't know” of 55 not rightNot run89.9%492 of 547 rightNot run4.6%Not runrun 1019Not runNot runNot runNot run
Logits Sigma-1 Duo Open · 57BNot run99.6%545 of 547 answered, 2 “I don't know”Not run8.1%44 of 545Not run4.3%2 “I don't know” of 46 not rightNot run91.6%501 of 547 rightNot run1.8%Not runrun 1231Not runNot runNot runNot run
GLiDE 27B Open · 27BNot run96.0%525 of 547 answered, 22 “I don't know”Not run6.9%36 of 525Not run37.9%22 “I don't know” of 58 not rightNot run89.4%489 of 547 rightNot run9.3%Not runrun 903Not runNot runNot runNot run
d1-ultra Closed · Efficient95.1%520 of 547 answered, 27 “I don't know”90.7%496 of 547 answered, 51 “I don't know”5.4%28 of 5207.9%39 of 49649.1%27 “I don't know” of 55 not right56.7%51 “I don't know” of 90 not right89.9%492 of 547 right83.5%457 of 547 right23.2%25.3%run 511run 4491870.0430.48
Sieve 31B v3 Open · 31BNot run99.8%546 of 547 answered, 1 “I don't know”Not run8.2%45 of 546Not run2.2%1 “I don't know” of 46 not rightNot run91.6%501 of 547 rightNot run0.8%Not runrun 1066Not runNot runNot runNot run
Turion Jev 31B Open · 31BNot run99.3%543 of 547 answered, 4 “I don't know”Not run7.9%43 of 543Not run8.5%4 “I don't know” of 47 not rightNot run91.4%500 of 547 rightNot run2.3%Not runrun 1077Not runNot runNot runNot run
Perplexity Decider 27B v1.2 API · with web Open · 27BNot run98.2%537 of 547 answered, 10 “I don't know”Not run10.4%56 of 537Not run15.2%10 “I don't know” of 66 not rightNot run87.9%481 of 547 rightNot run4.6%Not runrun 644Not runNot runNot runNot run
GLiDE v1.2 Open · 27B98.4%538 of 547 answered, 9 “I don't know”94.5%517 of 547 answered, 30 “I don't know”8.7%47 of 5387.0%36 of 51716.1%9 “I don't know” of 56 not right45.5%30 “I don't know” of 66 not right89.8%491 of 547 right87.9%481 of 547 right9.8%12.5%run 525run 52316270.131
d1.2-ultra Closed · Efficient89.0%487 of 547 answered, 60 “I don't know”80.1%438 of 547 answered, 109 “I don't know”3.7%18 of 4873.0%13 of 43876.9%60 “I don't know” of 78 not right89.3%109 “I don't know” of 122 not right85.7%469 of 547 right77.7%425 of 547 right17.7%26.0%run 1378run 13637120.361
Mammoth-1 Open · 27B95.4%522 of 547 answered, 25 “I don't know”93.2%510 of 547 answered, 37 “I don't know”6.1%32 of 5227.1%36 of 51043.9%25 “I don't know” of 57 not right50.7%37 “I don't know” of 73 not right89.6%490 of 547 right86.7%474 of 547 right7.4%9.8%run 519run 50519150.611
d3 Open · 26B100.0%547 of 547 answered, 0 “I don't know”99.8%546 of 547 answered, 1 “I don't know”9.5%52 of 54710.6%58 of 5460.0%0 “I don't know” of 52 not right1.7%1 “I don't know” of 59 not right90.5%495 of 547 right89.2%488 of 547 right1.4%2.9%run 544run 79324180.441
Drex Max Open · 27B99.8%546 of 547 answered, 1 “I don't know”99.6%545 of 547 answered, 2 “I don't know”9.0%49 of 54611.0%60 of 5452.0%1 “I don't know” of 50 not right3.2%2 “I don't know” of 62 not right90.9%497 of 547 right88.7%485 of 547 right0.9%1.7%run 542run 79828170.141
Perplexity Decider 27B v1.3 Open · 27BNot run99.1%542 of 547 answered, 5 “I don't know”Not run9.6%52 of 542Not run8.8%5 “I don't know” of 57 not rightNot run89.6%490 of 547 rightNot run3.8%Not runrun 1139Not runNot runNot runNot run
d3-ultra Open · 26B99.6%545 of 547 answered, 2 “I don't know”99.1%542 of 547 answered, 5 “I don't know”9.4%51 of 54510.3%56 of 5423.8%2 “I don't know” of 53 not right8.2%5 “I don't know” of 61 not right90.3%494 of 547 right88.8%486 of 547 right1.3%2.8%run 540run 79531260.61
d1.2 Closed · Efficient91.6%501 of 547 answered, 46 “I don't know”89.2%488 of 547 answered, 59 “I don't know”8.4%42 of 5017.6%37 of 48852.3%46 “I don't know” of 88 not right61.5%59 “I don't know” of 96 not right83.9%459 of 547 right82.4%451 of 547 right17.0%19.4%run 1377run 136119240.541
Perplexity Decider 27B v1.2 DGX Spark Open · 27B99.8%546 of 547 answered, 1 “I don't know”99.6%545 of 547 answered, 2 “I don't know”9.3%51 of 54611.7%64 of 5451.9%1 “I don't know” of 52 not right3.0%2 “I don't know” of 66 not right90.5%495 of 547 right87.9%481 of 547 right1.4%2.8%run 531run 52929160.0720.72
Drex Max v1.1 Closed · Efficient (provisional)96.7%529 of 547 answered, 18 “I don't know”92.1%504 of 547 answered, 43 “I don't know”9.1%48 of 5298.1%41 of 50427.3%18 “I don't know” of 66 not right51.2%43 “I don't know” of 84 not right87.9%481 of 547 right84.6%463 of 547 right10.2%14.3%run 513run 44720270.381
Perplexity Decider 27B v1.2 API Open · 27BNot run99.5%544 of 547 answered, 3 “I don't know”Not run11.6%63 of 544Not run4.5%3 “I don't know” of 66 not rightNot run87.9%481 of 547 rightNot run3.1%Not runrun 643Not runNot runNot runNot run
Jeffan-9B-1211 Open · 9.0BNot run100.0%547 of 547 answered, 0 “I don't know”Not run13.0%71 of 547Not run0.0%0 “I don't know” of 71 not rightNot run87.0%476 of 547 rightNot run3.7%Not runrun 1197Not runNot runNot runNot run
d1 Closed · EfficientNot run99.6%545 of 547 answered, 2 “I don't know”Not run12.3%67 of 545Not run2.9%2 “I don't know” of 69 not rightNot run87.4%478 of 547 rightNot run0.9%Not runrun 445Not runNot runNot runNot run
StartLux-Decision-v1.1-9B-Instant Open · 9.0BNot run99.6%545 of 547 answered, 2 “I don't know”Not run13.6%74 of 545Not run2.6%2 “I don't know” of 76 not rightNot run86.1%471 of 547 rightNot run1.8%Not runrun 1295Not runNot runNot runNot run
Clef Open · 27BNot run99.8%546 of 547 answered, 1 “I don't know”Not run13.7%75 of 546Not run1.3%1 “I don't know” of 76 not rightNot run86.1%471 of 547 rightNot run2.4%Not runrun 785Not runNot runNot runNot run
KnowLine-12B-Gen1 Open · 12BNot run99.8%546 of 547 answered, 1 “I don't know”Not run13.9%76 of 546Not run1.3%1 “I don't know” of 77 not rightNot run85.9%470 of 547 rightNot run1.3%Not runrun 1316Not runNot runNot runNot run
GLiDE 9B Open · 7.9BNot run97.4%533 of 547 answered, 14 “I don't know”Not run13.1%70 of 533Not run16.7%14 “I don't know” of 84 not rightNot run84.6%463 of 547 rightNot run7.8%Not runrun 911Not runNot runNot runNot run
OneJev-27B Open · 27BNot run99.5%544 of 547 answered, 3 “I don't know”Not run13.2%72 of 544Not run4.0%3 “I don't know” of 75 not rightNot run86.3%472 of 547 rightNot run2.3%Not runrun 780Not runNot runNot runNot run
Jev 1.14 Closed · EfficientNot run99.5%544 of 547 answered, 3 “I don't know”Not run13.2%72 of 544Not run4.0%3 “I don't know” of 75 not rightNot run86.3%472 of 547 rightNot run1.6%Not runrun 446Not runNot runNot runNot run
Cerno 9B v1.2 Open · 9.0BNot run99.6%545 of 547 answered, 2 “I don't know”Not run14.1%77 of 545Not run2.5%2 “I don't know” of 79 not rightNot run85.6%468 of 547 rightNot run2.2%Not runrun 1087Not runNot runNot runNot run
KnowLine-9B-Gen1 Open · 9.0BNot run99.8%546 of 547 answered, 1 “I don't know”Not run13.9%76 of 546Not run1.3%1 “I don't know” of 77 not rightNot run85.9%470 of 547 rightNot run1.6%Not runrun 1179Not runNot runNot runNot run
SubQ Snap 1.1 Closed · Efficient96.5%528 of 547 answered, 19 “I don't know”96.2%526 of 547 answered, 21 “I don't know”8.5%45 of 52812.0%63 of 52629.7%19 “I don't know” of 64 not right25.0%21 “I don't know” of 84 not right88.3%483 of 547 right84.6%463 of 547 right8.9%11.7%run 512run 44840220.030.36
Adjudex 9B v5 Open · 9.0BNot run100.0%547 of 547 answered, 0 “I don't know”Not run14.1%77 of 547Not run0.0%0 “I don't know” of 77 not rightNot run85.9%470 of 547 rightNot run0.6%Not runrun 1188Not runNot runNot runNot run
Mercury Decide Closed · EfficientNot run98.0%536 of 547 answered, 11 “I don't know”Not run13.2%71 of 536Not run13.4%11 “I don't know” of 82 not rightNot run85.0%465 of 547 rightNot run3.2%Not runrun 450Not runNot runNot runNot run
Microsoft-Decision-1 Closed · EfficientNot run99.5%544 of 547 answered, 3 “I don't know”Not run14.2%77 of 544Not run3.8%3 “I don't know” of 80 not rightNot run85.4%467 of 547 rightNot run3.6%Not runrun 451Not runNot runNot runNot run
Elephant-1 Open · 9.0B95.6%523 of 547 answered, 24 “I don't know”95.8%524 of 547 answered, 23 “I don't know”15.7%82 of 52312.8%67 of 52422.6%24 “I don't know” of 106 not right25.6%23 “I don't know” of 90 not right80.6%441 of 547 right83.5%457 of 547 right7.3%6.8%run 515run 50131460.110.99
Open-Jev-27B-v1.1 Open · 27BNot run97.1%531 of 547 answered, 16 “I don't know”Not run14.3%76 of 531Not run17.4%16 “I don't know” of 92 not rightNot run83.2%455 of 547 rightNot run15.2%Not runrun 763Not runNot runNot runNot run
KnowLine-9B-Gen2 Open · 9.0BNot run99.8%546 of 547 answered, 1 “I don't know”Not run14.3%78 of 546Not run1.3%1 “I don't know” of 79 not rightNot run85.6%468 of 547 rightNot run3.8%Not runrun 1323Not runNot runNot runNot run
celeris-1-decision Closed · EfficientNot run83.2%455 of 547 answered, 92 “I don't know”Not run8.1%37 of 455Not run71.3%92 “I don't know” of 129 not rightNot run76.4%418 of 547 rightNot run12.7%Not runrun 548Not runNot runNot runNot run
GPT-6 Luna Decisions API Closed · EfficientNot run98.2%537 of 547 answered, 10 “I don't know”Not run17.9%96 of 537Not run9.4%10 “I don't know” of 106 not rightNot run80.6%441 of 547 rightNot run3.0%Not runrun 444Not runNot runNot runNot run
Cygnet Open · 12BNot run98.2%537 of 547 answered, 10 “I don't know”Not run14.7%79 of 537Not run11.2%10 “I don't know” of 89 not rightNot run83.7%458 of 547 rightNot run3.5%Not runrun 755Not runNot runNot runNot run
Winnow-12B Open · 12BNot run97.3%532 of 547 answered, 15 “I don't know”Not run16.0%85 of 532Not run15.0%15 “I don't know” of 100 not rightNot run81.7%447 of 547 rightNot run3.1%Not runrun 748Not runNot runNot runNot run
Perplexity Decider Flash v1.3 Open · 4.0BNot run98.7%540 of 547 answered, 7 “I don't know”Not run16.7%90 of 540Not run7.2%7 “I don't know” of 97 not rightNot run82.3%450 of 547 rightNot run5.1%Not runrun 1128Not runNot runNot runNot run
Valen-Decision-1.5-9B Open · 9.0BNot run99.8%546 of 547 answered, 1 “I don't know”Not run15.4%84 of 546Not run1.2%1 “I don't know” of 85 not rightNot run84.5%462 of 547 rightNot run3.0%Not runrun 1280Not runNot runNot runNot run
Dexy-Decision-9B-V2 Open · 9.0BNot run99.8%546 of 547 answered, 1 “I don't know”Not run16.5%90 of 546Not run1.1%1 “I don't know” of 91 not rightNot run83.4%456 of 547 rightNot run0.7%Not runrun 1260Not runNot runNot runNot run
vera-medium-omni Open · 4.2BNot run98.7%540 of 547 answered, 7 “I don't know”Not run16.1%87 of 540Not run7.4%7 “I don't know” of 94 not rightNot run82.8%453 of 547 rightNot run5.7%Not runrun 1269Not runNot runNot runNot run
StartLux-Decision-v1.1-4B-Instant Open · 4.0BNot run100.0%547 of 547 answered, 0 “I don't know”Not run17.2%94 of 547Not run0.0%0 “I don't know” of 94 not rightNot run82.8%453 of 547 rightNot run3.2%Not runrun 1083Not runNot runNot runNot run
Clef-Flash Open · 9.0BNot run99.1%542 of 547 answered, 5 “I don't know”Not run16.6%90 of 542Not run5.3%5 “I don't know” of 95 not rightNot run82.6%452 of 547 rightNot run3.1%Not runrun 660Not runNot runNot runNot run
Verdin-4B-v1.1 Open · 4.0BNot run99.6%545 of 547 answered, 2 “I don't know”Not run17.2%94 of 545Not run2.1%2 “I don't know” of 96 not rightNot run82.4%451 of 547 rightNot run3.5%Not runrun 1249Not runNot runNot runNot run
Perplexity Decider Flash v1.2 Open · 4.0BNot run99.1%542 of 547 answered, 5 “I don't know”Not run18.5%100 of 542Not run4.8%5 “I don't know” of 105 not rightNot run80.8%442 of 547 rightNot run5.5%Not runrun 535Not runNot runNot runNot run
KnowLine 4B Gen5.5 Open · 4.0BNot run99.8%546 of 547 answered, 1 “I don't know”Not run19.2%105 of 546Not run0.9%1 “I don't know” of 106 not rightNot run80.6%441 of 547 rightNot run1.3%Not runrun 963Not runNot runNot runNot run
Chinese Decision Pointer 4B Open · 4.0BNot run99.3%543 of 547 answered, 4 “I don't know”Not run19.2%104 of 543Not run3.7%4 “I don't know” of 108 not rightNot run80.3%439 of 547 rightNot run2.1%Not runrun 1211Not runNot runNot runNot run
Metask-Jev-4B Open · 4.5BNot run99.6%545 of 547 answered, 2 “I don't know”Not run22.6%123 of 545Not run1.6%2 “I don't know” of 125 not rightNot run77.1%422 of 547 rightNot run3.8%Not runrun 741Not runNot runNot runNot run
KnowLine-2B-Gen1 Open · 2.0BNot run93.1%509 of 547 answered, 38 “I don't know”Not run26.5%135 of 509Not run22.0%38 “I don't know” of 173 not rightNot run68.4%374 of 547 rightNot run7.3%Not runrun 1059Not runNot runNot runNot run
GLiDE 2B Open · 1.9BNot run91.4%500 of 547 answered, 47 “I don't know”Not run26.4%132 of 500Not run26.3%47 “I don't know” of 179 not rightNot run67.3%368 of 547 rightNot run12.5%Not runrun 919Not runNot runNot runNot run
StartLux-Decision-v1.1-2B-Instant Open · 2.0BNot run94.1%515 of 547 answered, 32 “I don't know”Not run30.5%157 of 515Not run16.9%32 “I don't know” of 189 not rightNot run65.4%358 of 547 rightNot run6.7%Not runrun 975Not runNot runNot runNot run
d1-2B Open · 1.9BNot run72.4%396 of 547 answered, 151 “I don't know”Not run24.5%97 of 396Not run60.9%151 “I don't know” of 248 not rightNot run54.7%299 of 547 rightNot run26.9%Not runrun 710Not runNot runNot runNot run
GLiDE 0.8B Open · 0.8BNot run91.0%498 of 547 answered, 49 “I don't know”Not run35.5%177 of 498Not run21.7%49 “I don't know” of 226 not rightNot run58.7%321 of 547 rightNot run14.8%Not runrun 927Not runNot runNot runNot run
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BNot run97.8%535 of 547 answered, 12 “I don't know”Not run40.6%217 of 535Not run5.2%12 “I don't know” of 229 not rightNot run58.1%318 of 547 rightNot run6.6%Not runrun 1013Not runNot runNot runNot run
GLiDE 0.4B Open · 0.4BNot run72.0%394 of 547 answered, 153 “I don't know”Not run34.3%135 of 394Not run53.1%153 “I don't know” of 288 not rightNot run47.3%259 of 547 rightNot run23.3%Not runrun 935Not runNot runNot runNot run
OneJev1.6-0.8B Open · 0.8BNot run86.5%473 of 547 answered, 74 “I don't know”Not run39.5%187 of 473Not run28.4%74 “I don't know” of 261 not rightNot run52.3%286 of 547 rightNot run10.7%Not runrun 1027Not runNot runNot runNot run
KnowLine-0.8B-Gen1 Open · 0.8BNot run95.2%521 of 547 answered, 26 “I don't know”Not run41.7%217 of 521Not run10.7%26 “I don't know” of 243 not rightNot run55.6%304 of 547 rightNot run5.8%Not runrun 1003Not runNot runNot runNot run
GLiClass v4 Edge Open · 0.4BNot run90.5%495 of 547 answered, 52 “I don't know”Not run39.6%196 of 495Not run21.0%52 “I don't know” of 248 not rightNot run54.7%299 of 547 rightNot run11.8%Not runrun 1044Not runNot runNot runNot run
Strands Decider 2B Open · 2.0BNot run99.6%545 of 547 answered, 2 “I don't know”Not run55.2%301 of 545Not run0.7%2 “I don't know” of 303 not rightNot run44.6%244 of 547 rightNot run3.8%Not runrun 704Not runNot runNot runNot run
Laya Open · 0.4BNot run84.8%464 of 547 answered, 83 “I don't know”Not run79.1%367 of 464Not run18.4%83 “I don't know” of 450 not rightNot run17.7%97 of 547 rightNot run14.2%Not runrun 728Not runNot runNot runNot run
GLiNER2.5-multi-Decide Open · 0.3BNot run87.2%477 of 547 answered, 70 “I don't know”Not run76.1%363 of 477Not run16.2%70 “I don't know” of 433 not rightNot run20.8%114 of 547 rightNot run13.2%Not runrun 716Not runNot runNot runNot run
Laya Multilingual Open · 0.3BNot run62.0%339 of 547 answered, 208 “I don't know”Not run82.6%280 of 339Not run42.6%208 “I don't know” of 488 not rightNot run10.8%59 of 547 rightNot run34.7%Not runrun 734Not runNot runNot runNot run

Our method with “I don't know” in both languages

What is the same

In both languages the sixth option, F, is written “I don't know”, always the last option, never shuffled and never correct, and the instructions get the same added sentence: “If you do not know the answer, choose I don't know.” Both sides are counted on the same 547 kept questions.

What differs

The language of the question and its options (the English translation, as in the accuracy comparison above, against the Turkish original); the one word of the instructions that names the language of the question; and the options heading, “Seçenekler:” against “Options:”. The instructions of the Turkish runs:

Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose I don't know.

The instructions of the English runs:

Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.

The runs

The Turkish side is the Hallucination tab's option-word run (the option's word) on all 2,198 Turkish questions, counted on the 547 kept questions only: d1-ultra (run 511), GLiDE · v1.2 (run 525), d1.2-ultra (run 1378), Mammoth-1 (run 519), d3 (run 544), Drex Max (run 542), d3-ultra (run 540), d1.2 (run 1377), Perplexity Decider 27B v1.2 · DGX Spark (run 531), Drex Max v1.1 (run 513), SubQ Snap 1.1 (run 512) and Elephant-1 (run 515). The English side is the first English run with “I don't know” of the section above: KnowLine-31B-Gen1.5 (run 955), Copernicus-31B (run 1170), REAX 1.3 (run 831), KnowLine-31B-Gen2 (run 1329), Karpo Decision 31B (run 1161), Gemini Decision 1 · 20261111 (run 1019), Logits Sigma-1 Duo (run 1231), GLiDE 27B (run 903), d1-ultra (run 449), Sieve 31B v3 (run 1066), Turion Jev 31B (run 1077), Perplexity Decider 27B v1.2 · API · with web (run 644), GLiDE · v1.2 (run 523), d1.2-ultra (run 1363), Mammoth-1 (run 505), d3 (run 793), Drex Max (run 798), Perplexity Decider 27B v1.3 (run 1139), d3-ultra (run 795), d1.2 (run 1361), Perplexity Decider 27B v1.2 · DGX Spark (run 529), Drex Max v1.1 (run 447), Perplexity Decider 27B v1.2 · API (run 643), Jeffan-9B-1211 (run 1197), d1 (run 445), StartLux-Decision-v1.1-9B-Instant (run 1295), Clef (run 785), KnowLine-12B-Gen1 (run 1316), GLiDE 9B (run 911), OneJev-27B (run 780), Jev 1.14 (run 446), Cerno 9B v1.2 (run 1087), KnowLine-9B-Gen1 (run 1179), SubQ Snap 1.1 (run 448), Adjudex 9B v5 (run 1188), Mercury Decide (run 450), Microsoft-Decision-1 (run 451), Elephant-1 (run 501), Open-Jev-27B-v1.1 (run 763), KnowLine-9B-Gen2 (run 1323), celeris-1-decision (run 548), GPT-6 Luna · Decisions API (run 444), Cygnet (run 755), Winnow-12B (run 748), Perplexity Decider Flash v1.3 (run 1128), Valen-Decision-1.5-9B (run 1280), Dexy-Decision-9B-V2 (run 1260), vera-medium-omni (run 1269), StartLux-Decision-v1.1-4B-Instant (run 1083), Clef-Flash (run 660), Verdin-4B-v1.1 (run 1249), Perplexity Decider Flash v1.2 (run 535), KnowLine 4B Gen5.5 (run 963), Chinese Decision Pointer 4B (run 1211), Metask-Jev-4B (run 741), KnowLine-2B-Gen1 (run 1059), GLiDE 2B (run 919), StartLux-Decision-v1.1-2B-Instant (run 975), d1-2B (run 710), GLiDE 0.8B (run 927), StartLux-Decision-v1.1-0.8B-Instant (run 1013), GLiDE 0.4B (run 935), OneJev1.6-0.8B (run 1027), KnowLine-0.8B-Gen1 (run 1003), GLiClass v4 Edge (run 1044), Strands Decider 2B (run 704), Laya (run 728), GLiNER2.5-multi-Decide (run 716) and Laya Multilingual (run 734). First runs only: where a model has more runs, the others are not counted here. No run was made for this comparison.

English only, with no Turkish run with the option written “I don't know”: KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1.2 · API, Jeffan-9B-1211, d1, StartLux-Decision-v1.1-9B-Instant, Clef, KnowLine-12B-Gen1, GLiDE 9B, OneJev-27B, Jev 1.14, Cerno 9B v1.2, KnowLine-9B-Gen1, Adjudex 9B v5, Mercury Decide, Microsoft-Decision-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2, celeris-1-decision, GPT-6 Luna · Decisions API, Cygnet, Winnow-12B, Perplexity Decider Flash v1.3, Valen-Decision-1.5-9B, Dexy-Decision-9B-V2, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Clef-Flash, Verdin-4B-v1.1, Perplexity Decider Flash v1.2, KnowLine 4B Gen5.5, Chinese Decision Pointer 4B, Metask-Jev-4B, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, d1-2B, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1, GLiClass v4 Edge, Strands Decider 2B, Laya, GLiNER2.5-multi-Decide and Laya Multilingual. Their other side and paired columns read Not run; they are in no paired test and no Holm adjustment.

The test

A wrong answer is a valid choice other than “I don't know” that is not right. The paired test is the exact McNemar test on the questions only one of the two runs got wrong, with p adjusted by Holm's method over the 12 models with both runs: a family of its own, separate from every other family on the site. The comparison with “Bilmiyorum” in Turkish stays as published in the section above (its table).

Languages the model cards list

Of the 31 models whose cards or docs we checked, only Elephant-1, Mammoth-1, REAX 1.3, Copernicus-31B and vera-medium-omni name a language: English, by the card's language: en tag. Jev's docs call English its primary language. No card names Turkish. The other 103 English setups had their cards checked only for Turkish, on 16 November 2026, for the Turkish leaderboards' ‡ marks: Metask-Jev-4B and Erk-14B · reasoning off · chat list it.

Each model's card at the revision we ran (local weights) or its maker's model page and docs (APIs), searched for language statements and the Hugging Face language: field, on 2026-11-02. “not stated”: neither names a language.

ModelLanguages listedSourceChecked
d1-ultra Closed · Efficientnot stateddocs.liquid.ai2026-11-02
d1 Closed · Efficientnot stateddocs.liquid.ai2026-11-02
SubQ Snap 1.1 Closed · Efficientnot stateddocs.subq.ai2026-11-02
Elephant-1 Open · 9.0BEnglish (the card's language: en tag; its text names no language)huggingface.co2026-11-02
Mammoth-1 Open · 27BEnglish (the card's language: en tag; its text names no language)huggingface.co2026-11-02
REAX 1.3 Open · 57BEnglish (the card's language: en tag; its text names no language)huggingface.co2026-11-02
GLiDE v1.2 Open · 27Bnot statedhuggingface.co2026-11-02
GLiDE API Closed · Efficientnot stateddocs.fastino.ai2026-11-02
Perplexity Decider 27B v1.2 DGX Spark Open · 27Bnot statedhuggingface.co2026-11-02
Perplexity Decider Flash v1.2 Open · 4.0Bnot statedhuggingface.co2026-11-02
Perplexity Decider Flash v1.3 Open · 4.0Bnot statedhuggingface.co2026-11-02
Perplexity Decider 27B v1.3 Open · 27Bnot statedhuggingface.co2026-11-02
Karpo Decision 31B Open · 31Bnot statedhuggingface.co2026-11-02
Copernicus-31B Open · 31BEnglish (the card's language: en tag; its text names no language)huggingface.co2026-11-02
Adjudex 9B v5 Open · 9.0Bnot statedhuggingface.co2026-11-02
Jeffan-9B-1211 Open · 9.0Bnot statedhuggingface.co2026-11-02
Logits Sigma-1 Duo Open · 57Bnot statedhuggingface.co2026-11-02
vera-medium-omni Open · 4.2BEnglish (the card's language: en tag; its text names no language)huggingface.co2026-11-02
Dexy-Decision-9B-V2 Open · 9.0Bnot statedhuggingface.co2026-11-02
Valen-Decision-1.5-9B Open · 9.0Bnot statedhuggingface.co2026-11-02
Cerno 9B v1.2 Open · 9.0Bnot statedhuggingface.co2026-11-02
Jev 1.14 Closed · Efficient"English is the primary training language and where accuracy is currently best. Other languages, including CJK scripts, are handled but not equally well"docs.typesafe.ai2026-11-02
Microsoft-Decision-1 Closed · Efficientnot statedopenrouter.ai2026-11-02
Mercury Decide Closed · Efficientnot statedopenrouter.ai2026-11-02
celeris-1-decision Closed · Efficientnot stateddocs.celeris.ai2026-11-02
Clef Open · 27Bnot statedhuggingface.co2026-11-02
GPT-6 Luna Decisions API Closed · Efficientnot stateddevelopers.openai.com2026-11-02
Drex Max v1.1 Closed · Efficient (provisional)not statedconsole.nace.ai2026-11-02
Drex Max Open · 27Bnot statedhuggingface.co2026-11-02
d3 Open · 26Bnot statedhuggingface.co2026-11-02
d3-ultra Open · 26Bnot statedhuggingface.co2026-11-02