Hallucination

The Text tab's 2,198 questions, with “I don't know” as a sixth option

Non-hallucination, answered and error rate

Holding back by confidence

The answers each setup gave at 99% confidence or more, in its five-option run on the Text tab, the bar their share of the 2,198 questions; how many of them were wrong; and its ranking score (AUROC: how often a right answer has higher confidence than a wrong one, 0.5 chance, 1 perfect). No wrong answer at 99% first, by answers kept, then by the share wrong among the kept answers; a setup with no answers at 99% last · more kept and fewer wrong is better

Decision models

Confidence: option probabilities.

  1. Karpo Decision 31B50500.879
  2. Perplexity Decider 27B v1.2 DGX Spark36000.837
  3. Perplexity Decider 27B v1.2 API32700.843
  4. Gemini Decision 1 2026111131800.862
  5. Mammoth-128500.833
  6. Perplexity Decider 27B v1.327600.825
  7. d325000.835
  8. GLiDE 27B24700.843
  9. GLiDE v1.224000.839
  10. REAX 1.324000.868
  11. Drex Max v1.122000.834
  12. Elephant-115500.788
  13. KnowLine-31B-Gen1.514300.867
  14. GLiDE API12100.810
  15. Copernicus-31B10700.876
  16. d1-ultra9000.850
  17. KnowLine-31B-Gen28100.867
  18. KnowLine-9B-Gen16200.785
  19. Logits Sigma-1 Duo4200.855
  20. Turion Jev 31B4200.872
  21. KnowLine 4B Gen5.53900.751
  22. d1.23300.807
  23. Perplexity Decider Flash v1.32900.742
  24. GLiDE 9B2700.789
  25. Verdin-4B-v1.12700.753
  26. StartLux-Decision-v1.1-9B-Instant2000.783
  27. GLiDE 2B1900.640
  28. d1.2-ultra1100.825
  29. StartLux-Decision-v1.1-4B-Instant900.763
  30. Valen-Decision-1.5-9B700.767
  31. vera-medium-omni400.746
  32. Cygnet200.802
  33. OneJev-27B200.818
  34. Drex Max44310.826
  35. d3-ultra38410.833
  36. Perplexity Decider 27B v1.2 API · with web35110.826
  37. Sieve 31B v332910.873
  38. SubQ Snap 1.129910.836
  39. Winnow-12B16310.789
  40. Dexy-Decision-9B-V213810.773
  41. Adjudex 9B v513220.785
  42. Chinese Decision Pointer 4B5110.729
  43. GLiDE 0.8B1810.620
  44. Laya Multilingual110.454
  45. Cerno 9B v1.2No answers at this threshold––0.777
  46. ClefNo answers at this threshold––0.785
  47. Clef-FlashNo answers at this threshold––0.740
  48. d1-2BNo answers at this threshold––0.637
  49. GLiClass v4 EdgeNo answers at this threshold––0.558
  50. GLiDE 0.4BNo answers at this threshold––0.516
  51. GLiNER2.5-multi-DecideNo answers at this threshold––0.537
  52. Jeffan-9B-1211No answers at this threshold––0.798
  53. KnowLine-0.8B-Gen1No answers at this threshold––0.634
  54. KnowLine-12B-Gen1No answers at this threshold––0.810
  55. KnowLine-9B-Gen2No answers at this threshold––0.784
  56. LayaNo answers at this threshold––0.479
  57. Metask-Jev-4BNo answers at this threshold––0.725
  58. OneJev1.6-0.8BNo answers at this threshold––0.605
  59. Open-Jev-27B-v1.1No answers at this threshold––0.823
  60. Perplexity Decider Flash v1.2No answers at this threshold––0.757
  61. StartLux-Decision-v1.1-0.8B-InstantNo answers at this threshold––0.606
  62. StartLux-Decision-v1.1-2B-InstantNo answers at this threshold––0.663
  63. Strands Decider 2BNo answers at this threshold––0.647

Karpo Perplexity Gemini Mammoth vLLM-SR Fastino REAX Drex Elephant KnowLine Copernicus Liquid Logits Turion Jev Verdin StartLux Valen vera Cygnet OneJev Sieve SubQ Winnow Dexy Adjudex Chinese Decision Pointer Laya Cerno Clef GLiClass Jeffan Metask Open-Jev Strands

Chat models

Confidence: the confidence the model states for 38; option probabilities for 8; agreement across ten answers for 1.

  1. Inkling-Small high · chat · Jul 20263400.778
  2. GLM-5.3-Flash reasoning on · chat900.760
  3. GLM-5.3-Flash reasoning off · chat400.733
  4. Qwen4-Max high · chat300.857
  5. Claude Opus 5.5 low · chat · 1 per request100.882
  6. GLM-5.3 low · chat100.739
  7. Muse Ember max · chat100.800
  8. Gemini 4 Carbon high · chat1,79750.892
  9. GPT-6 Astra low · chat1,72550.918
  10. Gemini 4 Hydrogen high · chat1,43750.890
  11. GPT-6 Astra high · chat1,79370.890
  12. Gemini 3.8 Flash low · chat · 1 per request1,36560.842
  13. GPT-6 Astra medium · chat1,74980.897
  14. Gemini 3.8 Flash high · chat1,845100.833
  15. GPT-6.1 Sol high · chat1,826100.879
  16. GPT-6.1 Sol low · chat · 1 per request1,741140.859
  17. GPT-6.1 Sol low · chat1,610130.859
  18. GPT-6 Astra low · chat · 1 per request1,839150.861
  19. Gemini 4 Argon low · chat1,842230.855
  20. GPT-6.1 Terra high · chat1,363190.812
  21. Gemini 3.8 Flash low · chat976150.766
  22. GPT-6.1 Terra low · chat1,306270.820
  23. DeepSeek-V4.1-Flash reasoning off · chat · 1017option probabilities696160.828
  24. Erk-14B reasoning off · chatoption probabilities10150.754
  25. MiMo-V2.6-Flash high · chat1810.709
  26. DeepSeek-V4.1-Flash reasoning on · chat · 1017option probabilities2,1641490.614
  27. Kimi K3.1 Pro low · chat1,4361140.750
  28. Kimi K3.1 Pro no reasoning · chat442360.728
  29. Qwen3.5-397B-A17B high · chat7880.744
  30. Qwen3.8-27B stated confidence · reasoning off · chat810.515
  31. MiniMax-M3.1-Flash high · chat375540.692
  32. Qwen3.8-27B vote share · reasoning off · chatagreement across ten answers8751290.777
  33. Qwen3.8 Flash no reasoning · chat5590.660
  34. GPT-6 Luna low · chat8301370.685
  35. Gemma 4 31B reasoning off · chatoption probabilities1,7823320.833
  36. Qwen3.8-27B low · chatoption probabilities2,1455060.636
  37. Gemma 4 31B reasoning off · chat · 1 per requestoption probabilities1,8264870.820
  38. Qwen3-14B reasoning off · chatoption probabilities9753420.734
  39. Claude Fable 5.1 high · chatNo answers at this threshold––0.858
  40. Claude Haiku 5.5 max · chatNo answers at this threshold––0.819
  41. Claude Opus 5.5 high · chatNo answers at this threshold––0.837
  42. Claude Opus 5.5 low · chatNo answers at this threshold––0.840
  43. Claude Sonnet 5.5 high · chatNo answers at this threshold––0.781
  44. Claude Sonnet 5.5 high · chat · 1 per requestNo answers at this threshold––0.838
  45. Claude Sonnet 5.5 low · chatNo answers at this threshold––0.786
  46. GLiNER2.5 Multioption probabilities · No answers at this threshold––0.451
  47. Grok 4.7 high · chatNo answers at this threshold––0.849

Inkling GLM Qwen Claude Muse Gemini GPT DeepSeek Erk MiMo Kimi MiniMax Gemma Fastino Grok Confidence not measured (a plain answer line, no confidence): LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, Qwen3.5-4B · reasoning off · chat and Qwen3.5-4B · reasoning on · chat.

Holding back with an “I don't know” option

“I don't know” among the answers that were not right, with it offered as a sixth option on the 2,198 questions; beside it the share answered (right + wrong of 2,198) and the error rate among the answers (wrong of right + wrong), in the table's order · higher non-hallucination is better, read with the share answered; a model that always chose “I don't know” would reach 100% with no right answers

Decision models

  1. GLiDE 0.4B95.3%5.8%78.9%
  2. d1.2-ultra87.3%54.7%12.1%
  3. d1.279.4%55.7%20.7%
  4. d1-ultra75.1%71.9%13.0%
  5. Mammoth-168.7%68.6%20.8%
  6. d1-2B65.5%45.0%64.4%
  7. GLiDE API59.4%76.3%21.2%
  8. SubQ Snap 1.147.1%80.1%27.8%
  9. OneJev1.6-0.8B43.4%65.7%68.2%
  10. GLiDE 27B39.8%86.8%23.1%
  11. GLiDE v1.239.5%85.4%26.2%
  12. GLiDE 2B38.5%72.2%61.7%
  13. Drex Max v1.138.2%85.4%27.5%
  14. Elephant-134.8%81.6%42.3%
  15. GLiDE 0.8B29.2%78.2%67.7%
  16. Laya28.3%75.6%81.5%
  17. GLiDE 9B24.1%87.4%45.4%
  18. GLiClass v4 Edge22.3%81.9%77.0%
  19. StartLux-Decision-v1.1-2B-Instant21.5%84.8%65.0%
  20. Laya Multilingual14.8%87.3%83.8%
  21. Perplexity Decider Flash v1.214.7%91.8%51.7%
  22. StartLux-Decision-v1.1-0.8B-Instant12.4%90.9%70.5%
  23. Chinese Decision Pointer 4B11.2%93.8%52.5%
  24. Strands Decider 2B9.2%93.2%72.4%
  25. Clef-Flash9.0%95.0%53.1%
  26. vera-medium-omni8.9%95.2%51.4%
  27. Clef7.4%96.8%42.0%
  28. Perplexity Decider Flash v1.36.5%96.6%50.4%
  29. KnowLine-0.8B-Gen16.4%95.3%72.3%
  30. GLiNER2.5-multi-Decide6.4%94.8%79.9%
  31. KnowLine 4B Gen5.54.7%97.5%52.3%
  32. Jeffan-9B-12114.3%98.1%42.5%
  33. Winnow-12B4.3%98.0%44.8%
  34. Dexy-Decision-9B-V23.4%98.3%48.4%
  35. Turion Jev 31B3.3%99.3%20.0%
  36. Cygnet3.2%98.5%44.2%
  37. Perplexity Decider 27B v1.2 API · with web3.2%99.0%32.3%
  38. KnowLine-9B-Gen22.6%98.8%47.4%
  39. StartLux-Decision-v1.1-9B-Instant2.2%98.9%49.2%
  40. Gemini Decision 1 202611111.9%99.5%23.1%
  41. StartLux-Decision-v1.1-4B-Instant1.8%99.0%52.7%
  42. Open-Jev-27B-v1.11.4%99.4%44.2%
  43. d3-ultra1.4%99.5%32.2%
  44. Copernicus-31B1.2%99.8%18.3%
  45. Adjudex 9B v50.8%99.6%46.6%
  46. Metask-Jev-4B0.8%99.5%60.1%
  47. Perplexity Decider 27B v1.30.7%99.8%32.4%
  48. Karpo Decision 31B0.6%99.9%21.6%
  49. KnowLine-9B-Gen10.5%99.8%45.4%
  50. Verdin-4B-v1.10.4%99.8%53.3%
  51. REAX 1.30.4%99.9%21.6%
  52. Perplexity Decider 27B v1.2 API0.4%99.9%33.1%
  53. Logits Sigma-1 Duo0.4%99.9%24.7%
  54. Drex Max0.3%99.9%32.4%
  55. Perplexity Decider 27B v1.2 DGX Spark0.3%99.9%32.7%
  56. KnowLine-31B-Gen1.50.2%100.0%18.7%
  57. Cerno 9B v1.20.2%99.9%45.4%
  58. KnowLine-12B-Gen10.1%100.0%36.6%
  59. OneJev-27B0.1%100.0%40.6%
  60. Valen-Decision-1.5-9B0.1%100.0%52.7%
  61. KnowLine-31B-Gen20.0%100.0%18.6%
  62. Sieve 31B v30.0%100.0%25.2%
  63. d30.0%100.0%32.5%

Chose “I don't know” on fewer than 1% of the 2,198 questions: Turion Jev 31B, 15 times; Gemini Decision 1 · 20261111, 10 times; StartLux-Decision-v1.1-4B-Instant, 21 times; Open-Jev-27B-v1.1, 14 times; d3-ultra, 10 times; Copernicus-31B, 5 times; Adjudex 9B v5, 8 times; Metask-Jev-4B, 10 times; Perplexity Decider 27B v1.3, 5 times; Karpo Decision 31B, 3 times; KnowLine-9B-Gen1, 5 times; Verdin-4B-v1.1, 5 times; REAX 1.3, 2 times; Perplexity Decider 27B v1.2 · API, 3 times; Logits Sigma-1 Duo, 2 times; Drex Max, 2 times; Perplexity Decider 27B v1.2 · DGX Spark, 2 times; KnowLine-31B-Gen1.5, 1 time; Cerno 9B v1.2, 2 times; KnowLine-12B-Gen1, 1 time; OneJev-27B, 1 time; Valen-Decision-1.5-9B, 1 time; KnowLine-31B-Gen2, 0 times; Sieve 31B v3, 0 times; d3, 0 times. Their rate here is low (3.3% or less) because they almost never chose the option; the part Holding back by confidence shows how their confidence holds back wrong answers.

Non-hallucination Answered Error rate

Chat models

  1. LFM3-1.7B reasoning off · chat76.2%32.7%62.4%
  2. LFM3-1.7B reasoning on · chat67.0%45.5%58.8%
  3. LFM2.5-8B-A1B default reasoning · chat56.0%42.4%74.0%
  4. Grok 4.7 high · chat29.9%97.5%6.1%
  5. Gemma 4 E4B default reasoning · chat29.7%86.0%38.4%
  6. MiniCPM5-2B reasoning off · chat25.4%20.1%75.3%
  7. LFM2.5-2.6B default reasoning · chat24.7%81.6%63.2%
  8. Claude Haiku 5.5 max · chat22.0%67.7%9.3%
  9. Qwen3.5-2B reasoning off · chat20.8%75.6%72.3%
  10. Gemini 4 Carbon high · chat17.9%99.7%1.5%
  11. Gemini 4 Hydrogen high · chat17.4%99.1%1.7%
  12. GPT-6.1 Sol high · chat15.4%99.6%2.0%
  13. GPT-6 Astra high · chat14.3%99.7%1.9%
  14. GPT-6 Astra medium · chat12.7%99.7%2.2%
  15. GPT-6 Astra low · chat · 1 per request11.9%99.7%2.4%
  16. GPT-6.1 Sol low · chat · 1 per request11.6%99.6%2.8%
  17. GPT-6 Astra low · chat10.5%99.7%2.3%
  18. GPT-6.1 Sol low · chat9.2%99.7%2.7%
  19. GPT-6.1 Terra high · chat7.3%99.6%5.3%
  20. Gemini 4 Argon low · chat7.0%99.7%3.6%
  21. Qwen3.5-4B reasoning off · chat6.9%95.4%54.0%
  22. GLiNER2.5 Multi6.3%94.7%82.6%
  23. Gemini 3.8 Flash low · chat · 1 per request6.2%99.8%2.7%
  24. Claude Fable 5.1 high · chat6.0%99.8%3.6%
  25. Qwen3.8-27B low · chat5.9%97.9%22.9%
  26. MiniMax-M3.1-Flash high · chat5.5%97.6%32.8%
  27. Qwen3.8 Flash no reasoning · chat4.7%98.3%35.4%
  28. Gemini 3.8 Flash high · chat4.5%99.9%1.9%
  29. Gemma 4 31B reasoning off · chat4.1%98.9%26.1%
  30. MiMo-V2.6-Flash high · chat3.5%96.7%26.2%
  31. Qwen3.5-4B reasoning on · chat3.3%98.4%40.7%
  32. Inkling-Small high · chat · Jul 20263.1%99.3%21.4%
  33. GPT-6.1 Terra low · chat2.9%99.7%9.1%
  34. Claude Opus 5.5 high · chat2.7%99.9%4.9%
  35. Muse Ember max · chat2.1%97.5%13.1%
  36. Kimi K3.1 Pro low · chat2.0%99.7%15.7%
  37. Qwen3-14B reasoning off · chat1.9%99.0%53.8%
  38. Gemma 4 31B reasoning off · chat · 1 per request1.9%99.4%33.5%
  39. Gemini 3.8 Flash low · chat1.9%99.9%7.2%
  40. Claude Opus 5.5 low · chat1.8%99.9%7.7%
  41. Qwen3.5-397B-A17B high · chat1.5%99.7%17.8%
  42. Qwen3.8-27B vote share · reasoning off · chat1.3%99.4%45.6%
  43. DeepSeek-V4.1-Flash reasoning on · chat · 10171.3%99.9%7.0%
  44. GLM-5.3 low · chat1.1%99.0%23.9%
  45. GPT-6 Luna low · chat1.0%99.7%33.2%
  46. Claude Sonnet 5.5 high · chat · 1 per request0.8%99.9%11.1%
  47. Claude Opus 5.5 low · chat · 1 per request0.7%100.0%6.2%
  48. Qwen4-Max high · chat0.7%99.2%6.4%
  49. Claude Sonnet 5.5 high · chat0.7%99.9%12.8%
  50. Claude Sonnet 5.5 low · chat0.6%99.8%35.1%
  51. Qwen3.8-27B stated confidence · reasoning off · chat0.6%99.7%45.6%
  52. GLM-5.3-Flash reasoning on · chat0.6%91.9%16.6%
  53. Kimi K3.1 Pro no reasoning · chat0.5%99.9%28.3%
  54. GLM-5.3-Flash reasoning off · chat0.3%99.9%41.0%
  55. Erk-14B reasoning off · chat0.3%99.8%55.3%
  56. DeepSeek-V4.1-Flash reasoning off · chat · 10170.0%100.0%33.2%

Non-hallucination Answered Error rate

Saying “I don't know”, or held back by confidence

Each model's non-hallucination rate with “I don't know”, and its five-option run held back by its own confidence at about the same attempt rate · higher is better; beside them the run with the option's share answered and error rate among the answers

Decision models

  1. GLiDE 0.4B95.3%95.4%5.8%78.9%
  2. d1.2-ultra87.3%87.8%54.7%12.1%
  3. d1.279.4%81.3%55.7%20.7%
  4. d1-ultra75.1%78.7%71.9%13.0%
  5. Mammoth-168.7%70.9%68.6%20.8%
  6. d1-2B65.5%66.5%45.0%64.4%
  7. GLiDE API59.4%58.0%76.3%21.2%
  8. SubQ Snap 1.147.1%47.6%80.1%27.8%
  9. OneJev1.6-0.8B43.4%43.5%65.7%68.2%
  10. GLiDE 27B39.8%39.9%86.8%23.1%
  11. GLiDE v1.239.5%40.4%85.4%26.2%
  12. GLiDE 2B38.5%38.7%72.2%61.7%
  13. Drex Max v1.138.2%39.3%85.4%27.5%
  14. Elephant-134.8%35.5%81.6%42.3%
  15. GLiDE 0.8B29.2%28.9%78.2%67.7%
  16. Laya28.3%28.5%75.6%81.5%
  17. GLiDE 9B24.1%24.8%87.4%45.4%
  18. GLiClass v4 Edge22.3%22.6%81.9%77.0%
  19. StartLux-Decision-v1.1-2B-Instant21.5%21.8%84.8%65.0%
  20. Laya Multilingual14.8%15.0%87.3%83.8%
  21. Perplexity Decider Flash v1.214.7%15.2%91.8%51.7%
  22. StartLux-Decision-v1.1-0.8B-Instant12.4%12.3%90.9%70.5%
  23. Chinese Decision Pointer 4B11.2%11.3%93.8%52.5%
  24. Strands Decider 2B9.2%9.2%93.2%72.4%
  25. Clef-Flash9.0%9.0%95.0%53.1%
  26. vera-medium-omni8.9%9.2%95.2%51.4%
  27. Clef7.4%7.4%96.8%42.0%
  28. Perplexity Decider Flash v1.36.5%6.7%96.6%50.4%
  29. KnowLine-0.8B-Gen16.4%6.4%95.3%72.3%
  30. GLiNER2.5-multi-Decide6.4%6.4%94.8%79.9%
  31. KnowLine 4B Gen5.54.7%4.8%97.5%52.3%
  32. Jeffan-9B-12114.3%4.3%98.1%42.5%
  33. Winnow-12B4.3%4.3%98.0%44.8%
  34. Dexy-Decision-9B-V23.4%3.4%98.3%48.4%
  35. Turion Jev 31B3.3%3.4%99.3%20.0%
  36. Cygnet3.2%3.3%98.5%44.2%
  37. Perplexity Decider 27B v1.2 API · with web3.2%3.3%99.0%32.3%
  38. KnowLine-9B-Gen22.6%2.6%98.8%47.4%
  39. StartLux-Decision-v1.1-9B-Instant2.2%2.2%98.9%49.2%
  40. Gemini Decision 1 202611111.9%2.0%99.5%23.1%
  41. StartLux-Decision-v1.1-4B-Instant1.8%1.8%99.0%52.7%
  42. Open-Jev-27B-v1.11.4%1.4%99.4%44.2%
  43. d3-ultra1.4%1.4%99.5%32.2%
  44. Copernicus-31B1.2%1.2%99.8%18.3%
  45. Adjudex 9B v50.8%0.8%99.6%46.6%
  46. Metask-Jev-4B0.8%0.8%99.5%60.1%
  47. Perplexity Decider 27B v1.30.7%0.7%99.8%32.4%
  48. Karpo Decision 31B0.6%0.6%99.9%21.6%
  49. KnowLine-9B-Gen10.5%0.5%99.8%45.4%
  50. Verdin-4B-v1.10.4%0.4%99.8%53.3%
  51. REAX 1.30.4%0.4%99.9%21.6%
  52. Perplexity Decider 27B v1.2 API0.4%0.4%99.9%33.1%
  53. Logits Sigma-1 Duo0.4%0.4%99.9%24.7%
  54. Drex Max0.3%0.3%99.9%32.4%
  55. Perplexity Decider 27B v1.2 DGX Spark0.3%0.3%99.9%32.7%
  56. KnowLine-31B-Gen1.50.2%0.2%100.0%18.7%
  57. Cerno 9B v1.20.2%0.2%99.9%45.4%
  58. KnowLine-12B-Gen10.1%0.1%100.0%36.6%
  59. OneJev-27B0.1%0.1%100.0%40.6%
  60. Valen-Decision-1.5-9B0.1%0.1%100.0%52.7%
  61. KnowLine-31B-Gen20.0%0.0%100.0%18.6%
  62. Sieve 31B v30.0%0.0%100.0%25.2%
  63. d30.0%0.0%100.0%32.5%

With “I don't know” Held back by its own confidence (the derived rate)

Chat models

  1. Grok 4.7 high · chat29.9%26.9%97.5%6.1%
  2. Claude Haiku 5.5 max · chat22.0%83.1%67.7%9.3%
  3. Gemini 4 Carbon high · chat17.9%16.2%99.7%1.5%
  4. Gemini 4 Hydrogen high · chat17.4%37.5%99.1%1.7%
  5. GPT-6.1 Sol high · chat15.4%16.7%99.6%2.0%
  6. GPT-6 Astra high · chat14.3%14.3%99.7%1.9%
  7. GPT-6 Astra medium · chat12.7%11.9%99.7%2.2%
  8. GPT-6 Astra low · chat · 1 per request11.9%11.7%99.7%2.4%
  9. GPT-6.1 Sol low · chat · 1 per request11.6%10.5%99.6%2.8%
  10. GPT-6 Astra low · chat10.5%10.9%99.7%2.3%
  11. GPT-6.1 Sol low · chat9.2%7.9%99.7%2.7%
  12. GPT-6.1 Terra high · chat7.3%6.9%99.6%5.3%
  13. Gemini 4 Argon low · chat7.0%7.4%99.7%3.6%
  14. GLiNER2.5 Multi6.3%6.3%94.7%82.6%
  15. Gemini 3.8 Flash low · chat · 1 per request6.2%9.2%99.8%2.7%
  16. Claude Fable 5.1 high · chat6.0%8.0%99.8%3.6%
  17. Qwen3.8-27B low · chat5.9%8.6%97.9%22.9%
  18. MiniMax-M3.1-Flash high · chat5.5%6.3%97.6%32.8%
  19. Qwen3.8 Flash no reasoning · chat4.7%2.7%98.3%35.4%
  20. Gemini 3.8 Flash high · chat4.5%4.8%99.9%1.9%
  21. Gemma 4 31B reasoning off · chat4.1%4.0%98.9%26.1%
  22. MiMo-V2.6-Flash high · chat3.5%11.7%96.7%26.2%
  23. Inkling-Small high · chat · Jul 20263.1%4.7%99.3%21.4%
  24. GPT-6.1 Terra low · chat2.9%2.9%99.7%9.1%
  25. Claude Opus 5.5 high · chat2.7%0.9%99.9%4.9%
  26. Muse Ember max · chat2.1%17.0%97.5%13.1%
  27. Kimi K3.1 Pro low · chat2.0%0.8%99.7%15.7%
  28. Qwen3-14B reasoning off · chat1.9%1.9%99.0%53.8%
  29. Gemma 4 31B reasoning off · chat · 1 per request1.9%1.9%99.4%33.5%
  30. Gemini 3.8 Flash low · chat1.9%2.7%99.9%7.2%
  31. Claude Opus 5.5 low · chat1.8%0.9%99.9%7.7%
  32. Qwen3.5-397B-A17B high · chat1.5%1.3%99.7%17.8%
  33. Qwen3.8-27B vote share · reasoning off · chat1.3%2.5%99.4%45.6%
  34. DeepSeek-V4.1-Flash reasoning on · chat · 10171.3%17.7%99.9%7.0%
  35. GLM-5.3 low · chat1.1%3.2%99.0%23.9%
  36. GPT-6 Luna low · chat1.0%1.0%99.7%33.2%
  37. Claude Sonnet 5.5 high · chat · 1 per request0.8%1.3%99.9%11.1%
  38. Claude Opus 5.5 low · chat · 1 per request0.7%0.7%100.0%6.2%
  39. Qwen4-Max high · chat0.7%17.6%99.2%6.4%
  40. Claude Sonnet 5.5 high · chat0.7%0.3%99.9%12.8%
  41. Claude Sonnet 5.5 low · chat0.6%0.4%99.8%35.1%
  42. Qwen3.8-27B stated confidence · reasoning off · chat0.6%0.4%99.7%45.6%
  43. GLM-5.3-Flash reasoning on · chat0.6%37.1%91.9%16.6%
  44. Kimi K3.1 Pro no reasoning · chat0.5%0.5%99.9%28.3%
  45. GLM-5.3-Flash reasoning off · chat0.3%0.1%99.9%41.0%
  46. Erk-14B reasoning off · chat0.3%0.3%99.8%55.3%
  47. DeepSeek-V4.1-Flash reasoning off · chat · 10170.0%0.0%100.0%33.2%

With “I don't know” Held back by its own confidence (the derived rate) Held back by confidence not measured (a plain answer line, no confidence): LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, Qwen3.5-4B · reasoning off · chat and Qwen3.5-4B · reasoning on · chat.

Fewer wrong answers: “I don't know” or confidence?

Each decision model's first run with “I don't know” against its five-option run held back by its own confidence on as many questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 38 decision models that chose “I don't know” on at least 1% of the questions · the shorter bar left fewer wrong answers

  1. d1-ultra93 · 55
  2. Mammoth-166 · 35
  3. vera-medium-omni125 · 88
  4. Perplexity Decider Flash v1.2162 · 122
  5. d1.299 · 70
  6. KnowLine 4B Gen5.5103 · 74
  7. Perplexity Decider 27B v1.2 API · with web127 · 95
  8. GLiDE 9B143 · 111
  9. KnowLine-9B-Gen268 · 47
  10. StartLux-Decision-v1.1-9B-Instant67 · 48
  11. Perplexity Decider Flash v1.3102 · 80
  12. d1-2B194 · 165
  13. Elephant-1125 · 102
  14. GLiDE v1.277 · 59
  15. Drex Max v1.1125 · 103
  16. Winnow-12B94 · 75
  17. GLiDE API192 · 213
  18. GLiClass v4 Edge261 · 240
  19. Laya Multilingual261 · 240
  20. GLiDE 0.8B196 · 215
  21. Chinese Decision Pointer 4B146 · 132
  22. StartLux-Decision-v1.1-2B-Instant248 · 231
  23. d1.2-ultra44 · 37
  24. Jeffan-9B-121182 · 74
  25. Cygnet84 · 76
  26. Dexy-Decision-9B-V281 · 74
  27. Strands Decider 2B148 · 139
  28. SubQ Snap 1.1148 · 139
  29. GLiDE 2B211 · 201
  30. GLiNER2.5-multi-Decide212 · 204
  31. Laya404 · 394
  32. Clef83 · 78
  33. StartLux-Decision-v1.1-0.8B-Instant173 · 178
  34. GLiDE 27B63 · 60
  35. OneJev1.6-0.8B314 · 310
  36. GLiDE 0.4B53 · 51
  37. Clef-Flash109 · 108
  38. KnowLine-0.8B-Gen1122 · 123

Wrong only with “I don't know” Wrong only when held back by its own confidence No difference is significant (Holm-adjusted p below 0.05). Adjusted: p = 0.085, p = 0.098, p = 0.48, p = 0.72, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not in the test, as they chose “I don't know” on fewer than 1% of the questions: Turion Jev 31B, Gemini Decision 1 · 20261111, StartLux-Decision-v1.1-4B-Instant, Open-Jev-27B-v1.1, d3-ultra, Copernicus-31B, Adjudex 9B v5, Metask-Jev-4B, Perplexity Decider 27B v1.3, Karpo Decision 31B, KnowLine-9B-Gen1, Verdin-4B-v1.1, REAX 1.3, Perplexity Decider 27B v1.2 · API, Logits Sigma-1 Duo, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark, KnowLine-31B-Gen1.5, Cerno 9B v1.2, KnowLine-12B-Gen1, OneJev-27B, Valen-Decision-1.5-9B, KnowLine-31B-Gen2, Sieve 31B v3 and d3. Chat models are not in this test.

Accuracy with “I don't know”

Share of the 2,198 questions answered right with “I don't know” on offer · higher is better

Chat models

  1. Gemini 4 Carbon high · chat98.2%
  2. Gemini 3.8 Flash high · chat98.0%
  3. GPT-6 Astra high · chat97.8%
  4. GPT-6.1 Sol high · chat97.6%
  5. GPT-6 Astra medium · chat97.5%
  6. Gemini 4 Hydrogen high · chat97.4%
  7. GPT-6 Astra low · chat97.4%
  8. GPT-6 Astra low · chat · 1 per request97.3%
  9. Gemini 3.8 Flash low · chat · 1 per request97.1%
  10. GPT-6.1 Sol low · chat97.0%
  11. GPT-6.1 Sol low · chat · 1 per request96.9%
  12. Claude Fable 5.1 high · chat96.2%
  13. Gemini 4 Argon low · chat96.1%
  14. Claude Opus 5.5 high · chat94.9%
  15. GPT-6.1 Terra high · chat94.4%
  16. Claude Opus 5.5 low · chat · 1 per request93.7%
  17. DeepSeek-V4.1-Flash reasoning on · chat · 101792.9%
  18. Qwen4-Max high · chat92.9%
  19. Gemini 3.8 Flash low · chat92.7%
  20. Claude Opus 5.5 low · chat92.2%
  21. Grok 4.7 high · chat91.5%
  22. GPT-6.1 Terra low · chat90.7%
  23. Claude Sonnet 5.5 high · chat · 1 per request88.8%
  24. Claude Sonnet 5.5 high · chat87.1%
  25. Muse Ember max · chat84.8%
  26. Kimi K3.1 Pro low · chat84.1%
  27. Qwen3.5-397B-A17B high · chat82.0%
  28. Inkling-Small high · chat · Jul 202678.0%
  29. GLM-5.3-Flash reasoning on · chat76.6%
  30. Qwen3.8-27B low · chat75.4%
  31. GLM-5.3 low · chat75.4%
  32. Gemma 4 31B reasoning off · chat73.1%
  33. Kimi K3.1 Pro no reasoning · chat71.6%
  34. MiMo-V2.6-Flash high · chat71.3%
  35. DeepSeek-V4.1-Flash reasoning off · chat · 101766.8%
  36. GPT-6 Luna low · chat66.6%
  37. Gemma 4 31B reasoning off · chat · 1 per request66.1%
  38. MiniMax-M3.1-Flash high · chat65.6%
  39. Claude Sonnet 5.5 low · chat64.8%
  40. Qwen3.8 Flash no reasoning · chat63.5%
  41. Claude Haiku 5.5 max · chat61.4%
  42. GLM-5.3-Flash reasoning off · chat58.9%
  43. Qwen3.5-4B reasoning on · chat58.3%
  44. Qwen3.8-27B stated confidence · reasoning off · chat54.2%
  45. Qwen3.8-27B vote share · reasoning off · chat54.0%
  46. Gemma 4 E4B default reasoning · chat53.0%
  47. Qwen3-14B reasoning off · chat45.7%
  48. Erk-14B reasoning off · chat44.6%
  49. Qwen3.5-4B reasoning off · chat43.9%
  50. LFM2.5-2.6B default reasoning · chat30.0%
  51. Qwen3.5-2B reasoning off · chat20.9%
  52. LFM3-1.7B reasoning on · chat18.7%
  53. GLiNER2.5 Multi16.5%
  54. LFM3-1.7B reasoning off · chat12.3%
  55. LFM2.5-8B-A1B default reasoning · chat11.0%
  56. MiniCPM5-2B reasoning off · chat5.0%

Chat models: worst 5.0%best 98.2% (higher is better) The same model with five options (the Text tab)

A wrong answer counts as a hallucination: the model could have chosen “I don't know” instead.

More about this page

The English questions: the English option against the Turkish option

The 547 reading questions of the Turkish vs English tab, in English with a sixth option, “I don't know”, against the same models' answers with the Turkish option on the same questions in Turkish (this tab's runs counted on those questions, or runs on those questions only). Counts and the paired test: under Info; how the questions were translated and kept: Turkish vs English, Our method.

46 of the 67 models with both runs had a higher non-hallucination rate in English than in Turkish; 17 lower; four the same; 12 of the 67 differences in wrong answers are significant after Holm's correction: d1-2B (more wrong answers in Turkish), GLiDE 0.4B (more wrong answers in English), Laya Multilingual (more wrong answers in Turkish), OneJev1.6-0.8B (more wrong answers in Turkish), GLiDE 2B (more wrong answers in Turkish), GLiDE 0.8B (more wrong answers in Turkish), GLiClass v4 Edge (more wrong answers in Turkish), StartLux-Decision-v1.1-2B-Instant (more wrong answers in Turkish), GLiNER2.5-multi-Decide (more wrong answers in Turkish), KnowLine-0.8B-Gen1 (more wrong answers in Turkish), StartLux-Decision-v1.1-0.8B-Instant (more wrong answers in Turkish) and Metask-Jev-4B (more wrong answers in Turkish) (Paired test). celeris-1.1-decision has an English rate of 74.8% (98 “I don't know” of 131 not right) with an attempt rate of 82.1% (449 answered) but no Turkish run with the option on these questions, so no comparison or test. celeris-1-decision has an English rate of 71.3% (92 “I don't know” of 129 not right) with an attempt rate of 83.2% (455 answered) but no Turkish run with the option on these questions, so no comparison or test. KnowLine-2B-Gen1 has an English rate of 22.0% (38 “I don't know” of 173 not right) with an attempt rate of 93.1% (509 answered) but no Turkish run with the option on these questions, so no comparison or test. Read each rate with its attempt rate: a model that always chose the option would reach 100%.

The four models of the request: Holm's correction over the four

d1-ultra, SubQ Snap 1.1, Elephant-1 and Mammoth-1, as Ümit's follow-up of 2 November 2026 asked: each model's first run in English and in Turkish, each with the “I don't know” option in its language, on the 547 kept questions, and the exact McNemar test of wrong answers, with p adjusted by Holm's method over these four. None of the four differences in wrong answers is significant with Holm's correction over the four (adjusted p below 0.05); the smallest adjusted p is d1-ultra's, p = 0.077. The charts below and the Turkish vs English tab correct over all 67 compared models.

ModelEnglish non-hallucinationof answers not right, % “I don't know” · higher is better; right / wrong / optionAnswered in Englishnot “I don't know”: (right + wrong) / 547Error rate in Englishwith “I don't know”, of its answers: wrong / (right + wrong) · lower is betterTurkish non-hallucinationof answers not right, % the Turkish option · higher is better; right / wrong / optionAnswered in Turkishnot the Turkish option: (right + wrong) / 547Error rate in Turkishwith the Turkish option, of its answers: wrong / (right + wrong) · lower is betterWrong only in Englishquestions the English run got wrong and the Turkish run did notWrong only in Turkishquestions the Turkish run got wrong and the English run did notpexact McNemarHolm padjusted over the 4 models
d1-ultra Closed · Efficient56.7%457 / 39 / 5190.7%496 of 547 answered7.9%39 of 49653.6%491 / 26 / 3094.5%517 of 547 answered5.0%26 of 5172070.0190.077
SubQ Snap 1.1 Closed · Efficient25.0%463 / 63 / 2196.2%526 of 547 answered12.0%63 of 52627.1%477 / 51 / 1996.5%528 of 547 answered9.7%51 of 52839270.180.35
Elephant-1 Open · 9.0B25.6%457 / 67 / 2395.8%524 of 547 answered12.8%67 of 52415.5%450 / 82 / 1597.3%532 of 547 answered15.4%82 of 53231460.110.33
Mammoth-1 Open · 27B50.7%474 / 36 / 3793.2%510 of 547 answered7.1%36 of 51042.1%490 / 33 / 2495.6%523 of 547 answered6.3%33 of 52318150.730.73

In English, then in Turkish: the non-hallucination rate with its counts (right / wrong / option), Answered = (right + wrong) / 547 and the error rate = wrong / (right + wrong). * marks a Holm-adjusted p below 0.05 over the four.

Non-hallucination rate in Turkish and in English

The option among the answers that were not right, on the 547 kept questions: each model's first run in each language, ordered by the English rate · higher is better, read with the attempt rate (under the chart and in each bar's tooltip)

  1. d1.2-ultra76.2% · 89.3%
  2. d1.260.2% · 61.5%
  3. d1-2B43.8% · 60.9%
  4. d1-ultra53.6% · 56.7%
  5. GLiDE 0.4B89.6% · 53.1%
  6. Drex Max v1.115.0% · 51.2%
  7. Mammoth-142.1% · 50.7%
  8. GLiDE v1.211.3% · 45.5%
  9. Laya Multilingual25.7% · 42.6%
  10. GLiDE 27B18.8% · 37.9%
  11. OneJev1.6-0.8B30.7% · 28.4%
  12. GLiDE 2B24.0% · 26.3%
  13. Elephant-115.5% · 25.6%
  14. SubQ Snap 1.127.1% · 25.0%
  15. GLiDE 0.8B24.3% · 21.7%
  16. GLiClass v4 Edge14.1% · 21.0%
  17. Laya12.0% · 18.4%
  18. Open-Jev-27B-v1.17.1% · 17.4%
  19. StartLux-Decision-v1.1-2B-Instant12.3% · 16.9%
  20. GLiDE 9B9.8% · 16.7%
  21. GLiNER2.5-multi-Decide5.5% · 16.2%
  22. Perplexity Decider 27B v1.2 API · with web5.6% · 15.2%
  23. Winnow-12B8.6% · 15.0%
  24. Mercury Decide10.1% · 13.4%
  25. Cygnet5.8% · 11.2%
  26. KnowLine-0.8B-Gen14.4% · 10.7%
  27. GPT-6 Luna Decisions API7.1% · 9.4%
  28. Gemini Decision 1 2026111110.9% · 9.1%
  29. Perplexity Decider 27B v1.30.0% · 8.8%
  30. Turion Jev 31B5.7% · 8.5%
  31. d3-ultra2.1% · 8.2%
  32. vera-medium-omni4.2% · 7.4%
  33. Perplexity Decider Flash v1.37.6% · 7.2%
  34. REAX 1.32.8% · 6.7%
  35. Clef-Flash9.4% · 5.3%
  36. StartLux-Decision-v1.1-0.8B-Instant6.4% · 5.2%
  37. Perplexity Decider Flash v1.24.3% · 4.8%
  38. Perplexity Decider 27B v1.2 API0.0% · 4.5%
  39. Logits Sigma-1 Duo0.0% · 4.3%
  40. Jev 1.144.2% · 4.0%
  41. OneJev-27B1.6% · 4.0%
  42. Microsoft-Decision-12.2% · 3.8%
  43. Chinese Decision Pointer 4B3.0% · 3.7%
  44. Drex Max3.6% · 3.2%
  45. Perplexity Decider 27B v1.2 DGX Spark0.0% · 3.0%
  46. d11.1% · 2.9%
  47. StartLux-Decision-v1.1-9B-Instant1.0% · 2.6%
  48. Cerno 9B v1.20.0% · 2.5%
  49. Sieve 31B v30.0% · 2.2%
  50. Verdin-4B-v1.10.8% · 2.1%
  51. d30.0% · 1.7%
  52. Metask-Jev-4B1.1% · 1.6%
  53. Clef1.5% · 1.3%
  54. KnowLine-12B-Gen10.0% · 1.3%
  55. KnowLine-9B-Gen10.0% · 1.3%
  56. KnowLine-9B-Gen20.0% · 1.3%
  57. Valen-Decision-1.5-9B0.0% · 1.2%
  58. Dexy-Decision-9B-V21.1% · 1.1%
  59. KnowLine 4B Gen5.52.5% · 0.9%
  60. Strands Decider 2B1.5% · 0.7%
  61. Adjudex 9B v50.0% · 0.0%
  62. Copernicus-31B2.9% · 0.0%
  63. Jeffan-9B-12114.1% · 0.0%
  64. Karpo Decision 31B3.2% · 0.0%
  65. KnowLine-31B-Gen1.50.0% · 0.0%
  66. KnowLine-31B-Gen20.0% · 0.0%
  67. StartLux-Decision-v1.1-4B-Instant1.7% · 0.0%

Turkish option, the upper bar English option, the lower bar, striped Numbers on the right: Turkish · English. Liquid Fastino Drex Mammoth Laya OneJev Elephant SubQ GLiClass Open-Jev StartLux Perplexity Winnow Inception Cygnet KnowLine GPT Gemini Turion Jev vLLM-SR vera REAX Clef Logits Jev Microsoft Chinese Decision Pointer Cerno Sieve Verdin Metask Valen Dexy Strands Adjudex Copernicus Jeffan Karpo Attempt rate, Turkish · English: d1.2-ultra 88.8% · 80.1%; d1.2 90.3% · 89.2%; d1-2B 70.6% · 72.4%; d1-ultra 94.5% · 90.7%; GLiDE 0.4B 13.2% · 72.0%; Drex Max v1.1 98.4% · 92.1%; Mammoth-1 95.6% · 93.2%; GLiDE · v1.2 98.9% · 94.5%; Laya Multilingual 77.3% · 62.0%; GLiDE 27B 98.4% · 96.0%; OneJev1.6-0.8B 79.5% · 86.5%; GLiDE 2B 87.9% · 91.4%; Elephant-1 97.3% · 95.8%; SubQ Snap 1.1 96.5% · 96.2%; GLiDE 0.8B 86.5% · 91.0%; GLiClass v4 Edge 89.4% · 90.5%; Laya 90.3% · 84.8%; Open-Jev-27B-v1.1 98.9% · 97.1%; StartLux-Decision-v1.1-2B-Instant 94.0% · 94.1%; GLiDE 9B 98.4% · 97.4%; GLiNER2.5-multi-Decide 95.6% · 87.2%; Perplexity Decider 27B v1.2 · API · with web 99.5% · 98.2%; Winnow-12B 98.5% · 97.3%; Mercury Decide 98.5% · 98.0%; Cygnet 99.1% · 98.2%; KnowLine-0.8B-Gen1 97.3% · 95.2%; GPT-6 Luna · Decisions API 98.9% · 98.2%; Gemini Decision 1 · 20261111 99.1% · 99.1%; Perplexity Decider 27B v1.3 100.0% · 99.1%; Turion Jev 31B 99.6% · 99.3%; d3-ultra 99.8% · 99.1%; vera-medium-omni 99.1% · 98.7%; Perplexity Decider Flash v1.3 98.4% · 98.7%; REAX 1.3 99.8% · 99.5%; Clef-Flash 97.8% · 99.1%; StartLux-Decision-v1.1-0.8B-Instant 96.0% · 97.8%; Perplexity Decider Flash v1.2 98.9% · 99.1%; Perplexity Decider 27B v1.2 · API 100.0% · 99.5%; Logits Sigma-1 Duo 100.0% · 99.6%; Jev 1.14 99.5% · 99.5%; OneJev-27B 99.8% · 99.5%; Microsoft-Decision-1 99.6% · 99.5%; Chinese Decision Pointer 4B 99.3% · 99.3%; Drex Max 99.6% · 99.6%; Perplexity Decider 27B v1.2 · DGX Spark 100.0% · 99.6%; d1 99.8% · 99.6%; StartLux-Decision-v1.1-9B-Instant 99.8% · 99.6%; Cerno 9B v1.2 100.0% · 99.6%; Sieve 31B v3 100.0% · 99.8%; Verdin-4B-v1.1 99.8% · 99.6%; d3 100.0% · 99.8%; Metask-Jev-4B 99.6% · 99.6%; Clef 99.8% · 99.8%; KnowLine-12B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen1 100.0% · 99.8%; KnowLine-9B-Gen2 100.0% · 99.8%; Valen-Decision-1.5-9B 100.0% · 99.8%; Dexy-Decision-9B-V2 99.8% · 99.8%; KnowLine 4B Gen5.5 99.5% · 99.8%; Strands Decider 2B 99.1% · 99.6%; Adjudex 9B v5 100.0% · 100.0%; Copernicus-31B 99.8% · 100.0%; Jeffan-9B-1211 99.5% · 100.0%; Karpo Decision 31B 99.8% · 100.0%; KnowLine-31B-Gen1.5 100.0% · 100.0%; KnowLine-31B-Gen2 100.0% · 100.0%; StartLux-Decision-v1.1-4B-Instant 99.6% · 100.0%. A model that always chose the option would reach 100% with no right answers. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision and KnowLine-2B-Gen1 (in the table under Info).

Wrong only in Turkish, or only in English?

Each model's first run in each language, both with the option, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the decision models · the shorter bar is the language with fewer wrong answers

  1. GLiClass v4 Edge204 · 48 *
  2. d1-2B148 · 38 *
  3. KnowLine-0.8B-Gen1157 · 49 *
  4. StartLux-Decision-v1.1-0.8B-Instant151 · 48 *
  5. GLiDE 0.4B45 · 125 *
  6. StartLux-Decision-v1.1-2B-Instant125 · 47 *
  7. Laya Multilingual130 · 51 *
  8. GLiDE 2B125 · 48 *
  9. Metask-Jev-4B98 · 34 *
  10. OneJev1.6-0.8B121 · 55 *
  11. GLiNER2.5-multi-Decide95 · 46 *
  12. GLiDE 0.8B121 · 68 *
  13. Perplexity Decider Flash v1.269 · 36
  14. Verdin-4B-v1.170 · 37
  15. vera-medium-omni52 · 26
  16. KnowLine-9B-Gen140 · 18
  17. Clef-Flash52 · 27
  18. Valen-Decision-1.5-9B58 · 32
  19. d143 · 21
  20. StartLux-Decision-v1.1-4B-Instant57 · 33
  21. StartLux-Decision-v1.1-9B-Instant47 · 26
  22. d1-ultra7 · 20
  23. Chinese Decision Pointer 4B66 · 41
  24. Strands Decider 2B75 · 49
  25. KnowLine-9B-Gen240 · 23
  26. Perplexity Decider 27B v1.2 DGX Spark15 · 29
  27. Perplexity Decider Flash v1.357 · 37
  28. GPT-6 Luna Decisions API29 · 47
  29. Sieve 31B v311 · 23
  30. KnowLine-31B-Gen29 · 20
  31. Karpo Decision 31B13 · 25
  32. Perplexity Decider 27B v1.2 API15 · 27
  33. Elephant-146 · 31
  34. Turion Jev 31B11 · 21
  35. GLiDE v1.227 · 16
  36. GLiDE 9B38 · 25
  37. Laya120 · 97
  38. Gemini Decision 1 2026111111 · 20
  39. Cerno 9B v1.248 · 34
  40. SubQ Snap 1.127 · 39
  41. Microsoft-Decision-135 · 24
  42. Drex Max v1.130 · 20
  43. Adjudex 9B v544 · 32
  44. KnowLine 4B Gen5.553 · 40
  45. KnowLine-12B-Gen117 · 26
  46. d3-ultra23 · 33
  47. d1.2-ultra12 · 6
  48. Logits Sigma-1 Duo17 · 25
  49. Clef18 · 26
  50. REAX 1.313 · 20
  51. OneJev-27B25 · 34
  52. Copernicus-31B13 · 19
  53. Drex Max21 · 28
  54. d319 · 24
  55. Perplexity Decider 27B v1.2 API · with web21 · 26
  56. KnowLine-31B-Gen1.513 · 16
  57. GLiDE 27B18 · 15
  58. Mammoth-115 · 18
  59. Jev 1.1417 · 20
  60. d1.219 · 21
  61. Open-Jev-27B-v1.128 · 26
  62. Cygnet30 · 28
  63. Dexy-Decision-9B-V238 · 38
  64. Jeffan-9B-121131 · 31
  65. Mercury Decide29 · 29
  66. Perplexity Decider 27B v1.324 · 24
  67. Winnow-12B32 · 32

Wrong only in Turkish, the upper bar Wrong only in English, the lower bar, striped Numbers on the right: Turkish · English. GLiClass Liquid KnowLine StartLux Fastino Laya Metask OneJev Perplexity Verdin vera Clef Valen Chinese Decision Pointer Strands GPT Sieve Karpo Elephant Turion Jev Gemini Cerno SubQ Microsoft Drex Adjudex vLLM-SR Logits REAX Copernicus Mammoth Jev Open-Jev Cygnet Dexy Jeffan Inception Winnow * A significant difference (Holm-adjusted p below 0.05): GLiClass v4 Edge, d1-2B, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, StartLux-Decision-v1.1-2B-Instant, Laya Multilingual, GLiDE 2B, Metask-Jev-4B, OneJev1.6-0.8B, GLiNER2.5-multi-Decide and GLiDE 0.8B (p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p < 0.0001, p = 0.0026 and p = 0.0079). The other 55: p = 0.091, p = 0.099, p = 0.23, p = 0.28, p = 0.33, p = 0.4, p = 0.4, p = 0.71, p = 0.88, p = 0.88, p = 0.9, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with no Turkish run with the option on these questions: celeris-1.1-decision, celeris-1-decision and KnowLine-2B-Gen1 (in the table under Info).

The option's word: “I don't know” instead of “Bilmiyorum”

With “I don't know” in place of “Bilmiyorum”, 9 of the 13 models chose the option more often and 3 less often; 1 as often. Their non-hallucination rates moved by -3.0 to +15.8 points. In the exact McNemar test of wrong answers, with Holm's correction over the 13, “I don't know” left significantly fewer wrong answers for Mammoth-1, GLiDE · v1.2, Drex Max v1.1 and Elephant-1; for the other 9, the difference could be chance. Not run: Copernicus-31B, KnowLine-31B-Gen2, KnowLine-31B-Gen1.5, Turion Jev 31B, REAX 1.3, Karpo Decision 31B, Gemini Decision 1, Logits Sigma-1 Duo, Sieve 31B v3, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.2 · API, GLiDE 27B, KnowLine-12B-Gen1, OneJev-27B, Jeffan-9B-1211, Clef, Open-Jev-27B-v1.1, Cygnet, Cerno 9B v1.2, KnowLine-9B-Gen1, Winnow-12B, Adjudex 9B v5, KnowLine-9B-Gen2, Dexy-Decision-9B-V2, StartLux-Decision-v1.1-9B-Instant, Perplexity Decider Flash v1.3, GLiDE 9B, Valen-Decision-1.5-9B, StartLux-Decision-v1.1-4B-Instant, Verdin-4B-v1.1, KnowLine 4B Gen5.5, vera-medium-omni, Chinese Decision Pointer 4B, Clef-Flash, Perplexity Decider Flash v1.2, Metask-Jev-4B, StartLux-Decision-v1.1-2B-Instant, GLiDE 2B, StartLux-Decision-v1.1-0.8B-Instant, KnowLine-0.8B-Gen1, Strands Decider 2B, GLiDE 0.8B, OneJev1.6-0.8B, GLiNER2.5-multi-Decide, GLiClass v4 Edge, d1-2B, Laya Multilingual, Laya and GLiDE 0.4B (why).

The same 2,198 Turkish questions, with only the sixth option's word and the sentence that names it changed. A model that skipped “Bilmiyorum” because it did not recognise the Turkish word should choose “I don't know” more often. The languages each model's card or docs list are on the Turkish vs English tab (Languages the model cards list). Table and method: under Info.

Non-hallucination rate by the option's word

Each model's first run with “I don't know” and with “Bilmiyorum” · higher is better

  1. d1.2-ultra88.1% · 87.3%
  2. d1.276.4% · 79.4%
  3. d1-ultra74.0% · 75.1%
  4. Mammoth-173.2% · 68.7%
  5. GLiDE API63.4% · 59.4%
  6. SubQ Snap 1.152.3% · 47.1%
  7. GLiDE v1.249.9% · 39.5%
  8. Drex Max v1.154.1% · 38.2%
  9. Elephant-149.3% · 34.8%
  10. d3-ultra1.3% · 1.4%
  11. Drex Max0.3% · 0.3%
  12. Perplexity Decider 27B v1.2 DGX Spark0.5% · 0.3%
  13. d30.3% · 0.0%

With “I don't know” With “Bilmiyorum” (the table above) Read it with accuracy and the attempt rate, in the table under Info.

“I don't know” against confidence

With “I don't know” in place of “Bilmiyorum”, we compared each model's first run with “I don't know” against its five-option run held back by its own confidence at the same attempt rate (each cut-off reached the option run's answered count exactly for the 9 that chose “I don't know” on at least 1% of the questions). Confidence left fewer wrong answers for all 9, 9 to 49 fewer. In the exact McNemar test of wrong answers, with Holm's correction over the 9, confidence left significantly fewer for d1-ultra (216 wrong with “I don't know”, 167 by confidence; adjusted p = 0.00059); for the other 8, the difference could be chance. GLiDE · API is compared with its new five-option run 517, not run 129 (why).

A family of its own: the paired test with “Bilmiyorum” is the chart Fewer wrong answers above, and neither changes the other. Table and method: under Info.

Fewer wrong answers: “I don't know” or confidence?

Each model's first run with “I don't know” against its five-option run held back by its own confidence at about the same attempt rate: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 9 models that chose “I don't know” on at least 1% of the questions · the shorter bar left fewer wrong answers

  1. d1-ultra98 · 49 *
  2. Mammoth-164 · 38
  3. GLiDE v1.277 · 53
  4. Elephant-1120 · 92
  5. GLiDE API79 · 63
  6. Drex Max v1.1106 · 89
  7. d1.2107 · 90
  8. d1.2-ultra52 · 43
  9. SubQ Snap 1.1141 · 132

Wrong only with “I don't know” Wrong only when held back by its own confidence * A significant difference (Holm-adjusted p below 0.05): d1-ultra (p = 0.00059). The other 8: p = 0.1, p = 0.3, p = 0.38, p = 1, p = 1, p = 1, p = 1 and p = 1. Not in the test, as they chose “I don't know” on fewer than 1% of the questions: d3-ultra, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark and d3.

Hosted GLiDE against a new five-option run

Fastino changed its hosted model after 6 October, so run 129 (2026-10-06), the five-option run behind GLiDE · API's row in the table, is a different model from the one behind its “I don't know” run 351. A new five-option run, 517 (2026-11-02), scored 68.1% (1,497 of 2,198 right), against 67.9% for run 129. Held back by its own confidence on as many questions as run 351 said “I don't know” (521), the new run left 351 wrong answers, against 356 with “I don't know”: a derived non-hallucination rate of 59.7%, against 59.4% with the option and 58.0% from run 129 (exact McNemar test of the wrong answers, p = 0.76, not adjusted). The row in the table under Info is left as published, against run 129.

GLiDE · API: saying “I don't know”, or held back by confidence

Non-hallucination rate with “I don't know”, and with each five-option run held back by its own confidence at about the same attempt rate · higher is better

  1. With “I don't know”, run 35159.4%
  2. Confidence, new run 51759.7%
  3. Confidence, run 129 (as published)58.0%

With “I don't know” Held back by its own confidence (the derived rate) Table and method: under Info.

The same number of answers

Each model's five-option run keeps only its k most confident answers, k being the number of questions d1-ultra answered with “I don't know” on offer, and we count the wrong ones among them. In Turkish, k = 1,582, the answers of d1-ultra's run 511 with “I don't know” on the 2,198 questions: d1-ultra's five-option run (run 199) keeps 167 wrong of 1,582 (10.6%); after Holm's correction over the other 126 setups, of the 76 other decision models, 4 (Copernicus-31B, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2 and Turion Jev 31B) keep significantly fewer, 2.8 to 4.6 points fewer, 67 keep significantly more, 5.8 to 72.9 points more and for 5 (REAX 1.3 with 151, Karpo Decision 31B with 146, Gemini Decision 1 · 20261111 with 156, Logits Sigma-1 Duo with 209 and Sieve 31B v3 with 183) the difference could be chance, 3 of them keeping fewer than 167 but not significantly; of the 50 chat and general models, 27 keep significantly fewer, 3.4 to 10.3 points fewer, 19 keep significantly more, 4.6 to 72.9 points more and for 4 (Kimi K3.1 Pro · low · chat with 141, Qwen3.5-397B-A17B · high · chat with 158, Inkling-Small · high · chat · Jul 2026 with 181 and GLM-5.3-Flash · reasoning on · chat with 163) the difference could be chance, 3 of them keeping fewer than 167 but not significantly. Qwen3.5-4B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Gemma 4 E4B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Claude Haiku 5.5 · max · chat is left out: its run has 1,414 scored answers, fewer than k = 1,582; Qwen3.5-4B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-2.6B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-8B-A1B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; MiniCPM5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by. In English, k = 496, the answers of d1-ultra's run 449 with “I don't know” on the 547 kept reading questions: d1-ultra's five-option run (run 411) keeps 29 wrong of 496 (5.8%); after Holm's correction over the other 123 models, 17 keep significantly fewer, 3.8 to 5.4 points fewer, 43 keep significantly more, 4.4 to 79.2 points more and for 63 the difference could be chance; 22 of them keep fewer than 29 but not significantly. Qwen3.5-4B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Gemma 4 E4B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-4B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-2.6B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-8B-A1B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; MiniCPM5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by.

The difference from d1-ultra is tested by a paired bootstrap over questions, with p adjusted by Holm's method in each language. Each plot shows every number of answers, not only k: the line is the five-option run held back by its own confidence, the markers the model's runs with the option. Tables and method: under Info.

The same number of answers in Turkish

The 2,198 questions in Turkish: each model's five-option run answering only its most confident answers (the line), and its own runs with the option (the marker: “Bilmiyorum” “I don't know”); the table under Info has all 137 setups, the plot the 63 with runs with the option (every setup's curve under Info) · lower is better at the same coverage

0%20%40%60%80%100%0%23%46%69%92%Answered (coverage, of 2,198)Wrong of answersk = 1,582: d1-ultra's answers with “I don't know”k = 1,582Copernicus-31B: five options, run 1165, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 95 wrong (6.0%)KnowLine-31B-Gen1.5: five options, run 957, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 105 wrong (6.6%)KnowLine-31B-Gen2: five options, run 1314, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 122 wrong (7.7%)Turion Jev 31B: five options, run 1273, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 117 wrong (7.4%)REAX 1.3: five options, run 827, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 151 wrong (9.5%)Karpo Decision 31B: five options, run 1156, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 146 wrong (9.2%)Gemini Decision 1 · 20261111: five options, run 1021, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 156 wrong (9.9%)d1-ultra: five options, run 199, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 167 wrong (10.6%)Logits Sigma-1 Duo: five options, run 1221, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 209 wrong (13.2%)Sieve 31B v3: five options, run 1241, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 183 wrong (11.6%)GLiDE 27B: five options, run 899, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 259 wrong (16.4%)Perplexity Decider 27B v1.2 · API · with web: five options, run 640, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 313 wrong (19.8%)GLiDE · v1.2: five options, run 248, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 289 wrong (18.3%)Perplexity Decider 27B v1.3: five options, run 1134, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 320 wrong (20.2%)d1.2-ultra: five options, run 1347, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 323 wrong (20.4%)GLiDE · API: five options, run 129, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 325 wrong (20.5%)d3: five options, run 259, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 314 wrong (19.8%)d3-ultra: five options, run 269, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 315 wrong (19.9%)Drex Max: five options, run 322, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 321 wrong (20.3%)Drex Max v1.1: five options, run 326, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 321 wrong (20.3%)Mammoth-1: five options, run 374, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 322 wrong (20.4%)Perplexity Decider 27B v1.2 · DGX Spark: five options, run 217, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 328 wrong (20.7%)Perplexity Decider 27B v1.2 · API: five options, run 633, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 329 wrong (20.8%)SubQ Snap 1.1: five options, run 346, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 361 wrong (22.8%)KnowLine-12B-Gen1: five options, run 1309, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 418 wrong (26.4%)d1.2: five options, run 1346, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 436 wrong (27.6%)OneJev-27B: five options, run 264, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 455 wrong (28.8%)Clef: five options, run 125, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 512 wrong (32.4%)Jeffan-9B-1211: five options, run 1192, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 519 wrong (32.8%)Cygnet: five options, run 127, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 530 wrong (33.5%)Open-Jev-27B-v1.1: five options, run 45, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 504 wrong (31.9%)Winnow-12B: five options, run 128, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 540 wrong (34.1%)KnowLine-9B-Gen1: five options, run 1174, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 553 wrong (35.0%)Cerno 9B v1.2: five options, run 1201, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 559 wrong (35.3%)KnowLine-9B-Gen2: five options, run 1306, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 580 wrong (36.7%)Adjudex 9B v5: five options, run 1183, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 578 wrong (36.5%)Elephant-1: five options, run 373, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 586 wrong (37.0%)GLiDE 9B: five options, run 907, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 600 wrong (37.9%)Dexy-Decision-9B-V2: five options, run 1255, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 623 wrong (39.4%)StartLux-Decision-v1.1-9B-Instant: five options, run 1290, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 620 wrong (39.2%)Perplexity Decider Flash v1.3: five options, run 1123, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 667 wrong (42.2%)vera-medium-omni: five options, run 1264, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 665 wrong (42.0%)KnowLine 4B Gen5.5: five options, run 965, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 681 wrong (43.0%)StartLux-Decision-v1.1-4B-Instant: five options, run 1236, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 674 wrong (42.6%)Perplexity Decider Flash v1.2: five options, run 223, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 691 wrong (43.7%)Valen-Decision-1.5-9B: five options, run 1275, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 698 wrong (44.1%)Verdin-4B-v1.1: five options, run 1244, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 708 wrong (44.8%)Chinese Decision Pointer 4B: five options, run 1206, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 723 wrong (45.7%)Clef-Flash: five options, run 139, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 716 wrong (45.3%)Metask-Jev-4B: five options, run 126, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 841 wrong (53.2%)GLiDE 2B: five options, run 915, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 965 wrong (61.0%)StartLux-Decision-v1.1-2B-Instant: five options, run 977, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 972 wrong (61.4%)d1-2B: five options, run 340, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,060 wrong (67.0%)GLiDE 0.8B: five options, run 923, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,080 wrong (68.3%)OneJev1.6-0.8B: five options, run 1029, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,072 wrong (67.8%)StartLux-Decision-v1.1-0.8B-Instant: five options, run 1015, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,098 wrong (69.4%)Strands Decider 2B: five options, run 141, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,096 wrong (69.3%)KnowLine-0.8B-Gen1: five options, run 1005, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,107 wrong (70.0%)GLiClass v4 Edge: five options, run 1046, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,197 wrong (75.7%)GLiDE 0.4B: five options, run 931, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,211 wrong (76.5%)GLiNER2.5-multi-Decide: five options, run 135, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,248 wrong (78.9%)Laya: five options, run 51, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,273 wrong (80.5%)Laya Multilingual: five options, run 46, held back by confidence, from 1 to 2,198 answers; its 1,582 most confident answers: 1,321 wrong (83.5%)Copernicus-31B: “Bilmiyorum”, run 1172: 2,193 answered (99.8%), 402 wrong (18.3%)KnowLine-31B-Gen1.5: “Bilmiyorum”, run 959: 2,197 answered (100.0%), 411 wrong (18.7%)KnowLine-31B-Gen2: “Bilmiyorum”, run 1331: 2,198 answered (100.0%), 408 wrong (18.6%)Turion Jev 31B: “Bilmiyorum”, run 1285: 2,183 answered (99.3%), 436 wrong (20.0%)REAX 1.3: “Bilmiyorum”, run 833: 2,196 answered (99.9%), 475 wrong (21.6%)Karpo Decision 31B: “Bilmiyorum”, run 1163: 2,195 answered (99.9%), 475 wrong (21.6%)Gemini Decision 1 · 20261111: “Bilmiyorum”, run 1023: 2,188 answered (99.5%), 505 wrong (23.1%)d1-ultra: “Bilmiyorum”, run 350: 1,581 answered (71.9%), 205 wrong (13.0%)d1-ultra: “I don't know”, run 511: 1,582 answered (72.0%), 216 wrong (13.7%)Logits Sigma-1 Duo: “Bilmiyorum”, run 1233: 2,196 answered (99.9%), 543 wrong (24.7%)Sieve 31B v3: “Bilmiyorum”, run 1253: 2,198 answered (100.0%), 553 wrong (25.2%)GLiDE 27B: “Bilmiyorum”, run 905: 1,907 answered (86.8%), 441 wrong (23.1%)Perplexity Decider 27B v1.2 · API · with web: “Bilmiyorum”, run 642: 2,175 answered (99.0%), 702 wrong (32.3%)GLiDE · v1.2: “Bilmiyorum”, run 369: 1,877 answered (85.4%), 491 wrong (26.2%)GLiDE · v1.2: “I don't know”, run 525: 1,772 answered (80.6%), 427 wrong (24.1%)Perplexity Decider 27B v1.3: “Bilmiyorum”, run 1141: 2,193 answered (99.8%), 711 wrong (32.4%)d1.2-ultra: “Bilmiyorum”, run 1367: 1,202 answered (54.7%), 145 wrong (12.1%)d1.2-ultra: “I don't know”, run 1378: 1,180 answered (53.7%), 137 wrong (11.6%)GLiDE · API: “Bilmiyorum”, run 351: 1,677 answered (76.3%), 356 wrong (21.2%); made with Fastino's engine glide_v2, not run 129's glideGLiDE · API: “I don't know”, run 510: 1,617 answered (73.6%), 335 wrong (20.7%); made with Fastino's engine glide_v2, not run 129's glided3: “Bilmiyorum”, run 358: 2,198 answered (100.0%), 715 wrong (32.5%)d3: “I don't know”, run 544: 2,196 answered (99.9%), 714 wrong (32.5%)d3-ultra: “Bilmiyorum”, run 363: 2,188 answered (99.5%), 704 wrong (32.2%)d3-ultra: “I don't know”, run 540: 2,189 answered (99.6%), 704 wrong (32.2%)Drex Max: “Bilmiyorum”, run 367: 2,196 answered (99.9%), 711 wrong (32.4%)Drex Max: “I don't know”, run 542: 2,196 answered (99.9%), 712 wrong (32.4%)Drex Max v1.1: “Bilmiyorum”, run 352: 1,878 answered (85.4%), 517 wrong (27.5%)Drex Max v1.1: “I don't know”, run 513: 1,712 answered (77.9%), 413 wrong (24.1%)Mammoth-1: “Bilmiyorum”, run 386: 1,508 answered (68.6%), 314 wrong (20.8%)Mammoth-1: “I don't know”, run 519: 1,445 answered (65.7%), 276 wrong (19.1%)Perplexity Decider 27B v1.2 · DGX Spark: “Bilmiyorum”, run 372: 2,196 answered (99.9%), 717 wrong (32.7%)Perplexity Decider 27B v1.2 · DGX Spark: “I don't know”, run 531: 2,194 answered (99.8%), 728 wrong (33.2%)Perplexity Decider 27B v1.2 · API: “Bilmiyorum”, run 641: 2,195 answered (99.9%), 726 wrong (33.1%)SubQ Snap 1.1: “Bilmiyorum”, run 392: 1,761 answered (80.1%), 490 wrong (27.8%)SubQ Snap 1.1: “I don't know”, run 512: 1,700 answered (77.3%), 454 wrong (26.7%)KnowLine-12B-Gen1: “Bilmiyorum”, run 1318: 2,197 answered (100.0%), 804 wrong (36.6%)d1.2: “Bilmiyorum”, run 1365: 1,225 answered (55.7%), 253 wrong (20.7%)d1.2: “I don't know”, run 1377: 1,297 answered (59.0%), 279 wrong (21.5%)OneJev-27B: “Bilmiyorum”, run 782: 2,197 answered (100.0%), 892 wrong (40.6%)Clef: “Bilmiyorum”, run 788: 2,127 answered (96.8%), 894 wrong (42.0%)Jeffan-9B-1211: “Bilmiyorum”, run 1199: 2,157 answered (98.1%), 917 wrong (42.5%)Cygnet: “Bilmiyorum”, run 757: 2,166 answered (98.5%), 957 wrong (44.2%)Open-Jev-27B-v1.1: “Bilmiyorum”, run 768: 2,184 answered (99.4%), 965 wrong (44.2%)Winnow-12B: “Bilmiyorum”, run 750: 2,155 answered (98.0%), 965 wrong (44.8%)KnowLine-9B-Gen1: “Bilmiyorum”, run 1181: 2,193 answered (99.8%), 996 wrong (45.4%)Cerno 9B v1.2: “Bilmiyorum”, run 1204: 2,196 answered (99.9%), 998 wrong (45.4%)KnowLine-9B-Gen2: “Bilmiyorum”, run 1325: 2,171 answered (98.8%), 1,030 wrong (47.4%)Adjudex 9B v5: “Bilmiyorum”, run 1190: 2,190 answered (99.6%), 1,021 wrong (46.6%)Elephant-1: “Bilmiyorum”, run 382: 1,793 answered (81.6%), 758 wrong (42.3%)Elephant-1: “I don't know”, run 515: 1,594 answered (72.5%), 622 wrong (39.0%)GLiDE 9B: “Bilmiyorum”, run 913: 1,921 answered (87.4%), 872 wrong (45.4%)Dexy-Decision-9B-V2: “Bilmiyorum”, run 1262: 2,161 answered (98.3%), 1,046 wrong (48.4%)StartLux-Decision-v1.1-9B-Instant: “Bilmiyorum”, run 1297: 2,174 answered (98.9%), 1,069 wrong (49.2%)Perplexity Decider Flash v1.3: “Bilmiyorum”, run 1130: 2,123 answered (96.6%), 1,071 wrong (50.4%)vera-medium-omni: “Bilmiyorum”, run 1271: 2,093 answered (95.2%), 1,075 wrong (51.4%)KnowLine 4B Gen5.5: “Bilmiyorum”, run 967: 2,143 answered (97.5%), 1,121 wrong (52.3%)StartLux-Decision-v1.1-4B-Instant: “Bilmiyorum”, run 1239: 2,177 answered (99.0%), 1,147 wrong (52.7%)Perplexity Decider Flash v1.2: “Bilmiyorum”, run 791: 2,018 answered (91.8%), 1,044 wrong (51.7%)Valen-Decision-1.5-9B: “Bilmiyorum”, run 1282: 2,197 answered (100.0%), 1,157 wrong (52.7%)Verdin-4B-v1.1: “Bilmiyorum”, run 1251: 2,193 answered (99.8%), 1,169 wrong (53.3%)Chinese Decision Pointer 4B: “Bilmiyorum”, run 1213: 2,062 answered (93.8%), 1,082 wrong (52.5%)Clef-Flash: “Bilmiyorum”, run 664: 2,088 answered (95.0%), 1,108 wrong (53.1%)Metask-Jev-4B: “Bilmiyorum”, run 743: 2,188 answered (99.5%), 1,315 wrong (60.1%)GLiDE 2B: “Bilmiyorum”, run 921: 1,586 answered (72.2%), 978 wrong (61.7%)StartLux-Decision-v1.1-2B-Instant: “Bilmiyorum”, run 979: 1,865 answered (84.8%), 1,213 wrong (65.0%)d1-2B: “Bilmiyorum”, run 712: 989 answered (45.0%), 637 wrong (64.4%)GLiDE 0.8B: “Bilmiyorum”, run 929: 1,718 answered (78.2%), 1,163 wrong (67.7%)OneJev1.6-0.8B: “Bilmiyorum”, run 1031: 1,444 answered (65.7%), 985 wrong (68.2%)StartLux-Decision-v1.1-0.8B-Instant: “Bilmiyorum”, run 1017: 1,999 answered (90.9%), 1,409 wrong (70.5%)Strands Decider 2B: “Bilmiyorum”, run 706: 2,048 answered (93.2%), 1,483 wrong (72.4%)KnowLine-0.8B-Gen1: “Bilmiyorum”, run 1007: 2,094 answered (95.3%), 1,515 wrong (72.3%)GLiClass v4 Edge: “Bilmiyorum”, run 1048: 1,800 answered (81.9%), 1,386 wrong (77.0%)GLiDE 0.4B: “Bilmiyorum”, run 937: 128 answered (5.8%), 101 wrong (78.9%)GLiNER2.5-multi-Decide: “Bilmiyorum”, run 718: 2,084 answered (94.8%), 1,666 wrong (79.9%)Laya: “Bilmiyorum”, run 730: 1,662 answered (75.6%), 1,355 wrong (81.5%)Laya Multilingual: “Bilmiyorum”, run 737: 1,918 answered (87.3%), 1,608 wrong (83.8%)
  • k = 1,582: d1-ultra's answers with “I don't know”
  • Copernicus-31B: five options, run 1165 · “Bilmiyorum”, run 1172
  • KnowLine-31B-Gen1.5: five options, run 957 · “Bilmiyorum”, run 959
  • KnowLine-31B-Gen2: five options, run 1314 · “Bilmiyorum”, run 1331
  • Turion Jev 31B: five options, run 1273 · “Bilmiyorum”, run 1285
  • REAX 1.3: five options, run 827 · “Bilmiyorum”, run 833
  • Karpo Decision 31B: five options, run 1156 · “Bilmiyorum”, run 1163
  • Gemini Decision 1 · 20261111: five options, run 1021 · “Bilmiyorum”, run 1023
  • d1-ultra: five options, run 199 · “Bilmiyorum”, run 350 · “I don't know”, run 511
  • Logits Sigma-1 Duo: five options, run 1221 · “Bilmiyorum”, run 1233
  • Sieve 31B v3: five options, run 1241 · “Bilmiyorum”, run 1253
  • GLiDE 27B: five options, run 899 · “Bilmiyorum”, run 905
  • Perplexity Decider 27B v1.2 · API · with web: five options, run 640 · “Bilmiyorum”, run 642
  • GLiDE · v1.2: five options, run 248 · “Bilmiyorum”, run 369 · “I don't know”, run 525
  • Perplexity Decider 27B v1.3: five options, run 1134 · “Bilmiyorum”, run 1141
  • d1.2-ultra: five options, run 1347 · “Bilmiyorum”, run 1367 · “I don't know”, run 1378
  • GLiDE · API: five options, run 129 · “Bilmiyorum”, run 351 · “I don't know”, run 510 (runs 351 and 510 made with Fastino's engine glide_v2, not run 129's glide)
  • d3: five options, run 259 · “Bilmiyorum”, run 358 · “I don't know”, run 544
  • d3-ultra: five options, run 269 · “Bilmiyorum”, run 363 · “I don't know”, run 540
  • Drex Max: five options, run 322 · “Bilmiyorum”, run 367 · “I don't know”, run 542
  • Drex Max v1.1: five options, run 326 · “Bilmiyorum”, run 352 · “I don't know”, run 513
  • Mammoth-1: five options, run 374 · “Bilmiyorum”, run 386 · “I don't know”, run 519
  • Perplexity Decider 27B v1.2 · DGX Spark: five options, run 217 · “Bilmiyorum”, run 372 · “I don't know”, run 531
  • Perplexity Decider 27B v1.2 · API: five options, run 633 · “Bilmiyorum”, run 641
  • SubQ Snap 1.1: five options, run 346 · “Bilmiyorum”, run 392 · “I don't know”, run 512
  • KnowLine-12B-Gen1: five options, run 1309 · “Bilmiyorum”, run 1318
  • d1.2: five options, run 1346 · “Bilmiyorum”, run 1365 · “I don't know”, run 1377
  • OneJev-27B: five options, run 264 · “Bilmiyorum”, run 782
  • Clef: five options, run 125 · “Bilmiyorum”, run 788
  • Jeffan-9B-1211: five options, run 1192 · “Bilmiyorum”, run 1199
  • Cygnet: five options, run 127 · “Bilmiyorum”, run 757
  • Open-Jev-27B-v1.1: five options, run 45 · “Bilmiyorum”, run 768
  • Winnow-12B: five options, run 128 · “Bilmiyorum”, run 750
  • KnowLine-9B-Gen1: five options, run 1174 · “Bilmiyorum”, run 1181
  • Cerno 9B v1.2: five options, run 1201 · “Bilmiyorum”, run 1204
  • KnowLine-9B-Gen2: five options, run 1306 · “Bilmiyorum”, run 1325
  • Adjudex 9B v5: five options, run 1183 · “Bilmiyorum”, run 1190
  • Elephant-1: five options, run 373 · “Bilmiyorum”, run 382 · “I don't know”, run 515
  • GLiDE 9B: five options, run 907 · “Bilmiyorum”, run 913
  • Dexy-Decision-9B-V2: five options, run 1255 · “Bilmiyorum”, run 1262
  • StartLux-Decision-v1.1-9B-Instant: five options, run 1290 · “Bilmiyorum”, run 1297
  • Perplexity Decider Flash v1.3: five options, run 1123 · “Bilmiyorum”, run 1130
  • vera-medium-omni: five options, run 1264 · “Bilmiyorum”, run 1271
  • KnowLine 4B Gen5.5: five options, run 965 · “Bilmiyorum”, run 967
  • StartLux-Decision-v1.1-4B-Instant: five options, run 1236 · “Bilmiyorum”, run 1239
  • Perplexity Decider Flash v1.2: five options, run 223 · “Bilmiyorum”, run 791
  • Valen-Decision-1.5-9B: five options, run 1275 · “Bilmiyorum”, run 1282
  • Verdin-4B-v1.1: five options, run 1244 · “Bilmiyorum”, run 1251
  • Chinese Decision Pointer 4B: five options, run 1206 · “Bilmiyorum”, run 1213
  • Clef-Flash: five options, run 139 · “Bilmiyorum”, run 664
  • Metask-Jev-4B: five options, run 126 · “Bilmiyorum”, run 743
  • GLiDE 2B: five options, run 915 · “Bilmiyorum”, run 921
  • StartLux-Decision-v1.1-2B-Instant: five options, run 977 · “Bilmiyorum”, run 979
  • d1-2B: five options, run 340 · “Bilmiyorum”, run 712
  • GLiDE 0.8B: five options, run 923 · “Bilmiyorum”, run 929
  • OneJev1.6-0.8B: five options, run 1029 · “Bilmiyorum”, run 1031
  • StartLux-Decision-v1.1-0.8B-Instant: five options, run 1015 · “Bilmiyorum”, run 1017
  • Strands Decider 2B: five options, run 141 · “Bilmiyorum”, run 706
  • KnowLine-0.8B-Gen1: five options, run 1005 · “Bilmiyorum”, run 1007
  • GLiClass v4 Edge: five options, run 1046 · “Bilmiyorum”, run 1048
  • GLiDE 0.4B: five options, run 931 · “Bilmiyorum”, run 937
  • GLiNER2.5-multi-Decide: five options, run 135 · “Bilmiyorum”, run 718
  • Laya: five options, run 51 · “Bilmiyorum”, run 730
  • Laya Multilingual: five options, run 46 · “Bilmiyorum”, run 737

The same number of answers in English

The 547 kept reading questions in English: each model's five-option run answering only its most confident answers (the line), and its own runs with the option (the marker: “I don't know”); not drawn: Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat and Qwen3.5-2B · reasoning off · chat, not measured (a plain answer line, no confidence) · lower is better at the same coverage

0%20%40%60%80%100%0%25%50%75%100%Answered (coverage, of 547)Wrong of answersk = 496: d1-ultra's answers with “I don't know”k = 496Claude Fable 5.2 · max · chat: five options, run 971, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 2 wrong (0.4%)Gemini 4 Hydrogen · high · chat: five options, run 1107, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 3 wrong (0.6%)GPT-6 Astra · low · chat: five options, run 616, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 4 wrong (0.8%)GPT-6 Astra · high · chat: five options, run 615, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 5 wrong (1.0%)GPT-6.1 Terra · high · chat: five options, run 620, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 8 wrong (1.6%)GPT-6.1 Terra · low · chat: five options, run 621, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 8 wrong (1.6%)Claude Fable 5.1 · high · chat: five options, run 580, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 10 wrong (2.0%)Gemini 3.8 Flash · high · chat: five options, run 583, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 9 wrong (1.8%)Gemini 4 Argon · low · chat: five options, run 593, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 7 wrong (1.4%)GPT-6 Astra · medium · chat: five options, run 624, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 5 wrong (1.0%)GPT-6.1 Sol · low · chat: five options, run 619, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 6 wrong (1.2%)GPT-6 Astra · low · chat · 1 per request: five options, run 622, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 7 wrong (1.4%)GPT-6.1 Sol · high · chat: five options, run 618, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 6 wrong (1.2%)Gemini 3.8 Flash · low · chat · 1 per request: five options, run 589, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 10 wrong (2.0%)GPT-6.1 Sol · low · chat · 1 per request: five options, run 623, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 8 wrong (1.6%)Claude Opus 5.5 · low · chat · 1 per request: five options, run 592, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 10 wrong (2.0%)Claude Opus 5.5 · high · chat: five options, run 587, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 13 wrong (2.6%)Qwen4-Max · high · chat: five options, run 1334, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 14 wrong (2.8%)Gemini 3.8 Flash · low · chat: five options, run 578, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 14 wrong (2.8%)Gemini 4 Carbon · high · chat: five options, run 1055, held back by confidence, from 1 to 531 answers; its 496 most confident answers: 2 wrong (0.4%)KnowLine-31B-Gen1.5: five options, run 953, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 14 wrong (2.8%)Grok 4.7 · high · chat: five options, run 579, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 14 wrong (2.8%)Claude Opus 5.5 · low · chat: five options, run 588, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 19 wrong (3.8%)Copernicus-31B: five options, run 1168, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 16 wrong (3.2%)Claude Haiku 5.5 · max · chat: five options, run 949, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 20 wrong (4.0%)DeepSeek-V4.1-Flash · reasoning on · chat · 1017: five options, run 808, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 36 wrong (7.3%)REAX 1.3: five options, run 829, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 16 wrong (3.2%)Kimi K3.1 Pro · low · chat: five options, run 584, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 24 wrong (4.8%)KnowLine-31B-Gen2: five options, run 1327, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 16 wrong (3.2%)Karpo Decision 31B: five options, run 1159, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 22 wrong (4.4%)Claude Sonnet 5.5 · high · chat · 1 per request: five options, run 594, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 22 wrong (4.4%)Kimi K3.1 Pro · no reasoning · chat: five options, run 582, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 30 wrong (6.0%)Qwen3.5-397B-A17B · high · chat: five options, run 871, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 26 wrong (5.2%)Gemini Decision 1 · 20261111: five options, run 1009, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 25 wrong (5.0%)Logits Sigma-1 Duo: five options, run 1229, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 23 wrong (4.6%)GLiDE 27B: five options, run 901, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 27 wrong (5.4%)d1-ultra: five options, run 411, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 29 wrong (5.8%)Sieve 31B v3: five options, run 1064, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 24 wrong (4.8%)Turion Jev 31B: five options, run 1075, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 23 wrong (4.6%)Gemma 4 31B · reasoning off · chat: five options, run 774, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 29 wrong (5.8%)Perplexity Decider 27B v1.2 · API · with web: five options, run 639, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 37 wrong (7.5%)GLiDE · v1.2: five options, run 521, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 26 wrong (5.2%)d1.2-ultra: five options, run 1359, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 24 wrong (4.8%)Mammoth-1: five options, run 503, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 31 wrong (6.2%)Gemma 4 31B · reasoning off · chat · 1 per request: five options, run 783, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 36 wrong (7.3%)d3: five options, run 554, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 29 wrong (5.8%)Drex Max: five options, run 552, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 28 wrong (5.6%)GLM-5.3-Flash · reasoning on · chat: five options, run 823, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 34 wrong (6.9%)Muse Ember · max · chat: five options, run 942, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 36 wrong (7.3%)Perplexity Decider 27B v1.3: five options, run 1137, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 35 wrong (7.1%)d3-ultra: five options, run 550, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 31 wrong (6.2%)d1.2: five options, run 1357, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 31 wrong (6.2%)MiMo-V2.6-Flash · high · chat: five options, run 878, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 35 wrong (7.1%)Qwen3.8-27B · low · chat: five options, run 667, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 48 wrong (9.7%)Perplexity Decider 27B v1.2 · DGX Spark: five options, run 527, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 34 wrong (6.9%)Drex Max v1.1: five options, run 399, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 32 wrong (6.5%)Perplexity Decider 27B v1: five options, run 626, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 35 wrong (7.1%)Perplexity Decider 27B v1.2 · API: five options, run 637, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 36 wrong (7.3%)Inkling-Small · high · chat · Jul 2026: five options, run 867, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 44 wrong (8.9%)d1.1: five options, run 565, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 42 wrong (8.5%)GLM-5.3 · low · chat: five options, run 577, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 49 wrong (9.9%)Jeffan-9B-1211: five options, run 1195, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 46 wrong (9.3%)GLM-5.3-Flash · reasoning off · chat: five options, run 820, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 50 wrong (10.1%)DeepSeek-V4.1-Flash · reasoning off · chat · 1017: five options, run 804, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 43 wrong (8.7%)Qwen3.8 Flash · no reasoning · chat: five options, run 986, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 55 wrong (11.1%)d1: five options, run 430, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 44 wrong (8.9%)StartLux-Decision-v1.1-9B-Instant: five options, run 1293, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 47 wrong (9.5%)Clef: five options, run 613, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 44 wrong (8.9%)KnowLine-12B-Gen1: five options, run 1313, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 43 wrong (8.7%)GLiDE 9B: five options, run 909, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 46 wrong (9.3%)GPT-6 Luna · low · chat: five options, run 617, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 56 wrong (11.3%)OneJev-27B: five options, run 778, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 47 wrong (9.5%)Claude Sonnet 5.5 · high · chat: five options, run 590, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 43 wrong (8.7%)Jev 1.14: five options, run 410, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 44 wrong (8.9%)Cerno 9B v1.2: five options, run 1085, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 52 wrong (10.5%)Drex 2.0: five options, run 566, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 51 wrong (10.3%)Jev 1.13: five options, run 564, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 45 wrong (9.1%)KnowLine-9B-Gen1: five options, run 1177, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 49 wrong (9.9%)SubQ Snap 1.1: five options, run 413, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 44 wrong (8.9%)SubQ Snap Preview: five options, run 595, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 44 wrong (8.9%)Adjudex 9B v5: five options, run 1186, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 48 wrong (9.7%)Mercury Decide: five options, run 398, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 50 wrong (10.1%)Microsoft-Decision-1: five options, run 412, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 52 wrong (10.5%)Elephant-1: five options, run 499, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 55 wrong (11.1%)Open-Jev-27B-v1.1: five options, run 760, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 48 wrong (9.7%)KnowLine-9B-Gen2: five options, run 1321, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 54 wrong (10.9%)Qwen3.8-27B · stated confidence · reasoning off · chat: five options, run 672, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 75 wrong (15.1%)Qwen3.8-27B · yes/no scoring: five options, run 696, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 54 wrong (10.9%)MiniMax-M3.1-Flash · high · chat: five options, run 863, held back by confidence, from 1 to 546 answers; its 496 most confident answers: 58 wrong (11.7%)celeris-1-decision: five options, run 546, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 56 wrong (11.3%)GPT-6 Luna · Decisions API: five options, run 397, held back by confidence, from 1 to 545 answers; its 496 most confident answers: 58 wrong (11.7%)Claude Sonnet 5.5 · low · chat: five options, run 591, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 59 wrong (11.9%)Cygnet: five options, run 753, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 60 wrong (12.1%)Qwen3.8-27B · vote share · reasoning off · chat: five options, run 678, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 51 wrong (10.3%)Winnow-12B: five options, run 746, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 60 wrong (12.1%)Perplexity Decider Flash v1.3: five options, run 1126, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 59 wrong (11.9%)Valen-Decision-1.5-9B: five options, run 1278, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 57 wrong (11.5%)Dexy-Decision-9B-V2: five options, run 1258, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 61 wrong (12.3%)vera-medium-omni: five options, run 1267, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 59 wrong (11.9%)StartLux-Decision-v1.1-4B-Instant: five options, run 1081, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 62 wrong (12.5%)Clef-Flash: five options, run 614, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 67 wrong (13.5%)Verdin-4B-v1.1: five options, run 1247, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 63 wrong (12.7%)Perplexity Decider Flash v1.2: five options, run 533, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 62 wrong (12.5%)KnowLine 4B Gen5.5: five options, run 961, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 74 wrong (14.9%)Chinese Decision Pointer 4B: five options, run 1209, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 74 wrong (14.9%)Qwen3-14B · reasoning off · chat: five options, run 770, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 77 wrong (15.5%)Metask-Jev-4B: five options, run 739, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 90 wrong (18.1%)Erk-14B · reasoning off · chat: five options, run 762, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 117 wrong (23.6%)KnowLine-2B-Gen1: five options, run 1057, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 123 wrong (24.8%)GLiDE 2B: five options, run 917, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 125 wrong (25.2%)StartLux-Decision-v1.1-2B-Instant: five options, run 973, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 129 wrong (26.0%)d1-2B: five options, run 708, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 146 wrong (29.4%)GLiDE 0.8B: five options, run 925, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 182 wrong (36.7%)StartLux-Decision-v1.1-0.8B-Instant: five options, run 1011, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 186 wrong (37.5%)GLiDE 0.4B: five options, run 933, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 181 wrong (36.5%)OneJev1.6-0.8B: five options, run 1025, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 188 wrong (37.9%)KnowLine-0.8B-Gen1: five options, run 1001, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 188 wrong (37.9%)GLiClass v4 Edge: five options, run 1042, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 195 wrong (39.3%)Decider 1: five options, run 596, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 210 wrong (42.3%)Strands Decider 2B: five options, run 702, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 255 wrong (51.4%)Laya: five options, run 726, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 381 wrong (76.8%)GLiNER2.5-multi-Decide: five options, run 714, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 388 wrong (78.2%)GLiNER2.5 Multi: five options, run 720, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 413 wrong (83.3%)Laya Multilingual: five options, run 732, held back by confidence, from 1 to 547 answers; its 496 most confident answers: 422 wrong (85.1%)KnowLine-31B-Gen1.5: “I don't know”, run 955: 547 answered (100.0%), 37 wrong (6.8%)Copernicus-31B: “I don't know”, run 1170: 547 answered (100.0%), 39 wrong (7.1%)REAX 1.3: “I don't know”, run 831: 544 answered (99.5%), 42 wrong (7.7%)KnowLine-31B-Gen2: “I don't know”, run 1329: 547 answered (100.0%), 40 wrong (7.3%)Karpo Decision 31B: “I don't know”, run 1161: 547 answered (100.0%), 42 wrong (7.7%)Gemini Decision 1 · 20261111: “I don't know”, run 1019: 542 answered (99.1%), 50 wrong (9.2%)Logits Sigma-1 Duo: “I don't know”, run 1231: 545 answered (99.6%), 44 wrong (8.1%)GLiDE 27B: “I don't know”, run 903: 525 answered (96.0%), 36 wrong (6.9%)d1-ultra: “I don't know”, run 449: 496 answered (90.7%), 39 wrong (7.9%)Sieve 31B v3: “I don't know”, run 1066: 546 answered (99.8%), 45 wrong (8.2%)Turion Jev 31B: “I don't know”, run 1077: 543 answered (99.3%), 43 wrong (7.9%)Perplexity Decider 27B v1.2 · API · with web: “I don't know”, run 644: 537 answered (98.2%), 56 wrong (10.4%)GLiDE · v1.2: “I don't know”, run 523: 517 answered (94.5%), 36 wrong (7.0%)d1.2-ultra: “I don't know”, run 1363: 438 answered (80.1%), 13 wrong (3.0%)Mammoth-1: “I don't know”, run 505: 510 answered (93.2%), 36 wrong (7.1%)d3: “I don't know”, run 793: 546 answered (99.8%), 58 wrong (10.6%)Drex Max: “I don't know”, run 798: 545 answered (99.6%), 60 wrong (11.0%)Perplexity Decider 27B v1.3: “I don't know”, run 1139: 542 answered (99.1%), 52 wrong (9.6%)d3-ultra: “I don't know”, run 795: 542 answered (99.1%), 56 wrong (10.3%)d1.2: “I don't know”, run 1361: 488 answered (89.2%), 37 wrong (7.6%)Perplexity Decider 27B v1.2 · DGX Spark: “I don't know”, run 529: 545 answered (99.6%), 64 wrong (11.7%)Drex Max v1.1: “I don't know”, run 447: 504 answered (92.1%), 41 wrong (8.1%)Perplexity Decider 27B v1.2 · API: “I don't know”, run 643: 544 answered (99.5%), 63 wrong (11.6%)Jeffan-9B-1211: “I don't know”, run 1197: 547 answered (100.0%), 71 wrong (13.0%)d1: “I don't know”, run 445: 545 answered (99.6%), 67 wrong (12.3%)StartLux-Decision-v1.1-9B-Instant: “I don't know”, run 1295: 545 answered (99.6%), 74 wrong (13.6%)Clef: “I don't know”, run 785: 546 answered (99.8%), 75 wrong (13.7%)KnowLine-12B-Gen1: “I don't know”, run 1316: 546 answered (99.8%), 76 wrong (13.9%)GLiDE 9B: “I don't know”, run 911: 533 answered (97.4%), 70 wrong (13.1%)OneJev-27B: “I don't know”, run 780: 544 answered (99.5%), 72 wrong (13.2%)Jev 1.14: “I don't know”, run 446: 544 answered (99.5%), 72 wrong (13.2%)Cerno 9B v1.2: “I don't know”, run 1087: 545 answered (99.6%), 77 wrong (14.1%)KnowLine-9B-Gen1: “I don't know”, run 1179: 546 answered (99.8%), 76 wrong (13.9%)SubQ Snap 1.1: “I don't know”, run 448: 526 answered (96.2%), 63 wrong (12.0%)Adjudex 9B v5: “I don't know”, run 1188: 547 answered (100.0%), 77 wrong (14.1%)Mercury Decide: “I don't know”, run 450: 536 answered (98.0%), 71 wrong (13.2%)Microsoft-Decision-1: “I don't know”, run 451: 544 answered (99.5%), 77 wrong (14.2%)Elephant-1: “I don't know”, run 501: 524 answered (95.8%), 67 wrong (12.8%)Open-Jev-27B-v1.1: “I don't know”, run 763: 531 answered (97.1%), 76 wrong (14.3%)KnowLine-9B-Gen2: “I don't know”, run 1323: 546 answered (99.8%), 78 wrong (14.3%)celeris-1-decision: “I don't know”, run 548: 455 answered (83.2%), 37 wrong (8.1%)GPT-6 Luna · Decisions API: “I don't know”, run 444: 537 answered (98.2%), 96 wrong (17.9%)Cygnet: “I don't know”, run 755: 537 answered (98.2%), 79 wrong (14.7%)Winnow-12B: “I don't know”, run 748: 532 answered (97.3%), 85 wrong (16.0%)Perplexity Decider Flash v1.3: “I don't know”, run 1128: 540 answered (98.7%), 90 wrong (16.7%)Valen-Decision-1.5-9B: “I don't know”, run 1280: 546 answered (99.8%), 84 wrong (15.4%)Dexy-Decision-9B-V2: “I don't know”, run 1260: 546 answered (99.8%), 90 wrong (16.5%)vera-medium-omni: “I don't know”, run 1269: 540 answered (98.7%), 87 wrong (16.1%)StartLux-Decision-v1.1-4B-Instant: “I don't know”, run 1083: 547 answered (100.0%), 94 wrong (17.2%)Clef-Flash: “I don't know”, run 660: 542 answered (99.1%), 90 wrong (16.6%)Verdin-4B-v1.1: “I don't know”, run 1249: 545 answered (99.6%), 94 wrong (17.2%)Perplexity Decider Flash v1.2: “I don't know”, run 535: 542 answered (99.1%), 100 wrong (18.5%)KnowLine 4B Gen5.5: “I don't know”, run 963: 546 answered (99.8%), 105 wrong (19.2%)Chinese Decision Pointer 4B: “I don't know”, run 1211: 543 answered (99.3%), 104 wrong (19.2%)Metask-Jev-4B: “I don't know”, run 741: 545 answered (99.6%), 123 wrong (22.6%)KnowLine-2B-Gen1: “I don't know”, run 1059: 509 answered (93.1%), 135 wrong (26.5%)GLiDE 2B: “I don't know”, run 919: 500 answered (91.4%), 132 wrong (26.4%)StartLux-Decision-v1.1-2B-Instant: “I don't know”, run 975: 515 answered (94.1%), 157 wrong (30.5%)d1-2B: “I don't know”, run 710: 396 answered (72.4%), 97 wrong (24.5%)GLiDE 0.8B: “I don't know”, run 927: 498 answered (91.0%), 177 wrong (35.5%)StartLux-Decision-v1.1-0.8B-Instant: “I don't know”, run 1013: 535 answered (97.8%), 217 wrong (40.6%)GLiDE 0.4B: “I don't know”, run 935: 394 answered (72.0%), 135 wrong (34.3%)OneJev1.6-0.8B: “I don't know”, run 1027: 473 answered (86.5%), 187 wrong (39.5%)KnowLine-0.8B-Gen1: “I don't know”, run 1003: 521 answered (95.2%), 217 wrong (41.7%)GLiClass v4 Edge: “I don't know”, run 1044: 495 answered (90.5%), 196 wrong (39.6%)Strands Decider 2B: “I don't know”, run 704: 545 answered (99.6%), 301 wrong (55.2%)Laya: “I don't know”, run 728: 464 answered (84.8%), 367 wrong (79.1%)GLiNER2.5-multi-Decide: “I don't know”, run 716: 477 answered (87.2%), 363 wrong (76.1%)Laya Multilingual: “I don't know”, run 734: 339 answered (62.0%), 280 wrong (82.6%)
  • k = 496: d1-ultra's answers with “I don't know”
  • Claude Fable 5.2 · max · chat: five options, run 971
  • Gemini 4 Hydrogen · high · chat: five options, run 1107
  • GPT-6 Astra · low · chat: five options, run 616
  • GPT-6 Astra · high · chat: five options, run 615
  • GPT-6.1 Terra · high · chat: five options, run 620
  • GPT-6.1 Terra · low · chat: five options, run 621
  • Claude Fable 5.1 · high · chat: five options, run 580
  • Gemini 3.8 Flash · high · chat: five options, run 583
  • Gemini 4 Argon · low · chat: five options, run 593
  • GPT-6 Astra · medium · chat: five options, run 624
  • GPT-6.1 Sol · low · chat: five options, run 619
  • GPT-6 Astra · low · chat · 1 per request: five options, run 622
  • GPT-6.1 Sol · high · chat: five options, run 618
  • Gemini 3.8 Flash · low · chat · 1 per request: five options, run 589
  • GPT-6.1 Sol · low · chat · 1 per request: five options, run 623
  • Claude Opus 5.5 · low · chat · 1 per request: five options, run 592
  • Claude Opus 5.5 · high · chat: five options, run 587
  • Qwen4-Max · high · chat: five options, run 1334
  • Gemini 3.8 Flash · low · chat: five options, run 578
  • Gemini 4 Carbon · high · chat: five options, run 1055
  • KnowLine-31B-Gen1.5: five options, run 953 · “I don't know”, run 955
  • Grok 4.7 · high · chat: five options, run 579
  • Claude Opus 5.5 · low · chat: five options, run 588
  • Copernicus-31B: five options, run 1168 · “I don't know”, run 1170
  • Claude Haiku 5.5 · max · chat: five options, run 949
  • DeepSeek-V4.1-Flash · reasoning on · chat · 1017: five options, run 808
  • REAX 1.3: five options, run 829 · “I don't know”, run 831
  • Kimi K3.1 Pro · low · chat: five options, run 584
  • KnowLine-31B-Gen2: five options, run 1327 · “I don't know”, run 1329
  • Karpo Decision 31B: five options, run 1159 · “I don't know”, run 1161
  • Claude Sonnet 5.5 · high · chat · 1 per request: five options, run 594
  • Kimi K3.1 Pro · no reasoning · chat: five options, run 582
  • Qwen3.5-397B-A17B · high · chat: five options, run 871
  • Gemini Decision 1 · 20261111: five options, run 1009 · “I don't know”, run 1019
  • Logits Sigma-1 Duo: five options, run 1229 · “I don't know”, run 1231
  • GLiDE 27B: five options, run 901 · “I don't know”, run 903
  • d1-ultra: five options, run 411 · “I don't know”, run 449
  • Sieve 31B v3: five options, run 1064 · “I don't know”, run 1066
  • Turion Jev 31B: five options, run 1075 · “I don't know”, run 1077
  • Gemma 4 31B · reasoning off · chat: five options, run 774
  • Perplexity Decider 27B v1.2 · API · with web: five options, run 639 · “I don't know”, run 644
  • GLiDE · v1.2: five options, run 521 · “I don't know”, run 523
  • d1.2-ultra: five options, run 1359 · “I don't know”, run 1363
  • Mammoth-1: five options, run 503 · “I don't know”, run 505
  • Gemma 4 31B · reasoning off · chat · 1 per request: five options, run 783
  • d3: five options, run 554 · “I don't know”, run 793
  • Drex Max: five options, run 552 · “I don't know”, run 798
  • GLM-5.3-Flash · reasoning on · chat: five options, run 823
  • Muse Ember · max · chat: five options, run 942
  • Perplexity Decider 27B v1.3: five options, run 1137 · “I don't know”, run 1139
  • d3-ultra: five options, run 550 · “I don't know”, run 795
  • d1.2: five options, run 1357 · “I don't know”, run 1361
  • MiMo-V2.6-Flash · high · chat: five options, run 878
  • Qwen3.8-27B · low · chat: five options, run 667
  • Perplexity Decider 27B v1.2 · DGX Spark: five options, run 527 · “I don't know”, run 529
  • Drex Max v1.1: five options, run 399 · “I don't know”, run 447
  • Perplexity Decider 27B v1: five options, run 626
  • Perplexity Decider 27B v1.2 · API: five options, run 637 · “I don't know”, run 643
  • Inkling-Small · high · chat · Jul 2026: five options, run 867
  • d1.1: five options, run 565
  • GLM-5.3 · low · chat: five options, run 577
  • Jeffan-9B-1211: five options, run 1195 · “I don't know”, run 1197
  • GLM-5.3-Flash · reasoning off · chat: five options, run 820
  • DeepSeek-V4.1-Flash · reasoning off · chat · 1017: five options, run 804
  • Qwen3.8 Flash · no reasoning · chat: five options, run 986
  • d1: five options, run 430 · “I don't know”, run 445
  • StartLux-Decision-v1.1-9B-Instant: five options, run 1293 · “I don't know”, run 1295
  • Clef: five options, run 613 · “I don't know”, run 785
  • KnowLine-12B-Gen1: five options, run 1313 · “I don't know”, run 1316
  • GLiDE 9B: five options, run 909 · “I don't know”, run 911
  • GPT-6 Luna · low · chat: five options, run 617
  • OneJev-27B: five options, run 778 · “I don't know”, run 780
  • Claude Sonnet 5.5 · high · chat: five options, run 590
  • Jev 1.14: five options, run 410 · “I don't know”, run 446
  • Cerno 9B v1.2: five options, run 1085 · “I don't know”, run 1087
  • Drex 2.0: five options, run 566
  • Jev 1.13: five options, run 564
  • KnowLine-9B-Gen1: five options, run 1177 · “I don't know”, run 1179
  • SubQ Snap 1.1: five options, run 413 · “I don't know”, run 448
  • SubQ Snap Preview: five options, run 595
  • Adjudex 9B v5: five options, run 1186 · “I don't know”, run 1188
  • Mercury Decide: five options, run 398 · “I don't know”, run 450
  • Microsoft-Decision-1: five options, run 412 · “I don't know”, run 451
  • Elephant-1: five options, run 499 · “I don't know”, run 501
  • Open-Jev-27B-v1.1: five options, run 760 · “I don't know”, run 763
  • KnowLine-9B-Gen2: five options, run 1321 · “I don't know”, run 1323
  • Qwen3.8-27B · stated confidence · reasoning off · chat: five options, run 672
  • Qwen3.8-27B · yes/no scoring: five options, run 696
  • MiniMax-M3.1-Flash · high · chat: five options, run 863
  • celeris-1-decision: five options, run 546 · “I don't know”, run 548
  • GPT-6 Luna · Decisions API: five options, run 397 · “I don't know”, run 444
  • Claude Sonnet 5.5 · low · chat: five options, run 591
  • Cygnet: five options, run 753 · “I don't know”, run 755
  • Qwen3.8-27B · vote share · reasoning off · chat: five options, run 678
  • Winnow-12B: five options, run 746 · “I don't know”, run 748
  • Perplexity Decider Flash v1.3: five options, run 1126 · “I don't know”, run 1128
  • Valen-Decision-1.5-9B: five options, run 1278 · “I don't know”, run 1280
  • Dexy-Decision-9B-V2: five options, run 1258 · “I don't know”, run 1260
  • vera-medium-omni: five options, run 1267 · “I don't know”, run 1269
  • StartLux-Decision-v1.1-4B-Instant: five options, run 1081 · “I don't know”, run 1083
  • Clef-Flash: five options, run 614 · “I don't know”, run 660
  • Verdin-4B-v1.1: five options, run 1247 · “I don't know”, run 1249
  • Perplexity Decider Flash v1.2: five options, run 533 · “I don't know”, run 535
  • KnowLine 4B Gen5.5: five options, run 961 · “I don't know”, run 963
  • Chinese Decision Pointer 4B: five options, run 1209 · “I don't know”, run 1211
  • Qwen3-14B · reasoning off · chat: five options, run 770
  • Metask-Jev-4B: five options, run 739 · “I don't know”, run 741
  • Erk-14B · reasoning off · chat: five options, run 762
  • KnowLine-2B-Gen1: five options, run 1057 · “I don't know”, run 1059
  • GLiDE 2B: five options, run 917 · “I don't know”, run 919
  • StartLux-Decision-v1.1-2B-Instant: five options, run 973 · “I don't know”, run 975
  • d1-2B: five options, run 708 · “I don't know”, run 710
  • GLiDE 0.8B: five options, run 925 · “I don't know”, run 927
  • StartLux-Decision-v1.1-0.8B-Instant: five options, run 1011 · “I don't know”, run 1013
  • GLiDE 0.4B: five options, run 933 · “I don't know”, run 935
  • OneJev1.6-0.8B: five options, run 1025 · “I don't know”, run 1027
  • KnowLine-0.8B-Gen1: five options, run 1001 · “I don't know”, run 1003
  • GLiClass v4 Edge: five options, run 1042 · “I don't know”, run 1044
  • Decider 1: five options, run 596
  • Strands Decider 2B: five options, run 702 · “I don't know”, run 704
  • Laya: five options, run 726 · “I don't know”, run 728
  • GLiNER2.5-multi-Decide: five options, run 714 · “I don't know”, run 716
  • GLiNER2.5 Multi: five options, run 720
  • Laya Multilingual: five options, run 732 · “I don't know”, run 734

Answers at 95%+ confidence

An answer counts when its chosen option has probability at least the cutoff; an answer without a score or without a valid choice never counts; wrong is a counted answer that is not right. Each decision model is counted on its Text tab run (its first five-option Turkish run, the run the site already uses), on the Text tab's 2,198 questions, keeping the answers whose chosen option has a probability of at least 95%. These runs have no “I don't know” option.

Every decision setup

Ordered by % wrong (wrong / answers kept), then by answers kept, more first; a setup that keeps no answers is last (No answers at this threshold: its % wrong is undefined).

ModelRunAnswers kept (at 95%+)Wrong% wrong
d1-ultra Closed · Efficient19954900.0%0 of 549
GLiDE 27B Open · 27B89954800.0%0 of 548
Copernicus-31B Open · 31B116548300.0%0 of 483
KnowLine-31B-Gen2 Open · 31B131445600.0%0 of 456
d1.2-ultra Closed · Efficient134738900.0%0 of 389
Logits Sigma-1 Duo Open · 57B122135100.0%0 of 351
GLiDE API Closed · Efficient12930200.0%0 of 302
Perplexity Decider 27B v1 Open · 27B4429300.0%0 of 293
GLiDE 9B Open · 7.9B90720800.0%0 of 208
Drex 2.0 Open · 35B MoE · 3.0B active33418000.0%0 of 180
OneJev-27B Open · 27B26416600.0%0 of 166
celeris-1-decision Closed · Efficient15012300.0%0 of 123
KnowLine-12B-Gen1 Open · 12B13091500.0%0 of 15
KnowLine-9B-Gen2 Open · 9.0B13061000.0%0 of 10
StartLux-Decision-v1.1-2B-Instant Open · 2.0B977700.0%0 of 7
Jeffan-9B-1211 Open · 9.0B1192600.0%0 of 6
d1-2B Open · 1.9B340400.0%0 of 4
GLiClass v4 Edge Open · 0.4B1046200.0%0 of 2
Decider 1 Closed · Efficient243100.0%0 of 1
REAX 1.3 Open · 57B82765710.2%1 of 657
KnowLine-31B-Gen1.5 Open · 31B95762810.2%1 of 628
SubQ Snap Preview Closed · Efficient17733910.3%1 of 339
Turion Jev 31B Open · 31B127330410.3%1 of 304
d1.2 Closed · Efficient134626910.4%1 of 269
GLiDE v1.2 Open · 27B24848520.4%2 of 485
d1.1 Closed · Efficient25433620.6%2 of 336
Drex Max v1.1 Closed · Efficient (provisional)32649330.6%3 of 493
Perplexity Decider Flash v1.2 Open · 4.0B22315410.6%1 of 154
Gemini Decision 1 20261111 Closed · Efficient102176660.8%6 of 766
d3 Open · 26B25950340.8%4 of 503
Perplexity Decider 27B v1.3 Open · 27B113461150.8%5 of 611
Karpo Decision 31B Open · 31B115687080.9%8 of 870
Perplexity Decider 27B v1.1 Open · 27B14558961.0%6 of 589
Jev 1.14 Closed · Efficient19439041.0%4 of 390
Valen-Decision-1.5-9B Open · 9.0B12759211.1%1 of 92
Sieve 31B v3 Open · 31B124172781.1%8 of 727
Cygnet Open · 12B12733541.2%4 of 335
Mammoth-1 Open · 27B37466791.3%9 of 667
vera-medium-omni Open · 4.2B12647211.4%1 of 72
Perplexity Decider 27B v1.2 API · with web Open · 27B64064191.4%9 of 641
Clef Open · 27B12534851.4%5 of 348
Perplexity Decider 27B v1.2 API Open · 27B633610101.6%10 of 610
Perplexity Decider 27B v1.2 DGX Spark Open · 27B217633111.7%11 of 633
Jev 1.13 Closed · Efficient4944081.8%8 of 440
StartLux-Decision-v1.1-4B-Instant Open · 4.0B123615631.9%3 of 156
KnowLine-9B-Gen1 Open · 9.0B117430062.0%6 of 300
Dexy-Decision-9B-V2 Open · 9.0B125529362.0%6 of 293
Cerno 9B v1.2 Open · 9.0B120113932.2%3 of 139
KnowLine 4B Gen5.5 Open · 4.0B96527462.2%6 of 274
StartLux-Decision-v1.1-9B-Instant Open · 9.0B129021952.3%5 of 219
SubQ Snap 1.1 Closed · Efficient346665182.7%18 of 665
GPT-6 Luna Decisions API Closed · Efficient143542152.8%15 of 542
Metask-Jev-4B Open · 4.5B1263213.1%1 of 32
Perplexity Decider Flash v1.3 Open · 4.0B112324683.3%8 of 246
Drex Max Open · 27B322736243.3%24 of 736
Elephant-1 Open · 9.0B373393133.3%13 of 393
d3-ultra Open · 26B269833303.6%30 of 833
Adjudex 9B v5 Open · 9.0B1183357133.6%13 of 357
KnowLine-2B-Gen1 Open · 2.0B10792613.8%1 of 26
d1 Closed · Efficient52502204.0%20 of 502
Verdin-4B-v1.1 Open · 4.0B1244202115.4%11 of 202
Clef-Flash Open · 9.0B1395647.1%4 of 56
Chinese Decision Pointer 4B Open · 4.0B1206193147.3%14 of 193
Microsoft-Decision-1 Closed · Efficient205378297.7%29 of 378
Winnow-12B Open · 12B128641558.6%55 of 641
Mercury Decide Closed · Efficient204775678.6%67 of 775
Strands Decider 2B Open · 2.0B14110110.0%1 of 10
GLiDE 2B Open · 1.9B9151011110.9%11 of 101
GLiDE 0.8B Open · 0.8B923851720.0%17 of 85
OneJev1.6-0.8B Open · 0.8B10297228.6%2 of 7
KnowLine-0.8B-Gen1 Open · 0.8B10052150.0%1 of 2
Laya Multilingual Open · 0.3B46141178.6%11 of 14
GLiDE 0.4B Open · 0.4B931No answers at this threshold––
GLiNER2.5-multi-Decide Open · 0.3B135No answers at this threshold––
Laya Open · 0.4B51No answers at this threshold––
Open-Jev-27B-v1.1 Open · 27B45No answers at this threshold––
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B1015No answers at this threshold––

d1-ultra's repeat runs

The same questions and settings, run again; run 199 is the Text tab run above, shown for context.

RunAnswers keptWrong% wrong
d1-ultra Closed · Efficientrun 199, Text tab run54900.0%0 of 549
d1-ultra Closed · Efficientrun 200, repeat run53800.0%0 of 538
d1-ultra Closed · Efficientrun 201, repeat run54200.0%0 of 542
d1-ultra Closed · Efficientrun 202, repeat run53400.0%0 of 534
d1-ultra Closed · Efficientrun 203, repeat run55000.0%0 of 550

The three models Artificial Analysis ties at 65

The three models its Decision Index v1.0 puts level at 65 (its post), each with its Text tab run and its repeat runs, one row per run. Each column is a cutoff: % wrong of the answers kept at that probability or higher, with wrong of kept below.

Model and run50%70%80%90%95%99%
d1-ultra Closed · Efficientrun 199, Text tab run13.1%224 of 1,7127.4%99 of 1,3354.2%46 of 1,0941.0%8 of 8020.0%0 of 5490.0%0 of 90
d1-ultra Closed · Efficientrun 200, repeat run13.4%229 of 1,7057.0%93 of 1,3203.6%39 of 1,0890.8%6 of 7950.0%0 of 5380.0%0 of 86
d1-ultra Closed · Efficientrun 201, repeat run14.0%241 of 1,7197.3%96 of 1,3134.3%47 of 1,0911.3%10 of 7940.0%0 of 5420.0%0 of 96
d1-ultra Closed · Efficientrun 202, repeat run13.3%228 of 1,7136.4%84 of 1,3154.5%49 of 1,0881.0%8 of 7960.0%0 of 5340.0%0 of 91
d1-ultra Closed · Efficientrun 203, repeat run13.7%233 of 1,7067.3%97 of 1,3244.0%43 of 1,0801.1%9 of 8030.0%0 of 5500.0%0 of 90
SubQ Snap 1.1 Closed · Efficientrun 346, Text tab run27.9%499 of 1,79117.3%229 of 1,32312.4%138 of 1,1176.2%52 of 8392.7%18 of 6650.3%1 of 299
SubQ Snap 1.1 Closed · Efficientrun 387, repeat run27.9%499 of 1,79117.3%229 of 1,32312.4%138 of 1,1176.2%52 of 8392.7%18 of 6650.3%1 of 299
SubQ Snap 1.1 Closed · Efficientrun 388, repeat run27.9%499 of 1,79117.3%229 of 1,32312.4%138 of 1,1176.2%52 of 8392.7%18 of 6650.3%1 of 299
SubQ Snap 1.1 Closed · Efficientrun 389, repeat run27.9%499 of 1,79117.3%229 of 1,32312.4%138 of 1,1176.2%52 of 8392.7%18 of 6650.3%1 of 299
SubQ Snap 1.1 Closed · Efficientrun 390, repeat run27.9%499 of 1,79117.3%229 of 1,32312.4%138 of 1,1176.2%52 of 8392.7%18 of 6650.3%1 of 299
Elephant-1 Open · 9.0Brun 373, Text tab run33.2%455 of 1,37021.3%195 of 91714.0%102 of 7266.9%35 of 5103.3%13 of 3930.0%0 of 155
Elephant-1 Open · 9.0Brun 375, repeat run33.2%455 of 1,37021.3%195 of 91714.0%102 of 7266.9%35 of 5103.3%13 of 3930.0%0 of 155
Elephant-1 Open · 9.0Brun 376, repeat run33.2%455 of 1,37021.3%195 of 91714.0%102 of 7266.9%35 of 5103.3%13 of 3930.0%0 of 155
Elephant-1 Open · 9.0Brun 378, repeat run33.2%455 of 1,37021.3%195 of 91714.0%102 of 7266.9%35 of 5103.3%13 of 3930.0%0 of 155
Elephant-1 Open · 9.0Brun 379, repeat run33.2%455 of 1,37021.3%195 of 91714.0%102 of 7266.9%35 of 5103.3%13 of 3930.0%0 of 155

At every cutoff up to 98% (each whole percent), d1-ultra's worst run has a lower % wrong than the best run of SubQ Snap 1.1 and of Elephant-1; at 99%, all three are at or near 0 (highest: 0.3%).

In English (the 547 kept reading questions of the Turkish vs English tab), at 95%, d1-ultra has 1 wrong answer in each of its 5 runs, always question 280; each run keeps 230 to 237 answers (runs 411, 418, 422, 426 and 428).

Info The table, what we found and our method

About this test

Every question of the Text tab gets one more option after its five answers: F, “I don't know”, written in Turkish in the prompt (“Bilmiyorum”). It is always the last option, it is never shuffled and it is never correct. The System One instructions get one added sentence: “If you do not know the answer, choose Bilmiyorum.” A chat model's instructions end with its own sentence instead: “Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.” Everything else is as on the Text tab: the same 2,198 scored questions and answer key, and each model's own settings from its text run, its questions per request too (one for every decision model).

The models: the ten most accurate decision models on the Text tab at results 25290875, plus Base Labs' Elephant-1 and Mammoth-1 and SubQ Snap 1.1 once their text runs are published. With them, every chat model setup of the Text tab as its run finishes: 56 so far, one still running in Turkish. Each has one full run; a model whose repeated text runs changed answers gets five.

Download: results.json (every number on this page, under dont_know) and answers.json (each run's answers, under dont_know_runs).

Results

The rows are ranked by the first column, the non-hallucination rate, highest first: the share of “I don't know” among the answers that were not right (“I don't know” / (“I don't know” + wrong)), so the first model is the one that most often said it did not know instead of answering wrongly. Beside it: Answered, how often the model chose one of the question's own options (right + wrong, out of all 2,198 questions), and the error rate, the share of those answers that were wrong (wrong / (right + wrong)). Read the ranking with Answered, the error rate and accuracy: a model that always chose “I don't know” would rank first with no right answers. Each row shows the model's first run; under Runs, a model with more than one run (d1-ultra and GLiDE · API) shows the range over its runs and how many answers changed between them. Out of all 2,198 questions: accuracy counts the right answers; Answered (the attempt rate) counts the right and wrong answers. Formulas and how to read them: Our method.

Decision models

ModelNon-hallucinationof answers not right, % “I don't know” · higher is betterAnswerednot “I don't know”: (right + wrong) / 2,198Error rateof its answers: wrong / (right + wrong) · lower is betterDerived non-hallucination ratefrom the 5-option text run; higher is betterAccuracyhigher is betterMean probability of “I don't know”higher means more probability on the option; neither better nor worse on its ownText accuracy with five optionsthe Text tab's number; higher is betterRunsrange over the runs where more than one
GLiDE 0.4B Open · 0.4B95.3%2,070 “I don't know” of 2,171 not right5.8%128 of 2,198 answered, 2,070 “I don't know”78.9%101 of 12895.4%cut-off 0.4443: 29 right + 99 wrong answered, 2,070 held back, attempt rate 5.8%1.2%27 of 2,198 right39.1%22.2%run 931run 937
d1.2-ultra Closed · Efficient87.3%996 “I don't know” of 1,141 not right54.7%1,202 of 2,198 answered, 996 “I don't know”12.1%145 of 1,20287.8%cut-off 0.5947: 1,064 right + 138 wrong answered, 996 held back, attempt rate 54.7%48.1%1,057 of 2,198 right34.6%68.1%run 1347run 1367
d1.2 Closed · Efficient79.4%973 “I don't know” of 1,226 not right55.7%1,225 of 2,198 answered, 973 “I don't know”20.7%253 of 1,22581.3%cut-off 0.4936: 1,001 right + 224 wrong answered, 973 held back, attempt rate 55.7%44.2%972 of 2,198 right31.5%61.7%run 1346run 1365
d1-ultra Closed · Efficient75.1%617 “I don't know” of 822 not right71.9%1,581 of 2,198 answered, 617 “I don't know”13.0%205 of 1,58178.7%cut-off 0.5632: 1,414 right + 167 wrong answered, 617 held back, attempt rate 71.9%62.6%1,376 of 2,198 right30.7%77.1%run 199runs 350, 354, 360, 365, 370Answered 71.9% to 74.0%Non-hallucination 71.2% to 75.1%Accuracy 62.6% to 63.5%Mean probability 30.6% to 30.8%518 answers changed between runs
Mammoth-1 Open · 27B68.7%690 “I don't know” of 1,004 not right68.6%1,508 of 2,198 answered, 690 “I don't know”20.8%314 of 1,50870.9%cut-off 0.5697: 1,225 right + 283 wrong answered, 690 held back, attempt rate 68.6%54.3%1,194 of 2,198 right24.5%67.1%run 374run 386
d1-2B Open · 1.9B65.5%1,209 “I don't know” of 1,846 not right45.0%989 of 2,198 answered, 1,209 “I don't know”64.4%637 of 98966.5%cut-off 0.3129: 381 right + 608 wrong answered, 1,209 held back, attempt rate 45.0%16.0%352 of 2,198 right26.3%29.4%run 340run 712
GLiDE API Closed · Efficient59.4%521 “I don't know” of 877 not right76.3%1,677 of 2,198 answered, 521 “I don't know”21.2%356 of 1,67758.0%cut-off 0.5206: 1,300 right + 377 wrong answered, 521 held back, attempt rate 76.3%60.1%1,321 of 2,198 right22.0%67.9%run 129runs 351, 353, 357, 361, 364Answered 75.5% to 76.3%Non-hallucination 59.4% to 59.9%Accuracy 59.1% to 60.1%Mean probability 21.6% to 22.0%189 answers changed between runs
SubQ Snap 1.1 Closed · Efficient47.1%437 “I don't know” of 927 not right80.1%1,761 of 2,198 answered, 437 “I don't know”27.8%490 of 1,76147.6%cut-off 0.5124: 1,280 right + 481 wrong answered, 437 held back, attempt rate 80.1%57.8%1,271 of 2,198 right20.1%63.9%run 346run 392
OneJev1.6-0.8B Open · 0.8B43.4%754 “I don't know” of 1,739 not right65.7%1,444 of 2,198 answered, 754 “I don't know”68.2%985 of 1,44443.5%cut-off 0.2972: 463 right + 981 wrong answered, 754 held back, attempt rate 65.7%20.9%459 of 2,198 right22.2%29.1%run 1029run 1031
GLiDE 27B Open · 27B39.8%291 “I don't know” of 732 not right86.8%1,907 of 2,198 answered, 291 “I don't know”23.1%441 of 1,90739.9%cut-off 0.4133: 1,469 right + 438 wrong answered, 291 held back, attempt rate 86.8%66.7%1,466 of 2,198 right16.8%71.1%run 899run 905
GLiDE v1.2 Open · 27B39.5%321 “I don't know” of 812 not right85.4%1,877 of 2,198 answered, 321 “I don't know”26.2%491 of 1,87740.4%cut-off 0.4134: 1,404 right + 473 wrong answered, 321 held back, attempt rate 85.4%63.1%1,386 of 2,198 right18.6%68.5%run 248run 369
GLiDE 2B Open · 1.9B38.5%612 “I don't know” of 1,590 not right72.2%1,586 of 2,198 answered, 612 “I don't know”61.7%978 of 1,58638.7%cut-off 0.4078: 618 right + 968 wrong answered, 612 held back, attempt rate 72.2%27.7%608 of 2,198 right23.5%35.3%run 915run 921
Drex Max v1.1 Closed · Efficient (provisional)38.2%320 “I don't know” of 837 not right85.4%1,878 of 2,198 answered, 320 “I don't know”27.5%517 of 1,87839.3%cut-off 0.3806: 1,383 right + 495 wrong answered, 320 held back, attempt rate 85.4%61.9%1,361 of 2,198 right17.0%67.4%run 326run 352
Elephant-1 Open · 9.0B34.8%405 “I don't know” of 1,163 not right81.6%1,793 of 2,198 answered, 405 “I don't know”42.3%758 of 1,79335.5%cut-off 0.3883: 1,058 right + 735 wrong answered, 405 held back, attempt rate 81.6%47.1%1,035 of 2,198 right16.0%52.4%run 373run 382
GLiDE 0.8B Open · 0.8B29.2%480 “I don't know” of 1,643 not right78.2%1,718 of 2,198 answered, 480 “I don't know”67.7%1,163 of 1,71828.9%cut-off 0.3650: 536 right + 1,182 wrong answered, 480 held back, attempt rate 78.2%25.3%555 of 2,198 right19.9%29.3%run 923run 929
Laya Open · 0.4B28.3%536 “I don't know” of 1,891 not right75.6%1,662 of 2,198 answered, 536 “I don't know”81.5%1,355 of 1,66228.5%cut-off 0.2527: 316 right + 1,345 wrong answered, 537 held back, attempt rate 75.6%14.0%307 of 2,198 right18.6%19.0%run 51run 730
GLiDE 9B Open · 7.9B24.1%277 “I don't know” of 1,149 not right87.4%1,921 of 2,198 answered, 277 “I don't know”45.4%872 of 1,92124.8%cut-off 0.3391: 1,081 right + 840 wrong answered, 277 held back, attempt rate 87.4%47.7%1,049 of 2,198 right15.2%52.0%run 907run 913
GLiClass v4 Edge Open · 0.4B22.3%398 “I don't know” of 1,784 not right81.9%1,800 of 2,198 answered, 398 “I don't know”77.0%1,386 of 1,80022.6%cut-off 0.2423: 435 right + 1,365 wrong answered, 398 held back, attempt rate 81.9%18.8%414 of 2,198 right14.5%23.3%run 1046run 1048
StartLux-Decision-v1.1-2B-Instant Open · 2.0B21.5%333 “I don't know” of 1,546 not right84.8%1,865 of 2,198 answered, 333 “I don't know”65.0%1,213 of 1,86521.8%cut-off 0.2682: 669 right + 1,196 wrong answered, 333 held back, attempt rate 84.8%29.7%652 of 2,198 right13.2%34.3%run 977run 979
Laya Multilingual Open · 0.3B14.8%280 “I don't know” of 1,888 not right87.3%1,918 of 2,198 answered, 280 “I don't know”83.8%1,608 of 1,91815.0%cut-off 0.2794: 331 right + 1,587 wrong answered, 280 held back, attempt rate 87.3%14.1%310 of 2,198 right14.6%17.9%run 46run 737
Perplexity Decider Flash v1.2 Open · 4.0B14.7%180 “I don't know” of 1,224 not right91.8%2,018 of 2,198 answered, 180 “I don't know”51.7%1,044 of 2,01815.2%cut-off 0.2883: 1,014 right + 1,004 wrong answered, 180 held back, attempt rate 91.8%44.3%974 of 2,198 right12.5%47.8%run 223run 791
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B12.4%199 “I don't know” of 1,608 not right90.9%1,999 of 2,198 answered, 199 “I don't know”70.5%1,409 of 1,99912.3%cut-off 0.2318: 585 right + 1,414 wrong answered, 199 held back, attempt rate 90.9%26.8%590 of 2,198 right9.4%28.6%run 1015run 1017
Chinese Decision Pointer 4B Open · 4.0B11.2%136 “I don't know” of 1,218 not right93.8%2,062 of 2,198 answered, 136 “I don't know”52.5%1,082 of 2,06211.3%cut-off 0.3048: 994 right + 1,068 wrong answered, 136 held back, attempt rate 93.8%44.6%980 of 2,198 right6.8%46.7%run 1206run 1213
Strands Decider 2B Open · 2.0B9.2%150 “I don't know” of 1,633 not right93.2%2,048 of 2,198 answered, 150 “I don't know”72.4%1,483 of 2,0489.2%cut-off 0.2554: 574 right + 1,474 wrong answered, 150 held back, attempt rate 93.2%25.7%565 of 2,198 right8.8%27.5%run 141run 706
Clef-Flash Open · 9.0B9.0%110 “I don't know” of 1,218 not right95.0%2,088 of 2,198 answered, 110 “I don't know”53.1%1,108 of 2,0889.0%cut-off 0.2929: 981 right + 1,107 wrong answered, 110 held back, attempt rate 95.0%44.6%980 of 2,198 right9.1%45.7%run 139run 664
vera-medium-omni Open · 4.2B8.9%105 “I don't know” of 1,180 not right95.2%2,093 of 2,198 answered, 105 “I don't know”51.4%1,075 of 2,0939.2%cut-off 0.2717: 1,055 right + 1,038 wrong answered, 105 held back, attempt rate 95.2%46.3%1,018 of 2,198 right9.1%49.1%run 1264run 1271
Clef Open · 27B7.4%71 “I don't know” of 965 not right96.8%2,127 of 2,198 answered, 71 “I don't know”42.0%894 of 2,1277.4%cut-off 0.3189: 1,238 right + 889 wrong answered, 71 held back, attempt rate 96.8%56.1%1,233 of 2,198 right10.1%57.3%run 125run 788
Perplexity Decider Flash v1.3 Open · 4.0B6.5%75 “I don't know” of 1,146 not right96.6%2,123 of 2,198 answered, 75 “I don't know”50.4%1,071 of 2,1236.7%cut-off 0.2769: 1,074 right + 1,049 wrong answered, 75 held back, attempt rate 96.6%47.9%1,052 of 2,198 right9.1%49.7%run 1123run 1130
KnowLine-0.8B-Gen1 Open · 0.8B6.4%104 “I don't know” of 1,619 not right95.3%2,094 of 2,198 answered, 104 “I don't know”72.3%1,515 of 2,0946.4%cut-off 0.2348: 578 right + 1,516 wrong answered, 104 held back, attempt rate 95.3%26.3%579 of 2,198 right8.1%27.1%run 1005run 1007
GLiNER2.5-multi-Decide Open · 0.3B6.4%114 “I don't know” of 1,780 not right94.8%2,084 of 2,198 answered, 114 “I don't know”79.9%1,666 of 2,0846.4%cut-off 0.2553: 426 right + 1,658 wrong answered, 114 held back, attempt rate 94.8%19.0%418 of 2,198 right10.7%20.2%run 135run 718
KnowLine 4B Gen5.5 Open · 4.0B4.7%55 “I don't know” of 1,176 not right97.5%2,143 of 2,198 answered, 55 “I don't know”52.3%1,121 of 2,1434.8%cut-off 0.2688: 1,051 right + 1,092 wrong answered, 55 held back, attempt rate 97.5%46.5%1,022 of 2,198 right2.8%48.5%run 965run 967
Jeffan-9B-1211 Open · 9.0B4.3%41 “I don't know” of 958 not right98.1%2,157 of 2,198 answered, 41 “I don't know”42.5%917 of 2,1574.3%cut-off 0.2414: 1,248 right + 909 wrong answered, 41 held back, attempt rate 98.1%56.4%1,240 of 2,198 right5.8%57.0%run 1192run 1199
Winnow-12B Open · 12B4.3%43 “I don't know” of 1,008 not right98.0%2,155 of 2,198 answered, 43 “I don't know”44.8%965 of 2,1554.3%cut-off 0.2923: 1,209 right + 946 wrong answered, 43 held back, attempt rate 98.0%54.1%1,190 of 2,198 right3.5%55.5%run 128run 750
Dexy-Decision-9B-V2 Open · 9.0B3.4%37 “I don't know” of 1,083 not right98.3%2,161 of 2,198 answered, 37 “I don't know”48.4%1,046 of 2,1613.4%cut-off 0.2579: 1,122 right + 1,039 wrong answered, 37 held back, attempt rate 98.3%50.7%1,115 of 2,198 right2.7%51.6%run 1255run 1262
Turion Jev 31B Open · 31B3.3%15 “I don't know” of 451 not right99.3%2,183 of 2,198 answered, 15 “I don't know”20.0%436 of 2,1833.4%cut-off 0.2468: 1,755 right + 428 wrong answered, 15 held back, attempt rate 99.3%79.5%1,747 of 2,198 right3.1%80.1%run 1273run 1285
Cygnet Open · 12B3.2%32 “I don't know” of 989 not right98.5%2,166 of 2,198 answered, 32 “I don't know”44.2%957 of 2,1663.3%cut-off 0.2955: 1,217 right + 949 wrong answered, 32 held back, attempt rate 98.5%55.0%1,209 of 2,198 right5.1%55.7%run 127run 757
Perplexity Decider 27B v1.2 API · with web Open · 27B3.2%23 “I don't know” of 725 not right99.0%2,175 of 2,198 answered, 23 “I don't know”32.3%702 of 2,1753.3%cut-off 0.2556: 1,505 right + 670 wrong answered, 23 held back, attempt rate 99.0%67.0%1,473 of 2,198 right5.6%68.7%run 640run 642
KnowLine-9B-Gen2 Open · 9.0B2.6%27 “I don't know” of 1,057 not right98.8%2,171 of 2,198 answered, 27 “I don't know”47.4%1,030 of 2,1712.6%cut-off 0.2371: 1,162 right + 1,009 wrong answered, 27 held back, attempt rate 98.8%51.9%1,141 of 2,198 right6.2%53.3%run 1306run 1325
StartLux-Decision-v1.1-9B-Instant Open · 9.0B2.2%24 “I don't know” of 1,093 not right98.9%2,174 of 2,198 answered, 24 “I don't know”49.2%1,069 of 2,1742.2%cut-off 0.2431: 1,124 right + 1,050 wrong answered, 24 held back, attempt rate 98.9%50.3%1,105 of 2,198 right4.3%51.2%run 1290run 1297
Gemini Decision 1 20261111 Closed · Efficient1.9%10 “I don't know” of 515 not right99.5%2,188 of 2,198 answered, 10 “I don't know”23.1%505 of 2,1882.0%cut-off 0.2613: 1,702 right + 486 wrong answered, 10 held back, attempt rate 99.5%76.6%1,683 of 2,198 right5.5%77.5%run 1021run 1023
StartLux-Decision-v1.1-4B-Instant Open · 4.0B1.8%21 “I don't know” of 1,168 not right99.0%2,177 of 2,198 answered, 21 “I don't know”52.7%1,147 of 2,1771.8%cut-off 0.2403: 1,048 right + 1,129 wrong answered, 21 held back, attempt rate 99.0%46.9%1,030 of 2,198 right5.5%47.9%run 1236run 1239
Open-Jev-27B-v1.1 Open · 27B1.4%14 “I don't know” of 979 not right99.4%2,184 of 2,198 answered, 14 “I don't know”44.2%965 of 2,1841.4%cut-off 0.2077: 1,222 right + 962 wrong answered, 14 held back, attempt rate 99.4%55.5%1,219 of 2,198 right12.7%55.7%run 45run 768
d3-ultra Open · 26B1.4%10 “I don't know” of 714 not right99.5%2,188 of 2,198 answered, 10 “I don't know”32.2%704 of 2,1881.4%cut-off 0.2876: 1,486 right + 702 wrong answered, 10 held back, attempt rate 99.5%67.5%1,484 of 2,198 right2.5%67.8%run 269run 363
Copernicus-31B Open · 31B1.2%5 “I don't know” of 407 not right99.8%2,193 of 2,198 answered, 5 “I don't know”18.3%402 of 2,1931.2%cut-off 0.2499: 1,796 right + 397 wrong answered, 5 held back, attempt rate 99.8%81.5%1,791 of 2,198 right2.0%81.8%run 1165run 1172
Adjudex 9B v5 Open · 9.0B0.8%8 “I don't know” of 1,029 not right99.6%2,190 of 2,198 answered, 8 “I don't know”46.6%1,021 of 2,1900.8%cut-off 0.2339: 1,167 right + 1,023 wrong answered, 8 held back, attempt rate 99.6%53.2%1,169 of 2,198 right1.6%53.1%run 1183run 1190
Metask-Jev-4B Open · 4.5B0.8%10 “I don't know” of 1,325 not right99.5%2,188 of 2,198 answered, 10 “I don't know”60.1%1,315 of 2,1880.8%cut-off 0.2132: 892 right + 1,296 wrong answered, 10 held back, attempt rate 99.5%39.7%873 of 2,198 right6.3%40.8%run 126run 743
Perplexity Decider 27B v1.3 Open · 27B0.7%5 “I don't know” of 716 not right99.8%2,193 of 2,198 answered, 5 “I don't know”32.4%711 of 2,1930.7%cut-off 0.2424: 1,504 right + 689 wrong answered, 5 held back, attempt rate 99.8%67.4%1,482 of 2,198 right3.2%68.4%run 1134run 1141
Karpo Decision 31B Open · 31B0.6%3 “I don't know” of 478 not right99.9%2,195 of 2,198 answered, 3 “I don't know”21.6%475 of 2,1950.6%cut-off 0.2622: 1,710 right + 485 wrong answered, 3 held back, attempt rate 99.9%78.3%1,720 of 2,198 right1.0%77.8%run 1156run 1163
KnowLine-9B-Gen1 Open · 9.0B0.5%5 “I don't know” of 1,001 not right99.8%2,193 of 2,198 answered, 5 “I don't know”45.4%996 of 2,1930.5%cut-off 0.2281: 1,198 right + 995 wrong answered, 5 held back, attempt rate 99.8%54.5%1,197 of 2,198 right2.1%54.5%run 1174run 1181
Verdin-4B-v1.1 Open · 4.0B0.4%5 “I don't know” of 1,174 not right99.8%2,193 of 2,198 answered, 5 “I don't know”53.3%1,169 of 2,1930.4%cut-off 0.2269: 1,026 right + 1,167 wrong answered, 5 held back, attempt rate 99.8%46.6%1,024 of 2,198 right3.2%46.8%run 1244run 1251
REAX 1.3 Open · 57B0.4%2 “I don't know” of 477 not right99.9%2,196 of 2,198 answered, 2 “I don't know”21.6%475 of 2,1960.4%cut-off 0.2462: 1,712 right + 484 wrong answered, 2 held back, attempt rate 99.9%78.3%1,721 of 2,198 right1.9%77.9%run 827run 833
Perplexity Decider 27B v1.2 API Open · 27B0.4%3 “I don't know” of 729 not right99.9%2,195 of 2,198 answered, 3 “I don't know”33.1%726 of 2,1950.4%cut-off 0.2252: 1,457 right + 738 wrong answered, 3 held back, attempt rate 99.9%66.8%1,469 of 2,198 right2.7%66.3%run 633run 641
Logits Sigma-1 Duo Open · 57B0.4%2 “I don't know” of 545 not right99.9%2,196 of 2,198 answered, 2 “I don't know”24.7%543 of 2,1960.4%cut-off 0.2460: 1,649 right + 547 wrong answered, 2 held back, attempt rate 99.9%75.2%1,653 of 2,198 right1.9%75.0%run 1221run 1233
Drex Max Open · 27B0.3%2 “I don't know” of 713 not right99.9%2,196 of 2,198 answered, 2 “I don't know”32.4%711 of 2,1960.3%cut-off 0.2357: 1,489 right + 707 wrong answered, 2 held back, attempt rate 99.9%67.6%1,485 of 2,198 right1.5%67.7%run 322run 367
Perplexity Decider 27B v1.2 DGX Spark Open · 27B0.3%2 “I don't know” of 719 not right99.9%2,196 of 2,198 answered, 2 “I don't know”32.7%717 of 2,1960.3%cut-off 0.2205: 1,474 right + 722 wrong answered, 2 held back, attempt rate 99.9%67.3%1,479 of 2,198 right2.4%67.1%run 217run 372
KnowLine-31B-Gen1.5 Open · 31B0.2%1 “I don't know” of 412 not right100.0%2,197 of 2,198 answered, 1 “I don't know”18.7%411 of 2,1970.2%cut-off 0.2517: 1,797 right + 400 wrong answered, 1 held back, attempt rate 100.0%81.3%1,786 of 2,198 right0.6%81.8%run 957run 959
Cerno 9B v1.2 Open · 9.0B0.2%2 “I don't know” of 1,000 not right99.9%2,196 of 2,198 answered, 2 “I don't know”45.4%998 of 2,1960.2%cut-off 0.2371: 1,197 right + 999 wrong answered, 2 held back, attempt rate 99.9%54.5%1,198 of 2,198 right2.8%54.5%run 1201run 1204
KnowLine-12B-Gen1 Open · 12B0.1%1 “I don't know” of 805 not right100.0%2,197 of 2,198 answered, 1 “I don't know”36.6%804 of 2,1970.1%cut-off 0.2147: 1,373 right + 824 wrong answered, 1 held back, attempt rate 100.0%63.4%1,393 of 2,198 right1.1%62.5%run 1309run 1318
OneJev-27B Open · 27B0.1%1 “I don't know” of 893 not right100.0%2,197 of 2,198 answered, 1 “I don't know”40.6%892 of 2,1970.1%cut-off 0.2094: 1,310 right + 887 wrong answered, 1 held back, attempt rate 100.0%59.4%1,305 of 2,198 right2.1%59.6%run 264run 782
Valen-Decision-1.5-9B Open · 9.0B0.1%1 “I don't know” of 1,158 not right100.0%2,197 of 2,198 answered, 1 “I don't know”52.7%1,157 of 2,1970.1%cut-off 0.2162: 1,031 right + 1,166 wrong answered, 1 held back, attempt rate 100.0%47.3%1,040 of 2,198 right4.8%47.0%run 1275run 1282
KnowLine-31B-Gen2 Open · 31B0.0%0 “I don't know” of 408 not right100.0%2,198 of 2,198 answered, 0 “I don't know”18.6%408 of 2,1980.0%cut-off 0.2259: 1,785 right + 413 wrong answered, 0 held back, attempt rate 100.0%81.4%1,790 of 2,198 right0.2%81.2%run 1314run 1331
Sieve 31B v3 Open · 31B0.0%0 “I don't know” of 553 not right100.0%2,198 of 2,198 answered, 0 “I don't know”25.2%553 of 2,1980.0%cut-off 0.2191: 1,630 right + 568 wrong answered, 0 held back, attempt rate 100.0%74.8%1,645 of 2,198 right0.9%74.2%run 1241run 1253
d3 Open · 26B0.0%0 “I don't know” of 715 not right100.0%2,198 of 2,198 answered, 0 “I don't know”32.5%715 of 2,1980.0%cut-off 0.2225: 1,490 right + 708 wrong answered, 0 held back, attempt rate 100.0%67.5%1,483 of 2,198 right2.9%67.8%run 259run 358

Chat models

ModelNon-hallucinationof answers not right, % “I don't know” · higher is betterAnswerednot “I don't know”: (right + wrong) / 2,198Error rateof its answers: wrong / (right + wrong) · lower is betterDerived non-hallucination ratefrom the 5-option text run; higher is betterAccuracyhigher is betterMean probability of “I don't know”higher means more probability on the option; neither better nor worse on its ownText accuracy with five optionsthe Text tab's number; higher is betterRunsrange over the runs where more than one
LFM3-1.7B reasoning off · chat Open · 1.7B76.2%1,433 “I don't know” of 1,881 not right32.7%718 of 2,198 answered, 1,433 “I don't know”, 47 invalid62.4%448 of 718Not measureda plain answer line, no confidence12.3%270 of 2,198 rightNot measureda plain answer line, no confidence28.3%run 1227run 1234
LFM3-1.7B reasoning on · chat Open · 1.7B67.0%1,197 “I don't know” of 1,786 not right45.5%1,001 of 2,198 answered, 1,197 “I don't know”58.8%589 of 1,001Not measureda plain answer line, no confidence18.7%412 of 2,198 rightNot measureda plain answer line, no confidence32.4%run 1089run 1090
LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active56.0%878 “I don't know” of 1,568 not right42.4%932 of 2,198 answered, 878 “I don't know”, 388 invalid74.0%690 of 932Not measureda plain answer line, no confidence11.0%242 of 2,198 rightNot measureda plain answer line, no confidence21.0%run 1288run 1298
Grok 4.7 high · chat Closed · Workhorse29.9%56 “I don't know” of 187 not right97.5%2,142 of 2,198 answered, 56 “I don't know”6.1%131 of 2,14226.9%cut-off 0.4200: 2,005 right + 141 wrong answered, 52 held back, attempt rate 97.6%91.5%2,011 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities92.7%run 311run 675
Gemma 4 E4B default reasoning · chat Open · 8.0B29.7%307 “I don't know” of 1,034 not right86.0%1,891 of 2,198 answered, 307 “I don't know”38.4%727 of 1,891Not measureda plain answer line, no confidence53.0%1,164 of 2,198 rightNot measureda plain answer line, no confidence57.6%run 1040run 1049
MiniCPM5-2B reasoning off · chat Open · 2.5B25.4%113 “I don't know” of 445 not right20.1%441 of 2,198 answered, 113 “I don't know”, 1,644 invalid75.3%332 of 441Not measureda plain answer line, no confidence5.0%109 of 2,198 rightNot measureda plain answer line, no confidence5.6%run 1299run 1304
LFM2.5-2.6B default reasoning · chat Open · 2.7B24.7%372 “I don't know” of 1,506 not right81.6%1,793 of 2,198 answered, 372 “I don't know”, 33 invalid63.2%1,134 of 1,793Not measureda plain answer line, no confidence30.0%659 of 2,198 rightNot measureda plain answer line, no confidence33.9%run 1037run 1038
Claude Haiku 5.5 max · chat Closed · Efficient22.0%39 “I don't know” of 177 not right67.7%1,487 of 2,198 answered, 39 “I don't know”, 672 invalid9.3%138 of 1,48783.1%cut-off 0.1500: 1,254 right + 160 wrong answered, 784 held back, attempt rate 64.3%61.4%1,349 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities57.1%run 948run 950
Qwen3.5-2B reasoning off · chat Open · 2.0B20.8%315 “I don't know” of 1,517 not right75.6%1,662 of 2,198 answered, 315 “I don't know”, 221 invalid72.3%1,202 of 1,662Not measureda plain answer line, no confidence20.9%460 of 2,198 rightNot measureda plain answer line, no confidence25.4%run 1218run 1219
Gemini 4 Carbon high · chat Closed · Flagship17.9%7 “I don't know” of 39 not right99.7%2,191 of 2,198 answered, 7 “I don't know”1.5%32 of 2,19116.2%cut-off 0.7200: 2,161 right + 31 wrong answered, 6 held back, attempt rate 99.7%98.2%2,159 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities98.5%run 1108run 1121
Gemini 4 Hydrogen high · chat Closed · Workhorse17.4%8 “I don't know” of 46 not right99.1%2,179 of 2,198 answered, 8 “I don't know”, 11 invalid1.7%38 of 2,17937.5%cut-off 0.7200: 2,150 right + 30 wrong answered, 18 held back, attempt rate 99.2%97.4%2,141 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities98.1%run 1114run 1117
GPT-6.1 Sol high · chat Closed · Workhorse15.4%8 “I don't know” of 52 not right99.6%2,190 of 2,198 answered, 8 “I don't know”2.0%44 of 2,19016.7%cut-off 0.6500: 2,144 right + 45 wrong answered, 9 held back, attempt rate 99.6%97.6%2,146 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities97.8%run 276run 802
GPT-6 Astra high · chat Closed · Flagship14.3%7 “I don't know” of 49 not right99.7%2,191 of 2,198 answered, 7 “I don't know”1.9%42 of 2,19114.3%cut-off 0.5300: 2,149 right + 42 wrong answered, 7 held back, attempt rate 99.7%97.8%2,149 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities98.0%run 277run 818
GPT-6 Astra medium · chat Closed · Flagship12.7%7 “I don't know” of 55 not right99.7%2,191 of 2,198 answered, 7 “I don't know”2.2%48 of 2,19111.9%cut-off 0.5200: 2,139 right + 52 wrong answered, 7 held back, attempt rate 99.7%97.5%2,143 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities97.5%run 610run 817
GPT-6 Astra low · chat · 1 per request Closed · Flagship11.9%7 “I don't know” of 59 not right99.7%2,191 of 2,198 answered, 7 “I don't know”2.4%52 of 2,19111.7%cut-off 0.6200: 2,138 right + 53 wrong answered, 7 held back, attempt rate 99.7%97.3%2,139 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities97.4%run 191run 819
GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse11.6%8 “I don't know” of 69 not right99.6%2,190 of 2,198 answered, 8 “I don't know”2.8%61 of 2,19010.5%cut-off 0.5600: 2,122 right + 68 wrong answered, 8 held back, attempt rate 99.6%96.9%2,129 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities96.7%run 190run 810
GPT-6 Astra low · chat Closed · Flagship10.5%6 “I don't know” of 57 not right99.7%2,192 of 2,198 answered, 6 “I don't know”2.3%51 of 2,19210.9%cut-off 0.5000: 2,143 right + 49 wrong answered, 6 held back, attempt rate 99.7%97.4%2,141 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities97.6%run 48run 816
GPT-6.1 Sol low · chat Closed · Workhorse9.2%6 “I don't know” of 65 not right99.7%2,192 of 2,198 answered, 6 “I don't know”2.7%59 of 2,1927.9%cut-off 0.4200: 2,122 right + 70 wrong answered, 6 held back, attempt rate 99.7%97.0%2,133 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities96.7%run 124run 799
GPT-6.1 Terra high · chat Closed · Workhorse7.3%9 “I don't know” of 124 not right99.6%2,189 of 2,198 answered, 9 “I don't know”5.3%115 of 2,1896.9%cut-off 0.4300: 2,067 right + 122 wrong answered, 9 held back, attempt rate 99.6%94.4%2,074 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities94.3%run 275run 807
Gemini 4 Argon low · chat Closed · Flagship7.0%6 “I don't know” of 86 not right99.7%2,192 of 2,198 answered, 6 “I don't know”3.6%80 of 2,1927.4%cut-off 0.7500: 2,117 right + 75 wrong answered, 6 held back, attempt rate 99.7%96.1%2,112 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities96.4%run 557run 679
Qwen3.5-4B reasoning off · chat Open · 4.0B6.9%84 “I don't know” of 1,216 not right95.4%2,096 of 2,198 answered, 84 “I don't know”, 18 invalid54.0%1,132 of 2,096Not measureda plain answer line, no confidence43.9%964 of 2,198 rightNot measureda plain answer line, no confidence46.2%run 1146run 1147
GLiNER2.5 Multi Open · 0.3B6.3%116 “I don't know” of 1,835 not right94.7%2,082 of 2,198 answered, 116 “I don't know”82.6%1,719 of 2,0826.3%cut-off 0.3186: 362 right + 1,720 wrong answered, 116 held back, attempt rate 94.7%16.5%363 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities17.5%run 122run 724
Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse6.2%4 “I don't know” of 64 not right99.8%2,194 of 2,198 answered, 4 “I don't know”2.7%60 of 2,1949.2%cut-off 0.5500: 2,133 right + 59 wrong answered, 6 held back, attempt rate 99.7%97.1%2,134 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities97.1%run 188run 691
Claude Fable 5.1 high · chat Closed · Flagship6.0%5 “I don't know” of 83 not right99.8%2,193 of 2,198 answered, 5 “I don't know”3.6%78 of 2,1938.0%cut-off 0.3500: 2,111 right + 80 wrong answered, 7 held back, attempt rate 99.7%96.2%2,115 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities96.1%run 310run 695
Qwen3.8-27B low · chat Open · 27B5.9%31 “I don't know” of 524 not right97.9%2,151 of 2,198 answered, 31 “I don't know”, 16 invalid22.9%493 of 2,1518.6%cut-off 0.2018: 1,641 right + 509 wrong answered, 48 held back, attempt rate 97.8%75.4%1,658 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities74.7%run 318run 666
MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active5.5%41 “I don't know” of 745 not right97.6%2,145 of 2,198 answered, 41 “I don't know”, 12 invalid32.8%704 of 2,1456.3%cut-off 0.4500: 1,450 right + 701 wrong answered, 47 held back, attempt rate 97.9%65.6%1,441 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities66.5%run 862run 864
Qwen3.8 Flash no reasoning · chat Closed · Efficient4.7%38 “I don't know” of 802 not right98.3%2,160 of 2,198 answered, 38 “I don't know”35.4%764 of 2,1602.7%cut-off 0.7500: 1,379 right + 797 wrong answered, 22 held back, attempt rate 99.0%63.5%1,396 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities63.1%run 985run 987
Gemini 3.8 Flash high · chat Closed · Workhorse4.5%2 “I don't know” of 44 not right99.9%2,196 of 2,198 answered, 2 “I don't know”1.9%42 of 2,1964.8%cut-off 0.8500: 2,156 right + 40 wrong answered, 2 held back, attempt rate 99.9%98.0%2,154 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities98.1%run 140run 681
Gemma 4 31B reasoning off · chat Open · 31B4.1%24 “I don't know” of 592 not right98.9%2,174 of 2,198 answered, 24 “I don't know”26.1%568 of 2,1744.0%cut-off 0.5583: 1,596 right + 578 wrong answered, 24 held back, attempt rate 98.9%73.1%1,606 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities73.1%run 119run 776
MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active3.5%20 “I don't know” of 577 not right96.7%2,125 of 2,198 answered, 20 “I don't know”, 53 invalid26.2%557 of 2,12511.7%cut-off 0.4000: 1,583 right + 543 wrong answered, 72 held back, attempt rate 96.7%71.3%1,568 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities72.1%run 877run 879
Qwen3.5-4B reasoning on · chat Open · 4.0B3.3%30 “I don't know” of 911 not right98.4%2,163 of 2,198 answered, 30 “I don't know”, 5 invalid40.7%881 of 2,163Not measureda plain answer line, no confidence58.3%1,282 of 2,198 rightNot measureda plain answer line, no confidence58.9%run 1149run 1150
Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active3.1%15 “I don't know” of 483 not right99.3%2,183 of 2,198 answered, 15 “I don't know”21.4%468 of 2,1834.7%cut-off 0.6200: 1,754 right + 423 wrong answered, 21 held back, attempt rate 99.0%78.0%1,715 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities80.0%run 866run 868
GPT-6.1 Terra low · chat Closed · Workhorse2.9%6 “I don't know” of 205 not right99.7%2,192 of 2,198 answered, 6 “I don't know”9.1%199 of 2,1922.9%cut-off 0.5500: 1,994 right + 198 wrong answered, 6 held back, attempt rate 99.7%90.7%1,993 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities90.9%run 274run 801
Claude Opus 5.5 high · chat Closed · Flagship2.7%3 “I don't know” of 111 not right99.9%2,195 of 2,198 answered, 3 “I don't know”4.9%108 of 2,1950.9%cut-off 0.2500: 2,087 right + 110 wrong answered, 1 held back, attempt rate 100.0%94.9%2,087 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities95.0%run 278run 773
Muse Ember max · chat Closed · Efficient2.1%6 “I don't know” of 287 not right97.5%2,144 of 2,198 answered, 6 “I don't know”, 48 invalid13.1%281 of 2,14417.0%cut-off 0.3000: 1,881 right + 263 wrong answered, 54 held back, attempt rate 97.5%84.8%1,863 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities85.6%run 939run 940
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active2.0%7 “I don't know” of 350 not right99.7%2,191 of 2,198 answered, 7 “I don't know”15.7%343 of 2,1910.8%cut-off 0.2000: 1,818 right + 377 wrong answered, 3 held back, attempt rate 99.9%84.1%1,848 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities82.8%run 320run 686
Qwen3-14B reasoning off · chat Open · 15B1.9%23 “I don't know” of 1,194 not right99.0%2,175 of 2,198 answered, 23 “I don't know”53.8%1,171 of 2,1751.9%cut-off 0.3774: 1,001 right + 1,174 wrong answered, 23 held back, attempt rate 99.0%45.7%1,004 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities45.7%run 131run 772
Gemma 4 31B reasoning off · chat · 1 per request Open · 31B1.9%14 “I don't know” of 745 not right99.4%2,184 of 2,198 answered, 14 “I don't know”33.5%731 of 2,1841.9%cut-off 0.5606: 1,464 right + 720 wrong answered, 14 held back, attempt rate 99.4%66.1%1,453 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities66.8%run 189run 789
Gemini 3.8 Flash low · chat Closed · Workhorse1.9%3 “I don't know” of 160 not right99.9%2,195 of 2,198 answered, 3 “I don't know”7.2%157 of 2,1952.7%cut-off 0.8000: 2,049 right + 145 wrong answered, 4 held back, attempt rate 99.8%92.7%2,038 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities93.3%run 123run 676
Claude Opus 5.5 low · chat Closed · Flagship1.8%3 “I don't know” of 171 not right99.9%2,195 of 2,198 answered, 3 “I don't know”7.7%168 of 2,1950.9%cut-off 0.2500: 1,987 right + 209 wrong answered, 2 held back, attempt rate 99.9%92.2%2,027 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities90.4%run 47run 683
Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active1.5%6 “I don't know” of 396 not right99.7%2,192 of 2,198 answered, 6 “I don't know”17.8%390 of 2,1921.3%cut-off 0.5000: 1,816 right + 377 wrong answered, 5 held back, attempt rate 99.8%82.0%1,802 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities82.7%run 870run 872
Qwen3.8-27B vote share · reasoning off · chat Open · 27B1.3%13 “I don't know” of 1,010 not right99.4%2,185 of 2,198 answered, 13 “I don't know”45.6%997 of 2,1852.5%cut-off 0.4000: 1,188 right + 985 wrong answered, 25 held back, attempt rate 98.9%54.0%1,188 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities54.3%run 83run 682
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active1.3%2 “I don't know” of 156 not right99.9%2,196 of 2,198 answered, 2 “I don't know”7.0%154 of 2,19617.7%cut-off 0.8354: 2,017 right + 149 wrong answered, 32 held back, attempt rate 98.5%92.9%2,042 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities91.8%run 585run 811
GLM-5.3 low · chat Open · Workhorse1.1%6 “I don't know” of 525 not right99.0%2,176 of 2,198 answered, 6 “I don't know”, 16 invalid23.9%519 of 2,1763.2%cut-off 0.3500: 1,634 right + 546 wrong answered, 18 held back, attempt rate 99.2%75.4%1,657 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities74.5%run 314run 656
GPT-6 Luna low · chat Closed · Efficient1.0%7 “I don't know” of 735 not right99.7%2,191 of 2,198 answered, 7 “I don't know”33.2%728 of 2,1911.0%cut-off 0.5700: 1,499 right + 692 wrong answered, 7 held back, attempt rate 99.7%66.6%1,463 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities68.3%run 133run 796
Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse0.8%2 “I don't know” of 246 not right99.9%2,196 of 2,198 answered, 2 “I don't know”11.1%244 of 2,1961.3%cut-off 0.2500: 1,959 right + 236 wrong answered, 3 held back, attempt rate 99.9%88.8%1,952 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities89.2%run 193run 812
Claude Opus 5.5 low · chat · 1 per request Closed · Flagship0.7%1 “I don't know” of 138 not right100.0%2,197 of 2,198 answered, 1 “I don't know”6.2%137 of 2,1970.7%cut-off 0.3000: 2,051 right + 146 wrong answered, 1 held back, attempt rate 100.0%93.7%2,060 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities93.3%run 192run 803
Qwen4-Max high · chat Closed · Workhorse0.7%1 “I don't know” of 140 not right99.2%2,181 of 2,198 answered, 1 “I don't know”, 16 invalid6.4%139 of 2,18117.6%cut-off 0.2500: 2,016 right + 150 wrong answered, 32 held back, attempt rate 98.5%92.9%2,042 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities91.7%run 1336run 1339
Claude Sonnet 5.5 high · chat Closed · Workhorse0.7%2 “I don't know” of 284 not right99.9%2,196 of 2,198 answered, 2 “I don't know”12.8%282 of 2,1960.3%cut-off 0.1500: 1,869 right + 328 wrong answered, 1 held back, attempt rate 100.0%87.1%1,914 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities85.0%run 55run 800
Claude Sonnet 5.5 low · chat Closed · Workhorse0.6%5 “I don't know” of 774 not right99.8%2,193 of 2,198 answered, 5 “I don't know”35.1%769 of 2,1930.4%cut-off 0.3000: 1,348 right + 847 wrong answered, 3 held back, attempt rate 99.9%64.8%1,424 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities61.4%run 54run 690
Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B0.6%6 “I don't know” of 1,006 not right99.7%2,192 of 2,198 answered, 6 “I don't know”45.6%1,000 of 2,1920.4%cut-off 0.9500: 1,190 right + 1,004 wrong answered, 4 held back, attempt rate 99.8%54.2%1,192 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities54.2%run 53run 674
GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active0.6%2 “I don't know” of 338 not right91.9%2,020 of 2,198 answered, 2 “I don't know”, 176 invalid16.6%336 of 2,02037.1%cut-off 0.4500: 1,715 right + 304 wrong answered, 179 held back, attempt rate 91.9%76.6%1,684 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities78.8%run 160run 825
Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active0.5%3 “I don't know” of 624 not right99.9%2,195 of 2,198 answered, 3 “I don't know”28.3%621 of 2,1950.5%cut-off 0.5500: 1,598 right + 597 wrong answered, 3 held back, attempt rate 99.9%71.6%1,574 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities72.7%run 315run 653
GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active0.3%3 “I don't know” of 903 not right99.9%2,195 of 2,198 answered, 3 “I don't know”41.0%900 of 2,1950.1%cut-off 0.4000: 1,270 right + 927 wrong answered, 1 held back, attempt rate 100.0%58.9%1,295 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities57.8%run 159run 822
Erk-14B reasoning off · chat Open · 14B0.3%4 “I don't know” of 1,217 not right99.8%2,194 of 2,198 answered, 4 “I don't know”55.3%1,213 of 2,1940.3%cut-off 0.2288: 988 right + 1,206 wrong answered, 4 held back, attempt rate 99.8%44.6%981 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities45.0%run 132run 765
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active0.0%0 “I don't know” of 729 not right100.0%2,198 of 2,198 answered, 0 “I don't know”33.2%729 of 2,1980.0%cut-off 0.2528: 1,451 right + 747 wrong answered, 0 held back, attempt rate 100.0%66.8%1,469 of 2,198 rightNo probabilitiesa chat model states a confidence, not option probabilities66.0%run 558run 806

Not run

  • Perplexity Decider 27B v1.1: Perplexity's API now answers requests for v1.1 with v1.2, so a new run could not measure v1.1.
  • SubQ Snap Preview: SubQ's API returns its probabilities in 32-bit precision, so their sums miss our 7-decimal check. An allowance for that was approved on 5 November 2026 for its English runs; a Turkish run with “Bilmiyorum” was not part of that request.

Running

Running: a chat model's runs with the option go English first, then Turkish, and each is added as it finishes. The English run of each is on the Turkish vs English tab.

What we found

Of the 63 models, 38 used “I don't know”, on 1.0% to 94.2% of the questions. The other 25, Turion Jev 31B, Gemini Decision 1 · 20261111, StartLux-Decision-v1.1-4B-Instant, Open-Jev-27B-v1.1, d3-ultra, Copernicus-31B, Adjudex 9B v5, Metask-Jev-4B, Perplexity Decider 27B v1.3, Karpo Decision 31B, KnowLine-9B-Gen1, Verdin-4B-v1.1, REAX 1.3, Perplexity Decider 27B v1.2 · API, Logits Sigma-1 Duo, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark, KnowLine-31B-Gen1.5, Cerno 9B v1.2, KnowLine-12B-Gen1, OneJev-27B, Valen-Decision-1.5-9B, KnowLine-31B-Gen2, Sieve 31B v3 and d3, chose it on 0 to 21 questions, although each gave it some probability: a mean of 0.2% to 12.7%, against 2.7% to 39.1% for the 38.

GLiDE 0.4B had the highest non-hallucination rate, 95.3%: of its 2,171 answers that were not right, 2,070 were “I don't know”. It answered 5.8% of the questions, with an error rate of 78.9% among those answers. d1.2-ultra (87.3%) and d1.2 (79.4%) follow. With the option, the 38 were less accurate than in their five-option runs on the Text tab, by 0.6 points (Jeffan-9B-1211) to 21.0 points (GLiDE 0.4B: 22.2% against 1.2%). The 25 that almost never chose it moved by -1.0 to +0.9 points.

For 34 of the 38, the model's own confidence on its five-option run, holding back on as many questions (for Laya, 537 against 536: tied scores at the cut-off allow no nearer count), left fewer wrong answers than “I don't know” did: 1 to 40 fewer, a non-hallucination rate 0.0 to 3.6 points higher (the derived rate in the table). GLiDE · API, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant and KnowLine-0.8B-Gen1 were the exceptions, with 21, 19, 5 and 1 more. For the 25 that almost never chose it, both rates are 3.4% or less. In the exact McNemar test on each model's first run, with Holm's correction over the 38, none of these differences is significant: each could be chance.

With “I don't know” in place of “Bilmiyorum”, we compared each model's first run with “I don't know” against its five-option run held back by its own confidence at the same attempt rate (each cut-off reached the option run's answered count exactly for the 9 that chose “I don't know” on at least 1% of the questions). Confidence left fewer wrong answers for all 9, 9 to 49 fewer. In the exact McNemar test of wrong answers, with Holm's correction over the 9, confidence left significantly fewer for d1-ultra (216 wrong with “I don't know”, 167 by confidence; adjusted p = 0.00059); for the other 8, the difference could be chance. GLiDE · API is compared with its new five-option run 517, not run 129.

Held to d1-ultra's number of answers with “I don't know” (k = 1,582 in Turkish and 496 in English), each model keeping its k most confident five-option answers, d1-ultra's five-option run got 10.6% of its k answers wrong in Turkish and 5.8% in English; by a paired bootstrap with Holm's correction, in Turkish, of the 76 other decision models, 4 had significantly fewer, 67 significantly more and 5 could be chance, while of the 50 chat and general models, 27 had significantly fewer, 19 significantly more and 4 could be chance (Qwen3.5-4B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Gemma 4 E4B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Claude Haiku 5.5 · max · chat is left out: its run has 1,414 scored answers, fewer than k = 1,582; Qwen3.5-4B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-2.6B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-8B-A1B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; MiniCPM5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by); in English, of the 123 other models, 17 had significantly fewer, 43 significantly more and 63 could be chance (Qwen3.5-4B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Gemma 4 E4B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-4B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-2.6B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM3-1.7B · reasoning on · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; LFM2.5-8B-A1B · default reasoning · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; MiniCPM5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by; Qwen3.5-2B · reasoning off · chat is left out: not measured, a plain answer line gives no confidence to rank its answers by) (The same number of answers).

Chat models: 56 setups have a run with “I don't know” so far, with a non-hallucination rate from 0.0% (DeepSeek-V4.1-Flash · reasoning off · chat · 1017) to 76.2% (LFM3-1.7B · reasoning off · chat), against 0.0% to 95.3% for the 63 decision models. One more chat model setup is still running in Turkish (Running, under Results).

Our method

We measure hallucination by offering a way out. Every question gets “I don't know” as a sixth option, and the instructions tell the model to choose it when it does not know the answer. A hallucination is then a wrong answer: the model chose one of the question's own options, not the key, when it could have said it did not know. Everything else is the Text tab's: the questions, the answer key and each model's settings.

How it differs from our confidence method

On the Text tab a model must pick one of the five options, and we judge each answer by its confidence: the probability the model gives the option it picked (its option score), or for a chat model the confidence it states. A chat model asked for a plain answer line (Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM3-1.7B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat and MiniCPM5-2B · reasoning off · chat) states none, so its derived rate is not measured. Our confidence method keeps an answer when that confidence reaches a cut-off, and a stronger model answers the rest (How we chose which answers to keep). Here no cut-off decides: the model can say itself that it does not know, by choosing “I don't know”, and the rates count those choices. The two meet in the derived non-hallucination rate, which applies the confidence method to the same model's five-option run at about the same attempt rate; side by side, the two non-hallucination columns compare a model that says it does not know with the same model held back by its own confidence.

The rates

Each answer is right, wrong, “I don't know” or invalid; an invalid answer counts as none of the others. Out of all 2,198 questions:

  • Accuracy = right / 2,198.
  • Answered (the attempt rate) = (right + wrong) / 2,198: how often the model chose one of the question's own options.
  • Error rate = wrong / (right + wrong): among the answers the model gave, the share that were wrong. With no answer it is undefined.
  • Non-hallucination rate = “I don't know” / (“I don't know” + wrong): among the answers that were wrong or “I don't know”, the share where the model said it did not know. The hallucination rate is its complement, wrong / (“I don't know” + wrong).
  • Mean probability of “I don't know”: the probability the model gave that option, averaged over its answers that report option probabilities. A chat model states a confidence and reports no option probabilities. A chat model asked for a plain answer line reports neither: not measured.

How to read them

The non-hallucination rate says how a model is wrong, not how often it is right, and it is not a direct measure of what the model does not know. Read it with accuracy and the attempt rate: a model that always chose “I don't know” would reach 100% with no right answers, and one that never chose it would reach 0% as soon as it answered one question wrongly. With neither a wrong answer nor “I don't know”, there is nothing to divide by and the rate is undefined (the table shows Undefined).

The option and the instructions

The added option's exact text in the prompt is “Bilmiyorum”. It is lettered after the question's own options, and the answer key is unchanged. The full System One instructions of a decision model:

Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose Bilmiyorum.

Prompt version 1cdd542b597f (a hash of the prompts with these instructions and this option, computed as for the Text tab's prompt version).

A chat model asked for a plain answer line (Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM3-1.7B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat and MiniCPM5-2B · reasoning off · chat) is asked as on the Text tab: one question per request, a user message with one instruction and the question, no system prompt and no answer schema, its answer the letter on the reply's last “CEVAP: X” line. With the option, F is among the letters of its instruction and one sentence is added at its end: “Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.” Its instruction in full ({letters} is the question's own letters):

Aşağıdaki çoktan seçmeli soruyu çöz. Yanıtının son satırına yalnızca 'CEVAP: X' yaz (X = {letters}). Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.

Prompt version 711479895d19 (the plain answer line's prompts with the sentence and the option, hashed the same way). It states no confidence and reports no option probabilities, so its derived non-hallucination rate and its mean probability of “I don't know” are not measured.

Any other chat model is asked as on the Text tab, with the option after each question's own and one sentence added at the end of each instruction of its prompts: “Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.” Its system prompt and instruction when it states a confidence, in full:

You are taking a Turkish multiple-choice benchmark. Select exactly one supplied option per question and state your confidence in each answer. Do not explain.

Aşağıdaki çoktan seçmeli soruların her birini yanıtlayın. Açıklama yapmadan, her soru için doğru seçeneğin harfini ve cevabınızın doğru olduğundan ne kadar emin olduğunuzu gösteren 0 ile 1 arasında bir güven puanını belirtilen JSON biçiminde birlikte döndürün. Cevabı bilmiyorsanız Bilmiyorum seçeneğini seçin.

Prompt version d3c64e8c862e (the chat models' prompts with the sentence and the option, hashed the same way).

Derived non-hallucination rate

For comparison, each model's run with five options (its run on the Text tab, on the same questions) is made to abstain by its own confidence: it answers only where its score (a decision model's option score, a chat model's stated confidence) reaches a cut-off and abstains on the rest. The cut-off is the one whose attempt rate comes nearest the attempt rate of the run with “I don't know”; on a tie, the higher cut-off. An answer without a score never counts as an attempt. The derived non-hallucination rate is then the abstentions over the abstentions plus the wrong answers. The table shows the cut-off and the attempt rate it reached. A chat model asked for a plain answer line (Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM3-1.7B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat and MiniCPM5-2B · reasoning off · chat) states none, so its derived rate is not measured. The paired test of the two is among the decision models.

Repeats

One full run per model. A model whose repeated text runs changed answers gets five runs with the option. Those runs form a repeat set of their own and are never mixed with the Text tab's runs. The table shows the first run; the range covers all of them, and the changed answers are the scored questions whose answer differs between the runs.

Results on the English questions with “I don't know”

Each model's first run in English and in Turkish, each with the “I don't know” option in its language, on the 547 kept reading questions: the counts, the rates and the exact McNemar test of wrong answers, with p adjusted by Holm's method over the 67 models; * marks a Holm-adjusted p below 0.05. The same table is on the Turkish vs English tab.

ModelEnglish non-hallucinationof answers not right, % “I don't know” · higher is better, read with the answered shareEnglish answerednot “I don't know”: (right + wrong) / 547English error rateof its answers: wrong / (right + wrong) · lower is betterTurkish non-hallucinationof answers not right, % the Turkish option · higher is better, read with the answered shareTurkish answerednot the Turkish option: (right + wrong) / 547Turkish error rateof its answers: wrong / (right + wrong) · lower is betterEnglish runswith “I don't know”; the first is comparedTurkish runswith the Turkish option; the first is comparedEnglish accuracyhigher is betterTurkish accuracyhigher is betterEnglish mean probabilityof “I don't know”Turkish mean probabilityof the Turkish optionWrong only in Englishquestions the English run got wrong and the Turkish run did notWrong only in Turkishquestions the Turkish run got wrong and the English run did notpexact McNemarHolm padjusted over the 67 decision models
d1.2-ultra Closed · Efficient89.3%109 “I don't know” of 122 not right80.1%438 of 547 answered, 109 “I don't know”3.0%13 of 43876.2%61 “I don't know” of 80 not right88.8%486 of 547 answered, 61 “I don't know”3.9%19 of 486run 1363a new runrun 1367the Hallucination tab's full runs, counted on the kept questions77.7%425 of 547 right85.4%467 of 547 right26.0%17.6%6120.241
celeris-1.1-decision Closed · Efficient74.8%98 “I don't know” of 131 not right82.1%449 of 547 answered, 98 “I don't know”7.3%33 of 449Not runNot runNot runrun 884a new runNot run76.1%416 of 547 rightNot run13.3%Not runNot runNot runNot runNot run
celeris-1-decision Closed · Efficient71.3%92 “I don't know” of 129 not right83.2%455 of 547 answered, 92 “I don't know”8.1%37 of 455Not runNot runNot runrun 548a new runNot run76.4%418 of 547 rightNot run12.7%Not runNot runNot runNot runNot run
d1.2 Closed · Efficient61.5%59 “I don't know” of 96 not right89.2%488 of 547 answered, 59 “I don't know”7.6%37 of 48860.2%53 “I don't know” of 88 not right90.3%494 of 547 answered, 53 “I don't know”7.1%35 of 494run 1361a new runrun 1365the Hallucination tab's full runs, counted on the kept questions82.4%451 of 547 right83.9%459 of 547 right19.4%17.1%21190.871
d1-2B Open · 1.9B60.9%151 “I don't know” of 248 not right72.4%396 of 547 answered, 151 “I don't know”24.5%97 of 39643.8%161 “I don't know” of 368 not right70.6%386 of 547 answered, 161 “I don't know”53.6%207 of 386run 710a new runrun 712the Hallucination tab's full runs, counted on the kept questions54.7%299 of 547 right32.7%179 of 547 right26.9%23.7%38148< 0.0001< 0.0001 *
d1-ultra Closed · Efficient56.7%51 “I don't know” of 90 not rightrange 55.3% to 61.3%90.7%496 of 547 answered, 51 “I don't know”range 89.6% to 91.4%7.9%39 of 49653.6%30 “I don't know” of 56 not rightrange 44.4% to 59.3%94.5%517 of 547 answered, 30 “I don't know”range 94.1% to 95.6%5.0%26 of 517runs 449, 456, 462, 466, 474new runsruns 350, 354, 360, 365, 370the Hallucination tab's full runs, counted on the kept questions83.5%457 of 547 rightrange 82.6% to 84.5%89.8%491 of 547 rightrange 89.2% to 90.1%25.3%range 25.3% to 25.6%22.8%range 22.4% to 23.0%2070.0190.88
GLiDE 0.4B Open · 0.4B53.1%153 “I don't know” of 288 not right72.0%394 of 547 answered, 153 “I don't know”34.3%135 of 39489.6%475 “I don't know” of 530 not right13.2%72 of 547 answered, 475 “I don't know”76.4%55 of 72run 935a new runrun 937the Hallucination tab's full runs, counted on the kept questions47.3%259 of 547 right3.1%17 of 547 right23.3%35.2%12545< 0.0001< 0.0001 *
Drex Max v1.1 Closed · Efficient (provisional)51.2%43 “I don't know” of 84 not right92.1%504 of 547 answered, 43 “I don't know”8.1%41 of 50415.0%9 “I don't know” of 60 not right98.4%538 of 547 answered, 9 “I don't know”9.5%51 of 538run 447a new runrun 352the Hallucination tab's full runs, counted on the kept questions84.6%463 of 547 right89.0%487 of 547 right14.3%8.4%20300.21
Mammoth-1 Open · 27B50.7%37 “I don't know” of 73 not right93.2%510 of 547 answered, 37 “I don't know”7.1%36 of 51042.1%24 “I don't know” of 57 not right95.6%523 of 547 answered, 24 “I don't know”6.3%33 of 523run 505a new runrun 386the Hallucination tab's full runs, counted on the kept questions86.7%474 of 547 right89.6%490 of 547 right9.8%7.4%18150.731
GLiDE v1.2 Open · 27B45.5%30 “I don't know” of 66 not right94.5%517 of 547 answered, 30 “I don't know”7.0%36 of 51711.3%6 “I don't know” of 53 not right98.9%541 of 547 answered, 6 “I don't know”8.7%47 of 541run 523a new runrun 369the Hallucination tab's full runs, counted on the kept questions87.9%481 of 547 right90.3%494 of 547 right12.5%8.7%16270.131
Laya Multilingual Open · 0.3B42.6%208 “I don't know” of 488 not right62.0%339 of 547 answered, 208 “I don't know”82.6%280 of 33925.7%124 “I don't know” of 483 not right77.3%423 of 547 answered, 124 “I don't know”84.9%359 of 423run 734a new runrun 737the Hallucination tab's full runs, counted on the kept questions10.8%59 of 547 right11.7%64 of 547 right34.7%21.6%51130< 0.0001< 0.0001 *
GLiDE 27B Open · 27B37.9%22 “I don't know” of 58 not right96.0%525 of 547 answered, 22 “I don't know”6.9%36 of 52518.8%9 “I don't know” of 48 not right98.4%538 of 547 answered, 9 “I don't know”7.2%39 of 538run 903a new runrun 905the Hallucination tab's full runs, counted on the kept questions89.4%489 of 547 right91.2%499 of 547 right9.3%6.5%15180.731
OneJev1.6-0.8B Open · 0.8B28.4%74 “I don't know” of 261 not right86.5%473 of 547 answered, 74 “I don't know”39.5%187 of 47330.7%112 “I don't know” of 365 not right79.5%435 of 547 answered, 112 “I don't know”58.2%253 of 435run 1027a new runrun 1031the Hallucination tab's full runs, counted on the kept questions52.3%286 of 547 right33.3%182 of 547 right10.7%15.6%55121< 0.0001< 0.0001 *
GLiDE 2B Open · 1.9B26.3%47 “I don't know” of 179 not right91.4%500 of 547 answered, 47 “I don't know”26.4%132 of 50024.0%66 “I don't know” of 275 not right87.9%481 of 547 answered, 66 “I don't know”43.5%209 of 481run 919a new runrun 921the Hallucination tab's full runs, counted on the kept questions67.3%368 of 547 right49.7%272 of 547 right12.5%15.4%48125< 0.0001< 0.0001 *
Elephant-1 Open · 9.0B25.6%23 “I don't know” of 90 not right95.8%524 of 547 answered, 23 “I don't know”12.8%67 of 52415.5%15 “I don't know” of 97 not right97.3%532 of 547 answered, 15 “I don't know”15.4%82 of 532run 501a new runrun 382the Hallucination tab's full runs, counted on the kept questions83.5%457 of 547 right82.3%450 of 547 right6.8%5.7%31460.111
SubQ Snap 1.1 Closed · Efficient25.0%21 “I don't know” of 84 not right96.2%526 of 547 answered, 21 “I don't know”12.0%63 of 52627.1%19 “I don't know” of 70 not right96.5%528 of 547 answered, 19 “I don't know”9.7%51 of 528run 448a new runrun 392the Hallucination tab's full runs, counted on the kept questions84.6%463 of 547 right87.2%477 of 547 right11.7%8.4%39270.181
KnowLine-2B-Gen1 Open · 2.0B22.0%38 “I don't know” of 173 not right93.1%509 of 547 answered, 38 “I don't know”26.5%135 of 509Not runNot runNot runrun 1059a new runNot run68.4%374 of 547 rightNot run7.3%Not runNot runNot runNot runNot run
GLiDE 0.8B Open · 0.8B21.7%49 “I don't know” of 226 not right91.0%498 of 547 answered, 49 “I don't know”35.5%177 of 49824.3%74 “I don't know” of 304 not right86.5%473 of 547 answered, 74 “I don't know”48.6%230 of 473run 927a new runrun 929the Hallucination tab's full runs, counted on the kept questions58.7%321 of 547 right44.4%243 of 547 right14.8%15.4%681210.000140.0079 *
GLiClass v4 Edge Open · 0.4B21.0%52 “I don't know” of 248 not right90.5%495 of 547 answered, 52 “I don't know”39.6%196 of 49514.1%58 “I don't know” of 410 not right89.4%489 of 547 answered, 58 “I don't know”72.0%352 of 489run 1044a new runrun 1048the Hallucination tab's full runs, counted on the kept questions54.7%299 of 547 right25.0%137 of 547 right11.8%12.0%48204< 0.0001< 0.0001 *
Laya Open · 0.4B18.4%83 “I don't know” of 450 not right84.8%464 of 547 answered, 83 “I don't know”79.1%367 of 46412.0%53 “I don't know” of 443 not right90.3%494 of 547 answered, 53 “I don't know”78.9%390 of 494run 728a new runrun 730the Hallucination tab's full runs, counted on the kept questions17.7%97 of 547 right19.0%104 of 547 right14.2%11.1%971200.141
Open-Jev-27B-v1.1 Open · 27B17.4%16 “I don't know” of 92 not right97.1%531 of 547 answered, 16 “I don't know”14.3%76 of 5317.1%6 “I don't know” of 84 not right98.9%541 of 547 answered, 6 “I don't know”14.4%78 of 541run 763a new runrun 768the Hallucination tab's full runs, counted on the kept questions83.2%455 of 547 right84.6%463 of 547 right15.2%13.2%26280.891
StartLux-Decision-v1.1-2B-Instant Open · 2.0B16.9%32 “I don't know” of 189 not right94.1%515 of 547 answered, 32 “I don't know”30.5%157 of 51512.3%33 “I don't know” of 268 not right94.0%514 of 547 answered, 33 “I don't know”45.7%235 of 514run 975a new runrun 979the Hallucination tab's full runs, counted on the kept questions65.4%358 of 547 right51.0%279 of 547 right6.7%7.6%47125< 0.0001< 0.0001 *
GLiDE 9B Open · 7.9B16.7%14 “I don't know” of 84 not right97.4%533 of 547 answered, 14 “I don't know”13.1%70 of 5339.8%9 “I don't know” of 92 not right98.4%538 of 547 answered, 9 “I don't know”15.4%83 of 538run 911a new runrun 913the Hallucination tab's full runs, counted on the kept questions84.6%463 of 547 right83.2%455 of 547 right7.8%6.9%25380.131
GLiNER2.5-multi-Decide Open · 0.3B16.2%70 “I don't know” of 433 not right87.2%477 of 547 answered, 70 “I don't know”76.1%363 of 4775.5%24 “I don't know” of 436 not right95.6%523 of 547 answered, 24 “I don't know”78.8%412 of 523run 716a new runrun 718the Hallucination tab's full runs, counted on the kept questions20.8%114 of 547 right20.3%111 of 547 right13.2%10.4%4695< 0.00010.0026 *
Perplexity Decider 27B v1.2 API · with web Open · 27B15.2%10 “I don't know” of 66 not right98.2%537 of 547 answered, 10 “I don't know”10.4%56 of 5375.6%3 “I don't know” of 54 not right99.5%544 of 547 answered, 3 “I don't know”9.4%51 of 544run 644a new runrun 642the Hallucination tab's full runs, counted on the kept questions87.9%481 of 547 right90.1%493 of 547 right4.6%2.2%26210.561
Winnow-12B Open · 12B15.0%15 “I don't know” of 100 not right97.3%532 of 547 answered, 15 “I don't know”16.0%85 of 5328.6%8 “I don't know” of 93 not right98.5%539 of 547 answered, 8 “I don't know”15.8%85 of 539run 748a new runrun 750the Hallucination tab's full runs, counted on the kept questions81.7%447 of 547 right83.0%454 of 547 right3.1%1.8%323211
Mercury Decide Closed · Efficient13.4%11 “I don't know” of 82 not rightrange 11.1% to 13.4%98.0%536 of 547 answered, 11 “I don't know”range 98.0% to 98.4%13.2%71 of 53610.1%8 “I don't know” of 79 not rightrange 10.1% to 10.5%98.5%539 of 547 answered, 8 “I don't know”the same in every run13.2%71 of 539runs 450, 460, 469, 478, 484new runsruns 489, 491, 493, 495, 497new runs on the kept questions85.0%465 of 547 rightrange 85.0% to 85.4%85.6%468 of 547 rightrange 85.6% to 86.1%3.2%range 3.1% to 3.2%1.9%range 1.9% to 2.0%292911
Cygnet Open · 12B11.2%10 “I don't know” of 89 not right98.2%537 of 547 answered, 10 “I don't know”14.7%79 of 5375.8%5 “I don't know” of 86 not right99.1%542 of 547 answered, 5 “I don't know”14.9%81 of 542run 755a new runrun 757the Hallucination tab's full runs, counted on the kept questions83.7%458 of 547 right84.3%461 of 547 right3.5%2.8%28300.91
KnowLine-0.8B-Gen1 Open · 0.8B10.7%26 “I don't know” of 243 not right95.2%521 of 547 answered, 26 “I don't know”41.7%217 of 5214.4%15 “I don't know” of 340 not right97.3%532 of 547 answered, 15 “I don't know”61.1%325 of 532run 1003a new runrun 1007the Hallucination tab's full runs, counted on the kept questions55.6%304 of 547 right37.8%207 of 547 right5.8%6.0%49157< 0.0001< 0.0001 *
GPT-6 Luna Decisions API Closed · Efficient9.4%10 “I don't know” of 106 not right98.2%537 of 547 answered, 10 “I don't know”17.9%96 of 5377.1%6 “I don't know” of 84 not right98.9%541 of 547 answered, 6 “I don't know”14.4%78 of 541run 444a new runrun 453a new run on the kept questions80.6%441 of 547 right84.6%463 of 547 right3.0%2.7%47290.051
Gemini Decision 1 20261111 Closed · Efficient9.1%5 “I don't know” of 55 not right99.1%542 of 547 answered, 5 “I don't know”9.2%50 of 54210.9%5 “I don't know” of 46 not right99.1%542 of 547 answered, 5 “I don't know”7.6%41 of 542run 1019a new runrun 1023the Hallucination tab's full runs, counted on the kept questions89.9%492 of 547 right91.6%501 of 547 right4.6%4.2%20110.151
Perplexity Decider 27B v1.3 Open · 27B8.8%5 “I don't know” of 57 not right99.1%542 of 547 answered, 5 “I don't know”9.6%52 of 5420.0%0 “I don't know” of 52 not right100.0%547 of 547 answered, 0 “I don't know”9.5%52 of 547run 1139a new runrun 1141the Hallucination tab's full runs, counted on the kept questions89.6%490 of 547 right90.5%495 of 547 right3.8%1.7%242411
Turion Jev 31B Open · 31B8.5%4 “I don't know” of 47 not right99.3%543 of 547 answered, 4 “I don't know”7.9%43 of 5435.7%2 “I don't know” of 35 not right99.6%545 of 547 answered, 2 “I don't know”6.1%33 of 545run 1077a new runrun 1285the Hallucination tab's full runs, counted on the kept questions91.4%500 of 547 right93.6%512 of 547 right2.3%1.6%21110.111
d3-ultra Open · 26B8.2%5 “I don't know” of 61 not right99.1%542 of 547 answered, 5 “I don't know”10.3%56 of 5422.1%1 “I don't know” of 47 not right99.8%546 of 547 answered, 1 “I don't know”8.4%46 of 546run 795a new runrun 363the Hallucination tab's full runs, counted on the kept questions88.8%486 of 547 right91.4%500 of 547 right2.8%1.0%33230.231
vera-medium-omni Open · 4.2B7.4%7 “I don't know” of 94 not right98.7%540 of 547 answered, 7 “I don't know”16.1%87 of 5404.2%5 “I don't know” of 118 not right99.1%542 of 547 answered, 5 “I don't know”20.8%113 of 542run 1269a new runrun 1271the Hallucination tab's full runs, counted on the kept questions82.8%453 of 547 right78.4%429 of 547 right5.7%5.3%26520.00430.23
Perplexity Decider Flash v1.3 Open · 4.0B7.2%7 “I don't know” of 97 not right98.7%540 of 547 answered, 7 “I don't know”16.7%90 of 5407.6%9 “I don't know” of 119 not right98.4%538 of 547 answered, 9 “I don't know”20.4%110 of 538run 1128a new runrun 1130the Hallucination tab's full runs, counted on the kept questions82.3%450 of 547 right78.2%428 of 547 right5.1%5.0%37570.0491
REAX 1.3 Open · 57B6.7%3 “I don't know” of 45 not right99.5%544 of 547 answered, 3 “I don't know”7.7%42 of 5442.8%1 “I don't know” of 36 not right99.8%546 of 547 answered, 1 “I don't know”6.4%35 of 546run 831a new runrun 833the Hallucination tab's full runs, counted on the kept questions91.8%502 of 547 right93.4%511 of 547 right1.9%0.7%20130.31
Clef-Flash Open · 9.0B5.3%5 “I don't know” of 95 not right99.1%542 of 547 answered, 5 “I don't know”16.6%90 of 5429.4%12 “I don't know” of 127 not right97.8%535 of 547 answered, 12 “I don't know”21.5%115 of 535run 660a new runrun 664the Hallucination tab's full runs, counted on the kept questions82.6%452 of 547 right76.8%420 of 547 right3.1%4.8%27520.00660.33
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B5.2%12 “I don't know” of 229 not right97.8%535 of 547 answered, 12 “I don't know”40.6%217 of 5356.4%22 “I don't know” of 342 not right96.0%525 of 547 answered, 22 “I don't know”61.0%320 of 525run 1013a new runrun 1017the Hallucination tab's full runs, counted on the kept questions58.1%318 of 547 right37.5%205 of 547 right6.6%7.3%48151< 0.0001< 0.0001 *
Perplexity Decider Flash v1.2 Open · 4.0B4.8%5 “I don't know” of 105 not right99.1%542 of 547 answered, 5 “I don't know”18.5%100 of 5424.3%6 “I don't know” of 139 not right98.9%541 of 547 answered, 6 “I don't know”24.6%133 of 541run 535a new runrun 537a new run on the kept questions80.8%442 of 547 right74.6%408 of 547 right5.5%6.3%36690.00170.091
Perplexity Decider 27B v1.2 API Open · 27B4.5%3 “I don't know” of 66 not right99.5%544 of 547 answered, 3 “I don't know”11.6%63 of 5440.0%0 “I don't know” of 51 not right100.0%547 of 547 answered, 0 “I don't know”9.3%51 of 547run 643a new runrun 641the Hallucination tab's full runs, counted on the kept questions87.9%481 of 547 right90.7%496 of 547 right3.1%1.1%27150.0881
Logits Sigma-1 Duo Open · 57B4.3%2 “I don't know” of 46 not right99.6%545 of 547 answered, 2 “I don't know”8.1%44 of 5450.0%0 “I don't know” of 36 not right100.0%547 of 547 answered, 0 “I don't know”6.6%36 of 547run 1231a new runrun 1233the Hallucination tab's full runs, counted on the kept questions91.6%501 of 547 right93.4%511 of 547 right1.8%0.8%25170.281
Jev 1.14 Closed · Efficient4.0%3 “I don't know” of 75 not rightrange 4.0% to 5.2%99.5%544 of 547 answered, 3 “I don't know”range 99.3% to 99.5%13.2%72 of 5444.2%3 “I don't know” of 72 not rightrange 4.0% to 4.5%99.5%544 of 547 answered, 3 “I don't know”the same in every run12.7%69 of 544runs 446, 454, 457, 461, 464new runsruns 470, 473, 476, 480, 482new runs on the kept questions86.3%472 of 547 rightrange 85.9% to 86.7%86.8%475 of 547 rightrange 86.3% to 87.8%1.6%range 1.6% to 1.6%1.7%range 1.6% to 1.7%20170.741
OneJev-27B Open · 27B4.0%3 “I don't know” of 75 not right99.5%544 of 547 answered, 3 “I don't know”13.2%72 of 5441.6%1 “I don't know” of 64 not right99.8%546 of 547 answered, 1 “I don't know”11.5%63 of 546run 780a new runrun 782the Hallucination tab's full runs, counted on the kept questions86.3%472 of 547 right88.3%483 of 547 right2.3%1.3%34250.31
Microsoft-Decision-1 Closed · Efficient3.8%3 “I don't know” of 80 not rightrange 3.8% to 4.0%99.5%544 of 547 answered, 3 “I don't know”the same in every run14.2%77 of 5442.2%2 “I don't know” of 90 not rightrange 2.2% to 4.3%99.6%545 of 547 answered, 2 “I don't know”range 99.3% to 99.6%16.1%88 of 545runs 451, 459, 467, 477, 483new runsruns 487, 490, 492, 494, 496new runs on the kept questions85.4%467 of 547 rightrange 85.4% to 86.3%83.5%457 of 547 rightrange 82.8% to 83.5%3.6%range 3.6% to 3.6%2.3%range 2.3% to 2.3%24350.191
Chinese Decision Pointer 4B Open · 4.0B3.7%4 “I don't know” of 108 not right99.3%543 of 547 answered, 4 “I don't know”19.2%104 of 5433.0%4 “I don't know” of 133 not right99.3%543 of 547 answered, 4 “I don't know”23.8%129 of 543run 1211a new runrun 1213the Hallucination tab's full runs, counted on the kept questions80.3%439 of 547 right75.7%414 of 547 right2.1%1.9%41660.020.9
Drex Max Open · 27B3.2%2 “I don't know” of 62 not right99.6%545 of 547 answered, 2 “I don't know”11.0%60 of 5453.6%2 “I don't know” of 55 not right99.6%545 of 547 answered, 2 “I don't know”9.7%53 of 545run 798a new runrun 367the Hallucination tab's full runs, counted on the kept questions88.7%485 of 547 right89.9%492 of 547 right1.7%0.6%28210.391
Perplexity Decider 27B v1.2 DGX Spark Open · 27B3.0%2 “I don't know” of 66 not right99.6%545 of 547 answered, 2 “I don't know”11.7%64 of 5450.0%0 “I don't know” of 50 not right100.0%547 of 547 answered, 0 “I don't know”9.1%50 of 547run 529a new runrun 372the Hallucination tab's full runs, counted on the kept questions87.9%481 of 547 right90.9%497 of 547 right2.8%0.9%29150.0491
d1 Closed · Efficient2.9%2 “I don't know” of 69 not rightrange 2.6% to 2.9%99.6%545 of 547 answered, 2 “I don't know”the same in every run12.3%67 of 5451.1%1 “I don't know” of 90 not rightrange 0.0% to 1.1%99.8%546 of 547 answered, 1 “I don't know”range 99.8% to 100.0%16.3%89 of 546runs 445, 455, 458, 463, 465new runsruns 472, 475, 479, 481, 485new runs on the kept questions87.4%478 of 547 rightrange 85.9% to 87.4%83.5%457 of 547 rightrange 83.0% to 85.0%0.9%range 0.9% to 1.0%1.3%range 1.1% to 1.3%21430.00810.4
StartLux-Decision-v1.1-9B-Instant Open · 9.0B2.6%2 “I don't know” of 76 not right99.6%545 of 547 answered, 2 “I don't know”13.6%74 of 5451.0%1 “I don't know” of 96 not right99.8%546 of 547 answered, 1 “I don't know”17.4%95 of 546run 1295a new runrun 1297the Hallucination tab's full runs, counted on the kept questions86.1%471 of 547 right82.4%451 of 547 right1.8%1.4%26470.0190.88
Cerno 9B v1.2 Open · 9.0B2.5%2 “I don't know” of 79 not right99.6%545 of 547 answered, 2 “I don't know”14.1%77 of 5450.0%0 “I don't know” of 91 not right100.0%547 of 547 answered, 0 “I don't know”16.6%91 of 547run 1087a new runrun 1204the Hallucination tab's full runs, counted on the kept questions85.6%468 of 547 right83.4%456 of 547 right2.2%1.6%34480.151
Sieve 31B v3 Open · 31B2.2%1 “I don't know” of 46 not right99.8%546 of 547 answered, 1 “I don't know”8.2%45 of 5460.0%0 “I don't know” of 33 not right100.0%547 of 547 answered, 0 “I don't know”6.0%33 of 547run 1066a new runrun 1253the Hallucination tab's full runs, counted on the kept questions91.6%501 of 547 right94.0%514 of 547 right0.8%0.3%23110.0581
Verdin-4B-v1.1 Open · 4.0B2.1%2 “I don't know” of 96 not right99.6%545 of 547 answered, 2 “I don't know”17.2%94 of 5450.8%1 “I don't know” of 128 not right99.8%546 of 547 answered, 1 “I don't know”23.3%127 of 546run 1249a new runrun 1251the Hallucination tab's full runs, counted on the kept questions82.4%451 of 547 right76.6%419 of 547 right3.5%2.2%37700.00180.099
d3 Open · 26B1.7%1 “I don't know” of 59 not right99.8%546 of 547 answered, 1 “I don't know”10.6%58 of 5460.0%0 “I don't know” of 53 not right100.0%547 of 547 answered, 0 “I don't know”9.7%53 of 547run 793a new runrun 358the Hallucination tab's full runs, counted on the kept questions89.2%488 of 547 right90.3%494 of 547 right2.9%1.1%24190.541
Metask-Jev-4B Open · 4.5B1.6%2 “I don't know” of 125 not right99.6%545 of 547 answered, 2 “I don't know”22.6%123 of 5451.1%2 “I don't know” of 189 not right99.6%545 of 547 answered, 2 “I don't know”34.3%187 of 545run 741a new runrun 743the Hallucination tab's full runs, counted on the kept questions77.1%422 of 547 right65.4%358 of 547 right3.8%4.1%3498< 0.0001< 0.0001 *
Clef Open · 27B1.3%1 “I don't know” of 76 not right99.8%546 of 547 answered, 1 “I don't know”13.7%75 of 5461.5%1 “I don't know” of 68 not right99.8%546 of 547 answered, 1 “I don't know”12.3%67 of 546run 785a new runrun 788the Hallucination tab's full runs, counted on the kept questions86.1%471 of 547 right87.6%479 of 547 right2.4%3.6%26180.291
KnowLine-12B-Gen1 Open · 12B1.3%1 “I don't know” of 77 not right99.8%546 of 547 answered, 1 “I don't know”13.9%76 of 5460.0%0 “I don't know” of 67 not right100.0%547 of 547 answered, 0 “I don't know”12.2%67 of 547run 1316a new runrun 1318the Hallucination tab's full runs, counted on the kept questions85.9%470 of 547 right87.8%480 of 547 right1.3%0.6%26170.221
KnowLine-9B-Gen1 Open · 9.0B1.3%1 “I don't know” of 77 not right99.8%546 of 547 answered, 1 “I don't know”13.9%76 of 5460.0%0 “I don't know” of 98 not right100.0%547 of 547 answered, 0 “I don't know”17.9%98 of 547run 1179a new runrun 1181the Hallucination tab's full runs, counted on the kept questions85.9%470 of 547 right82.1%449 of 547 right1.6%0.7%18400.00540.28
KnowLine-9B-Gen2 Open · 9.0B1.3%1 “I don't know” of 79 not right99.8%546 of 547 answered, 1 “I don't know”14.3%78 of 5460.0%0 “I don't know” of 95 not right100.0%547 of 547 answered, 0 “I don't know”17.4%95 of 547run 1323a new runrun 1325the Hallucination tab's full runs, counted on the kept questions85.6%468 of 547 right82.6%452 of 547 right3.8%3.5%23400.0431
Valen-Decision-1.5-9B Open · 9.0B1.2%1 “I don't know” of 85 not right99.8%546 of 547 answered, 1 “I don't know”15.4%84 of 5460.0%0 “I don't know” of 110 not right100.0%547 of 547 answered, 0 “I don't know”20.1%110 of 547run 1280a new runrun 1282the Hallucination tab's full runs, counted on the kept questions84.5%462 of 547 right79.9%437 of 547 right3.0%2.7%32580.0080.4
Dexy-Decision-9B-V2 Open · 9.0B1.1%1 “I don't know” of 91 not right99.8%546 of 547 answered, 1 “I don't know”16.5%90 of 5461.1%1 “I don't know” of 91 not right99.8%546 of 547 answered, 1 “I don't know”16.5%90 of 546run 1260a new runrun 1262the Hallucination tab's full runs, counted on the kept questions83.4%456 of 547 right83.4%456 of 547 right0.7%0.6%383811
KnowLine 4B Gen5.5 Open · 4.0B0.9%1 “I don't know” of 106 not right99.8%546 of 547 answered, 1 “I don't know”19.2%105 of 5462.5%3 “I don't know” of 121 not right99.5%544 of 547 answered, 3 “I don't know”21.7%118 of 544run 963a new runrun 967the Hallucination tab's full runs, counted on the kept questions80.6%441 of 547 right77.9%426 of 547 right1.3%1.0%40530.211
Strands Decider 2B Open · 2.0B0.7%2 “I don't know” of 303 not right99.6%545 of 547 answered, 2 “I don't know”55.2%301 of 5451.5%5 “I don't know” of 332 not right99.1%542 of 547 answered, 5 “I don't know”60.3%327 of 542run 704a new runrun 706the Hallucination tab's full runs, counted on the kept questions44.6%244 of 547 right39.3%215 of 547 right3.8%4.9%49750.0241
Adjudex 9B v5 Open · 9.0B0.0%0 “I don't know” of 77 not right100.0%547 of 547 answered, 0 “I don't know”14.1%77 of 5470.0%0 “I don't know” of 89 not right100.0%547 of 547 answered, 0 “I don't know”16.3%89 of 547run 1188a new runrun 1190the Hallucination tab's full runs, counted on the kept questions85.9%470 of 547 right83.7%458 of 547 right0.6%0.3%32440.211
Copernicus-31B Open · 31B0.0%0 “I don't know” of 39 not right100.0%547 of 547 answered, 0 “I don't know”7.1%39 of 5472.9%1 “I don't know” of 34 not right99.8%546 of 547 answered, 1 “I don't know”6.0%33 of 546run 1170a new runrun 1172the Hallucination tab's full runs, counted on the kept questions92.9%508 of 547 right93.8%513 of 547 right1.0%0.7%19130.381
Jeffan-9B-1211 Open · 9.0B0.0%0 “I don't know” of 71 not right100.0%547 of 547 answered, 0 “I don't know”13.0%71 of 5474.1%3 “I don't know” of 74 not right99.5%544 of 547 answered, 3 “I don't know”13.1%71 of 544run 1197a new runrun 1199the Hallucination tab's full runs, counted on the kept questions87.0%476 of 547 right86.5%473 of 547 right3.7%3.7%313111
Karpo Decision 31B Open · 31B0.0%0 “I don't know” of 42 not right100.0%547 of 547 answered, 0 “I don't know”7.7%42 of 5473.2%1 “I don't know” of 31 not right99.8%546 of 547 answered, 1 “I don't know”5.5%30 of 546run 1161a new runrun 1163the Hallucination tab's full runs, counted on the kept questions92.3%505 of 547 right94.3%516 of 547 right0.6%0.3%25130.0731
KnowLine-31B-Gen1.5 Open · 31B0.0%0 “I don't know” of 37 not right100.0%547 of 547 answered, 0 “I don't know”6.8%37 of 5470.0%0 “I don't know” of 34 not right100.0%547 of 547 answered, 0 “I don't know”6.2%34 of 547run 955a new runrun 959the Hallucination tab's full runs, counted on the kept questions93.2%510 of 547 right93.8%513 of 547 right0.2%0.2%16130.711
KnowLine-31B-Gen2 Open · 31B0.0%0 “I don't know” of 40 not right100.0%547 of 547 answered, 0 “I don't know”7.3%40 of 5470.0%0 “I don't know” of 29 not right100.0%547 of 547 answered, 0 “I don't know”5.3%29 of 547run 1329a new runrun 1331the Hallucination tab's full runs, counted on the kept questions92.7%507 of 547 right94.7%518 of 547 right0.2%0.1%2090.0611
StartLux-Decision-v1.1-4B-Instant Open · 4.0B0.0%0 “I don't know” of 94 not right100.0%547 of 547 answered, 0 “I don't know”17.2%94 of 5471.7%2 “I don't know” of 120 not right99.6%545 of 547 answered, 2 “I don't know”21.7%118 of 545run 1083a new runrun 1239the Hallucination tab's full runs, counted on the kept questions82.8%453 of 547 right78.1%427 of 547 right3.2%2.9%33570.0150.71

Results with “I don't know” instead of “Bilmiyorum”

In the tab's order. Each row shows the model's first run with each word on the same 2,198 questions; the “Bilmiyorum” run is the one in the table above. With “I don't know”, then with “Bilmiyorum”: the non-hallucination rate (the option / (the option + wrong)), Answered = (right + wrong) / 2,198 and the error rate = wrong / (right + wrong); then accuracy with each. Wrong only with one word: the questions only one of the two runs got wrong, by the exact McNemar test, with p adjusted by Holm's method over the models compared; * marks a Holm-adjusted p below 0.05.

ModelNon-hallucination with “I don't know”of answers not right, % “I don't know” · higher is betterAnswered with “I don't know”not “I don't know”: (right + wrong) / 2,198Error rate with “I don't know”of its answers: wrong / (right + wrong) · lower is betterNon-hallucination with “Bilmiyorum”of answers not right, % “Bilmiyorum” · higher is betterAnswered with “Bilmiyorum”not “Bilmiyorum”: (right + wrong) / 2,198Error rate with “Bilmiyorum”of its answers: wrong / (right + wrong) · lower is betterAccuracy with “I don't know”higher is betterAccuracy with “Bilmiyorum”higher is betterWrong only with one wordexact McNemar, Holm-adjustedRunsrun, start date and prompt version
d1.2-ultra Closed · Efficient88.1%1,018 “I don't know” of 1,155 not right53.7%1,180 of 2,198 answered, 1,018 “I don't know”11.6%137 of 1,18087.3%996 “Bilmiyorum” of 1,141 not right54.7%1,202 of 2,198 answered, 996 “Bilmiyorum”12.1%145 of 1,20247.5%1,043 of 2,198 right48.1%1,057 of 2,198 right33 · 4133 wrong only with “I don't know”, 41 only with “Bilmiyorum”p = 0.42, Holm-adjusted p = 11378 · 1367“I don't know”: run 1378, 2026-11-16, prompt version fb17d583a3af“Bilmiyorum”: run 1367, 2026-11-16, prompt version 1cdd542b597f
d1.2 Closed · Efficient76.4%901 “I don't know” of 1,180 not right59.0%1,297 of 2,198 answered, 901 “I don't know”21.5%279 of 1,29779.4%973 “Bilmiyorum” of 1,226 not right55.7%1,225 of 2,198 answered, 973 “Bilmiyorum”20.7%253 of 1,22546.3%1,018 of 2,198 right44.2%972 of 2,198 right95 · 6995 wrong only with “I don't know”, 69 only with “Bilmiyorum”p = 0.051, Holm-adjusted p = 0.41377 · 1365“I don't know”: run 1377, 2026-11-16, prompt version fb17d583a3af“Bilmiyorum”: run 1365, 2026-11-16, prompt version 1cdd542b597f
d1-ultra Closed · Efficient74.0%616 “I don't know” of 832 not right72.0%1,582 of 2,198 answered, 616 “I don't know”13.7%216 of 1,58275.1%617 “Bilmiyorum” of 822 not right71.9%1,581 of 2,198 answered, 617 “Bilmiyorum”13.0%205 of 1,58162.1%1,366 of 2,198 right62.6%1,376 of 2,198 right66 · 5566 wrong only with “I don't know”, 55 only with “Bilmiyorum”p = 0.36, Holm-adjusted p = 1511 · 350“I don't know”: run 511, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 350, 2026-11-01, prompt version 1cdd542b597f
Mammoth-1 Open · 27B73.2%753 “I don't know” of 1,029 not right65.7%1,445 of 2,198 answered, 753 “I don't know”19.1%276 of 1,44568.7%690 “Bilmiyorum” of 1,004 not right68.6%1,508 of 2,198 answered, 690 “Bilmiyorum”20.8%314 of 1,50853.2%1,169 of 2,198 right54.3%1,194 of 2,198 right9 · 479 wrong only with “I don't know”, 47 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 *519 · 386“I don't know”: run 519, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 386, 2026-11-01, prompt version 1cdd542b597f
GLiDE API Closed · Efficient63.4%581 “I don't know” of 916 not right73.6%1,617 of 2,198 answered, 581 “I don't know”20.7%335 of 1,61759.4%521 “Bilmiyorum” of 877 not right76.3%1,677 of 2,198 answered, 521 “Bilmiyorum”21.2%356 of 1,67758.3%1,282 of 2,198 right60.1%1,321 of 2,198 right52 · 7352 wrong only with “I don't know”, 73 only with “Bilmiyorum”p = 0.073, Holm-adjusted p = 0.51510 · 351“I don't know”: run 510, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 351, 2026-11-01, prompt version 1cdd542b597f
SubQ Snap 1.1 Closed · Efficient52.3%498 “I don't know” of 952 not right77.3%1,700 of 2,198 answered, 498 “I don't know”26.7%454 of 1,70047.1%437 “Bilmiyorum” of 927 not right80.1%1,761 of 2,198 answered, 437 “Bilmiyorum”27.8%490 of 1,76156.7%1,246 of 2,198 right57.8%1,271 of 2,198 right108 · 144108 wrong only with “I don't know”, 144 only with “Bilmiyorum”p = 0.027, Holm-adjusted p = 0.25512 · 392“I don't know”: run 512, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 392, 2026-11-01, prompt version 1cdd542b597f
GLiDE v1.2 Open · 27B49.9%426 “I don't know” of 853 not right80.6%1,772 of 2,198 answered, 426 “I don't know”24.1%427 of 1,77239.5%321 “Bilmiyorum” of 812 not right85.4%1,877 of 2,198 answered, 321 “Bilmiyorum”26.2%491 of 1,87761.2%1,345 of 2,198 right63.1%1,386 of 2,198 right19 · 8319 wrong only with “I don't know”, 83 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 *525 · 369“I don't know”: run 525, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 369, 2026-11-01, prompt version 1cdd542b597f
Drex Max v1.1 Closed · Efficient (provisional)54.1%486 “I don't know” of 899 not right77.9%1,712 of 2,198 answered, 486 “I don't know”24.1%413 of 1,71238.2%320 “Bilmiyorum” of 837 not right85.4%1,878 of 2,198 answered, 320 “Bilmiyorum”27.5%517 of 1,87859.1%1,299 of 2,198 right61.9%1,361 of 2,198 right40 · 14440 wrong only with “I don't know”, 144 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 *513 · 352“I don't know”: run 513, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 352, 2026-11-01, prompt version 1cdd542b597f
Elephant-1 Open · 9.0B49.3%604 “I don't know” of 1,226 not right72.5%1,594 of 2,198 answered, 604 “I don't know”39.0%622 of 1,59434.8%405 “Bilmiyorum” of 1,163 not right81.6%1,793 of 2,198 answered, 405 “Bilmiyorum”42.3%758 of 1,79344.2%972 of 2,198 right47.1%1,035 of 2,198 right18 · 15418 wrong only with “I don't know”, 154 only with “Bilmiyorum”p < 0.0001, Holm-adjusted p < 0.0001 *515 · 382“I don't know”: run 515, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 382, 2026-11-01, prompt version 1cdd542b597f
d3-ultra Open · 26B1.3%9 “I don't know” of 713 not right99.6%2,189 of 2,198 answered, 9 “I don't know”32.2%704 of 2,1891.4%10 “Bilmiyorum” of 714 not right99.5%2,188 of 2,198 answered, 10 “Bilmiyorum”32.2%704 of 2,18867.6%1,485 of 2,198 right67.5%1,484 of 2,198 right39 · 3939 wrong only with “I don't know”, 39 only with “Bilmiyorum”p = 1, Holm-adjusted p = 1540 · 363“I don't know”: run 540, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 363, 2026-11-01, prompt version 1cdd542b597f
Drex Max Open · 27B0.3%2 “I don't know” of 714 not right99.9%2,196 of 2,198 answered, 2 “I don't know”32.4%712 of 2,1960.3%2 “Bilmiyorum” of 713 not right99.9%2,196 of 2,198 answered, 2 “Bilmiyorum”32.4%711 of 2,19667.5%1,484 of 2,198 right67.6%1,485 of 2,198 right29 · 2829 wrong only with “I don't know”, 28 only with “Bilmiyorum”p = 1, Holm-adjusted p = 1542 · 367“I don't know”: run 542, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 367, 2026-11-01, prompt version 1cdd542b597f
Perplexity Decider 27B v1.2 DGX Spark Open · 27B0.5%4 “I don't know” of 732 not right99.8%2,194 of 2,198 answered, 4 “I don't know”33.2%728 of 2,1940.3%2 “Bilmiyorum” of 719 not right99.9%2,196 of 2,198 answered, 2 “Bilmiyorum”32.7%717 of 2,19666.7%1,466 of 2,198 right67.3%1,479 of 2,198 right40 · 2940 wrong only with “I don't know”, 29 only with “Bilmiyorum”p = 0.23, Holm-adjusted p = 1531 · 372“I don't know”: run 531, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 372, 2026-11-01, prompt version 1cdd542b597f
d3 Open · 26B0.3%2 “I don't know” of 716 not right99.9%2,196 of 2,198 answered, 2 “I don't know”32.5%714 of 2,1960.0%0 “Bilmiyorum” of 715 not right100.0%2,198 of 2,198 answered, 0 “Bilmiyorum”32.5%715 of 2,19867.4%1,482 of 2,198 right67.5%1,483 of 2,198 right21 · 2221 wrong only with “I don't know”, 22 only with “Bilmiyorum”p = 1, Holm-adjusted p = 1544 · 358“I don't know”: run 544, 2026-11-02, prompt version fb17d583a3af“Bilmiyorum”: run 358, 2026-11-01, prompt version 1cdd542b597f

Not run

  • Copernicus-31B: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • KnowLine-31B-Gen2: added to the Hallucination tab on 16 November 2026 (run on our GPU; Ümit #9814), after the option-word check; not asked with “I don't know” yet.
  • KnowLine-31B-Gen1.5: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • Turion Jev 31B: added to the Hallucination tab on 15 November 2026 (run on our DGX Spark), after the option-word check; not asked with “I don't know” yet.
  • REAX 1.3: added to the Hallucination tab on 6 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • Karpo Decision 31B: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616), after the option-word check; not asked with “I don't know” yet.
  • Gemini Decision 1: added to the Hallucination tab on 11 November 2026 (Google's Decisions API through OpenRouter), after the option-word check; not asked with “I don't know” yet.
  • Logits Sigma-1 Duo: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • Sieve 31B v3: added to the Hallucination tab on 15 November 2026 (run on our DGX Spark), after the option-word check; not asked with “I don't know” yet.
  • Perplexity Decider 27B v1.3: added to the Hallucination tab on 14 November 2026 (run on our GPU; Ümit #7470), after the option-word check; not asked with “I don't know” yet.
  • Perplexity Decider 27B v1.2: added to the Hallucination tab on 5 November 2026 (Ümit #1229), after the option-word check; not asked with “I don't know” yet.
  • Perplexity Decider 27B v1.2: added to the Hallucination tab on 5 November 2026 (Ümit #1229), after the option-word check; not asked with “I don't know” yet.
  • GLiDE 27B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • KnowLine-12B-Gen1: added to the Hallucination tab on 16 November 2026 (run on our GPU; Ümit #9814), after the option-word check; not asked with “I don't know” yet.
  • OneJev-27B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • Jeffan-9B-1211: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • Clef: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • Open-Jev-27B-v1.1: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • Cygnet: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • Cerno 9B v1.2: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • KnowLine-9B-Gen1: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • Winnow-12B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • Adjudex 9B v5: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • KnowLine-9B-Gen2: added to the Hallucination tab on 16 November 2026 (run on our GPU; Ümit #9814), after the option-word check; not asked with “I don't know” yet.
  • Dexy-Decision-9B-V2: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • StartLux-Decision-v1.1-9B-Instant: added to the Hallucination tab on 15 November 2026 (run on our DGX Spark), after the option-word check; not asked with “I don't know” yet.
  • Perplexity Decider Flash v1.3: added to the Hallucination tab on 14 November 2026 (run on our GPU; Ümit #7470), after the option-word check; not asked with “I don't know” yet.
  • GLiDE 9B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • Valen-Decision-1.5-9B: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • StartLux-Decision-v1.1-4B-Instant: added to the Hallucination tab on 15 November 2026 (run on our DGX Spark), after the option-word check; not asked with “I don't know” yet.
  • Verdin-4B-v1.1: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • KnowLine 4B Gen5.5: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • vera-medium-omni: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • Chinese Decision Pointer 4B: added to the Hallucination tab on 15 November 2026 (run on our GPU; Ümit #7616, #8157), after the option-word check; not asked with “I don't know” yet.
  • Clef-Flash: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • Perplexity Decider Flash v1.2: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • Metask-Jev-4B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • StartLux-Decision-v1.1-2B-Instant: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • GLiDE 2B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • StartLux-Decision-v1.1-0.8B-Instant: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • KnowLine-0.8B-Gen1: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • Strands Decider 2B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • GLiDE 0.8B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • OneJev1.6-0.8B: added to the Hallucination tab on 11 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • GLiNER2.5-multi-Decide: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • GLiClass v4 Edge: added to the Hallucination tab on 12 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.
  • d1-2B: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • Laya Multilingual: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • Laya: added to the Hallucination tab on 5 November 2026 with the runs on our GPUs, after the option-word check; not asked with “I don't know” yet.
  • GLiDE 0.4B: added to the Hallucination tab on 10 November 2026 (run on our GPU), after the option-word check; not asked with “I don't know” yet.

Our method with “I don't know”

Only two things changed from the runs above: the sixth option's word and the sentence of the System One instructions that names it. The option “Bilmiyorum” became “I don't know”, and the sentence “If you do not know the answer, choose Bilmiyorum.” became “If you do not know the answer, choose I don't know.” Everything else is as above: the same 2,198 Turkish questions and answer key, the option always last and never correct, one question per request and each model's own settings. The full instructions:

Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose I don't know.

Prompt version fb17d583a3af with “I don't know”, against 1cdd542b597f with “Bilmiyorum”. One full run per model with “I don't know”; the table gives each run's ID and start date (UTC).

How to read the comparison

A model that did not recognise the Turkish word and so ignored the option should choose the English one more often, and its non-hallucination rate should rise. A model that chooses both about as often did not skip “Bilmiyorum” for its word. The rates are those above, and the same limits hold: the non-hallucination rate says how a model is wrong, not how often it is right, and it is not a direct measure of what the model does not know; read it with accuracy and the attempt rate, as a model that always chose the option would reach 100% with no right answers. With neither a wrong answer nor the option, the rate is undefined.

Results: “I don't know” against confidence

In the tab's order. Each row: the model's first run with “I don't know” and its five-option run held back by its own confidence on the same 2,198 questions. Answered = right + wrong; coverage = answered / 2,198. Wrong only on one side: the questions only one of the two got wrong, by the exact McNemar test, with p adjusted by Holm's method over the 9 models that chose “I don't know” on at least 1% of the questions; * marks a Holm-adjusted p below 0.05. Not in the family: d3-ultra, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark and d3.

ModelRunswith “I don't know” · five optionsCut-offand its neighbouring candidatesAnsweredtarget with the option · achieved by confidence (coverage)Wrongwith “I don't know” · by confidence; lower is betterWrong only on one sidewith “I don't know” · by confidencepexact McNemarHolm padjusted over the family
d1.2-ultra Closed · Efficient1378 · 13470.6035next lower 0.6034: 1,181 answerednext higher 0.6038: 1,179 answered1,180 · 1,180target 1,180 (53.7%), 1,018 “I don't know”achieved 1,180 (53.7%), 1,018 held back137 · 12852 · 430.411
d1.2 Closed · Efficient1377 · 13460.4689next lower 0.4687: 1,298 answerednext higher 0.4690: 1,296 answered1,297 · 1,297target 1,297 (59.0%), 901 “I don't know”achieved 1,297 (59.0%), 901 held back279 · 262107 · 900.251
d1-ultra Closed · Efficient511 · 1990.5630next lower 0.5622: 1,583 answerednext higher 0.5632: 1,581 answered1,582 · 1,582target 1,582 (72.0%), 616 “I don't know”achieved 1,582 (72.0%), 616 held back216 · 16798 · 49< 0.00010.00059 *
Mammoth-1 Open · 27B519 · 3740.5951next lower 0.5948: 1,446 answerednext higher 0.5956: 1,444 answered1,445 · 1,445target 1,445 (65.7%), 753 “I don't know”achieved 1,445 (65.7%), 753 held back276 · 25064 · 380.0130.1
GLiDE API Closed · Efficient510 · 517run 517 in place of the Text tab's run 1290.5222next lower 0.5220: 1,618 answerednext higher 0.5231: 1,616 answered1,617 · 1,617target 1,617 (73.6%), 581 “I don't know”achieved 1,617 (73.6%), 581 held back335 · 31979 · 630.211
SubQ Snap 1.1 Closed · Efficient512 · 3460.5372next lower 0.5371: 1,701 answerednext higher 0.5380: 1,699 answered1,700 · 1,700target 1,700 (77.3%), 498 “I don't know”achieved 1,700 (77.3%), 498 held back454 · 445141 · 1320.631
GLiDE v1.2 Open · 27B525 · 2480.4535next lower 0.4534: 1,773 answerednext higher 0.4536: 1,771 answered1,772 · 1,772target 1,772 (80.6%), 426 “I don't know”achieved 1,772 (80.6%), 426 held back427 · 40377 · 530.0430.3
Drex Max v1.1 Closed · Efficient (provisional)513 · 3260.4478next lower 0.4470: 1,713 answerednext higher 0.4479: 1,711 answered1,712 · 1,712target 1,712 (77.9%), 486 “I don't know”achieved 1,712 (77.9%), 486 held back413 · 396106 · 890.251
Elephant-1 Open · 9.0B515 · 3730.4376next lower 0.4374: 1,595 answerednext higher 0.4378: 1,593 answered1,594 · 1,594target 1,594 (72.5%), 604 “I don't know”achieved 1,594 (72.5%), 604 held back622 · 594120 · 920.0630.38
d3-ultra Open · 26B540 · 2690.2839next lower 0.2830: 2,190 answerednext higher 0.2876: 2,188 answered2,189 · 2,189target 2,189 (99.6%), 9 “I don't know”achieved 2,189 (99.6%), 9 held back704 · 70354 · 531not in the family
Drex Max Open · 27B542 · 3220.2357next lower 0.2348: 2,197 answerednext higher 0.2375: 2,195 answered2,196 · 2,196target 2,196 (99.9%), 2 “I don't know”achieved 2,196 (99.9%), 2 held back712 · 70749 · 440.68not in the family
Perplexity Decider 27B v1.2 DGX Spark Open · 27B531 · 2170.2217next lower 0.2205: 2,195 answerednext higher 0.2223: 2,193 answered2,194 · 2,194target 2,194 (99.8%), 4 “I don't know”achieved 2,194 (99.8%), 4 held back728 · 72052 · 440.48not in the family
d3 Open · 26B544 · 2590.2331next lower 0.2294: 2,197 answerednext higher 0.2352: 2,195 answered2,196 · 2,196target 2,196 (99.9%), 2 “I don't know”achieved 2,196 (99.9%), 2 held back714 · 70639 · 310.4not in the family

Our method for “I don't know” against confidence

The derived method above, applied to the runs with “I don't know”: each model's five-option run answers only where its score reaches a cut-off and abstains on the rest. The candidate cut-offs are the run's distinct scores; the one chosen is the one whose attempt rate comes nearest the attempt rate of the run with “I don't know”, and on a tie the higher cut-off. An answer without a score never counts as an attempt. Answers with equal scores enter together, so the achieved answered count can differ from the target; the table shows the chosen cut-off with the next lower and next higher candidates and the answers each would give.

The five-option run is each model's run on the Text tab, as in the table above. For GLiDE · API, the five-option run is 517, not the Text tab's run 129: Fastino changed its hosted model after 6 October, so run 129 is a different model from the one behind run 510, and run 517 is a new five-option run of the same hosted model (Our method for the hosted GLiDE rerun).

The paired test is the exact McNemar test of wrong answers, the run with “I don't know” first and confidence second, as for “Bilmiyorum”. Holm's correction runs over the models that chose “I don't know” on at least 1% of the scored questions: GLiDE · v1.2, d1-ultra, Drex Max v1.1, GLiDE · API, SubQ Snap 1.1, Mammoth-1, d1.2-ultra, Elephant-1 and Liquid d1.2. The others, d3-ultra, Drex Max, Perplexity Decider 27B v1.2 · DGX Spark and d3, chose it too rarely for the two sides to differ by more than a few answers; their p is shown and not adjusted.

This is a family of its own, from runs already published: it changes none of the numbers above, and it is separate from the comparisons with Holm over 4, 13 and 11 models (the four models of the request, the English questions and the option's word).

Hosted GLiDE: the comparison redone with run 517

Both against GLiDE · API's first run with “I don't know”, run 351. Run 129 is the Text tab's run and the row in the table above; run 517 is new and replaces it here only.

Five-option runrun, date and prompt versionText accuracy with five optionshigher is betterDerived non-hallucination ratehigher is betterWrong only on one sideexact McNemar
run 1292026-10-06, as published67.9%1,492 of 2,198 right58.0%cut-off 0.5206: 1,300 right + 377 wrong answered, 521 held back, attempt rate 76.3%192 with “I don't know”, 213 by confidencep = 0.32, Holm-adjusted p = 1
run 5172026-11-02, prompt version 254597b8be1668.1%1,497 of 2,198 right59.7%cut-off 0.4956: 1,326 right + 351 wrong answered, 521 held back, attempt rate 76.3%90 with “I don't know”, 85 by confidencep = 0.76, not adjusted

Our method for the hosted GLiDE rerun

Fastino changed the hosted model behind fastino/GLiDE after 6 October. Run 129 was made on 6 October 2026, so it is a different model from the one that answered with “I don't know” (run 351), and the derived rate in the table above compares two models. Run 517 is a new five-option run of the same hosted model, started 2 November 2026, made as on the Text tab: the same 2,198 questions, one question per request, the main prompt version 254597b8be16. Its derived non-hallucination rate and paired test are computed as above, from run 517 instead of run 129; the p value is not adjusted, as it is one test.

The evidence for “a different model” and “the same model” is the engine each response names: every answer of run 129 names Fastino's engine glide; every answer from 10 October on names glide_v2, including all of run 351 and its repeats and all of run 517. Fastino publishes no version to pin, so that name is the only identity we can check: it cannot show that the weights behind glide_v2 stayed the same between 1 and 2 November.

The row in the table above is left as published, against run 129, and run 517 is kept out of the Text tab: its leaderboard, repeats and every other list.

Results: the same number of answers

One table per language, ordered by wrong of the k answers, fewest first; the Turkish one has every main run of the Text tab (137 setups), grouped as its leaderboard groups them and labelled as it labels them. Each row: the model's five-option run held to its k most confident answers, its wrong answers among them, the difference from d1-ultra in points with its 95% bootstrap interval, p and Holm p (* marks a Holm-adjusted p below 0.05), and for context the model's own runs with the option (answered = right + wrong, and wrong of answers = wrong / answered), which are not part of the test; a setup without one reads none.

Turkish: k = 1,582 of the 2,198 questions

Each setup's curve over every number of answers: Every setup's curve in Turkish, below.

Decision models

ModelFive-option runthe run whose k answers are keptWrong of the k answerswrong / 1,582; lower is betterDifference from d1-ultrapoints, with its 95% interval; positive = more wrong answersppaired bootstrap, two-sidedHolm padjusted over the 126Runs with the optionfor context: answered, wrong of answers
Copernicus-31B Open · 31B11656.0%95 of 1,582-4.6-6.3 to -3.20.00020.025 *1172“Bilmiyorum”, run 1172: 2,193 answered (99.8%), 18.3% wrong (402)
KnowLine-31B-Gen1.5 Open · 31B9576.6%105 of 1,582-3.9-5.6 to -2.50.00020.025 *959“Bilmiyorum”, run 959: 2,197 answered (100.0%), 18.7% wrong (411)
Turion Jev 31B Open · 31B12737.4%117 of 1,582-3.2-4.7 to -1.50.00020.025 *1285“Bilmiyorum”, run 1285: 2,183 answered (99.3%), 20.0% wrong (436)
KnowLine-31B-Gen2 Open · 31B13147.7%122 of 1,582-2.8-4.7 to -1.50.00020.025 *1331“Bilmiyorum”, run 1331: 2,198 answered (100.0%), 18.6% wrong (408)
Karpo Decision 31B Open · 31B11569.2%146 of 1,582-1.3-3.1 to +0.20.0840.591163“Bilmiyorum”, run 1163: 2,195 answered (99.9%), 21.6% wrong (475)
REAX 1.3 Open · 57B8279.5%151 of 1,582-1.0-2.7 to +0.50.181833“Bilmiyorum”, run 833: 2,196 answered (99.9%), 21.6% wrong (475)
Gemini Decision 1 20261111 Closed · Efficient10219.9%156 of 1,582-0.7-2.5 to +1.00.4511023“Bilmiyorum”, run 1023: 2,188 answered (99.5%), 23.1% wrong (505)
d1-ultra Closed · Efficient19910.6%167 of 1,582referenceits own run 511 with “I don't know”: 216 wrong of 1,582 (13.7%), not part of the testreferencereference350 · 511“Bilmiyorum”, run 350: 1,581 answered (71.9%), 13.0% wrong (205)“I don't know”, run 511: 1,582 answered (72.0%), 13.7% wrong (216)
Sieve 31B v3 Open · 31B124111.6%183 of 1,582+1.0-0.9 to +2.80.3211253“Bilmiyorum”, run 1253: 2,198 answered (100.0%), 25.2% wrong (553)
Logits Sigma-1 Duo Open · 57B122113.2%209 of 1,582+2.7+0.6 to +4.20.00920.0831233“Bilmiyorum”, run 1233: 2,196 answered (99.9%), 24.7% wrong (543)
GLiDE 27B Open · 27B89916.4%259 of 1,582+5.8+3.7 to +7.60.00020.025 *905“Bilmiyorum”, run 905: 1,907 answered (86.8%), 23.1% wrong (441)
GLiDE v1.2 Open · 27B24818.3%289 of 1,582+7.7+5.8 to +9.70.00020.025 *369 · 525“Bilmiyorum”, run 369: 1,877 answered (85.4%), 26.2% wrong (491)“I don't know”, run 525: 1,772 answered (80.6%), 24.1% wrong (427)
Perplexity Decider 27B v1.2 API · with web Open · 27B64019.8%313 of 1,582+9.2+7.1 to +11.10.00020.025 *642“Bilmiyorum”, run 642: 2,175 answered (99.0%), 32.3% wrong (702)
d3 Open · 26B25919.8%314 of 1,582+9.3+7.3 to +11.30.00020.025 *358 · 544“Bilmiyorum”, run 358: 2,198 answered (100.0%), 32.5% wrong (715)“I don't know”, run 544: 2,196 answered (99.9%), 32.5% wrong (714)
d3-ultra Open · 26B26919.9%315 of 1,582+9.4+7.2 to +11.30.00020.025 *363 · 540“Bilmiyorum”, run 363: 2,188 answered (99.5%), 32.2% wrong (704)“I don't know”, run 540: 2,189 answered (99.6%), 32.2% wrong (704)
Perplexity Decider 27B v1.1 Open · 27B14520.1%318 of 1,582+9.5+7.6 to +11.60.00020.025 *noneno run with the option
Perplexity Decider 27B v1.3 Open · 27B113420.2%320 of 1,582+9.7+7.7 to +11.60.00020.025 *1141“Bilmiyorum”, run 1141: 2,193 answered (99.8%), 32.4% wrong (711)
Drex Max Open · 27B32220.3%321 of 1,582+9.7+7.6 to +11.70.00020.025 *367 · 542“Bilmiyorum”, run 367: 2,196 answered (99.9%), 32.4% wrong (711)“I don't know”, run 542: 2,196 answered (99.9%), 32.4% wrong (712)
Drex Max v1.1 Closed · Efficient (provisional)32620.3%321 of 1,582+9.7+7.8 to +11.80.00020.025 *352 · 513“Bilmiyorum”, run 352: 1,878 answered (85.4%), 27.5% wrong (517)“I don't know”, run 513: 1,712 answered (77.9%), 24.1% wrong (413)
Mammoth-1 Open · 27B37420.4%322 of 1,582+9.8+7.8 to +11.80.00020.025 *386 · 519“Bilmiyorum”, run 386: 1,508 answered (68.6%), 20.8% wrong (314)“I don't know”, run 519: 1,445 answered (65.7%), 19.1% wrong (276)
d1.2-ultra Closed · Efficient134720.4%323 of 1,582+9.9+7.9 to +11.80.00020.025 *1367 · 1378“Bilmiyorum”, run 1367: 1,202 answered (54.7%), 12.1% wrong (145)“I don't know”, run 1378: 1,180 answered (53.7%), 11.6% wrong (137)
GLiDE API Closed · Efficient12920.5%325 of 1,582+10.0+7.9 to +12.00.00020.025 *351 · 510“Bilmiyorum”, run 351: 1,677 answered (76.3%), 21.2% wrong (356)“I don't know”, run 510: 1,617 answered (73.6%), 20.7% wrong (335)Made with Fastino's engine glide_v2, not run 129's glide
Perplexity Decider 27B v1.2 DGX Spark Open · 27B21720.7%328 of 1,582+10.2+8.0 to +12.10.00020.025 *372 · 531“Bilmiyorum”, run 372: 2,196 answered (99.9%), 32.7% wrong (717)“I don't know”, run 531: 2,194 answered (99.8%), 33.2% wrong (728)
Perplexity Decider 27B v1.2 API Open · 27B63320.8%329 of 1,582+10.2+8.0 to +12.30.00020.025 *641“Bilmiyorum”, run 641: 2,195 answered (99.9%), 33.1% wrong (726)
SubQ Snap Preview Closed · Efficient17722.5%356 of 1,582+11.9+9.7 to +13.90.00020.025 *noneno run with the option
SubQ Snap 1.1 Closed · Efficient34622.8%361 of 1,582+12.3+10.1 to +14.50.00020.025 *392 · 512“Bilmiyorum”, run 392: 1,761 answered (80.1%), 27.8% wrong (490)“I don't know”, run 512: 1,700 answered (77.3%), 26.7% wrong (454)
GPT-6 Luna Decisions API Closed · Efficient14323.5%372 of 1,582+13.0+10.7 to +15.10.00020.025 *noneno run with the option
KnowLine-12B-Gen1 Open · 12B130926.4%418 of 1,582+15.9+13.7 to +18.10.00020.025 *1318“Bilmiyorum”, run 1318: 2,197 answered (100.0%), 36.6% wrong (804)
Perplexity Decider 27B v1 Open · 27B4427.2%430 of 1,582+16.6+14.5 to +18.90.00020.025 *noneno run with the option
d1.1 Closed · Efficient25427.4%434 of 1,582+16.9+14.4 to +19.00.00020.025 *noneno run with the option
d1.2 Closed · Efficient134627.6%436 of 1,582+17.0+14.7 to +19.20.00020.025 *1365 · 1377“Bilmiyorum”, run 1365: 1,225 answered (55.7%), 20.7% wrong (253)“I don't know”, run 1377: 1,297 answered (59.0%), 21.5% wrong (279)
celeris-1-decision Closed · Efficient15028.4%450 of 1,582+17.9+15.4 to +20.00.00020.025 *noneno run with the option
OneJev-27B Open · 27B26428.8%455 of 1,582+18.2+15.9 to +20.40.00020.025 *782“Bilmiyorum”, run 782: 2,197 answered (100.0%), 40.6% wrong (892)
Jev 1.14 Closed · Efficient19430.7%485 of 1,582+20.1+17.9 to +22.40.00020.025 *noneno run with the option
Mercury Decide Closed · Efficient20430.8%487 of 1,582+20.2+17.8 to +22.60.00020.025 *noneno run with the option
Jev 1.13 Closed · Efficient4931.5%498 of 1,582+20.9+18.5 to +23.20.00020.025 *noneno run with the option
Open-Jev-27B-v1.1 Open · 27B4531.9%504 of 1,582+21.3+18.8 to +23.60.00020.025 *768“Bilmiyorum”, run 768: 2,184 answered (99.4%), 44.2% wrong (965)
Clef Open · 27B12532.4%512 of 1,582+21.8+19.5 to +24.20.00020.025 *788“Bilmiyorum”, run 788: 2,127 answered (96.8%), 42.0% wrong (894)
d1 Closed · Efficient5232.6%516 of 1,582+22.1+19.7 to +24.30.00020.025 *noneno run with the option
Jeffan-9B-1211 Open · 9.0B119232.8%519 of 1,582+22.3+19.8 to +24.50.00020.025 *1199“Bilmiyorum”, run 1199: 2,157 answered (98.1%), 42.5% wrong (917)
Drex 2.0 Open · 35B MoE · 3.0B active33433.4%528 of 1,582+22.8+20.4 to +25.00.00020.025 *noneno run with the option
Cygnet Open · 12B12733.5%530 of 1,582+22.9+20.4 to +25.30.00020.025 *757“Bilmiyorum”, run 757: 2,166 answered (98.5%), 44.2% wrong (957)
Winnow-12B Open · 12B12834.1%540 of 1,582+23.6+21.0 to +25.90.00020.025 *750“Bilmiyorum”, run 750: 2,155 answered (98.0%), 44.8% wrong (965)
KnowLine-9B-Gen1 Open · 9.0B117435.0%553 of 1,582+24.4+22.0 to +26.90.00020.025 *1181“Bilmiyorum”, run 1181: 2,193 answered (99.8%), 45.4% wrong (996)
Cerno 9B v1.2 Open · 9.0B120135.3%559 of 1,582+24.8+22.4 to +27.10.00020.025 *1204“Bilmiyorum”, run 1204: 2,196 answered (99.9%), 45.4% wrong (998)
Adjudex 9B v5 Open · 9.0B118336.5%578 of 1,582+26.0+23.6 to +28.30.00020.025 *1190“Bilmiyorum”, run 1190: 2,190 answered (99.6%), 46.6% wrong (1,021)
KnowLine-9B-Gen2 Open · 9.0B130636.7%580 of 1,582+26.1+23.8 to +28.60.00020.025 *1325“Bilmiyorum”, run 1325: 2,171 answered (98.8%), 47.4% wrong (1,030)
Elephant-1 Open · 9.0B37337.0%586 of 1,582+26.5+24.1 to +29.00.00020.025 *382 · 515“Bilmiyorum”, run 382: 1,793 answered (81.6%), 42.3% wrong (758)“I don't know”, run 515: 1,594 answered (72.5%), 39.0% wrong (622)
GLiDE 9B Open · 7.9B90737.9%600 of 1,582+27.4+24.8 to +29.60.00020.025 *913“Bilmiyorum”, run 913: 1,921 answered (87.4%), 45.4% wrong (872)
StartLux-Decision-v1.1-9B-Instant Open · 9.0B129039.2%620 of 1,582+28.6+26.1 to +31.00.00020.025 *1297“Bilmiyorum”, run 1297: 2,174 answered (98.9%), 49.2% wrong (1,069)
Dexy-Decision-9B-V2 Open · 9.0B125539.4%623 of 1,582+28.8+26.2 to +31.20.00020.025 *1262“Bilmiyorum”, run 1262: 2,161 answered (98.3%), 48.4% wrong (1,046)
vera-medium-omni Open · 4.2B126442.0%665 of 1,582+31.5+28.8 to +33.90.00020.025 *1271“Bilmiyorum”, run 1271: 2,093 answered (95.2%), 51.4% wrong (1,075)
Perplexity Decider Flash v1.3 Open · 4.0B112342.2%667 of 1,582+31.6+29.1 to +34.10.00020.025 *1130“Bilmiyorum”, run 1130: 2,123 answered (96.6%), 50.4% wrong (1,071)
StartLux-Decision-v1.1-4B-Instant Open · 4.0B123642.6%674 of 1,582+32.0+29.4 to +34.40.00020.025 *1239“Bilmiyorum”, run 1239: 2,177 answered (99.0%), 52.7% wrong (1,147)
KnowLine 4B Gen5.5 Open · 4.0B96543.0%681 of 1,582+32.5+29.8 to +34.80.00020.025 *967“Bilmiyorum”, run 967: 2,143 answered (97.5%), 52.3% wrong (1,121)
Perplexity Decider Flash v1.2 Open · 4.0B22343.7%691 of 1,582+33.1+30.6 to +35.60.00020.025 *791“Bilmiyorum”, run 791: 2,018 answered (91.8%), 51.7% wrong (1,044)
Valen-Decision-1.5-9B Open · 9.0B127544.1%698 of 1,582+33.6+30.9 to +36.00.00020.025 *1282“Bilmiyorum”, run 1282: 2,197 answered (100.0%), 52.7% wrong (1,157)
Verdin-4B-v1.1 Open · 4.0B124444.8%708 of 1,582+34.2+31.5 to +36.70.00020.025 *1251“Bilmiyorum”, run 1251: 2,193 answered (99.8%), 53.3% wrong (1,169)
Clef-Flash Open · 9.0B13945.3%716 of 1,582+34.7+32.0 to +37.20.00020.025 *664“Bilmiyorum”, run 664: 2,088 answered (95.0%), 53.1% wrong (1,108)
Chinese Decision Pointer 4B Open · 4.0B120645.7%723 of 1,582+35.1+32.5 to +37.60.00020.025 *1213“Bilmiyorum”, run 1213: 2,062 answered (93.8%), 52.5% wrong (1,082)
Microsoft-Decision-1 Closed · Efficient20545.9%726 of 1,582+35.3+32.6 to +37.80.00020.025 *noneno run with the option
Metask-Jev-4B Open · 4.5B12653.2%841 of 1,582+42.6+39.8 to +45.10.00020.025 *743“Bilmiyorum”, run 743: 2,188 answered (99.5%), 60.1% wrong (1,315)
KnowLine-2B-Gen1 Open · 2.0B107958.2%920 of 1,582+47.6+44.8 to +50.20.00020.025 *noneno run with the option
GLiDE 2B Open · 1.9B91561.0%965 of 1,582+50.4+47.6 to +53.20.00020.025 *921“Bilmiyorum”, run 921: 1,586 answered (72.2%), 61.7% wrong (978)
StartLux-Decision-v1.1-2B-Instant Open · 2.0B97761.4%972 of 1,582+50.9+48.1 to +53.50.00020.025 *979“Bilmiyorum”, run 979: 1,865 answered (84.8%), 65.0% wrong (1,213)
d1-2B Open · 1.9B34067.0%1,060 of 1,582+56.4+53.7 to +59.00.00020.025 *712“Bilmiyorum”, run 712: 989 answered (45.0%), 64.4% wrong (637)
OneJev1.6-0.8B Open · 0.8B102967.8%1,072 of 1,582+57.2+54.5 to +59.80.00020.025 *1031“Bilmiyorum”, run 1031: 1,444 answered (65.7%), 68.2% wrong (985)
GLiDE 0.8B Open · 0.8B92368.3%1,080 of 1,582+57.7+54.9 to +60.10.00020.025 *929“Bilmiyorum”, run 929: 1,718 answered (78.2%), 67.7% wrong (1,163)
Strands Decider 2B Open · 2.0B14169.3%1,096 of 1,582+58.7+55.9 to +61.30.00020.025 *706“Bilmiyorum”, run 706: 2,048 answered (93.2%), 72.4% wrong (1,483)
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B101569.4%1,098 of 1,582+58.8+56.1 to +61.30.00020.025 *1017“Bilmiyorum”, run 1017: 1,999 answered (90.9%), 70.5% wrong (1,409)
KnowLine-0.8B-Gen1 Open · 0.8B100570.0%1,107 of 1,582+59.4+56.6 to +61.90.00020.025 *1007“Bilmiyorum”, run 1007: 2,094 answered (95.3%), 72.3% wrong (1,515)
Decider 1 Closed · Efficient24370.8%1,120 of 1,582+60.2+57.5 to +62.80.00020.025 *noneno run with the option
GLiClass v4 Edge Open · 0.4B104675.7%1,197 of 1,582+65.1+62.3 to +67.50.00020.025 *1048“Bilmiyorum”, run 1048: 1,800 answered (81.9%), 77.0% wrong (1,386)
GLiDE 0.4B Open · 0.4B93176.5%1,211 of 1,582+66.0+63.3 to +68.50.00020.025 *937“Bilmiyorum”, run 937: 128 answered (5.8%), 78.9% wrong (101)
GLiNER2.5-multi-Decide Open · 0.3B13578.9%1,248 of 1,582+68.3+65.6 to +70.70.00020.025 *718“Bilmiyorum”, run 718: 2,084 answered (94.8%), 79.9% wrong (1,666)
Laya Open · 0.4B5180.5%1,273 of 1,582+69.9+67.3 to +72.40.00020.025 *730“Bilmiyorum”, run 730: 1,662 answered (75.6%), 81.5% wrong (1,355)
Laya Multilingual Open · 0.3B4683.5%1,321 of 1,582+72.9+70.4 to +75.30.00020.025 *737“Bilmiyorum”, run 737: 1,918 answered (87.3%), 83.8% wrong (1,608)

Other models: chat and general models

ModelFive-option runthe run whose k answers are keptWrong of the k answerswrong / 1,582; lower is betterDifference from d1-ultrapoints, with its 95% interval; positive = more wrong answersppaired bootstrap, two-sidedHolm padjusted over the 126Runs with the optionfor context: answered, wrong of answers
Gemini 4 Carbon high · chat Closed · Flagship11080.3%4 of 1,582-10.3-12.1 to -8.80.00020.025 *noneno run with the option
GPT-6 Astra low · chat Closed · Flagship480.3%4 of 1,582-10.3-12.1 to -8.80.00020.025 *noneno run with the option
Claude Fable 5.2 max · chat Closed · Flagship9800.3%5 of 1,582-10.2-12.0 to -8.80.00020.025 *noneno run with the option
Gemini 4 Hydrogen high · chat Closed · Workhorse11140.3%5 of 1,582-10.2-12.0 to -8.70.00020.025 *noneno run with the option
GPT-6 Astra high · chat Closed · Flagship2770.4%6 of 1,582-10.2-11.9 to -8.70.00020.025 *noneno run with the option
GPT-6 Astra medium · chat Closed · Flagship6100.4%6 of 1,582-10.2-11.9 to -8.70.00020.025 *noneno run with the option
Gemini 3.8 Flash high · chat Closed · Workhorse1400.4%7 of 1,582-10.1-11.8 to -8.60.00020.025 *noneno run with the option
GPT-6.1 Sol high · chat Closed · Workhorse2760.5%8 of 1,582-10.1-11.8 to -8.50.00020.025 *noneno run with the option
GPT-6 Astra low · chat · 1 per request Closed · Flagship1910.7%11 of 1,582-9.9-11.6 to -8.30.00020.025 *noneno run with the option
GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse1900.8%12 of 1,582-9.8-11.5 to -8.30.00020.025 *noneno run with the option
Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse1880.8%13 of 1,582-9.7-11.5 to -8.20.00020.025 *noneno run with the option
GPT-6.1 Sol low · chat Closed · Workhorse1240.8%13 of 1,582-9.7-11.5 to -8.20.00020.025 *noneno run with the option
Gemini 4 Argon low · chat Closed · Flagship5571.1%17 of 1,582-9.5-11.2 to -8.00.00020.025 *noneno run with the option
Claude Fable 5.1 high · chat Closed · Flagship3101.2%19 of 1,582-9.4-11.1 to -8.00.00020.025 *noneno run with the option
Claude Opus 5.5 high · chat Closed · Flagship2781.5%23 of 1,582-9.1-10.8 to -7.60.00020.025 *noneno run with the option
Claude Opus 5.5 low · chat · 1 per request Closed · Flagship1921.5%23 of 1,582-9.1-10.7 to -7.60.00020.025 *noneno run with the option
GPT-6.1 Terra high · chat Closed · Workhorse2752.0%31 of 1,582-8.6-10.3 to -7.10.00020.025 *noneno run with the option
Grok 4.7 high · chat Closed · Workhorse3112.0%31 of 1,582-8.6-10.3 to -7.10.00020.025 *noneno run with the option
Qwen4-Max high · chat Closed · Workhorse13362.0%31 of 1,582-8.6-10.4 to -7.10.00020.025 *noneno run with the option
Gemini 3.8 Flash low · chat Closed · Workhorse1233.2%50 of 1,582-7.4-9.2 to -5.80.00020.025 *noneno run with the option
GPT-6.1 Terra low · chat Closed · Workhorse2743.2%50 of 1,582-7.4-9.2 to -6.00.00020.025 *noneno run with the option
Claude Opus 5.5 low · chat Closed · Flagship473.2%51 of 1,582-7.3-9.0 to -5.90.00020.025 *noneno run with the option
Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse1934.1%65 of 1,582-6.4-8.1 to -4.90.00020.025 *noneno run with the option
Muse Ember max · chat Closed · Efficient9395.4%86 of 1,582-5.1-7.0 to -3.40.00020.025 *noneno run with the option
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active5855.5%87 of 1,582-5.1-7.0 to -3.40.00020.025 *noneno run with the option
Claude Sonnet 5.5 high · chat Closed · Workhorse557.1%113 of 1,582-3.4-5.3 to -1.80.00020.025 *noneno run with the option
DeepSeek-V4.1-Flash reasoning on · chat · first release Open · 552B MoE · 16B active1217.1%113 of 1,582-3.4-5.4 to -1.80.00020.025 *noneno run with the option
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active3208.9%141 of 1,582-1.6-3.5 to +0.00.0510.41noneno run with the option
Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active87010.0%158 of 1,582-0.6-2.5 to +1.10.451noneno run with the option
GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active16010.3%163 of 1,582-0.3-2.2 to +1.40.711noneno run with the option
Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active86611.4%181 of 1,582+0.9-1.1 to +2.70.451noneno run with the option
Gemma 4 31B reasoning off · chat Open · 31B11915.1%239 of 1,582+4.6+2.5 to +6.10.00020.025 *noneno run with the option
GLM-5.3 low · chat Open · Workhorse31416.4%260 of 1,582+5.9+3.8 to +7.80.00020.025 *noneno run with the option
Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active31518.6%294 of 1,582+8.0+5.8 to +9.90.00020.025 *noneno run with the option
MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active87718.9%299 of 1,582+8.3+6.1 to +10.40.00020.025 *noneno run with the option
Qwen3.8-27B low · chat Open · 27B31819.6%310 of 1,582+9.0+6.7 to +10.90.00020.025 *noneno run with the option
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active55821.8%345 of 1,582+11.3+8.9 to +13.30.00020.025 *noneno run with the option
DeepSeek-V4.1-Flash reasoning off · chat · first release Open · 552B MoE · 16B active12021.9%346 of 1,582+11.3+9.3 to +13.60.00020.025 *noneno run with the option
Gemma 4 31B reasoning off · chat · 1 per request Open · 31B18921.9%346 of 1,582+11.3+9.0 to +13.30.00020.025 *noneno run with the option
GPT-6 Luna low · chat Closed · Efficient13324.5%388 of 1,582+14.0+11.8 to +16.20.00020.025 *noneno run with the option
MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active86225.7%406 of 1,582+15.1+12.8 to +17.40.00020.025 *noneno run with the option
Claude Sonnet 5.5 low · chat Closed · Workhorse5427.8%440 of 1,582+17.3+14.9 to +19.50.00020.025 *noneno run with the option
Qwen3.8 Flash no reasoning · chat Closed · Efficient98529.2%462 of 1,582+18.6+16.4 to +20.70.00020.025 *noneno run with the option
GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active15932.6%515 of 1,582+22.0+19.5 to +24.30.00020.025 *noneno run with the option
Qwen3.8-27B vote share · reasoning off · chat Open · 27B8335.3%559 of 1,582+24.8+22.2 to +27.00.00020.025 *noneno run with the option
Qwen3.8-27B yes/no scoring Open · 27B5037.9%599 of 1,582+27.3+24.9 to +29.70.00020.025 *noneno run with the option
Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B5340.1%635 of 1,582+29.6+27.1 to +31.90.00020.025 *noneno run with the option
Erk-14B reasoning off · chat Open · 14B13246.5%735 of 1,582+35.9+33.1 to +38.40.00020.025 *noneno run with the option
Qwen3-14B reasoning off · chat Open · 15B13147.1%745 of 1,582+36.5+33.6 to +38.90.00020.025 *noneno run with the option
GLiNER2.5 Multi Open · 0.3B12283.5%1,321 of 1,582+72.9+70.4 to +75.20.00020.025 *noneno run with the option
Claude Haiku 5.5 max · chat Closed · Efficient948—left out: its run has 1,414 scored answers, fewer than k = 1,582—left out, fewer than k scored answers——noneno run with the option
Gemma 4 E4B default reasoning · chat Open · 8.0B1040Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
LFM2.5-2.6B default reasoning · chat Open · 2.7B1037Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active1288Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
LFM3-1.7B reasoning off · chat Open · 1.7B1227Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
LFM3-1.7B reasoning on · chat Open · 1.7B1089Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
MiniCPM5-2B reasoning off · chat Open · 2.5B1299Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
Qwen3.5-2B reasoning off · chat Open · 2.0B1218Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
Qwen3.5-4B reasoning off · chat Open · 4.0B1146Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
Qwen3.5-4B reasoning on · chat Open · 4.0B1149Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option

English: k = 496 of the 547 kept reading questions

ModelFive-option runthe run whose k answers are keptWrong of the k answerswrong / 496; lower is betterDifference from d1-ultrapoints, with its 95% interval; positive = more wrong answersppaired bootstrap, two-sidedHolm padjusted over the 123Runs with the optionfor context: answered, wrong of answers
Claude Fable 5.2 max · chat Closed · Flagship9710.4%2 of 496-5.4-7.7 to -3.20.00020.025 *noneno run with the option
Gemini 4 Carbon high · chat Closed · Flagship10550.4%2 of 496-5.4-7.5 to -3.40.00020.025 *noneno run with the option
Gemini 4 Hydrogen high · chat Closed · Workhorse11070.6%3 of 496-5.2-7.5 to -3.20.00020.025 *noneno run with the option
GPT-6 Astra low · chat Closed · Flagship6160.8%4 of 496-5.0-7.1 to -3.00.00020.025 *noneno run with the option
GPT-6 Astra high · chat Closed · Flagship6151.0%5 of 496-4.8-6.9 to -2.80.00020.025 *noneno run with the option
GPT-6 Astra medium · chat Closed · Flagship6241.0%5 of 496-4.8-6.7 to -2.60.00020.025 *noneno run with the option
GPT-6.1 Sol high · chat Closed · Workhorse6181.2%6 of 496-4.6-6.7 to -2.60.00020.025 *noneno run with the option
GPT-6.1 Sol low · chat Closed · Workhorse6191.2%6 of 496-4.6-6.7 to -2.60.00020.025 *noneno run with the option
Gemini 4 Argon low · chat Closed · Flagship5931.4%7 of 496-4.4-6.5 to -2.40.00020.025 *noneno run with the option
GPT-6 Astra low · chat · 1 per request Closed · Flagship6221.4%7 of 496-4.4-6.7 to -2.40.00020.025 *noneno run with the option
GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse6231.6%8 of 496-4.2-6.5 to -2.20.00020.025 *noneno run with the option
GPT-6.1 Terra high · chat Closed · Workhorse6201.6%8 of 496-4.2-6.0 to -2.20.00020.025 *noneno run with the option
GPT-6.1 Terra low · chat Closed · Workhorse6211.6%8 of 496-4.2-6.2 to -2.20.00020.025 *noneno run with the option
Gemini 3.8 Flash high · chat Closed · Workhorse5831.8%9 of 496-4.0-6.2 to -2.00.00020.025 *noneno run with the option
Claude Fable 5.1 high · chat Closed · Flagship5802.0%10 of 496-3.8-6.0 to -2.00.00040.028 *noneno run with the option
Claude Opus 5.5 low · chat · 1 per request Closed · Flagship5922.0%10 of 496-3.8-5.6 to -1.80.00040.028 *noneno run with the option
Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse5892.0%10 of 496-3.8-5.8 to -1.80.00040.028 *noneno run with the option
Claude Opus 5.5 high · chat Closed · Flagship5872.6%13 of 496-3.2-5.0 to -1.20.0020.12noneno run with the option
Gemini 3.8 Flash low · chat Closed · Workhorse5782.8%14 of 496-3.0-4.8 to -1.00.0050.24noneno run with the option
Grok 4.7 high · chat Closed · Workhorse5792.8%14 of 496-3.0-5.0 to -1.00.00220.13noneno run with the option
KnowLine-31B-Gen1.5 Open · 31B9532.8%14 of 496-3.0-4.8 to -1.20.00280.15955“I don't know”, run 955: 547 answered (100.0%), 6.8% wrong (37)
Qwen4-Max high · chat Closed · Workhorse13342.8%14 of 496-3.0-5.0 to -1.20.00180.11noneno run with the option
Copernicus-31B Open · 31B11683.2%16 of 496-2.6-4.4 to -1.00.00320.171170“I don't know”, run 1170: 547 answered (100.0%), 7.1% wrong (39)
KnowLine-31B-Gen2 Open · 31B13273.2%16 of 496-2.6-4.4 to -0.60.0120.461329“I don't know”, run 1329: 547 answered (100.0%), 7.3% wrong (40)
REAX 1.3 Open · 57B8293.2%16 of 496-2.6-4.2 to -0.60.0110.46831“I don't know”, run 831: 544 answered (99.5%), 7.7% wrong (42)
Claude Opus 5.5 low · chat Closed · Flagship5883.8%19 of 496-2.0-4.2 to +0.00.0531noneno run with the option
Claude Haiku 5.5 max · chat Closed · Efficient9494.0%20 of 496-1.8-4.0 to +0.40.121noneno run with the option
Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse5944.4%22 of 496-1.4-3.4 to +0.80.31noneno run with the option
Karpo Decision 31B Open · 31B11594.4%22 of 496-1.4-3.2 to +0.60.2411161“I don't know”, run 1161: 547 answered (100.0%), 7.7% wrong (42)
Logits Sigma-1 Duo Open · 57B12294.6%23 of 496-1.2-2.8 to +0.60.2811231“I don't know”, run 1231: 545 answered (99.6%), 8.1% wrong (44)
Turion Jev 31B Open · 31B10754.6%23 of 496-1.2-3.0 to +0.80.311077“I don't know”, run 1077: 543 answered (99.3%), 7.9% wrong (43)
d1.2-ultra Closed · Efficient13594.8%24 of 496-1.0-3.0 to +1.20.4211363“I don't know”, run 1363: 438 answered (80.1%), 3.0% wrong (13)
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active5844.8%24 of 496-1.0-3.2 to +1.20.421noneno run with the option
Sieve 31B v3 Open · 31B10644.8%24 of 496-1.0-3.2 to +1.00.3811066“I don't know”, run 1066: 546 answered (99.8%), 8.2% wrong (45)
Gemini Decision 1 20261111 Closed · Efficient10095.0%25 of 496-0.8-2.8 to +1.40.5111019“I don't know”, run 1019: 542 answered (99.1%), 9.2% wrong (50)
GLiDE v1.2 Open · 27B5215.2%26 of 496-0.6-2.4 to +1.60.761523“I don't know”, run 523: 517 answered (94.5%), 7.0% wrong (36)
Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active8715.2%26 of 496-0.6-2.4 to +1.60.781noneno run with the option
GLiDE 27B Open · 27B9015.4%27 of 496-0.4-2.8 to +1.40.61903“I don't know”, run 903: 525 answered (96.0%), 6.9% wrong (36)
Drex Max Open · 27B5525.6%28 of 496-0.2-2.2 to +2.00.981798“I don't know”, run 798: 545 answered (99.6%), 11.0% wrong (60)
d1-ultra Closed · Efficient4115.8%29 of 496referenceits own run 449 with “I don't know”: 39 wrong of 496 (7.9%), not part of the testreferencereference449“I don't know”, run 449: 496 answered (90.7%), 7.9% wrong (39)
d3 Open · 26B5545.8%29 of 496+0.0-1.8 to +2.20.931793“I don't know”, run 793: 546 answered (99.8%), 10.6% wrong (58)
Gemma 4 31B reasoning off · chat Open · 31B7745.8%29 of 496+0.0-2.2 to +2.00.991noneno run with the option
Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active5826.0%30 of 496+0.2-2.2 to +2.211noneno run with the option
d1.2 Closed · Efficient13576.2%31 of 496+0.4-1.6 to +2.60.7411361“I don't know”, run 1361: 488 answered (89.2%), 7.6% wrong (37)
d3-ultra Open · 26B5506.2%31 of 496+0.4-1.6 to +2.40.771795“I don't know”, run 795: 542 answered (99.1%), 10.3% wrong (56)
Mammoth-1 Open · 27B5036.2%31 of 496+0.4-1.8 to +2.40.841505“I don't know”, run 505: 510 answered (93.2%), 7.1% wrong (36)
Drex Max v1.1 Closed · Efficient (provisional)3996.5%32 of 496+0.6-1.6 to +2.80.641447“I don't know”, run 447: 504 answered (92.1%), 8.1% wrong (41)
GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active8236.9%34 of 496+1.0-1.4 to +3.40.431noneno run with the option
Perplexity Decider 27B v1.2 DGX Spark Open · 27B5276.9%34 of 496+1.0-1.0 to +3.20.321529“I don't know”, run 529: 545 answered (99.6%), 11.7% wrong (64)
MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active8787.1%35 of 496+1.2-1.0 to +3.40.351noneno run with the option
Perplexity Decider 27B v1 Open · 27B6267.1%35 of 496+1.2-0.8 to +3.60.221noneno run with the option
Perplexity Decider 27B v1.3 Open · 27B11377.1%35 of 496+1.2-1.0 to +3.20.3411139“I don't know”, run 1139: 542 answered (99.1%), 9.6% wrong (52)
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active8087.3%36 of 496+1.4-1.0 to +3.80.31noneno run with the option
Gemma 4 31B reasoning off · chat · 1 per request Open · 31B7837.3%36 of 496+1.4-0.8 to +3.60.261noneno run with the option
Muse Ember max · chat Closed · Efficient9427.3%36 of 496+1.4-0.6 to +4.00.191noneno run with the option
Perplexity Decider 27B v1.2 API Open · 27B6377.3%36 of 496+1.4-0.6 to +3.80.171643“I don't know”, run 643: 544 answered (99.5%), 11.6% wrong (63)
Perplexity Decider 27B v1.2 API · with web Open · 27B6397.5%37 of 496+1.6-0.4 to +3.80.161644“I don't know”, run 644: 537 answered (98.2%), 10.4% wrong (56)
d1.1 Closed · Efficient5658.5%42 of 496+2.6+0.2 to +4.80.0381noneno run with the option
Claude Sonnet 5.5 high · chat Closed · Workhorse5908.7%43 of 496+2.8+0.2 to +5.80.0421noneno run with the option
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active8048.7%43 of 496+2.8+0.4 to +5.00.0280.95noneno run with the option
KnowLine-12B-Gen1 Open · 12B13138.7%43 of 496+2.8+0.4 to +5.00.0210.761316“I don't know”, run 1316: 546 answered (99.8%), 13.9% wrong (76)
Clef Open · 27B6138.9%44 of 496+3.0+0.6 to +5.20.0140.56785“I don't know”, run 785: 546 answered (99.8%), 13.7% wrong (75)
d1 Closed · Efficient4308.9%44 of 496+3.0+0.8 to +5.60.0170.62445“I don't know”, run 445: 545 answered (99.6%), 12.3% wrong (67)
Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active8678.9%44 of 496+3.0+0.6 to +5.40.0140.56noneno run with the option
Jev 1.14 Closed · Efficient4108.9%44 of 496+3.0+0.4 to +5.20.0240.83446“I don't know”, run 446: 544 answered (99.5%), 13.2% wrong (72)
SubQ Snap 1.1 Closed · Efficient4138.9%44 of 496+3.0+1.0 to +5.80.0080.36448“I don't know”, run 448: 526 answered (96.2%), 12.0% wrong (63)
SubQ Snap Preview Closed · Efficient5958.9%44 of 496+3.0+1.0 to +5.80.0080.36noneno run with the option
Jev 1.13 Closed · Efficient5649.1%45 of 496+3.2+1.2 to +5.60.0040.2noneno run with the option
GLiDE 9B Open · 7.9B9099.3%46 of 496+3.4+1.0 to +6.00.00980.42911“I don't know”, run 911: 533 answered (97.4%), 13.1% wrong (70)
Jeffan-9B-1211 Open · 9.0B11959.3%46 of 496+3.4+0.8 to +6.00.010.431197“I don't know”, run 1197: 547 answered (100.0%), 13.0% wrong (71)
OneJev-27B Open · 27B7789.5%47 of 496+3.6+1.2 to +6.00.0050.24780“I don't know”, run 780: 544 answered (99.5%), 13.2% wrong (72)
StartLux-Decision-v1.1-9B-Instant Open · 9.0B12939.5%47 of 496+3.6+1.2 to +6.50.00680.311295“I don't know”, run 1295: 545 answered (99.6%), 13.6% wrong (74)
Adjudex 9B v5 Open · 9.0B11869.7%48 of 496+3.8+1.2 to +6.50.00380.191188“I don't know”, run 1188: 547 answered (100.0%), 14.1% wrong (77)
Open-Jev-27B-v1.1 Open · 27B7609.7%48 of 496+3.8+1.4 to +6.50.00280.15763“I don't know”, run 763: 531 answered (97.1%), 14.3% wrong (76)
Qwen3.8-27B low · chat Open · 27B6679.7%48 of 496+3.8+1.4 to +6.50.00240.13noneno run with the option
GLM-5.3 low · chat Open · Workhorse5779.9%49 of 496+4.0+1.4 to +6.50.00320.17noneno run with the option
KnowLine-9B-Gen1 Open · 9.0B11779.9%49 of 496+4.0+1.2 to +6.70.00440.221179“I don't know”, run 1179: 546 answered (99.8%), 13.9% wrong (76)
GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active82010.1%50 of 496+4.2+1.6 to +6.90.00140.087noneno run with the option
Mercury Decide Closed · Efficient39810.1%50 of 496+4.2+1.6 to +6.70.00140.087450“I don't know”, run 450: 536 answered (98.0%), 13.2% wrong (71)
Drex 2.0 Open · 35B MoE · 3.0B active56610.3%51 of 496+4.4+2.0 to +7.10.00040.028 *noneno run with the option
Qwen3.8-27B vote share · reasoning off · chat Open · 27B67810.3%51 of 496+4.4+2.0 to +7.30.00040.028 *noneno run with the option
Cerno 9B v1.2 Open · 9.0B108510.5%52 of 496+4.6+1.8 to +6.90.00140.0871087“I don't know”, run 1087: 545 answered (99.6%), 14.1% wrong (77)
Microsoft-Decision-1 Closed · Efficient41210.5%52 of 496+4.6+2.2 to +7.30.00060.039 *451“I don't know”, run 451: 544 answered (99.5%), 14.2% wrong (77)
KnowLine-9B-Gen2 Open · 9.0B132110.9%54 of 496+5.0+2.4 to +7.90.0010.0631323“I don't know”, run 1323: 546 answered (99.8%), 14.3% wrong (78)
Qwen3.8-27B yes/no scoring Open · 27B69610.9%54 of 496+5.0+2.4 to +7.70.00020.025 *noneno run with the option
Elephant-1 Open · 9.0B49911.1%55 of 496+5.2+2.6 to +7.90.00020.025 *501“I don't know”, run 501: 524 answered (95.8%), 12.8% wrong (67)
Qwen3.8 Flash no reasoning · chat Closed · Efficient98611.1%55 of 496+5.2+2.6 to +7.90.00020.025 *noneno run with the option
celeris-1-decision Closed · Efficient54611.3%56 of 496+5.4+2.6 to +8.10.00020.025 *548“I don't know”, run 548: 455 answered (83.2%), 8.1% wrong (37)
GPT-6 Luna low · chat Closed · Efficient61711.3%56 of 496+5.4+2.6 to +8.50.00060.039 *noneno run with the option
Valen-Decision-1.5-9B Open · 9.0B127811.5%57 of 496+5.6+3.2 to +8.70.00020.025 *1280“I don't know”, run 1280: 546 answered (99.8%), 15.4% wrong (84)
GPT-6 Luna Decisions API Closed · Efficient39711.7%58 of 496+5.8+3.2 to +8.90.00020.025 *444“I don't know”, run 444: 537 answered (98.2%), 17.9% wrong (96)
MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active86311.7%58 of 496+5.8+3.4 to +8.70.00020.025 *noneno run with the option
Claude Sonnet 5.5 low · chat Closed · Workhorse59111.9%59 of 496+6.0+3.2 to +9.30.00020.025 *noneno run with the option
Perplexity Decider Flash v1.3 Open · 4.0B112611.9%59 of 496+6.0+3.2 to +9.10.00020.025 *1128“I don't know”, run 1128: 540 answered (98.7%), 16.7% wrong (90)
vera-medium-omni Open · 4.2B126711.9%59 of 496+6.0+3.2 to +8.70.00020.025 *1269“I don't know”, run 1269: 540 answered (98.7%), 16.1% wrong (87)
Cygnet Open · 12B75312.1%60 of 496+6.2+3.6 to +8.90.00020.025 *755“I don't know”, run 755: 537 answered (98.2%), 14.7% wrong (79)
Winnow-12B Open · 12B74612.1%60 of 496+6.2+3.6 to +9.10.00020.025 *748“I don't know”, run 748: 532 answered (97.3%), 16.0% wrong (85)
Dexy-Decision-9B-V2 Open · 9.0B125812.3%61 of 496+6.5+3.4 to +9.30.00020.025 *1260“I don't know”, run 1260: 546 answered (99.8%), 16.5% wrong (90)
Perplexity Decider Flash v1.2 Open · 4.0B53312.5%62 of 496+6.7+4.0 to +9.90.00020.025 *535“I don't know”, run 535: 542 answered (99.1%), 18.5% wrong (100)
StartLux-Decision-v1.1-4B-Instant Open · 4.0B108112.5%62 of 496+6.7+4.0 to +10.10.00020.025 *1083“I don't know”, run 1083: 547 answered (100.0%), 17.2% wrong (94)
Verdin-4B-v1.1 Open · 4.0B124712.7%63 of 496+6.9+3.8 to +9.90.00020.025 *1249“I don't know”, run 1249: 545 answered (99.6%), 17.2% wrong (94)
Clef-Flash Open · 9.0B61413.5%67 of 496+7.7+4.6 to +10.70.00020.025 *660“I don't know”, run 660: 542 answered (99.1%), 16.6% wrong (90)
Chinese Decision Pointer 4B Open · 4.0B120914.9%74 of 496+9.1+6.2 to +12.50.00020.025 *1211“I don't know”, run 1211: 543 answered (99.3%), 19.2% wrong (104)
KnowLine 4B Gen5.5 Open · 4.0B96114.9%74 of 496+9.1+6.2 to +12.30.00020.025 *963“I don't know”, run 963: 546 answered (99.8%), 19.2% wrong (105)
Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B67215.1%75 of 496+9.3+6.3 to +12.10.00020.025 *noneno run with the option
Qwen3-14B reasoning off · chat Open · 15B77015.5%77 of 496+9.7+6.5 to +13.10.00020.025 *noneno run with the option
Metask-Jev-4B Open · 4.5B73918.1%90 of 496+12.3+9.1 to +15.50.00020.025 *741“I don't know”, run 741: 545 answered (99.6%), 22.6% wrong (123)
Erk-14B reasoning off · chat Open · 14B76223.6%117 of 496+17.7+14.1 to +21.60.00020.025 *noneno run with the option
KnowLine-2B-Gen1 Open · 2.0B105724.8%123 of 496+19.0+15.1 to +23.20.00020.025 *1059“I don't know”, run 1059: 509 answered (93.1%), 26.5% wrong (135)
GLiDE 2B Open · 1.9B91725.2%125 of 496+19.4+15.7 to +23.00.00020.025 *919“I don't know”, run 919: 500 answered (91.4%), 26.4% wrong (132)
StartLux-Decision-v1.1-2B-Instant Open · 2.0B97326.0%129 of 496+20.2+16.5 to +24.20.00020.025 *975“I don't know”, run 975: 515 answered (94.1%), 30.5% wrong (157)
d1-2B Open · 1.9B70829.4%146 of 496+23.6+19.8 to +27.80.00020.025 *710“I don't know”, run 710: 396 answered (72.4%), 24.5% wrong (97)
GLiDE 0.4B Open · 0.4B93336.5%181 of 496+30.6+26.6 to +35.10.00020.025 *935“I don't know”, run 935: 394 answered (72.0%), 34.3% wrong (135)
GLiDE 0.8B Open · 0.8B92536.7%182 of 496+30.8+26.4 to +35.10.00020.025 *927“I don't know”, run 927: 498 answered (91.0%), 35.5% wrong (177)
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B101137.5%186 of 496+31.7+27.2 to +35.90.00020.025 *1013“I don't know”, run 1013: 535 answered (97.8%), 40.6% wrong (217)
KnowLine-0.8B-Gen1 Open · 0.8B100137.9%188 of 496+32.1+27.6 to +36.50.00020.025 *1003“I don't know”, run 1003: 521 answered (95.2%), 41.7% wrong (217)
OneJev1.6-0.8B Open · 0.8B102537.9%188 of 496+32.1+27.8 to +36.50.00020.025 *1027“I don't know”, run 1027: 473 answered (86.5%), 39.5% wrong (187)
GLiClass v4 Edge Open · 0.4B104239.3%195 of 496+33.5+29.0 to +37.70.00020.025 *1044“I don't know”, run 1044: 495 answered (90.5%), 39.6% wrong (196)
Decider 1 Closed · Efficient59642.3%210 of 496+36.5+32.1 to +40.90.00020.025 *noneno run with the option
Strands Decider 2B Open · 2.0B70251.4%255 of 496+45.6+40.9 to +50.40.00020.025 *704“I don't know”, run 704: 545 answered (99.6%), 55.2% wrong (301)
Laya Open · 0.4B72676.8%381 of 496+71.0+66.5 to +75.40.00020.025 *728“I don't know”, run 728: 464 answered (84.8%), 79.1% wrong (367)
GLiNER2.5-multi-Decide Open · 0.3B71478.2%388 of 496+72.4+68.3 to +76.40.00020.025 *716“I don't know”, run 716: 477 answered (87.2%), 76.1% wrong (363)
GLiNER2.5 Multi Open · 0.3B72083.3%413 of 496+77.4+73.8 to +81.50.00020.025 *noneno run with the option
Laya Multilingual Open · 0.3B73285.1%422 of 496+79.2+75.2 to +83.10.00020.025 *734“I don't know”, run 734: 339 answered (62.0%), 82.6% wrong (280)
Gemma 4 E4B default reasoning · chat Open · 8.0B889Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
LFM2.5-2.6B default reasoning · chat Open · 2.7B1034Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active886Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
LFM3-1.7B reasoning off · chat Open · 1.7B1223Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
LFM3-1.7B reasoning on · chat Open · 1.7B1061Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
MiniCPM5-2B reasoning off · chat Open · 2.5B1094Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
Qwen3.5-2B reasoning off · chat Open · 2.0B1215Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
Qwen3.5-4B reasoning off · chat Open · 4.0B1143Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option
Qwen3.5-4B reasoning on · chat Open · 4.0B1072Not measureda plain answer line, no confidence—left out, not measured——noneno run with the option

Every setup's curve in Turkish

One small plot per setup, all 137, grouped and ordered as the Turkish table above, on the same axes: answered across (0 to 100% of the 2,198 questions), wrong of answers up (0 to 92%; anything higher is drawn at the top edge). The line is the setup's five-option run held back by its own confidence, from 1 answer to its scored answers; the dashed line is d1-ultra's (run 199) in every plot, and the dashed vertical line is k = 1,582. The markers are the setup's runs with the option ( “Bilmiyorum” “I don't know”). Each line is thinned to one vertex per 1% of coverage, always keeping the first, the last and the point at k; every vertex is a curve point.

Decision models

  • Copernicus-31B: five options, run 1165, from 1 to 2,198 answers; 95 wrong of its 1,582 most confidentCopernicus-31B: “Bilmiyorum”, run 1172: 2,193 answered, 402 wrong
    Copernicus-31B Open · 31B Wrong of the 1,582 answers: 95 (6.0%), -4.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1172, 18.3% wrong of 2,193
  • KnowLine-31B-Gen1.5: five options, run 957, from 1 to 2,198 answers; 105 wrong of its 1,582 most confidentKnowLine-31B-Gen1.5: “Bilmiyorum”, run 959: 2,197 answered, 411 wrong
    KnowLine-31B-Gen1.5 Open · 31B Wrong of the 1,582 answers: 105 (6.6%), -3.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 959, 18.7% wrong of 2,197
  • Turion Jev 31B: five options, run 1273, from 1 to 2,198 answers; 117 wrong of its 1,582 most confidentTurion Jev 31B: “Bilmiyorum”, run 1285: 2,183 answered, 436 wrong
    Turion Jev 31B Open · 31B Wrong of the 1,582 answers: 117 (7.4%), -3.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1285, 20.0% wrong of 2,183
  • KnowLine-31B-Gen2: five options, run 1314, from 1 to 2,198 answers; 122 wrong of its 1,582 most confidentKnowLine-31B-Gen2: “Bilmiyorum”, run 1331: 2,198 answered, 408 wrong
    KnowLine-31B-Gen2 Open · 31B Wrong of the 1,582 answers: 122 (7.7%), -2.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1331, 18.6% wrong of 2,198
  • Karpo Decision 31B: five options, run 1156, from 1 to 2,198 answers; 146 wrong of its 1,582 most confidentKarpo Decision 31B: “Bilmiyorum”, run 1163: 2,195 answered, 475 wrong
    Karpo Decision 31B Open · 31B Wrong of the 1,582 answers: 146 (9.2%), -1.3 points against d1-ultra, Holm p = 0.59
    Runs with the option: “Bilmiyorum” run 1163, 21.6% wrong of 2,195
  • REAX 1.3: five options, run 827, from 1 to 2,198 answers; 151 wrong of its 1,582 most confidentREAX 1.3: “Bilmiyorum”, run 833: 2,196 answered, 475 wrong
    REAX 1.3 Open · 57B Wrong of the 1,582 answers: 151 (9.5%), -1.0 points against d1-ultra, Holm p = 1
    Runs with the option: “Bilmiyorum” run 833, 21.6% wrong of 2,196
  • Gemini Decision 1 · 20261111: five options, run 1021, from 1 to 2,198 answers; 156 wrong of its 1,582 most confidentGemini Decision 1 · 20261111: “Bilmiyorum”, run 1023: 2,188 answered, 505 wrong
    Gemini Decision 1 20261111 Closed · Efficient Wrong of the 1,582 answers: 156 (9.9%), -0.7 points against d1-ultra, Holm p = 1
    Runs with the option: “Bilmiyorum” run 1023, 23.1% wrong of 2,188
  • d1-ultra: five options, run 199, from 1 to 2,198 answers; 167 wrong of its 1,582 most confidentd1-ultra: “Bilmiyorum”, run 350: 1,581 answered, 205 wrongd1-ultra: “I don't know”, run 511: 1,582 answered, 216 wrong
    d1-ultra Closed · Efficient Wrong of the 1,582 answers: 167 (10.6%), the reference, dashed in every plot
    Runs with the option: “Bilmiyorum” run 350, 13.0% wrong of 1,581 and “I don't know” run 511, 13.7% wrong of 1,582
  • Sieve 31B v3: five options, run 1241, from 1 to 2,198 answers; 183 wrong of its 1,582 most confidentSieve 31B v3: “Bilmiyorum”, run 1253: 2,198 answered, 553 wrong
    Sieve 31B v3 Open · 31B Wrong of the 1,582 answers: 183 (11.6%), +1.0 points against d1-ultra, Holm p = 1
    Runs with the option: “Bilmiyorum” run 1253, 25.2% wrong of 2,198
  • Logits Sigma-1 Duo: five options, run 1221, from 1 to 2,198 answers; 209 wrong of its 1,582 most confidentLogits Sigma-1 Duo: “Bilmiyorum”, run 1233: 2,196 answered, 543 wrong
    Logits Sigma-1 Duo Open · 57B Wrong of the 1,582 answers: 209 (13.2%), +2.7 points against d1-ultra, Holm p = 0.083
    Runs with the option: “Bilmiyorum” run 1233, 24.7% wrong of 2,196
  • GLiDE 27B: five options, run 899, from 1 to 2,198 answers; 259 wrong of its 1,582 most confidentGLiDE 27B: “Bilmiyorum”, run 905: 1,907 answered, 441 wrong
    GLiDE 27B Open · 27B Wrong of the 1,582 answers: 259 (16.4%), +5.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 905, 23.1% wrong of 1,907
  • GLiDE · v1.2: five options, run 248, from 1 to 2,198 answers; 289 wrong of its 1,582 most confidentGLiDE · v1.2: “Bilmiyorum”, run 369: 1,877 answered, 491 wrongGLiDE · v1.2: “I don't know”, run 525: 1,772 answered, 427 wrong
    GLiDE v1.2 Open · 27B Wrong of the 1,582 answers: 289 (18.3%), +7.7 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 369, 26.2% wrong of 1,877 and “I don't know” run 525, 24.1% wrong of 1,772
  • Perplexity Decider 27B v1.2 · API · with web: five options, run 640, from 1 to 2,198 answers; 313 wrong of its 1,582 most confidentPerplexity Decider 27B v1.2 · API · with web: “Bilmiyorum”, run 642: 2,175 answered, 702 wrong
    Perplexity Decider 27B v1.2 API · with web Open · 27B Wrong of the 1,582 answers: 313 (19.8%), +9.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 642, 32.3% wrong of 2,175
  • d3: five options, run 259, from 1 to 2,198 answers; 314 wrong of its 1,582 most confidentd3: “Bilmiyorum”, run 358: 2,198 answered, 715 wrongd3: “I don't know”, run 544: 2,196 answered, 714 wrong
    d3 Open · 26B Wrong of the 1,582 answers: 314 (19.8%), +9.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 358, 32.5% wrong of 2,198 and “I don't know” run 544, 32.5% wrong of 2,196
  • d3-ultra: five options, run 269, from 1 to 2,198 answers; 315 wrong of its 1,582 most confidentd3-ultra: “Bilmiyorum”, run 363: 2,188 answered, 704 wrongd3-ultra: “I don't know”, run 540: 2,189 answered, 704 wrong
    d3-ultra Open · 26B Wrong of the 1,582 answers: 315 (19.9%), +9.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 363, 32.2% wrong of 2,188 and “I don't know” run 540, 32.2% wrong of 2,189
  • Perplexity Decider 27B v1.1: five options, run 145, from 1 to 2,198 answers; 318 wrong of its 1,582 most confident
    Perplexity Decider 27B v1.1 Open · 27B Wrong of the 1,582 answers: 318 (20.1%), +9.5 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: Not run: Perplexity's API now answers requests for v1.1 with v1.2, so a new run could not measure v1.1.
  • Perplexity Decider 27B v1.3: five options, run 1134, from 1 to 2,198 answers; 320 wrong of its 1,582 most confidentPerplexity Decider 27B v1.3: “Bilmiyorum”, run 1141: 2,193 answered, 711 wrong
    Perplexity Decider 27B v1.3 Open · 27B Wrong of the 1,582 answers: 320 (20.2%), +9.7 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1141, 32.4% wrong of 2,193
  • Drex Max: five options, run 322, from 1 to 2,198 answers; 321 wrong of its 1,582 most confidentDrex Max: “Bilmiyorum”, run 367: 2,196 answered, 711 wrongDrex Max: “I don't know”, run 542: 2,196 answered, 712 wrong
    Drex Max Open · 27B Wrong of the 1,582 answers: 321 (20.3%), +9.7 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 367, 32.4% wrong of 2,196 and “I don't know” run 542, 32.4% wrong of 2,196
  • Drex Max v1.1: five options, run 326, from 1 to 2,198 answers; 321 wrong of its 1,582 most confidentDrex Max v1.1: “Bilmiyorum”, run 352: 1,878 answered, 517 wrongDrex Max v1.1: “I don't know”, run 513: 1,712 answered, 413 wrong
    Drex Max v1.1 Closed · Efficient (provisional) Wrong of the 1,582 answers: 321 (20.3%), +9.7 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 352, 27.5% wrong of 1,878 and “I don't know” run 513, 24.1% wrong of 1,712
  • Mammoth-1: five options, run 374, from 1 to 2,198 answers; 322 wrong of its 1,582 most confidentMammoth-1: “Bilmiyorum”, run 386: 1,508 answered, 314 wrongMammoth-1: “I don't know”, run 519: 1,445 answered, 276 wrong
    Mammoth-1 Open · 27B Wrong of the 1,582 answers: 322 (20.4%), +9.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 386, 20.8% wrong of 1,508 and “I don't know” run 519, 19.1% wrong of 1,445
  • d1.2-ultra: five options, run 1347, from 1 to 2,198 answers; 323 wrong of its 1,582 most confidentd1.2-ultra: “Bilmiyorum”, run 1367: 1,202 answered, 145 wrongd1.2-ultra: “I don't know”, run 1378: 1,180 answered, 137 wrong
    d1.2-ultra Closed · Efficient Wrong of the 1,582 answers: 323 (20.4%), +9.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1367, 12.1% wrong of 1,202 and “I don't know” run 1378, 11.6% wrong of 1,180
  • GLiDE · API: five options, run 129, from 1 to 2,198 answers; 325 wrong of its 1,582 most confidentGLiDE · API: “Bilmiyorum”, run 351: 1,677 answered, 356 wrong; made with Fastino's engine glide_v2, not run 129's glideGLiDE · API: “I don't know”, run 510: 1,617 answered, 335 wrong; made with Fastino's engine glide_v2, not run 129's glide
    GLiDE API Closed · Efficient Wrong of the 1,582 answers: 325 (20.5%), +10.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 351, 21.2% wrong of 1,677 and “I don't know” run 510, 20.7% wrong of 1,617 (made with Fastino's engine glide_v2, not run 129's glide)
  • Perplexity Decider 27B v1.2 · DGX Spark: five options, run 217, from 1 to 2,198 answers; 328 wrong of its 1,582 most confidentPerplexity Decider 27B v1.2 · DGX Spark: “Bilmiyorum”, run 372: 2,196 answered, 717 wrongPerplexity Decider 27B v1.2 · DGX Spark: “I don't know”, run 531: 2,194 answered, 728 wrong
    Perplexity Decider 27B v1.2 DGX Spark Open · 27B Wrong of the 1,582 answers: 328 (20.7%), +10.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 372, 32.7% wrong of 2,196 and “I don't know” run 531, 33.2% wrong of 2,194
  • Perplexity Decider 27B v1.2 · API: five options, run 633, from 1 to 2,198 answers; 329 wrong of its 1,582 most confidentPerplexity Decider 27B v1.2 · API: “Bilmiyorum”, run 641: 2,195 answered, 726 wrong
    Perplexity Decider 27B v1.2 API Open · 27B Wrong of the 1,582 answers: 329 (20.8%), +10.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 641, 33.1% wrong of 2,195
  • SubQ Snap Preview: five options, run 177, from 1 to 2,198 answers; 356 wrong of its 1,582 most confident
    SubQ Snap Preview Closed · Efficient Wrong of the 1,582 answers: 356 (22.5%), +11.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: Not run: SubQ's API returns its probabilities in 32-bit precision, so their sums miss our 7-decimal check. An allowance for that was approved on 5 November 2026 for its English runs; a Turkish run with “Bilmiyorum” was not part of that request.
  • SubQ Snap 1.1: five options, run 346, from 1 to 2,198 answers; 361 wrong of its 1,582 most confidentSubQ Snap 1.1: “Bilmiyorum”, run 392: 1,761 answered, 490 wrongSubQ Snap 1.1: “I don't know”, run 512: 1,700 answered, 454 wrong
    SubQ Snap 1.1 Closed · Efficient Wrong of the 1,582 answers: 361 (22.8%), +12.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 392, 27.8% wrong of 1,761 and “I don't know” run 512, 26.7% wrong of 1,700
  • GPT-6 Luna · Decisions API: five options, run 143, from 1 to 2,194 answers; 372 wrong of its 1,582 most confident
    GPT-6 Luna Decisions API Closed · Efficient Wrong of the 1,582 answers: 372 (23.5%), +13.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • KnowLine-12B-Gen1: five options, run 1309, from 1 to 2,198 answers; 418 wrong of its 1,582 most confidentKnowLine-12B-Gen1: “Bilmiyorum”, run 1318: 2,197 answered, 804 wrong
    KnowLine-12B-Gen1 Open · 12B Wrong of the 1,582 answers: 418 (26.4%), +15.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1318, 36.6% wrong of 2,197
  • Perplexity Decider 27B v1: five options, run 44, from 1 to 2,198 answers; 430 wrong of its 1,582 most confident
    Perplexity Decider 27B v1 Open · 27B Wrong of the 1,582 answers: 430 (27.2%), +16.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • d1.1: five options, run 254, from 1 to 2,198 answers; 434 wrong of its 1,582 most confident
    d1.1 Closed · Efficient Wrong of the 1,582 answers: 434 (27.4%), +16.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • d1.2: five options, run 1346, from 1 to 2,198 answers; 436 wrong of its 1,582 most confidentd1.2: “Bilmiyorum”, run 1365: 1,225 answered, 253 wrongd1.2: “I don't know”, run 1377: 1,297 answered, 279 wrong
    d1.2 Closed · Efficient Wrong of the 1,582 answers: 436 (27.6%), +17.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1365, 20.7% wrong of 1,225 and “I don't know” run 1377, 21.5% wrong of 1,297
  • celeris-1-decision: five options, run 150, from 1 to 2,198 answers; 450 wrong of its 1,582 most confident
    celeris-1-decision Closed · Efficient Wrong of the 1,582 answers: 450 (28.4%), +17.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • OneJev-27B: five options, run 264, from 1 to 2,198 answers; 455 wrong of its 1,582 most confidentOneJev-27B: “Bilmiyorum”, run 782: 2,197 answered, 892 wrong
    OneJev-27B Open · 27B Wrong of the 1,582 answers: 455 (28.8%), +18.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 782, 40.6% wrong of 2,197
  • Jev 1.14: five options, run 194, from 1 to 2,198 answers; 485 wrong of its 1,582 most confident
    Jev 1.14 Closed · Efficient Wrong of the 1,582 answers: 485 (30.7%), +20.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Mercury Decide: five options, run 204, from 1 to 2,198 answers; 487 wrong of its 1,582 most confident
    Mercury Decide Closed · Efficient Wrong of the 1,582 answers: 487 (30.8%), +20.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Jev 1.13: five options, run 49, from 1 to 2,198 answers; 498 wrong of its 1,582 most confident
    Jev 1.13 Closed · Efficient Wrong of the 1,582 answers: 498 (31.5%), +20.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Open-Jev-27B-v1.1: five options, run 45, from 1 to 2,198 answers; 504 wrong of its 1,582 most confidentOpen-Jev-27B-v1.1: “Bilmiyorum”, run 768: 2,184 answered, 965 wrong
    Open-Jev-27B-v1.1 Open · 27B Wrong of the 1,582 answers: 504 (31.9%), +21.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 768, 44.2% wrong of 2,184
  • Clef: five options, run 125, from 1 to 2,198 answers; 512 wrong of its 1,582 most confidentClef: “Bilmiyorum”, run 788: 2,127 answered, 894 wrong
    Clef Open · 27B Wrong of the 1,582 answers: 512 (32.4%), +21.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 788, 42.0% wrong of 2,127
  • d1: five options, run 52, from 1 to 2,198 answers; 516 wrong of its 1,582 most confident
    d1 Closed · Efficient Wrong of the 1,582 answers: 516 (32.6%), +22.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Jeffan-9B-1211: five options, run 1192, from 1 to 2,198 answers; 519 wrong of its 1,582 most confidentJeffan-9B-1211: “Bilmiyorum”, run 1199: 2,157 answered, 917 wrong
    Jeffan-9B-1211 Open · 9.0B Wrong of the 1,582 answers: 519 (32.8%), +22.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1199, 42.5% wrong of 2,157
  • Drex 2.0: five options, run 334, from 1 to 2,198 answers; 528 wrong of its 1,582 most confident
    Drex 2.0 Open · 35B MoE · 3.0B active Wrong of the 1,582 answers: 528 (33.4%), +22.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Cygnet: five options, run 127, from 1 to 2,198 answers; 530 wrong of its 1,582 most confidentCygnet: “Bilmiyorum”, run 757: 2,166 answered, 957 wrong
    Cygnet Open · 12B Wrong of the 1,582 answers: 530 (33.5%), +22.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 757, 44.2% wrong of 2,166
  • Winnow-12B: five options, run 128, from 1 to 2,198 answers; 540 wrong of its 1,582 most confidentWinnow-12B: “Bilmiyorum”, run 750: 2,155 answered, 965 wrong
    Winnow-12B Open · 12B Wrong of the 1,582 answers: 540 (34.1%), +23.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 750, 44.8% wrong of 2,155
  • KnowLine-9B-Gen1: five options, run 1174, from 1 to 2,198 answers; 553 wrong of its 1,582 most confidentKnowLine-9B-Gen1: “Bilmiyorum”, run 1181: 2,193 answered, 996 wrong
    KnowLine-9B-Gen1 Open · 9.0B Wrong of the 1,582 answers: 553 (35.0%), +24.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1181, 45.4% wrong of 2,193
  • Cerno 9B v1.2: five options, run 1201, from 1 to 2,198 answers; 559 wrong of its 1,582 most confidentCerno 9B v1.2: “Bilmiyorum”, run 1204: 2,196 answered, 998 wrong
    Cerno 9B v1.2 Open · 9.0B Wrong of the 1,582 answers: 559 (35.3%), +24.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1204, 45.4% wrong of 2,196
  • Adjudex 9B v5: five options, run 1183, from 1 to 2,198 answers; 578 wrong of its 1,582 most confidentAdjudex 9B v5: “Bilmiyorum”, run 1190: 2,190 answered, 1,021 wrong
    Adjudex 9B v5 Open · 9.0B Wrong of the 1,582 answers: 578 (36.5%), +26.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1190, 46.6% wrong of 2,190
  • KnowLine-9B-Gen2: five options, run 1306, from 1 to 2,198 answers; 580 wrong of its 1,582 most confidentKnowLine-9B-Gen2: “Bilmiyorum”, run 1325: 2,171 answered, 1,030 wrong
    KnowLine-9B-Gen2 Open · 9.0B Wrong of the 1,582 answers: 580 (36.7%), +26.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1325, 47.4% wrong of 2,171
  • Elephant-1: five options, run 373, from 1 to 2,198 answers; 586 wrong of its 1,582 most confidentElephant-1: “Bilmiyorum”, run 382: 1,793 answered, 758 wrongElephant-1: “I don't know”, run 515: 1,594 answered, 622 wrong
    Elephant-1 Open · 9.0B Wrong of the 1,582 answers: 586 (37.0%), +26.5 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 382, 42.3% wrong of 1,793 and “I don't know” run 515, 39.0% wrong of 1,594
  • GLiDE 9B: five options, run 907, from 1 to 2,198 answers; 600 wrong of its 1,582 most confidentGLiDE 9B: “Bilmiyorum”, run 913: 1,921 answered, 872 wrong
    GLiDE 9B Open · 7.9B Wrong of the 1,582 answers: 600 (37.9%), +27.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 913, 45.4% wrong of 1,921
  • StartLux-Decision-v1.1-9B-Instant: five options, run 1290, from 1 to 2,198 answers; 620 wrong of its 1,582 most confidentStartLux-Decision-v1.1-9B-Instant: “Bilmiyorum”, run 1297: 2,174 answered, 1,069 wrong
    StartLux-Decision-v1.1-9B-Instant Open · 9.0B Wrong of the 1,582 answers: 620 (39.2%), +28.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1297, 49.2% wrong of 2,174
  • Dexy-Decision-9B-V2: five options, run 1255, from 1 to 2,198 answers; 623 wrong of its 1,582 most confidentDexy-Decision-9B-V2: “Bilmiyorum”, run 1262: 2,161 answered, 1,046 wrong
    Dexy-Decision-9B-V2 Open · 9.0B Wrong of the 1,582 answers: 623 (39.4%), +28.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1262, 48.4% wrong of 2,161
  • vera-medium-omni: five options, run 1264, from 1 to 2,198 answers; 665 wrong of its 1,582 most confidentvera-medium-omni: “Bilmiyorum”, run 1271: 2,093 answered, 1,075 wrong
    vera-medium-omni Open · 4.2B Wrong of the 1,582 answers: 665 (42.0%), +31.5 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1271, 51.4% wrong of 2,093
  • Perplexity Decider Flash v1.3: five options, run 1123, from 1 to 2,198 answers; 667 wrong of its 1,582 most confidentPerplexity Decider Flash v1.3: “Bilmiyorum”, run 1130: 2,123 answered, 1,071 wrong
    Perplexity Decider Flash v1.3 Open · 4.0B Wrong of the 1,582 answers: 667 (42.2%), +31.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1130, 50.4% wrong of 2,123
  • StartLux-Decision-v1.1-4B-Instant: five options, run 1236, from 1 to 2,198 answers; 674 wrong of its 1,582 most confidentStartLux-Decision-v1.1-4B-Instant: “Bilmiyorum”, run 1239: 2,177 answered, 1,147 wrong
    StartLux-Decision-v1.1-4B-Instant Open · 4.0B Wrong of the 1,582 answers: 674 (42.6%), +32.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1239, 52.7% wrong of 2,177
  • KnowLine 4B Gen5.5: five options, run 965, from 1 to 2,198 answers; 681 wrong of its 1,582 most confidentKnowLine 4B Gen5.5: “Bilmiyorum”, run 967: 2,143 answered, 1,121 wrong
    KnowLine 4B Gen5.5 Open · 4.0B Wrong of the 1,582 answers: 681 (43.0%), +32.5 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 967, 52.3% wrong of 2,143
  • Perplexity Decider Flash v1.2: five options, run 223, from 1 to 2,198 answers; 691 wrong of its 1,582 most confidentPerplexity Decider Flash v1.2: “Bilmiyorum”, run 791: 2,018 answered, 1,044 wrong
    Perplexity Decider Flash v1.2 Open · 4.0B Wrong of the 1,582 answers: 691 (43.7%), +33.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 791, 51.7% wrong of 2,018
  • Valen-Decision-1.5-9B: five options, run 1275, from 1 to 2,198 answers; 698 wrong of its 1,582 most confidentValen-Decision-1.5-9B: “Bilmiyorum”, run 1282: 2,197 answered, 1,157 wrong
    Valen-Decision-1.5-9B Open · 9.0B Wrong of the 1,582 answers: 698 (44.1%), +33.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1282, 52.7% wrong of 2,197
  • Verdin-4B-v1.1: five options, run 1244, from 1 to 2,198 answers; 708 wrong of its 1,582 most confidentVerdin-4B-v1.1: “Bilmiyorum”, run 1251: 2,193 answered, 1,169 wrong
    Verdin-4B-v1.1 Open · 4.0B Wrong of the 1,582 answers: 708 (44.8%), +34.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1251, 53.3% wrong of 2,193
  • Clef-Flash: five options, run 139, from 1 to 2,198 answers; 716 wrong of its 1,582 most confidentClef-Flash: “Bilmiyorum”, run 664: 2,088 answered, 1,108 wrong
    Clef-Flash Open · 9.0B Wrong of the 1,582 answers: 716 (45.3%), +34.7 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 664, 53.1% wrong of 2,088
  • Chinese Decision Pointer 4B: five options, run 1206, from 1 to 2,198 answers; 723 wrong of its 1,582 most confidentChinese Decision Pointer 4B: “Bilmiyorum”, run 1213: 2,062 answered, 1,082 wrong
    Chinese Decision Pointer 4B Open · 4.0B Wrong of the 1,582 answers: 723 (45.7%), +35.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1213, 52.5% wrong of 2,062
  • Microsoft-Decision-1: five options, run 205, from 1 to 2,198 answers; 726 wrong of its 1,582 most confident
    Microsoft-Decision-1 Closed · Efficient Wrong of the 1,582 answers: 726 (45.9%), +35.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Metask-Jev-4B: five options, run 126, from 1 to 2,198 answers; 841 wrong of its 1,582 most confidentMetask-Jev-4B: “Bilmiyorum”, run 743: 2,188 answered, 1,315 wrong
    Metask-Jev-4B Open · 4.5B Wrong of the 1,582 answers: 841 (53.2%), +42.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 743, 60.1% wrong of 2,188
  • KnowLine-2B-Gen1: five options, run 1079, from 1 to 2,198 answers; 920 wrong of its 1,582 most confident
    KnowLine-2B-Gen1 Open · 2.0B Wrong of the 1,582 answers: 920 (58.2%), +47.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GLiDE 2B: five options, run 915, from 1 to 2,198 answers; 965 wrong of its 1,582 most confidentGLiDE 2B: “Bilmiyorum”, run 921: 1,586 answered, 978 wrong
    GLiDE 2B Open · 1.9B Wrong of the 1,582 answers: 965 (61.0%), +50.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 921, 61.7% wrong of 1,586
  • StartLux-Decision-v1.1-2B-Instant: five options, run 977, from 1 to 2,198 answers; 972 wrong of its 1,582 most confidentStartLux-Decision-v1.1-2B-Instant: “Bilmiyorum”, run 979: 1,865 answered, 1,213 wrong
    StartLux-Decision-v1.1-2B-Instant Open · 2.0B Wrong of the 1,582 answers: 972 (61.4%), +50.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 979, 65.0% wrong of 1,865
  • d1-2B: five options, run 340, from 1 to 2,198 answers; 1,060 wrong of its 1,582 most confidentd1-2B: “Bilmiyorum”, run 712: 989 answered, 637 wrong
    d1-2B Open · 1.9B Wrong of the 1,582 answers: 1,060 (67.0%), +56.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 712, 64.4% wrong of 989
  • OneJev1.6-0.8B: five options, run 1029, from 1 to 2,198 answers; 1,072 wrong of its 1,582 most confidentOneJev1.6-0.8B: “Bilmiyorum”, run 1031: 1,444 answered, 985 wrong
    OneJev1.6-0.8B Open · 0.8B Wrong of the 1,582 answers: 1,072 (67.8%), +57.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1031, 68.2% wrong of 1,444
  • GLiDE 0.8B: five options, run 923, from 1 to 2,198 answers; 1,080 wrong of its 1,582 most confidentGLiDE 0.8B: “Bilmiyorum”, run 929: 1,718 answered, 1,163 wrong
    GLiDE 0.8B Open · 0.8B Wrong of the 1,582 answers: 1,080 (68.3%), +57.7 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 929, 67.7% wrong of 1,718
  • Strands Decider 2B: five options, run 141, from 1 to 2,198 answers; 1,096 wrong of its 1,582 most confidentStrands Decider 2B: “Bilmiyorum”, run 706: 2,048 answered, 1,483 wrong
    Strands Decider 2B Open · 2.0B Wrong of the 1,582 answers: 1,096 (69.3%), +58.7 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 706, 72.4% wrong of 2,048
  • StartLux-Decision-v1.1-0.8B-Instant: five options, run 1015, from 1 to 2,198 answers; 1,098 wrong of its 1,582 most confidentStartLux-Decision-v1.1-0.8B-Instant: “Bilmiyorum”, run 1017: 1,999 answered, 1,409 wrong
    StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B Wrong of the 1,582 answers: 1,098 (69.4%), +58.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1017, 70.5% wrong of 1,999
  • KnowLine-0.8B-Gen1: five options, run 1005, from 1 to 2,198 answers; 1,107 wrong of its 1,582 most confidentKnowLine-0.8B-Gen1: “Bilmiyorum”, run 1007: 2,094 answered, 1,515 wrong
    KnowLine-0.8B-Gen1 Open · 0.8B Wrong of the 1,582 answers: 1,107 (70.0%), +59.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1007, 72.3% wrong of 2,094
  • Decider 1: five options, run 243, from 1 to 2,198 answers; 1,120 wrong of its 1,582 most confident
    Decider 1 Closed · Efficient Wrong of the 1,582 answers: 1,120 (70.8%), +60.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GLiClass v4 Edge: five options, run 1046, from 1 to 2,198 answers; 1,197 wrong of its 1,582 most confidentGLiClass v4 Edge: “Bilmiyorum”, run 1048: 1,800 answered, 1,386 wrong
    GLiClass v4 Edge Open · 0.4B Wrong of the 1,582 answers: 1,197 (75.7%), +65.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 1048, 77.0% wrong of 1,800
  • GLiDE 0.4B: five options, run 931, from 1 to 2,198 answers; 1,211 wrong of its 1,582 most confidentGLiDE 0.4B: “Bilmiyorum”, run 937: 128 answered, 101 wrong
    GLiDE 0.4B Open · 0.4B Wrong of the 1,582 answers: 1,211 (76.5%), +66.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 937, 78.9% wrong of 128
  • GLiNER2.5-multi-Decide: five options, run 135, from 1 to 2,198 answers; 1,248 wrong of its 1,582 most confidentGLiNER2.5-multi-Decide: “Bilmiyorum”, run 718: 2,084 answered, 1,666 wrong
    GLiNER2.5-multi-Decide Open · 0.3B Wrong of the 1,582 answers: 1,248 (78.9%), +68.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 718, 79.9% wrong of 2,084
  • Laya: five options, run 51, from 1 to 2,198 answers; 1,273 wrong of its 1,582 most confidentLaya: “Bilmiyorum”, run 730: 1,662 answered, 1,355 wrong
    Laya Open · 0.4B Wrong of the 1,582 answers: 1,273 (80.5%), +69.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 730, 81.5% wrong of 1,662
  • Laya Multilingual: five options, run 46, from 1 to 2,198 answers; 1,321 wrong of its 1,582 most confidentLaya Multilingual: “Bilmiyorum”, run 737: 1,918 answered, 1,608 wrong
    Laya Multilingual Open · 0.3B Wrong of the 1,582 answers: 1,321 (83.5%), +72.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: “Bilmiyorum” run 737, 83.8% wrong of 1,918

Other models: chat and general models

  • Gemini 4 Carbon · high · chat: five options, run 1108, from 1 to 2,198 answers; 4 wrong of its 1,582 most confident
    Gemini 4 Carbon high · chat Closed · Flagship Wrong of the 1,582 answers: 4 (0.3%), -10.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6 Astra · low · chat: five options, run 48, from 1 to 2,198 answers; 4 wrong of its 1,582 most confident
    GPT-6 Astra low · chat Closed · Flagship Wrong of the 1,582 answers: 4 (0.3%), -10.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Claude Fable 5.2 · max · chat: five options, run 980, from 1 to 2,198 answers; 5 wrong of its 1,582 most confident
    Claude Fable 5.2 max · chat Closed · Flagship Wrong of the 1,582 answers: 5 (0.3%), -10.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Gemini 4 Hydrogen · high · chat: five options, run 1114, from 1 to 2,198 answers; 5 wrong of its 1,582 most confident
    Gemini 4 Hydrogen high · chat Closed · Workhorse Wrong of the 1,582 answers: 5 (0.3%), -10.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6 Astra · high · chat: five options, run 277, from 1 to 2,198 answers; 6 wrong of its 1,582 most confident
    GPT-6 Astra high · chat Closed · Flagship Wrong of the 1,582 answers: 6 (0.4%), -10.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6 Astra · medium · chat: five options, run 610, from 1 to 2,198 answers; 6 wrong of its 1,582 most confident
    GPT-6 Astra medium · chat Closed · Flagship Wrong of the 1,582 answers: 6 (0.4%), -10.2 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Gemini 3.8 Flash · high · chat: five options, run 140, from 1 to 2,198 answers; 7 wrong of its 1,582 most confident
    Gemini 3.8 Flash high · chat Closed · Workhorse Wrong of the 1,582 answers: 7 (0.4%), -10.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6.1 Sol · high · chat: five options, run 276, from 1 to 2,198 answers; 8 wrong of its 1,582 most confident
    GPT-6.1 Sol high · chat Closed · Workhorse Wrong of the 1,582 answers: 8 (0.5%), -10.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6 Astra · low · chat · 1 per request: five options, run 191, from 1 to 2,198 answers; 11 wrong of its 1,582 most confident
    GPT-6 Astra low · chat · 1 per request Closed · Flagship Wrong of the 1,582 answers: 11 (0.7%), -9.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6.1 Sol · low · chat · 1 per request: five options, run 190, from 1 to 2,198 answers; 12 wrong of its 1,582 most confident
    GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse Wrong of the 1,582 answers: 12 (0.8%), -9.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Gemini 3.8 Flash · low · chat · 1 per request: five options, run 188, from 1 to 2,198 answers; 13 wrong of its 1,582 most confident
    Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse Wrong of the 1,582 answers: 13 (0.8%), -9.7 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6.1 Sol · low · chat: five options, run 124, from 1 to 2,198 answers; 13 wrong of its 1,582 most confident
    GPT-6.1 Sol low · chat Closed · Workhorse Wrong of the 1,582 answers: 13 (0.8%), -9.7 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Gemini 4 Argon · low · chat: five options, run 557, from 1 to 2,198 answers; 17 wrong of its 1,582 most confident
    Gemini 4 Argon low · chat Closed · Flagship Wrong of the 1,582 answers: 17 (1.1%), -9.5 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Claude Fable 5.1 · high · chat: five options, run 310, from 1 to 2,198 answers; 19 wrong of its 1,582 most confident
    Claude Fable 5.1 high · chat Closed · Flagship Wrong of the 1,582 answers: 19 (1.2%), -9.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Claude Opus 5.5 · high · chat: five options, run 278, from 1 to 2,198 answers; 23 wrong of its 1,582 most confident
    Claude Opus 5.5 high · chat Closed · Flagship Wrong of the 1,582 answers: 23 (1.5%), -9.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Claude Opus 5.5 · low · chat · 1 per request: five options, run 192, from 1 to 2,198 answers; 23 wrong of its 1,582 most confident
    Claude Opus 5.5 low · chat · 1 per request Closed · Flagship Wrong of the 1,582 answers: 23 (1.5%), -9.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6.1 Terra · high · chat: five options, run 275, from 1 to 2,198 answers; 31 wrong of its 1,582 most confident
    GPT-6.1 Terra high · chat Closed · Workhorse Wrong of the 1,582 answers: 31 (2.0%), -8.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Grok 4.7 · high · chat: five options, run 311, from 1 to 2,198 answers; 31 wrong of its 1,582 most confident
    Grok 4.7 high · chat Closed · Workhorse Wrong of the 1,582 answers: 31 (2.0%), -8.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Qwen4-Max · high · chat: five options, run 1336, from 1 to 2,166 answers; 31 wrong of its 1,582 most confident
    Qwen4-Max high · chat Closed · Workhorse Wrong of the 1,582 answers: 31 (2.0%), -8.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Gemini 3.8 Flash · low · chat: five options, run 123, from 1 to 2,198 answers; 50 wrong of its 1,582 most confident
    Gemini 3.8 Flash low · chat Closed · Workhorse Wrong of the 1,582 answers: 50 (3.2%), -7.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6.1 Terra · low · chat: five options, run 274, from 1 to 2,198 answers; 50 wrong of its 1,582 most confident
    GPT-6.1 Terra low · chat Closed · Workhorse Wrong of the 1,582 answers: 50 (3.2%), -7.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Claude Opus 5.5 · low · chat: five options, run 47, from 1 to 2,198 answers; 51 wrong of its 1,582 most confident
    Claude Opus 5.5 low · chat Closed · Flagship Wrong of the 1,582 answers: 51 (3.2%), -7.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Claude Sonnet 5.5 · high · chat · 1 per request: five options, run 193, from 1 to 2,198 answers; 65 wrong of its 1,582 most confident
    Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse Wrong of the 1,582 answers: 65 (4.1%), -6.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Muse Ember · max · chat: five options, run 939, from 1 to 2,150 answers; 86 wrong of its 1,582 most confident
    Muse Ember max · chat Closed · Efficient Wrong of the 1,582 answers: 86 (5.4%), -5.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • DeepSeek-V4.1-Flash · reasoning on · chat · 1017: five options, run 585, from 1 to 2,166 answers; 87 wrong of its 1,582 most confident
    DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active Wrong of the 1,582 answers: 87 (5.5%), -5.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Claude Sonnet 5.5 · high · chat: five options, run 55, from 1 to 2,198 answers; 113 wrong of its 1,582 most confident
    Claude Sonnet 5.5 high · chat Closed · Workhorse Wrong of the 1,582 answers: 113 (7.1%), -3.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • DeepSeek-V4.1-Flash · reasoning on · chat · first release: five options, run 121, from 1 to 2,198 answers; 113 wrong of its 1,582 most confident
    DeepSeek-V4.1-Flash reasoning on · chat · first release Open · 552B MoE · 16B active Wrong of the 1,582 answers: 113 (7.1%), -3.4 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Kimi K3.1 Pro · low · chat: five options, run 320, from 1 to 2,198 answers; 141 wrong of its 1,582 most confident
    Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active Wrong of the 1,582 answers: 141 (8.9%), -1.6 points against d1-ultra, Holm p = 0.41
    Runs with the option: No run with the option yet
  • Qwen3.5-397B-A17B · high · chat: five options, run 870, from 1 to 2,198 answers; 158 wrong of its 1,582 most confident
    Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active Wrong of the 1,582 answers: 158 (10.0%), -0.6 points against d1-ultra, Holm p = 1
    Runs with the option: No run with the option yet
  • GLM-5.3-Flash · reasoning on · chat: five options, run 160, from 1 to 2,070 answers; 163 wrong of its 1,582 most confident
    GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active Wrong of the 1,582 answers: 163 (10.3%), -0.3 points against d1-ultra, Holm p = 1
    Runs with the option: No run with the option yet
  • Inkling-Small · high · chat · Jul 2026: five options, run 866, from 1 to 2,198 answers; 181 wrong of its 1,582 most confident
    Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Wrong of the 1,582 answers: 181 (11.4%), +0.9 points against d1-ultra, Holm p = 1
    Runs with the option: No run with the option yet
  • Gemma 4 31B · reasoning off · chat: five options, run 119, from 1 to 2,198 answers; 239 wrong of its 1,582 most confident
    Gemma 4 31B reasoning off · chat Open · 31B Wrong of the 1,582 answers: 239 (15.1%), +4.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GLM-5.3 · low · chat: five options, run 314, from 1 to 2,198 answers; 260 wrong of its 1,582 most confident
    GLM-5.3 low · chat Open · Workhorse Wrong of the 1,582 answers: 260 (16.4%), +5.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Kimi K3.1 Pro · no reasoning · chat: five options, run 315, from 1 to 2,198 answers; 294 wrong of its 1,582 most confident
    Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active Wrong of the 1,582 answers: 294 (18.6%), +8.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • MiMo-V2.6-Flash · high · chat: five options, run 877, from 1 to 2,134 answers; 299 wrong of its 1,582 most confident
    MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active Wrong of the 1,582 answers: 299 (18.9%), +8.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Qwen3.8-27B · low · chat: five options, run 318, from 1 to 2,150 answers; 310 wrong of its 1,582 most confident
    Qwen3.8-27B low · chat Open · 27B Wrong of the 1,582 answers: 310 (19.6%), +9.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • DeepSeek-V4.1-Flash · reasoning off · chat · 1017: five options, run 558, from 1 to 2,198 answers; 345 wrong of its 1,582 most confident
    DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active Wrong of the 1,582 answers: 345 (21.8%), +11.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • DeepSeek-V4.1-Flash · reasoning off · chat · first release: five options, run 120, from 1 to 2,198 answers; 346 wrong of its 1,582 most confident
    DeepSeek-V4.1-Flash reasoning off · chat · first release Open · 552B MoE · 16B active Wrong of the 1,582 answers: 346 (21.9%), +11.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Gemma 4 31B · reasoning off · chat · 1 per request: five options, run 189, from 1 to 2,198 answers; 346 wrong of its 1,582 most confident
    Gemma 4 31B reasoning off · chat · 1 per request Open · 31B Wrong of the 1,582 answers: 346 (21.9%), +11.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GPT-6 Luna · low · chat: five options, run 133, from 1 to 2,198 answers; 388 wrong of its 1,582 most confident
    GPT-6 Luna low · chat Closed · Efficient Wrong of the 1,582 answers: 388 (24.5%), +14.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • MiniMax-M3.1-Flash · high · chat: five options, run 862, from 1 to 2,196 answers; 406 wrong of its 1,582 most confident
    MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active Wrong of the 1,582 answers: 406 (25.7%), +15.1 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Claude Sonnet 5.5 · low · chat: five options, run 54, from 1 to 2,198 answers; 440 wrong of its 1,582 most confident
    Claude Sonnet 5.5 low · chat Closed · Workhorse Wrong of the 1,582 answers: 440 (27.8%), +17.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Qwen3.8 Flash · no reasoning · chat: five options, run 985, from 1 to 2,198 answers; 462 wrong of its 1,582 most confident
    Qwen3.8 Flash no reasoning · chat Closed · Efficient Wrong of the 1,582 answers: 462 (29.2%), +18.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GLM-5.3-Flash · reasoning off · chat: five options, run 159, from 1 to 2,198 answers; 515 wrong of its 1,582 most confident
    GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active Wrong of the 1,582 answers: 515 (32.6%), +22.0 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Qwen3.8-27B · vote share · reasoning off · chat: five options, run 83, from 1 to 2,198 answers; 559 wrong of its 1,582 most confident
    Qwen3.8-27B vote share · reasoning off · chat Open · 27B Wrong of the 1,582 answers: 559 (35.3%), +24.8 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Qwen3.8-27B · yes/no scoring: five options, run 50, from 1 to 2,198 answers; 599 wrong of its 1,582 most confident
    Qwen3.8-27B yes/no scoring Open · 27B Wrong of the 1,582 answers: 599 (37.9%), +27.3 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Qwen3.8-27B · stated confidence · reasoning off · chat: five options, run 53, from 1 to 2,198 answers; 635 wrong of its 1,582 most confident
    Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B Wrong of the 1,582 answers: 635 (40.1%), +29.6 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Erk-14B · reasoning off · chat: five options, run 132, from 1 to 2,198 answers; 735 wrong of its 1,582 most confident
    Erk-14B reasoning off · chat Open · 14B Wrong of the 1,582 answers: 735 (46.5%), +35.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Qwen3-14B · reasoning off · chat: five options, run 131, from 1 to 2,198 answers; 745 wrong of its 1,582 most confident
    Qwen3-14B reasoning off · chat Open · 15B Wrong of the 1,582 answers: 745 (47.1%), +36.5 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • GLiNER2.5 Multi: five options, run 122, from 1 to 2,198 answers; 1,321 wrong of its 1,582 most confident
    GLiNER2.5 Multi Open · 0.3B Wrong of the 1,582 answers: 1,321 (83.5%), +72.9 points against d1-ultra, Holm p = 0.025 *
    Runs with the option: No run with the option yet
  • Claude Haiku 5.5 · max · chat: five options, run 948, from 1 to 1,414 answers; left out: fewer than k = 1,582 scored answers
    Claude Haiku 5.5 max · chat Closed · Efficient Wrong of the 1,582 answers: — (left out: its run has 1,414 scored answers, fewer than k), not tested
    Runs with the option: No run with the option yet
  • Gemma 4 E4B default reasoning · chat Open · 8.0B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
    Runs with the option: No run with the option yet
  • LFM2.5-2.6B default reasoning · chat Open · 2.7B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
    Runs with the option: No run with the option yet
  • LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
    Runs with the option: No run with the option yet
  • LFM3-1.7B reasoning off · chat Open · 1.7B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
    Runs with the option: No run with the option yet
  • LFM3-1.7B reasoning on · chat Open · 1.7B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
    Runs with the option: No run with the option yet
  • MiniCPM5-2B reasoning off · chat Open · 2.5B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
    Runs with the option: No run with the option yet
  • Qwen3.5-2B reasoning off · chat Open · 2.0B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
    Runs with the option: No run with the option yet
  • Qwen3.5-4B reasoning off · chat Open · 4.0B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
    Runs with the option: No run with the option yet
  • Qwen3.5-4B reasoning on · chat Open · 4.0B Wrong of the 1,582 answers: — (not measured: a plain answer line, no confidence), not tested
    Runs with the option: No run with the option yet

Our method for the same number of answers

Each model's five-option run answers only its k most confident questions, by its score (the probability it gave the option it picked), and holds back on the rest; we count its wrong answers among the k. Wrong of the k answers = wrong / k. k is the number of questions d1-ultra answered (right + wrong) with the option on offer: 1,582 in its run 511 with “I don't know” on the 2,198 Turkish questions (the option's word, above), and 496 in its run 449, its first English run with “I don't know”, on the 547 kept reading questions. Every model in a language is held to the same k.

The answers are ranked by score, highest first, and equal scores by question ID, lowest first (never by whether the answer is right); the first k are kept, so the wrong count is always a whole number. An answer without a score or a valid choice is never kept.

The difference is a model's wrong of the k answers minus d1-ultra's, in points. Its uncertainty comes from a paired bootstrap over questions: 10,000 resamples of the questions with replacement, from a new random generator with the numeric seed 20261103 for each language, the same resample for every model, a question drawn twice counting twice and k fixed. In each resample every run's drawn answers are ranked again by score and question ID, the copies of a drawn question sitting together, and the first k kept, so the count stays whole. Two-sided p = min(1, 2 × min(1 + #{d ≤ 0}, 1 + #{d ≥ 0}) / 10,001), where #{d ≤ 0} is the number of resamples whose difference d is 0 or less and #{d ≥ 0} the number whose difference is 0 or more. Its floor is 2/10,001 (p = 0.0002), when no resample falls on the other side of zero: a p at the floor says the difference held in every resample, not that p is exactly that value. The 95% interval is the percentile interval: from the 250th smallest to the 250th largest of the 10,000 resampled differences.

Holm's method adjusts p over the other 126 setups in Turkish (every main run of the Text tab, decision models and chat and general models alike) and the other 123 models in English, each language a family of its own, separate from the tab's other comparisons. The resamples do not depend on which models are compared, so a model's p is the same in any family; only Holm's adjustment grows with the family. A run with fewer than k scored answers keeps no k answers: it is left out of the bootstrap and of Holm's adjustment, and its line ends before k. A setup asked for a plain answer line states no confidence and reports no option probabilities, so its answers have no score to rank by: not measured, it is left out of the bootstrap and of Holm's adjustment and has no line. For GLiDE · API, the Turkish five-option run is the Text tab's run 129, made with Fastino's engine glide; its runs with the option, 351 and 510, answered with glide_v2, a different hosted model (Our method for the hosted GLiDE rerun), so its line and its markers come from two models, as the legend and the table say.

The plots show every number of answers, not only k. Each line is a five-option run under the same selection, one point per number of answers from 1 to the run's scored answers: answered / questions across, wrong / answered up. There is no point at 0 answers, where the share wrong is undefined, and a line ends at its run's scored answers, without being extended to 100%. The Turkish plot draws the 63 setups with runs with the option; the English curves come from this comparison too, so they can differ slightly from those of the Turkish vs English tab (Risk and coverage), where answers with equal scores enter together. A line skips vertices closer than 0.25% of coverage to the previous one drawn, always keeping the first, the last and the point at k, so every vertex drawn is a curve point; the y axis fits the markers and every curve point at 1% coverage or more, and anything higher (the first few answers, where one wrong answer is a large share) is drawn at the top edge. The markers are each model's own runs with the option: in Turkish its first run with “Bilmiyorum” (the table above) and its run with “I don't know” (the option's word); in English its first English run with “I don't know”.

Limits: k comes from one model, d1-ultra, and a different k gives different numbers (the plots show them all). The k answers kept are each model's own most confident ones, so the models are counted on different questions. d1-ultra's own runs with the option made 216 wrong of 1,582 in Turkish and 39 of 496 in English, against 167 and 29 for its five-option run held to the same k: the comparison is between five-option runs held back by confidence, not between runs with the option.