Math · I don't know
I don't know on Math
2,163 questions, asked in English. How to read this page · Method
Decision models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI Celeris Fastino Base Labs Subquadratic Nace.AI Other Perplexity StartLux EldanRing TypeSafe tattrongvu victorbennett vLLM Semantic Router Microsoft Zefan Cai Blockbrain Labs MachinePulse Google scar-ai quincyqiang PelaAI Inception Strands Agents newsbang Valen-Team REAX Metask Lab Turion.AI Sthānika AI Logits Cloudflare OmniJev Poindexter Labs Patrick-18 Apertix AI A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Liquid AI Celeris Fastino Base Labs Subquadratic Nace.AI Other Perplexity StartLux EldanRing TypeSafe tattrongvu victorbennett vLLM Semantic Router Microsoft Zefan Cai Blockbrain Labs MachinePulse Google scar-ai quincyqiang PelaAI Inception Strands Agents newsbang Valen-Team REAX Metask Lab Turion.AI Sthānika AI Logits Cloudflare OmniJev Poindexter Labs Patrick-18 Apertix AI Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient | — | 2,163 | 1,202 | 121 | 840 | 87.4% | 61.2% | 9.1% | 90.9% | 90.6% |
| d1.2-ultra Closed · Undisclosed | — | 2,163 | 1,126 | 138 | 899 | 86.7% | 58.4% | 10.9% | 89.1% | 88.8% |
| celeris-1.1-decision Closed · Efficient | — | 2,163 | 1,174 | 169 | 820 | 82.9% | 62.1% | 12.6% | 87.4% | 84.6% |
| celeris-1-decision Closed · Efficient | — | 2,163 | 1,164 | 173 | 826 | 82.7% | 61.8% | 12.9% | 87.1% | 84.7% |
| d1.1 Closed · Efficient | — | 2,163 | 1,007 | 202 | 954 | 82.5% | 55.9% | 16.7% | 83.3% | 84.5% |
| d1-2B Open · 1.9B | — | 2,163 | 375 | 354 | 1,434 | 80.2% | 33.7% | 48.6% | 51.4% | 80.8% |
| GLiDE 0.4B Open · 0.4B | — | 2,163 | 157 | 546 | 1,460 | 72.8% | 32.5% | 77.7% | 22.3% | 72.2% |
| Mammoth-1 Open · 27B | — | 2,163 | 1,412 | 221 | 530 | 70.6% | 75.5% | 13.5% | 86.5% | 71.7% |
| Elephant-1 Open · 9.0B | — | 2,163 | 1,069 | 383 | 711 | 65.0% | 67.1% | 26.4% | 73.6% | 66.3% |
| d1.2 Closed · Undisclosed | — | 2,163 | 1,255 | 397 | 511 | 56.3% | 76.4% | 24.0% | 76.0% | 56.3% |
| SubQ Snap 1.1 Closed · Efficient | — | 2,163 | 1,319 | 401 | 443 | 52.5% | 79.5% | 23.3% | 76.7% | 53.3% |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 2,163 | 1,388 | 438 | 337 | 43.5% | 84.4% | 24.0% | 76.0% | 45.4% |
| GLiDE API Closed · Efficient | — | 2,163 | 1,406 | 450 | 307 | 40.6% | 85.8% | 24.2% | 75.8% | 42.2% |
| GLiDE v1.2 Open · 27B | — | 2,163 | 1,428 | 472 | 263 | 35.8% | 87.8% | 24.8% | 75.2% | 35.8% |
| GLiDE 9B Open · 7.9B | — | 2,163 | 1,162 | 755 | 246 | 24.6% | 88.6% | 39.4% | 60.6% | 25.3% |
| GLiDE 27B Open · 27B | — | 2,163 | 1,516 | 507 | 140 | 21.6% | 93.5% | 25.1% | 74.9% | 21.7% |
| GLiDE 0.8B Open · 0.8B | — | 2,163 | 598 | 1,239 | 326 | 20.8% | 84.9% | 67.4% | 32.6% | 21.0% |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 2,163 | 1,199 | 764 | 200 | 20.7% | 90.8% | 38.9% | 61.1% | 21.3% |
| GLiDE 2B Open · 1.9B | — | 2,163 | 740 | 1,177 | 246 | 17.3% | 88.6% | 61.4% | 38.6% | 17.2% |
| GLiClass v4 Edge Open · 0.4B | — | 2,163 | 380 | 1,476 | 307 | 17.2% | 85.8% | 79.5% | 20.5% | 17.6% |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 2,163 | 1,492 | 568 | 103 | 15.4% | 95.2% | 27.6% | 72.4% | 15.0% |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 2,163 | 1,125 | 899 | 139 | 13.4% | 93.6% | 44.4% | 55.6% | 14.0% |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 2,163 | 1,136 | 904 | 123 | 12.0% | 94.3% | 44.3% | 55.7% | 12.3% |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 2,163 | 592 | 1,394 | 177 | 11.3% | 91.8% | 70.2% | 29.8% | 11.4% |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 2,163 | 1,527 | 576 | 60 | 9.4% | 97.2% | 27.4% | 72.6% | 9.8% |
| Winnow-12B Open · 12B | — | 2,163 | 956 | 1,096 | 111 | 9.2% | 94.9% | 53.4% | 46.6% | 9.5% |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 2,163 | 1,187 | 894 | 82 | 8.4% | 96.2% | 43.0% | 57.0% | 8.4% |
| Jev 1.14 Closed · Efficient | — | 2,163 | 1,386 | 717 | 60 | 7.7% | 97.2% | 34.1% | 65.9% | 7.1% |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 2,163 | 793 | 1,272 | 98 | 7.2% | 95.5% | 61.6% | 38.4% | 7.2% |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 2,163 | 1,490 | 626 | 47 | 7.0% | 97.8% | 29.6% | 70.4% | 7.1% |
| Jeffan-9B-1211 Open · 9.0B | — | 2,163 | 1,271 | 833 | 59 | 6.6% | 97.3% | 39.6% | 60.4% | 6.7% |
| Verdin-4B-v1.1 Open · 4.0B | — | 2,163 | 1,075 | 1,018 | 70 | 6.4% | 96.8% | 48.6% | 51.4% | 6.6% |
| Jev 1.13 Closed · Efficient | — | 2,163 | 1,370 | 748 | 45 | 5.7% | 97.9% | 35.3% | 64.7% | 5.0% |
| d3-ultra Open · 26B | — | 2,163 | 1,527 | 602 | 34 | 5.3% | 98.4% | 28.3% | 71.7% | 5.4% |
| Microsoft-Decision-1 Closed · Efficient | — | 2,163 | 1,254 | 863 | 46 | 5.1% | 97.9% | 40.8% | 59.2% | 5.2% |
| Open-Jev-27B-v1.1 Open · 27B | — | 2,163 | 1,154 | 959 | 50 | 5.0% | 97.7% | 45.4% | 54.6% | 5.0% |
| Cygnet Open · 12B | — | 2,163 | 942 | 1,161 | 60 | 4.9% | 97.2% | 55.2% | 44.8% | 4.9% |
| Karpo Decision 31B Open · 31B | — | 2,163 | 1,745 | 398 | 20 | 4.8% | 99.1% | 18.6% | 81.4% | 5.0% |
| d3 Open · 26B | — | 2,163 | 1,509 | 623 | 31 | 4.7% | 98.6% | 29.2% | 70.8% | 4.8% |
| Drex Max Open · 27B | — | 2,163 | 1,515 | 619 | 29 | 4.5% | 98.7% | 29.0% | 71.0% | 4.5% |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 2,163 | 1,239 | 885 | 39 | 4.2% | 98.2% | 41.7% | 58.3% | 4.3% |
| Gemini Decision 1 20261111 Closed · Efficient | — | 2,163 | 1,496 | 641 | 26 | 3.9% | 98.8% | 30.0% | 70.0% | 4.2% |
| vera-medium-omni Open · 4.2B | — | 2,163 | 1,222 | 905 | 36 | 3.8% | 98.3% | 42.5% | 57.5% | 3.8% |
| Chinese Decision Pointer 4B Open · 4.0B | — | 2,163 | 1,081 | 1,045 | 37 | 3.4% | 98.3% | 49.2% | 50.8% | 3.4% |
| KnowLine-9B-Gen2 Open · 9.0B | — | 2,163 | 1,289 | 846 | 28 | 3.2% | 98.7% | 39.6% | 60.4% | 3.2% |
| Mercury Decide Closed · Efficient | — | 2,163 | 1,156 | 975 | 32 | 3.2% | 98.5% | 45.8% | 54.2% | 3.2% |
| Strands Decider 2B Open · 2.0B | — | 2,163 | 586 | 1,527 | 50 | 3.2% | 97.7% | 72.3% | 27.7% | 3.2% |
| Adjudex 9B v5 Open · 9.0B | — | 2,163 | 1,321 | 816 | 26 | 3.1% | 98.8% | 38.2% | 61.8% | 3.1% |
| d1 Closed · Efficient | — | 2,163 | 1,161 | 973 | 29 | 2.9% | 98.7% | 45.6% | 54.4% | 3.0% |
| Valen-Decision-1.5-9B Open · 9.0B | — | 2,163 | 1,179 | 958 | 26 | 2.6% | 98.8% | 44.8% | 55.2% | 2.7% |
| REAX 1.3 Open · 57B | — | 2,163 | 1,688 | 464 | 11 | 2.3% | 99.5% | 21.6% | 78.4% | 2.3% |
| Metask-Jev-4B Open · 4.5B | — | 2,163 | 813 | 1,320 | 30 | 2.2% | 98.6% | 61.9% | 38.1% | 2.2% |
| Turion Jev 31B Open · 31B | — | 2,163 | 1,521 | 628 | 14 | 2.2% | 99.4% | 29.2% | 70.8% | 2.1% |
| Sieve 31B v3 Open · 31B | — | 2,163 | 1,688 | 465 | 10 | 2.1% | 99.5% | 21.6% | 78.4% | 2.0% |
| Logits Sigma-1 Duo Open · 57B | — | 2,163 | 1,627 | 525 | 11 | 2.1% | 99.5% | 24.4% | 75.6% | 2.0% |
| Clef-Flash Open · 9.0B | — | 2,163 | 1,053 | 1,088 | 22 | 2.0% | 99.0% | 50.8% | 49.2% | 2.0% |
| OneJev-27B Open · 27B | — | 2,163 | 1,160 | 985 | 18 | 1.8% | 99.2% | 45.9% | 54.1% | 1.8% |
| Copernicus-31B Open · 31B | — | 2,163 | 1,698 | 457 | 8 | 1.7% | 99.6% | 21.2% | 78.8% | 1.6% |
| KnowLine-2B-Gen1 Open · 2.0B | — | 2,163 | 744 | 1,395 | 24 | 1.7% | 98.9% | 65.2% | 34.8% | 1.7% |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 2,163 | 1,263 | 885 | 15 | 1.7% | 99.3% | 41.2% | 58.8% | 1.7% |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 2,163 | 1,190 | 957 | 16 | 1.6% | 99.3% | 44.6% | 55.4% | 1.7% |
| Clef Open · 27B | — | 2,163 | 1,114 | 1,032 | 17 | 1.6% | 99.2% | 48.1% | 51.9% | 1.6% |
| Cerno 9B v1.2 Open · 9.0B | — | 2,163 | 1,237 | 915 | 11 | 1.2% | 99.5% | 42.5% | 57.5% | 1.2% |
| KnowLine-12B-Gen1 Open · 12B | — | 2,163 | 1,431 | 724 | 8 | 1.1% | 99.6% | 33.6% | 66.4% | 1.1% |
| KnowLine-31B-Gen1.5 Open · 31B | — | 2,163 | 1,692 | 466 | 5 | 1.1% | 99.8% | 21.6% | 78.4% | 1.0% |
| KnowLine-9B-Gen1 Open · 9.0B | — | 2,163 | 1,301 | 854 | 8 | 0.9% | 99.6% | 39.6% | 60.4% | 0.9% |
| OneJev1.6-0.8B Open · 0.8B | — | 2,163 | 603 | 1,550 | 10 | 0.6% | 99.5% | 72.0% | 28.0% | 0.6% |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 2,163 | 612 | 1,546 | 5 | 0.3% | 99.8% | 71.6% | 28.4% | 0.3% |
| KnowLine-31B-Gen2 Open · 31B | — | 2,163 | 1,733 | 429 | 1 | 0.2% | 100.0% | 19.8% | 80.2% | 0.2% |
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Non-hallucination with “I don't know” 4.8% · answered 99.1% Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Non-hallucination with “I don't know” 0.2% · answered 100.0% Logic 58.9% | — | 80.3%78.5%–81.9% | ab | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Non-hallucination with “I don't know” 2.3% · answered 99.5% Logic 59.3% | — | 78.0%76.2%–79.7% | bc | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 1.1% · answered 99.8% Logic 59.0% | — | 77.4%75.6%–79.1% | cd | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Non-hallucination with “I don't know” 1.7% · answered 99.6% Logic 58.7% | — | 77.3%75.5%–79.1% | cd | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 2.1% · answered 99.5% Logic 57.9% | — | 77.1%75.3%–78.8% | cd | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Non-hallucination with “I don't know” 2.1% · answered 99.5% Logic 57.8% | — | 75.2%73.4%–77.0% | de | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Non-hallucination with “I don't know” 87.4% · answered 61.2% Logic 54.9% | — | 74.1%72.2%–75.9% | defg | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Non-hallucination with “I don't know” 70.6% · answered 75.5% Logic 54.5% | — | 74.0%72.1%–75.8% | defh | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 9.4% · answered 97.2% Logic 54.8% | — | 72.5%70.6%–74.3% | efghi | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Non-hallucination with “I don't know” 21.6% · answered 93.5% Logic 58.4% | — | 72.0%70.1%–73.8% | efghij | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Non-hallucination with “I don't know” 5.3% · answered 98.4% Logic 54.6% | — | 71.5%69.6%–73.4% | fghij | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 3.9% · answered 98.8% Logic 50.0% | — | 71.3%69.4%–73.2% | efghij | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 40.6% · answered 85.8% Logic 54.5% | — | 70.8%68.8%–72.7% | fghij | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Non-hallucination with “I don't know” 4.7% · answered 98.6% Logic 54.5% | — | 70.4%68.5%–72.3% | gij | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Non-hallucination with “I don't know” 4.5% · answered 98.7% Logic 53.1% | — | 70.3%68.4%–72.2% | gijk | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 43.5% · answered 84.4% Logic 54.8% | — | 70.1%68.1%–72.0% | ijk | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.7% · answered 61.8% Logic 47.3% | — | 70.0%68.0%–71.9% | hijk | 1,514 / 2,163 | No answers at this threshold | 0.858 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Non-hallucination with “I don't know” 35.8% · answered 87.8% Logic 55.2% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.9% · answered 62.1% Logic 47.3% | — | 69.9%67.9%–71.8% | ijkl | 1,511 / 2,163 | No answers at this threshold | 0.859 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 86.7% · answered 58.4% Logic 51.6% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 7.0% · answered 97.8% Logic 53.3% | — | 69.8%67.8%–71.7% | ijkl | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 15.4% · answered 95.2% Logic 52.5% | with web | 69.3%67.3%–71.2% | jklm | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 2.2% · answered 99.4% Logic 52.3% | — | 69.0%67.0%–70.9% | ijklmn | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Non-hallucination with “I don't know” 52.5% · answered 79.5% Logic 52.4% | — | 68.6%66.6%–70.5% | jklmn | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Non-hallucination with “I don't know” 1.1% · answered 99.6% Logic 50.6% | — | 66.3%64.2%–68.2% | klmno | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 56.3% · answered 76.4% Logic 49.2% | — | 65.5%63.5%–67.5% | lmnop | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.5% · answered 55.9% Logic 46.0% | — | 65.2%63.2%–67.2% | mnop | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 7.7% · answered 97.2% Logic 48.2% | — | 65.1%63.1%–67.1% | nop | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 5.7% · answered 97.9% Logic 46.3% | — | 64.1%62.0%–66.1% | opq | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 3.1% · answered 98.8% Logic 46.4% | — | 62.1%60.0%–64.1% | opqr | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Non-hallucination with “I don't know” 65.0% · answered 67.1% Logic 46.4% | — | 61.3%59.3%–63.4% | pqr | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 3.2% · answered 98.7% Logic 43.9% | — | 60.1%58.0%–62.1% | qrs | 1,300 / 2,163 | No answers at this threshold | 0.803 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 6.6% · answered 97.3% Logic 48.2% | — | 59.8%57.7%–61.9% | qrs | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 0.9% · answered 99.6% Logic 44.3% | — | 59.8%57.7%–61.8% | qrs | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Non-hallucination with “I don't know” 5.1% · answered 97.9% Logic 44.4% | — | 59.4%57.3%–61.4% | rst | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Non-hallucination with “I don't know” 20.7% · answered 90.8% Logic 41.9% | — | 59.0%56.9%–61.0% | rstu | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Non-hallucination with “I don't know” 1.7% · answered 99.3% Logic 45.3% | — | 58.8%56.7%–60.9% | rstu | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Non-hallucination with “I don't know” 4.2% · answered 98.2% Logic 46.6% | — | 58.4%56.3%–60.5% | rstuv | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Non-hallucination with “I don't know” 24.6% · answered 88.6% Logic 47.8% | — | 57.9%55.8%–60.0% | rstuvw | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 3.8% · answered 98.3% Logic 44.1% | — | 56.8%54.7%–58.9% | stuvw | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Non-hallucination with “I don't know” 1.2% · answered 99.5% Logic 43.3% | — | 56.0%53.9%–58.1% | stuvw | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Non-hallucination with “I don't know” 1.6% · answered 99.3% Logic 42.3% | — | 56.0%53.9%–58.1% | stuvw | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Non-hallucination with “I don't know” 8.4% · answered 96.2% Logic 44.7% | — | 55.9%53.8%–58.0% | stuvw | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Non-hallucination with “I don't know” 13.4% · answered 93.6% Logic 45.9% | — | 55.8%53.7%–57.9% | stuvwx | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 2.9% · answered 98.7% Logic 40.8% | — | 55.6%53.5%–57.7% | stuvwxy | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 2.6% · answered 98.8% Logic 44.4% | — | 55.3%53.2%–57.4% | tuvwxy | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Non-hallucination with “I don't know” 12.0% · answered 94.3% Logic 43.1% | — | 55.2%53.1%–57.3% | tuvwxy | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Non-hallucination with “I don't know” 5.0% · answered 97.7% Logic 42.6% | — | 54.6%52.4%–56.6% | uvwxyz | 1,180 / 2,163 | No answers at this threshold | 0.764 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Non-hallucination with “I don't know” 1.8% · answered 99.2% Logic 40.6% | — | 53.8%51.7%–55.9% | vwxyzA | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 3.2% · answered 98.5% Logic 45.1% | — | 53.5%51.4%–55.6% | wxyzA | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 6.4% · answered 96.8% Logic 39.7% | — | 51.5%49.4%–53.6% | yzAB | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Non-hallucination with “I don't know” 1.6% · answered 99.2% Logic 40.8% | — | 51.1%49.0%–53.2% | xyzAB | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 3.4% · answered 98.3% Logic 40.4% | — | 50.4%48.3%–52.5% | zAB | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Non-hallucination with “I don't know” 2.0% · answered 99.0% Logic 38.9% | — | 49.8%47.7%–51.9% | AB | 1,077 / 2,163 | No answers at this threshold | 0.730 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Non-hallucination with “I don't know” 9.2% · answered 94.9% Logic 39.7% | — | 46.9%44.8%–49.0% | BC | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Non-hallucination with “I don't know” 4.9% · answered 97.2% Logic 39.3% | — | 44.3%42.3%–46.4% | C | 959 / 2,163 | No answers at this threshold | 0.735 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Non-hallucination with “I don't know” 2.2% · answered 98.6% Logic 32.1% | — | 39.2%37.1%–41.2% | D | 847 / 2,163 | No answers at this threshold | 0.651 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Non-hallucination with “I don't know” 7.2% · answered 95.5% Logic 30.8% | — | 38.7%36.6%–40.7% | D | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Non-hallucination with “I don't know” 80.2% · answered 33.7% Logic 30.7% | — | 37.1%35.1%–39.1% | D | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Non-hallucination with “I don't know” 17.3% · answered 88.6% Logic 32.3% | — | 36.8%34.7%–38.8% | D | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Non-hallucination with “I don't know” 1.7% · answered 98.9% Logic 31.7% | — | 35.3%33.3%–37.3% | DE | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Non-hallucination with “I don't know” 20.8% · answered 84.9% Logic 29.2% | — | 32.1%30.2%–34.1% | EF | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Non-hallucination with “I don't know” 11.3% · answered 91.8% Logic 24.7% | — | 29.7%27.8%–31.6% | F | 642 / 2,163 | No answers at this threshold | 0.583 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Non-hallucination with “I don't know” 3.2% · answered 97.7% Logic 22.4% | — | 28.7%26.8%–30.6% | F | 620 / 2,163 | No answers at this threshold | 0.569 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Non-hallucination with “I don't know” 0.6% · answered 99.5% Logic 27.0% | — | 28.2%26.4%–30.2% | F | 611 / 2,163 | No answers at this threshold | 0.592 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Non-hallucination with “I don't know” 0.3% · answered 99.8% Logic 26.7% | — | 28.2%26.3%–30.1% | F | 610 / 2,163 | No answers at this threshold | 0.596 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Non-hallucination with “I don't know” 17.2% · answered 85.8% Logic 23.6% | — | 22.7%20.9%–24.5% | G | 490 / 2,163 | No answers at this threshold | 0.501 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Non-hallucination with “I don't know” 72.8% · answered 32.5% Logic 23.2% | — | 20.6%18.9%–22.3% | G | 445 / 2,163 | No answers at this threshold | 0.500 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Chat models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
xAI Google OpenAI Liquid AI Other Meta Anthropic Thinking Machines IBM Alibaba OpenBMB Xiaomi MiniMax eCloud Technology KRAFTON DeepSeek Z.ai A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Grok 4.7 chat Closed · Undisclosed | — | 1,987 | 1,947 | 1 | 39 | 97.5% | 98.0% | 0.1% | 99.9% | Not measured |
| Gemini 4 Hydrogen chat | — | 2,163 | 2,142 | 3 | 18 | 85.7% | 99.2% | 0.1% | 99.9% | Not measured |
| GPT-6 Astra chat Closed · Undisclosed | — | 2,163 | 2,149 | 3 | 11 | 78.6% | 99.5% | 0.1% | 99.9% | Not measured |
| GPT-6.1 Sol chat Closed · Undisclosed | — | 2,163 | 2,149 | 3 | 11 | 78.6% | 99.5% | 0.1% | 99.9% | Not measured |
| GPT-6.1 Terra chat Closed · Undisclosed | — | 2,163 | 2,136 | 6 | 21 | 77.8% | 99.0% | 0.3% | 99.7% | Not measured |
| Gemini 4 Carbon chat Closed · Undisclosed | — | 2,163 | 2,147 | 4 | 12 | 75.0% | 99.4% | 0.2% | 99.8% | Not measured |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | 2,163 | 2,128 | 10 | 25 | 71.4% | 98.8% | 0.5% | 99.5% | Not measured |
| GPT-6 Luna chat Closed · Efficient | — | 1,075 | 1,001 | 23 | 51 | 68.9% | 95.3% | 2.2% | 97.8% | Not measured |
| LFM3-1.7B off · chat Open · 1.7B | — | 2,163 | 2,132 | 10 | 21 | 67.7% | 99.0% | 0.5% | 99.5% | Not measured |
| LFM3-1.7B on · chat Open · 1.7B | — | 2,163 | 2,129 | 11 | 23 | 67.6% | 98.9% | 0.5% | 99.5% | Not measured |
| Muse Ember max · chat Closed · Efficient | — | 2,163 | 2,149 | 5 | 9 | 64.3% | 99.6% | 0.2% | 99.8% | Not measured |
| Claude Opus 5.5 chat Closed · Flagship | — | 2,163 | 2,155 | 3 | 5 | 62.5% | 99.8% | 0.1% | 99.9% | Not measured |
| Claude Sonnet 5.5 chat Closed · Workhorse | — | 2,163 | 2,155 | 3 | 5 | 62.5% | 99.8% | 0.1% | 99.9% | Not measured |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | 2,163 | 2,150 | 5 | 8 | 61.5% | 99.6% | 0.2% | 99.8% | Not measured |
| Granite-4.2-3B high · chat Open · 3.0B | — | 2,163 | 2,137 | 10 | 16 | 61.5% | 99.3% | 0.5% | 99.5% | Not measured |
| Claude Fable 5.1 chat Closed · Flagship | — | 2,163 | 2,153 | 4 | 6 | 60.0% | 99.7% | 0.2% | 99.8% | Not measured |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | 2,163 | 2,152 | 5 | 6 | 54.5% | 99.7% | 0.2% | 99.8% | Not measured |
| Qwen4-Max high · chat Closed · Undisclosed | — | 2,163 | 2,152 | 5 | 6 | 54.5% | 99.7% | 0.2% | 99.8% | Not measured |
| MiniCPM5-2B on · chat Open · 2.5B | — | 2,163 | 2,135 | 13 | 15 | 53.6% | 99.3% | 0.6% | 99.4% | Not measured |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | 2,163 | 2,149 | 7 | 7 | 50.0% | 99.7% | 0.3% | 99.7% | Not measured |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | 2,163 | 2,119 | 24 | 20 | 45.5% | 99.1% | 1.1% | 98.9% | Not measured |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 2,163 | 2,039 | 68 | 56 | 45.2% | 97.4% | 3.2% | 96.8% | Not measured |
| Gemma 4 E4B chat Open · 8.0B | — | 2,163 | 2,113 | 28 | 22 | 44.0% | 99.0% | 1.3% | 98.7% | Not measured |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | 2,163 | 2,131 | 19 | 13 | 40.6% | 99.4% | 0.9% | 99.1% | Not measured |
| Qwen3-14B off · chat Open · 15B | — | 2,163 | 2,046 | 71 | 46 | 39.3% | 97.9% | 3.4% | 96.6% | Not measured |
| Gemma 4 E4B on · chat Open · 8.0B | — | 2,163 | 2,117 | 29 | 17 | 37.0% | 99.2% | 1.4% | 98.6% | Not measured |
| Qwen3.5-4B off · chat Open · 4.7B | — | 2,163 | 2,138 | 16 | 9 | 36.0% | 99.6% | 0.7% | 99.3% | Not measured |
| Erk-14B off · chat Open · 15B | — | 2,163 | 1,901 | 168 | 94 | 35.9% | 95.7% | 8.1% | 91.9% | Not measured |
| Qwen3.5-2B off · chat Open · 2.0B | — | 2,163 | 2,032 | 90 | 41 | 31.3% | 98.1% | 4.2% | 95.8% | Not measured |
| MiniCPM5-2B off · chat Open · 2.5B | — | 2,163 | 2,076 | 60 | 27 | 31.0% | 98.8% | 2.8% | 97.2% | Not measured |
| LFM2.5-2.6B on · chat Open · 2.7B | — | 2,163 | 2,072 | 67 | 24 | 26.4% | 98.9% | 3.1% | 96.9% | Not measured |
| Gemma 4 E2B chat Open · 5.1B | — | 2,163 | 2,088 | 56 | 19 | 25.3% | 99.1% | 2.6% | 97.4% | Not measured |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | 2,163 | 2,102 | 47 | 14 | 23.0% | 99.4% | 2.2% | 97.8% | Not measured |
| Raon-1 nano off · chat Open · 3.1B | — | 2,163 | 1,959 | 159 | 45 | 22.1% | 97.9% | 7.5% | 92.5% | Not measured |
| Gemma 4 31B chat Open · 31B | — | 2,163 | 2,149 | 11 | 3 | 21.4% | 99.9% | 0.5% | 99.5% | Not measured |
| Qwen3.5-4B on · chat Open · 4.7B | — | 2,163 | 2,149 | 11 | 3 | 21.4% | 99.9% | 0.5% | 99.5% | Not measured |
| Qwen3.5-2B on · chat Open · 2.0B | — | 2,163 | 2,099 | 51 | 13 | 20.3% | 99.4% | 2.4% | 97.6% | Not measured |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | 2,163 | 2,153 | 8 | 2 | 20.0% | 99.9% | 0.4% | 99.6% | Not measured |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | 2,163 | 2,140 | 19 | 4 | 17.4% | 99.8% | 0.9% | 99.1% | Not measured |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | 2,163 | 2,145 | 15 | 3 | 16.7% | 99.9% | 0.7% | 99.3% | Not measured |
| Gemma 4 E2B on · chat Open · 5.1B | — | 2,163 | 2,084 | 66 | 13 | 16.5% | 99.4% | 3.1% | 96.9% | Not measured |
| Gemma 4 12B Unified chat Open · 12B | — | 2,163 | 2,142 | 18 | 3 | 14.3% | 99.9% | 0.8% | 99.2% | Not measured |
| Qwen3.8-27B low · chat Open · 27B | — | 2,163 | 2,140 | 20 | 3 | 13.0% | 99.9% | 0.9% | 99.1% | Not measured |
The F option not measured for 5 setups: DeepSeek-V4-Flash (chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E2B (on · chat · 32k cap), Raon-1 nano (on · chat).
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC); Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)); Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC); Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test); GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 62.5% · answered 99.8% Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 54.5% · answered 99.7% Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 60.0% · answered 99.7% Logic 95.2% | — | 99.7%99.4%–99.9% | abc | 2,157 / 2,163 | Not measured | Not measured | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 62.5% · answered 99.8% Logic 95.5% | — | 99.7%99.3%–99.8% | abd | 2,156 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Non-hallucination with “I don't know” 20.0% · answered 99.9% Logic 96.2% | — | 99.6%99.3%–99.8% | abcdef | 2,155 / 2,163 | Not measured | Not measured | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 78.6% · answered 99.5% Logic 96.7% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 78.6% · answered 99.5% Logic 96.5% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Non-hallucination with “I don't know” 54.5% · answered 99.7% Logic 95.4% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 75.0% · answered 99.4% Logic 96.7% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Non-hallucination with “I don't know” 85.7% · answered 99.2% Logic 96.1% | — | 99.6%99.2%–99.8% | abcdef | 2,154 / 2,163 | Not measured | Not measured | 85.7%answered 99% | 4,400 ms | 6,680 ms | — |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 64.3% · answered 99.6% Logic 95.5% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 97.5% · answered 98.0% Logic running | — | 99.5%99.2%–99.7% | abcdefg | 2,153 / 2,163 | Not measured | Not measured | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Non-hallucination with “I don't know” 61.5% · answered 99.6% Logic 95.6% | — | 99.5%99.2%–99.7% | abcdef | 2,153 / 2,163 | Not measured | Not measured | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 16.7% · answered 99.9% Logic 94.0% | — | 99.5%99.1%–99.7% | abcdefgh | 2,152 / 2,163 | Not measured | Not measured | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Non-hallucination with “I don't know” 50.0% · answered 99.7% Logic 95.1% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Non-hallucination with “I don't know” 21.4% · answered 99.9% Logic 92.4% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Non-hallucination with “I don't know” 21.4% · answered 99.9% Logic 93.3% | — | 99.4%99.0%–99.6% | abcdefgh | 2,150 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Logic running | — | 99.3%98.9%–99.6% | abcdefghij | 2,148 / 2,163 | Not measured | Not measured | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Non-hallucination with “I don't know” 53.6% · answered 99.3% Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | abcdefghi | 2,147 / 2,163seed 0 run | Not measured | Not measured | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 77.8% · answered 99.0% Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Non-hallucination with “I don't know” 61.5% · answered 99.3% Logic 91.6% | — | 99.2%98.7%–99.5% | abcdefghi | 2,146 / 2,163 | Not measured | Not measured | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Non-hallucination with “I don't know” 13.0% · answered 99.9% Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Non-hallucination with “I don't know” 71.4% · answered 98.8% Logic 86.7% | — | 99.0%98.5%–99.4% | bcdefghij | 2,142 / 2,163 | Not measured | Not measured | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abcdefghij | 2,142 / 2,163seed 0 run | Not measured | Not measured | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Non-hallucination with “I don't know” 67.6% · answered 98.9% Logic 84.2% | — | 98.9%98.8%–99.1% min–max of 3 seeds | cefghijk | 2,139 / 2,163seed 0 run | Not measured | Not measured | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Non-hallucination with “I don't know” 14.3% · answered 99.9% Logic 87.7% | — | 98.8%98.3%–99.2% | dfghijkl | 2,138 / 2,163 | Not measured | Not measured | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Non-hallucination with “I don't know” 17.4% · answered 99.8% Logic 93.2% | — | 98.8%98.2%–99.2% | fghijkl | 2,137 / 2,163 | Not measured | Not measured | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Non-hallucination with “I don't know” 67.7% · answered 99.0% Logic 85.5% | — | 98.7%98.1%–99.1% | ghijkl | 2,135 / 2,163 | Not measured | Not measured | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 45.5% · answered 99.1% Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | efghijkl | 2,136 / 2,163seed 0 run | Not measured | Not measured | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% | — | 98.5%97.9%–99.0% | hijkl | 2,131 / 2,163 | Not measured | Not measured | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Non-hallucination with “I don't know” 36.0% · answered 99.6% Logic 90.1% | — | 98.5%97.9%–98.9% | hijklm | 2,130 / 2,163 | Not measured | Not measured | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 37.0% · answered 99.2% Logic 82.6% | — | 98.2%97.6%–98.7% | ijklm | 2,125 / 2,163 | Not measured | Not measured | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Non-hallucination with “I don't know” 40.6% · answered 99.4% Logic 94.0% | — | 98.2%97.6%–98.7% | ijklmn | 2,125 / 2,163 | Not measured | Not measured | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Non-hallucination with “I don't know” 31.0% · answered 98.8% Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | ijklmn | 2,126 / 2,163seed 0 run | Not measured | Not measured | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 44.0% · answered 99.0% Logic 83.2% | — | 98.1%97.4%–98.6% | jklmn | 2,122 / 2,163 | Not measured | Not measured | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% | — | 97.8%97.1%–98.4% | klmno | 2,116 / 2,163 | Not measured | Not measured | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 23.0% · answered 99.4% Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | lmnop | 2,111 / 2,163seed 0 run | Not measured | Not measured | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% | — | 97.0%96.2%–97.6% | mnop | 2,098 / 2,163 | Not measured | Not measured | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 16.5% · answered 99.4% Logic 74.6% | — | 96.9%96.1%–97.6% | mnop | 2,097 / 2,163 | Not measured | Not measured | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 25.3% · answered 99.1% Logic 72.9% | — | 96.8%95.9%–97.4% | nopq | 2,093 / 2,163 | Not measured | Not measured | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Non-hallucination with “I don't know” 45.2% · answered 97.4% Logic 60.2% | — | 96.3%95.4%–97.0% | opqr | 2,082 / 2,163 | Not measured | Not measured | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Non-hallucination with “I don't know” 39.3% · answered 97.9% Logic 65.3% | — | 95.8%94.9%–96.6% | pqrs | 2,072 / 2,163 | Not measured | Not measured | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Non-hallucination with “I don't know” 20.3% · answered 99.4% Logic 70.3% | — | 94.7%93.7%–95.6% | qrs | 2,049 / 2,163 | Not measured | Not measured | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Non-hallucination with “I don't know” 31.3% · answered 98.1% Logic 64.8% | — | 94.6%93.6%–95.5% | rs | 2,047 / 2,163 | Not measured | Not measured | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Non-hallucination with “I don't know” 26.4% · answered 98.9% Logic 75.4% | — | 94.0%92.9%–94.9% | rs | 2,033 / 2,163 | Not measured | Not measured | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 68.9% · answered 95.3% Logic 58.0% | — | 93.7%92.6%–94.6% | s | 2,026 / 2,163 | Not measured | Not measured | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Non-hallucination with “I don't know” 35.9% · answered 95.7% Logic 54.2% | — | 90.8%89.5%–91.9% | t | 1,963 / 2,163 | Not measured | Not measured | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Non-hallucination with “I don't know” 22.1% · answered 97.9% Logic 65.1% | — | 89.6%88.2%–90.8% | t | 1,937 / 2,163 | Not measured | Not measured | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI Fastino Other StartLux Perplexity victorbennett scar-ai quincyqiang Strands Agents Metask Lab PelaAI OmniJev A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Liquid AI Fastino Other StartLux Perplexity victorbennett scar-ai quincyqiang Strands Agents Metask Lab PelaAI OmniJev Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-2B Open · 1.9B | — | 2,163 | 375 | 354 | 1,434 | 80.2% | 33.7% | 48.6% | 51.4% | 80.8% |
| GLiDE 0.4B Open · 0.4B | — | 2,163 | 157 | 546 | 1,460 | 72.8% | 32.5% | 77.7% | 22.3% | 72.2% |
| GLiDE 0.8B Open · 0.8B | — | 2,163 | 598 | 1,239 | 326 | 20.8% | 84.9% | 67.4% | 32.6% | 21.0% |
| GLiDE 2B Open · 1.9B | — | 2,163 | 740 | 1,177 | 246 | 17.3% | 88.6% | 61.4% | 38.6% | 17.2% |
| GLiClass v4 Edge Open · 0.4B | — | 2,163 | 380 | 1,476 | 307 | 17.2% | 85.8% | 79.5% | 20.5% | 17.6% |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 2,163 | 1,125 | 899 | 139 | 13.4% | 93.6% | 44.4% | 55.6% | 14.0% |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 2,163 | 1,136 | 904 | 123 | 12.0% | 94.3% | 44.3% | 55.7% | 12.3% |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 2,163 | 592 | 1,394 | 177 | 11.3% | 91.8% | 70.2% | 29.8% | 11.4% |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 2,163 | 1,187 | 894 | 82 | 8.4% | 96.2% | 43.0% | 57.0% | 8.4% |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 2,163 | 793 | 1,272 | 98 | 7.2% | 95.5% | 61.6% | 38.4% | 7.2% |
| Verdin-4B-v1.1 Open · 4.0B | — | 2,163 | 1,075 | 1,018 | 70 | 6.4% | 96.8% | 48.6% | 51.4% | 6.6% |
| vera-medium-omni Open · 4.2B | — | 2,163 | 1,222 | 905 | 36 | 3.8% | 98.3% | 42.5% | 57.5% | 3.8% |
| Chinese Decision Pointer 4B Open · 4.0B | — | 2,163 | 1,081 | 1,045 | 37 | 3.4% | 98.3% | 49.2% | 50.8% | 3.4% |
| Strands Decider 2B Open · 2.0B | — | 2,163 | 586 | 1,527 | 50 | 3.2% | 97.7% | 72.3% | 27.7% | 3.2% |
| Metask-Jev-4B Open · 4.5B | — | 2,163 | 813 | 1,320 | 30 | 2.2% | 98.6% | 61.9% | 38.1% | 2.2% |
| KnowLine-2B-Gen1 Open · 2.0B | — | 2,163 | 744 | 1,395 | 24 | 1.7% | 98.9% | 65.2% | 34.8% | 1.7% |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 2,163 | 1,190 | 957 | 16 | 1.6% | 99.3% | 44.6% | 55.4% | 1.7% |
| OneJev1.6-0.8B Open · 0.8B | — | 2,163 | 603 | 1,550 | 10 | 0.6% | 99.5% | 72.0% | 28.0% | 0.6% |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 2,163 | 612 | 1,546 | 5 | 0.3% | 99.8% | 71.6% | 28.4% | 0.3% |
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 3.8% · answered 98.3% Logic 44.1% | — | 56.8%54.7%–58.9% | a | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Non-hallucination with “I don't know” 1.6% · answered 99.3% Logic 42.3% | — | 56.0%53.9%–58.1% | a | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Non-hallucination with “I don't know” 8.4% · answered 96.2% Logic 44.7% | — | 55.9%53.8%–58.0% | a | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Non-hallucination with “I don't know” 13.4% · answered 93.6% Logic 45.9% | — | 55.8%53.7%–57.9% | a | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Non-hallucination with “I don't know” 12.0% · answered 94.3% Logic 43.1% | — | 55.2%53.1%–57.3% | a | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 6.4% · answered 96.8% Logic 39.7% | — | 51.5%49.4%–53.6% | b | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 3.4% · answered 98.3% Logic 40.4% | — | 50.4%48.3%–52.5% | b | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Non-hallucination with “I don't know” 2.2% · answered 98.6% Logic 32.1% | — | 39.2%37.1%–41.2% | c | 847 / 2,163 | No answers at this threshold | 0.651 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Non-hallucination with “I don't know” 7.2% · answered 95.5% Logic 30.8% | — | 38.7%36.6%–40.7% | c | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Non-hallucination with “I don't know” 80.2% · answered 33.7% Logic 30.7% | — | 37.1%35.1%–39.1% | c | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Non-hallucination with “I don't know” 17.3% · answered 88.6% Logic 32.3% | — | 36.8%34.7%–38.8% | c | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Non-hallucination with “I don't know” 1.7% · answered 98.9% Logic 31.7% | — | 35.3%33.3%–37.3% | cd | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Non-hallucination with “I don't know” 20.8% · answered 84.9% Logic 29.2% | — | 32.1%30.2%–34.1% | de | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Non-hallucination with “I don't know” 11.3% · answered 91.8% Logic 24.7% | — | 29.7%27.8%–31.6% | ef | 642 / 2,163 | No answers at this threshold | 0.583 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Non-hallucination with “I don't know” 3.2% · answered 97.7% Logic 22.4% | — | 28.7%26.8%–30.6% | ef | 620 / 2,163 | No answers at this threshold | 0.569 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Non-hallucination with “I don't know” 0.6% · answered 99.5% Logic 27.0% | — | 28.2%26.4%–30.2% | f | 611 / 2,163 | No answers at this threshold | 0.592 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Non-hallucination with “I don't know” 0.3% · answered 99.8% Logic 26.7% | — | 28.2%26.3%–30.1% | f | 610 / 2,163 | No answers at this threshold | 0.596 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Non-hallucination with “I don't know” 17.2% · answered 85.8% Logic 23.6% | — | 22.7%20.9%–24.5% | g | 490 / 2,163 | No answers at this threshold | 0.501 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Non-hallucination with “I don't know” 72.8% · answered 32.5% Logic 23.2% | — | 20.6%18.9%–22.3% | g | 445 / 2,163 | No answers at this threshold | 0.500 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Chat models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Other IBM OpenBMB Alibaba Liquid AI Google KRAFTON A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| LFM3-1.7B off · chat Open · 1.7B | — | 2,163 | 2,132 | 10 | 21 | 67.7% | 99.0% | 0.5% | 99.5% | Not measured |
| LFM3-1.7B on · chat Open · 1.7B | — | 2,163 | 2,129 | 11 | 23 | 67.6% | 98.9% | 0.5% | 99.5% | Not measured |
| Granite-4.2-3B high · chat Open · 3.0B | — | 2,163 | 2,137 | 10 | 16 | 61.5% | 99.3% | 0.5% | 99.5% | Not measured |
| MiniCPM5-2B on · chat Open · 2.5B | — | 2,163 | 2,135 | 13 | 15 | 53.6% | 99.3% | 0.6% | 99.4% | Not measured |
| Qwen3.5-4B off · chat Open · 4.7B | — | 2,163 | 2,138 | 16 | 9 | 36.0% | 99.6% | 0.7% | 99.3% | Not measured |
| Qwen3.5-2B off · chat Open · 2.0B | — | 2,163 | 2,032 | 90 | 41 | 31.3% | 98.1% | 4.2% | 95.8% | Not measured |
| MiniCPM5-2B off · chat Open · 2.5B | — | 2,163 | 2,076 | 60 | 27 | 31.0% | 98.8% | 2.8% | 97.2% | Not measured |
| LFM2.5-2.6B on · chat Open · 2.7B | — | 2,163 | 2,072 | 67 | 24 | 26.4% | 98.9% | 3.1% | 96.9% | Not measured |
| Gemma 4 E2B chat Open · 5.1B | — | 2,163 | 2,088 | 56 | 19 | 25.3% | 99.1% | 2.6% | 97.4% | Not measured |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | 2,163 | 2,102 | 47 | 14 | 23.0% | 99.4% | 2.2% | 97.8% | Not measured |
| Raon-1 nano off · chat Open · 3.1B | — | 2,163 | 1,959 | 159 | 45 | 22.1% | 97.9% | 7.5% | 92.5% | Not measured |
| Qwen3.5-4B on · chat Open · 4.7B | — | 2,163 | 2,149 | 11 | 3 | 21.4% | 99.9% | 0.5% | 99.5% | Not measured |
| Qwen3.5-2B on · chat Open · 2.0B | — | 2,163 | 2,099 | 51 | 13 | 20.3% | 99.4% | 2.4% | 97.6% | Not measured |
| Gemma 4 E2B on · chat Open · 5.1B | — | 2,163 | 2,084 | 66 | 13 | 16.5% | 99.4% | 3.1% | 96.9% | Not measured |
The F option not measured for 3 setups: LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), Gemma 4 E2B (on · chat · 32k cap), Raon-1 nano (on · chat).
Not ranked (only complete runs are): Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Non-hallucination with “I don't know” 21.4% · answered 99.9% Logic 92.4% | — | 99.4%99.0%–99.7% | a | 2,151 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Non-hallucination with “I don't know” 53.6% · answered 99.3% Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | ab | 2,147 / 2,163seed 0 run | Not measured | Not measured | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Non-hallucination with “I don't know” 61.5% · answered 99.3% Logic 91.6% | — | 99.2%98.7%–99.5% | ab | 2,146 / 2,163 | Not measured | Not measured | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abc | 2,142 / 2,163seed 0 run | Not measured | Not measured | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Non-hallucination with “I don't know” 67.6% · answered 98.9% Logic 84.2% | — | 98.9%98.8%–99.1% min–max of 3 seeds | abc | 2,139 / 2,163seed 0 run | Not measured | Not measured | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Non-hallucination with “I don't know” 67.7% · answered 99.0% Logic 85.5% | — | 98.7%98.1%–99.1% | bcd | 2,135 / 2,163 | Not measured | Not measured | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Non-hallucination with “I don't know” 36.0% · answered 99.6% Logic 90.1% | — | 98.5%97.9%–98.9% | bcde | 2,130 / 2,163 | Not measured | Not measured | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Non-hallucination with “I don't know” 31.0% · answered 98.8% Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | cdef | 2,126 / 2,163seed 0 run | Not measured | Not measured | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% | — | 97.8%97.1%–98.4% | defg | 2,116 / 2,163 | Not measured | Not measured | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 23.0% · answered 99.4% Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | efg | 2,111 / 2,163seed 0 run | Not measured | Not measured | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% | — | 97.0%96.2%–97.6% | fg | 2,098 / 2,163 | Not measured | Not measured | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 16.5% · answered 99.4% Logic 74.6% | — | 96.9%96.1%–97.6% | g | 2,097 / 2,163 | Not measured | Not measured | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 25.3% · answered 99.1% Logic 72.9% | — | 96.8%95.9%–97.4% | g | 2,093 / 2,163 | Not measured | Not measured | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Non-hallucination with “I don't know” 20.3% · answered 99.4% Logic 70.3% | — | 94.7%93.7%–95.6% | h | 2,049 / 2,163 | Not measured | Not measured | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Non-hallucination with “I don't know” 31.3% · answered 98.1% Logic 64.8% | — | 94.6%93.6%–95.5% | h | 2,047 / 2,163 | Not measured | Not measured | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Non-hallucination with “I don't know” 26.4% · answered 98.9% Logic 75.4% | — | 94.0%92.9%–94.9% | h | 2,033 / 2,163 | Not measured | Not measured | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Non-hallucination with “I don't know” 22.1% · answered 97.9% Logic 65.1% | — | 89.6%88.2%–90.8% | i | 1,937 / 2,163 | Not measured | Not measured | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Decision models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Base Labs Fastino EldanRing tattrongvu Blockbrain Labs StartLux PelaAI newsbang Valen-Team Cloudflare Patrick-18 Apertix AI A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Base Labs Fastino EldanRing tattrongvu Blockbrain Labs StartLux PelaAI newsbang Valen-Team Cloudflare Patrick-18 Apertix AI Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Elephant-1 Open · 9.0B | — | 2,163 | 1,069 | 383 | 711 | 65.0% | 67.1% | 26.4% | 73.6% | 66.3% |
| GLiDE 9B Open · 7.9B | — | 2,163 | 1,162 | 755 | 246 | 24.6% | 88.6% | 39.4% | 60.6% | 25.3% |
| Winnow-12B Open · 12B | — | 2,163 | 956 | 1,096 | 111 | 9.2% | 94.9% | 53.4% | 46.6% | 9.5% |
| Jeffan-9B-1211 Open · 9.0B | — | 2,163 | 1,271 | 833 | 59 | 6.6% | 97.3% | 39.6% | 60.4% | 6.7% |
| Cygnet Open · 12B | — | 2,163 | 942 | 1,161 | 60 | 4.9% | 97.2% | 55.2% | 44.8% | 4.9% |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 2,163 | 1,239 | 885 | 39 | 4.2% | 98.2% | 41.7% | 58.3% | 4.3% |
| KnowLine-9B-Gen2 Open · 9.0B | — | 2,163 | 1,289 | 846 | 28 | 3.2% | 98.7% | 39.6% | 60.4% | 3.2% |
| Adjudex 9B v5 Open · 9.0B | — | 2,163 | 1,321 | 816 | 26 | 3.1% | 98.8% | 38.2% | 61.8% | 3.1% |
| Valen-Decision-1.5-9B Open · 9.0B | — | 2,163 | 1,179 | 958 | 26 | 2.6% | 98.8% | 44.8% | 55.2% | 2.7% |
| Clef-Flash Open · 9.0B | — | 2,163 | 1,053 | 1,088 | 22 | 2.0% | 99.0% | 50.8% | 49.2% | 2.0% |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 2,163 | 1,263 | 885 | 15 | 1.7% | 99.3% | 41.2% | 58.8% | 1.7% |
| Cerno 9B v1.2 Open · 9.0B | — | 2,163 | 1,237 | 915 | 11 | 1.2% | 99.5% | 42.5% | 57.5% | 1.2% |
| KnowLine-12B-Gen1 Open · 12B | — | 2,163 | 1,431 | 724 | 8 | 1.1% | 99.6% | 33.6% | 66.4% | 1.1% |
| KnowLine-9B-Gen1 Open · 9.0B | — | 2,163 | 1,301 | 854 | 8 | 0.9% | 99.6% | 39.6% | 60.4% | 0.9% |
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Non-hallucination with “I don't know” 1.1% · answered 99.6% Logic 50.6% | — | 66.3%64.2%–68.2% | a | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 3.1% · answered 98.8% Logic 46.4% | — | 62.1%60.0%–64.1% | b | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Non-hallucination with “I don't know” 65.0% · answered 67.1% Logic 46.4% | — | 61.3%59.3%–63.4% | bc | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 3.2% · answered 98.7% Logic 43.9% | — | 60.1%58.0%–62.1% | bcd | 1,300 / 2,163 | No answers at this threshold | 0.803 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 6.6% · answered 97.3% Logic 48.2% | — | 59.8%57.7%–61.9% | bcd | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 0.9% · answered 99.6% Logic 44.3% | — | 59.8%57.7%–61.8% | bcd | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Non-hallucination with “I don't know” 1.7% · answered 99.3% Logic 45.3% | — | 58.8%56.7%–60.9% | cde | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Non-hallucination with “I don't know” 4.2% · answered 98.2% Logic 46.6% | — | 58.4%56.3%–60.5% | cde | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Non-hallucination with “I don't know” 24.6% · answered 88.6% Logic 47.8% | — | 57.9%55.8%–60.0% | de | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Non-hallucination with “I don't know” 1.2% · answered 99.5% Logic 43.3% | — | 56.0%53.9%–58.1% | e | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 2.6% · answered 98.8% Logic 44.4% | — | 55.3%53.2%–57.4% | e | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Non-hallucination with “I don't know” 2.0% · answered 99.0% Logic 38.9% | — | 49.8%47.7%–51.9% | f | 1,077 / 2,163 | No answers at this threshold | 0.730 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Non-hallucination with “I don't know” 9.2% · answered 94.9% Logic 39.7% | — | 46.9%44.8%–49.0% | fg | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Non-hallucination with “I don't know” 4.9% · answered 97.2% Logic 39.3% | — | 44.3%42.3%–46.4% | g | 959 / 2,163 | No answers at this threshold | 0.735 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Chat models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Google Liquid AI Alibaba eCloud Technology A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | 2,163 | 2,119 | 24 | 20 | 45.5% | 99.1% | 1.1% | 98.9% | Not measured |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 2,163 | 2,039 | 68 | 56 | 45.2% | 97.4% | 3.2% | 96.8% | Not measured |
| Gemma 4 E4B chat Open · 8.0B | — | 2,163 | 2,113 | 28 | 22 | 44.0% | 99.0% | 1.3% | 98.7% | Not measured |
| Qwen3-14B off · chat Open · 15B | — | 2,163 | 2,046 | 71 | 46 | 39.3% | 97.9% | 3.4% | 96.6% | Not measured |
| Gemma 4 E4B on · chat Open · 8.0B | — | 2,163 | 2,117 | 29 | 17 | 37.0% | 99.2% | 1.4% | 98.6% | Not measured |
| Erk-14B off · chat Open · 15B | — | 2,163 | 1,901 | 168 | 94 | 35.9% | 95.7% | 8.1% | 91.9% | Not measured |
| Gemma 4 12B Unified chat Open · 12B | — | 2,163 | 2,142 | 18 | 3 | 14.3% | 99.9% | 0.8% | 99.2% | Not measured |
The F option not measured for 1 setup: Gemma 4 E4B (on · chat · 32k cap).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Non-hallucination with “I don't know” 14.3% · answered 99.9% Logic 87.7% | — | 98.8%98.3%–99.2% | a | 2,138 / 2,163 | Not measured | Not measured | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 45.5% · answered 99.1% Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | a | 2,136 / 2,163seed 0 run | Not measured | Not measured | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% | — | 98.5%97.9%–99.0% | a | 2,131 / 2,163 | Not measured | Not measured | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 37.0% · answered 99.2% Logic 82.6% | — | 98.2%97.6%–98.7% | a | 2,125 / 2,163 | Not measured | Not measured | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 44.0% · answered 99.0% Logic 83.2% | — | 98.1%97.4%–98.6% | a | 2,122 / 2,163 | Not measured | Not measured | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Non-hallucination with “I don't know” 45.2% · answered 97.4% Logic 60.2% | — | 96.3%95.4%–97.0% | b | 2,082 / 2,163 | Not measured | Not measured | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Non-hallucination with “I don't know” 39.3% · answered 97.9% Logic 65.3% | — | 95.8%94.9%–96.6% | b | 2,072 / 2,163 | Not measured | Not measured | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Non-hallucination with “I don't know” 35.9% · answered 95.7% Logic 54.2% | — | 90.8%89.5%–91.9% | c | 1,963 / 2,163 | Not measured | Not measured | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Decision models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Base Labs Fastino Nace.AI Perplexity vLLM Semantic Router Zefan Cai MachinePulse Turion.AI Sthānika AI OmniJev Poindexter Labs Cloudflare PelaAI A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Base Labs Fastino Nace.AI Perplexity vLLM Semantic Router Zefan Cai MachinePulse Turion.AI Sthānika AI OmniJev Poindexter Labs Cloudflare PelaAI Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Mammoth-1 Open · 27B | — | 2,163 | 1,412 | 221 | 530 | 70.6% | 75.5% | 13.5% | 86.5% | 71.7% |
| GLiDE v1.2 Open · 27B | — | 2,163 | 1,428 | 472 | 263 | 35.8% | 87.8% | 24.8% | 75.2% | 35.8% |
| GLiDE 27B Open · 27B | — | 2,163 | 1,516 | 507 | 140 | 21.6% | 93.5% | 25.1% | 74.9% | 21.7% |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 2,163 | 1,199 | 764 | 200 | 20.7% | 90.8% | 38.9% | 61.1% | 21.3% |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 2,163 | 1,492 | 568 | 103 | 15.4% | 95.2% | 27.6% | 72.4% | 15.0% |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 2,163 | 1,527 | 576 | 60 | 9.4% | 97.2% | 27.4% | 72.6% | 9.8% |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 2,163 | 1,490 | 626 | 47 | 7.0% | 97.8% | 29.6% | 70.4% | 7.1% |
| d3-ultra Open · 26B | — | 2,163 | 1,527 | 602 | 34 | 5.3% | 98.4% | 28.3% | 71.7% | 5.4% |
| Open-Jev-27B-v1.1 Open · 27B | — | 2,163 | 1,154 | 959 | 50 | 5.0% | 97.7% | 45.4% | 54.6% | 5.0% |
| Karpo Decision 31B Open · 31B | — | 2,163 | 1,745 | 398 | 20 | 4.8% | 99.1% | 18.6% | 81.4% | 5.0% |
| d3 Open · 26B | — | 2,163 | 1,509 | 623 | 31 | 4.7% | 98.6% | 29.2% | 70.8% | 4.8% |
| Drex Max Open · 27B | — | 2,163 | 1,515 | 619 | 29 | 4.5% | 98.7% | 29.0% | 71.0% | 4.5% |
| Turion Jev 31B Open · 31B | — | 2,163 | 1,521 | 628 | 14 | 2.2% | 99.4% | 29.2% | 70.8% | 2.1% |
| Sieve 31B v3 Open · 31B | — | 2,163 | 1,688 | 465 | 10 | 2.1% | 99.5% | 21.6% | 78.4% | 2.0% |
| OneJev-27B Open · 27B | — | 2,163 | 1,160 | 985 | 18 | 1.8% | 99.2% | 45.9% | 54.1% | 1.8% |
| Copernicus-31B Open · 31B | — | 2,163 | 1,698 | 457 | 8 | 1.7% | 99.6% | 21.2% | 78.8% | 1.6% |
| Clef Open · 27B | — | 2,163 | 1,114 | 1,032 | 17 | 1.6% | 99.2% | 48.1% | 51.9% | 1.6% |
| KnowLine-31B-Gen1.5 Open · 31B | — | 2,163 | 1,692 | 466 | 5 | 1.1% | 99.8% | 21.6% | 78.4% | 1.0% |
| KnowLine-31B-Gen2 Open · 31B | — | 2,163 | 1,733 | 429 | 1 | 0.2% | 100.0% | 19.8% | 80.2% | 0.2% |
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Non-hallucination with “I don't know” 4.8% · answered 99.1% Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Non-hallucination with “I don't know” 0.2% · answered 100.0% Logic 58.9% | — | 80.3%78.5%–81.9% | a | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 1.1% · answered 99.8% Logic 59.0% | — | 77.4%75.6%–79.1% | b | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Non-hallucination with “I don't know” 1.7% · answered 99.6% Logic 58.7% | — | 77.3%75.5%–79.1% | b | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 2.1% · answered 99.5% Logic 57.9% | — | 77.1%75.3%–78.8% | bc | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Non-hallucination with “I don't know” 70.6% · answered 75.5% Logic 54.5% | — | 74.0%72.1%–75.8% | cd | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 9.4% · answered 97.2% Logic 54.8% | — | 72.5%70.6%–74.3% | de | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Non-hallucination with “I don't know” 21.6% · answered 93.5% Logic 58.4% | — | 72.0%70.1%–73.8% | def | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Non-hallucination with “I don't know” 5.3% · answered 98.4% Logic 54.6% | — | 71.5%69.6%–73.4% | def | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Non-hallucination with “I don't know” 4.7% · answered 98.6% Logic 54.5% | — | 70.4%68.5%–72.3% | ef | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Non-hallucination with “I don't know” 4.5% · answered 98.7% Logic 53.1% | — | 70.3%68.4%–72.2% | ef | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Non-hallucination with “I don't know” 35.8% · answered 87.8% Logic 55.2% | — | 69.9%67.9%–71.8% | ef | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 7.0% · answered 97.8% Logic 53.3% | — | 69.8%67.8%–71.7% | f | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 15.4% · answered 95.2% Logic 52.5% | with web | 69.3%67.3%–71.2% | f | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 2.2% · answered 99.4% Logic 52.3% | — | 69.0%67.0%–70.9% | f | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Non-hallucination with “I don't know” 20.7% · answered 90.8% Logic 41.9% | — | 59.0%56.9%–61.0% | g | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Non-hallucination with “I don't know” 5.0% · answered 97.7% Logic 42.6% | — | 54.6%52.4%–56.6% | h | 1,180 / 2,163 | No answers at this threshold | 0.764 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Non-hallucination with “I don't know” 1.8% · answered 99.2% Logic 40.6% | — | 53.8%51.7%–55.9% | h | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Non-hallucination with “I don't know” 1.6% · answered 99.2% Logic 40.8% | — | 51.1%49.0%–53.2% | h | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Chat models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI Google Alibaba A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | 2,163 | 2,128 | 10 | 25 | 71.4% | 98.8% | 0.5% | 99.5% | Not measured |
| Gemma 4 31B chat Open · 31B | — | 2,163 | 2,149 | 11 | 3 | 21.4% | 99.9% | 0.5% | 99.5% | Not measured |
| Qwen3.8-27B low · chat Open · 27B | — | 2,163 | 2,140 | 20 | 3 | 13.0% | 99.9% | 0.9% | 99.1% | Not measured |
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Non-hallucination with “I don't know” 21.4% · answered 99.9% Logic 93.3% | — | 99.4%99.0%–99.6% | a | 2,150 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Non-hallucination with “I don't know” 13.0% · answered 99.9% Logic 94.0% | — | 99.2%98.7%–99.5% | a | 2,146 / 2,163 | Not measured | Not measured | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Non-hallucination with “I don't know” 71.4% · answered 98.8% Logic 86.7% | — | 99.0%98.5%–99.4% | a | 2,142 / 2,163 | Not measured | Not measured | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Decision models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
REAX Logits A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
REAX Logits Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 2,163 | 1,688 | 464 | 11 | 2.3% | 99.5% | 21.6% | 78.4% | 2.3% |
| Logits Sigma-1 Duo Open · 57B | — | 2,163 | 1,627 | 525 | 11 | 2.1% | 99.5% | 24.4% | 75.6% | 2.0% |
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Non-hallucination with “I don't know” 2.3% · answered 99.5% Logic 59.3% | — | 78.0%76.2%–79.7% | a | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Non-hallucination with “I don't know” 2.1% · answered 99.5% Logic 57.8% | — | 75.2%73.4%–77.0% | b | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Chat models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Thinking Machines Xiaomi MiniMax DeepSeek Z.ai A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | 2,163 | 2,150 | 5 | 8 | 61.5% | 99.6% | 0.2% | 99.8% | Not measured |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | 2,163 | 2,152 | 5 | 6 | 54.5% | 99.7% | 0.2% | 99.8% | Not measured |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | 2,163 | 2,149 | 7 | 7 | 50.0% | 99.7% | 0.3% | 99.7% | Not measured |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | 2,163 | 2,131 | 19 | 13 | 40.6% | 99.4% | 0.9% | 99.1% | Not measured |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | 2,163 | 2,153 | 8 | 2 | 20.0% | 99.9% | 0.4% | 99.6% | Not measured |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | 2,163 | 2,140 | 19 | 4 | 17.4% | 99.8% | 0.9% | 99.1% | Not measured |
The F option not measured for 1 setup: DeepSeek-V4-Flash (chat).
Not ranked (only complete runs are): Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Non-hallucination with “I don't know” 20.0% · answered 99.9% Logic 96.2% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Non-hallucination with “I don't know” 54.5% · answered 99.7% Logic 95.4% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Non-hallucination with “I don't know” 61.5% · answered 99.6% Logic 95.6% | — | 99.5%99.2%–99.7% | a | 2,153 / 2,163 | Not measured | Not measured | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Non-hallucination with “I don't know” 50.0% · answered 99.7% Logic 95.1% | — | 99.4%99.0%–99.7% | ab | 2,151 / 2,163 | Not measured | Not measured | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Logic running | — | 99.3%98.9%–99.6% | ab | 2,148 / 2,163 | Not measured | Not measured | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Non-hallucination with “I don't know” 17.4% · answered 99.8% Logic 93.2% | — | 98.8%98.2%–99.2% | bc | 2,137 / 2,163 | Not measured | Not measured | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Non-hallucination with “I don't know” 40.6% · answered 99.4% Logic 94.0% | — | 98.2%97.6%–98.7% | c | 2,125 / 2,163 | Not measured | Not measured | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Decision models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI Celeris Subquadratic Nace.AI Fastino TypeSafe Microsoft Google Inception A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Liquid AI Celeris Subquadratic Nace.AI Fastino TypeSafe Microsoft Google Inception Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient | — | 2,163 | 1,202 | 121 | 840 | 87.4% | 61.2% | 9.1% | 90.9% | 90.6% |
| celeris-1.1-decision Closed · Efficient | — | 2,163 | 1,174 | 169 | 820 | 82.9% | 62.1% | 12.6% | 87.4% | 84.6% |
| celeris-1-decision Closed · Efficient | — | 2,163 | 1,164 | 173 | 826 | 82.7% | 61.8% | 12.9% | 87.1% | 84.7% |
| d1.1 Closed · Efficient | — | 2,163 | 1,007 | 202 | 954 | 82.5% | 55.9% | 16.7% | 83.3% | 84.5% |
| SubQ Snap 1.1 Closed · Efficient | — | 2,163 | 1,319 | 401 | 443 | 52.5% | 79.5% | 23.3% | 76.7% | 53.3% |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 2,163 | 1,388 | 438 | 337 | 43.5% | 84.4% | 24.0% | 76.0% | 45.4% |
| GLiDE API Closed · Efficient | — | 2,163 | 1,406 | 450 | 307 | 40.6% | 85.8% | 24.2% | 75.8% | 42.2% |
| Jev 1.14 Closed · Efficient | — | 2,163 | 1,386 | 717 | 60 | 7.7% | 97.2% | 34.1% | 65.9% | 7.1% |
| Jev 1.13 Closed · Efficient | — | 2,163 | 1,370 | 748 | 45 | 5.7% | 97.9% | 35.3% | 64.7% | 5.0% |
| Microsoft-Decision-1 Closed · Efficient | — | 2,163 | 1,254 | 863 | 46 | 5.1% | 97.9% | 40.8% | 59.2% | 5.2% |
| Gemini Decision 1 20261111 Closed · Efficient | — | 2,163 | 1,496 | 641 | 26 | 3.9% | 98.8% | 30.0% | 70.0% | 4.2% |
| Mercury Decide Closed · Efficient | — | 2,163 | 1,156 | 975 | 32 | 3.2% | 98.5% | 45.8% | 54.2% | 3.2% |
| d1 Closed · Efficient | — | 2,163 | 1,161 | 973 | 29 | 2.9% | 98.7% | 45.6% | 54.4% | 3.0% |
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Non-hallucination with “I don't know” 87.4% · answered 61.2% Logic 54.9% | — | 74.1%72.2%–75.9% | a | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 3.9% · answered 98.8% Logic 50.0% | — | 71.3%69.4%–73.2% | ab | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 40.6% · answered 85.8% Logic 54.5% | — | 70.8%68.8%–72.7% | b | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 43.5% · answered 84.4% Logic 54.8% | — | 70.1%68.1%–72.0% | b | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.7% · answered 61.8% Logic 47.3% | — | 70.0%68.0%–71.9% | b | 1,514 / 2,163 | No answers at this threshold | 0.858 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.9% · answered 62.1% Logic 47.3% | — | 69.9%67.9%–71.8% | b | 1,511 / 2,163 | No answers at this threshold | 0.859 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Non-hallucination with “I don't know” 52.5% · answered 79.5% Logic 52.4% | — | 68.6%66.6%–70.5% | b | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.5% · answered 55.9% Logic 46.0% | — | 65.2%63.2%–67.2% | c | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 7.7% · answered 97.2% Logic 48.2% | — | 65.1%63.1%–67.1% | c | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 5.7% · answered 97.9% Logic 46.3% | — | 64.1%62.0%–66.1% | c | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Non-hallucination with “I don't know” 5.1% · answered 97.9% Logic 44.4% | — | 59.4%57.3%–61.4% | d | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 2.9% · answered 98.7% Logic 40.8% | — | 55.6%53.5%–57.7% | e | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 3.2% · answered 98.5% Logic 45.1% | — | 53.5%51.4%–55.6% | e | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Chat models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
OpenAI Meta A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna chat Closed · Efficient | — | 1,075 | 1,001 | 23 | 51 | 68.9% | 95.3% | 2.2% | 97.8% | Not measured |
| Muse Ember max · chat Closed · Efficient | — | 2,163 | 2,149 | 5 | 9 | 64.3% | 99.6% | 0.2% | 99.8% | Not measured |
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 64.3% · answered 99.6% Logic 95.5% | — | 99.6%99.2%–99.8% | a | 2,154 / 2,163 | Not measured | Not measured | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 68.9% · answered 95.3% Logic 58.0% | — | 93.7%92.6%–94.6% | b | 2,026 / 2,163 | Not measured | Not measured | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Decision models
No workhorse decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Chat models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Anthropic A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse | — | 2,163 | 2,155 | 3 | 5 | 62.5% | 99.8% | 0.1% | 99.9% | Not measured |
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 62.5% · answered 99.8% Logic 95.5% | — | 99.7%99.3%–99.8% | a | 2,156 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Decision models
No flagship decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Chat models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Anthropic A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship | — | 2,163 | 2,155 | 3 | 5 | 62.5% | 99.8% | 0.1% | 99.9% | Not measured |
| Claude Fable 5.1 chat Closed · Flagship | — | 2,163 | 2,153 | 4 | 6 | 60.0% | 99.7% | 0.2% | 99.8% | Not measured |
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 62.5% · answered 99.8% Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 60.0% · answered 99.7% Logic 95.2% | — | 99.7%99.4%–99.9% | a | 2,157 / 2,163 | Not measured | Not measured | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Decision models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Liquid AI Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed | — | 2,163 | 1,126 | 138 | 899 | 86.7% | 58.4% | 10.9% | 89.1% | 88.8% |
| d1.2 Closed · Undisclosed | — | 2,163 | 1,255 | 397 | 511 | 56.3% | 76.4% | 24.0% | 76.0% | 56.3% |
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 86.7% · answered 58.4% Logic 51.6% | — | 69.9%67.9%–71.8% | a | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 56.3% · answered 76.4% Logic 49.2% | — | 65.5%63.5%–67.5% | b | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Chat models
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
xAI OpenAI Google Alibaba A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Grok 4.7 chat Closed · Undisclosed | — | 1,987 | 1,947 | 1 | 39 | 97.5% | 98.0% | 0.1% | 99.9% | Not measured |
| GPT-6 Astra chat Closed · Undisclosed | — | 2,163 | 2,149 | 3 | 11 | 78.6% | 99.5% | 0.1% | 99.9% | Not measured |
| GPT-6.1 Sol chat Closed · Undisclosed | — | 2,163 | 2,149 | 3 | 11 | 78.6% | 99.5% | 0.1% | 99.9% | Not measured |
| GPT-6.1 Terra chat Closed · Undisclosed | — | 2,163 | 2,136 | 6 | 21 | 77.8% | 99.0% | 0.3% | 99.7% | Not measured |
| Gemini 4 Carbon chat Closed · Undisclosed | — | 2,163 | 2,147 | 4 | 12 | 75.0% | 99.4% | 0.2% | 99.8% | Not measured |
| Qwen4-Max high · chat Closed · Undisclosed | — | 2,163 | 2,152 | 5 | 6 | 54.5% | 99.7% | 0.2% | 99.8% | Not measured |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | 2,163 | 2,145 | 15 | 3 | 16.7% | 99.9% | 0.7% | 99.3% | Not measured |
Not ranked (only complete runs are): Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 54.5% · answered 99.7% Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 78.6% · answered 99.5% Logic 96.7% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 78.6% · answered 99.5% Logic 96.5% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 75.0% · answered 99.4% Logic 96.7% | — | 99.6%99.2%–99.8% | ab | 2,154 / 2,163 | Not measured | Not measured | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 97.5% · answered 98.0% Logic running | — | 99.5%99.2%–99.7% | ab | 2,153 / 2,163 | Not measured | Not measured | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 16.7% · answered 99.9% Logic 94.0% | — | 99.5%99.1%–99.7% | ab | 2,152 / 2,163 | Not measured | Not measured | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 77.8% · answered 99.0% Logic 94.0% | — | 99.2%98.7%–99.5% | b | 2,146 / 2,163 | Not measured | Not measured | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI Celeris Fastino Base Labs Subquadratic Nace.AI Other Perplexity StartLux EldanRing TypeSafe tattrongvu victorbennett vLLM Semantic Router Microsoft Zefan Cai Blockbrain Labs MachinePulse Google scar-ai quincyqiang PelaAI Inception Strands Agents newsbang Valen-Team REAX Metask Lab Turion.AI Sthānika AI Logits Cloudflare OmniJev Poindexter Labs Patrick-18 Apertix AI A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Liquid AI Celeris Fastino Base Labs Subquadratic Nace.AI Other Perplexity StartLux EldanRing TypeSafe tattrongvu victorbennett vLLM Semantic Router Microsoft Zefan Cai Blockbrain Labs MachinePulse Google scar-ai quincyqiang PelaAI Inception Strands Agents newsbang Valen-Team REAX Metask Lab Turion.AI Sthānika AI Logits Cloudflare OmniJev Poindexter Labs Patrick-18 Apertix AI Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient | — | 2,163 | 1,202 | 121 | 840 | 87.4% | 61.2% | 9.1% | 90.9% | 90.6% |
| d1.2-ultra Closed · Undisclosed | — | 2,163 | 1,126 | 138 | 899 | 86.7% | 58.4% | 10.9% | 89.1% | 88.8% |
| celeris-1.1-decision Closed · Efficient | — | 2,163 | 1,174 | 169 | 820 | 82.9% | 62.1% | 12.6% | 87.4% | 84.6% |
| celeris-1-decision Closed · Efficient | — | 2,163 | 1,164 | 173 | 826 | 82.7% | 61.8% | 12.9% | 87.1% | 84.7% |
| d1.1 Closed · Efficient | — | 2,163 | 1,007 | 202 | 954 | 82.5% | 55.9% | 16.7% | 83.3% | 84.5% |
| d1-2B Open · 1.9B | — | 2,163 | 375 | 354 | 1,434 | 80.2% | 33.7% | 48.6% | 51.4% | 80.8% |
| GLiDE 0.4B Open · 0.4B | — | 2,163 | 157 | 546 | 1,460 | 72.8% | 32.5% | 77.7% | 22.3% | 72.2% |
| Mammoth-1 Open · 27B | — | 2,163 | 1,412 | 221 | 530 | 70.6% | 75.5% | 13.5% | 86.5% | 71.7% |
| Elephant-1 Open · 9.0B | — | 2,163 | 1,069 | 383 | 711 | 65.0% | 67.1% | 26.4% | 73.6% | 66.3% |
| d1.2 Closed · Undisclosed | — | 2,163 | 1,255 | 397 | 511 | 56.3% | 76.4% | 24.0% | 76.0% | 56.3% |
| SubQ Snap 1.1 Closed · Efficient | — | 2,163 | 1,319 | 401 | 443 | 52.5% | 79.5% | 23.3% | 76.7% | 53.3% |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 2,163 | 1,388 | 438 | 337 | 43.5% | 84.4% | 24.0% | 76.0% | 45.4% |
| GLiDE API Closed · Efficient | — | 2,163 | 1,406 | 450 | 307 | 40.6% | 85.8% | 24.2% | 75.8% | 42.2% |
| GLiDE v1.2 Open · 27B | — | 2,163 | 1,428 | 472 | 263 | 35.8% | 87.8% | 24.8% | 75.2% | 35.8% |
| GLiDE 9B Open · 7.9B | — | 2,163 | 1,162 | 755 | 246 | 24.6% | 88.6% | 39.4% | 60.6% | 25.3% |
| GLiDE 27B Open · 27B | — | 2,163 | 1,516 | 507 | 140 | 21.6% | 93.5% | 25.1% | 74.9% | 21.7% |
| GLiDE 0.8B Open · 0.8B | — | 2,163 | 598 | 1,239 | 326 | 20.8% | 84.9% | 67.4% | 32.6% | 21.0% |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 2,163 | 1,199 | 764 | 200 | 20.7% | 90.8% | 38.9% | 61.1% | 21.3% |
| GLiDE 2B Open · 1.9B | — | 2,163 | 740 | 1,177 | 246 | 17.3% | 88.6% | 61.4% | 38.6% | 17.2% |
| GLiClass v4 Edge Open · 0.4B | — | 2,163 | 380 | 1,476 | 307 | 17.2% | 85.8% | 79.5% | 20.5% | 17.6% |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 2,163 | 1,492 | 568 | 103 | 15.4% | 95.2% | 27.6% | 72.4% | 15.0% |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 2,163 | 1,125 | 899 | 139 | 13.4% | 93.6% | 44.4% | 55.6% | 14.0% |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 2,163 | 1,136 | 904 | 123 | 12.0% | 94.3% | 44.3% | 55.7% | 12.3% |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 2,163 | 592 | 1,394 | 177 | 11.3% | 91.8% | 70.2% | 29.8% | 11.4% |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 2,163 | 1,527 | 576 | 60 | 9.4% | 97.2% | 27.4% | 72.6% | 9.8% |
| Winnow-12B Open · 12B | — | 2,163 | 956 | 1,096 | 111 | 9.2% | 94.9% | 53.4% | 46.6% | 9.5% |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 2,163 | 1,187 | 894 | 82 | 8.4% | 96.2% | 43.0% | 57.0% | 8.4% |
| Jev 1.14 Closed · Efficient | — | 2,163 | 1,386 | 717 | 60 | 7.7% | 97.2% | 34.1% | 65.9% | 7.1% |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 2,163 | 793 | 1,272 | 98 | 7.2% | 95.5% | 61.6% | 38.4% | 7.2% |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 2,163 | 1,490 | 626 | 47 | 7.0% | 97.8% | 29.6% | 70.4% | 7.1% |
| Jeffan-9B-1211 Open · 9.0B | — | 2,163 | 1,271 | 833 | 59 | 6.6% | 97.3% | 39.6% | 60.4% | 6.7% |
| Verdin-4B-v1.1 Open · 4.0B | — | 2,163 | 1,075 | 1,018 | 70 | 6.4% | 96.8% | 48.6% | 51.4% | 6.6% |
| Jev 1.13 Closed · Efficient | — | 2,163 | 1,370 | 748 | 45 | 5.7% | 97.9% | 35.3% | 64.7% | 5.0% |
| d3-ultra Open · 26B | — | 2,163 | 1,527 | 602 | 34 | 5.3% | 98.4% | 28.3% | 71.7% | 5.4% |
| Microsoft-Decision-1 Closed · Efficient | — | 2,163 | 1,254 | 863 | 46 | 5.1% | 97.9% | 40.8% | 59.2% | 5.2% |
| Open-Jev-27B-v1.1 Open · 27B | — | 2,163 | 1,154 | 959 | 50 | 5.0% | 97.7% | 45.4% | 54.6% | 5.0% |
| Cygnet Open · 12B | — | 2,163 | 942 | 1,161 | 60 | 4.9% | 97.2% | 55.2% | 44.8% | 4.9% |
| Karpo Decision 31B Open · 31B | — | 2,163 | 1,745 | 398 | 20 | 4.8% | 99.1% | 18.6% | 81.4% | 5.0% |
| d3 Open · 26B | — | 2,163 | 1,509 | 623 | 31 | 4.7% | 98.6% | 29.2% | 70.8% | 4.8% |
| Drex Max Open · 27B | — | 2,163 | 1,515 | 619 | 29 | 4.5% | 98.7% | 29.0% | 71.0% | 4.5% |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 2,163 | 1,239 | 885 | 39 | 4.2% | 98.2% | 41.7% | 58.3% | 4.3% |
| Gemini Decision 1 20261111 Closed · Efficient | — | 2,163 | 1,496 | 641 | 26 | 3.9% | 98.8% | 30.0% | 70.0% | 4.2% |
| vera-medium-omni Open · 4.2B | — | 2,163 | 1,222 | 905 | 36 | 3.8% | 98.3% | 42.5% | 57.5% | 3.8% |
| Chinese Decision Pointer 4B Open · 4.0B | — | 2,163 | 1,081 | 1,045 | 37 | 3.4% | 98.3% | 49.2% | 50.8% | 3.4% |
| KnowLine-9B-Gen2 Open · 9.0B | — | 2,163 | 1,289 | 846 | 28 | 3.2% | 98.7% | 39.6% | 60.4% | 3.2% |
| Mercury Decide Closed · Efficient | — | 2,163 | 1,156 | 975 | 32 | 3.2% | 98.5% | 45.8% | 54.2% | 3.2% |
| Strands Decider 2B Open · 2.0B | — | 2,163 | 586 | 1,527 | 50 | 3.2% | 97.7% | 72.3% | 27.7% | 3.2% |
| Adjudex 9B v5 Open · 9.0B | — | 2,163 | 1,321 | 816 | 26 | 3.1% | 98.8% | 38.2% | 61.8% | 3.1% |
| d1 Closed · Efficient | — | 2,163 | 1,161 | 973 | 29 | 2.9% | 98.7% | 45.6% | 54.4% | 3.0% |
| Valen-Decision-1.5-9B Open · 9.0B | — | 2,163 | 1,179 | 958 | 26 | 2.6% | 98.8% | 44.8% | 55.2% | 2.7% |
| REAX 1.3 Open · 57B | — | 2,163 | 1,688 | 464 | 11 | 2.3% | 99.5% | 21.6% | 78.4% | 2.3% |
| Metask-Jev-4B Open · 4.5B | — | 2,163 | 813 | 1,320 | 30 | 2.2% | 98.6% | 61.9% | 38.1% | 2.2% |
| Turion Jev 31B Open · 31B | — | 2,163 | 1,521 | 628 | 14 | 2.2% | 99.4% | 29.2% | 70.8% | 2.1% |
| Sieve 31B v3 Open · 31B | — | 2,163 | 1,688 | 465 | 10 | 2.1% | 99.5% | 21.6% | 78.4% | 2.0% |
| Logits Sigma-1 Duo Open · 57B | — | 2,163 | 1,627 | 525 | 11 | 2.1% | 99.5% | 24.4% | 75.6% | 2.0% |
| Clef-Flash Open · 9.0B | — | 2,163 | 1,053 | 1,088 | 22 | 2.0% | 99.0% | 50.8% | 49.2% | 2.0% |
| OneJev-27B Open · 27B | — | 2,163 | 1,160 | 985 | 18 | 1.8% | 99.2% | 45.9% | 54.1% | 1.8% |
| Copernicus-31B Open · 31B | — | 2,163 | 1,698 | 457 | 8 | 1.7% | 99.6% | 21.2% | 78.8% | 1.6% |
| KnowLine-2B-Gen1 Open · 2.0B | — | 2,163 | 744 | 1,395 | 24 | 1.7% | 98.9% | 65.2% | 34.8% | 1.7% |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 2,163 | 1,263 | 885 | 15 | 1.7% | 99.3% | 41.2% | 58.8% | 1.7% |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 2,163 | 1,190 | 957 | 16 | 1.6% | 99.3% | 44.6% | 55.4% | 1.7% |
| Clef Open · 27B | — | 2,163 | 1,114 | 1,032 | 17 | 1.6% | 99.2% | 48.1% | 51.9% | 1.6% |
| Cerno 9B v1.2 Open · 9.0B | — | 2,163 | 1,237 | 915 | 11 | 1.2% | 99.5% | 42.5% | 57.5% | 1.2% |
| KnowLine-12B-Gen1 Open · 12B | — | 2,163 | 1,431 | 724 | 8 | 1.1% | 99.6% | 33.6% | 66.4% | 1.1% |
| KnowLine-31B-Gen1.5 Open · 31B | — | 2,163 | 1,692 | 466 | 5 | 1.1% | 99.8% | 21.6% | 78.4% | 1.0% |
| KnowLine-9B-Gen1 Open · 9.0B | — | 2,163 | 1,301 | 854 | 8 | 0.9% | 99.6% | 39.6% | 60.4% | 0.9% |
| OneJev1.6-0.8B Open · 0.8B | — | 2,163 | 603 | 1,550 | 10 | 0.6% | 99.5% | 72.0% | 28.0% | 0.6% |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 2,163 | 612 | 1,546 | 5 | 0.3% | 99.8% | 71.6% | 28.4% | 0.3% |
| KnowLine-31B-Gen2 Open · 31B | — | 2,163 | 1,733 | 429 | 1 | 0.2% | 100.0% | 19.8% | 80.2% | 0.2% |
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Non-hallucination with “I don't know” 4.8% · answered 99.1% Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Non-hallucination with “I don't know” 0.2% · answered 100.0% Logic 58.9% | — | 80.3%78.5%–81.9% | ab | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Non-hallucination with “I don't know” 2.3% · answered 99.5% Logic 59.3% | — | 78.0%76.2%–79.7% | bc | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 1.1% · answered 99.8% Logic 59.0% | — | 77.4%75.6%–79.1% | cd | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Non-hallucination with “I don't know” 1.7% · answered 99.6% Logic 58.7% | — | 77.3%75.5%–79.1% | cd | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 2.1% · answered 99.5% Logic 57.9% | — | 77.1%75.3%–78.8% | cd | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Non-hallucination with “I don't know” 2.1% · answered 99.5% Logic 57.8% | — | 75.2%73.4%–77.0% | de | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Non-hallucination with “I don't know” 87.4% · answered 61.2% Logic 54.9% | — | 74.1%72.2%–75.9% | defg | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Non-hallucination with “I don't know” 70.6% · answered 75.5% Logic 54.5% | — | 74.0%72.1%–75.8% | defh | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 9.4% · answered 97.2% Logic 54.8% | — | 72.5%70.6%–74.3% | efghi | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Non-hallucination with “I don't know” 21.6% · answered 93.5% Logic 58.4% | — | 72.0%70.1%–73.8% | efghij | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Non-hallucination with “I don't know” 5.3% · answered 98.4% Logic 54.6% | — | 71.5%69.6%–73.4% | fghij | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 3.9% · answered 98.8% Logic 50.0% | — | 71.3%69.4%–73.2% | efghij | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 40.6% · answered 85.8% Logic 54.5% | — | 70.8%68.8%–72.7% | fghij | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Non-hallucination with “I don't know” 4.7% · answered 98.6% Logic 54.5% | — | 70.4%68.5%–72.3% | gij | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Non-hallucination with “I don't know” 4.5% · answered 98.7% Logic 53.1% | — | 70.3%68.4%–72.2% | gijk | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 43.5% · answered 84.4% Logic 54.8% | — | 70.1%68.1%–72.0% | ijk | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.7% · answered 61.8% Logic 47.3% | — | 70.0%68.0%–71.9% | hijk | 1,514 / 2,163 | No answers at this threshold | 0.858 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Non-hallucination with “I don't know” 35.8% · answered 87.8% Logic 55.2% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.9% · answered 62.1% Logic 47.3% | — | 69.9%67.9%–71.8% | ijkl | 1,511 / 2,163 | No answers at this threshold | 0.859 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 86.7% · answered 58.4% Logic 51.6% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 7.0% · answered 97.8% Logic 53.3% | — | 69.8%67.8%–71.7% | ijkl | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 15.4% · answered 95.2% Logic 52.5% | with web | 69.3%67.3%–71.2% | jklm | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 2.2% · answered 99.4% Logic 52.3% | — | 69.0%67.0%–70.9% | ijklmn | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Non-hallucination with “I don't know” 52.5% · answered 79.5% Logic 52.4% | — | 68.6%66.6%–70.5% | jklmn | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Non-hallucination with “I don't know” 1.1% · answered 99.6% Logic 50.6% | — | 66.3%64.2%–68.2% | klmno | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 56.3% · answered 76.4% Logic 49.2% | — | 65.5%63.5%–67.5% | lmnop | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.5% · answered 55.9% Logic 46.0% | — | 65.2%63.2%–67.2% | mnop | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 7.7% · answered 97.2% Logic 48.2% | — | 65.1%63.1%–67.1% | nop | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 5.7% · answered 97.9% Logic 46.3% | — | 64.1%62.0%–66.1% | opq | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 3.1% · answered 98.8% Logic 46.4% | — | 62.1%60.0%–64.1% | opqr | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Non-hallucination with “I don't know” 65.0% · answered 67.1% Logic 46.4% | — | 61.3%59.3%–63.4% | pqr | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 3.2% · answered 98.7% Logic 43.9% | — | 60.1%58.0%–62.1% | qrs | 1,300 / 2,163 | No answers at this threshold | 0.803 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 6.6% · answered 97.3% Logic 48.2% | — | 59.8%57.7%–61.9% | qrs | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 0.9% · answered 99.6% Logic 44.3% | — | 59.8%57.7%–61.8% | qrs | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Non-hallucination with “I don't know” 5.1% · answered 97.9% Logic 44.4% | — | 59.4%57.3%–61.4% | rst | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Non-hallucination with “I don't know” 20.7% · answered 90.8% Logic 41.9% | — | 59.0%56.9%–61.0% | rstu | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Non-hallucination with “I don't know” 1.7% · answered 99.3% Logic 45.3% | — | 58.8%56.7%–60.9% | rstu | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Non-hallucination with “I don't know” 4.2% · answered 98.2% Logic 46.6% | — | 58.4%56.3%–60.5% | rstuv | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Non-hallucination with “I don't know” 24.6% · answered 88.6% Logic 47.8% | — | 57.9%55.8%–60.0% | rstuvw | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 3.8% · answered 98.3% Logic 44.1% | — | 56.8%54.7%–58.9% | stuvw | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Non-hallucination with “I don't know” 1.2% · answered 99.5% Logic 43.3% | — | 56.0%53.9%–58.1% | stuvw | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Non-hallucination with “I don't know” 1.6% · answered 99.3% Logic 42.3% | — | 56.0%53.9%–58.1% | stuvw | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Non-hallucination with “I don't know” 8.4% · answered 96.2% Logic 44.7% | — | 55.9%53.8%–58.0% | stuvw | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Non-hallucination with “I don't know” 13.4% · answered 93.6% Logic 45.9% | — | 55.8%53.7%–57.9% | stuvwx | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 2.9% · answered 98.7% Logic 40.8% | — | 55.6%53.5%–57.7% | stuvwxy | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 2.6% · answered 98.8% Logic 44.4% | — | 55.3%53.2%–57.4% | tuvwxy | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Non-hallucination with “I don't know” 12.0% · answered 94.3% Logic 43.1% | — | 55.2%53.1%–57.3% | tuvwxy | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Non-hallucination with “I don't know” 5.0% · answered 97.7% Logic 42.6% | — | 54.6%52.4%–56.6% | uvwxyz | 1,180 / 2,163 | No answers at this threshold | 0.764 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Non-hallucination with “I don't know” 1.8% · answered 99.2% Logic 40.6% | — | 53.8%51.7%–55.9% | vwxyzA | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 3.2% · answered 98.5% Logic 45.1% | — | 53.5%51.4%–55.6% | wxyzA | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 6.4% · answered 96.8% Logic 39.7% | — | 51.5%49.4%–53.6% | yzAB | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Non-hallucination with “I don't know” 1.6% · answered 99.2% Logic 40.8% | — | 51.1%49.0%–53.2% | xyzAB | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 3.4% · answered 98.3% Logic 40.4% | — | 50.4%48.3%–52.5% | zAB | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Non-hallucination with “I don't know” 2.0% · answered 99.0% Logic 38.9% | — | 49.8%47.7%–51.9% | AB | 1,077 / 2,163 | No answers at this threshold | 0.730 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Non-hallucination with “I don't know” 9.2% · answered 94.9% Logic 39.7% | — | 46.9%44.8%–49.0% | BC | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Non-hallucination with “I don't know” 4.9% · answered 97.2% Logic 39.3% | — | 44.3%42.3%–46.4% | C | 959 / 2,163 | No answers at this threshold | 0.735 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Non-hallucination with “I don't know” 2.2% · answered 98.6% Logic 32.1% | — | 39.2%37.1%–41.2% | D | 847 / 2,163 | No answers at this threshold | 0.651 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Non-hallucination with “I don't know” 7.2% · answered 95.5% Logic 30.8% | — | 38.7%36.6%–40.7% | D | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Non-hallucination with “I don't know” 80.2% · answered 33.7% Logic 30.7% | — | 37.1%35.1%–39.1% | D | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Non-hallucination with “I don't know” 17.3% · answered 88.6% Logic 32.3% | — | 36.8%34.7%–38.8% | D | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Non-hallucination with “I don't know” 1.7% · answered 98.9% Logic 31.7% | — | 35.3%33.3%–37.3% | DE | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Non-hallucination with “I don't know” 20.8% · answered 84.9% Logic 29.2% | — | 32.1%30.2%–34.1% | EF | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Non-hallucination with “I don't know” 11.3% · answered 91.8% Logic 24.7% | — | 29.7%27.8%–31.6% | F | 642 / 2,163 | No answers at this threshold | 0.583 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Non-hallucination with “I don't know” 3.2% · answered 97.7% Logic 22.4% | — | 28.7%26.8%–30.6% | F | 620 / 2,163 | No answers at this threshold | 0.569 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Non-hallucination with “I don't know” 0.6% · answered 99.5% Logic 27.0% | — | 28.2%26.4%–30.2% | F | 611 / 2,163 | No answers at this threshold | 0.592 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Non-hallucination with “I don't know” 0.3% · answered 99.8% Logic 26.7% | — | 28.2%26.3%–30.1% | F | 610 / 2,163 | No answers at this threshold | 0.596 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Non-hallucination with “I don't know” 17.2% · answered 85.8% Logic 23.6% | — | 22.7%20.9%–24.5% | G | 490 / 2,163 | No answers at this threshold | 0.501 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Non-hallucination with “I don't know” 72.8% · answered 32.5% Logic 23.2% | — | 20.6%18.9%–22.3% | G | 445 / 2,163 | No answers at this threshold | 0.500 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI Fastino Other StartLux Perplexity victorbennett scar-ai quincyqiang Strands Agents Metask Lab PelaAI OmniJev A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Liquid AI Fastino Other StartLux Perplexity victorbennett scar-ai quincyqiang Strands Agents Metask Lab PelaAI OmniJev Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-2B Open · 1.9B | — | 2,163 | 375 | 354 | 1,434 | 80.2% | 33.7% | 48.6% | 51.4% | 80.8% |
| GLiDE 0.4B Open · 0.4B | — | 2,163 | 157 | 546 | 1,460 | 72.8% | 32.5% | 77.7% | 22.3% | 72.2% |
| GLiDE 0.8B Open · 0.8B | — | 2,163 | 598 | 1,239 | 326 | 20.8% | 84.9% | 67.4% | 32.6% | 21.0% |
| GLiDE 2B Open · 1.9B | — | 2,163 | 740 | 1,177 | 246 | 17.3% | 88.6% | 61.4% | 38.6% | 17.2% |
| GLiClass v4 Edge Open · 0.4B | — | 2,163 | 380 | 1,476 | 307 | 17.2% | 85.8% | 79.5% | 20.5% | 17.6% |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 2,163 | 1,125 | 899 | 139 | 13.4% | 93.6% | 44.4% | 55.6% | 14.0% |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 2,163 | 1,136 | 904 | 123 | 12.0% | 94.3% | 44.3% | 55.7% | 12.3% |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 2,163 | 592 | 1,394 | 177 | 11.3% | 91.8% | 70.2% | 29.8% | 11.4% |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 2,163 | 1,187 | 894 | 82 | 8.4% | 96.2% | 43.0% | 57.0% | 8.4% |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 2,163 | 793 | 1,272 | 98 | 7.2% | 95.5% | 61.6% | 38.4% | 7.2% |
| Verdin-4B-v1.1 Open · 4.0B | — | 2,163 | 1,075 | 1,018 | 70 | 6.4% | 96.8% | 48.6% | 51.4% | 6.6% |
| vera-medium-omni Open · 4.2B | — | 2,163 | 1,222 | 905 | 36 | 3.8% | 98.3% | 42.5% | 57.5% | 3.8% |
| Chinese Decision Pointer 4B Open · 4.0B | — | 2,163 | 1,081 | 1,045 | 37 | 3.4% | 98.3% | 49.2% | 50.8% | 3.4% |
| Strands Decider 2B Open · 2.0B | — | 2,163 | 586 | 1,527 | 50 | 3.2% | 97.7% | 72.3% | 27.7% | 3.2% |
| Metask-Jev-4B Open · 4.5B | — | 2,163 | 813 | 1,320 | 30 | 2.2% | 98.6% | 61.9% | 38.1% | 2.2% |
| KnowLine-2B-Gen1 Open · 2.0B | — | 2,163 | 744 | 1,395 | 24 | 1.7% | 98.9% | 65.2% | 34.8% | 1.7% |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 2,163 | 1,190 | 957 | 16 | 1.6% | 99.3% | 44.6% | 55.4% | 1.7% |
| OneJev1.6-0.8B Open · 0.8B | — | 2,163 | 603 | 1,550 | 10 | 0.6% | 99.5% | 72.0% | 28.0% | 0.6% |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 2,163 | 612 | 1,546 | 5 | 0.3% | 99.8% | 71.6% | 28.4% | 0.3% |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 3.8% · answered 98.3% Logic 44.1% | — | 56.8%54.7%–58.9% | a | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Non-hallucination with “I don't know” 1.6% · answered 99.3% Logic 42.3% | — | 56.0%53.9%–58.1% | a | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Non-hallucination with “I don't know” 8.4% · answered 96.2% Logic 44.7% | — | 55.9%53.8%–58.0% | a | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Non-hallucination with “I don't know” 13.4% · answered 93.6% Logic 45.9% | — | 55.8%53.7%–57.9% | a | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Non-hallucination with “I don't know” 12.0% · answered 94.3% Logic 43.1% | — | 55.2%53.1%–57.3% | a | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 6.4% · answered 96.8% Logic 39.7% | — | 51.5%49.4%–53.6% | b | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Non-hallucination with “I don't know” 3.4% · answered 98.3% Logic 40.4% | — | 50.4%48.3%–52.5% | b | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Non-hallucination with “I don't know” 2.2% · answered 98.6% Logic 32.1% | — | 39.2%37.1%–41.2% | c | 847 / 2,163 | No answers at this threshold | 0.651 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Non-hallucination with “I don't know” 7.2% · answered 95.5% Logic 30.8% | — | 38.7%36.6%–40.7% | c | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Non-hallucination with “I don't know” 80.2% · answered 33.7% Logic 30.7% | — | 37.1%35.1%–39.1% | c | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Non-hallucination with “I don't know” 17.3% · answered 88.6% Logic 32.3% | — | 36.8%34.7%–38.8% | c | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Non-hallucination with “I don't know” 1.7% · answered 98.9% Logic 31.7% | — | 35.3%33.3%–37.3% | cd | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Non-hallucination with “I don't know” 20.8% · answered 84.9% Logic 29.2% | — | 32.1%30.2%–34.1% | de | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Non-hallucination with “I don't know” 11.3% · answered 91.8% Logic 24.7% | — | 29.7%27.8%–31.6% | ef | 642 / 2,163 | No answers at this threshold | 0.583 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Non-hallucination with “I don't know” 3.2% · answered 97.7% Logic 22.4% | — | 28.7%26.8%–30.6% | ef | 620 / 2,163 | No answers at this threshold | 0.569 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Non-hallucination with “I don't know” 0.6% · answered 99.5% Logic 27.0% | — | 28.2%26.4%–30.2% | f | 611 / 2,163 | No answers at this threshold | 0.592 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Non-hallucination with “I don't know” 0.3% · answered 99.8% Logic 26.7% | — | 28.2%26.3%–30.1% | f | 610 / 2,163 | No answers at this threshold | 0.596 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Non-hallucination with “I don't know” 17.2% · answered 85.8% Logic 23.6% | — | 22.7%20.9%–24.5% | g | 490 / 2,163 | No answers at this threshold | 0.501 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Non-hallucination with “I don't know” 72.8% · answered 32.5% Logic 23.2% | — | 20.6%18.9%–22.3% | g | 445 / 2,163 | No answers at this threshold | 0.500 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Base Labs Fastino EldanRing tattrongvu Blockbrain Labs StartLux PelaAI newsbang Valen-Team Cloudflare Patrick-18 Apertix AI A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Base Labs Fastino EldanRing tattrongvu Blockbrain Labs StartLux PelaAI newsbang Valen-Team Cloudflare Patrick-18 Apertix AI Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Elephant-1 Open · 9.0B | — | 2,163 | 1,069 | 383 | 711 | 65.0% | 67.1% | 26.4% | 73.6% | 66.3% |
| GLiDE 9B Open · 7.9B | — | 2,163 | 1,162 | 755 | 246 | 24.6% | 88.6% | 39.4% | 60.6% | 25.3% |
| Winnow-12B Open · 12B | — | 2,163 | 956 | 1,096 | 111 | 9.2% | 94.9% | 53.4% | 46.6% | 9.5% |
| Jeffan-9B-1211 Open · 9.0B | — | 2,163 | 1,271 | 833 | 59 | 6.6% | 97.3% | 39.6% | 60.4% | 6.7% |
| Cygnet Open · 12B | — | 2,163 | 942 | 1,161 | 60 | 4.9% | 97.2% | 55.2% | 44.8% | 4.9% |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 2,163 | 1,239 | 885 | 39 | 4.2% | 98.2% | 41.7% | 58.3% | 4.3% |
| KnowLine-9B-Gen2 Open · 9.0B | — | 2,163 | 1,289 | 846 | 28 | 3.2% | 98.7% | 39.6% | 60.4% | 3.2% |
| Adjudex 9B v5 Open · 9.0B | — | 2,163 | 1,321 | 816 | 26 | 3.1% | 98.8% | 38.2% | 61.8% | 3.1% |
| Valen-Decision-1.5-9B Open · 9.0B | — | 2,163 | 1,179 | 958 | 26 | 2.6% | 98.8% | 44.8% | 55.2% | 2.7% |
| Clef-Flash Open · 9.0B | — | 2,163 | 1,053 | 1,088 | 22 | 2.0% | 99.0% | 50.8% | 49.2% | 2.0% |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 2,163 | 1,263 | 885 | 15 | 1.7% | 99.3% | 41.2% | 58.8% | 1.7% |
| Cerno 9B v1.2 Open · 9.0B | — | 2,163 | 1,237 | 915 | 11 | 1.2% | 99.5% | 42.5% | 57.5% | 1.2% |
| KnowLine-12B-Gen1 Open · 12B | — | 2,163 | 1,431 | 724 | 8 | 1.1% | 99.6% | 33.6% | 66.4% | 1.1% |
| KnowLine-9B-Gen1 Open · 9.0B | — | 2,163 | 1,301 | 854 | 8 | 0.9% | 99.6% | 39.6% | 60.4% | 0.9% |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Non-hallucination with “I don't know” 1.1% · answered 99.6% Logic 50.6% | — | 66.3%64.2%–68.2% | a | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 3.1% · answered 98.8% Logic 46.4% | — | 62.1%60.0%–64.1% | b | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Non-hallucination with “I don't know” 65.0% · answered 67.1% Logic 46.4% | — | 61.3%59.3%–63.4% | bc | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 3.2% · answered 98.7% Logic 43.9% | — | 60.1%58.0%–62.1% | bcd | 1,300 / 2,163 | No answers at this threshold | 0.803 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 6.6% · answered 97.3% Logic 48.2% | — | 59.8%57.7%–61.9% | bcd | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 0.9% · answered 99.6% Logic 44.3% | — | 59.8%57.7%–61.8% | bcd | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Non-hallucination with “I don't know” 1.7% · answered 99.3% Logic 45.3% | — | 58.8%56.7%–60.9% | cde | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Non-hallucination with “I don't know” 4.2% · answered 98.2% Logic 46.6% | — | 58.4%56.3%–60.5% | cde | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Non-hallucination with “I don't know” 24.6% · answered 88.6% Logic 47.8% | — | 57.9%55.8%–60.0% | de | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Non-hallucination with “I don't know” 1.2% · answered 99.5% Logic 43.3% | — | 56.0%53.9%–58.1% | e | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Non-hallucination with “I don't know” 2.6% · answered 98.8% Logic 44.4% | — | 55.3%53.2%–57.4% | e | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Non-hallucination with “I don't know” 2.0% · answered 99.0% Logic 38.9% | — | 49.8%47.7%–51.9% | f | 1,077 / 2,163 | No answers at this threshold | 0.730 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Non-hallucination with “I don't know” 9.2% · answered 94.9% Logic 39.7% | — | 46.9%44.8%–49.0% | fg | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Non-hallucination with “I don't know” 4.9% · answered 97.2% Logic 39.3% | — | 44.3%42.3%–46.4% | g | 959 / 2,163 | No answers at this threshold | 0.735 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Base Labs Fastino Nace.AI Perplexity vLLM Semantic Router Zefan Cai MachinePulse Turion.AI Sthānika AI OmniJev Poindexter Labs Cloudflare PelaAI A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Base Labs Fastino Nace.AI Perplexity vLLM Semantic Router Zefan Cai MachinePulse Turion.AI Sthānika AI OmniJev Poindexter Labs Cloudflare PelaAI Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Mammoth-1 Open · 27B | — | 2,163 | 1,412 | 221 | 530 | 70.6% | 75.5% | 13.5% | 86.5% | 71.7% |
| GLiDE v1.2 Open · 27B | — | 2,163 | 1,428 | 472 | 263 | 35.8% | 87.8% | 24.8% | 75.2% | 35.8% |
| GLiDE 27B Open · 27B | — | 2,163 | 1,516 | 507 | 140 | 21.6% | 93.5% | 25.1% | 74.9% | 21.7% |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 2,163 | 1,199 | 764 | 200 | 20.7% | 90.8% | 38.9% | 61.1% | 21.3% |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 2,163 | 1,492 | 568 | 103 | 15.4% | 95.2% | 27.6% | 72.4% | 15.0% |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 2,163 | 1,527 | 576 | 60 | 9.4% | 97.2% | 27.4% | 72.6% | 9.8% |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 2,163 | 1,490 | 626 | 47 | 7.0% | 97.8% | 29.6% | 70.4% | 7.1% |
| d3-ultra Open · 26B | — | 2,163 | 1,527 | 602 | 34 | 5.3% | 98.4% | 28.3% | 71.7% | 5.4% |
| Open-Jev-27B-v1.1 Open · 27B | — | 2,163 | 1,154 | 959 | 50 | 5.0% | 97.7% | 45.4% | 54.6% | 5.0% |
| Karpo Decision 31B Open · 31B | — | 2,163 | 1,745 | 398 | 20 | 4.8% | 99.1% | 18.6% | 81.4% | 5.0% |
| d3 Open · 26B | — | 2,163 | 1,509 | 623 | 31 | 4.7% | 98.6% | 29.2% | 70.8% | 4.8% |
| Drex Max Open · 27B | — | 2,163 | 1,515 | 619 | 29 | 4.5% | 98.7% | 29.0% | 71.0% | 4.5% |
| Turion Jev 31B Open · 31B | — | 2,163 | 1,521 | 628 | 14 | 2.2% | 99.4% | 29.2% | 70.8% | 2.1% |
| Sieve 31B v3 Open · 31B | — | 2,163 | 1,688 | 465 | 10 | 2.1% | 99.5% | 21.6% | 78.4% | 2.0% |
| OneJev-27B Open · 27B | — | 2,163 | 1,160 | 985 | 18 | 1.8% | 99.2% | 45.9% | 54.1% | 1.8% |
| Copernicus-31B Open · 31B | — | 2,163 | 1,698 | 457 | 8 | 1.7% | 99.6% | 21.2% | 78.8% | 1.6% |
| Clef Open · 27B | — | 2,163 | 1,114 | 1,032 | 17 | 1.6% | 99.2% | 48.1% | 51.9% | 1.6% |
| KnowLine-31B-Gen1.5 Open · 31B | — | 2,163 | 1,692 | 466 | 5 | 1.1% | 99.8% | 21.6% | 78.4% | 1.0% |
| KnowLine-31B-Gen2 Open · 31B | — | 2,163 | 1,733 | 429 | 1 | 0.2% | 100.0% | 19.8% | 80.2% | 0.2% |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Non-hallucination with “I don't know” 4.8% · answered 99.1% Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Non-hallucination with “I don't know” 0.2% · answered 100.0% Logic 58.9% | — | 80.3%78.5%–81.9% | a | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 1.1% · answered 99.8% Logic 59.0% | — | 77.4%75.6%–79.1% | b | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Non-hallucination with “I don't know” 1.7% · answered 99.6% Logic 58.7% | — | 77.3%75.5%–79.1% | b | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 2.1% · answered 99.5% Logic 57.9% | — | 77.1%75.3%–78.8% | bc | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Non-hallucination with “I don't know” 70.6% · answered 75.5% Logic 54.5% | — | 74.0%72.1%–75.8% | cd | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 9.4% · answered 97.2% Logic 54.8% | — | 72.5%70.6%–74.3% | de | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Non-hallucination with “I don't know” 21.6% · answered 93.5% Logic 58.4% | — | 72.0%70.1%–73.8% | def | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Non-hallucination with “I don't know” 5.3% · answered 98.4% Logic 54.6% | — | 71.5%69.6%–73.4% | def | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Non-hallucination with “I don't know” 4.7% · answered 98.6% Logic 54.5% | — | 70.4%68.5%–72.3% | ef | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Non-hallucination with “I don't know” 4.5% · answered 98.7% Logic 53.1% | — | 70.3%68.4%–72.2% | ef | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Non-hallucination with “I don't know” 35.8% · answered 87.8% Logic 55.2% | — | 69.9%67.9%–71.8% | ef | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 7.0% · answered 97.8% Logic 53.3% | — | 69.8%67.8%–71.7% | f | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Non-hallucination with “I don't know” 15.4% · answered 95.2% Logic 52.5% | with web | 69.3%67.3%–71.2% | f | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Non-hallucination with “I don't know” 2.2% · answered 99.4% Logic 52.3% | — | 69.0%67.0%–70.9% | f | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Non-hallucination with “I don't know” 20.7% · answered 90.8% Logic 41.9% | — | 59.0%56.9%–61.0% | g | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Non-hallucination with “I don't know” 5.0% · answered 97.7% Logic 42.6% | — | 54.6%52.4%–56.6% | h | 1,180 / 2,163 | No answers at this threshold | 0.764 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Non-hallucination with “I don't know” 1.8% · answered 99.2% Logic 40.6% | — | 53.8%51.7%–55.9% | h | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Non-hallucination with “I don't know” 1.6% · answered 99.2% Logic 40.8% | — | 51.1%49.0%–53.2% | h | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
REAX Logits A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
REAX Logits Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 2,163 | 1,688 | 464 | 11 | 2.3% | 99.5% | 21.6% | 78.4% | 2.3% |
| Logits Sigma-1 Duo Open · 57B | — | 2,163 | 1,627 | 525 | 11 | 2.1% | 99.5% | 24.4% | 75.6% | 2.0% |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Non-hallucination with “I don't know” 2.3% · answered 99.5% Logic 59.3% | — | 78.0%76.2%–79.7% | a | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Non-hallucination with “I don't know” 2.1% · answered 99.5% Logic 57.8% | — | 75.2%73.4%–77.0% | b | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI Celeris Subquadratic Nace.AI Fastino TypeSafe Microsoft Google Inception A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Liquid AI Celeris Subquadratic Nace.AI Fastino TypeSafe Microsoft Google Inception Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient | — | 2,163 | 1,202 | 121 | 840 | 87.4% | 61.2% | 9.1% | 90.9% | 90.6% |
| celeris-1.1-decision Closed · Efficient | — | 2,163 | 1,174 | 169 | 820 | 82.9% | 62.1% | 12.6% | 87.4% | 84.6% |
| celeris-1-decision Closed · Efficient | — | 2,163 | 1,164 | 173 | 826 | 82.7% | 61.8% | 12.9% | 87.1% | 84.7% |
| d1.1 Closed · Efficient | — | 2,163 | 1,007 | 202 | 954 | 82.5% | 55.9% | 16.7% | 83.3% | 84.5% |
| SubQ Snap 1.1 Closed · Efficient | — | 2,163 | 1,319 | 401 | 443 | 52.5% | 79.5% | 23.3% | 76.7% | 53.3% |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 2,163 | 1,388 | 438 | 337 | 43.5% | 84.4% | 24.0% | 76.0% | 45.4% |
| GLiDE API Closed · Efficient | — | 2,163 | 1,406 | 450 | 307 | 40.6% | 85.8% | 24.2% | 75.8% | 42.2% |
| Jev 1.14 Closed · Efficient | — | 2,163 | 1,386 | 717 | 60 | 7.7% | 97.2% | 34.1% | 65.9% | 7.1% |
| Jev 1.13 Closed · Efficient | — | 2,163 | 1,370 | 748 | 45 | 5.7% | 97.9% | 35.3% | 64.7% | 5.0% |
| Microsoft-Decision-1 Closed · Efficient | — | 2,163 | 1,254 | 863 | 46 | 5.1% | 97.9% | 40.8% | 59.2% | 5.2% |
| Gemini Decision 1 20261111 Closed · Efficient | — | 2,163 | 1,496 | 641 | 26 | 3.9% | 98.8% | 30.0% | 70.0% | 4.2% |
| Mercury Decide Closed · Efficient | — | 2,163 | 1,156 | 975 | 32 | 3.2% | 98.5% | 45.8% | 54.2% | 3.2% |
| d1 Closed · Efficient | — | 2,163 | 1,161 | 973 | 29 | 2.9% | 98.7% | 45.6% | 54.4% | 3.0% |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Non-hallucination with “I don't know” 87.4% · answered 61.2% Logic 54.9% | — | 74.1%72.2%–75.9% | a | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 3.9% · answered 98.8% Logic 50.0% | — | 71.3%69.4%–73.2% | ab | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 40.6% · answered 85.8% Logic 54.5% | — | 70.8%68.8%–72.7% | b | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 43.5% · answered 84.4% Logic 54.8% | — | 70.1%68.1%–72.0% | b | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.7% · answered 61.8% Logic 47.3% | — | 70.0%68.0%–71.9% | b | 1,514 / 2,163 | No answers at this threshold | 0.858 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.9% · answered 62.1% Logic 47.3% | — | 69.9%67.9%–71.8% | b | 1,511 / 2,163 | No answers at this threshold | 0.859 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Non-hallucination with “I don't know” 52.5% · answered 79.5% Logic 52.4% | — | 68.6%66.6%–70.5% | b | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 82.5% · answered 55.9% Logic 46.0% | — | 65.2%63.2%–67.2% | c | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 7.7% · answered 97.2% Logic 48.2% | — | 65.1%63.1%–67.1% | c | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 5.7% · answered 97.9% Logic 46.3% | — | 64.1%62.0%–66.1% | c | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Non-hallucination with “I don't know” 5.1% · answered 97.9% Logic 44.4% | — | 59.4%57.3%–61.4% | d | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 2.9% · answered 98.7% Logic 40.8% | — | 55.6%53.5%–57.7% | e | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 3.2% · answered 98.5% Logic 45.1% | — | 53.5%51.4%–55.6% | e | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
No workhorse decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
No flagship decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI A setup that always chose F would reach 100% with no right answers.
Saying “I don't know”, or held back by confidence
Each setup's non-hallucination rate with F, and its five-option run held back by its own confidence at about the same attempt rate (it answers only where the chosen option's probability reaches the cut-off that keeps that share; tb-next's derived rate) · higher is better
Liquid AI Solid: with F Striped: held back by its own confidence
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed | — | 2,163 | 1,126 | 138 | 899 | 86.7% | 58.4% | 10.9% | 89.1% | 88.8% |
| d1.2 Closed · Undisclosed | — | 2,163 | 1,255 | 397 | 511 | 56.3% | 76.4% | 24.0% | 76.0% | 56.3% |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 86.7% · answered 58.4% Logic 51.6% | — | 69.9%67.9%–71.8% | a | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Non-hallucination with “I don't know” 56.3% · answered 76.4% Logic 49.2% | — | 65.5%63.5%–67.5% | b | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
xAI Google OpenAI Liquid AI Other Meta Anthropic Thinking Machines IBM Alibaba OpenBMB Xiaomi MiniMax eCloud Technology KRAFTON DeepSeek Z.ai A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Grok 4.7 chat Closed · Undisclosed | — | 1,987 | 1,947 | 1 | 39 | 97.5% | 98.0% | 0.1% | 99.9% | Not measured |
| Gemini 4 Hydrogen chat | — | 2,163 | 2,142 | 3 | 18 | 85.7% | 99.2% | 0.1% | 99.9% | Not measured |
| GPT-6 Astra chat Closed · Undisclosed | — | 2,163 | 2,149 | 3 | 11 | 78.6% | 99.5% | 0.1% | 99.9% | Not measured |
| GPT-6.1 Sol chat Closed · Undisclosed | — | 2,163 | 2,149 | 3 | 11 | 78.6% | 99.5% | 0.1% | 99.9% | Not measured |
| GPT-6.1 Terra chat Closed · Undisclosed | — | 2,163 | 2,136 | 6 | 21 | 77.8% | 99.0% | 0.3% | 99.7% | Not measured |
| Gemini 4 Carbon chat Closed · Undisclosed | — | 2,163 | 2,147 | 4 | 12 | 75.0% | 99.4% | 0.2% | 99.8% | Not measured |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | 2,163 | 2,128 | 10 | 25 | 71.4% | 98.8% | 0.5% | 99.5% | Not measured |
| GPT-6 Luna chat Closed · Efficient | — | 1,075 | 1,001 | 23 | 51 | 68.9% | 95.3% | 2.2% | 97.8% | Not measured |
| LFM3-1.7B off · chat Open · 1.7B | — | 2,163 | 2,132 | 10 | 21 | 67.7% | 99.0% | 0.5% | 99.5% | Not measured |
| LFM3-1.7B on · chat Open · 1.7B | — | 2,163 | 2,129 | 11 | 23 | 67.6% | 98.9% | 0.5% | 99.5% | Not measured |
| Muse Ember max · chat Closed · Efficient | — | 2,163 | 2,149 | 5 | 9 | 64.3% | 99.6% | 0.2% | 99.8% | Not measured |
| Claude Opus 5.5 chat Closed · Flagship | — | 2,163 | 2,155 | 3 | 5 | 62.5% | 99.8% | 0.1% | 99.9% | Not measured |
| Claude Sonnet 5.5 chat Closed · Workhorse | — | 2,163 | 2,155 | 3 | 5 | 62.5% | 99.8% | 0.1% | 99.9% | Not measured |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | 2,163 | 2,150 | 5 | 8 | 61.5% | 99.6% | 0.2% | 99.8% | Not measured |
| Granite-4.2-3B high · chat Open · 3.0B | — | 2,163 | 2,137 | 10 | 16 | 61.5% | 99.3% | 0.5% | 99.5% | Not measured |
| Claude Fable 5.1 chat Closed · Flagship | — | 2,163 | 2,153 | 4 | 6 | 60.0% | 99.7% | 0.2% | 99.8% | Not measured |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | 2,163 | 2,152 | 5 | 6 | 54.5% | 99.7% | 0.2% | 99.8% | Not measured |
| Qwen4-Max high · chat Closed · Undisclosed | — | 2,163 | 2,152 | 5 | 6 | 54.5% | 99.7% | 0.2% | 99.8% | Not measured |
| MiniCPM5-2B on · chat Open · 2.5B | — | 2,163 | 2,135 | 13 | 15 | 53.6% | 99.3% | 0.6% | 99.4% | Not measured |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | 2,163 | 2,149 | 7 | 7 | 50.0% | 99.7% | 0.3% | 99.7% | Not measured |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | 2,163 | 2,119 | 24 | 20 | 45.5% | 99.1% | 1.1% | 98.9% | Not measured |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 2,163 | 2,039 | 68 | 56 | 45.2% | 97.4% | 3.2% | 96.8% | Not measured |
| Gemma 4 E4B chat Open · 8.0B | — | 2,163 | 2,113 | 28 | 22 | 44.0% | 99.0% | 1.3% | 98.7% | Not measured |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | 2,163 | 2,131 | 19 | 13 | 40.6% | 99.4% | 0.9% | 99.1% | Not measured |
| Qwen3-14B off · chat Open · 15B | — | 2,163 | 2,046 | 71 | 46 | 39.3% | 97.9% | 3.4% | 96.6% | Not measured |
| Gemma 4 E4B on · chat Open · 8.0B | — | 2,163 | 2,117 | 29 | 17 | 37.0% | 99.2% | 1.4% | 98.6% | Not measured |
| Qwen3.5-4B off · chat Open · 4.7B | — | 2,163 | 2,138 | 16 | 9 | 36.0% | 99.6% | 0.7% | 99.3% | Not measured |
| Erk-14B off · chat Open · 15B | — | 2,163 | 1,901 | 168 | 94 | 35.9% | 95.7% | 8.1% | 91.9% | Not measured |
| Qwen3.5-2B off · chat Open · 2.0B | — | 2,163 | 2,032 | 90 | 41 | 31.3% | 98.1% | 4.2% | 95.8% | Not measured |
| MiniCPM5-2B off · chat Open · 2.5B | — | 2,163 | 2,076 | 60 | 27 | 31.0% | 98.8% | 2.8% | 97.2% | Not measured |
| LFM2.5-2.6B on · chat Open · 2.7B | — | 2,163 | 2,072 | 67 | 24 | 26.4% | 98.9% | 3.1% | 96.9% | Not measured |
| Gemma 4 E2B chat Open · 5.1B | — | 2,163 | 2,088 | 56 | 19 | 25.3% | 99.1% | 2.6% | 97.4% | Not measured |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | 2,163 | 2,102 | 47 | 14 | 23.0% | 99.4% | 2.2% | 97.8% | Not measured |
| Raon-1 nano off · chat Open · 3.1B | — | 2,163 | 1,959 | 159 | 45 | 22.1% | 97.9% | 7.5% | 92.5% | Not measured |
| Gemma 4 31B chat Open · 31B | — | 2,163 | 2,149 | 11 | 3 | 21.4% | 99.9% | 0.5% | 99.5% | Not measured |
| Qwen3.5-4B on · chat Open · 4.7B | — | 2,163 | 2,149 | 11 | 3 | 21.4% | 99.9% | 0.5% | 99.5% | Not measured |
| Qwen3.5-2B on · chat Open · 2.0B | — | 2,163 | 2,099 | 51 | 13 | 20.3% | 99.4% | 2.4% | 97.6% | Not measured |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | 2,163 | 2,153 | 8 | 2 | 20.0% | 99.9% | 0.4% | 99.6% | Not measured |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | 2,163 | 2,140 | 19 | 4 | 17.4% | 99.8% | 0.9% | 99.1% | Not measured |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | 2,163 | 2,145 | 15 | 3 | 16.7% | 99.9% | 0.7% | 99.3% | Not measured |
| Gemma 4 E2B on · chat Open · 5.1B | — | 2,163 | 2,084 | 66 | 13 | 16.5% | 99.4% | 3.1% | 96.9% | Not measured |
| Gemma 4 12B Unified chat Open · 12B | — | 2,163 | 2,142 | 18 | 3 | 14.3% | 99.9% | 0.8% | 99.2% | Not measured |
| Qwen3.8-27B low · chat Open · 27B | — | 2,163 | 2,140 | 20 | 3 | 13.0% | 99.9% | 0.9% | 99.1% | Not measured |
The F option not measured for 5 setups: DeepSeek-V4-Flash (chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E2B (on · chat · 32k cap), Raon-1 nano (on · chat).
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC); Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)); Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC); Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test); GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 62.5% · answered 99.8% Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 54.5% · answered 99.7% Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 60.0% · answered 99.7% Logic 95.2% | — | 99.7%99.4%–99.9% | abc | 2,157 / 2,163 | Not measured | Not measured | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 62.5% · answered 99.8% Logic 95.5% | — | 99.7%99.3%–99.8% | abd | 2,156 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Non-hallucination with “I don't know” 20.0% · answered 99.9% Logic 96.2% | — | 99.6%99.3%–99.8% | abcdef | 2,155 / 2,163 | Not measured | Not measured | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 78.6% · answered 99.5% Logic 96.7% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 78.6% · answered 99.5% Logic 96.5% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Non-hallucination with “I don't know” 54.5% · answered 99.7% Logic 95.4% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 75.0% · answered 99.4% Logic 96.7% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Non-hallucination with “I don't know” 85.7% · answered 99.2% Logic 96.1% | — | 99.6%99.2%–99.8% | abcdef | 2,154 / 2,163 | Not measured | Not measured | 85.7%answered 99% | 4,400 ms | 6,680 ms | — |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 64.3% · answered 99.6% Logic 95.5% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 97.5% · answered 98.0% Logic running | — | 99.5%99.2%–99.7% | abcdefg | 2,153 / 2,163 | Not measured | Not measured | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Non-hallucination with “I don't know” 61.5% · answered 99.6% Logic 95.6% | — | 99.5%99.2%–99.7% | abcdef | 2,153 / 2,163 | Not measured | Not measured | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 16.7% · answered 99.9% Logic 94.0% | — | 99.5%99.1%–99.7% | abcdefgh | 2,152 / 2,163 | Not measured | Not measured | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Non-hallucination with “I don't know” 50.0% · answered 99.7% Logic 95.1% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Non-hallucination with “I don't know” 21.4% · answered 99.9% Logic 92.4% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Non-hallucination with “I don't know” 21.4% · answered 99.9% Logic 93.3% | — | 99.4%99.0%–99.6% | abcdefgh | 2,150 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Logic running | — | 99.3%98.9%–99.6% | abcdefghij | 2,148 / 2,163 | Not measured | Not measured | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Non-hallucination with “I don't know” 53.6% · answered 99.3% Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | abcdefghi | 2,147 / 2,163seed 0 run | Not measured | Not measured | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 77.8% · answered 99.0% Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Non-hallucination with “I don't know” 61.5% · answered 99.3% Logic 91.6% | — | 99.2%98.7%–99.5% | abcdefghi | 2,146 / 2,163 | Not measured | Not measured | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Non-hallucination with “I don't know” 13.0% · answered 99.9% Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Non-hallucination with “I don't know” 71.4% · answered 98.8% Logic 86.7% | — | 99.0%98.5%–99.4% | bcdefghij | 2,142 / 2,163 | Not measured | Not measured | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abcdefghij | 2,142 / 2,163seed 0 run | Not measured | Not measured | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Non-hallucination with “I don't know” 67.6% · answered 98.9% Logic 84.2% | — | 98.9%98.8%–99.1% min–max of 3 seeds | cefghijk | 2,139 / 2,163seed 0 run | Not measured | Not measured | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Non-hallucination with “I don't know” 14.3% · answered 99.9% Logic 87.7% | — | 98.8%98.3%–99.2% | dfghijkl | 2,138 / 2,163 | Not measured | Not measured | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Non-hallucination with “I don't know” 17.4% · answered 99.8% Logic 93.2% | — | 98.8%98.2%–99.2% | fghijkl | 2,137 / 2,163 | Not measured | Not measured | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Non-hallucination with “I don't know” 67.7% · answered 99.0% Logic 85.5% | — | 98.7%98.1%–99.1% | ghijkl | 2,135 / 2,163 | Not measured | Not measured | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 45.5% · answered 99.1% Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | efghijkl | 2,136 / 2,163seed 0 run | Not measured | Not measured | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% | — | 98.5%97.9%–99.0% | hijkl | 2,131 / 2,163 | Not measured | Not measured | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Non-hallucination with “I don't know” 36.0% · answered 99.6% Logic 90.1% | — | 98.5%97.9%–98.9% | hijklm | 2,130 / 2,163 | Not measured | Not measured | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 37.0% · answered 99.2% Logic 82.6% | — | 98.2%97.6%–98.7% | ijklm | 2,125 / 2,163 | Not measured | Not measured | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Non-hallucination with “I don't know” 40.6% · answered 99.4% Logic 94.0% | — | 98.2%97.6%–98.7% | ijklmn | 2,125 / 2,163 | Not measured | Not measured | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Non-hallucination with “I don't know” 31.0% · answered 98.8% Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | ijklmn | 2,126 / 2,163seed 0 run | Not measured | Not measured | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 44.0% · answered 99.0% Logic 83.2% | — | 98.1%97.4%–98.6% | jklmn | 2,122 / 2,163 | Not measured | Not measured | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% | — | 97.8%97.1%–98.4% | klmno | 2,116 / 2,163 | Not measured | Not measured | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 23.0% · answered 99.4% Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | lmnop | 2,111 / 2,163seed 0 run | Not measured | Not measured | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% | — | 97.0%96.2%–97.6% | mnop | 2,098 / 2,163 | Not measured | Not measured | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 16.5% · answered 99.4% Logic 74.6% | — | 96.9%96.1%–97.6% | mnop | 2,097 / 2,163 | Not measured | Not measured | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 25.3% · answered 99.1% Logic 72.9% | — | 96.8%95.9%–97.4% | nopq | 2,093 / 2,163 | Not measured | Not measured | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Non-hallucination with “I don't know” 45.2% · answered 97.4% Logic 60.2% | — | 96.3%95.4%–97.0% | opqr | 2,082 / 2,163 | Not measured | Not measured | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Non-hallucination with “I don't know” 39.3% · answered 97.9% Logic 65.3% | — | 95.8%94.9%–96.6% | pqrs | 2,072 / 2,163 | Not measured | Not measured | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Non-hallucination with “I don't know” 20.3% · answered 99.4% Logic 70.3% | — | 94.7%93.7%–95.6% | qrs | 2,049 / 2,163 | Not measured | Not measured | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Non-hallucination with “I don't know” 31.3% · answered 98.1% Logic 64.8% | — | 94.6%93.6%–95.5% | rs | 2,047 / 2,163 | Not measured | Not measured | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Non-hallucination with “I don't know” 26.4% · answered 98.9% Logic 75.4% | — | 94.0%92.9%–94.9% | rs | 2,033 / 2,163 | Not measured | Not measured | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 68.9% · answered 95.3% Logic 58.0% | — | 93.7%92.6%–94.6% | s | 2,026 / 2,163 | Not measured | Not measured | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Non-hallucination with “I don't know” 35.9% · answered 95.7% Logic 54.2% | — | 90.8%89.5%–91.9% | t | 1,963 / 2,163 | Not measured | Not measured | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Non-hallucination with “I don't know” 22.1% · answered 97.9% Logic 65.1% | — | 89.6%88.2%–90.8% | t | 1,937 / 2,163 | Not measured | Not measured | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Other IBM OpenBMB Alibaba Liquid AI Google KRAFTON A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| LFM3-1.7B off · chat Open · 1.7B | — | 2,163 | 2,132 | 10 | 21 | 67.7% | 99.0% | 0.5% | 99.5% | Not measured |
| LFM3-1.7B on · chat Open · 1.7B | — | 2,163 | 2,129 | 11 | 23 | 67.6% | 98.9% | 0.5% | 99.5% | Not measured |
| Granite-4.2-3B high · chat Open · 3.0B | — | 2,163 | 2,137 | 10 | 16 | 61.5% | 99.3% | 0.5% | 99.5% | Not measured |
| MiniCPM5-2B on · chat Open · 2.5B | — | 2,163 | 2,135 | 13 | 15 | 53.6% | 99.3% | 0.6% | 99.4% | Not measured |
| Qwen3.5-4B off · chat Open · 4.7B | — | 2,163 | 2,138 | 16 | 9 | 36.0% | 99.6% | 0.7% | 99.3% | Not measured |
| Qwen3.5-2B off · chat Open · 2.0B | — | 2,163 | 2,032 | 90 | 41 | 31.3% | 98.1% | 4.2% | 95.8% | Not measured |
| MiniCPM5-2B off · chat Open · 2.5B | — | 2,163 | 2,076 | 60 | 27 | 31.0% | 98.8% | 2.8% | 97.2% | Not measured |
| LFM2.5-2.6B on · chat Open · 2.7B | — | 2,163 | 2,072 | 67 | 24 | 26.4% | 98.9% | 3.1% | 96.9% | Not measured |
| Gemma 4 E2B chat Open · 5.1B | — | 2,163 | 2,088 | 56 | 19 | 25.3% | 99.1% | 2.6% | 97.4% | Not measured |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | 2,163 | 2,102 | 47 | 14 | 23.0% | 99.4% | 2.2% | 97.8% | Not measured |
| Raon-1 nano off · chat Open · 3.1B | — | 2,163 | 1,959 | 159 | 45 | 22.1% | 97.9% | 7.5% | 92.5% | Not measured |
| Qwen3.5-4B on · chat Open · 4.7B | — | 2,163 | 2,149 | 11 | 3 | 21.4% | 99.9% | 0.5% | 99.5% | Not measured |
| Qwen3.5-2B on · chat Open · 2.0B | — | 2,163 | 2,099 | 51 | 13 | 20.3% | 99.4% | 2.4% | 97.6% | Not measured |
| Gemma 4 E2B on · chat Open · 5.1B | — | 2,163 | 2,084 | 66 | 13 | 16.5% | 99.4% | 3.1% | 96.9% | Not measured |
The F option not measured for 3 setups: LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), Gemma 4 E2B (on · chat · 32k cap), Raon-1 nano (on · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Non-hallucination with “I don't know” 21.4% · answered 99.9% Logic 92.4% | — | 99.4%99.0%–99.7% | a | 2,151 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Non-hallucination with “I don't know” 53.6% · answered 99.3% Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | ab | 2,147 / 2,163seed 0 run | Not measured | Not measured | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Non-hallucination with “I don't know” 61.5% · answered 99.3% Logic 91.6% | — | 99.2%98.7%–99.5% | ab | 2,146 / 2,163 | Not measured | Not measured | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abc | 2,142 / 2,163seed 0 run | Not measured | Not measured | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Non-hallucination with “I don't know” 67.6% · answered 98.9% Logic 84.2% | — | 98.9%98.8%–99.1% min–max of 3 seeds | abc | 2,139 / 2,163seed 0 run | Not measured | Not measured | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Non-hallucination with “I don't know” 67.7% · answered 99.0% Logic 85.5% | — | 98.7%98.1%–99.1% | bcd | 2,135 / 2,163 | Not measured | Not measured | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Non-hallucination with “I don't know” 36.0% · answered 99.6% Logic 90.1% | — | 98.5%97.9%–98.9% | bcde | 2,130 / 2,163 | Not measured | Not measured | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Non-hallucination with “I don't know” 31.0% · answered 98.8% Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | cdef | 2,126 / 2,163seed 0 run | Not measured | Not measured | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% | — | 97.8%97.1%–98.4% | defg | 2,116 / 2,163 | Not measured | Not measured | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 23.0% · answered 99.4% Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | efg | 2,111 / 2,163seed 0 run | Not measured | Not measured | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% | — | 97.0%96.2%–97.6% | fg | 2,098 / 2,163 | Not measured | Not measured | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 16.5% · answered 99.4% Logic 74.6% | — | 96.9%96.1%–97.6% | g | 2,097 / 2,163 | Not measured | Not measured | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Non-hallucination with “I don't know” 25.3% · answered 99.1% Logic 72.9% | — | 96.8%95.9%–97.4% | g | 2,093 / 2,163 | Not measured | Not measured | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Non-hallucination with “I don't know” 20.3% · answered 99.4% Logic 70.3% | — | 94.7%93.7%–95.6% | h | 2,049 / 2,163 | Not measured | Not measured | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Non-hallucination with “I don't know” 31.3% · answered 98.1% Logic 64.8% | — | 94.6%93.6%–95.5% | h | 2,047 / 2,163 | Not measured | Not measured | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Non-hallucination with “I don't know” 26.4% · answered 98.9% Logic 75.4% | — | 94.0%92.9%–94.9% | h | 2,033 / 2,163 | Not measured | Not measured | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Non-hallucination with “I don't know” 22.1% · answered 97.9% Logic 65.1% | — | 89.6%88.2%–90.8% | i | 1,937 / 2,163 | Not measured | Not measured | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Google Liquid AI Alibaba eCloud Technology A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | 2,163 | 2,119 | 24 | 20 | 45.5% | 99.1% | 1.1% | 98.9% | Not measured |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 2,163 | 2,039 | 68 | 56 | 45.2% | 97.4% | 3.2% | 96.8% | Not measured |
| Gemma 4 E4B chat Open · 8.0B | — | 2,163 | 2,113 | 28 | 22 | 44.0% | 99.0% | 1.3% | 98.7% | Not measured |
| Qwen3-14B off · chat Open · 15B | — | 2,163 | 2,046 | 71 | 46 | 39.3% | 97.9% | 3.4% | 96.6% | Not measured |
| Gemma 4 E4B on · chat Open · 8.0B | — | 2,163 | 2,117 | 29 | 17 | 37.0% | 99.2% | 1.4% | 98.6% | Not measured |
| Erk-14B off · chat Open · 15B | — | 2,163 | 1,901 | 168 | 94 | 35.9% | 95.7% | 8.1% | 91.9% | Not measured |
| Gemma 4 12B Unified chat Open · 12B | — | 2,163 | 2,142 | 18 | 3 | 14.3% | 99.9% | 0.8% | 99.2% | Not measured |
The F option not measured for 1 setup: Gemma 4 E4B (on · chat · 32k cap).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Non-hallucination with “I don't know” 14.3% · answered 99.9% Logic 87.7% | — | 98.8%98.3%–99.2% | a | 2,138 / 2,163 | Not measured | Not measured | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 45.5% · answered 99.1% Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | a | 2,136 / 2,163seed 0 run | Not measured | Not measured | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% | — | 98.5%97.9%–99.0% | a | 2,131 / 2,163 | Not measured | Not measured | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 37.0% · answered 99.2% Logic 82.6% | — | 98.2%97.6%–98.7% | a | 2,125 / 2,163 | Not measured | Not measured | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Non-hallucination with “I don't know” 44.0% · answered 99.0% Logic 83.2% | — | 98.1%97.4%–98.6% | a | 2,122 / 2,163 | Not measured | Not measured | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Non-hallucination with “I don't know” 45.2% · answered 97.4% Logic 60.2% | — | 96.3%95.4%–97.0% | b | 2,082 / 2,163 | Not measured | Not measured | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Non-hallucination with “I don't know” 39.3% · answered 97.9% Logic 65.3% | — | 95.8%94.9%–96.6% | b | 2,072 / 2,163 | Not measured | Not measured | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Non-hallucination with “I don't know” 35.9% · answered 95.7% Logic 54.2% | — | 90.8%89.5%–91.9% | c | 1,963 / 2,163 | Not measured | Not measured | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Liquid AI Google Alibaba A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | 2,163 | 2,128 | 10 | 25 | 71.4% | 98.8% | 0.5% | 99.5% | Not measured |
| Gemma 4 31B chat Open · 31B | — | 2,163 | 2,149 | 11 | 3 | 21.4% | 99.9% | 0.5% | 99.5% | Not measured |
| Qwen3.8-27B low · chat Open · 27B | — | 2,163 | 2,140 | 20 | 3 | 13.0% | 99.9% | 0.9% | 99.1% | Not measured |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Non-hallucination with “I don't know” 21.4% · answered 99.9% Logic 93.3% | — | 99.4%99.0%–99.6% | a | 2,150 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Non-hallucination with “I don't know” 13.0% · answered 99.9% Logic 94.0% | — | 99.2%98.7%–99.5% | a | 2,146 / 2,163 | Not measured | Not measured | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Non-hallucination with “I don't know” 71.4% · answered 98.8% Logic 86.7% | — | 99.0%98.5%–99.4% | a | 2,142 / 2,163 | Not measured | Not measured | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Thinking Machines Xiaomi MiniMax DeepSeek Z.ai A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | 2,163 | 2,150 | 5 | 8 | 61.5% | 99.6% | 0.2% | 99.8% | Not measured |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | 2,163 | 2,152 | 5 | 6 | 54.5% | 99.7% | 0.2% | 99.8% | Not measured |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | 2,163 | 2,149 | 7 | 7 | 50.0% | 99.7% | 0.3% | 99.7% | Not measured |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | 2,163 | 2,131 | 19 | 13 | 40.6% | 99.4% | 0.9% | 99.1% | Not measured |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | 2,163 | 2,153 | 8 | 2 | 20.0% | 99.9% | 0.4% | 99.6% | Not measured |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | 2,163 | 2,140 | 19 | 4 | 17.4% | 99.8% | 0.9% | 99.1% | Not measured |
The F option not measured for 1 setup: DeepSeek-V4-Flash (chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Non-hallucination with “I don't know” 20.0% · answered 99.9% Logic 96.2% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Non-hallucination with “I don't know” 54.5% · answered 99.7% Logic 95.4% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Non-hallucination with “I don't know” 61.5% · answered 99.6% Logic 95.6% | — | 99.5%99.2%–99.7% | a | 2,153 / 2,163 | Not measured | Not measured | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Non-hallucination with “I don't know” 50.0% · answered 99.7% Logic 95.1% | — | 99.4%99.0%–99.7% | ab | 2,151 / 2,163 | Not measured | Not measured | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Logic running | — | 99.3%98.9%–99.6% | ab | 2,148 / 2,163 | Not measured | Not measured | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Non-hallucination with “I don't know” 17.4% · answered 99.8% Logic 93.2% | — | 98.8%98.2%–99.2% | bc | 2,137 / 2,163 | Not measured | Not measured | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Non-hallucination with “I don't know” 40.6% · answered 99.4% Logic 94.0% | — | 98.2%97.6%–98.7% | c | 2,125 / 2,163 | Not measured | Not measured | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
OpenAI Meta A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Luna chat Closed · Efficient | — | 1,075 | 1,001 | 23 | 51 | 68.9% | 95.3% | 2.2% | 97.8% | Not measured |
| Muse Ember max · chat Closed · Efficient | — | 2,163 | 2,149 | 5 | 9 | 64.3% | 99.6% | 0.2% | 99.8% | Not measured |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 64.3% · answered 99.6% Logic 95.5% | — | 99.6%99.2%–99.8% | a | 2,154 / 2,163 | Not measured | Not measured | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 68.9% · answered 95.3% Logic 58.0% | — | 93.7%92.6%–94.6% | b | 2,026 / 2,163 | Not measured | Not measured | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Anthropic A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse | — | 2,163 | 2,155 | 3 | 5 | 62.5% | 99.8% | 0.1% | 99.9% | Not measured |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 62.5% · answered 99.8% Logic 95.5% | — | 99.7%99.3%–99.8% | a | 2,156 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
Anthropic A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship | — | 2,163 | 2,155 | 3 | 5 | 62.5% | 99.8% | 0.1% | 99.9% | Not measured |
| Claude Fable 5.1 chat Closed · Flagship | — | 2,163 | 2,153 | 4 | 6 | 60.0% | 99.7% | 0.2% | 99.8% | Not measured |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 62.5% · answered 99.8% Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 60.0% · answered 99.7% Logic 95.2% | — | 99.7%99.4%–99.9% | a | 2,157 / 2,163 | Not measured | Not measured | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Explicit “I don't know” option
Non-hallucination with “I don't know”
F among the answers that were not right (F / (F + wrong)), with F offered as a sixth option; beside it the share answered (right + wrong of N) and the error rate among the answers (wrong / (right + wrong)) · higher non-hallucination is better, read with the share answered
xAI OpenAI Google Alibaba A setup that always chose F would reach 100% with no right answers.
Counts with F on offer
| Setup | Web | Nquestions answered | Right | Wronga letter A–E, or none | I don't knowoption F | Non-hallucinationof answers not right, % “I don't know” · higher is better · F / (F + wrong) | Answerednot “I don't know”: (right + wrong) / N | Error rateof its answers: wrong / (right + wrong) · lower is better | Accuracy when answeringright / (right + wrong) | Held back by confidencenon-hallucination at the same attempt rate |
|---|---|---|---|---|---|---|---|---|---|---|
| Grok 4.7 chat Closed · Undisclosed | — | 1,987 | 1,947 | 1 | 39 | 97.5% | 98.0% | 0.1% | 99.9% | Not measured |
| GPT-6 Astra chat Closed · Undisclosed | — | 2,163 | 2,149 | 3 | 11 | 78.6% | 99.5% | 0.1% | 99.9% | Not measured |
| GPT-6.1 Sol chat Closed · Undisclosed | — | 2,163 | 2,149 | 3 | 11 | 78.6% | 99.5% | 0.1% | 99.9% | Not measured |
| GPT-6.1 Terra chat Closed · Undisclosed | — | 2,163 | 2,136 | 6 | 21 | 77.8% | 99.0% | 0.3% | 99.7% | Not measured |
| Gemini 4 Carbon chat Closed · Undisclosed | — | 2,163 | 2,147 | 4 | 12 | 75.0% | 99.4% | 0.2% | 99.8% | Not measured |
| Qwen4-Max high · chat Closed · Undisclosed | — | 2,163 | 2,152 | 5 | 6 | 54.5% | 99.7% | 0.2% | 99.8% | Not measured |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | 2,163 | 2,145 | 15 | 3 | 16.7% | 99.9% | 0.7% | 99.3% | Not measured |
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 54.5% · answered 99.7% Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 78.6% · answered 99.5% Logic 96.7% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 78.6% · answered 99.5% Logic 96.5% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 75.0% · answered 99.4% Logic 96.7% | — | 99.6%99.2%–99.8% | ab | 2,154 / 2,163 | Not measured | Not measured | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 97.5% · answered 98.0% Logic running | — | 99.5%99.2%–99.7% | ab | 2,153 / 2,163 | Not measured | Not measured | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Non-hallucination with “I don't know” 16.7% · answered 99.9% Logic 94.0% | — | 99.5%99.1%–99.7% | ab | 2,152 / 2,163 | Not measured | Not measured | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Non-hallucination with “I don't know” 77.8% · answered 99.0% Logic 94.0% | — | 99.2%98.7%–99.5% | b | 2,146 / 2,163 | Not measured | Not measured | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.