Math · Confidence
Confidence on Math
2,163 questions, asked in English. How to read this page · Method
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
MachinePulse Base Labs Perplexity Poindexter Labs Nace.AI TypeSafe Sthānika AI Google Liquid AI REAX Fastino Valen-Team PelaAI StartLux tattrongvu Turion.AI scar-ai Microsoft OmniJev Logits Cloudflare vLLM Semantic Router Blockbrain Labs Other Metask Lab Zefan Cai Strands Agents Celeris Subquadratic Inception Apertix AI Patrick-18 newsbang quincyqiang victorbennett EldanRing Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | Decision | 81.6%2,163 | 88.9%86.7% kept (1,876) | 94.2%73.7% kept (1,594) | 98.0%58.1% kept (1,256) | 100.0%38.7% kept (837) |
| Mammoth-1 Open · 27B | — | Decision | 74.0%2,163 | 85.3%79.1% kept (1,712) | 94.3%61.8% kept (1,336) | 98.5%45.7% kept (988) | 100.0%19.0% kept (412) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 69.8%2,163 | 83.0%73.5% kept (1,590) | 93.1%56.3% kept (1,217) | 98.2%38.7% kept (838) | 100.0%15.3% kept (331) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 69.3%2,163 | 82.5%73.5% kept (1,590) | 92.8%56.2% kept (1,215) | 97.7%38.8% kept (840) | 100.0%15.1% kept (326) |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 72.5%2,163 | 81.2%81.1% kept (1,754) | 90.0%64.0% kept (1,384) | 96.3%46.1% kept (998) | 100.0%14.9% kept (323) |
| Copernicus-31B Open · 31B | — | Decision | 77.3%2,163 | 91.3%73.4% kept (1,587) | 97.2%57.5% kept (1,244) | 99.4%39.0% kept (843) | 100.0%14.7% kept (318) |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 70.1%2,163 | 79.7%79.8% kept (1,725) | 89.3%62.5% kept (1,352) | 97.3%43.4% kept (938) | 100.0%14.6% kept (315) |
| Jev 1.13 Closed · Efficient | — | Decision | 64.1%2,163 | 80.3%65.2% kept (1,410) | 91.3%42.7% kept (924) | 98.3%25.1% kept (543) | 100.0%11.5% kept (248) |
| Sieve 31B v3 Open · 31B | — | Decision | 77.1%2,163 | 90.0%75.3% kept (1,628) | 95.9%59.6% kept (1,289) | 99.3%41.1% kept (888) | 100.0%9.9% kept (215) |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 71.3%2,163 | 85.3%72.1% kept (1,560) | 92.7%55.2% kept (1,194) | 98.2%36.3% kept (785) | 100.0%9.9% kept (214) |
| d1-ultra Closed · Efficient | — | Decision | 74.1%2,163 | 87.8%73.4% kept (1,588) | 94.0%59.0% kept (1,276) | 98.5%40.8% kept (882) | 100.0%9.8% kept (213) |
| Jev 1.14 Closed · Efficient | — | Decision | 65.1%2,163 | 82.2%61.7% kept (1,335) | 92.8%40.5% kept (876) | 98.3%24.0% kept (519) | 100.0%9.3% kept (202) |
| d1.1 Closed · Efficient | — | Decision | 65.2%2,163 | 82.7%61.1% kept (1,322) | 92.6%41.5% kept (897) | 97.6%24.9% kept (538) | 100.0%8.5% kept (183) |
| REAX 1.3 Open · 57B | — | Decision | 78.0%2,163 | 90.9%76.9% kept (1,664) | 97.5%58.5% kept (1,265) | 99.9%40.5% kept (877) | 100.0%8.2% kept (178) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 59.0%2,163 | 80.4%53.0% kept (1,147) | 92.2%35.0% kept (758) | 97.8%20.6% kept (445) | 100.0%8.0% kept (174) |
| GLiDE API Closed · Efficient | — | Decision | 70.8%2,163 | 81.6%77.6% kept (1,678) | 92.0%58.2% kept (1,258) | 98.6%37.4% kept (808) | 100.0%7.4% kept (161) |
| d1 Closed · Efficient | — | Decision | 55.6%2,163 | 73.6%55.1% kept (1,192) | 85.9%32.1% kept (694) | 95.4%17.0% kept (368) | 100.0%7.0% kept (152) |
| d1.2 Closed · Undisclosed | — | Decision | 65.5%2,163 | 83.5%60.9% kept (1,318) | 94.4%41.3% kept (894) | 99.2%23.7% kept (512) | 100.0%6.5% kept (141) |
| GLiDE v1.2 Open · 27B | — | Decision | 69.9%2,163 | 81.1%75.9% kept (1,641) | 92.6%56.4% kept (1,219) | 99.1%34.4% kept (745) | 100.0%6.1% kept (131) |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 55.3%2,163 | 74.3%54.9% kept (1,187) | 86.7%32.3% kept (698) | 94.9%17.3% kept (374) | 100.0%6.0% kept (130) |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 59.8%2,163 | 76.7%58.4% kept (1,264) | 89.3%36.8% kept (797) | 98.0%20.5% kept (444) | 100.0%5.7% kept (123) |
| d1.2-ultra Closed · Undisclosed | — | Decision | 69.9%2,163 | 87.4%66.6% kept (1,441) | 94.8%48.6% kept (1,052) | 99.7%30.5% kept (660) | 100.0%5.3% kept (115) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 55.8%2,163 | 70.8%62.3% kept (1,348) | 81.1%40.5% kept (876) | 94.0%20.8% kept (450) | 100.0%4.4% kept (96) |
| GLiDE 9B Open · 7.9B | — | Decision | 57.9%2,163 | 70.8%68.3% kept (1,478) | 84.7%42.8% kept (926) | 94.7%20.9% kept (451) | 100.0%3.4% kept (73) |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 59.8%2,163 | 77.8%60.3% kept (1,304) | 90.4%38.1% kept (824) | 98.5%18.2% kept (393) | 100.0%3.0% kept (64) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 56.0%2,163 | 67.1%67.4% kept (1,457) | 79.3%41.9% kept (907) | 92.8%21.3% kept (460) | 100.0%2.5% kept (54) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 58.4%2,163 | 74.9%58.2% kept (1,258) | 85.1%37.4% kept (808) | 96.6%19.1% kept (413) | 100.0%2.3% kept (49) |
| Turion Jev 31B Open · 31B | — | Decision | 69.0%2,163 | 84.7%68.7% kept (1,487) | 92.8%51.1% kept (1,106) | 99.8%26.0% kept (562) | 100.0%2.0% kept (43) |
| vera-medium-omni Open · 4.2B | — | Decision | 56.8%2,163 | 75.4%55.2% kept (1,193) | 88.5%29.6% kept (641) | 98.9%12.3% kept (267) | 100.0%1.5% kept (33) |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 59.4%2,163 | 73.4%63.9% kept (1,382) | 88.5%36.2% kept (782) | 98.0%13.8% kept (299) | 100.0%1.2% kept (27) |
| OneJev-27B Open · 27B | — | Decision | 53.8%2,163 | 81.8%39.5% kept (855) | 92.5%23.3% kept (504) | 98.3%10.6% kept (229) | 100.0%1.2% kept (26) |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 75.2%2,163 | 89.2%73.5% kept (1,589) | 98.3%52.4% kept (1,134) | 100.0%26.7% kept (578) | 100.0%0.7% kept (15) |
| Clef Open · 27B | — | Decision | 51.1%2,163 | 65.2%62.2% kept (1,346) | 78.9%36.8% kept (796) | 90.8%17.6% kept (381) | 100.0%0.6% kept (12) |
| d3-ultra Open · 26B | — | Decision | 71.5%2,163 | 78.5%85.2% kept (1,842) | 86.2%68.6% kept (1,484) | 97.4%43.7% kept (946) | 100.0%0.5% kept (11) |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 35.3%2,163 | 52.1%28.2% kept (609) | 67.2%8.9% kept (192) | 88.5%2.4% kept (52) | 100.0%0.4% kept (8) |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 66.3%2,163 | 83.7%60.8% kept (1,315) | 92.1%42.2% kept (912) | 99.1%19.6% kept (425) | 100.0%0.3% kept (7) |
| GLiDE 0.8B Open · 0.8B | — | Decision | 32.1%2,163 | 38.1%44.3% kept (959) | 47.5%16.8% kept (364) | 73.0%2.9% kept (63) | 100.0%0.2% kept (4) |
| d1-2B Open · 1.9B | — | Decision | 37.1%2,163 | 58.0%25.0% kept (541) | 74.5%7.4% kept (161) | 83.7%2.3% kept (49) | 100.0%0.2% kept (4) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 38.7%2,163 | 68.1%21.1% kept (457) | 83.6%7.9% kept (171) | 92.0%2.3% kept (50) | 100.0%0.1% kept (2) |
| Clef-Flash Open · 9.0B | — | Decision | 49.8%2,163 | 68.7%44.7% kept (967) | 81.3%23.3% kept (504) | 94.4%7.4% kept (161) | none kept |
| Cygnet Open · 12B | — | Decision | 44.3%2,163 | 65.9%41.9% kept (907) | 84.2%14.9% kept (322) | 98.7%3.5% kept (75) | none kept |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 22.7%2,163 | 26.9%3.6% kept (78) | 38.5%0.6% kept (13) | 100.0%0.0% kept (1) | none kept |
| GLiDE 0.4B Open · 0.4B | — | Decision | 20.6%2,163 | 23.5%5.5% kept (119) | 33.3%1.0% kept (21) | none kept | none kept |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 28.2%2,163 | 41.7%15.5% kept (336) | 54.9%3.3% kept (71) | 75.0%0.2% kept (4) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 60.1%2,163 | 80.7%52.3% kept (1,131) | 92.5%30.9% kept (668) | 99.6%10.3% kept (223) | none kept |
| Metask-Jev-4B Open · 4.5B | — | Decision | 39.2%2,163 | 63.1%19.8% kept (428) | 77.5%8.6% kept (187) | 88.2%1.6% kept (34) | none kept |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 28.2%2,163 | 41.8%15.3% kept (330) | 54.1%3.4% kept (74) | 66.7%0.7% kept (15) | none kept |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 54.6%2,163 | 95.8%10.0% kept (216) | 100.0%1.2% kept (25) | none kept | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 29.7%2,163 | 45.9%10.6% kept (229) | 61.0%3.6% kept (77) | 92.3%0.6% kept (13) | none kept |
| Strands Decider 2B Open · 2.0B | — | Decision | 28.7%2,163 | 45.9%11.8% kept (255) | 69.2%1.2% kept (26) | 100.0%0.0% kept (1) | none kept |
| celeris-1-decision Closed · Efficient | — | Decision | 70.0%2,163 | 80.9%77.1% kept (1,667) | 90.9%58.3% kept (1,262) | 96.5%40.4% kept (874) | none kept |
| celeris-1.1-decision Closed · Efficient | — | Decision | 69.9%2,163 | 80.9%77.2% kept (1,669) | 90.8%58.3% kept (1,262) | 96.7%40.5% kept (876) | none kept |
| Drex Max Open · 27B | — | Decision | 70.3%2,163 | 76.2%86.7% kept (1,875) | 83.3%72.2% kept (1,561) | 92.9%55.9% kept (1,209) | 98.6%33.7% kept (729) |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 80.3%2,163 | 90.0%80.7% kept (1,746) | 95.3%66.5% kept (1,438) | 99.3%48.1% kept (1,040) | 99.8%21.7% kept (470) |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 77.4%2,163 | 89.5%77.0% kept (1,665) | 95.3%61.3% kept (1,327) | 99.1%42.7% kept (923) | 99.8%21.5% kept (465) |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 68.6%2,163 | 76.4%82.0% kept (1,774) | 84.1%66.4% kept (1,436) | 93.7%46.3% kept (1,002) | 99.7%16.7% kept (361) |
| Mercury Decide Closed · Efficient | — | Decision | 53.5%2,163 | 65.1%71.2% kept (1,539) | 75.3%49.8% kept (1,077) | 85.3%32.0% kept (692) | 96.8%13.0% kept (281) |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 56.0%2,163 | 64.6%75.7% kept (1,638) | 74.3%55.1% kept (1,192) | 86.2%33.8% kept (731) | 96.1%11.7% kept (254) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 58.8%2,163 | 69.2%74.8% kept (1,619) | 79.2%53.7% kept (1,162) | 91.9%32.1% kept (694) | 98.6%9.7% kept (210) |
| GLiDE 27B Open · 27B | — | Decision | 72.0%2,163 | 82.1%78.7% kept (1,703) | 91.2%60.6% kept (1,311) | 98.7%39.5% kept (855) | 99.5%9.2% kept (199) |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 62.1%2,163 | 74.7%68.7% kept (1,486) | 86.2%46.0% kept (994) | 96.3%27.2% kept (589) | 99.5%9.2% kept (198) |
| Elephant-1 Open · 9.0B | — | Decision | 61.3%2,163 | 75.7%65.6% kept (1,419) | 86.7%44.0% kept (952) | 95.8%26.2% kept (567) | 99.5%8.5% kept (183) |
| d3 Open · 26B | — | Decision | 70.4%2,163 | 85.3%71.8% kept (1,553) | 95.2%53.7% kept (1,161) | 99.3%34.0% kept (735) | 99.4%7.5% kept (163) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 55.2%2,163 | 69.1%65.3% kept (1,412) | 79.3%46.3% kept (1,001) | 88.3%25.8% kept (557) | 96.8%7.2% kept (155) |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 50.4%2,163 | 60.9%66.6% kept (1,440) | 74.3%39.1% kept (845) | 86.2%20.4% kept (442) | 96.1%7.2% kept (155) |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 51.5%2,163 | 69.1%51.5% kept (1,115) | 81.6%29.9% kept (646) | 91.6%16.0% kept (346) | 96.9%6.0% kept (130) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 55.9%2,163 | 67.9%69.5% kept (1,504) | 77.7%49.1% kept (1,061) | 89.5%27.7% kept (599) | 99.1%5.3% kept (114) |
| Winnow-12B Open · 12B | — | Decision | 46.9%2,163 | 64.6%47.3% kept (1,023) | 77.3%30.4% kept (657) | 89.5%16.2% kept (351) | 95.2%1.0% kept (21) |
| GLiDE 2B Open · 1.9B | — | Decision | 36.8%2,163 | 45.5%49.8% kept (1,078) | 56.1%21.8% kept (471) | 73.8%5.6% kept (122) | 93.8%0.7% kept (16) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | Decision | 837 / 00.903 | 85 / 00.899 | 34 / 00.801 | 3 / 00.833 | 34 / 00.880 | 38 / 00.933 | 50 / 00.890 | 83 / 00.952 | 45 / 00.879 | 66 / 00.915 | 53 / 00.950 | 91 / 00.902 | 114 / 00.893 | 141 / 00.904 |
| Mammoth-1 Open · 27B | — | Decision | 412 / 00.895 | 27 / 00.885 | 9 / 00.766 | 1 / 00.788 | 11 / 00.821 | 14 / 00.876 | 30 / 00.897 | 41 / 00.909 | 17 / 00.893 | 47 / 00.930 | 31 / 00.904 | 53 / 00.900 | 59 / 00.896 | 72 / 00.903 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 331 / 00.869 | 22 / 00.861 | 6 / 00.818 | 1 / 00.806 | 8 / 00.772 | 10 / 00.823 | 16 / 00.819 | 34 / 00.961 | 34 / 00.901 | 43 / 00.903 | 33 / 00.932 | 32 / 00.840 | 47 / 00.875 | 45 / 00.863 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 326 / 00.873 | 23 / 00.850 | 9 / 00.842 | 1 / 00.571 | 6 / 00.827 | 14 / 00.799 | 18 / 00.854 | 28 / 00.927 | 36 / 00.869 | 43 / 00.887 | 31 / 00.946 | 29 / 00.851 | 41 / 00.885 | 47 / 00.884 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 323 / 00.856 | 26 / 00.828 | 8 / 00.776 | 0 / 00.577 | 8 / 00.785 | 15 / 00.845 | 17 / 00.790 | 36 / 00.935 | 27 / 00.839 | 25 / 00.900 | 22 / 00.933 | 35 / 00.845 | 49 / 00.871 | 55 / 00.834 |
| Copernicus-31B Open · 31B | — | Decision | 318 / 00.899 | 27 / 00.901 | 13 / 00.865 | 1 / 00.800 | 9 / 00.881 | 17 / 00.892 | 20 / 00.881 | 42 / 00.980 | 12 / 00.884 | 25 / 00.876 | 19 / 00.937 | 29 / 00.882 | 48 / 00.914 | 56 / 00.875 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 315 / 00.867 | 23 / 00.839 | 8 / 00.850 | 1 / 00.471 | 14 / 00.799 | 9 / 00.861 | 15 / 00.817 | 28 / 00.903 | 31 / 00.840 | 47 / 00.910 | 30 / 00.919 | 36 / 00.866 | 39 / 00.853 | 34 / 00.893 |
| Jev 1.13 Closed · Efficient | — | Decision | 248 / 00.834 | 17 / 00.836 | 8 / 00.756 | 0 / 00.671 | 9 / 00.761 | 8 / 00.797 | 9 / 00.835 | 23 / 00.894 | 20 / 00.801 | 30 / 00.868 | 17 / 00.831 | 24 / 00.821 | 41 / 00.850 | 42 / 00.818 |
| Sieve 31B v3 Open · 31B | — | Decision | 215 / 00.891 | 24 / 00.871 | 6 / 00.848 | 0 / 00.903 | 7 / 00.856 | 8 / 00.898 | 10 / 00.863 | 26 / 00.956 | 15 / 00.918 | 22 / 00.907 | 14 / 00.939 | 24 / 00.895 | 31 / 00.879 | 28 / 00.848 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 214 / 00.859 | 15 / 00.862 | 6 / 00.794 | 2 / 00.729 | 0 / 00.789 | 5 / 00.854 | 10 / 00.851 | 18 / 00.882 | 13 / 00.797 | 23 / 00.833 | 15 / 00.799 | 31 / 00.859 | 38 / 00.907 | 38 / 00.858 |
| d1-ultra Closed · Efficient | — | Decision | 213 / 00.876 | 23 / 00.865 | 8 / 00.755 | 0 / 00.650 | 5 / 00.813 | 9 / 00.896 | 13 / 00.887 | 12 / 00.910 | 13 / 00.941 | 18 / 00.902 | 13 / 00.893 | 20 / 00.845 | 30 / 00.885 | 49 / 00.888 |
| Jev 1.14 Closed · Efficient | — | Decision | 202 / 00.829 | 16 / 00.854 | 7 / 00.791 | 0 / 00.686 | 8 / 00.700 | 5 / 00.778 | 8 / 00.810 | 18 / 00.910 | 14 / 00.788 | 25 / 00.862 | 18 / 00.826 | 21 / 00.781 | 27 / 00.849 | 35 / 00.834 |
| d1.1 Closed · Efficient | — | Decision | 183 / 00.829 | 13 / 00.811 | 5 / 00.768 | 0 / 00.714 | 4 / 00.721 | 3 / 00.825 | 7 / 00.796 | 16 / 00.848 | 14 / 00.854 | 19 / 00.870 | 15 / 00.849 | 23 / 00.815 | 30 / 00.798 | 34 / 00.835 |
| REAX 1.3 Open · 57B | — | Decision | 178 / 00.903 | 11 / 00.888 | 7 / 00.826 | 1 / 00.818 | 7 / 00.869 | 9 / 00.886 | 11 / 00.917 | 30 / 00.955 | 10 / 00.946 | 14 / 00.881 | 7 / 00.935 | 22 / 00.887 | 25 / 00.913 | 24 / 00.900 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 174 / 00.813 | 8 / 00.819 | 6 / 00.817 | 0 / 00.533 | 0 / 00.740 | 5 / 00.798 | 13 / 00.841 | 20 / 00.905 | 9 / 00.768 | 16 / 00.837 | 14 / 00.766 | 22 / 00.762 | 35 / 00.836 | 26 / 00.803 |
| GLiDE API Closed · Efficient | — | Decision | 161 / 00.865 | 17 / 00.874 | 5 / 00.746 | 1 / 00.625 | 11 / 00.778 | 4 / 00.853 | 4 / 00.840 | 20 / 00.987 | 13 / 00.872 | 29 / 00.885 | 11 / 00.924 | 15 / 00.896 | 17 / 00.811 | 14 / 00.881 |
| d1 Closed · Efficient | — | Decision | 152 / 00.767 | 14 / 00.765 | 5 / 00.653 | 0 / 00.530 | 4 / 00.700 | 3 / 00.789 | 6 / 00.752 | 9 / 00.761 | 11 / 00.726 | 21 / 00.802 | 11 / 00.738 | 20 / 00.769 | 23 / 00.777 | 25 / 00.784 |
| d1.2 Closed · Undisclosed | — | Decision | 141 / 00.838 | 8 / 00.833 | 5 / 00.800 | 0 / 00.557 | 3 / 00.708 | 3 / 00.840 | 9 / 00.794 | 20 / 00.896 | 8 / 00.862 | 12 / 00.843 | 10 / 00.797 | 24 / 00.843 | 20 / 00.838 | 19 / 00.845 |
| GLiDE v1.2 Open · 27B | — | Decision | 131 / 00.862 | 12 / 00.857 | 3 / 00.740 | 0 / 00.528 | 5 / 00.777 | 4 / 00.834 | 5 / 00.838 | 20 / 00.963 | 14 / 00.865 | 22 / 00.912 | 7 / 00.948 | 15 / 00.873 | 14 / 00.849 | 10 / 00.877 |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 130 / 00.790 | 16 / 00.837 | 3 / 00.663 | 0 / 00.667 | 7 / 00.778 | 4 / 00.849 | 5 / 00.753 | 17 / 00.876 | 9 / 00.765 | 12 / 00.777 | 7 / 00.834 | 13 / 00.782 | 16 / 00.795 | 21 / 00.776 |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 123 / 00.799 | 13 / 00.816 | 5 / 00.712 | 0 / 00.595 | 6 / 00.674 | 6 / 00.803 | 5 / 00.760 | 19 / 00.873 | 6 / 00.799 | 14 / 00.792 | 7 / 00.780 | 17 / 00.803 | 11 / 00.821 | 14 / 00.817 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 115 / 00.876 | 6 / 00.891 | 5 / 00.839 | 1 / 00.889 | 4 / 00.774 | 4 / 00.851 | 5 / 00.801 | 18 / 00.960 | 10 / 00.855 | 8 / 00.912 | 10 / 00.902 | 15 / 00.859 | 11 / 00.855 | 18 / 00.868 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 96 / 00.779 | 11 / 00.818 | 1 / 00.707 | 0 / 00.524 | 2 / 00.676 | 2 / 00.743 | 7 / 00.748 | 18 / 00.814 | 6 / 00.808 | 11 / 00.806 | 6 / 00.744 | 12 / 00.768 | 12 / 00.778 | 8 / 00.775 |
| GLiDE 9B Open · 7.9B | — | Decision | 73 / 00.798 | 7 / 00.800 | 2 / 00.737 | 0 / 00.731 | 3 / 00.700 | 1 / 00.716 | 4 / 00.752 | 15 / 00.805 | 3 / 00.774 | 10 / 00.873 | 4 / 00.853 | 7 / 00.776 | 8 / 00.851 | 9 / 00.765 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 64 / 00.814 | 4 / 00.865 | 4 / 00.677 | 0 / 00.769 | 3 / 00.732 | 3 / 00.844 | 3 / 00.745 | 15 / 00.811 | 4 / 00.797 | 6 / 00.888 | 2 / 00.825 | 5 / 00.804 | 7 / 00.808 | 8 / 00.768 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 54 / 00.765 | 3 / 00.791 | 2 / 00.611 | 0 / 00.500 | 1 / 00.688 | 1 / 00.814 | 4 / 00.699 | 11 / 00.845 | 1 / 00.842 | 7 / 00.813 | 1 / 00.742 | 7 / 00.743 | 9 / 00.803 | 7 / 00.747 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 49 / 00.781 | 3 / 00.777 | 1 / 00.678 | 0 / 00.394 | 1 / 00.703 | 1 / 00.730 | 3 / 00.735 | 11 / 00.887 | 3 / 00.774 | 6 / 00.803 | 5 / 00.830 | 4 / 00.775 | 2 / 00.788 | 9 / 00.779 |
| Turion Jev 31B Open · 31B | — | Decision | 43 / 00.864 | 5 / 00.868 | 1 / 00.770 | 0 / 00.757 | 1 / 00.809 | 0 / 00.904 | 3 / 00.817 | 3 / 00.926 | 1 / 00.854 | 6 / 00.823 | 1 / 00.940 | 8 / 00.853 | 5 / 00.866 | 9 / 00.852 |
| vera-medium-omni Open · 4.2B | — | Decision | 33 / 00.789 | 2 / 00.782 | 3 / 00.704 | 0 / 00.636 | 1 / 00.695 | 1 / 00.836 | 1 / 00.706 | 13 / 00.860 | 0 / 00.848 | 2 / 00.837 | 1 / 00.797 | 4 / 00.770 | 1 / 00.809 | 4 / 00.784 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 27 / 00.787 | 5 / 00.789 | 0 / 00.696 | 0 / 00.615 | 1 / 00.670 | 0 / 00.747 | 0 / 00.750 | 4 / 00.848 | 4 / 00.806 | 5 / 00.837 | 1 / 00.801 | 2 / 00.768 | 3 / 00.824 | 2 / 00.794 |
| OneJev-27B Open · 27B | — | Decision | 26 / 00.783 | 2 / 00.768 | 0 / 00.645 | 0 / 00.806 | 1 / 00.750 | 0 / 00.767 | 1 / 00.778 | 2 / 00.884 | 1 / 00.758 | 1 / 00.811 | 3 / 00.787 | 4 / 00.757 | 6 / 00.785 | 5 / 00.789 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 15 / 00.899 | 2 / 00.900 | 0 / 00.841 | 0 / 00.667 | 0 / 00.847 | 1 / 00.888 | 0 / 00.872 | 2 / 00.950 | 0 / 00.919 | 1 / 00.910 | 1 / 00.939 | 4 / 00.895 | 1 / 00.871 | 3 / 00.911 |
| Clef Open · 27B | — | Decision | 12 / 00.767 | 2 / 00.751 | 1 / 00.708 | 0 / 00.875 | 2 / 00.723 | 0 / 00.720 | 0 / 00.798 | 3 / 00.778 | 0 / 00.793 | 1 / 00.744 | 0 / 00.878 | 2 / 00.748 | 0 / 00.802 | 1 / 00.774 |
| d3-ultra Open · 26B | — | Decision | 11 / 00.857 | 1 / 00.842 | 1 / 00.778 | 0 / 00.764 | 0 / 00.820 | 0 / 00.731 | 0 / 00.827 | 3 / 00.942 | 0 / 00.819 | 1 / 00.901 | 2 / 00.848 | 1 / 00.871 | 1 / 00.850 | 1 / 00.874 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 8 / 00.636 | 0 / 00.570 | 0 / 00.578 | 0 / 00.533 | 0 / 00.508 | 0 / 00.540 | 0 / 00.655 | 0 / 00.667 | 1 / 00.779 | 4 / 00.736 | 1 / 00.699 | 1 / 00.644 | 0 / 00.656 | 1 / 00.587 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 7 / 00.819 | 2 / 00.838 | 0 / 00.736 | 0 / 00.814 | 0 / 00.752 | 0 / 00.840 | 0 / 00.833 | 5 / 00.915 | 0 / 00.800 | 0 / 00.824 | 0 / 00.916 | 0 / 00.775 | 0 / 00.857 | 0 / 00.794 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 4 / 00.589 | 1 / 00.538 | 0 / 00.498 | 0 / 00.467 | 0 / 00.610 | 0 / 00.579 | 0 / 00.525 | 0 / 00.710 | 1 / 00.604 | 0 / 00.601 | 0 / 00.661 | 1 / 00.673 | 0 / 00.564 | 1 / 00.616 |
| d1-2B Open · 1.9B | — | Decision | 4 / 00.656 | 0 / 00.634 | 0 / 00.629 | 0 / 00.318 | 0 / 00.572 | 1 / 00.582 | 0 / 00.589 | 0 / 00.721 | 1 / 00.778 | 1 / 00.625 | 0 / 00.718 | 1 / 00.709 | 0 / 00.657 | 0 / 00.677 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 2 / 00.685 | 0 / 00.661 | 0 / 00.601 | 0 / 00.333 | 0 / 00.560 | 0 / 00.665 | 0 / 00.655 | 0 / 00.788 | 0 / 00.813 | 1 / 00.755 | 0 / 00.703 | 0 / 00.614 | 0 / 00.658 | 1 / 00.679 |
| Clef-Flash Open · 9.0B | — | Decision | No answers at this threshold0.730 | 0 / 00.711 | 0 / 00.642 | 0 / 00.361 | 0 / 00.614 | 0 / 00.736 | 0 / 00.670 | 0 / 00.766 | 0 / 00.789 | 0 / 00.744 | 0 / 00.823 | 0 / 00.722 | 0 / 00.804 | 0 / 00.737 |
| Cygnet Open · 12B | — | Decision | No answers at this threshold0.735 | 0 / 00.730 | 0 / 00.617 | 0 / 00.600 | 0 / 00.690 | 0 / 00.639 | 0 / 00.726 | 0 / 00.800 | 0 / 00.788 | 0 / 00.796 | 0 / 00.697 | 0 / 00.680 | 0 / 00.768 | 0 / 00.761 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | No answers at this threshold0.501 | 0 / 00.524 | 0 / 00.554 | 0 / 00.550 | 0 / 00.476 | 0 / 00.474 | 0 / 00.505 | 0 / 00.412 | 0 / 00.522 | 0 / 00.412 | 0 / 00.460 | 0 / 00.566 | 0 / 00.543 | 0 / 00.518 |
| GLiDE 0.4B Open · 0.4B | — | Decision | No answers at this threshold0.500 | 0 / 00.496 | 0 / 00.528 | 0 / 00.875 | 0 / 00.509 | 0 / 00.606 | 0 / 00.458 | 0 / 00.370 | 0 / 00.632 | 0 / 00.450 | 0 / 00.478 | 0 / 00.493 | 0 / 00.498 | 0 / 00.515 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | No answers at this threshold0.596 | 0 / 00.486 | 0 / 00.603 | 0 / 00.667 | 0 / 00.628 | 0 / 00.573 | 0 / 00.490 | 0 / 00.683 | 0 / 00.792 | 0 / 00.660 | 0 / 00.653 | 0 / 00.603 | 0 / 00.557 | 0 / 00.565 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | No answers at this threshold0.803 | 0 / 00.788 | 0 / 00.711 | 0 / 00.712 | 0 / 00.671 | 0 / 00.801 | 0 / 00.819 | 0 / 00.834 | 0 / 00.841 | 0 / 00.847 | 0 / 00.824 | 0 / 00.774 | 0 / 00.806 | 0 / 00.817 |
| Metask-Jev-4B Open · 4.5B | — | Decision | No answers at this threshold0.651 | 0 / 00.614 | 0 / 00.544 | 0 / 00.933 | 0 / 00.617 | 0 / 00.737 | 0 / 00.613 | 0 / 00.633 | 0 / 00.619 | 0 / 00.666 | 0 / 00.727 | 0 / 00.660 | 0 / 00.624 | 0 / 00.664 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | No answers at this threshold0.592 | 0 / 00.548 | 0 / 00.601 | 0 / 00.533 | 0 / 00.519 | 0 / 00.678 | 0 / 00.474 | 0 / 00.543 | 0 / 00.726 | 0 / 00.690 | 0 / 00.619 | 0 / 00.605 | 0 / 00.609 | 0 / 00.559 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | No answers at this threshold0.764 | 0 / 00.816 | 0 / 00.607 | 0 / 00.625 | 0 / 00.674 | 0 / 00.731 | 0 / 00.751 | 0 / 00.762 | 0 / 00.745 | 0 / 00.847 | 0 / 00.818 | 0 / 00.776 | 0 / 00.727 | 0 / 00.749 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | No answers at this threshold0.583 | 0 / 00.611 | 0 / 00.460 | 0 / 00.667 | 0 / 00.546 | 0 / 00.440 | 0 / 00.612 | 0 / 00.567 | 0 / 00.693 | 0 / 00.672 | 0 / 00.676 | 0 / 00.551 | 0 / 00.503 | 0 / 00.580 |
| Strands Decider 2B Open · 2.0B | — | Decision | No answers at this threshold0.569 | 0 / 00.510 | 0 / 00.458 | 0 / 00.615 | 0 / 00.534 | 0 / 00.589 | 0 / 00.583 | 0 / 00.635 | 0 / 00.615 | 0 / 00.574 | 0 / 00.632 | 0 / 00.556 | 0 / 00.542 | 0 / 00.627 |
| celeris-1-decision Closed · Efficient | — | Decision | No answers at this threshold0.858 | 0 / 00.857 | 0 / 00.787 | 0 / 00.807 | 0 / 00.794 | 0 / 00.884 | 0 / 00.896 | 0 / 00.952 | 0 / 00.840 | 0 / 00.881 | 0 / 00.836 | 0 / 00.838 | 0 / 00.869 | 0 / 00.849 |
| celeris-1.1-decision Closed · Efficient | — | Decision | No answers at this threshold0.859 | 0 / 00.863 | 0 / 00.790 | 0 / 00.875 | 0 / 00.790 | 0 / 00.863 | 0 / 00.896 | 0 / 00.944 | 0 / 00.853 | 0 / 00.877 | 0 / 00.812 | 0 / 00.852 | 0 / 00.867 | 0 / 00.855 |
| Drex Max Open · 27B | — | Decision | 719 / 100.861 | 67 / 10.834 | 25 / 00.803 | 3 / 00.750 | 32 / 00.799 | 28 / 00.867 | 36 / 40.794 | 63 / 00.923 | 42 / 20.877 | 75 / 00.894 | 44 / 00.908 | 83 / 00.876 | 96 / 20.855 | 125 / 10.847 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 469 / 10.898 | 38 / 00.916 | 15 / 00.862 | 2 / 00.742 | 14 / 00.847 | 24 / 00.902 | 37 / 00.921 | 70 / 00.956 | 12 / 00.871 | 28 / 00.882 | 28 / 00.916 | 56 / 10.863 | 68 / 00.908 | 77 / 00.886 |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 464 / 10.895 | 38 / 00.874 | 15 / 00.850 | 1 / 00.750 | 14 / 00.881 | 22 / 00.903 | 32 / 00.900 | 60 / 00.956 | 18 / 00.912 | 35 / 00.872 | 33 / 00.904 | 55 / 10.885 | 64 / 00.914 | 77 / 00.888 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 360 / 10.839 | 28 / 00.819 | 12 / 00.784 | 1 / 00.722 | 15 / 00.780 | 10 / 00.808 | 19 / 00.754 | 47 / 00.929 | 35 / 00.829 | 31 / 00.832 | 28 / 00.931 | 37 / 10.802 | 45 / 00.871 | 52 / 00.848 |
| Mercury Decide Closed · Efficient | — | Decision | 272 / 90.789 | 25 / 00.811 | 9 / 00.692 | 1 / 00.729 | 8 / 00.748 | 6 / 00.687 | 11 / 30.718 | 26 / 10.859 | 23 / 20.801 | 31 / 10.808 | 18 / 00.821 | 22 / 10.751 | 44 / 10.785 | 48 / 00.822 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 244 / 100.780 | 24 / 00.806 | 9 / 00.677 | 1 / 10.561 | 8 / 00.679 | 6 / 00.797 | 11 / 00.751 | 31 / 00.843 | 17 / 00.747 | 22 / 20.806 | 13 / 00.812 | 31 / 20.720 | 33 / 30.836 | 38 / 20.776 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 207 / 30.804 | 18 / 10.778 | 5 / 00.684 | 0 / 00.606 | 7 / 00.731 | 7 / 10.745 | 8 / 00.759 | 29 / 10.839 | 10 / 00.829 | 18 / 00.822 | 12 / 00.853 | 33 / 00.824 | 24 / 00.814 | 36 / 00.820 |
| GLiDE 27B Open · 27B | — | Decision | 198 / 10.864 | 17 / 10.858 | 6 / 00.787 | 0 / 00.569 | 8 / 00.761 | 5 / 00.846 | 7 / 00.870 | 22 / 00.948 | 20 / 00.938 | 36 / 00.873 | 18 / 00.905 | 21 / 00.861 | 20 / 00.848 | 18 / 00.875 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 197 / 10.806 | 18 / 00.815 | 4 / 00.701 | 0 / 00.729 | 6 / 00.684 | 8 / 00.764 | 9 / 00.797 | 31 / 00.867 | 12 / 10.814 | 21 / 00.820 | 16 / 00.840 | 23 / 00.813 | 21 / 00.812 | 28 / 00.806 |
| Elephant-1 Open · 9.0B | — | Decision | 182 / 10.803 | 14 / 00.821 | 5 / 00.711 | 0 / 00.486 | 6 / 00.684 | 6 / 00.823 | 10 / 00.730 | 28 / 00.833 | 10 / 10.785 | 15 / 00.868 | 14 / 00.879 | 25 / 00.778 | 24 / 00.834 | 25 / 00.774 |
| d3 Open · 26B | — | Decision | 162 / 10.878 | 13 / 10.867 | 5 / 00.791 | 1 / 00.806 | 6 / 00.811 | 7 / 00.907 | 8 / 00.847 | 21 / 00.955 | 15 / 00.904 | 25 / 00.905 | 13 / 00.893 | 21 / 00.886 | 13 / 00.875 | 14 / 00.872 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 150 / 50.781 | 7 / 10.781 | 4 / 00.688 | 0 / 00.300 | 2 / 00.694 | 3 / 00.780 | 9 / 00.732 | 26 / 00.884 | 10 / 10.807 | 21 / 00.854 | 11 / 10.773 | 15 / 10.730 | 19 / 00.806 | 23 / 10.753 |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 149 / 60.746 | 14 / 20.793 | 5 / 00.668 | 0 / 00.424 | 5 / 00.719 | 2 / 00.797 | 5 / 20.655 | 16 / 00.735 | 8 / 10.772 | 19 / 00.732 | 7 / 00.768 | 16 / 00.771 | 24 / 10.729 | 28 / 00.726 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 126 / 40.752 | 14 / 10.731 | 5 / 00.719 | 0 / 00.548 | 4 / 00.677 | 6 / 00.743 | 8 / 10.763 | 16 / 00.824 | 8 / 00.774 | 14 / 00.799 | 9 / 20.772 | 17 / 00.761 | 12 / 00.772 | 13 / 00.696 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 113 / 10.787 | 6 / 00.765 | 5 / 00.695 | 0 / 00.450 | 2 / 00.701 | 3 / 00.836 | 9 / 00.788 | 23 / 00.913 | 7 / 00.830 | 12 / 00.797 | 7 / 00.792 | 11 / 00.764 | 15 / 00.792 | 13 / 10.765 |
| Winnow-12B Open · 12B | — | Decision | 20 / 10.743 | 5 / 00.742 | 1 / 00.638 | 0 / 00.686 | 0 / 00.670 | 1 / 00.642 | 0 / 10.743 | 1 / 00.787 | 1 / 00.742 | 5 / 00.731 | 3 / 00.785 | 1 / 00.746 | 1 / 00.758 | 1 / 00.774 |
| GLiDE 2B Open · 1.9B | — | Decision | 15 / 10.635 | 3 / 10.625 | 0 / 00.569 | 0 / 00.519 | 1 / 00.554 | 0 / 00.590 | 1 / 00.676 | 1 / 00.646 | 1 / 00.728 | 3 / 00.663 | 2 / 00.596 | 1 / 00.617 | 1 / 00.616 | 1 / 00.639 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 48 setups: Claude Opus 5.5 (chat), Qwen4-Max (high · chat), Claude Fable 5.1 (chat), Claude Sonnet 5.5 (chat), DeepSeek-V4.1-Flash (high · chat · 1017), GPT-6 Astra (chat), GPT-6.1 Sol (chat), Inkling-Small (high · chat · Jul 2026), Gemini 4 Carbon (chat), Gemini 4 Hydrogen (chat), Muse Ember (max · chat), Grok 4.7 (chat), Inkling 2 Small (high · chat), Qwen3.8 Flash (off · chat), MiMo-V2.6-Flash (on · chat), Qwen3.5-4B (on · chat), Gemma 4 31B (chat), DeepSeek-V4-Flash (chat), MiniCPM5-2B (on · chat), GPT-6.1 Terra (chat), Granite-4.2-3B (high · chat), Qwen3.8-27B (low · chat), LFM2.5-24B-A2B (on · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), LFM3-1.7B (on · chat), Gemma 4 12B Unified (chat), GLM-5.3-Flash (high · chat), LFM3-1.7B (off · chat), Gemma 4 E4B (on · chat · 32k cap · sampled), Gemma 4 E4B (on · chat · 32k cap), Qwen3.5-4B (off · chat), Gemma 4 E4B (on · chat), MiniMax-M3.1-Flash (high · chat), MiniCPM5-2B (off · chat), Gemma 4 E4B (chat), Gemma 4 E2B (on · chat · 32k cap), Gemma 4 E2B (on · chat · 32k cap · sampled), Raon-1 nano (on · chat), Gemma 4 E2B (on · chat), Gemma 4 E2B (chat), LFM2.5-8B-A1B (on · chat), Qwen3-14B (off · chat), Qwen3.5-2B (on · chat), Qwen3.5-2B (off · chat), LFM2.5-2.6B (on · chat), GPT-6 Luna (chat), Erk-14B (off · chat), Raon-1 nano (off · chat).
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC); Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)); Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC); Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test); GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 96.2% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Logic 96.3% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.2% | — | Chat | 99.7%99.4%–99.9% | ab | 2,157 / 2,163 | Not measured | Not measured | — | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.5% | — | Chat | 99.7%99.3%–99.8% | abcd | 2,156 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Logic 96.2% | — | Chat | 99.6%99.3%–99.8% | abcdefg | 2,155 / 2,163 | Not measured | Not measured | — | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Logic 96.7% | — | Chat | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Logic 96.5% | — | Chat | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Logic 95.4% | — | Chat | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Logic 96.7% | — | Chat | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Logic 96.1% | — | Chat | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 85.7%answered 99% | 4,400 ms | 6,680 ms | — |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Logic 95.5% | — | Chat | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Logic running | — | Chat | 99.5%99.2%–99.7% | abcdefg | 2,153 / 2,163 | Not measured | Not measured | — | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Logic 95.6% | — | Chat | 99.5%99.2%–99.7% | abcdef | 2,153 / 2,163 | Not measured | Not measured | — | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Logic 94.0% | — | Chat | 99.5%99.1%–99.7% | abcdefgh | 2,152 / 2,163 | Not measured | Not measured | — | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Logic 95.1% | — | Chat | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | — | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Logic 92.4% | — | Chat | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B | — | Chat | 99.4%99.0%–99.6% | abcdefghi | 2,150 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Logic running | — | Chat | 99.3%98.9%–99.6% | abcdefghijk | 2,148 / 2,163 | Not measured | Not measured | — | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Logic 88.9% | — | Chat | 99.2%99.2%–99.3% min–max of 3 seeds | abcdefghij | 2,147 / 2,163seed 0 run | Not measured | Not measured | — | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | abcdefghijk | 2,146 / 2,163 | Not measured | Not measured | — | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Logic 91.6% | — | Chat | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | — | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | abcdefghijk | 2,146 / 2,163 | Not measured | Not measured | — | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Logic 86.7% | — | Chat | 99.0%98.5%–99.4% | bcdefghijk | 2,142 / 2,163 | Not measured | Not measured | — | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% | — | Chat | 98.9%98.8%–99.0% min–max of 3 seeds | abcdefghijkl | 2,142 / 2,163seed 0 run | Not measured | Not measured | — | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B | — | Chat | 98.9%98.8%–99.1% min–max of 3 seeds | bcdefghijkl | 2,139 / 2,163seed 0 run | Not measured | Not measured | — | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Logic 87.7% | — | Chat | 98.8%98.3%–99.2% | cdefghijklm | 2,138 / 2,163 | Not measured | Not measured | — | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Logic 93.2% | — | Chat | 98.8%98.2%–99.2% | dfghijklm | 2,137 / 2,163 | Not measured | Not measured | — | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Logic 85.5% | — | Chat | 98.7%98.1%–99.1% | ghijklm | 2,135 / 2,163 | Not measured | Not measured | — | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.9% | — | Chat | 98.5%98.4%–98.8% min–max of 3 seeds | efghijklm | 2,136 / 2,163seed 0 run | Not measured | Not measured | — | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% | — | Chat | 98.5%97.9%–99.0% | hijklmn | 2,131 / 2,163 | Not measured | Not measured | — | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Logic 90.1% | — | Chat | 98.5%97.9%–98.9% | hijklmno | 2,130 / 2,163 | Not measured | Not measured | — | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 82.6% | — | Chat | 98.2%97.6%–98.7% | jklmno | 2,125 / 2,163 | Not measured | Not measured | — | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Logic 94.0% | — | Chat | 98.2%97.6%–98.7% | ijklmnop | 2,125 / 2,163 | Not measured | Not measured | — | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Logic 83.6% | — | Chat | 98.2%98.1%–98.3% min–max of 3 seeds | jklmnop | 2,126 / 2,163seed 0 run | Not measured | Not measured | — | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 83.2% | — | Chat | 98.1%97.4%–98.6% | klmnop | 2,122 / 2,163 | Not measured | Not measured | — | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% | — | Chat | 97.8%97.1%–98.4% | lmnopq | 2,116 / 2,163 | Not measured | Not measured | — | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 76.8% | — | Chat | 97.7%97.6%–97.8% min–max of 3 seeds | mnopqr | 2,111 / 2,163seed 0 run | Not measured | Not measured | — | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% | — | Chat | 97.0%96.2%–97.6% | nopqr | 2,098 / 2,163 | Not measured | Not measured | — | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 74.6% | — | Chat | 96.9%96.1%–97.6% | opqr | 2,097 / 2,163 | Not measured | Not measured | — | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 72.9% | — | Chat | 96.8%95.9%–97.4% | pqrs | 2,093 / 2,163 | Not measured | Not measured | — | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Logic 60.2% | — | Chat | 96.3%95.4%–97.0% | qrst | 2,082 / 2,163 | Not measured | Not measured | — | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Logic 65.3% | — | Chat | 95.8%94.9%–96.6% | rstu | 2,072 / 2,163 | Not measured | Not measured | — | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B | — | Chat | 94.7%93.7%–95.6% | stu | 2,049 / 2,163 | Not measured | Not measured | — | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Logic 64.8% | — | Chat | 94.6%93.6%–95.5% | stu | 2,047 / 2,163 | Not measured | Not measured | — | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Logic 75.4% | — | Chat | 94.0%92.9%–94.9% | tu | 2,033 / 2,163 | Not measured | Not measured | — | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Logic 58.0% | — | Chat | 93.7%92.6%–94.6% | u | 2,026 / 2,163 | Not measured | Not measured | — | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Logic 54.2% | — | Chat | 90.8%89.5%–91.9% | v | 1,963 / 2,163 | Not measured | Not measured | — | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 65.1% | — | Chat | 89.6%88.2%–90.8% | v | 1,937 / 2,163 | Not measured | Not measured | — | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 837 answers, 837 right · 0 wrong · AUROC 0.903 Logic 60.7% | — | Decision | 81.6%79.9%–83.1% | w | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 85.99chosen-answer Brier 0.0958 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 470 answers, 469 right · 1 wrong · AUROC 0.898 Logic 58.9% | — | Decision | 80.3%78.5%–81.9% | wx | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 85.05chosen-answer Brier 0.1016 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 178 answers, 178 right · 0 wrong · AUROC 0.903 Logic 59.3% | — | Decision | 78.0%76.2%–79.7% | xy | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 83.57chosen-answer Brier 0.1084 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 465 answers, 464 right · 1 wrong · AUROC 0.895 Logic 59.0% | — | Decision | 77.4%75.6%–79.1% | yz | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 83.18chosen-answer Brier 0.1103 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 318 answers, 318 right · 0 wrong · AUROC 0.899 Logic 58.7% | — | Decision | 77.3%75.5%–79.1% | yz | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 83.06chosen-answer Brier 0.1122 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 215 answers, 215 right · 0 wrong · AUROC 0.891 Logic 57.9% | — | Decision | 77.1%75.3%–78.8% | yz | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 82.89chosen-answer Brier 0.1133 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 15 answers, 15 right · 0 wrong · AUROC 0.899 Logic 57.8% | — | Decision | 75.2%73.4%–77.0% | zA | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 81.60chosen-answer Brier 0.1201 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 213 answers, 213 right · 0 wrong · AUROC 0.876 Logic 54.9% | — | Decision | 74.1%72.2%–75.9% | yzAB | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 80.88chosen-answer Brier 0.1235 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 412 answers, 412 right · 0 wrong · AUROC 0.895 Logic 54.5% | — | Decision | 74.0%72.1%–75.8% | zAB | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 81.16chosen-answer Brier 0.1164 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 323 answers, 323 right · 0 wrong · AUROC 0.856 Logic 54.8% | — | Decision | 72.5%70.6%–74.3% | ABC | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 79.32chosen-answer Brier 0.1385 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 199 answers, 198 right · 1 wrong · AUROC 0.864 Logic 58.4% | — | Decision | 72.0%70.1%–73.8% | ABCD | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 79.22chosen-answer Brier 0.1354 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 11 answers, 11 right · 0 wrong · AUROC 0.857 Logic 54.6% | — | Decision | 71.5%69.6%–73.4% | BCD | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 78.39chosen-answer Brier 0.1474 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 214 answers, 214 right · 0 wrong · AUROC 0.859 Logic 50.0% | — | Decision | 71.3%69.4%–73.2% | ABCD | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 78.84chosen-answer Brier 0.1366 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 161 answers, 161 right · 0 wrong · AUROC 0.865 Logic 54.5% | — | Decision | 70.8%68.8%–72.7% | BCD | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 78.56chosen-answer Brier 0.1367 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 163 answers, 162 right · 1 wrong · AUROC 0.878 Logic 54.5% | — | Decision | 70.4%68.5%–72.3% | BCDE | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 78.71chosen-answer Brier 0.1299 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 729 answers, 719 right · 10 wrong · AUROC 0.861 Logic 53.1% | — | Decision | 70.3%68.4%–72.2% | BCDE | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 77.10chosen-answer Brier 0.1611 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 315 answers, 315 right · 0 wrong · AUROC 0.867 Logic 54.8% | — | Decision | 70.1%68.1%–72.0% | BCDE | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 78.01chosen-answer Brier 0.1407 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.858 Logic 47.3% | — | Decision | 70.0%68.0%–71.9% | BCDEF | 1,514 / 2,163 | No answers at this threshold | 0.858 | 77.96chosen-answer Brier 0.1407 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 131 answers, 131 right · 0 wrong · AUROC 0.862 Logic 55.2% | — | Decision | 69.9%67.9%–71.8% | CDEF | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 77.95chosen-answer Brier 0.1396 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.859 Logic 47.3% | — | Decision | 69.9%67.9%–71.8% | BCDEF | 1,511 / 2,163 | No answers at this threshold | 0.859 | 77.91chosen-answer Brier 0.1403 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 115 answers, 115 right · 0 wrong · AUROC 0.876 Logic 51.6% | — | Decision | 69.9%67.9%–71.8% | CDE | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 78.23chosen-answer Brier 0.1340 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 331 answers, 331 right · 0 wrong · AUROC 0.869 Logic 53.3% | — | Decision | 69.8%67.8%–71.7% | CDEF | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 78.16chosen-answer Brier 0.1345 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 326 answers, 326 right · 0 wrong · AUROC 0.873 Logic 52.5% | with web | Decision | 69.3%67.3%–71.2% | CDEFG | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 77.94chosen-answer Brier 0.1342 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 43 answers, 43 right · 0 wrong · AUROC 0.864 Logic 52.3% | — | Decision | 69.0%67.0%–70.9% | CDEFG | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 77.52chosen-answer Brier 0.1399 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 361 answers, 360 right · 1 wrong · AUROC 0.839 Logic 52.4% | — | Decision | 68.6%66.6%–70.5% | DEFG | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 76.15chosen-answer Brier 0.1626 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.819 Logic 50.6% | — | Decision | 66.3%64.2%–68.2% | EFGH | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 74.96chosen-answer Brier 0.1632 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 141 answers, 141 right · 0 wrong · AUROC 0.838 Logic 49.2% | — | Decision | 65.5%63.5%–67.5% | FGHI | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 74.98chosen-answer Brier 0.1556 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 183 answers, 183 right · 0 wrong · AUROC 0.829 Logic 46.0% | — | Decision | 65.2%63.2%–67.2% | GHI | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 74.62chosen-answer Brier 0.1595 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 202 answers, 202 right · 0 wrong · AUROC 0.829 Logic 48.2% | — | Decision | 65.1%63.1%–67.1% | GHI | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 74.56chosen-answer Brier 0.1602 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 248 answers, 248 right · 0 wrong · AUROC 0.834 Logic 46.3% | — | Decision | 64.1%62.0%–66.1% | HIJ | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 74.08chosen-answer Brier 0.1592 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 198 answers, 197 right · 1 wrong · AUROC 0.806 Logic 46.4% | — | Decision | 62.1%60.0%–64.1% | HIJK | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 72.28chosen-answer Brier 0.1752 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 183 answers, 182 right · 1 wrong · AUROC 0.803 Logic 46.4% | — | Decision | 61.3%59.3%–63.4% | IJK | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 71.86chosen-answer Brier 0.1763 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.803 Logic 43.9% | — | Decision | 60.1%58.0%–62.1% | JKL | 1,300 / 2,163 | No answers at this threshold | 0.803 | 71.15chosen-answer Brier 0.1779 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 64 answers, 64 right · 0 wrong · AUROC 0.814 Logic 48.2% | — | Decision | 59.8%57.7%–61.9% | JKL | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 71.32chosen-answer Brier 0.1719 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 123 answers, 123 right · 0 wrong · AUROC 0.799 Logic 44.3% | — | Decision | 59.8%57.7%–61.8% | JKL | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 71.08chosen-answer Brier 0.1762 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 27 answers, 27 right · 0 wrong · AUROC 0.787 Logic 44.4% | — | Decision | 59.4%57.3%–61.4% | KLM | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 70.49chosen-answer Brier 0.1839 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 174 answers, 174 right · 0 wrong · AUROC 0.813 Logic 41.9% | — | Decision | 59.0%56.9%–61.0% | KLM | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 70.98chosen-answer Brier 0.1703 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 210 answers, 207 right · 3 wrong · AUROC 0.804 Logic 45.3% | — | Decision | 58.8%56.7%–60.9% | KLM | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 69.67chosen-answer Brier 0.1946 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 49 answers, 49 right · 0 wrong · AUROC 0.781 Logic 46.6% | — | Decision | 58.4%56.3%–60.5% | KLMN | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 69.90chosen-answer Brier 0.1864 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 73 answers, 73 right · 0 wrong · AUROC 0.798 Logic 47.8% | — | Decision | 57.9%55.8%–60.0% | KLMN | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 69.69chosen-answer Brier 0.1855 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 33 answers, 33 right · 0 wrong · AUROC 0.789 Logic 44.1% | — | Decision | 56.8%54.7%–58.9% | LMN | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 69.18chosen-answer Brier 0.1846 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 254 answers, 244 right · 10 wrong · AUROC 0.780 Logic 43.3% | — | Decision | 56.0%53.9%–58.1% | LMNOP | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 67.15chosen-answer Brier 0.2174 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.765 Logic 42.3% | — | Decision | 56.0%53.9%–58.1% | LMNO | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 67.89chosen-answer Brier 0.2021 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 114 answers, 113 right · 1 wrong · AUROC 0.787 Logic 44.7% | — | Decision | 55.9%53.8%–58.0% | LMNO | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 67.89chosen-answer Brier 0.2015 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 96 answers, 96 right · 0 wrong · AUROC 0.779 Logic 45.9% | — | Decision | 55.8%53.7%–57.9% | LMNO | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 68.25chosen-answer Brier 0.1934 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 152 answers, 152 right · 0 wrong · AUROC 0.767 Logic 40.8% | — | Decision | 55.6%53.5%–57.7% | LMNOP | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 68.16chosen-answer Brier 0.1930 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 130 answers, 130 right · 0 wrong · AUROC 0.790 Logic 44.4% | — | Decision | 55.3%53.2%–57.4% | MNOP | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 68.37chosen-answer Brier 0.1860 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 155 answers, 150 right · 5 wrong · AUROC 0.781 Logic 43.1% | — | Decision | 55.2%53.1%–57.3% | MNOP | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 67.61chosen-answer Brier 0.2002 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.764 Logic 42.6% | — | Decision | 54.6%52.4%–56.6% | MNOPQ | 1,180 / 2,163 | No answers at this threshold | 0.764 | 64.53chosen-answer Brier 0.2549 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 26 answers, 26 right · 0 wrong · AUROC 0.783 Logic 40.6% | — | Decision | 53.8%51.7%–55.9% | NOPQ | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 67.54chosen-answer Brier 0.1869 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 281 answers, 272 right · 9 wrong · AUROC 0.789 Logic 45.1% | — | Decision | 53.5%51.4%–55.6% | NOPQ | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 66.02chosen-answer Brier 0.2146 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 130 answers, 126 right · 4 wrong · AUROC 0.752 Logic 39.7% | — | Decision | 51.5%49.4%–53.6% | PQR | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 65.62chosen-answer Brier 0.2026 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 12 answers, 12 right · 0 wrong · AUROC 0.767 Logic 40.8% | — | Decision | 51.1%49.0%–53.2% | OPQR | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 65.20chosen-answer Brier 0.2070 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 155 answers, 149 right · 6 wrong · AUROC 0.746 Logic 40.4% | — | Decision | 50.4%48.3%–52.5% | QR | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 64.11chosen-answer Brier 0.2221 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.730 Logic 38.9% | — | Decision | 49.8%47.7%–51.9% | QR | 1,077 / 2,163 | No answers at this threshold | 0.730 | 64.44chosen-answer Brier 0.2092 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 21 answers, 20 right · 1 wrong · AUROC 0.743 Logic 39.7% | — | Decision | 46.9%44.8%–49.0% | RS | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 63.00chosen-answer Brier 0.2087 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.735 Logic 39.3% | — | Decision | 44.3%42.3%–46.4% | S | 959 / 2,163 | No answers at this threshold | 0.735 | 61.87chosen-answer Brier 0.2060 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.651 Logic 32.1% | — | Decision | 39.2%37.1%–41.2% | T | 847 / 2,163 | No answers at this threshold | 0.651 | 58.63chosen-answer Brier 0.2190 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.685 Logic 30.8% | — | Decision | 38.7%36.6%–40.7% | T | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 58.95chosen-answer Brier 0.2075 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.656 Logic 30.7% | — | Decision | 37.1%35.1%–39.1% | TU | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 57.72chosen-answer Brier 0.2163 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 16 answers, 15 right · 1 wrong · AUROC 0.635 Logic 32.3% | — | Decision | 36.8%34.7%–38.8% | T | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 55.70chosen-answer Brier 0.2536 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.636 Logic 31.7% | — | Decision | 35.3%33.3%–37.3% | TU | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 56.49chosen-answer Brier 0.2229 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.589 Logic 29.2% | — | Decision | 32.1%30.2%–34.1% | UV | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 53.21chosen-answer Brier 0.2566 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.583 Logic 24.7% | — | Decision | 29.7%27.8%–31.6% | V | 642 / 2,163 | No answers at this threshold | 0.583 | 54.40chosen-answer Brier 0.2088 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.569 Logic 22.4% | — | Decision | 28.7%26.8%–30.6% | V | 620 / 2,163 | No answers at this threshold | 0.569 | 53.90chosen-answer Brier 0.2086 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: No answers at this threshold · AUROC 0.592 Logic 27.0% | — | Decision | 28.2%26.4%–30.2% | V | 611 / 2,163 | No answers at this threshold | 0.592 | 53.62chosen-answer Brier 0.2100 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.596 Logic 26.7% | — | Decision | 28.2%26.3%–30.1% | V | 610 / 2,163 | No answers at this threshold | 0.596 | 53.60chosen-answer Brier 0.2100 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.501 Logic 23.6% | — | Decision | 22.7%20.9%–24.5% | W | 490 / 2,163 | No answers at this threshold | 0.501 | 52.01chosen-answer Brier 0.1863 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.500 Logic 23.2% | — | Decision | 20.6%18.9%–22.3% | W | 445 / 2,163 | No answers at this threshold | 0.500 | 50.98chosen-answer Brier 0.1862 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
StartLux PelaAI scar-ai Fastino Liquid AI Other Metask Lab OmniJev Strands Agents Perplexity quincyqiang victorbennett Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 55.8%2,163 | 70.8%62.3% kept (1,348) | 81.1%40.5% kept (876) | 94.0%20.8% kept (450) | 100.0%4.4% kept (96) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 56.0%2,163 | 67.1%67.4% kept (1,457) | 79.3%41.9% kept (907) | 92.8%21.3% kept (460) | 100.0%2.5% kept (54) |
| vera-medium-omni Open · 4.2B | — | Decision | 56.8%2,163 | 75.4%55.2% kept (1,193) | 88.5%29.6% kept (641) | 98.9%12.3% kept (267) | 100.0%1.5% kept (33) |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 35.3%2,163 | 52.1%28.2% kept (609) | 67.2%8.9% kept (192) | 88.5%2.4% kept (52) | 100.0%0.4% kept (8) |
| GLiDE 0.8B Open · 0.8B | — | Decision | 32.1%2,163 | 38.1%44.3% kept (959) | 47.5%16.8% kept (364) | 73.0%2.9% kept (63) | 100.0%0.2% kept (4) |
| d1-2B Open · 1.9B | — | Decision | 37.1%2,163 | 58.0%25.0% kept (541) | 74.5%7.4% kept (161) | 83.7%2.3% kept (49) | 100.0%0.2% kept (4) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 38.7%2,163 | 68.1%21.1% kept (457) | 83.6%7.9% kept (171) | 92.0%2.3% kept (50) | 100.0%0.1% kept (2) |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 22.7%2,163 | 26.9%3.6% kept (78) | 38.5%0.6% kept (13) | 100.0%0.0% kept (1) | none kept |
| GLiDE 0.4B Open · 0.4B | — | Decision | 20.6%2,163 | 23.5%5.5% kept (119) | 33.3%1.0% kept (21) | none kept | none kept |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 28.2%2,163 | 41.7%15.5% kept (336) | 54.9%3.3% kept (71) | 75.0%0.2% kept (4) | none kept |
| Metask-Jev-4B Open · 4.5B | — | Decision | 39.2%2,163 | 63.1%19.8% kept (428) | 77.5%8.6% kept (187) | 88.2%1.6% kept (34) | none kept |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 28.2%2,163 | 41.8%15.3% kept (330) | 54.1%3.4% kept (74) | 66.7%0.7% kept (15) | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 29.7%2,163 | 45.9%10.6% kept (229) | 61.0%3.6% kept (77) | 92.3%0.6% kept (13) | none kept |
| Strands Decider 2B Open · 2.0B | — | Decision | 28.7%2,163 | 45.9%11.8% kept (255) | 69.2%1.2% kept (26) | 100.0%0.0% kept (1) | none kept |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 55.2%2,163 | 69.1%65.3% kept (1,412) | 79.3%46.3% kept (1,001) | 88.3%25.8% kept (557) | 96.8%7.2% kept (155) |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 50.4%2,163 | 60.9%66.6% kept (1,440) | 74.3%39.1% kept (845) | 86.2%20.4% kept (442) | 96.1%7.2% kept (155) |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 51.5%2,163 | 69.1%51.5% kept (1,115) | 81.6%29.9% kept (646) | 91.6%16.0% kept (346) | 96.9%6.0% kept (130) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 55.9%2,163 | 67.9%69.5% kept (1,504) | 77.7%49.1% kept (1,061) | 89.5%27.7% kept (599) | 99.1%5.3% kept (114) |
| GLiDE 2B Open · 1.9B | — | Decision | 36.8%2,163 | 45.5%49.8% kept (1,078) | 56.1%21.8% kept (471) | 73.8%5.6% kept (122) | 93.8%0.7% kept (16) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 96 / 00.779 | 11 / 00.818 | 1 / 00.707 | 0 / 00.524 | 2 / 00.676 | 2 / 00.743 | 7 / 00.748 | 18 / 00.814 | 6 / 00.808 | 11 / 00.806 | 6 / 00.744 | 12 / 00.768 | 12 / 00.778 | 8 / 00.775 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 54 / 00.765 | 3 / 00.791 | 2 / 00.611 | 0 / 00.500 | 1 / 00.688 | 1 / 00.814 | 4 / 00.699 | 11 / 00.845 | 1 / 00.842 | 7 / 00.813 | 1 / 00.742 | 7 / 00.743 | 9 / 00.803 | 7 / 00.747 |
| vera-medium-omni Open · 4.2B | — | Decision | 33 / 00.789 | 2 / 00.782 | 3 / 00.704 | 0 / 00.636 | 1 / 00.695 | 1 / 00.836 | 1 / 00.706 | 13 / 00.860 | 0 / 00.848 | 2 / 00.837 | 1 / 00.797 | 4 / 00.770 | 1 / 00.809 | 4 / 00.784 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 8 / 00.636 | 0 / 00.570 | 0 / 00.578 | 0 / 00.533 | 0 / 00.508 | 0 / 00.540 | 0 / 00.655 | 0 / 00.667 | 1 / 00.779 | 4 / 00.736 | 1 / 00.699 | 1 / 00.644 | 0 / 00.656 | 1 / 00.587 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 4 / 00.589 | 1 / 00.538 | 0 / 00.498 | 0 / 00.467 | 0 / 00.610 | 0 / 00.579 | 0 / 00.525 | 0 / 00.710 | 1 / 00.604 | 0 / 00.601 | 0 / 00.661 | 1 / 00.673 | 0 / 00.564 | 1 / 00.616 |
| d1-2B Open · 1.9B | — | Decision | 4 / 00.656 | 0 / 00.634 | 0 / 00.629 | 0 / 00.318 | 0 / 00.572 | 1 / 00.582 | 0 / 00.589 | 0 / 00.721 | 1 / 00.778 | 1 / 00.625 | 0 / 00.718 | 1 / 00.709 | 0 / 00.657 | 0 / 00.677 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 2 / 00.685 | 0 / 00.661 | 0 / 00.601 | 0 / 00.333 | 0 / 00.560 | 0 / 00.665 | 0 / 00.655 | 0 / 00.788 | 0 / 00.813 | 1 / 00.755 | 0 / 00.703 | 0 / 00.614 | 0 / 00.658 | 1 / 00.679 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | No answers at this threshold0.501 | 0 / 00.524 | 0 / 00.554 | 0 / 00.550 | 0 / 00.476 | 0 / 00.474 | 0 / 00.505 | 0 / 00.412 | 0 / 00.522 | 0 / 00.412 | 0 / 00.460 | 0 / 00.566 | 0 / 00.543 | 0 / 00.518 |
| GLiDE 0.4B Open · 0.4B | — | Decision | No answers at this threshold0.500 | 0 / 00.496 | 0 / 00.528 | 0 / 00.875 | 0 / 00.509 | 0 / 00.606 | 0 / 00.458 | 0 / 00.370 | 0 / 00.632 | 0 / 00.450 | 0 / 00.478 | 0 / 00.493 | 0 / 00.498 | 0 / 00.515 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | No answers at this threshold0.596 | 0 / 00.486 | 0 / 00.603 | 0 / 00.667 | 0 / 00.628 | 0 / 00.573 | 0 / 00.490 | 0 / 00.683 | 0 / 00.792 | 0 / 00.660 | 0 / 00.653 | 0 / 00.603 | 0 / 00.557 | 0 / 00.565 |
| Metask-Jev-4B Open · 4.5B | — | Decision | No answers at this threshold0.651 | 0 / 00.614 | 0 / 00.544 | 0 / 00.933 | 0 / 00.617 | 0 / 00.737 | 0 / 00.613 | 0 / 00.633 | 0 / 00.619 | 0 / 00.666 | 0 / 00.727 | 0 / 00.660 | 0 / 00.624 | 0 / 00.664 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | No answers at this threshold0.592 | 0 / 00.548 | 0 / 00.601 | 0 / 00.533 | 0 / 00.519 | 0 / 00.678 | 0 / 00.474 | 0 / 00.543 | 0 / 00.726 | 0 / 00.690 | 0 / 00.619 | 0 / 00.605 | 0 / 00.609 | 0 / 00.559 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | No answers at this threshold0.583 | 0 / 00.611 | 0 / 00.460 | 0 / 00.667 | 0 / 00.546 | 0 / 00.440 | 0 / 00.612 | 0 / 00.567 | 0 / 00.693 | 0 / 00.672 | 0 / 00.676 | 0 / 00.551 | 0 / 00.503 | 0 / 00.580 |
| Strands Decider 2B Open · 2.0B | — | Decision | No answers at this threshold0.569 | 0 / 00.510 | 0 / 00.458 | 0 / 00.615 | 0 / 00.534 | 0 / 00.589 | 0 / 00.583 | 0 / 00.635 | 0 / 00.615 | 0 / 00.574 | 0 / 00.632 | 0 / 00.556 | 0 / 00.542 | 0 / 00.627 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 150 / 50.781 | 7 / 10.781 | 4 / 00.688 | 0 / 00.300 | 2 / 00.694 | 3 / 00.780 | 9 / 00.732 | 26 / 00.884 | 10 / 10.807 | 21 / 00.854 | 11 / 10.773 | 15 / 10.730 | 19 / 00.806 | 23 / 10.753 |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 149 / 60.746 | 14 / 20.793 | 5 / 00.668 | 0 / 00.424 | 5 / 00.719 | 2 / 00.797 | 5 / 20.655 | 16 / 00.735 | 8 / 10.772 | 19 / 00.732 | 7 / 00.768 | 16 / 00.771 | 24 / 10.729 | 28 / 00.726 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 126 / 40.752 | 14 / 10.731 | 5 / 00.719 | 0 / 00.548 | 4 / 00.677 | 6 / 00.743 | 8 / 10.763 | 16 / 00.824 | 8 / 00.774 | 14 / 00.799 | 9 / 20.772 | 17 / 00.761 | 12 / 00.772 | 13 / 00.696 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 113 / 10.787 | 6 / 00.765 | 5 / 00.695 | 0 / 00.450 | 2 / 00.701 | 3 / 00.836 | 9 / 00.788 | 23 / 00.913 | 7 / 00.830 | 12 / 00.797 | 7 / 00.792 | 11 / 00.764 | 15 / 00.792 | 13 / 10.765 |
| GLiDE 2B Open · 1.9B | — | Decision | 15 / 10.635 | 3 / 10.625 | 0 / 00.569 | 0 / 00.519 | 1 / 00.554 | 0 / 00.590 | 1 / 00.676 | 1 / 00.646 | 1 / 00.728 | 3 / 00.663 | 2 / 00.596 | 1 / 00.617 | 1 / 00.616 | 1 / 00.639 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 17 setups: Qwen3.5-4B (on · chat), MiniCPM5-2B (on · chat), Granite-4.2-3B (high · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), LFM3-1.7B (on · chat), LFM3-1.7B (off · chat), Qwen3.5-4B (off · chat), MiniCPM5-2B (off · chat), Gemma 4 E2B (on · chat · 32k cap), Gemma 4 E2B (on · chat · 32k cap · sampled), Raon-1 nano (on · chat), Gemma 4 E2B (on · chat), Gemma 4 E2B (chat), Qwen3.5-2B (on · chat), Qwen3.5-2B (off · chat), LFM2.5-2.6B (on · chat), Raon-1 nano (off · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Logic 92.4% | — | Chat | 99.4%99.0%–99.7% | a | 2,151 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Logic 88.9% | — | Chat | 99.2%99.2%–99.3% min–max of 3 seeds | ab | 2,147 / 2,163seed 0 run | Not measured | Not measured | — | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Logic 91.6% | — | Chat | 99.2%98.7%–99.5% | ab | 2,146 / 2,163 | Not measured | Not measured | — | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% | — | Chat | 98.9%98.8%–99.0% min–max of 3 seeds | abc | 2,142 / 2,163seed 0 run | Not measured | Not measured | — | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B | — | Chat | 98.9%98.8%–99.1% min–max of 3 seeds | abcd | 2,139 / 2,163seed 0 run | Not measured | Not measured | — | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Logic 85.5% | — | Chat | 98.7%98.1%–99.1% | abcde | 2,135 / 2,163 | Not measured | Not measured | — | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Logic 90.1% | — | Chat | 98.5%97.9%–98.9% | bcde | 2,130 / 2,163 | Not measured | Not measured | — | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Logic 83.6% | — | Chat | 98.2%98.1%–98.3% min–max of 3 seeds | cdef | 2,126 / 2,163seed 0 run | Not measured | Not measured | — | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% | — | Chat | 97.8%97.1%–98.4% | defg | 2,116 / 2,163 | Not measured | Not measured | — | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 76.8% | — | Chat | 97.7%97.6%–97.8% min–max of 3 seeds | efg | 2,111 / 2,163seed 0 run | Not measured | Not measured | — | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% | — | Chat | 97.0%96.2%–97.6% | fg | 2,098 / 2,163 | Not measured | Not measured | — | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 74.6% | — | Chat | 96.9%96.1%–97.6% | fg | 2,097 / 2,163 | Not measured | Not measured | — | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 72.9% | — | Chat | 96.8%95.9%–97.4% | g | 2,093 / 2,163 | Not measured | Not measured | — | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B | — | Chat | 94.7%93.7%–95.6% | h | 2,049 / 2,163 | Not measured | Not measured | — | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Logic 64.8% | — | Chat | 94.6%93.6%–95.5% | h | 2,047 / 2,163 | Not measured | Not measured | — | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Logic 75.4% | — | Chat | 94.0%92.9%–94.9% | h | 2,033 / 2,163 | Not measured | Not measured | — | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 65.1% | — | Chat | 89.6%88.2%–90.8% | i | 1,937 / 2,163 | Not measured | Not measured | — | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 33 answers, 33 right · 0 wrong · AUROC 0.789 Logic 44.1% | — | Decision | 56.8%54.7%–58.9% | j | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 69.18chosen-answer Brier 0.1846 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.765 Logic 42.3% | — | Decision | 56.0%53.9%–58.1% | j | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 67.89chosen-answer Brier 0.2021 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 114 answers, 113 right · 1 wrong · AUROC 0.787 Logic 44.7% | — | Decision | 55.9%53.8%–58.0% | j | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 67.89chosen-answer Brier 0.2015 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 96 answers, 96 right · 0 wrong · AUROC 0.779 Logic 45.9% | — | Decision | 55.8%53.7%–57.9% | j | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 68.25chosen-answer Brier 0.1934 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 155 answers, 150 right · 5 wrong · AUROC 0.781 Logic 43.1% | — | Decision | 55.2%53.1%–57.3% | j | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 67.61chosen-answer Brier 0.2002 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 130 answers, 126 right · 4 wrong · AUROC 0.752 Logic 39.7% | — | Decision | 51.5%49.4%–53.6% | k | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 65.62chosen-answer Brier 0.2026 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 155 answers, 149 right · 6 wrong · AUROC 0.746 Logic 40.4% | — | Decision | 50.4%48.3%–52.5% | k | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 64.11chosen-answer Brier 0.2221 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.651 Logic 32.1% | — | Decision | 39.2%37.1%–41.2% | l | 847 / 2,163 | No answers at this threshold | 0.651 | 58.63chosen-answer Brier 0.2190 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.685 Logic 30.8% | — | Decision | 38.7%36.6%–40.7% | l | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 58.95chosen-answer Brier 0.2075 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.656 Logic 30.7% | — | Decision | 37.1%35.1%–39.1% | l | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 57.72chosen-answer Brier 0.2163 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 16 answers, 15 right · 1 wrong · AUROC 0.635 Logic 32.3% | — | Decision | 36.8%34.7%–38.8% | l | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 55.70chosen-answer Brier 0.2536 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.636 Logic 31.7% | — | Decision | 35.3%33.3%–37.3% | lm | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 56.49chosen-answer Brier 0.2229 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.589 Logic 29.2% | — | Decision | 32.1%30.2%–34.1% | mn | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 53.21chosen-answer Brier 0.2566 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.583 Logic 24.7% | — | Decision | 29.7%27.8%–31.6% | n | 642 / 2,163 | No answers at this threshold | 0.583 | 54.40chosen-answer Brier 0.2088 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.569 Logic 22.4% | — | Decision | 28.7%26.8%–30.6% | n | 620 / 2,163 | No answers at this threshold | 0.569 | 53.90chosen-answer Brier 0.2086 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: No answers at this threshold · AUROC 0.592 Logic 27.0% | — | Decision | 28.2%26.4%–30.2% | n | 611 / 2,163 | No answers at this threshold | 0.592 | 53.62chosen-answer Brier 0.2100 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.596 Logic 26.7% | — | Decision | 28.2%26.3%–30.1% | n | 610 / 2,163 | No answers at this threshold | 0.596 | 53.60chosen-answer Brier 0.2100 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.501 Logic 23.6% | — | Decision | 22.7%20.9%–24.5% | o | 490 / 2,163 | No answers at this threshold | 0.501 | 52.01chosen-answer Brier 0.1863 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.500 Logic 23.2% | — | Decision | 20.6%18.9%–22.3% | o | 445 / 2,163 | No answers at this threshold | 0.500 | 50.98chosen-answer Brier 0.1862 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Valen-Team PelaAI Fastino tattrongvu StartLux Cloudflare Blockbrain Labs Apertix AI Patrick-18 newsbang Base Labs EldanRing Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 55.3%2,163 | 74.3%54.9% kept (1,187) | 86.7%32.3% kept (698) | 94.9%17.3% kept (374) | 100.0%6.0% kept (130) |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 59.8%2,163 | 76.7%58.4% kept (1,264) | 89.3%36.8% kept (797) | 98.0%20.5% kept (444) | 100.0%5.7% kept (123) |
| GLiDE 9B Open · 7.9B | — | Decision | 57.9%2,163 | 70.8%68.3% kept (1,478) | 84.7%42.8% kept (926) | 94.7%20.9% kept (451) | 100.0%3.4% kept (73) |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 59.8%2,163 | 77.8%60.3% kept (1,304) | 90.4%38.1% kept (824) | 98.5%18.2% kept (393) | 100.0%3.0% kept (64) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 58.4%2,163 | 74.9%58.2% kept (1,258) | 85.1%37.4% kept (808) | 96.6%19.1% kept (413) | 100.0%2.3% kept (49) |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 66.3%2,163 | 83.7%60.8% kept (1,315) | 92.1%42.2% kept (912) | 99.1%19.6% kept (425) | 100.0%0.3% kept (7) |
| Clef-Flash Open · 9.0B | — | Decision | 49.8%2,163 | 68.7%44.7% kept (967) | 81.3%23.3% kept (504) | 94.4%7.4% kept (161) | none kept |
| Cygnet Open · 12B | — | Decision | 44.3%2,163 | 65.9%41.9% kept (907) | 84.2%14.9% kept (322) | 98.7%3.5% kept (75) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 60.1%2,163 | 80.7%52.3% kept (1,131) | 92.5%30.9% kept (668) | 99.6%10.3% kept (223) | none kept |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 56.0%2,163 | 64.6%75.7% kept (1,638) | 74.3%55.1% kept (1,192) | 86.2%33.8% kept (731) | 96.1%11.7% kept (254) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 58.8%2,163 | 69.2%74.8% kept (1,619) | 79.2%53.7% kept (1,162) | 91.9%32.1% kept (694) | 98.6%9.7% kept (210) |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 62.1%2,163 | 74.7%68.7% kept (1,486) | 86.2%46.0% kept (994) | 96.3%27.2% kept (589) | 99.5%9.2% kept (198) |
| Elephant-1 Open · 9.0B | — | Decision | 61.3%2,163 | 75.7%65.6% kept (1,419) | 86.7%44.0% kept (952) | 95.8%26.2% kept (567) | 99.5%8.5% kept (183) |
| Winnow-12B Open · 12B | — | Decision | 46.9%2,163 | 64.6%47.3% kept (1,023) | 77.3%30.4% kept (657) | 89.5%16.2% kept (351) | 95.2%1.0% kept (21) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 130 / 00.790 | 16 / 00.837 | 3 / 00.663 | 0 / 00.667 | 7 / 00.778 | 4 / 00.849 | 5 / 00.753 | 17 / 00.876 | 9 / 00.765 | 12 / 00.777 | 7 / 00.834 | 13 / 00.782 | 16 / 00.795 | 21 / 00.776 |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 123 / 00.799 | 13 / 00.816 | 5 / 00.712 | 0 / 00.595 | 6 / 00.674 | 6 / 00.803 | 5 / 00.760 | 19 / 00.873 | 6 / 00.799 | 14 / 00.792 | 7 / 00.780 | 17 / 00.803 | 11 / 00.821 | 14 / 00.817 |
| GLiDE 9B Open · 7.9B | — | Decision | 73 / 00.798 | 7 / 00.800 | 2 / 00.737 | 0 / 00.731 | 3 / 00.700 | 1 / 00.716 | 4 / 00.752 | 15 / 00.805 | 3 / 00.774 | 10 / 00.873 | 4 / 00.853 | 7 / 00.776 | 8 / 00.851 | 9 / 00.765 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 64 / 00.814 | 4 / 00.865 | 4 / 00.677 | 0 / 00.769 | 3 / 00.732 | 3 / 00.844 | 3 / 00.745 | 15 / 00.811 | 4 / 00.797 | 6 / 00.888 | 2 / 00.825 | 5 / 00.804 | 7 / 00.808 | 8 / 00.768 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 49 / 00.781 | 3 / 00.777 | 1 / 00.678 | 0 / 00.394 | 1 / 00.703 | 1 / 00.730 | 3 / 00.735 | 11 / 00.887 | 3 / 00.774 | 6 / 00.803 | 5 / 00.830 | 4 / 00.775 | 2 / 00.788 | 9 / 00.779 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 7 / 00.819 | 2 / 00.838 | 0 / 00.736 | 0 / 00.814 | 0 / 00.752 | 0 / 00.840 | 0 / 00.833 | 5 / 00.915 | 0 / 00.800 | 0 / 00.824 | 0 / 00.916 | 0 / 00.775 | 0 / 00.857 | 0 / 00.794 |
| Clef-Flash Open · 9.0B | — | Decision | No answers at this threshold0.730 | 0 / 00.711 | 0 / 00.642 | 0 / 00.361 | 0 / 00.614 | 0 / 00.736 | 0 / 00.670 | 0 / 00.766 | 0 / 00.789 | 0 / 00.744 | 0 / 00.823 | 0 / 00.722 | 0 / 00.804 | 0 / 00.737 |
| Cygnet Open · 12B | — | Decision | No answers at this threshold0.735 | 0 / 00.730 | 0 / 00.617 | 0 / 00.600 | 0 / 00.690 | 0 / 00.639 | 0 / 00.726 | 0 / 00.800 | 0 / 00.788 | 0 / 00.796 | 0 / 00.697 | 0 / 00.680 | 0 / 00.768 | 0 / 00.761 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | No answers at this threshold0.803 | 0 / 00.788 | 0 / 00.711 | 0 / 00.712 | 0 / 00.671 | 0 / 00.801 | 0 / 00.819 | 0 / 00.834 | 0 / 00.841 | 0 / 00.847 | 0 / 00.824 | 0 / 00.774 | 0 / 00.806 | 0 / 00.817 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 244 / 100.780 | 24 / 00.806 | 9 / 00.677 | 1 / 10.561 | 8 / 00.679 | 6 / 00.797 | 11 / 00.751 | 31 / 00.843 | 17 / 00.747 | 22 / 20.806 | 13 / 00.812 | 31 / 20.720 | 33 / 30.836 | 38 / 20.776 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 207 / 30.804 | 18 / 10.778 | 5 / 00.684 | 0 / 00.606 | 7 / 00.731 | 7 / 10.745 | 8 / 00.759 | 29 / 10.839 | 10 / 00.829 | 18 / 00.822 | 12 / 00.853 | 33 / 00.824 | 24 / 00.814 | 36 / 00.820 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 197 / 10.806 | 18 / 00.815 | 4 / 00.701 | 0 / 00.729 | 6 / 00.684 | 8 / 00.764 | 9 / 00.797 | 31 / 00.867 | 12 / 10.814 | 21 / 00.820 | 16 / 00.840 | 23 / 00.813 | 21 / 00.812 | 28 / 00.806 |
| Elephant-1 Open · 9.0B | — | Decision | 182 / 10.803 | 14 / 00.821 | 5 / 00.711 | 0 / 00.486 | 6 / 00.684 | 6 / 00.823 | 10 / 00.730 | 28 / 00.833 | 10 / 10.785 | 15 / 00.868 | 14 / 00.879 | 25 / 00.778 | 24 / 00.834 | 25 / 00.774 |
| Winnow-12B Open · 12B | — | Decision | 20 / 10.743 | 5 / 00.742 | 1 / 00.638 | 0 / 00.686 | 0 / 00.670 | 1 / 00.642 | 0 / 10.743 | 1 / 00.787 | 1 / 00.742 | 5 / 00.731 | 3 / 00.785 | 1 / 00.746 | 1 / 00.758 | 1 / 00.774 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 8 setups: Gemma 4 12B Unified (chat), Gemma 4 E4B (on · chat · 32k cap · sampled), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E4B (on · chat), Gemma 4 E4B (chat), LFM2.5-8B-A1B (on · chat), Qwen3-14B (off · chat), Erk-14B (off · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Logic 87.7% | — | Chat | 98.8%98.3%–99.2% | a | 2,138 / 2,163 | Not measured | Not measured | — | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.9% | — | Chat | 98.5%98.4%–98.8% min–max of 3 seeds | a | 2,136 / 2,163seed 0 run | Not measured | Not measured | — | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% | — | Chat | 98.5%97.9%–99.0% | a | 2,131 / 2,163 | Not measured | Not measured | — | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 82.6% | — | Chat | 98.2%97.6%–98.7% | a | 2,125 / 2,163 | Not measured | Not measured | — | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 83.2% | — | Chat | 98.1%97.4%–98.6% | a | 2,122 / 2,163 | Not measured | Not measured | — | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Logic 60.2% | — | Chat | 96.3%95.4%–97.0% | b | 2,082 / 2,163 | Not measured | Not measured | — | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Logic 65.3% | — | Chat | 95.8%94.9%–96.6% | b | 2,072 / 2,163 | Not measured | Not measured | — | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Logic 54.2% | — | Chat | 90.8%89.5%–91.9% | c | 1,963 / 2,163 | Not measured | Not measured | — | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.819 Logic 50.6% | — | Decision | 66.3%64.2%–68.2% | d | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 74.96chosen-answer Brier 0.1632 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 198 answers, 197 right · 1 wrong · AUROC 0.806 Logic 46.4% | — | Decision | 62.1%60.0%–64.1% | e | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 72.28chosen-answer Brier 0.1752 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 183 answers, 182 right · 1 wrong · AUROC 0.803 Logic 46.4% | — | Decision | 61.3%59.3%–63.4% | ef | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 71.86chosen-answer Brier 0.1763 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.803 Logic 43.9% | — | Decision | 60.1%58.0%–62.1% | efg | 1,300 / 2,163 | No answers at this threshold | 0.803 | 71.15chosen-answer Brier 0.1779 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 64 answers, 64 right · 0 wrong · AUROC 0.814 Logic 48.2% | — | Decision | 59.8%57.7%–61.9% | efg | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 71.32chosen-answer Brier 0.1719 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 123 answers, 123 right · 0 wrong · AUROC 0.799 Logic 44.3% | — | Decision | 59.8%57.7%–61.8% | efg | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 71.08chosen-answer Brier 0.1762 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 210 answers, 207 right · 3 wrong · AUROC 0.804 Logic 45.3% | — | Decision | 58.8%56.7%–60.9% | fgh | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 69.67chosen-answer Brier 0.1946 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 49 answers, 49 right · 0 wrong · AUROC 0.781 Logic 46.6% | — | Decision | 58.4%56.3%–60.5% | fgh | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 69.90chosen-answer Brier 0.1864 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 73 answers, 73 right · 0 wrong · AUROC 0.798 Logic 47.8% | — | Decision | 57.9%55.8%–60.0% | gh | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 69.69chosen-answer Brier 0.1855 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 254 answers, 244 right · 10 wrong · AUROC 0.780 Logic 43.3% | — | Decision | 56.0%53.9%–58.1% | h | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 67.15chosen-answer Brier 0.2174 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 130 answers, 130 right · 0 wrong · AUROC 0.790 Logic 44.4% | — | Decision | 55.3%53.2%–57.4% | h | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 68.37chosen-answer Brier 0.1860 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.730 Logic 38.9% | — | Decision | 49.8%47.7%–51.9% | i | 1,077 / 2,163 | No answers at this threshold | 0.730 | 64.44chosen-answer Brier 0.2092 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 21 answers, 20 right · 1 wrong · AUROC 0.743 Logic 39.7% | — | Decision | 46.9%44.8%–49.0% | ij | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 63.00chosen-answer Brier 0.2087 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.735 Logic 39.3% | — | Decision | 44.3%42.3%–46.4% | j | 959 / 2,163 | No answers at this threshold | 0.735 | 61.87chosen-answer Brier 0.2060 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
MachinePulse Base Labs Perplexity Poindexter Labs Sthānika AI Nace.AI Fastino Turion.AI OmniJev Cloudflare vLLM Semantic Router Zefan Cai PelaAI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | Decision | 81.6%2,163 | 88.9%86.7% kept (1,876) | 94.2%73.7% kept (1,594) | 98.0%58.1% kept (1,256) | 100.0%38.7% kept (837) |
| Mammoth-1 Open · 27B | — | Decision | 74.0%2,163 | 85.3%79.1% kept (1,712) | 94.3%61.8% kept (1,336) | 98.5%45.7% kept (988) | 100.0%19.0% kept (412) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 69.8%2,163 | 83.0%73.5% kept (1,590) | 93.1%56.3% kept (1,217) | 98.2%38.7% kept (838) | 100.0%15.3% kept (331) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 69.3%2,163 | 82.5%73.5% kept (1,590) | 92.8%56.2% kept (1,215) | 97.7%38.8% kept (840) | 100.0%15.1% kept (326) |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 72.5%2,163 | 81.2%81.1% kept (1,754) | 90.0%64.0% kept (1,384) | 96.3%46.1% kept (998) | 100.0%14.9% kept (323) |
| Copernicus-31B Open · 31B | — | Decision | 77.3%2,163 | 91.3%73.4% kept (1,587) | 97.2%57.5% kept (1,244) | 99.4%39.0% kept (843) | 100.0%14.7% kept (318) |
| Sieve 31B v3 Open · 31B | — | Decision | 77.1%2,163 | 90.0%75.3% kept (1,628) | 95.9%59.6% kept (1,289) | 99.3%41.1% kept (888) | 100.0%9.9% kept (215) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 59.0%2,163 | 80.4%53.0% kept (1,147) | 92.2%35.0% kept (758) | 97.8%20.6% kept (445) | 100.0%8.0% kept (174) |
| GLiDE v1.2 Open · 27B | — | Decision | 69.9%2,163 | 81.1%75.9% kept (1,641) | 92.6%56.4% kept (1,219) | 99.1%34.4% kept (745) | 100.0%6.1% kept (131) |
| Turion Jev 31B Open · 31B | — | Decision | 69.0%2,163 | 84.7%68.7% kept (1,487) | 92.8%51.1% kept (1,106) | 99.8%26.0% kept (562) | 100.0%2.0% kept (43) |
| OneJev-27B Open · 27B | — | Decision | 53.8%2,163 | 81.8%39.5% kept (855) | 92.5%23.3% kept (504) | 98.3%10.6% kept (229) | 100.0%1.2% kept (26) |
| Clef Open · 27B | — | Decision | 51.1%2,163 | 65.2%62.2% kept (1,346) | 78.9%36.8% kept (796) | 90.8%17.6% kept (381) | 100.0%0.6% kept (12) |
| d3-ultra Open · 26B | — | Decision | 71.5%2,163 | 78.5%85.2% kept (1,842) | 86.2%68.6% kept (1,484) | 97.4%43.7% kept (946) | 100.0%0.5% kept (11) |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 54.6%2,163 | 95.8%10.0% kept (216) | 100.0%1.2% kept (25) | none kept | none kept |
| Drex Max Open · 27B | — | Decision | 70.3%2,163 | 76.2%86.7% kept (1,875) | 83.3%72.2% kept (1,561) | 92.9%55.9% kept (1,209) | 98.6%33.7% kept (729) |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 80.3%2,163 | 90.0%80.7% kept (1,746) | 95.3%66.5% kept (1,438) | 99.3%48.1% kept (1,040) | 99.8%21.7% kept (470) |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 77.4%2,163 | 89.5%77.0% kept (1,665) | 95.3%61.3% kept (1,327) | 99.1%42.7% kept (923) | 99.8%21.5% kept (465) |
| GLiDE 27B Open · 27B | — | Decision | 72.0%2,163 | 82.1%78.7% kept (1,703) | 91.2%60.6% kept (1,311) | 98.7%39.5% kept (855) | 99.5%9.2% kept (199) |
| d3 Open · 26B | — | Decision | 70.4%2,163 | 85.3%71.8% kept (1,553) | 95.2%53.7% kept (1,161) | 99.3%34.0% kept (735) | 99.4%7.5% kept (163) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | Decision | 837 / 00.903 | 85 / 00.899 | 34 / 00.801 | 3 / 00.833 | 34 / 00.880 | 38 / 00.933 | 50 / 00.890 | 83 / 00.952 | 45 / 00.879 | 66 / 00.915 | 53 / 00.950 | 91 / 00.902 | 114 / 00.893 | 141 / 00.904 |
| Mammoth-1 Open · 27B | — | Decision | 412 / 00.895 | 27 / 00.885 | 9 / 00.766 | 1 / 00.788 | 11 / 00.821 | 14 / 00.876 | 30 / 00.897 | 41 / 00.909 | 17 / 00.893 | 47 / 00.930 | 31 / 00.904 | 53 / 00.900 | 59 / 00.896 | 72 / 00.903 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 331 / 00.869 | 22 / 00.861 | 6 / 00.818 | 1 / 00.806 | 8 / 00.772 | 10 / 00.823 | 16 / 00.819 | 34 / 00.961 | 34 / 00.901 | 43 / 00.903 | 33 / 00.932 | 32 / 00.840 | 47 / 00.875 | 45 / 00.863 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 326 / 00.873 | 23 / 00.850 | 9 / 00.842 | 1 / 00.571 | 6 / 00.827 | 14 / 00.799 | 18 / 00.854 | 28 / 00.927 | 36 / 00.869 | 43 / 00.887 | 31 / 00.946 | 29 / 00.851 | 41 / 00.885 | 47 / 00.884 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 323 / 00.856 | 26 / 00.828 | 8 / 00.776 | 0 / 00.577 | 8 / 00.785 | 15 / 00.845 | 17 / 00.790 | 36 / 00.935 | 27 / 00.839 | 25 / 00.900 | 22 / 00.933 | 35 / 00.845 | 49 / 00.871 | 55 / 00.834 |
| Copernicus-31B Open · 31B | — | Decision | 318 / 00.899 | 27 / 00.901 | 13 / 00.865 | 1 / 00.800 | 9 / 00.881 | 17 / 00.892 | 20 / 00.881 | 42 / 00.980 | 12 / 00.884 | 25 / 00.876 | 19 / 00.937 | 29 / 00.882 | 48 / 00.914 | 56 / 00.875 |
| Sieve 31B v3 Open · 31B | — | Decision | 215 / 00.891 | 24 / 00.871 | 6 / 00.848 | 0 / 00.903 | 7 / 00.856 | 8 / 00.898 | 10 / 00.863 | 26 / 00.956 | 15 / 00.918 | 22 / 00.907 | 14 / 00.939 | 24 / 00.895 | 31 / 00.879 | 28 / 00.848 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 174 / 00.813 | 8 / 00.819 | 6 / 00.817 | 0 / 00.533 | 0 / 00.740 | 5 / 00.798 | 13 / 00.841 | 20 / 00.905 | 9 / 00.768 | 16 / 00.837 | 14 / 00.766 | 22 / 00.762 | 35 / 00.836 | 26 / 00.803 |
| GLiDE v1.2 Open · 27B | — | Decision | 131 / 00.862 | 12 / 00.857 | 3 / 00.740 | 0 / 00.528 | 5 / 00.777 | 4 / 00.834 | 5 / 00.838 | 20 / 00.963 | 14 / 00.865 | 22 / 00.912 | 7 / 00.948 | 15 / 00.873 | 14 / 00.849 | 10 / 00.877 |
| Turion Jev 31B Open · 31B | — | Decision | 43 / 00.864 | 5 / 00.868 | 1 / 00.770 | 0 / 00.757 | 1 / 00.809 | 0 / 00.904 | 3 / 00.817 | 3 / 00.926 | 1 / 00.854 | 6 / 00.823 | 1 / 00.940 | 8 / 00.853 | 5 / 00.866 | 9 / 00.852 |
| OneJev-27B Open · 27B | — | Decision | 26 / 00.783 | 2 / 00.768 | 0 / 00.645 | 0 / 00.806 | 1 / 00.750 | 0 / 00.767 | 1 / 00.778 | 2 / 00.884 | 1 / 00.758 | 1 / 00.811 | 3 / 00.787 | 4 / 00.757 | 6 / 00.785 | 5 / 00.789 |
| Clef Open · 27B | — | Decision | 12 / 00.767 | 2 / 00.751 | 1 / 00.708 | 0 / 00.875 | 2 / 00.723 | 0 / 00.720 | 0 / 00.798 | 3 / 00.778 | 0 / 00.793 | 1 / 00.744 | 0 / 00.878 | 2 / 00.748 | 0 / 00.802 | 1 / 00.774 |
| d3-ultra Open · 26B | — | Decision | 11 / 00.857 | 1 / 00.842 | 1 / 00.778 | 0 / 00.764 | 0 / 00.820 | 0 / 00.731 | 0 / 00.827 | 3 / 00.942 | 0 / 00.819 | 1 / 00.901 | 2 / 00.848 | 1 / 00.871 | 1 / 00.850 | 1 / 00.874 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | No answers at this threshold0.764 | 0 / 00.816 | 0 / 00.607 | 0 / 00.625 | 0 / 00.674 | 0 / 00.731 | 0 / 00.751 | 0 / 00.762 | 0 / 00.745 | 0 / 00.847 | 0 / 00.818 | 0 / 00.776 | 0 / 00.727 | 0 / 00.749 |
| Drex Max Open · 27B | — | Decision | 719 / 100.861 | 67 / 10.834 | 25 / 00.803 | 3 / 00.750 | 32 / 00.799 | 28 / 00.867 | 36 / 40.794 | 63 / 00.923 | 42 / 20.877 | 75 / 00.894 | 44 / 00.908 | 83 / 00.876 | 96 / 20.855 | 125 / 10.847 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 469 / 10.898 | 38 / 00.916 | 15 / 00.862 | 2 / 00.742 | 14 / 00.847 | 24 / 00.902 | 37 / 00.921 | 70 / 00.956 | 12 / 00.871 | 28 / 00.882 | 28 / 00.916 | 56 / 10.863 | 68 / 00.908 | 77 / 00.886 |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 464 / 10.895 | 38 / 00.874 | 15 / 00.850 | 1 / 00.750 | 14 / 00.881 | 22 / 00.903 | 32 / 00.900 | 60 / 00.956 | 18 / 00.912 | 35 / 00.872 | 33 / 00.904 | 55 / 10.885 | 64 / 00.914 | 77 / 00.888 |
| GLiDE 27B Open · 27B | — | Decision | 198 / 10.864 | 17 / 10.858 | 6 / 00.787 | 0 / 00.569 | 8 / 00.761 | 5 / 00.846 | 7 / 00.870 | 22 / 00.948 | 20 / 00.938 | 36 / 00.873 | 18 / 00.905 | 21 / 00.861 | 20 / 00.848 | 18 / 00.875 |
| d3 Open · 26B | — | Decision | 162 / 10.878 | 13 / 10.867 | 5 / 00.791 | 1 / 00.806 | 6 / 00.811 | 7 / 00.907 | 8 / 00.847 | 21 / 00.955 | 15 / 00.904 | 25 / 00.905 | 13 / 00.893 | 21 / 00.886 | 13 / 00.875 | 14 / 00.872 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 3 setups: Gemma 4 31B (chat), Qwen3.8-27B (low · chat), LFM2.5-24B-A2B (on · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B | — | Chat | 99.4%99.0%–99.6% | a | 2,150 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | a | 2,146 / 2,163 | Not measured | Not measured | — | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Logic 86.7% | — | Chat | 99.0%98.5%–99.4% | a | 2,142 / 2,163 | Not measured | Not measured | — | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 837 answers, 837 right · 0 wrong · AUROC 0.903 Logic 60.7% | — | Decision | 81.6%79.9%–83.1% | b | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 85.99chosen-answer Brier 0.0958 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 470 answers, 469 right · 1 wrong · AUROC 0.898 Logic 58.9% | — | Decision | 80.3%78.5%–81.9% | b | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 85.05chosen-answer Brier 0.1016 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 465 answers, 464 right · 1 wrong · AUROC 0.895 Logic 59.0% | — | Decision | 77.4%75.6%–79.1% | c | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 83.18chosen-answer Brier 0.1103 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 318 answers, 318 right · 0 wrong · AUROC 0.899 Logic 58.7% | — | Decision | 77.3%75.5%–79.1% | c | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 83.06chosen-answer Brier 0.1122 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 215 answers, 215 right · 0 wrong · AUROC 0.891 Logic 57.9% | — | Decision | 77.1%75.3%–78.8% | cd | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 82.89chosen-answer Brier 0.1133 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 412 answers, 412 right · 0 wrong · AUROC 0.895 Logic 54.5% | — | Decision | 74.0%72.1%–75.8% | de | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 81.16chosen-answer Brier 0.1164 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 323 answers, 323 right · 0 wrong · AUROC 0.856 Logic 54.8% | — | Decision | 72.5%70.6%–74.3% | ef | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 79.32chosen-answer Brier 0.1385 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 199 answers, 198 right · 1 wrong · AUROC 0.864 Logic 58.4% | — | Decision | 72.0%70.1%–73.8% | efg | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 79.22chosen-answer Brier 0.1354 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 11 answers, 11 right · 0 wrong · AUROC 0.857 Logic 54.6% | — | Decision | 71.5%69.6%–73.4% | efg | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 78.39chosen-answer Brier 0.1474 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 163 answers, 162 right · 1 wrong · AUROC 0.878 Logic 54.5% | — | Decision | 70.4%68.5%–72.3% | fg | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 78.71chosen-answer Brier 0.1299 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 729 answers, 719 right · 10 wrong · AUROC 0.861 Logic 53.1% | — | Decision | 70.3%68.4%–72.2% | fg | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 77.10chosen-answer Brier 0.1611 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 131 answers, 131 right · 0 wrong · AUROC 0.862 Logic 55.2% | — | Decision | 69.9%67.9%–71.8% | fg | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 77.95chosen-answer Brier 0.1396 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 331 answers, 331 right · 0 wrong · AUROC 0.869 Logic 53.3% | — | Decision | 69.8%67.8%–71.7% | g | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 78.16chosen-answer Brier 0.1345 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 326 answers, 326 right · 0 wrong · AUROC 0.873 Logic 52.5% | with web | Decision | 69.3%67.3%–71.2% | g | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 77.94chosen-answer Brier 0.1342 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 43 answers, 43 right · 0 wrong · AUROC 0.864 Logic 52.3% | — | Decision | 69.0%67.0%–70.9% | g | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 77.52chosen-answer Brier 0.1399 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 174 answers, 174 right · 0 wrong · AUROC 0.813 Logic 41.9% | — | Decision | 59.0%56.9%–61.0% | h | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 70.98chosen-answer Brier 0.1703 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.764 Logic 42.6% | — | Decision | 54.6%52.4%–56.6% | i | 1,180 / 2,163 | No answers at this threshold | 0.764 | 64.53chosen-answer Brier 0.2549 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 26 answers, 26 right · 0 wrong · AUROC 0.783 Logic 40.6% | — | Decision | 53.8%51.7%–55.9% | i | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 67.54chosen-answer Brier 0.1869 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 12 answers, 12 right · 0 wrong · AUROC 0.767 Logic 40.8% | — | Decision | 51.1%49.0%–53.2% | i | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 65.20chosen-answer Brier 0.2070 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
REAX Logits Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | Decision | 78.0%2,163 | 90.9%76.9% kept (1,664) | 97.5%58.5% kept (1,265) | 99.9%40.5% kept (877) | 100.0%8.2% kept (178) |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 75.2%2,163 | 89.2%73.5% kept (1,589) | 98.3%52.4% kept (1,134) | 100.0%26.7% kept (578) | 100.0%0.7% kept (15) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | Decision | 178 / 00.903 | 11 / 00.888 | 7 / 00.826 | 1 / 00.818 | 7 / 00.869 | 9 / 00.886 | 11 / 00.917 | 30 / 00.955 | 10 / 00.946 | 14 / 00.881 | 7 / 00.935 | 22 / 00.887 | 25 / 00.913 | 24 / 00.900 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 15 / 00.899 | 2 / 00.900 | 0 / 00.841 | 0 / 00.667 | 0 / 00.847 | 1 / 00.888 | 0 / 00.872 | 2 / 00.950 | 0 / 00.919 | 1 / 00.910 | 1 / 00.939 | 4 / 00.895 | 1 / 00.871 | 3 / 00.911 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 7 setups: DeepSeek-V4.1-Flash (high · chat · 1017), Inkling-Small (high · chat · Jul 2026), Inkling 2 Small (high · chat), MiMo-V2.6-Flash (on · chat), DeepSeek-V4-Flash (chat), GLM-5.3-Flash (high · chat), MiniMax-M3.1-Flash (high · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Logic 96.2% | — | Chat | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | — | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Logic 95.4% | — | Chat | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Logic 95.6% | — | Chat | 99.5%99.2%–99.7% | a | 2,153 / 2,163 | Not measured | Not measured | — | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Logic 95.1% | — | Chat | 99.4%99.0%–99.7% | ab | 2,151 / 2,163 | Not measured | Not measured | — | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Logic running | — | Chat | 99.3%98.9%–99.6% | ab | 2,148 / 2,163 | Not measured | Not measured | — | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Logic 93.2% | — | Chat | 98.8%98.2%–99.2% | bc | 2,137 / 2,163 | Not measured | Not measured | — | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Logic 94.0% | — | Chat | 98.2%97.6%–98.7% | c | 2,125 / 2,163 | Not measured | Not measured | — | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 178 answers, 178 right · 0 wrong · AUROC 0.903 Logic 59.3% | — | Decision | 78.0%76.2%–79.7% | d | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 83.57chosen-answer Brier 0.1084 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 15 answers, 15 right · 0 wrong · AUROC 0.899 Logic 57.8% | — | Decision | 75.2%73.4%–77.0% | e | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 81.60chosen-answer Brier 0.1201 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Nace.AI TypeSafe Google Liquid AI Fastino Microsoft Celeris Subquadratic Inception Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 70.1%2,163 | 79.7%79.8% kept (1,725) | 89.3%62.5% kept (1,352) | 97.3%43.4% kept (938) | 100.0%14.6% kept (315) |
| Jev 1.13 Closed · Efficient | — | Decision | 64.1%2,163 | 80.3%65.2% kept (1,410) | 91.3%42.7% kept (924) | 98.3%25.1% kept (543) | 100.0%11.5% kept (248) |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 71.3%2,163 | 85.3%72.1% kept (1,560) | 92.7%55.2% kept (1,194) | 98.2%36.3% kept (785) | 100.0%9.9% kept (214) |
| d1-ultra Closed · Efficient | — | Decision | 74.1%2,163 | 87.8%73.4% kept (1,588) | 94.0%59.0% kept (1,276) | 98.5%40.8% kept (882) | 100.0%9.8% kept (213) |
| Jev 1.14 Closed · Efficient | — | Decision | 65.1%2,163 | 82.2%61.7% kept (1,335) | 92.8%40.5% kept (876) | 98.3%24.0% kept (519) | 100.0%9.3% kept (202) |
| d1.1 Closed · Efficient | — | Decision | 65.2%2,163 | 82.7%61.1% kept (1,322) | 92.6%41.5% kept (897) | 97.6%24.9% kept (538) | 100.0%8.5% kept (183) |
| GLiDE API Closed · Efficient | — | Decision | 70.8%2,163 | 81.6%77.6% kept (1,678) | 92.0%58.2% kept (1,258) | 98.6%37.4% kept (808) | 100.0%7.4% kept (161) |
| d1 Closed · Efficient | — | Decision | 55.6%2,163 | 73.6%55.1% kept (1,192) | 85.9%32.1% kept (694) | 95.4%17.0% kept (368) | 100.0%7.0% kept (152) |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 59.4%2,163 | 73.4%63.9% kept (1,382) | 88.5%36.2% kept (782) | 98.0%13.8% kept (299) | 100.0%1.2% kept (27) |
| celeris-1-decision Closed · Efficient | — | Decision | 70.0%2,163 | 80.9%77.1% kept (1,667) | 90.9%58.3% kept (1,262) | 96.5%40.4% kept (874) | none kept |
| celeris-1.1-decision Closed · Efficient | — | Decision | 69.9%2,163 | 80.9%77.2% kept (1,669) | 90.8%58.3% kept (1,262) | 96.7%40.5% kept (876) | none kept |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 68.6%2,163 | 76.4%82.0% kept (1,774) | 84.1%66.4% kept (1,436) | 93.7%46.3% kept (1,002) | 99.7%16.7% kept (361) |
| Mercury Decide Closed · Efficient | — | Decision | 53.5%2,163 | 65.1%71.2% kept (1,539) | 75.3%49.8% kept (1,077) | 85.3%32.0% kept (692) | 96.8%13.0% kept (281) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 315 / 00.867 | 23 / 00.839 | 8 / 00.850 | 1 / 00.471 | 14 / 00.799 | 9 / 00.861 | 15 / 00.817 | 28 / 00.903 | 31 / 00.840 | 47 / 00.910 | 30 / 00.919 | 36 / 00.866 | 39 / 00.853 | 34 / 00.893 |
| Jev 1.13 Closed · Efficient | — | Decision | 248 / 00.834 | 17 / 00.836 | 8 / 00.756 | 0 / 00.671 | 9 / 00.761 | 8 / 00.797 | 9 / 00.835 | 23 / 00.894 | 20 / 00.801 | 30 / 00.868 | 17 / 00.831 | 24 / 00.821 | 41 / 00.850 | 42 / 00.818 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 214 / 00.859 | 15 / 00.862 | 6 / 00.794 | 2 / 00.729 | 0 / 00.789 | 5 / 00.854 | 10 / 00.851 | 18 / 00.882 | 13 / 00.797 | 23 / 00.833 | 15 / 00.799 | 31 / 00.859 | 38 / 00.907 | 38 / 00.858 |
| d1-ultra Closed · Efficient | — | Decision | 213 / 00.876 | 23 / 00.865 | 8 / 00.755 | 0 / 00.650 | 5 / 00.813 | 9 / 00.896 | 13 / 00.887 | 12 / 00.910 | 13 / 00.941 | 18 / 00.902 | 13 / 00.893 | 20 / 00.845 | 30 / 00.885 | 49 / 00.888 |
| Jev 1.14 Closed · Efficient | — | Decision | 202 / 00.829 | 16 / 00.854 | 7 / 00.791 | 0 / 00.686 | 8 / 00.700 | 5 / 00.778 | 8 / 00.810 | 18 / 00.910 | 14 / 00.788 | 25 / 00.862 | 18 / 00.826 | 21 / 00.781 | 27 / 00.849 | 35 / 00.834 |
| d1.1 Closed · Efficient | — | Decision | 183 / 00.829 | 13 / 00.811 | 5 / 00.768 | 0 / 00.714 | 4 / 00.721 | 3 / 00.825 | 7 / 00.796 | 16 / 00.848 | 14 / 00.854 | 19 / 00.870 | 15 / 00.849 | 23 / 00.815 | 30 / 00.798 | 34 / 00.835 |
| GLiDE API Closed · Efficient | — | Decision | 161 / 00.865 | 17 / 00.874 | 5 / 00.746 | 1 / 00.625 | 11 / 00.778 | 4 / 00.853 | 4 / 00.840 | 20 / 00.987 | 13 / 00.872 | 29 / 00.885 | 11 / 00.924 | 15 / 00.896 | 17 / 00.811 | 14 / 00.881 |
| d1 Closed · Efficient | — | Decision | 152 / 00.767 | 14 / 00.765 | 5 / 00.653 | 0 / 00.530 | 4 / 00.700 | 3 / 00.789 | 6 / 00.752 | 9 / 00.761 | 11 / 00.726 | 21 / 00.802 | 11 / 00.738 | 20 / 00.769 | 23 / 00.777 | 25 / 00.784 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 27 / 00.787 | 5 / 00.789 | 0 / 00.696 | 0 / 00.615 | 1 / 00.670 | 0 / 00.747 | 0 / 00.750 | 4 / 00.848 | 4 / 00.806 | 5 / 00.837 | 1 / 00.801 | 2 / 00.768 | 3 / 00.824 | 2 / 00.794 |
| celeris-1-decision Closed · Efficient | — | Decision | No answers at this threshold0.858 | 0 / 00.857 | 0 / 00.787 | 0 / 00.807 | 0 / 00.794 | 0 / 00.884 | 0 / 00.896 | 0 / 00.952 | 0 / 00.840 | 0 / 00.881 | 0 / 00.836 | 0 / 00.838 | 0 / 00.869 | 0 / 00.849 |
| celeris-1.1-decision Closed · Efficient | — | Decision | No answers at this threshold0.859 | 0 / 00.863 | 0 / 00.790 | 0 / 00.875 | 0 / 00.790 | 0 / 00.863 | 0 / 00.896 | 0 / 00.944 | 0 / 00.853 | 0 / 00.877 | 0 / 00.812 | 0 / 00.852 | 0 / 00.867 | 0 / 00.855 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 360 / 10.839 | 28 / 00.819 | 12 / 00.784 | 1 / 00.722 | 15 / 00.780 | 10 / 00.808 | 19 / 00.754 | 47 / 00.929 | 35 / 00.829 | 31 / 00.832 | 28 / 00.931 | 37 / 10.802 | 45 / 00.871 | 52 / 00.848 |
| Mercury Decide Closed · Efficient | — | Decision | 272 / 90.789 | 25 / 00.811 | 9 / 00.692 | 1 / 00.729 | 8 / 00.748 | 6 / 00.687 | 11 / 30.718 | 26 / 10.859 | 23 / 20.801 | 31 / 10.808 | 18 / 00.821 | 22 / 10.751 | 44 / 10.785 | 48 / 00.822 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 2 setups: Muse Ember (max · chat), GPT-6 Luna (chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Logic 95.5% | — | Chat | 99.6%99.2%–99.8% | a | 2,154 / 2,163 | Not measured | Not measured | — | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Logic 58.0% | — | Chat | 93.7%92.6%–94.6% | b | 2,026 / 2,163 | Not measured | Not measured | — | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 213 answers, 213 right · 0 wrong · AUROC 0.876 Logic 54.9% | — | Decision | 74.1%72.2%–75.9% | c | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 80.88chosen-answer Brier 0.1235 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 214 answers, 214 right · 0 wrong · AUROC 0.859 Logic 50.0% | — | Decision | 71.3%69.4%–73.2% | cd | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 78.84chosen-answer Brier 0.1366 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 161 answers, 161 right · 0 wrong · AUROC 0.865 Logic 54.5% | — | Decision | 70.8%68.8%–72.7% | d | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 78.56chosen-answer Brier 0.1367 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 315 answers, 315 right · 0 wrong · AUROC 0.867 Logic 54.8% | — | Decision | 70.1%68.1%–72.0% | d | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 78.01chosen-answer Brier 0.1407 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.858 Logic 47.3% | — | Decision | 70.0%68.0%–71.9% | d | 1,514 / 2,163 | No answers at this threshold | 0.858 | 77.96chosen-answer Brier 0.1407 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.859 Logic 47.3% | — | Decision | 69.9%67.9%–71.8% | d | 1,511 / 2,163 | No answers at this threshold | 0.859 | 77.91chosen-answer Brier 0.1403 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 361 answers, 360 right · 1 wrong · AUROC 0.839 Logic 52.4% | — | Decision | 68.6%66.6%–70.5% | d | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 76.15chosen-answer Brier 0.1626 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 183 answers, 183 right · 0 wrong · AUROC 0.829 Logic 46.0% | — | Decision | 65.2%63.2%–67.2% | e | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 74.62chosen-answer Brier 0.1595 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 202 answers, 202 right · 0 wrong · AUROC 0.829 Logic 48.2% | — | Decision | 65.1%63.1%–67.1% | e | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 74.56chosen-answer Brier 0.1602 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 248 answers, 248 right · 0 wrong · AUROC 0.834 Logic 46.3% | — | Decision | 64.1%62.0%–66.1% | e | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 74.08chosen-answer Brier 0.1592 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 27 answers, 27 right · 0 wrong · AUROC 0.787 Logic 44.4% | — | Decision | 59.4%57.3%–61.4% | f | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 70.49chosen-answer Brier 0.1839 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 152 answers, 152 right · 0 wrong · AUROC 0.767 Logic 40.8% | — | Decision | 55.6%53.5%–57.7% | g | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 68.16chosen-answer Brier 0.1930 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 281 answers, 272 right · 9 wrong · AUROC 0.789 Logic 45.1% | — | Decision | 53.5%51.4%–55.6% | g | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 66.02chosen-answer Brier 0.2146 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.5% | — | Chat | 99.7%99.3%–99.8% | a | 2,156 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 96.2% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.2% | — | Chat | 99.7%99.4%–99.9% | a | 2,157 / 2,163 | Not measured | Not measured | — | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Liquid AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | Decision | 65.5%2,163 | 83.5%60.9% kept (1,318) | 94.4%41.3% kept (894) | 99.2%23.7% kept (512) | 100.0%6.5% kept (141) |
| d1.2-ultra Closed · Undisclosed | — | Decision | 69.9%2,163 | 87.4%66.6% kept (1,441) | 94.8%48.6% kept (1,052) | 99.7%30.5% kept (660) | 100.0%5.3% kept (115) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | Decision | 141 / 00.838 | 8 / 00.833 | 5 / 00.800 | 0 / 00.557 | 3 / 00.708 | 3 / 00.840 | 9 / 00.794 | 20 / 00.896 | 8 / 00.862 | 12 / 00.843 | 10 / 00.797 | 24 / 00.843 | 20 / 00.838 | 19 / 00.845 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 115 / 00.876 | 6 / 00.891 | 5 / 00.839 | 1 / 00.889 | 4 / 00.774 | 4 / 00.851 | 5 / 00.801 | 18 / 00.960 | 10 / 00.855 | 8 / 00.912 | 10 / 00.902 | 15 / 00.859 | 11 / 00.855 | 18 / 00.868 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 7 setups: Qwen4-Max (high · chat), GPT-6 Astra (chat), GPT-6.1 Sol (chat), Gemini 4 Carbon (chat), Grok 4.7 (chat), Qwen3.8 Flash (off · chat), GPT-6.1 Terra (chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Logic 96.3% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Logic 96.7% | — | Chat | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Logic 96.5% | — | Chat | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Logic 96.7% | — | Chat | 99.6%99.2%–99.8% | ab | 2,154 / 2,163 | Not measured | Not measured | — | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Logic running | — | Chat | 99.5%99.2%–99.7% | ab | 2,153 / 2,163 | Not measured | Not measured | — | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Logic 94.0% | — | Chat | 99.5%99.1%–99.7% | ab | 2,152 / 2,163 | Not measured | Not measured | — | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | b | 2,146 / 2,163 | Not measured | Not measured | — | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 115 answers, 115 right · 0 wrong · AUROC 0.876 Logic 51.6% | — | Decision | 69.9%67.9%–71.8% | c | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 78.23chosen-answer Brier 0.1340 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 141 answers, 141 right · 0 wrong · AUROC 0.838 Logic 49.2% | — | Decision | 65.5%63.5%–67.5% | d | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 74.98chosen-answer Brier 0.1556 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
MachinePulse Base Labs Perplexity Poindexter Labs Nace.AI TypeSafe Sthānika AI Google Liquid AI REAX Fastino Valen-Team PelaAI StartLux tattrongvu Turion.AI scar-ai Microsoft OmniJev Logits Cloudflare vLLM Semantic Router Blockbrain Labs Other Metask Lab Zefan Cai Strands Agents Celeris Subquadratic Inception Apertix AI Patrick-18 newsbang quincyqiang victorbennett EldanRing Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | 81.6%2,163 | 88.9%86.7% kept (1,876) | 94.2%73.7% kept (1,594) | 98.0%58.1% kept (1,256) | 100.0%38.7% kept (837) |
| Mammoth-1 Open · 27B | — | 74.0%2,163 | 85.3%79.1% kept (1,712) | 94.3%61.8% kept (1,336) | 98.5%45.7% kept (988) | 100.0%19.0% kept (412) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 69.8%2,163 | 83.0%73.5% kept (1,590) | 93.1%56.3% kept (1,217) | 98.2%38.7% kept (838) | 100.0%15.3% kept (331) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 69.3%2,163 | 82.5%73.5% kept (1,590) | 92.8%56.2% kept (1,215) | 97.7%38.8% kept (840) | 100.0%15.1% kept (326) |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 72.5%2,163 | 81.2%81.1% kept (1,754) | 90.0%64.0% kept (1,384) | 96.3%46.1% kept (998) | 100.0%14.9% kept (323) |
| Copernicus-31B Open · 31B | — | 77.3%2,163 | 91.3%73.4% kept (1,587) | 97.2%57.5% kept (1,244) | 99.4%39.0% kept (843) | 100.0%14.7% kept (318) |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 70.1%2,163 | 79.7%79.8% kept (1,725) | 89.3%62.5% kept (1,352) | 97.3%43.4% kept (938) | 100.0%14.6% kept (315) |
| Jev 1.13 Closed · Efficient | — | 64.1%2,163 | 80.3%65.2% kept (1,410) | 91.3%42.7% kept (924) | 98.3%25.1% kept (543) | 100.0%11.5% kept (248) |
| Sieve 31B v3 Open · 31B | — | 77.1%2,163 | 90.0%75.3% kept (1,628) | 95.9%59.6% kept (1,289) | 99.3%41.1% kept (888) | 100.0%9.9% kept (215) |
| Gemini Decision 1 20261111 Closed · Efficient | — | 71.3%2,163 | 85.3%72.1% kept (1,560) | 92.7%55.2% kept (1,194) | 98.2%36.3% kept (785) | 100.0%9.9% kept (214) |
| d1-ultra Closed · Efficient | — | 74.1%2,163 | 87.8%73.4% kept (1,588) | 94.0%59.0% kept (1,276) | 98.5%40.8% kept (882) | 100.0%9.8% kept (213) |
| Jev 1.14 Closed · Efficient | — | 65.1%2,163 | 82.2%61.7% kept (1,335) | 92.8%40.5% kept (876) | 98.3%24.0% kept (519) | 100.0%9.3% kept (202) |
| d1.1 Closed · Efficient | — | 65.2%2,163 | 82.7%61.1% kept (1,322) | 92.6%41.5% kept (897) | 97.6%24.9% kept (538) | 100.0%8.5% kept (183) |
| REAX 1.3 Open · 57B | — | 78.0%2,163 | 90.9%76.9% kept (1,664) | 97.5%58.5% kept (1,265) | 99.9%40.5% kept (877) | 100.0%8.2% kept (178) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 59.0%2,163 | 80.4%53.0% kept (1,147) | 92.2%35.0% kept (758) | 97.8%20.6% kept (445) | 100.0%8.0% kept (174) |
| GLiDE API Closed · Efficient | — | 70.8%2,163 | 81.6%77.6% kept (1,678) | 92.0%58.2% kept (1,258) | 98.6%37.4% kept (808) | 100.0%7.4% kept (161) |
| d1 Closed · Efficient | — | 55.6%2,163 | 73.6%55.1% kept (1,192) | 85.9%32.1% kept (694) | 95.4%17.0% kept (368) | 100.0%7.0% kept (152) |
| d1.2 Closed · Undisclosed | — | 65.5%2,163 | 83.5%60.9% kept (1,318) | 94.4%41.3% kept (894) | 99.2%23.7% kept (512) | 100.0%6.5% kept (141) |
| GLiDE v1.2 Open · 27B | — | 69.9%2,163 | 81.1%75.9% kept (1,641) | 92.6%56.4% kept (1,219) | 99.1%34.4% kept (745) | 100.0%6.1% kept (131) |
| Valen-Decision-1.5-9B Open · 9.0B | — | 55.3%2,163 | 74.3%54.9% kept (1,187) | 86.7%32.3% kept (698) | 94.9%17.3% kept (374) | 100.0%6.0% kept (130) |
| KnowLine-9B-Gen1 Open · 9.0B | — | 59.8%2,163 | 76.7%58.4% kept (1,264) | 89.3%36.8% kept (797) | 98.0%20.5% kept (444) | 100.0%5.7% kept (123) |
| d1.2-ultra Closed · Undisclosed | — | 69.9%2,163 | 87.4%66.6% kept (1,441) | 94.8%48.6% kept (1,052) | 99.7%30.5% kept (660) | 100.0%5.3% kept (115) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 55.8%2,163 | 70.8%62.3% kept (1,348) | 81.1%40.5% kept (876) | 94.0%20.8% kept (450) | 100.0%4.4% kept (96) |
| GLiDE 9B Open · 7.9B | — | 57.9%2,163 | 70.8%68.3% kept (1,478) | 84.7%42.8% kept (926) | 94.7%20.9% kept (451) | 100.0%3.4% kept (73) |
| Jeffan-9B-1211 Open · 9.0B | — | 59.8%2,163 | 77.8%60.3% kept (1,304) | 90.4%38.1% kept (824) | 98.5%18.2% kept (393) | 100.0%3.0% kept (64) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 56.0%2,163 | 67.1%67.4% kept (1,457) | 79.3%41.9% kept (907) | 92.8%21.3% kept (460) | 100.0%2.5% kept (54) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 58.4%2,163 | 74.9%58.2% kept (1,258) | 85.1%37.4% kept (808) | 96.6%19.1% kept (413) | 100.0%2.3% kept (49) |
| Turion Jev 31B Open · 31B | — | 69.0%2,163 | 84.7%68.7% kept (1,487) | 92.8%51.1% kept (1,106) | 99.8%26.0% kept (562) | 100.0%2.0% kept (43) |
| vera-medium-omni Open · 4.2B | — | 56.8%2,163 | 75.4%55.2% kept (1,193) | 88.5%29.6% kept (641) | 98.9%12.3% kept (267) | 100.0%1.5% kept (33) |
| Microsoft-Decision-1 Closed · Efficient | — | 59.4%2,163 | 73.4%63.9% kept (1,382) | 88.5%36.2% kept (782) | 98.0%13.8% kept (299) | 100.0%1.2% kept (27) |
| OneJev-27B Open · 27B | — | 53.8%2,163 | 81.8%39.5% kept (855) | 92.5%23.3% kept (504) | 98.3%10.6% kept (229) | 100.0%1.2% kept (26) |
| Logits Sigma-1 Duo Open · 57B | — | 75.2%2,163 | 89.2%73.5% kept (1,589) | 98.3%52.4% kept (1,134) | 100.0%26.7% kept (578) | 100.0%0.7% kept (15) |
| Clef Open · 27B | — | 51.1%2,163 | 65.2%62.2% kept (1,346) | 78.9%36.8% kept (796) | 90.8%17.6% kept (381) | 100.0%0.6% kept (12) |
| d3-ultra Open · 26B | — | 71.5%2,163 | 78.5%85.2% kept (1,842) | 86.2%68.6% kept (1,484) | 97.4%43.7% kept (946) | 100.0%0.5% kept (11) |
| KnowLine-2B-Gen1 Open · 2.0B | — | 35.3%2,163 | 52.1%28.2% kept (609) | 67.2%8.9% kept (192) | 88.5%2.4% kept (52) | 100.0%0.4% kept (8) |
| KnowLine-12B-Gen1 Open · 12B | — | 66.3%2,163 | 83.7%60.8% kept (1,315) | 92.1%42.2% kept (912) | 99.1%19.6% kept (425) | 100.0%0.3% kept (7) |
| GLiDE 0.8B Open · 0.8B | — | 32.1%2,163 | 38.1%44.3% kept (959) | 47.5%16.8% kept (364) | 73.0%2.9% kept (63) | 100.0%0.2% kept (4) |
| d1-2B Open · 1.9B | — | 37.1%2,163 | 58.0%25.0% kept (541) | 74.5%7.4% kept (161) | 83.7%2.3% kept (49) | 100.0%0.2% kept (4) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 38.7%2,163 | 68.1%21.1% kept (457) | 83.6%7.9% kept (171) | 92.0%2.3% kept (50) | 100.0%0.1% kept (2) |
| Clef-Flash Open · 9.0B | — | 49.8%2,163 | 68.7%44.7% kept (967) | 81.3%23.3% kept (504) | 94.4%7.4% kept (161) | none kept |
| Cygnet Open · 12B | — | 44.3%2,163 | 65.9%41.9% kept (907) | 84.2%14.9% kept (322) | 98.7%3.5% kept (75) | none kept |
| GLiClass v4 Edge Open · 0.4B | — | 22.7%2,163 | 26.9%3.6% kept (78) | 38.5%0.6% kept (13) | 100.0%0.0% kept (1) | none kept |
| GLiDE 0.4B Open · 0.4B | — | 20.6%2,163 | 23.5%5.5% kept (119) | 33.3%1.0% kept (21) | none kept | none kept |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 28.2%2,163 | 41.7%15.5% kept (336) | 54.9%3.3% kept (71) | 75.0%0.2% kept (4) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | 60.1%2,163 | 80.7%52.3% kept (1,131) | 92.5%30.9% kept (668) | 99.6%10.3% kept (223) | none kept |
| Metask-Jev-4B Open · 4.5B | — | 39.2%2,163 | 63.1%19.8% kept (428) | 77.5%8.6% kept (187) | 88.2%1.6% kept (34) | none kept |
| OneJev1.6-0.8B Open · 0.8B | — | 28.2%2,163 | 41.8%15.3% kept (330) | 54.1%3.4% kept (74) | 66.7%0.7% kept (15) | none kept |
| Open-Jev-27B-v1.1 Open · 27B | — | 54.6%2,163 | 95.8%10.0% kept (216) | 100.0%1.2% kept (25) | none kept | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 29.7%2,163 | 45.9%10.6% kept (229) | 61.0%3.6% kept (77) | 92.3%0.6% kept (13) | none kept |
| Strands Decider 2B Open · 2.0B | — | 28.7%2,163 | 45.9%11.8% kept (255) | 69.2%1.2% kept (26) | 100.0%0.0% kept (1) | none kept |
| celeris-1-decision Closed · Efficient | — | 70.0%2,163 | 80.9%77.1% kept (1,667) | 90.9%58.3% kept (1,262) | 96.5%40.4% kept (874) | none kept |
| celeris-1.1-decision Closed · Efficient | — | 69.9%2,163 | 80.9%77.2% kept (1,669) | 90.8%58.3% kept (1,262) | 96.7%40.5% kept (876) | none kept |
| Drex Max Open · 27B | — | 70.3%2,163 | 76.2%86.7% kept (1,875) | 83.3%72.2% kept (1,561) | 92.9%55.9% kept (1,209) | 98.6%33.7% kept (729) |
| KnowLine-31B-Gen2 Open · 31B | — | 80.3%2,163 | 90.0%80.7% kept (1,746) | 95.3%66.5% kept (1,438) | 99.3%48.1% kept (1,040) | 99.8%21.7% kept (470) |
| KnowLine-31B-Gen1.5 Open · 31B | — | 77.4%2,163 | 89.5%77.0% kept (1,665) | 95.3%61.3% kept (1,327) | 99.1%42.7% kept (923) | 99.8%21.5% kept (465) |
| SubQ Snap 1.1 Closed · Efficient | — | 68.6%2,163 | 76.4%82.0% kept (1,774) | 84.1%66.4% kept (1,436) | 93.7%46.3% kept (1,002) | 99.7%16.7% kept (361) |
| Mercury Decide Closed · Efficient | — | 53.5%2,163 | 65.1%71.2% kept (1,539) | 75.3%49.8% kept (1,077) | 85.3%32.0% kept (692) | 96.8%13.0% kept (281) |
| Cerno 9B v1.2 Open · 9.0B | — | 56.0%2,163 | 64.6%75.7% kept (1,638) | 74.3%55.1% kept (1,192) | 86.2%33.8% kept (731) | 96.1%11.7% kept (254) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 58.8%2,163 | 69.2%74.8% kept (1,619) | 79.2%53.7% kept (1,162) | 91.9%32.1% kept (694) | 98.6%9.7% kept (210) |
| GLiDE 27B Open · 27B | — | 72.0%2,163 | 82.1%78.7% kept (1,703) | 91.2%60.6% kept (1,311) | 98.7%39.5% kept (855) | 99.5%9.2% kept (199) |
| Adjudex 9B v5 Open · 9.0B | — | 62.1%2,163 | 74.7%68.7% kept (1,486) | 86.2%46.0% kept (994) | 96.3%27.2% kept (589) | 99.5%9.2% kept (198) |
| Elephant-1 Open · 9.0B | — | 61.3%2,163 | 75.7%65.6% kept (1,419) | 86.7%44.0% kept (952) | 95.8%26.2% kept (567) | 99.5%8.5% kept (183) |
| d3 Open · 26B | — | 70.4%2,163 | 85.3%71.8% kept (1,553) | 95.2%53.7% kept (1,161) | 99.3%34.0% kept (735) | 99.4%7.5% kept (163) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 55.2%2,163 | 69.1%65.3% kept (1,412) | 79.3%46.3% kept (1,001) | 88.3%25.8% kept (557) | 96.8%7.2% kept (155) |
| Chinese Decision Pointer 4B Open · 4.0B | — | 50.4%2,163 | 60.9%66.6% kept (1,440) | 74.3%39.1% kept (845) | 86.2%20.4% kept (442) | 96.1%7.2% kept (155) |
| Verdin-4B-v1.1 Open · 4.0B | — | 51.5%2,163 | 69.1%51.5% kept (1,115) | 81.6%29.9% kept (646) | 91.6%16.0% kept (346) | 96.9%6.0% kept (130) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 55.9%2,163 | 67.9%69.5% kept (1,504) | 77.7%49.1% kept (1,061) | 89.5%27.7% kept (599) | 99.1%5.3% kept (114) |
| Winnow-12B Open · 12B | — | 46.9%2,163 | 64.6%47.3% kept (1,023) | 77.3%30.4% kept (657) | 89.5%16.2% kept (351) | 95.2%1.0% kept (21) |
| GLiDE 2B Open · 1.9B | — | 36.8%2,163 | 45.5%49.8% kept (1,078) | 56.1%21.8% kept (471) | 73.8%5.6% kept (122) | 93.8%0.7% kept (16) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | 837 / 00.903 | 85 / 00.899 | 34 / 00.801 | 3 / 00.833 | 34 / 00.880 | 38 / 00.933 | 50 / 00.890 | 83 / 00.952 | 45 / 00.879 | 66 / 00.915 | 53 / 00.950 | 91 / 00.902 | 114 / 00.893 | 141 / 00.904 |
| Mammoth-1 Open · 27B | — | 412 / 00.895 | 27 / 00.885 | 9 / 00.766 | 1 / 00.788 | 11 / 00.821 | 14 / 00.876 | 30 / 00.897 | 41 / 00.909 | 17 / 00.893 | 47 / 00.930 | 31 / 00.904 | 53 / 00.900 | 59 / 00.896 | 72 / 00.903 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 331 / 00.869 | 22 / 00.861 | 6 / 00.818 | 1 / 00.806 | 8 / 00.772 | 10 / 00.823 | 16 / 00.819 | 34 / 00.961 | 34 / 00.901 | 43 / 00.903 | 33 / 00.932 | 32 / 00.840 | 47 / 00.875 | 45 / 00.863 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 326 / 00.873 | 23 / 00.850 | 9 / 00.842 | 1 / 00.571 | 6 / 00.827 | 14 / 00.799 | 18 / 00.854 | 28 / 00.927 | 36 / 00.869 | 43 / 00.887 | 31 / 00.946 | 29 / 00.851 | 41 / 00.885 | 47 / 00.884 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 323 / 00.856 | 26 / 00.828 | 8 / 00.776 | 0 / 00.577 | 8 / 00.785 | 15 / 00.845 | 17 / 00.790 | 36 / 00.935 | 27 / 00.839 | 25 / 00.900 | 22 / 00.933 | 35 / 00.845 | 49 / 00.871 | 55 / 00.834 |
| Copernicus-31B Open · 31B | — | 318 / 00.899 | 27 / 00.901 | 13 / 00.865 | 1 / 00.800 | 9 / 00.881 | 17 / 00.892 | 20 / 00.881 | 42 / 00.980 | 12 / 00.884 | 25 / 00.876 | 19 / 00.937 | 29 / 00.882 | 48 / 00.914 | 56 / 00.875 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 315 / 00.867 | 23 / 00.839 | 8 / 00.850 | 1 / 00.471 | 14 / 00.799 | 9 / 00.861 | 15 / 00.817 | 28 / 00.903 | 31 / 00.840 | 47 / 00.910 | 30 / 00.919 | 36 / 00.866 | 39 / 00.853 | 34 / 00.893 |
| Jev 1.13 Closed · Efficient | — | 248 / 00.834 | 17 / 00.836 | 8 / 00.756 | 0 / 00.671 | 9 / 00.761 | 8 / 00.797 | 9 / 00.835 | 23 / 00.894 | 20 / 00.801 | 30 / 00.868 | 17 / 00.831 | 24 / 00.821 | 41 / 00.850 | 42 / 00.818 |
| Sieve 31B v3 Open · 31B | — | 215 / 00.891 | 24 / 00.871 | 6 / 00.848 | 0 / 00.903 | 7 / 00.856 | 8 / 00.898 | 10 / 00.863 | 26 / 00.956 | 15 / 00.918 | 22 / 00.907 | 14 / 00.939 | 24 / 00.895 | 31 / 00.879 | 28 / 00.848 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 214 / 00.859 | 15 / 00.862 | 6 / 00.794 | 2 / 00.729 | 0 / 00.789 | 5 / 00.854 | 10 / 00.851 | 18 / 00.882 | 13 / 00.797 | 23 / 00.833 | 15 / 00.799 | 31 / 00.859 | 38 / 00.907 | 38 / 00.858 |
| d1-ultra Closed · Efficient | — | 213 / 00.876 | 23 / 00.865 | 8 / 00.755 | 0 / 00.650 | 5 / 00.813 | 9 / 00.896 | 13 / 00.887 | 12 / 00.910 | 13 / 00.941 | 18 / 00.902 | 13 / 00.893 | 20 / 00.845 | 30 / 00.885 | 49 / 00.888 |
| Jev 1.14 Closed · Efficient | — | 202 / 00.829 | 16 / 00.854 | 7 / 00.791 | 0 / 00.686 | 8 / 00.700 | 5 / 00.778 | 8 / 00.810 | 18 / 00.910 | 14 / 00.788 | 25 / 00.862 | 18 / 00.826 | 21 / 00.781 | 27 / 00.849 | 35 / 00.834 |
| d1.1 Closed · Efficient | — | 183 / 00.829 | 13 / 00.811 | 5 / 00.768 | 0 / 00.714 | 4 / 00.721 | 3 / 00.825 | 7 / 00.796 | 16 / 00.848 | 14 / 00.854 | 19 / 00.870 | 15 / 00.849 | 23 / 00.815 | 30 / 00.798 | 34 / 00.835 |
| REAX 1.3 Open · 57B | — | 178 / 00.903 | 11 / 00.888 | 7 / 00.826 | 1 / 00.818 | 7 / 00.869 | 9 / 00.886 | 11 / 00.917 | 30 / 00.955 | 10 / 00.946 | 14 / 00.881 | 7 / 00.935 | 22 / 00.887 | 25 / 00.913 | 24 / 00.900 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 174 / 00.813 | 8 / 00.819 | 6 / 00.817 | 0 / 00.533 | 0 / 00.740 | 5 / 00.798 | 13 / 00.841 | 20 / 00.905 | 9 / 00.768 | 16 / 00.837 | 14 / 00.766 | 22 / 00.762 | 35 / 00.836 | 26 / 00.803 |
| GLiDE API Closed · Efficient | — | 161 / 00.865 | 17 / 00.874 | 5 / 00.746 | 1 / 00.625 | 11 / 00.778 | 4 / 00.853 | 4 / 00.840 | 20 / 00.987 | 13 / 00.872 | 29 / 00.885 | 11 / 00.924 | 15 / 00.896 | 17 / 00.811 | 14 / 00.881 |
| d1 Closed · Efficient | — | 152 / 00.767 | 14 / 00.765 | 5 / 00.653 | 0 / 00.530 | 4 / 00.700 | 3 / 00.789 | 6 / 00.752 | 9 / 00.761 | 11 / 00.726 | 21 / 00.802 | 11 / 00.738 | 20 / 00.769 | 23 / 00.777 | 25 / 00.784 |
| d1.2 Closed · Undisclosed | — | 141 / 00.838 | 8 / 00.833 | 5 / 00.800 | 0 / 00.557 | 3 / 00.708 | 3 / 00.840 | 9 / 00.794 | 20 / 00.896 | 8 / 00.862 | 12 / 00.843 | 10 / 00.797 | 24 / 00.843 | 20 / 00.838 | 19 / 00.845 |
| GLiDE v1.2 Open · 27B | — | 131 / 00.862 | 12 / 00.857 | 3 / 00.740 | 0 / 00.528 | 5 / 00.777 | 4 / 00.834 | 5 / 00.838 | 20 / 00.963 | 14 / 00.865 | 22 / 00.912 | 7 / 00.948 | 15 / 00.873 | 14 / 00.849 | 10 / 00.877 |
| Valen-Decision-1.5-9B Open · 9.0B | — | 130 / 00.790 | 16 / 00.837 | 3 / 00.663 | 0 / 00.667 | 7 / 00.778 | 4 / 00.849 | 5 / 00.753 | 17 / 00.876 | 9 / 00.765 | 12 / 00.777 | 7 / 00.834 | 13 / 00.782 | 16 / 00.795 | 21 / 00.776 |
| KnowLine-9B-Gen1 Open · 9.0B | — | 123 / 00.799 | 13 / 00.816 | 5 / 00.712 | 0 / 00.595 | 6 / 00.674 | 6 / 00.803 | 5 / 00.760 | 19 / 00.873 | 6 / 00.799 | 14 / 00.792 | 7 / 00.780 | 17 / 00.803 | 11 / 00.821 | 14 / 00.817 |
| d1.2-ultra Closed · Undisclosed | — | 115 / 00.876 | 6 / 00.891 | 5 / 00.839 | 1 / 00.889 | 4 / 00.774 | 4 / 00.851 | 5 / 00.801 | 18 / 00.960 | 10 / 00.855 | 8 / 00.912 | 10 / 00.902 | 15 / 00.859 | 11 / 00.855 | 18 / 00.868 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 96 / 00.779 | 11 / 00.818 | 1 / 00.707 | 0 / 00.524 | 2 / 00.676 | 2 / 00.743 | 7 / 00.748 | 18 / 00.814 | 6 / 00.808 | 11 / 00.806 | 6 / 00.744 | 12 / 00.768 | 12 / 00.778 | 8 / 00.775 |
| GLiDE 9B Open · 7.9B | — | 73 / 00.798 | 7 / 00.800 | 2 / 00.737 | 0 / 00.731 | 3 / 00.700 | 1 / 00.716 | 4 / 00.752 | 15 / 00.805 | 3 / 00.774 | 10 / 00.873 | 4 / 00.853 | 7 / 00.776 | 8 / 00.851 | 9 / 00.765 |
| Jeffan-9B-1211 Open · 9.0B | — | 64 / 00.814 | 4 / 00.865 | 4 / 00.677 | 0 / 00.769 | 3 / 00.732 | 3 / 00.844 | 3 / 00.745 | 15 / 00.811 | 4 / 00.797 | 6 / 00.888 | 2 / 00.825 | 5 / 00.804 | 7 / 00.808 | 8 / 00.768 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 54 / 00.765 | 3 / 00.791 | 2 / 00.611 | 0 / 00.500 | 1 / 00.688 | 1 / 00.814 | 4 / 00.699 | 11 / 00.845 | 1 / 00.842 | 7 / 00.813 | 1 / 00.742 | 7 / 00.743 | 9 / 00.803 | 7 / 00.747 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 49 / 00.781 | 3 / 00.777 | 1 / 00.678 | 0 / 00.394 | 1 / 00.703 | 1 / 00.730 | 3 / 00.735 | 11 / 00.887 | 3 / 00.774 | 6 / 00.803 | 5 / 00.830 | 4 / 00.775 | 2 / 00.788 | 9 / 00.779 |
| Turion Jev 31B Open · 31B | — | 43 / 00.864 | 5 / 00.868 | 1 / 00.770 | 0 / 00.757 | 1 / 00.809 | 0 / 00.904 | 3 / 00.817 | 3 / 00.926 | 1 / 00.854 | 6 / 00.823 | 1 / 00.940 | 8 / 00.853 | 5 / 00.866 | 9 / 00.852 |
| vera-medium-omni Open · 4.2B | — | 33 / 00.789 | 2 / 00.782 | 3 / 00.704 | 0 / 00.636 | 1 / 00.695 | 1 / 00.836 | 1 / 00.706 | 13 / 00.860 | 0 / 00.848 | 2 / 00.837 | 1 / 00.797 | 4 / 00.770 | 1 / 00.809 | 4 / 00.784 |
| Microsoft-Decision-1 Closed · Efficient | — | 27 / 00.787 | 5 / 00.789 | 0 / 00.696 | 0 / 00.615 | 1 / 00.670 | 0 / 00.747 | 0 / 00.750 | 4 / 00.848 | 4 / 00.806 | 5 / 00.837 | 1 / 00.801 | 2 / 00.768 | 3 / 00.824 | 2 / 00.794 |
| OneJev-27B Open · 27B | — | 26 / 00.783 | 2 / 00.768 | 0 / 00.645 | 0 / 00.806 | 1 / 00.750 | 0 / 00.767 | 1 / 00.778 | 2 / 00.884 | 1 / 00.758 | 1 / 00.811 | 3 / 00.787 | 4 / 00.757 | 6 / 00.785 | 5 / 00.789 |
| Logits Sigma-1 Duo Open · 57B | — | 15 / 00.899 | 2 / 00.900 | 0 / 00.841 | 0 / 00.667 | 0 / 00.847 | 1 / 00.888 | 0 / 00.872 | 2 / 00.950 | 0 / 00.919 | 1 / 00.910 | 1 / 00.939 | 4 / 00.895 | 1 / 00.871 | 3 / 00.911 |
| Clef Open · 27B | — | 12 / 00.767 | 2 / 00.751 | 1 / 00.708 | 0 / 00.875 | 2 / 00.723 | 0 / 00.720 | 0 / 00.798 | 3 / 00.778 | 0 / 00.793 | 1 / 00.744 | 0 / 00.878 | 2 / 00.748 | 0 / 00.802 | 1 / 00.774 |
| d3-ultra Open · 26B | — | 11 / 00.857 | 1 / 00.842 | 1 / 00.778 | 0 / 00.764 | 0 / 00.820 | 0 / 00.731 | 0 / 00.827 | 3 / 00.942 | 0 / 00.819 | 1 / 00.901 | 2 / 00.848 | 1 / 00.871 | 1 / 00.850 | 1 / 00.874 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 8 / 00.636 | 0 / 00.570 | 0 / 00.578 | 0 / 00.533 | 0 / 00.508 | 0 / 00.540 | 0 / 00.655 | 0 / 00.667 | 1 / 00.779 | 4 / 00.736 | 1 / 00.699 | 1 / 00.644 | 0 / 00.656 | 1 / 00.587 |
| KnowLine-12B-Gen1 Open · 12B | — | 7 / 00.819 | 2 / 00.838 | 0 / 00.736 | 0 / 00.814 | 0 / 00.752 | 0 / 00.840 | 0 / 00.833 | 5 / 00.915 | 0 / 00.800 | 0 / 00.824 | 0 / 00.916 | 0 / 00.775 | 0 / 00.857 | 0 / 00.794 |
| GLiDE 0.8B Open · 0.8B | — | 4 / 00.589 | 1 / 00.538 | 0 / 00.498 | 0 / 00.467 | 0 / 00.610 | 0 / 00.579 | 0 / 00.525 | 0 / 00.710 | 1 / 00.604 | 0 / 00.601 | 0 / 00.661 | 1 / 00.673 | 0 / 00.564 | 1 / 00.616 |
| d1-2B Open · 1.9B | — | 4 / 00.656 | 0 / 00.634 | 0 / 00.629 | 0 / 00.318 | 0 / 00.572 | 1 / 00.582 | 0 / 00.589 | 0 / 00.721 | 1 / 00.778 | 1 / 00.625 | 0 / 00.718 | 1 / 00.709 | 0 / 00.657 | 0 / 00.677 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 2 / 00.685 | 0 / 00.661 | 0 / 00.601 | 0 / 00.333 | 0 / 00.560 | 0 / 00.665 | 0 / 00.655 | 0 / 00.788 | 0 / 00.813 | 1 / 00.755 | 0 / 00.703 | 0 / 00.614 | 0 / 00.658 | 1 / 00.679 |
| Clef-Flash Open · 9.0B | — | No answers at this threshold0.730 | 0 / 00.711 | 0 / 00.642 | 0 / 00.361 | 0 / 00.614 | 0 / 00.736 | 0 / 00.670 | 0 / 00.766 | 0 / 00.789 | 0 / 00.744 | 0 / 00.823 | 0 / 00.722 | 0 / 00.804 | 0 / 00.737 |
| Cygnet Open · 12B | — | No answers at this threshold0.735 | 0 / 00.730 | 0 / 00.617 | 0 / 00.600 | 0 / 00.690 | 0 / 00.639 | 0 / 00.726 | 0 / 00.800 | 0 / 00.788 | 0 / 00.796 | 0 / 00.697 | 0 / 00.680 | 0 / 00.768 | 0 / 00.761 |
| GLiClass v4 Edge Open · 0.4B | — | No answers at this threshold0.501 | 0 / 00.524 | 0 / 00.554 | 0 / 00.550 | 0 / 00.476 | 0 / 00.474 | 0 / 00.505 | 0 / 00.412 | 0 / 00.522 | 0 / 00.412 | 0 / 00.460 | 0 / 00.566 | 0 / 00.543 | 0 / 00.518 |
| GLiDE 0.4B Open · 0.4B | — | No answers at this threshold0.500 | 0 / 00.496 | 0 / 00.528 | 0 / 00.875 | 0 / 00.509 | 0 / 00.606 | 0 / 00.458 | 0 / 00.370 | 0 / 00.632 | 0 / 00.450 | 0 / 00.478 | 0 / 00.493 | 0 / 00.498 | 0 / 00.515 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | No answers at this threshold0.596 | 0 / 00.486 | 0 / 00.603 | 0 / 00.667 | 0 / 00.628 | 0 / 00.573 | 0 / 00.490 | 0 / 00.683 | 0 / 00.792 | 0 / 00.660 | 0 / 00.653 | 0 / 00.603 | 0 / 00.557 | 0 / 00.565 |
| KnowLine-9B-Gen2 Open · 9.0B | — | No answers at this threshold0.803 | 0 / 00.788 | 0 / 00.711 | 0 / 00.712 | 0 / 00.671 | 0 / 00.801 | 0 / 00.819 | 0 / 00.834 | 0 / 00.841 | 0 / 00.847 | 0 / 00.824 | 0 / 00.774 | 0 / 00.806 | 0 / 00.817 |
| Metask-Jev-4B Open · 4.5B | — | No answers at this threshold0.651 | 0 / 00.614 | 0 / 00.544 | 0 / 00.933 | 0 / 00.617 | 0 / 00.737 | 0 / 00.613 | 0 / 00.633 | 0 / 00.619 | 0 / 00.666 | 0 / 00.727 | 0 / 00.660 | 0 / 00.624 | 0 / 00.664 |
| OneJev1.6-0.8B Open · 0.8B | — | No answers at this threshold0.592 | 0 / 00.548 | 0 / 00.601 | 0 / 00.533 | 0 / 00.519 | 0 / 00.678 | 0 / 00.474 | 0 / 00.543 | 0 / 00.726 | 0 / 00.690 | 0 / 00.619 | 0 / 00.605 | 0 / 00.609 | 0 / 00.559 |
| Open-Jev-27B-v1.1 Open · 27B | — | No answers at this threshold0.764 | 0 / 00.816 | 0 / 00.607 | 0 / 00.625 | 0 / 00.674 | 0 / 00.731 | 0 / 00.751 | 0 / 00.762 | 0 / 00.745 | 0 / 00.847 | 0 / 00.818 | 0 / 00.776 | 0 / 00.727 | 0 / 00.749 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | No answers at this threshold0.583 | 0 / 00.611 | 0 / 00.460 | 0 / 00.667 | 0 / 00.546 | 0 / 00.440 | 0 / 00.612 | 0 / 00.567 | 0 / 00.693 | 0 / 00.672 | 0 / 00.676 | 0 / 00.551 | 0 / 00.503 | 0 / 00.580 |
| Strands Decider 2B Open · 2.0B | — | No answers at this threshold0.569 | 0 / 00.510 | 0 / 00.458 | 0 / 00.615 | 0 / 00.534 | 0 / 00.589 | 0 / 00.583 | 0 / 00.635 | 0 / 00.615 | 0 / 00.574 | 0 / 00.632 | 0 / 00.556 | 0 / 00.542 | 0 / 00.627 |
| celeris-1-decision Closed · Efficient | — | No answers at this threshold0.858 | 0 / 00.857 | 0 / 00.787 | 0 / 00.807 | 0 / 00.794 | 0 / 00.884 | 0 / 00.896 | 0 / 00.952 | 0 / 00.840 | 0 / 00.881 | 0 / 00.836 | 0 / 00.838 | 0 / 00.869 | 0 / 00.849 |
| celeris-1.1-decision Closed · Efficient | — | No answers at this threshold0.859 | 0 / 00.863 | 0 / 00.790 | 0 / 00.875 | 0 / 00.790 | 0 / 00.863 | 0 / 00.896 | 0 / 00.944 | 0 / 00.853 | 0 / 00.877 | 0 / 00.812 | 0 / 00.852 | 0 / 00.867 | 0 / 00.855 |
| Drex Max Open · 27B | — | 719 / 100.861 | 67 / 10.834 | 25 / 00.803 | 3 / 00.750 | 32 / 00.799 | 28 / 00.867 | 36 / 40.794 | 63 / 00.923 | 42 / 20.877 | 75 / 00.894 | 44 / 00.908 | 83 / 00.876 | 96 / 20.855 | 125 / 10.847 |
| KnowLine-31B-Gen2 Open · 31B | — | 469 / 10.898 | 38 / 00.916 | 15 / 00.862 | 2 / 00.742 | 14 / 00.847 | 24 / 00.902 | 37 / 00.921 | 70 / 00.956 | 12 / 00.871 | 28 / 00.882 | 28 / 00.916 | 56 / 10.863 | 68 / 00.908 | 77 / 00.886 |
| KnowLine-31B-Gen1.5 Open · 31B | — | 464 / 10.895 | 38 / 00.874 | 15 / 00.850 | 1 / 00.750 | 14 / 00.881 | 22 / 00.903 | 32 / 00.900 | 60 / 00.956 | 18 / 00.912 | 35 / 00.872 | 33 / 00.904 | 55 / 10.885 | 64 / 00.914 | 77 / 00.888 |
| SubQ Snap 1.1 Closed · Efficient | — | 360 / 10.839 | 28 / 00.819 | 12 / 00.784 | 1 / 00.722 | 15 / 00.780 | 10 / 00.808 | 19 / 00.754 | 47 / 00.929 | 35 / 00.829 | 31 / 00.832 | 28 / 00.931 | 37 / 10.802 | 45 / 00.871 | 52 / 00.848 |
| Mercury Decide Closed · Efficient | — | 272 / 90.789 | 25 / 00.811 | 9 / 00.692 | 1 / 00.729 | 8 / 00.748 | 6 / 00.687 | 11 / 30.718 | 26 / 10.859 | 23 / 20.801 | 31 / 10.808 | 18 / 00.821 | 22 / 10.751 | 44 / 10.785 | 48 / 00.822 |
| Cerno 9B v1.2 Open · 9.0B | — | 244 / 100.780 | 24 / 00.806 | 9 / 00.677 | 1 / 10.561 | 8 / 00.679 | 6 / 00.797 | 11 / 00.751 | 31 / 00.843 | 17 / 00.747 | 22 / 20.806 | 13 / 00.812 | 31 / 20.720 | 33 / 30.836 | 38 / 20.776 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 207 / 30.804 | 18 / 10.778 | 5 / 00.684 | 0 / 00.606 | 7 / 00.731 | 7 / 10.745 | 8 / 00.759 | 29 / 10.839 | 10 / 00.829 | 18 / 00.822 | 12 / 00.853 | 33 / 00.824 | 24 / 00.814 | 36 / 00.820 |
| GLiDE 27B Open · 27B | — | 198 / 10.864 | 17 / 10.858 | 6 / 00.787 | 0 / 00.569 | 8 / 00.761 | 5 / 00.846 | 7 / 00.870 | 22 / 00.948 | 20 / 00.938 | 36 / 00.873 | 18 / 00.905 | 21 / 00.861 | 20 / 00.848 | 18 / 00.875 |
| Adjudex 9B v5 Open · 9.0B | — | 197 / 10.806 | 18 / 00.815 | 4 / 00.701 | 0 / 00.729 | 6 / 00.684 | 8 / 00.764 | 9 / 00.797 | 31 / 00.867 | 12 / 10.814 | 21 / 00.820 | 16 / 00.840 | 23 / 00.813 | 21 / 00.812 | 28 / 00.806 |
| Elephant-1 Open · 9.0B | — | 182 / 10.803 | 14 / 00.821 | 5 / 00.711 | 0 / 00.486 | 6 / 00.684 | 6 / 00.823 | 10 / 00.730 | 28 / 00.833 | 10 / 10.785 | 15 / 00.868 | 14 / 00.879 | 25 / 00.778 | 24 / 00.834 | 25 / 00.774 |
| d3 Open · 26B | — | 162 / 10.878 | 13 / 10.867 | 5 / 00.791 | 1 / 00.806 | 6 / 00.811 | 7 / 00.907 | 8 / 00.847 | 21 / 00.955 | 15 / 00.904 | 25 / 00.905 | 13 / 00.893 | 21 / 00.886 | 13 / 00.875 | 14 / 00.872 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 150 / 50.781 | 7 / 10.781 | 4 / 00.688 | 0 / 00.300 | 2 / 00.694 | 3 / 00.780 | 9 / 00.732 | 26 / 00.884 | 10 / 10.807 | 21 / 00.854 | 11 / 10.773 | 15 / 10.730 | 19 / 00.806 | 23 / 10.753 |
| Chinese Decision Pointer 4B Open · 4.0B | — | 149 / 60.746 | 14 / 20.793 | 5 / 00.668 | 0 / 00.424 | 5 / 00.719 | 2 / 00.797 | 5 / 20.655 | 16 / 00.735 | 8 / 10.772 | 19 / 00.732 | 7 / 00.768 | 16 / 00.771 | 24 / 10.729 | 28 / 00.726 |
| Verdin-4B-v1.1 Open · 4.0B | — | 126 / 40.752 | 14 / 10.731 | 5 / 00.719 | 0 / 00.548 | 4 / 00.677 | 6 / 00.743 | 8 / 10.763 | 16 / 00.824 | 8 / 00.774 | 14 / 00.799 | 9 / 20.772 | 17 / 00.761 | 12 / 00.772 | 13 / 00.696 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 113 / 10.787 | 6 / 00.765 | 5 / 00.695 | 0 / 00.450 | 2 / 00.701 | 3 / 00.836 | 9 / 00.788 | 23 / 00.913 | 7 / 00.830 | 12 / 00.797 | 7 / 00.792 | 11 / 00.764 | 15 / 00.792 | 13 / 10.765 |
| Winnow-12B Open · 12B | — | 20 / 10.743 | 5 / 00.742 | 1 / 00.638 | 0 / 00.686 | 0 / 00.670 | 1 / 00.642 | 0 / 10.743 | 1 / 00.787 | 1 / 00.742 | 5 / 00.731 | 3 / 00.785 | 1 / 00.746 | 1 / 00.758 | 1 / 00.774 |
| GLiDE 2B Open · 1.9B | — | 15 / 10.635 | 3 / 10.625 | 0 / 00.569 | 0 / 00.519 | 1 / 00.554 | 0 / 00.590 | 1 / 00.676 | 1 / 00.646 | 1 / 00.728 | 3 / 00.663 | 2 / 00.596 | 1 / 00.617 | 1 / 00.616 | 1 / 00.639 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 837 answers, 837 right · 0 wrong · AUROC 0.903 Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 85.99chosen-answer Brier 0.0958 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 470 answers, 469 right · 1 wrong · AUROC 0.898 Logic 58.9% | — | 80.3%78.5%–81.9% | ab | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 85.05chosen-answer Brier 0.1016 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 178 answers, 178 right · 0 wrong · AUROC 0.903 Logic 59.3% | — | 78.0%76.2%–79.7% | bc | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 83.57chosen-answer Brier 0.1084 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 465 answers, 464 right · 1 wrong · AUROC 0.895 Logic 59.0% | — | 77.4%75.6%–79.1% | cd | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 83.18chosen-answer Brier 0.1103 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 318 answers, 318 right · 0 wrong · AUROC 0.899 Logic 58.7% | — | 77.3%75.5%–79.1% | cd | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 83.06chosen-answer Brier 0.1122 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 215 answers, 215 right · 0 wrong · AUROC 0.891 Logic 57.9% | — | 77.1%75.3%–78.8% | cd | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 82.89chosen-answer Brier 0.1133 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 15 answers, 15 right · 0 wrong · AUROC 0.899 Logic 57.8% | — | 75.2%73.4%–77.0% | de | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 81.60chosen-answer Brier 0.1201 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 213 answers, 213 right · 0 wrong · AUROC 0.876 Logic 54.9% | — | 74.1%72.2%–75.9% | defg | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 80.88chosen-answer Brier 0.1235 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 412 answers, 412 right · 0 wrong · AUROC 0.895 Logic 54.5% | — | 74.0%72.1%–75.8% | defh | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 81.16chosen-answer Brier 0.1164 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 323 answers, 323 right · 0 wrong · AUROC 0.856 Logic 54.8% | — | 72.5%70.6%–74.3% | efghi | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 79.32chosen-answer Brier 0.1385 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 199 answers, 198 right · 1 wrong · AUROC 0.864 Logic 58.4% | — | 72.0%70.1%–73.8% | efghij | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 79.22chosen-answer Brier 0.1354 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 11 answers, 11 right · 0 wrong · AUROC 0.857 Logic 54.6% | — | 71.5%69.6%–73.4% | fghij | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 78.39chosen-answer Brier 0.1474 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 214 answers, 214 right · 0 wrong · AUROC 0.859 Logic 50.0% | — | 71.3%69.4%–73.2% | efghij | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 78.84chosen-answer Brier 0.1366 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 161 answers, 161 right · 0 wrong · AUROC 0.865 Logic 54.5% | — | 70.8%68.8%–72.7% | fghij | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 78.56chosen-answer Brier 0.1367 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 163 answers, 162 right · 1 wrong · AUROC 0.878 Logic 54.5% | — | 70.4%68.5%–72.3% | gij | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 78.71chosen-answer Brier 0.1299 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 729 answers, 719 right · 10 wrong · AUROC 0.861 Logic 53.1% | — | 70.3%68.4%–72.2% | gijk | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 77.10chosen-answer Brier 0.1611 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 315 answers, 315 right · 0 wrong · AUROC 0.867 Logic 54.8% | — | 70.1%68.1%–72.0% | ijk | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 78.01chosen-answer Brier 0.1407 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.858 Logic 47.3% | — | 70.0%68.0%–71.9% | hijk | 1,514 / 2,163 | No answers at this threshold | 0.858 | 77.96chosen-answer Brier 0.1407 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 131 answers, 131 right · 0 wrong · AUROC 0.862 Logic 55.2% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 77.95chosen-answer Brier 0.1396 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.859 Logic 47.3% | — | 69.9%67.9%–71.8% | ijkl | 1,511 / 2,163 | No answers at this threshold | 0.859 | 77.91chosen-answer Brier 0.1403 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 115 answers, 115 right · 0 wrong · AUROC 0.876 Logic 51.6% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 78.23chosen-answer Brier 0.1340 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 331 answers, 331 right · 0 wrong · AUROC 0.869 Logic 53.3% | — | 69.8%67.8%–71.7% | ijkl | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 78.16chosen-answer Brier 0.1345 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 326 answers, 326 right · 0 wrong · AUROC 0.873 Logic 52.5% | with web | 69.3%67.3%–71.2% | jklm | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 77.94chosen-answer Brier 0.1342 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 43 answers, 43 right · 0 wrong · AUROC 0.864 Logic 52.3% | — | 69.0%67.0%–70.9% | ijklmn | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 77.52chosen-answer Brier 0.1399 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 361 answers, 360 right · 1 wrong · AUROC 0.839 Logic 52.4% | — | 68.6%66.6%–70.5% | jklmn | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 76.15chosen-answer Brier 0.1626 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.819 Logic 50.6% | — | 66.3%64.2%–68.2% | klmno | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 74.96chosen-answer Brier 0.1632 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 141 answers, 141 right · 0 wrong · AUROC 0.838 Logic 49.2% | — | 65.5%63.5%–67.5% | lmnop | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 74.98chosen-answer Brier 0.1556 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 183 answers, 183 right · 0 wrong · AUROC 0.829 Logic 46.0% | — | 65.2%63.2%–67.2% | mnop | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 74.62chosen-answer Brier 0.1595 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 202 answers, 202 right · 0 wrong · AUROC 0.829 Logic 48.2% | — | 65.1%63.1%–67.1% | nop | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 74.56chosen-answer Brier 0.1602 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 248 answers, 248 right · 0 wrong · AUROC 0.834 Logic 46.3% | — | 64.1%62.0%–66.1% | opq | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 74.08chosen-answer Brier 0.1592 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 198 answers, 197 right · 1 wrong · AUROC 0.806 Logic 46.4% | — | 62.1%60.0%–64.1% | opqr | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 72.28chosen-answer Brier 0.1752 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 183 answers, 182 right · 1 wrong · AUROC 0.803 Logic 46.4% | — | 61.3%59.3%–63.4% | pqr | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 71.86chosen-answer Brier 0.1763 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.803 Logic 43.9% | — | 60.1%58.0%–62.1% | qrs | 1,300 / 2,163 | No answers at this threshold | 0.803 | 71.15chosen-answer Brier 0.1779 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 64 answers, 64 right · 0 wrong · AUROC 0.814 Logic 48.2% | — | 59.8%57.7%–61.9% | qrs | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 71.32chosen-answer Brier 0.1719 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 123 answers, 123 right · 0 wrong · AUROC 0.799 Logic 44.3% | — | 59.8%57.7%–61.8% | qrs | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 71.08chosen-answer Brier 0.1762 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 27 answers, 27 right · 0 wrong · AUROC 0.787 Logic 44.4% | — | 59.4%57.3%–61.4% | rst | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 70.49chosen-answer Brier 0.1839 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 174 answers, 174 right · 0 wrong · AUROC 0.813 Logic 41.9% | — | 59.0%56.9%–61.0% | rstu | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 70.98chosen-answer Brier 0.1703 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 210 answers, 207 right · 3 wrong · AUROC 0.804 Logic 45.3% | — | 58.8%56.7%–60.9% | rstu | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 69.67chosen-answer Brier 0.1946 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 49 answers, 49 right · 0 wrong · AUROC 0.781 Logic 46.6% | — | 58.4%56.3%–60.5% | rstuv | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 69.90chosen-answer Brier 0.1864 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 73 answers, 73 right · 0 wrong · AUROC 0.798 Logic 47.8% | — | 57.9%55.8%–60.0% | rstuvw | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 69.69chosen-answer Brier 0.1855 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 33 answers, 33 right · 0 wrong · AUROC 0.789 Logic 44.1% | — | 56.8%54.7%–58.9% | stuvw | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 69.18chosen-answer Brier 0.1846 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 254 answers, 244 right · 10 wrong · AUROC 0.780 Logic 43.3% | — | 56.0%53.9%–58.1% | stuvw | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 67.15chosen-answer Brier 0.2174 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.765 Logic 42.3% | — | 56.0%53.9%–58.1% | stuvw | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 67.89chosen-answer Brier 0.2021 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 114 answers, 113 right · 1 wrong · AUROC 0.787 Logic 44.7% | — | 55.9%53.8%–58.0% | stuvw | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 67.89chosen-answer Brier 0.2015 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 96 answers, 96 right · 0 wrong · AUROC 0.779 Logic 45.9% | — | 55.8%53.7%–57.9% | stuvwx | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 68.25chosen-answer Brier 0.1934 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 152 answers, 152 right · 0 wrong · AUROC 0.767 Logic 40.8% | — | 55.6%53.5%–57.7% | stuvwxy | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 68.16chosen-answer Brier 0.1930 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 130 answers, 130 right · 0 wrong · AUROC 0.790 Logic 44.4% | — | 55.3%53.2%–57.4% | tuvwxy | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 68.37chosen-answer Brier 0.1860 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 155 answers, 150 right · 5 wrong · AUROC 0.781 Logic 43.1% | — | 55.2%53.1%–57.3% | tuvwxy | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 67.61chosen-answer Brier 0.2002 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.764 Logic 42.6% | — | 54.6%52.4%–56.6% | uvwxyz | 1,180 / 2,163 | No answers at this threshold | 0.764 | 64.53chosen-answer Brier 0.2549 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 26 answers, 26 right · 0 wrong · AUROC 0.783 Logic 40.6% | — | 53.8%51.7%–55.9% | vwxyzA | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 67.54chosen-answer Brier 0.1869 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 281 answers, 272 right · 9 wrong · AUROC 0.789 Logic 45.1% | — | 53.5%51.4%–55.6% | wxyzA | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 66.02chosen-answer Brier 0.2146 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 130 answers, 126 right · 4 wrong · AUROC 0.752 Logic 39.7% | — | 51.5%49.4%–53.6% | yzAB | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 65.62chosen-answer Brier 0.2026 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 12 answers, 12 right · 0 wrong · AUROC 0.767 Logic 40.8% | — | 51.1%49.0%–53.2% | xyzAB | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 65.20chosen-answer Brier 0.2070 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 155 answers, 149 right · 6 wrong · AUROC 0.746 Logic 40.4% | — | 50.4%48.3%–52.5% | zAB | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 64.11chosen-answer Brier 0.2221 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.730 Logic 38.9% | — | 49.8%47.7%–51.9% | AB | 1,077 / 2,163 | No answers at this threshold | 0.730 | 64.44chosen-answer Brier 0.2092 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 21 answers, 20 right · 1 wrong · AUROC 0.743 Logic 39.7% | — | 46.9%44.8%–49.0% | BC | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 63.00chosen-answer Brier 0.2087 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.735 Logic 39.3% | — | 44.3%42.3%–46.4% | C | 959 / 2,163 | No answers at this threshold | 0.735 | 61.87chosen-answer Brier 0.2060 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.651 Logic 32.1% | — | 39.2%37.1%–41.2% | D | 847 / 2,163 | No answers at this threshold | 0.651 | 58.63chosen-answer Brier 0.2190 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.685 Logic 30.8% | — | 38.7%36.6%–40.7% | D | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 58.95chosen-answer Brier 0.2075 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.656 Logic 30.7% | — | 37.1%35.1%–39.1% | D | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 57.72chosen-answer Brier 0.2163 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 16 answers, 15 right · 1 wrong · AUROC 0.635 Logic 32.3% | — | 36.8%34.7%–38.8% | D | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 55.70chosen-answer Brier 0.2536 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.636 Logic 31.7% | — | 35.3%33.3%–37.3% | DE | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 56.49chosen-answer Brier 0.2229 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.589 Logic 29.2% | — | 32.1%30.2%–34.1% | EF | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 53.21chosen-answer Brier 0.2566 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.583 Logic 24.7% | — | 29.7%27.8%–31.6% | F | 642 / 2,163 | No answers at this threshold | 0.583 | 54.40chosen-answer Brier 0.2088 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.569 Logic 22.4% | — | 28.7%26.8%–30.6% | F | 620 / 2,163 | No answers at this threshold | 0.569 | 53.90chosen-answer Brier 0.2086 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: No answers at this threshold · AUROC 0.592 Logic 27.0% | — | 28.2%26.4%–30.2% | F | 611 / 2,163 | No answers at this threshold | 0.592 | 53.62chosen-answer Brier 0.2100 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.596 Logic 26.7% | — | 28.2%26.3%–30.1% | F | 610 / 2,163 | No answers at this threshold | 0.596 | 53.60chosen-answer Brier 0.2100 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.501 Logic 23.6% | — | 22.7%20.9%–24.5% | G | 490 / 2,163 | No answers at this threshold | 0.501 | 52.01chosen-answer Brier 0.1863 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.500 Logic 23.2% | — | 20.6%18.9%–22.3% | G | 445 / 2,163 | No answers at this threshold | 0.500 | 50.98chosen-answer Brier 0.1862 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
StartLux PelaAI scar-ai Fastino Liquid AI Other Metask Lab OmniJev Strands Agents Perplexity quincyqiang victorbennett Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 55.8%2,163 | 70.8%62.3% kept (1,348) | 81.1%40.5% kept (876) | 94.0%20.8% kept (450) | 100.0%4.4% kept (96) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 56.0%2,163 | 67.1%67.4% kept (1,457) | 79.3%41.9% kept (907) | 92.8%21.3% kept (460) | 100.0%2.5% kept (54) |
| vera-medium-omni Open · 4.2B | — | 56.8%2,163 | 75.4%55.2% kept (1,193) | 88.5%29.6% kept (641) | 98.9%12.3% kept (267) | 100.0%1.5% kept (33) |
| KnowLine-2B-Gen1 Open · 2.0B | — | 35.3%2,163 | 52.1%28.2% kept (609) | 67.2%8.9% kept (192) | 88.5%2.4% kept (52) | 100.0%0.4% kept (8) |
| GLiDE 0.8B Open · 0.8B | — | 32.1%2,163 | 38.1%44.3% kept (959) | 47.5%16.8% kept (364) | 73.0%2.9% kept (63) | 100.0%0.2% kept (4) |
| d1-2B Open · 1.9B | — | 37.1%2,163 | 58.0%25.0% kept (541) | 74.5%7.4% kept (161) | 83.7%2.3% kept (49) | 100.0%0.2% kept (4) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 38.7%2,163 | 68.1%21.1% kept (457) | 83.6%7.9% kept (171) | 92.0%2.3% kept (50) | 100.0%0.1% kept (2) |
| GLiClass v4 Edge Open · 0.4B | — | 22.7%2,163 | 26.9%3.6% kept (78) | 38.5%0.6% kept (13) | 100.0%0.0% kept (1) | none kept |
| GLiDE 0.4B Open · 0.4B | — | 20.6%2,163 | 23.5%5.5% kept (119) | 33.3%1.0% kept (21) | none kept | none kept |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 28.2%2,163 | 41.7%15.5% kept (336) | 54.9%3.3% kept (71) | 75.0%0.2% kept (4) | none kept |
| Metask-Jev-4B Open · 4.5B | — | 39.2%2,163 | 63.1%19.8% kept (428) | 77.5%8.6% kept (187) | 88.2%1.6% kept (34) | none kept |
| OneJev1.6-0.8B Open · 0.8B | — | 28.2%2,163 | 41.8%15.3% kept (330) | 54.1%3.4% kept (74) | 66.7%0.7% kept (15) | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 29.7%2,163 | 45.9%10.6% kept (229) | 61.0%3.6% kept (77) | 92.3%0.6% kept (13) | none kept |
| Strands Decider 2B Open · 2.0B | — | 28.7%2,163 | 45.9%11.8% kept (255) | 69.2%1.2% kept (26) | 100.0%0.0% kept (1) | none kept |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 55.2%2,163 | 69.1%65.3% kept (1,412) | 79.3%46.3% kept (1,001) | 88.3%25.8% kept (557) | 96.8%7.2% kept (155) |
| Chinese Decision Pointer 4B Open · 4.0B | — | 50.4%2,163 | 60.9%66.6% kept (1,440) | 74.3%39.1% kept (845) | 86.2%20.4% kept (442) | 96.1%7.2% kept (155) |
| Verdin-4B-v1.1 Open · 4.0B | — | 51.5%2,163 | 69.1%51.5% kept (1,115) | 81.6%29.9% kept (646) | 91.6%16.0% kept (346) | 96.9%6.0% kept (130) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 55.9%2,163 | 67.9%69.5% kept (1,504) | 77.7%49.1% kept (1,061) | 89.5%27.7% kept (599) | 99.1%5.3% kept (114) |
| GLiDE 2B Open · 1.9B | — | 36.8%2,163 | 45.5%49.8% kept (1,078) | 56.1%21.8% kept (471) | 73.8%5.6% kept (122) | 93.8%0.7% kept (16) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 96 / 00.779 | 11 / 00.818 | 1 / 00.707 | 0 / 00.524 | 2 / 00.676 | 2 / 00.743 | 7 / 00.748 | 18 / 00.814 | 6 / 00.808 | 11 / 00.806 | 6 / 00.744 | 12 / 00.768 | 12 / 00.778 | 8 / 00.775 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 54 / 00.765 | 3 / 00.791 | 2 / 00.611 | 0 / 00.500 | 1 / 00.688 | 1 / 00.814 | 4 / 00.699 | 11 / 00.845 | 1 / 00.842 | 7 / 00.813 | 1 / 00.742 | 7 / 00.743 | 9 / 00.803 | 7 / 00.747 |
| vera-medium-omni Open · 4.2B | — | 33 / 00.789 | 2 / 00.782 | 3 / 00.704 | 0 / 00.636 | 1 / 00.695 | 1 / 00.836 | 1 / 00.706 | 13 / 00.860 | 0 / 00.848 | 2 / 00.837 | 1 / 00.797 | 4 / 00.770 | 1 / 00.809 | 4 / 00.784 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 8 / 00.636 | 0 / 00.570 | 0 / 00.578 | 0 / 00.533 | 0 / 00.508 | 0 / 00.540 | 0 / 00.655 | 0 / 00.667 | 1 / 00.779 | 4 / 00.736 | 1 / 00.699 | 1 / 00.644 | 0 / 00.656 | 1 / 00.587 |
| GLiDE 0.8B Open · 0.8B | — | 4 / 00.589 | 1 / 00.538 | 0 / 00.498 | 0 / 00.467 | 0 / 00.610 | 0 / 00.579 | 0 / 00.525 | 0 / 00.710 | 1 / 00.604 | 0 / 00.601 | 0 / 00.661 | 1 / 00.673 | 0 / 00.564 | 1 / 00.616 |
| d1-2B Open · 1.9B | — | 4 / 00.656 | 0 / 00.634 | 0 / 00.629 | 0 / 00.318 | 0 / 00.572 | 1 / 00.582 | 0 / 00.589 | 0 / 00.721 | 1 / 00.778 | 1 / 00.625 | 0 / 00.718 | 1 / 00.709 | 0 / 00.657 | 0 / 00.677 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 2 / 00.685 | 0 / 00.661 | 0 / 00.601 | 0 / 00.333 | 0 / 00.560 | 0 / 00.665 | 0 / 00.655 | 0 / 00.788 | 0 / 00.813 | 1 / 00.755 | 0 / 00.703 | 0 / 00.614 | 0 / 00.658 | 1 / 00.679 |
| GLiClass v4 Edge Open · 0.4B | — | No answers at this threshold0.501 | 0 / 00.524 | 0 / 00.554 | 0 / 00.550 | 0 / 00.476 | 0 / 00.474 | 0 / 00.505 | 0 / 00.412 | 0 / 00.522 | 0 / 00.412 | 0 / 00.460 | 0 / 00.566 | 0 / 00.543 | 0 / 00.518 |
| GLiDE 0.4B Open · 0.4B | — | No answers at this threshold0.500 | 0 / 00.496 | 0 / 00.528 | 0 / 00.875 | 0 / 00.509 | 0 / 00.606 | 0 / 00.458 | 0 / 00.370 | 0 / 00.632 | 0 / 00.450 | 0 / 00.478 | 0 / 00.493 | 0 / 00.498 | 0 / 00.515 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | No answers at this threshold0.596 | 0 / 00.486 | 0 / 00.603 | 0 / 00.667 | 0 / 00.628 | 0 / 00.573 | 0 / 00.490 | 0 / 00.683 | 0 / 00.792 | 0 / 00.660 | 0 / 00.653 | 0 / 00.603 | 0 / 00.557 | 0 / 00.565 |
| Metask-Jev-4B Open · 4.5B | — | No answers at this threshold0.651 | 0 / 00.614 | 0 / 00.544 | 0 / 00.933 | 0 / 00.617 | 0 / 00.737 | 0 / 00.613 | 0 / 00.633 | 0 / 00.619 | 0 / 00.666 | 0 / 00.727 | 0 / 00.660 | 0 / 00.624 | 0 / 00.664 |
| OneJev1.6-0.8B Open · 0.8B | — | No answers at this threshold0.592 | 0 / 00.548 | 0 / 00.601 | 0 / 00.533 | 0 / 00.519 | 0 / 00.678 | 0 / 00.474 | 0 / 00.543 | 0 / 00.726 | 0 / 00.690 | 0 / 00.619 | 0 / 00.605 | 0 / 00.609 | 0 / 00.559 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | No answers at this threshold0.583 | 0 / 00.611 | 0 / 00.460 | 0 / 00.667 | 0 / 00.546 | 0 / 00.440 | 0 / 00.612 | 0 / 00.567 | 0 / 00.693 | 0 / 00.672 | 0 / 00.676 | 0 / 00.551 | 0 / 00.503 | 0 / 00.580 |
| Strands Decider 2B Open · 2.0B | — | No answers at this threshold0.569 | 0 / 00.510 | 0 / 00.458 | 0 / 00.615 | 0 / 00.534 | 0 / 00.589 | 0 / 00.583 | 0 / 00.635 | 0 / 00.615 | 0 / 00.574 | 0 / 00.632 | 0 / 00.556 | 0 / 00.542 | 0 / 00.627 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 150 / 50.781 | 7 / 10.781 | 4 / 00.688 | 0 / 00.300 | 2 / 00.694 | 3 / 00.780 | 9 / 00.732 | 26 / 00.884 | 10 / 10.807 | 21 / 00.854 | 11 / 10.773 | 15 / 10.730 | 19 / 00.806 | 23 / 10.753 |
| Chinese Decision Pointer 4B Open · 4.0B | — | 149 / 60.746 | 14 / 20.793 | 5 / 00.668 | 0 / 00.424 | 5 / 00.719 | 2 / 00.797 | 5 / 20.655 | 16 / 00.735 | 8 / 10.772 | 19 / 00.732 | 7 / 00.768 | 16 / 00.771 | 24 / 10.729 | 28 / 00.726 |
| Verdin-4B-v1.1 Open · 4.0B | — | 126 / 40.752 | 14 / 10.731 | 5 / 00.719 | 0 / 00.548 | 4 / 00.677 | 6 / 00.743 | 8 / 10.763 | 16 / 00.824 | 8 / 00.774 | 14 / 00.799 | 9 / 20.772 | 17 / 00.761 | 12 / 00.772 | 13 / 00.696 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 113 / 10.787 | 6 / 00.765 | 5 / 00.695 | 0 / 00.450 | 2 / 00.701 | 3 / 00.836 | 9 / 00.788 | 23 / 00.913 | 7 / 00.830 | 12 / 00.797 | 7 / 00.792 | 11 / 00.764 | 15 / 00.792 | 13 / 10.765 |
| GLiDE 2B Open · 1.9B | — | 15 / 10.635 | 3 / 10.625 | 0 / 00.569 | 0 / 00.519 | 1 / 00.554 | 0 / 00.590 | 1 / 00.676 | 1 / 00.646 | 1 / 00.728 | 3 / 00.663 | 2 / 00.596 | 1 / 00.617 | 1 / 00.616 | 1 / 00.639 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 33 answers, 33 right · 0 wrong · AUROC 0.789 Logic 44.1% | — | 56.8%54.7%–58.9% | a | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 69.18chosen-answer Brier 0.1846 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.765 Logic 42.3% | — | 56.0%53.9%–58.1% | a | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 67.89chosen-answer Brier 0.2021 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 114 answers, 113 right · 1 wrong · AUROC 0.787 Logic 44.7% | — | 55.9%53.8%–58.0% | a | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 67.89chosen-answer Brier 0.2015 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 96 answers, 96 right · 0 wrong · AUROC 0.779 Logic 45.9% | — | 55.8%53.7%–57.9% | a | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 68.25chosen-answer Brier 0.1934 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 155 answers, 150 right · 5 wrong · AUROC 0.781 Logic 43.1% | — | 55.2%53.1%–57.3% | a | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 67.61chosen-answer Brier 0.2002 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 130 answers, 126 right · 4 wrong · AUROC 0.752 Logic 39.7% | — | 51.5%49.4%–53.6% | b | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 65.62chosen-answer Brier 0.2026 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 155 answers, 149 right · 6 wrong · AUROC 0.746 Logic 40.4% | — | 50.4%48.3%–52.5% | b | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 64.11chosen-answer Brier 0.2221 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.651 Logic 32.1% | — | 39.2%37.1%–41.2% | c | 847 / 2,163 | No answers at this threshold | 0.651 | 58.63chosen-answer Brier 0.2190 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.685 Logic 30.8% | — | 38.7%36.6%–40.7% | c | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 58.95chosen-answer Brier 0.2075 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.656 Logic 30.7% | — | 37.1%35.1%–39.1% | c | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 57.72chosen-answer Brier 0.2163 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 16 answers, 15 right · 1 wrong · AUROC 0.635 Logic 32.3% | — | 36.8%34.7%–38.8% | c | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 55.70chosen-answer Brier 0.2536 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.636 Logic 31.7% | — | 35.3%33.3%–37.3% | cd | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 56.49chosen-answer Brier 0.2229 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.589 Logic 29.2% | — | 32.1%30.2%–34.1% | de | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 53.21chosen-answer Brier 0.2566 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.583 Logic 24.7% | — | 29.7%27.8%–31.6% | ef | 642 / 2,163 | No answers at this threshold | 0.583 | 54.40chosen-answer Brier 0.2088 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.569 Logic 22.4% | — | 28.7%26.8%–30.6% | ef | 620 / 2,163 | No answers at this threshold | 0.569 | 53.90chosen-answer Brier 0.2086 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: No answers at this threshold · AUROC 0.592 Logic 27.0% | — | 28.2%26.4%–30.2% | f | 611 / 2,163 | No answers at this threshold | 0.592 | 53.62chosen-answer Brier 0.2100 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.596 Logic 26.7% | — | 28.2%26.3%–30.1% | f | 610 / 2,163 | No answers at this threshold | 0.596 | 53.60chosen-answer Brier 0.2100 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.501 Logic 23.6% | — | 22.7%20.9%–24.5% | g | 490 / 2,163 | No answers at this threshold | 0.501 | 52.01chosen-answer Brier 0.1863 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.500 Logic 23.2% | — | 20.6%18.9%–22.3% | g | 445 / 2,163 | No answers at this threshold | 0.500 | 50.98chosen-answer Brier 0.1862 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Valen-Team PelaAI Fastino tattrongvu StartLux Cloudflare Blockbrain Labs Apertix AI Patrick-18 newsbang Base Labs EldanRing Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Valen-Decision-1.5-9B Open · 9.0B | — | 55.3%2,163 | 74.3%54.9% kept (1,187) | 86.7%32.3% kept (698) | 94.9%17.3% kept (374) | 100.0%6.0% kept (130) |
| KnowLine-9B-Gen1 Open · 9.0B | — | 59.8%2,163 | 76.7%58.4% kept (1,264) | 89.3%36.8% kept (797) | 98.0%20.5% kept (444) | 100.0%5.7% kept (123) |
| GLiDE 9B Open · 7.9B | — | 57.9%2,163 | 70.8%68.3% kept (1,478) | 84.7%42.8% kept (926) | 94.7%20.9% kept (451) | 100.0%3.4% kept (73) |
| Jeffan-9B-1211 Open · 9.0B | — | 59.8%2,163 | 77.8%60.3% kept (1,304) | 90.4%38.1% kept (824) | 98.5%18.2% kept (393) | 100.0%3.0% kept (64) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 58.4%2,163 | 74.9%58.2% kept (1,258) | 85.1%37.4% kept (808) | 96.6%19.1% kept (413) | 100.0%2.3% kept (49) |
| KnowLine-12B-Gen1 Open · 12B | — | 66.3%2,163 | 83.7%60.8% kept (1,315) | 92.1%42.2% kept (912) | 99.1%19.6% kept (425) | 100.0%0.3% kept (7) |
| Clef-Flash Open · 9.0B | — | 49.8%2,163 | 68.7%44.7% kept (967) | 81.3%23.3% kept (504) | 94.4%7.4% kept (161) | none kept |
| Cygnet Open · 12B | — | 44.3%2,163 | 65.9%41.9% kept (907) | 84.2%14.9% kept (322) | 98.7%3.5% kept (75) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | 60.1%2,163 | 80.7%52.3% kept (1,131) | 92.5%30.9% kept (668) | 99.6%10.3% kept (223) | none kept |
| Cerno 9B v1.2 Open · 9.0B | — | 56.0%2,163 | 64.6%75.7% kept (1,638) | 74.3%55.1% kept (1,192) | 86.2%33.8% kept (731) | 96.1%11.7% kept (254) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 58.8%2,163 | 69.2%74.8% kept (1,619) | 79.2%53.7% kept (1,162) | 91.9%32.1% kept (694) | 98.6%9.7% kept (210) |
| Adjudex 9B v5 Open · 9.0B | — | 62.1%2,163 | 74.7%68.7% kept (1,486) | 86.2%46.0% kept (994) | 96.3%27.2% kept (589) | 99.5%9.2% kept (198) |
| Elephant-1 Open · 9.0B | — | 61.3%2,163 | 75.7%65.6% kept (1,419) | 86.7%44.0% kept (952) | 95.8%26.2% kept (567) | 99.5%8.5% kept (183) |
| Winnow-12B Open · 12B | — | 46.9%2,163 | 64.6%47.3% kept (1,023) | 77.3%30.4% kept (657) | 89.5%16.2% kept (351) | 95.2%1.0% kept (21) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Valen-Decision-1.5-9B Open · 9.0B | — | 130 / 00.790 | 16 / 00.837 | 3 / 00.663 | 0 / 00.667 | 7 / 00.778 | 4 / 00.849 | 5 / 00.753 | 17 / 00.876 | 9 / 00.765 | 12 / 00.777 | 7 / 00.834 | 13 / 00.782 | 16 / 00.795 | 21 / 00.776 |
| KnowLine-9B-Gen1 Open · 9.0B | — | 123 / 00.799 | 13 / 00.816 | 5 / 00.712 | 0 / 00.595 | 6 / 00.674 | 6 / 00.803 | 5 / 00.760 | 19 / 00.873 | 6 / 00.799 | 14 / 00.792 | 7 / 00.780 | 17 / 00.803 | 11 / 00.821 | 14 / 00.817 |
| GLiDE 9B Open · 7.9B | — | 73 / 00.798 | 7 / 00.800 | 2 / 00.737 | 0 / 00.731 | 3 / 00.700 | 1 / 00.716 | 4 / 00.752 | 15 / 00.805 | 3 / 00.774 | 10 / 00.873 | 4 / 00.853 | 7 / 00.776 | 8 / 00.851 | 9 / 00.765 |
| Jeffan-9B-1211 Open · 9.0B | — | 64 / 00.814 | 4 / 00.865 | 4 / 00.677 | 0 / 00.769 | 3 / 00.732 | 3 / 00.844 | 3 / 00.745 | 15 / 00.811 | 4 / 00.797 | 6 / 00.888 | 2 / 00.825 | 5 / 00.804 | 7 / 00.808 | 8 / 00.768 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 49 / 00.781 | 3 / 00.777 | 1 / 00.678 | 0 / 00.394 | 1 / 00.703 | 1 / 00.730 | 3 / 00.735 | 11 / 00.887 | 3 / 00.774 | 6 / 00.803 | 5 / 00.830 | 4 / 00.775 | 2 / 00.788 | 9 / 00.779 |
| KnowLine-12B-Gen1 Open · 12B | — | 7 / 00.819 | 2 / 00.838 | 0 / 00.736 | 0 / 00.814 | 0 / 00.752 | 0 / 00.840 | 0 / 00.833 | 5 / 00.915 | 0 / 00.800 | 0 / 00.824 | 0 / 00.916 | 0 / 00.775 | 0 / 00.857 | 0 / 00.794 |
| Clef-Flash Open · 9.0B | — | No answers at this threshold0.730 | 0 / 00.711 | 0 / 00.642 | 0 / 00.361 | 0 / 00.614 | 0 / 00.736 | 0 / 00.670 | 0 / 00.766 | 0 / 00.789 | 0 / 00.744 | 0 / 00.823 | 0 / 00.722 | 0 / 00.804 | 0 / 00.737 |
| Cygnet Open · 12B | — | No answers at this threshold0.735 | 0 / 00.730 | 0 / 00.617 | 0 / 00.600 | 0 / 00.690 | 0 / 00.639 | 0 / 00.726 | 0 / 00.800 | 0 / 00.788 | 0 / 00.796 | 0 / 00.697 | 0 / 00.680 | 0 / 00.768 | 0 / 00.761 |
| KnowLine-9B-Gen2 Open · 9.0B | — | No answers at this threshold0.803 | 0 / 00.788 | 0 / 00.711 | 0 / 00.712 | 0 / 00.671 | 0 / 00.801 | 0 / 00.819 | 0 / 00.834 | 0 / 00.841 | 0 / 00.847 | 0 / 00.824 | 0 / 00.774 | 0 / 00.806 | 0 / 00.817 |
| Cerno 9B v1.2 Open · 9.0B | — | 244 / 100.780 | 24 / 00.806 | 9 / 00.677 | 1 / 10.561 | 8 / 00.679 | 6 / 00.797 | 11 / 00.751 | 31 / 00.843 | 17 / 00.747 | 22 / 20.806 | 13 / 00.812 | 31 / 20.720 | 33 / 30.836 | 38 / 20.776 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 207 / 30.804 | 18 / 10.778 | 5 / 00.684 | 0 / 00.606 | 7 / 00.731 | 7 / 10.745 | 8 / 00.759 | 29 / 10.839 | 10 / 00.829 | 18 / 00.822 | 12 / 00.853 | 33 / 00.824 | 24 / 00.814 | 36 / 00.820 |
| Adjudex 9B v5 Open · 9.0B | — | 197 / 10.806 | 18 / 00.815 | 4 / 00.701 | 0 / 00.729 | 6 / 00.684 | 8 / 00.764 | 9 / 00.797 | 31 / 00.867 | 12 / 10.814 | 21 / 00.820 | 16 / 00.840 | 23 / 00.813 | 21 / 00.812 | 28 / 00.806 |
| Elephant-1 Open · 9.0B | — | 182 / 10.803 | 14 / 00.821 | 5 / 00.711 | 0 / 00.486 | 6 / 00.684 | 6 / 00.823 | 10 / 00.730 | 28 / 00.833 | 10 / 10.785 | 15 / 00.868 | 14 / 00.879 | 25 / 00.778 | 24 / 00.834 | 25 / 00.774 |
| Winnow-12B Open · 12B | — | 20 / 10.743 | 5 / 00.742 | 1 / 00.638 | 0 / 00.686 | 0 / 00.670 | 1 / 00.642 | 0 / 10.743 | 1 / 00.787 | 1 / 00.742 | 5 / 00.731 | 3 / 00.785 | 1 / 00.746 | 1 / 00.758 | 1 / 00.774 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.819 Logic 50.6% | — | 66.3%64.2%–68.2% | a | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 74.96chosen-answer Brier 0.1632 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 198 answers, 197 right · 1 wrong · AUROC 0.806 Logic 46.4% | — | 62.1%60.0%–64.1% | b | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 72.28chosen-answer Brier 0.1752 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 183 answers, 182 right · 1 wrong · AUROC 0.803 Logic 46.4% | — | 61.3%59.3%–63.4% | bc | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 71.86chosen-answer Brier 0.1763 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.803 Logic 43.9% | — | 60.1%58.0%–62.1% | bcd | 1,300 / 2,163 | No answers at this threshold | 0.803 | 71.15chosen-answer Brier 0.1779 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 64 answers, 64 right · 0 wrong · AUROC 0.814 Logic 48.2% | — | 59.8%57.7%–61.9% | bcd | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 71.32chosen-answer Brier 0.1719 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 123 answers, 123 right · 0 wrong · AUROC 0.799 Logic 44.3% | — | 59.8%57.7%–61.8% | bcd | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 71.08chosen-answer Brier 0.1762 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 210 answers, 207 right · 3 wrong · AUROC 0.804 Logic 45.3% | — | 58.8%56.7%–60.9% | cde | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 69.67chosen-answer Brier 0.1946 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 49 answers, 49 right · 0 wrong · AUROC 0.781 Logic 46.6% | — | 58.4%56.3%–60.5% | cde | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 69.90chosen-answer Brier 0.1864 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 73 answers, 73 right · 0 wrong · AUROC 0.798 Logic 47.8% | — | 57.9%55.8%–60.0% | de | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 69.69chosen-answer Brier 0.1855 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 254 answers, 244 right · 10 wrong · AUROC 0.780 Logic 43.3% | — | 56.0%53.9%–58.1% | e | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 67.15chosen-answer Brier 0.2174 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 130 answers, 130 right · 0 wrong · AUROC 0.790 Logic 44.4% | — | 55.3%53.2%–57.4% | e | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 68.37chosen-answer Brier 0.1860 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.730 Logic 38.9% | — | 49.8%47.7%–51.9% | f | 1,077 / 2,163 | No answers at this threshold | 0.730 | 64.44chosen-answer Brier 0.2092 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 21 answers, 20 right · 1 wrong · AUROC 0.743 Logic 39.7% | — | 46.9%44.8%–49.0% | fg | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 63.00chosen-answer Brier 0.2087 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.735 Logic 39.3% | — | 44.3%42.3%–46.4% | g | 959 / 2,163 | No answers at this threshold | 0.735 | 61.87chosen-answer Brier 0.2060 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
MachinePulse Base Labs Perplexity Poindexter Labs Sthānika AI Nace.AI Fastino Turion.AI OmniJev Cloudflare vLLM Semantic Router Zefan Cai PelaAI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | 81.6%2,163 | 88.9%86.7% kept (1,876) | 94.2%73.7% kept (1,594) | 98.0%58.1% kept (1,256) | 100.0%38.7% kept (837) |
| Mammoth-1 Open · 27B | — | 74.0%2,163 | 85.3%79.1% kept (1,712) | 94.3%61.8% kept (1,336) | 98.5%45.7% kept (988) | 100.0%19.0% kept (412) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 69.8%2,163 | 83.0%73.5% kept (1,590) | 93.1%56.3% kept (1,217) | 98.2%38.7% kept (838) | 100.0%15.3% kept (331) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 69.3%2,163 | 82.5%73.5% kept (1,590) | 92.8%56.2% kept (1,215) | 97.7%38.8% kept (840) | 100.0%15.1% kept (326) |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 72.5%2,163 | 81.2%81.1% kept (1,754) | 90.0%64.0% kept (1,384) | 96.3%46.1% kept (998) | 100.0%14.9% kept (323) |
| Copernicus-31B Open · 31B | — | 77.3%2,163 | 91.3%73.4% kept (1,587) | 97.2%57.5% kept (1,244) | 99.4%39.0% kept (843) | 100.0%14.7% kept (318) |
| Sieve 31B v3 Open · 31B | — | 77.1%2,163 | 90.0%75.3% kept (1,628) | 95.9%59.6% kept (1,289) | 99.3%41.1% kept (888) | 100.0%9.9% kept (215) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 59.0%2,163 | 80.4%53.0% kept (1,147) | 92.2%35.0% kept (758) | 97.8%20.6% kept (445) | 100.0%8.0% kept (174) |
| GLiDE v1.2 Open · 27B | — | 69.9%2,163 | 81.1%75.9% kept (1,641) | 92.6%56.4% kept (1,219) | 99.1%34.4% kept (745) | 100.0%6.1% kept (131) |
| Turion Jev 31B Open · 31B | — | 69.0%2,163 | 84.7%68.7% kept (1,487) | 92.8%51.1% kept (1,106) | 99.8%26.0% kept (562) | 100.0%2.0% kept (43) |
| OneJev-27B Open · 27B | — | 53.8%2,163 | 81.8%39.5% kept (855) | 92.5%23.3% kept (504) | 98.3%10.6% kept (229) | 100.0%1.2% kept (26) |
| Clef Open · 27B | — | 51.1%2,163 | 65.2%62.2% kept (1,346) | 78.9%36.8% kept (796) | 90.8%17.6% kept (381) | 100.0%0.6% kept (12) |
| d3-ultra Open · 26B | — | 71.5%2,163 | 78.5%85.2% kept (1,842) | 86.2%68.6% kept (1,484) | 97.4%43.7% kept (946) | 100.0%0.5% kept (11) |
| Open-Jev-27B-v1.1 Open · 27B | — | 54.6%2,163 | 95.8%10.0% kept (216) | 100.0%1.2% kept (25) | none kept | none kept |
| Drex Max Open · 27B | — | 70.3%2,163 | 76.2%86.7% kept (1,875) | 83.3%72.2% kept (1,561) | 92.9%55.9% kept (1,209) | 98.6%33.7% kept (729) |
| KnowLine-31B-Gen2 Open · 31B | — | 80.3%2,163 | 90.0%80.7% kept (1,746) | 95.3%66.5% kept (1,438) | 99.3%48.1% kept (1,040) | 99.8%21.7% kept (470) |
| KnowLine-31B-Gen1.5 Open · 31B | — | 77.4%2,163 | 89.5%77.0% kept (1,665) | 95.3%61.3% kept (1,327) | 99.1%42.7% kept (923) | 99.8%21.5% kept (465) |
| GLiDE 27B Open · 27B | — | 72.0%2,163 | 82.1%78.7% kept (1,703) | 91.2%60.6% kept (1,311) | 98.7%39.5% kept (855) | 99.5%9.2% kept (199) |
| d3 Open · 26B | — | 70.4%2,163 | 85.3%71.8% kept (1,553) | 95.2%53.7% kept (1,161) | 99.3%34.0% kept (735) | 99.4%7.5% kept (163) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | 837 / 00.903 | 85 / 00.899 | 34 / 00.801 | 3 / 00.833 | 34 / 00.880 | 38 / 00.933 | 50 / 00.890 | 83 / 00.952 | 45 / 00.879 | 66 / 00.915 | 53 / 00.950 | 91 / 00.902 | 114 / 00.893 | 141 / 00.904 |
| Mammoth-1 Open · 27B | — | 412 / 00.895 | 27 / 00.885 | 9 / 00.766 | 1 / 00.788 | 11 / 00.821 | 14 / 00.876 | 30 / 00.897 | 41 / 00.909 | 17 / 00.893 | 47 / 00.930 | 31 / 00.904 | 53 / 00.900 | 59 / 00.896 | 72 / 00.903 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 331 / 00.869 | 22 / 00.861 | 6 / 00.818 | 1 / 00.806 | 8 / 00.772 | 10 / 00.823 | 16 / 00.819 | 34 / 00.961 | 34 / 00.901 | 43 / 00.903 | 33 / 00.932 | 32 / 00.840 | 47 / 00.875 | 45 / 00.863 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 326 / 00.873 | 23 / 00.850 | 9 / 00.842 | 1 / 00.571 | 6 / 00.827 | 14 / 00.799 | 18 / 00.854 | 28 / 00.927 | 36 / 00.869 | 43 / 00.887 | 31 / 00.946 | 29 / 00.851 | 41 / 00.885 | 47 / 00.884 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 323 / 00.856 | 26 / 00.828 | 8 / 00.776 | 0 / 00.577 | 8 / 00.785 | 15 / 00.845 | 17 / 00.790 | 36 / 00.935 | 27 / 00.839 | 25 / 00.900 | 22 / 00.933 | 35 / 00.845 | 49 / 00.871 | 55 / 00.834 |
| Copernicus-31B Open · 31B | — | 318 / 00.899 | 27 / 00.901 | 13 / 00.865 | 1 / 00.800 | 9 / 00.881 | 17 / 00.892 | 20 / 00.881 | 42 / 00.980 | 12 / 00.884 | 25 / 00.876 | 19 / 00.937 | 29 / 00.882 | 48 / 00.914 | 56 / 00.875 |
| Sieve 31B v3 Open · 31B | — | 215 / 00.891 | 24 / 00.871 | 6 / 00.848 | 0 / 00.903 | 7 / 00.856 | 8 / 00.898 | 10 / 00.863 | 26 / 00.956 | 15 / 00.918 | 22 / 00.907 | 14 / 00.939 | 24 / 00.895 | 31 / 00.879 | 28 / 00.848 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 174 / 00.813 | 8 / 00.819 | 6 / 00.817 | 0 / 00.533 | 0 / 00.740 | 5 / 00.798 | 13 / 00.841 | 20 / 00.905 | 9 / 00.768 | 16 / 00.837 | 14 / 00.766 | 22 / 00.762 | 35 / 00.836 | 26 / 00.803 |
| GLiDE v1.2 Open · 27B | — | 131 / 00.862 | 12 / 00.857 | 3 / 00.740 | 0 / 00.528 | 5 / 00.777 | 4 / 00.834 | 5 / 00.838 | 20 / 00.963 | 14 / 00.865 | 22 / 00.912 | 7 / 00.948 | 15 / 00.873 | 14 / 00.849 | 10 / 00.877 |
| Turion Jev 31B Open · 31B | — | 43 / 00.864 | 5 / 00.868 | 1 / 00.770 | 0 / 00.757 | 1 / 00.809 | 0 / 00.904 | 3 / 00.817 | 3 / 00.926 | 1 / 00.854 | 6 / 00.823 | 1 / 00.940 | 8 / 00.853 | 5 / 00.866 | 9 / 00.852 |
| OneJev-27B Open · 27B | — | 26 / 00.783 | 2 / 00.768 | 0 / 00.645 | 0 / 00.806 | 1 / 00.750 | 0 / 00.767 | 1 / 00.778 | 2 / 00.884 | 1 / 00.758 | 1 / 00.811 | 3 / 00.787 | 4 / 00.757 | 6 / 00.785 | 5 / 00.789 |
| Clef Open · 27B | — | 12 / 00.767 | 2 / 00.751 | 1 / 00.708 | 0 / 00.875 | 2 / 00.723 | 0 / 00.720 | 0 / 00.798 | 3 / 00.778 | 0 / 00.793 | 1 / 00.744 | 0 / 00.878 | 2 / 00.748 | 0 / 00.802 | 1 / 00.774 |
| d3-ultra Open · 26B | — | 11 / 00.857 | 1 / 00.842 | 1 / 00.778 | 0 / 00.764 | 0 / 00.820 | 0 / 00.731 | 0 / 00.827 | 3 / 00.942 | 0 / 00.819 | 1 / 00.901 | 2 / 00.848 | 1 / 00.871 | 1 / 00.850 | 1 / 00.874 |
| Open-Jev-27B-v1.1 Open · 27B | — | No answers at this threshold0.764 | 0 / 00.816 | 0 / 00.607 | 0 / 00.625 | 0 / 00.674 | 0 / 00.731 | 0 / 00.751 | 0 / 00.762 | 0 / 00.745 | 0 / 00.847 | 0 / 00.818 | 0 / 00.776 | 0 / 00.727 | 0 / 00.749 |
| Drex Max Open · 27B | — | 719 / 100.861 | 67 / 10.834 | 25 / 00.803 | 3 / 00.750 | 32 / 00.799 | 28 / 00.867 | 36 / 40.794 | 63 / 00.923 | 42 / 20.877 | 75 / 00.894 | 44 / 00.908 | 83 / 00.876 | 96 / 20.855 | 125 / 10.847 |
| KnowLine-31B-Gen2 Open · 31B | — | 469 / 10.898 | 38 / 00.916 | 15 / 00.862 | 2 / 00.742 | 14 / 00.847 | 24 / 00.902 | 37 / 00.921 | 70 / 00.956 | 12 / 00.871 | 28 / 00.882 | 28 / 00.916 | 56 / 10.863 | 68 / 00.908 | 77 / 00.886 |
| KnowLine-31B-Gen1.5 Open · 31B | — | 464 / 10.895 | 38 / 00.874 | 15 / 00.850 | 1 / 00.750 | 14 / 00.881 | 22 / 00.903 | 32 / 00.900 | 60 / 00.956 | 18 / 00.912 | 35 / 00.872 | 33 / 00.904 | 55 / 10.885 | 64 / 00.914 | 77 / 00.888 |
| GLiDE 27B Open · 27B | — | 198 / 10.864 | 17 / 10.858 | 6 / 00.787 | 0 / 00.569 | 8 / 00.761 | 5 / 00.846 | 7 / 00.870 | 22 / 00.948 | 20 / 00.938 | 36 / 00.873 | 18 / 00.905 | 21 / 00.861 | 20 / 00.848 | 18 / 00.875 |
| d3 Open · 26B | — | 162 / 10.878 | 13 / 10.867 | 5 / 00.791 | 1 / 00.806 | 6 / 00.811 | 7 / 00.907 | 8 / 00.847 | 21 / 00.955 | 15 / 00.904 | 25 / 00.905 | 13 / 00.893 | 21 / 00.886 | 13 / 00.875 | 14 / 00.872 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 837 answers, 837 right · 0 wrong · AUROC 0.903 Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 85.99chosen-answer Brier 0.0958 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 470 answers, 469 right · 1 wrong · AUROC 0.898 Logic 58.9% | — | 80.3%78.5%–81.9% | a | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 85.05chosen-answer Brier 0.1016 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 465 answers, 464 right · 1 wrong · AUROC 0.895 Logic 59.0% | — | 77.4%75.6%–79.1% | b | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 83.18chosen-answer Brier 0.1103 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 318 answers, 318 right · 0 wrong · AUROC 0.899 Logic 58.7% | — | 77.3%75.5%–79.1% | b | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 83.06chosen-answer Brier 0.1122 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 215 answers, 215 right · 0 wrong · AUROC 0.891 Logic 57.9% | — | 77.1%75.3%–78.8% | bc | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 82.89chosen-answer Brier 0.1133 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 412 answers, 412 right · 0 wrong · AUROC 0.895 Logic 54.5% | — | 74.0%72.1%–75.8% | cd | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 81.16chosen-answer Brier 0.1164 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 323 answers, 323 right · 0 wrong · AUROC 0.856 Logic 54.8% | — | 72.5%70.6%–74.3% | de | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 79.32chosen-answer Brier 0.1385 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 199 answers, 198 right · 1 wrong · AUROC 0.864 Logic 58.4% | — | 72.0%70.1%–73.8% | def | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 79.22chosen-answer Brier 0.1354 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 11 answers, 11 right · 0 wrong · AUROC 0.857 Logic 54.6% | — | 71.5%69.6%–73.4% | def | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 78.39chosen-answer Brier 0.1474 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 163 answers, 162 right · 1 wrong · AUROC 0.878 Logic 54.5% | — | 70.4%68.5%–72.3% | ef | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 78.71chosen-answer Brier 0.1299 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 729 answers, 719 right · 10 wrong · AUROC 0.861 Logic 53.1% | — | 70.3%68.4%–72.2% | ef | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 77.10chosen-answer Brier 0.1611 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 131 answers, 131 right · 0 wrong · AUROC 0.862 Logic 55.2% | — | 69.9%67.9%–71.8% | ef | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 77.95chosen-answer Brier 0.1396 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 331 answers, 331 right · 0 wrong · AUROC 0.869 Logic 53.3% | — | 69.8%67.8%–71.7% | f | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 78.16chosen-answer Brier 0.1345 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 326 answers, 326 right · 0 wrong · AUROC 0.873 Logic 52.5% | with web | 69.3%67.3%–71.2% | f | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 77.94chosen-answer Brier 0.1342 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 43 answers, 43 right · 0 wrong · AUROC 0.864 Logic 52.3% | — | 69.0%67.0%–70.9% | f | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 77.52chosen-answer Brier 0.1399 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 174 answers, 174 right · 0 wrong · AUROC 0.813 Logic 41.9% | — | 59.0%56.9%–61.0% | g | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 70.98chosen-answer Brier 0.1703 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.764 Logic 42.6% | — | 54.6%52.4%–56.6% | h | 1,180 / 2,163 | No answers at this threshold | 0.764 | 64.53chosen-answer Brier 0.2549 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 26 answers, 26 right · 0 wrong · AUROC 0.783 Logic 40.6% | — | 53.8%51.7%–55.9% | h | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 67.54chosen-answer Brier 0.1869 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 12 answers, 12 right · 0 wrong · AUROC 0.767 Logic 40.8% | — | 51.1%49.0%–53.2% | h | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 65.20chosen-answer Brier 0.2070 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
REAX Logits Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 78.0%2,163 | 90.9%76.9% kept (1,664) | 97.5%58.5% kept (1,265) | 99.9%40.5% kept (877) | 100.0%8.2% kept (178) |
| Logits Sigma-1 Duo Open · 57B | — | 75.2%2,163 | 89.2%73.5% kept (1,589) | 98.3%52.4% kept (1,134) | 100.0%26.7% kept (578) | 100.0%0.7% kept (15) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 178 / 00.903 | 11 / 00.888 | 7 / 00.826 | 1 / 00.818 | 7 / 00.869 | 9 / 00.886 | 11 / 00.917 | 30 / 00.955 | 10 / 00.946 | 14 / 00.881 | 7 / 00.935 | 22 / 00.887 | 25 / 00.913 | 24 / 00.900 |
| Logits Sigma-1 Duo Open · 57B | — | 15 / 00.899 | 2 / 00.900 | 0 / 00.841 | 0 / 00.667 | 0 / 00.847 | 1 / 00.888 | 0 / 00.872 | 2 / 00.950 | 0 / 00.919 | 1 / 00.910 | 1 / 00.939 | 4 / 00.895 | 1 / 00.871 | 3 / 00.911 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 178 answers, 178 right · 0 wrong · AUROC 0.903 Logic 59.3% | — | 78.0%76.2%–79.7% | a | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 83.57chosen-answer Brier 0.1084 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 15 answers, 15 right · 0 wrong · AUROC 0.899 Logic 57.8% | — | 75.2%73.4%–77.0% | b | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 81.60chosen-answer Brier 0.1201 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Nace.AI TypeSafe Google Liquid AI Fastino Microsoft Celeris Subquadratic Inception Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Drex Max v1.1 Closed · Efficient (provisional) | — | 70.1%2,163 | 79.7%79.8% kept (1,725) | 89.3%62.5% kept (1,352) | 97.3%43.4% kept (938) | 100.0%14.6% kept (315) |
| Jev 1.13 Closed · Efficient | — | 64.1%2,163 | 80.3%65.2% kept (1,410) | 91.3%42.7% kept (924) | 98.3%25.1% kept (543) | 100.0%11.5% kept (248) |
| Gemini Decision 1 20261111 Closed · Efficient | — | 71.3%2,163 | 85.3%72.1% kept (1,560) | 92.7%55.2% kept (1,194) | 98.2%36.3% kept (785) | 100.0%9.9% kept (214) |
| d1-ultra Closed · Efficient | — | 74.1%2,163 | 87.8%73.4% kept (1,588) | 94.0%59.0% kept (1,276) | 98.5%40.8% kept (882) | 100.0%9.8% kept (213) |
| Jev 1.14 Closed · Efficient | — | 65.1%2,163 | 82.2%61.7% kept (1,335) | 92.8%40.5% kept (876) | 98.3%24.0% kept (519) | 100.0%9.3% kept (202) |
| d1.1 Closed · Efficient | — | 65.2%2,163 | 82.7%61.1% kept (1,322) | 92.6%41.5% kept (897) | 97.6%24.9% kept (538) | 100.0%8.5% kept (183) |
| GLiDE API Closed · Efficient | — | 70.8%2,163 | 81.6%77.6% kept (1,678) | 92.0%58.2% kept (1,258) | 98.6%37.4% kept (808) | 100.0%7.4% kept (161) |
| d1 Closed · Efficient | — | 55.6%2,163 | 73.6%55.1% kept (1,192) | 85.9%32.1% kept (694) | 95.4%17.0% kept (368) | 100.0%7.0% kept (152) |
| Microsoft-Decision-1 Closed · Efficient | — | 59.4%2,163 | 73.4%63.9% kept (1,382) | 88.5%36.2% kept (782) | 98.0%13.8% kept (299) | 100.0%1.2% kept (27) |
| celeris-1-decision Closed · Efficient | — | 70.0%2,163 | 80.9%77.1% kept (1,667) | 90.9%58.3% kept (1,262) | 96.5%40.4% kept (874) | none kept |
| celeris-1.1-decision Closed · Efficient | — | 69.9%2,163 | 80.9%77.2% kept (1,669) | 90.8%58.3% kept (1,262) | 96.7%40.5% kept (876) | none kept |
| SubQ Snap 1.1 Closed · Efficient | — | 68.6%2,163 | 76.4%82.0% kept (1,774) | 84.1%66.4% kept (1,436) | 93.7%46.3% kept (1,002) | 99.7%16.7% kept (361) |
| Mercury Decide Closed · Efficient | — | 53.5%2,163 | 65.1%71.2% kept (1,539) | 75.3%49.8% kept (1,077) | 85.3%32.0% kept (692) | 96.8%13.0% kept (281) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Drex Max v1.1 Closed · Efficient (provisional) | — | 315 / 00.867 | 23 / 00.839 | 8 / 00.850 | 1 / 00.471 | 14 / 00.799 | 9 / 00.861 | 15 / 00.817 | 28 / 00.903 | 31 / 00.840 | 47 / 00.910 | 30 / 00.919 | 36 / 00.866 | 39 / 00.853 | 34 / 00.893 |
| Jev 1.13 Closed · Efficient | — | 248 / 00.834 | 17 / 00.836 | 8 / 00.756 | 0 / 00.671 | 9 / 00.761 | 8 / 00.797 | 9 / 00.835 | 23 / 00.894 | 20 / 00.801 | 30 / 00.868 | 17 / 00.831 | 24 / 00.821 | 41 / 00.850 | 42 / 00.818 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 214 / 00.859 | 15 / 00.862 | 6 / 00.794 | 2 / 00.729 | 0 / 00.789 | 5 / 00.854 | 10 / 00.851 | 18 / 00.882 | 13 / 00.797 | 23 / 00.833 | 15 / 00.799 | 31 / 00.859 | 38 / 00.907 | 38 / 00.858 |
| d1-ultra Closed · Efficient | — | 213 / 00.876 | 23 / 00.865 | 8 / 00.755 | 0 / 00.650 | 5 / 00.813 | 9 / 00.896 | 13 / 00.887 | 12 / 00.910 | 13 / 00.941 | 18 / 00.902 | 13 / 00.893 | 20 / 00.845 | 30 / 00.885 | 49 / 00.888 |
| Jev 1.14 Closed · Efficient | — | 202 / 00.829 | 16 / 00.854 | 7 / 00.791 | 0 / 00.686 | 8 / 00.700 | 5 / 00.778 | 8 / 00.810 | 18 / 00.910 | 14 / 00.788 | 25 / 00.862 | 18 / 00.826 | 21 / 00.781 | 27 / 00.849 | 35 / 00.834 |
| d1.1 Closed · Efficient | — | 183 / 00.829 | 13 / 00.811 | 5 / 00.768 | 0 / 00.714 | 4 / 00.721 | 3 / 00.825 | 7 / 00.796 | 16 / 00.848 | 14 / 00.854 | 19 / 00.870 | 15 / 00.849 | 23 / 00.815 | 30 / 00.798 | 34 / 00.835 |
| GLiDE API Closed · Efficient | — | 161 / 00.865 | 17 / 00.874 | 5 / 00.746 | 1 / 00.625 | 11 / 00.778 | 4 / 00.853 | 4 / 00.840 | 20 / 00.987 | 13 / 00.872 | 29 / 00.885 | 11 / 00.924 | 15 / 00.896 | 17 / 00.811 | 14 / 00.881 |
| d1 Closed · Efficient | — | 152 / 00.767 | 14 / 00.765 | 5 / 00.653 | 0 / 00.530 | 4 / 00.700 | 3 / 00.789 | 6 / 00.752 | 9 / 00.761 | 11 / 00.726 | 21 / 00.802 | 11 / 00.738 | 20 / 00.769 | 23 / 00.777 | 25 / 00.784 |
| Microsoft-Decision-1 Closed · Efficient | — | 27 / 00.787 | 5 / 00.789 | 0 / 00.696 | 0 / 00.615 | 1 / 00.670 | 0 / 00.747 | 0 / 00.750 | 4 / 00.848 | 4 / 00.806 | 5 / 00.837 | 1 / 00.801 | 2 / 00.768 | 3 / 00.824 | 2 / 00.794 |
| celeris-1-decision Closed · Efficient | — | No answers at this threshold0.858 | 0 / 00.857 | 0 / 00.787 | 0 / 00.807 | 0 / 00.794 | 0 / 00.884 | 0 / 00.896 | 0 / 00.952 | 0 / 00.840 | 0 / 00.881 | 0 / 00.836 | 0 / 00.838 | 0 / 00.869 | 0 / 00.849 |
| celeris-1.1-decision Closed · Efficient | — | No answers at this threshold0.859 | 0 / 00.863 | 0 / 00.790 | 0 / 00.875 | 0 / 00.790 | 0 / 00.863 | 0 / 00.896 | 0 / 00.944 | 0 / 00.853 | 0 / 00.877 | 0 / 00.812 | 0 / 00.852 | 0 / 00.867 | 0 / 00.855 |
| SubQ Snap 1.1 Closed · Efficient | — | 360 / 10.839 | 28 / 00.819 | 12 / 00.784 | 1 / 00.722 | 15 / 00.780 | 10 / 00.808 | 19 / 00.754 | 47 / 00.929 | 35 / 00.829 | 31 / 00.832 | 28 / 00.931 | 37 / 10.802 | 45 / 00.871 | 52 / 00.848 |
| Mercury Decide Closed · Efficient | — | 272 / 90.789 | 25 / 00.811 | 9 / 00.692 | 1 / 00.729 | 8 / 00.748 | 6 / 00.687 | 11 / 30.718 | 26 / 10.859 | 23 / 20.801 | 31 / 10.808 | 18 / 00.821 | 22 / 10.751 | 44 / 10.785 | 48 / 00.822 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 213 answers, 213 right · 0 wrong · AUROC 0.876 Logic 54.9% | — | 74.1%72.2%–75.9% | a | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 80.88chosen-answer Brier 0.1235 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 214 answers, 214 right · 0 wrong · AUROC 0.859 Logic 50.0% | — | 71.3%69.4%–73.2% | ab | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 78.84chosen-answer Brier 0.1366 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 161 answers, 161 right · 0 wrong · AUROC 0.865 Logic 54.5% | — | 70.8%68.8%–72.7% | b | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 78.56chosen-answer Brier 0.1367 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 315 answers, 315 right · 0 wrong · AUROC 0.867 Logic 54.8% | — | 70.1%68.1%–72.0% | b | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 78.01chosen-answer Brier 0.1407 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.858 Logic 47.3% | — | 70.0%68.0%–71.9% | b | 1,514 / 2,163 | No answers at this threshold | 0.858 | 77.96chosen-answer Brier 0.1407 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.859 Logic 47.3% | — | 69.9%67.9%–71.8% | b | 1,511 / 2,163 | No answers at this threshold | 0.859 | 77.91chosen-answer Brier 0.1403 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 361 answers, 360 right · 1 wrong · AUROC 0.839 Logic 52.4% | — | 68.6%66.6%–70.5% | b | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 76.15chosen-answer Brier 0.1626 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 183 answers, 183 right · 0 wrong · AUROC 0.829 Logic 46.0% | — | 65.2%63.2%–67.2% | c | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 74.62chosen-answer Brier 0.1595 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 202 answers, 202 right · 0 wrong · AUROC 0.829 Logic 48.2% | — | 65.1%63.1%–67.1% | c | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 74.56chosen-answer Brier 0.1602 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 248 answers, 248 right · 0 wrong · AUROC 0.834 Logic 46.3% | — | 64.1%62.0%–66.1% | c | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 74.08chosen-answer Brier 0.1592 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 27 answers, 27 right · 0 wrong · AUROC 0.787 Logic 44.4% | — | 59.4%57.3%–61.4% | d | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 70.49chosen-answer Brier 0.1839 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 152 answers, 152 right · 0 wrong · AUROC 0.767 Logic 40.8% | — | 55.6%53.5%–57.7% | e | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 68.16chosen-answer Brier 0.1930 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 281 answers, 272 right · 9 wrong · AUROC 0.789 Logic 45.1% | — | 53.5%51.4%–55.6% | e | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 66.02chosen-answer Brier 0.2146 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
No workhorse decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
No flagship decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Liquid AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | 65.5%2,163 | 83.5%60.9% kept (1,318) | 94.4%41.3% kept (894) | 99.2%23.7% kept (512) | 100.0%6.5% kept (141) |
| d1.2-ultra Closed · Undisclosed | — | 69.9%2,163 | 87.4%66.6% kept (1,441) | 94.8%48.6% kept (1,052) | 99.7%30.5% kept (660) | 100.0%5.3% kept (115) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Basic conceptskept R / W · AUROC | Digits and place valuekept R / W · AUROC | Number baseskept R / W · AUROC | Division and divisibilitykept R / W · AUROC | Prime factorisation and GCD/LCMkept R / W · AUROC | Rational numberskept R / W · AUROC | Decimalskept R / W · AUROC | Orderingkept R / W · AUROC | Simple inequalitieskept R / W · AUROC | Absolute valuekept R / W · AUROC | Exponentskept R / W · AUROC | Radicalskept R / W · AUROC | Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | 141 / 00.838 | 8 / 00.833 | 5 / 00.800 | 0 / 00.557 | 3 / 00.708 | 3 / 00.840 | 9 / 00.794 | 20 / 00.896 | 8 / 00.862 | 12 / 00.843 | 10 / 00.797 | 24 / 00.843 | 20 / 00.838 | 19 / 00.845 |
| d1.2-ultra Closed · Undisclosed | — | 115 / 00.876 | 6 / 00.891 | 5 / 00.839 | 1 / 00.889 | 4 / 00.774 | 4 / 00.851 | 5 / 00.801 | 18 / 00.960 | 10 / 00.855 | 8 / 00.912 | 10 / 00.902 | 15 / 00.859 | 11 / 00.855 | 18 / 00.868 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 115 answers, 115 right · 0 wrong · AUROC 0.876 Logic 51.6% | — | 69.9%67.9%–71.8% | a | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 78.23chosen-answer Brier 0.1340 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 141 answers, 141 right · 0 wrong · AUROC 0.838 Logic 49.2% | — | 65.5%63.5%–67.5% | b | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 74.98chosen-answer Brier 0.1556 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC); Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)); Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC); Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test); GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.2% | — | 99.7%99.4%–99.9% | abc | 2,157 / 2,163 | Not measured | Not measured | — | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.5% | — | 99.7%99.3%–99.8% | abd | 2,156 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Logic 96.2% | — | 99.6%99.3%–99.8% | abcdef | 2,155 / 2,163 | Not measured | Not measured | — | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Logic 96.7% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Logic 96.5% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Logic 95.4% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Logic 96.7% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Logic 96.1% | — | 99.6%99.2%–99.8% | abcdef | 2,154 / 2,163 | Not measured | Not measured | — | 85.7%answered 99% | 4,400 ms | 6,680 ms | — |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Logic 95.5% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Logic running | — | 99.5%99.2%–99.7% | abcdefg | 2,153 / 2,163 | Not measured | Not measured | — | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Logic 95.6% | — | 99.5%99.2%–99.7% | abcdef | 2,153 / 2,163 | Not measured | Not measured | — | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Logic 94.0% | — | 99.5%99.1%–99.7% | abcdefgh | 2,152 / 2,163 | Not measured | Not measured | — | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Logic 95.1% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | — | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Logic 92.4% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B | — | 99.4%99.0%–99.6% | abcdefgh | 2,150 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Logic running | — | 99.3%98.9%–99.6% | abcdefghij | 2,148 / 2,163 | Not measured | Not measured | — | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | abcdefghi | 2,147 / 2,163seed 0 run | Not measured | Not measured | — | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | — | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Logic 91.6% | — | 99.2%98.7%–99.5% | abcdefghi | 2,146 / 2,163 | Not measured | Not measured | — | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | — | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Logic 86.7% | — | 99.0%98.5%–99.4% | bcdefghij | 2,142 / 2,163 | Not measured | Not measured | — | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abcdefghij | 2,142 / 2,163seed 0 run | Not measured | Not measured | — | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B | — | 98.9%98.8%–99.1% min–max of 3 seeds | cefghijk | 2,139 / 2,163seed 0 run | Not measured | Not measured | — | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Logic 87.7% | — | 98.8%98.3%–99.2% | dfghijkl | 2,138 / 2,163 | Not measured | Not measured | — | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Logic 93.2% | — | 98.8%98.2%–99.2% | fghijkl | 2,137 / 2,163 | Not measured | Not measured | — | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Logic 85.5% | — | 98.7%98.1%–99.1% | ghijkl | 2,135 / 2,163 | Not measured | Not measured | — | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | efghijkl | 2,136 / 2,163seed 0 run | Not measured | Not measured | — | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% | — | 98.5%97.9%–99.0% | hijkl | 2,131 / 2,163 | Not measured | Not measured | — | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Logic 90.1% | — | 98.5%97.9%–98.9% | hijklm | 2,130 / 2,163 | Not measured | Not measured | — | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 82.6% | — | 98.2%97.6%–98.7% | ijklm | 2,125 / 2,163 | Not measured | Not measured | — | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Logic 94.0% | — | 98.2%97.6%–98.7% | ijklmn | 2,125 / 2,163 | Not measured | Not measured | — | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | ijklmn | 2,126 / 2,163seed 0 run | Not measured | Not measured | — | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 83.2% | — | 98.1%97.4%–98.6% | jklmn | 2,122 / 2,163 | Not measured | Not measured | — | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% | — | 97.8%97.1%–98.4% | klmno | 2,116 / 2,163 | Not measured | Not measured | — | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | lmnop | 2,111 / 2,163seed 0 run | Not measured | Not measured | — | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% | — | 97.0%96.2%–97.6% | mnop | 2,098 / 2,163 | Not measured | Not measured | — | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 74.6% | — | 96.9%96.1%–97.6% | mnop | 2,097 / 2,163 | Not measured | Not measured | — | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 72.9% | — | 96.8%95.9%–97.4% | nopq | 2,093 / 2,163 | Not measured | Not measured | — | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Logic 60.2% | — | 96.3%95.4%–97.0% | opqr | 2,082 / 2,163 | Not measured | Not measured | — | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Logic 65.3% | — | 95.8%94.9%–96.6% | pqrs | 2,072 / 2,163 | Not measured | Not measured | — | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B | — | 94.7%93.7%–95.6% | qrs | 2,049 / 2,163 | Not measured | Not measured | — | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Logic 64.8% | — | 94.6%93.6%–95.5% | rs | 2,047 / 2,163 | Not measured | Not measured | — | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Logic 75.4% | — | 94.0%92.9%–94.9% | rs | 2,033 / 2,163 | Not measured | Not measured | — | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Logic 58.0% | — | 93.7%92.6%–94.6% | s | 2,026 / 2,163 | Not measured | Not measured | — | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Logic 54.2% | — | 90.8%89.5%–91.9% | t | 1,963 / 2,163 | Not measured | Not measured | — | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 65.1% | — | 89.6%88.2%–90.8% | t | 1,937 / 2,163 | Not measured | Not measured | — | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Logic 92.4% | — | 99.4%99.0%–99.7% | a | 2,151 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | ab | 2,147 / 2,163seed 0 run | Not measured | Not measured | — | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Logic 91.6% | — | 99.2%98.7%–99.5% | ab | 2,146 / 2,163 | Not measured | Not measured | — | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abc | 2,142 / 2,163seed 0 run | Not measured | Not measured | — | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B | — | 98.9%98.8%–99.1% min–max of 3 seeds | abc | 2,139 / 2,163seed 0 run | Not measured | Not measured | — | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Logic 85.5% | — | 98.7%98.1%–99.1% | bcd | 2,135 / 2,163 | Not measured | Not measured | — | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Logic 90.1% | — | 98.5%97.9%–98.9% | bcde | 2,130 / 2,163 | Not measured | Not measured | — | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | cdef | 2,126 / 2,163seed 0 run | Not measured | Not measured | — | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% | — | 97.8%97.1%–98.4% | defg | 2,116 / 2,163 | Not measured | Not measured | — | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | efg | 2,111 / 2,163seed 0 run | Not measured | Not measured | — | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% | — | 97.0%96.2%–97.6% | fg | 2,098 / 2,163 | Not measured | Not measured | — | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 74.6% | — | 96.9%96.1%–97.6% | g | 2,097 / 2,163 | Not measured | Not measured | — | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 72.9% | — | 96.8%95.9%–97.4% | g | 2,093 / 2,163 | Not measured | Not measured | — | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B | — | 94.7%93.7%–95.6% | h | 2,049 / 2,163 | Not measured | Not measured | — | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Logic 64.8% | — | 94.6%93.6%–95.5% | h | 2,047 / 2,163 | Not measured | Not measured | — | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Logic 75.4% | — | 94.0%92.9%–94.9% | h | 2,033 / 2,163 | Not measured | Not measured | — | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 65.1% | — | 89.6%88.2%–90.8% | i | 1,937 / 2,163 | Not measured | Not measured | — | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Logic 87.7% | — | 98.8%98.3%–99.2% | a | 2,138 / 2,163 | Not measured | Not measured | — | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | a | 2,136 / 2,163seed 0 run | Not measured | Not measured | — | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% | — | 98.5%97.9%–99.0% | a | 2,131 / 2,163 | Not measured | Not measured | — | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 82.6% | — | 98.2%97.6%–98.7% | a | 2,125 / 2,163 | Not measured | Not measured | — | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 83.2% | — | 98.1%97.4%–98.6% | a | 2,122 / 2,163 | Not measured | Not measured | — | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Logic 60.2% | — | 96.3%95.4%–97.0% | b | 2,082 / 2,163 | Not measured | Not measured | — | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Logic 65.3% | — | 95.8%94.9%–96.6% | b | 2,072 / 2,163 | Not measured | Not measured | — | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Logic 54.2% | — | 90.8%89.5%–91.9% | c | 1,963 / 2,163 | Not measured | Not measured | — | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B | — | 99.4%99.0%–99.6% | a | 2,150 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Logic 94.0% | — | 99.2%98.7%–99.5% | a | 2,146 / 2,163 | Not measured | Not measured | — | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Logic 86.7% | — | 99.0%98.5%–99.4% | a | 2,142 / 2,163 | Not measured | Not measured | — | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Logic 96.2% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | — | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Logic 95.4% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Logic 95.6% | — | 99.5%99.2%–99.7% | a | 2,153 / 2,163 | Not measured | Not measured | — | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Logic 95.1% | — | 99.4%99.0%–99.7% | ab | 2,151 / 2,163 | Not measured | Not measured | — | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Logic running | — | 99.3%98.9%–99.6% | ab | 2,148 / 2,163 | Not measured | Not measured | — | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Logic 93.2% | — | 98.8%98.2%–99.2% | bc | 2,137 / 2,163 | Not measured | Not measured | — | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Logic 94.0% | — | 98.2%97.6%–98.7% | c | 2,125 / 2,163 | Not measured | Not measured | — | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Logic 95.5% | — | 99.6%99.2%–99.8% | a | 2,154 / 2,163 | Not measured | Not measured | — | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Logic 58.0% | — | 93.7%92.6%–94.6% | b | 2,026 / 2,163 | Not measured | Not measured | — | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.5% | — | 99.7%99.3%–99.8% | a | 2,156 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.2% | — | 99.7%99.4%–99.9% | a | 2,157 / 2,163 | Not measured | Not measured | — | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Logic 96.7% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Logic 96.5% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Logic 96.7% | — | 99.6%99.2%–99.8% | ab | 2,154 / 2,163 | Not measured | Not measured | — | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Logic running | — | 99.5%99.2%–99.7% | ab | 2,153 / 2,163 | Not measured | Not measured | — | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Logic 94.0% | — | 99.5%99.1%–99.7% | ab | 2,152 / 2,163 | Not measured | Not measured | — | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Logic 94.0% | — | 99.2%98.7%–99.5% | b | 2,146 / 2,163 | Not measured | Not measured | — | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.