Logic · Confidence
Confidence on Logic
1,125 questions, asked in English. How to read this page · Method
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
PelaAI Poindexter Labs TypeSafe Perplexity Patrick-18 vLLM Semantic Router Base Labs Fastino REAX Liquid AI Google quincyqiang victorbennett Nace.AI EldanRing StartLux Microsoft tattrongvu Logits Turion.AI Cloudflare scar-ai OmniJev Blockbrain Labs Other Metask Lab Zefan Cai Strands Agents Celeris MachinePulse Inception Sthānika AI Subquadratic newsbang Valen-Team Apertix AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 59.0%1,125 | 78.3%54.6% kept (614) | 89.4%37.1% kept (417) | 96.1%20.7% kept (233) | 100.0%7.9% kept (89) |
| Copernicus-31B Open · 31B | — | Decision | 58.7%1,125 | 82.3%47.6% kept (536) | 91.9%30.6% kept (344) | 99.0%17.2% kept (193) | 100.0%5.9% kept (66) |
| Jev 1.13 Closed · Efficient | — | Decision | 46.3%1,125 | 68.4%44.7% kept (503) | 83.5%22.6% kept (254) | 94.4%11.2% kept (126) | 100.0%5.8% kept (65) |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 54.8%1,125 | 70.5%63.6% kept (715) | 82.3%41.7% kept (469) | 93.2%22.3% kept (251) | 100.0%5.5% kept (62) |
| Jev 1.14 Closed · Efficient | — | Decision | 48.2%1,125 | 72.6%39.9% kept (449) | 84.2%21.4% kept (241) | 92.6%10.8% kept (121) | 100.0%5.4% kept (61) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 45.3%1,125 | 58.8%58.8% kept (662) | 74.4%34.4% kept (387) | 91.7%15.0% kept (169) | 100.0%5.0% kept (56) |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 58.9%1,125 | 79.2%57.8% kept (650) | 89.1%38.2% kept (430) | 97.4%20.9% kept (235) | 100.0%4.8% kept (54) |
| d3 Open · 26B | — | Decision | 54.5%1,125 | 75.4%53.1% kept (597) | 90.5%31.7% kept (357) | 95.8%17.1% kept (192) | 100.0%4.8% kept (54) |
| Elephant-1 Open · 9.0B | — | Decision | 46.4%1,125 | 66.5%47.8% kept (538) | 84.2%27.0% kept (304) | 94.9%14.0% kept (157) | 100.0%4.7% kept (53) |
| GLiDE API Closed · Efficient | — | Decision | 54.5%1,125 | 72.9%58.5% kept (658) | 85.0%36.1% kept (406) | 96.8%16.7% kept (188) | 100.0%4.6% kept (52) |
| REAX 1.3 Open · 57B | — | Decision | 59.3%1,125 | 77.6%58.3% kept (656) | 90.8%34.7% kept (390) | 98.2%19.6% kept (221) | 100.0%4.5% kept (51) |
| d1.1 Closed · Efficient | — | Decision | 46.0%1,125 | 70.3%39.6% kept (445) | 84.6%22.5% kept (253) | 94.7%11.7% kept (132) | 100.0%4.3% kept (48) |
| GLiDE v1.2 Open · 27B | — | Decision | 55.2%1,125 | 71.2%59.6% kept (671) | 89.2%33.6% kept (378) | 97.7%15.7% kept (177) | 100.0%4.0% kept (45) |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 50.0%1,125 | 79.4%34.5% kept (388) | 91.5%18.9% kept (213) | 97.5%10.8% kept (121) | 100.0%3.8% kept (43) |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 40.4%1,125 | 54.7%49.4% kept (556) | 68.8%25.6% kept (288) | 83.1%12.1% kept (136) | 100.0%3.5% kept (39) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 43.1%1,125 | 65.9%36.3% kept (408) | 82.4%18.7% kept (210) | 92.3%10.4% kept (117) | 100.0%3.1% kept (35) |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 39.7%1,125 | 59.2%39.8% kept (448) | 73.6%20.9% kept (235) | 87.0%9.6% kept (108) | 100.0%3.1% kept (35) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 41.9%1,125 | 76.8%22.2% kept (250) | 82.2%12.0% kept (135) | 93.1%7.7% kept (87) | 100.0%2.9% kept (33) |
| GLiDE 9B Open · 7.9B | — | Decision | 47.8%1,125 | 66.4%47.4% kept (533) | 87.9%22.8% kept (256) | 97.5%10.7% kept (120) | 100.0%2.8% kept (32) |
| Winnow-12B Open · 12B | — | Decision | 39.7%1,125 | 57.3%45.2% kept (508) | 65.5%32.4% kept (365) | 74.5%19.2% kept (216) | 100.0%2.8% kept (32) |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 44.3%1,125 | 68.7%31.2% kept (351) | 85.3%16.4% kept (184) | 97.9%8.4% kept (95) | 100.0%2.7% kept (30) |
| d1.2 Closed · Undisclosed | — | Decision | 49.2%1,125 | 73.8%35.6% kept (400) | 87.5%19.9% kept (224) | 95.3%11.4% kept (128) | 100.0%2.5% kept (28) |
| d3-ultra Open · 26B | — | Decision | 54.6%1,125 | 62.7%75.7% kept (852) | 73.6%50.9% kept (573) | 93.0%25.3% kept (285) | 100.0%2.5% kept (28) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 45.9%1,125 | 68.4%36.8% kept (414) | 82.7%18.0% kept (202) | 97.8%8.3% kept (93) | 100.0%1.9% kept (21) |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 44.4%1,125 | 61.9%43.9% kept (494) | 78.7%19.6% kept (221) | 94.4%8.0% kept (90) | 100.0%1.8% kept (20) |
| d1-ultra Closed · Efficient | — | Decision | 54.9%1,125 | 76.4%48.5% kept (546) | 86.2%29.7% kept (334) | 96.6%15.6% kept (176) | 100.0%1.8% kept (20) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 44.7%1,125 | 63.9%47.6% kept (535) | 77.7%27.6% kept (310) | 92.2%11.5% kept (129) | 100.0%1.5% kept (17) |
| GLiDE 2B Open · 1.9B | — | Decision | 32.3%1,125 | 46.3%34.8% kept (391) | 54.7%15.1% kept (170) | 79.7%5.2% kept (59) | 100.0%1.2% kept (14) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 46.6%1,125 | 71.5%38.3% kept (431) | 84.6%19.6% kept (221) | 98.9%8.2% kept (92) | 100.0%1.2% kept (14) |
| d1.2-ultra Closed · Undisclosed | — | Decision | 51.6%1,125 | 77.1%44.7% kept (503) | 88.6%28.2% kept (317) | 96.9%14.4% kept (162) | 100.0%1.2% kept (13) |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 48.2%1,125 | 71.1%41.5% kept (467) | 86.4%22.9% kept (258) | 98.1%9.6% kept (108) | 100.0%0.9% kept (10) |
| GLiDE 0.8B Open · 0.8B | — | Decision | 29.2%1,125 | 46.1%24.1% kept (271) | 54.7%9.4% kept (106) | 62.5%3.6% kept (40) | 100.0%0.8% kept (9) |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 31.7%1,125 | 51.2%18.4% kept (207) | 69.1%7.2% kept (81) | 87.5%2.8% kept (32) | 100.0%0.7% kept (8) |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 57.8%1,125 | 75.4%57.5% kept (647) | 92.0%31.0% kept (349) | 97.6%14.6% kept (164) | 100.0%0.7% kept (8) |
| Turion Jev 31B Open · 31B | — | Decision | 52.3%1,125 | 81.5%37.9% kept (426) | 92.5%23.6% kept (266) | 97.7%11.6% kept (130) | 100.0%0.6% kept (7) |
| Clef Open · 27B | — | Decision | 40.8%1,125 | 53.0%54.5% kept (613) | 65.5%25.8% kept (290) | 85.3%9.1% kept (102) | 100.0%0.5% kept (6) |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 50.6%1,125 | 74.3%44.6% kept (502) | 90.5%25.3% kept (285) | 99.1%9.7% kept (109) | 100.0%0.5% kept (6) |
| vera-medium-omni Open · 4.2B | — | Decision | 44.1%1,125 | 72.7%29.3% kept (330) | 85.6%11.1% kept (125) | 100.0%4.9% kept (55) | 100.0%0.5% kept (6) |
| OneJev-27B Open · 27B | — | Decision | 40.6%1,125 | 74.4%18.4% kept (207) | 91.1%10.0% kept (112) | 100.0%4.6% kept (52) | 100.0%0.4% kept (4) |
| d1-2B Open · 1.9B | — | Decision | 30.7%1,125 | 52.0%15.2% kept (171) | 73.3%5.3% kept (60) | 100.0%1.7% kept (19) | 100.0%0.3% kept (3) |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 26.7%1,125 | 47.4%10.1% kept (114) | 63.9%3.2% kept (36) | 92.9%1.2% kept (14) | 100.0%0.2% kept (2) |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 27.0%1,125 | 36.3%15.9% kept (179) | 44.6%5.0% kept (56) | 83.3%0.5% kept (6) | 100.0%0.2% kept (2) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 30.8%1,125 | 59.7%13.2% kept (149) | 73.8%5.4% kept (61) | 100.0%2.0% kept (22) | 100.0%0.2% kept (2) |
| Clef-Flash Open · 9.0B | — | Decision | 38.9%1,125 | 52.4%40.9% kept (460) | 59.7%20.5% kept (231) | 77.9%6.0% kept (68) | none kept |
| Cygnet Open · 12B | — | Decision | 39.3%1,125 | 58.1%43.3% kept (487) | 71.2%18.2% kept (205) | 89.5%5.1% kept (57) | none kept |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 23.6%1,125 | 35.6%9.2% kept (104) | 30.8%2.3% kept (26) | 50.0%0.5% kept (6) | none kept |
| GLiDE 0.4B Open · 0.4B | — | Decision | 23.2%1,125 | 33.3%5.6% kept (63) | 40.0%1.8% kept (20) | 100.0%0.3% kept (3) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 43.9%1,125 | 70.9%32.1% kept (361) | 86.9%14.9% kept (168) | 100.0%4.1% kept (46) | none kept |
| Metask-Jev-4B Open · 4.5B | — | Decision | 32.1%1,125 | 66.9%12.4% kept (139) | 81.0%7.0% kept (79) | 90.5%1.9% kept (21) | none kept |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 42.6%1,125 | 95.5%5.9% kept (66) | 100.0%0.4% kept (4) | none kept | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 24.7%1,125 | 46.6%9.2% kept (103) | 65.9%3.6% kept (41) | 90.0%0.9% kept (10) | none kept |
| Strands Decider 2B Open · 2.0B | — | Decision | 22.4%1,125 | 38.8%6.0% kept (67) | 58.8%1.5% kept (17) | 100.0%0.2% kept (2) | none kept |
| celeris-1-decision Closed · Efficient | — | Decision | 47.3%1,125 | 61.9%53.7% kept (604) | 76.5%30.6% kept (344) | 90.5%15.9% kept (179) | none kept |
| celeris-1.1-decision Closed · Efficient | — | Decision | 47.3%1,125 | 62.9%52.7% kept (593) | 76.8%29.9% kept (336) | 90.8%16.4% kept (184) | none kept |
| Karpo Decision 31B Open · 31B | — | Decision | 60.7%1,125 | 75.3%65.2% kept (734) | 89.4%44.5% kept (501) | 96.3%28.8% kept (324) | 98.3%15.3% kept (172) |
| Drex Max Open · 27B | — | Decision | 53.1%1,125 | 63.7%72.1% kept (811) | 74.1%49.7% kept (559) | 86.5%30.8% kept (347) | 94.8%13.8% kept (155) |
| Mammoth-1 Open · 27B | — | Decision | 54.5%1,125 | 74.1%55.3% kept (622) | 86.3%36.3% kept (408) | 95.4%21.3% kept (240) | 98.1%9.6% kept (108) |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 54.8%1,125 | 71.8%58.0% kept (652) | 86.2%35.6% kept (400) | 95.5%19.9% kept (224) | 98.9%8.0% kept (90) |
| Mercury Decide Closed · Efficient | — | Decision | 45.1%1,125 | 61.0%53.6% kept (603) | 70.3%38.0% kept (428) | 79.9%22.1% kept (249) | 92.3%8.1% kept (91) |
| Sieve 31B v3 Open · 31B | — | Decision | 57.9%1,125 | 78.1%53.2% kept (599) | 89.3%35.7% kept (402) | 96.3%21.6% kept (243) | 97.5%7.2% kept (81) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 52.5%1,125 | 75.0%49.3% kept (555) | 88.5%29.3% kept (330) | 95.7%16.7% kept (188) | 98.7%7.0% kept (79) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 53.3%1,125 | 77.0%47.6% kept (535) | 90.7%28.8% kept (324) | 95.9%17.2% kept (193) | 97.5%7.1% kept (80) |
| GLiDE 27B Open · 27B | — | Decision | 58.4%1,125 | 76.8%60.3% kept (678) | 88.6%38.3% kept (431) | 97.7%18.9% kept (213) | 98.7%6.7% kept (75) |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 52.4%1,125 | 65.0%66.5% kept (748) | 77.9%42.2% kept (475) | 88.7%23.6% kept (266) | 97.2%6.4% kept (72) |
| d1 Closed · Efficient | — | Decision | 40.8%1,125 | 59.9%41.0% kept (461) | 74.2%21.7% kept (244) | 87.9%10.3% kept (116) | 98.2%5.0% kept (56) |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 46.4%1,125 | 68.9%44.4% kept (499) | 83.2%25.3% kept (285) | 95.8%12.7% kept (143) | 98.2%4.9% kept (55) |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 44.4%1,125 | 63.2%37.7% kept (424) | 77.9%17.7% kept (199) | 90.2%8.2% kept (92) | 94.6%3.3% kept (37) |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 43.3%1,125 | 56.1%59.4% kept (668) | 70.8%33.5% kept (377) | 89.6%14.6% kept (164) | 96.3%2.4% kept (27) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 42.3%1,125 | 55.4%51.6% kept (581) | 68.8%27.4% kept (308) | 86.4%11.1% kept (125) | 94.4%1.6% kept (18) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 89 / 00.804 | 25 / 00.765 | 14 / 00.764 | 50 / 00.904 |
| Copernicus-31B Open · 31B | — | Decision | 66 / 00.801 | 15 / 00.746 | 12 / 00.775 | 39 / 00.935 |
| Jev 1.13 Closed · Efficient | — | Decision | 65 / 00.764 | 7 / 00.708 | 5 / 00.714 | 53 / 00.875 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 62 / 00.803 | 22 / 00.752 | 5 / 00.764 | 35 / 00.878 |
| Jev 1.14 Closed · Efficient | — | Decision | 61 / 00.750 | 5 / 00.671 | 8 / 00.711 | 48 / 00.856 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 56 / 00.759 | 7 / 00.712 | 9 / 00.689 | 40 / 00.910 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 54 / 00.825 | 26 / 00.792 | 5 / 00.785 | 23 / 00.907 |
| d3 Open · 26B | — | Decision | 54 / 00.806 | 3 / 00.755 | 11 / 00.761 | 40 / 00.863 |
| Elephant-1 Open · 9.0B | — | Decision | 53 / 00.768 | 5 / 00.671 | 9 / 00.725 | 39 / 00.925 |
| GLiDE API Closed · Efficient | — | Decision | 52 / 00.814 | 7 / 00.777 | 7 / 00.765 | 38 / 00.866 |
| REAX 1.3 Open · 57B | — | Decision | 51 / 00.816 | 12 / 00.750 | 11 / 00.804 | 28 / 00.921 |
| d1.1 Closed · Efficient | — | Decision | 48 / 00.748 | 4 / 00.676 | 6 / 00.712 | 38 / 00.906 |
| GLiDE v1.2 Open · 27B | — | Decision | 45 / 00.793 | 3 / 00.732 | 6 / 00.739 | 36 / 00.906 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 43 / 00.768 | 5 / 00.766 | 0 / 00.699 | 38 / 00.885 |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 39 / 00.724 | 1 / 00.644 | 8 / 00.699 | 30 / 00.866 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 35 / 00.729 | 3 / 00.642 | 1 / 00.693 | 31 / 00.879 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 35 / 00.729 | 2 / 00.648 | 5 / 00.672 | 28 / 00.884 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 33 / 00.717 | 3 / 00.652 | 0 / 00.690 | 30 / 00.815 |
| GLiDE 9B Open · 7.9B | — | Decision | 32 / 00.749 | 3 / 00.637 | 4 / 00.729 | 25 / 00.911 |
| Winnow-12B Open · 12B | — | Decision | 32 / 00.727 | 1 / 00.608 | 6 / 00.670 | 25 / 00.835 |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 30 / 00.725 | 3 / 00.638 | 0 / 00.692 | 27 / 00.843 |
| d1.2 Closed · Undisclosed | — | Decision | 28 / 00.742 | 3 / 00.676 | 2 / 00.719 | 23 / 00.874 |
| d3-ultra Open · 26B | — | Decision | 28 / 00.778 | 9 / 00.707 | 10 / 00.761 | 9 / 00.853 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 21 / 00.720 | 1 / 00.633 | 2 / 00.672 | 18 / 00.853 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 20 / 00.712 | 1 / 00.628 | 2 / 00.683 | 17 / 00.823 |
| d1-ultra Closed · Efficient | — | Decision | 20 / 00.783 | 12 / 00.759 | 3 / 00.715 | 5 / 00.846 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 17 / 00.757 | 4 / 00.663 | 1 / 00.731 | 12 / 00.860 |
| GLiDE 2B Open · 1.9B | — | Decision | 14 / 00.632 | 1 / 00.525 | 1 / 00.637 | 12 / 00.787 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 14 / 00.762 | 0 / 00.675 | 3 / 00.713 | 11 / 00.854 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 13 / 00.800 | 4 / 00.741 | 4 / 00.772 | 5 / 00.837 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 10 / 00.753 | 2 / 00.663 | 5 / 00.704 | 3 / 00.866 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 9 / 00.617 | 0 / 00.557 | 0 / 00.608 | 9 / 00.700 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 8 / 00.601 | 0 / 00.501 | 0 / 00.571 | 8 / 00.756 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 8 / 00.796 | 1 / 00.738 | 1 / 00.764 | 6 / 00.880 |
| Turion Jev 31B Open · 31B | — | Decision | 7 / 00.782 | 1 / 00.702 | 0 / 00.756 | 6 / 00.875 |
| Clef Open · 27B | — | Decision | 6 / 00.697 | 2 / 00.607 | 0 / 00.704 | 4 / 00.846 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 6 / 00.774 | 1 / 00.701 | 1 / 00.711 | 4 / 00.937 |
| vera-medium-omni Open · 4.2B | — | Decision | 6 / 00.739 | 0 / 00.659 | 0 / 00.710 | 6 / 00.876 |
| OneJev-27B Open · 27B | — | Decision | 4 / 00.726 | 0 / 00.633 | 0 / 00.712 | 4 / 00.845 |
| d1-2B Open · 1.9B | — | Decision | 3 / 00.620 | 0 / 00.549 | 0 / 00.606 | 3 / 00.787 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 2 / 00.573 | 0 / 00.520 | 0 / 00.531 | 2 / 00.746 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 2 / 00.545 | 0 / 00.508 | 0 / 00.538 | 2 / 00.608 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 2 / 00.640 | 0 / 00.539 | 0 / 00.652 | 2 / 00.767 |
| Clef-Flash Open · 9.0B | — | Decision | No answers at this threshold0.659 | 0 / 00.537 | 0 / 00.670 | 0 / 00.790 |
| Cygnet Open · 12B | — | Decision | No answers at this threshold0.720 | 0 / 00.604 | 0 / 00.676 | 0 / 00.848 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | No answers at this threshold0.527 | 0 / 00.516 | 0 / 00.547 | 0 / 00.584 |
| GLiDE 0.4B Open · 0.4B | — | Decision | No answers at this threshold0.556 | 0 / 00.531 | 0 / 00.545 | 0 / 00.639 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | No answers at this threshold0.730 | 0 / 00.672 | 0 / 00.663 | 0 / 00.859 |
| Metask-Jev-4B Open · 4.5B | — | Decision | No answers at this threshold0.660 | 0 / 00.591 | 0 / 00.604 | 0 / 00.830 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | No answers at this threshold0.733 | 0 / 00.623 | 0 / 00.716 | 0 / 00.861 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | No answers at this threshold0.641 | 0 / 00.580 | 0 / 00.615 | 0 / 00.694 |
| Strands Decider 2B Open · 2.0B | — | Decision | No answers at this threshold0.568 | 0 / 00.544 | 0 / 00.562 | 0 / 00.634 |
| celeris-1-decision Closed · Efficient | — | Decision | No answers at this threshold0.735 | 0 / 00.706 | 0 / 00.679 | 0 / 00.867 |
| celeris-1.1-decision Closed · Efficient | — | Decision | No answers at this threshold0.733 | 0 / 00.709 | 0 / 00.679 | 0 / 00.853 |
| Karpo Decision 31B Open · 31B | — | Decision | 169 / 30.816 | 58 / 00.805 | 33 / 20.758 | 78 / 10.877 |
| Drex Max Open · 27B | — | Decision | 147 / 80.786 | 34 / 00.729 | 39 / 50.758 | 74 / 30.838 |
| Mammoth-1 Open · 27B | — | Decision | 106 / 20.795 | 21 / 00.703 | 13 / 00.797 | 72 / 20.905 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 89 / 10.793 | 7 / 00.729 | 24 / 00.771 | 58 / 10.877 |
| Mercury Decide Closed · Efficient | — | Decision | 84 / 70.746 | 14 / 20.670 | 28 / 10.744 | 42 / 40.775 |
| Sieve 31B v3 Open · 31B | — | Decision | 79 / 20.797 | 9 / 00.764 | 16 / 20.745 | 54 / 00.861 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 78 / 10.800 | 8 / 00.728 | 12 / 00.787 | 58 / 10.860 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 78 / 20.798 | 9 / 00.730 | 12 / 00.783 | 57 / 20.791 |
| GLiDE 27B Open · 27B | — | Decision | 74 / 10.822 | 6 / 00.754 | 17 / 00.800 | 51 / 10.888 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 70 / 20.787 | 17 / 00.736 | 18 / 10.743 | 35 / 10.857 |
| d1 Closed · Efficient | — | Decision | 55 / 10.723 | 3 / 00.630 | 5 / 00.683 | 47 / 10.828 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 54 / 10.771 | 4 / 00.706 | 2 / 00.730 | 48 / 10.869 |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 35 / 20.701 | 5 / 10.598 | 5 / 10.709 | 25 / 00.785 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 26 / 10.752 | 8 / 10.645 | 3 / 00.745 | 15 / 00.881 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 17 / 10.708 | 2 / 00.634 | 3 / 10.641 | 12 / 00.881 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 47 setups: GPT-6 Astra (chat), Gemini 4 Carbon (chat), GPT-6.1 Sol (chat), Qwen4-Max (high · chat), Claude Opus 5.5 (chat), DeepSeek-V4.1-Flash (high · chat · 1017), Gemini 4 Hydrogen (chat), Inkling 2 Small (high · chat), Claude Sonnet 5.5 (chat), Muse Ember (max · chat), Inkling-Small (high · chat · Jul 2026), Claude Fable 5.1 (chat), MiMo-V2.6-Flash (on · chat), Qwen3.5-397B-A17B (on · chat), Qwen3.8-27B (low · chat), GPT-6.1 Terra (chat), MiniMax-M3.1-Flash (high · chat), Qwen3.8 Flash (off · chat), Gemma 4 31B (chat), GLM-5.3-Flash (high · chat), Qwen3.5-4B (on · chat), Granite-4.2-3B (high · chat), Qwen3.5-4B (off · chat), MiniCPM5-2B (on · chat), Gemma 4 12B Unified (chat), LFM2.5-24B-A2B (on · chat), Gemma 4 E4B (on · chat · 32k cap · sampled), LFM3-1.7B (off · chat), Gemma 4 E4B (on · chat · 32k cap), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), LFM3-1.7B (on · chat), MiniCPM5-2B (off · chat), Gemma 4 E4B (chat), Gemma 4 E4B (on · chat), Gemma 4 E2B (on · chat · 32k cap · sampled), Gemma 4 E2B (on · chat · 32k cap), LFM2.5-2.6B (on · chat), Gemma 4 E2B (on · chat), Gemma 4 E2B (chat), Raon-1 nano (on · chat), Qwen3.5-2B (on · chat), Qwen3-14B (off · chat), Raon-1 nano (off · chat), Qwen3.5-2B (off · chat), LFM2.5-8B-A1B (on · chat), GPT-6 Luna (chat), Erk-14B (off · chat).
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 1,125 of 1,125); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Not ranked (only complete runs are): Spark-X2.6-4B (high · chat) (not running now, 20 of 1,125 stored, last answer 13 Nov 02:18 UTC); Grok 4.7 (chat) (paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)); Claude Fable 5.2 (chat) (not running now, 147 of 1,125 stored, last answer 12 Nov 11:32 UTC); Granite-4.2-3B (off · chat) (still running, 259 of 1,125); GPT-6 Luna (max · chat) (paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)); DeepSeek-V4-Flash (chat) (sample: a random subset by design, 317 answers stored, not the full 1,125); Claude Haiku 5.5 (max · chat) (paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Math 99.6% | — | Chat | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 56.4%answered 97% | 6,590 ms | 14,330 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Math 99.6% | — | Chat | 96.7%95.5%–97.6% | ab | 1,088 / 1,125 | Not measured | Not measured | — | 45.5%answered 98% | 44,520 ms | 206,660 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Math 99.6% | — | Chat | 96.5%95.3%–97.5% | a | 1,086 / 1,125 | Not measured | Not measured | — | 63.2%answered 97% | 7,070 ms | 16,200 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Math 99.8% | — | Chat | 96.3%95.0%–97.2% | abc | 1,083 / 1,125 | Not measured | Not measured | — | 0.0%answered 100% | 11,570 ms | 38,640 ms | $5.690$5,690 |
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.8% | — | Chat | 96.2%94.9%–97.2% | abcde | 1,082 / 1,125 | Not measured | Not measured | — | 38.2%answered 98% | 5,830 ms | 12,240 ms | $19.540$19,540 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Math 99.6% | — | Chat | 96.2%94.9%–97.2% | abcde | 1,082 / 1,125 | Not measured | Not measured | — | 3.8%answered 100% | 6,680 ms | 40,040 ms | $1.647$1,647 |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Math 99.6% | — | Chat | 96.1%94.8%–97.1% | abdf | 1,081 / 1,125 | Not measured | Not measured | — | 76.1%answered 95% | 8,210 ms | 24,090 ms | — |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Math 99.5% | — | Chat | 95.6%94.3%–96.7% | abcdefg | 1,076 / 1,125 | Not measured | Not measured | — | 37.3%answered 98% | 13,180 ms | 144,250 ms | $4.121$4,121 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.7% | — | Chat | 95.5%94.1%–96.5% | abcdefg | 1,074 / 1,125 | Not measured | Not measured | — | 41.3%answered 98% | 6,710 ms | 14,890 ms | $11.480$11,480 API list-price equivalent |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Math 99.6% | — | Chat | 95.5%94.1%–96.5% | abcdefg | 1,074 / 1,125 | Not measured | Not measured | — | 22.6%answered 99% | 3,110 ms | 14,020 ms | $1.820$1,820 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Math 99.6% | — | Chat | 95.4%94.0%–96.5% | abcdefg | 1,073 / 1,125 | Not measured | Not measured | — | 50.0%answered 97% | 13,290 ms | 78,510 ms | $5.489$5,489 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.7% | — | Chat | 95.2%93.8%–96.3% | abcdefgh | 1,071 / 1,125 | Not measured | Not measured | — | 8.3%answered 100% | 3,820 ms | 13,460 ms | $0.000$0.00 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Math 99.4% | — | Chat | 95.1%93.7%–96.2% | abcdefgh | 1,070 / 1,125 | Not measured | Not measured | — | 23.3%answered 98% | 24,740 ms | 179,930 ms | $1.390$1,390 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active Qwen3.5-397B-A17B on · chat · Alibaba · chat model · reasoning on Parameters: 397B MoE · 17B active Math running | — | Chat | 95.1%93.7%–96.2% | abcdefgh | 1,070 / 1,125 | Not measured | Not measured | — | 31.7%answered 98% | 84,650 ms | 167,520 ms | $13.426$13,426 |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Math 99.2% | — | Chat | 94.0%92.5%–95.3% | bcdefghi | 1,058 / 1,125 | Not measured | Not measured | — | 39.1%answered 97% | 18,710 ms | 101,030 ms | $4.884$4,884estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Math 99.2% | — | Chat | 94.0%92.4%–95.2% | ceghi | 1,057 / 1,125 | Not measured | Not measured | — | 61.1%answered 95% | 5,640 ms | 10,330 ms | $0.000$0.00 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Math 98.2% | — | Chat | 94.0%92.4%–95.2% | bcdefghi | 1,057 / 1,125 | Not measured | Not measured | — | 38.3%answered 97% | 5,610 ms | 37,430 ms | $0.803$803 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Math 99.5% | — | Chat | 94.0%92.4%–95.2% | defghi | 1,057 / 1,125 | Not measured | Not measured | — | 34.6%answered 98% | 17,550 ms | 73,220 ms | $1.376$1,376 |
| Gemma 4 31B chat Open · 31B | — | Chat | 93.3%91.7%–94.6% | fghij | 1,050 / 1,125 | Not measured | Not measured | — | 16.3%answered 99% | 18,290 ms | 39,790 ms | $1.244$1,244estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Math 98.8% | — | Chat | 93.2%91.5%–94.5% | ghij | 1,048 / 1,125 | Not measured | Not measured | — | 4.2%answered 100% | 12,050 ms | 82,830 ms | $1.057$1,057 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Math 99.4% | — | Chat | 92.4%90.7%–93.8% | hij | 1,039 / 1,125 | Not measured | Not measured | — | 36.1%answered 97% | 142,520 ms | 246,280 ms | $3.525$3,525estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Math 99.2% | — | Chat | 91.6%89.9%–93.1% | ijk | 1,031 / 1,125 | Not measured | Not measured | — | 53.2%answered 94% | 221,350 ms | 1,229,550 ms | $10.893$10,893estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Math 98.5% | — | Chat | 90.1%88.3%–91.7% | jkl | 1,014 / 1,125 | Not measured | Not measured | — | 33.6%answered 96% | 45,380 ms | 127,510 ms | $1.420$1,420estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Math 99.2% | — | Chat | 88.9%88.3%–89.6% min–max of 3 seeds | klm | 998 / 1,125seed 0 run | Not measured | Not measured | — | 51.5%answered 92% | 101,100 ms | 333,427 ms | $3.335$3,335estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Math 98.8% | — | Chat | 87.7%85.7%–89.5% | klmn | 987 / 1,125 | Not measured | Not measured | — | 15.5%answered 98% | 42,060 ms | 82,940 ms | $1.292$1,292estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Math 99.0% | — | Chat | 86.7%84.6%–88.5% | lmno | 975 / 1,125 | Not measured | Not measured | — | 37.8%answered 95% | 228,980 ms | 776,930 ms | $7.562$7,562estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.5% | — | Chat | 85.9%85.5%–86.1% min–max of 3 seeds | lmno | 968 / 1,125seed 0 run | Not measured | Not measured | — | 55.9%answered 90% | 31,567 ms | 83,467 ms | $0.993$993estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Math 98.7% | — | Chat | 85.5%83.3%–87.4% | lmno | 962 / 1,125 | Not measured | Not measured | — | 54.2%answered 90% | 163,870 ms | 656,750 ms | $0.487$487estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.5% | — | Chat | 85.0%82.8%–86.9% | mno | 956 / 1,125 | Not measured | Not measured | — | Not measured | 57,900 ms | 109,440 ms | $1.503$1,503estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Math 98.9% | — | Chat | 84.6%84.4%–85.0% min–max of 3 seeds | mno | 956 / 1,125seed 0 run | Not measured | Not measured | — | Not measured | 27,950 ms | 107,497 ms | $1.040$1,040estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B | — | Chat | 84.2%83.9%–84.4% min–max of 3 seeds | mno | 950 / 1,125seed 0 run | Not measured | Not measured | — | 43.6%answered 91% | 29,847 ms | 113,220 ms | $1.017$1,017estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Math 98.2% | — | Chat | 83.6%83.1%–83.9% min–max of 3 seeds | no | 942 / 1,125seed 0 run | Not measured | Not measured | — | 43.9%answered 88% | 46,453 ms | 111,607 ms | $1.327$1,327estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.1% | — | Chat | 83.2%80.9%–85.3% | no | 936 / 1,125 | Not measured | Not measured | — | 52.4%answered 88% | 32,690 ms | 64,070 ms | $0.842$842estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.2% | — | Chat | 82.6%80.3%–84.7% | o | 929 / 1,125 | Not measured | Not measured | — | 45.1%answered 91% | 284,080 ms | 539,980 ms | $0.650$650estimate · DGX Spark |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 97.7% | — | Chat | 76.8%76.0%–77.6% min–max of 3 seeds | p | 855 / 1,125seed 0 run | Not measured | Not measured | — | 32.6%answered 91% | 21,223 ms | 45,077 ms | $0.574$574estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 97.8% | — | Chat | 75.9%73.3%–78.3% | p | 854 / 1,125 | Not measured | Not measured | — | Not measured | 41,350 ms | 70,680 ms | $1.193$1,193estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Math 94.0% | — | Chat | 75.4%72.8%–77.8% | p | 848 / 1,125 | Not measured | Not measured | — | 44.3%answered 86% | 81,330 ms | 224,000 ms | $2.480$2,480estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 96.9% | — | Chat | 74.6%72.0%–77.0% | p | 839 / 1,125 | Not measured | Not measured | — | 26.4%answered 93% | 173,220 ms | 297,090 ms | $0.363$363estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 96.8% | — | Chat | 72.9%70.2%–75.4% | p | 820 / 1,125 | Not measured | Not measured | — | 34.0%answered 90% | 21,390 ms | 35,160 ms | $0.513$513estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Math 97.0% | — | Chat | 70.9%68.2%–73.5% | pq | 798 / 1,125 | Not measured | Not measured | — | 10.5%answered 95% | 493,230 ms† | 1,396,520 ms† | $15.330$15,330estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B on · chat Open · 2.0B | — | Chat | 70.3%67.6%–72.9% | pq | 791 / 1,125 | Not measured | Not measured | — | 33.1%answered 91% | 186,330 ms | 272,210 ms | $4.330$4,330estimate · RTX PRO 6000 Blackwell |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Math 95.8% | — | Chat | 65.3%62.5%–68.1% | qr | 735 / 1,125 | Not measured | Not measured | — | 16.1%answered 94% | 16,560 ms | 178,680 ms | $1.772$1,772estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Math 89.6% | — | Chat | 65.1%62.2%–67.8% | qrs | 732 / 1,125 | Not measured | Not measured | — | 35.7%answered 87% | 276,720 ms† | 965,630 ms† | $9.270$9,270estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Math 94.6% | — | Chat | 64.8%62.0%–67.5% | qrs | 729 / 1,125 | Not measured | Not measured | — | 35.2%answered 86% | 71,290 ms | 113,470 ms | $1.605$1,605estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 172 answers, 169 right · 3 wrong · AUROC 0.816 Math 81.6% | — | Decision | 60.7%57.8%–63.5% | rstu | 683 / 1,125 | 169 right of 172 kept3 wrong · 953 skipped | 0.816 | 71.80chosen-answer Brier 0.1711 | 6.6%answered 97% | 173 ms | 211 ms | $0.128$128estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Math 96.3% | — | Chat | 60.2%57.3%–63.0% | rstuvwxyzA | 677 / 1,125 | Not measured | Not measured | — | 58.3%answered 70% | 138,140 ms† | 403,380 ms† | $4.664$4,664estimate · RTX PRO 6000 Blackwell† |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 51 answers, 51 right · 0 wrong · AUROC 0.816 Math 78.0% | — | Decision | 59.3%56.4%–62.1% | rstv | 667 / 1,125 | 51 right of 51 kept0 mistakes · 1,074 skipped | 0.816 | 71.07chosen-answer Brier 0.1714 | 5.3%answered 98% | 215 ms | 278 ms | $0.160$160estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 89 answers, 89 right · 0 wrong · AUROC 0.804 Math 77.4% | — | Decision | 59.0%56.1%–61.9% | rstuvw | 664 / 1,125 | 89 right of 89 kept0 mistakes · 1,036 skipped | 0.804 | 70.72chosen-answer Brier 0.1757 | 0.4%answered 100% | 983 ms | 1,279 ms | $0.061$61.43estimate · DGX Spark |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.825 Math 80.3% | — | Decision | 58.9%56.0%–61.8% | rstuvw | 663 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.825 | 71.09chosen-answer Brier 0.1675 | 0.0%answered 100% | 151 ms | 204 ms | $0.112$112estimate · RTX PRO 6000 Blackwell |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 66 answers, 66 right · 0 wrong · AUROC 0.801 Math 77.3% | — | Decision | 58.7%55.8%–61.5% | rstuvwx | 660 / 1,125 | 66 right of 66 kept0 mistakes · 1,059 skipped | 0.801 | 70.48chosen-answer Brier 0.1771 | 1.9%answered 99% | 185 ms | 255 ms | $0.136$136estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 75 answers, 74 right · 1 wrong · AUROC 0.822 Math 72.0% | — | Decision | 58.4%55.5%–61.2% | rstuvwxy | 657 / 1,125 | 74 right of 75 kept1 wrong · 1,050 skipped | 0.822 | 70.67chosen-answer Brier 0.1707 | 44.3%answered 79% | 544 ms | 655 ms | $0.033$33.16estimate · DGX Spark |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Math 93.7% | — | Chat | 58.0%55.0%–60.8% | stuvwxyzAB | 652 / 1,125 | Not measured | Not measured | — | 55.1%answered 72% | 3,860 ms | 5,820 ms | $0.000$0.00 |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 81 answers, 79 right · 2 wrong · AUROC 0.797 Math 77.1% | — | Decision | 57.9%55.0%–60.7% | rstuvwxz | 651 / 1,125 | 79 right of 81 kept2 wrong · 1,044 skipped | 0.797 | 70.03chosen-answer Brier 0.1781 | 5.5%answered 98% | 1,050 ms | 1,335 ms | $0.063$62.95estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.796 Math 75.2% | — | Decision | 57.8%54.9%–60.6% | rstuvwxyzA | 650 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.796 | 69.90chosen-answer Brier 0.1798 | 7.1%answered 97% | 285 ms | 399 ms | $0.211$211estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 45 answers, 45 right · 0 wrong · AUROC 0.793 Math 69.9% | — | Decision | 55.2%52.3%–58.1% | tuvwxyzABC | 621 / 1,125 | 45 right of 45 kept0 mistakes · 1,080 skipped | 0.793 | 68.39chosen-answer Brier 0.1842 | 50.2%answered 73% | 111 ms | 140 ms | $0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 20 answers, 20 right · 0 wrong · AUROC 0.783 Math 74.1% | — | Decision | 54.9%52.0%–57.8% | tuvwxyzABC | 618 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.783 | 68.09chosen-answer Brier 0.1876 | 91.0%answered 35% | 182 ms | 425 ms | $0.013$12.64 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 90 answers, 89 right · 1 wrong · AUROC 0.793 Math 70.1% | — | Decision | 54.8%51.8%–57.6% | tuvwxyzABCD | 616 / 1,125 | 89 right of 90 kept1 wrong · 1,035 skipped | 0.793 | 68.07chosen-answer Brier 0.1861 | 54.1%answered 69% | 329 ms | 1,540 ms | $0.022$22.37 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 62 answers, 62 right · 0 wrong · AUROC 0.803 Math 72.5% | — | Decision | 54.8%51.8%–57.6% | tuvwxyzABCD | 616 / 1,125 | 62 right of 62 kept0 mistakes · 1,063 skipped | 0.803 | 67.98chosen-answer Brier 0.1881 | 18.1%answered 91% | 108 ms | 131 ms | $0.081$81.16estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 28 answers, 28 right · 0 wrong · AUROC 0.778 Math 71.5% | — | Decision | 54.6%51.7%–57.5% | uwxyzABCDE | 614 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.778 | 66.46chosen-answer Brier 0.2166 | 10.1%answered 95% | 156 ms | 185 ms | $0.109$109estimate · RTX PRO 6000 Blackwell |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 52 answers, 52 right · 0 wrong · AUROC 0.814 Math 70.8% | — | Decision | 54.5%51.6%–57.4% | vwxyzABCD | 613 / 1,125 | 52 right of 52 kept0 mistakes · 1,073 skipped | 0.814 | 68.27chosen-answer Brier 0.1794 | 60.0%answered 67% | 271 ms | 291 ms | $0.052$52.03 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 108 answers, 106 right · 2 wrong · AUROC 0.795 Math 74.0% | — | Decision | 54.5%51.6%–57.4% | vwxyzABCDE | 613 / 1,125 | 106 right of 108 kept2 wrong · 1,017 skipped | 0.795 | 68.07chosen-answer Brier 0.1835 | 72.3%answered 58% | 221 ms | 244 ms | $0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.806 Math 70.4% | — | Decision | 54.5%51.6%–57.4% | vwxyzABCD | 613 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.806 | 68.34chosen-answer Brier 0.1781 | 14.8%answered 93% | 76 ms | 98 ms | $0.057$57.18estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Math 90.8% | — | Chat | 54.2%51.3%–57.1% | tuvwxyzABCDEFG | 610 / 1,125 | Not measured | Not measured | — | 30.7%answered 84% | 10,150 ms | 18,410 ms | $0.390$390estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 80 answers, 78 right · 2 wrong · AUROC 0.798 Math 69.8% | — | Decision | 53.3%50.4%–56.2% | wxyzABCDEF | 600 / 1,125 | 78 right of 80 kept2 wrong · 1,045 skipped | 0.798 | 67.64chosen-answer Brier 0.1805 | 14.4%answered 93% | 238 ms | 552 ms | $0.007$7.46 |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 155 answers, 147 right · 8 wrong · AUROC 0.786 Math 70.3% | — | Decision | 53.1%50.1%–56.0% | wxyzABCDEF | 597 / 1,125 | 147 right of 155 kept8 wrong · 970 skipped | 0.786 | 65.68chosen-answer Brier 0.2170 | 7.3%answered 97% | 106 ms | 143 ms | $0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 79 answers, 78 right · 1 wrong · AUROC 0.800 Math 69.3% | with web | Decision | 52.5%49.6%–55.4% | xyzABCDEFGH | 591 / 1,125 | 78 right of 79 kept1 wrong · 1,046 skipped | 0.800 | 67.24chosen-answer Brier 0.1806 | 37.7%answered 80% | 448 ms | 827 ms | $1.030$1,030 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 72 answers, 70 right · 2 wrong · AUROC 0.787 Math 68.6% | — | Decision | 52.4%49.4%–55.3% | zABCDEFGHI | 589 / 1,125 | 70 right of 72 kept2 wrong · 1,053 skipped | 0.787 | 65.94chosen-answer Brier 0.2047 | 65.7%answered 61% | 384 ms | 393 ms | $0.015$14.92 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.782 Math 69.0% | — | Decision | 52.3%49.3%–55.2% | yABCDEFGHI | 588 / 1,125 | 7 right of 7 kept0 mistakes · 1,118 skipped | 0.782 | 66.89chosen-answer Brier 0.1849 | 1.5%answered 99% | 385 ms | 533 ms | $0.025$24.91estimate · DGX Spark |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.800 Math 69.9% | — | Decision | 51.6%48.6%–54.5% | BCDEFGHIJ | 580 / 1,125 | 13 right of 13 kept0 mistakes · 1,112 skipped | 0.800 | 66.77chosen-answer Brier 0.1802 | 94.5%answered 28% | 220 ms | 341 ms | $0.013$12.64 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.774 Math 66.3% | — | Decision | 50.6%47.7%–53.5% | BCDEFGHIJK | 569 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.774 | 65.90chosen-answer Brier 0.1878 | 8.8%answered 95% | 68 ms | 88 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 43 answers, 43 right · 0 wrong · AUROC 0.768 Math 71.3% | — | Decision | 50.0%47.1%–53.0% | CDEFGHIJKL | 563 / 1,125 | 43 right of 43 kept0 mistakes · 1,082 skipped | 0.768 | 65.37chosen-answer Brier 0.1930 | 2.0%answered 99% | 228 ms | 344 ms | $0.037$36.77 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 28 answers, 28 right · 0 wrong · AUROC 0.742 Math 65.5% | — | Decision | 49.2%46.2%–52.1% | CDEFGHIJKLM | 553 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.742 | 64.51chosen-answer Brier 0.2013 | 78.8%answered 43% | 222 ms | 390 ms | $0.006$6.02 |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 10 answers, 10 right · 0 wrong · AUROC 0.753 Math 59.8% | — | Decision | 48.2%45.3%–51.1% | EFGHIJKLMN | 542 / 1,125 | 10 right of 10 kept0 mistakes · 1,115 skipped | 0.753 | 64.33chosen-answer Brier 0.1952 | 23.4%answered 87% | 52 ms | 66 ms | $0.039$39.35estimate · RTX PRO 6000 Blackwell |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 61 answers, 61 right · 0 wrong · AUROC 0.750 Math 65.1% | — | Decision | 48.2%45.3%–51.1% | DEFGHIJKLMN | 542 / 1,125 | 61 right of 61 kept0 mistakes · 1,064 skipped | 0.750 | 64.14chosen-answer Brier 0.1990 | 24.5%answered 86% | 166 ms | 193 ms | $0.026$25.64 |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 32 answers, 32 right · 0 wrong · AUROC 0.749 Math 57.9% | — | Decision | 47.8%44.9%–50.7% | FGHIJKLMN | 538 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.749 | 63.95chosen-answer Brier 0.1993 | 30.4%answered 82% | 208 ms | 251 ms | $0.013$13.29estimate · DGX Spark |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.735 Math 70.0% | — | Decision | 47.3%44.4%–50.2% | FGHIJKLMNO | 532 / 1,125 | No answers at this threshold | 0.735 | 62.94chosen-answer Brier 0.2141 | 92.3%answered 28% | 390 ms | 394 ms | $0.015$15.40 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.733 Math 69.9% | — | Decision | 47.3%44.4%–50.2% | FGHIJKLMNO | 532 / 1,125 | No answers at this threshold | 0.733 | 62.92chosen-answer Brier 0.2144 | 91.0%answered 29% | 383 ms | 390 ms | $0.002$1.54 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 14 answers, 14 right · 0 wrong · AUROC 0.762 Math 58.4% | — | Decision | 46.6%43.7%–49.5% | GHIJKLMNOP | 524 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.762 | 63.55chosen-answer Brier 0.1947 | 17.4%answered 90% | 160 ms | 175 ms | $0.010$10.04estimate · DGX Spark |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 55 answers, 54 right · 1 wrong · AUROC 0.771 Math 62.1% | — | Decision | 46.4%43.5%–49.3% | GHIJKLMNOP | 522 / 1,125 | 54 right of 55 kept1 wrong · 1,070 skipped | 0.771 | 63.48chosen-answer Brier 0.1945 | 12.2%answered 93% | 41 ms | 46 ms | $0.031$31.36estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 53 answers, 53 right · 0 wrong · AUROC 0.768 Math 61.3% | — | Decision | 46.4%43.5%–49.3% | GHIJKLMNOP | 522 / 1,125 | 53 right of 53 kept0 mistakes · 1,072 skipped | 0.768 | 63.39chosen-answer Brier 0.1962 | 75.6%answered 47% | 79 ms | 98 ms | $0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 65 answers, 65 right · 0 wrong · AUROC 0.764 Math 64.1% | — | Decision | 46.3%43.4%–49.2% | HIJKLMNOP | 521 / 1,125 | 65 right of 65 kept0 mistakes · 1,060 skipped | 0.764 | 63.36chosen-answer Brier 0.1959 | 15.8%answered 91% | 167 ms | 203 ms | $0.026$25.64 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 48 answers, 48 right · 0 wrong · AUROC 0.748 Math 65.2% | — | Decision | 46.0%43.2%–49.0% | IJKLMNOPQR | 518 / 1,125 | 48 right of 48 kept0 mistakes · 1,077 skipped | 0.748 | 63.14chosen-answer Brier 0.1977 | 90.6%answered 28% | 176 ms | 272 ms | $0.006$6.02 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 21 answers, 21 right · 0 wrong · AUROC 0.720 Math 55.8% | — | Decision | 45.9%43.0%–48.8% | IJKLMNOPQS | 516 / 1,125 | 21 right of 21 kept0 mistakes · 1,104 skipped | 0.720 | 62.63chosen-answer Brier 0.2061 | 16.0%answered 91% | 73 ms | 84 ms | $0.005$4.62estimate · DGX Spark |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 56 answers, 56 right · 0 wrong · AUROC 0.759 Math 58.8% | — | Decision | 45.3%42.4%–48.3% | JKLMNOPQRST | 510 / 1,125 | 56 right of 56 kept0 mistakes · 1,069 skipped | 0.759 | 61.81chosen-answer Brier 0.2171 | 12.4%answered 93% | 47 ms | 53 ms | $0.034$33.81estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 91 answers, 84 right · 7 wrong · AUROC 0.746 Math 53.5% | — | Decision | 45.1%42.2%–48.0% | KLMNOPQRST | 507 / 1,125 | 84 right of 91 kept7 wrong · 1,034 skipped | 0.746 | 61.18chosen-answer Brier 0.2272 | 20.5%answered 88% | 301 ms | 360 ms | $0.008$7.52 |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 17 answers, 17 right · 0 wrong · AUROC 0.757 Math 55.9% | — | Decision | 44.7%41.8%–47.6% | KLMNOPQRST | 503 / 1,125 | 17 right of 17 kept0 mistakes · 1,108 skipped | 0.757 | 62.14chosen-answer Brier 0.2042 | 24.4%answered 85% | 33 ms | 41 ms | $0.026$26.44estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 20 answers, 20 right · 0 wrong · AUROC 0.712 Math 59.4% | — | Decision | 44.4%41.6%–47.4% | KLMNOPQRST | 500 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.712 | 61.53chosen-answer Brier 0.2139 | 19.8%answered 88% | 315 ms | 362 ms | $0.013$13.42 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 37 answers, 35 right · 2 wrong · AUROC 0.701 Math 55.3% | — | Decision | 44.4%41.5%–47.3% | KLMNOPQRST | 499 / 1,125 | 35 right of 37 kept2 wrong · 1,088 skipped | 0.701 | 61.42chosen-answer Brier 0.2151 | 13.6%answered 92% | 60 ms | 67 ms | $0.042$42.42estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 30 answers, 30 right · 0 wrong · AUROC 0.725 Math 59.8% | — | Decision | 44.3%41.4%–47.2% | KLMNOPQRST | 498 / 1,125 | 30 right of 30 kept0 mistakes · 1,095 skipped | 0.725 | 61.97chosen-answer Brier 0.2032 | 6.7%answered 96% | 50 ms | 61 ms | $0.036$36.27estimate · RTX PRO 6000 Blackwell |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.739 Math 56.8% | — | Decision | 44.1%41.2%–47.0% | LMNOPQRST | 496 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.739 | 61.90chosen-answer Brier 0.2028 | 13.6%answered 92% | 32 ms | 37 ms | $0.026$25.82estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.730 Math 60.1% | — | Decision | 43.9%41.0%–46.8% | LMNOPQRST | 494 / 1,125 | No answers at this threshold | 0.730 | 61.90chosen-answer Brier 0.2011 | 19.5%answered 88% | 52 ms | 65 ms | $0.040$39.96estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 27 answers, 26 right · 1 wrong · AUROC 0.752 Math 56.0% | — | Decision | 43.3%40.4%–46.2% | LMNOPQRST | 487 / 1,125 | 26 right of 27 kept1 wrong · 1,098 skipped | 0.752 | 60.41chosen-answer Brier 0.2247 | 6.0%answered 97% | 209 ms | 269 ms | $0.013$13.35estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 35 answers, 35 right · 0 wrong · AUROC 0.729 Math 55.2% | — | Decision | 43.1%40.2%–46.0% | LMNOPQRST | 485 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 61.34chosen-answer Brier 0.2043 | 28.9%answered 82% | 45 ms | 55 ms | $0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.733 Math 54.6% | — | Decision | 42.6%39.7%–45.5% | MNOPQRST | 479 / 1,125 | No answers at this threshold | 0.733 | 59.78chosen-answer Brier 0.2302 | 28.2%answered 82% | 413 ms | 576 ms | $0.294$294estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 18 answers, 17 right · 1 wrong · AUROC 0.708 Math 56.0% | — | Decision | 42.3%39.5%–45.2% | NOPQRST | 476 / 1,125 | 17 right of 18 kept1 wrong · 1,107 skipped | 0.708 | 59.66chosen-answer Brier 0.2298 | 8.3%answered 95% | 96 ms | 115 ms | $0.006$5.89estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 33 answers, 33 right · 0 wrong · AUROC 0.717 Math 59.0% | — | Decision | 41.9%39.0%–44.8% | NOPQRST | 471 / 1,125 | 33 right of 33 kept0 mistakes · 1,092 skipped | 0.717 | 60.73chosen-answer Brier 0.2040 | 37.7%answered 75% | 345 ms | 504 ms | $0.018$18.41 |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.697 Math 51.1% | — | Decision | 40.8%38.0%–43.7% | OPQRST | 459 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.697 | 58.55chosen-answer Brier 0.2369 | 3.6%answered 98% | 196 ms | 222 ms | $0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 56 answers, 55 right · 1 wrong · AUROC 0.723 Math 55.6% | — | Decision | 40.8%38.0%–43.7% | OPQRST | 459 / 1,125 | 55 right of 56 kept1 wrong · 1,069 skipped | 0.723 | 59.76chosen-answer Brier 0.2128 | 11.3%answered 93% | 188 ms | 290 ms | $0.006$6.02 |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.726 Math 53.8% | — | Decision | 40.6%37.8%–43.5% | OPQRST | 457 / 1,125 | 4 right of 4 kept0 mistakes · 1,121 skipped | 0.726 | 60.27chosen-answer Brier 0.2009 | 1.2%answered 99% | 108 ms | 136 ms | $0.086$86.07estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.724 Math 50.4% | — | Decision | 40.4%37.5%–43.3% | PQRST | 454 / 1,125 | 39 right of 39 kept0 mistakes · 1,086 skipped | 0.724 | 58.90chosen-answer Brier 0.2256 | 14.6%answered 91% | 38 ms | 48 ms | $0.030$29.51estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 35 answers, 35 right · 0 wrong · AUROC 0.729 Math 51.5% | — | Decision | 39.7%36.9%–42.6% | RT | 447 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 59.26chosen-answer Brier 0.2122 | 12.6%answered 92% | 45 ms | 51 ms | $0.033$33.20estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 32 answers, 32 right · 0 wrong · AUROC 0.727 Math 46.9% | — | Decision | 39.7%36.9%–42.6% | QRST | 447 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.727 | 58.43chosen-answer Brier 0.2287 | 29.7%answered 80% | 66 ms | 91 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.720 Math 44.3% | — | Decision | 39.3%36.5%–42.2% | STU | 442 / 1,125 | No answers at this threshold | 0.720 | 58.97chosen-answer Brier 0.2135 | 11.3%answered 93% | 40 ms | 48 ms | $0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.659 Math 49.8% | — | Decision | 38.9%36.1%–41.8% | TU | 438 / 1,125 | No answers at this threshold | 0.659 | 57.70chosen-answer Brier 0.2353 | 14.4%answered 91% | 110 ms | 137 ms | $0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 14 answers, 14 right · 0 wrong · AUROC 0.632 Math 36.8% | — | Decision | 32.3%29.6%–35.1% | UV | 363 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.632 | 54.39chosen-answer Brier 0.2349 | 32.9%answered 75% | 34 ms | 42 ms | $0.002$2.26estimate · DGX Spark |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.660 Math 39.2% | — | Decision | 32.1%29.4%–34.9% | UV | 361 / 1,125 | No answers at this threshold | 0.660 | 56.30chosen-answer Brier 0.1950 | 1.5%answered 99% | 123 ms | 152 ms | $0.091$90.99estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.601 Math 35.3% | — | Decision | 31.7%29.1%–34.5% | V | 357 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.601 | 55.15chosen-answer Brier 0.2143 | 15.7%answered 89% | 59 ms | 70 ms | $0.004$3.78estimate · DGX Spark |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.640 Math 38.7% | — | Decision | 30.8%28.1%–33.5% | VW | 346 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.640 | 55.51chosen-answer Brier 0.1975 | 31.4%answered 76% | 40 ms | 45 ms | $0.003$2.52estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 3 answers, 3 right · 0 wrong · AUROC 0.620 Math 37.1% | — | Decision | 30.7%28.0%–33.4% | VW | 345 / 1,125 | 3 right of 3 kept0 mistakes · 1,122 skipped | 0.620 | 55.05chosen-answer Brier 0.2058 | 86.5%answered 22% | 17 ms | 23 ms | $0.015$14.76estimate · RTX PRO 6000 Blackwell |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 9 answers, 9 right · 0 wrong · AUROC 0.617 Math 32.1% | — | Decision | 29.2%26.6%–31.9% | VWX | 328 / 1,125 | 9 right of 9 kept0 mistakes · 1,116 skipped | 0.617 | 53.53chosen-answer Brier 0.2210 | 26.1%answered 80% | 23 ms | 29 ms | $0.002$1.58estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.545 Math 28.2% | — | Decision | 27.0%24.5%–29.7% | VWX | 304 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.545 | 52.72chosen-answer Brier 0.2159 | 4.6%answered 97% | 16 ms | 19 ms | $0.001$1.16estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.573 Math 28.2% | — | Decision | 26.7%24.2%–29.3% | VWX | 300 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.573 | 53.45chosen-answer Brier 0.1976 | 1.2%answered 99% | 31 ms | 39 ms | $0.002$2.00estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.641 Math 29.7% | — | Decision | 24.7%22.3%–27.3% | WX | 278 / 1,125 | No answers at this threshold | 0.641 | 53.22chosen-answer Brier 0.1827 | 21.4%answered 83% | 27 ms | 31 ms | $0.002$1.79estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.527 Math 22.7% | — | Decision | 23.6%21.2%–26.1% | WX | 265 / 1,125 | No answers at this threshold | 0.527 | 51.99chosen-answer Brier 0.1958 | 12.4%answered 90% | 15 ms | 28 ms | $0.001$1.31estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.556 Math 20.6% | — | Decision | 23.2%20.8%–25.8% | X | 261 / 1,125 | No answers at this threshold | 0.556 | 52.19chosen-answer Brier 0.1883 | 76.1%answered 30% | 13 ms | 16 ms | $0.001$0.95estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.568 Math 28.7% | — | Decision | 22.4%20.1%–24.9% | X | 252 / 1,125 | No answers at this threshold | 0.568 | 51.86chosen-answer Brier 0.1868 | 8.3%answered 94% | 36 ms | 43 ms | $0.028$27.67estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
quincyqiang Perplexity victorbennett StartLux Fastino PelaAI scar-ai Liquid AI OmniJev Other Metask Lab Strands Agents Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 40.4%1,125 | 54.7%49.4% kept (556) | 68.8%25.6% kept (288) | 83.1%12.1% kept (136) | 100.0%3.5% kept (39) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 43.1%1,125 | 65.9%36.3% kept (408) | 82.4%18.7% kept (210) | 92.3%10.4% kept (117) | 100.0%3.1% kept (35) |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 39.7%1,125 | 59.2%39.8% kept (448) | 73.6%20.9% kept (235) | 87.0%9.6% kept (108) | 100.0%3.1% kept (35) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 45.9%1,125 | 68.4%36.8% kept (414) | 82.7%18.0% kept (202) | 97.8%8.3% kept (93) | 100.0%1.9% kept (21) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 44.7%1,125 | 63.9%47.6% kept (535) | 77.7%27.6% kept (310) | 92.2%11.5% kept (129) | 100.0%1.5% kept (17) |
| GLiDE 2B Open · 1.9B | — | Decision | 32.3%1,125 | 46.3%34.8% kept (391) | 54.7%15.1% kept (170) | 79.7%5.2% kept (59) | 100.0%1.2% kept (14) |
| GLiDE 0.8B Open · 0.8B | — | Decision | 29.2%1,125 | 46.1%24.1% kept (271) | 54.7%9.4% kept (106) | 62.5%3.6% kept (40) | 100.0%0.8% kept (9) |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 31.7%1,125 | 51.2%18.4% kept (207) | 69.1%7.2% kept (81) | 87.5%2.8% kept (32) | 100.0%0.7% kept (8) |
| vera-medium-omni Open · 4.2B | — | Decision | 44.1%1,125 | 72.7%29.3% kept (330) | 85.6%11.1% kept (125) | 100.0%4.9% kept (55) | 100.0%0.5% kept (6) |
| d1-2B Open · 1.9B | — | Decision | 30.7%1,125 | 52.0%15.2% kept (171) | 73.3%5.3% kept (60) | 100.0%1.7% kept (19) | 100.0%0.3% kept (3) |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 26.7%1,125 | 47.4%10.1% kept (114) | 63.9%3.2% kept (36) | 92.9%1.2% kept (14) | 100.0%0.2% kept (2) |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 27.0%1,125 | 36.3%15.9% kept (179) | 44.6%5.0% kept (56) | 83.3%0.5% kept (6) | 100.0%0.2% kept (2) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 30.8%1,125 | 59.7%13.2% kept (149) | 73.8%5.4% kept (61) | 100.0%2.0% kept (22) | 100.0%0.2% kept (2) |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 23.6%1,125 | 35.6%9.2% kept (104) | 30.8%2.3% kept (26) | 50.0%0.5% kept (6) | none kept |
| GLiDE 0.4B Open · 0.4B | — | Decision | 23.2%1,125 | 33.3%5.6% kept (63) | 40.0%1.8% kept (20) | 100.0%0.3% kept (3) | none kept |
| Metask-Jev-4B Open · 4.5B | — | Decision | 32.1%1,125 | 66.9%12.4% kept (139) | 81.0%7.0% kept (79) | 90.5%1.9% kept (21) | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 24.7%1,125 | 46.6%9.2% kept (103) | 65.9%3.6% kept (41) | 90.0%0.9% kept (10) | none kept |
| Strands Decider 2B Open · 2.0B | — | Decision | 22.4%1,125 | 38.8%6.0% kept (67) | 58.8%1.5% kept (17) | 100.0%0.2% kept (2) | none kept |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 42.3%1,125 | 55.4%51.6% kept (581) | 68.8%27.4% kept (308) | 86.4%11.1% kept (125) | 94.4%1.6% kept (18) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|---|
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 39 / 00.724 | 1 / 00.644 | 8 / 00.699 | 30 / 00.866 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 35 / 00.729 | 3 / 00.642 | 1 / 00.693 | 31 / 00.879 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 35 / 00.729 | 2 / 00.648 | 5 / 00.672 | 28 / 00.884 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 21 / 00.720 | 1 / 00.633 | 2 / 00.672 | 18 / 00.853 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 17 / 00.757 | 4 / 00.663 | 1 / 00.731 | 12 / 00.860 |
| GLiDE 2B Open · 1.9B | — | Decision | 14 / 00.632 | 1 / 00.525 | 1 / 00.637 | 12 / 00.787 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 9 / 00.617 | 0 / 00.557 | 0 / 00.608 | 9 / 00.700 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 8 / 00.601 | 0 / 00.501 | 0 / 00.571 | 8 / 00.756 |
| vera-medium-omni Open · 4.2B | — | Decision | 6 / 00.739 | 0 / 00.659 | 0 / 00.710 | 6 / 00.876 |
| d1-2B Open · 1.9B | — | Decision | 3 / 00.620 | 0 / 00.549 | 0 / 00.606 | 3 / 00.787 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 2 / 00.573 | 0 / 00.520 | 0 / 00.531 | 2 / 00.746 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 2 / 00.545 | 0 / 00.508 | 0 / 00.538 | 2 / 00.608 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 2 / 00.640 | 0 / 00.539 | 0 / 00.652 | 2 / 00.767 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | No answers at this threshold0.527 | 0 / 00.516 | 0 / 00.547 | 0 / 00.584 |
| GLiDE 0.4B Open · 0.4B | — | Decision | No answers at this threshold0.556 | 0 / 00.531 | 0 / 00.545 | 0 / 00.639 |
| Metask-Jev-4B Open · 4.5B | — | Decision | No answers at this threshold0.660 | 0 / 00.591 | 0 / 00.604 | 0 / 00.830 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | No answers at this threshold0.641 | 0 / 00.580 | 0 / 00.615 | 0 / 00.694 |
| Strands Decider 2B Open · 2.0B | — | Decision | No answers at this threshold0.568 | 0 / 00.544 | 0 / 00.562 | 0 / 00.634 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 17 / 10.708 | 2 / 00.634 | 3 / 10.641 | 12 / 00.881 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 17 setups: Qwen3.5-4B (on · chat), Granite-4.2-3B (high · chat), Qwen3.5-4B (off · chat), MiniCPM5-2B (on · chat), LFM3-1.7B (off · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), LFM3-1.7B (on · chat), MiniCPM5-2B (off · chat), Gemma 4 E2B (on · chat · 32k cap · sampled), Gemma 4 E2B (on · chat · 32k cap), LFM2.5-2.6B (on · chat), Gemma 4 E2B (on · chat), Gemma 4 E2B (chat), Raon-1 nano (on · chat), Qwen3.5-2B (on · chat), Raon-1 nano (off · chat), Qwen3.5-2B (off · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Spark-X2.6-4B (high · chat) (not running now, 20 of 1,125 stored, last answer 13 Nov 02:18 UTC); Granite-4.2-3B (off · chat) (still running, 259 of 1,125).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Math 99.4% | — | Chat | 92.4%90.7%–93.8% | a | 1,039 / 1,125 | Not measured | Not measured | — | 36.1%answered 97% | 142,520 ms | 246,280 ms | $3.525$3,525estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Math 99.2% | — | Chat | 91.6%89.9%–93.1% | ab | 1,031 / 1,125 | Not measured | Not measured | — | 53.2%answered 94% | 221,350 ms | 1,229,550 ms | $10.893$10,893estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Math 98.5% | — | Chat | 90.1%88.3%–91.7% | ab | 1,014 / 1,125 | Not measured | Not measured | — | 33.6%answered 96% | 45,380 ms | 127,510 ms | $1.420$1,420estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Math 99.2% | — | Chat | 88.9%88.3%–89.6% min–max of 3 seeds | bc | 998 / 1,125seed 0 run | Not measured | Not measured | — | 51.5%answered 92% | 101,100 ms | 333,427 ms | $3.335$3,335estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Math 98.7% | — | Chat | 85.5%83.3%–87.4% | cd | 962 / 1,125 | Not measured | Not measured | — | 54.2%answered 90% | 163,870 ms | 656,750 ms | $0.487$487estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Math 98.9% | — | Chat | 84.6%84.4%–85.0% min–max of 3 seeds | cd | 956 / 1,125seed 0 run | Not measured | Not measured | — | Not measured | 27,950 ms | 107,497 ms | $1.040$1,040estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B | — | Chat | 84.2%83.9%–84.4% min–max of 3 seeds | d | 950 / 1,125seed 0 run | Not measured | Not measured | — | 43.6%answered 91% | 29,847 ms | 113,220 ms | $1.017$1,017estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Math 98.2% | — | Chat | 83.6%83.1%–83.9% min–max of 3 seeds | d | 942 / 1,125seed 0 run | Not measured | Not measured | — | 43.9%answered 88% | 46,453 ms | 111,607 ms | $1.327$1,327estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 97.7% | — | Chat | 76.8%76.0%–77.6% min–max of 3 seeds | e | 855 / 1,125seed 0 run | Not measured | Not measured | — | 32.6%answered 91% | 21,223 ms | 45,077 ms | $0.574$574estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 97.8% | — | Chat | 75.9%73.3%–78.3% | e | 854 / 1,125 | Not measured | Not measured | — | Not measured | 41,350 ms | 70,680 ms | $1.193$1,193estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Math 94.0% | — | Chat | 75.4%72.8%–77.8% | ef | 848 / 1,125 | Not measured | Not measured | — | 44.3%answered 86% | 81,330 ms | 224,000 ms | $2.480$2,480estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 96.9% | — | Chat | 74.6%72.0%–77.0% | ef | 839 / 1,125 | Not measured | Not measured | — | 26.4%answered 93% | 173,220 ms | 297,090 ms | $0.363$363estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 96.8% | — | Chat | 72.9%70.2%–75.4% | ef | 820 / 1,125 | Not measured | Not measured | — | 34.0%answered 90% | 21,390 ms | 35,160 ms | $0.513$513estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Math 97.0% | — | Chat | 70.9%68.2%–73.5% | ef | 798 / 1,125 | Not measured | Not measured | — | 10.5%answered 95% | 493,230 ms† | 1,396,520 ms† | $15.330$15,330estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B on · chat Open · 2.0B | — | Chat | 70.3%67.6%–72.9% | fg | 791 / 1,125 | Not measured | Not measured | — | 33.1%answered 91% | 186,330 ms | 272,210 ms | $4.330$4,330estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Math 89.6% | — | Chat | 65.1%62.2%–67.8% | gh | 732 / 1,125 | Not measured | Not measured | — | 35.7%answered 87% | 276,720 ms† | 965,630 ms† | $9.270$9,270estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Math 94.6% | — | Chat | 64.8%62.0%–67.5% | h | 729 / 1,125 | Not measured | Not measured | — | 35.2%answered 86% | 71,290 ms | 113,470 ms | $1.605$1,605estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 21 answers, 21 right · 0 wrong · AUROC 0.720 Math 55.8% | — | Decision | 45.9%43.0%–48.8% | i | 516 / 1,125 | 21 right of 21 kept0 mistakes · 1,104 skipped | 0.720 | 62.63chosen-answer Brier 0.2061 | 16.0%answered 91% | 73 ms | 84 ms | $0.005$4.62estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 17 answers, 17 right · 0 wrong · AUROC 0.757 Math 55.9% | — | Decision | 44.7%41.8%–47.6% | ij | 503 / 1,125 | 17 right of 17 kept0 mistakes · 1,108 skipped | 0.757 | 62.14chosen-answer Brier 0.2042 | 24.4%answered 85% | 33 ms | 41 ms | $0.026$26.44estimate · RTX PRO 6000 Blackwell |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.739 Math 56.8% | — | Decision | 44.1%41.2%–47.0% | ijk | 496 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.739 | 61.90chosen-answer Brier 0.2028 | 13.6%answered 92% | 32 ms | 37 ms | $0.026$25.82estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 35 answers, 35 right · 0 wrong · AUROC 0.729 Math 55.2% | — | Decision | 43.1%40.2%–46.0% | ijk | 485 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 61.34chosen-answer Brier 0.2043 | 28.9%answered 82% | 45 ms | 55 ms | $0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 18 answers, 17 right · 1 wrong · AUROC 0.708 Math 56.0% | — | Decision | 42.3%39.5%–45.2% | ijk | 476 / 1,125 | 17 right of 18 kept1 wrong · 1,107 skipped | 0.708 | 59.66chosen-answer Brier 0.2298 | 8.3%answered 95% | 96 ms | 115 ms | $0.006$5.89estimate · DGX Spark |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.724 Math 50.4% | — | Decision | 40.4%37.5%–43.3% | jk | 454 / 1,125 | 39 right of 39 kept0 mistakes · 1,086 skipped | 0.724 | 58.90chosen-answer Brier 0.2256 | 14.6%answered 91% | 38 ms | 48 ms | $0.030$29.51estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 35 answers, 35 right · 0 wrong · AUROC 0.729 Math 51.5% | — | Decision | 39.7%36.9%–42.6% | k | 447 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 59.26chosen-answer Brier 0.2122 | 12.6%answered 92% | 45 ms | 51 ms | $0.033$33.20estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 14 answers, 14 right · 0 wrong · AUROC 0.632 Math 36.8% | — | Decision | 32.3%29.6%–35.1% | l | 363 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.632 | 54.39chosen-answer Brier 0.2349 | 32.9%answered 75% | 34 ms | 42 ms | $0.002$2.26estimate · DGX Spark |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.660 Math 39.2% | — | Decision | 32.1%29.4%–34.9% | l | 361 / 1,125 | No answers at this threshold | 0.660 | 56.30chosen-answer Brier 0.1950 | 1.5%answered 99% | 123 ms | 152 ms | $0.091$90.99estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.601 Math 35.3% | — | Decision | 31.7%29.1%–34.5% | l | 357 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.601 | 55.15chosen-answer Brier 0.2143 | 15.7%answered 89% | 59 ms | 70 ms | $0.004$3.78estimate · DGX Spark |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.640 Math 38.7% | — | Decision | 30.8%28.1%–33.5% | l | 346 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.640 | 55.51chosen-answer Brier 0.1975 | 31.4%answered 76% | 40 ms | 45 ms | $0.003$2.52estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 3 answers, 3 right · 0 wrong · AUROC 0.620 Math 37.1% | — | Decision | 30.7%28.0%–33.4% | lm | 345 / 1,125 | 3 right of 3 kept0 mistakes · 1,122 skipped | 0.620 | 55.05chosen-answer Brier 0.2058 | 86.5%answered 22% | 17 ms | 23 ms | $0.015$14.76estimate · RTX PRO 6000 Blackwell |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 9 answers, 9 right · 0 wrong · AUROC 0.617 Math 32.1% | — | Decision | 29.2%26.6%–31.9% | lmn | 328 / 1,125 | 9 right of 9 kept0 mistakes · 1,116 skipped | 0.617 | 53.53chosen-answer Brier 0.2210 | 26.1%answered 80% | 23 ms | 29 ms | $0.002$1.58estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.545 Math 28.2% | — | Decision | 27.0%24.5%–29.7% | lmno | 304 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.545 | 52.72chosen-answer Brier 0.2159 | 4.6%answered 97% | 16 ms | 19 ms | $0.001$1.16estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.573 Math 28.2% | — | Decision | 26.7%24.2%–29.3% | lmno | 300 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.573 | 53.45chosen-answer Brier 0.1976 | 1.2%answered 99% | 31 ms | 39 ms | $0.002$2.00estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.641 Math 29.7% | — | Decision | 24.7%22.3%–27.3% | mno | 278 / 1,125 | No answers at this threshold | 0.641 | 53.22chosen-answer Brier 0.1827 | 21.4%answered 83% | 27 ms | 31 ms | $0.002$1.79estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.527 Math 22.7% | — | Decision | 23.6%21.2%–26.1% | no | 265 / 1,125 | No answers at this threshold | 0.527 | 51.99chosen-answer Brier 0.1958 | 12.4%answered 90% | 15 ms | 28 ms | $0.001$1.31estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.556 Math 20.6% | — | Decision | 23.2%20.8%–25.8% | o | 261 / 1,125 | No answers at this threshold | 0.556 | 52.19chosen-answer Brier 0.1883 | 76.1%answered 30% | 13 ms | 16 ms | $0.001$0.95estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.568 Math 28.7% | — | Decision | 22.4%20.1%–24.9% | o | 252 / 1,125 | No answers at this threshold | 0.568 | 51.86chosen-answer Brier 0.1868 | 8.3%answered 94% | 36 ms | 43 ms | $0.028$27.67estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Patrick-18 Base Labs Fastino EldanRing PelaAI StartLux tattrongvu Cloudflare Blockbrain Labs newsbang Valen-Team Apertix AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 45.3%1,125 | 58.8%58.8% kept (662) | 74.4%34.4% kept (387) | 91.7%15.0% kept (169) | 100.0%5.0% kept (56) |
| Elephant-1 Open · 9.0B | — | Decision | 46.4%1,125 | 66.5%47.8% kept (538) | 84.2%27.0% kept (304) | 94.9%14.0% kept (157) | 100.0%4.7% kept (53) |
| GLiDE 9B Open · 7.9B | — | Decision | 47.8%1,125 | 66.4%47.4% kept (533) | 87.9%22.8% kept (256) | 97.5%10.7% kept (120) | 100.0%2.8% kept (32) |
| Winnow-12B Open · 12B | — | Decision | 39.7%1,125 | 57.3%45.2% kept (508) | 65.5%32.4% kept (365) | 74.5%19.2% kept (216) | 100.0%2.8% kept (32) |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 44.3%1,125 | 68.7%31.2% kept (351) | 85.3%16.4% kept (184) | 97.9%8.4% kept (95) | 100.0%2.7% kept (30) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 46.6%1,125 | 71.5%38.3% kept (431) | 84.6%19.6% kept (221) | 98.9%8.2% kept (92) | 100.0%1.2% kept (14) |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 48.2%1,125 | 71.1%41.5% kept (467) | 86.4%22.9% kept (258) | 98.1%9.6% kept (108) | 100.0%0.9% kept (10) |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 50.6%1,125 | 74.3%44.6% kept (502) | 90.5%25.3% kept (285) | 99.1%9.7% kept (109) | 100.0%0.5% kept (6) |
| Clef-Flash Open · 9.0B | — | Decision | 38.9%1,125 | 52.4%40.9% kept (460) | 59.7%20.5% kept (231) | 77.9%6.0% kept (68) | none kept |
| Cygnet Open · 12B | — | Decision | 39.3%1,125 | 58.1%43.3% kept (487) | 71.2%18.2% kept (205) | 89.5%5.1% kept (57) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 43.9%1,125 | 70.9%32.1% kept (361) | 86.9%14.9% kept (168) | 100.0%4.1% kept (46) | none kept |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 46.4%1,125 | 68.9%44.4% kept (499) | 83.2%25.3% kept (285) | 95.8%12.7% kept (143) | 98.2%4.9% kept (55) |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 44.4%1,125 | 63.2%37.7% kept (424) | 77.9%17.7% kept (199) | 90.2%8.2% kept (92) | 94.6%3.3% kept (37) |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 43.3%1,125 | 56.1%59.4% kept (668) | 70.8%33.5% kept (377) | 89.6%14.6% kept (164) | 96.3%2.4% kept (27) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|---|
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 56 / 00.759 | 7 / 00.712 | 9 / 00.689 | 40 / 00.910 |
| Elephant-1 Open · 9.0B | — | Decision | 53 / 00.768 | 5 / 00.671 | 9 / 00.725 | 39 / 00.925 |
| GLiDE 9B Open · 7.9B | — | Decision | 32 / 00.749 | 3 / 00.637 | 4 / 00.729 | 25 / 00.911 |
| Winnow-12B Open · 12B | — | Decision | 32 / 00.727 | 1 / 00.608 | 6 / 00.670 | 25 / 00.835 |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 30 / 00.725 | 3 / 00.638 | 0 / 00.692 | 27 / 00.843 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 14 / 00.762 | 0 / 00.675 | 3 / 00.713 | 11 / 00.854 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 10 / 00.753 | 2 / 00.663 | 5 / 00.704 | 3 / 00.866 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 6 / 00.774 | 1 / 00.701 | 1 / 00.711 | 4 / 00.937 |
| Clef-Flash Open · 9.0B | — | Decision | No answers at this threshold0.659 | 0 / 00.537 | 0 / 00.670 | 0 / 00.790 |
| Cygnet Open · 12B | — | Decision | No answers at this threshold0.720 | 0 / 00.604 | 0 / 00.676 | 0 / 00.848 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | No answers at this threshold0.730 | 0 / 00.672 | 0 / 00.663 | 0 / 00.859 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 54 / 10.771 | 4 / 00.706 | 2 / 00.730 | 48 / 10.869 |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 35 / 20.701 | 5 / 10.598 | 5 / 10.709 | 25 / 00.785 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 26 / 10.752 | 8 / 10.645 | 3 / 00.745 | 15 / 00.881 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 8 setups: Gemma 4 12B Unified (chat), Gemma 4 E4B (on · chat · 32k cap · sampled), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E4B (chat), Gemma 4 E4B (on · chat), Qwen3-14B (off · chat), LFM2.5-8B-A1B (on · chat), Erk-14B (off · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Math 98.8% | — | Chat | 87.7%85.7%–89.5% | a | 987 / 1,125 | Not measured | Not measured | — | 15.5%answered 98% | 42,060 ms | 82,940 ms | $1.292$1,292estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.5% | — | Chat | 85.9%85.5%–86.1% min–max of 3 seeds | ab | 968 / 1,125seed 0 run | Not measured | Not measured | — | 55.9%answered 90% | 31,567 ms | 83,467 ms | $0.993$993estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.5% | — | Chat | 85.0%82.8%–86.9% | ab | 956 / 1,125 | Not measured | Not measured | — | Not measured | 57,900 ms | 109,440 ms | $1.503$1,503estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.1% | — | Chat | 83.2%80.9%–85.3% | b | 936 / 1,125 | Not measured | Not measured | — | 52.4%answered 88% | 32,690 ms | 64,070 ms | $0.842$842estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.2% | — | Chat | 82.6%80.3%–84.7% | b | 929 / 1,125 | Not measured | Not measured | — | 45.1%answered 91% | 284,080 ms | 539,980 ms | $0.650$650estimate · DGX Spark |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Math 95.8% | — | Chat | 65.3%62.5%–68.1% | c | 735 / 1,125 | Not measured | Not measured | — | 16.1%answered 94% | 16,560 ms | 178,680 ms | $1.772$1,772estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Math 96.3% | — | Chat | 60.2%57.3%–63.0% | cd | 677 / 1,125 | Not measured | Not measured | — | 58.3%answered 70% | 138,140 ms† | 403,380 ms† | $4.664$4,664estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Math 90.8% | — | Chat | 54.2%51.3%–57.1% | de | 610 / 1,125 | Not measured | Not measured | — | 30.7%answered 84% | 10,150 ms | 18,410 ms | $0.390$390estimate · RTX PRO 6000 Blackwell |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.774 Math 66.3% | — | Decision | 50.6%47.7%–53.5% | ef | 569 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.774 | 65.90chosen-answer Brier 0.1878 | 8.8%answered 95% | 68 ms | 88 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 10 answers, 10 right · 0 wrong · AUROC 0.753 Math 59.8% | — | Decision | 48.2%45.3%–51.1% | efg | 542 / 1,125 | 10 right of 10 kept0 mistakes · 1,115 skipped | 0.753 | 64.33chosen-answer Brier 0.1952 | 23.4%answered 87% | 52 ms | 66 ms | $0.039$39.35estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 32 answers, 32 right · 0 wrong · AUROC 0.749 Math 57.9% | — | Decision | 47.8%44.9%–50.7% | efg | 538 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.749 | 63.95chosen-answer Brier 0.1993 | 30.4%answered 82% | 208 ms | 251 ms | $0.013$13.29estimate · DGX Spark |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 14 answers, 14 right · 0 wrong · AUROC 0.762 Math 58.4% | — | Decision | 46.6%43.7%–49.5% | fg | 524 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.762 | 63.55chosen-answer Brier 0.1947 | 17.4%answered 90% | 160 ms | 175 ms | $0.010$10.04estimate · DGX Spark |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 55 answers, 54 right · 1 wrong · AUROC 0.771 Math 62.1% | — | Decision | 46.4%43.5%–49.3% | fg | 522 / 1,125 | 54 right of 55 kept1 wrong · 1,070 skipped | 0.771 | 63.48chosen-answer Brier 0.1945 | 12.2%answered 93% | 41 ms | 46 ms | $0.031$31.36estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 53 answers, 53 right · 0 wrong · AUROC 0.768 Math 61.3% | — | Decision | 46.4%43.5%–49.3% | fg | 522 / 1,125 | 53 right of 53 kept0 mistakes · 1,072 skipped | 0.768 | 63.39chosen-answer Brier 0.1962 | 75.6%answered 47% | 79 ms | 98 ms | $0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 56 answers, 56 right · 0 wrong · AUROC 0.759 Math 58.8% | — | Decision | 45.3%42.4%–48.3% | fgh | 510 / 1,125 | 56 right of 56 kept0 mistakes · 1,069 skipped | 0.759 | 61.81chosen-answer Brier 0.2171 | 12.4%answered 93% | 47 ms | 53 ms | $0.034$33.81estimate · RTX PRO 6000 Blackwell |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 37 answers, 35 right · 2 wrong · AUROC 0.701 Math 55.3% | — | Decision | 44.4%41.5%–47.3% | ghi | 499 / 1,125 | 35 right of 37 kept2 wrong · 1,088 skipped | 0.701 | 61.42chosen-answer Brier 0.2151 | 13.6%answered 92% | 60 ms | 67 ms | $0.042$42.42estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 30 answers, 30 right · 0 wrong · AUROC 0.725 Math 59.8% | — | Decision | 44.3%41.4%–47.2% | ghi | 498 / 1,125 | 30 right of 30 kept0 mistakes · 1,095 skipped | 0.725 | 61.97chosen-answer Brier 0.2032 | 6.7%answered 96% | 50 ms | 61 ms | $0.036$36.27estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.730 Math 60.1% | — | Decision | 43.9%41.0%–46.8% | ghi | 494 / 1,125 | No answers at this threshold | 0.730 | 61.90chosen-answer Brier 0.2011 | 19.5%answered 88% | 52 ms | 65 ms | $0.040$39.96estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 27 answers, 26 right · 1 wrong · AUROC 0.752 Math 56.0% | — | Decision | 43.3%40.4%–46.2% | ghi | 487 / 1,125 | 26 right of 27 kept1 wrong · 1,098 skipped | 0.752 | 60.41chosen-answer Brier 0.2247 | 6.0%answered 97% | 209 ms | 269 ms | $0.013$13.35estimate · DGX Spark |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 32 answers, 32 right · 0 wrong · AUROC 0.727 Math 46.9% | — | Decision | 39.7%36.9%–42.6% | hi | 447 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.727 | 58.43chosen-answer Brier 0.2287 | 29.7%answered 80% | 66 ms | 91 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.720 Math 44.3% | — | Decision | 39.3%36.5%–42.2% | i | 442 / 1,125 | No answers at this threshold | 0.720 | 58.97chosen-answer Brier 0.2135 | 11.3%answered 93% | 40 ms | 48 ms | $0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.659 Math 49.8% | — | Decision | 38.9%36.1%–41.8% | i | 438 / 1,125 | No answers at this threshold | 0.659 | 57.70chosen-answer Brier 0.2353 | 14.4%answered 91% | 110 ms | 137 ms | $0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
PelaAI Poindexter Labs Perplexity vLLM Semantic Router Fastino Nace.AI Turion.AI Cloudflare OmniJev Zefan Cai MachinePulse Base Labs Sthānika AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 59.0%1,125 | 78.3%54.6% kept (614) | 89.4%37.1% kept (417) | 96.1%20.7% kept (233) | 100.0%7.9% kept (89) |
| Copernicus-31B Open · 31B | — | Decision | 58.7%1,125 | 82.3%47.6% kept (536) | 91.9%30.6% kept (344) | 99.0%17.2% kept (193) | 100.0%5.9% kept (66) |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 54.8%1,125 | 70.5%63.6% kept (715) | 82.3%41.7% kept (469) | 93.2%22.3% kept (251) | 100.0%5.5% kept (62) |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 58.9%1,125 | 79.2%57.8% kept (650) | 89.1%38.2% kept (430) | 97.4%20.9% kept (235) | 100.0%4.8% kept (54) |
| d3 Open · 26B | — | Decision | 54.5%1,125 | 75.4%53.1% kept (597) | 90.5%31.7% kept (357) | 95.8%17.1% kept (192) | 100.0%4.8% kept (54) |
| GLiDE v1.2 Open · 27B | — | Decision | 55.2%1,125 | 71.2%59.6% kept (671) | 89.2%33.6% kept (378) | 97.7%15.7% kept (177) | 100.0%4.0% kept (45) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 41.9%1,125 | 76.8%22.2% kept (250) | 82.2%12.0% kept (135) | 93.1%7.7% kept (87) | 100.0%2.9% kept (33) |
| d3-ultra Open · 26B | — | Decision | 54.6%1,125 | 62.7%75.7% kept (852) | 73.6%50.9% kept (573) | 93.0%25.3% kept (285) | 100.0%2.5% kept (28) |
| Turion Jev 31B Open · 31B | — | Decision | 52.3%1,125 | 81.5%37.9% kept (426) | 92.5%23.6% kept (266) | 97.7%11.6% kept (130) | 100.0%0.6% kept (7) |
| Clef Open · 27B | — | Decision | 40.8%1,125 | 53.0%54.5% kept (613) | 65.5%25.8% kept (290) | 85.3%9.1% kept (102) | 100.0%0.5% kept (6) |
| OneJev-27B Open · 27B | — | Decision | 40.6%1,125 | 74.4%18.4% kept (207) | 91.1%10.0% kept (112) | 100.0%4.6% kept (52) | 100.0%0.4% kept (4) |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 42.6%1,125 | 95.5%5.9% kept (66) | 100.0%0.4% kept (4) | none kept | none kept |
| Karpo Decision 31B Open · 31B | — | Decision | 60.7%1,125 | 75.3%65.2% kept (734) | 89.4%44.5% kept (501) | 96.3%28.8% kept (324) | 98.3%15.3% kept (172) |
| Drex Max Open · 27B | — | Decision | 53.1%1,125 | 63.7%72.1% kept (811) | 74.1%49.7% kept (559) | 86.5%30.8% kept (347) | 94.8%13.8% kept (155) |
| Mammoth-1 Open · 27B | — | Decision | 54.5%1,125 | 74.1%55.3% kept (622) | 86.3%36.3% kept (408) | 95.4%21.3% kept (240) | 98.1%9.6% kept (108) |
| Sieve 31B v3 Open · 31B | — | Decision | 57.9%1,125 | 78.1%53.2% kept (599) | 89.3%35.7% kept (402) | 96.3%21.6% kept (243) | 97.5%7.2% kept (81) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 52.5%1,125 | 75.0%49.3% kept (555) | 88.5%29.3% kept (330) | 95.7%16.7% kept (188) | 98.7%7.0% kept (79) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 53.3%1,125 | 77.0%47.6% kept (535) | 90.7%28.8% kept (324) | 95.9%17.2% kept (193) | 97.5%7.1% kept (80) |
| GLiDE 27B Open · 27B | — | Decision | 58.4%1,125 | 76.8%60.3% kept (678) | 88.6%38.3% kept (431) | 97.7%18.9% kept (213) | 98.7%6.7% kept (75) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 89 / 00.804 | 25 / 00.765 | 14 / 00.764 | 50 / 00.904 |
| Copernicus-31B Open · 31B | — | Decision | 66 / 00.801 | 15 / 00.746 | 12 / 00.775 | 39 / 00.935 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 62 / 00.803 | 22 / 00.752 | 5 / 00.764 | 35 / 00.878 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 54 / 00.825 | 26 / 00.792 | 5 / 00.785 | 23 / 00.907 |
| d3 Open · 26B | — | Decision | 54 / 00.806 | 3 / 00.755 | 11 / 00.761 | 40 / 00.863 |
| GLiDE v1.2 Open · 27B | — | Decision | 45 / 00.793 | 3 / 00.732 | 6 / 00.739 | 36 / 00.906 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 33 / 00.717 | 3 / 00.652 | 0 / 00.690 | 30 / 00.815 |
| d3-ultra Open · 26B | — | Decision | 28 / 00.778 | 9 / 00.707 | 10 / 00.761 | 9 / 00.853 |
| Turion Jev 31B Open · 31B | — | Decision | 7 / 00.782 | 1 / 00.702 | 0 / 00.756 | 6 / 00.875 |
| Clef Open · 27B | — | Decision | 6 / 00.697 | 2 / 00.607 | 0 / 00.704 | 4 / 00.846 |
| OneJev-27B Open · 27B | — | Decision | 4 / 00.726 | 0 / 00.633 | 0 / 00.712 | 4 / 00.845 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | No answers at this threshold0.733 | 0 / 00.623 | 0 / 00.716 | 0 / 00.861 |
| Karpo Decision 31B Open · 31B | — | Decision | 169 / 30.816 | 58 / 00.805 | 33 / 20.758 | 78 / 10.877 |
| Drex Max Open · 27B | — | Decision | 147 / 80.786 | 34 / 00.729 | 39 / 50.758 | 74 / 30.838 |
| Mammoth-1 Open · 27B | — | Decision | 106 / 20.795 | 21 / 00.703 | 13 / 00.797 | 72 / 20.905 |
| Sieve 31B v3 Open · 31B | — | Decision | 79 / 20.797 | 9 / 00.764 | 16 / 20.745 | 54 / 00.861 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 78 / 10.800 | 8 / 00.728 | 12 / 00.787 | 58 / 10.860 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 78 / 20.798 | 9 / 00.730 | 12 / 00.783 | 57 / 20.791 |
| GLiDE 27B Open · 27B | — | Decision | 74 / 10.822 | 6 / 00.754 | 17 / 00.800 | 51 / 10.888 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 3 setups: Qwen3.8-27B (low · chat), Gemma 4 31B (chat), LFM2.5-24B-A2B (on · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Math 99.2% | — | Chat | 94.0%92.5%–95.3% | a | 1,058 / 1,125 | Not measured | Not measured | — | 39.1%answered 97% | 18,710 ms | 101,030 ms | $4.884$4,884estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B | — | Chat | 93.3%91.7%–94.6% | a | 1,050 / 1,125 | Not measured | Not measured | — | 16.3%answered 99% | 18,290 ms | 39,790 ms | $1.244$1,244estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Math 99.0% | — | Chat | 86.7%84.6%–88.5% | b | 975 / 1,125 | Not measured | Not measured | — | 37.8%answered 95% | 228,980 ms | 776,930 ms | $7.562$7,562estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 172 answers, 169 right · 3 wrong · AUROC 0.816 Math 81.6% | — | Decision | 60.7%57.8%–63.5% | c | 683 / 1,125 | 169 right of 172 kept3 wrong · 953 skipped | 0.816 | 71.80chosen-answer Brier 0.1711 | 6.6%answered 97% | 173 ms | 211 ms | $0.128$128estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 89 answers, 89 right · 0 wrong · AUROC 0.804 Math 77.4% | — | Decision | 59.0%56.1%–61.9% | cd | 664 / 1,125 | 89 right of 89 kept0 mistakes · 1,036 skipped | 0.804 | 70.72chosen-answer Brier 0.1757 | 0.4%answered 100% | 983 ms | 1,279 ms | $0.061$61.43estimate · DGX Spark |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.825 Math 80.3% | — | Decision | 58.9%56.0%–61.8% | cd | 663 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.825 | 71.09chosen-answer Brier 0.1675 | 0.0%answered 100% | 151 ms | 204 ms | $0.112$112estimate · RTX PRO 6000 Blackwell |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 66 answers, 66 right · 0 wrong · AUROC 0.801 Math 77.3% | — | Decision | 58.7%55.8%–61.5% | cd | 660 / 1,125 | 66 right of 66 kept0 mistakes · 1,059 skipped | 0.801 | 70.48chosen-answer Brier 0.1771 | 1.9%answered 99% | 185 ms | 255 ms | $0.136$136estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 75 answers, 74 right · 1 wrong · AUROC 0.822 Math 72.0% | — | Decision | 58.4%55.5%–61.2% | cd | 657 / 1,125 | 74 right of 75 kept1 wrong · 1,050 skipped | 0.822 | 70.67chosen-answer Brier 0.1707 | 44.3%answered 79% | 544 ms | 655 ms | $0.033$33.16estimate · DGX Spark |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 81 answers, 79 right · 2 wrong · AUROC 0.797 Math 77.1% | — | Decision | 57.9%55.0%–60.7% | cde | 651 / 1,125 | 79 right of 81 kept2 wrong · 1,044 skipped | 0.797 | 70.03chosen-answer Brier 0.1781 | 5.5%answered 98% | 1,050 ms | 1,335 ms | $0.063$62.95estimate · DGX Spark |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 45 answers, 45 right · 0 wrong · AUROC 0.793 Math 69.9% | — | Decision | 55.2%52.3%–58.1% | def | 621 / 1,125 | 45 right of 45 kept0 mistakes · 1,080 skipped | 0.793 | 68.39chosen-answer Brier 0.1842 | 50.2%answered 73% | 111 ms | 140 ms | $0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 62 answers, 62 right · 0 wrong · AUROC 0.803 Math 72.5% | — | Decision | 54.8%51.8%–57.6% | def | 616 / 1,125 | 62 right of 62 kept0 mistakes · 1,063 skipped | 0.803 | 67.98chosen-answer Brier 0.1881 | 18.1%answered 91% | 108 ms | 131 ms | $0.081$81.16estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 28 answers, 28 right · 0 wrong · AUROC 0.778 Math 71.5% | — | Decision | 54.6%51.7%–57.5% | def | 614 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.778 | 66.46chosen-answer Brier 0.2166 | 10.1%answered 95% | 156 ms | 185 ms | $0.109$109estimate · RTX PRO 6000 Blackwell |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 108 answers, 106 right · 2 wrong · AUROC 0.795 Math 74.0% | — | Decision | 54.5%51.6%–57.4% | def | 613 / 1,125 | 106 right of 108 kept2 wrong · 1,017 skipped | 0.795 | 68.07chosen-answer Brier 0.1835 | 72.3%answered 58% | 221 ms | 244 ms | $0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.806 Math 70.4% | — | Decision | 54.5%51.6%–57.4% | def | 613 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.806 | 68.34chosen-answer Brier 0.1781 | 14.8%answered 93% | 76 ms | 98 ms | $0.057$57.18estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 80 answers, 78 right · 2 wrong · AUROC 0.798 Math 69.8% | — | Decision | 53.3%50.4%–56.2% | ef | 600 / 1,125 | 78 right of 80 kept2 wrong · 1,045 skipped | 0.798 | 67.64chosen-answer Brier 0.1805 | 14.4%answered 93% | 238 ms | 552 ms | $0.007$7.46 |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 155 answers, 147 right · 8 wrong · AUROC 0.786 Math 70.3% | — | Decision | 53.1%50.1%–56.0% | ef | 597 / 1,125 | 147 right of 155 kept8 wrong · 970 skipped | 0.786 | 65.68chosen-answer Brier 0.2170 | 7.3%answered 97% | 106 ms | 143 ms | $0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 79 answers, 78 right · 1 wrong · AUROC 0.800 Math 69.3% | with web | Decision | 52.5%49.6%–55.4% | f | 591 / 1,125 | 78 right of 79 kept1 wrong · 1,046 skipped | 0.800 | 67.24chosen-answer Brier 0.1806 | 37.7%answered 80% | 448 ms | 827 ms | $1.030$1,030 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.782 Math 69.0% | — | Decision | 52.3%49.3%–55.2% | f | 588 / 1,125 | 7 right of 7 kept0 mistakes · 1,118 skipped | 0.782 | 66.89chosen-answer Brier 0.1849 | 1.5%answered 99% | 385 ms | 533 ms | $0.025$24.91estimate · DGX Spark |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.733 Math 54.6% | — | Decision | 42.6%39.7%–45.5% | g | 479 / 1,125 | No answers at this threshold | 0.733 | 59.78chosen-answer Brier 0.2302 | 28.2%answered 82% | 413 ms | 576 ms | $0.294$294estimate · RTX PRO 6000 Blackwell |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 33 answers, 33 right · 0 wrong · AUROC 0.717 Math 59.0% | — | Decision | 41.9%39.0%–44.8% | g | 471 / 1,125 | 33 right of 33 kept0 mistakes · 1,092 skipped | 0.717 | 60.73chosen-answer Brier 0.2040 | 37.7%answered 75% | 345 ms | 504 ms | $0.018$18.41 |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.697 Math 51.1% | — | Decision | 40.8%38.0%–43.7% | g | 459 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.697 | 58.55chosen-answer Brier 0.2369 | 3.6%answered 98% | 196 ms | 222 ms | $0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.726 Math 53.8% | — | Decision | 40.6%37.8%–43.5% | g | 457 / 1,125 | 4 right of 4 kept0 mistakes · 1,121 skipped | 0.726 | 60.27chosen-answer Brier 0.2009 | 1.2%answered 99% | 108 ms | 136 ms | $0.086$86.07estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
REAX Logits Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | Decision | 59.3%1,125 | 77.6%58.3% kept (656) | 90.8%34.7% kept (390) | 98.2%19.6% kept (221) | 100.0%4.5% kept (51) |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 57.8%1,125 | 75.4%57.5% kept (647) | 92.0%31.0% kept (349) | 97.6%14.6% kept (164) | 100.0%0.7% kept (8) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | Decision | 51 / 00.816 | 12 / 00.750 | 11 / 00.804 | 28 / 00.921 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 8 / 00.796 | 1 / 00.738 | 1 / 00.764 | 6 / 00.880 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 7 setups: DeepSeek-V4.1-Flash (high · chat · 1017), Inkling 2 Small (high · chat), Inkling-Small (high · chat · Jul 2026), MiMo-V2.6-Flash (on · chat), Qwen3.5-397B-A17B (on · chat), MiniMax-M3.1-Flash (high · chat), GLM-5.3-Flash (high · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): DeepSeek-V4-Flash (chat) (sample: a random subset by design, 317 answers stored, not the full 1,125).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Math 99.6% | — | Chat | 96.2%94.9%–97.2% | a | 1,082 / 1,125 | Not measured | Not measured | — | 3.8%answered 100% | 6,680 ms | 40,040 ms | $1.647$1,647 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Math 99.5% | — | Chat | 95.6%94.3%–96.7% | ab | 1,076 / 1,125 | Not measured | Not measured | — | 37.3%answered 98% | 13,180 ms | 144,250 ms | $4.121$4,121 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Math 99.6% | — | Chat | 95.4%94.0%–96.5% | ab | 1,073 / 1,125 | Not measured | Not measured | — | 50.0%answered 97% | 13,290 ms | 78,510 ms | $5.489$5,489 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Math 99.4% | — | Chat | 95.1%93.7%–96.2% | abc | 1,070 / 1,125 | Not measured | Not measured | — | 23.3%answered 98% | 24,740 ms | 179,930 ms | $1.390$1,390 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active Qwen3.5-397B-A17B on · chat · Alibaba · chat model · reasoning on Parameters: 397B MoE · 17B active Math running | — | Chat | 95.1%93.7%–96.2% | abc | 1,070 / 1,125 | Not measured | Not measured | — | 31.7%answered 98% | 84,650 ms | 167,520 ms | $13.426$13,426 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Math 98.2% | — | Chat | 94.0%92.4%–95.2% | bc | 1,057 / 1,125 | Not measured | Not measured | — | 38.3%answered 97% | 5,610 ms | 37,430 ms | $0.803$803 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Math 98.8% | — | Chat | 93.2%91.5%–94.5% | c | 1,048 / 1,125 | Not measured | Not measured | — | 4.2%answered 100% | 12,050 ms | 82,830 ms | $1.057$1,057 |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 51 answers, 51 right · 0 wrong · AUROC 0.816 Math 78.0% | — | Decision | 59.3%56.4%–62.1% | d | 667 / 1,125 | 51 right of 51 kept0 mistakes · 1,074 skipped | 0.816 | 71.07chosen-answer Brier 0.1714 | 5.3%answered 98% | 215 ms | 278 ms | $0.160$160estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.796 Math 75.2% | — | Decision | 57.8%54.9%–60.6% | d | 650 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.796 | 69.90chosen-answer Brier 0.1798 | 7.1%answered 97% | 285 ms | 399 ms | $0.211$211estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
TypeSafe Fastino Liquid AI Google Microsoft Celeris Nace.AI Inception Subquadratic Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Jev 1.13 Closed · Efficient | — | Decision | 46.3%1,125 | 68.4%44.7% kept (503) | 83.5%22.6% kept (254) | 94.4%11.2% kept (126) | 100.0%5.8% kept (65) |
| Jev 1.14 Closed · Efficient | — | Decision | 48.2%1,125 | 72.6%39.9% kept (449) | 84.2%21.4% kept (241) | 92.6%10.8% kept (121) | 100.0%5.4% kept (61) |
| GLiDE API Closed · Efficient | — | Decision | 54.5%1,125 | 72.9%58.5% kept (658) | 85.0%36.1% kept (406) | 96.8%16.7% kept (188) | 100.0%4.6% kept (52) |
| d1.1 Closed · Efficient | — | Decision | 46.0%1,125 | 70.3%39.6% kept (445) | 84.6%22.5% kept (253) | 94.7%11.7% kept (132) | 100.0%4.3% kept (48) |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 50.0%1,125 | 79.4%34.5% kept (388) | 91.5%18.9% kept (213) | 97.5%10.8% kept (121) | 100.0%3.8% kept (43) |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 44.4%1,125 | 61.9%43.9% kept (494) | 78.7%19.6% kept (221) | 94.4%8.0% kept (90) | 100.0%1.8% kept (20) |
| d1-ultra Closed · Efficient | — | Decision | 54.9%1,125 | 76.4%48.5% kept (546) | 86.2%29.7% kept (334) | 96.6%15.6% kept (176) | 100.0%1.8% kept (20) |
| celeris-1-decision Closed · Efficient | — | Decision | 47.3%1,125 | 61.9%53.7% kept (604) | 76.5%30.6% kept (344) | 90.5%15.9% kept (179) | none kept |
| celeris-1.1-decision Closed · Efficient | — | Decision | 47.3%1,125 | 62.9%52.7% kept (593) | 76.8%29.9% kept (336) | 90.8%16.4% kept (184) | none kept |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 54.8%1,125 | 71.8%58.0% kept (652) | 86.2%35.6% kept (400) | 95.5%19.9% kept (224) | 98.9%8.0% kept (90) |
| Mercury Decide Closed · Efficient | — | Decision | 45.1%1,125 | 61.0%53.6% kept (603) | 70.3%38.0% kept (428) | 79.9%22.1% kept (249) | 92.3%8.1% kept (91) |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 52.4%1,125 | 65.0%66.5% kept (748) | 77.9%42.2% kept (475) | 88.7%23.6% kept (266) | 97.2%6.4% kept (72) |
| d1 Closed · Efficient | — | Decision | 40.8%1,125 | 59.9%41.0% kept (461) | 74.2%21.7% kept (244) | 87.9%10.3% kept (116) | 98.2%5.0% kept (56) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|---|
| Jev 1.13 Closed · Efficient | — | Decision | 65 / 00.764 | 7 / 00.708 | 5 / 00.714 | 53 / 00.875 |
| Jev 1.14 Closed · Efficient | — | Decision | 61 / 00.750 | 5 / 00.671 | 8 / 00.711 | 48 / 00.856 |
| GLiDE API Closed · Efficient | — | Decision | 52 / 00.814 | 7 / 00.777 | 7 / 00.765 | 38 / 00.866 |
| d1.1 Closed · Efficient | — | Decision | 48 / 00.748 | 4 / 00.676 | 6 / 00.712 | 38 / 00.906 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 43 / 00.768 | 5 / 00.766 | 0 / 00.699 | 38 / 00.885 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 20 / 00.712 | 1 / 00.628 | 2 / 00.683 | 17 / 00.823 |
| d1-ultra Closed · Efficient | — | Decision | 20 / 00.783 | 12 / 00.759 | 3 / 00.715 | 5 / 00.846 |
| celeris-1-decision Closed · Efficient | — | Decision | No answers at this threshold0.735 | 0 / 00.706 | 0 / 00.679 | 0 / 00.867 |
| celeris-1.1-decision Closed · Efficient | — | Decision | No answers at this threshold0.733 | 0 / 00.709 | 0 / 00.679 | 0 / 00.853 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 89 / 10.793 | 7 / 00.729 | 24 / 00.771 | 58 / 10.877 |
| Mercury Decide Closed · Efficient | — | Decision | 84 / 70.746 | 14 / 20.670 | 28 / 10.744 | 42 / 40.775 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 70 / 20.787 | 17 / 00.736 | 18 / 10.743 | 35 / 10.857 |
| d1 Closed · Efficient | — | Decision | 55 / 10.723 | 3 / 00.630 | 5 / 00.683 | 47 / 10.828 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 2 setups: Muse Ember (max · chat), GPT-6 Luna (chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 1,125 of 1,125); shown once.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)); Claude Haiku 5.5 (max · chat) (paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Math 99.6% | — | Chat | 95.5%94.1%–96.5% | a | 1,074 / 1,125 | Not measured | Not measured | — | 22.6%answered 99% | 3,110 ms | 14,020 ms | $1.820$1,820 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Math 93.7% | — | Chat | 58.0%55.0%–60.8% | b | 652 / 1,125 | Not measured | Not measured | — | 55.1%answered 72% | 3,860 ms | 5,820 ms | $0.000$0.00 |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 20 answers, 20 right · 0 wrong · AUROC 0.783 Math 74.1% | — | Decision | 54.9%52.0%–57.8% | bc | 618 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.783 | 68.09chosen-answer Brier 0.1876 | 91.0%answered 35% | 182 ms | 425 ms | $0.013$12.64 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 90 answers, 89 right · 1 wrong · AUROC 0.793 Math 70.1% | — | Decision | 54.8%51.8%–57.6% | bc | 616 / 1,125 | 89 right of 90 kept1 wrong · 1,035 skipped | 0.793 | 68.07chosen-answer Brier 0.1861 | 54.1%answered 69% | 329 ms | 1,540 ms | $0.022$22.37 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 52 answers, 52 right · 0 wrong · AUROC 0.814 Math 70.8% | — | Decision | 54.5%51.6%–57.4% | bc | 613 / 1,125 | 52 right of 52 kept0 mistakes · 1,073 skipped | 0.814 | 68.27chosen-answer Brier 0.1794 | 60.0%answered 67% | 271 ms | 291 ms | $0.052$52.03 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 72 answers, 70 right · 2 wrong · AUROC 0.787 Math 68.6% | — | Decision | 52.4%49.4%–55.3% | bcd | 589 / 1,125 | 70 right of 72 kept2 wrong · 1,053 skipped | 0.787 | 65.94chosen-answer Brier 0.2047 | 65.7%answered 61% | 384 ms | 393 ms | $0.015$14.92 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 43 answers, 43 right · 0 wrong · AUROC 0.768 Math 71.3% | — | Decision | 50.0%47.1%–53.0% | cde | 563 / 1,125 | 43 right of 43 kept0 mistakes · 1,082 skipped | 0.768 | 65.37chosen-answer Brier 0.1930 | 2.0%answered 99% | 228 ms | 344 ms | $0.037$36.77 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 61 answers, 61 right · 0 wrong · AUROC 0.750 Math 65.1% | — | Decision | 48.2%45.3%–51.1% | de | 542 / 1,125 | 61 right of 61 kept0 mistakes · 1,064 skipped | 0.750 | 64.14chosen-answer Brier 0.1990 | 24.5%answered 86% | 166 ms | 193 ms | $0.026$25.64 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.735 Math 70.0% | — | Decision | 47.3%44.4%–50.2% | de | 532 / 1,125 | No answers at this threshold | 0.735 | 62.94chosen-answer Brier 0.2141 | 92.3%answered 28% | 390 ms | 394 ms | $0.015$15.40 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.733 Math 69.9% | — | Decision | 47.3%44.4%–50.2% | de | 532 / 1,125 | No answers at this threshold | 0.733 | 62.92chosen-answer Brier 0.2144 | 91.0%answered 29% | 383 ms | 390 ms | $0.002$1.54 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 65 answers, 65 right · 0 wrong · AUROC 0.764 Math 64.1% | — | Decision | 46.3%43.4%–49.2% | e | 521 / 1,125 | 65 right of 65 kept0 mistakes · 1,060 skipped | 0.764 | 63.36chosen-answer Brier 0.1959 | 15.8%answered 91% | 167 ms | 203 ms | $0.026$25.64 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 48 answers, 48 right · 0 wrong · AUROC 0.748 Math 65.2% | — | Decision | 46.0%43.2%–49.0% | e | 518 / 1,125 | 48 right of 48 kept0 mistakes · 1,077 skipped | 0.748 | 63.14chosen-answer Brier 0.1977 | 90.6%answered 28% | 176 ms | 272 ms | $0.006$6.02 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 91 answers, 84 right · 7 wrong · AUROC 0.746 Math 53.5% | — | Decision | 45.1%42.2%–48.0% | ef | 507 / 1,125 | 84 right of 91 kept7 wrong · 1,034 skipped | 0.746 | 61.18chosen-answer Brier 0.2272 | 20.5%answered 88% | 301 ms | 360 ms | $0.008$7.52 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 20 answers, 20 right · 0 wrong · AUROC 0.712 Math 59.4% | — | Decision | 44.4%41.6%–47.4% | ef | 500 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.712 | 61.53chosen-answer Brier 0.2139 | 19.8%answered 88% | 315 ms | 362 ms | $0.013$13.42 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 56 answers, 55 right · 1 wrong · AUROC 0.723 Math 55.6% | — | Decision | 40.8%38.0%–43.7% | f | 459 / 1,125 | 55 right of 56 kept1 wrong · 1,069 skipped | 0.723 | 59.76chosen-answer Brier 0.2128 | 11.3%answered 93% | 188 ms | 290 ms | $0.006$6.02 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.7% | — | Chat | 95.5%94.1%–96.5% | a | 1,074 / 1,125 | Not measured | Not measured | — | 41.3%answered 98% | 6,710 ms | 14,890 ms | $11.480$11,480 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 147 of 1,125 stored, last answer 12 Nov 11:32 UTC).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.8% | — | Chat | 96.2%94.9%–97.2% | a | 1,082 / 1,125 | Not measured | Not measured | — | 38.2%answered 98% | 5,830 ms | 12,240 ms | $19.540$19,540 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.7% | — | Chat | 95.2%93.8%–96.3% | b | 1,071 / 1,125 | Not measured | Not measured | — | 8.3%answered 100% | 3,820 ms | 13,460 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Liquid AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | Decision | 49.2%1,125 | 73.8%35.6% kept (400) | 87.5%19.9% kept (224) | 95.3%11.4% kept (128) | 100.0%2.5% kept (28) |
| d1.2-ultra Closed · Undisclosed | — | Decision | 51.6%1,125 | 77.1%44.7% kept (503) | 88.6%28.2% kept (317) | 96.9%14.4% kept (162) | 100.0%1.2% kept (13) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | Decision | 28 / 00.742 | 3 / 00.676 | 2 / 00.719 | 23 / 00.874 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 13 / 00.800 | 4 / 00.741 | 4 / 00.772 | 5 / 00.837 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 6 setups: GPT-6 Astra (chat), Gemini 4 Carbon (chat), GPT-6.1 Sol (chat), Qwen4-Max (high · chat), GPT-6.1 Terra (chat), Qwen3.8 Flash (off · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Grok 4.7 (chat) (paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Math 99.6% | — | Chat | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 56.4%answered 97% | 6,590 ms | 14,330 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Math 99.6% | — | Chat | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 45.5%answered 98% | 44,520 ms | 206,660 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Math 99.6% | — | Chat | 96.5%95.3%–97.5% | a | 1,086 / 1,125 | Not measured | Not measured | — | 63.2%answered 97% | 7,070 ms | 16,200 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Math 99.8% | — | Chat | 96.3%95.0%–97.2% | a | 1,083 / 1,125 | Not measured | Not measured | — | 0.0%answered 100% | 11,570 ms | 38,640 ms | $5.690$5,690 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Math 99.2% | — | Chat | 94.0%92.4%–95.2% | b | 1,057 / 1,125 | Not measured | Not measured | — | 61.1%answered 95% | 5,640 ms | 10,330 ms | $0.000$0.00 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Math 99.5% | — | Chat | 94.0%92.4%–95.2% | b | 1,057 / 1,125 | Not measured | Not measured | — | 34.6%answered 98% | 17,550 ms | 73,220 ms | $1.376$1,376 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.800 Math 69.9% | — | Decision | 51.6%48.6%–54.5% | c | 580 / 1,125 | 13 right of 13 kept0 mistakes · 1,112 skipped | 0.800 | 66.77chosen-answer Brier 0.1802 | 94.5%answered 28% | 220 ms | 341 ms | $0.013$12.64 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 28 answers, 28 right · 0 wrong · AUROC 0.742 Math 65.5% | — | Decision | 49.2%46.2%–52.1% | c | 553 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.742 | 64.51chosen-answer Brier 0.2013 | 78.8%answered 43% | 222 ms | 390 ms | $0.006$6.02 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
PelaAI Poindexter Labs TypeSafe Perplexity Patrick-18 vLLM Semantic Router Base Labs Fastino REAX Liquid AI Google quincyqiang victorbennett Nace.AI EldanRing StartLux Microsoft tattrongvu Logits Turion.AI Cloudflare scar-ai OmniJev Blockbrain Labs Other Metask Lab Zefan Cai Strands Agents Celeris MachinePulse Inception Sthānika AI Subquadratic newsbang Valen-Team Apertix AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | — | 59.0%1,125 | 78.3%54.6% kept (614) | 89.4%37.1% kept (417) | 96.1%20.7% kept (233) | 100.0%7.9% kept (89) |
| Copernicus-31B Open · 31B | — | 58.7%1,125 | 82.3%47.6% kept (536) | 91.9%30.6% kept (344) | 99.0%17.2% kept (193) | 100.0%5.9% kept (66) |
| Jev 1.13 Closed · Efficient | — | 46.3%1,125 | 68.4%44.7% kept (503) | 83.5%22.6% kept (254) | 94.4%11.2% kept (126) | 100.0%5.8% kept (65) |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 54.8%1,125 | 70.5%63.6% kept (715) | 82.3%41.7% kept (469) | 93.2%22.3% kept (251) | 100.0%5.5% kept (62) |
| Jev 1.14 Closed · Efficient | — | 48.2%1,125 | 72.6%39.9% kept (449) | 84.2%21.4% kept (241) | 92.6%10.8% kept (121) | 100.0%5.4% kept (61) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 45.3%1,125 | 58.8%58.8% kept (662) | 74.4%34.4% kept (387) | 91.7%15.0% kept (169) | 100.0%5.0% kept (56) |
| KnowLine-31B-Gen2 Open · 31B | — | 58.9%1,125 | 79.2%57.8% kept (650) | 89.1%38.2% kept (430) | 97.4%20.9% kept (235) | 100.0%4.8% kept (54) |
| d3 Open · 26B | — | 54.5%1,125 | 75.4%53.1% kept (597) | 90.5%31.7% kept (357) | 95.8%17.1% kept (192) | 100.0%4.8% kept (54) |
| Elephant-1 Open · 9.0B | — | 46.4%1,125 | 66.5%47.8% kept (538) | 84.2%27.0% kept (304) | 94.9%14.0% kept (157) | 100.0%4.7% kept (53) |
| GLiDE API Closed · Efficient | — | 54.5%1,125 | 72.9%58.5% kept (658) | 85.0%36.1% kept (406) | 96.8%16.7% kept (188) | 100.0%4.6% kept (52) |
| REAX 1.3 Open · 57B | — | 59.3%1,125 | 77.6%58.3% kept (656) | 90.8%34.7% kept (390) | 98.2%19.6% kept (221) | 100.0%4.5% kept (51) |
| d1.1 Closed · Efficient | — | 46.0%1,125 | 70.3%39.6% kept (445) | 84.6%22.5% kept (253) | 94.7%11.7% kept (132) | 100.0%4.3% kept (48) |
| GLiDE v1.2 Open · 27B | — | 55.2%1,125 | 71.2%59.6% kept (671) | 89.2%33.6% kept (378) | 97.7%15.7% kept (177) | 100.0%4.0% kept (45) |
| Gemini Decision 1 20261111 Closed · Efficient | — | 50.0%1,125 | 79.4%34.5% kept (388) | 91.5%18.9% kept (213) | 97.5%10.8% kept (121) | 100.0%3.8% kept (43) |
| Chinese Decision Pointer 4B Open · 4.0B | — | 40.4%1,125 | 54.7%49.4% kept (556) | 68.8%25.6% kept (288) | 83.1%12.1% kept (136) | 100.0%3.5% kept (39) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 43.1%1,125 | 65.9%36.3% kept (408) | 82.4%18.7% kept (210) | 92.3%10.4% kept (117) | 100.0%3.1% kept (35) |
| Verdin-4B-v1.1 Open · 4.0B | — | 39.7%1,125 | 59.2%39.8% kept (448) | 73.6%20.9% kept (235) | 87.0%9.6% kept (108) | 100.0%3.1% kept (35) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 41.9%1,125 | 76.8%22.2% kept (250) | 82.2%12.0% kept (135) | 93.1%7.7% kept (87) | 100.0%2.9% kept (33) |
| GLiDE 9B Open · 7.9B | — | 47.8%1,125 | 66.4%47.4% kept (533) | 87.9%22.8% kept (256) | 97.5%10.7% kept (120) | 100.0%2.8% kept (32) |
| Winnow-12B Open · 12B | — | 39.7%1,125 | 57.3%45.2% kept (508) | 65.5%32.4% kept (365) | 74.5%19.2% kept (216) | 100.0%2.8% kept (32) |
| KnowLine-9B-Gen1 Open · 9.0B | — | 44.3%1,125 | 68.7%31.2% kept (351) | 85.3%16.4% kept (184) | 97.9%8.4% kept (95) | 100.0%2.7% kept (30) |
| d1.2 Closed · Undisclosed | — | 49.2%1,125 | 73.8%35.6% kept (400) | 87.5%19.9% kept (224) | 95.3%11.4% kept (128) | 100.0%2.5% kept (28) |
| d3-ultra Open · 26B | — | 54.6%1,125 | 62.7%75.7% kept (852) | 73.6%50.9% kept (573) | 93.0%25.3% kept (285) | 100.0%2.5% kept (28) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 45.9%1,125 | 68.4%36.8% kept (414) | 82.7%18.0% kept (202) | 97.8%8.3% kept (93) | 100.0%1.9% kept (21) |
| Microsoft-Decision-1 Closed · Efficient | — | 44.4%1,125 | 61.9%43.9% kept (494) | 78.7%19.6% kept (221) | 94.4%8.0% kept (90) | 100.0%1.8% kept (20) |
| d1-ultra Closed · Efficient | — | 54.9%1,125 | 76.4%48.5% kept (546) | 86.2%29.7% kept (334) | 96.6%15.6% kept (176) | 100.0%1.8% kept (20) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 44.7%1,125 | 63.9%47.6% kept (535) | 77.7%27.6% kept (310) | 92.2%11.5% kept (129) | 100.0%1.5% kept (17) |
| GLiDE 2B Open · 1.9B | — | 32.3%1,125 | 46.3%34.8% kept (391) | 54.7%15.1% kept (170) | 79.7%5.2% kept (59) | 100.0%1.2% kept (14) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 46.6%1,125 | 71.5%38.3% kept (431) | 84.6%19.6% kept (221) | 98.9%8.2% kept (92) | 100.0%1.2% kept (14) |
| d1.2-ultra Closed · Undisclosed | — | 51.6%1,125 | 77.1%44.7% kept (503) | 88.6%28.2% kept (317) | 96.9%14.4% kept (162) | 100.0%1.2% kept (13) |
| Jeffan-9B-1211 Open · 9.0B | — | 48.2%1,125 | 71.1%41.5% kept (467) | 86.4%22.9% kept (258) | 98.1%9.6% kept (108) | 100.0%0.9% kept (10) |
| GLiDE 0.8B Open · 0.8B | — | 29.2%1,125 | 46.1%24.1% kept (271) | 54.7%9.4% kept (106) | 62.5%3.6% kept (40) | 100.0%0.8% kept (9) |
| KnowLine-2B-Gen1 Open · 2.0B | — | 31.7%1,125 | 51.2%18.4% kept (207) | 69.1%7.2% kept (81) | 87.5%2.8% kept (32) | 100.0%0.7% kept (8) |
| Logits Sigma-1 Duo Open · 57B | — | 57.8%1,125 | 75.4%57.5% kept (647) | 92.0%31.0% kept (349) | 97.6%14.6% kept (164) | 100.0%0.7% kept (8) |
| Turion Jev 31B Open · 31B | — | 52.3%1,125 | 81.5%37.9% kept (426) | 92.5%23.6% kept (266) | 97.7%11.6% kept (130) | 100.0%0.6% kept (7) |
| Clef Open · 27B | — | 40.8%1,125 | 53.0%54.5% kept (613) | 65.5%25.8% kept (290) | 85.3%9.1% kept (102) | 100.0%0.5% kept (6) |
| KnowLine-12B-Gen1 Open · 12B | — | 50.6%1,125 | 74.3%44.6% kept (502) | 90.5%25.3% kept (285) | 99.1%9.7% kept (109) | 100.0%0.5% kept (6) |
| vera-medium-omni Open · 4.2B | — | 44.1%1,125 | 72.7%29.3% kept (330) | 85.6%11.1% kept (125) | 100.0%4.9% kept (55) | 100.0%0.5% kept (6) |
| OneJev-27B Open · 27B | — | 40.6%1,125 | 74.4%18.4% kept (207) | 91.1%10.0% kept (112) | 100.0%4.6% kept (52) | 100.0%0.4% kept (4) |
| d1-2B Open · 1.9B | — | 30.7%1,125 | 52.0%15.2% kept (171) | 73.3%5.3% kept (60) | 100.0%1.7% kept (19) | 100.0%0.3% kept (3) |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 26.7%1,125 | 47.4%10.1% kept (114) | 63.9%3.2% kept (36) | 92.9%1.2% kept (14) | 100.0%0.2% kept (2) |
| OneJev1.6-0.8B Open · 0.8B | — | 27.0%1,125 | 36.3%15.9% kept (179) | 44.6%5.0% kept (56) | 83.3%0.5% kept (6) | 100.0%0.2% kept (2) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 30.8%1,125 | 59.7%13.2% kept (149) | 73.8%5.4% kept (61) | 100.0%2.0% kept (22) | 100.0%0.2% kept (2) |
| Clef-Flash Open · 9.0B | — | 38.9%1,125 | 52.4%40.9% kept (460) | 59.7%20.5% kept (231) | 77.9%6.0% kept (68) | none kept |
| Cygnet Open · 12B | — | 39.3%1,125 | 58.1%43.3% kept (487) | 71.2%18.2% kept (205) | 89.5%5.1% kept (57) | none kept |
| GLiClass v4 Edge Open · 0.4B | — | 23.6%1,125 | 35.6%9.2% kept (104) | 30.8%2.3% kept (26) | 50.0%0.5% kept (6) | none kept |
| GLiDE 0.4B Open · 0.4B | — | 23.2%1,125 | 33.3%5.6% kept (63) | 40.0%1.8% kept (20) | 100.0%0.3% kept (3) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | 43.9%1,125 | 70.9%32.1% kept (361) | 86.9%14.9% kept (168) | 100.0%4.1% kept (46) | none kept |
| Metask-Jev-4B Open · 4.5B | — | 32.1%1,125 | 66.9%12.4% kept (139) | 81.0%7.0% kept (79) | 90.5%1.9% kept (21) | none kept |
| Open-Jev-27B-v1.1 Open · 27B | — | 42.6%1,125 | 95.5%5.9% kept (66) | 100.0%0.4% kept (4) | none kept | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 24.7%1,125 | 46.6%9.2% kept (103) | 65.9%3.6% kept (41) | 90.0%0.9% kept (10) | none kept |
| Strands Decider 2B Open · 2.0B | — | 22.4%1,125 | 38.8%6.0% kept (67) | 58.8%1.5% kept (17) | 100.0%0.2% kept (2) | none kept |
| celeris-1-decision Closed · Efficient | — | 47.3%1,125 | 61.9%53.7% kept (604) | 76.5%30.6% kept (344) | 90.5%15.9% kept (179) | none kept |
| celeris-1.1-decision Closed · Efficient | — | 47.3%1,125 | 62.9%52.7% kept (593) | 76.8%29.9% kept (336) | 90.8%16.4% kept (184) | none kept |
| Karpo Decision 31B Open · 31B | — | 60.7%1,125 | 75.3%65.2% kept (734) | 89.4%44.5% kept (501) | 96.3%28.8% kept (324) | 98.3%15.3% kept (172) |
| Drex Max Open · 27B | — | 53.1%1,125 | 63.7%72.1% kept (811) | 74.1%49.7% kept (559) | 86.5%30.8% kept (347) | 94.8%13.8% kept (155) |
| Mammoth-1 Open · 27B | — | 54.5%1,125 | 74.1%55.3% kept (622) | 86.3%36.3% kept (408) | 95.4%21.3% kept (240) | 98.1%9.6% kept (108) |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 54.8%1,125 | 71.8%58.0% kept (652) | 86.2%35.6% kept (400) | 95.5%19.9% kept (224) | 98.9%8.0% kept (90) |
| Mercury Decide Closed · Efficient | — | 45.1%1,125 | 61.0%53.6% kept (603) | 70.3%38.0% kept (428) | 79.9%22.1% kept (249) | 92.3%8.1% kept (91) |
| Sieve 31B v3 Open · 31B | — | 57.9%1,125 | 78.1%53.2% kept (599) | 89.3%35.7% kept (402) | 96.3%21.6% kept (243) | 97.5%7.2% kept (81) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 52.5%1,125 | 75.0%49.3% kept (555) | 88.5%29.3% kept (330) | 95.7%16.7% kept (188) | 98.7%7.0% kept (79) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 53.3%1,125 | 77.0%47.6% kept (535) | 90.7%28.8% kept (324) | 95.9%17.2% kept (193) | 97.5%7.1% kept (80) |
| GLiDE 27B Open · 27B | — | 58.4%1,125 | 76.8%60.3% kept (678) | 88.6%38.3% kept (431) | 97.7%18.9% kept (213) | 98.7%6.7% kept (75) |
| SubQ Snap 1.1 Closed · Efficient | — | 52.4%1,125 | 65.0%66.5% kept (748) | 77.9%42.2% kept (475) | 88.7%23.6% kept (266) | 97.2%6.4% kept (72) |
| d1 Closed · Efficient | — | 40.8%1,125 | 59.9%41.0% kept (461) | 74.2%21.7% kept (244) | 87.9%10.3% kept (116) | 98.2%5.0% kept (56) |
| Adjudex 9B v5 Open · 9.0B | — | 46.4%1,125 | 68.9%44.4% kept (499) | 83.2%25.3% kept (285) | 95.8%12.7% kept (143) | 98.2%4.9% kept (55) |
| Valen-Decision-1.5-9B Open · 9.0B | — | 44.4%1,125 | 63.2%37.7% kept (424) | 77.9%17.7% kept (199) | 90.2%8.2% kept (92) | 94.6%3.3% kept (37) |
| Cerno 9B v1.2 Open · 9.0B | — | 43.3%1,125 | 56.1%59.4% kept (668) | 70.8%33.5% kept (377) | 89.6%14.6% kept (164) | 96.3%2.4% kept (27) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 42.3%1,125 | 55.4%51.6% kept (581) | 68.8%27.4% kept (308) | 86.4%11.1% kept (125) | 94.4%1.6% kept (18) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | — | 89 / 00.804 | 25 / 00.765 | 14 / 00.764 | 50 / 00.904 |
| Copernicus-31B Open · 31B | — | 66 / 00.801 | 15 / 00.746 | 12 / 00.775 | 39 / 00.935 |
| Jev 1.13 Closed · Efficient | — | 65 / 00.764 | 7 / 00.708 | 5 / 00.714 | 53 / 00.875 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 62 / 00.803 | 22 / 00.752 | 5 / 00.764 | 35 / 00.878 |
| Jev 1.14 Closed · Efficient | — | 61 / 00.750 | 5 / 00.671 | 8 / 00.711 | 48 / 00.856 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 56 / 00.759 | 7 / 00.712 | 9 / 00.689 | 40 / 00.910 |
| KnowLine-31B-Gen2 Open · 31B | — | 54 / 00.825 | 26 / 00.792 | 5 / 00.785 | 23 / 00.907 |
| d3 Open · 26B | — | 54 / 00.806 | 3 / 00.755 | 11 / 00.761 | 40 / 00.863 |
| Elephant-1 Open · 9.0B | — | 53 / 00.768 | 5 / 00.671 | 9 / 00.725 | 39 / 00.925 |
| GLiDE API Closed · Efficient | — | 52 / 00.814 | 7 / 00.777 | 7 / 00.765 | 38 / 00.866 |
| REAX 1.3 Open · 57B | — | 51 / 00.816 | 12 / 00.750 | 11 / 00.804 | 28 / 00.921 |
| d1.1 Closed · Efficient | — | 48 / 00.748 | 4 / 00.676 | 6 / 00.712 | 38 / 00.906 |
| GLiDE v1.2 Open · 27B | — | 45 / 00.793 | 3 / 00.732 | 6 / 00.739 | 36 / 00.906 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 43 / 00.768 | 5 / 00.766 | 0 / 00.699 | 38 / 00.885 |
| Chinese Decision Pointer 4B Open · 4.0B | — | 39 / 00.724 | 1 / 00.644 | 8 / 00.699 | 30 / 00.866 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 35 / 00.729 | 3 / 00.642 | 1 / 00.693 | 31 / 00.879 |
| Verdin-4B-v1.1 Open · 4.0B | — | 35 / 00.729 | 2 / 00.648 | 5 / 00.672 | 28 / 00.884 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 33 / 00.717 | 3 / 00.652 | 0 / 00.690 | 30 / 00.815 |
| GLiDE 9B Open · 7.9B | — | 32 / 00.749 | 3 / 00.637 | 4 / 00.729 | 25 / 00.911 |
| Winnow-12B Open · 12B | — | 32 / 00.727 | 1 / 00.608 | 6 / 00.670 | 25 / 00.835 |
| KnowLine-9B-Gen1 Open · 9.0B | — | 30 / 00.725 | 3 / 00.638 | 0 / 00.692 | 27 / 00.843 |
| d1.2 Closed · Undisclosed | — | 28 / 00.742 | 3 / 00.676 | 2 / 00.719 | 23 / 00.874 |
| d3-ultra Open · 26B | — | 28 / 00.778 | 9 / 00.707 | 10 / 00.761 | 9 / 00.853 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 21 / 00.720 | 1 / 00.633 | 2 / 00.672 | 18 / 00.853 |
| Microsoft-Decision-1 Closed · Efficient | — | 20 / 00.712 | 1 / 00.628 | 2 / 00.683 | 17 / 00.823 |
| d1-ultra Closed · Efficient | — | 20 / 00.783 | 12 / 00.759 | 3 / 00.715 | 5 / 00.846 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 17 / 00.757 | 4 / 00.663 | 1 / 00.731 | 12 / 00.860 |
| GLiDE 2B Open · 1.9B | — | 14 / 00.632 | 1 / 00.525 | 1 / 00.637 | 12 / 00.787 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 14 / 00.762 | 0 / 00.675 | 3 / 00.713 | 11 / 00.854 |
| d1.2-ultra Closed · Undisclosed | — | 13 / 00.800 | 4 / 00.741 | 4 / 00.772 | 5 / 00.837 |
| Jeffan-9B-1211 Open · 9.0B | — | 10 / 00.753 | 2 / 00.663 | 5 / 00.704 | 3 / 00.866 |
| GLiDE 0.8B Open · 0.8B | — | 9 / 00.617 | 0 / 00.557 | 0 / 00.608 | 9 / 00.700 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 8 / 00.601 | 0 / 00.501 | 0 / 00.571 | 8 / 00.756 |
| Logits Sigma-1 Duo Open · 57B | — | 8 / 00.796 | 1 / 00.738 | 1 / 00.764 | 6 / 00.880 |
| Turion Jev 31B Open · 31B | — | 7 / 00.782 | 1 / 00.702 | 0 / 00.756 | 6 / 00.875 |
| Clef Open · 27B | — | 6 / 00.697 | 2 / 00.607 | 0 / 00.704 | 4 / 00.846 |
| KnowLine-12B-Gen1 Open · 12B | — | 6 / 00.774 | 1 / 00.701 | 1 / 00.711 | 4 / 00.937 |
| vera-medium-omni Open · 4.2B | — | 6 / 00.739 | 0 / 00.659 | 0 / 00.710 | 6 / 00.876 |
| OneJev-27B Open · 27B | — | 4 / 00.726 | 0 / 00.633 | 0 / 00.712 | 4 / 00.845 |
| d1-2B Open · 1.9B | — | 3 / 00.620 | 0 / 00.549 | 0 / 00.606 | 3 / 00.787 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 2 / 00.573 | 0 / 00.520 | 0 / 00.531 | 2 / 00.746 |
| OneJev1.6-0.8B Open · 0.8B | — | 2 / 00.545 | 0 / 00.508 | 0 / 00.538 | 2 / 00.608 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 2 / 00.640 | 0 / 00.539 | 0 / 00.652 | 2 / 00.767 |
| Clef-Flash Open · 9.0B | — | No answers at this threshold0.659 | 0 / 00.537 | 0 / 00.670 | 0 / 00.790 |
| Cygnet Open · 12B | — | No answers at this threshold0.720 | 0 / 00.604 | 0 / 00.676 | 0 / 00.848 |
| GLiClass v4 Edge Open · 0.4B | — | No answers at this threshold0.527 | 0 / 00.516 | 0 / 00.547 | 0 / 00.584 |
| GLiDE 0.4B Open · 0.4B | — | No answers at this threshold0.556 | 0 / 00.531 | 0 / 00.545 | 0 / 00.639 |
| KnowLine-9B-Gen2 Open · 9.0B | — | No answers at this threshold0.730 | 0 / 00.672 | 0 / 00.663 | 0 / 00.859 |
| Metask-Jev-4B Open · 4.5B | — | No answers at this threshold0.660 | 0 / 00.591 | 0 / 00.604 | 0 / 00.830 |
| Open-Jev-27B-v1.1 Open · 27B | — | No answers at this threshold0.733 | 0 / 00.623 | 0 / 00.716 | 0 / 00.861 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | No answers at this threshold0.641 | 0 / 00.580 | 0 / 00.615 | 0 / 00.694 |
| Strands Decider 2B Open · 2.0B | — | No answers at this threshold0.568 | 0 / 00.544 | 0 / 00.562 | 0 / 00.634 |
| celeris-1-decision Closed · Efficient | — | No answers at this threshold0.735 | 0 / 00.706 | 0 / 00.679 | 0 / 00.867 |
| celeris-1.1-decision Closed · Efficient | — | No answers at this threshold0.733 | 0 / 00.709 | 0 / 00.679 | 0 / 00.853 |
| Karpo Decision 31B Open · 31B | — | 169 / 30.816 | 58 / 00.805 | 33 / 20.758 | 78 / 10.877 |
| Drex Max Open · 27B | — | 147 / 80.786 | 34 / 00.729 | 39 / 50.758 | 74 / 30.838 |
| Mammoth-1 Open · 27B | — | 106 / 20.795 | 21 / 00.703 | 13 / 00.797 | 72 / 20.905 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 89 / 10.793 | 7 / 00.729 | 24 / 00.771 | 58 / 10.877 |
| Mercury Decide Closed · Efficient | — | 84 / 70.746 | 14 / 20.670 | 28 / 10.744 | 42 / 40.775 |
| Sieve 31B v3 Open · 31B | — | 79 / 20.797 | 9 / 00.764 | 16 / 20.745 | 54 / 00.861 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 78 / 10.800 | 8 / 00.728 | 12 / 00.787 | 58 / 10.860 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 78 / 20.798 | 9 / 00.730 | 12 / 00.783 | 57 / 20.791 |
| GLiDE 27B Open · 27B | — | 74 / 10.822 | 6 / 00.754 | 17 / 00.800 | 51 / 10.888 |
| SubQ Snap 1.1 Closed · Efficient | — | 70 / 20.787 | 17 / 00.736 | 18 / 10.743 | 35 / 10.857 |
| d1 Closed · Efficient | — | 55 / 10.723 | 3 / 00.630 | 5 / 00.683 | 47 / 10.828 |
| Adjudex 9B v5 Open · 9.0B | — | 54 / 10.771 | 4 / 00.706 | 2 / 00.730 | 48 / 10.869 |
| Valen-Decision-1.5-9B Open · 9.0B | — | 35 / 20.701 | 5 / 10.598 | 5 / 10.709 | 25 / 00.785 |
| Cerno 9B v1.2 Open · 9.0B | — | 26 / 10.752 | 8 / 10.645 | 3 / 00.745 | 15 / 00.881 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 17 / 10.708 | 2 / 00.634 | 3 / 10.641 | 12 / 00.881 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 1,125 of 1,125); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 172 answers, 169 right · 3 wrong · AUROC 0.816 Math 81.6% | — | 60.7%57.8%–63.5% | ab | 683 / 1,125 | 169 right of 172 kept3 wrong · 953 skipped | 0.816 | 71.80chosen-answer Brier 0.1711 | 6.6%answered 97% | 173 ms | 211 ms | $0.128$128estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 51 answers, 51 right · 0 wrong · AUROC 0.816 Math 78.0% | — | 59.3%56.4%–62.1% | ac | 667 / 1,125 | 51 right of 51 kept0 mistakes · 1,074 skipped | 0.816 | 71.07chosen-answer Brier 0.1714 | 5.3%answered 98% | 215 ms | 278 ms | $0.160$160estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 89 answers, 89 right · 0 wrong · AUROC 0.804 Math 77.4% | — | 59.0%56.1%–61.9% | abcd | 664 / 1,125 | 89 right of 89 kept0 mistakes · 1,036 skipped | 0.804 | 70.72chosen-answer Brier 0.1757 | 0.4%answered 100% | 983 ms | 1,279 ms | $0.061$61.43estimate · DGX Spark |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.825 Math 80.3% | — | 58.9%56.0%–61.8% | abcd | 663 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.825 | 71.09chosen-answer Brier 0.1675 | 0.0%answered 100% | 151 ms | 204 ms | $0.112$112estimate · RTX PRO 6000 Blackwell |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 66 answers, 66 right · 0 wrong · AUROC 0.801 Math 77.3% | — | 58.7%55.8%–61.5% | abcd | 660 / 1,125 | 66 right of 66 kept0 mistakes · 1,059 skipped | 0.801 | 70.48chosen-answer Brier 0.1771 | 1.9%answered 99% | 185 ms | 255 ms | $0.136$136estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 75 answers, 74 right · 1 wrong · AUROC 0.822 Math 72.0% | — | 58.4%55.5%–61.2% | abcde | 657 / 1,125 | 74 right of 75 kept1 wrong · 1,050 skipped | 0.822 | 70.67chosen-answer Brier 0.1707 | 44.3%answered 79% | 544 ms | 655 ms | $0.033$33.16estimate · DGX Spark |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 81 answers, 79 right · 2 wrong · AUROC 0.797 Math 77.1% | — | 57.9%55.0%–60.7% | abcdf | 651 / 1,125 | 79 right of 81 kept2 wrong · 1,044 skipped | 0.797 | 70.03chosen-answer Brier 0.1781 | 5.5%answered 98% | 1,050 ms | 1,335 ms | $0.063$62.95estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.796 Math 75.2% | — | 57.8%54.9%–60.6% | abcdefg | 650 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.796 | 69.90chosen-answer Brier 0.1798 | 7.1%answered 97% | 285 ms | 399 ms | $0.211$211estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 45 answers, 45 right · 0 wrong · AUROC 0.793 Math 69.9% | — | 55.2%52.3%–58.1% | abcdefgh | 621 / 1,125 | 45 right of 45 kept0 mistakes · 1,080 skipped | 0.793 | 68.39chosen-answer Brier 0.1842 | 50.2%answered 73% | 111 ms | 140 ms | $0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 20 answers, 20 right · 0 wrong · AUROC 0.783 Math 74.1% | — | 54.9%52.0%–57.8% | abcdefgh | 618 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.783 | 68.09chosen-answer Brier 0.1876 | 91.0%answered 35% | 182 ms | 425 ms | $0.013$12.64 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 90 answers, 89 right · 1 wrong · AUROC 0.793 Math 70.1% | — | 54.8%51.8%–57.6% | abcdefghi | 616 / 1,125 | 89 right of 90 kept1 wrong · 1,035 skipped | 0.793 | 68.07chosen-answer Brier 0.1861 | 54.1%answered 69% | 329 ms | 1,540 ms | $0.022$22.37 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 62 answers, 62 right · 0 wrong · AUROC 0.803 Math 72.5% | — | 54.8%51.8%–57.6% | abcdefgh | 616 / 1,125 | 62 right of 62 kept0 mistakes · 1,063 skipped | 0.803 | 67.98chosen-answer Brier 0.1881 | 18.1%answered 91% | 108 ms | 131 ms | $0.081$81.16estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 28 answers, 28 right · 0 wrong · AUROC 0.778 Math 71.5% | — | 54.6%51.7%–57.5% | bdefghij | 614 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.778 | 66.46chosen-answer Brier 0.2166 | 10.1%answered 95% | 156 ms | 185 ms | $0.109$109estimate · RTX PRO 6000 Blackwell |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 52 answers, 52 right · 0 wrong · AUROC 0.814 Math 70.8% | — | 54.5%51.6%–57.4% | cdefghi | 613 / 1,125 | 52 right of 52 kept0 mistakes · 1,073 skipped | 0.814 | 68.27chosen-answer Brier 0.1794 | 60.0%answered 67% | 271 ms | 291 ms | $0.052$52.03 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 108 answers, 106 right · 2 wrong · AUROC 0.795 Math 74.0% | — | 54.5%51.6%–57.4% | cdefghij | 613 / 1,125 | 106 right of 108 kept2 wrong · 1,017 skipped | 0.795 | 68.07chosen-answer Brier 0.1835 | 72.3%answered 58% | 221 ms | 244 ms | $0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.806 Math 70.4% | — | 54.5%51.6%–57.4% | cdefgh | 613 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.806 | 68.34chosen-answer Brier 0.1781 | 14.8%answered 93% | 76 ms | 98 ms | $0.057$57.18estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 80 answers, 78 right · 2 wrong · AUROC 0.798 Math 69.8% | — | 53.3%50.4%–56.2% | defghijk | 600 / 1,125 | 78 right of 80 kept2 wrong · 1,045 skipped | 0.798 | 67.64chosen-answer Brier 0.1805 | 14.4%answered 93% | 238 ms | 552 ms | $0.007$7.46 |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 155 answers, 147 right · 8 wrong · AUROC 0.786 Math 70.3% | — | 53.1%50.1%–56.0% | defghijk | 597 / 1,125 | 147 right of 155 kept8 wrong · 970 skipped | 0.786 | 65.68chosen-answer Brier 0.2170 | 7.3%answered 97% | 106 ms | 143 ms | $0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 79 answers, 78 right · 1 wrong · AUROC 0.800 Math 69.3% | with web | 52.5%49.6%–55.4% | fghijkl | 591 / 1,125 | 78 right of 79 kept1 wrong · 1,046 skipped | 0.800 | 67.24chosen-answer Brier 0.1806 | 37.7%answered 80% | 448 ms | 827 ms | $1.030$1,030 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 72 answers, 70 right · 2 wrong · AUROC 0.787 Math 68.6% | — | 52.4%49.4%–55.3% | fghijklm | 589 / 1,125 | 70 right of 72 kept2 wrong · 1,053 skipped | 0.787 | 65.94chosen-answer Brier 0.2047 | 65.7%answered 61% | 384 ms | 393 ms | $0.015$14.92 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.782 Math 69.0% | — | 52.3%49.3%–55.2% | eghijklmn | 588 / 1,125 | 7 right of 7 kept0 mistakes · 1,118 skipped | 0.782 | 66.89chosen-answer Brier 0.1849 | 1.5%answered 99% | 385 ms | 533 ms | $0.025$24.91estimate · DGX Spark |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.800 Math 69.9% | — | 51.6%48.6%–54.5% | hijklmno | 580 / 1,125 | 13 right of 13 kept0 mistakes · 1,112 skipped | 0.800 | 66.77chosen-answer Brier 0.1802 | 94.5%answered 28% | 220 ms | 341 ms | $0.013$12.64 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.774 Math 66.3% | — | 50.6%47.7%–53.5% | hijklmnop | 569 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.774 | 65.90chosen-answer Brier 0.1878 | 8.8%answered 95% | 68 ms | 88 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 43 answers, 43 right · 0 wrong · AUROC 0.768 Math 71.3% | — | 50.0%47.1%–53.0% | hijklmnopq | 563 / 1,125 | 43 right of 43 kept0 mistakes · 1,082 skipped | 0.768 | 65.37chosen-answer Brier 0.1930 | 2.0%answered 99% | 228 ms | 344 ms | $0.037$36.77 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 28 answers, 28 right · 0 wrong · AUROC 0.742 Math 65.5% | — | 49.2%46.2%–52.1% | hijklmnopqr | 553 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.742 | 64.51chosen-answer Brier 0.2013 | 78.8%answered 43% | 222 ms | 390 ms | $0.006$6.02 |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 10 answers, 10 right · 0 wrong · AUROC 0.753 Math 59.8% | — | 48.2%45.3%–51.1% | jklmnopqrs | 542 / 1,125 | 10 right of 10 kept0 mistakes · 1,115 skipped | 0.753 | 64.33chosen-answer Brier 0.1952 | 23.4%answered 87% | 52 ms | 66 ms | $0.039$39.35estimate · RTX PRO 6000 Blackwell |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 61 answers, 61 right · 0 wrong · AUROC 0.750 Math 65.1% | — | 48.2%45.3%–51.1% | ijklmnopqrs | 542 / 1,125 | 61 right of 61 kept0 mistakes · 1,064 skipped | 0.750 | 64.14chosen-answer Brier 0.1990 | 24.5%answered 86% | 166 ms | 193 ms | $0.026$25.64 |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 32 answers, 32 right · 0 wrong · AUROC 0.749 Math 57.9% | — | 47.8%44.9%–50.7% | klmnopqrs | 538 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.749 | 63.95chosen-answer Brier 0.1993 | 30.4%answered 82% | 208 ms | 251 ms | $0.013$13.29estimate · DGX Spark |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.735 Math 70.0% | — | 47.3%44.4%–50.2% | klmnopqrstu | 532 / 1,125 | No answers at this threshold | 0.735 | 62.94chosen-answer Brier 0.2141 | 92.3%answered 28% | 390 ms | 394 ms | $0.015$15.40 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.733 Math 69.9% | — | 47.3%44.4%–50.2% | klmnopqrst | 532 / 1,125 | No answers at this threshold | 0.733 | 62.92chosen-answer Brier 0.2144 | 91.0%answered 29% | 383 ms | 390 ms | $0.002$1.54 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 14 answers, 14 right · 0 wrong · AUROC 0.762 Math 58.4% | — | 46.6%43.7%–49.5% | lmnopqrstuv | 524 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.762 | 63.55chosen-answer Brier 0.1947 | 17.4%answered 90% | 160 ms | 175 ms | $0.010$10.04estimate · DGX Spark |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 55 answers, 54 right · 1 wrong · AUROC 0.771 Math 62.1% | — | 46.4%43.5%–49.3% | lmnopqrstuv | 522 / 1,125 | 54 right of 55 kept1 wrong · 1,070 skipped | 0.771 | 63.48chosen-answer Brier 0.1945 | 12.2%answered 93% | 41 ms | 46 ms | $0.031$31.36estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 53 answers, 53 right · 0 wrong · AUROC 0.768 Math 61.3% | — | 46.4%43.5%–49.3% | lmnopqrstuv | 522 / 1,125 | 53 right of 53 kept0 mistakes · 1,072 skipped | 0.768 | 63.39chosen-answer Brier 0.1962 | 75.6%answered 47% | 79 ms | 98 ms | $0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 65 answers, 65 right · 0 wrong · AUROC 0.764 Math 64.1% | — | 46.3%43.4%–49.2% | lmnopqrstuv | 521 / 1,125 | 65 right of 65 kept0 mistakes · 1,060 skipped | 0.764 | 63.36chosen-answer Brier 0.1959 | 15.8%answered 91% | 167 ms | 203 ms | $0.026$25.64 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 48 answers, 48 right · 0 wrong · AUROC 0.748 Math 65.2% | — | 46.0%43.2%–49.0% | mnopqrstuvw | 518 / 1,125 | 48 right of 48 kept0 mistakes · 1,077 skipped | 0.748 | 63.14chosen-answer Brier 0.1977 | 90.6%answered 28% | 176 ms | 272 ms | $0.006$6.02 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 21 answers, 21 right · 0 wrong · AUROC 0.720 Math 55.8% | — | 45.9%43.0%–48.8% | nopqrstuvx | 516 / 1,125 | 21 right of 21 kept0 mistakes · 1,104 skipped | 0.720 | 62.63chosen-answer Brier 0.2061 | 16.0%answered 91% | 73 ms | 84 ms | $0.005$4.62estimate · DGX Spark |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 56 answers, 56 right · 0 wrong · AUROC 0.759 Math 58.8% | — | 45.3%42.4%–48.3% | opqrstuvwxy | 510 / 1,125 | 56 right of 56 kept0 mistakes · 1,069 skipped | 0.759 | 61.81chosen-answer Brier 0.2171 | 12.4%answered 93% | 47 ms | 53 ms | $0.034$33.81estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 91 answers, 84 right · 7 wrong · AUROC 0.746 Math 53.5% | — | 45.1%42.2%–48.0% | pqrstuvwxy | 507 / 1,125 | 84 right of 91 kept7 wrong · 1,034 skipped | 0.746 | 61.18chosen-answer Brier 0.2272 | 20.5%answered 88% | 301 ms | 360 ms | $0.008$7.52 |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 17 answers, 17 right · 0 wrong · AUROC 0.757 Math 55.9% | — | 44.7%41.8%–47.6% | pqrstuvwxy | 503 / 1,125 | 17 right of 17 kept0 mistakes · 1,108 skipped | 0.757 | 62.14chosen-answer Brier 0.2042 | 24.4%answered 85% | 33 ms | 41 ms | $0.026$26.44estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 20 answers, 20 right · 0 wrong · AUROC 0.712 Math 59.4% | — | 44.4%41.6%–47.4% | pqrstuvwxy | 500 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.712 | 61.53chosen-answer Brier 0.2139 | 19.8%answered 88% | 315 ms | 362 ms | $0.013$13.42 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 37 answers, 35 right · 2 wrong · AUROC 0.701 Math 55.3% | — | 44.4%41.5%–47.3% | pqrstuvwxy | 499 / 1,125 | 35 right of 37 kept2 wrong · 1,088 skipped | 0.701 | 61.42chosen-answer Brier 0.2151 | 13.6%answered 92% | 60 ms | 67 ms | $0.042$42.42estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 30 answers, 30 right · 0 wrong · AUROC 0.725 Math 59.8% | — | 44.3%41.4%–47.2% | pqrstuvwxy | 498 / 1,125 | 30 right of 30 kept0 mistakes · 1,095 skipped | 0.725 | 61.97chosen-answer Brier 0.2032 | 6.7%answered 96% | 50 ms | 61 ms | $0.036$36.27estimate · RTX PRO 6000 Blackwell |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.739 Math 56.8% | — | 44.1%41.2%–47.0% | qrstuvwxy | 496 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.739 | 61.90chosen-answer Brier 0.2028 | 13.6%answered 92% | 32 ms | 37 ms | $0.026$25.82estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.730 Math 60.1% | — | 43.9%41.0%–46.8% | qrstuvwxy | 494 / 1,125 | No answers at this threshold | 0.730 | 61.90chosen-answer Brier 0.2011 | 19.5%answered 88% | 52 ms | 65 ms | $0.040$39.96estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 27 answers, 26 right · 1 wrong · AUROC 0.752 Math 56.0% | — | 43.3%40.4%–46.2% | rstuvwxy | 487 / 1,125 | 26 right of 27 kept1 wrong · 1,098 skipped | 0.752 | 60.41chosen-answer Brier 0.2247 | 6.0%answered 97% | 209 ms | 269 ms | $0.013$13.35estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 35 answers, 35 right · 0 wrong · AUROC 0.729 Math 55.2% | — | 43.1%40.2%–46.0% | qrstuvwxy | 485 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 61.34chosen-answer Brier 0.2043 | 28.9%answered 82% | 45 ms | 55 ms | $0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.733 Math 54.6% | — | 42.6%39.7%–45.5% | rstuvwxy | 479 / 1,125 | No answers at this threshold | 0.733 | 59.78chosen-answer Brier 0.2302 | 28.2%answered 82% | 413 ms | 576 ms | $0.294$294estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 18 answers, 17 right · 1 wrong · AUROC 0.708 Math 56.0% | — | 42.3%39.5%–45.2% | stuvwxy | 476 / 1,125 | 17 right of 18 kept1 wrong · 1,107 skipped | 0.708 | 59.66chosen-answer Brier 0.2298 | 8.3%answered 95% | 96 ms | 115 ms | $0.006$5.89estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 33 answers, 33 right · 0 wrong · AUROC 0.717 Math 59.0% | — | 41.9%39.0%–44.8% | stuvwxy | 471 / 1,125 | 33 right of 33 kept0 mistakes · 1,092 skipped | 0.717 | 60.73chosen-answer Brier 0.2040 | 37.7%answered 75% | 345 ms | 504 ms | $0.018$18.41 |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.697 Math 51.1% | — | 40.8%38.0%–43.7% | uvwxy | 459 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.697 | 58.55chosen-answer Brier 0.2369 | 3.6%answered 98% | 196 ms | 222 ms | $0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 56 answers, 55 right · 1 wrong · AUROC 0.723 Math 55.6% | — | 40.8%38.0%–43.7% | tuvwxy | 459 / 1,125 | 55 right of 56 kept1 wrong · 1,069 skipped | 0.723 | 59.76chosen-answer Brier 0.2128 | 11.3%answered 93% | 188 ms | 290 ms | $0.006$6.02 |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.726 Math 53.8% | — | 40.6%37.8%–43.5% | tuvwxy | 457 / 1,125 | 4 right of 4 kept0 mistakes · 1,121 skipped | 0.726 | 60.27chosen-answer Brier 0.2009 | 1.2%answered 99% | 108 ms | 136 ms | $0.086$86.07estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.724 Math 50.4% | — | 40.4%37.5%–43.3% | vwxy | 454 / 1,125 | 39 right of 39 kept0 mistakes · 1,086 skipped | 0.724 | 58.90chosen-answer Brier 0.2256 | 14.6%answered 91% | 38 ms | 48 ms | $0.030$29.51estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 35 answers, 35 right · 0 wrong · AUROC 0.729 Math 51.5% | — | 39.7%36.9%–42.6% | wy | 447 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 59.26chosen-answer Brier 0.2122 | 12.6%answered 92% | 45 ms | 51 ms | $0.033$33.20estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 32 answers, 32 right · 0 wrong · AUROC 0.727 Math 46.9% | — | 39.7%36.9%–42.6% | xy | 447 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.727 | 58.43chosen-answer Brier 0.2287 | 29.7%answered 80% | 66 ms | 91 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.720 Math 44.3% | — | 39.3%36.5%–42.2% | xy | 442 / 1,125 | No answers at this threshold | 0.720 | 58.97chosen-answer Brier 0.2135 | 11.3%answered 93% | 40 ms | 48 ms | $0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.659 Math 49.8% | — | 38.9%36.1%–41.8% | yz | 438 / 1,125 | No answers at this threshold | 0.659 | 57.70chosen-answer Brier 0.2353 | 14.4%answered 91% | 110 ms | 137 ms | $0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 14 answers, 14 right · 0 wrong · AUROC 0.632 Math 36.8% | — | 32.3%29.6%–35.1% | zA | 363 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.632 | 54.39chosen-answer Brier 0.2349 | 32.9%answered 75% | 34 ms | 42 ms | $0.002$2.26estimate · DGX Spark |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.660 Math 39.2% | — | 32.1%29.4%–34.9% | zA | 361 / 1,125 | No answers at this threshold | 0.660 | 56.30chosen-answer Brier 0.1950 | 1.5%answered 99% | 123 ms | 152 ms | $0.091$90.99estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.601 Math 35.3% | — | 31.7%29.1%–34.5% | A | 357 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.601 | 55.15chosen-answer Brier 0.2143 | 15.7%answered 89% | 59 ms | 70 ms | $0.004$3.78estimate · DGX Spark |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.640 Math 38.7% | — | 30.8%28.1%–33.5% | AB | 346 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.640 | 55.51chosen-answer Brier 0.1975 | 31.4%answered 76% | 40 ms | 45 ms | $0.003$2.52estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 3 answers, 3 right · 0 wrong · AUROC 0.620 Math 37.1% | — | 30.7%28.0%–33.4% | AB | 345 / 1,125 | 3 right of 3 kept0 mistakes · 1,122 skipped | 0.620 | 55.05chosen-answer Brier 0.2058 | 86.5%answered 22% | 17 ms | 23 ms | $0.015$14.76estimate · RTX PRO 6000 Blackwell |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 9 answers, 9 right · 0 wrong · AUROC 0.617 Math 32.1% | — | 29.2%26.6%–31.9% | ABC | 328 / 1,125 | 9 right of 9 kept0 mistakes · 1,116 skipped | 0.617 | 53.53chosen-answer Brier 0.2210 | 26.1%answered 80% | 23 ms | 29 ms | $0.002$1.58estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.545 Math 28.2% | — | 27.0%24.5%–29.7% | ABC | 304 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.545 | 52.72chosen-answer Brier 0.2159 | 4.6%answered 97% | 16 ms | 19 ms | $0.001$1.16estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.573 Math 28.2% | — | 26.7%24.2%–29.3% | ABC | 300 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.573 | 53.45chosen-answer Brier 0.1976 | 1.2%answered 99% | 31 ms | 39 ms | $0.002$2.00estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.641 Math 29.7% | — | 24.7%22.3%–27.3% | BC | 278 / 1,125 | No answers at this threshold | 0.641 | 53.22chosen-answer Brier 0.1827 | 21.4%answered 83% | 27 ms | 31 ms | $0.002$1.79estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.527 Math 22.7% | — | 23.6%21.2%–26.1% | BC | 265 / 1,125 | No answers at this threshold | 0.527 | 51.99chosen-answer Brier 0.1958 | 12.4%answered 90% | 15 ms | 28 ms | $0.001$1.31estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.556 Math 20.6% | — | 23.2%20.8%–25.8% | C | 261 / 1,125 | No answers at this threshold | 0.556 | 52.19chosen-answer Brier 0.1883 | 76.1%answered 30% | 13 ms | 16 ms | $0.001$0.95estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.568 Math 28.7% | — | 22.4%20.1%–24.9% | C | 252 / 1,125 | No answers at this threshold | 0.568 | 51.86chosen-answer Brier 0.1868 | 8.3%answered 94% | 36 ms | 43 ms | $0.028$27.67estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
quincyqiang Perplexity victorbennett StartLux Fastino PelaAI scar-ai Liquid AI OmniJev Other Metask Lab Strands Agents Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Chinese Decision Pointer 4B Open · 4.0B | — | 40.4%1,125 | 54.7%49.4% kept (556) | 68.8%25.6% kept (288) | 83.1%12.1% kept (136) | 100.0%3.5% kept (39) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 43.1%1,125 | 65.9%36.3% kept (408) | 82.4%18.7% kept (210) | 92.3%10.4% kept (117) | 100.0%3.1% kept (35) |
| Verdin-4B-v1.1 Open · 4.0B | — | 39.7%1,125 | 59.2%39.8% kept (448) | 73.6%20.9% kept (235) | 87.0%9.6% kept (108) | 100.0%3.1% kept (35) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 45.9%1,125 | 68.4%36.8% kept (414) | 82.7%18.0% kept (202) | 97.8%8.3% kept (93) | 100.0%1.9% kept (21) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 44.7%1,125 | 63.9%47.6% kept (535) | 77.7%27.6% kept (310) | 92.2%11.5% kept (129) | 100.0%1.5% kept (17) |
| GLiDE 2B Open · 1.9B | — | 32.3%1,125 | 46.3%34.8% kept (391) | 54.7%15.1% kept (170) | 79.7%5.2% kept (59) | 100.0%1.2% kept (14) |
| GLiDE 0.8B Open · 0.8B | — | 29.2%1,125 | 46.1%24.1% kept (271) | 54.7%9.4% kept (106) | 62.5%3.6% kept (40) | 100.0%0.8% kept (9) |
| KnowLine-2B-Gen1 Open · 2.0B | — | 31.7%1,125 | 51.2%18.4% kept (207) | 69.1%7.2% kept (81) | 87.5%2.8% kept (32) | 100.0%0.7% kept (8) |
| vera-medium-omni Open · 4.2B | — | 44.1%1,125 | 72.7%29.3% kept (330) | 85.6%11.1% kept (125) | 100.0%4.9% kept (55) | 100.0%0.5% kept (6) |
| d1-2B Open · 1.9B | — | 30.7%1,125 | 52.0%15.2% kept (171) | 73.3%5.3% kept (60) | 100.0%1.7% kept (19) | 100.0%0.3% kept (3) |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 26.7%1,125 | 47.4%10.1% kept (114) | 63.9%3.2% kept (36) | 92.9%1.2% kept (14) | 100.0%0.2% kept (2) |
| OneJev1.6-0.8B Open · 0.8B | — | 27.0%1,125 | 36.3%15.9% kept (179) | 44.6%5.0% kept (56) | 83.3%0.5% kept (6) | 100.0%0.2% kept (2) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 30.8%1,125 | 59.7%13.2% kept (149) | 73.8%5.4% kept (61) | 100.0%2.0% kept (22) | 100.0%0.2% kept (2) |
| GLiClass v4 Edge Open · 0.4B | — | 23.6%1,125 | 35.6%9.2% kept (104) | 30.8%2.3% kept (26) | 50.0%0.5% kept (6) | none kept |
| GLiDE 0.4B Open · 0.4B | — | 23.2%1,125 | 33.3%5.6% kept (63) | 40.0%1.8% kept (20) | 100.0%0.3% kept (3) | none kept |
| Metask-Jev-4B Open · 4.5B | — | 32.1%1,125 | 66.9%12.4% kept (139) | 81.0%7.0% kept (79) | 90.5%1.9% kept (21) | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 24.7%1,125 | 46.6%9.2% kept (103) | 65.9%3.6% kept (41) | 90.0%0.9% kept (10) | none kept |
| Strands Decider 2B Open · 2.0B | — | 22.4%1,125 | 38.8%6.0% kept (67) | 58.8%1.5% kept (17) | 100.0%0.2% kept (2) | none kept |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 42.3%1,125 | 55.4%51.6% kept (581) | 68.8%27.4% kept (308) | 86.4%11.1% kept (125) | 94.4%1.6% kept (18) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|
| Chinese Decision Pointer 4B Open · 4.0B | — | 39 / 00.724 | 1 / 00.644 | 8 / 00.699 | 30 / 00.866 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 35 / 00.729 | 3 / 00.642 | 1 / 00.693 | 31 / 00.879 |
| Verdin-4B-v1.1 Open · 4.0B | — | 35 / 00.729 | 2 / 00.648 | 5 / 00.672 | 28 / 00.884 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 21 / 00.720 | 1 / 00.633 | 2 / 00.672 | 18 / 00.853 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 17 / 00.757 | 4 / 00.663 | 1 / 00.731 | 12 / 00.860 |
| GLiDE 2B Open · 1.9B | — | 14 / 00.632 | 1 / 00.525 | 1 / 00.637 | 12 / 00.787 |
| GLiDE 0.8B Open · 0.8B | — | 9 / 00.617 | 0 / 00.557 | 0 / 00.608 | 9 / 00.700 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 8 / 00.601 | 0 / 00.501 | 0 / 00.571 | 8 / 00.756 |
| vera-medium-omni Open · 4.2B | — | 6 / 00.739 | 0 / 00.659 | 0 / 00.710 | 6 / 00.876 |
| d1-2B Open · 1.9B | — | 3 / 00.620 | 0 / 00.549 | 0 / 00.606 | 3 / 00.787 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 2 / 00.573 | 0 / 00.520 | 0 / 00.531 | 2 / 00.746 |
| OneJev1.6-0.8B Open · 0.8B | — | 2 / 00.545 | 0 / 00.508 | 0 / 00.538 | 2 / 00.608 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 2 / 00.640 | 0 / 00.539 | 0 / 00.652 | 2 / 00.767 |
| GLiClass v4 Edge Open · 0.4B | — | No answers at this threshold0.527 | 0 / 00.516 | 0 / 00.547 | 0 / 00.584 |
| GLiDE 0.4B Open · 0.4B | — | No answers at this threshold0.556 | 0 / 00.531 | 0 / 00.545 | 0 / 00.639 |
| Metask-Jev-4B Open · 4.5B | — | No answers at this threshold0.660 | 0 / 00.591 | 0 / 00.604 | 0 / 00.830 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | No answers at this threshold0.641 | 0 / 00.580 | 0 / 00.615 | 0 / 00.694 |
| Strands Decider 2B Open · 2.0B | — | No answers at this threshold0.568 | 0 / 00.544 | 0 / 00.562 | 0 / 00.634 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 17 / 10.708 | 2 / 00.634 | 3 / 10.641 | 12 / 00.881 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 21 answers, 21 right · 0 wrong · AUROC 0.720 Math 55.8% | — | 45.9%43.0%–48.8% | a | 516 / 1,125 | 21 right of 21 kept0 mistakes · 1,104 skipped | 0.720 | 62.63chosen-answer Brier 0.2061 | 16.0%answered 91% | 73 ms | 84 ms | $0.005$4.62estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 17 answers, 17 right · 0 wrong · AUROC 0.757 Math 55.9% | — | 44.7%41.8%–47.6% | ab | 503 / 1,125 | 17 right of 17 kept0 mistakes · 1,108 skipped | 0.757 | 62.14chosen-answer Brier 0.2042 | 24.4%answered 85% | 33 ms | 41 ms | $0.026$26.44estimate · RTX PRO 6000 Blackwell |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.739 Math 56.8% | — | 44.1%41.2%–47.0% | abc | 496 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.739 | 61.90chosen-answer Brier 0.2028 | 13.6%answered 92% | 32 ms | 37 ms | $0.026$25.82estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 35 answers, 35 right · 0 wrong · AUROC 0.729 Math 55.2% | — | 43.1%40.2%–46.0% | abc | 485 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 61.34chosen-answer Brier 0.2043 | 28.9%answered 82% | 45 ms | 55 ms | $0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 18 answers, 17 right · 1 wrong · AUROC 0.708 Math 56.0% | — | 42.3%39.5%–45.2% | abc | 476 / 1,125 | 17 right of 18 kept1 wrong · 1,107 skipped | 0.708 | 59.66chosen-answer Brier 0.2298 | 8.3%answered 95% | 96 ms | 115 ms | $0.006$5.89estimate · DGX Spark |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.724 Math 50.4% | — | 40.4%37.5%–43.3% | bc | 454 / 1,125 | 39 right of 39 kept0 mistakes · 1,086 skipped | 0.724 | 58.90chosen-answer Brier 0.2256 | 14.6%answered 91% | 38 ms | 48 ms | $0.030$29.51estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 35 answers, 35 right · 0 wrong · AUROC 0.729 Math 51.5% | — | 39.7%36.9%–42.6% | c | 447 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 59.26chosen-answer Brier 0.2122 | 12.6%answered 92% | 45 ms | 51 ms | $0.033$33.20estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 14 answers, 14 right · 0 wrong · AUROC 0.632 Math 36.8% | — | 32.3%29.6%–35.1% | d | 363 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.632 | 54.39chosen-answer Brier 0.2349 | 32.9%answered 75% | 34 ms | 42 ms | $0.002$2.26estimate · DGX Spark |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.660 Math 39.2% | — | 32.1%29.4%–34.9% | d | 361 / 1,125 | No answers at this threshold | 0.660 | 56.30chosen-answer Brier 0.1950 | 1.5%answered 99% | 123 ms | 152 ms | $0.091$90.99estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.601 Math 35.3% | — | 31.7%29.1%–34.5% | d | 357 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.601 | 55.15chosen-answer Brier 0.2143 | 15.7%answered 89% | 59 ms | 70 ms | $0.004$3.78estimate · DGX Spark |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.640 Math 38.7% | — | 30.8%28.1%–33.5% | d | 346 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.640 | 55.51chosen-answer Brier 0.1975 | 31.4%answered 76% | 40 ms | 45 ms | $0.003$2.52estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 3 answers, 3 right · 0 wrong · AUROC 0.620 Math 37.1% | — | 30.7%28.0%–33.4% | d | 345 / 1,125 | 3 right of 3 kept0 mistakes · 1,122 skipped | 0.620 | 55.05chosen-answer Brier 0.2058 | 86.5%answered 22% | 17 ms | 23 ms | $0.015$14.76estimate · RTX PRO 6000 Blackwell |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 9 answers, 9 right · 0 wrong · AUROC 0.617 Math 32.1% | — | 29.2%26.6%–31.9% | de | 328 / 1,125 | 9 right of 9 kept0 mistakes · 1,116 skipped | 0.617 | 53.53chosen-answer Brier 0.2210 | 26.1%answered 80% | 23 ms | 29 ms | $0.002$1.58estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.545 Math 28.2% | — | 27.0%24.5%–29.7% | def | 304 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.545 | 52.72chosen-answer Brier 0.2159 | 4.6%answered 97% | 16 ms | 19 ms | $0.001$1.16estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.573 Math 28.2% | — | 26.7%24.2%–29.3% | def | 300 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.573 | 53.45chosen-answer Brier 0.1976 | 1.2%answered 99% | 31 ms | 39 ms | $0.002$2.00estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.641 Math 29.7% | — | 24.7%22.3%–27.3% | ef | 278 / 1,125 | No answers at this threshold | 0.641 | 53.22chosen-answer Brier 0.1827 | 21.4%answered 83% | 27 ms | 31 ms | $0.002$1.79estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.527 Math 22.7% | — | 23.6%21.2%–26.1% | ef | 265 / 1,125 | No answers at this threshold | 0.527 | 51.99chosen-answer Brier 0.1958 | 12.4%answered 90% | 15 ms | 28 ms | $0.001$1.31estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.556 Math 20.6% | — | 23.2%20.8%–25.8% | f | 261 / 1,125 | No answers at this threshold | 0.556 | 52.19chosen-answer Brier 0.1883 | 76.1%answered 30% | 13 ms | 16 ms | $0.001$0.95estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.568 Math 28.7% | — | 22.4%20.1%–24.9% | f | 252 / 1,125 | No answers at this threshold | 0.568 | 51.86chosen-answer Brier 0.1868 | 8.3%answered 94% | 36 ms | 43 ms | $0.028$27.67estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Patrick-18 Base Labs Fastino EldanRing PelaAI StartLux tattrongvu Cloudflare Blockbrain Labs newsbang Valen-Team Apertix AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Dexy-Decision-9B-V2 Open · 9.0B | — | 45.3%1,125 | 58.8%58.8% kept (662) | 74.4%34.4% kept (387) | 91.7%15.0% kept (169) | 100.0%5.0% kept (56) |
| Elephant-1 Open · 9.0B | — | 46.4%1,125 | 66.5%47.8% kept (538) | 84.2%27.0% kept (304) | 94.9%14.0% kept (157) | 100.0%4.7% kept (53) |
| GLiDE 9B Open · 7.9B | — | 47.8%1,125 | 66.4%47.4% kept (533) | 87.9%22.8% kept (256) | 97.5%10.7% kept (120) | 100.0%2.8% kept (32) |
| Winnow-12B Open · 12B | — | 39.7%1,125 | 57.3%45.2% kept (508) | 65.5%32.4% kept (365) | 74.5%19.2% kept (216) | 100.0%2.8% kept (32) |
| KnowLine-9B-Gen1 Open · 9.0B | — | 44.3%1,125 | 68.7%31.2% kept (351) | 85.3%16.4% kept (184) | 97.9%8.4% kept (95) | 100.0%2.7% kept (30) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 46.6%1,125 | 71.5%38.3% kept (431) | 84.6%19.6% kept (221) | 98.9%8.2% kept (92) | 100.0%1.2% kept (14) |
| Jeffan-9B-1211 Open · 9.0B | — | 48.2%1,125 | 71.1%41.5% kept (467) | 86.4%22.9% kept (258) | 98.1%9.6% kept (108) | 100.0%0.9% kept (10) |
| KnowLine-12B-Gen1 Open · 12B | — | 50.6%1,125 | 74.3%44.6% kept (502) | 90.5%25.3% kept (285) | 99.1%9.7% kept (109) | 100.0%0.5% kept (6) |
| Clef-Flash Open · 9.0B | — | 38.9%1,125 | 52.4%40.9% kept (460) | 59.7%20.5% kept (231) | 77.9%6.0% kept (68) | none kept |
| Cygnet Open · 12B | — | 39.3%1,125 | 58.1%43.3% kept (487) | 71.2%18.2% kept (205) | 89.5%5.1% kept (57) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | 43.9%1,125 | 70.9%32.1% kept (361) | 86.9%14.9% kept (168) | 100.0%4.1% kept (46) | none kept |
| Adjudex 9B v5 Open · 9.0B | — | 46.4%1,125 | 68.9%44.4% kept (499) | 83.2%25.3% kept (285) | 95.8%12.7% kept (143) | 98.2%4.9% kept (55) |
| Valen-Decision-1.5-9B Open · 9.0B | — | 44.4%1,125 | 63.2%37.7% kept (424) | 77.9%17.7% kept (199) | 90.2%8.2% kept (92) | 94.6%3.3% kept (37) |
| Cerno 9B v1.2 Open · 9.0B | — | 43.3%1,125 | 56.1%59.4% kept (668) | 70.8%33.5% kept (377) | 89.6%14.6% kept (164) | 96.3%2.4% kept (27) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|
| Dexy-Decision-9B-V2 Open · 9.0B | — | 56 / 00.759 | 7 / 00.712 | 9 / 00.689 | 40 / 00.910 |
| Elephant-1 Open · 9.0B | — | 53 / 00.768 | 5 / 00.671 | 9 / 00.725 | 39 / 00.925 |
| GLiDE 9B Open · 7.9B | — | 32 / 00.749 | 3 / 00.637 | 4 / 00.729 | 25 / 00.911 |
| Winnow-12B Open · 12B | — | 32 / 00.727 | 1 / 00.608 | 6 / 00.670 | 25 / 00.835 |
| KnowLine-9B-Gen1 Open · 9.0B | — | 30 / 00.725 | 3 / 00.638 | 0 / 00.692 | 27 / 00.843 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 14 / 00.762 | 0 / 00.675 | 3 / 00.713 | 11 / 00.854 |
| Jeffan-9B-1211 Open · 9.0B | — | 10 / 00.753 | 2 / 00.663 | 5 / 00.704 | 3 / 00.866 |
| KnowLine-12B-Gen1 Open · 12B | — | 6 / 00.774 | 1 / 00.701 | 1 / 00.711 | 4 / 00.937 |
| Clef-Flash Open · 9.0B | — | No answers at this threshold0.659 | 0 / 00.537 | 0 / 00.670 | 0 / 00.790 |
| Cygnet Open · 12B | — | No answers at this threshold0.720 | 0 / 00.604 | 0 / 00.676 | 0 / 00.848 |
| KnowLine-9B-Gen2 Open · 9.0B | — | No answers at this threshold0.730 | 0 / 00.672 | 0 / 00.663 | 0 / 00.859 |
| Adjudex 9B v5 Open · 9.0B | — | 54 / 10.771 | 4 / 00.706 | 2 / 00.730 | 48 / 10.869 |
| Valen-Decision-1.5-9B Open · 9.0B | — | 35 / 20.701 | 5 / 10.598 | 5 / 10.709 | 25 / 00.785 |
| Cerno 9B v1.2 Open · 9.0B | — | 26 / 10.752 | 8 / 10.645 | 3 / 00.745 | 15 / 00.881 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.774 Math 66.3% | — | 50.6%47.7%–53.5% | a | 569 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.774 | 65.90chosen-answer Brier 0.1878 | 8.8%answered 95% | 68 ms | 88 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 10 answers, 10 right · 0 wrong · AUROC 0.753 Math 59.8% | — | 48.2%45.3%–51.1% | ab | 542 / 1,125 | 10 right of 10 kept0 mistakes · 1,115 skipped | 0.753 | 64.33chosen-answer Brier 0.1952 | 23.4%answered 87% | 52 ms | 66 ms | $0.039$39.35estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 32 answers, 32 right · 0 wrong · AUROC 0.749 Math 57.9% | — | 47.8%44.9%–50.7% | ab | 538 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.749 | 63.95chosen-answer Brier 0.1993 | 30.4%answered 82% | 208 ms | 251 ms | $0.013$13.29estimate · DGX Spark |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 14 answers, 14 right · 0 wrong · AUROC 0.762 Math 58.4% | — | 46.6%43.7%–49.5% | ab | 524 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.762 | 63.55chosen-answer Brier 0.1947 | 17.4%answered 90% | 160 ms | 175 ms | $0.010$10.04estimate · DGX Spark |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 55 answers, 54 right · 1 wrong · AUROC 0.771 Math 62.1% | — | 46.4%43.5%–49.3% | ab | 522 / 1,125 | 54 right of 55 kept1 wrong · 1,070 skipped | 0.771 | 63.48chosen-answer Brier 0.1945 | 12.2%answered 93% | 41 ms | 46 ms | $0.031$31.36estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 53 answers, 53 right · 0 wrong · AUROC 0.768 Math 61.3% | — | 46.4%43.5%–49.3% | ab | 522 / 1,125 | 53 right of 53 kept0 mistakes · 1,072 skipped | 0.768 | 63.39chosen-answer Brier 0.1962 | 75.6%answered 47% | 79 ms | 98 ms | $0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 56 answers, 56 right · 0 wrong · AUROC 0.759 Math 58.8% | — | 45.3%42.4%–48.3% | ab | 510 / 1,125 | 56 right of 56 kept0 mistakes · 1,069 skipped | 0.759 | 61.81chosen-answer Brier 0.2171 | 12.4%answered 93% | 47 ms | 53 ms | $0.034$33.81estimate · RTX PRO 6000 Blackwell |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 37 answers, 35 right · 2 wrong · AUROC 0.701 Math 55.3% | — | 44.4%41.5%–47.3% | bc | 499 / 1,125 | 35 right of 37 kept2 wrong · 1,088 skipped | 0.701 | 61.42chosen-answer Brier 0.2151 | 13.6%answered 92% | 60 ms | 67 ms | $0.042$42.42estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 30 answers, 30 right · 0 wrong · AUROC 0.725 Math 59.8% | — | 44.3%41.4%–47.2% | bc | 498 / 1,125 | 30 right of 30 kept0 mistakes · 1,095 skipped | 0.725 | 61.97chosen-answer Brier 0.2032 | 6.7%answered 96% | 50 ms | 61 ms | $0.036$36.27estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.730 Math 60.1% | — | 43.9%41.0%–46.8% | bc | 494 / 1,125 | No answers at this threshold | 0.730 | 61.90chosen-answer Brier 0.2011 | 19.5%answered 88% | 52 ms | 65 ms | $0.040$39.96estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 27 answers, 26 right · 1 wrong · AUROC 0.752 Math 56.0% | — | 43.3%40.4%–46.2% | bc | 487 / 1,125 | 26 right of 27 kept1 wrong · 1,098 skipped | 0.752 | 60.41chosen-answer Brier 0.2247 | 6.0%answered 97% | 209 ms | 269 ms | $0.013$13.35estimate · DGX Spark |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 32 answers, 32 right · 0 wrong · AUROC 0.727 Math 46.9% | — | 39.7%36.9%–42.6% | c | 447 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.727 | 58.43chosen-answer Brier 0.2287 | 29.7%answered 80% | 66 ms | 91 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.720 Math 44.3% | — | 39.3%36.5%–42.2% | c | 442 / 1,125 | No answers at this threshold | 0.720 | 58.97chosen-answer Brier 0.2135 | 11.3%answered 93% | 40 ms | 48 ms | $0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.659 Math 49.8% | — | 38.9%36.1%–41.8% | c | 438 / 1,125 | No answers at this threshold | 0.659 | 57.70chosen-answer Brier 0.2353 | 14.4%answered 91% | 110 ms | 137 ms | $0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
PelaAI Poindexter Labs Perplexity vLLM Semantic Router Fastino Nace.AI Turion.AI Cloudflare OmniJev Zefan Cai MachinePulse Base Labs Sthānika AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | — | 59.0%1,125 | 78.3%54.6% kept (614) | 89.4%37.1% kept (417) | 96.1%20.7% kept (233) | 100.0%7.9% kept (89) |
| Copernicus-31B Open · 31B | — | 58.7%1,125 | 82.3%47.6% kept (536) | 91.9%30.6% kept (344) | 99.0%17.2% kept (193) | 100.0%5.9% kept (66) |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 54.8%1,125 | 70.5%63.6% kept (715) | 82.3%41.7% kept (469) | 93.2%22.3% kept (251) | 100.0%5.5% kept (62) |
| KnowLine-31B-Gen2 Open · 31B | — | 58.9%1,125 | 79.2%57.8% kept (650) | 89.1%38.2% kept (430) | 97.4%20.9% kept (235) | 100.0%4.8% kept (54) |
| d3 Open · 26B | — | 54.5%1,125 | 75.4%53.1% kept (597) | 90.5%31.7% kept (357) | 95.8%17.1% kept (192) | 100.0%4.8% kept (54) |
| GLiDE v1.2 Open · 27B | — | 55.2%1,125 | 71.2%59.6% kept (671) | 89.2%33.6% kept (378) | 97.7%15.7% kept (177) | 100.0%4.0% kept (45) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 41.9%1,125 | 76.8%22.2% kept (250) | 82.2%12.0% kept (135) | 93.1%7.7% kept (87) | 100.0%2.9% kept (33) |
| d3-ultra Open · 26B | — | 54.6%1,125 | 62.7%75.7% kept (852) | 73.6%50.9% kept (573) | 93.0%25.3% kept (285) | 100.0%2.5% kept (28) |
| Turion Jev 31B Open · 31B | — | 52.3%1,125 | 81.5%37.9% kept (426) | 92.5%23.6% kept (266) | 97.7%11.6% kept (130) | 100.0%0.6% kept (7) |
| Clef Open · 27B | — | 40.8%1,125 | 53.0%54.5% kept (613) | 65.5%25.8% kept (290) | 85.3%9.1% kept (102) | 100.0%0.5% kept (6) |
| OneJev-27B Open · 27B | — | 40.6%1,125 | 74.4%18.4% kept (207) | 91.1%10.0% kept (112) | 100.0%4.6% kept (52) | 100.0%0.4% kept (4) |
| Open-Jev-27B-v1.1 Open · 27B | — | 42.6%1,125 | 95.5%5.9% kept (66) | 100.0%0.4% kept (4) | none kept | none kept |
| Karpo Decision 31B Open · 31B | — | 60.7%1,125 | 75.3%65.2% kept (734) | 89.4%44.5% kept (501) | 96.3%28.8% kept (324) | 98.3%15.3% kept (172) |
| Drex Max Open · 27B | — | 53.1%1,125 | 63.7%72.1% kept (811) | 74.1%49.7% kept (559) | 86.5%30.8% kept (347) | 94.8%13.8% kept (155) |
| Mammoth-1 Open · 27B | — | 54.5%1,125 | 74.1%55.3% kept (622) | 86.3%36.3% kept (408) | 95.4%21.3% kept (240) | 98.1%9.6% kept (108) |
| Sieve 31B v3 Open · 31B | — | 57.9%1,125 | 78.1%53.2% kept (599) | 89.3%35.7% kept (402) | 96.3%21.6% kept (243) | 97.5%7.2% kept (81) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 52.5%1,125 | 75.0%49.3% kept (555) | 88.5%29.3% kept (330) | 95.7%16.7% kept (188) | 98.7%7.0% kept (79) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 53.3%1,125 | 77.0%47.6% kept (535) | 90.7%28.8% kept (324) | 95.9%17.2% kept (193) | 97.5%7.1% kept (80) |
| GLiDE 27B Open · 27B | — | 58.4%1,125 | 76.8%60.3% kept (678) | 88.6%38.3% kept (431) | 97.7%18.9% kept (213) | 98.7%6.7% kept (75) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Open · 31B | — | 89 / 00.804 | 25 / 00.765 | 14 / 00.764 | 50 / 00.904 |
| Copernicus-31B Open · 31B | — | 66 / 00.801 | 15 / 00.746 | 12 / 00.775 | 39 / 00.935 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 62 / 00.803 | 22 / 00.752 | 5 / 00.764 | 35 / 00.878 |
| KnowLine-31B-Gen2 Open · 31B | — | 54 / 00.825 | 26 / 00.792 | 5 / 00.785 | 23 / 00.907 |
| d3 Open · 26B | — | 54 / 00.806 | 3 / 00.755 | 11 / 00.761 | 40 / 00.863 |
| GLiDE v1.2 Open · 27B | — | 45 / 00.793 | 3 / 00.732 | 6 / 00.739 | 36 / 00.906 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 33 / 00.717 | 3 / 00.652 | 0 / 00.690 | 30 / 00.815 |
| d3-ultra Open · 26B | — | 28 / 00.778 | 9 / 00.707 | 10 / 00.761 | 9 / 00.853 |
| Turion Jev 31B Open · 31B | — | 7 / 00.782 | 1 / 00.702 | 0 / 00.756 | 6 / 00.875 |
| Clef Open · 27B | — | 6 / 00.697 | 2 / 00.607 | 0 / 00.704 | 4 / 00.846 |
| OneJev-27B Open · 27B | — | 4 / 00.726 | 0 / 00.633 | 0 / 00.712 | 4 / 00.845 |
| Open-Jev-27B-v1.1 Open · 27B | — | No answers at this threshold0.733 | 0 / 00.623 | 0 / 00.716 | 0 / 00.861 |
| Karpo Decision 31B Open · 31B | — | 169 / 30.816 | 58 / 00.805 | 33 / 20.758 | 78 / 10.877 |
| Drex Max Open · 27B | — | 147 / 80.786 | 34 / 00.729 | 39 / 50.758 | 74 / 30.838 |
| Mammoth-1 Open · 27B | — | 106 / 20.795 | 21 / 00.703 | 13 / 00.797 | 72 / 20.905 |
| Sieve 31B v3 Open · 31B | — | 79 / 20.797 | 9 / 00.764 | 16 / 20.745 | 54 / 00.861 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 78 / 10.800 | 8 / 00.728 | 12 / 00.787 | 58 / 10.860 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 78 / 20.798 | 9 / 00.730 | 12 / 00.783 | 57 / 20.791 |
| GLiDE 27B Open · 27B | — | 74 / 10.822 | 6 / 00.754 | 17 / 00.800 | 51 / 10.888 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 172 answers, 169 right · 3 wrong · AUROC 0.816 Math 81.6% | — | 60.7%57.8%–63.5% | a | 683 / 1,125 | 169 right of 172 kept3 wrong · 953 skipped | 0.816 | 71.80chosen-answer Brier 0.1711 | 6.6%answered 97% | 173 ms | 211 ms | $0.128$128estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 89 answers, 89 right · 0 wrong · AUROC 0.804 Math 77.4% | — | 59.0%56.1%–61.9% | ab | 664 / 1,125 | 89 right of 89 kept0 mistakes · 1,036 skipped | 0.804 | 70.72chosen-answer Brier 0.1757 | 0.4%answered 100% | 983 ms | 1,279 ms | $0.061$61.43estimate · DGX Spark |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.825 Math 80.3% | — | 58.9%56.0%–61.8% | ab | 663 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.825 | 71.09chosen-answer Brier 0.1675 | 0.0%answered 100% | 151 ms | 204 ms | $0.112$112estimate · RTX PRO 6000 Blackwell |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 66 answers, 66 right · 0 wrong · AUROC 0.801 Math 77.3% | — | 58.7%55.8%–61.5% | ab | 660 / 1,125 | 66 right of 66 kept0 mistakes · 1,059 skipped | 0.801 | 70.48chosen-answer Brier 0.1771 | 1.9%answered 99% | 185 ms | 255 ms | $0.136$136estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 75 answers, 74 right · 1 wrong · AUROC 0.822 Math 72.0% | — | 58.4%55.5%–61.2% | ab | 657 / 1,125 | 74 right of 75 kept1 wrong · 1,050 skipped | 0.822 | 70.67chosen-answer Brier 0.1707 | 44.3%answered 79% | 544 ms | 655 ms | $0.033$33.16estimate · DGX Spark |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 81 answers, 79 right · 2 wrong · AUROC 0.797 Math 77.1% | — | 57.9%55.0%–60.7% | abc | 651 / 1,125 | 79 right of 81 kept2 wrong · 1,044 skipped | 0.797 | 70.03chosen-answer Brier 0.1781 | 5.5%answered 98% | 1,050 ms | 1,335 ms | $0.063$62.95estimate · DGX Spark |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 45 answers, 45 right · 0 wrong · AUROC 0.793 Math 69.9% | — | 55.2%52.3%–58.1% | bcd | 621 / 1,125 | 45 right of 45 kept0 mistakes · 1,080 skipped | 0.793 | 68.39chosen-answer Brier 0.1842 | 50.2%answered 73% | 111 ms | 140 ms | $0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 62 answers, 62 right · 0 wrong · AUROC 0.803 Math 72.5% | — | 54.8%51.8%–57.6% | bcd | 616 / 1,125 | 62 right of 62 kept0 mistakes · 1,063 skipped | 0.803 | 67.98chosen-answer Brier 0.1881 | 18.1%answered 91% | 108 ms | 131 ms | $0.081$81.16estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 28 answers, 28 right · 0 wrong · AUROC 0.778 Math 71.5% | — | 54.6%51.7%–57.5% | bcd | 614 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.778 | 66.46chosen-answer Brier 0.2166 | 10.1%answered 95% | 156 ms | 185 ms | $0.109$109estimate · RTX PRO 6000 Blackwell |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 108 answers, 106 right · 2 wrong · AUROC 0.795 Math 74.0% | — | 54.5%51.6%–57.4% | bcd | 613 / 1,125 | 106 right of 108 kept2 wrong · 1,017 skipped | 0.795 | 68.07chosen-answer Brier 0.1835 | 72.3%answered 58% | 221 ms | 244 ms | $0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 54 answers, 54 right · 0 wrong · AUROC 0.806 Math 70.4% | — | 54.5%51.6%–57.4% | bcd | 613 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.806 | 68.34chosen-answer Brier 0.1781 | 14.8%answered 93% | 76 ms | 98 ms | $0.057$57.18estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 80 answers, 78 right · 2 wrong · AUROC 0.798 Math 69.8% | — | 53.3%50.4%–56.2% | cd | 600 / 1,125 | 78 right of 80 kept2 wrong · 1,045 skipped | 0.798 | 67.64chosen-answer Brier 0.1805 | 14.4%answered 93% | 238 ms | 552 ms | $0.007$7.46 |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 155 answers, 147 right · 8 wrong · AUROC 0.786 Math 70.3% | — | 53.1%50.1%–56.0% | cd | 597 / 1,125 | 147 right of 155 kept8 wrong · 970 skipped | 0.786 | 65.68chosen-answer Brier 0.2170 | 7.3%answered 97% | 106 ms | 143 ms | $0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 79 answers, 78 right · 1 wrong · AUROC 0.800 Math 69.3% | with web | 52.5%49.6%–55.4% | d | 591 / 1,125 | 78 right of 79 kept1 wrong · 1,046 skipped | 0.800 | 67.24chosen-answer Brier 0.1806 | 37.7%answered 80% | 448 ms | 827 ms | $1.030$1,030 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.782 Math 69.0% | — | 52.3%49.3%–55.2% | d | 588 / 1,125 | 7 right of 7 kept0 mistakes · 1,118 skipped | 0.782 | 66.89chosen-answer Brier 0.1849 | 1.5%answered 99% | 385 ms | 533 ms | $0.025$24.91estimate · DGX Spark |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.733 Math 54.6% | — | 42.6%39.7%–45.5% | e | 479 / 1,125 | No answers at this threshold | 0.733 | 59.78chosen-answer Brier 0.2302 | 28.2%answered 82% | 413 ms | 576 ms | $0.294$294estimate · RTX PRO 6000 Blackwell |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 33 answers, 33 right · 0 wrong · AUROC 0.717 Math 59.0% | — | 41.9%39.0%–44.8% | e | 471 / 1,125 | 33 right of 33 kept0 mistakes · 1,092 skipped | 0.717 | 60.73chosen-answer Brier 0.2040 | 37.7%answered 75% | 345 ms | 504 ms | $0.018$18.41 |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 6 answers, 6 right · 0 wrong · AUROC 0.697 Math 51.1% | — | 40.8%38.0%–43.7% | e | 459 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.697 | 58.55chosen-answer Brier 0.2369 | 3.6%answered 98% | 196 ms | 222 ms | $0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.726 Math 53.8% | — | 40.6%37.8%–43.5% | e | 457 / 1,125 | 4 right of 4 kept0 mistakes · 1,121 skipped | 0.726 | 60.27chosen-answer Brier 0.2009 | 1.2%answered 99% | 108 ms | 136 ms | $0.086$86.07estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
REAX Logits Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 59.3%1,125 | 77.6%58.3% kept (656) | 90.8%34.7% kept (390) | 98.2%19.6% kept (221) | 100.0%4.5% kept (51) |
| Logits Sigma-1 Duo Open · 57B | — | 57.8%1,125 | 75.4%57.5% kept (647) | 92.0%31.0% kept (349) | 97.6%14.6% kept (164) | 100.0%0.7% kept (8) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 51 / 00.816 | 12 / 00.750 | 11 / 00.804 | 28 / 00.921 |
| Logits Sigma-1 Duo Open · 57B | — | 8 / 00.796 | 1 / 00.738 | 1 / 00.764 | 6 / 00.880 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 51 answers, 51 right · 0 wrong · AUROC 0.816 Math 78.0% | — | 59.3%56.4%–62.1% | a | 667 / 1,125 | 51 right of 51 kept0 mistakes · 1,074 skipped | 0.816 | 71.07chosen-answer Brier 0.1714 | 5.3%answered 98% | 215 ms | 278 ms | $0.160$160estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 8 answers, 8 right · 0 wrong · AUROC 0.796 Math 75.2% | — | 57.8%54.9%–60.6% | a | 650 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.796 | 69.90chosen-answer Brier 0.1798 | 7.1%answered 97% | 285 ms | 399 ms | $0.211$211estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
TypeSafe Fastino Liquid AI Google Microsoft Celeris Nace.AI Inception Subquadratic Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Jev 1.13 Closed · Efficient | — | 46.3%1,125 | 68.4%44.7% kept (503) | 83.5%22.6% kept (254) | 94.4%11.2% kept (126) | 100.0%5.8% kept (65) |
| Jev 1.14 Closed · Efficient | — | 48.2%1,125 | 72.6%39.9% kept (449) | 84.2%21.4% kept (241) | 92.6%10.8% kept (121) | 100.0%5.4% kept (61) |
| GLiDE API Closed · Efficient | — | 54.5%1,125 | 72.9%58.5% kept (658) | 85.0%36.1% kept (406) | 96.8%16.7% kept (188) | 100.0%4.6% kept (52) |
| d1.1 Closed · Efficient | — | 46.0%1,125 | 70.3%39.6% kept (445) | 84.6%22.5% kept (253) | 94.7%11.7% kept (132) | 100.0%4.3% kept (48) |
| Gemini Decision 1 20261111 Closed · Efficient | — | 50.0%1,125 | 79.4%34.5% kept (388) | 91.5%18.9% kept (213) | 97.5%10.8% kept (121) | 100.0%3.8% kept (43) |
| Microsoft-Decision-1 Closed · Efficient | — | 44.4%1,125 | 61.9%43.9% kept (494) | 78.7%19.6% kept (221) | 94.4%8.0% kept (90) | 100.0%1.8% kept (20) |
| d1-ultra Closed · Efficient | — | 54.9%1,125 | 76.4%48.5% kept (546) | 86.2%29.7% kept (334) | 96.6%15.6% kept (176) | 100.0%1.8% kept (20) |
| celeris-1-decision Closed · Efficient | — | 47.3%1,125 | 61.9%53.7% kept (604) | 76.5%30.6% kept (344) | 90.5%15.9% kept (179) | none kept |
| celeris-1.1-decision Closed · Efficient | — | 47.3%1,125 | 62.9%52.7% kept (593) | 76.8%29.9% kept (336) | 90.8%16.4% kept (184) | none kept |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 54.8%1,125 | 71.8%58.0% kept (652) | 86.2%35.6% kept (400) | 95.5%19.9% kept (224) | 98.9%8.0% kept (90) |
| Mercury Decide Closed · Efficient | — | 45.1%1,125 | 61.0%53.6% kept (603) | 70.3%38.0% kept (428) | 79.9%22.1% kept (249) | 92.3%8.1% kept (91) |
| SubQ Snap 1.1 Closed · Efficient | — | 52.4%1,125 | 65.0%66.5% kept (748) | 77.9%42.2% kept (475) | 88.7%23.6% kept (266) | 97.2%6.4% kept (72) |
| d1 Closed · Efficient | — | 40.8%1,125 | 59.9%41.0% kept (461) | 74.2%21.7% kept (244) | 87.9%10.3% kept (116) | 98.2%5.0% kept (56) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|
| Jev 1.13 Closed · Efficient | — | 65 / 00.764 | 7 / 00.708 | 5 / 00.714 | 53 / 00.875 |
| Jev 1.14 Closed · Efficient | — | 61 / 00.750 | 5 / 00.671 | 8 / 00.711 | 48 / 00.856 |
| GLiDE API Closed · Efficient | — | 52 / 00.814 | 7 / 00.777 | 7 / 00.765 | 38 / 00.866 |
| d1.1 Closed · Efficient | — | 48 / 00.748 | 4 / 00.676 | 6 / 00.712 | 38 / 00.906 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 43 / 00.768 | 5 / 00.766 | 0 / 00.699 | 38 / 00.885 |
| Microsoft-Decision-1 Closed · Efficient | — | 20 / 00.712 | 1 / 00.628 | 2 / 00.683 | 17 / 00.823 |
| d1-ultra Closed · Efficient | — | 20 / 00.783 | 12 / 00.759 | 3 / 00.715 | 5 / 00.846 |
| celeris-1-decision Closed · Efficient | — | No answers at this threshold0.735 | 0 / 00.706 | 0 / 00.679 | 0 / 00.867 |
| celeris-1.1-decision Closed · Efficient | — | No answers at this threshold0.733 | 0 / 00.709 | 0 / 00.679 | 0 / 00.853 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 89 / 10.793 | 7 / 00.729 | 24 / 00.771 | 58 / 10.877 |
| Mercury Decide Closed · Efficient | — | 84 / 70.746 | 14 / 20.670 | 28 / 10.744 | 42 / 40.775 |
| SubQ Snap 1.1 Closed · Efficient | — | 70 / 20.787 | 17 / 00.736 | 18 / 10.743 | 35 / 10.857 |
| d1 Closed · Efficient | — | 55 / 10.723 | 3 / 00.630 | 5 / 00.683 | 47 / 10.828 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 1,125 of 1,125); shown once.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 20 answers, 20 right · 0 wrong · AUROC 0.783 Math 74.1% | — | 54.9%52.0%–57.8% | a | 618 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.783 | 68.09chosen-answer Brier 0.1876 | 91.0%answered 35% | 182 ms | 425 ms | $0.013$12.64 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 90 answers, 89 right · 1 wrong · AUROC 0.793 Math 70.1% | — | 54.8%51.8%–57.6% | a | 616 / 1,125 | 89 right of 90 kept1 wrong · 1,035 skipped | 0.793 | 68.07chosen-answer Brier 0.1861 | 54.1%answered 69% | 329 ms | 1,540 ms | $0.022$22.37 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 52 answers, 52 right · 0 wrong · AUROC 0.814 Math 70.8% | — | 54.5%51.6%–57.4% | a | 613 / 1,125 | 52 right of 52 kept0 mistakes · 1,073 skipped | 0.814 | 68.27chosen-answer Brier 0.1794 | 60.0%answered 67% | 271 ms | 291 ms | $0.052$52.03 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 72 answers, 70 right · 2 wrong · AUROC 0.787 Math 68.6% | — | 52.4%49.4%–55.3% | ab | 589 / 1,125 | 70 right of 72 kept2 wrong · 1,053 skipped | 0.787 | 65.94chosen-answer Brier 0.2047 | 65.7%answered 61% | 384 ms | 393 ms | $0.015$14.92 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 43 answers, 43 right · 0 wrong · AUROC 0.768 Math 71.3% | — | 50.0%47.1%–53.0% | abc | 563 / 1,125 | 43 right of 43 kept0 mistakes · 1,082 skipped | 0.768 | 65.37chosen-answer Brier 0.1930 | 2.0%answered 99% | 228 ms | 344 ms | $0.037$36.77 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 61 answers, 61 right · 0 wrong · AUROC 0.750 Math 65.1% | — | 48.2%45.3%–51.1% | bcd | 542 / 1,125 | 61 right of 61 kept0 mistakes · 1,064 skipped | 0.750 | 64.14chosen-answer Brier 0.1990 | 24.5%answered 86% | 166 ms | 193 ms | $0.026$25.64 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.735 Math 70.0% | — | 47.3%44.4%–50.2% | bcd | 532 / 1,125 | No answers at this threshold | 0.735 | 62.94chosen-answer Brier 0.2141 | 92.3%answered 28% | 390 ms | 394 ms | $0.015$15.40 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.733 Math 69.9% | — | 47.3%44.4%–50.2% | bcd | 532 / 1,125 | No answers at this threshold | 0.733 | 62.92chosen-answer Brier 0.2144 | 91.0%answered 29% | 383 ms | 390 ms | $0.002$1.54 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 65 answers, 65 right · 0 wrong · AUROC 0.764 Math 64.1% | — | 46.3%43.4%–49.2% | cd | 521 / 1,125 | 65 right of 65 kept0 mistakes · 1,060 skipped | 0.764 | 63.36chosen-answer Brier 0.1959 | 15.8%answered 91% | 167 ms | 203 ms | $0.026$25.64 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 48 answers, 48 right · 0 wrong · AUROC 0.748 Math 65.2% | — | 46.0%43.2%–49.0% | cd | 518 / 1,125 | 48 right of 48 kept0 mistakes · 1,077 skipped | 0.748 | 63.14chosen-answer Brier 0.1977 | 90.6%answered 28% | 176 ms | 272 ms | $0.006$6.02 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 91 answers, 84 right · 7 wrong · AUROC 0.746 Math 53.5% | — | 45.1%42.2%–48.0% | cde | 507 / 1,125 | 84 right of 91 kept7 wrong · 1,034 skipped | 0.746 | 61.18chosen-answer Brier 0.2272 | 20.5%answered 88% | 301 ms | 360 ms | $0.008$7.52 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 20 answers, 20 right · 0 wrong · AUROC 0.712 Math 59.4% | — | 44.4%41.6%–47.4% | de | 500 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.712 | 61.53chosen-answer Brier 0.2139 | 19.8%answered 88% | 315 ms | 362 ms | $0.013$13.42 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 56 answers, 55 right · 1 wrong · AUROC 0.723 Math 55.6% | — | 40.8%38.0%–43.7% | e | 459 / 1,125 | 55 right of 56 kept1 wrong · 1,069 skipped | 0.723 | 59.76chosen-answer Brier 0.2128 | 11.3%answered 93% | 188 ms | 290 ms | $0.006$6.02 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
No workhorse decision-model setup has answers on Logic yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
No flagship decision-model setup has answers on Logic yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Liquid AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | 49.2%1,125 | 73.8%35.6% kept (400) | 87.5%19.9% kept (224) | 95.3%11.4% kept (128) | 100.0%2.5% kept (28) |
| d1.2-ultra Closed · Undisclosed | — | 51.6%1,125 | 77.1%44.7% kept (503) | 88.6%28.2% kept (317) | 96.9%14.4% kept (162) | 100.0%1.2% kept (13) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Numerical reasoningkept R / W · AUROC | Verbal reasoningkept R / W · AUROC | Formal logickept R / W · AUROC |
|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | 28 / 00.742 | 3 / 00.676 | 2 / 00.719 | 23 / 00.874 |
| d1.2-ultra Closed · Undisclosed | — | 13 / 00.800 | 4 / 00.741 | 4 / 00.772 | 5 / 00.837 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.800 Math 69.9% | — | 51.6%48.6%–54.5% | a | 580 / 1,125 | 13 right of 13 kept0 mistakes · 1,112 skipped | 0.800 | 66.77chosen-answer Brier 0.1802 | 94.5%answered 28% | 220 ms | 341 ms | $0.013$12.64 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 28 answers, 28 right · 0 wrong · AUROC 0.742 Math 65.5% | — | 49.2%46.2%–52.1% | a | 553 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.742 | 64.51chosen-answer Brier 0.2013 | 78.8%answered 43% | 222 ms | 390 ms | $0.006$6.02 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Not ranked (only complete runs are): Spark-X2.6-4B (high · chat) (not running now, 20 of 1,125 stored, last answer 13 Nov 02:18 UTC); Grok 4.7 (chat) (paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)); Claude Fable 5.2 (chat) (not running now, 147 of 1,125 stored, last answer 12 Nov 11:32 UTC); Granite-4.2-3B (off · chat) (still running, 259 of 1,125); GPT-6 Luna (max · chat) (paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)); DeepSeek-V4-Flash (chat) (sample: a random subset by design, 317 answers stored, not the full 1,125); Claude Haiku 5.5 (max · chat) (paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Math 99.6% | — | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 56.4%answered 97% | 6,590 ms | 14,330 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Math 99.6% | — | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 45.5%answered 98% | 44,520 ms | 206,660 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Math 99.6% | — | 96.5%95.3%–97.5% | a | 1,086 / 1,125 | Not measured | Not measured | — | 63.2%answered 97% | 7,070 ms | 16,200 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Math 99.8% | — | 96.3%95.0%–97.2% | ab | 1,083 / 1,125 | Not measured | Not measured | — | 0.0%answered 100% | 11,570 ms | 38,640 ms | $5.690$5,690 |
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.8% | — | 96.2%94.9%–97.2% | abc | 1,082 / 1,125 | Not measured | Not measured | — | 38.2%answered 98% | 5,830 ms | 12,240 ms | $19.540$19,540 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Math 99.6% | — | 96.2%94.9%–97.2% | abc | 1,082 / 1,125 | Not measured | Not measured | — | 3.8%answered 100% | 6,680 ms | 40,040 ms | $1.647$1,647 |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Math 99.6% | — | 96.1%94.8%–97.1% | abcd | 1,081 / 1,125 | Not measured | Not measured | — | 76.1%answered 95% | 8,210 ms | 24,090 ms | — |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Math 99.5% | — | 95.6%94.3%–96.7% | abcde | 1,076 / 1,125 | Not measured | Not measured | — | 37.3%answered 98% | 13,180 ms | 144,250 ms | $4.121$4,121 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.7% | — | 95.5%94.1%–96.5% | abcdef | 1,074 / 1,125 | Not measured | Not measured | — | 41.3%answered 98% | 6,710 ms | 14,890 ms | $11.480$11,480 API list-price equivalent |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Math 99.6% | — | 95.5%94.1%–96.5% | abcdef | 1,074 / 1,125 | Not measured | Not measured | — | 22.6%answered 99% | 3,110 ms | 14,020 ms | $1.820$1,820 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Math 99.6% | — | 95.4%94.0%–96.5% | abcdef | 1,073 / 1,125 | Not measured | Not measured | — | 50.0%answered 97% | 13,290 ms | 78,510 ms | $5.489$5,489 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.7% | — | 95.2%93.8%–96.3% | abcdef | 1,071 / 1,125 | Not measured | Not measured | — | 8.3%answered 100% | 3,820 ms | 13,460 ms | $0.000$0.00 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Math 99.4% | — | 95.1%93.7%–96.2% | abcdef | 1,070 / 1,125 | Not measured | Not measured | — | 23.3%answered 98% | 24,740 ms | 179,930 ms | $1.390$1,390 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active Qwen3.5-397B-A17B on · chat · Alibaba · chat model · reasoning on Parameters: 397B MoE · 17B active Math running | — | 95.1%93.7%–96.2% | abcdef | 1,070 / 1,125 | Not measured | Not measured | — | 31.7%answered 98% | 84,650 ms | 167,520 ms | $13.426$13,426 |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Math 99.2% | — | 94.0%92.5%–95.3% | bcdefg | 1,058 / 1,125 | Not measured | Not measured | — | 39.1%answered 97% | 18,710 ms | 101,030 ms | $4.884$4,884estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Math 99.2% | — | 94.0%92.4%–95.2% | efg | 1,057 / 1,125 | Not measured | Not measured | — | 61.1%answered 95% | 5,640 ms | 10,330 ms | $0.000$0.00 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Math 98.2% | — | 94.0%92.4%–95.2% | cdefg | 1,057 / 1,125 | Not measured | Not measured | — | 38.3%answered 97% | 5,610 ms | 37,430 ms | $0.803$803 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Math 99.5% | — | 94.0%92.4%–95.2% | defg | 1,057 / 1,125 | Not measured | Not measured | — | 34.6%answered 98% | 17,550 ms | 73,220 ms | $1.376$1,376 |
| Gemma 4 31B chat Open · 31B | — | 93.3%91.7%–94.6% | efgh | 1,050 / 1,125 | Not measured | Not measured | — | 16.3%answered 99% | 18,290 ms | 39,790 ms | $1.244$1,244estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Math 98.8% | — | 93.2%91.5%–94.5% | fgh | 1,048 / 1,125 | Not measured | Not measured | — | 4.2%answered 100% | 12,050 ms | 82,830 ms | $1.057$1,057 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Math 99.4% | — | 92.4%90.7%–93.8% | gh | 1,039 / 1,125 | Not measured | Not measured | — | 36.1%answered 97% | 142,520 ms | 246,280 ms | $3.525$3,525estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Math 99.2% | — | 91.6%89.9%–93.1% | ghi | 1,031 / 1,125 | Not measured | Not measured | — | 53.2%answered 94% | 221,350 ms | 1,229,550 ms | $10.893$10,893estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Math 98.5% | — | 90.1%88.3%–91.7% | hij | 1,014 / 1,125 | Not measured | Not measured | — | 33.6%answered 96% | 45,380 ms | 127,510 ms | $1.420$1,420estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Math 99.2% | — | 88.9%88.3%–89.6% min–max of 3 seeds | ijk | 998 / 1,125seed 0 run | Not measured | Not measured | — | 51.5%answered 92% | 101,100 ms | 333,427 ms | $3.335$3,335estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Math 98.8% | — | 87.7%85.7%–89.5% | jkl | 987 / 1,125 | Not measured | Not measured | — | 15.5%answered 98% | 42,060 ms | 82,940 ms | $1.292$1,292estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Math 99.0% | — | 86.7%84.6%–88.5% | jklm | 975 / 1,125 | Not measured | Not measured | — | 37.8%answered 95% | 228,980 ms | 776,930 ms | $7.562$7,562estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.5% | — | 85.9%85.5%–86.1% min–max of 3 seeds | klm | 968 / 1,125seed 0 run | Not measured | Not measured | — | 55.9%answered 90% | 31,567 ms | 83,467 ms | $0.993$993estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Math 98.7% | — | 85.5%83.3%–87.4% | klm | 962 / 1,125 | Not measured | Not measured | — | 54.2%answered 90% | 163,870 ms | 656,750 ms | $0.487$487estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.5% | — | 85.0%82.8%–86.9% | klm | 956 / 1,125 | Not measured | Not measured | — | Not measured | 57,900 ms | 109,440 ms | $1.503$1,503estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Math 98.9% | — | 84.6%84.4%–85.0% min–max of 3 seeds | klm | 956 / 1,125seed 0 run | Not measured | Not measured | — | Not measured | 27,950 ms | 107,497 ms | $1.040$1,040estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B | — | 84.2%83.9%–84.4% min–max of 3 seeds | klm | 950 / 1,125seed 0 run | Not measured | Not measured | — | 43.6%answered 91% | 29,847 ms | 113,220 ms | $1.017$1,017estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Math 98.2% | — | 83.6%83.1%–83.9% min–max of 3 seeds | lm | 942 / 1,125seed 0 run | Not measured | Not measured | — | 43.9%answered 88% | 46,453 ms | 111,607 ms | $1.327$1,327estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.1% | — | 83.2%80.9%–85.3% | m | 936 / 1,125 | Not measured | Not measured | — | 52.4%answered 88% | 32,690 ms | 64,070 ms | $0.842$842estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.2% | — | 82.6%80.3%–84.7% | m | 929 / 1,125 | Not measured | Not measured | — | 45.1%answered 91% | 284,080 ms | 539,980 ms | $0.650$650estimate · DGX Spark |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 97.7% | — | 76.8%76.0%–77.6% min–max of 3 seeds | n | 855 / 1,125seed 0 run | Not measured | Not measured | — | 32.6%answered 91% | 21,223 ms | 45,077 ms | $0.574$574estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 97.8% | — | 75.9%73.3%–78.3% | n | 854 / 1,125 | Not measured | Not measured | — | Not measured | 41,350 ms | 70,680 ms | $1.193$1,193estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Math 94.0% | — | 75.4%72.8%–77.8% | n | 848 / 1,125 | Not measured | Not measured | — | 44.3%answered 86% | 81,330 ms | 224,000 ms | $2.480$2,480estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 96.9% | — | 74.6%72.0%–77.0% | n | 839 / 1,125 | Not measured | Not measured | — | 26.4%answered 93% | 173,220 ms | 297,090 ms | $0.363$363estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 96.8% | — | 72.9%70.2%–75.4% | n | 820 / 1,125 | Not measured | Not measured | — | 34.0%answered 90% | 21,390 ms | 35,160 ms | $0.513$513estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Math 97.0% | — | 70.9%68.2%–73.5% | no | 798 / 1,125 | Not measured | Not measured | — | 10.5%answered 95% | 493,230 ms† | 1,396,520 ms† | $15.330$15,330estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B on · chat Open · 2.0B | — | 70.3%67.6%–72.9% | nop | 791 / 1,125 | Not measured | Not measured | — | 33.1%answered 91% | 186,330 ms | 272,210 ms | $4.330$4,330estimate · RTX PRO 6000 Blackwell |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Math 95.8% | — | 65.3%62.5%–68.1% | opq | 735 / 1,125 | Not measured | Not measured | — | 16.1%answered 94% | 16,560 ms | 178,680 ms | $1.772$1,772estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Math 89.6% | — | 65.1%62.2%–67.8% | opq | 732 / 1,125 | Not measured | Not measured | — | 35.7%answered 87% | 276,720 ms† | 965,630 ms† | $9.270$9,270estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Math 94.6% | — | 64.8%62.0%–67.5% | pq | 729 / 1,125 | Not measured | Not measured | — | 35.2%answered 86% | 71,290 ms | 113,470 ms | $1.605$1,605estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Math 96.3% | — | 60.2%57.3%–63.0% | qr | 677 / 1,125 | Not measured | Not measured | — | 58.3%answered 70% | 138,140 ms† | 403,380 ms† | $4.664$4,664estimate · RTX PRO 6000 Blackwell† |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Math 93.7% | — | 58.0%55.0%–60.8% | r | 652 / 1,125 | Not measured | Not measured | — | 55.1%answered 72% | 3,860 ms | 5,820 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Math 90.8% | — | 54.2%51.3%–57.1% | r | 610 / 1,125 | Not measured | Not measured | — | 30.7%answered 84% | 10,150 ms | 18,410 ms | $0.390$390estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Spark-X2.6-4B (high · chat) (not running now, 20 of 1,125 stored, last answer 13 Nov 02:18 UTC); Granite-4.2-3B (off · chat) (still running, 259 of 1,125).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Math 99.4% | — | 92.4%90.7%–93.8% | a | 1,039 / 1,125 | Not measured | Not measured | — | 36.1%answered 97% | 142,520 ms | 246,280 ms | $3.525$3,525estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Math 99.2% | — | 91.6%89.9%–93.1% | a | 1,031 / 1,125 | Not measured | Not measured | — | 53.2%answered 94% | 221,350 ms | 1,229,550 ms | $10.893$10,893estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Math 98.5% | — | 90.1%88.3%–91.7% | ab | 1,014 / 1,125 | Not measured | Not measured | — | 33.6%answered 96% | 45,380 ms | 127,510 ms | $1.420$1,420estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Math 99.2% | — | 88.9%88.3%–89.6% min–max of 3 seeds | bc | 998 / 1,125seed 0 run | Not measured | Not measured | — | 51.5%answered 92% | 101,100 ms | 333,427 ms | $3.335$3,335estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Math 98.7% | — | 85.5%83.3%–87.4% | cd | 962 / 1,125 | Not measured | Not measured | — | 54.2%answered 90% | 163,870 ms | 656,750 ms | $0.487$487estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Math 98.9% | — | 84.6%84.4%–85.0% min–max of 3 seeds | d | 956 / 1,125seed 0 run | Not measured | Not measured | — | Not measured | 27,950 ms | 107,497 ms | $1.040$1,040estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B | — | 84.2%83.9%–84.4% min–max of 3 seeds | d | 950 / 1,125seed 0 run | Not measured | Not measured | — | 43.6%answered 91% | 29,847 ms | 113,220 ms | $1.017$1,017estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Math 98.2% | — | 83.6%83.1%–83.9% min–max of 3 seeds | d | 942 / 1,125seed 0 run | Not measured | Not measured | — | 43.9%answered 88% | 46,453 ms | 111,607 ms | $1.327$1,327estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 97.7% | — | 76.8%76.0%–77.6% min–max of 3 seeds | e | 855 / 1,125seed 0 run | Not measured | Not measured | — | 32.6%answered 91% | 21,223 ms | 45,077 ms | $0.574$574estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 97.8% | — | 75.9%73.3%–78.3% | e | 854 / 1,125 | Not measured | Not measured | — | Not measured | 41,350 ms | 70,680 ms | $1.193$1,193estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Math 94.0% | — | 75.4%72.8%–77.8% | ef | 848 / 1,125 | Not measured | Not measured | — | 44.3%answered 86% | 81,330 ms | 224,000 ms | $2.480$2,480estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 96.9% | — | 74.6%72.0%–77.0% | efg | 839 / 1,125 | Not measured | Not measured | — | 26.4%answered 93% | 173,220 ms | 297,090 ms | $0.363$363estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Math 96.8% | — | 72.9%70.2%–75.4% | efg | 820 / 1,125 | Not measured | Not measured | — | 34.0%answered 90% | 21,390 ms | 35,160 ms | $0.513$513estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Math 97.0% | — | 70.9%68.2%–73.5% | fg | 798 / 1,125 | Not measured | Not measured | — | 10.5%answered 95% | 493,230 ms† | 1,396,520 ms† | $15.330$15,330estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B on · chat Open · 2.0B | — | 70.3%67.6%–72.9% | g | 791 / 1,125 | Not measured | Not measured | — | 33.1%answered 91% | 186,330 ms | 272,210 ms | $4.330$4,330estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Math 89.6% | — | 65.1%62.2%–67.8% | h | 732 / 1,125 | Not measured | Not measured | — | 35.7%answered 87% | 276,720 ms† | 965,630 ms† | $9.270$9,270estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Math 94.6% | — | 64.8%62.0%–67.5% | h | 729 / 1,125 | Not measured | Not measured | — | 35.2%answered 86% | 71,290 ms | 113,470 ms | $1.605$1,605estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Math 98.8% | — | 87.7%85.7%–89.5% | a | 987 / 1,125 | Not measured | Not measured | — | 15.5%answered 98% | 42,060 ms | 82,940 ms | $1.292$1,292estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.5% | — | 85.9%85.5%–86.1% min–max of 3 seeds | ab | 968 / 1,125seed 0 run | Not measured | Not measured | — | 55.9%answered 90% | 31,567 ms | 83,467 ms | $0.993$993estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.5% | — | 85.0%82.8%–86.9% | abc | 956 / 1,125 | Not measured | Not measured | — | Not measured | 57,900 ms | 109,440 ms | $1.503$1,503estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.1% | — | 83.2%80.9%–85.3% | bc | 936 / 1,125 | Not measured | Not measured | — | 52.4%answered 88% | 32,690 ms | 64,070 ms | $0.842$842estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Math 98.2% | — | 82.6%80.3%–84.7% | c | 929 / 1,125 | Not measured | Not measured | — | 45.1%answered 91% | 284,080 ms | 539,980 ms | $0.650$650estimate · DGX Spark |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Math 95.8% | — | 65.3%62.5%–68.1% | d | 735 / 1,125 | Not measured | Not measured | — | 16.1%answered 94% | 16,560 ms | 178,680 ms | $1.772$1,772estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Math 96.3% | — | 60.2%57.3%–63.0% | e | 677 / 1,125 | Not measured | Not measured | — | 58.3%answered 70% | 138,140 ms† | 403,380 ms† | $4.664$4,664estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Math 90.8% | — | 54.2%51.3%–57.1% | f | 610 / 1,125 | Not measured | Not measured | — | 30.7%answered 84% | 10,150 ms | 18,410 ms | $0.390$390estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Math 99.2% | — | 94.0%92.5%–95.3% | a | 1,058 / 1,125 | Not measured | Not measured | — | 39.1%answered 97% | 18,710 ms | 101,030 ms | $4.884$4,884estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B | — | 93.3%91.7%–94.6% | a | 1,050 / 1,125 | Not measured | Not measured | — | 16.3%answered 99% | 18,290 ms | 39,790 ms | $1.244$1,244estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Math 99.0% | — | 86.7%84.6%–88.5% | b | 975 / 1,125 | Not measured | Not measured | — | 37.8%answered 95% | 228,980 ms | 776,930 ms | $7.562$7,562estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): DeepSeek-V4-Flash (chat) (sample: a random subset by design, 317 answers stored, not the full 1,125).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Math 99.6% | — | 96.2%94.9%–97.2% | a | 1,082 / 1,125 | Not measured | Not measured | — | 3.8%answered 100% | 6,680 ms | 40,040 ms | $1.647$1,647 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Math 99.5% | — | 95.6%94.3%–96.7% | ab | 1,076 / 1,125 | Not measured | Not measured | — | 37.3%answered 98% | 13,180 ms | 144,250 ms | $4.121$4,121 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Math 99.6% | — | 95.4%94.0%–96.5% | ab | 1,073 / 1,125 | Not measured | Not measured | — | 50.0%answered 97% | 13,290 ms | 78,510 ms | $5.489$5,489 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Math 99.4% | — | 95.1%93.7%–96.2% | abc | 1,070 / 1,125 | Not measured | Not measured | — | 23.3%answered 98% | 24,740 ms | 179,930 ms | $1.390$1,390 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active Qwen3.5-397B-A17B on · chat · Alibaba · chat model · reasoning on Parameters: 397B MoE · 17B active Math running | — | 95.1%93.7%–96.2% | abc | 1,070 / 1,125 | Not measured | Not measured | — | 31.7%answered 98% | 84,650 ms | 167,520 ms | $13.426$13,426 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Math 98.2% | — | 94.0%92.4%–95.2% | bc | 1,057 / 1,125 | Not measured | Not measured | — | 38.3%answered 97% | 5,610 ms | 37,430 ms | $0.803$803 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Math 98.8% | — | 93.2%91.5%–94.5% | c | 1,048 / 1,125 | Not measured | Not measured | — | 4.2%answered 100% | 12,050 ms | 82,830 ms | $1.057$1,057 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)); Claude Haiku 5.5 (max · chat) (paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Math 99.6% | — | 95.5%94.1%–96.5% | a | 1,074 / 1,125 | Not measured | Not measured | — | 22.6%answered 99% | 3,110 ms | 14,020 ms | $1.820$1,820 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Math 93.7% | — | 58.0%55.0%–60.8% | b | 652 / 1,125 | Not measured | Not measured | — | 55.1%answered 72% | 3,860 ms | 5,820 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.7% | — | 95.5%94.1%–96.5% | a | 1,074 / 1,125 | Not measured | Not measured | — | 41.3%answered 98% | 6,710 ms | 14,890 ms | $11.480$11,480 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 147 of 1,125 stored, last answer 12 Nov 11:32 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.8% | — | 96.2%94.9%–97.2% | a | 1,082 / 1,125 | Not measured | Not measured | — | 38.2%answered 98% | 5,830 ms | 12,240 ms | $19.540$19,540 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Math 99.7% | — | 95.2%93.8%–96.3% | b | 1,071 / 1,125 | Not measured | Not measured | — | 8.3%answered 100% | 3,820 ms | 13,460 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Grok 4.7 (chat) (paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Math 99.6% | — | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 56.4%answered 97% | 6,590 ms | 14,330 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Math 99.6% | — | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 45.5%answered 98% | 44,520 ms | 206,660 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Math 99.6% | — | 96.5%95.3%–97.5% | a | 1,086 / 1,125 | Not measured | Not measured | — | 63.2%answered 97% | 7,070 ms | 16,200 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Math 99.8% | — | 96.3%95.0%–97.2% | a | 1,083 / 1,125 | Not measured | Not measured | — | 0.0%answered 100% | 11,570 ms | 38,640 ms | $5.690$5,690 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Math 99.2% | — | 94.0%92.4%–95.2% | b | 1,057 / 1,125 | Not measured | Not measured | — | 61.1%answered 95% | 5,640 ms | 10,330 ms | $0.000$0.00 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Math 99.5% | — | 94.0%92.4%–95.2% | b | 1,057 / 1,125 | Not measured | Not measured | — | 34.6%answered 98% | 17,550 ms | 73,220 ms | $1.376$1,376 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.