Logic and Math · Confidence
Confidence on Logic and Math
Logic and Math pooled: 3,288 questions; only setups that finished both. How to read this page · Method
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Perplexity Poindexter Labs TypeSafe Google Liquid AI REAX Fastino Nace.AI PelaAI StartLux tattrongvu Turion.AI Microsoft vLLM Semantic Router scar-ai OmniJev Logits Cloudflare Blockbrain Labs Other Metask Lab Zefan Cai Strands Agents Celeris MachinePulse Base Labs Subquadratic Inception Sthānika AI Apertix AI Patrick-18 newsbang quincyqiang Valen-Team victorbennett EldanRing Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 66.4%3,288 | 78.1%75.1% kept (2,469) | 88.0%56.4% kept (1,853) | 95.7%38.0% kept (1,249) | 100.0%11.7% kept (385) |
| Copernicus-31B Open · 31B | — | Decision | 71.0%3,288 | 89.0%64.6% kept (2,123) | 96.0%48.3% kept (1,588) | 99.3%31.5% kept (1,036) | 100.0%11.7% kept (384) |
| Jev 1.13 Closed · Efficient | — | Decision | 58.0%3,288 | 77.2%58.2% kept (1,913) | 89.6%35.8% kept (1,178) | 97.6%20.3% kept (669) | 100.0%9.5% kept (313) |
| Jev 1.14 Closed · Efficient | — | Decision | 59.3%3,288 | 79.8%54.3% kept (1,784) | 91.0%34.0% kept (1,117) | 97.2%19.5% kept (640) | 100.0%8.0% kept (263) |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 64.1%3,288 | 84.1%59.2% kept (1,948) | 92.5%42.8% kept (1,407) | 98.1%27.6% kept (906) | 100.0%7.8% kept (257) |
| d1-ultra Closed · Efficient | — | Decision | 67.5%3,288 | 84.9%64.9% kept (2,134) | 92.4%49.0% kept (1,610) | 98.2%32.2% kept (1,058) | 100.0%7.1% kept (233) |
| d1.1 Closed · Efficient | — | Decision | 58.6%3,288 | 79.6%53.7% kept (1,767) | 90.9%35.0% kept (1,150) | 97.0%20.4% kept (670) | 100.0%7.0% kept (231) |
| REAX 1.3 Open · 57B | — | Decision | 71.6%3,288 | 87.1%70.6% kept (2,320) | 96.0%50.3% kept (1,655) | 99.5%33.4% kept (1,098) | 100.0%7.0% kept (229) |
| GLiDE API Closed · Efficient | — | Decision | 65.2%3,288 | 79.2%71.0% kept (2,336) | 90.3%50.6% kept (1,664) | 98.3%30.3% kept (996) | 100.0%6.5% kept (213) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 53.1%3,288 | 79.7%42.5% kept (1,397) | 90.7%27.2% kept (893) | 97.0%16.2% kept (532) | 100.0%6.3% kept (207) |
| GLiDE v1.2 Open · 27B | — | Decision | 64.8%3,288 | 78.2%70.3% kept (2,312) | 91.8%48.6% kept (1,597) | 98.8%28.0% kept (922) | 100.0%5.4% kept (176) |
| d1.2 Closed · Undisclosed | — | Decision | 59.9%3,288 | 81.2%52.3% kept (1,718) | 93.0%34.0% kept (1,118) | 98.4%19.5% kept (640) | 100.0%5.1% kept (169) |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 54.5%3,288 | 75.0%49.1% kept (1,615) | 88.6%29.8% kept (981) | 98.0%16.4% kept (539) | 100.0%4.7% kept (153) |
| d1.2-ultra Closed · Undisclosed | — | Decision | 63.6%3,288 | 84.7%59.1% kept (1,944) | 93.4%41.6% kept (1,369) | 99.1%25.0% kept (822) | 100.0%3.9% kept (128) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 52.4%3,288 | 70.2%53.6% kept (1,762) | 81.4%32.8% kept (1,078) | 94.7%16.5% kept (543) | 100.0%3.6% kept (117) |
| GLiDE 9B Open · 7.9B | — | Decision | 54.5%3,288 | 69.6%61.2% kept (2,011) | 85.4%35.9% kept (1,182) | 95.3%17.4% kept (571) | 100.0%3.2% kept (105) |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 55.8%3,288 | 76.0%53.9% kept (1,771) | 89.5%32.9% kept (1,082) | 98.4%15.2% kept (501) | 100.0%2.3% kept (74) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 54.4%3,288 | 74.0%51.4% kept (1,689) | 85.0%31.3% kept (1,029) | 97.0%15.4% kept (505) | 100.0%1.9% kept (63) |
| Turion Jev 31B Open · 31B | — | Decision | 63.3%3,288 | 84.0%58.2% kept (1,913) | 92.7%41.7% kept (1,372) | 99.4%21.0% kept (692) | 100.0%1.5% kept (50) |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 54.3%3,288 | 70.4%57.1% kept (1,876) | 86.3%30.5% kept (1,003) | 97.2%11.8% kept (389) | 100.0%1.4% kept (47) |
| d3-ultra Open · 26B | — | Decision | 65.7%3,288 | 73.5%81.9% kept (2,694) | 82.7%62.6% kept (2,057) | 96.3%37.4% kept (1,231) | 100.0%1.2% kept (39) |
| vera-medium-omni Open · 4.2B | — | Decision | 52.5%3,288 | 74.9%46.3% kept (1,523) | 88.0%23.3% kept (766) | 99.1%9.8% kept (322) | 100.0%1.2% kept (39) |
| OneJev-27B Open · 27B | — | Decision | 49.3%3,288 | 80.3%32.3% kept (1,062) | 92.2%18.7% kept (616) | 98.6%8.5% kept (281) | 100.0%0.9% kept (30) |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 69.3%3,288 | 85.2%68.0% kept (2,236) | 96.8%45.1% kept (1,483) | 99.5%22.6% kept (742) | 100.0%0.7% kept (23) |
| Clef Open · 27B | — | Decision | 47.6%3,288 | 61.4%59.6% kept (1,959) | 75.3%33.0% kept (1,086) | 89.6%14.7% kept (483) | 100.0%0.5% kept (18) |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 34.1%3,288 | 51.8%24.8% kept (816) | 67.8%8.3% kept (273) | 88.1%2.6% kept (84) | 100.0%0.5% kept (16) |
| GLiDE 0.8B Open · 0.8B | — | Decision | 31.1%3,288 | 39.8%37.4% kept (1,230) | 49.1%14.3% kept (470) | 68.9%3.1% kept (103) | 100.0%0.4% kept (13) |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 60.9%3,288 | 81.1%55.3% kept (1,817) | 91.7%36.4% kept (1,197) | 99.1%16.2% kept (534) | 100.0%0.4% kept (13) |
| d1-2B Open · 1.9B | — | Decision | 34.9%3,288 | 56.6%21.7% kept (712) | 74.2%6.7% kept (221) | 88.2%2.1% kept (68) | 100.0%0.2% kept (7) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 35.9%3,288 | 66.0%18.4% kept (606) | 81.0%7.1% kept (232) | 94.4%2.2% kept (72) | 100.0%0.1% kept (4) |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 27.7%3,288 | 43.1%13.7% kept (450) | 57.9%3.3% kept (107) | 88.9%0.5% kept (18) | 100.0%0.1% kept (2) |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 27.8%3,288 | 39.9%15.5% kept (509) | 50.0%4.0% kept (130) | 71.4%0.6% kept (21) | 100.0%0.1% kept (2) |
| Clef-Flash Open · 9.0B | — | Decision | 46.1%3,288 | 63.4%43.4% kept (1,427) | 74.6%22.4% kept (735) | 89.5%7.0% kept (229) | none kept |
| Cygnet Open · 12B | — | Decision | 42.6%3,288 | 63.2%42.4% kept (1,394) | 79.1%16.0% kept (527) | 94.7%4.0% kept (132) | none kept |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 23.0%3,288 | 31.9%5.5% kept (182) | 33.3%1.2% kept (39) | 57.1%0.2% kept (7) | none kept |
| GLiDE 0.4B Open · 0.4B | — | Decision | 21.5%3,288 | 26.9%5.5% kept (182) | 36.6%1.2% kept (41) | 100.0%0.1% kept (3) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 54.6%3,288 | 78.4%45.4% kept (1,492) | 91.4%25.4% kept (836) | 99.6%8.2% kept (269) | none kept |
| Metask-Jev-4B Open · 4.5B | — | Decision | 36.7%3,288 | 64.0%17.2% kept (567) | 78.6%8.1% kept (266) | 89.1%1.7% kept (55) | none kept |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 50.5%3,288 | 95.7%8.6% kept (282) | 100.0%0.9% kept (29) | none kept | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 28.0%3,288 | 46.1%10.1% kept (332) | 62.7%3.6% kept (118) | 91.3%0.7% kept (23) | none kept |
| Strands Decider 2B Open · 2.0B | — | Decision | 26.5%3,288 | 44.4%9.8% kept (322) | 65.1%1.3% kept (43) | 100.0%0.1% kept (3) | none kept |
| celeris-1-decision Closed · Efficient | — | Decision | 62.2%3,288 | 75.8%69.1% kept (2,271) | 87.8%48.8% kept (1,606) | 95.4%32.0% kept (1,053) | none kept |
| celeris-1.1-decision Closed · Efficient | — | Decision | 62.1%3,288 | 76.2%68.8% kept (2,262) | 87.9%48.6% kept (1,598) | 95.7%32.2% kept (1,060) | none kept |
| Karpo Decision 31B Open · 31B | — | Decision | 74.4%3,288 | 85.1%79.4% kept (2,610) | 93.0%63.7% kept (2,095) | 97.7%48.1% kept (1,580) | 99.7%30.7% kept (1,009) |
| Drex Max Open · 27B | — | Decision | 64.4%3,288 | 72.4%81.7% kept (2,686) | 80.8%64.5% kept (2,120) | 91.5%47.3% kept (1,556) | 98.0%26.9% kept (884) |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 71.1%3,288 | 86.5%69.3% kept (2,279) | 93.9%53.0% kept (1,744) | 98.5%35.2% kept (1,156) | 99.8%16.8% kept (554) |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 73.0%3,288 | 87.1%72.9% kept (2,396) | 93.9%56.8% kept (1,868) | 99.0%38.8% kept (1,275) | 99.8%15.9% kept (524) |
| Mammoth-1 Open · 27B | — | Decision | 67.3%3,288 | 82.3%71.0% kept (2,334) | 92.4%53.0% kept (1,744) | 97.9%37.3% kept (1,228) | 99.6%15.8% kept (520) |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 63.0%3,288 | 73.0%76.7% kept (2,522) | 82.6%58.1% kept (1,911) | 92.7%38.6% kept (1,268) | 99.3%13.2% kept (433) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 64.1%3,288 | 81.5%64.6% kept (2,125) | 92.6%46.9% kept (1,541) | 97.8%31.4% kept (1,031) | 99.5%12.5% kept (411) |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 64.8%3,288 | 77.5%72.3% kept (2,377) | 88.6%53.3% kept (1,752) | 97.0%35.3% kept (1,162) | 99.8%12.3% kept (405) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 63.6%3,288 | 80.6%65.2% kept (2,145) | 91.8%47.0% kept (1,545) | 97.4%31.3% kept (1,028) | 99.8%12.3% kept (405) |
| Mercury Decide Closed · Efficient | — | Decision | 50.6%3,288 | 64.0%65.1% kept (2,142) | 73.9%45.8% kept (1,505) | 83.8%28.6% kept (941) | 95.7%11.3% kept (372) |
| Sieve 31B v3 Open · 31B | — | Decision | 70.5%3,288 | 86.8%67.7% kept (2,227) | 94.3%51.4% kept (1,691) | 98.7%34.4% kept (1,131) | 99.3%9.0% kept (296) |
| GLiDE 27B Open · 27B | — | Decision | 67.3%3,288 | 80.6%72.4% kept (2,381) | 90.6%53.0% kept (1,742) | 98.5%32.5% kept (1,068) | 99.3%8.3% kept (274) |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 51.7%3,288 | 62.1%70.1% kept (2,306) | 73.5%47.7% kept (1,569) | 86.8%27.2% kept (895) | 96.1%8.5% kept (281) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 54.2%3,288 | 66.2%69.4% kept (2,281) | 78.0%47.1% kept (1,549) | 91.9%26.2% kept (863) | 98.9%8.1% kept (266) |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 56.7%3,288 | 73.2%60.4% kept (1,985) | 85.5%38.9% kept (1,279) | 96.2%22.3% kept (732) | 99.2%7.7% kept (253) |
| Elephant-1 Open · 9.0B | — | Decision | 56.2%3,288 | 73.2%59.5% kept (1,957) | 86.1%38.2% kept (1,256) | 95.6%22.0% kept (724) | 99.6%7.2% kept (236) |
| d3 Open · 26B | — | Decision | 65.0%3,288 | 82.6%65.4% kept (2,150) | 94.1%46.2% kept (1,518) | 98.6%28.2% kept (927) | 99.5%6.6% kept (217) |
| d1 Closed · Efficient | — | Decision | 50.5%3,288 | 69.8%50.3% kept (1,653) | 82.8%28.5% kept (938) | 93.6%14.7% kept (484) | 99.5%6.3% kept (208) |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 47.0%3,288 | 59.2%60.7% kept (1,996) | 72.9%34.5% kept (1,133) | 85.5%17.6% kept (578) | 96.9%5.9% kept (194) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 51.1%3,288 | 68.4%55.4% kept (1,820) | 79.9%36.8% kept (1,211) | 89.0%20.5% kept (674) | 97.4%5.8% kept (190) |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 51.6%3,288 | 71.4%49.0% kept (1,611) | 84.7%27.3% kept (897) | 94.0%14.2% kept (466) | 98.8%5.1% kept (167) |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 47.5%3,288 | 66.3%47.5% kept (1,563) | 79.5%26.8% kept (881) | 90.5%13.8% kept (454) | 97.6%5.0% kept (165) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 52.1%3,288 | 66.8%62.0% kept (2,039) | 77.7%41.7% kept (1,371) | 90.0%22.1% kept (728) | 99.2%4.0% kept (131) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 51.3%3,288 | 63.8%62.0% kept (2,038) | 76.6%37.0% kept (1,215) | 91.5%17.8% kept (585) | 98.6%2.2% kept (72) |
| Winnow-12B Open · 12B | — | Decision | 44.4%3,288 | 62.2%46.6% kept (1,531) | 73.1%31.1% kept (1,022) | 83.8%17.2% kept (567) | 98.1%1.6% kept (53) |
| GLiDE 2B Open · 1.9B | — | Decision | 35.2%3,288 | 45.7%44.7% kept (1,469) | 55.7%19.5% kept (641) | 75.7%5.5% kept (181) | 96.7%0.9% kept (30) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 385 / 00.844 | 22 / 00.752 | 5 / 00.764 | 35 / 00.878 | 26 / 00.828 | 8 / 00.776 | 0 / 00.577 | 8 / 00.785 | 15 / 00.845 | 17 / 00.790 | 36 / 00.935 | 27 / 00.839 | 25 / 00.900 | 22 / 00.933 | 35 / 00.845 | 49 / 00.871 | 55 / 00.834 |
| Copernicus-31B Open · 31B | — | Decision | 384 / 00.873 | 15 / 00.746 | 12 / 00.775 | 39 / 00.935 | 27 / 00.901 | 13 / 00.865 | 1 / 00.800 | 9 / 00.881 | 17 / 00.892 | 20 / 00.881 | 42 / 00.980 | 12 / 00.884 | 25 / 00.876 | 19 / 00.937 | 29 / 00.882 | 48 / 00.914 | 56 / 00.875 |
| Jev 1.13 Closed · Efficient | — | Decision | 313 / 00.817 | 7 / 00.708 | 5 / 00.714 | 53 / 00.875 | 17 / 00.836 | 8 / 00.756 | 0 / 00.671 | 9 / 00.761 | 8 / 00.797 | 9 / 00.835 | 23 / 00.894 | 20 / 00.801 | 30 / 00.868 | 17 / 00.831 | 24 / 00.821 | 41 / 00.850 | 42 / 00.818 |
| Jev 1.14 Closed · Efficient | — | Decision | 263 / 00.810 | 5 / 00.671 | 8 / 00.711 | 48 / 00.856 | 16 / 00.854 | 7 / 00.791 | 0 / 00.686 | 8 / 00.700 | 5 / 00.778 | 8 / 00.810 | 18 / 00.910 | 14 / 00.788 | 25 / 00.862 | 18 / 00.826 | 21 / 00.781 | 27 / 00.849 | 35 / 00.834 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 257 / 00.838 | 5 / 00.766 | 0 / 00.699 | 38 / 00.885 | 15 / 00.862 | 6 / 00.794 | 2 / 00.729 | 0 / 00.789 | 5 / 00.854 | 10 / 00.851 | 18 / 00.882 | 13 / 00.797 | 23 / 00.833 | 15 / 00.799 | 31 / 00.859 | 38 / 00.907 | 38 / 00.858 |
| d1-ultra Closed · Efficient | — | Decision | 233 / 00.852 | 12 / 00.759 | 3 / 00.715 | 5 / 00.846 | 23 / 00.865 | 8 / 00.755 | 0 / 00.650 | 5 / 00.813 | 9 / 00.896 | 13 / 00.887 | 12 / 00.910 | 13 / 00.941 | 18 / 00.902 | 13 / 00.893 | 20 / 00.845 | 30 / 00.885 | 49 / 00.888 |
| d1.1 Closed · Efficient | — | Decision | 231 / 00.811 | 4 / 00.676 | 6 / 00.712 | 38 / 00.906 | 13 / 00.811 | 5 / 00.768 | 0 / 00.714 | 4 / 00.721 | 3 / 00.825 | 7 / 00.796 | 16 / 00.848 | 14 / 00.854 | 19 / 00.870 | 15 / 00.849 | 23 / 00.815 | 30 / 00.798 | 34 / 00.835 |
| REAX 1.3 Open · 57B | — | Decision | 229 / 00.878 | 12 / 00.750 | 11 / 00.804 | 28 / 00.921 | 11 / 00.888 | 7 / 00.826 | 1 / 00.818 | 7 / 00.869 | 9 / 00.886 | 11 / 00.917 | 30 / 00.955 | 10 / 00.946 | 14 / 00.881 | 7 / 00.935 | 22 / 00.887 | 25 / 00.913 | 24 / 00.900 |
| GLiDE API Closed · Efficient | — | Decision | 213 / 00.852 | 7 / 00.777 | 7 / 00.765 | 38 / 00.866 | 17 / 00.874 | 5 / 00.746 | 1 / 00.625 | 11 / 00.778 | 4 / 00.853 | 4 / 00.840 | 20 / 00.987 | 13 / 00.872 | 29 / 00.885 | 11 / 00.924 | 15 / 00.896 | 17 / 00.811 | 14 / 00.881 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 207 / 00.789 | 3 / 00.652 | 0 / 00.690 | 30 / 00.815 | 8 / 00.819 | 6 / 00.817 | 0 / 00.533 | 0 / 00.740 | 5 / 00.798 | 13 / 00.841 | 20 / 00.905 | 9 / 00.768 | 16 / 00.837 | 14 / 00.766 | 22 / 00.762 | 35 / 00.836 | 26 / 00.803 |
| GLiDE v1.2 Open · 27B | — | Decision | 176 / 00.843 | 3 / 00.732 | 6 / 00.739 | 36 / 00.906 | 12 / 00.857 | 3 / 00.740 | 0 / 00.528 | 5 / 00.777 | 4 / 00.834 | 5 / 00.838 | 20 / 00.963 | 14 / 00.865 | 22 / 00.912 | 7 / 00.948 | 15 / 00.873 | 14 / 00.849 | 10 / 00.877 |
| d1.2 Closed · Undisclosed | — | Decision | 169 / 00.812 | 3 / 00.676 | 2 / 00.719 | 23 / 00.874 | 8 / 00.833 | 5 / 00.800 | 0 / 00.557 | 3 / 00.708 | 3 / 00.840 | 9 / 00.794 | 20 / 00.896 | 8 / 00.862 | 12 / 00.843 | 10 / 00.797 | 24 / 00.843 | 20 / 00.838 | 19 / 00.845 |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 153 / 00.782 | 3 / 00.638 | 0 / 00.692 | 27 / 00.843 | 13 / 00.816 | 5 / 00.712 | 0 / 00.595 | 6 / 00.674 | 6 / 00.803 | 5 / 00.760 | 19 / 00.873 | 6 / 00.799 | 14 / 00.792 | 7 / 00.780 | 17 / 00.803 | 11 / 00.821 | 14 / 00.817 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 128 / 00.856 | 4 / 00.741 | 4 / 00.772 | 5 / 00.837 | 6 / 00.891 | 5 / 00.839 | 1 / 00.889 | 4 / 00.774 | 4 / 00.851 | 5 / 00.801 | 18 / 00.960 | 10 / 00.855 | 8 / 00.912 | 10 / 00.902 | 15 / 00.859 | 11 / 00.855 | 18 / 00.868 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 117 / 00.763 | 1 / 00.633 | 2 / 00.672 | 18 / 00.853 | 11 / 00.818 | 1 / 00.707 | 0 / 00.524 | 2 / 00.676 | 2 / 00.743 | 7 / 00.748 | 18 / 00.814 | 6 / 00.808 | 11 / 00.806 | 6 / 00.744 | 12 / 00.768 | 12 / 00.778 | 8 / 00.775 |
| GLiDE 9B Open · 7.9B | — | Decision | 105 / 00.786 | 3 / 00.637 | 4 / 00.729 | 25 / 00.911 | 7 / 00.800 | 2 / 00.737 | 0 / 00.731 | 3 / 00.700 | 1 / 00.716 | 4 / 00.752 | 15 / 00.805 | 3 / 00.774 | 10 / 00.873 | 4 / 00.853 | 7 / 00.776 | 8 / 00.851 | 9 / 00.765 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 74 / 00.798 | 2 / 00.663 | 5 / 00.704 | 3 / 00.866 | 4 / 00.865 | 4 / 00.677 | 0 / 00.769 | 3 / 00.732 | 3 / 00.844 | 3 / 00.745 | 15 / 00.811 | 4 / 00.797 | 6 / 00.888 | 2 / 00.825 | 5 / 00.804 | 7 / 00.808 | 8 / 00.768 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 63 / 00.779 | 0 / 00.675 | 3 / 00.713 | 11 / 00.854 | 3 / 00.777 | 1 / 00.678 | 0 / 00.394 | 1 / 00.703 | 1 / 00.730 | 3 / 00.735 | 11 / 00.887 | 3 / 00.774 | 6 / 00.803 | 5 / 00.830 | 4 / 00.775 | 2 / 00.788 | 9 / 00.779 |
| Turion Jev 31B Open · 31B | — | Decision | 50 / 00.844 | 1 / 00.702 | 0 / 00.756 | 6 / 00.875 | 5 / 00.868 | 1 / 00.770 | 0 / 00.757 | 1 / 00.809 | 0 / 00.904 | 3 / 00.817 | 3 / 00.926 | 1 / 00.854 | 6 / 00.823 | 1 / 00.940 | 8 / 00.853 | 5 / 00.866 | 9 / 00.852 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 47 / 00.769 | 1 / 00.628 | 2 / 00.683 | 17 / 00.823 | 5 / 00.789 | 0 / 00.696 | 0 / 00.615 | 1 / 00.670 | 0 / 00.747 | 0 / 00.750 | 4 / 00.848 | 4 / 00.806 | 5 / 00.837 | 1 / 00.801 | 2 / 00.768 | 3 / 00.824 | 2 / 00.794 |
| d3-ultra Open · 26B | — | Decision | 39 / 00.835 | 9 / 00.707 | 10 / 00.761 | 9 / 00.853 | 1 / 00.842 | 1 / 00.778 | 0 / 00.764 | 0 / 00.820 | 0 / 00.731 | 0 / 00.827 | 3 / 00.942 | 0 / 00.819 | 1 / 00.901 | 2 / 00.848 | 1 / 00.871 | 1 / 00.850 | 1 / 00.874 |
| vera-medium-omni Open · 4.2B | — | Decision | 39 / 00.777 | 0 / 00.659 | 0 / 00.710 | 6 / 00.876 | 2 / 00.782 | 3 / 00.704 | 0 / 00.636 | 1 / 00.695 | 1 / 00.836 | 1 / 00.706 | 13 / 00.860 | 0 / 00.848 | 2 / 00.837 | 1 / 00.797 | 4 / 00.770 | 1 / 00.809 | 4 / 00.784 |
| OneJev-27B Open · 27B | — | Decision | 30 / 00.770 | 0 / 00.633 | 0 / 00.712 | 4 / 00.845 | 2 / 00.768 | 0 / 00.645 | 0 / 00.806 | 1 / 00.750 | 0 / 00.767 | 1 / 00.778 | 2 / 00.884 | 1 / 00.758 | 1 / 00.811 | 3 / 00.787 | 4 / 00.757 | 6 / 00.785 | 5 / 00.789 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 23 / 00.868 | 1 / 00.738 | 1 / 00.764 | 6 / 00.880 | 2 / 00.900 | 0 / 00.841 | 0 / 00.667 | 0 / 00.847 | 1 / 00.888 | 0 / 00.872 | 2 / 00.950 | 0 / 00.919 | 1 / 00.910 | 1 / 00.939 | 4 / 00.895 | 1 / 00.871 | 3 / 00.911 |
| Clef Open · 27B | — | Decision | 18 / 00.747 | 2 / 00.607 | 0 / 00.704 | 4 / 00.846 | 2 / 00.751 | 1 / 00.708 | 0 / 00.875 | 2 / 00.723 | 0 / 00.720 | 0 / 00.798 | 3 / 00.778 | 0 / 00.793 | 1 / 00.744 | 0 / 00.878 | 2 / 00.748 | 0 / 00.802 | 1 / 00.774 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 16 / 00.626 | 0 / 00.501 | 0 / 00.571 | 8 / 00.756 | 0 / 00.570 | 0 / 00.578 | 0 / 00.533 | 0 / 00.508 | 0 / 00.540 | 0 / 00.655 | 0 / 00.667 | 1 / 00.779 | 4 / 00.736 | 1 / 00.699 | 1 / 00.644 | 0 / 00.656 | 1 / 00.587 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 13 / 00.602 | 0 / 00.557 | 0 / 00.608 | 9 / 00.700 | 1 / 00.538 | 0 / 00.498 | 0 / 00.467 | 0 / 00.610 | 0 / 00.579 | 0 / 00.525 | 0 / 00.710 | 1 / 00.604 | 0 / 00.601 | 0 / 00.661 | 1 / 00.673 | 0 / 00.564 | 1 / 00.616 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 13 / 00.810 | 1 / 00.701 | 1 / 00.711 | 4 / 00.937 | 2 / 00.838 | 0 / 00.736 | 0 / 00.814 | 0 / 00.752 | 0 / 00.840 | 0 / 00.833 | 5 / 00.915 | 0 / 00.800 | 0 / 00.824 | 0 / 00.916 | 0 / 00.775 | 0 / 00.857 | 0 / 00.794 |
| d1-2B Open · 1.9B | — | Decision | 7 / 00.648 | 0 / 00.549 | 0 / 00.606 | 3 / 00.787 | 0 / 00.634 | 0 / 00.629 | 0 / 00.318 | 0 / 00.572 | 1 / 00.582 | 0 / 00.589 | 0 / 00.721 | 1 / 00.778 | 1 / 00.625 | 0 / 00.718 | 1 / 00.709 | 0 / 00.657 | 0 / 00.677 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 4 / 00.675 | 0 / 00.539 | 0 / 00.652 | 2 / 00.767 | 0 / 00.661 | 0 / 00.601 | 0 / 00.333 | 0 / 00.560 | 0 / 00.665 | 0 / 00.655 | 0 / 00.788 | 0 / 00.813 | 1 / 00.755 | 0 / 00.703 | 0 / 00.614 | 0 / 00.658 | 1 / 00.679 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 2 / 00.590 | 0 / 00.520 | 0 / 00.531 | 2 / 00.746 | 0 / 00.486 | 0 / 00.603 | 0 / 00.667 | 0 / 00.628 | 0 / 00.573 | 0 / 00.490 | 0 / 00.683 | 0 / 00.792 | 0 / 00.660 | 0 / 00.653 | 0 / 00.603 | 0 / 00.557 | 0 / 00.565 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 2 / 00.576 | 0 / 00.508 | 0 / 00.538 | 2 / 00.608 | 0 / 00.548 | 0 / 00.601 | 0 / 00.533 | 0 / 00.519 | 0 / 00.678 | 0 / 00.474 | 0 / 00.543 | 0 / 00.726 | 0 / 00.690 | 0 / 00.619 | 0 / 00.605 | 0 / 00.609 | 0 / 00.559 |
| Clef-Flash Open · 9.0B | — | Decision | No answers at this threshold0.707 | 0 / 00.537 | 0 / 00.670 | 0 / 00.790 | 0 / 00.711 | 0 / 00.642 | 0 / 00.361 | 0 / 00.614 | 0 / 00.736 | 0 / 00.670 | 0 / 00.766 | 0 / 00.789 | 0 / 00.744 | 0 / 00.823 | 0 / 00.722 | 0 / 00.804 | 0 / 00.737 |
| Cygnet Open · 12B | — | Decision | No answers at this threshold0.729 | 0 / 00.604 | 0 / 00.676 | 0 / 00.848 | 0 / 00.730 | 0 / 00.617 | 0 / 00.600 | 0 / 00.690 | 0 / 00.639 | 0 / 00.726 | 0 / 00.800 | 0 / 00.788 | 0 / 00.796 | 0 / 00.697 | 0 / 00.680 | 0 / 00.768 | 0 / 00.761 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | No answers at this threshold0.511 | 0 / 00.516 | 0 / 00.547 | 0 / 00.584 | 0 / 00.524 | 0 / 00.554 | 0 / 00.550 | 0 / 00.476 | 0 / 00.474 | 0 / 00.505 | 0 / 00.412 | 0 / 00.522 | 0 / 00.412 | 0 / 00.460 | 0 / 00.566 | 0 / 00.543 | 0 / 00.518 |
| GLiDE 0.4B Open · 0.4B | — | Decision | No answers at this threshold0.520 | 0 / 00.531 | 0 / 00.545 | 0 / 00.639 | 0 / 00.496 | 0 / 00.528 | 0 / 00.875 | 0 / 00.509 | 0 / 00.606 | 0 / 00.458 | 0 / 00.370 | 0 / 00.632 | 0 / 00.450 | 0 / 00.478 | 0 / 00.493 | 0 / 00.498 | 0 / 00.515 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | No answers at this threshold0.786 | 0 / 00.672 | 0 / 00.663 | 0 / 00.859 | 0 / 00.788 | 0 / 00.711 | 0 / 00.712 | 0 / 00.671 | 0 / 00.801 | 0 / 00.819 | 0 / 00.834 | 0 / 00.841 | 0 / 00.847 | 0 / 00.824 | 0 / 00.774 | 0 / 00.806 | 0 / 00.817 |
| Metask-Jev-4B Open · 4.5B | — | Decision | No answers at this threshold0.659 | 0 / 00.591 | 0 / 00.604 | 0 / 00.830 | 0 / 00.614 | 0 / 00.544 | 0 / 00.933 | 0 / 00.617 | 0 / 00.737 | 0 / 00.613 | 0 / 00.633 | 0 / 00.619 | 0 / 00.666 | 0 / 00.727 | 0 / 00.660 | 0 / 00.624 | 0 / 00.664 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | No answers at this threshold0.759 | 0 / 00.623 | 0 / 00.716 | 0 / 00.861 | 0 / 00.816 | 0 / 00.607 | 0 / 00.625 | 0 / 00.674 | 0 / 00.731 | 0 / 00.751 | 0 / 00.762 | 0 / 00.745 | 0 / 00.847 | 0 / 00.818 | 0 / 00.776 | 0 / 00.727 | 0 / 00.749 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | No answers at this threshold0.605 | 0 / 00.580 | 0 / 00.615 | 0 / 00.694 | 0 / 00.611 | 0 / 00.460 | 0 / 00.667 | 0 / 00.546 | 0 / 00.440 | 0 / 00.612 | 0 / 00.567 | 0 / 00.693 | 0 / 00.672 | 0 / 00.676 | 0 / 00.551 | 0 / 00.503 | 0 / 00.580 |
| Strands Decider 2B Open · 2.0B | — | Decision | No answers at this threshold0.573 | 0 / 00.544 | 0 / 00.562 | 0 / 00.634 | 0 / 00.510 | 0 / 00.458 | 0 / 00.615 | 0 / 00.534 | 0 / 00.589 | 0 / 00.583 | 0 / 00.635 | 0 / 00.615 | 0 / 00.574 | 0 / 00.632 | 0 / 00.556 | 0 / 00.542 | 0 / 00.627 |
| celeris-1-decision Closed · Efficient | — | Decision | No answers at this threshold0.829 | 0 / 00.706 | 0 / 00.679 | 0 / 00.867 | 0 / 00.857 | 0 / 00.787 | 0 / 00.807 | 0 / 00.794 | 0 / 00.884 | 0 / 00.896 | 0 / 00.952 | 0 / 00.840 | 0 / 00.881 | 0 / 00.836 | 0 / 00.838 | 0 / 00.869 | 0 / 00.849 |
| celeris-1.1-decision Closed · Efficient | — | Decision | No answers at this threshold0.830 | 0 / 00.709 | 0 / 00.679 | 0 / 00.853 | 0 / 00.863 | 0 / 00.790 | 0 / 00.875 | 0 / 00.790 | 0 / 00.863 | 0 / 00.896 | 0 / 00.944 | 0 / 00.853 | 0 / 00.877 | 0 / 00.812 | 0 / 00.852 | 0 / 00.867 | 0 / 00.855 |
| Karpo Decision 31B Open · 31B | — | Decision | 1006 / 30.882 | 58 / 00.805 | 33 / 20.758 | 78 / 10.877 | 85 / 00.899 | 34 / 00.801 | 3 / 00.833 | 34 / 00.880 | 38 / 00.933 | 50 / 00.890 | 83 / 00.952 | 45 / 00.879 | 66 / 00.915 | 53 / 00.950 | 91 / 00.902 | 114 / 00.893 | 141 / 00.904 |
| Drex Max Open · 27B | — | Decision | 866 / 180.841 | 34 / 00.729 | 39 / 50.758 | 74 / 30.838 | 67 / 10.834 | 25 / 00.803 | 3 / 00.750 | 32 / 00.799 | 28 / 00.867 | 36 / 40.794 | 63 / 00.923 | 42 / 20.877 | 75 / 00.894 | 44 / 00.908 | 83 / 00.876 | 96 / 20.855 | 125 / 10.847 |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 553 / 10.870 | 25 / 00.765 | 14 / 00.764 | 50 / 00.904 | 38 / 00.874 | 15 / 00.850 | 1 / 00.750 | 14 / 00.881 | 22 / 00.903 | 32 / 00.900 | 60 / 00.956 | 18 / 00.912 | 35 / 00.872 | 33 / 00.904 | 55 / 10.885 | 64 / 00.914 | 77 / 00.888 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 523 / 10.882 | 26 / 00.792 | 5 / 00.785 | 23 / 00.907 | 38 / 00.916 | 15 / 00.862 | 2 / 00.742 | 14 / 00.847 | 24 / 00.902 | 37 / 00.921 | 70 / 00.956 | 12 / 00.871 | 28 / 00.882 | 28 / 00.916 | 56 / 10.863 | 68 / 00.908 | 77 / 00.886 |
| Mammoth-1 Open · 27B | — | Decision | 518 / 20.868 | 21 / 00.703 | 13 / 00.797 | 72 / 20.905 | 27 / 00.885 | 9 / 00.766 | 1 / 00.788 | 11 / 00.821 | 14 / 00.876 | 30 / 00.897 | 41 / 00.909 | 17 / 00.893 | 47 / 00.930 | 31 / 00.904 | 53 / 00.900 | 59 / 00.896 | 72 / 00.903 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 430 / 30.827 | 17 / 00.736 | 18 / 10.743 | 35 / 10.857 | 28 / 00.819 | 12 / 00.784 | 1 / 00.722 | 15 / 00.780 | 10 / 00.808 | 19 / 00.754 | 47 / 00.929 | 35 / 00.829 | 31 / 00.832 | 28 / 00.931 | 37 / 10.802 | 45 / 00.871 | 52 / 00.848 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 409 / 20.849 | 9 / 00.730 | 12 / 00.783 | 57 / 20.791 | 22 / 00.861 | 6 / 00.818 | 1 / 00.806 | 8 / 00.772 | 10 / 00.823 | 16 / 00.819 | 34 / 00.961 | 34 / 00.901 | 43 / 00.903 | 33 / 00.932 | 32 / 00.840 | 47 / 00.875 | 45 / 00.863 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 404 / 10.846 | 7 / 00.729 | 24 / 00.771 | 58 / 10.877 | 23 / 00.839 | 8 / 00.850 | 1 / 00.471 | 14 / 00.799 | 9 / 00.861 | 15 / 00.817 | 28 / 00.903 | 31 / 00.840 | 47 / 00.910 | 30 / 00.919 | 36 / 00.866 | 39 / 00.853 | 34 / 00.893 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 404 / 10.854 | 8 / 00.728 | 12 / 00.787 | 58 / 10.860 | 23 / 00.850 | 9 / 00.842 | 1 / 00.571 | 6 / 00.827 | 14 / 00.799 | 18 / 00.854 | 28 / 00.927 | 36 / 00.869 | 43 / 00.887 | 31 / 00.946 | 29 / 00.851 | 41 / 00.885 | 47 / 00.884 |
| Mercury Decide Closed · Efficient | — | Decision | 356 / 160.777 | 14 / 20.670 | 28 / 10.744 | 42 / 40.775 | 25 / 00.811 | 9 / 00.692 | 1 / 00.729 | 8 / 00.748 | 6 / 00.687 | 11 / 30.718 | 26 / 10.859 | 23 / 20.801 | 31 / 10.808 | 18 / 00.821 | 22 / 10.751 | 44 / 10.785 | 48 / 00.822 |
| Sieve 31B v3 Open · 31B | — | Decision | 294 / 20.865 | 9 / 00.764 | 16 / 20.745 | 54 / 00.861 | 24 / 00.871 | 6 / 00.848 | 0 / 00.903 | 7 / 00.856 | 8 / 00.898 | 10 / 00.863 | 26 / 00.956 | 15 / 00.918 | 22 / 00.907 | 14 / 00.939 | 24 / 00.895 | 31 / 00.879 | 28 / 00.848 |
| GLiDE 27B Open · 27B | — | Decision | 272 / 20.852 | 6 / 00.754 | 17 / 00.800 | 51 / 10.888 | 17 / 10.858 | 6 / 00.787 | 0 / 00.569 | 8 / 00.761 | 5 / 00.846 | 7 / 00.870 | 22 / 00.948 | 20 / 00.938 | 36 / 00.873 | 18 / 00.905 | 21 / 00.861 | 20 / 00.848 | 18 / 00.875 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 270 / 110.776 | 8 / 10.645 | 3 / 00.745 | 15 / 00.881 | 24 / 00.806 | 9 / 00.677 | 1 / 10.561 | 8 / 00.679 | 6 / 00.797 | 11 / 00.751 | 31 / 00.843 | 17 / 00.747 | 22 / 20.806 | 13 / 00.812 | 31 / 20.720 | 33 / 30.836 | 38 / 20.776 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 263 / 30.795 | 7 / 00.712 | 9 / 00.689 | 40 / 00.910 | 18 / 10.778 | 5 / 00.684 | 0 / 00.606 | 7 / 00.731 | 7 / 10.745 | 8 / 00.759 | 29 / 10.839 | 10 / 00.829 | 18 / 00.822 | 12 / 00.853 | 33 / 00.824 | 24 / 00.814 | 36 / 00.820 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 251 / 20.802 | 4 / 00.706 | 2 / 00.730 | 48 / 10.869 | 18 / 00.815 | 4 / 00.701 | 0 / 00.729 | 6 / 00.684 | 8 / 00.764 | 9 / 00.797 | 31 / 00.867 | 12 / 10.814 | 21 / 00.820 | 16 / 00.840 | 23 / 00.813 | 21 / 00.812 | 28 / 00.806 |
| Elephant-1 Open · 9.0B | — | Decision | 235 / 10.797 | 5 / 00.671 | 9 / 00.725 | 39 / 00.925 | 14 / 00.821 | 5 / 00.711 | 0 / 00.486 | 6 / 00.684 | 6 / 00.823 | 10 / 00.730 | 28 / 00.833 | 10 / 10.785 | 15 / 00.868 | 14 / 00.879 | 25 / 00.778 | 24 / 00.834 | 25 / 00.774 |
| d3 Open · 26B | — | Decision | 216 / 10.858 | 3 / 00.755 | 11 / 00.761 | 40 / 00.863 | 13 / 10.867 | 5 / 00.791 | 1 / 00.806 | 6 / 00.811 | 7 / 00.907 | 8 / 00.847 | 21 / 00.955 | 15 / 00.904 | 25 / 00.905 | 13 / 00.893 | 21 / 00.886 | 13 / 00.875 | 14 / 00.872 |
| d1 Closed · Efficient | — | Decision | 207 / 10.757 | 3 / 00.630 | 5 / 00.683 | 47 / 10.828 | 14 / 00.765 | 5 / 00.653 | 0 / 00.530 | 4 / 00.700 | 3 / 00.789 | 6 / 00.752 | 9 / 00.761 | 11 / 00.726 | 21 / 00.802 | 11 / 00.738 | 20 / 00.769 | 23 / 00.777 | 25 / 00.784 |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 188 / 60.743 | 1 / 00.644 | 8 / 00.699 | 30 / 00.866 | 14 / 20.793 | 5 / 00.668 | 0 / 00.424 | 5 / 00.719 | 2 / 00.797 | 5 / 20.655 | 16 / 00.735 | 8 / 10.772 | 19 / 00.732 | 7 / 00.768 | 16 / 00.771 | 24 / 10.729 | 28 / 00.726 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 185 / 50.769 | 3 / 00.642 | 1 / 00.693 | 31 / 00.879 | 7 / 10.781 | 4 / 00.688 | 0 / 00.300 | 2 / 00.694 | 3 / 00.780 | 9 / 00.732 | 26 / 00.884 | 10 / 10.807 | 21 / 00.854 | 11 / 10.773 | 15 / 10.730 | 19 / 00.806 | 23 / 10.753 |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 165 / 20.765 | 5 / 10.598 | 5 / 10.709 | 25 / 00.785 | 16 / 00.837 | 3 / 00.663 | 0 / 00.667 | 7 / 00.778 | 4 / 00.849 | 5 / 00.753 | 17 / 00.876 | 9 / 00.765 | 12 / 00.777 | 7 / 00.834 | 13 / 00.782 | 16 / 00.795 | 21 / 00.776 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 161 / 40.747 | 2 / 00.648 | 5 / 00.672 | 28 / 00.884 | 14 / 10.731 | 5 / 00.719 | 0 / 00.548 | 4 / 00.677 | 6 / 00.743 | 8 / 10.763 | 16 / 00.824 | 8 / 00.774 | 14 / 00.799 | 9 / 20.772 | 17 / 00.761 | 12 / 00.772 | 13 / 00.696 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 130 / 10.782 | 4 / 00.663 | 1 / 00.731 | 12 / 00.860 | 6 / 00.765 | 5 / 00.695 | 0 / 00.450 | 2 / 00.701 | 3 / 00.836 | 9 / 00.788 | 23 / 00.913 | 7 / 00.830 | 12 / 00.797 | 7 / 00.792 | 11 / 00.764 | 15 / 00.792 | 13 / 10.765 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 71 / 10.752 | 2 / 00.634 | 3 / 10.641 | 12 / 00.881 | 3 / 00.791 | 2 / 00.611 | 0 / 00.500 | 1 / 00.688 | 1 / 00.814 | 4 / 00.699 | 11 / 00.845 | 1 / 00.842 | 7 / 00.813 | 1 / 00.742 | 7 / 00.743 | 9 / 00.803 | 7 / 00.747 |
| Winnow-12B Open · 12B | — | Decision | 52 / 10.737 | 1 / 00.608 | 6 / 00.670 | 25 / 00.835 | 5 / 00.742 | 1 / 00.638 | 0 / 00.686 | 0 / 00.670 | 1 / 00.642 | 0 / 10.743 | 1 / 00.787 | 1 / 00.742 | 5 / 00.731 | 3 / 00.785 | 1 / 00.746 | 1 / 00.758 | 1 / 00.774 |
| GLiDE 2B Open · 1.9B | — | Decision | 29 / 10.637 | 1 / 00.525 | 1 / 00.637 | 12 / 00.787 | 3 / 10.625 | 0 / 00.569 | 0 / 00.519 | 1 / 00.554 | 0 / 00.590 | 1 / 00.676 | 1 / 00.646 | 1 / 00.728 | 3 / 00.663 | 2 / 00.596 | 1 / 00.617 | 1 / 00.616 | 1 / 00.639 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 46 setups: GPT-6 Astra (chat), Gemini 4 Carbon (chat), GPT-6.1 Sol (chat), Qwen4-Max (high · chat), Claude Opus 5.5 (chat), DeepSeek-V4.1-Flash (high · chat · 1017), Gemini 4 Hydrogen (chat), Claude Sonnet 5.5 (chat), Inkling 2 Small (high · chat), Claude Fable 5.1 (chat), Inkling-Small (high · chat · Jul 2026), Muse Ember (max · chat), MiMo-V2.6-Flash (on · chat), Qwen3.8 Flash (off · chat), Qwen3.8-27B (low · chat), GPT-6.1 Terra (chat), Gemma 4 31B (chat), Qwen3.5-4B (on · chat), GLM-5.3-Flash (high · chat), MiniMax-M3.1-Flash (high · chat), Granite-4.2-3B (high · chat), MiniCPM5-2B (on · chat), Qwen3.5-4B (off · chat), Gemma 4 12B Unified (chat), LFM2.5-24B-A2B (on · chat), Gemma 4 E4B (on · chat · 32k cap · sampled), LFM3-1.7B (off · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), LFM3-1.7B (on · chat), Gemma 4 E4B (on · chat · 32k cap), MiniCPM5-2B (off · chat), Gemma 4 E4B (chat), Gemma 4 E4B (on · chat), Gemma 4 E2B (on · chat · 32k cap · sampled), Gemma 4 E2B (on · chat · 32k cap), Gemma 4 E2B (on · chat), Gemma 4 E2B (chat), Raon-1 nano (on · chat), LFM2.5-2.6B (on · chat), Qwen3.5-2B (on · chat), Qwen3-14B (off · chat), Qwen3.5-2B (off · chat), LFM2.5-8B-A1B (on · chat), GPT-6 Luna (chat), Raon-1 nano (off · chat), Erk-14B (off · chat).
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 3,276 of 3,288); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 9 setups not yet: DeepSeek-V4-Flash (chat) (missing Logic); Qwen3.5-397B-A17B (on · chat) (missing Math); GPT-6 Luna (max · chat) (missing Logic and Math); Claude Haiku 5.5 (max · chat) (missing Logic and Math); Claude Fable 5.2 (chat) (missing Logic and Math); Grok 4.7 (chat) (missing Logic); Spark-X2.6-4B (high · chat) (missing Logic and Math); Granite-4.2-3B (off · chat) (missing Logic and Math); Gemini 4 Carbon (high · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Logic 96.7% · Math 99.6% | — | Chat | 98.6%98.2%–99.0% | a | 3,243 / 3,288 | Not measured | Not measured | — | 60.9%answered 99% | 5,580 ms | 9,580 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Logic 96.7% · Math 99.6% | — | Chat | 98.6%98.1%–98.9% | a | 3,242 / 3,288 | Not measured | Not measured | — | 53.3%answered 99% | 16,350 ms | 90,630 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Logic 96.5% · Math 99.6% | — | Chat | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | — | 66.2%answered 99% | 5,240 ms | 10,410 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Logic 96.3% · Math 99.8% | — | Chat | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | — | 10.7%answered 100% | 5,700 ms | 19,750 ms | $2.919$2,919 |
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 96.2% · Math 99.8% | — | Chat | 98.5%98.1%–98.9% | a | 3,240 / 3,288 | Not measured | Not measured | — | 41.3%answered 99% | 3,600 ms | 8,380 ms | $13.179$13,179 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Logic 96.2% · Math 99.6% | — | Chat | 98.4%98.0%–98.8% | ab | 3,237 / 3,288 | Not measured | Not measured | — | 6.3%answered 100% | 2,270 ms | 14,870 ms | $0.713$713 |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Logic 96.1% · Math 99.6% | — | Chat | 98.4%97.9%–98.8% | abc | 3,235 / 3,288 | Not measured | Not measured | — | 78.3%answered 98% | 4,920 ms | 12,150 ms | — |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.5% · Math 99.7% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,230 / 3,288 | Not measured | Not measured | — | 43.7%answered 99% | 4,030 ms | 10,080 ms | $7.042$7,042 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Logic 95.6% · Math 99.5% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,229 / 3,288 | Not measured | Not measured | — | 41.7%answered 99% | 2,870 ms | 35,530 ms | $1.830$1,830 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.2% · Math 99.7% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | — | 15.7%answered 100% | 4,940 ms | 8,620 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Logic 95.4% · Math 99.6% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | — | 50.6%answered 99% | 2,750 ms | 34,100 ms | $2.506$2,506 |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Logic 95.5% · Math 99.6% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | — | 31.3%answered 99% | 1,460 ms | 6,300 ms | $0.863$863 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Logic 95.1% · Math 99.4% | — | Chat | 98.0%97.4%–98.4% | abcde | 3,221 / 3,288 | Not measured | Not measured | — | 27.6%answered 99% | 10,590 ms | 53,970 ms | $0.615$615 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Logic 94.0% · Math 99.5% | — | Chat | 97.6%97.0%–98.1% | bcdef | 3,209 / 3,288 | Not measured | Not measured | — | 31.3%answered 99% | 6,780 ms | 36,290 ms | $0.732$732 |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Logic 94.0% · Math 99.2% | — | Chat | 97.4%96.8%–97.9% | cdef | 3,204 / 3,288 | Not measured | Not measured | — | 33.9%answered 99% | 11,200 ms | 39,560 ms | $2.020$2,020estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Logic 94.0% · Math 99.2% | — | Chat | 97.4%96.8%–97.9% | def | 3,203 / 3,288 | Not measured | Not measured | — | 65.0%answered 98% | 4,440 ms | 8,290 ms | $0.000$0.00 |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Logic 93.3% · Math 99.4% | — | Chat | 97.3%96.7%–97.8% | def | 3,200 / 3,288 | Not measured | Not measured | — | 17.0%answered 99% | 12,575 ms | 25,280 ms | $0.775$775estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Logic 92.4% · Math 99.4% | — | Chat | 97.0%96.4%–97.5% | ef | 3,190 / 3,288 | Not measured | Not measured | — | 34.2%answered 99% | 63,270 ms | 193,950 ms | $2.042$2,042estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Logic 93.2% · Math 98.8% | — | Chat | 96.9%96.2%–97.4% | efg | 3,185 / 3,288 | Not measured | Not measured | — | 7.4%answered 100% | 4,640 ms | 25,900 ms | $0.471$471 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Logic 94.0% · Math 98.2% | — | Chat | 96.8%96.1%–97.3% | fg | 3,182 / 3,288 | Not measured | Not measured | — | 38.9%answered 99% | 2,170 ms | 15,760 ms | $0.342$342 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Logic 91.6% · Math 99.2% | — | Chat | 96.6%96.0%–97.2% | fg | 3,177 / 3,288 | Not measured | Not measured | — | 54.7%answered 98% | 25,615 ms | 523,100 ms | $4.241$4,241estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Logic 88.9% · Math 99.2% | — | Chat | 95.7%95.4%–96.0% min–max of 3 seeds | gh | 3,145 / 3,288seed 0 run | Not measured | Not measured | — | 51.8%answered 97% | 19,260 ms | 194,917 ms | $1.410$1,410estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Logic 90.1% · Math 98.5% | — | Chat | 95.6%94.9%–96.3% | ghi | 3,144 / 3,288 | Not measured | Not measured | — | 34.0%answered 98% | 14,295 ms | 78,230 ms | $0.683$683estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Logic 87.7% · Math 98.8% | — | Chat | 95.0%94.2%–95.7% | hij | 3,125 / 3,288 | Not measured | Not measured | — | 15.3%answered 99% | 22,345 ms | 53,390 ms | $0.749$749estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Logic 86.7% · Math 99.0% | — | Chat | 94.8%94.0%–95.5% | hijk | 3,117 / 3,288 | Not measured | Not measured | — | 44.0%answered 97% | 42,425 ms | 432,510 ms | $3.205$3,205estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.9% · Math 98.5% | — | Chat | 94.2%94.0%–94.4% min–max of 3 seeds | hijkl | 3,104 / 3,288seed 0 run | Not measured | Not measured | — | 54.1%answered 96% | 13,945 ms | 51,987 ms | $0.549$549estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Logic 85.5% · Math 98.7% | — | Chat | 94.2%93.3%–94.9% | hijkl | 3,097 / 3,288 | Not measured | Not measured | — | 55.9%answered 96% | 29,040 ms | 343,820 ms | $0.206$206estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% · Math 98.9% | — | Chat | 94.0%93.8%–94.2% min–max of 3 seeds | hijkl | 3,098 / 3,288seed 0 run | Not measured | Not measured | — | Not measured | 5,667 ms | 58,313 ms | $0.449$449estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Logic 84.2% · Math 98.9% | — | Chat | 93.9%93.89%–93.95% min–max of 3 seeds | ijkl | 3,089 / 3,288seed 0 run | Not measured | Not measured | — | 46.7%answered 96% | 5,630 ms | 63,067 ms | $0.435$435estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% · Math 98.5% | — | Chat | 93.9%93.0%–94.7% | jkl | 3,087 / 3,288 | Not measured | Not measured | — | Not measured | 25,530 ms | 77,380 ms | $0.855$855estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Logic 83.6% · Math 98.2% | — | Chat | 93.2%92.9%–93.3% min–max of 3 seeds | jkl | 3,068 / 3,288seed 0 run | Not measured | Not measured | — | 41.0%answered 95% | 9,065 ms | 75,703 ms | $0.598$598estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 83.2% · Math 98.1% | — | Chat | 93.0%92.1%–93.8% | kl | 3,058 / 3,288 | Not measured | Not measured | — | 51.0%answered 95% | 13,150 ms | 44,990 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 82.6% · Math 98.2% | — | Chat | 92.9%92.0%–93.7% | l | 3,054 / 3,288 | Not measured | Not measured | — | 43.7%answered 96% | 131,535 ms | 392,620 ms | $0.375$375estimate · DGX Spark |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 76.8% · Math 97.7% | — | Chat | 90.5%90.2%–90.9% min–max of 3 seeds | mn | 2,966 / 3,288seed 0 run | Not measured | Not measured | — | 31.0%answered 97% | 8,873 ms | 32,503 ms | $0.325$325estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% · Math 97.8% | — | Chat | 90.3%89.3%–91.3% | m | 2,970 / 3,288 | Not measured | Not measured | — | Not measured | 19,100 ms | 54,630 ms | $0.678$678estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 74.6% · Math 96.9% | — | Chat | 89.3%88.2%–90.3% | mno | 2,936 / 3,288 | Not measured | Not measured | — | 24.3%answered 97% | 77,885 ms | 232,800 ms | $0.215$215estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 72.9% · Math 96.8% | — | Chat | 88.6%87.5%–89.6% | mnop | 2,913 / 3,288 | Not measured | Not measured | — | 32.3%answered 96% | 9,935 ms | 27,690 ms | $0.313$313estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% · Math 97.0% | — | Chat | 88.1%86.9%–89.1% | nop | 2,896 / 3,288 | Not measured | Not measured | — | Not measured | 54,985 ms† | 987,660 ms† | $6.344$6,344estimate · RTX PRO 6000 Blackwell† |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Logic 75.4% · Math 94.0% | — | Chat | 87.6%86.5%–88.7% | opq | 2,881 / 3,288 | Not measured | Not measured | — | 40.6%answered 95% | 14,035 ms | 142,970 ms | $1.046$1,046estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Logic 70.3% · Math 94.7% | — | Chat | 86.4%85.2%–87.5% | pqr | 2,840 / 3,288 | Not measured | Not measured | — | 30.9%answered 97% | 121,840 ms | 241,590 ms | $3.106$3,106estimate · RTX PRO 6000 Blackwell |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Logic 65.3% · Math 95.8% | — | Chat | 85.4%84.1%–86.5% | qr | 2,807 / 3,288 | Not measured | Not measured | — | 20.9%answered 96% | 9,380 ms | 35,110 ms | $0.840$840estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Logic 64.8% · Math 94.6% | — | Chat | 84.4%83.1%–85.6% | r | 2,776 / 3,288 | Not measured | Not measured | — | 34.3%answered 94% | 17,805 ms | 92,100 ms | $0.788$788estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Logic 60.2% · Math 96.3% | — | Chat | 83.9%82.6%–85.1% | rs | 2,759 / 3,288 | Not measured | Not measured | — | 55.9%answered 88% | 20,095 ms† | 230,570 ms† | $1.913$1,913estimate · RTX PRO 6000 Blackwell† |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Logic 58.0% · Math 93.7% | — | Chat | 81.4%80.1%–82.7% | s | 2,678 / 3,288 | Not measured | Not measured | — | Not measured | 3,830 ms | 5,300 ms | $0.000$0.00 |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 65.1% · Math 89.6% | — | Chat | 81.2%79.8%–82.5% | st | 2,669 / 3,288 | Not measured | Not measured | — | 31.1%answered 94% | 42,585 ms† | 593,950 ms† | $4.422$4,422estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Logic 54.2% · Math 90.8% | — | Chat | 78.3%76.8%–79.6% | t | 2,573 / 3,288 | Not measured | Not measured | — | 32.3%answered 92% | 8,740 ms | 16,420 ms | $0.308$308estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 1,009 answers, 1,006 right · 3 wrong · AUROC 0.882 Logic 60.7% · Math 81.6% | — | Decision | 74.4%72.9%–75.9% | u | 2,447 / 3,288 | 1,006 right of 1,009 kept3 wrong · 2,279 skipped | 0.882 | 81.13chosen-answer Brier 0.1215 | 5.7%answered 98% | 162 ms | 198 ms | $0.118$118estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 524 answers, 523 right · 1 wrong · AUROC 0.882 Logic 58.9% · Math 80.3% | — | Decision | 73.0%71.4%–74.5% | uv | 2,399 / 3,288 | 523 right of 524 kept1 wrong · 2,764 skipped | 0.882 | 80.27chosen-answer Brier 0.1242 | 0.1%answered 100% | 116 ms | 177 ms | $0.093$92.98estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 229 answers, 229 right · 0 wrong · AUROC 0.878 Logic 59.3% · Math 78.0% | — | Decision | 71.6%70.0%–73.1% | v | 2,354 / 3,288 | 229 right of 229 kept0 mistakes · 3,059 skipped | 0.878 | 79.30chosen-answer Brier 0.1300 | 3.8%answered 99% | 200 ms | 260 ms | $0.148$148estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 554 answers, 553 right · 1 wrong · AUROC 0.870 Logic 59.0% · Math 77.4% | — | Decision | 71.1%69.5%–72.6% | vw | 2,338 / 3,288 | 553 right of 554 kept1 wrong · 2,734 skipped | 0.870 | 78.92chosen-answer Brier 0.1327 | 0.7%answered 100% | 811 ms | 1,111 ms | $0.052$51.85estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 384 answers, 384 right · 0 wrong · AUROC 0.873 Logic 58.7% · Math 77.3% | — | Decision | 71.0%69.4%–72.5% | vw | 2,333 / 3,288 | 384 right of 384 kept0 mistakes · 2,904 skipped | 0.873 | 78.76chosen-answer Brier 0.1344 | 1.8%answered 99% | 128 ms | 218 ms | $0.102$102estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 296 answers, 294 right · 2 wrong · AUROC 0.865 Logic 57.9% · Math 77.1% | — | Decision | 70.5%68.9%–72.1% | vwx | 2,319 / 3,288 | 294 right of 296 kept2 wrong · 2,992 skipped | 0.865 | 78.49chosen-answer Brier 0.1354 | 3.8%answered 99% | 676 ms | 1,173 ms | $0.047$47.29estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 23 answers, 23 right · 0 wrong · AUROC 0.868 Logic 57.8% · Math 75.2% | — | Decision | 69.3%67.7%–70.8% | wxy | 2,277 / 3,288 | 23 right of 23 kept0 mistakes · 3,265 skipped | 0.868 | 77.60chosen-answer Brier 0.1405 | 4.4%answered 99% | 207 ms | 324 ms | $0.165$165estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 233 answers, 233 right · 0 wrong · AUROC 0.852 Logic 54.9% · Math 74.1% | — | Decision | 67.5%65.9%–69.1% | xyzA | 2,221 / 3,288 | 233 right of 233 kept0 mistakes · 3,055 skipped | 0.852 | 76.50chosen-answer Brier 0.1455 | 89.1%answered 52% | 180 ms | 426 ms | $0.008$8.37 |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 274 answers, 272 right · 2 wrong · AUROC 0.852 Logic 58.4% · Math 72.0% | — | Decision | 67.3%65.7%–68.9% | xyz | 2,214 / 3,288 | 272 right of 274 kept2 wrong · 3,014 skipped | 0.852 | 76.29chosen-answer Brier 0.1475 | 31.9%answered 89% | 450 ms | 587 ms | $0.029$28.55estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 520 answers, 518 right · 2 wrong · AUROC 0.868 Logic 54.5% · Math 74.0% | — | Decision | 67.3%65.7%–68.9% | yzA | 2,213 / 3,288 | 518 right of 520 kept2 wrong · 2,768 skipped | 0.868 | 76.68chosen-answer Brier 0.1394 | 71.4%answered 69% | 150 ms | 232 ms | $0.028$27.98 list$0.120$120estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 385 answers, 385 right · 0 wrong · AUROC 0.844 Logic 54.8% · Math 72.5% | — | Decision | 66.4%64.8%–68.0% | zAB | 2,184 / 3,288 | 385 right of 385 kept0 mistakes · 2,903 skipped | 0.844 | 75.44chosen-answer Brier 0.1554 | 13.5%answered 95% | 97 ms | 125 ms | $0.071$71.10estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.835 Logic 54.6% · Math 71.5% | — | Decision | 65.7%64.1%–67.3% | zABC | 2,161 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.835 | 74.31chosen-answer Brier 0.1711 | 7.5%answered 97% | 129 ms | 168 ms | $0.096$95.71estimate · RTX PRO 6000 Blackwell |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 213 answers, 213 right · 0 wrong · AUROC 0.852 Logic 54.5% · Math 70.8% | — | Decision | 65.2%63.6%–66.8% | zABC | 2,144 / 3,288 | 213 right of 213 kept0 mistakes · 3,075 skipped | 0.852 | 75.04chosen-answer Brier 0.1513 | 49.2%answered 80% | 266 ms | 284 ms | $0.036$36.42 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 217 answers, 216 right · 1 wrong · AUROC 0.858 Logic 54.5% · Math 70.4% | — | Decision | 65.0%63.3%–66.6% | zABC | 2,136 / 3,288 | 216 right of 217 kept1 wrong · 3,071 skipped | 0.858 | 75.16chosen-answer Brier 0.1464 | 9.3%answered 97% | 57 ms | 86 ms | $0.046$46.28estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 405 answers, 404 right · 1 wrong · AUROC 0.846 Logic 54.8% · Math 70.1% | — | Decision | 64.8%63.2%–66.5% | zABC | 2,132 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.846 | 74.61chosen-answer Brier 0.1562 | 48.3%answered 79% | 327 ms | 518 ms | $0.016$16.13 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 176 answers, 176 right · 0 wrong · AUROC 0.843 Logic 55.2% · Math 69.9% | — | Decision | 64.8%63.2%–66.5% | ABC | 2,132 / 3,288 | 176 right of 176 kept0 mistakes · 3,112 skipped | 0.843 | 74.68chosen-answer Brier 0.1548 | 42.3%answered 83% | 81 ms | 136 ms | $0.010$9.71 list$0.066$66.05estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 884 answers, 866 right · 18 wrong · AUROC 0.841 Logic 53.1% · Math 70.3% | — | Decision | 64.4%62.8%–66.0% | zABC | 2,118 / 3,288 | 866 right of 884 kept18 wrong · 2,404 skipped | 0.841 | 73.20chosen-answer Brier 0.1802 | 5.7%answered 98% | 81 ms | 130 ms | $0.016$16.31 list$0.064$63.74estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 411 answers, 409 right · 2 wrong · AUROC 0.849 Logic 53.3% · Math 69.8% | — | Decision | 64.1%62.5%–65.8% | BCD | 2,109 / 3,288 | 409 right of 411 kept2 wrong · 2,877 skipped | 0.849 | 74.56chosen-answer Brier 0.1502 | 10.3%answered 96% | 230 ms | 439 ms | $0.005$5.38 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 257 answers, 257 right · 0 wrong · AUROC 0.838 Logic 50.0% · Math 71.3% | — | Decision | 64.1%62.4%–65.7% | zABCD | 2,106 / 3,288 | 257 right of 257 kept0 mistakes · 3,031 skipped | 0.838 | 74.23chosen-answer Brier 0.1559 | 3.0%answered 99% | 227 ms | 347 ms | $0.027$27.07 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 128 answers, 128 right · 0 wrong · AUROC 0.856 Logic 51.6% · Math 69.9% | — | Decision | 63.6%61.9%–65.2% | BCD | 2,091 / 3,288 | 128 right of 128 kept0 mistakes · 3,160 skipped | 0.856 | 74.31chosen-answer Brier 0.1498 | 90.2%answered 48% | 217 ms | 347 ms | $0.008$8.37 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 405 answers, 404 right · 1 wrong · AUROC 0.854 Logic 52.5% · Math 69.3% | with web | Decision | 63.6%61.9%–65.2% | CD | 2,090 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.854 | 74.28chosen-answer Brier 0.1501 | 25.8%answered 90% | 430 ms | 707 ms | $1.025$1,025 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 50 answers, 50 right · 0 wrong · AUROC 0.844 Logic 52.3% · Math 69.0% | — | Decision | 63.3%61.6%–64.9% | BCDE | 2,081 / 3,288 | 50 right of 50 kept0 mistakes · 3,238 skipped | 0.844 | 73.88chosen-answer Brier 0.1553 | 1.9%answered 99% | 301 ms | 448 ms | $0.020$20.01estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 433 answers, 430 right · 3 wrong · AUROC 0.827 Logic 52.4% · Math 68.6% | — | Decision | 63.0%61.4%–64.7% | CDE | 2,072 / 3,288 | 430 right of 433 kept3 wrong · 2,855 skipped | 0.827 | 72.66chosen-answer Brier 0.1770 | 58.3%answered 73% | 382 ms | 391 ms | $0.011$10.75 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.829 Logic 47.3% · Math 70.0% | — | Decision | 62.2%60.6%–63.9% | CDEF | 2,046 / 3,288 | No answers at this threshold | 0.829 | 72.82chosen-answer Brier 0.1659 | 87.2%answered 50% | 388 ms | 392 ms | $0.011$11.25 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.830 Logic 47.3% · Math 69.9% | — | Decision | 62.1%60.5%–63.8% | CDEF | 2,043 / 3,288 | No answers at this threshold | 0.830 | 72.78chosen-answer Brier 0.1657 | 86.7%answered 51% | 381 ms | 388 ms | $0.001$1.13 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.810 Logic 50.6% · Math 66.3% | — | Decision | 60.9%59.2%–62.5% | DEFG | 2,002 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.810 | 71.86chosen-answer Brier 0.1716 | 4.5%answered 98% | 51 ms | 76 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 169 answers, 169 right · 0 wrong · AUROC 0.812 Logic 49.2% · Math 65.5% | — | Decision | 59.9%58.2%–61.6% | EFGH | 1,970 / 3,288 | 169 right of 169 kept0 mistakes · 3,119 skipped | 0.812 | 71.40chosen-answer Brier 0.1712 | 66.9%answered 65% | 208 ms | 390 ms | $0.004$3.98 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 263 answers, 263 right · 0 wrong · AUROC 0.810 Logic 48.2% · Math 65.1% | — | Decision | 59.3%57.6%–61.0% | FGHI | 1,951 / 3,288 | 263 right of 263 kept0 mistakes · 3,025 skipped | 0.810 | 71.00chosen-answer Brier 0.1734 | 15.4%answered 93% | 164 ms | 188 ms | $0.021$21.27 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 231 answers, 231 right · 0 wrong · AUROC 0.811 Logic 46.0% · Math 65.2% | — | Decision | 58.6%56.9%–60.3% | FGHI | 1,928 / 3,288 | 231 right of 231 kept0 mistakes · 3,057 skipped | 0.811 | 70.69chosen-answer Brier 0.1726 | 86.0%answered 46% | 174 ms | 285 ms | $0.004$3.98 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 313 answers, 313 right · 0 wrong · AUROC 0.817 Logic 46.3% · Math 64.1% | — | Decision | 58.0%56.3%–59.7% | GHIJ | 1,907 / 3,288 | 313 right of 313 kept0 mistakes · 2,975 skipped | 0.817 | 70.41chosen-answer Brier 0.1718 | 10.2%answered 96% | 166 ms | 205 ms | $0.021$21.27 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 253 answers, 251 right · 2 wrong · AUROC 0.802 Logic 46.4% · Math 62.1% | — | Decision | 56.7%55.0%–58.4% | HIJK | 1,865 / 3,288 | 251 right of 253 kept2 wrong · 3,035 skipped | 0.802 | 69.27chosen-answer Brier 0.1818 | 7.0%answered 97% | 38 ms | 45 ms | $0.028$28.19estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 236 answers, 235 right · 1 wrong · AUROC 0.797 Logic 46.4% · Math 61.3% | — | Decision | 56.2%54.5%–57.9% | IJK | 1,849 / 3,288 | 235 right of 236 kept1 wrong · 3,052 skipped | 0.797 | 68.96chosen-answer Brier 0.1831 | 69.4%answered 60% | 80 ms | 94 ms | $0.010$9.79 list$0.058$58.27estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 74 answers, 74 right · 0 wrong · AUROC 0.798 Logic 48.2% · Math 59.8% | — | Decision | 55.8%54.1%–57.5% | IJKL | 1,836 / 3,288 | 74 right of 74 kept0 mistakes · 3,214 skipped | 0.798 | 68.92chosen-answer Brier 0.1799 | 13.5%answered 94% | 39 ms | 53 ms | $0.032$32.40estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.786 Logic 43.9% · Math 60.1% | — | Decision | 54.6%52.9%–56.3% | JKLM | 1,794 / 3,288 | No answers at this threshold | 0.786 | 67.99chosen-answer Brier 0.1858 | 10.4%answered 95% | 42 ms | 57 ms | $0.034$34.08estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 105 answers, 105 right · 0 wrong · AUROC 0.786 Logic 47.8% · Math 57.9% | — | Decision | 54.5%52.8%–56.2% | JKLM | 1,791 / 3,288 | 105 right of 105 kept0 mistakes · 3,183 skipped | 0.786 | 67.72chosen-answer Brier 0.1902 | 26.9%answered 86% | 177 ms | 222 ms | $0.011$11.45estimate · DGX Spark |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 153 answers, 153 right · 0 wrong · AUROC 0.782 Logic 44.3% · Math 59.8% | — | Decision | 54.5%52.8%–56.2% | JKLMN | 1,791 / 3,288 | 153 right of 153 kept0 mistakes · 3,135 skipped | 0.782 | 67.96chosen-answer Brier 0.1854 | 3.4%answered 98% | 38 ms | 55 ms | $0.030$29.87estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 63 answers, 63 right · 0 wrong · AUROC 0.779 Logic 46.6% · Math 58.4% | — | Decision | 54.4%52.7%–56.1% | KLMNO | 1,788 / 3,288 | 63 right of 63 kept0 mistakes · 3,225 skipped | 0.779 | 67.73chosen-answer Brier 0.1893 | 9.6%answered 95% | 142 ms | 165 ms | $0.009$8.92estimate · DGX Spark |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 47 answers, 47 right · 0 wrong · AUROC 0.769 Logic 44.4% · Math 59.4% | — | Decision | 54.3%52.6%–56.0% | KLMNOP | 1,784 / 3,288 | 47 right of 47 kept0 mistakes · 3,241 skipped | 0.769 | 67.42chosen-answer Brier 0.1941 | 11.4%answered 94% | 312 ms | 360 ms | $0.009$9.07 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 266 answers, 263 right · 3 wrong · AUROC 0.795 Logic 45.3% · Math 58.8% | — | Decision | 54.2%52.5%–55.9% | KLMNOP | 1,782 / 3,288 | 263 right of 266 kept3 wrong · 3,022 skipped | 0.795 | 66.98chosen-answer Brier 0.2023 | 6.1%answered 97% | 46 ms | 51 ms | $0.033$32.61estimate · RTX PRO 6000 Blackwell |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 207 answers, 207 right · 0 wrong · AUROC 0.789 Logic 41.9% · Math 59.0% | — | Decision | 53.1%51.4%–54.8% | KLMNOP | 1,747 / 3,288 | 207 right of 207 kept0 mistakes · 3,081 skipped | 0.789 | 67.48chosen-answer Brier 0.1818 | 28.1%answered 85% | 339 ms | 472 ms | $0.012$12.17 |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.777 Logic 44.1% · Math 56.8% | — | Decision | 52.5%50.8%–54.2% | LMNOPQ | 1,725 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.777 | 66.69chosen-answer Brier 0.1909 | 7.9%answered 96% | 31 ms | 36 ms | $0.024$24.19estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 117 answers, 117 right · 0 wrong · AUROC 0.763 Logic 45.9% · Math 55.8% | — | Decision | 52.4%50.7%–54.1% | LMNOPQ | 1,724 / 3,288 | 117 right of 117 kept0 mistakes · 3,171 skipped | 0.763 | 66.33chosen-answer Brier 0.1978 | 14.4%answered 93% | 68 ms | 80 ms | $0.004$4.28estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 131 answers, 130 right · 1 wrong · AUROC 0.782 Logic 44.7% · Math 55.9% | — | Decision | 52.1%50.4%–53.8% | MNOPQ | 1,713 / 3,288 | 130 right of 131 kept1 wrong · 3,157 skipped | 0.782 | 65.93chosen-answer Brier 0.2024 | 15.0%answered 92% | 29 ms | 35 ms | $0.024$23.77estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 281 answers, 270 right · 11 wrong · AUROC 0.776 Logic 43.3% · Math 56.0% | — | Decision | 51.7%50.0%–53.4% | MNOPQ | 1,699 / 3,288 | 270 right of 281 kept11 wrong · 3,007 skipped | 0.776 | 64.84chosen-answer Brier 0.2199 | 3.1%answered 99% | 163 ms | 240 ms | $0.011$10.82estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 167 answers, 165 right · 2 wrong · AUROC 0.765 Logic 44.4% · Math 55.3% | — | Decision | 51.6%49.9%–53.3% | MNOPQ | 1,696 / 3,288 | 165 right of 167 kept2 wrong · 3,121 skipped | 0.765 | 65.99chosen-answer Brier 0.1960 | 7.1%answered 96% | 59 ms | 65 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 72 answers, 71 right · 1 wrong · AUROC 0.752 Logic 42.3% · Math 56.0% | — | Decision | 51.3%49.6%–53.0% | MNOPQR | 1,687 / 3,288 | 71 right of 72 kept1 wrong · 3,216 skipped | 0.752 | 65.07chosen-answer Brier 0.2116 | 4.4%answered 98% | 76 ms | 104 ms | $0.005$5.02estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 190 answers, 185 right · 5 wrong · AUROC 0.769 Logic 43.1% · Math 55.2% | — | Decision | 51.1%49.4%–52.8% | MNOPQR | 1,680 / 3,288 | 185 right of 190 kept5 wrong · 3,098 skipped | 0.769 | 65.47chosen-answer Brier 0.2016 | 18.9%answered 90% | 41 ms | 51 ms | $0.003$2.72 list$0.032$31.55estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 372 answers, 356 right · 16 wrong · AUROC 0.777 Logic 45.1% · Math 53.5% | — | Decision | 50.6%48.9%–52.3% | NOPQRS | 1,664 / 3,288 | 356 right of 372 kept16 wrong · 2,916 skipped | 0.777 | 64.36chosen-answer Brier 0.2189 | 10.1%answered 95% | 293 ms | 349 ms | $0.005$5.46 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 208 answers, 207 right · 1 wrong · AUROC 0.757 Logic 40.8% · Math 55.6% | — | Decision | 50.5%48.8%–52.3% | OPQRS | 1,662 / 3,288 | 207 right of 208 kept1 wrong · 3,080 skipped | 0.757 | 65.29chosen-answer Brier 0.1998 | 6.4%answered 97% | 185 ms | 292 ms | $0.004$3.98 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.759 Logic 42.6% · Math 54.6% | — | Decision | 50.5%48.7%–52.2% | PQRS | 1,659 / 3,288 | No answers at this threshold | 0.759 | 62.91chosen-answer Brier 0.2465 | 14.5%answered 92% | 250 ms | 482 ms | $0.212$212estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 30 answers, 30 right · 0 wrong · AUROC 0.770 Logic 40.6% · Math 53.8% | — | Decision | 49.3%47.6%–51.0% | QRST | 1,620 / 3,288 | 30 right of 30 kept0 mistakes · 3,258 skipped | 0.770 | 65.05chosen-answer Brier 0.1917 | 1.6%answered 99% | 80 ms | 134 ms | $0.069$69.42estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 18 answers, 18 right · 0 wrong · AUROC 0.747 Logic 40.8% · Math 51.1% | — | Decision | 47.6%45.9%–49.3% | RSTU | 1,564 / 3,288 | 18 right of 18 kept0 mistakes · 3,270 skipped | 0.747 | 62.92chosen-answer Brier 0.2172 | 2.4%answered 99% | 185 ms | 216 ms | $0.081$81.46 list$0.135$135estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 165 answers, 161 right · 4 wrong · AUROC 0.747 Logic 39.7% · Math 51.5% | — | Decision | 47.5%45.8%–49.2% | STU | 1,561 / 3,288 | 161 right of 165 kept4 wrong · 3,123 skipped | 0.747 | 63.44chosen-answer Brier 0.2059 | 8.9%answered 95% | 46 ms | 52 ms | $0.033$33.45estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 194 answers, 188 right · 6 wrong · AUROC 0.743 Logic 40.4% · Math 50.4% | — | Decision | 47.0%45.3%–48.7% | STU | 1,545 / 3,288 | 188 right of 194 kept6 wrong · 3,094 skipped | 0.743 | 62.33chosen-answer Brier 0.2233 | 7.8%answered 96% | 30 ms | 42 ms | $0.025$25.03estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.707 Logic 38.9% · Math 49.8% | — | Decision | 46.1%44.4%–47.8% | TUV | 1,515 / 3,288 | No answers at this threshold | 0.707 | 62.13chosen-answer Brier 0.2181 | 6.9%answered 96% | 103 ms | 131 ms | $0.013$12.90 list$0.075$75.10estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 53 answers, 52 right · 1 wrong · AUROC 0.737 Logic 39.7% · Math 46.9% | — | Decision | 44.4%42.7%–46.1% | UV | 1,461 / 3,288 | 52 right of 53 kept1 wrong · 3,235 skipped | 0.737 | 61.44chosen-answer Brier 0.2155 | 17.1%answered 90% | 55 ms | 72 ms | $0.043$42.91estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.729 Logic 39.3% · Math 44.3% | — | Decision | 42.6%40.9%–44.3% | V | 1,401 / 3,288 | No answers at this threshold | 0.729 | 60.88chosen-answer Brier 0.2086 | 7.3%answered 96% | 38 ms | 44 ms | $0.029$29.34 list$0.029$28.61estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.659 Logic 32.1% · Math 39.2% | — | Decision | 36.7%35.1%–38.4% | W | 1,208 / 3,288 | No answers at this threshold | 0.659 | 57.83chosen-answer Brier 0.2108 | 2.0%answered 99% | 117 ms | 146 ms | $0.086$85.83estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.675 Logic 30.8% · Math 38.7% | — | Decision | 35.9%34.3%–37.6% | W | 1,182 / 3,288 | 4 right of 4 kept0 mistakes · 3,284 skipped | 0.675 | 57.77chosen-answer Brier 0.2041 | 16.4%answered 89% | 36 ms | 42 ms | $0.002$2.36estimate · DGX Spark |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 30 answers, 29 right · 1 wrong · AUROC 0.637 Logic 32.3% · Math 36.8% | — | Decision | 35.2%33.6%–36.9% | W | 1,158 / 3,288 | 29 right of 30 kept1 wrong · 3,258 skipped | 0.637 | 55.25chosen-answer Brier 0.2472 | 23.1%answered 84% | 28 ms | 37 ms | $0.002$1.94estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.648 Logic 30.7% · Math 37.1% | — | Decision | 34.9%33.3%–36.5% | WX | 1,147 / 3,288 | 7 right of 7 kept0 mistakes · 3,281 skipped | 0.648 | 56.81chosen-answer Brier 0.2127 | 82.5%answered 30% | 17 ms | 21 ms | $0.014$14.30estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 16 answers, 16 right · 0 wrong · AUROC 0.626 Logic 31.7% · Math 35.3% | — | Decision | 34.1%32.5%–35.7% | WX | 1,120 / 3,288 | 16 right of 16 kept0 mistakes · 3,272 skipped | 0.626 | 56.04chosen-answer Brier 0.2199 | 6.8%answered 95% | 51 ms | 63 ms | $0.003$3.40estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.602 Logic 29.2% · Math 32.1% | — | Decision | 31.1%29.5%–32.7% | XY | 1,022 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.602 | 53.32chosen-answer Brier 0.2444 | 22.7%answered 83% | 20 ms | 25 ms | $0.001$1.38estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.605 Logic 24.7% · Math 29.7% | — | Decision | 28.0%26.5%–29.5% | YZ | 920 / 3,288 | No answers at this threshold | 0.605 | 54.00chosen-answer Brier 0.1999 | 14.9%answered 89% | 24 ms | 29 ms | $0.002$1.62estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.576 Logic 27.0% · Math 28.2% | — | Decision | 27.8%26.3%–29.4% | YZ | 915 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.576 | 53.31chosen-answer Brier 0.2120 | 2.0%answered 99% | 15 ms | 17 ms | $0.001$1.11estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.590 Logic 26.7% · Math 28.2% | — | Decision | 27.7%26.2%–29.2% | YZ | 910 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.590 | 53.55chosen-answer Brier 0.2057 | 0.6%answered 100% | 26 ms | 35 ms | $0.002$1.80estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.573 Logic 22.4% · Math 28.7% | — | Decision | 26.5%25.0%–28.1% | Zα | 872 / 3,288 | No answers at this threshold | 0.573 | 53.20chosen-answer Brier 0.2012 | 5.0%answered 96% | 35 ms | 42 ms | $0.027$26.72estimate · RTX PRO 6000 Blackwell |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.511 Logic 23.6% · Math 22.7% | — | Decision | 23.0%21.6%–24.4% | αβ | 755 / 3,288 | No answers at this threshold | 0.511 | 52.00chosen-answer Brier 0.1896 | 15.6%answered 87% | 12 ms | 23 ms | $0.001$1.01estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.520 Logic 23.2% · Math 20.6% | — | Decision | 21.5%20.1%–22.9% | β | 706 / 3,288 | No answers at this threshold | 0.520 | 51.39chosen-answer Brier 0.1869 | 73.9%answered 32% | 11 ms | 14 ms | $0.001$0.83estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
StartLux scar-ai PelaAI Fastino Liquid AI OmniJev Other Metask Lab Strands Agents quincyqiang Perplexity victorbennett Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 52.4%3,288 | 70.2%53.6% kept (1,762) | 81.4%32.8% kept (1,078) | 94.7%16.5% kept (543) | 100.0%3.6% kept (117) |
| vera-medium-omni Open · 4.2B | — | Decision | 52.5%3,288 | 74.9%46.3% kept (1,523) | 88.0%23.3% kept (766) | 99.1%9.8% kept (322) | 100.0%1.2% kept (39) |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 34.1%3,288 | 51.8%24.8% kept (816) | 67.8%8.3% kept (273) | 88.1%2.6% kept (84) | 100.0%0.5% kept (16) |
| GLiDE 0.8B Open · 0.8B | — | Decision | 31.1%3,288 | 39.8%37.4% kept (1,230) | 49.1%14.3% kept (470) | 68.9%3.1% kept (103) | 100.0%0.4% kept (13) |
| d1-2B Open · 1.9B | — | Decision | 34.9%3,288 | 56.6%21.7% kept (712) | 74.2%6.7% kept (221) | 88.2%2.1% kept (68) | 100.0%0.2% kept (7) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 35.9%3,288 | 66.0%18.4% kept (606) | 81.0%7.1% kept (232) | 94.4%2.2% kept (72) | 100.0%0.1% kept (4) |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 27.7%3,288 | 43.1%13.7% kept (450) | 57.9%3.3% kept (107) | 88.9%0.5% kept (18) | 100.0%0.1% kept (2) |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 27.8%3,288 | 39.9%15.5% kept (509) | 50.0%4.0% kept (130) | 71.4%0.6% kept (21) | 100.0%0.1% kept (2) |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 23.0%3,288 | 31.9%5.5% kept (182) | 33.3%1.2% kept (39) | 57.1%0.2% kept (7) | none kept |
| GLiDE 0.4B Open · 0.4B | — | Decision | 21.5%3,288 | 26.9%5.5% kept (182) | 36.6%1.2% kept (41) | 100.0%0.1% kept (3) | none kept |
| Metask-Jev-4B Open · 4.5B | — | Decision | 36.7%3,288 | 64.0%17.2% kept (567) | 78.6%8.1% kept (266) | 89.1%1.7% kept (55) | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 28.0%3,288 | 46.1%10.1% kept (332) | 62.7%3.6% kept (118) | 91.3%0.7% kept (23) | none kept |
| Strands Decider 2B Open · 2.0B | — | Decision | 26.5%3,288 | 44.4%9.8% kept (322) | 65.1%1.3% kept (43) | 100.0%0.1% kept (3) | none kept |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 47.0%3,288 | 59.2%60.7% kept (1,996) | 72.9%34.5% kept (1,133) | 85.5%17.6% kept (578) | 96.9%5.9% kept (194) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 51.1%3,288 | 68.4%55.4% kept (1,820) | 79.9%36.8% kept (1,211) | 89.0%20.5% kept (674) | 97.4%5.8% kept (190) |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 47.5%3,288 | 66.3%47.5% kept (1,563) | 79.5%26.8% kept (881) | 90.5%13.8% kept (454) | 97.6%5.0% kept (165) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 52.1%3,288 | 66.8%62.0% kept (2,039) | 77.7%41.7% kept (1,371) | 90.0%22.1% kept (728) | 99.2%4.0% kept (131) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 51.3%3,288 | 63.8%62.0% kept (2,038) | 76.6%37.0% kept (1,215) | 91.5%17.8% kept (585) | 98.6%2.2% kept (72) |
| GLiDE 2B Open · 1.9B | — | Decision | 35.2%3,288 | 45.7%44.7% kept (1,469) | 55.7%19.5% kept (641) | 75.7%5.5% kept (181) | 96.7%0.9% kept (30) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 117 / 00.763 | 1 / 00.633 | 2 / 00.672 | 18 / 00.853 | 11 / 00.818 | 1 / 00.707 | 0 / 00.524 | 2 / 00.676 | 2 / 00.743 | 7 / 00.748 | 18 / 00.814 | 6 / 00.808 | 11 / 00.806 | 6 / 00.744 | 12 / 00.768 | 12 / 00.778 | 8 / 00.775 |
| vera-medium-omni Open · 4.2B | — | Decision | 39 / 00.777 | 0 / 00.659 | 0 / 00.710 | 6 / 00.876 | 2 / 00.782 | 3 / 00.704 | 0 / 00.636 | 1 / 00.695 | 1 / 00.836 | 1 / 00.706 | 13 / 00.860 | 0 / 00.848 | 2 / 00.837 | 1 / 00.797 | 4 / 00.770 | 1 / 00.809 | 4 / 00.784 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 16 / 00.626 | 0 / 00.501 | 0 / 00.571 | 8 / 00.756 | 0 / 00.570 | 0 / 00.578 | 0 / 00.533 | 0 / 00.508 | 0 / 00.540 | 0 / 00.655 | 0 / 00.667 | 1 / 00.779 | 4 / 00.736 | 1 / 00.699 | 1 / 00.644 | 0 / 00.656 | 1 / 00.587 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 13 / 00.602 | 0 / 00.557 | 0 / 00.608 | 9 / 00.700 | 1 / 00.538 | 0 / 00.498 | 0 / 00.467 | 0 / 00.610 | 0 / 00.579 | 0 / 00.525 | 0 / 00.710 | 1 / 00.604 | 0 / 00.601 | 0 / 00.661 | 1 / 00.673 | 0 / 00.564 | 1 / 00.616 |
| d1-2B Open · 1.9B | — | Decision | 7 / 00.648 | 0 / 00.549 | 0 / 00.606 | 3 / 00.787 | 0 / 00.634 | 0 / 00.629 | 0 / 00.318 | 0 / 00.572 | 1 / 00.582 | 0 / 00.589 | 0 / 00.721 | 1 / 00.778 | 1 / 00.625 | 0 / 00.718 | 1 / 00.709 | 0 / 00.657 | 0 / 00.677 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 4 / 00.675 | 0 / 00.539 | 0 / 00.652 | 2 / 00.767 | 0 / 00.661 | 0 / 00.601 | 0 / 00.333 | 0 / 00.560 | 0 / 00.665 | 0 / 00.655 | 0 / 00.788 | 0 / 00.813 | 1 / 00.755 | 0 / 00.703 | 0 / 00.614 | 0 / 00.658 | 1 / 00.679 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 2 / 00.590 | 0 / 00.520 | 0 / 00.531 | 2 / 00.746 | 0 / 00.486 | 0 / 00.603 | 0 / 00.667 | 0 / 00.628 | 0 / 00.573 | 0 / 00.490 | 0 / 00.683 | 0 / 00.792 | 0 / 00.660 | 0 / 00.653 | 0 / 00.603 | 0 / 00.557 | 0 / 00.565 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 2 / 00.576 | 0 / 00.508 | 0 / 00.538 | 2 / 00.608 | 0 / 00.548 | 0 / 00.601 | 0 / 00.533 | 0 / 00.519 | 0 / 00.678 | 0 / 00.474 | 0 / 00.543 | 0 / 00.726 | 0 / 00.690 | 0 / 00.619 | 0 / 00.605 | 0 / 00.609 | 0 / 00.559 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | No answers at this threshold0.511 | 0 / 00.516 | 0 / 00.547 | 0 / 00.584 | 0 / 00.524 | 0 / 00.554 | 0 / 00.550 | 0 / 00.476 | 0 / 00.474 | 0 / 00.505 | 0 / 00.412 | 0 / 00.522 | 0 / 00.412 | 0 / 00.460 | 0 / 00.566 | 0 / 00.543 | 0 / 00.518 |
| GLiDE 0.4B Open · 0.4B | — | Decision | No answers at this threshold0.520 | 0 / 00.531 | 0 / 00.545 | 0 / 00.639 | 0 / 00.496 | 0 / 00.528 | 0 / 00.875 | 0 / 00.509 | 0 / 00.606 | 0 / 00.458 | 0 / 00.370 | 0 / 00.632 | 0 / 00.450 | 0 / 00.478 | 0 / 00.493 | 0 / 00.498 | 0 / 00.515 |
| Metask-Jev-4B Open · 4.5B | — | Decision | No answers at this threshold0.659 | 0 / 00.591 | 0 / 00.604 | 0 / 00.830 | 0 / 00.614 | 0 / 00.544 | 0 / 00.933 | 0 / 00.617 | 0 / 00.737 | 0 / 00.613 | 0 / 00.633 | 0 / 00.619 | 0 / 00.666 | 0 / 00.727 | 0 / 00.660 | 0 / 00.624 | 0 / 00.664 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | No answers at this threshold0.605 | 0 / 00.580 | 0 / 00.615 | 0 / 00.694 | 0 / 00.611 | 0 / 00.460 | 0 / 00.667 | 0 / 00.546 | 0 / 00.440 | 0 / 00.612 | 0 / 00.567 | 0 / 00.693 | 0 / 00.672 | 0 / 00.676 | 0 / 00.551 | 0 / 00.503 | 0 / 00.580 |
| Strands Decider 2B Open · 2.0B | — | Decision | No answers at this threshold0.573 | 0 / 00.544 | 0 / 00.562 | 0 / 00.634 | 0 / 00.510 | 0 / 00.458 | 0 / 00.615 | 0 / 00.534 | 0 / 00.589 | 0 / 00.583 | 0 / 00.635 | 0 / 00.615 | 0 / 00.574 | 0 / 00.632 | 0 / 00.556 | 0 / 00.542 | 0 / 00.627 |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 188 / 60.743 | 1 / 00.644 | 8 / 00.699 | 30 / 00.866 | 14 / 20.793 | 5 / 00.668 | 0 / 00.424 | 5 / 00.719 | 2 / 00.797 | 5 / 20.655 | 16 / 00.735 | 8 / 10.772 | 19 / 00.732 | 7 / 00.768 | 16 / 00.771 | 24 / 10.729 | 28 / 00.726 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 185 / 50.769 | 3 / 00.642 | 1 / 00.693 | 31 / 00.879 | 7 / 10.781 | 4 / 00.688 | 0 / 00.300 | 2 / 00.694 | 3 / 00.780 | 9 / 00.732 | 26 / 00.884 | 10 / 10.807 | 21 / 00.854 | 11 / 10.773 | 15 / 10.730 | 19 / 00.806 | 23 / 10.753 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 161 / 40.747 | 2 / 00.648 | 5 / 00.672 | 28 / 00.884 | 14 / 10.731 | 5 / 00.719 | 0 / 00.548 | 4 / 00.677 | 6 / 00.743 | 8 / 10.763 | 16 / 00.824 | 8 / 00.774 | 14 / 00.799 | 9 / 20.772 | 17 / 00.761 | 12 / 00.772 | 13 / 00.696 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 130 / 10.782 | 4 / 00.663 | 1 / 00.731 | 12 / 00.860 | 6 / 00.765 | 5 / 00.695 | 0 / 00.450 | 2 / 00.701 | 3 / 00.836 | 9 / 00.788 | 23 / 00.913 | 7 / 00.830 | 12 / 00.797 | 7 / 00.792 | 11 / 00.764 | 15 / 00.792 | 13 / 10.765 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 71 / 10.752 | 2 / 00.634 | 3 / 10.641 | 12 / 00.881 | 3 / 00.791 | 2 / 00.611 | 0 / 00.500 | 1 / 00.688 | 1 / 00.814 | 4 / 00.699 | 11 / 00.845 | 1 / 00.842 | 7 / 00.813 | 1 / 00.742 | 7 / 00.743 | 9 / 00.803 | 7 / 00.747 |
| GLiDE 2B Open · 1.9B | — | Decision | 29 / 10.637 | 1 / 00.525 | 1 / 00.637 | 12 / 00.787 | 3 / 10.625 | 0 / 00.569 | 0 / 00.519 | 1 / 00.554 | 0 / 00.590 | 1 / 00.676 | 1 / 00.646 | 1 / 00.728 | 3 / 00.663 | 2 / 00.596 | 1 / 00.617 | 1 / 00.616 | 1 / 00.639 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 17 setups: Qwen3.5-4B (on · chat), Granite-4.2-3B (high · chat), MiniCPM5-2B (on · chat), Qwen3.5-4B (off · chat), LFM3-1.7B (off · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), LFM3-1.7B (on · chat), MiniCPM5-2B (off · chat), Gemma 4 E2B (on · chat · 32k cap · sampled), Gemma 4 E2B (on · chat · 32k cap), Gemma 4 E2B (on · chat), Gemma 4 E2B (chat), Raon-1 nano (on · chat), LFM2.5-2.6B (on · chat), Qwen3.5-2B (on · chat), Qwen3.5-2B (off · chat), Raon-1 nano (off · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: Spark-X2.6-4B (high · chat) (missing Logic and Math); Granite-4.2-3B (off · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Logic 92.4% · Math 99.4% | — | Chat | 97.0%96.4%–97.5% | a | 3,190 / 3,288 | Not measured | Not measured | — | 34.2%answered 99% | 63,270 ms | 193,950 ms | $2.042$2,042estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Logic 91.6% · Math 99.2% | — | Chat | 96.6%96.0%–97.2% | ab | 3,177 / 3,288 | Not measured | Not measured | — | 54.7%answered 98% | 25,615 ms | 523,100 ms | $4.241$4,241estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Logic 88.9% · Math 99.2% | — | Chat | 95.7%95.4%–96.0% min–max of 3 seeds | b | 3,145 / 3,288seed 0 run | Not measured | Not measured | — | 51.8%answered 97% | 19,260 ms | 194,917 ms | $1.410$1,410estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Logic 90.1% · Math 98.5% | — | Chat | 95.6%94.9%–96.3% | bc | 3,144 / 3,288 | Not measured | Not measured | — | 34.0%answered 98% | 14,295 ms | 78,230 ms | $0.683$683estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Logic 85.5% · Math 98.7% | — | Chat | 94.2%93.3%–94.9% | cd | 3,097 / 3,288 | Not measured | Not measured | — | 55.9%answered 96% | 29,040 ms | 343,820 ms | $0.206$206estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% · Math 98.9% | — | Chat | 94.0%93.8%–94.2% min–max of 3 seeds | cd | 3,098 / 3,288seed 0 run | Not measured | Not measured | — | Not measured | 5,667 ms | 58,313 ms | $0.449$449estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Logic 84.2% · Math 98.9% | — | Chat | 93.9%93.89%–93.95% min–max of 3 seeds | d | 3,089 / 3,288seed 0 run | Not measured | Not measured | — | 46.7%answered 96% | 5,630 ms | 63,067 ms | $0.435$435estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Logic 83.6% · Math 98.2% | — | Chat | 93.2%92.9%–93.3% min–max of 3 seeds | d | 3,068 / 3,288seed 0 run | Not measured | Not measured | — | 41.0%answered 95% | 9,065 ms | 75,703 ms | $0.598$598estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 76.8% · Math 97.7% | — | Chat | 90.5%90.2%–90.9% min–max of 3 seeds | e | 2,966 / 3,288seed 0 run | Not measured | Not measured | — | 31.0%answered 97% | 8,873 ms | 32,503 ms | $0.325$325estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% · Math 97.8% | — | Chat | 90.3%89.3%–91.3% | e | 2,970 / 3,288 | Not measured | Not measured | — | Not measured | 19,100 ms | 54,630 ms | $0.678$678estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 74.6% · Math 96.9% | — | Chat | 89.3%88.2%–90.3% | ef | 2,936 / 3,288 | Not measured | Not measured | — | 24.3%answered 97% | 77,885 ms | 232,800 ms | $0.215$215estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 72.9% · Math 96.8% | — | Chat | 88.6%87.5%–89.6% | ef | 2,913 / 3,288 | Not measured | Not measured | — | 32.3%answered 96% | 9,935 ms | 27,690 ms | $0.313$313estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% · Math 97.0% | — | Chat | 88.1%86.9%–89.1% | fg | 2,896 / 3,288 | Not measured | Not measured | — | Not measured | 54,985 ms† | 987,660 ms† | $6.344$6,344estimate · RTX PRO 6000 Blackwell† |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Logic 75.4% · Math 94.0% | — | Chat | 87.6%86.5%–88.7% | fg | 2,881 / 3,288 | Not measured | Not measured | — | 40.6%answered 95% | 14,035 ms | 142,970 ms | $1.046$1,046estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Logic 70.3% · Math 94.7% | — | Chat | 86.4%85.2%–87.5% | gh | 2,840 / 3,288 | Not measured | Not measured | — | 30.9%answered 97% | 121,840 ms | 241,590 ms | $3.106$3,106estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Logic 64.8% · Math 94.6% | — | Chat | 84.4%83.1%–85.6% | h | 2,776 / 3,288 | Not measured | Not measured | — | 34.3%answered 94% | 17,805 ms | 92,100 ms | $0.788$788estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 65.1% · Math 89.6% | — | Chat | 81.2%79.8%–82.5% | i | 2,669 / 3,288 | Not measured | Not measured | — | 31.1%answered 94% | 42,585 ms† | 593,950 ms† | $4.422$4,422estimate · RTX PRO 6000 Blackwell† |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.777 Logic 44.1% · Math 56.8% | — | Decision | 52.5%50.8%–54.2% | j | 1,725 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.777 | 66.69chosen-answer Brier 0.1909 | 7.9%answered 96% | 31 ms | 36 ms | $0.024$24.19estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 117 answers, 117 right · 0 wrong · AUROC 0.763 Logic 45.9% · Math 55.8% | — | Decision | 52.4%50.7%–54.1% | j | 1,724 / 3,288 | 117 right of 117 kept0 mistakes · 3,171 skipped | 0.763 | 66.33chosen-answer Brier 0.1978 | 14.4%answered 93% | 68 ms | 80 ms | $0.004$4.28estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 131 answers, 130 right · 1 wrong · AUROC 0.782 Logic 44.7% · Math 55.9% | — | Decision | 52.1%50.4%–53.8% | j | 1,713 / 3,288 | 130 right of 131 kept1 wrong · 3,157 skipped | 0.782 | 65.93chosen-answer Brier 0.2024 | 15.0%answered 92% | 29 ms | 35 ms | $0.024$23.77estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 72 answers, 71 right · 1 wrong · AUROC 0.752 Logic 42.3% · Math 56.0% | — | Decision | 51.3%49.6%–53.0% | j | 1,687 / 3,288 | 71 right of 72 kept1 wrong · 3,216 skipped | 0.752 | 65.07chosen-answer Brier 0.2116 | 4.4%answered 98% | 76 ms | 104 ms | $0.005$5.02estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 190 answers, 185 right · 5 wrong · AUROC 0.769 Logic 43.1% · Math 55.2% | — | Decision | 51.1%49.4%–52.8% | j | 1,680 / 3,288 | 185 right of 190 kept5 wrong · 3,098 skipped | 0.769 | 65.47chosen-answer Brier 0.2016 | 18.9%answered 90% | 41 ms | 51 ms | $0.003$2.72 list$0.032$31.55estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 165 answers, 161 right · 4 wrong · AUROC 0.747 Logic 39.7% · Math 51.5% | — | Decision | 47.5%45.8%–49.2% | k | 1,561 / 3,288 | 161 right of 165 kept4 wrong · 3,123 skipped | 0.747 | 63.44chosen-answer Brier 0.2059 | 8.9%answered 95% | 46 ms | 52 ms | $0.033$33.45estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 194 answers, 188 right · 6 wrong · AUROC 0.743 Logic 40.4% · Math 50.4% | — | Decision | 47.0%45.3%–48.7% | k | 1,545 / 3,288 | 188 right of 194 kept6 wrong · 3,094 skipped | 0.743 | 62.33chosen-answer Brier 0.2233 | 7.8%answered 96% | 30 ms | 42 ms | $0.025$25.03estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.659 Logic 32.1% · Math 39.2% | — | Decision | 36.7%35.1%–38.4% | l | 1,208 / 3,288 | No answers at this threshold | 0.659 | 57.83chosen-answer Brier 0.2108 | 2.0%answered 99% | 117 ms | 146 ms | $0.086$85.83estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.675 Logic 30.8% · Math 38.7% | — | Decision | 35.9%34.3%–37.6% | l | 1,182 / 3,288 | 4 right of 4 kept0 mistakes · 3,284 skipped | 0.675 | 57.77chosen-answer Brier 0.2041 | 16.4%answered 89% | 36 ms | 42 ms | $0.002$2.36estimate · DGX Spark |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 30 answers, 29 right · 1 wrong · AUROC 0.637 Logic 32.3% · Math 36.8% | — | Decision | 35.2%33.6%–36.9% | l | 1,158 / 3,288 | 29 right of 30 kept1 wrong · 3,258 skipped | 0.637 | 55.25chosen-answer Brier 0.2472 | 23.1%answered 84% | 28 ms | 37 ms | $0.002$1.94estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.648 Logic 30.7% · Math 37.1% | — | Decision | 34.9%33.3%–36.5% | l | 1,147 / 3,288 | 7 right of 7 kept0 mistakes · 3,281 skipped | 0.648 | 56.81chosen-answer Brier 0.2127 | 82.5%answered 30% | 17 ms | 21 ms | $0.014$14.30estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 16 answers, 16 right · 0 wrong · AUROC 0.626 Logic 31.7% · Math 35.3% | — | Decision | 34.1%32.5%–35.7% | lm | 1,120 / 3,288 | 16 right of 16 kept0 mistakes · 3,272 skipped | 0.626 | 56.04chosen-answer Brier 0.2199 | 6.8%answered 95% | 51 ms | 63 ms | $0.003$3.40estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.602 Logic 29.2% · Math 32.1% | — | Decision | 31.1%29.5%–32.7% | mn | 1,022 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.602 | 53.32chosen-answer Brier 0.2444 | 22.7%answered 83% | 20 ms | 25 ms | $0.001$1.38estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.605 Logic 24.7% · Math 29.7% | — | Decision | 28.0%26.5%–29.5% | no | 920 / 3,288 | No answers at this threshold | 0.605 | 54.00chosen-answer Brier 0.1999 | 14.9%answered 89% | 24 ms | 29 ms | $0.002$1.62estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.576 Logic 27.0% · Math 28.2% | — | Decision | 27.8%26.3%–29.4% | o | 915 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.576 | 53.31chosen-answer Brier 0.2120 | 2.0%answered 99% | 15 ms | 17 ms | $0.001$1.11estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.590 Logic 26.7% · Math 28.2% | — | Decision | 27.7%26.2%–29.2% | o | 910 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.590 | 53.55chosen-answer Brier 0.2057 | 0.6%answered 100% | 26 ms | 35 ms | $0.002$1.80estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.573 Logic 22.4% · Math 28.7% | — | Decision | 26.5%25.0%–28.1% | o | 872 / 3,288 | No answers at this threshold | 0.573 | 53.20chosen-answer Brier 0.2012 | 5.0%answered 96% | 35 ms | 42 ms | $0.027$26.72estimate · RTX PRO 6000 Blackwell |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.511 Logic 23.6% · Math 22.7% | — | Decision | 23.0%21.6%–24.4% | p | 755 / 3,288 | No answers at this threshold | 0.511 | 52.00chosen-answer Brier 0.1896 | 15.6%answered 87% | 12 ms | 23 ms | $0.001$1.01estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.520 Logic 23.2% · Math 20.6% | — | Decision | 21.5%20.1%–22.9% | p | 706 / 3,288 | No answers at this threshold | 0.520 | 51.39chosen-answer Brier 0.1869 | 73.9%answered 32% | 11 ms | 14 ms | $0.001$0.83estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
PelaAI Fastino tattrongvu StartLux Cloudflare Blockbrain Labs Apertix AI Patrick-18 newsbang Base Labs Valen-Team EldanRing Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 54.5%3,288 | 75.0%49.1% kept (1,615) | 88.6%29.8% kept (981) | 98.0%16.4% kept (539) | 100.0%4.7% kept (153) |
| GLiDE 9B Open · 7.9B | — | Decision | 54.5%3,288 | 69.6%61.2% kept (2,011) | 85.4%35.9% kept (1,182) | 95.3%17.4% kept (571) | 100.0%3.2% kept (105) |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 55.8%3,288 | 76.0%53.9% kept (1,771) | 89.5%32.9% kept (1,082) | 98.4%15.2% kept (501) | 100.0%2.3% kept (74) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 54.4%3,288 | 74.0%51.4% kept (1,689) | 85.0%31.3% kept (1,029) | 97.0%15.4% kept (505) | 100.0%1.9% kept (63) |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 60.9%3,288 | 81.1%55.3% kept (1,817) | 91.7%36.4% kept (1,197) | 99.1%16.2% kept (534) | 100.0%0.4% kept (13) |
| Clef-Flash Open · 9.0B | — | Decision | 46.1%3,288 | 63.4%43.4% kept (1,427) | 74.6%22.4% kept (735) | 89.5%7.0% kept (229) | none kept |
| Cygnet Open · 12B | — | Decision | 42.6%3,288 | 63.2%42.4% kept (1,394) | 79.1%16.0% kept (527) | 94.7%4.0% kept (132) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 54.6%3,288 | 78.4%45.4% kept (1,492) | 91.4%25.4% kept (836) | 99.6%8.2% kept (269) | none kept |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 51.7%3,288 | 62.1%70.1% kept (2,306) | 73.5%47.7% kept (1,569) | 86.8%27.2% kept (895) | 96.1%8.5% kept (281) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 54.2%3,288 | 66.2%69.4% kept (2,281) | 78.0%47.1% kept (1,549) | 91.9%26.2% kept (863) | 98.9%8.1% kept (266) |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 56.7%3,288 | 73.2%60.4% kept (1,985) | 85.5%38.9% kept (1,279) | 96.2%22.3% kept (732) | 99.2%7.7% kept (253) |
| Elephant-1 Open · 9.0B | — | Decision | 56.2%3,288 | 73.2%59.5% kept (1,957) | 86.1%38.2% kept (1,256) | 95.6%22.0% kept (724) | 99.6%7.2% kept (236) |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 51.6%3,288 | 71.4%49.0% kept (1,611) | 84.7%27.3% kept (897) | 94.0%14.2% kept (466) | 98.8%5.1% kept (167) |
| Winnow-12B Open · 12B | — | Decision | 44.4%3,288 | 62.2%46.6% kept (1,531) | 73.1%31.1% kept (1,022) | 83.8%17.2% kept (567) | 98.1%1.6% kept (53) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 153 / 00.782 | 3 / 00.638 | 0 / 00.692 | 27 / 00.843 | 13 / 00.816 | 5 / 00.712 | 0 / 00.595 | 6 / 00.674 | 6 / 00.803 | 5 / 00.760 | 19 / 00.873 | 6 / 00.799 | 14 / 00.792 | 7 / 00.780 | 17 / 00.803 | 11 / 00.821 | 14 / 00.817 |
| GLiDE 9B Open · 7.9B | — | Decision | 105 / 00.786 | 3 / 00.637 | 4 / 00.729 | 25 / 00.911 | 7 / 00.800 | 2 / 00.737 | 0 / 00.731 | 3 / 00.700 | 1 / 00.716 | 4 / 00.752 | 15 / 00.805 | 3 / 00.774 | 10 / 00.873 | 4 / 00.853 | 7 / 00.776 | 8 / 00.851 | 9 / 00.765 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 74 / 00.798 | 2 / 00.663 | 5 / 00.704 | 3 / 00.866 | 4 / 00.865 | 4 / 00.677 | 0 / 00.769 | 3 / 00.732 | 3 / 00.844 | 3 / 00.745 | 15 / 00.811 | 4 / 00.797 | 6 / 00.888 | 2 / 00.825 | 5 / 00.804 | 7 / 00.808 | 8 / 00.768 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 63 / 00.779 | 0 / 00.675 | 3 / 00.713 | 11 / 00.854 | 3 / 00.777 | 1 / 00.678 | 0 / 00.394 | 1 / 00.703 | 1 / 00.730 | 3 / 00.735 | 11 / 00.887 | 3 / 00.774 | 6 / 00.803 | 5 / 00.830 | 4 / 00.775 | 2 / 00.788 | 9 / 00.779 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 13 / 00.810 | 1 / 00.701 | 1 / 00.711 | 4 / 00.937 | 2 / 00.838 | 0 / 00.736 | 0 / 00.814 | 0 / 00.752 | 0 / 00.840 | 0 / 00.833 | 5 / 00.915 | 0 / 00.800 | 0 / 00.824 | 0 / 00.916 | 0 / 00.775 | 0 / 00.857 | 0 / 00.794 |
| Clef-Flash Open · 9.0B | — | Decision | No answers at this threshold0.707 | 0 / 00.537 | 0 / 00.670 | 0 / 00.790 | 0 / 00.711 | 0 / 00.642 | 0 / 00.361 | 0 / 00.614 | 0 / 00.736 | 0 / 00.670 | 0 / 00.766 | 0 / 00.789 | 0 / 00.744 | 0 / 00.823 | 0 / 00.722 | 0 / 00.804 | 0 / 00.737 |
| Cygnet Open · 12B | — | Decision | No answers at this threshold0.729 | 0 / 00.604 | 0 / 00.676 | 0 / 00.848 | 0 / 00.730 | 0 / 00.617 | 0 / 00.600 | 0 / 00.690 | 0 / 00.639 | 0 / 00.726 | 0 / 00.800 | 0 / 00.788 | 0 / 00.796 | 0 / 00.697 | 0 / 00.680 | 0 / 00.768 | 0 / 00.761 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | No answers at this threshold0.786 | 0 / 00.672 | 0 / 00.663 | 0 / 00.859 | 0 / 00.788 | 0 / 00.711 | 0 / 00.712 | 0 / 00.671 | 0 / 00.801 | 0 / 00.819 | 0 / 00.834 | 0 / 00.841 | 0 / 00.847 | 0 / 00.824 | 0 / 00.774 | 0 / 00.806 | 0 / 00.817 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 270 / 110.776 | 8 / 10.645 | 3 / 00.745 | 15 / 00.881 | 24 / 00.806 | 9 / 00.677 | 1 / 10.561 | 8 / 00.679 | 6 / 00.797 | 11 / 00.751 | 31 / 00.843 | 17 / 00.747 | 22 / 20.806 | 13 / 00.812 | 31 / 20.720 | 33 / 30.836 | 38 / 20.776 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 263 / 30.795 | 7 / 00.712 | 9 / 00.689 | 40 / 00.910 | 18 / 10.778 | 5 / 00.684 | 0 / 00.606 | 7 / 00.731 | 7 / 10.745 | 8 / 00.759 | 29 / 10.839 | 10 / 00.829 | 18 / 00.822 | 12 / 00.853 | 33 / 00.824 | 24 / 00.814 | 36 / 00.820 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 251 / 20.802 | 4 / 00.706 | 2 / 00.730 | 48 / 10.869 | 18 / 00.815 | 4 / 00.701 | 0 / 00.729 | 6 / 00.684 | 8 / 00.764 | 9 / 00.797 | 31 / 00.867 | 12 / 10.814 | 21 / 00.820 | 16 / 00.840 | 23 / 00.813 | 21 / 00.812 | 28 / 00.806 |
| Elephant-1 Open · 9.0B | — | Decision | 235 / 10.797 | 5 / 00.671 | 9 / 00.725 | 39 / 00.925 | 14 / 00.821 | 5 / 00.711 | 0 / 00.486 | 6 / 00.684 | 6 / 00.823 | 10 / 00.730 | 28 / 00.833 | 10 / 10.785 | 15 / 00.868 | 14 / 00.879 | 25 / 00.778 | 24 / 00.834 | 25 / 00.774 |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 165 / 20.765 | 5 / 10.598 | 5 / 10.709 | 25 / 00.785 | 16 / 00.837 | 3 / 00.663 | 0 / 00.667 | 7 / 00.778 | 4 / 00.849 | 5 / 00.753 | 17 / 00.876 | 9 / 00.765 | 12 / 00.777 | 7 / 00.834 | 13 / 00.782 | 16 / 00.795 | 21 / 00.776 |
| Winnow-12B Open · 12B | — | Decision | 52 / 10.737 | 1 / 00.608 | 6 / 00.670 | 25 / 00.835 | 5 / 00.742 | 1 / 00.638 | 0 / 00.686 | 0 / 00.670 | 1 / 00.642 | 0 / 10.743 | 1 / 00.787 | 1 / 00.742 | 5 / 00.731 | 3 / 00.785 | 1 / 00.746 | 1 / 00.758 | 1 / 00.774 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 8 setups: Gemma 4 12B Unified (chat), Gemma 4 E4B (on · chat · 32k cap · sampled), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E4B (chat), Gemma 4 E4B (on · chat), Qwen3-14B (off · chat), LFM2.5-8B-A1B (on · chat), Erk-14B (off · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Logic 87.7% · Math 98.8% | — | Chat | 95.0%94.2%–95.7% | a | 3,125 / 3,288 | Not measured | Not measured | — | 15.3%answered 99% | 22,345 ms | 53,390 ms | $0.749$749estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.9% · Math 98.5% | — | Chat | 94.2%94.0%–94.4% min–max of 3 seeds | ab | 3,104 / 3,288seed 0 run | Not measured | Not measured | — | 54.1%answered 96% | 13,945 ms | 51,987 ms | $0.549$549estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% · Math 98.5% | — | Chat | 93.9%93.0%–94.7% | abc | 3,087 / 3,288 | Not measured | Not measured | — | Not measured | 25,530 ms | 77,380 ms | $0.855$855estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 83.2% · Math 98.1% | — | Chat | 93.0%92.1%–93.8% | bc | 3,058 / 3,288 | Not measured | Not measured | — | 51.0%answered 95% | 13,150 ms | 44,990 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 82.6% · Math 98.2% | — | Chat | 92.9%92.0%–93.7% | c | 3,054 / 3,288 | Not measured | Not measured | — | 43.7%answered 96% | 131,535 ms | 392,620 ms | $0.375$375estimate · DGX Spark |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Logic 65.3% · Math 95.8% | — | Chat | 85.4%84.1%–86.5% | d | 2,807 / 3,288 | Not measured | Not measured | — | 20.9%answered 96% | 9,380 ms | 35,110 ms | $0.840$840estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Logic 60.2% · Math 96.3% | — | Chat | 83.9%82.6%–85.1% | d | 2,759 / 3,288 | Not measured | Not measured | — | 55.9%answered 88% | 20,095 ms† | 230,570 ms† | $1.913$1,913estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Logic 54.2% · Math 90.8% | — | Chat | 78.3%76.8%–79.6% | e | 2,573 / 3,288 | Not measured | Not measured | — | 32.3%answered 92% | 8,740 ms | 16,420 ms | $0.308$308estimate · RTX PRO 6000 Blackwell |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.810 Logic 50.6% · Math 66.3% | — | Decision | 60.9%59.2%–62.5% | f | 2,002 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.810 | 71.86chosen-answer Brier 0.1716 | 4.5%answered 98% | 51 ms | 76 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 253 answers, 251 right · 2 wrong · AUROC 0.802 Logic 46.4% · Math 62.1% | — | Decision | 56.7%55.0%–58.4% | g | 1,865 / 3,288 | 251 right of 253 kept2 wrong · 3,035 skipped | 0.802 | 69.27chosen-answer Brier 0.1818 | 7.0%answered 97% | 38 ms | 45 ms | $0.028$28.19estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 236 answers, 235 right · 1 wrong · AUROC 0.797 Logic 46.4% · Math 61.3% | — | Decision | 56.2%54.5%–57.9% | g | 1,849 / 3,288 | 235 right of 236 kept1 wrong · 3,052 skipped | 0.797 | 68.96chosen-answer Brier 0.1831 | 69.4%answered 60% | 80 ms | 94 ms | $0.010$9.79 list$0.058$58.27estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 74 answers, 74 right · 0 wrong · AUROC 0.798 Logic 48.2% · Math 59.8% | — | Decision | 55.8%54.1%–57.5% | g | 1,836 / 3,288 | 74 right of 74 kept0 mistakes · 3,214 skipped | 0.798 | 68.92chosen-answer Brier 0.1799 | 13.5%answered 94% | 39 ms | 53 ms | $0.032$32.40estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.786 Logic 43.9% · Math 60.1% | — | Decision | 54.6%52.9%–56.3% | g | 1,794 / 3,288 | No answers at this threshold | 0.786 | 67.99chosen-answer Brier 0.1858 | 10.4%answered 95% | 42 ms | 57 ms | $0.034$34.08estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 105 answers, 105 right · 0 wrong · AUROC 0.786 Logic 47.8% · Math 57.9% | — | Decision | 54.5%52.8%–56.2% | ghi | 1,791 / 3,288 | 105 right of 105 kept0 mistakes · 3,183 skipped | 0.786 | 67.72chosen-answer Brier 0.1902 | 26.9%answered 86% | 177 ms | 222 ms | $0.011$11.45estimate · DGX Spark |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 153 answers, 153 right · 0 wrong · AUROC 0.782 Logic 44.3% · Math 59.8% | — | Decision | 54.5%52.8%–56.2% | gh | 1,791 / 3,288 | 153 right of 153 kept0 mistakes · 3,135 skipped | 0.782 | 67.96chosen-answer Brier 0.1854 | 3.4%answered 98% | 38 ms | 55 ms | $0.030$29.87estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 63 answers, 63 right · 0 wrong · AUROC 0.779 Logic 46.6% · Math 58.4% | — | Decision | 54.4%52.7%–56.1% | ghi | 1,788 / 3,288 | 63 right of 63 kept0 mistakes · 3,225 skipped | 0.779 | 67.73chosen-answer Brier 0.1893 | 9.6%answered 95% | 142 ms | 165 ms | $0.009$8.92estimate · DGX Spark |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 266 answers, 263 right · 3 wrong · AUROC 0.795 Logic 45.3% · Math 58.8% | — | Decision | 54.2%52.5%–55.9% | ghi | 1,782 / 3,288 | 263 right of 266 kept3 wrong · 3,022 skipped | 0.795 | 66.98chosen-answer Brier 0.2023 | 6.1%answered 97% | 46 ms | 51 ms | $0.033$32.61estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 281 answers, 270 right · 11 wrong · AUROC 0.776 Logic 43.3% · Math 56.0% | — | Decision | 51.7%50.0%–53.4% | hi | 1,699 / 3,288 | 270 right of 281 kept11 wrong · 3,007 skipped | 0.776 | 64.84chosen-answer Brier 0.2199 | 3.1%answered 99% | 163 ms | 240 ms | $0.011$10.82estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 167 answers, 165 right · 2 wrong · AUROC 0.765 Logic 44.4% · Math 55.3% | — | Decision | 51.6%49.9%–53.3% | i | 1,696 / 3,288 | 165 right of 167 kept2 wrong · 3,121 skipped | 0.765 | 65.99chosen-answer Brier 0.1960 | 7.1%answered 96% | 59 ms | 65 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.707 Logic 38.9% · Math 49.8% | — | Decision | 46.1%44.4%–47.8% | j | 1,515 / 3,288 | No answers at this threshold | 0.707 | 62.13chosen-answer Brier 0.2181 | 6.9%answered 96% | 103 ms | 131 ms | $0.013$12.90 list$0.075$75.10estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 53 answers, 52 right · 1 wrong · AUROC 0.737 Logic 39.7% · Math 46.9% | — | Decision | 44.4%42.7%–46.1% | jk | 1,461 / 3,288 | 52 right of 53 kept1 wrong · 3,235 skipped | 0.737 | 61.44chosen-answer Brier 0.2155 | 17.1%answered 90% | 55 ms | 72 ms | $0.043$42.91estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.729 Logic 39.3% · Math 44.3% | — | Decision | 42.6%40.9%–44.3% | k | 1,401 / 3,288 | No answers at this threshold | 0.729 | 60.88chosen-answer Brier 0.2086 | 7.3%answered 96% | 38 ms | 44 ms | $0.029$29.34 list$0.029$28.61estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Perplexity Poindexter Labs Nace.AI Fastino Turion.AI vLLM Semantic Router OmniJev Cloudflare Zefan Cai MachinePulse PelaAI Base Labs Sthānika AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 66.4%3,288 | 78.1%75.1% kept (2,469) | 88.0%56.4% kept (1,853) | 95.7%38.0% kept (1,249) | 100.0%11.7% kept (385) |
| Copernicus-31B Open · 31B | — | Decision | 71.0%3,288 | 89.0%64.6% kept (2,123) | 96.0%48.3% kept (1,588) | 99.3%31.5% kept (1,036) | 100.0%11.7% kept (384) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 53.1%3,288 | 79.7%42.5% kept (1,397) | 90.7%27.2% kept (893) | 97.0%16.2% kept (532) | 100.0%6.3% kept (207) |
| GLiDE v1.2 Open · 27B | — | Decision | 64.8%3,288 | 78.2%70.3% kept (2,312) | 91.8%48.6% kept (1,597) | 98.8%28.0% kept (922) | 100.0%5.4% kept (176) |
| Turion Jev 31B Open · 31B | — | Decision | 63.3%3,288 | 84.0%58.2% kept (1,913) | 92.7%41.7% kept (1,372) | 99.4%21.0% kept (692) | 100.0%1.5% kept (50) |
| d3-ultra Open · 26B | — | Decision | 65.7%3,288 | 73.5%81.9% kept (2,694) | 82.7%62.6% kept (2,057) | 96.3%37.4% kept (1,231) | 100.0%1.2% kept (39) |
| OneJev-27B Open · 27B | — | Decision | 49.3%3,288 | 80.3%32.3% kept (1,062) | 92.2%18.7% kept (616) | 98.6%8.5% kept (281) | 100.0%0.9% kept (30) |
| Clef Open · 27B | — | Decision | 47.6%3,288 | 61.4%59.6% kept (1,959) | 75.3%33.0% kept (1,086) | 89.6%14.7% kept (483) | 100.0%0.5% kept (18) |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 50.5%3,288 | 95.7%8.6% kept (282) | 100.0%0.9% kept (29) | none kept | none kept |
| Karpo Decision 31B Open · 31B | — | Decision | 74.4%3,288 | 85.1%79.4% kept (2,610) | 93.0%63.7% kept (2,095) | 97.7%48.1% kept (1,580) | 99.7%30.7% kept (1,009) |
| Drex Max Open · 27B | — | Decision | 64.4%3,288 | 72.4%81.7% kept (2,686) | 80.8%64.5% kept (2,120) | 91.5%47.3% kept (1,556) | 98.0%26.9% kept (884) |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 71.1%3,288 | 86.5%69.3% kept (2,279) | 93.9%53.0% kept (1,744) | 98.5%35.2% kept (1,156) | 99.8%16.8% kept (554) |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 73.0%3,288 | 87.1%72.9% kept (2,396) | 93.9%56.8% kept (1,868) | 99.0%38.8% kept (1,275) | 99.8%15.9% kept (524) |
| Mammoth-1 Open · 27B | — | Decision | 67.3%3,288 | 82.3%71.0% kept (2,334) | 92.4%53.0% kept (1,744) | 97.9%37.3% kept (1,228) | 99.6%15.8% kept (520) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 64.1%3,288 | 81.5%64.6% kept (2,125) | 92.6%46.9% kept (1,541) | 97.8%31.4% kept (1,031) | 99.5%12.5% kept (411) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 63.6%3,288 | 80.6%65.2% kept (2,145) | 91.8%47.0% kept (1,545) | 97.4%31.3% kept (1,028) | 99.8%12.3% kept (405) |
| Sieve 31B v3 Open · 31B | — | Decision | 70.5%3,288 | 86.8%67.7% kept (2,227) | 94.3%51.4% kept (1,691) | 98.7%34.4% kept (1,131) | 99.3%9.0% kept (296) |
| GLiDE 27B Open · 27B | — | Decision | 67.3%3,288 | 80.6%72.4% kept (2,381) | 90.6%53.0% kept (1,742) | 98.5%32.5% kept (1,068) | 99.3%8.3% kept (274) |
| d3 Open · 26B | — | Decision | 65.0%3,288 | 82.6%65.4% kept (2,150) | 94.1%46.2% kept (1,518) | 98.6%28.2% kept (927) | 99.5%6.6% kept (217) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 385 / 00.844 | 22 / 00.752 | 5 / 00.764 | 35 / 00.878 | 26 / 00.828 | 8 / 00.776 | 0 / 00.577 | 8 / 00.785 | 15 / 00.845 | 17 / 00.790 | 36 / 00.935 | 27 / 00.839 | 25 / 00.900 | 22 / 00.933 | 35 / 00.845 | 49 / 00.871 | 55 / 00.834 |
| Copernicus-31B Open · 31B | — | Decision | 384 / 00.873 | 15 / 00.746 | 12 / 00.775 | 39 / 00.935 | 27 / 00.901 | 13 / 00.865 | 1 / 00.800 | 9 / 00.881 | 17 / 00.892 | 20 / 00.881 | 42 / 00.980 | 12 / 00.884 | 25 / 00.876 | 19 / 00.937 | 29 / 00.882 | 48 / 00.914 | 56 / 00.875 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 207 / 00.789 | 3 / 00.652 | 0 / 00.690 | 30 / 00.815 | 8 / 00.819 | 6 / 00.817 | 0 / 00.533 | 0 / 00.740 | 5 / 00.798 | 13 / 00.841 | 20 / 00.905 | 9 / 00.768 | 16 / 00.837 | 14 / 00.766 | 22 / 00.762 | 35 / 00.836 | 26 / 00.803 |
| GLiDE v1.2 Open · 27B | — | Decision | 176 / 00.843 | 3 / 00.732 | 6 / 00.739 | 36 / 00.906 | 12 / 00.857 | 3 / 00.740 | 0 / 00.528 | 5 / 00.777 | 4 / 00.834 | 5 / 00.838 | 20 / 00.963 | 14 / 00.865 | 22 / 00.912 | 7 / 00.948 | 15 / 00.873 | 14 / 00.849 | 10 / 00.877 |
| Turion Jev 31B Open · 31B | — | Decision | 50 / 00.844 | 1 / 00.702 | 0 / 00.756 | 6 / 00.875 | 5 / 00.868 | 1 / 00.770 | 0 / 00.757 | 1 / 00.809 | 0 / 00.904 | 3 / 00.817 | 3 / 00.926 | 1 / 00.854 | 6 / 00.823 | 1 / 00.940 | 8 / 00.853 | 5 / 00.866 | 9 / 00.852 |
| d3-ultra Open · 26B | — | Decision | 39 / 00.835 | 9 / 00.707 | 10 / 00.761 | 9 / 00.853 | 1 / 00.842 | 1 / 00.778 | 0 / 00.764 | 0 / 00.820 | 0 / 00.731 | 0 / 00.827 | 3 / 00.942 | 0 / 00.819 | 1 / 00.901 | 2 / 00.848 | 1 / 00.871 | 1 / 00.850 | 1 / 00.874 |
| OneJev-27B Open · 27B | — | Decision | 30 / 00.770 | 0 / 00.633 | 0 / 00.712 | 4 / 00.845 | 2 / 00.768 | 0 / 00.645 | 0 / 00.806 | 1 / 00.750 | 0 / 00.767 | 1 / 00.778 | 2 / 00.884 | 1 / 00.758 | 1 / 00.811 | 3 / 00.787 | 4 / 00.757 | 6 / 00.785 | 5 / 00.789 |
| Clef Open · 27B | — | Decision | 18 / 00.747 | 2 / 00.607 | 0 / 00.704 | 4 / 00.846 | 2 / 00.751 | 1 / 00.708 | 0 / 00.875 | 2 / 00.723 | 0 / 00.720 | 0 / 00.798 | 3 / 00.778 | 0 / 00.793 | 1 / 00.744 | 0 / 00.878 | 2 / 00.748 | 0 / 00.802 | 1 / 00.774 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | No answers at this threshold0.759 | 0 / 00.623 | 0 / 00.716 | 0 / 00.861 | 0 / 00.816 | 0 / 00.607 | 0 / 00.625 | 0 / 00.674 | 0 / 00.731 | 0 / 00.751 | 0 / 00.762 | 0 / 00.745 | 0 / 00.847 | 0 / 00.818 | 0 / 00.776 | 0 / 00.727 | 0 / 00.749 |
| Karpo Decision 31B Open · 31B | — | Decision | 1006 / 30.882 | 58 / 00.805 | 33 / 20.758 | 78 / 10.877 | 85 / 00.899 | 34 / 00.801 | 3 / 00.833 | 34 / 00.880 | 38 / 00.933 | 50 / 00.890 | 83 / 00.952 | 45 / 00.879 | 66 / 00.915 | 53 / 00.950 | 91 / 00.902 | 114 / 00.893 | 141 / 00.904 |
| Drex Max Open · 27B | — | Decision | 866 / 180.841 | 34 / 00.729 | 39 / 50.758 | 74 / 30.838 | 67 / 10.834 | 25 / 00.803 | 3 / 00.750 | 32 / 00.799 | 28 / 00.867 | 36 / 40.794 | 63 / 00.923 | 42 / 20.877 | 75 / 00.894 | 44 / 00.908 | 83 / 00.876 | 96 / 20.855 | 125 / 10.847 |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 553 / 10.870 | 25 / 00.765 | 14 / 00.764 | 50 / 00.904 | 38 / 00.874 | 15 / 00.850 | 1 / 00.750 | 14 / 00.881 | 22 / 00.903 | 32 / 00.900 | 60 / 00.956 | 18 / 00.912 | 35 / 00.872 | 33 / 00.904 | 55 / 10.885 | 64 / 00.914 | 77 / 00.888 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 523 / 10.882 | 26 / 00.792 | 5 / 00.785 | 23 / 00.907 | 38 / 00.916 | 15 / 00.862 | 2 / 00.742 | 14 / 00.847 | 24 / 00.902 | 37 / 00.921 | 70 / 00.956 | 12 / 00.871 | 28 / 00.882 | 28 / 00.916 | 56 / 10.863 | 68 / 00.908 | 77 / 00.886 |
| Mammoth-1 Open · 27B | — | Decision | 518 / 20.868 | 21 / 00.703 | 13 / 00.797 | 72 / 20.905 | 27 / 00.885 | 9 / 00.766 | 1 / 00.788 | 11 / 00.821 | 14 / 00.876 | 30 / 00.897 | 41 / 00.909 | 17 / 00.893 | 47 / 00.930 | 31 / 00.904 | 53 / 00.900 | 59 / 00.896 | 72 / 00.903 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 409 / 20.849 | 9 / 00.730 | 12 / 00.783 | 57 / 20.791 | 22 / 00.861 | 6 / 00.818 | 1 / 00.806 | 8 / 00.772 | 10 / 00.823 | 16 / 00.819 | 34 / 00.961 | 34 / 00.901 | 43 / 00.903 | 33 / 00.932 | 32 / 00.840 | 47 / 00.875 | 45 / 00.863 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 404 / 10.854 | 8 / 00.728 | 12 / 00.787 | 58 / 10.860 | 23 / 00.850 | 9 / 00.842 | 1 / 00.571 | 6 / 00.827 | 14 / 00.799 | 18 / 00.854 | 28 / 00.927 | 36 / 00.869 | 43 / 00.887 | 31 / 00.946 | 29 / 00.851 | 41 / 00.885 | 47 / 00.884 |
| Sieve 31B v3 Open · 31B | — | Decision | 294 / 20.865 | 9 / 00.764 | 16 / 20.745 | 54 / 00.861 | 24 / 00.871 | 6 / 00.848 | 0 / 00.903 | 7 / 00.856 | 8 / 00.898 | 10 / 00.863 | 26 / 00.956 | 15 / 00.918 | 22 / 00.907 | 14 / 00.939 | 24 / 00.895 | 31 / 00.879 | 28 / 00.848 |
| GLiDE 27B Open · 27B | — | Decision | 272 / 20.852 | 6 / 00.754 | 17 / 00.800 | 51 / 10.888 | 17 / 10.858 | 6 / 00.787 | 0 / 00.569 | 8 / 00.761 | 5 / 00.846 | 7 / 00.870 | 22 / 00.948 | 20 / 00.938 | 36 / 00.873 | 18 / 00.905 | 21 / 00.861 | 20 / 00.848 | 18 / 00.875 |
| d3 Open · 26B | — | Decision | 216 / 10.858 | 3 / 00.755 | 11 / 00.761 | 40 / 00.863 | 13 / 10.867 | 5 / 00.791 | 1 / 00.806 | 6 / 00.811 | 7 / 00.907 | 8 / 00.847 | 21 / 00.955 | 15 / 00.904 | 25 / 00.905 | 13 / 00.893 | 21 / 00.886 | 13 / 00.875 | 14 / 00.872 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 3 setups: Qwen3.8-27B (low · chat), Gemma 4 31B (chat), LFM2.5-24B-A2B (on · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Logic 94.0% · Math 99.2% | — | Chat | 97.4%96.8%–97.9% | a | 3,204 / 3,288 | Not measured | Not measured | — | 33.9%answered 99% | 11,200 ms | 39,560 ms | $2.020$2,020estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Logic 93.3% · Math 99.4% | — | Chat | 97.3%96.7%–97.8% | a | 3,200 / 3,288 | Not measured | Not measured | — | 17.0%answered 99% | 12,575 ms | 25,280 ms | $0.775$775estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Logic 86.7% · Math 99.0% | — | Chat | 94.8%94.0%–95.5% | b | 3,117 / 3,288 | Not measured | Not measured | — | 44.0%answered 97% | 42,425 ms | 432,510 ms | $3.205$3,205estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 1,009 answers, 1,006 right · 3 wrong · AUROC 0.882 Logic 60.7% · Math 81.6% | — | Decision | 74.4%72.9%–75.9% | c | 2,447 / 3,288 | 1,006 right of 1,009 kept3 wrong · 2,279 skipped | 0.882 | 81.13chosen-answer Brier 0.1215 | 5.7%answered 98% | 162 ms | 198 ms | $0.118$118estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 524 answers, 523 right · 1 wrong · AUROC 0.882 Logic 58.9% · Math 80.3% | — | Decision | 73.0%71.4%–74.5% | c | 2,399 / 3,288 | 523 right of 524 kept1 wrong · 2,764 skipped | 0.882 | 80.27chosen-answer Brier 0.1242 | 0.1%answered 100% | 116 ms | 177 ms | $0.093$92.98estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 554 answers, 553 right · 1 wrong · AUROC 0.870 Logic 59.0% · Math 77.4% | — | Decision | 71.1%69.5%–72.6% | d | 2,338 / 3,288 | 553 right of 554 kept1 wrong · 2,734 skipped | 0.870 | 78.92chosen-answer Brier 0.1327 | 0.7%answered 100% | 811 ms | 1,111 ms | $0.052$51.85estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 384 answers, 384 right · 0 wrong · AUROC 0.873 Logic 58.7% · Math 77.3% | — | Decision | 71.0%69.4%–72.5% | d | 2,333 / 3,288 | 384 right of 384 kept0 mistakes · 2,904 skipped | 0.873 | 78.76chosen-answer Brier 0.1344 | 1.8%answered 99% | 128 ms | 218 ms | $0.102$102estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 296 answers, 294 right · 2 wrong · AUROC 0.865 Logic 57.9% · Math 77.1% | — | Decision | 70.5%68.9%–72.1% | d | 2,319 / 3,288 | 294 right of 296 kept2 wrong · 2,992 skipped | 0.865 | 78.49chosen-answer Brier 0.1354 | 3.8%answered 99% | 676 ms | 1,173 ms | $0.047$47.29estimate · DGX Spark |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 274 answers, 272 right · 2 wrong · AUROC 0.852 Logic 58.4% · Math 72.0% | — | Decision | 67.3%65.7%–68.9% | e | 2,214 / 3,288 | 272 right of 274 kept2 wrong · 3,014 skipped | 0.852 | 76.29chosen-answer Brier 0.1475 | 31.9%answered 89% | 450 ms | 587 ms | $0.029$28.55estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 520 answers, 518 right · 2 wrong · AUROC 0.868 Logic 54.5% · Math 74.0% | — | Decision | 67.3%65.7%–68.9% | ef | 2,213 / 3,288 | 518 right of 520 kept2 wrong · 2,768 skipped | 0.868 | 76.68chosen-answer Brier 0.1394 | 71.4%answered 69% | 150 ms | 232 ms | $0.028$27.98 list$0.120$120estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 385 answers, 385 right · 0 wrong · AUROC 0.844 Logic 54.8% · Math 72.5% | — | Decision | 66.4%64.8%–68.0% | efg | 2,184 / 3,288 | 385 right of 385 kept0 mistakes · 2,903 skipped | 0.844 | 75.44chosen-answer Brier 0.1554 | 13.5%answered 95% | 97 ms | 125 ms | $0.071$71.10estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.835 Logic 54.6% · Math 71.5% | — | Decision | 65.7%64.1%–67.3% | efgh | 2,161 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.835 | 74.31chosen-answer Brier 0.1711 | 7.5%answered 97% | 129 ms | 168 ms | $0.096$95.71estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 217 answers, 216 right · 1 wrong · AUROC 0.858 Logic 54.5% · Math 70.4% | — | Decision | 65.0%63.3%–66.6% | fgh | 2,136 / 3,288 | 216 right of 217 kept1 wrong · 3,071 skipped | 0.858 | 75.16chosen-answer Brier 0.1464 | 9.3%answered 97% | 57 ms | 86 ms | $0.046$46.28estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 176 answers, 176 right · 0 wrong · AUROC 0.843 Logic 55.2% · Math 69.9% | — | Decision | 64.8%63.2%–66.5% | fgh | 2,132 / 3,288 | 176 right of 176 kept0 mistakes · 3,112 skipped | 0.843 | 74.68chosen-answer Brier 0.1548 | 42.3%answered 83% | 81 ms | 136 ms | $0.010$9.71 list$0.066$66.05estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 884 answers, 866 right · 18 wrong · AUROC 0.841 Logic 53.1% · Math 70.3% | — | Decision | 64.4%62.8%–66.0% | gh | 2,118 / 3,288 | 866 right of 884 kept18 wrong · 2,404 skipped | 0.841 | 73.20chosen-answer Brier 0.1802 | 5.7%answered 98% | 81 ms | 130 ms | $0.016$16.31 list$0.064$63.74estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 411 answers, 409 right · 2 wrong · AUROC 0.849 Logic 53.3% · Math 69.8% | — | Decision | 64.1%62.5%–65.8% | h | 2,109 / 3,288 | 409 right of 411 kept2 wrong · 2,877 skipped | 0.849 | 74.56chosen-answer Brier 0.1502 | 10.3%answered 96% | 230 ms | 439 ms | $0.005$5.38 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 405 answers, 404 right · 1 wrong · AUROC 0.854 Logic 52.5% · Math 69.3% | with web | Decision | 63.6%61.9%–65.2% | h | 2,090 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.854 | 74.28chosen-answer Brier 0.1501 | 25.8%answered 90% | 430 ms | 707 ms | $1.025$1,025 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 50 answers, 50 right · 0 wrong · AUROC 0.844 Logic 52.3% · Math 69.0% | — | Decision | 63.3%61.6%–64.9% | h | 2,081 / 3,288 | 50 right of 50 kept0 mistakes · 3,238 skipped | 0.844 | 73.88chosen-answer Brier 0.1553 | 1.9%answered 99% | 301 ms | 448 ms | $0.020$20.01estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 207 answers, 207 right · 0 wrong · AUROC 0.789 Logic 41.9% · Math 59.0% | — | Decision | 53.1%51.4%–54.8% | i | 1,747 / 3,288 | 207 right of 207 kept0 mistakes · 3,081 skipped | 0.789 | 67.48chosen-answer Brier 0.1818 | 28.1%answered 85% | 339 ms | 472 ms | $0.012$12.17 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.759 Logic 42.6% · Math 54.6% | — | Decision | 50.5%48.7%–52.2% | ij | 1,659 / 3,288 | No answers at this threshold | 0.759 | 62.91chosen-answer Brier 0.2465 | 14.5%answered 92% | 250 ms | 482 ms | $0.212$212estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 30 answers, 30 right · 0 wrong · AUROC 0.770 Logic 40.6% · Math 53.8% | — | Decision | 49.3%47.6%–51.0% | j | 1,620 / 3,288 | 30 right of 30 kept0 mistakes · 3,258 skipped | 0.770 | 65.05chosen-answer Brier 0.1917 | 1.6%answered 99% | 80 ms | 134 ms | $0.069$69.42estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 18 answers, 18 right · 0 wrong · AUROC 0.747 Logic 40.8% · Math 51.1% | — | Decision | 47.6%45.9%–49.3% | j | 1,564 / 3,288 | 18 right of 18 kept0 mistakes · 3,270 skipped | 0.747 | 62.92chosen-answer Brier 0.2172 | 2.4%answered 99% | 185 ms | 216 ms | $0.081$81.46 list$0.135$135estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
REAX Logits Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | Decision | 71.6%3,288 | 87.1%70.6% kept (2,320) | 96.0%50.3% kept (1,655) | 99.5%33.4% kept (1,098) | 100.0%7.0% kept (229) |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 69.3%3,288 | 85.2%68.0% kept (2,236) | 96.8%45.1% kept (1,483) | 99.5%22.6% kept (742) | 100.0%0.7% kept (23) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | Decision | 229 / 00.878 | 12 / 00.750 | 11 / 00.804 | 28 / 00.921 | 11 / 00.888 | 7 / 00.826 | 1 / 00.818 | 7 / 00.869 | 9 / 00.886 | 11 / 00.917 | 30 / 00.955 | 10 / 00.946 | 14 / 00.881 | 7 / 00.935 | 22 / 00.887 | 25 / 00.913 | 24 / 00.900 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 23 / 00.868 | 1 / 00.738 | 1 / 00.764 | 6 / 00.880 | 2 / 00.900 | 0 / 00.841 | 0 / 00.667 | 0 / 00.847 | 1 / 00.888 | 0 / 00.872 | 2 / 00.950 | 0 / 00.919 | 1 / 00.910 | 1 / 00.939 | 4 / 00.895 | 1 / 00.871 | 3 / 00.911 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 6 setups: DeepSeek-V4.1-Flash (high · chat · 1017), Inkling 2 Small (high · chat), Inkling-Small (high · chat · Jul 2026), MiMo-V2.6-Flash (on · chat), GLM-5.3-Flash (high · chat), MiniMax-M3.1-Flash (high · chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: DeepSeek-V4-Flash (chat) (missing Logic); Qwen3.5-397B-A17B (on · chat) (missing Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Logic 96.2% · Math 99.6% | — | Chat | 98.4%98.0%–98.8% | a | 3,237 / 3,288 | Not measured | Not measured | — | 6.3%answered 100% | 2,270 ms | 14,870 ms | $0.713$713 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Logic 95.6% · Math 99.5% | — | Chat | 98.2%97.7%–98.6% | a | 3,229 / 3,288 | Not measured | Not measured | — | 41.7%answered 99% | 2,870 ms | 35,530 ms | $1.830$1,830 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Logic 95.4% · Math 99.6% | — | Chat | 98.2%97.7%–98.6% | a | 3,228 / 3,288 | Not measured | Not measured | — | 50.6%answered 99% | 2,750 ms | 34,100 ms | $2.506$2,506 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Logic 95.1% · Math 99.4% | — | Chat | 98.0%97.4%–98.4% | a | 3,221 / 3,288 | Not measured | Not measured | — | 27.6%answered 99% | 10,590 ms | 53,970 ms | $0.615$615 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Logic 93.2% · Math 98.8% | — | Chat | 96.9%96.2%–97.4% | b | 3,185 / 3,288 | Not measured | Not measured | — | 7.4%answered 100% | 4,640 ms | 25,900 ms | $0.471$471 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Logic 94.0% · Math 98.2% | — | Chat | 96.8%96.1%–97.3% | b | 3,182 / 3,288 | Not measured | Not measured | — | 38.9%answered 99% | 2,170 ms | 15,760 ms | $0.342$342 |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 229 answers, 229 right · 0 wrong · AUROC 0.878 Logic 59.3% · Math 78.0% | — | Decision | 71.6%70.0%–73.1% | c | 2,354 / 3,288 | 229 right of 229 kept0 mistakes · 3,059 skipped | 0.878 | 79.30chosen-answer Brier 0.1300 | 3.8%answered 99% | 200 ms | 260 ms | $0.148$148estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 23 answers, 23 right · 0 wrong · AUROC 0.868 Logic 57.8% · Math 75.2% | — | Decision | 69.3%67.7%–70.8% | d | 2,277 / 3,288 | 23 right of 23 kept0 mistakes · 3,265 skipped | 0.868 | 77.60chosen-answer Brier 0.1405 | 4.4%answered 99% | 207 ms | 324 ms | $0.165$165estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
TypeSafe Google Liquid AI Fastino Microsoft Celeris Subquadratic Nace.AI Inception Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| Jev 1.13 Closed · Efficient | — | Decision | 58.0%3,288 | 77.2%58.2% kept (1,913) | 89.6%35.8% kept (1,178) | 97.6%20.3% kept (669) | 100.0%9.5% kept (313) |
| Jev 1.14 Closed · Efficient | — | Decision | 59.3%3,288 | 79.8%54.3% kept (1,784) | 91.0%34.0% kept (1,117) | 97.2%19.5% kept (640) | 100.0%8.0% kept (263) |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 64.1%3,288 | 84.1%59.2% kept (1,948) | 92.5%42.8% kept (1,407) | 98.1%27.6% kept (906) | 100.0%7.8% kept (257) |
| d1-ultra Closed · Efficient | — | Decision | 67.5%3,288 | 84.9%64.9% kept (2,134) | 92.4%49.0% kept (1,610) | 98.2%32.2% kept (1,058) | 100.0%7.1% kept (233) |
| d1.1 Closed · Efficient | — | Decision | 58.6%3,288 | 79.6%53.7% kept (1,767) | 90.9%35.0% kept (1,150) | 97.0%20.4% kept (670) | 100.0%7.0% kept (231) |
| GLiDE API Closed · Efficient | — | Decision | 65.2%3,288 | 79.2%71.0% kept (2,336) | 90.3%50.6% kept (1,664) | 98.3%30.3% kept (996) | 100.0%6.5% kept (213) |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 54.3%3,288 | 70.4%57.1% kept (1,876) | 86.3%30.5% kept (1,003) | 97.2%11.8% kept (389) | 100.0%1.4% kept (47) |
| celeris-1-decision Closed · Efficient | — | Decision | 62.2%3,288 | 75.8%69.1% kept (2,271) | 87.8%48.8% kept (1,606) | 95.4%32.0% kept (1,053) | none kept |
| celeris-1.1-decision Closed · Efficient | — | Decision | 62.1%3,288 | 76.2%68.8% kept (2,262) | 87.9%48.6% kept (1,598) | 95.7%32.2% kept (1,060) | none kept |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 63.0%3,288 | 73.0%76.7% kept (2,522) | 82.6%58.1% kept (1,911) | 92.7%38.6% kept (1,268) | 99.3%13.2% kept (433) |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 64.8%3,288 | 77.5%72.3% kept (2,377) | 88.6%53.3% kept (1,752) | 97.0%35.3% kept (1,162) | 99.8%12.3% kept (405) |
| Mercury Decide Closed · Efficient | — | Decision | 50.6%3,288 | 64.0%65.1% kept (2,142) | 73.9%45.8% kept (1,505) | 83.8%28.6% kept (941) | 95.7%11.3% kept (372) |
| d1 Closed · Efficient | — | Decision | 50.5%3,288 | 69.8%50.3% kept (1,653) | 82.8%28.5% kept (938) | 93.6%14.7% kept (484) | 99.5%6.3% kept (208) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Jev 1.13 Closed · Efficient | — | Decision | 313 / 00.817 | 7 / 00.708 | 5 / 00.714 | 53 / 00.875 | 17 / 00.836 | 8 / 00.756 | 0 / 00.671 | 9 / 00.761 | 8 / 00.797 | 9 / 00.835 | 23 / 00.894 | 20 / 00.801 | 30 / 00.868 | 17 / 00.831 | 24 / 00.821 | 41 / 00.850 | 42 / 00.818 |
| Jev 1.14 Closed · Efficient | — | Decision | 263 / 00.810 | 5 / 00.671 | 8 / 00.711 | 48 / 00.856 | 16 / 00.854 | 7 / 00.791 | 0 / 00.686 | 8 / 00.700 | 5 / 00.778 | 8 / 00.810 | 18 / 00.910 | 14 / 00.788 | 25 / 00.862 | 18 / 00.826 | 21 / 00.781 | 27 / 00.849 | 35 / 00.834 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 257 / 00.838 | 5 / 00.766 | 0 / 00.699 | 38 / 00.885 | 15 / 00.862 | 6 / 00.794 | 2 / 00.729 | 0 / 00.789 | 5 / 00.854 | 10 / 00.851 | 18 / 00.882 | 13 / 00.797 | 23 / 00.833 | 15 / 00.799 | 31 / 00.859 | 38 / 00.907 | 38 / 00.858 |
| d1-ultra Closed · Efficient | — | Decision | 233 / 00.852 | 12 / 00.759 | 3 / 00.715 | 5 / 00.846 | 23 / 00.865 | 8 / 00.755 | 0 / 00.650 | 5 / 00.813 | 9 / 00.896 | 13 / 00.887 | 12 / 00.910 | 13 / 00.941 | 18 / 00.902 | 13 / 00.893 | 20 / 00.845 | 30 / 00.885 | 49 / 00.888 |
| d1.1 Closed · Efficient | — | Decision | 231 / 00.811 | 4 / 00.676 | 6 / 00.712 | 38 / 00.906 | 13 / 00.811 | 5 / 00.768 | 0 / 00.714 | 4 / 00.721 | 3 / 00.825 | 7 / 00.796 | 16 / 00.848 | 14 / 00.854 | 19 / 00.870 | 15 / 00.849 | 23 / 00.815 | 30 / 00.798 | 34 / 00.835 |
| GLiDE API Closed · Efficient | — | Decision | 213 / 00.852 | 7 / 00.777 | 7 / 00.765 | 38 / 00.866 | 17 / 00.874 | 5 / 00.746 | 1 / 00.625 | 11 / 00.778 | 4 / 00.853 | 4 / 00.840 | 20 / 00.987 | 13 / 00.872 | 29 / 00.885 | 11 / 00.924 | 15 / 00.896 | 17 / 00.811 | 14 / 00.881 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 47 / 00.769 | 1 / 00.628 | 2 / 00.683 | 17 / 00.823 | 5 / 00.789 | 0 / 00.696 | 0 / 00.615 | 1 / 00.670 | 0 / 00.747 | 0 / 00.750 | 4 / 00.848 | 4 / 00.806 | 5 / 00.837 | 1 / 00.801 | 2 / 00.768 | 3 / 00.824 | 2 / 00.794 |
| celeris-1-decision Closed · Efficient | — | Decision | No answers at this threshold0.829 | 0 / 00.706 | 0 / 00.679 | 0 / 00.867 | 0 / 00.857 | 0 / 00.787 | 0 / 00.807 | 0 / 00.794 | 0 / 00.884 | 0 / 00.896 | 0 / 00.952 | 0 / 00.840 | 0 / 00.881 | 0 / 00.836 | 0 / 00.838 | 0 / 00.869 | 0 / 00.849 |
| celeris-1.1-decision Closed · Efficient | — | Decision | No answers at this threshold0.830 | 0 / 00.709 | 0 / 00.679 | 0 / 00.853 | 0 / 00.863 | 0 / 00.790 | 0 / 00.875 | 0 / 00.790 | 0 / 00.863 | 0 / 00.896 | 0 / 00.944 | 0 / 00.853 | 0 / 00.877 | 0 / 00.812 | 0 / 00.852 | 0 / 00.867 | 0 / 00.855 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 430 / 30.827 | 17 / 00.736 | 18 / 10.743 | 35 / 10.857 | 28 / 00.819 | 12 / 00.784 | 1 / 00.722 | 15 / 00.780 | 10 / 00.808 | 19 / 00.754 | 47 / 00.929 | 35 / 00.829 | 31 / 00.832 | 28 / 00.931 | 37 / 10.802 | 45 / 00.871 | 52 / 00.848 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 404 / 10.846 | 7 / 00.729 | 24 / 00.771 | 58 / 10.877 | 23 / 00.839 | 8 / 00.850 | 1 / 00.471 | 14 / 00.799 | 9 / 00.861 | 15 / 00.817 | 28 / 00.903 | 31 / 00.840 | 47 / 00.910 | 30 / 00.919 | 36 / 00.866 | 39 / 00.853 | 34 / 00.893 |
| Mercury Decide Closed · Efficient | — | Decision | 356 / 160.777 | 14 / 20.670 | 28 / 10.744 | 42 / 40.775 | 25 / 00.811 | 9 / 00.692 | 1 / 00.729 | 8 / 00.748 | 6 / 00.687 | 11 / 30.718 | 26 / 10.859 | 23 / 20.801 | 31 / 10.808 | 18 / 00.821 | 22 / 10.751 | 44 / 10.785 | 48 / 00.822 |
| d1 Closed · Efficient | — | Decision | 207 / 10.757 | 3 / 00.630 | 5 / 00.683 | 47 / 10.828 | 14 / 00.765 | 5 / 00.653 | 0 / 00.530 | 4 / 00.700 | 3 / 00.789 | 6 / 00.752 | 9 / 00.761 | 11 / 00.726 | 21 / 00.802 | 11 / 00.738 | 20 / 00.769 | 23 / 00.777 | 25 / 00.784 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 2 setups: Muse Ember (max · chat), GPT-6 Luna (chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 3,276 of 3,288); shown once.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: GPT-6 Luna (max · chat) (missing Logic and Math); Claude Haiku 5.5 (max · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Logic 95.5% · Math 99.6% | — | Chat | 98.2%97.7%–98.6% | a | 3,228 / 3,288 | Not measured | Not measured | — | 31.3%answered 99% | 1,460 ms | 6,300 ms | $0.863$863 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Logic 58.0% · Math 93.7% | — | Chat | 81.4%80.1%–82.7% | b | 2,678 / 3,288 | Not measured | Not measured | — | Not measured | 3,830 ms | 5,300 ms | $0.000$0.00 |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 233 answers, 233 right · 0 wrong · AUROC 0.852 Logic 54.9% · Math 74.1% | — | Decision | 67.5%65.9%–69.1% | c | 2,221 / 3,288 | 233 right of 233 kept0 mistakes · 3,055 skipped | 0.852 | 76.50chosen-answer Brier 0.1455 | 89.1%answered 52% | 180 ms | 426 ms | $0.008$8.37 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 213 answers, 213 right · 0 wrong · AUROC 0.852 Logic 54.5% · Math 70.8% | — | Decision | 65.2%63.6%–66.8% | cd | 2,144 / 3,288 | 213 right of 213 kept0 mistakes · 3,075 skipped | 0.852 | 75.04chosen-answer Brier 0.1513 | 49.2%answered 80% | 266 ms | 284 ms | $0.036$36.42 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 405 answers, 404 right · 1 wrong · AUROC 0.846 Logic 54.8% · Math 70.1% | — | Decision | 64.8%63.2%–66.5% | de | 2,132 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.846 | 74.61chosen-answer Brier 0.1562 | 48.3%answered 79% | 327 ms | 518 ms | $0.016$16.13 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 257 answers, 257 right · 0 wrong · AUROC 0.838 Logic 50.0% · Math 71.3% | — | Decision | 64.1%62.4%–65.7% | de | 2,106 / 3,288 | 257 right of 257 kept0 mistakes · 3,031 skipped | 0.838 | 74.23chosen-answer Brier 0.1559 | 3.0%answered 99% | 227 ms | 347 ms | $0.027$27.07 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 433 answers, 430 right · 3 wrong · AUROC 0.827 Logic 52.4% · Math 68.6% | — | Decision | 63.0%61.4%–64.7% | de | 2,072 / 3,288 | 430 right of 433 kept3 wrong · 2,855 skipped | 0.827 | 72.66chosen-answer Brier 0.1770 | 58.3%answered 73% | 382 ms | 391 ms | $0.011$10.75 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.829 Logic 47.3% · Math 70.0% | — | Decision | 62.2%60.6%–63.9% | e | 2,046 / 3,288 | No answers at this threshold | 0.829 | 72.82chosen-answer Brier 0.1659 | 87.2%answered 50% | 388 ms | 392 ms | $0.011$11.25 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.830 Logic 47.3% · Math 69.9% | — | Decision | 62.1%60.5%–63.8% | e | 2,043 / 3,288 | No answers at this threshold | 0.830 | 72.78chosen-answer Brier 0.1657 | 86.7%answered 51% | 381 ms | 388 ms | $0.001$1.13 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 263 answers, 263 right · 0 wrong · AUROC 0.810 Logic 48.2% · Math 65.1% | — | Decision | 59.3%57.6%–61.0% | f | 1,951 / 3,288 | 263 right of 263 kept0 mistakes · 3,025 skipped | 0.810 | 71.00chosen-answer Brier 0.1734 | 15.4%answered 93% | 164 ms | 188 ms | $0.021$21.27 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 231 answers, 231 right · 0 wrong · AUROC 0.811 Logic 46.0% · Math 65.2% | — | Decision | 58.6%56.9%–60.3% | f | 1,928 / 3,288 | 231 right of 231 kept0 mistakes · 3,057 skipped | 0.811 | 70.69chosen-answer Brier 0.1726 | 86.0%answered 46% | 174 ms | 285 ms | $0.004$3.98 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 313 answers, 313 right · 0 wrong · AUROC 0.817 Logic 46.3% · Math 64.1% | — | Decision | 58.0%56.3%–59.7% | f | 1,907 / 3,288 | 313 right of 313 kept0 mistakes · 2,975 skipped | 0.817 | 70.41chosen-answer Brier 0.1718 | 10.2%answered 96% | 166 ms | 205 ms | $0.021$21.27 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 47 answers, 47 right · 0 wrong · AUROC 0.769 Logic 44.4% · Math 59.4% | — | Decision | 54.3%52.6%–56.0% | g | 1,784 / 3,288 | 47 right of 47 kept0 mistakes · 3,241 skipped | 0.769 | 67.42chosen-answer Brier 0.1941 | 11.4%answered 94% | 312 ms | 360 ms | $0.009$9.07 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 372 answers, 356 right · 16 wrong · AUROC 0.777 Logic 45.1% · Math 53.5% | — | Decision | 50.6%48.9%–52.3% | h | 1,664 / 3,288 | 356 right of 372 kept16 wrong · 2,916 skipped | 0.777 | 64.36chosen-answer Brier 0.2189 | 10.1%answered 95% | 293 ms | 349 ms | $0.005$5.46 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 208 answers, 207 right · 1 wrong · AUROC 0.757 Logic 40.8% · Math 55.6% | — | Decision | 50.5%48.8%–52.3% | h | 1,662 / 3,288 | 207 right of 208 kept1 wrong · 3,080 skipped | 0.757 | 65.29chosen-answer Brier 0.1998 | 6.4%answered 97% | 185 ms | 292 ms | $0.004$3.98 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.5% · Math 99.7% | — | Chat | 98.2%97.7%–98.6% | a | 3,230 / 3,288 | Not measured | Not measured | — | 43.7%answered 99% | 4,030 ms | 10,080 ms | $7.042$7,042 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 1 setup not yet: Claude Fable 5.2 (chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 96.2% · Math 99.8% | — | Chat | 98.5%98.1%–98.9% | a | 3,240 / 3,288 | Not measured | Not measured | — | 41.3%answered 99% | 3,600 ms | 8,380 ms | $13.179$13,179 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.2% · Math 99.7% | — | Chat | 98.2%97.7%–98.6% | b | 3,228 / 3,288 | Not measured | Not measured | — | 15.7%answered 100% | 4,940 ms | 8,620 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Liquid AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | Models | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | Decision | 59.9%3,288 | 81.2%52.3% kept (1,718) | 93.0%34.0% kept (1,118) | 98.4%19.5% kept (640) | 100.0%5.1% kept (169) |
| d1.2-ultra Closed · Undisclosed | — | Decision | 63.6%3,288 | 84.7%59.1% kept (1,944) | 93.4%41.6% kept (1,369) | 99.1%25.0% kept (822) | 100.0%3.9% kept (128) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Models | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | Decision | 169 / 00.812 | 3 / 00.676 | 2 / 00.719 | 23 / 00.874 | 8 / 00.833 | 5 / 00.800 | 0 / 00.557 | 3 / 00.708 | 3 / 00.840 | 9 / 00.794 | 20 / 00.896 | 8 / 00.862 | 12 / 00.843 | 10 / 00.797 | 24 / 00.843 | 20 / 00.838 | 19 / 00.845 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 128 / 00.856 | 4 / 00.741 | 4 / 00.772 | 5 / 00.837 | 6 / 00.891 | 5 / 00.839 | 1 / 00.889 | 4 / 00.774 | 4 / 00.851 | 5 / 00.801 | 18 / 00.960 | 10 / 00.855 | 8 / 00.912 | 10 / 00.902 | 15 / 00.859 | 11 / 00.855 | 18 / 00.868 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Confidence not measured for 6 setups: GPT-6 Astra (chat), Gemini 4 Carbon (chat), GPT-6.1 Sol (chat), Qwen4-Max (high · chat), Qwen3.8 Flash (off · chat), GPT-6.1 Terra (chat).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: Grok 4.7 (chat) (missing Logic); Gemini 4 Carbon (high · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Logic 96.7% · Math 99.6% | — | Chat | 98.6%98.2%–99.0% | a | 3,243 / 3,288 | Not measured | Not measured | — | 60.9%answered 99% | 5,580 ms | 9,580 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Logic 96.7% · Math 99.6% | — | Chat | 98.6%98.1%–98.9% | a | 3,242 / 3,288 | Not measured | Not measured | — | 53.3%answered 99% | 16,350 ms | 90,630 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Logic 96.5% · Math 99.6% | — | Chat | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | — | 66.2%answered 99% | 5,240 ms | 10,410 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Logic 96.3% · Math 99.8% | — | Chat | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | — | 10.7%answered 100% | 5,700 ms | 19,750 ms | $2.919$2,919 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Logic 94.0% · Math 99.5% | — | Chat | 97.6%97.0%–98.1% | b | 3,209 / 3,288 | Not measured | Not measured | — | 31.3%answered 99% | 6,780 ms | 36,290 ms | $0.732$732 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Logic 94.0% · Math 99.2% | — | Chat | 97.4%96.8%–97.9% | b | 3,203 / 3,288 | Not measured | Not measured | — | 65.0%answered 98% | 4,440 ms | 8,290 ms | $0.000$0.00 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 128 answers, 128 right · 0 wrong · AUROC 0.856 Logic 51.6% · Math 69.9% | — | Decision | 63.6%61.9%–65.2% | c | 2,091 / 3,288 | 128 right of 128 kept0 mistakes · 3,160 skipped | 0.856 | 74.31chosen-answer Brier 0.1498 | 90.2%answered 48% | 217 ms | 347 ms | $0.008$8.37 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 169 answers, 169 right · 0 wrong · AUROC 0.812 Logic 49.2% · Math 65.5% | — | Decision | 59.9%58.2%–61.6% | d | 1,970 / 3,288 | 169 right of 169 kept0 mistakes · 3,119 skipped | 0.812 | 71.40chosen-answer Brier 0.1712 | 66.9%answered 65% | 208 ms | 390 ms | $0.004$3.98 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Perplexity Poindexter Labs TypeSafe Google Liquid AI REAX Fastino Nace.AI PelaAI StartLux tattrongvu Turion.AI Microsoft vLLM Semantic Router scar-ai OmniJev Logits Cloudflare Blockbrain Labs Other Metask Lab Zefan Cai Strands Agents Celeris MachinePulse Base Labs Subquadratic Inception Sthānika AI Apertix AI Patrick-18 newsbang quincyqiang Valen-Team victorbennett EldanRing Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 66.4%3,288 | 78.1%75.1% kept (2,469) | 88.0%56.4% kept (1,853) | 95.7%38.0% kept (1,249) | 100.0%11.7% kept (385) |
| Copernicus-31B Open · 31B | — | 71.0%3,288 | 89.0%64.6% kept (2,123) | 96.0%48.3% kept (1,588) | 99.3%31.5% kept (1,036) | 100.0%11.7% kept (384) |
| Jev 1.13 Closed · Efficient | — | 58.0%3,288 | 77.2%58.2% kept (1,913) | 89.6%35.8% kept (1,178) | 97.6%20.3% kept (669) | 100.0%9.5% kept (313) |
| Jev 1.14 Closed · Efficient | — | 59.3%3,288 | 79.8%54.3% kept (1,784) | 91.0%34.0% kept (1,117) | 97.2%19.5% kept (640) | 100.0%8.0% kept (263) |
| Gemini Decision 1 20261111 Closed · Efficient | — | 64.1%3,288 | 84.1%59.2% kept (1,948) | 92.5%42.8% kept (1,407) | 98.1%27.6% kept (906) | 100.0%7.8% kept (257) |
| d1-ultra Closed · Efficient | — | 67.5%3,288 | 84.9%64.9% kept (2,134) | 92.4%49.0% kept (1,610) | 98.2%32.2% kept (1,058) | 100.0%7.1% kept (233) |
| d1.1 Closed · Efficient | — | 58.6%3,288 | 79.6%53.7% kept (1,767) | 90.9%35.0% kept (1,150) | 97.0%20.4% kept (670) | 100.0%7.0% kept (231) |
| REAX 1.3 Open · 57B | — | 71.6%3,288 | 87.1%70.6% kept (2,320) | 96.0%50.3% kept (1,655) | 99.5%33.4% kept (1,098) | 100.0%7.0% kept (229) |
| GLiDE API Closed · Efficient | — | 65.2%3,288 | 79.2%71.0% kept (2,336) | 90.3%50.6% kept (1,664) | 98.3%30.3% kept (996) | 100.0%6.5% kept (213) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 53.1%3,288 | 79.7%42.5% kept (1,397) | 90.7%27.2% kept (893) | 97.0%16.2% kept (532) | 100.0%6.3% kept (207) |
| GLiDE v1.2 Open · 27B | — | 64.8%3,288 | 78.2%70.3% kept (2,312) | 91.8%48.6% kept (1,597) | 98.8%28.0% kept (922) | 100.0%5.4% kept (176) |
| d1.2 Closed · Undisclosed | — | 59.9%3,288 | 81.2%52.3% kept (1,718) | 93.0%34.0% kept (1,118) | 98.4%19.5% kept (640) | 100.0%5.1% kept (169) |
| KnowLine-9B-Gen1 Open · 9.0B | — | 54.5%3,288 | 75.0%49.1% kept (1,615) | 88.6%29.8% kept (981) | 98.0%16.4% kept (539) | 100.0%4.7% kept (153) |
| d1.2-ultra Closed · Undisclosed | — | 63.6%3,288 | 84.7%59.1% kept (1,944) | 93.4%41.6% kept (1,369) | 99.1%25.0% kept (822) | 100.0%3.9% kept (128) |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 52.4%3,288 | 70.2%53.6% kept (1,762) | 81.4%32.8% kept (1,078) | 94.7%16.5% kept (543) | 100.0%3.6% kept (117) |
| GLiDE 9B Open · 7.9B | — | 54.5%3,288 | 69.6%61.2% kept (2,011) | 85.4%35.9% kept (1,182) | 95.3%17.4% kept (571) | 100.0%3.2% kept (105) |
| Jeffan-9B-1211 Open · 9.0B | — | 55.8%3,288 | 76.0%53.9% kept (1,771) | 89.5%32.9% kept (1,082) | 98.4%15.2% kept (501) | 100.0%2.3% kept (74) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 54.4%3,288 | 74.0%51.4% kept (1,689) | 85.0%31.3% kept (1,029) | 97.0%15.4% kept (505) | 100.0%1.9% kept (63) |
| Turion Jev 31B Open · 31B | — | 63.3%3,288 | 84.0%58.2% kept (1,913) | 92.7%41.7% kept (1,372) | 99.4%21.0% kept (692) | 100.0%1.5% kept (50) |
| Microsoft-Decision-1 Closed · Efficient | — | 54.3%3,288 | 70.4%57.1% kept (1,876) | 86.3%30.5% kept (1,003) | 97.2%11.8% kept (389) | 100.0%1.4% kept (47) |
| d3-ultra Open · 26B | — | 65.7%3,288 | 73.5%81.9% kept (2,694) | 82.7%62.6% kept (2,057) | 96.3%37.4% kept (1,231) | 100.0%1.2% kept (39) |
| vera-medium-omni Open · 4.2B | — | 52.5%3,288 | 74.9%46.3% kept (1,523) | 88.0%23.3% kept (766) | 99.1%9.8% kept (322) | 100.0%1.2% kept (39) |
| OneJev-27B Open · 27B | — | 49.3%3,288 | 80.3%32.3% kept (1,062) | 92.2%18.7% kept (616) | 98.6%8.5% kept (281) | 100.0%0.9% kept (30) |
| Logits Sigma-1 Duo Open · 57B | — | 69.3%3,288 | 85.2%68.0% kept (2,236) | 96.8%45.1% kept (1,483) | 99.5%22.6% kept (742) | 100.0%0.7% kept (23) |
| Clef Open · 27B | — | 47.6%3,288 | 61.4%59.6% kept (1,959) | 75.3%33.0% kept (1,086) | 89.6%14.7% kept (483) | 100.0%0.5% kept (18) |
| KnowLine-2B-Gen1 Open · 2.0B | — | 34.1%3,288 | 51.8%24.8% kept (816) | 67.8%8.3% kept (273) | 88.1%2.6% kept (84) | 100.0%0.5% kept (16) |
| GLiDE 0.8B Open · 0.8B | — | 31.1%3,288 | 39.8%37.4% kept (1,230) | 49.1%14.3% kept (470) | 68.9%3.1% kept (103) | 100.0%0.4% kept (13) |
| KnowLine-12B-Gen1 Open · 12B | — | 60.9%3,288 | 81.1%55.3% kept (1,817) | 91.7%36.4% kept (1,197) | 99.1%16.2% kept (534) | 100.0%0.4% kept (13) |
| d1-2B Open · 1.9B | — | 34.9%3,288 | 56.6%21.7% kept (712) | 74.2%6.7% kept (221) | 88.2%2.1% kept (68) | 100.0%0.2% kept (7) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 35.9%3,288 | 66.0%18.4% kept (606) | 81.0%7.1% kept (232) | 94.4%2.2% kept (72) | 100.0%0.1% kept (4) |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 27.7%3,288 | 43.1%13.7% kept (450) | 57.9%3.3% kept (107) | 88.9%0.5% kept (18) | 100.0%0.1% kept (2) |
| OneJev1.6-0.8B Open · 0.8B | — | 27.8%3,288 | 39.9%15.5% kept (509) | 50.0%4.0% kept (130) | 71.4%0.6% kept (21) | 100.0%0.1% kept (2) |
| Clef-Flash Open · 9.0B | — | 46.1%3,288 | 63.4%43.4% kept (1,427) | 74.6%22.4% kept (735) | 89.5%7.0% kept (229) | none kept |
| Cygnet Open · 12B | — | 42.6%3,288 | 63.2%42.4% kept (1,394) | 79.1%16.0% kept (527) | 94.7%4.0% kept (132) | none kept |
| GLiClass v4 Edge Open · 0.4B | — | 23.0%3,288 | 31.9%5.5% kept (182) | 33.3%1.2% kept (39) | 57.1%0.2% kept (7) | none kept |
| GLiDE 0.4B Open · 0.4B | — | 21.5%3,288 | 26.9%5.5% kept (182) | 36.6%1.2% kept (41) | 100.0%0.1% kept (3) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | 54.6%3,288 | 78.4%45.4% kept (1,492) | 91.4%25.4% kept (836) | 99.6%8.2% kept (269) | none kept |
| Metask-Jev-4B Open · 4.5B | — | 36.7%3,288 | 64.0%17.2% kept (567) | 78.6%8.1% kept (266) | 89.1%1.7% kept (55) | none kept |
| Open-Jev-27B-v1.1 Open · 27B | — | 50.5%3,288 | 95.7%8.6% kept (282) | 100.0%0.9% kept (29) | none kept | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 28.0%3,288 | 46.1%10.1% kept (332) | 62.7%3.6% kept (118) | 91.3%0.7% kept (23) | none kept |
| Strands Decider 2B Open · 2.0B | — | 26.5%3,288 | 44.4%9.8% kept (322) | 65.1%1.3% kept (43) | 100.0%0.1% kept (3) | none kept |
| celeris-1-decision Closed · Efficient | — | 62.2%3,288 | 75.8%69.1% kept (2,271) | 87.8%48.8% kept (1,606) | 95.4%32.0% kept (1,053) | none kept |
| celeris-1.1-decision Closed · Efficient | — | 62.1%3,288 | 76.2%68.8% kept (2,262) | 87.9%48.6% kept (1,598) | 95.7%32.2% kept (1,060) | none kept |
| Karpo Decision 31B Open · 31B | — | 74.4%3,288 | 85.1%79.4% kept (2,610) | 93.0%63.7% kept (2,095) | 97.7%48.1% kept (1,580) | 99.7%30.7% kept (1,009) |
| Drex Max Open · 27B | — | 64.4%3,288 | 72.4%81.7% kept (2,686) | 80.8%64.5% kept (2,120) | 91.5%47.3% kept (1,556) | 98.0%26.9% kept (884) |
| KnowLine-31B-Gen1.5 Open · 31B | — | 71.1%3,288 | 86.5%69.3% kept (2,279) | 93.9%53.0% kept (1,744) | 98.5%35.2% kept (1,156) | 99.8%16.8% kept (554) |
| KnowLine-31B-Gen2 Open · 31B | — | 73.0%3,288 | 87.1%72.9% kept (2,396) | 93.9%56.8% kept (1,868) | 99.0%38.8% kept (1,275) | 99.8%15.9% kept (524) |
| Mammoth-1 Open · 27B | — | 67.3%3,288 | 82.3%71.0% kept (2,334) | 92.4%53.0% kept (1,744) | 97.9%37.3% kept (1,228) | 99.6%15.8% kept (520) |
| SubQ Snap 1.1 Closed · Efficient | — | 63.0%3,288 | 73.0%76.7% kept (2,522) | 82.6%58.1% kept (1,911) | 92.7%38.6% kept (1,268) | 99.3%13.2% kept (433) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 64.1%3,288 | 81.5%64.6% kept (2,125) | 92.6%46.9% kept (1,541) | 97.8%31.4% kept (1,031) | 99.5%12.5% kept (411) |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 64.8%3,288 | 77.5%72.3% kept (2,377) | 88.6%53.3% kept (1,752) | 97.0%35.3% kept (1,162) | 99.8%12.3% kept (405) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 63.6%3,288 | 80.6%65.2% kept (2,145) | 91.8%47.0% kept (1,545) | 97.4%31.3% kept (1,028) | 99.8%12.3% kept (405) |
| Mercury Decide Closed · Efficient | — | 50.6%3,288 | 64.0%65.1% kept (2,142) | 73.9%45.8% kept (1,505) | 83.8%28.6% kept (941) | 95.7%11.3% kept (372) |
| Sieve 31B v3 Open · 31B | — | 70.5%3,288 | 86.8%67.7% kept (2,227) | 94.3%51.4% kept (1,691) | 98.7%34.4% kept (1,131) | 99.3%9.0% kept (296) |
| GLiDE 27B Open · 27B | — | 67.3%3,288 | 80.6%72.4% kept (2,381) | 90.6%53.0% kept (1,742) | 98.5%32.5% kept (1,068) | 99.3%8.3% kept (274) |
| Cerno 9B v1.2 Open · 9.0B | — | 51.7%3,288 | 62.1%70.1% kept (2,306) | 73.5%47.7% kept (1,569) | 86.8%27.2% kept (895) | 96.1%8.5% kept (281) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 54.2%3,288 | 66.2%69.4% kept (2,281) | 78.0%47.1% kept (1,549) | 91.9%26.2% kept (863) | 98.9%8.1% kept (266) |
| Adjudex 9B v5 Open · 9.0B | — | 56.7%3,288 | 73.2%60.4% kept (1,985) | 85.5%38.9% kept (1,279) | 96.2%22.3% kept (732) | 99.2%7.7% kept (253) |
| Elephant-1 Open · 9.0B | — | 56.2%3,288 | 73.2%59.5% kept (1,957) | 86.1%38.2% kept (1,256) | 95.6%22.0% kept (724) | 99.6%7.2% kept (236) |
| d3 Open · 26B | — | 65.0%3,288 | 82.6%65.4% kept (2,150) | 94.1%46.2% kept (1,518) | 98.6%28.2% kept (927) | 99.5%6.6% kept (217) |
| d1 Closed · Efficient | — | 50.5%3,288 | 69.8%50.3% kept (1,653) | 82.8%28.5% kept (938) | 93.6%14.7% kept (484) | 99.5%6.3% kept (208) |
| Chinese Decision Pointer 4B Open · 4.0B | — | 47.0%3,288 | 59.2%60.7% kept (1,996) | 72.9%34.5% kept (1,133) | 85.5%17.6% kept (578) | 96.9%5.9% kept (194) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 51.1%3,288 | 68.4%55.4% kept (1,820) | 79.9%36.8% kept (1,211) | 89.0%20.5% kept (674) | 97.4%5.8% kept (190) |
| Valen-Decision-1.5-9B Open · 9.0B | — | 51.6%3,288 | 71.4%49.0% kept (1,611) | 84.7%27.3% kept (897) | 94.0%14.2% kept (466) | 98.8%5.1% kept (167) |
| Verdin-4B-v1.1 Open · 4.0B | — | 47.5%3,288 | 66.3%47.5% kept (1,563) | 79.5%26.8% kept (881) | 90.5%13.8% kept (454) | 97.6%5.0% kept (165) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 52.1%3,288 | 66.8%62.0% kept (2,039) | 77.7%41.7% kept (1,371) | 90.0%22.1% kept (728) | 99.2%4.0% kept (131) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 51.3%3,288 | 63.8%62.0% kept (2,038) | 76.6%37.0% kept (1,215) | 91.5%17.8% kept (585) | 98.6%2.2% kept (72) |
| Winnow-12B Open · 12B | — | 44.4%3,288 | 62.2%46.6% kept (1,531) | 73.1%31.1% kept (1,022) | 83.8%17.2% kept (567) | 98.1%1.6% kept (53) |
| GLiDE 2B Open · 1.9B | — | 35.2%3,288 | 45.7%44.7% kept (1,469) | 55.7%19.5% kept (641) | 75.7%5.5% kept (181) | 96.7%0.9% kept (30) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 385 / 00.844 | 22 / 00.752 | 5 / 00.764 | 35 / 00.878 | 26 / 00.828 | 8 / 00.776 | 0 / 00.577 | 8 / 00.785 | 15 / 00.845 | 17 / 00.790 | 36 / 00.935 | 27 / 00.839 | 25 / 00.900 | 22 / 00.933 | 35 / 00.845 | 49 / 00.871 | 55 / 00.834 |
| Copernicus-31B Open · 31B | — | 384 / 00.873 | 15 / 00.746 | 12 / 00.775 | 39 / 00.935 | 27 / 00.901 | 13 / 00.865 | 1 / 00.800 | 9 / 00.881 | 17 / 00.892 | 20 / 00.881 | 42 / 00.980 | 12 / 00.884 | 25 / 00.876 | 19 / 00.937 | 29 / 00.882 | 48 / 00.914 | 56 / 00.875 |
| Jev 1.13 Closed · Efficient | — | 313 / 00.817 | 7 / 00.708 | 5 / 00.714 | 53 / 00.875 | 17 / 00.836 | 8 / 00.756 | 0 / 00.671 | 9 / 00.761 | 8 / 00.797 | 9 / 00.835 | 23 / 00.894 | 20 / 00.801 | 30 / 00.868 | 17 / 00.831 | 24 / 00.821 | 41 / 00.850 | 42 / 00.818 |
| Jev 1.14 Closed · Efficient | — | 263 / 00.810 | 5 / 00.671 | 8 / 00.711 | 48 / 00.856 | 16 / 00.854 | 7 / 00.791 | 0 / 00.686 | 8 / 00.700 | 5 / 00.778 | 8 / 00.810 | 18 / 00.910 | 14 / 00.788 | 25 / 00.862 | 18 / 00.826 | 21 / 00.781 | 27 / 00.849 | 35 / 00.834 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 257 / 00.838 | 5 / 00.766 | 0 / 00.699 | 38 / 00.885 | 15 / 00.862 | 6 / 00.794 | 2 / 00.729 | 0 / 00.789 | 5 / 00.854 | 10 / 00.851 | 18 / 00.882 | 13 / 00.797 | 23 / 00.833 | 15 / 00.799 | 31 / 00.859 | 38 / 00.907 | 38 / 00.858 |
| d1-ultra Closed · Efficient | — | 233 / 00.852 | 12 / 00.759 | 3 / 00.715 | 5 / 00.846 | 23 / 00.865 | 8 / 00.755 | 0 / 00.650 | 5 / 00.813 | 9 / 00.896 | 13 / 00.887 | 12 / 00.910 | 13 / 00.941 | 18 / 00.902 | 13 / 00.893 | 20 / 00.845 | 30 / 00.885 | 49 / 00.888 |
| d1.1 Closed · Efficient | — | 231 / 00.811 | 4 / 00.676 | 6 / 00.712 | 38 / 00.906 | 13 / 00.811 | 5 / 00.768 | 0 / 00.714 | 4 / 00.721 | 3 / 00.825 | 7 / 00.796 | 16 / 00.848 | 14 / 00.854 | 19 / 00.870 | 15 / 00.849 | 23 / 00.815 | 30 / 00.798 | 34 / 00.835 |
| REAX 1.3 Open · 57B | — | 229 / 00.878 | 12 / 00.750 | 11 / 00.804 | 28 / 00.921 | 11 / 00.888 | 7 / 00.826 | 1 / 00.818 | 7 / 00.869 | 9 / 00.886 | 11 / 00.917 | 30 / 00.955 | 10 / 00.946 | 14 / 00.881 | 7 / 00.935 | 22 / 00.887 | 25 / 00.913 | 24 / 00.900 |
| GLiDE API Closed · Efficient | — | 213 / 00.852 | 7 / 00.777 | 7 / 00.765 | 38 / 00.866 | 17 / 00.874 | 5 / 00.746 | 1 / 00.625 | 11 / 00.778 | 4 / 00.853 | 4 / 00.840 | 20 / 00.987 | 13 / 00.872 | 29 / 00.885 | 11 / 00.924 | 15 / 00.896 | 17 / 00.811 | 14 / 00.881 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 207 / 00.789 | 3 / 00.652 | 0 / 00.690 | 30 / 00.815 | 8 / 00.819 | 6 / 00.817 | 0 / 00.533 | 0 / 00.740 | 5 / 00.798 | 13 / 00.841 | 20 / 00.905 | 9 / 00.768 | 16 / 00.837 | 14 / 00.766 | 22 / 00.762 | 35 / 00.836 | 26 / 00.803 |
| GLiDE v1.2 Open · 27B | — | 176 / 00.843 | 3 / 00.732 | 6 / 00.739 | 36 / 00.906 | 12 / 00.857 | 3 / 00.740 | 0 / 00.528 | 5 / 00.777 | 4 / 00.834 | 5 / 00.838 | 20 / 00.963 | 14 / 00.865 | 22 / 00.912 | 7 / 00.948 | 15 / 00.873 | 14 / 00.849 | 10 / 00.877 |
| d1.2 Closed · Undisclosed | — | 169 / 00.812 | 3 / 00.676 | 2 / 00.719 | 23 / 00.874 | 8 / 00.833 | 5 / 00.800 | 0 / 00.557 | 3 / 00.708 | 3 / 00.840 | 9 / 00.794 | 20 / 00.896 | 8 / 00.862 | 12 / 00.843 | 10 / 00.797 | 24 / 00.843 | 20 / 00.838 | 19 / 00.845 |
| KnowLine-9B-Gen1 Open · 9.0B | — | 153 / 00.782 | 3 / 00.638 | 0 / 00.692 | 27 / 00.843 | 13 / 00.816 | 5 / 00.712 | 0 / 00.595 | 6 / 00.674 | 6 / 00.803 | 5 / 00.760 | 19 / 00.873 | 6 / 00.799 | 14 / 00.792 | 7 / 00.780 | 17 / 00.803 | 11 / 00.821 | 14 / 00.817 |
| d1.2-ultra Closed · Undisclosed | — | 128 / 00.856 | 4 / 00.741 | 4 / 00.772 | 5 / 00.837 | 6 / 00.891 | 5 / 00.839 | 1 / 00.889 | 4 / 00.774 | 4 / 00.851 | 5 / 00.801 | 18 / 00.960 | 10 / 00.855 | 8 / 00.912 | 10 / 00.902 | 15 / 00.859 | 11 / 00.855 | 18 / 00.868 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 117 / 00.763 | 1 / 00.633 | 2 / 00.672 | 18 / 00.853 | 11 / 00.818 | 1 / 00.707 | 0 / 00.524 | 2 / 00.676 | 2 / 00.743 | 7 / 00.748 | 18 / 00.814 | 6 / 00.808 | 11 / 00.806 | 6 / 00.744 | 12 / 00.768 | 12 / 00.778 | 8 / 00.775 |
| GLiDE 9B Open · 7.9B | — | 105 / 00.786 | 3 / 00.637 | 4 / 00.729 | 25 / 00.911 | 7 / 00.800 | 2 / 00.737 | 0 / 00.731 | 3 / 00.700 | 1 / 00.716 | 4 / 00.752 | 15 / 00.805 | 3 / 00.774 | 10 / 00.873 | 4 / 00.853 | 7 / 00.776 | 8 / 00.851 | 9 / 00.765 |
| Jeffan-9B-1211 Open · 9.0B | — | 74 / 00.798 | 2 / 00.663 | 5 / 00.704 | 3 / 00.866 | 4 / 00.865 | 4 / 00.677 | 0 / 00.769 | 3 / 00.732 | 3 / 00.844 | 3 / 00.745 | 15 / 00.811 | 4 / 00.797 | 6 / 00.888 | 2 / 00.825 | 5 / 00.804 | 7 / 00.808 | 8 / 00.768 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 63 / 00.779 | 0 / 00.675 | 3 / 00.713 | 11 / 00.854 | 3 / 00.777 | 1 / 00.678 | 0 / 00.394 | 1 / 00.703 | 1 / 00.730 | 3 / 00.735 | 11 / 00.887 | 3 / 00.774 | 6 / 00.803 | 5 / 00.830 | 4 / 00.775 | 2 / 00.788 | 9 / 00.779 |
| Turion Jev 31B Open · 31B | — | 50 / 00.844 | 1 / 00.702 | 0 / 00.756 | 6 / 00.875 | 5 / 00.868 | 1 / 00.770 | 0 / 00.757 | 1 / 00.809 | 0 / 00.904 | 3 / 00.817 | 3 / 00.926 | 1 / 00.854 | 6 / 00.823 | 1 / 00.940 | 8 / 00.853 | 5 / 00.866 | 9 / 00.852 |
| Microsoft-Decision-1 Closed · Efficient | — | 47 / 00.769 | 1 / 00.628 | 2 / 00.683 | 17 / 00.823 | 5 / 00.789 | 0 / 00.696 | 0 / 00.615 | 1 / 00.670 | 0 / 00.747 | 0 / 00.750 | 4 / 00.848 | 4 / 00.806 | 5 / 00.837 | 1 / 00.801 | 2 / 00.768 | 3 / 00.824 | 2 / 00.794 |
| d3-ultra Open · 26B | — | 39 / 00.835 | 9 / 00.707 | 10 / 00.761 | 9 / 00.853 | 1 / 00.842 | 1 / 00.778 | 0 / 00.764 | 0 / 00.820 | 0 / 00.731 | 0 / 00.827 | 3 / 00.942 | 0 / 00.819 | 1 / 00.901 | 2 / 00.848 | 1 / 00.871 | 1 / 00.850 | 1 / 00.874 |
| vera-medium-omni Open · 4.2B | — | 39 / 00.777 | 0 / 00.659 | 0 / 00.710 | 6 / 00.876 | 2 / 00.782 | 3 / 00.704 | 0 / 00.636 | 1 / 00.695 | 1 / 00.836 | 1 / 00.706 | 13 / 00.860 | 0 / 00.848 | 2 / 00.837 | 1 / 00.797 | 4 / 00.770 | 1 / 00.809 | 4 / 00.784 |
| OneJev-27B Open · 27B | — | 30 / 00.770 | 0 / 00.633 | 0 / 00.712 | 4 / 00.845 | 2 / 00.768 | 0 / 00.645 | 0 / 00.806 | 1 / 00.750 | 0 / 00.767 | 1 / 00.778 | 2 / 00.884 | 1 / 00.758 | 1 / 00.811 | 3 / 00.787 | 4 / 00.757 | 6 / 00.785 | 5 / 00.789 |
| Logits Sigma-1 Duo Open · 57B | — | 23 / 00.868 | 1 / 00.738 | 1 / 00.764 | 6 / 00.880 | 2 / 00.900 | 0 / 00.841 | 0 / 00.667 | 0 / 00.847 | 1 / 00.888 | 0 / 00.872 | 2 / 00.950 | 0 / 00.919 | 1 / 00.910 | 1 / 00.939 | 4 / 00.895 | 1 / 00.871 | 3 / 00.911 |
| Clef Open · 27B | — | 18 / 00.747 | 2 / 00.607 | 0 / 00.704 | 4 / 00.846 | 2 / 00.751 | 1 / 00.708 | 0 / 00.875 | 2 / 00.723 | 0 / 00.720 | 0 / 00.798 | 3 / 00.778 | 0 / 00.793 | 1 / 00.744 | 0 / 00.878 | 2 / 00.748 | 0 / 00.802 | 1 / 00.774 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 16 / 00.626 | 0 / 00.501 | 0 / 00.571 | 8 / 00.756 | 0 / 00.570 | 0 / 00.578 | 0 / 00.533 | 0 / 00.508 | 0 / 00.540 | 0 / 00.655 | 0 / 00.667 | 1 / 00.779 | 4 / 00.736 | 1 / 00.699 | 1 / 00.644 | 0 / 00.656 | 1 / 00.587 |
| GLiDE 0.8B Open · 0.8B | — | 13 / 00.602 | 0 / 00.557 | 0 / 00.608 | 9 / 00.700 | 1 / 00.538 | 0 / 00.498 | 0 / 00.467 | 0 / 00.610 | 0 / 00.579 | 0 / 00.525 | 0 / 00.710 | 1 / 00.604 | 0 / 00.601 | 0 / 00.661 | 1 / 00.673 | 0 / 00.564 | 1 / 00.616 |
| KnowLine-12B-Gen1 Open · 12B | — | 13 / 00.810 | 1 / 00.701 | 1 / 00.711 | 4 / 00.937 | 2 / 00.838 | 0 / 00.736 | 0 / 00.814 | 0 / 00.752 | 0 / 00.840 | 0 / 00.833 | 5 / 00.915 | 0 / 00.800 | 0 / 00.824 | 0 / 00.916 | 0 / 00.775 | 0 / 00.857 | 0 / 00.794 |
| d1-2B Open · 1.9B | — | 7 / 00.648 | 0 / 00.549 | 0 / 00.606 | 3 / 00.787 | 0 / 00.634 | 0 / 00.629 | 0 / 00.318 | 0 / 00.572 | 1 / 00.582 | 0 / 00.589 | 0 / 00.721 | 1 / 00.778 | 1 / 00.625 | 0 / 00.718 | 1 / 00.709 | 0 / 00.657 | 0 / 00.677 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 4 / 00.675 | 0 / 00.539 | 0 / 00.652 | 2 / 00.767 | 0 / 00.661 | 0 / 00.601 | 0 / 00.333 | 0 / 00.560 | 0 / 00.665 | 0 / 00.655 | 0 / 00.788 | 0 / 00.813 | 1 / 00.755 | 0 / 00.703 | 0 / 00.614 | 0 / 00.658 | 1 / 00.679 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 2 / 00.590 | 0 / 00.520 | 0 / 00.531 | 2 / 00.746 | 0 / 00.486 | 0 / 00.603 | 0 / 00.667 | 0 / 00.628 | 0 / 00.573 | 0 / 00.490 | 0 / 00.683 | 0 / 00.792 | 0 / 00.660 | 0 / 00.653 | 0 / 00.603 | 0 / 00.557 | 0 / 00.565 |
| OneJev1.6-0.8B Open · 0.8B | — | 2 / 00.576 | 0 / 00.508 | 0 / 00.538 | 2 / 00.608 | 0 / 00.548 | 0 / 00.601 | 0 / 00.533 | 0 / 00.519 | 0 / 00.678 | 0 / 00.474 | 0 / 00.543 | 0 / 00.726 | 0 / 00.690 | 0 / 00.619 | 0 / 00.605 | 0 / 00.609 | 0 / 00.559 |
| Clef-Flash Open · 9.0B | — | No answers at this threshold0.707 | 0 / 00.537 | 0 / 00.670 | 0 / 00.790 | 0 / 00.711 | 0 / 00.642 | 0 / 00.361 | 0 / 00.614 | 0 / 00.736 | 0 / 00.670 | 0 / 00.766 | 0 / 00.789 | 0 / 00.744 | 0 / 00.823 | 0 / 00.722 | 0 / 00.804 | 0 / 00.737 |
| Cygnet Open · 12B | — | No answers at this threshold0.729 | 0 / 00.604 | 0 / 00.676 | 0 / 00.848 | 0 / 00.730 | 0 / 00.617 | 0 / 00.600 | 0 / 00.690 | 0 / 00.639 | 0 / 00.726 | 0 / 00.800 | 0 / 00.788 | 0 / 00.796 | 0 / 00.697 | 0 / 00.680 | 0 / 00.768 | 0 / 00.761 |
| GLiClass v4 Edge Open · 0.4B | — | No answers at this threshold0.511 | 0 / 00.516 | 0 / 00.547 | 0 / 00.584 | 0 / 00.524 | 0 / 00.554 | 0 / 00.550 | 0 / 00.476 | 0 / 00.474 | 0 / 00.505 | 0 / 00.412 | 0 / 00.522 | 0 / 00.412 | 0 / 00.460 | 0 / 00.566 | 0 / 00.543 | 0 / 00.518 |
| GLiDE 0.4B Open · 0.4B | — | No answers at this threshold0.520 | 0 / 00.531 | 0 / 00.545 | 0 / 00.639 | 0 / 00.496 | 0 / 00.528 | 0 / 00.875 | 0 / 00.509 | 0 / 00.606 | 0 / 00.458 | 0 / 00.370 | 0 / 00.632 | 0 / 00.450 | 0 / 00.478 | 0 / 00.493 | 0 / 00.498 | 0 / 00.515 |
| KnowLine-9B-Gen2 Open · 9.0B | — | No answers at this threshold0.786 | 0 / 00.672 | 0 / 00.663 | 0 / 00.859 | 0 / 00.788 | 0 / 00.711 | 0 / 00.712 | 0 / 00.671 | 0 / 00.801 | 0 / 00.819 | 0 / 00.834 | 0 / 00.841 | 0 / 00.847 | 0 / 00.824 | 0 / 00.774 | 0 / 00.806 | 0 / 00.817 |
| Metask-Jev-4B Open · 4.5B | — | No answers at this threshold0.659 | 0 / 00.591 | 0 / 00.604 | 0 / 00.830 | 0 / 00.614 | 0 / 00.544 | 0 / 00.933 | 0 / 00.617 | 0 / 00.737 | 0 / 00.613 | 0 / 00.633 | 0 / 00.619 | 0 / 00.666 | 0 / 00.727 | 0 / 00.660 | 0 / 00.624 | 0 / 00.664 |
| Open-Jev-27B-v1.1 Open · 27B | — | No answers at this threshold0.759 | 0 / 00.623 | 0 / 00.716 | 0 / 00.861 | 0 / 00.816 | 0 / 00.607 | 0 / 00.625 | 0 / 00.674 | 0 / 00.731 | 0 / 00.751 | 0 / 00.762 | 0 / 00.745 | 0 / 00.847 | 0 / 00.818 | 0 / 00.776 | 0 / 00.727 | 0 / 00.749 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | No answers at this threshold0.605 | 0 / 00.580 | 0 / 00.615 | 0 / 00.694 | 0 / 00.611 | 0 / 00.460 | 0 / 00.667 | 0 / 00.546 | 0 / 00.440 | 0 / 00.612 | 0 / 00.567 | 0 / 00.693 | 0 / 00.672 | 0 / 00.676 | 0 / 00.551 | 0 / 00.503 | 0 / 00.580 |
| Strands Decider 2B Open · 2.0B | — | No answers at this threshold0.573 | 0 / 00.544 | 0 / 00.562 | 0 / 00.634 | 0 / 00.510 | 0 / 00.458 | 0 / 00.615 | 0 / 00.534 | 0 / 00.589 | 0 / 00.583 | 0 / 00.635 | 0 / 00.615 | 0 / 00.574 | 0 / 00.632 | 0 / 00.556 | 0 / 00.542 | 0 / 00.627 |
| celeris-1-decision Closed · Efficient | — | No answers at this threshold0.829 | 0 / 00.706 | 0 / 00.679 | 0 / 00.867 | 0 / 00.857 | 0 / 00.787 | 0 / 00.807 | 0 / 00.794 | 0 / 00.884 | 0 / 00.896 | 0 / 00.952 | 0 / 00.840 | 0 / 00.881 | 0 / 00.836 | 0 / 00.838 | 0 / 00.869 | 0 / 00.849 |
| celeris-1.1-decision Closed · Efficient | — | No answers at this threshold0.830 | 0 / 00.709 | 0 / 00.679 | 0 / 00.853 | 0 / 00.863 | 0 / 00.790 | 0 / 00.875 | 0 / 00.790 | 0 / 00.863 | 0 / 00.896 | 0 / 00.944 | 0 / 00.853 | 0 / 00.877 | 0 / 00.812 | 0 / 00.852 | 0 / 00.867 | 0 / 00.855 |
| Karpo Decision 31B Open · 31B | — | 1006 / 30.882 | 58 / 00.805 | 33 / 20.758 | 78 / 10.877 | 85 / 00.899 | 34 / 00.801 | 3 / 00.833 | 34 / 00.880 | 38 / 00.933 | 50 / 00.890 | 83 / 00.952 | 45 / 00.879 | 66 / 00.915 | 53 / 00.950 | 91 / 00.902 | 114 / 00.893 | 141 / 00.904 |
| Drex Max Open · 27B | — | 866 / 180.841 | 34 / 00.729 | 39 / 50.758 | 74 / 30.838 | 67 / 10.834 | 25 / 00.803 | 3 / 00.750 | 32 / 00.799 | 28 / 00.867 | 36 / 40.794 | 63 / 00.923 | 42 / 20.877 | 75 / 00.894 | 44 / 00.908 | 83 / 00.876 | 96 / 20.855 | 125 / 10.847 |
| KnowLine-31B-Gen1.5 Open · 31B | — | 553 / 10.870 | 25 / 00.765 | 14 / 00.764 | 50 / 00.904 | 38 / 00.874 | 15 / 00.850 | 1 / 00.750 | 14 / 00.881 | 22 / 00.903 | 32 / 00.900 | 60 / 00.956 | 18 / 00.912 | 35 / 00.872 | 33 / 00.904 | 55 / 10.885 | 64 / 00.914 | 77 / 00.888 |
| KnowLine-31B-Gen2 Open · 31B | — | 523 / 10.882 | 26 / 00.792 | 5 / 00.785 | 23 / 00.907 | 38 / 00.916 | 15 / 00.862 | 2 / 00.742 | 14 / 00.847 | 24 / 00.902 | 37 / 00.921 | 70 / 00.956 | 12 / 00.871 | 28 / 00.882 | 28 / 00.916 | 56 / 10.863 | 68 / 00.908 | 77 / 00.886 |
| Mammoth-1 Open · 27B | — | 518 / 20.868 | 21 / 00.703 | 13 / 00.797 | 72 / 20.905 | 27 / 00.885 | 9 / 00.766 | 1 / 00.788 | 11 / 00.821 | 14 / 00.876 | 30 / 00.897 | 41 / 00.909 | 17 / 00.893 | 47 / 00.930 | 31 / 00.904 | 53 / 00.900 | 59 / 00.896 | 72 / 00.903 |
| SubQ Snap 1.1 Closed · Efficient | — | 430 / 30.827 | 17 / 00.736 | 18 / 10.743 | 35 / 10.857 | 28 / 00.819 | 12 / 00.784 | 1 / 00.722 | 15 / 00.780 | 10 / 00.808 | 19 / 00.754 | 47 / 00.929 | 35 / 00.829 | 31 / 00.832 | 28 / 00.931 | 37 / 10.802 | 45 / 00.871 | 52 / 00.848 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 409 / 20.849 | 9 / 00.730 | 12 / 00.783 | 57 / 20.791 | 22 / 00.861 | 6 / 00.818 | 1 / 00.806 | 8 / 00.772 | 10 / 00.823 | 16 / 00.819 | 34 / 00.961 | 34 / 00.901 | 43 / 00.903 | 33 / 00.932 | 32 / 00.840 | 47 / 00.875 | 45 / 00.863 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 404 / 10.846 | 7 / 00.729 | 24 / 00.771 | 58 / 10.877 | 23 / 00.839 | 8 / 00.850 | 1 / 00.471 | 14 / 00.799 | 9 / 00.861 | 15 / 00.817 | 28 / 00.903 | 31 / 00.840 | 47 / 00.910 | 30 / 00.919 | 36 / 00.866 | 39 / 00.853 | 34 / 00.893 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 404 / 10.854 | 8 / 00.728 | 12 / 00.787 | 58 / 10.860 | 23 / 00.850 | 9 / 00.842 | 1 / 00.571 | 6 / 00.827 | 14 / 00.799 | 18 / 00.854 | 28 / 00.927 | 36 / 00.869 | 43 / 00.887 | 31 / 00.946 | 29 / 00.851 | 41 / 00.885 | 47 / 00.884 |
| Mercury Decide Closed · Efficient | — | 356 / 160.777 | 14 / 20.670 | 28 / 10.744 | 42 / 40.775 | 25 / 00.811 | 9 / 00.692 | 1 / 00.729 | 8 / 00.748 | 6 / 00.687 | 11 / 30.718 | 26 / 10.859 | 23 / 20.801 | 31 / 10.808 | 18 / 00.821 | 22 / 10.751 | 44 / 10.785 | 48 / 00.822 |
| Sieve 31B v3 Open · 31B | — | 294 / 20.865 | 9 / 00.764 | 16 / 20.745 | 54 / 00.861 | 24 / 00.871 | 6 / 00.848 | 0 / 00.903 | 7 / 00.856 | 8 / 00.898 | 10 / 00.863 | 26 / 00.956 | 15 / 00.918 | 22 / 00.907 | 14 / 00.939 | 24 / 00.895 | 31 / 00.879 | 28 / 00.848 |
| GLiDE 27B Open · 27B | — | 272 / 20.852 | 6 / 00.754 | 17 / 00.800 | 51 / 10.888 | 17 / 10.858 | 6 / 00.787 | 0 / 00.569 | 8 / 00.761 | 5 / 00.846 | 7 / 00.870 | 22 / 00.948 | 20 / 00.938 | 36 / 00.873 | 18 / 00.905 | 21 / 00.861 | 20 / 00.848 | 18 / 00.875 |
| Cerno 9B v1.2 Open · 9.0B | — | 270 / 110.776 | 8 / 10.645 | 3 / 00.745 | 15 / 00.881 | 24 / 00.806 | 9 / 00.677 | 1 / 10.561 | 8 / 00.679 | 6 / 00.797 | 11 / 00.751 | 31 / 00.843 | 17 / 00.747 | 22 / 20.806 | 13 / 00.812 | 31 / 20.720 | 33 / 30.836 | 38 / 20.776 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 263 / 30.795 | 7 / 00.712 | 9 / 00.689 | 40 / 00.910 | 18 / 10.778 | 5 / 00.684 | 0 / 00.606 | 7 / 00.731 | 7 / 10.745 | 8 / 00.759 | 29 / 10.839 | 10 / 00.829 | 18 / 00.822 | 12 / 00.853 | 33 / 00.824 | 24 / 00.814 | 36 / 00.820 |
| Adjudex 9B v5 Open · 9.0B | — | 251 / 20.802 | 4 / 00.706 | 2 / 00.730 | 48 / 10.869 | 18 / 00.815 | 4 / 00.701 | 0 / 00.729 | 6 / 00.684 | 8 / 00.764 | 9 / 00.797 | 31 / 00.867 | 12 / 10.814 | 21 / 00.820 | 16 / 00.840 | 23 / 00.813 | 21 / 00.812 | 28 / 00.806 |
| Elephant-1 Open · 9.0B | — | 235 / 10.797 | 5 / 00.671 | 9 / 00.725 | 39 / 00.925 | 14 / 00.821 | 5 / 00.711 | 0 / 00.486 | 6 / 00.684 | 6 / 00.823 | 10 / 00.730 | 28 / 00.833 | 10 / 10.785 | 15 / 00.868 | 14 / 00.879 | 25 / 00.778 | 24 / 00.834 | 25 / 00.774 |
| d3 Open · 26B | — | 216 / 10.858 | 3 / 00.755 | 11 / 00.761 | 40 / 00.863 | 13 / 10.867 | 5 / 00.791 | 1 / 00.806 | 6 / 00.811 | 7 / 00.907 | 8 / 00.847 | 21 / 00.955 | 15 / 00.904 | 25 / 00.905 | 13 / 00.893 | 21 / 00.886 | 13 / 00.875 | 14 / 00.872 |
| d1 Closed · Efficient | — | 207 / 10.757 | 3 / 00.630 | 5 / 00.683 | 47 / 10.828 | 14 / 00.765 | 5 / 00.653 | 0 / 00.530 | 4 / 00.700 | 3 / 00.789 | 6 / 00.752 | 9 / 00.761 | 11 / 00.726 | 21 / 00.802 | 11 / 00.738 | 20 / 00.769 | 23 / 00.777 | 25 / 00.784 |
| Chinese Decision Pointer 4B Open · 4.0B | — | 188 / 60.743 | 1 / 00.644 | 8 / 00.699 | 30 / 00.866 | 14 / 20.793 | 5 / 00.668 | 0 / 00.424 | 5 / 00.719 | 2 / 00.797 | 5 / 20.655 | 16 / 00.735 | 8 / 10.772 | 19 / 00.732 | 7 / 00.768 | 16 / 00.771 | 24 / 10.729 | 28 / 00.726 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 185 / 50.769 | 3 / 00.642 | 1 / 00.693 | 31 / 00.879 | 7 / 10.781 | 4 / 00.688 | 0 / 00.300 | 2 / 00.694 | 3 / 00.780 | 9 / 00.732 | 26 / 00.884 | 10 / 10.807 | 21 / 00.854 | 11 / 10.773 | 15 / 10.730 | 19 / 00.806 | 23 / 10.753 |
| Valen-Decision-1.5-9B Open · 9.0B | — | 165 / 20.765 | 5 / 10.598 | 5 / 10.709 | 25 / 00.785 | 16 / 00.837 | 3 / 00.663 | 0 / 00.667 | 7 / 00.778 | 4 / 00.849 | 5 / 00.753 | 17 / 00.876 | 9 / 00.765 | 12 / 00.777 | 7 / 00.834 | 13 / 00.782 | 16 / 00.795 | 21 / 00.776 |
| Verdin-4B-v1.1 Open · 4.0B | — | 161 / 40.747 | 2 / 00.648 | 5 / 00.672 | 28 / 00.884 | 14 / 10.731 | 5 / 00.719 | 0 / 00.548 | 4 / 00.677 | 6 / 00.743 | 8 / 10.763 | 16 / 00.824 | 8 / 00.774 | 14 / 00.799 | 9 / 20.772 | 17 / 00.761 | 12 / 00.772 | 13 / 00.696 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 130 / 10.782 | 4 / 00.663 | 1 / 00.731 | 12 / 00.860 | 6 / 00.765 | 5 / 00.695 | 0 / 00.450 | 2 / 00.701 | 3 / 00.836 | 9 / 00.788 | 23 / 00.913 | 7 / 00.830 | 12 / 00.797 | 7 / 00.792 | 11 / 00.764 | 15 / 00.792 | 13 / 10.765 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 71 / 10.752 | 2 / 00.634 | 3 / 10.641 | 12 / 00.881 | 3 / 00.791 | 2 / 00.611 | 0 / 00.500 | 1 / 00.688 | 1 / 00.814 | 4 / 00.699 | 11 / 00.845 | 1 / 00.842 | 7 / 00.813 | 1 / 00.742 | 7 / 00.743 | 9 / 00.803 | 7 / 00.747 |
| Winnow-12B Open · 12B | — | 52 / 10.737 | 1 / 00.608 | 6 / 00.670 | 25 / 00.835 | 5 / 00.742 | 1 / 00.638 | 0 / 00.686 | 0 / 00.670 | 1 / 00.642 | 0 / 10.743 | 1 / 00.787 | 1 / 00.742 | 5 / 00.731 | 3 / 00.785 | 1 / 00.746 | 1 / 00.758 | 1 / 00.774 |
| GLiDE 2B Open · 1.9B | — | 29 / 10.637 | 1 / 00.525 | 1 / 00.637 | 12 / 00.787 | 3 / 10.625 | 0 / 00.569 | 0 / 00.519 | 1 / 00.554 | 0 / 00.590 | 1 / 00.676 | 1 / 00.646 | 1 / 00.728 | 3 / 00.663 | 2 / 00.596 | 1 / 00.617 | 1 / 00.616 | 1 / 00.639 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 3,276 of 3,288); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 1,009 answers, 1,006 right · 3 wrong · AUROC 0.882 Logic 60.7% · Math 81.6% | — | 74.4%72.9%–75.9% | a | 2,447 / 3,288 | 1,006 right of 1,009 kept3 wrong · 2,279 skipped | 0.882 | 81.13chosen-answer Brier 0.1215 | 5.7%answered 98% | 162 ms | 198 ms | $0.118$118estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 524 answers, 523 right · 1 wrong · AUROC 0.882 Logic 58.9% · Math 80.3% | — | 73.0%71.4%–74.5% | ab | 2,399 / 3,288 | 523 right of 524 kept1 wrong · 2,764 skipped | 0.882 | 80.27chosen-answer Brier 0.1242 | 0.1%answered 100% | 116 ms | 177 ms | $0.093$92.98estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 229 answers, 229 right · 0 wrong · AUROC 0.878 Logic 59.3% · Math 78.0% | — | 71.6%70.0%–73.1% | b | 2,354 / 3,288 | 229 right of 229 kept0 mistakes · 3,059 skipped | 0.878 | 79.30chosen-answer Brier 0.1300 | 3.8%answered 99% | 200 ms | 260 ms | $0.148$148estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 554 answers, 553 right · 1 wrong · AUROC 0.870 Logic 59.0% · Math 77.4% | — | 71.1%69.5%–72.6% | bc | 2,338 / 3,288 | 553 right of 554 kept1 wrong · 2,734 skipped | 0.870 | 78.92chosen-answer Brier 0.1327 | 0.7%answered 100% | 811 ms | 1,111 ms | $0.052$51.85estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 384 answers, 384 right · 0 wrong · AUROC 0.873 Logic 58.7% · Math 77.3% | — | 71.0%69.4%–72.5% | bc | 2,333 / 3,288 | 384 right of 384 kept0 mistakes · 2,904 skipped | 0.873 | 78.76chosen-answer Brier 0.1344 | 1.8%answered 99% | 128 ms | 218 ms | $0.102$102estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 296 answers, 294 right · 2 wrong · AUROC 0.865 Logic 57.9% · Math 77.1% | — | 70.5%68.9%–72.1% | bcd | 2,319 / 3,288 | 294 right of 296 kept2 wrong · 2,992 skipped | 0.865 | 78.49chosen-answer Brier 0.1354 | 3.8%answered 99% | 676 ms | 1,173 ms | $0.047$47.29estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 23 answers, 23 right · 0 wrong · AUROC 0.868 Logic 57.8% · Math 75.2% | — | 69.3%67.7%–70.8% | cde | 2,277 / 3,288 | 23 right of 23 kept0 mistakes · 3,265 skipped | 0.868 | 77.60chosen-answer Brier 0.1405 | 4.4%answered 99% | 207 ms | 324 ms | $0.165$165estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 233 answers, 233 right · 0 wrong · AUROC 0.852 Logic 54.9% · Math 74.1% | — | 67.5%65.9%–69.1% | defg | 2,221 / 3,288 | 233 right of 233 kept0 mistakes · 3,055 skipped | 0.852 | 76.50chosen-answer Brier 0.1455 | 89.1%answered 52% | 180 ms | 426 ms | $0.008$8.37 |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 274 answers, 272 right · 2 wrong · AUROC 0.852 Logic 58.4% · Math 72.0% | — | 67.3%65.7%–68.9% | efh | 2,214 / 3,288 | 272 right of 274 kept2 wrong · 3,014 skipped | 0.852 | 76.29chosen-answer Brier 0.1475 | 31.9%answered 89% | 450 ms | 587 ms | $0.029$28.55estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 520 answers, 518 right · 2 wrong · AUROC 0.868 Logic 54.5% · Math 74.0% | — | 67.3%65.7%–68.9% | efghi | 2,213 / 3,288 | 518 right of 520 kept2 wrong · 2,768 skipped | 0.868 | 76.68chosen-answer Brier 0.1394 | 71.4%answered 69% | 150 ms | 232 ms | $0.028$27.98 list$0.120$120estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 385 answers, 385 right · 0 wrong · AUROC 0.844 Logic 54.8% · Math 72.5% | — | 66.4%64.8%–68.0% | fghij | 2,184 / 3,288 | 385 right of 385 kept0 mistakes · 2,903 skipped | 0.844 | 75.44chosen-answer Brier 0.1554 | 13.5%answered 95% | 97 ms | 125 ms | $0.071$71.10estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.835 Logic 54.6% · Math 71.5% | — | 65.7%64.1%–67.3% | fghijk | 2,161 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.835 | 74.31chosen-answer Brier 0.1711 | 7.5%answered 97% | 129 ms | 168 ms | $0.096$95.71estimate · RTX PRO 6000 Blackwell |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 213 answers, 213 right · 0 wrong · AUROC 0.852 Logic 54.5% · Math 70.8% | — | 65.2%63.6%–66.8% | fghijkl | 2,144 / 3,288 | 213 right of 213 kept0 mistakes · 3,075 skipped | 0.852 | 75.04chosen-answer Brier 0.1513 | 49.2%answered 80% | 266 ms | 284 ms | $0.036$36.42 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 217 answers, 216 right · 1 wrong · AUROC 0.858 Logic 54.5% · Math 70.4% | — | 65.0%63.3%–66.6% | fghijkl | 2,136 / 3,288 | 216 right of 217 kept1 wrong · 3,071 skipped | 0.858 | 75.16chosen-answer Brier 0.1464 | 9.3%answered 97% | 57 ms | 86 ms | $0.046$46.28estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 405 answers, 404 right · 1 wrong · AUROC 0.846 Logic 54.8% · Math 70.1% | — | 64.8%63.2%–66.5% | fghijkl | 2,132 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.846 | 74.61chosen-answer Brier 0.1562 | 48.3%answered 79% | 327 ms | 518 ms | $0.016$16.13 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 176 answers, 176 right · 0 wrong · AUROC 0.843 Logic 55.2% · Math 69.9% | — | 64.8%63.2%–66.5% | gijkl | 2,132 / 3,288 | 176 right of 176 kept0 mistakes · 3,112 skipped | 0.843 | 74.68chosen-answer Brier 0.1548 | 42.3%answered 83% | 81 ms | 136 ms | $0.010$9.71 list$0.066$66.05estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 884 answers, 866 right · 18 wrong · AUROC 0.841 Logic 53.1% · Math 70.3% | — | 64.4%62.8%–66.0% | gijkl | 2,118 / 3,288 | 866 right of 884 kept18 wrong · 2,404 skipped | 0.841 | 73.20chosen-answer Brier 0.1802 | 5.7%answered 98% | 81 ms | 130 ms | $0.016$16.31 list$0.064$63.74estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 411 answers, 409 right · 2 wrong · AUROC 0.849 Logic 53.3% · Math 69.8% | — | 64.1%62.5%–65.8% | jklm | 2,109 / 3,288 | 409 right of 411 kept2 wrong · 2,877 skipped | 0.849 | 74.56chosen-answer Brier 0.1502 | 10.3%answered 96% | 230 ms | 439 ms | $0.005$5.38 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 257 answers, 257 right · 0 wrong · AUROC 0.838 Logic 50.0% · Math 71.3% | — | 64.1%62.4%–65.7% | hijklm | 2,106 / 3,288 | 257 right of 257 kept0 mistakes · 3,031 skipped | 0.838 | 74.23chosen-answer Brier 0.1559 | 3.0%answered 99% | 227 ms | 347 ms | $0.027$27.07 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 128 answers, 128 right · 0 wrong · AUROC 0.856 Logic 51.6% · Math 69.9% | — | 63.6%61.9%–65.2% | jklm | 2,091 / 3,288 | 128 right of 128 kept0 mistakes · 3,160 skipped | 0.856 | 74.31chosen-answer Brier 0.1498 | 90.2%answered 48% | 217 ms | 347 ms | $0.008$8.37 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 405 answers, 404 right · 1 wrong · AUROC 0.854 Logic 52.5% · Math 69.3% | with web | 63.6%61.9%–65.2% | klm | 2,090 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.854 | 74.28chosen-answer Brier 0.1501 | 25.8%answered 90% | 430 ms | 707 ms | $1.025$1,025 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 50 answers, 50 right · 0 wrong · AUROC 0.844 Logic 52.3% · Math 69.0% | — | 63.3%61.6%–64.9% | jklmn | 2,081 / 3,288 | 50 right of 50 kept0 mistakes · 3,238 skipped | 0.844 | 73.88chosen-answer Brier 0.1553 | 1.9%answered 99% | 301 ms | 448 ms | $0.020$20.01estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 433 answers, 430 right · 3 wrong · AUROC 0.827 Logic 52.4% · Math 68.6% | — | 63.0%61.4%–64.7% | klmn | 2,072 / 3,288 | 430 right of 433 kept3 wrong · 2,855 skipped | 0.827 | 72.66chosen-answer Brier 0.1770 | 58.3%answered 73% | 382 ms | 391 ms | $0.011$10.75 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.829 Logic 47.3% · Math 70.0% | — | 62.2%60.6%–63.9% | lmno | 2,046 / 3,288 | No answers at this threshold | 0.829 | 72.82chosen-answer Brier 0.1659 | 87.2%answered 50% | 388 ms | 392 ms | $0.011$11.25 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.830 Logic 47.3% · Math 69.9% | — | 62.1%60.5%–63.8% | lmnop | 2,043 / 3,288 | No answers at this threshold | 0.830 | 72.78chosen-answer Brier 0.1657 | 86.7%answered 51% | 381 ms | 388 ms | $0.001$1.13 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.810 Logic 50.6% · Math 66.3% | — | 60.9%59.2%–62.5% | mnopq | 2,002 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.810 | 71.86chosen-answer Brier 0.1716 | 4.5%answered 98% | 51 ms | 76 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 169 answers, 169 right · 0 wrong · AUROC 0.812 Logic 49.2% · Math 65.5% | — | 59.9%58.2%–61.6% | nopqr | 1,970 / 3,288 | 169 right of 169 kept0 mistakes · 3,119 skipped | 0.812 | 71.40chosen-answer Brier 0.1712 | 66.9%answered 65% | 208 ms | 390 ms | $0.004$3.98 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 263 answers, 263 right · 0 wrong · AUROC 0.810 Logic 48.2% · Math 65.1% | — | 59.3%57.6%–61.0% | opqrs | 1,951 / 3,288 | 263 right of 263 kept0 mistakes · 3,025 skipped | 0.810 | 71.00chosen-answer Brier 0.1734 | 15.4%answered 93% | 164 ms | 188 ms | $0.021$21.27 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 231 answers, 231 right · 0 wrong · AUROC 0.811 Logic 46.0% · Math 65.2% | — | 58.6%56.9%–60.3% | pqrs | 1,928 / 3,288 | 231 right of 231 kept0 mistakes · 3,057 skipped | 0.811 | 70.69chosen-answer Brier 0.1726 | 86.0%answered 46% | 174 ms | 285 ms | $0.004$3.98 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 313 answers, 313 right · 0 wrong · AUROC 0.817 Logic 46.3% · Math 64.1% | — | 58.0%56.3%–59.7% | qrst | 1,907 / 3,288 | 313 right of 313 kept0 mistakes · 2,975 skipped | 0.817 | 70.41chosen-answer Brier 0.1718 | 10.2%answered 96% | 166 ms | 205 ms | $0.021$21.27 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 253 answers, 251 right · 2 wrong · AUROC 0.802 Logic 46.4% · Math 62.1% | — | 56.7%55.0%–58.4% | rstu | 1,865 / 3,288 | 251 right of 253 kept2 wrong · 3,035 skipped | 0.802 | 69.27chosen-answer Brier 0.1818 | 7.0%answered 97% | 38 ms | 45 ms | $0.028$28.19estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 236 answers, 235 right · 1 wrong · AUROC 0.797 Logic 46.4% · Math 61.3% | — | 56.2%54.5%–57.9% | stu | 1,849 / 3,288 | 235 right of 236 kept1 wrong · 3,052 skipped | 0.797 | 68.96chosen-answer Brier 0.1831 | 69.4%answered 60% | 80 ms | 94 ms | $0.010$9.79 list$0.058$58.27estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 74 answers, 74 right · 0 wrong · AUROC 0.798 Logic 48.2% · Math 59.8% | — | 55.8%54.1%–57.5% | stu | 1,836 / 3,288 | 74 right of 74 kept0 mistakes · 3,214 skipped | 0.798 | 68.92chosen-answer Brier 0.1799 | 13.5%answered 94% | 39 ms | 53 ms | $0.032$32.40estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.786 Logic 43.9% · Math 60.1% | — | 54.6%52.9%–56.3% | tuv | 1,794 / 3,288 | No answers at this threshold | 0.786 | 67.99chosen-answer Brier 0.1858 | 10.4%answered 95% | 42 ms | 57 ms | $0.034$34.08estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 105 answers, 105 right · 0 wrong · AUROC 0.786 Logic 47.8% · Math 57.9% | — | 54.5%52.8%–56.2% | uv | 1,791 / 3,288 | 105 right of 105 kept0 mistakes · 3,183 skipped | 0.786 | 67.72chosen-answer Brier 0.1902 | 26.9%answered 86% | 177 ms | 222 ms | $0.011$11.45estimate · DGX Spark |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 153 answers, 153 right · 0 wrong · AUROC 0.782 Logic 44.3% · Math 59.8% | — | 54.5%52.8%–56.2% | uv | 1,791 / 3,288 | 153 right of 153 kept0 mistakes · 3,135 skipped | 0.782 | 67.96chosen-answer Brier 0.1854 | 3.4%answered 98% | 38 ms | 55 ms | $0.030$29.87estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 63 answers, 63 right · 0 wrong · AUROC 0.779 Logic 46.6% · Math 58.4% | — | 54.4%52.7%–56.1% | uv | 1,788 / 3,288 | 63 right of 63 kept0 mistakes · 3,225 skipped | 0.779 | 67.73chosen-answer Brier 0.1893 | 9.6%answered 95% | 142 ms | 165 ms | $0.009$8.92estimate · DGX Spark |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 47 answers, 47 right · 0 wrong · AUROC 0.769 Logic 44.4% · Math 59.4% | — | 54.3%52.6%–56.0% | uvw | 1,784 / 3,288 | 47 right of 47 kept0 mistakes · 3,241 skipped | 0.769 | 67.42chosen-answer Brier 0.1941 | 11.4%answered 94% | 312 ms | 360 ms | $0.009$9.07 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 266 answers, 263 right · 3 wrong · AUROC 0.795 Logic 45.3% · Math 58.8% | — | 54.2%52.5%–55.9% | uvw | 1,782 / 3,288 | 263 right of 266 kept3 wrong · 3,022 skipped | 0.795 | 66.98chosen-answer Brier 0.2023 | 6.1%answered 97% | 46 ms | 51 ms | $0.033$32.61estimate · RTX PRO 6000 Blackwell |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 207 answers, 207 right · 0 wrong · AUROC 0.789 Logic 41.9% · Math 59.0% | — | 53.1%51.4%–54.8% | uvwx | 1,747 / 3,288 | 207 right of 207 kept0 mistakes · 3,081 skipped | 0.789 | 67.48chosen-answer Brier 0.1818 | 28.1%answered 85% | 339 ms | 472 ms | $0.012$12.17 |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.777 Logic 44.1% · Math 56.8% | — | 52.5%50.8%–54.2% | vwxy | 1,725 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.777 | 66.69chosen-answer Brier 0.1909 | 7.9%answered 96% | 31 ms | 36 ms | $0.024$24.19estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 117 answers, 117 right · 0 wrong · AUROC 0.763 Logic 45.9% · Math 55.8% | — | 52.4%50.7%–54.1% | vwxy | 1,724 / 3,288 | 117 right of 117 kept0 mistakes · 3,171 skipped | 0.763 | 66.33chosen-answer Brier 0.1978 | 14.4%answered 93% | 68 ms | 80 ms | $0.004$4.28estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 131 answers, 130 right · 1 wrong · AUROC 0.782 Logic 44.7% · Math 55.9% | — | 52.1%50.4%–53.8% | vwxy | 1,713 / 3,288 | 130 right of 131 kept1 wrong · 3,157 skipped | 0.782 | 65.93chosen-answer Brier 0.2024 | 15.0%answered 92% | 29 ms | 35 ms | $0.024$23.77estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 281 answers, 270 right · 11 wrong · AUROC 0.776 Logic 43.3% · Math 56.0% | — | 51.7%50.0%–53.4% | vwxy | 1,699 / 3,288 | 270 right of 281 kept11 wrong · 3,007 skipped | 0.776 | 64.84chosen-answer Brier 0.2199 | 3.1%answered 99% | 163 ms | 240 ms | $0.011$10.82estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 167 answers, 165 right · 2 wrong · AUROC 0.765 Logic 44.4% · Math 55.3% | — | 51.6%49.9%–53.3% | vwxy | 1,696 / 3,288 | 165 right of 167 kept2 wrong · 3,121 skipped | 0.765 | 65.99chosen-answer Brier 0.1960 | 7.1%answered 96% | 59 ms | 65 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 72 answers, 71 right · 1 wrong · AUROC 0.752 Logic 42.3% · Math 56.0% | — | 51.3%49.6%–53.0% | vwxyz | 1,687 / 3,288 | 71 right of 72 kept1 wrong · 3,216 skipped | 0.752 | 65.07chosen-answer Brier 0.2116 | 4.4%answered 98% | 76 ms | 104 ms | $0.005$5.02estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 190 answers, 185 right · 5 wrong · AUROC 0.769 Logic 43.1% · Math 55.2% | — | 51.1%49.4%–52.8% | vwxyz | 1,680 / 3,288 | 185 right of 190 kept5 wrong · 3,098 skipped | 0.769 | 65.47chosen-answer Brier 0.2016 | 18.9%answered 90% | 41 ms | 51 ms | $0.003$2.72 list$0.032$31.55estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 372 answers, 356 right · 16 wrong · AUROC 0.777 Logic 45.1% · Math 53.5% | — | 50.6%48.9%–52.3% | wxyzA | 1,664 / 3,288 | 356 right of 372 kept16 wrong · 2,916 skipped | 0.777 | 64.36chosen-answer Brier 0.2189 | 10.1%answered 95% | 293 ms | 349 ms | $0.005$5.46 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 208 answers, 207 right · 1 wrong · AUROC 0.757 Logic 40.8% · Math 55.6% | — | 50.5%48.8%–52.3% | wxyzA | 1,662 / 3,288 | 207 right of 208 kept1 wrong · 3,080 skipped | 0.757 | 65.29chosen-answer Brier 0.1998 | 6.4%answered 97% | 185 ms | 292 ms | $0.004$3.98 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.759 Logic 42.6% · Math 54.6% | — | 50.5%48.7%–52.2% | xyzA | 1,659 / 3,288 | No answers at this threshold | 0.759 | 62.91chosen-answer Brier 0.2465 | 14.5%answered 92% | 250 ms | 482 ms | $0.212$212estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 30 answers, 30 right · 0 wrong · AUROC 0.770 Logic 40.6% · Math 53.8% | — | 49.3%47.6%–51.0% | yzAB | 1,620 / 3,288 | 30 right of 30 kept0 mistakes · 3,258 skipped | 0.770 | 65.05chosen-answer Brier 0.1917 | 1.6%answered 99% | 80 ms | 134 ms | $0.069$69.42estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 18 answers, 18 right · 0 wrong · AUROC 0.747 Logic 40.8% · Math 51.1% | — | 47.6%45.9%–49.3% | zABC | 1,564 / 3,288 | 18 right of 18 kept0 mistakes · 3,270 skipped | 0.747 | 62.92chosen-answer Brier 0.2172 | 2.4%answered 99% | 185 ms | 216 ms | $0.081$81.46 list$0.135$135estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 165 answers, 161 right · 4 wrong · AUROC 0.747 Logic 39.7% · Math 51.5% | — | 47.5%45.8%–49.2% | ABC | 1,561 / 3,288 | 161 right of 165 kept4 wrong · 3,123 skipped | 0.747 | 63.44chosen-answer Brier 0.2059 | 8.9%answered 95% | 46 ms | 52 ms | $0.033$33.45estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 194 answers, 188 right · 6 wrong · AUROC 0.743 Logic 40.4% · Math 50.4% | — | 47.0%45.3%–48.7% | ABC | 1,545 / 3,288 | 188 right of 194 kept6 wrong · 3,094 skipped | 0.743 | 62.33chosen-answer Brier 0.2233 | 7.8%answered 96% | 30 ms | 42 ms | $0.025$25.03estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.707 Logic 38.9% · Math 49.8% | — | 46.1%44.4%–47.8% | BCD | 1,515 / 3,288 | No answers at this threshold | 0.707 | 62.13chosen-answer Brier 0.2181 | 6.9%answered 96% | 103 ms | 131 ms | $0.013$12.90 list$0.075$75.10estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 53 answers, 52 right · 1 wrong · AUROC 0.737 Logic 39.7% · Math 46.9% | — | 44.4%42.7%–46.1% | CD | 1,461 / 3,288 | 52 right of 53 kept1 wrong · 3,235 skipped | 0.737 | 61.44chosen-answer Brier 0.2155 | 17.1%answered 90% | 55 ms | 72 ms | $0.043$42.91estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.729 Logic 39.3% · Math 44.3% | — | 42.6%40.9%–44.3% | D | 1,401 / 3,288 | No answers at this threshold | 0.729 | 60.88chosen-answer Brier 0.2086 | 7.3%answered 96% | 38 ms | 44 ms | $0.029$29.34 list$0.029$28.61estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.659 Logic 32.1% · Math 39.2% | — | 36.7%35.1%–38.4% | E | 1,208 / 3,288 | No answers at this threshold | 0.659 | 57.83chosen-answer Brier 0.2108 | 2.0%answered 99% | 117 ms | 146 ms | $0.086$85.83estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.675 Logic 30.8% · Math 38.7% | — | 35.9%34.3%–37.6% | E | 1,182 / 3,288 | 4 right of 4 kept0 mistakes · 3,284 skipped | 0.675 | 57.77chosen-answer Brier 0.2041 | 16.4%answered 89% | 36 ms | 42 ms | $0.002$2.36estimate · DGX Spark |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 30 answers, 29 right · 1 wrong · AUROC 0.637 Logic 32.3% · Math 36.8% | — | 35.2%33.6%–36.9% | E | 1,158 / 3,288 | 29 right of 30 kept1 wrong · 3,258 skipped | 0.637 | 55.25chosen-answer Brier 0.2472 | 23.1%answered 84% | 28 ms | 37 ms | $0.002$1.94estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.648 Logic 30.7% · Math 37.1% | — | 34.9%33.3%–36.5% | EF | 1,147 / 3,288 | 7 right of 7 kept0 mistakes · 3,281 skipped | 0.648 | 56.81chosen-answer Brier 0.2127 | 82.5%answered 30% | 17 ms | 21 ms | $0.014$14.30estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 16 answers, 16 right · 0 wrong · AUROC 0.626 Logic 31.7% · Math 35.3% | — | 34.1%32.5%–35.7% | EF | 1,120 / 3,288 | 16 right of 16 kept0 mistakes · 3,272 skipped | 0.626 | 56.04chosen-answer Brier 0.2199 | 6.8%answered 95% | 51 ms | 63 ms | $0.003$3.40estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.602 Logic 29.2% · Math 32.1% | — | 31.1%29.5%–32.7% | FG | 1,022 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.602 | 53.32chosen-answer Brier 0.2444 | 22.7%answered 83% | 20 ms | 25 ms | $0.001$1.38estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.605 Logic 24.7% · Math 29.7% | — | 28.0%26.5%–29.5% | GH | 920 / 3,288 | No answers at this threshold | 0.605 | 54.00chosen-answer Brier 0.1999 | 14.9%answered 89% | 24 ms | 29 ms | $0.002$1.62estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.576 Logic 27.0% · Math 28.2% | — | 27.8%26.3%–29.4% | GH | 915 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.576 | 53.31chosen-answer Brier 0.2120 | 2.0%answered 99% | 15 ms | 17 ms | $0.001$1.11estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.590 Logic 26.7% · Math 28.2% | — | 27.7%26.2%–29.2% | GH | 910 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.590 | 53.55chosen-answer Brier 0.2057 | 0.6%answered 100% | 26 ms | 35 ms | $0.002$1.80estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.573 Logic 22.4% · Math 28.7% | — | 26.5%25.0%–28.1% | HI | 872 / 3,288 | No answers at this threshold | 0.573 | 53.20chosen-answer Brier 0.2012 | 5.0%answered 96% | 35 ms | 42 ms | $0.027$26.72estimate · RTX PRO 6000 Blackwell |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.511 Logic 23.6% · Math 22.7% | — | 23.0%21.6%–24.4% | IJ | 755 / 3,288 | No answers at this threshold | 0.511 | 52.00chosen-answer Brier 0.1896 | 15.6%answered 87% | 12 ms | 23 ms | $0.001$1.01estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.520 Logic 23.2% · Math 20.6% | — | 21.5%20.1%–22.9% | J | 706 / 3,288 | No answers at this threshold | 0.520 | 51.39chosen-answer Brier 0.1869 | 73.9%answered 32% | 11 ms | 14 ms | $0.001$0.83estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
StartLux scar-ai PelaAI Fastino Liquid AI OmniJev Other Metask Lab Strands Agents quincyqiang Perplexity victorbennett Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 52.4%3,288 | 70.2%53.6% kept (1,762) | 81.4%32.8% kept (1,078) | 94.7%16.5% kept (543) | 100.0%3.6% kept (117) |
| vera-medium-omni Open · 4.2B | — | 52.5%3,288 | 74.9%46.3% kept (1,523) | 88.0%23.3% kept (766) | 99.1%9.8% kept (322) | 100.0%1.2% kept (39) |
| KnowLine-2B-Gen1 Open · 2.0B | — | 34.1%3,288 | 51.8%24.8% kept (816) | 67.8%8.3% kept (273) | 88.1%2.6% kept (84) | 100.0%0.5% kept (16) |
| GLiDE 0.8B Open · 0.8B | — | 31.1%3,288 | 39.8%37.4% kept (1,230) | 49.1%14.3% kept (470) | 68.9%3.1% kept (103) | 100.0%0.4% kept (13) |
| d1-2B Open · 1.9B | — | 34.9%3,288 | 56.6%21.7% kept (712) | 74.2%6.7% kept (221) | 88.2%2.1% kept (68) | 100.0%0.2% kept (7) |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 35.9%3,288 | 66.0%18.4% kept (606) | 81.0%7.1% kept (232) | 94.4%2.2% kept (72) | 100.0%0.1% kept (4) |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 27.7%3,288 | 43.1%13.7% kept (450) | 57.9%3.3% kept (107) | 88.9%0.5% kept (18) | 100.0%0.1% kept (2) |
| OneJev1.6-0.8B Open · 0.8B | — | 27.8%3,288 | 39.9%15.5% kept (509) | 50.0%4.0% kept (130) | 71.4%0.6% kept (21) | 100.0%0.1% kept (2) |
| GLiClass v4 Edge Open · 0.4B | — | 23.0%3,288 | 31.9%5.5% kept (182) | 33.3%1.2% kept (39) | 57.1%0.2% kept (7) | none kept |
| GLiDE 0.4B Open · 0.4B | — | 21.5%3,288 | 26.9%5.5% kept (182) | 36.6%1.2% kept (41) | 100.0%0.1% kept (3) | none kept |
| Metask-Jev-4B Open · 4.5B | — | 36.7%3,288 | 64.0%17.2% kept (567) | 78.6%8.1% kept (266) | 89.1%1.7% kept (55) | none kept |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 28.0%3,288 | 46.1%10.1% kept (332) | 62.7%3.6% kept (118) | 91.3%0.7% kept (23) | none kept |
| Strands Decider 2B Open · 2.0B | — | 26.5%3,288 | 44.4%9.8% kept (322) | 65.1%1.3% kept (43) | 100.0%0.1% kept (3) | none kept |
| Chinese Decision Pointer 4B Open · 4.0B | — | 47.0%3,288 | 59.2%60.7% kept (1,996) | 72.9%34.5% kept (1,133) | 85.5%17.6% kept (578) | 96.9%5.9% kept (194) |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 51.1%3,288 | 68.4%55.4% kept (1,820) | 79.9%36.8% kept (1,211) | 89.0%20.5% kept (674) | 97.4%5.8% kept (190) |
| Verdin-4B-v1.1 Open · 4.0B | — | 47.5%3,288 | 66.3%47.5% kept (1,563) | 79.5%26.8% kept (881) | 90.5%13.8% kept (454) | 97.6%5.0% kept (165) |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 52.1%3,288 | 66.8%62.0% kept (2,039) | 77.7%41.7% kept (1,371) | 90.0%22.1% kept (728) | 99.2%4.0% kept (131) |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 51.3%3,288 | 63.8%62.0% kept (2,038) | 76.6%37.0% kept (1,215) | 91.5%17.8% kept (585) | 98.6%2.2% kept (72) |
| GLiDE 2B Open · 1.9B | — | 35.2%3,288 | 45.7%44.7% kept (1,469) | 55.7%19.5% kept (641) | 75.7%5.5% kept (181) | 96.7%0.9% kept (30) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 117 / 00.763 | 1 / 00.633 | 2 / 00.672 | 18 / 00.853 | 11 / 00.818 | 1 / 00.707 | 0 / 00.524 | 2 / 00.676 | 2 / 00.743 | 7 / 00.748 | 18 / 00.814 | 6 / 00.808 | 11 / 00.806 | 6 / 00.744 | 12 / 00.768 | 12 / 00.778 | 8 / 00.775 |
| vera-medium-omni Open · 4.2B | — | 39 / 00.777 | 0 / 00.659 | 0 / 00.710 | 6 / 00.876 | 2 / 00.782 | 3 / 00.704 | 0 / 00.636 | 1 / 00.695 | 1 / 00.836 | 1 / 00.706 | 13 / 00.860 | 0 / 00.848 | 2 / 00.837 | 1 / 00.797 | 4 / 00.770 | 1 / 00.809 | 4 / 00.784 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 16 / 00.626 | 0 / 00.501 | 0 / 00.571 | 8 / 00.756 | 0 / 00.570 | 0 / 00.578 | 0 / 00.533 | 0 / 00.508 | 0 / 00.540 | 0 / 00.655 | 0 / 00.667 | 1 / 00.779 | 4 / 00.736 | 1 / 00.699 | 1 / 00.644 | 0 / 00.656 | 1 / 00.587 |
| GLiDE 0.8B Open · 0.8B | — | 13 / 00.602 | 0 / 00.557 | 0 / 00.608 | 9 / 00.700 | 1 / 00.538 | 0 / 00.498 | 0 / 00.467 | 0 / 00.610 | 0 / 00.579 | 0 / 00.525 | 0 / 00.710 | 1 / 00.604 | 0 / 00.601 | 0 / 00.661 | 1 / 00.673 | 0 / 00.564 | 1 / 00.616 |
| d1-2B Open · 1.9B | — | 7 / 00.648 | 0 / 00.549 | 0 / 00.606 | 3 / 00.787 | 0 / 00.634 | 0 / 00.629 | 0 / 00.318 | 0 / 00.572 | 1 / 00.582 | 0 / 00.589 | 0 / 00.721 | 1 / 00.778 | 1 / 00.625 | 0 / 00.718 | 1 / 00.709 | 0 / 00.657 | 0 / 00.677 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 4 / 00.675 | 0 / 00.539 | 0 / 00.652 | 2 / 00.767 | 0 / 00.661 | 0 / 00.601 | 0 / 00.333 | 0 / 00.560 | 0 / 00.665 | 0 / 00.655 | 0 / 00.788 | 0 / 00.813 | 1 / 00.755 | 0 / 00.703 | 0 / 00.614 | 0 / 00.658 | 1 / 00.679 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 2 / 00.590 | 0 / 00.520 | 0 / 00.531 | 2 / 00.746 | 0 / 00.486 | 0 / 00.603 | 0 / 00.667 | 0 / 00.628 | 0 / 00.573 | 0 / 00.490 | 0 / 00.683 | 0 / 00.792 | 0 / 00.660 | 0 / 00.653 | 0 / 00.603 | 0 / 00.557 | 0 / 00.565 |
| OneJev1.6-0.8B Open · 0.8B | — | 2 / 00.576 | 0 / 00.508 | 0 / 00.538 | 2 / 00.608 | 0 / 00.548 | 0 / 00.601 | 0 / 00.533 | 0 / 00.519 | 0 / 00.678 | 0 / 00.474 | 0 / 00.543 | 0 / 00.726 | 0 / 00.690 | 0 / 00.619 | 0 / 00.605 | 0 / 00.609 | 0 / 00.559 |
| GLiClass v4 Edge Open · 0.4B | — | No answers at this threshold0.511 | 0 / 00.516 | 0 / 00.547 | 0 / 00.584 | 0 / 00.524 | 0 / 00.554 | 0 / 00.550 | 0 / 00.476 | 0 / 00.474 | 0 / 00.505 | 0 / 00.412 | 0 / 00.522 | 0 / 00.412 | 0 / 00.460 | 0 / 00.566 | 0 / 00.543 | 0 / 00.518 |
| GLiDE 0.4B Open · 0.4B | — | No answers at this threshold0.520 | 0 / 00.531 | 0 / 00.545 | 0 / 00.639 | 0 / 00.496 | 0 / 00.528 | 0 / 00.875 | 0 / 00.509 | 0 / 00.606 | 0 / 00.458 | 0 / 00.370 | 0 / 00.632 | 0 / 00.450 | 0 / 00.478 | 0 / 00.493 | 0 / 00.498 | 0 / 00.515 |
| Metask-Jev-4B Open · 4.5B | — | No answers at this threshold0.659 | 0 / 00.591 | 0 / 00.604 | 0 / 00.830 | 0 / 00.614 | 0 / 00.544 | 0 / 00.933 | 0 / 00.617 | 0 / 00.737 | 0 / 00.613 | 0 / 00.633 | 0 / 00.619 | 0 / 00.666 | 0 / 00.727 | 0 / 00.660 | 0 / 00.624 | 0 / 00.664 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | No answers at this threshold0.605 | 0 / 00.580 | 0 / 00.615 | 0 / 00.694 | 0 / 00.611 | 0 / 00.460 | 0 / 00.667 | 0 / 00.546 | 0 / 00.440 | 0 / 00.612 | 0 / 00.567 | 0 / 00.693 | 0 / 00.672 | 0 / 00.676 | 0 / 00.551 | 0 / 00.503 | 0 / 00.580 |
| Strands Decider 2B Open · 2.0B | — | No answers at this threshold0.573 | 0 / 00.544 | 0 / 00.562 | 0 / 00.634 | 0 / 00.510 | 0 / 00.458 | 0 / 00.615 | 0 / 00.534 | 0 / 00.589 | 0 / 00.583 | 0 / 00.635 | 0 / 00.615 | 0 / 00.574 | 0 / 00.632 | 0 / 00.556 | 0 / 00.542 | 0 / 00.627 |
| Chinese Decision Pointer 4B Open · 4.0B | — | 188 / 60.743 | 1 / 00.644 | 8 / 00.699 | 30 / 00.866 | 14 / 20.793 | 5 / 00.668 | 0 / 00.424 | 5 / 00.719 | 2 / 00.797 | 5 / 20.655 | 16 / 00.735 | 8 / 10.772 | 19 / 00.732 | 7 / 00.768 | 16 / 00.771 | 24 / 10.729 | 28 / 00.726 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 185 / 50.769 | 3 / 00.642 | 1 / 00.693 | 31 / 00.879 | 7 / 10.781 | 4 / 00.688 | 0 / 00.300 | 2 / 00.694 | 3 / 00.780 | 9 / 00.732 | 26 / 00.884 | 10 / 10.807 | 21 / 00.854 | 11 / 10.773 | 15 / 10.730 | 19 / 00.806 | 23 / 10.753 |
| Verdin-4B-v1.1 Open · 4.0B | — | 161 / 40.747 | 2 / 00.648 | 5 / 00.672 | 28 / 00.884 | 14 / 10.731 | 5 / 00.719 | 0 / 00.548 | 4 / 00.677 | 6 / 00.743 | 8 / 10.763 | 16 / 00.824 | 8 / 00.774 | 14 / 00.799 | 9 / 20.772 | 17 / 00.761 | 12 / 00.772 | 13 / 00.696 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 130 / 10.782 | 4 / 00.663 | 1 / 00.731 | 12 / 00.860 | 6 / 00.765 | 5 / 00.695 | 0 / 00.450 | 2 / 00.701 | 3 / 00.836 | 9 / 00.788 | 23 / 00.913 | 7 / 00.830 | 12 / 00.797 | 7 / 00.792 | 11 / 00.764 | 15 / 00.792 | 13 / 10.765 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 71 / 10.752 | 2 / 00.634 | 3 / 10.641 | 12 / 00.881 | 3 / 00.791 | 2 / 00.611 | 0 / 00.500 | 1 / 00.688 | 1 / 00.814 | 4 / 00.699 | 11 / 00.845 | 1 / 00.842 | 7 / 00.813 | 1 / 00.742 | 7 / 00.743 | 9 / 00.803 | 7 / 00.747 |
| GLiDE 2B Open · 1.9B | — | 29 / 10.637 | 1 / 00.525 | 1 / 00.637 | 12 / 00.787 | 3 / 10.625 | 0 / 00.569 | 0 / 00.519 | 1 / 00.554 | 0 / 00.590 | 1 / 00.676 | 1 / 00.646 | 1 / 00.728 | 3 / 00.663 | 2 / 00.596 | 1 / 00.617 | 1 / 00.616 | 1 / 00.639 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.777 Logic 44.1% · Math 56.8% | — | 52.5%50.8%–54.2% | a | 1,725 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.777 | 66.69chosen-answer Brier 0.1909 | 7.9%answered 96% | 31 ms | 36 ms | $0.024$24.19estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Kept at ≥99%: 117 answers, 117 right · 0 wrong · AUROC 0.763 Logic 45.9% · Math 55.8% | — | 52.4%50.7%–54.1% | a | 1,724 / 3,288 | 117 right of 117 kept0 mistakes · 3,171 skipped | 0.763 | 66.33chosen-answer Brier 0.1978 | 14.4%answered 93% | 68 ms | 80 ms | $0.004$4.28estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 131 answers, 130 right · 1 wrong · AUROC 0.782 Logic 44.7% · Math 55.9% | — | 52.1%50.4%–53.8% | a | 1,713 / 3,288 | 130 right of 131 kept1 wrong · 3,157 skipped | 0.782 | 65.93chosen-answer Brier 0.2024 | 15.0%answered 92% | 29 ms | 35 ms | $0.024$23.77estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Kept at ≥99%: 72 answers, 71 right · 1 wrong · AUROC 0.752 Logic 42.3% · Math 56.0% | — | 51.3%49.6%–53.0% | a | 1,687 / 3,288 | 71 right of 72 kept1 wrong · 3,216 skipped | 0.752 | 65.07chosen-answer Brier 0.2116 | 4.4%answered 98% | 76 ms | 104 ms | $0.005$5.02estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Kept at ≥99%: 190 answers, 185 right · 5 wrong · AUROC 0.769 Logic 43.1% · Math 55.2% | — | 51.1%49.4%–52.8% | a | 1,680 / 3,288 | 185 right of 190 kept5 wrong · 3,098 skipped | 0.769 | 65.47chosen-answer Brier 0.2016 | 18.9%answered 90% | 41 ms | 51 ms | $0.003$2.72 list$0.032$31.55estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 165 answers, 161 right · 4 wrong · AUROC 0.747 Logic 39.7% · Math 51.5% | — | 47.5%45.8%–49.2% | b | 1,561 / 3,288 | 161 right of 165 kept4 wrong · 3,123 skipped | 0.747 | 63.44chosen-answer Brier 0.2059 | 8.9%answered 95% | 46 ms | 52 ms | $0.033$33.45estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Kept at ≥99%: 194 answers, 188 right · 6 wrong · AUROC 0.743 Logic 40.4% · Math 50.4% | — | 47.0%45.3%–48.7% | b | 1,545 / 3,288 | 188 right of 194 kept6 wrong · 3,094 skipped | 0.743 | 62.33chosen-answer Brier 0.2233 | 7.8%answered 96% | 30 ms | 42 ms | $0.025$25.03estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Kept at ≥99%: No answers at this threshold · AUROC 0.659 Logic 32.1% · Math 39.2% | — | 36.7%35.1%–38.4% | c | 1,208 / 3,288 | No answers at this threshold | 0.659 | 57.83chosen-answer Brier 0.2108 | 2.0%answered 99% | 117 ms | 146 ms | $0.086$85.83estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Kept at ≥99%: 4 answers, 4 right · 0 wrong · AUROC 0.675 Logic 30.8% · Math 38.7% | — | 35.9%34.3%–37.6% | c | 1,182 / 3,288 | 4 right of 4 kept0 mistakes · 3,284 skipped | 0.675 | 57.77chosen-answer Brier 0.2041 | 16.4%answered 89% | 36 ms | 42 ms | $0.002$2.36estimate · DGX Spark |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Kept at ≥99%: 30 answers, 29 right · 1 wrong · AUROC 0.637 Logic 32.3% · Math 36.8% | — | 35.2%33.6%–36.9% | c | 1,158 / 3,288 | 29 right of 30 kept1 wrong · 3,258 skipped | 0.637 | 55.25chosen-answer Brier 0.2472 | 23.1%answered 84% | 28 ms | 37 ms | $0.002$1.94estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Kept at ≥99%: 7 answers, 7 right · 0 wrong · AUROC 0.648 Logic 30.7% · Math 37.1% | — | 34.9%33.3%–36.5% | c | 1,147 / 3,288 | 7 right of 7 kept0 mistakes · 3,281 skipped | 0.648 | 56.81chosen-answer Brier 0.2127 | 82.5%answered 30% | 17 ms | 21 ms | $0.014$14.30estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Kept at ≥99%: 16 answers, 16 right · 0 wrong · AUROC 0.626 Logic 31.7% · Math 35.3% | — | 34.1%32.5%–35.7% | cd | 1,120 / 3,288 | 16 right of 16 kept0 mistakes · 3,272 skipped | 0.626 | 56.04chosen-answer Brier 0.2199 | 6.8%answered 95% | 51 ms | 63 ms | $0.003$3.40estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.602 Logic 29.2% · Math 32.1% | — | 31.1%29.5%–32.7% | d | 1,022 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.602 | 53.32chosen-answer Brier 0.2444 | 22.7%answered 83% | 20 ms | 25 ms | $0.001$1.38estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: No answers at this threshold · AUROC 0.605 Logic 24.7% · Math 29.7% | — | 28.0%26.5%–29.5% | e | 920 / 3,288 | No answers at this threshold | 0.605 | 54.00chosen-answer Brier 0.1999 | 14.9%answered 89% | 24 ms | 29 ms | $0.002$1.62estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.576 Logic 27.0% · Math 28.2% | — | 27.8%26.3%–29.4% | e | 915 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.576 | 53.31chosen-answer Brier 0.2120 | 2.0%answered 99% | 15 ms | 17 ms | $0.001$1.11estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Kept at ≥99%: 2 answers, 2 right · 0 wrong · AUROC 0.590 Logic 26.7% · Math 28.2% | — | 27.7%26.2%–29.2% | e | 910 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.590 | 53.55chosen-answer Brier 0.2057 | 0.6%answered 100% | 26 ms | 35 ms | $0.002$1.80estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Kept at ≥99%: No answers at this threshold · AUROC 0.573 Logic 22.4% · Math 28.7% | — | 26.5%25.0%–28.1% | e | 872 / 3,288 | No answers at this threshold | 0.573 | 53.20chosen-answer Brier 0.2012 | 5.0%answered 96% | 35 ms | 42 ms | $0.027$26.72estimate · RTX PRO 6000 Blackwell |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Kept at ≥99%: No answers at this threshold · AUROC 0.511 Logic 23.6% · Math 22.7% | — | 23.0%21.6%–24.4% | f | 755 / 3,288 | No answers at this threshold | 0.511 | 52.00chosen-answer Brier 0.1896 | 15.6%answered 87% | 12 ms | 23 ms | $0.001$1.01estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Kept at ≥99%: No answers at this threshold · AUROC 0.520 Logic 23.2% · Math 20.6% | — | 21.5%20.1%–22.9% | f | 706 / 3,288 | No answers at this threshold | 0.520 | 51.39chosen-answer Brier 0.1869 | 73.9%answered 32% | 11 ms | 14 ms | $0.001$0.83estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
PelaAI Fastino tattrongvu StartLux Cloudflare Blockbrain Labs Apertix AI Patrick-18 newsbang Base Labs Valen-Team EldanRing Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| KnowLine-9B-Gen1 Open · 9.0B | — | 54.5%3,288 | 75.0%49.1% kept (1,615) | 88.6%29.8% kept (981) | 98.0%16.4% kept (539) | 100.0%4.7% kept (153) |
| GLiDE 9B Open · 7.9B | — | 54.5%3,288 | 69.6%61.2% kept (2,011) | 85.4%35.9% kept (1,182) | 95.3%17.4% kept (571) | 100.0%3.2% kept (105) |
| Jeffan-9B-1211 Open · 9.0B | — | 55.8%3,288 | 76.0%53.9% kept (1,771) | 89.5%32.9% kept (1,082) | 98.4%15.2% kept (501) | 100.0%2.3% kept (74) |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 54.4%3,288 | 74.0%51.4% kept (1,689) | 85.0%31.3% kept (1,029) | 97.0%15.4% kept (505) | 100.0%1.9% kept (63) |
| KnowLine-12B-Gen1 Open · 12B | — | 60.9%3,288 | 81.1%55.3% kept (1,817) | 91.7%36.4% kept (1,197) | 99.1%16.2% kept (534) | 100.0%0.4% kept (13) |
| Clef-Flash Open · 9.0B | — | 46.1%3,288 | 63.4%43.4% kept (1,427) | 74.6%22.4% kept (735) | 89.5%7.0% kept (229) | none kept |
| Cygnet Open · 12B | — | 42.6%3,288 | 63.2%42.4% kept (1,394) | 79.1%16.0% kept (527) | 94.7%4.0% kept (132) | none kept |
| KnowLine-9B-Gen2 Open · 9.0B | — | 54.6%3,288 | 78.4%45.4% kept (1,492) | 91.4%25.4% kept (836) | 99.6%8.2% kept (269) | none kept |
| Cerno 9B v1.2 Open · 9.0B | — | 51.7%3,288 | 62.1%70.1% kept (2,306) | 73.5%47.7% kept (1,569) | 86.8%27.2% kept (895) | 96.1%8.5% kept (281) |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 54.2%3,288 | 66.2%69.4% kept (2,281) | 78.0%47.1% kept (1,549) | 91.9%26.2% kept (863) | 98.9%8.1% kept (266) |
| Adjudex 9B v5 Open · 9.0B | — | 56.7%3,288 | 73.2%60.4% kept (1,985) | 85.5%38.9% kept (1,279) | 96.2%22.3% kept (732) | 99.2%7.7% kept (253) |
| Elephant-1 Open · 9.0B | — | 56.2%3,288 | 73.2%59.5% kept (1,957) | 86.1%38.2% kept (1,256) | 95.6%22.0% kept (724) | 99.6%7.2% kept (236) |
| Valen-Decision-1.5-9B Open · 9.0B | — | 51.6%3,288 | 71.4%49.0% kept (1,611) | 84.7%27.3% kept (897) | 94.0%14.2% kept (466) | 98.8%5.1% kept (167) |
| Winnow-12B Open · 12B | — | 44.4%3,288 | 62.2%46.6% kept (1,531) | 73.1%31.1% kept (1,022) | 83.8%17.2% kept (567) | 98.1%1.6% kept (53) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-9B-Gen1 Open · 9.0B | — | 153 / 00.782 | 3 / 00.638 | 0 / 00.692 | 27 / 00.843 | 13 / 00.816 | 5 / 00.712 | 0 / 00.595 | 6 / 00.674 | 6 / 00.803 | 5 / 00.760 | 19 / 00.873 | 6 / 00.799 | 14 / 00.792 | 7 / 00.780 | 17 / 00.803 | 11 / 00.821 | 14 / 00.817 |
| GLiDE 9B Open · 7.9B | — | 105 / 00.786 | 3 / 00.637 | 4 / 00.729 | 25 / 00.911 | 7 / 00.800 | 2 / 00.737 | 0 / 00.731 | 3 / 00.700 | 1 / 00.716 | 4 / 00.752 | 15 / 00.805 | 3 / 00.774 | 10 / 00.873 | 4 / 00.853 | 7 / 00.776 | 8 / 00.851 | 9 / 00.765 |
| Jeffan-9B-1211 Open · 9.0B | — | 74 / 00.798 | 2 / 00.663 | 5 / 00.704 | 3 / 00.866 | 4 / 00.865 | 4 / 00.677 | 0 / 00.769 | 3 / 00.732 | 3 / 00.844 | 3 / 00.745 | 15 / 00.811 | 4 / 00.797 | 6 / 00.888 | 2 / 00.825 | 5 / 00.804 | 7 / 00.808 | 8 / 00.768 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 63 / 00.779 | 0 / 00.675 | 3 / 00.713 | 11 / 00.854 | 3 / 00.777 | 1 / 00.678 | 0 / 00.394 | 1 / 00.703 | 1 / 00.730 | 3 / 00.735 | 11 / 00.887 | 3 / 00.774 | 6 / 00.803 | 5 / 00.830 | 4 / 00.775 | 2 / 00.788 | 9 / 00.779 |
| KnowLine-12B-Gen1 Open · 12B | — | 13 / 00.810 | 1 / 00.701 | 1 / 00.711 | 4 / 00.937 | 2 / 00.838 | 0 / 00.736 | 0 / 00.814 | 0 / 00.752 | 0 / 00.840 | 0 / 00.833 | 5 / 00.915 | 0 / 00.800 | 0 / 00.824 | 0 / 00.916 | 0 / 00.775 | 0 / 00.857 | 0 / 00.794 |
| Clef-Flash Open · 9.0B | — | No answers at this threshold0.707 | 0 / 00.537 | 0 / 00.670 | 0 / 00.790 | 0 / 00.711 | 0 / 00.642 | 0 / 00.361 | 0 / 00.614 | 0 / 00.736 | 0 / 00.670 | 0 / 00.766 | 0 / 00.789 | 0 / 00.744 | 0 / 00.823 | 0 / 00.722 | 0 / 00.804 | 0 / 00.737 |
| Cygnet Open · 12B | — | No answers at this threshold0.729 | 0 / 00.604 | 0 / 00.676 | 0 / 00.848 | 0 / 00.730 | 0 / 00.617 | 0 / 00.600 | 0 / 00.690 | 0 / 00.639 | 0 / 00.726 | 0 / 00.800 | 0 / 00.788 | 0 / 00.796 | 0 / 00.697 | 0 / 00.680 | 0 / 00.768 | 0 / 00.761 |
| KnowLine-9B-Gen2 Open · 9.0B | — | No answers at this threshold0.786 | 0 / 00.672 | 0 / 00.663 | 0 / 00.859 | 0 / 00.788 | 0 / 00.711 | 0 / 00.712 | 0 / 00.671 | 0 / 00.801 | 0 / 00.819 | 0 / 00.834 | 0 / 00.841 | 0 / 00.847 | 0 / 00.824 | 0 / 00.774 | 0 / 00.806 | 0 / 00.817 |
| Cerno 9B v1.2 Open · 9.0B | — | 270 / 110.776 | 8 / 10.645 | 3 / 00.745 | 15 / 00.881 | 24 / 00.806 | 9 / 00.677 | 1 / 10.561 | 8 / 00.679 | 6 / 00.797 | 11 / 00.751 | 31 / 00.843 | 17 / 00.747 | 22 / 20.806 | 13 / 00.812 | 31 / 20.720 | 33 / 30.836 | 38 / 20.776 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 263 / 30.795 | 7 / 00.712 | 9 / 00.689 | 40 / 00.910 | 18 / 10.778 | 5 / 00.684 | 0 / 00.606 | 7 / 00.731 | 7 / 10.745 | 8 / 00.759 | 29 / 10.839 | 10 / 00.829 | 18 / 00.822 | 12 / 00.853 | 33 / 00.824 | 24 / 00.814 | 36 / 00.820 |
| Adjudex 9B v5 Open · 9.0B | — | 251 / 20.802 | 4 / 00.706 | 2 / 00.730 | 48 / 10.869 | 18 / 00.815 | 4 / 00.701 | 0 / 00.729 | 6 / 00.684 | 8 / 00.764 | 9 / 00.797 | 31 / 00.867 | 12 / 10.814 | 21 / 00.820 | 16 / 00.840 | 23 / 00.813 | 21 / 00.812 | 28 / 00.806 |
| Elephant-1 Open · 9.0B | — | 235 / 10.797 | 5 / 00.671 | 9 / 00.725 | 39 / 00.925 | 14 / 00.821 | 5 / 00.711 | 0 / 00.486 | 6 / 00.684 | 6 / 00.823 | 10 / 00.730 | 28 / 00.833 | 10 / 10.785 | 15 / 00.868 | 14 / 00.879 | 25 / 00.778 | 24 / 00.834 | 25 / 00.774 |
| Valen-Decision-1.5-9B Open · 9.0B | — | 165 / 20.765 | 5 / 10.598 | 5 / 10.709 | 25 / 00.785 | 16 / 00.837 | 3 / 00.663 | 0 / 00.667 | 7 / 00.778 | 4 / 00.849 | 5 / 00.753 | 17 / 00.876 | 9 / 00.765 | 12 / 00.777 | 7 / 00.834 | 13 / 00.782 | 16 / 00.795 | 21 / 00.776 |
| Winnow-12B Open · 12B | — | 52 / 10.737 | 1 / 00.608 | 6 / 00.670 | 25 / 00.835 | 5 / 00.742 | 1 / 00.638 | 0 / 00.686 | 0 / 00.670 | 1 / 00.642 | 0 / 10.743 | 1 / 00.787 | 1 / 00.742 | 5 / 00.731 | 3 / 00.785 | 1 / 00.746 | 1 / 00.758 | 1 / 00.774 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Kept at ≥99%: 13 answers, 13 right · 0 wrong · AUROC 0.810 Logic 50.6% · Math 66.3% | — | 60.9%59.2%–62.5% | a | 2,002 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.810 | 71.86chosen-answer Brier 0.1716 | 4.5%answered 98% | 51 ms | 76 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 253 answers, 251 right · 2 wrong · AUROC 0.802 Logic 46.4% · Math 62.1% | — | 56.7%55.0%–58.4% | b | 1,865 / 3,288 | 251 right of 253 kept2 wrong · 3,035 skipped | 0.802 | 69.27chosen-answer Brier 0.1818 | 7.0%answered 97% | 38 ms | 45 ms | $0.028$28.19estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Kept at ≥99%: 236 answers, 235 right · 1 wrong · AUROC 0.797 Logic 46.4% · Math 61.3% | — | 56.2%54.5%–57.9% | b | 1,849 / 3,288 | 235 right of 236 kept1 wrong · 3,052 skipped | 0.797 | 68.96chosen-answer Brier 0.1831 | 69.4%answered 60% | 80 ms | 94 ms | $0.010$9.79 list$0.058$58.27estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 74 answers, 74 right · 0 wrong · AUROC 0.798 Logic 48.2% · Math 59.8% | — | 55.8%54.1%–57.5% | b | 1,836 / 3,288 | 74 right of 74 kept0 mistakes · 3,214 skipped | 0.798 | 68.92chosen-answer Brier 0.1799 | 13.5%answered 94% | 39 ms | 53 ms | $0.032$32.40estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: No answers at this threshold · AUROC 0.786 Logic 43.9% · Math 60.1% | — | 54.6%52.9%–56.3% | b | 1,794 / 3,288 | No answers at this threshold | 0.786 | 67.99chosen-answer Brier 0.1858 | 10.4%answered 95% | 42 ms | 57 ms | $0.034$34.08estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Kept at ≥99%: 105 answers, 105 right · 0 wrong · AUROC 0.786 Logic 47.8% · Math 57.9% | — | 54.5%52.8%–56.2% | bc | 1,791 / 3,288 | 105 right of 105 kept0 mistakes · 3,183 skipped | 0.786 | 67.72chosen-answer Brier 0.1902 | 26.9%answered 86% | 177 ms | 222 ms | $0.011$11.45estimate · DGX Spark |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 153 answers, 153 right · 0 wrong · AUROC 0.782 Logic 44.3% · Math 59.8% | — | 54.5%52.8%–56.2% | b | 1,791 / 3,288 | 153 right of 153 kept0 mistakes · 3,135 skipped | 0.782 | 67.96chosen-answer Brier 0.1854 | 3.4%answered 98% | 38 ms | 55 ms | $0.030$29.87estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Kept at ≥99%: 63 answers, 63 right · 0 wrong · AUROC 0.779 Logic 46.6% · Math 58.4% | — | 54.4%52.7%–56.1% | bc | 1,788 / 3,288 | 63 right of 63 kept0 mistakes · 3,225 skipped | 0.779 | 67.73chosen-answer Brier 0.1893 | 9.6%answered 95% | 142 ms | 165 ms | $0.009$8.92estimate · DGX Spark |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Kept at ≥99%: 266 answers, 263 right · 3 wrong · AUROC 0.795 Logic 45.3% · Math 58.8% | — | 54.2%52.5%–55.9% | bc | 1,782 / 3,288 | 263 right of 266 kept3 wrong · 3,022 skipped | 0.795 | 66.98chosen-answer Brier 0.2023 | 6.1%answered 97% | 46 ms | 51 ms | $0.033$32.61estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Kept at ≥99%: 281 answers, 270 right · 11 wrong · AUROC 0.776 Logic 43.3% · Math 56.0% | — | 51.7%50.0%–53.4% | c | 1,699 / 3,288 | 270 right of 281 kept11 wrong · 3,007 skipped | 0.776 | 64.84chosen-answer Brier 0.2199 | 3.1%answered 99% | 163 ms | 240 ms | $0.011$10.82estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Kept at ≥99%: 167 answers, 165 right · 2 wrong · AUROC 0.765 Logic 44.4% · Math 55.3% | — | 51.6%49.9%–53.3% | c | 1,696 / 3,288 | 165 right of 167 kept2 wrong · 3,121 skipped | 0.765 | 65.99chosen-answer Brier 0.1960 | 7.1%answered 96% | 59 ms | 65 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Kept at ≥99%: No answers at this threshold · AUROC 0.707 Logic 38.9% · Math 49.8% | — | 46.1%44.4%–47.8% | d | 1,515 / 3,288 | No answers at this threshold | 0.707 | 62.13chosen-answer Brier 0.2181 | 6.9%answered 96% | 103 ms | 131 ms | $0.013$12.90 list$0.075$75.10estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Kept at ≥99%: 53 answers, 52 right · 1 wrong · AUROC 0.737 Logic 39.7% · Math 46.9% | — | 44.4%42.7%–46.1% | de | 1,461 / 3,288 | 52 right of 53 kept1 wrong · 3,235 skipped | 0.737 | 61.44chosen-answer Brier 0.2155 | 17.1%answered 90% | 55 ms | 72 ms | $0.043$42.91estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Kept at ≥99%: No answers at this threshold · AUROC 0.729 Logic 39.3% · Math 44.3% | — | 42.6%40.9%–44.3% | e | 1,401 / 3,288 | No answers at this threshold | 0.729 | 60.88chosen-answer Brier 0.2086 | 7.3%answered 96% | 38 ms | 44 ms | $0.029$29.34 list$0.029$28.61estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Perplexity Poindexter Labs Nace.AI Fastino Turion.AI vLLM Semantic Router OmniJev Cloudflare Zefan Cai MachinePulse PelaAI Base Labs Sthānika AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 66.4%3,288 | 78.1%75.1% kept (2,469) | 88.0%56.4% kept (1,853) | 95.7%38.0% kept (1,249) | 100.0%11.7% kept (385) |
| Copernicus-31B Open · 31B | — | 71.0%3,288 | 89.0%64.6% kept (2,123) | 96.0%48.3% kept (1,588) | 99.3%31.5% kept (1,036) | 100.0%11.7% kept (384) |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 53.1%3,288 | 79.7%42.5% kept (1,397) | 90.7%27.2% kept (893) | 97.0%16.2% kept (532) | 100.0%6.3% kept (207) |
| GLiDE v1.2 Open · 27B | — | 64.8%3,288 | 78.2%70.3% kept (2,312) | 91.8%48.6% kept (1,597) | 98.8%28.0% kept (922) | 100.0%5.4% kept (176) |
| Turion Jev 31B Open · 31B | — | 63.3%3,288 | 84.0%58.2% kept (1,913) | 92.7%41.7% kept (1,372) | 99.4%21.0% kept (692) | 100.0%1.5% kept (50) |
| d3-ultra Open · 26B | — | 65.7%3,288 | 73.5%81.9% kept (2,694) | 82.7%62.6% kept (2,057) | 96.3%37.4% kept (1,231) | 100.0%1.2% kept (39) |
| OneJev-27B Open · 27B | — | 49.3%3,288 | 80.3%32.3% kept (1,062) | 92.2%18.7% kept (616) | 98.6%8.5% kept (281) | 100.0%0.9% kept (30) |
| Clef Open · 27B | — | 47.6%3,288 | 61.4%59.6% kept (1,959) | 75.3%33.0% kept (1,086) | 89.6%14.7% kept (483) | 100.0%0.5% kept (18) |
| Open-Jev-27B-v1.1 Open · 27B | — | 50.5%3,288 | 95.7%8.6% kept (282) | 100.0%0.9% kept (29) | none kept | none kept |
| Karpo Decision 31B Open · 31B | — | 74.4%3,288 | 85.1%79.4% kept (2,610) | 93.0%63.7% kept (2,095) | 97.7%48.1% kept (1,580) | 99.7%30.7% kept (1,009) |
| Drex Max Open · 27B | — | 64.4%3,288 | 72.4%81.7% kept (2,686) | 80.8%64.5% kept (2,120) | 91.5%47.3% kept (1,556) | 98.0%26.9% kept (884) |
| KnowLine-31B-Gen1.5 Open · 31B | — | 71.1%3,288 | 86.5%69.3% kept (2,279) | 93.9%53.0% kept (1,744) | 98.5%35.2% kept (1,156) | 99.8%16.8% kept (554) |
| KnowLine-31B-Gen2 Open · 31B | — | 73.0%3,288 | 87.1%72.9% kept (2,396) | 93.9%56.8% kept (1,868) | 99.0%38.8% kept (1,275) | 99.8%15.9% kept (524) |
| Mammoth-1 Open · 27B | — | 67.3%3,288 | 82.3%71.0% kept (2,334) | 92.4%53.0% kept (1,744) | 97.9%37.3% kept (1,228) | 99.6%15.8% kept (520) |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 64.1%3,288 | 81.5%64.6% kept (2,125) | 92.6%46.9% kept (1,541) | 97.8%31.4% kept (1,031) | 99.5%12.5% kept (411) |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 63.6%3,288 | 80.6%65.2% kept (2,145) | 91.8%47.0% kept (1,545) | 97.4%31.3% kept (1,028) | 99.8%12.3% kept (405) |
| Sieve 31B v3 Open · 31B | — | 70.5%3,288 | 86.8%67.7% kept (2,227) | 94.3%51.4% kept (1,691) | 98.7%34.4% kept (1,131) | 99.3%9.0% kept (296) |
| GLiDE 27B Open · 27B | — | 67.3%3,288 | 80.6%72.4% kept (2,381) | 90.6%53.0% kept (1,742) | 98.5%32.5% kept (1,068) | 99.3%8.3% kept (274) |
| d3 Open · 26B | — | 65.0%3,288 | 82.6%65.4% kept (2,150) | 94.1%46.2% kept (1,518) | 98.6%28.2% kept (927) | 99.5%6.6% kept (217) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 385 / 00.844 | 22 / 00.752 | 5 / 00.764 | 35 / 00.878 | 26 / 00.828 | 8 / 00.776 | 0 / 00.577 | 8 / 00.785 | 15 / 00.845 | 17 / 00.790 | 36 / 00.935 | 27 / 00.839 | 25 / 00.900 | 22 / 00.933 | 35 / 00.845 | 49 / 00.871 | 55 / 00.834 |
| Copernicus-31B Open · 31B | — | 384 / 00.873 | 15 / 00.746 | 12 / 00.775 | 39 / 00.935 | 27 / 00.901 | 13 / 00.865 | 1 / 00.800 | 9 / 00.881 | 17 / 00.892 | 20 / 00.881 | 42 / 00.980 | 12 / 00.884 | 25 / 00.876 | 19 / 00.937 | 29 / 00.882 | 48 / 00.914 | 56 / 00.875 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 207 / 00.789 | 3 / 00.652 | 0 / 00.690 | 30 / 00.815 | 8 / 00.819 | 6 / 00.817 | 0 / 00.533 | 0 / 00.740 | 5 / 00.798 | 13 / 00.841 | 20 / 00.905 | 9 / 00.768 | 16 / 00.837 | 14 / 00.766 | 22 / 00.762 | 35 / 00.836 | 26 / 00.803 |
| GLiDE v1.2 Open · 27B | — | 176 / 00.843 | 3 / 00.732 | 6 / 00.739 | 36 / 00.906 | 12 / 00.857 | 3 / 00.740 | 0 / 00.528 | 5 / 00.777 | 4 / 00.834 | 5 / 00.838 | 20 / 00.963 | 14 / 00.865 | 22 / 00.912 | 7 / 00.948 | 15 / 00.873 | 14 / 00.849 | 10 / 00.877 |
| Turion Jev 31B Open · 31B | — | 50 / 00.844 | 1 / 00.702 | 0 / 00.756 | 6 / 00.875 | 5 / 00.868 | 1 / 00.770 | 0 / 00.757 | 1 / 00.809 | 0 / 00.904 | 3 / 00.817 | 3 / 00.926 | 1 / 00.854 | 6 / 00.823 | 1 / 00.940 | 8 / 00.853 | 5 / 00.866 | 9 / 00.852 |
| d3-ultra Open · 26B | — | 39 / 00.835 | 9 / 00.707 | 10 / 00.761 | 9 / 00.853 | 1 / 00.842 | 1 / 00.778 | 0 / 00.764 | 0 / 00.820 | 0 / 00.731 | 0 / 00.827 | 3 / 00.942 | 0 / 00.819 | 1 / 00.901 | 2 / 00.848 | 1 / 00.871 | 1 / 00.850 | 1 / 00.874 |
| OneJev-27B Open · 27B | — | 30 / 00.770 | 0 / 00.633 | 0 / 00.712 | 4 / 00.845 | 2 / 00.768 | 0 / 00.645 | 0 / 00.806 | 1 / 00.750 | 0 / 00.767 | 1 / 00.778 | 2 / 00.884 | 1 / 00.758 | 1 / 00.811 | 3 / 00.787 | 4 / 00.757 | 6 / 00.785 | 5 / 00.789 |
| Clef Open · 27B | — | 18 / 00.747 | 2 / 00.607 | 0 / 00.704 | 4 / 00.846 | 2 / 00.751 | 1 / 00.708 | 0 / 00.875 | 2 / 00.723 | 0 / 00.720 | 0 / 00.798 | 3 / 00.778 | 0 / 00.793 | 1 / 00.744 | 0 / 00.878 | 2 / 00.748 | 0 / 00.802 | 1 / 00.774 |
| Open-Jev-27B-v1.1 Open · 27B | — | No answers at this threshold0.759 | 0 / 00.623 | 0 / 00.716 | 0 / 00.861 | 0 / 00.816 | 0 / 00.607 | 0 / 00.625 | 0 / 00.674 | 0 / 00.731 | 0 / 00.751 | 0 / 00.762 | 0 / 00.745 | 0 / 00.847 | 0 / 00.818 | 0 / 00.776 | 0 / 00.727 | 0 / 00.749 |
| Karpo Decision 31B Open · 31B | — | 1006 / 30.882 | 58 / 00.805 | 33 / 20.758 | 78 / 10.877 | 85 / 00.899 | 34 / 00.801 | 3 / 00.833 | 34 / 00.880 | 38 / 00.933 | 50 / 00.890 | 83 / 00.952 | 45 / 00.879 | 66 / 00.915 | 53 / 00.950 | 91 / 00.902 | 114 / 00.893 | 141 / 00.904 |
| Drex Max Open · 27B | — | 866 / 180.841 | 34 / 00.729 | 39 / 50.758 | 74 / 30.838 | 67 / 10.834 | 25 / 00.803 | 3 / 00.750 | 32 / 00.799 | 28 / 00.867 | 36 / 40.794 | 63 / 00.923 | 42 / 20.877 | 75 / 00.894 | 44 / 00.908 | 83 / 00.876 | 96 / 20.855 | 125 / 10.847 |
| KnowLine-31B-Gen1.5 Open · 31B | — | 553 / 10.870 | 25 / 00.765 | 14 / 00.764 | 50 / 00.904 | 38 / 00.874 | 15 / 00.850 | 1 / 00.750 | 14 / 00.881 | 22 / 00.903 | 32 / 00.900 | 60 / 00.956 | 18 / 00.912 | 35 / 00.872 | 33 / 00.904 | 55 / 10.885 | 64 / 00.914 | 77 / 00.888 |
| KnowLine-31B-Gen2 Open · 31B | — | 523 / 10.882 | 26 / 00.792 | 5 / 00.785 | 23 / 00.907 | 38 / 00.916 | 15 / 00.862 | 2 / 00.742 | 14 / 00.847 | 24 / 00.902 | 37 / 00.921 | 70 / 00.956 | 12 / 00.871 | 28 / 00.882 | 28 / 00.916 | 56 / 10.863 | 68 / 00.908 | 77 / 00.886 |
| Mammoth-1 Open · 27B | — | 518 / 20.868 | 21 / 00.703 | 13 / 00.797 | 72 / 20.905 | 27 / 00.885 | 9 / 00.766 | 1 / 00.788 | 11 / 00.821 | 14 / 00.876 | 30 / 00.897 | 41 / 00.909 | 17 / 00.893 | 47 / 00.930 | 31 / 00.904 | 53 / 00.900 | 59 / 00.896 | 72 / 00.903 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 409 / 20.849 | 9 / 00.730 | 12 / 00.783 | 57 / 20.791 | 22 / 00.861 | 6 / 00.818 | 1 / 00.806 | 8 / 00.772 | 10 / 00.823 | 16 / 00.819 | 34 / 00.961 | 34 / 00.901 | 43 / 00.903 | 33 / 00.932 | 32 / 00.840 | 47 / 00.875 | 45 / 00.863 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 404 / 10.854 | 8 / 00.728 | 12 / 00.787 | 58 / 10.860 | 23 / 00.850 | 9 / 00.842 | 1 / 00.571 | 6 / 00.827 | 14 / 00.799 | 18 / 00.854 | 28 / 00.927 | 36 / 00.869 | 43 / 00.887 | 31 / 00.946 | 29 / 00.851 | 41 / 00.885 | 47 / 00.884 |
| Sieve 31B v3 Open · 31B | — | 294 / 20.865 | 9 / 00.764 | 16 / 20.745 | 54 / 00.861 | 24 / 00.871 | 6 / 00.848 | 0 / 00.903 | 7 / 00.856 | 8 / 00.898 | 10 / 00.863 | 26 / 00.956 | 15 / 00.918 | 22 / 00.907 | 14 / 00.939 | 24 / 00.895 | 31 / 00.879 | 28 / 00.848 |
| GLiDE 27B Open · 27B | — | 272 / 20.852 | 6 / 00.754 | 17 / 00.800 | 51 / 10.888 | 17 / 10.858 | 6 / 00.787 | 0 / 00.569 | 8 / 00.761 | 5 / 00.846 | 7 / 00.870 | 22 / 00.948 | 20 / 00.938 | 36 / 00.873 | 18 / 00.905 | 21 / 00.861 | 20 / 00.848 | 18 / 00.875 |
| d3 Open · 26B | — | 216 / 10.858 | 3 / 00.755 | 11 / 00.761 | 40 / 00.863 | 13 / 10.867 | 5 / 00.791 | 1 / 00.806 | 6 / 00.811 | 7 / 00.907 | 8 / 00.847 | 21 / 00.955 | 15 / 00.904 | 25 / 00.905 | 13 / 00.893 | 21 / 00.886 | 13 / 00.875 | 14 / 00.872 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Kept at ≥99%: 1,009 answers, 1,006 right · 3 wrong · AUROC 0.882 Logic 60.7% · Math 81.6% | — | 74.4%72.9%–75.9% | a | 2,447 / 3,288 | 1,006 right of 1,009 kept3 wrong · 2,279 skipped | 0.882 | 81.13chosen-answer Brier 0.1215 | 5.7%answered 98% | 162 ms | 198 ms | $0.118$118estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Kept at ≥99%: 524 answers, 523 right · 1 wrong · AUROC 0.882 Logic 58.9% · Math 80.3% | — | 73.0%71.4%–74.5% | a | 2,399 / 3,288 | 523 right of 524 kept1 wrong · 2,764 skipped | 0.882 | 80.27chosen-answer Brier 0.1242 | 0.1%answered 100% | 116 ms | 177 ms | $0.093$92.98estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Kept at ≥99%: 554 answers, 553 right · 1 wrong · AUROC 0.870 Logic 59.0% · Math 77.4% | — | 71.1%69.5%–72.6% | b | 2,338 / 3,288 | 553 right of 554 kept1 wrong · 2,734 skipped | 0.870 | 78.92chosen-answer Brier 0.1327 | 0.7%answered 100% | 811 ms | 1,111 ms | $0.052$51.85estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Kept at ≥99%: 384 answers, 384 right · 0 wrong · AUROC 0.873 Logic 58.7% · Math 77.3% | — | 71.0%69.4%–72.5% | b | 2,333 / 3,288 | 384 right of 384 kept0 mistakes · 2,904 skipped | 0.873 | 78.76chosen-answer Brier 0.1344 | 1.8%answered 99% | 128 ms | 218 ms | $0.102$102estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Kept at ≥99%: 296 answers, 294 right · 2 wrong · AUROC 0.865 Logic 57.9% · Math 77.1% | — | 70.5%68.9%–72.1% | b | 2,319 / 3,288 | 294 right of 296 kept2 wrong · 2,992 skipped | 0.865 | 78.49chosen-answer Brier 0.1354 | 3.8%answered 99% | 676 ms | 1,173 ms | $0.047$47.29estimate · DGX Spark |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 274 answers, 272 right · 2 wrong · AUROC 0.852 Logic 58.4% · Math 72.0% | — | 67.3%65.7%–68.9% | c | 2,214 / 3,288 | 272 right of 274 kept2 wrong · 3,014 skipped | 0.852 | 76.29chosen-answer Brier 0.1475 | 31.9%answered 89% | 450 ms | 587 ms | $0.029$28.55estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Kept at ≥99%: 520 answers, 518 right · 2 wrong · AUROC 0.868 Logic 54.5% · Math 74.0% | — | 67.3%65.7%–68.9% | cd | 2,213 / 3,288 | 518 right of 520 kept2 wrong · 2,768 skipped | 0.868 | 76.68chosen-answer Brier 0.1394 | 71.4%answered 69% | 150 ms | 232 ms | $0.028$27.98 list$0.120$120estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 385 answers, 385 right · 0 wrong · AUROC 0.844 Logic 54.8% · Math 72.5% | — | 66.4%64.8%–68.0% | cde | 2,184 / 3,288 | 385 right of 385 kept0 mistakes · 2,903 skipped | 0.844 | 75.44chosen-answer Brier 0.1554 | 13.5%answered 95% | 97 ms | 125 ms | $0.071$71.10estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 39 answers, 39 right · 0 wrong · AUROC 0.835 Logic 54.6% · Math 71.5% | — | 65.7%64.1%–67.3% | cdef | 2,161 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.835 | 74.31chosen-answer Brier 0.1711 | 7.5%answered 97% | 129 ms | 168 ms | $0.096$95.71estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Kept at ≥99%: 217 answers, 216 right · 1 wrong · AUROC 0.858 Logic 54.5% · Math 70.4% | — | 65.0%63.3%–66.6% | def | 2,136 / 3,288 | 216 right of 217 kept1 wrong · 3,071 skipped | 0.858 | 75.16chosen-answer Brier 0.1464 | 9.3%answered 97% | 57 ms | 86 ms | $0.046$46.28estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Kept at ≥99%: 176 answers, 176 right · 0 wrong · AUROC 0.843 Logic 55.2% · Math 69.9% | — | 64.8%63.2%–66.5% | def | 2,132 / 3,288 | 176 right of 176 kept0 mistakes · 3,112 skipped | 0.843 | 74.68chosen-answer Brier 0.1548 | 42.3%answered 83% | 81 ms | 136 ms | $0.010$9.71 list$0.066$66.05estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Kept at ≥99%: 884 answers, 866 right · 18 wrong · AUROC 0.841 Logic 53.1% · Math 70.3% | — | 64.4%62.8%–66.0% | ef | 2,118 / 3,288 | 866 right of 884 kept18 wrong · 2,404 skipped | 0.841 | 73.20chosen-answer Brier 0.1802 | 5.7%answered 98% | 81 ms | 130 ms | $0.016$16.31 list$0.064$63.74estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 411 answers, 409 right · 2 wrong · AUROC 0.849 Logic 53.3% · Math 69.8% | — | 64.1%62.5%–65.8% | f | 2,109 / 3,288 | 409 right of 411 kept2 wrong · 2,877 skipped | 0.849 | 74.56chosen-answer Brier 0.1502 | 10.3%answered 96% | 230 ms | 439 ms | $0.005$5.38 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Kept at ≥99%: 405 answers, 404 right · 1 wrong · AUROC 0.854 Logic 52.5% · Math 69.3% | with web | 63.6%61.9%–65.2% | f | 2,090 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.854 | 74.28chosen-answer Brier 0.1501 | 25.8%answered 90% | 430 ms | 707 ms | $1.025$1,025 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Kept at ≥99%: 50 answers, 50 right · 0 wrong · AUROC 0.844 Logic 52.3% · Math 69.0% | — | 63.3%61.6%–64.9% | f | 2,081 / 3,288 | 50 right of 50 kept0 mistakes · 3,238 skipped | 0.844 | 73.88chosen-answer Brier 0.1553 | 1.9%answered 99% | 301 ms | 448 ms | $0.020$20.01estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Kept at ≥99%: 207 answers, 207 right · 0 wrong · AUROC 0.789 Logic 41.9% · Math 59.0% | — | 53.1%51.4%–54.8% | g | 1,747 / 3,288 | 207 right of 207 kept0 mistakes · 3,081 skipped | 0.789 | 67.48chosen-answer Brier 0.1818 | 28.1%answered 85% | 339 ms | 472 ms | $0.012$12.17 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Kept at ≥99%: No answers at this threshold · AUROC 0.759 Logic 42.6% · Math 54.6% | — | 50.5%48.7%–52.2% | gh | 1,659 / 3,288 | No answers at this threshold | 0.759 | 62.91chosen-answer Brier 0.2465 | 14.5%answered 92% | 250 ms | 482 ms | $0.212$212estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Kept at ≥99%: 30 answers, 30 right · 0 wrong · AUROC 0.770 Logic 40.6% · Math 53.8% | — | 49.3%47.6%–51.0% | h | 1,620 / 3,288 | 30 right of 30 kept0 mistakes · 3,258 skipped | 0.770 | 65.05chosen-answer Brier 0.1917 | 1.6%answered 99% | 80 ms | 134 ms | $0.069$69.42estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Kept at ≥99%: 18 answers, 18 right · 0 wrong · AUROC 0.747 Logic 40.8% · Math 51.1% | — | 47.6%45.9%–49.3% | h | 1,564 / 3,288 | 18 right of 18 kept0 mistakes · 3,270 skipped | 0.747 | 62.92chosen-answer Brier 0.2172 | 2.4%answered 99% | 185 ms | 216 ms | $0.081$81.46 list$0.135$135estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
REAX Logits Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 71.6%3,288 | 87.1%70.6% kept (2,320) | 96.0%50.3% kept (1,655) | 99.5%33.4% kept (1,098) | 100.0%7.0% kept (229) |
| Logits Sigma-1 Duo Open · 57B | — | 69.3%3,288 | 85.2%68.0% kept (2,236) | 96.8%45.1% kept (1,483) | 99.5%22.6% kept (742) | 100.0%0.7% kept (23) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 229 / 00.878 | 12 / 00.750 | 11 / 00.804 | 28 / 00.921 | 11 / 00.888 | 7 / 00.826 | 1 / 00.818 | 7 / 00.869 | 9 / 00.886 | 11 / 00.917 | 30 / 00.955 | 10 / 00.946 | 14 / 00.881 | 7 / 00.935 | 22 / 00.887 | 25 / 00.913 | 24 / 00.900 |
| Logits Sigma-1 Duo Open · 57B | — | 23 / 00.868 | 1 / 00.738 | 1 / 00.764 | 6 / 00.880 | 2 / 00.900 | 0 / 00.841 | 0 / 00.667 | 0 / 00.847 | 1 / 00.888 | 0 / 00.872 | 2 / 00.950 | 0 / 00.919 | 1 / 00.910 | 1 / 00.939 | 4 / 00.895 | 1 / 00.871 | 3 / 00.911 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Kept at ≥99%: 229 answers, 229 right · 0 wrong · AUROC 0.878 Logic 59.3% · Math 78.0% | — | 71.6%70.0%–73.1% | a | 2,354 / 3,288 | 229 right of 229 kept0 mistakes · 3,059 skipped | 0.878 | 79.30chosen-answer Brier 0.1300 | 3.8%answered 99% | 200 ms | 260 ms | $0.148$148estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Kept at ≥99%: 23 answers, 23 right · 0 wrong · AUROC 0.868 Logic 57.8% · Math 75.2% | — | 69.3%67.7%–70.8% | b | 2,277 / 3,288 | 23 right of 23 kept0 mistakes · 3,265 skipped | 0.868 | 77.60chosen-answer Brier 0.1405 | 4.4%answered 99% | 207 ms | 324 ms | $0.165$165estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
TypeSafe Google Liquid AI Fastino Microsoft Celeris Subquadratic Nace.AI Inception Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| Jev 1.13 Closed · Efficient | — | 58.0%3,288 | 77.2%58.2% kept (1,913) | 89.6%35.8% kept (1,178) | 97.6%20.3% kept (669) | 100.0%9.5% kept (313) |
| Jev 1.14 Closed · Efficient | — | 59.3%3,288 | 79.8%54.3% kept (1,784) | 91.0%34.0% kept (1,117) | 97.2%19.5% kept (640) | 100.0%8.0% kept (263) |
| Gemini Decision 1 20261111 Closed · Efficient | — | 64.1%3,288 | 84.1%59.2% kept (1,948) | 92.5%42.8% kept (1,407) | 98.1%27.6% kept (906) | 100.0%7.8% kept (257) |
| d1-ultra Closed · Efficient | — | 67.5%3,288 | 84.9%64.9% kept (2,134) | 92.4%49.0% kept (1,610) | 98.2%32.2% kept (1,058) | 100.0%7.1% kept (233) |
| d1.1 Closed · Efficient | — | 58.6%3,288 | 79.6%53.7% kept (1,767) | 90.9%35.0% kept (1,150) | 97.0%20.4% kept (670) | 100.0%7.0% kept (231) |
| GLiDE API Closed · Efficient | — | 65.2%3,288 | 79.2%71.0% kept (2,336) | 90.3%50.6% kept (1,664) | 98.3%30.3% kept (996) | 100.0%6.5% kept (213) |
| Microsoft-Decision-1 Closed · Efficient | — | 54.3%3,288 | 70.4%57.1% kept (1,876) | 86.3%30.5% kept (1,003) | 97.2%11.8% kept (389) | 100.0%1.4% kept (47) |
| celeris-1-decision Closed · Efficient | — | 62.2%3,288 | 75.8%69.1% kept (2,271) | 87.8%48.8% kept (1,606) | 95.4%32.0% kept (1,053) | none kept |
| celeris-1.1-decision Closed · Efficient | — | 62.1%3,288 | 76.2%68.8% kept (2,262) | 87.9%48.6% kept (1,598) | 95.7%32.2% kept (1,060) | none kept |
| SubQ Snap 1.1 Closed · Efficient | — | 63.0%3,288 | 73.0%76.7% kept (2,522) | 82.6%58.1% kept (1,911) | 92.7%38.6% kept (1,268) | 99.3%13.2% kept (433) |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 64.8%3,288 | 77.5%72.3% kept (2,377) | 88.6%53.3% kept (1,752) | 97.0%35.3% kept (1,162) | 99.8%12.3% kept (405) |
| Mercury Decide Closed · Efficient | — | 50.6%3,288 | 64.0%65.1% kept (2,142) | 73.9%45.8% kept (1,505) | 83.8%28.6% kept (941) | 95.7%11.3% kept (372) |
| d1 Closed · Efficient | — | 50.5%3,288 | 69.8%50.3% kept (1,653) | 82.8%28.5% kept (938) | 93.6%14.7% kept (484) | 99.5%6.3% kept (208) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Jev 1.13 Closed · Efficient | — | 313 / 00.817 | 7 / 00.708 | 5 / 00.714 | 53 / 00.875 | 17 / 00.836 | 8 / 00.756 | 0 / 00.671 | 9 / 00.761 | 8 / 00.797 | 9 / 00.835 | 23 / 00.894 | 20 / 00.801 | 30 / 00.868 | 17 / 00.831 | 24 / 00.821 | 41 / 00.850 | 42 / 00.818 |
| Jev 1.14 Closed · Efficient | — | 263 / 00.810 | 5 / 00.671 | 8 / 00.711 | 48 / 00.856 | 16 / 00.854 | 7 / 00.791 | 0 / 00.686 | 8 / 00.700 | 5 / 00.778 | 8 / 00.810 | 18 / 00.910 | 14 / 00.788 | 25 / 00.862 | 18 / 00.826 | 21 / 00.781 | 27 / 00.849 | 35 / 00.834 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 257 / 00.838 | 5 / 00.766 | 0 / 00.699 | 38 / 00.885 | 15 / 00.862 | 6 / 00.794 | 2 / 00.729 | 0 / 00.789 | 5 / 00.854 | 10 / 00.851 | 18 / 00.882 | 13 / 00.797 | 23 / 00.833 | 15 / 00.799 | 31 / 00.859 | 38 / 00.907 | 38 / 00.858 |
| d1-ultra Closed · Efficient | — | 233 / 00.852 | 12 / 00.759 | 3 / 00.715 | 5 / 00.846 | 23 / 00.865 | 8 / 00.755 | 0 / 00.650 | 5 / 00.813 | 9 / 00.896 | 13 / 00.887 | 12 / 00.910 | 13 / 00.941 | 18 / 00.902 | 13 / 00.893 | 20 / 00.845 | 30 / 00.885 | 49 / 00.888 |
| d1.1 Closed · Efficient | — | 231 / 00.811 | 4 / 00.676 | 6 / 00.712 | 38 / 00.906 | 13 / 00.811 | 5 / 00.768 | 0 / 00.714 | 4 / 00.721 | 3 / 00.825 | 7 / 00.796 | 16 / 00.848 | 14 / 00.854 | 19 / 00.870 | 15 / 00.849 | 23 / 00.815 | 30 / 00.798 | 34 / 00.835 |
| GLiDE API Closed · Efficient | — | 213 / 00.852 | 7 / 00.777 | 7 / 00.765 | 38 / 00.866 | 17 / 00.874 | 5 / 00.746 | 1 / 00.625 | 11 / 00.778 | 4 / 00.853 | 4 / 00.840 | 20 / 00.987 | 13 / 00.872 | 29 / 00.885 | 11 / 00.924 | 15 / 00.896 | 17 / 00.811 | 14 / 00.881 |
| Microsoft-Decision-1 Closed · Efficient | — | 47 / 00.769 | 1 / 00.628 | 2 / 00.683 | 17 / 00.823 | 5 / 00.789 | 0 / 00.696 | 0 / 00.615 | 1 / 00.670 | 0 / 00.747 | 0 / 00.750 | 4 / 00.848 | 4 / 00.806 | 5 / 00.837 | 1 / 00.801 | 2 / 00.768 | 3 / 00.824 | 2 / 00.794 |
| celeris-1-decision Closed · Efficient | — | No answers at this threshold0.829 | 0 / 00.706 | 0 / 00.679 | 0 / 00.867 | 0 / 00.857 | 0 / 00.787 | 0 / 00.807 | 0 / 00.794 | 0 / 00.884 | 0 / 00.896 | 0 / 00.952 | 0 / 00.840 | 0 / 00.881 | 0 / 00.836 | 0 / 00.838 | 0 / 00.869 | 0 / 00.849 |
| celeris-1.1-decision Closed · Efficient | — | No answers at this threshold0.830 | 0 / 00.709 | 0 / 00.679 | 0 / 00.853 | 0 / 00.863 | 0 / 00.790 | 0 / 00.875 | 0 / 00.790 | 0 / 00.863 | 0 / 00.896 | 0 / 00.944 | 0 / 00.853 | 0 / 00.877 | 0 / 00.812 | 0 / 00.852 | 0 / 00.867 | 0 / 00.855 |
| SubQ Snap 1.1 Closed · Efficient | — | 430 / 30.827 | 17 / 00.736 | 18 / 10.743 | 35 / 10.857 | 28 / 00.819 | 12 / 00.784 | 1 / 00.722 | 15 / 00.780 | 10 / 00.808 | 19 / 00.754 | 47 / 00.929 | 35 / 00.829 | 31 / 00.832 | 28 / 00.931 | 37 / 10.802 | 45 / 00.871 | 52 / 00.848 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 404 / 10.846 | 7 / 00.729 | 24 / 00.771 | 58 / 10.877 | 23 / 00.839 | 8 / 00.850 | 1 / 00.471 | 14 / 00.799 | 9 / 00.861 | 15 / 00.817 | 28 / 00.903 | 31 / 00.840 | 47 / 00.910 | 30 / 00.919 | 36 / 00.866 | 39 / 00.853 | 34 / 00.893 |
| Mercury Decide Closed · Efficient | — | 356 / 160.777 | 14 / 20.670 | 28 / 10.744 | 42 / 40.775 | 25 / 00.811 | 9 / 00.692 | 1 / 00.729 | 8 / 00.748 | 6 / 00.687 | 11 / 30.718 | 26 / 10.859 | 23 / 20.801 | 31 / 10.808 | 18 / 00.821 | 22 / 10.751 | 44 / 10.785 | 48 / 00.822 |
| d1 Closed · Efficient | — | 207 / 10.757 | 3 / 00.630 | 5 / 00.683 | 47 / 10.828 | 14 / 00.765 | 5 / 00.653 | 0 / 00.530 | 4 / 00.700 | 3 / 00.789 | 6 / 00.752 | 9 / 00.761 | 11 / 00.726 | 21 / 00.802 | 11 / 00.738 | 20 / 00.769 | 23 / 00.777 | 25 / 00.784 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 3,276 of 3,288); shown once.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Kept at ≥99%: 233 answers, 233 right · 0 wrong · AUROC 0.852 Logic 54.9% · Math 74.1% | — | 67.5%65.9%–69.1% | a | 2,221 / 3,288 | 233 right of 233 kept0 mistakes · 3,055 skipped | 0.852 | 76.50chosen-answer Brier 0.1455 | 89.1%answered 52% | 180 ms | 426 ms | $0.008$8.37 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Kept at ≥99%: 213 answers, 213 right · 0 wrong · AUROC 0.852 Logic 54.5% · Math 70.8% | — | 65.2%63.6%–66.8% | ab | 2,144 / 3,288 | 213 right of 213 kept0 mistakes · 3,075 skipped | 0.852 | 75.04chosen-answer Brier 0.1513 | 49.2%answered 80% | 266 ms | 284 ms | $0.036$36.42 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 405 answers, 404 right · 1 wrong · AUROC 0.846 Logic 54.8% · Math 70.1% | — | 64.8%63.2%–66.5% | bc | 2,132 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.846 | 74.61chosen-answer Brier 0.1562 | 48.3%answered 79% | 327 ms | 518 ms | $0.016$16.13 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Kept at ≥99%: 257 answers, 257 right · 0 wrong · AUROC 0.838 Logic 50.0% · Math 71.3% | — | 64.1%62.4%–65.7% | bc | 2,106 / 3,288 | 257 right of 257 kept0 mistakes · 3,031 skipped | 0.838 | 74.23chosen-answer Brier 0.1559 | 3.0%answered 99% | 227 ms | 347 ms | $0.027$27.07 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Kept at ≥99%: 433 answers, 430 right · 3 wrong · AUROC 0.827 Logic 52.4% · Math 68.6% | — | 63.0%61.4%–64.7% | bc | 2,072 / 3,288 | 430 right of 433 kept3 wrong · 2,855 skipped | 0.827 | 72.66chosen-answer Brier 0.1770 | 58.3%answered 73% | 382 ms | 391 ms | $0.011$10.75 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.829 Logic 47.3% · Math 70.0% | — | 62.2%60.6%–63.9% | c | 2,046 / 3,288 | No answers at this threshold | 0.829 | 72.82chosen-answer Brier 0.1659 | 87.2%answered 50% | 388 ms | 392 ms | $0.011$11.25 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Kept at ≥99%: No answers at this threshold · AUROC 0.830 Logic 47.3% · Math 69.9% | — | 62.1%60.5%–63.8% | c | 2,043 / 3,288 | No answers at this threshold | 0.830 | 72.78chosen-answer Brier 0.1657 | 86.7%answered 51% | 381 ms | 388 ms | $0.001$1.13 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 263 answers, 263 right · 0 wrong · AUROC 0.810 Logic 48.2% · Math 65.1% | — | 59.3%57.6%–61.0% | d | 1,951 / 3,288 | 263 right of 263 kept0 mistakes · 3,025 skipped | 0.810 | 71.00chosen-answer Brier 0.1734 | 15.4%answered 93% | 164 ms | 188 ms | $0.021$21.27 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 231 answers, 231 right · 0 wrong · AUROC 0.811 Logic 46.0% · Math 65.2% | — | 58.6%56.9%–60.3% | d | 1,928 / 3,288 | 231 right of 231 kept0 mistakes · 3,057 skipped | 0.811 | 70.69chosen-answer Brier 0.1726 | 86.0%answered 46% | 174 ms | 285 ms | $0.004$3.98 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Kept at ≥99%: 313 answers, 313 right · 0 wrong · AUROC 0.817 Logic 46.3% · Math 64.1% | — | 58.0%56.3%–59.7% | d | 1,907 / 3,288 | 313 right of 313 kept0 mistakes · 2,975 skipped | 0.817 | 70.41chosen-answer Brier 0.1718 | 10.2%answered 96% | 166 ms | 205 ms | $0.021$21.27 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Kept at ≥99%: 47 answers, 47 right · 0 wrong · AUROC 0.769 Logic 44.4% · Math 59.4% | — | 54.3%52.6%–56.0% | e | 1,784 / 3,288 | 47 right of 47 kept0 mistakes · 3,241 skipped | 0.769 | 67.42chosen-answer Brier 0.1941 | 11.4%answered 94% | 312 ms | 360 ms | $0.009$9.07 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Kept at ≥99%: 372 answers, 356 right · 16 wrong · AUROC 0.777 Logic 45.1% · Math 53.5% | — | 50.6%48.9%–52.3% | f | 1,664 / 3,288 | 356 right of 372 kept16 wrong · 2,916 skipped | 0.777 | 64.36chosen-answer Brier 0.2189 | 10.1%answered 95% | 293 ms | 349 ms | $0.005$5.46 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 208 answers, 207 right · 1 wrong · AUROC 0.757 Logic 40.8% · Math 55.6% | — | 50.5%48.8%–52.3% | f | 1,662 / 3,288 | 207 right of 208 kept1 wrong · 3,080 skipped | 0.757 | 65.29chosen-answer Brier 0.1998 | 6.4%answered 97% | 185 ms | 292 ms | $0.004$3.98 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
No workhorse decision-model setup has answers on Logic and Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
No flagship decision-model setup has answers on Logic and Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Answers kept at the ≥99% confidence threshold
Share of the N questions whose chosen option had probability ≥99% in the five-option (standard) run: right (family colour) and wrong (red); beside it the counts right / wrong, whether any was wrong, and the AUROC of the chosen option's probability (right against wrong answers, 0.5 = no signal) · more kept with no mistake is better
Liquid AI Kept but wrong
Accuracy by confidence cut-off
| Setup | Web | All answersaccuracy · N | ≥50%accuracy of the kept · share kept | ≥70%accuracy of the kept · share kept | ≥90%accuracy of the kept · share kept | ≥99%accuracy of the kept · share kept |
|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | 59.9%3,288 | 81.2%52.3% kept (1,718) | 93.0%34.0% kept (1,118) | 98.4%19.5% kept (640) | 100.0%5.1% kept (169) |
| d1.2-ultra Closed · Undisclosed | — | 63.6%3,288 | 84.7%59.1% kept (1,944) | 93.4%41.6% kept (1,369) | 99.1%25.0% kept (822) | 100.0%3.9% kept (128) |
Answering only where the chosen option's probability reaches the cut-off: the accuracy of the answers kept and, under it, the share of the N questions kept.
Confidence by topic
| Setup | Web | Allkept right / wrong · AUROC | Logic · Numerical reasoningkept R / W · AUROC | Logic · Verbal reasoningkept R / W · AUROC | Logic · Formal logickept R / W · AUROC | Math · Basic conceptskept R / W · AUROC | Math · Digits and place valuekept R / W · AUROC | Math · Number baseskept R / W · AUROC | Math · Division and divisibilitykept R / W · AUROC | Math · Prime factorisation and GCD/LCMkept R / W · AUROC | Math · Rational numberskept R / W · AUROC | Math · Decimalskept R / W · AUROC | Math · Orderingkept R / W · AUROC | Math · Simple inequalitieskept R / W · AUROC | Math · Absolute valuekept R / W · AUROC | Math · Exponentskept R / W · AUROC | Math · Radicalskept R / W · AUROC | Math · Factorisationkept R / W · AUROC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2 Closed · Undisclosed | — | 169 / 00.812 | 3 / 00.676 | 2 / 00.719 | 23 / 00.874 | 8 / 00.833 | 5 / 00.800 | 0 / 00.557 | 3 / 00.708 | 3 / 00.840 | 9 / 00.794 | 20 / 00.896 | 8 / 00.862 | 12 / 00.843 | 10 / 00.797 | 24 / 00.843 | 20 / 00.838 | 19 / 00.845 |
| d1.2-ultra Closed · Undisclosed | — | 128 / 00.856 | 4 / 00.741 | 4 / 00.772 | 5 / 00.837 | 6 / 00.891 | 5 / 00.839 | 1 / 00.889 | 4 / 00.774 | 4 / 00.851 | 5 / 00.801 | 18 / 00.960 | 10 / 00.855 | 8 / 00.912 | 10 / 00.902 | 15 / 00.859 | 11 / 00.855 | 18 / 00.868 |
Per topic: answers kept at ≥99% (right / wrong) and, under them, the AUROC of the chosen option's probability (– where a topic has only right or only wrong answers).
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 128 answers, 128 right · 0 wrong · AUROC 0.856 Logic 51.6% · Math 69.9% | — | 63.6%61.9%–65.2% | a | 2,091 / 3,288 | 128 right of 128 kept0 mistakes · 3,160 skipped | 0.856 | 74.31chosen-answer Brier 0.1498 | 90.2%answered 48% | 217 ms | 347 ms | $0.008$8.37 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Kept at ≥99%: 169 answers, 169 right · 0 wrong · AUROC 0.812 Logic 49.2% · Math 65.5% | — | 59.9%58.2%–61.6% | b | 1,970 / 3,288 | 169 right of 169 kept0 mistakes · 3,119 skipped | 0.812 | 71.40chosen-answer Brier 0.1712 | 66.9%answered 65% | 208 ms | 390 ms | $0.004$3.98 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 9 setups not yet: DeepSeek-V4-Flash (chat) (missing Logic); Qwen3.5-397B-A17B (on · chat) (missing Math); GPT-6 Luna (max · chat) (missing Logic and Math); Claude Haiku 5.5 (max · chat) (missing Logic and Math); Claude Fable 5.2 (chat) (missing Logic and Math); Grok 4.7 (chat) (missing Logic); Spark-X2.6-4B (high · chat) (missing Logic and Math); Granite-4.2-3B (off · chat) (missing Logic and Math); Gemini 4 Carbon (high · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Logic 96.7% · Math 99.6% | — | 98.6%98.2%–99.0% | a | 3,243 / 3,288 | Not measured | Not measured | — | 60.9%answered 99% | 5,580 ms | 9,580 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Logic 96.7% · Math 99.6% | — | 98.6%98.1%–98.9% | a | 3,242 / 3,288 | Not measured | Not measured | — | 53.3%answered 99% | 16,350 ms | 90,630 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Logic 96.5% · Math 99.6% | — | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | — | 66.2%answered 99% | 5,240 ms | 10,410 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Logic 96.3% · Math 99.8% | — | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | — | 10.7%answered 100% | 5,700 ms | 19,750 ms | $2.919$2,919 |
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 96.2% · Math 99.8% | — | 98.5%98.1%–98.9% | a | 3,240 / 3,288 | Not measured | Not measured | — | 41.3%answered 99% | 3,600 ms | 8,380 ms | $13.179$13,179 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Logic 96.2% · Math 99.6% | — | 98.4%98.0%–98.8% | a | 3,237 / 3,288 | Not measured | Not measured | — | 6.3%answered 100% | 2,270 ms | 14,870 ms | $0.713$713 |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Logic 96.1% · Math 99.6% | — | 98.4%97.9%–98.8% | ab | 3,235 / 3,288 | Not measured | Not measured | — | 78.3%answered 98% | 4,920 ms | 12,150 ms | — |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.5% · Math 99.7% | — | 98.2%97.7%–98.6% | abc | 3,230 / 3,288 | Not measured | Not measured | — | 43.7%answered 99% | 4,030 ms | 10,080 ms | $7.042$7,042 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Logic 95.6% · Math 99.5% | — | 98.2%97.7%–98.6% | abcd | 3,229 / 3,288 | Not measured | Not measured | — | 41.7%answered 99% | 2,870 ms | 35,530 ms | $1.830$1,830 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.2% · Math 99.7% | — | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | — | 15.7%answered 100% | 4,940 ms | 8,620 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Logic 95.4% · Math 99.6% | — | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | — | 50.6%answered 99% | 2,750 ms | 34,100 ms | $2.506$2,506 |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Logic 95.5% · Math 99.6% | — | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | — | 31.3%answered 99% | 1,460 ms | 6,300 ms | $0.863$863 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Logic 95.1% · Math 99.4% | — | 98.0%97.4%–98.4% | abcde | 3,221 / 3,288 | Not measured | Not measured | — | 27.6%answered 99% | 10,590 ms | 53,970 ms | $0.615$615 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Logic 94.0% · Math 99.5% | — | 97.6%97.0%–98.1% | bcdef | 3,209 / 3,288 | Not measured | Not measured | — | 31.3%answered 99% | 6,780 ms | 36,290 ms | $0.732$732 |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Logic 94.0% · Math 99.2% | — | 97.4%96.8%–97.9% | cdef | 3,204 / 3,288 | Not measured | Not measured | — | 33.9%answered 99% | 11,200 ms | 39,560 ms | $2.020$2,020estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Logic 94.0% · Math 99.2% | — | 97.4%96.8%–97.9% | cdef | 3,203 / 3,288 | Not measured | Not measured | — | 65.0%answered 98% | 4,440 ms | 8,290 ms | $0.000$0.00 |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Logic 93.3% · Math 99.4% | — | 97.3%96.7%–97.8% | def | 3,200 / 3,288 | Not measured | Not measured | — | 17.0%answered 99% | 12,575 ms | 25,280 ms | $0.775$775estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Logic 92.4% · Math 99.4% | — | 97.0%96.4%–97.5% | ef | 3,190 / 3,288 | Not measured | Not measured | — | 34.2%answered 99% | 63,270 ms | 193,950 ms | $2.042$2,042estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Logic 93.2% · Math 98.8% | — | 96.9%96.2%–97.4% | fg | 3,185 / 3,288 | Not measured | Not measured | — | 7.4%answered 100% | 4,640 ms | 25,900 ms | $0.471$471 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Logic 94.0% · Math 98.2% | — | 96.8%96.1%–97.3% | fg | 3,182 / 3,288 | Not measured | Not measured | — | 38.9%answered 99% | 2,170 ms | 15,760 ms | $0.342$342 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Logic 91.6% · Math 99.2% | — | 96.6%96.0%–97.2% | fg | 3,177 / 3,288 | Not measured | Not measured | — | 54.7%answered 98% | 25,615 ms | 523,100 ms | $4.241$4,241estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Logic 88.9% · Math 99.2% | — | 95.7%95.4%–96.0% min–max of 3 seeds | gh | 3,145 / 3,288seed 0 run | Not measured | Not measured | — | 51.8%answered 97% | 19,260 ms | 194,917 ms | $1.410$1,410estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Logic 90.1% · Math 98.5% | — | 95.6%94.9%–96.3% | gh | 3,144 / 3,288 | Not measured | Not measured | — | 34.0%answered 98% | 14,295 ms | 78,230 ms | $0.683$683estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Logic 87.7% · Math 98.8% | — | 95.0%94.2%–95.7% | hi | 3,125 / 3,288 | Not measured | Not measured | — | 15.3%answered 99% | 22,345 ms | 53,390 ms | $0.749$749estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Logic 86.7% · Math 99.0% | — | 94.8%94.0%–95.5% | hij | 3,117 / 3,288 | Not measured | Not measured | — | 44.0%answered 97% | 42,425 ms | 432,510 ms | $3.205$3,205estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.9% · Math 98.5% | — | 94.2%94.0%–94.4% min–max of 3 seeds | hijk | 3,104 / 3,288seed 0 run | Not measured | Not measured | — | 54.1%answered 96% | 13,945 ms | 51,987 ms | $0.549$549estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Logic 85.5% · Math 98.7% | — | 94.2%93.3%–94.9% | hijk | 3,097 / 3,288 | Not measured | Not measured | — | 55.9%answered 96% | 29,040 ms | 343,820 ms | $0.206$206estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% · Math 98.9% | — | 94.0%93.8%–94.2% min–max of 3 seeds | hijk | 3,098 / 3,288seed 0 run | Not measured | Not measured | — | Not measured | 5,667 ms | 58,313 ms | $0.449$449estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Logic 84.2% · Math 98.9% | — | 93.9%93.89%–93.95% min–max of 3 seeds | ijk | 3,089 / 3,288seed 0 run | Not measured | Not measured | — | 46.7%answered 96% | 5,630 ms | 63,067 ms | $0.435$435estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% · Math 98.5% | — | 93.9%93.0%–94.7% | ijk | 3,087 / 3,288 | Not measured | Not measured | — | Not measured | 25,530 ms | 77,380 ms | $0.855$855estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Logic 83.6% · Math 98.2% | — | 93.2%92.9%–93.3% min–max of 3 seeds | jk | 3,068 / 3,288seed 0 run | Not measured | Not measured | — | 41.0%answered 95% | 9,065 ms | 75,703 ms | $0.598$598estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 83.2% · Math 98.1% | — | 93.0%92.1%–93.8% | k | 3,058 / 3,288 | Not measured | Not measured | — | 51.0%answered 95% | 13,150 ms | 44,990 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 82.6% · Math 98.2% | — | 92.9%92.0%–93.7% | k | 3,054 / 3,288 | Not measured | Not measured | — | 43.7%answered 96% | 131,535 ms | 392,620 ms | $0.375$375estimate · DGX Spark |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 76.8% · Math 97.7% | — | 90.5%90.2%–90.9% min–max of 3 seeds | l | 2,966 / 3,288seed 0 run | Not measured | Not measured | — | 31.0%answered 97% | 8,873 ms | 32,503 ms | $0.325$325estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% · Math 97.8% | — | 90.3%89.3%–91.3% | l | 2,970 / 3,288 | Not measured | Not measured | — | Not measured | 19,100 ms | 54,630 ms | $0.678$678estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 74.6% · Math 96.9% | — | 89.3%88.2%–90.3% | lm | 2,936 / 3,288 | Not measured | Not measured | — | 24.3%answered 97% | 77,885 ms | 232,800 ms | $0.215$215estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 72.9% · Math 96.8% | — | 88.6%87.5%–89.6% | lmn | 2,913 / 3,288 | Not measured | Not measured | — | 32.3%answered 96% | 9,935 ms | 27,690 ms | $0.313$313estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% · Math 97.0% | — | 88.1%86.9%–89.1% | mn | 2,896 / 3,288 | Not measured | Not measured | — | Not measured | 54,985 ms† | 987,660 ms† | $6.344$6,344estimate · RTX PRO 6000 Blackwell† |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Logic 75.4% · Math 94.0% | — | 87.6%86.5%–88.7% | mno | 2,881 / 3,288 | Not measured | Not measured | — | 40.6%answered 95% | 14,035 ms | 142,970 ms | $1.046$1,046estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Logic 70.3% · Math 94.7% | — | 86.4%85.2%–87.5% | nop | 2,840 / 3,288 | Not measured | Not measured | — | 30.9%answered 97% | 121,840 ms | 241,590 ms | $3.106$3,106estimate · RTX PRO 6000 Blackwell |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Logic 65.3% · Math 95.8% | — | 85.4%84.1%–86.5% | op | 2,807 / 3,288 | Not measured | Not measured | — | 20.9%answered 96% | 9,380 ms | 35,110 ms | $0.840$840estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Logic 64.8% · Math 94.6% | — | 84.4%83.1%–85.6% | p | 2,776 / 3,288 | Not measured | Not measured | — | 34.3%answered 94% | 17,805 ms | 92,100 ms | $0.788$788estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Logic 60.2% · Math 96.3% | — | 83.9%82.6%–85.1% | pq | 2,759 / 3,288 | Not measured | Not measured | — | 55.9%answered 88% | 20,095 ms† | 230,570 ms† | $1.913$1,913estimate · RTX PRO 6000 Blackwell† |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Logic 58.0% · Math 93.7% | — | 81.4%80.1%–82.7% | q | 2,678 / 3,288 | Not measured | Not measured | — | Not measured | 3,830 ms | 5,300 ms | $0.000$0.00 |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 65.1% · Math 89.6% | — | 81.2%79.8%–82.5% | qr | 2,669 / 3,288 | Not measured | Not measured | — | 31.1%answered 94% | 42,585 ms† | 593,950 ms† | $4.422$4,422estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Logic 54.2% · Math 90.8% | — | 78.3%76.8%–79.6% | r | 2,573 / 3,288 | Not measured | Not measured | — | 32.3%answered 92% | 8,740 ms | 16,420 ms | $0.308$308estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: Spark-X2.6-4B (high · chat) (missing Logic and Math); Granite-4.2-3B (off · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Logic 92.4% · Math 99.4% | — | 97.0%96.4%–97.5% | a | 3,190 / 3,288 | Not measured | Not measured | — | 34.2%answered 99% | 63,270 ms | 193,950 ms | $2.042$2,042estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Logic 91.6% · Math 99.2% | — | 96.6%96.0%–97.2% | ab | 3,177 / 3,288 | Not measured | Not measured | — | 54.7%answered 98% | 25,615 ms | 523,100 ms | $4.241$4,241estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Logic 88.9% · Math 99.2% | — | 95.7%95.4%–96.0% min–max of 3 seeds | b | 3,145 / 3,288seed 0 run | Not measured | Not measured | — | 51.8%answered 97% | 19,260 ms | 194,917 ms | $1.410$1,410estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Logic 90.1% · Math 98.5% | — | 95.6%94.9%–96.3% | b | 3,144 / 3,288 | Not measured | Not measured | — | 34.0%answered 98% | 14,295 ms | 78,230 ms | $0.683$683estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Logic 85.5% · Math 98.7% | — | 94.2%93.3%–94.9% | c | 3,097 / 3,288 | Not measured | Not measured | — | 55.9%answered 96% | 29,040 ms | 343,820 ms | $0.206$206estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Logic 84.6% · Math 98.9% | — | 94.0%93.8%–94.2% min–max of 3 seeds | c | 3,098 / 3,288seed 0 run | Not measured | Not measured | — | Not measured | 5,667 ms | 58,313 ms | $0.449$449estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Logic 84.2% · Math 98.9% | — | 93.9%93.89%–93.95% min–max of 3 seeds | c | 3,089 / 3,288seed 0 run | Not measured | Not measured | — | 46.7%answered 96% | 5,630 ms | 63,067 ms | $0.435$435estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Logic 83.6% · Math 98.2% | — | 93.2%92.9%–93.3% min–max of 3 seeds | c | 3,068 / 3,288seed 0 run | Not measured | Not measured | — | 41.0%answered 95% | 9,065 ms | 75,703 ms | $0.598$598estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 76.8% · Math 97.7% | — | 90.5%90.2%–90.9% min–max of 3 seeds | de | 2,966 / 3,288seed 0 run | Not measured | Not measured | — | 31.0%answered 97% | 8,873 ms | 32,503 ms | $0.325$325estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 75.9% · Math 97.8% | — | 90.3%89.3%–91.3% | d | 2,970 / 3,288 | Not measured | Not measured | — | Not measured | 19,100 ms | 54,630 ms | $0.678$678estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 74.6% · Math 96.9% | — | 89.3%88.2%–90.3% | def | 2,936 / 3,288 | Not measured | Not measured | — | 24.3%answered 97% | 77,885 ms | 232,800 ms | $0.215$215estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Logic 72.9% · Math 96.8% | — | 88.6%87.5%–89.6% | ef | 2,913 / 3,288 | Not measured | Not measured | — | 32.3%answered 96% | 9,935 ms | 27,690 ms | $0.313$313estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 70.9% · Math 97.0% | — | 88.1%86.9%–89.1% | fg | 2,896 / 3,288 | Not measured | Not measured | — | Not measured | 54,985 ms† | 987,660 ms† | $6.344$6,344estimate · RTX PRO 6000 Blackwell† |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Logic 75.4% · Math 94.0% | — | 87.6%86.5%–88.7% | fg | 2,881 / 3,288 | Not measured | Not measured | — | 40.6%answered 95% | 14,035 ms | 142,970 ms | $1.046$1,046estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Logic 70.3% · Math 94.7% | — | 86.4%85.2%–87.5% | gh | 2,840 / 3,288 | Not measured | Not measured | — | 30.9%answered 97% | 121,840 ms | 241,590 ms | $3.106$3,106estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Logic 64.8% · Math 94.6% | — | 84.4%83.1%–85.6% | h | 2,776 / 3,288 | Not measured | Not measured | — | 34.3%answered 94% | 17,805 ms | 92,100 ms | $0.788$788estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Logic 65.1% · Math 89.6% | — | 81.2%79.8%–82.5% | i | 2,669 / 3,288 | Not measured | Not measured | — | 31.1%answered 94% | 42,585 ms† | 593,950 ms† | $4.422$4,422estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Logic 87.7% · Math 98.8% | — | 95.0%94.2%–95.7% | a | 3,125 / 3,288 | Not measured | Not measured | — | 15.3%answered 99% | 22,345 ms | 53,390 ms | $0.749$749estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.9% · Math 98.5% | — | 94.2%94.0%–94.4% min–max of 3 seeds | ab | 3,104 / 3,288seed 0 run | Not measured | Not measured | — | 54.1%answered 96% | 13,945 ms | 51,987 ms | $0.549$549estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Logic 85.0% · Math 98.5% | — | 93.9%93.0%–94.7% | bc | 3,087 / 3,288 | Not measured | Not measured | — | Not measured | 25,530 ms | 77,380 ms | $0.855$855estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 83.2% · Math 98.1% | — | 93.0%92.1%–93.8% | c | 3,058 / 3,288 | Not measured | Not measured | — | 51.0%answered 95% | 13,150 ms | 44,990 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Logic 82.6% · Math 98.2% | — | 92.9%92.0%–93.7% | c | 3,054 / 3,288 | Not measured | Not measured | — | 43.7%answered 96% | 131,535 ms | 392,620 ms | $0.375$375estimate · DGX Spark |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Logic 65.3% · Math 95.8% | — | 85.4%84.1%–86.5% | d | 2,807 / 3,288 | Not measured | Not measured | — | 20.9%answered 96% | 9,380 ms | 35,110 ms | $0.840$840estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Logic 60.2% · Math 96.3% | — | 83.9%82.6%–85.1% | d | 2,759 / 3,288 | Not measured | Not measured | — | 55.9%answered 88% | 20,095 ms† | 230,570 ms† | $1.913$1,913estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Logic 54.2% · Math 90.8% | — | 78.3%76.8%–79.6% | e | 2,573 / 3,288 | Not measured | Not measured | — | 32.3%answered 92% | 8,740 ms | 16,420 ms | $0.308$308estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Logic 94.0% · Math 99.2% | — | 97.4%96.8%–97.9% | a | 3,204 / 3,288 | Not measured | Not measured | — | 33.9%answered 99% | 11,200 ms | 39,560 ms | $2.020$2,020estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Logic 93.3% · Math 99.4% | — | 97.3%96.7%–97.8% | a | 3,200 / 3,288 | Not measured | Not measured | — | 17.0%answered 99% | 12,575 ms | 25,280 ms | $0.775$775estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Logic 86.7% · Math 99.0% | — | 94.8%94.0%–95.5% | b | 3,117 / 3,288 | Not measured | Not measured | — | 44.0%answered 97% | 42,425 ms | 432,510 ms | $3.205$3,205estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: DeepSeek-V4-Flash (chat) (missing Logic); Qwen3.5-397B-A17B (on · chat) (missing Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Logic 96.2% · Math 99.6% | — | 98.4%98.0%–98.8% | a | 3,237 / 3,288 | Not measured | Not measured | — | 6.3%answered 100% | 2,270 ms | 14,870 ms | $0.713$713 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Logic 95.6% · Math 99.5% | — | 98.2%97.7%–98.6% | a | 3,229 / 3,288 | Not measured | Not measured | — | 41.7%answered 99% | 2,870 ms | 35,530 ms | $1.830$1,830 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Logic 95.4% · Math 99.6% | — | 98.2%97.7%–98.6% | a | 3,228 / 3,288 | Not measured | Not measured | — | 50.6%answered 99% | 2,750 ms | 34,100 ms | $2.506$2,506 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Logic 95.1% · Math 99.4% | — | 98.0%97.4%–98.4% | a | 3,221 / 3,288 | Not measured | Not measured | — | 27.6%answered 99% | 10,590 ms | 53,970 ms | $0.615$615 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Logic 93.2% · Math 98.8% | — | 96.9%96.2%–97.4% | b | 3,185 / 3,288 | Not measured | Not measured | — | 7.4%answered 100% | 4,640 ms | 25,900 ms | $0.471$471 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Logic 94.0% · Math 98.2% | — | 96.8%96.1%–97.3% | b | 3,182 / 3,288 | Not measured | Not measured | — | 38.9%answered 99% | 2,170 ms | 15,760 ms | $0.342$342 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: GPT-6 Luna (max · chat) (missing Logic and Math); Claude Haiku 5.5 (max · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Logic 95.5% · Math 99.6% | — | 98.2%97.7%–98.6% | a | 3,228 / 3,288 | Not measured | Not measured | — | 31.3%answered 99% | 1,460 ms | 6,300 ms | $0.863$863 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Logic 58.0% · Math 93.7% | — | 81.4%80.1%–82.7% | b | 2,678 / 3,288 | Not measured | Not measured | — | Not measured | 3,830 ms | 5,300 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.5% · Math 99.7% | — | 98.2%97.7%–98.6% | a | 3,230 / 3,288 | Not measured | Not measured | — | 43.7%answered 99% | 4,030 ms | 10,080 ms | $7.042$7,042 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 1 setup not yet: Claude Fable 5.2 (chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 96.2% · Math 99.8% | — | 98.5%98.1%–98.9% | a | 3,240 / 3,288 | Not measured | Not measured | — | 41.3%answered 99% | 3,600 ms | 8,380 ms | $13.179$13,179 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Logic 95.2% · Math 99.7% | — | 98.2%97.7%–98.6% | b | 3,228 / 3,288 | Not measured | Not measured | — | 15.7%answered 100% | 4,940 ms | 8,620 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Confidence
Not measured: no setup here returns option probabilities (probs_json). Chat models give a letter only; decision models give a probability per option.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: Grok 4.7 (chat) (missing Logic); Gemini 4 Carbon (high · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Logic 96.7% · Math 99.6% | — | 98.6%98.2%–99.0% | a | 3,243 / 3,288 | Not measured | Not measured | — | 60.9%answered 99% | 5,580 ms | 9,580 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Logic 96.7% · Math 99.6% | — | 98.6%98.1%–98.9% | a | 3,242 / 3,288 | Not measured | Not measured | — | 53.3%answered 99% | 16,350 ms | 90,630 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Logic 96.5% · Math 99.6% | — | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | — | 66.2%answered 99% | 5,240 ms | 10,410 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Logic 96.3% · Math 99.8% | — | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | — | 10.7%answered 100% | 5,700 ms | 19,750 ms | $2.919$2,919 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Logic 94.0% · Math 99.5% | — | 97.6%97.0%–98.1% | b | 3,209 / 3,288 | Not measured | Not measured | — | 31.3%answered 99% | 6,780 ms | 36,290 ms | $0.732$732 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Logic 94.0% · Math 99.2% | — | 97.4%96.8%–97.9% | b | 3,203 / 3,288 | Not measured | Not measured | — | 65.0%answered 98% | 4,440 ms | 8,290 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.