Math · Accuracy
Accuracy on Math
2,163 questions, asked in English. How to read this page · Method
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Alibaba DeepSeek OpenAI Thinking Machines Google Meta xAI Xiaomi OpenBMB IBM Liquid AI Other Z.ai MiniMax KRAFTON eCloud Technology MachinePulse PelaAI REAX Poindexter Labs Sthānika AI Logits Base Labs Perplexity Fastino vLLM Semantic Router Nace.AI Celeris Turion.AI Subquadratic TypeSafe newsbang tattrongvu Microsoft Patrick-18 StartLux scar-ai Apertix AI Valen-Team Zefan Cai OmniJev Inception victorbennett Cloudflare quincyqiang EldanRing Blockbrain Labs Metask Lab Strands Agents Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship | — | Chat | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Qwen4-Max high · chat Closed · Undisclosed | — | Chat | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| Claude Fable 5.1 chat Closed · Flagship | — | Chat | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 99.4%156/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| Claude Sonnet 5.5 chat Closed · Workhorse | — | Chat | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | Chat | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 99.0%100/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| GPT-6 Astra chat Closed · Undisclosed | — | Chat | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| GPT-6.1 Sol chat Closed · Undisclosed | — | Chat | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | Chat | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| Gemini 4 Carbon chat Closed · Undisclosed | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 99.6%225/226 | 99.2%255/257 | 99.7%343/344 |
| Gemini 4 Hydrogen chat | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Muse Ember max · chat Closed · Efficient | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 99.0%100/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Grok 4.7 chat Closed · Undisclosed | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 98.8%159/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | Chat | 100.0%238/238 | 99.0%202/204 | 100.0%17/17 | 99.4%156/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 99.6%225/226 | 98.8%254/257 | 100.0%344/344 |
| Qwen3.5-4B on · chat Open · 4.7B | — | Chat | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Gemma 4 31B chat Open · 31B | — | Chat | 99.2%236/238 | 98.0%200/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 99.0%104/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active | — | Chat | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| MiniCPM5-2B on · chat Open · 2.5B | — | Chat | 99.2%236/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 97.1%102/105 | 100.0%226/226 | 99.2%255/257 | 99.4%342/344 |
| GPT-6.1 Terra chat Closed · Undisclosed | — | Chat | 99.6%237/238 | 98.5%201/204 | 100.0%17/17 | 99.4%156/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 98.1%158/161 | 100.0%105/105 | 99.6%225/226 | 98.8%254/257 | 99.7%343/344 |
| Granite-4.2-3B high · chat Open · 3.0B | — | Chat | 99.6%237/238 | 98.5%201/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Qwen3.8-27B low · chat Open · 27B | — | Chat | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 98.7%155/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 99.0%104/105 | 99.1%224/226 | 99.6%256/257 | 99.4%342/344 |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | Chat | 98.7%235/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 96.3%105/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 99.6%225/226 | 98.4%253/257 | 99.7%343/344 |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B | — | Chat | 98.3%234/238 | 97.5%199/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 98.8%159/161 | 98.1%103/105 | 99.6%225/226 | 99.6%256/257 | 100.0%344/344 |
| LFM3-1.7B on · chat Open · 1.7B | — | Chat | 98.3%234/238 | 97.5%199/204 | 100.0%17/17 | 100.0%157/157 | 97.2%106/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 99.4%342/344 |
| Gemma 4 12B Unified chat Open · 12B | — | Chat | 98.7%235/238 | 96.1%196/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 97.0%130/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | Chat | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 98.1%154/157 | 98.2%107/109 | 97.8%131/134 | 98.2%108/110 | 99.0%100/101 | 98.1%158/161 | 100.0%105/105 | 100.0%226/226 | 98.4%253/257 | 99.7%343/344 |
| LFM3-1.7B off · chat Open · 1.7B | — | Chat | 98.7%235/238 | 96.1%196/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 99.0%100/101 | 100.0%161/161 | 98.1%103/105 | 99.1%224/226 | 98.8%254/257 | 99.1%341/344 |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | Chat | 97.5%232/238 | 97.1%198/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 96.3%129/134 | 99.1%109/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 99.6%225/226 | 98.8%254/257 | 99.7%343/344 |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B | — | Chat | 97.5%232/238 | 97.5%199/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 95.5%128/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 98.7%223/226 | 98.4%253/257 | 100.0%344/344 |
| Qwen3.5-4B off · chat Open · 4.7B | — | Chat | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 97.5%153/157 | 96.3%105/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 98.7%223/226 | 98.1%252/257 | 99.1%341/344 |
| Gemma 4 E4B on · chat Open · 8.0B | — | Chat | 97.5%232/238 | 94.1%192/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 94.8%127/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 97.1%102/105 | 99.1%224/226 | 98.8%254/257 | 100.0%344/344 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | Chat | 98.7%235/238 | 97.5%199/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 98.5%132/134 | 96.4%106/110 | 95.0%96/101 | 98.8%159/161 | 99.0%104/105 | 98.2%222/226 | 98.1%252/257 | 98.8%340/344 |
| MiniCPM5-2B off · chat Open · 2.5B | — | Chat | 98.7%235/238 | 96.6%197/204 | 88.2%15/17 | 98.1%154/157 | 98.2%107/109 | 98.5%132/134 | 97.3%107/110 | 99.0%100/101 | 97.5%157/161 | 97.1%102/105 | 99.1%224/226 | 98.4%253/257 | 99.7%343/344 |
| Gemma 4 E4B chat Open · 8.0B | — | Chat | 98.3%234/238 | 95.1%194/204 | 100.0%17/17 | 98.7%155/157 | 99.1%108/109 | 95.5%128/134 | 98.2%108/110 | 98.0%99/101 | 98.1%158/161 | 96.2%101/105 | 98.7%223/226 | 99.2%255/257 | 99.4%342/344 |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B | — | Chat | 97.5%232/238 | 93.6%191/204 | 100.0%17/17 | 98.1%154/157 | 96.3%105/109 | 96.3%129/134 | 97.3%107/110 | 97.0%98/101 | 98.8%159/161 | 97.1%102/105 | 99.1%224/226 | 99.2%255/257 | 99.7%343/344 |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | Chat | 97.9%233/238 | 94.1%192/204 | 94.1%16/17 | 99.4%156/157 | 97.2%106/109 | 95.5%128/134 | 95.5%105/110 | 99.0%100/101 | 96.9%156/161 | 97.1%102/105 | 98.2%222/226 | 98.4%253/257 | 99.4%342/344 |
| Raon-1 nano on · chat Open · 3.1B | — | Chat | 96.6%230/238 | 90.2%184/204 | 100.0%17/17 | 98.1%154/157 | 96.3%105/109 | 97.8%131/134 | 97.3%107/110 | 97.0%98/101 | 99.4%160/161 | 98.1%103/105 | 98.2%222/226 | 97.7%251/257 | 97.7%336/344 |
| Gemma 4 E2B on · chat Open · 5.1B | — | Chat | 94.5%225/238 | 91.2%186/204 | 94.1%16/17 | 98.1%154/157 | 97.2%106/109 | 94.8%127/134 | 97.3%107/110 | 97.0%98/101 | 98.8%159/161 | 96.2%101/105 | 99.1%224/226 | 98.1%252/257 | 99.4%342/344 |
| Gemma 4 E2B chat Open · 5.1B | — | Chat | 96.2%229/238 | 90.7%185/204 | 94.1%16/17 | 97.5%153/157 | 98.2%107/109 | 95.5%128/134 | 97.3%107/110 | 98.0%99/101 | 96.3%155/161 | 96.2%101/105 | 98.7%223/226 | 97.3%250/257 | 98.8%340/344 |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | Chat | 96.2%229/238 | 92.6%189/204 | 94.1%16/17 | 99.4%156/157 | 93.6%102/109 | 97.8%131/134 | 98.2%108/110 | 92.1%93/101 | 97.5%157/161 | 92.4%97/105 | 97.8%221/226 | 97.3%250/257 | 96.8%333/344 |
| Qwen3-14B off · chat Open · 15B | — | Chat | 94.5%225/238 | 83.3%170/204 | 100.0%17/17 | 96.8%152/157 | 99.1%108/109 | 94.8%127/134 | 98.2%108/110 | 95.0%96/101 | 91.3%147/161 | 100.0%105/105 | 99.1%224/226 | 98.4%253/257 | 98.8%340/344 |
| Qwen3.5-2B on · chat Open · 2.0B | — | Chat | 95.4%227/238 | 79.9%163/204 | 88.2%15/17 | 98.1%154/157 | 89.9%98/109 | 91.8%123/134 | 97.3%107/110 | 96.0%97/101 | 95.7%154/161 | 93.3%98/105 | 98.2%222/226 | 97.3%250/257 | 99.1%341/344 |
| Qwen3.5-2B off · chat Open · 2.0B | — | Chat | 94.5%225/238 | 87.3%178/204 | 100.0%17/17 | 89.2%140/157 | 90.8%99/109 | 97.8%131/134 | 93.6%103/110 | 97.0%98/101 | 93.2%150/161 | 93.3%98/105 | 95.1%215/226 | 99.2%255/257 | 98.3%338/344 |
| LFM2.5-2.6B on · chat Open · 2.7B | — | Chat | 95.8%228/238 | 89.2%182/204 | 88.2%15/17 | 93.6%147/157 | 94.5%103/109 | 95.5%128/134 | 98.2%108/110 | 97.0%98/101 | 98.1%158/161 | 97.1%102/105 | 92.0%208/226 | 93.4%240/257 | 91.9%316/344 |
| GPT-6 Luna chat Closed · Efficient | — | Chat | 90.8%216/238 | 81.4%166/204 | 82.4%14/17 | 94.3%148/157 | 89.9%98/109 | 96.3%129/134 | 96.4%106/110 | 93.1%94/101 | 93.2%150/161 | 98.1%103/105 | 96.9%219/226 | 96.9%249/257 | 97.1%334/344 |
| Erk-14B off · chat Open · 15B | — | Chat | 91.6%218/238 | 72.1%147/204 | 100.0%17/17 | 83.4%131/157 | 82.6%90/109 | 94.0%126/134 | 94.5%104/110 | 94.1%95/101 | 89.4%144/161 | 97.1%102/105 | 95.6%216/226 | 94.9%244/257 | 95.6%329/344 |
| Raon-1 nano off · chat Open · 3.1B | — | Chat | 91.2%217/238 | 75.5%154/204 | 94.1%16/17 | 87.3%137/157 | 89.9%98/109 | 93.3%125/134 | 92.7%102/110 | 92.1%93/101 | 85.7%138/161 | 91.4%96/105 | 91.2%206/226 | 93.0%239/257 | 91.9%316/344 |
| Karpo Decision 31B Open · 31B | — | Decision | 77.7%185/238 | 65.2%133/204 | 64.7%11/17 | 68.8%108/157 | 74.3%81/109 | 85.1%114/134 | 92.7%102/110 | 92.1%93/101 | 88.2%142/161 | 81.9%86/105 | 85.8%194/226 | 83.7%215/257 | 87.2%300/344 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 75.6%180/238 | 60.3%123/204 | 64.7%11/17 | 70.1%110/157 | 75.2%82/109 | 79.1%106/134 | 90.9%100/110 | 93.1%94/101 | 85.1%137/161 | 81.9%86/105 | 88.5%200/226 | 82.9%213/257 | 85.5%294/344 |
| REAX 1.3 Open · 57B | — | Decision | 76.9%183/238 | 57.8%118/204 | 64.7%11/17 | 61.8%97/157 | 76.1%83/109 | 76.1%102/134 | 92.7%102/110 | 84.2%85/101 | 83.9%135/161 | 79.0%83/105 | 85.4%193/226 | 77.8%200/257 | 85.8%295/344 |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 76.9%183/238 | 58.8%120/204 | 52.9%9/17 | 63.7%100/157 | 70.6%77/109 | 76.1%102/134 | 88.2%97/110 | 89.1%90/101 | 82.0%132/161 | 79.0%83/105 | 85.0%192/226 | 80.2%206/257 | 82.3%283/344 |
| Copernicus-31B Open · 31B | — | Decision | 76.1%181/238 | 54.9%112/204 | 58.8%10/17 | 63.1%99/157 | 71.6%78/109 | 79.1%106/134 | 89.1%98/110 | 92.1%93/101 | 81.4%131/161 | 76.2%80/105 | 85.4%193/226 | 79.4%204/257 | 83.7%288/344 |
| Sieve 31B v3 Open · 31B | — | Decision | 75.6%180/238 | 55.9%114/204 | 52.9%9/17 | 63.7%100/157 | 69.7%76/109 | 81.3%109/134 | 90.9%100/110 | 86.1%87/101 | 80.1%129/161 | 80.0%84/105 | 82.7%187/226 | 79.4%204/257 | 84.0%289/344 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 71.4%170/238 | 51.5%105/204 | 64.7%11/17 | 59.9%94/157 | 75.2%82/109 | 75.4%101/134 | 90.0%99/110 | 85.1%86/101 | 81.4%131/161 | 75.2%79/105 | 81.4%184/226 | 78.6%202/257 | 82.3%283/344 |
| d1-ultra Closed · Efficient | — | Decision | 71.8%171/238 | 53.4%109/204 | 70.6%12/17 | 65.6%103/157 | 59.6%65/109 | 78.4%105/134 | 84.5%93/110 | 77.2%78/101 | 72.7%117/161 | 78.1%82/105 | 79.6%180/226 | 79.4%204/257 | 82.6%284/344 |
| Mammoth-1 Open · 27B | — | Decision | 68.1%162/238 | 50.0%102/204 | 35.3%6/17 | 57.3%90/157 | 62.4%68/109 | 77.6%104/134 | 88.2%97/110 | 81.2%82/101 | 79.5%128/161 | 79.0%83/105 | 84.5%191/226 | 78.6%202/257 | 82.8%285/344 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 69.7%166/238 | 52.5%107/204 | 76.5%13/17 | 55.4%87/157 | 60.6%66/109 | 77.6%104/134 | 86.4%95/110 | 83.2%84/101 | 77.6%125/161 | 73.3%77/105 | 76.5%173/226 | 76.7%197/257 | 79.7%274/344 |
| GLiDE 27B Open · 27B | — | Decision | 69.3%165/238 | 52.0%106/204 | 52.9%9/17 | 61.8%97/157 | 60.6%66/109 | 73.1%98/134 | 90.9%100/110 | 81.2%82/101 | 77.6%125/161 | 70.5%74/105 | 77.4%175/226 | 73.9%190/257 | 78.5%270/344 |
| d3-ultra Open · 26B | — | Decision | 67.2%160/238 | 50.0%102/204 | 52.9%9/17 | 56.1%88/157 | 73.4%80/109 | 69.4%93/134 | 88.2%97/110 | 85.1%86/101 | 77.6%125/161 | 75.2%79/105 | 75.2%170/226 | 72.0%185/257 | 79.4%273/344 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 65.1%155/238 | 48.0%98/204 | 58.8%10/17 | 61.8%97/157 | 65.1%71/109 | 75.4%101/134 | 81.8%90/110 | 79.2%80/101 | 70.8%114/161 | 76.2%80/105 | 77.0%174/226 | 76.7%197/257 | 80.2%276/344 |
| GLiDE API Closed · Efficient | — | Decision | 67.2%160/238 | 54.4%111/204 | 47.1%8/17 | 56.7%89/157 | 59.6%65/109 | 71.6%96/134 | 88.2%97/110 | 80.2%81/101 | 73.9%119/161 | 75.2%79/105 | 74.3%168/226 | 75.1%193/257 | 77.0%265/344 |
| d3 Open · 26B | — | Decision | 65.1%155/238 | 52.5%107/204 | 47.1%8/17 | 58.6%92/157 | 56.0%61/109 | 67.9%91/134 | 86.4%95/110 | 80.2%81/101 | 77.6%125/161 | 70.5%74/105 | 76.5%173/226 | 73.2%188/257 | 79.4%273/344 |
| Drex Max Open · 27B | — | Decision | 68.1%162/238 | 52.5%107/204 | 52.9%9/17 | 56.1%88/157 | 58.7%64/109 | 67.9%91/134 | 89.1%98/110 | 79.2%80/101 | 77.0%124/161 | 69.5%73/105 | 75.7%171/226 | 71.6%184/257 | 78.5%270/344 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 69.7%166/238 | 52.9%108/204 | 58.8%10/17 | 54.8%86/157 | 56.0%61/109 | 67.9%91/134 | 88.2%97/110 | 80.2%81/101 | 75.8%122/161 | 72.4%76/105 | 76.1%172/226 | 70.4%181/257 | 77.0%265/344 |
| celeris-1-decision Closed · Efficient | — | Decision | 72.7%173/238 | 53.4%109/204 | 58.8%10/17 | 57.3%90/157 | 59.6%65/109 | 70.1%94/134 | 92.7%102/110 | 75.2%76/101 | 68.9%111/161 | 69.5%73/105 | 72.6%164/226 | 73.5%189/257 | 75.0%258/344 |
| GLiDE v1.2 Open · 27B | — | Decision | 67.2%160/238 | 52.9%108/204 | 52.9%9/17 | 56.7%89/157 | 60.6%66/109 | 73.9%99/134 | 88.2%97/110 | 78.2%79/101 | 72.0%116/161 | 71.4%75/105 | 75.2%170/226 | 70.4%181/257 | 76.2%262/344 |
| celeris-1.1-decision Closed · Efficient | — | Decision | 71.4%170/238 | 53.4%109/204 | 52.9%9/17 | 58.6%92/157 | 59.6%65/109 | 70.1%94/134 | 92.7%102/110 | 74.3%75/101 | 68.9%111/161 | 70.5%74/105 | 72.1%163/226 | 73.5%189/257 | 75.0%258/344 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 67.2%160/238 | 46.1%94/204 | 47.1%8/17 | 54.8%86/157 | 59.6%65/109 | 75.4%101/134 | 84.5%93/110 | 82.2%83/101 | 75.8%122/161 | 69.5%73/105 | 75.2%170/226 | 73.2%188/257 | 77.9%268/344 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 66.0%157/238 | 50.5%103/204 | 52.9%9/17 | 55.4%87/157 | 60.6%66/109 | 72.4%97/134 | 83.6%92/110 | 76.2%77/101 | 74.5%120/161 | 70.5%74/105 | 77.4%175/226 | 70.8%182/257 | 78.5%270/344 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 68.1%162/238 | 48.5%99/204 | 58.8%10/17 | 56.7%89/157 | 61.5%67/109 | 67.9%91/134 | 82.7%91/110 | 80.2%81/101 | 75.2%121/161 | 66.7%70/105 | 77.9%176/226 | 70.4%181/257 | 75.9%261/344 |
| Turion Jev 31B Open · 31B | — | Decision | 70.2%167/238 | 41.2%84/204 | 41.2%7/17 | 56.1%88/157 | 56.9%62/109 | 76.1%102/134 | 81.8%90/110 | 84.2%85/101 | 76.4%123/161 | 71.4%75/105 | 73.0%165/226 | 73.2%188/257 | 74.7%257/344 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 65.1%155/238 | 51.5%105/204 | 52.9%9/17 | 52.9%83/157 | 67.0%73/109 | 70.1%94/134 | 82.7%91/110 | 76.2%77/101 | 74.5%120/161 | 72.4%76/105 | 74.8%169/226 | 68.1%175/257 | 74.4%256/344 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 66.8%159/238 | 55.9%114/204 | 41.2%7/17 | 56.1%88/157 | 54.1%59/109 | 65.7%88/134 | 77.3%85/110 | 77.2%78/101 | 67.7%109/161 | 69.5%73/105 | 67.3%152/226 | 66.1%170/257 | 73.0%251/344 |
| d1.2 Closed · Undisclosed | — | Decision | 62.6%149/238 | 40.2%82/204 | 41.2%7/17 | 52.9%83/157 | 60.6%66/109 | 65.7%88/134 | 75.5%83/110 | 78.2%79/101 | 66.5%107/161 | 67.6%71/105 | 72.1%163/226 | 73.2%188/257 | 73.0%251/344 |
| d1.1 Closed · Efficient | — | Decision | 63.0%150/238 | 44.1%90/204 | 41.2%7/17 | 50.3%79/157 | 56.0%61/109 | 63.4%85/134 | 72.7%80/110 | 73.3%74/101 | 64.6%104/161 | 66.7%70/105 | 72.1%163/226 | 75.9%195/257 | 73.3%252/344 |
| Jev 1.14 Closed · Efficient | — | Decision | 59.7%142/238 | 42.6%87/204 | 58.8%10/17 | 52.2%82/157 | 58.7%64/109 | 64.9%87/134 | 78.2%86/110 | 76.2%77/101 | 66.5%107/161 | 74.3%78/105 | 70.4%159/226 | 71.2%183/257 | 71.8%247/344 |
| Jev 1.13 Closed · Efficient | — | Decision | 56.7%135/238 | 47.5%97/204 | 41.2%7/17 | 51.0%80/157 | 55.0%60/109 | 63.4%85/134 | 78.2%86/110 | 76.2%77/101 | 64.0%103/161 | 74.3%78/105 | 69.9%158/226 | 70.0%180/257 | 69.8%240/344 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 61.3%146/238 | 43.1%88/204 | 41.2%7/17 | 51.0%80/157 | 52.3%57/109 | 63.4%85/134 | 76.4%84/110 | 64.4%65/101 | 64.0%103/161 | 64.8%68/105 | 64.2%145/226 | 68.9%177/257 | 69.2%238/344 |
| Elephant-1 Open · 9.0B | — | Decision | 59.2%141/238 | 40.2%82/204 | 41.2%7/17 | 52.2%82/157 | 56.0%61/109 | 63.4%85/134 | 72.7%80/110 | 72.3%73/101 | 60.9%98/161 | 65.7%69/105 | 63.7%144/226 | 64.2%165/257 | 69.8%240/344 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 57.6%137/238 | 41.2%84/204 | 35.3%6/17 | 49.0%77/157 | 50.5%55/109 | 60.4%81/134 | 77.3%85/110 | 64.4%65/101 | 62.7%101/161 | 66.7%70/105 | 66.4%150/226 | 67.3%173/257 | 62.8%216/344 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 58.4%139/238 | 38.2%78/204 | 23.5%4/17 | 38.9%61/157 | 53.2%58/109 | 56.7%76/134 | 73.6%81/110 | 71.3%72/101 | 64.0%103/161 | 63.8%67/105 | 64.6%146/226 | 63.4%163/257 | 71.5%246/344 |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 59.7%142/238 | 42.2%86/204 | 17.6%3/17 | 51.0%80/157 | 53.2%58/109 | 61.2%82/134 | 72.7%80/110 | 63.4%64/101 | 66.5%107/161 | 61.0%64/105 | 61.5%139/226 | 66.1%170/257 | 63.4%218/344 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 61.3%146/238 | 40.2%82/204 | 23.5%4/17 | 48.4%76/157 | 52.3%57/109 | 53.7%72/134 | 70.0%77/110 | 63.4%64/101 | 60.9%98/161 | 61.9%65/105 | 66.4%150/226 | 63.8%164/257 | 66.6%229/344 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 55.5%132/238 | 40.7%83/204 | 29.4%5/17 | 49.7%78/157 | 51.4%56/109 | 57.5%77/134 | 70.0%77/110 | 65.3%66/101 | 64.0%103/161 | 61.9%65/105 | 62.4%141/226 | 64.6%166/257 | 66.0%227/344 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 56.3%134/238 | 39.7%81/204 | 35.3%6/17 | 42.7%67/157 | 56.0%61/109 | 53.7%72/134 | 72.7%80/110 | 70.3%71/101 | 60.2%97/161 | 61.0%64/105 | 64.2%145/226 | 60.7%156/257 | 69.2%238/344 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 59.2%141/238 | 38.2%78/204 | 35.3%6/17 | 46.5%73/157 | 49.5%54/109 | 54.5%73/134 | 70.9%78/110 | 65.3%66/101 | 64.6%104/161 | 62.9%66/105 | 60.6%137/226 | 62.3%160/257 | 66.3%228/344 |
| GLiDE 9B Open · 7.9B | — | Decision | 51.3%122/238 | 38.2%78/204 | 23.5%4/17 | 43.3%68/157 | 48.6%53/109 | 55.2%74/134 | 73.6%81/110 | 63.4%64/101 | 62.7%101/161 | 67.6%71/105 | 62.8%142/226 | 64.2%165/257 | 66.9%230/344 |
| vera-medium-omni Open · 4.2B | — | Decision | 55.9%133/238 | 36.3%74/204 | 35.3%6/17 | 42.7%67/157 | 48.6%53/109 | 50.7%68/134 | 70.9%78/110 | 67.3%68/101 | 62.7%101/161 | 63.8%67/105 | 58.4%132/226 | 61.1%157/257 | 65.4%225/344 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 55.0%131/238 | 41.7%85/204 | 35.3%6/17 | 43.9%69/157 | 48.6%53/109 | 53.0%71/134 | 67.3%74/110 | 66.3%67/101 | 60.2%97/161 | 56.2%59/105 | 58.8%133/226 | 61.1%157/257 | 61.0%210/344 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 52.1%124/238 | 37.7%77/204 | 23.5%4/17 | 40.8%64/157 | 53.2%58/109 | 60.4%81/134 | 71.8%79/110 | 60.4%61/101 | 57.8%93/161 | 61.9%65/105 | 59.3%134/226 | 59.9%154/257 | 63.1%217/344 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 55.5%132/238 | 36.3%74/204 | 29.4%5/17 | 44.6%70/157 | 50.5%55/109 | 53.0%71/134 | 69.1%76/110 | 60.4%61/101 | 60.2%97/161 | 59.0%62/105 | 58.8%133/226 | 61.9%159/257 | 62.5%215/344 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 54.2%129/238 | 40.7%83/204 | 17.6%3/17 | 41.4%65/157 | 49.5%54/109 | 52.2%70/134 | 74.5%82/110 | 64.4%65/101 | 59.6%96/161 | 58.1%61/105 | 55.8%126/226 | 61.5%158/257 | 62.8%216/344 |
| d1 Closed · Efficient | — | Decision | 52.1%124/238 | 39.7%81/204 | 35.3%6/17 | 40.8%64/157 | 42.2%46/109 | 51.5%69/134 | 59.1%65/110 | 63.4%64/101 | 60.9%98/161 | 61.9%65/105 | 58.0%131/226 | 61.9%159/257 | 67.2%231/344 |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 52.9%126/238 | 34.8%71/204 | 47.1%8/17 | 46.5%73/157 | 45.9%50/109 | 53.0%71/134 | 61.8%68/110 | 65.3%66/101 | 54.7%88/161 | 57.1%60/105 | 58.4%132/226 | 65.0%167/257 | 63.1%217/344 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 51.7%123/238 | 39.2%80/204 | 29.4%5/17 | 40.8%64/157 | 52.3%57/109 | 57.5%77/134 | 70.9%78/110 | 63.4%64/101 | 56.5%91/161 | 57.1%60/105 | 59.7%135/226 | 61.9%159/257 | 58.7%202/344 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 48.7%116/238 | 36.3%74/204 | 47.1%8/17 | 48.4%76/157 | 38.5%42/109 | 47.0%63/134 | 73.6%81/110 | 68.3%69/101 | 62.1%100/161 | 53.3%56/105 | 56.2%127/226 | 61.5%158/257 | 61.0%210/344 |
| OneJev-27B Open · 27B | — | Decision | 52.9%126/238 | 32.8%67/204 | 47.1%8/17 | 43.9%69/157 | 38.5%42/109 | 47.0%63/134 | 67.3%74/110 | 65.3%66/101 | 60.2%97/161 | 55.2%58/105 | 54.0%122/226 | 59.9%154/257 | 63.1%217/344 |
| Mercury Decide Closed · Efficient | — | Decision | 56.7%135/238 | 36.3%74/204 | 41.2%7/17 | 38.9%61/157 | 44.0%48/109 | 44.0%59/134 | 67.3%74/110 | 75.2%76/101 | 57.1%92/161 | 56.2%59/105 | 50.9%115/226 | 58.8%151/257 | 59.9%206/344 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 55.9%133/238 | 34.3%70/204 | 17.6%3/17 | 33.8%53/157 | 54.1%59/109 | 44.0%59/134 | 64.5%71/110 | 60.4%61/101 | 50.3%81/161 | 45.7%48/105 | 55.3%125/226 | 59.9%154/257 | 57.3%197/344 |
| Clef Open · 27B | — | Decision | 54.6%130/238 | 36.3%74/204 | 52.9%9/17 | 38.9%61/157 | 40.4%44/109 | 36.6%49/134 | 59.1%65/110 | 57.4%58/101 | 57.8%93/161 | 49.5%52/105 | 53.1%120/226 | 54.1%139/257 | 61.3%211/344 |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 47.1%112/238 | 34.3%70/204 | 35.3%6/17 | 43.3%68/157 | 39.4%43/109 | 49.3%66/134 | 62.7%69/110 | 56.4%57/101 | 56.5%91/161 | 45.7%48/105 | 49.1%111/226 | 55.6%143/257 | 60.2%207/344 |
| Clef-Flash Open · 9.0B | — | Decision | 49.6%118/238 | 33.3%68/204 | 52.9%9/17 | 42.0%66/157 | 43.1%47/109 | 42.5%57/134 | 57.3%63/110 | 59.4%60/101 | 57.8%93/161 | 53.3%56/105 | 49.1%111/226 | 55.3%142/257 | 54.4%187/344 |
| Winnow-12B Open · 12B | — | Decision | 51.3%122/238 | 31.9%65/204 | 41.2%7/17 | 35.7%56/157 | 38.5%42/109 | 47.0%63/134 | 61.8%68/110 | 67.3%68/101 | 52.2%84/161 | 45.7%48/105 | 45.6%103/226 | 46.3%119/257 | 49.1%169/344 |
| Cygnet Open · 12B | — | Decision | 43.7%104/238 | 30.9%63/204 | 41.2%7/17 | 33.1%52/157 | 39.4%43/109 | 36.6%49/134 | 50.9%56/110 | 59.4%60/101 | 48.4%78/161 | 45.7%48/105 | 49.6%112/226 | 47.9%123/257 | 47.7%164/344 |
| Metask-Jev-4B Open · 4.5B | — | Decision | 40.3%96/238 | 31.4%64/204 | 11.8%2/17 | 30.6%48/157 | 32.1%35/109 | 32.1%43/134 | 39.1%43/110 | 42.6%43/101 | 43.5%70/161 | 34.3%36/105 | 38.1%86/226 | 47.9%123/257 | 45.9%158/344 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 33.6%80/238 | 25.0%51/204 | 35.3%6/17 | 32.5%51/157 | 34.9%38/109 | 32.8%44/134 | 50.0%55/110 | 54.5%55/101 | 46.0%74/161 | 46.7%49/105 | 40.3%91/226 | 43.6%112/257 | 37.8%130/344 |
| d1-2B Open · 1.9B | — | Decision | 29.4%70/238 | 33.3%68/204 | 35.3%6/17 | 37.6%59/157 | 35.8%39/109 | 34.3%46/134 | 46.4%51/110 | 39.6%40/101 | 32.9%53/161 | 39.0%41/105 | 41.6%94/226 | 40.5%104/257 | 38.1%131/344 |
| GLiDE 2B Open · 1.9B | — | Decision | 32.8%78/238 | 27.9%57/204 | 23.5%4/17 | 30.6%48/157 | 36.7%40/109 | 32.8%44/134 | 40.9%45/110 | 40.6%41/101 | 38.5%62/161 | 42.9%45/105 | 35.8%81/226 | 42.4%109/257 | 41.0%141/344 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 31.1%74/238 | 25.5%52/204 | 11.8%2/17 | 29.9%47/157 | 26.6%29/109 | 35.1%47/134 | 40.0%44/110 | 51.5%52/101 | 37.3%60/161 | 35.2%37/105 | 37.2%84/226 | 38.1%98/257 | 39.8%137/344 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 30.3%72/238 | 27.0%55/204 | 29.4%5/17 | 29.9%47/157 | 33.9%37/109 | 26.1%35/134 | 26.4%29/110 | 34.7%35/101 | 36.0%58/161 | 23.8%25/105 | 29.6%67/226 | 34.6%89/257 | 40.7%140/344 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 27.7%66/238 | 20.6%42/204 | 29.4%5/17 | 23.6%37/157 | 35.8%39/109 | 29.1%39/134 | 38.2%42/110 | 42.6%43/101 | 36.0%58/161 | 27.6%29/105 | 30.5%69/226 | 28.4%73/257 | 29.1%100/344 |
| Strands Decider 2B Open · 2.0B | — | Decision | 29.8%71/238 | 24.0%49/204 | 23.5%4/17 | 25.5%40/157 | 30.3%33/109 | 27.6%37/134 | 30.9%34/110 | 37.6%38/101 | 31.7%51/161 | 25.7%27/105 | 29.6%67/226 | 25.3%65/257 | 30.2%104/344 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 26.5%63/238 | 22.5%46/204 | 11.8%2/17 | 25.5%40/157 | 33.9%37/109 | 25.4%34/134 | 28.2%31/110 | 28.7%29/101 | 32.9%53/161 | 28.6%30/105 | 29.2%66/226 | 24.1%62/257 | 34.3%118/344 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 27.3%65/238 | 21.6%44/204 | 35.3%6/17 | 21.0%33/157 | 31.2%34/109 | 23.9%32/134 | 30.9%34/110 | 29.7%30/101 | 33.5%54/161 | 27.6%29/105 | 30.1%68/226 | 33.5%86/257 | 27.6%95/344 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 22.7%54/238 | 23.0%47/204 | 29.4%5/17 | 21.7%34/157 | 30.3%33/109 | 19.4%26/134 | 24.5%27/110 | 23.8%24/101 | 18.0%29/161 | 24.8%26/105 | 22.1%50/226 | 19.1%49/257 | 25.0%86/344 |
| GLiDE 0.4B Open · 0.4B | — | Decision | 19.3%46/238 | 17.6%36/204 | 5.9%1/17 | 23.6%37/157 | 25.7%28/109 | 18.7%25/134 | 10.9%12/110 | 22.8%23/101 | 23.6%38/161 | 19.0%20/105 | 22.6%51/226 | 18.3%47/257 | 23.5%81/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC); Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)); Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC); Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test); GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.8% (95%: 99.5%–99.9%) Logic 96.2% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Accuracy 99.8% (95%: 99.5%–99.9%) Logic 96.3% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.7% (95%: 99.4%–99.9%) Logic 95.2% | — | Chat | 99.7%99.4%–99.9% | ab | 2,157 / 2,163 | Not measured | Not measured | — | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.7% (95%: 99.3%–99.8%) Logic 95.5% | — | Chat | 99.7%99.3%–99.8% | abcd | 2,156 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.2% | — | Chat | 99.6%99.3%–99.8% | abcdefg | 2,155 / 2,163 | Not measured | Not measured | — | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.7% | — | Chat | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.5% | — | Chat | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Accuracy 99.6% (95%: 99.3%–99.8%) Logic 95.4% | — | Chat | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.6% (95%: 99.2%–99.8%) Logic 96.7% | — | Chat | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.2%–99.8%) Logic 96.1% | — | Chat | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 85.7%answered 99% | 4,400 ms | 6,680 ms | — |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Accuracy 99.6% (95%: 99.2%–99.8%) Logic 95.5% | — | Chat | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.5% (95%: 99.2%–99.7%) Logic running | — | Chat | 99.5%99.2%–99.7% | abcdefg | 2,153 / 2,163 | Not measured | Not measured | — | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Accuracy 99.5% (95%: 99.2%–99.7%) Logic 95.6% | — | Chat | 99.5%99.2%–99.7% | abcdef | 2,153 / 2,163 | Not measured | Not measured | — | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Accuracy 99.5% (95%: 99.1%–99.7%) Logic 94.0% | — | Chat | 99.5%99.1%–99.7% | abcdefgh | 2,152 / 2,163 | Not measured | Not measured | — | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Accuracy 99.4% (95%: 99.0%–99.7%) Logic 95.1% | — | Chat | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | — | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Accuracy 99.4% (95%: 99.0%–99.7%) Logic 92.4% | — | Chat | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Accuracy 99.4% (95%: 99.0%–99.6%) Logic 93.3% | — | Chat | 99.4%99.0%–99.6% | abcdefghi | 2,150 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Accuracy 99.3% (95%: 98.9%–99.6%) Logic running | — | Chat | 99.3%98.9%–99.6% | abcdefghijk | 2,148 / 2,163 | Not measured | Not measured | — | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Accuracy 99.2% (mean of 3 seeds; min–max 99.2%–99.3%) Logic 88.9% | — | Chat | 99.2%99.2%–99.3% min–max of 3 seeds | abcdefghij | 2,147 / 2,163seed 0 run | Not measured | Not measured | — | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.2% (95%: 98.7%–99.5%) Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | abcdefghijk | 2,146 / 2,163 | Not measured | Not measured | — | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Accuracy 99.2% (95%: 98.7%–99.5%) Logic 91.6% | — | Chat | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | — | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Accuracy 99.2% (95%: 98.7%–99.5%) Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | abcdefghijk | 2,146 / 2,163 | Not measured | Not measured | — | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Accuracy 99.0% (95%: 98.5%–99.4%) Logic 86.7% | — | Chat | 99.0%98.5%–99.4% | bcdefghijk | 2,142 / 2,163 | Not measured | Not measured | — | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Accuracy 98.9% (mean of 3 seeds; min–max 98.8%–99.0%) Logic 84.6% | — | Chat | 98.9%98.8%–99.0% min–max of 3 seeds | abcdefghijkl | 2,142 / 2,163seed 0 run | Not measured | Not measured | — | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Accuracy 98.9% (mean of 3 seeds; min–max 98.8%–99.1%) Logic 84.2% | — | Chat | 98.9%98.8%–99.1% min–max of 3 seeds | bcdefghijkl | 2,139 / 2,163seed 0 run | Not measured | Not measured | — | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Accuracy 98.8% (95%: 98.3%–99.2%) Logic 87.7% | — | Chat | 98.8%98.3%–99.2% | cdefghijklm | 2,138 / 2,163 | Not measured | Not measured | — | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Accuracy 98.8% (95%: 98.2%–99.2%) Logic 93.2% | — | Chat | 98.8%98.2%–99.2% | dfghijklm | 2,137 / 2,163 | Not measured | Not measured | — | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Accuracy 98.7% (95%: 98.1%–99.1%) Logic 85.5% | — | Chat | 98.7%98.1%–99.1% | ghijklm | 2,135 / 2,163 | Not measured | Not measured | — | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.5% (mean of 3 seeds; min–max 98.4%–98.8%) Logic 85.9% | — | Chat | 98.5%98.4%–98.8% min–max of 3 seeds | efghijklm | 2,136 / 2,163seed 0 run | Not measured | Not measured | — | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.5% (95%: 97.9%–99.0%) Logic 85.0% | — | Chat | 98.5%97.9%–99.0% | hijklmn | 2,131 / 2,163 | Not measured | Not measured | — | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Accuracy 98.5% (95%: 97.9%–98.9%) Logic 90.1% | — | Chat | 98.5%97.9%–98.9% | hijklmno | 2,130 / 2,163 | Not measured | Not measured | — | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.2% (95%: 97.6%–98.7%) Logic 82.6% | — | Chat | 98.2%97.6%–98.7% | jklmno | 2,125 / 2,163 | Not measured | Not measured | — | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Accuracy 98.2% (95%: 97.6%–98.7%) Logic 94.0% | — | Chat | 98.2%97.6%–98.7% | ijklmnop | 2,125 / 2,163 | Not measured | Not measured | — | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Accuracy 98.2% (mean of 3 seeds; min–max 98.1%–98.3%) Logic 83.6% | — | Chat | 98.2%98.1%–98.3% min–max of 3 seeds | jklmnop | 2,126 / 2,163seed 0 run | Not measured | Not measured | — | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.1% (95%: 97.4%–98.6%) Logic 83.2% | — | Chat | 98.1%97.4%–98.6% | klmnop | 2,122 / 2,163 | Not measured | Not measured | — | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 97.8% (95%: 97.1%–98.4%) Logic 75.9% | — | Chat | 97.8%97.1%–98.4% | lmnopq | 2,116 / 2,163 | Not measured | Not measured | — | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 97.7% (mean of 3 seeds; min–max 97.6%–97.8%) Logic 76.8% | — | Chat | 97.7%97.6%–97.8% min–max of 3 seeds | mnopqr | 2,111 / 2,163seed 0 run | Not measured | Not measured | — | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 97.0% (95%: 96.2%–97.6%) Logic 70.9% | — | Chat | 97.0%96.2%–97.6% | nopqr | 2,098 / 2,163 | Not measured | Not measured | — | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 96.9% (95%: 96.1%–97.6%) Logic 74.6% | — | Chat | 96.9%96.1%–97.6% | opqr | 2,097 / 2,163 | Not measured | Not measured | — | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 96.8% (95%: 95.9%–97.4%) Logic 72.9% | — | Chat | 96.8%95.9%–97.4% | pqrs | 2,093 / 2,163 | Not measured | Not measured | — | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Accuracy 96.3% (95%: 95.4%–97.0%) Logic 60.2% | — | Chat | 96.3%95.4%–97.0% | qrst | 2,082 / 2,163 | Not measured | Not measured | — | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Accuracy 95.8% (95%: 94.9%–96.6%) Logic 65.3% | — | Chat | 95.8%94.9%–96.6% | rstu | 2,072 / 2,163 | Not measured | Not measured | — | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Accuracy 94.7% (95%: 93.7%–95.6%) Logic 70.3% | — | Chat | 94.7%93.7%–95.6% | stu | 2,049 / 2,163 | Not measured | Not measured | — | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Accuracy 94.6% (95%: 93.6%–95.5%) Logic 64.8% | — | Chat | 94.6%93.6%–95.5% | stu | 2,047 / 2,163 | Not measured | Not measured | — | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Accuracy 94.0% (95%: 92.9%–94.9%) Logic 75.4% | — | Chat | 94.0%92.9%–94.9% | tu | 2,033 / 2,163 | Not measured | Not measured | — | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 93.7% (95%: 92.6%–94.6%) Logic 58.0% | — | Chat | 93.7%92.6%–94.6% | u | 2,026 / 2,163 | Not measured | Not measured | — | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Accuracy 90.8% (95%: 89.5%–91.9%) Logic 54.2% | — | Chat | 90.8%89.5%–91.9% | v | 1,963 / 2,163 | Not measured | Not measured | — | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 89.6% (95%: 88.2%–90.8%) Logic 65.1% | — | Chat | 89.6%88.2%–90.8% | v | 1,937 / 2,163 | Not measured | Not measured | — | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Accuracy 81.6% (95%: 79.9%–83.1%) Logic 60.7% | — | Decision | 81.6%79.9%–83.1% | w | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 85.99chosen-answer Brier 0.0958 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Accuracy 80.3% (95%: 78.5%–81.9%) Logic 58.9% | — | Decision | 80.3%78.5%–81.9% | wx | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 85.05chosen-answer Brier 0.1016 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Accuracy 78.0% (95%: 76.2%–79.7%) Logic 59.3% | — | Decision | 78.0%76.2%–79.7% | xy | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 83.57chosen-answer Brier 0.1084 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Accuracy 77.4% (95%: 75.6%–79.1%) Logic 59.0% | — | Decision | 77.4%75.6%–79.1% | yz | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 83.18chosen-answer Brier 0.1103 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Accuracy 77.3% (95%: 75.5%–79.1%) Logic 58.7% | — | Decision | 77.3%75.5%–79.1% | yz | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 83.06chosen-answer Brier 0.1122 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Accuracy 77.1% (95%: 75.3%–78.8%) Logic 57.9% | — | Decision | 77.1%75.3%–78.8% | yz | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 82.89chosen-answer Brier 0.1133 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Accuracy 75.2% (95%: 73.4%–77.0%) Logic 57.8% | — | Decision | 75.2%73.4%–77.0% | zA | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 81.60chosen-answer Brier 0.1201 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Accuracy 74.1% (95%: 72.2%–75.9%) Logic 54.9% | — | Decision | 74.1%72.2%–75.9% | yzAB | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 80.88chosen-answer Brier 0.1235 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Accuracy 74.0% (95%: 72.1%–75.8%) Logic 54.5% | — | Decision | 74.0%72.1%–75.8% | zAB | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 81.16chosen-answer Brier 0.1164 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Accuracy 72.5% (95%: 70.6%–74.3%) Logic 54.8% | — | Decision | 72.5%70.6%–74.3% | ABC | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 79.32chosen-answer Brier 0.1385 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Accuracy 72.0% (95%: 70.1%–73.8%) Logic 58.4% | — | Decision | 72.0%70.1%–73.8% | ABCD | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 79.22chosen-answer Brier 0.1354 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 71.5% (95%: 69.6%–73.4%) Logic 54.6% | — | Decision | 71.5%69.6%–73.4% | BCD | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 78.39chosen-answer Brier 0.1474 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Accuracy 71.3% (95%: 69.4%–73.2%) Logic 50.0% | — | Decision | 71.3%69.4%–73.2% | ABCD | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 78.84chosen-answer Brier 0.1366 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Accuracy 70.8% (95%: 68.8%–72.7%) Logic 54.5% | — | Decision | 70.8%68.8%–72.7% | BCD | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 78.56chosen-answer Brier 0.1367 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 70.4% (95%: 68.5%–72.3%) Logic 54.5% | — | Decision | 70.4%68.5%–72.3% | BCDE | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 78.71chosen-answer Brier 0.1299 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Accuracy 70.3% (95%: 68.4%–72.2%) Logic 53.1% | — | Decision | 70.3%68.4%–72.2% | BCDE | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 77.10chosen-answer Brier 0.1611 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Accuracy 70.1% (95%: 68.1%–72.0%) Logic 54.8% | — | Decision | 70.1%68.1%–72.0% | BCDE | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 78.01chosen-answer Brier 0.1407 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 70.0% (95%: 68.0%–71.9%) Logic 47.3% | — | Decision | 70.0%68.0%–71.9% | BCDEF | 1,514 / 2,163 | No answers at this threshold | 0.858 | 77.96chosen-answer Brier 0.1407 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Accuracy 69.9% (95%: 67.9%–71.8%) Logic 55.2% | — | Decision | 69.9%67.9%–71.8% | CDEF | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 77.95chosen-answer Brier 0.1396 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 69.9% (95%: 67.9%–71.8%) Logic 47.3% | — | Decision | 69.9%67.9%–71.8% | BCDEF | 1,511 / 2,163 | No answers at this threshold | 0.859 | 77.91chosen-answer Brier 0.1403 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 69.9% (95%: 67.9%–71.8%) Logic 51.6% | — | Decision | 69.9%67.9%–71.8% | CDE | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 78.23chosen-answer Brier 0.1340 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Accuracy 69.8% (95%: 67.8%–71.7%) Logic 53.3% | — | Decision | 69.8%67.8%–71.7% | CDEF | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 78.16chosen-answer Brier 0.1345 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Accuracy 69.3% (95%: 67.3%–71.2%) Logic 52.5% | with web | Decision | 69.3%67.3%–71.2% | CDEFG | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 77.94chosen-answer Brier 0.1342 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Accuracy 69.0% (95%: 67.0%–70.9%) Logic 52.3% | — | Decision | 69.0%67.0%–70.9% | CDEFG | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 77.52chosen-answer Brier 0.1399 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Accuracy 68.6% (95%: 66.6%–70.5%) Logic 52.4% | — | Decision | 68.6%66.6%–70.5% | DEFG | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 76.15chosen-answer Brier 0.1626 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Accuracy 66.3% (95%: 64.2%–68.2%) Logic 50.6% | — | Decision | 66.3%64.2%–68.2% | EFGH | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 74.96chosen-answer Brier 0.1632 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 65.5% (95%: 63.5%–67.5%) Logic 49.2% | — | Decision | 65.5%63.5%–67.5% | FGHI | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 74.98chosen-answer Brier 0.1556 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 65.2% (95%: 63.2%–67.2%) Logic 46.0% | — | Decision | 65.2%63.2%–67.2% | GHI | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 74.62chosen-answer Brier 0.1595 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 65.1% (95%: 63.1%–67.1%) Logic 48.2% | — | Decision | 65.1%63.1%–67.1% | GHI | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 74.56chosen-answer Brier 0.1602 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 64.1% (95%: 62.0%–66.1%) Logic 46.3% | — | Decision | 64.1%62.0%–66.1% | HIJ | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 74.08chosen-answer Brier 0.1592 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 62.1% (95%: 60.0%–64.1%) Logic 46.4% | — | Decision | 62.1%60.0%–64.1% | HIJK | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 72.28chosen-answer Brier 0.1752 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Accuracy 61.3% (95%: 59.3%–63.4%) Logic 46.4% | — | Decision | 61.3%59.3%–63.4% | IJK | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 71.86chosen-answer Brier 0.1763 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 60.1% (95%: 58.0%–62.1%) Logic 43.9% | — | Decision | 60.1%58.0%–62.1% | JKL | 1,300 / 2,163 | No answers at this threshold | 0.803 | 71.15chosen-answer Brier 0.1779 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 59.8% (95%: 57.7%–61.9%) Logic 48.2% | — | Decision | 59.8%57.7%–61.9% | JKL | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 71.32chosen-answer Brier 0.1719 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 59.8% (95%: 57.7%–61.8%) Logic 44.3% | — | Decision | 59.8%57.7%–61.8% | JKL | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 71.08chosen-answer Brier 0.1762 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Accuracy 59.4% (95%: 57.3%–61.4%) Logic 44.4% | — | Decision | 59.4%57.3%–61.4% | KLM | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 70.49chosen-answer Brier 0.1839 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Accuracy 59.0% (95%: 56.9%–61.0%) Logic 41.9% | — | Decision | 59.0%56.9%–61.0% | KLM | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 70.98chosen-answer Brier 0.1703 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Accuracy 58.8% (95%: 56.7%–60.9%) Logic 45.3% | — | Decision | 58.8%56.7%–60.9% | KLM | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 69.67chosen-answer Brier 0.1946 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Accuracy 58.4% (95%: 56.3%–60.5%) Logic 46.6% | — | Decision | 58.4%56.3%–60.5% | KLMN | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 69.90chosen-answer Brier 0.1864 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Accuracy 57.9% (95%: 55.8%–60.0%) Logic 47.8% | — | Decision | 57.9%55.8%–60.0% | KLMN | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 69.69chosen-answer Brier 0.1855 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Accuracy 56.8% (95%: 54.7%–58.9%) Logic 44.1% | — | Decision | 56.8%54.7%–58.9% | LMN | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 69.18chosen-answer Brier 0.1846 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Accuracy 56.0% (95%: 53.9%–58.1%) Logic 43.3% | — | Decision | 56.0%53.9%–58.1% | LMNOP | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 67.15chosen-answer Brier 0.2174 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Accuracy 56.0% (95%: 53.9%–58.1%) Logic 42.3% | — | Decision | 56.0%53.9%–58.1% | LMNO | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 67.89chosen-answer Brier 0.2021 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Accuracy 55.9% (95%: 53.8%–58.0%) Logic 44.7% | — | Decision | 55.9%53.8%–58.0% | LMNO | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 67.89chosen-answer Brier 0.2015 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Accuracy 55.8% (95%: 53.7%–57.9%) Logic 45.9% | — | Decision | 55.8%53.7%–57.9% | LMNO | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 68.25chosen-answer Brier 0.1934 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 55.6% (95%: 53.5%–57.7%) Logic 40.8% | — | Decision | 55.6%53.5%–57.7% | LMNOP | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 68.16chosen-answer Brier 0.1930 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 55.3% (95%: 53.2%–57.4%) Logic 44.4% | — | Decision | 55.3%53.2%–57.4% | MNOP | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 68.37chosen-answer Brier 0.1860 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Accuracy 55.2% (95%: 53.1%–57.3%) Logic 43.1% | — | Decision | 55.2%53.1%–57.3% | MNOP | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 67.61chosen-answer Brier 0.2002 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Accuracy 54.6% (95%: 52.4%–56.6%) Logic 42.6% | — | Decision | 54.6%52.4%–56.6% | MNOPQ | 1,180 / 2,163 | No answers at this threshold | 0.764 | 64.53chosen-answer Brier 0.2549 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Accuracy 53.8% (95%: 51.7%–55.9%) Logic 40.6% | — | Decision | 53.8%51.7%–55.9% | NOPQ | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 67.54chosen-answer Brier 0.1869 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Accuracy 53.5% (95%: 51.4%–55.6%) Logic 45.1% | — | Decision | 53.5%51.4%–55.6% | NOPQ | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 66.02chosen-answer Brier 0.2146 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 51.5% (95%: 49.4%–53.6%) Logic 39.7% | — | Decision | 51.5%49.4%–53.6% | PQR | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 65.62chosen-answer Brier 0.2026 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Accuracy 51.1% (95%: 49.0%–53.2%) Logic 40.8% | — | Decision | 51.1%49.0%–53.2% | OPQR | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 65.20chosen-answer Brier 0.2070 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 50.4% (95%: 48.3%–52.5%) Logic 40.4% | — | Decision | 50.4%48.3%–52.5% | QR | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 64.11chosen-answer Brier 0.2221 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Accuracy 49.8% (95%: 47.7%–51.9%) Logic 38.9% | — | Decision | 49.8%47.7%–51.9% | QR | 1,077 / 2,163 | No answers at this threshold | 0.730 | 64.44chosen-answer Brier 0.2092 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Accuracy 46.9% (95%: 44.8%–49.0%) Logic 39.7% | — | Decision | 46.9%44.8%–49.0% | RS | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 63.00chosen-answer Brier 0.2087 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Accuracy 44.3% (95%: 42.3%–46.4%) Logic 39.3% | — | Decision | 44.3%42.3%–46.4% | S | 959 / 2,163 | No answers at this threshold | 0.735 | 61.87chosen-answer Brier 0.2060 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Accuracy 39.2% (95%: 37.1%–41.2%) Logic 32.1% | — | Decision | 39.2%37.1%–41.2% | T | 847 / 2,163 | No answers at this threshold | 0.651 | 58.63chosen-answer Brier 0.2190 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Accuracy 38.7% (95%: 36.6%–40.7%) Logic 30.8% | — | Decision | 38.7%36.6%–40.7% | T | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 58.95chosen-answer Brier 0.2075 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Accuracy 37.1% (95%: 35.1%–39.1%) Logic 30.7% | — | Decision | 37.1%35.1%–39.1% | TU | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 57.72chosen-answer Brier 0.2163 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Accuracy 36.8% (95%: 34.7%–38.8%) Logic 32.3% | — | Decision | 36.8%34.7%–38.8% | T | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 55.70chosen-answer Brier 0.2536 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Accuracy 35.3% (95%: 33.3%–37.3%) Logic 31.7% | — | Decision | 35.3%33.3%–37.3% | TU | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 56.49chosen-answer Brier 0.2229 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Accuracy 32.1% (95%: 30.2%–34.1%) Logic 29.2% | — | Decision | 32.1%30.2%–34.1% | UV | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 53.21chosen-answer Brier 0.2566 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 29.7% (95%: 27.8%–31.6%) Logic 24.7% | — | Decision | 29.7%27.8%–31.6% | V | 642 / 2,163 | No answers at this threshold | 0.583 | 54.40chosen-answer Brier 0.2088 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Accuracy 28.7% (95%: 26.8%–30.6%) Logic 22.4% | — | Decision | 28.7%26.8%–30.6% | V | 620 / 2,163 | No answers at this threshold | 0.569 | 53.90chosen-answer Brier 0.2086 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Accuracy 28.2% (95%: 26.4%–30.2%) Logic 27.0% | — | Decision | 28.2%26.4%–30.2% | V | 611 / 2,163 | No answers at this threshold | 0.592 | 53.62chosen-answer Brier 0.2100 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 28.2% (95%: 26.3%–30.1%) Logic 26.7% | — | Decision | 28.2%26.3%–30.1% | V | 610 / 2,163 | No answers at this threshold | 0.596 | 53.60chosen-answer Brier 0.2100 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Accuracy 22.7% (95%: 20.9%–24.5%) Logic 23.6% | — | Decision | 22.7%20.9%–24.5% | W | 490 / 2,163 | No answers at this threshold | 0.501 | 52.01chosen-answer Brier 0.1863 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Accuracy 20.6% (95%: 18.9%–22.3%) Logic 23.2% | — | Decision | 20.6%18.9%–22.3% | W | 445 / 2,163 | No answers at this threshold | 0.500 | 50.98chosen-answer Brier 0.1862 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Alibaba OpenBMB IBM Other Google KRAFTON Liquid AI scar-ai PelaAI Perplexity StartLux victorbennett quincyqiang Metask Lab Fastino Strands Agents OmniJev Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B | — | Chat | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| MiniCPM5-2B on · chat Open · 2.5B | — | Chat | 99.2%236/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 97.1%102/105 | 100.0%226/226 | 99.2%255/257 | 99.4%342/344 |
| Granite-4.2-3B high · chat Open · 3.0B | — | Chat | 99.6%237/238 | 98.5%201/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B | — | Chat | 98.3%234/238 | 97.5%199/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 98.8%159/161 | 98.1%103/105 | 99.6%225/226 | 99.6%256/257 | 100.0%344/344 |
| LFM3-1.7B on · chat Open · 1.7B | — | Chat | 98.3%234/238 | 97.5%199/204 | 100.0%17/17 | 100.0%157/157 | 97.2%106/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 99.4%342/344 |
| LFM3-1.7B off · chat Open · 1.7B | — | Chat | 98.7%235/238 | 96.1%196/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 99.0%100/101 | 100.0%161/161 | 98.1%103/105 | 99.1%224/226 | 98.8%254/257 | 99.1%341/344 |
| Qwen3.5-4B off · chat Open · 4.7B | — | Chat | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 97.5%153/157 | 96.3%105/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 98.7%223/226 | 98.1%252/257 | 99.1%341/344 |
| MiniCPM5-2B off · chat Open · 2.5B | — | Chat | 98.7%235/238 | 96.6%197/204 | 88.2%15/17 | 98.1%154/157 | 98.2%107/109 | 98.5%132/134 | 97.3%107/110 | 99.0%100/101 | 97.5%157/161 | 97.1%102/105 | 99.1%224/226 | 98.4%253/257 | 99.7%343/344 |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B | — | Chat | 97.5%232/238 | 93.6%191/204 | 100.0%17/17 | 98.1%154/157 | 96.3%105/109 | 96.3%129/134 | 97.3%107/110 | 97.0%98/101 | 98.8%159/161 | 97.1%102/105 | 99.1%224/226 | 99.2%255/257 | 99.7%343/344 |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | Chat | 97.9%233/238 | 94.1%192/204 | 94.1%16/17 | 99.4%156/157 | 97.2%106/109 | 95.5%128/134 | 95.5%105/110 | 99.0%100/101 | 96.9%156/161 | 97.1%102/105 | 98.2%222/226 | 98.4%253/257 | 99.4%342/344 |
| Raon-1 nano on · chat Open · 3.1B | — | Chat | 96.6%230/238 | 90.2%184/204 | 100.0%17/17 | 98.1%154/157 | 96.3%105/109 | 97.8%131/134 | 97.3%107/110 | 97.0%98/101 | 99.4%160/161 | 98.1%103/105 | 98.2%222/226 | 97.7%251/257 | 97.7%336/344 |
| Gemma 4 E2B on · chat Open · 5.1B | — | Chat | 94.5%225/238 | 91.2%186/204 | 94.1%16/17 | 98.1%154/157 | 97.2%106/109 | 94.8%127/134 | 97.3%107/110 | 97.0%98/101 | 98.8%159/161 | 96.2%101/105 | 99.1%224/226 | 98.1%252/257 | 99.4%342/344 |
| Gemma 4 E2B chat Open · 5.1B | — | Chat | 96.2%229/238 | 90.7%185/204 | 94.1%16/17 | 97.5%153/157 | 98.2%107/109 | 95.5%128/134 | 97.3%107/110 | 98.0%99/101 | 96.3%155/161 | 96.2%101/105 | 98.7%223/226 | 97.3%250/257 | 98.8%340/344 |
| Qwen3.5-2B on · chat Open · 2.0B | — | Chat | 95.4%227/238 | 79.9%163/204 | 88.2%15/17 | 98.1%154/157 | 89.9%98/109 | 91.8%123/134 | 97.3%107/110 | 96.0%97/101 | 95.7%154/161 | 93.3%98/105 | 98.2%222/226 | 97.3%250/257 | 99.1%341/344 |
| Qwen3.5-2B off · chat Open · 2.0B | — | Chat | 94.5%225/238 | 87.3%178/204 | 100.0%17/17 | 89.2%140/157 | 90.8%99/109 | 97.8%131/134 | 93.6%103/110 | 97.0%98/101 | 93.2%150/161 | 93.3%98/105 | 95.1%215/226 | 99.2%255/257 | 98.3%338/344 |
| LFM2.5-2.6B on · chat Open · 2.7B | — | Chat | 95.8%228/238 | 89.2%182/204 | 88.2%15/17 | 93.6%147/157 | 94.5%103/109 | 95.5%128/134 | 98.2%108/110 | 97.0%98/101 | 98.1%158/161 | 97.1%102/105 | 92.0%208/226 | 93.4%240/257 | 91.9%316/344 |
| Raon-1 nano off · chat Open · 3.1B | — | Chat | 91.2%217/238 | 75.5%154/204 | 94.1%16/17 | 87.3%137/157 | 89.9%98/109 | 93.3%125/134 | 92.7%102/110 | 92.1%93/101 | 85.7%138/161 | 91.4%96/105 | 91.2%206/226 | 93.0%239/257 | 91.9%316/344 |
| vera-medium-omni Open · 4.2B | — | Decision | 55.9%133/238 | 36.3%74/204 | 35.3%6/17 | 42.7%67/157 | 48.6%53/109 | 50.7%68/134 | 70.9%78/110 | 67.3%68/101 | 62.7%101/161 | 63.8%67/105 | 58.4%132/226 | 61.1%157/257 | 65.4%225/344 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 52.1%124/238 | 37.7%77/204 | 23.5%4/17 | 40.8%64/157 | 53.2%58/109 | 60.4%81/134 | 71.8%79/110 | 60.4%61/101 | 57.8%93/161 | 61.9%65/105 | 59.3%134/226 | 59.9%154/257 | 63.1%217/344 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 55.5%132/238 | 36.3%74/204 | 29.4%5/17 | 44.6%70/157 | 50.5%55/109 | 53.0%71/134 | 69.1%76/110 | 60.4%61/101 | 60.2%97/161 | 59.0%62/105 | 58.8%133/226 | 61.9%159/257 | 62.5%215/344 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 54.2%129/238 | 40.7%83/204 | 17.6%3/17 | 41.4%65/157 | 49.5%54/109 | 52.2%70/134 | 74.5%82/110 | 64.4%65/101 | 59.6%96/161 | 58.1%61/105 | 55.8%126/226 | 61.5%158/257 | 62.8%216/344 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 51.7%123/238 | 39.2%80/204 | 29.4%5/17 | 40.8%64/157 | 52.3%57/109 | 57.5%77/134 | 70.9%78/110 | 63.4%64/101 | 56.5%91/161 | 57.1%60/105 | 59.7%135/226 | 61.9%159/257 | 58.7%202/344 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 55.9%133/238 | 34.3%70/204 | 17.6%3/17 | 33.8%53/157 | 54.1%59/109 | 44.0%59/134 | 64.5%71/110 | 60.4%61/101 | 50.3%81/161 | 45.7%48/105 | 55.3%125/226 | 59.9%154/257 | 57.3%197/344 |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 47.1%112/238 | 34.3%70/204 | 35.3%6/17 | 43.3%68/157 | 39.4%43/109 | 49.3%66/134 | 62.7%69/110 | 56.4%57/101 | 56.5%91/161 | 45.7%48/105 | 49.1%111/226 | 55.6%143/257 | 60.2%207/344 |
| Metask-Jev-4B Open · 4.5B | — | Decision | 40.3%96/238 | 31.4%64/204 | 11.8%2/17 | 30.6%48/157 | 32.1%35/109 | 32.1%43/134 | 39.1%43/110 | 42.6%43/101 | 43.5%70/161 | 34.3%36/105 | 38.1%86/226 | 47.9%123/257 | 45.9%158/344 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 33.6%80/238 | 25.0%51/204 | 35.3%6/17 | 32.5%51/157 | 34.9%38/109 | 32.8%44/134 | 50.0%55/110 | 54.5%55/101 | 46.0%74/161 | 46.7%49/105 | 40.3%91/226 | 43.6%112/257 | 37.8%130/344 |
| d1-2B Open · 1.9B | — | Decision | 29.4%70/238 | 33.3%68/204 | 35.3%6/17 | 37.6%59/157 | 35.8%39/109 | 34.3%46/134 | 46.4%51/110 | 39.6%40/101 | 32.9%53/161 | 39.0%41/105 | 41.6%94/226 | 40.5%104/257 | 38.1%131/344 |
| GLiDE 2B Open · 1.9B | — | Decision | 32.8%78/238 | 27.9%57/204 | 23.5%4/17 | 30.6%48/157 | 36.7%40/109 | 32.8%44/134 | 40.9%45/110 | 40.6%41/101 | 38.5%62/161 | 42.9%45/105 | 35.8%81/226 | 42.4%109/257 | 41.0%141/344 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 31.1%74/238 | 25.5%52/204 | 11.8%2/17 | 29.9%47/157 | 26.6%29/109 | 35.1%47/134 | 40.0%44/110 | 51.5%52/101 | 37.3%60/161 | 35.2%37/105 | 37.2%84/226 | 38.1%98/257 | 39.8%137/344 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 30.3%72/238 | 27.0%55/204 | 29.4%5/17 | 29.9%47/157 | 33.9%37/109 | 26.1%35/134 | 26.4%29/110 | 34.7%35/101 | 36.0%58/161 | 23.8%25/105 | 29.6%67/226 | 34.6%89/257 | 40.7%140/344 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 27.7%66/238 | 20.6%42/204 | 29.4%5/17 | 23.6%37/157 | 35.8%39/109 | 29.1%39/134 | 38.2%42/110 | 42.6%43/101 | 36.0%58/161 | 27.6%29/105 | 30.5%69/226 | 28.4%73/257 | 29.1%100/344 |
| Strands Decider 2B Open · 2.0B | — | Decision | 29.8%71/238 | 24.0%49/204 | 23.5%4/17 | 25.5%40/157 | 30.3%33/109 | 27.6%37/134 | 30.9%34/110 | 37.6%38/101 | 31.7%51/161 | 25.7%27/105 | 29.6%67/226 | 25.3%65/257 | 30.2%104/344 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 26.5%63/238 | 22.5%46/204 | 11.8%2/17 | 25.5%40/157 | 33.9%37/109 | 25.4%34/134 | 28.2%31/110 | 28.7%29/101 | 32.9%53/161 | 28.6%30/105 | 29.2%66/226 | 24.1%62/257 | 34.3%118/344 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 27.3%65/238 | 21.6%44/204 | 35.3%6/17 | 21.0%33/157 | 31.2%34/109 | 23.9%32/134 | 30.9%34/110 | 29.7%30/101 | 33.5%54/161 | 27.6%29/105 | 30.1%68/226 | 33.5%86/257 | 27.6%95/344 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 22.7%54/238 | 23.0%47/204 | 29.4%5/17 | 21.7%34/157 | 30.3%33/109 | 19.4%26/134 | 24.5%27/110 | 23.8%24/101 | 18.0%29/161 | 24.8%26/105 | 22.1%50/226 | 19.1%49/257 | 25.0%86/344 |
| GLiDE 0.4B Open · 0.4B | — | Decision | 19.3%46/238 | 17.6%36/204 | 5.9%1/17 | 23.6%37/157 | 25.7%28/109 | 18.7%25/134 | 10.9%12/110 | 22.8%23/101 | 23.6%38/161 | 19.0%20/105 | 22.6%51/226 | 18.3%47/257 | 23.5%81/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Accuracy 99.4% (95%: 99.0%–99.7%) Logic 92.4% | — | Chat | 99.4%99.0%–99.7% | a | 2,151 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Accuracy 99.2% (mean of 3 seeds; min–max 99.2%–99.3%) Logic 88.9% | — | Chat | 99.2%99.2%–99.3% min–max of 3 seeds | ab | 2,147 / 2,163seed 0 run | Not measured | Not measured | — | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Accuracy 99.2% (95%: 98.7%–99.5%) Logic 91.6% | — | Chat | 99.2%98.7%–99.5% | ab | 2,146 / 2,163 | Not measured | Not measured | — | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Accuracy 98.9% (mean of 3 seeds; min–max 98.8%–99.0%) Logic 84.6% | — | Chat | 98.9%98.8%–99.0% min–max of 3 seeds | abc | 2,142 / 2,163seed 0 run | Not measured | Not measured | — | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Accuracy 98.9% (mean of 3 seeds; min–max 98.8%–99.1%) Logic 84.2% | — | Chat | 98.9%98.8%–99.1% min–max of 3 seeds | abcd | 2,139 / 2,163seed 0 run | Not measured | Not measured | — | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Accuracy 98.7% (95%: 98.1%–99.1%) Logic 85.5% | — | Chat | 98.7%98.1%–99.1% | abcde | 2,135 / 2,163 | Not measured | Not measured | — | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Accuracy 98.5% (95%: 97.9%–98.9%) Logic 90.1% | — | Chat | 98.5%97.9%–98.9% | bcde | 2,130 / 2,163 | Not measured | Not measured | — | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Accuracy 98.2% (mean of 3 seeds; min–max 98.1%–98.3%) Logic 83.6% | — | Chat | 98.2%98.1%–98.3% min–max of 3 seeds | cdef | 2,126 / 2,163seed 0 run | Not measured | Not measured | — | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 97.8% (95%: 97.1%–98.4%) Logic 75.9% | — | Chat | 97.8%97.1%–98.4% | defg | 2,116 / 2,163 | Not measured | Not measured | — | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 97.7% (mean of 3 seeds; min–max 97.6%–97.8%) Logic 76.8% | — | Chat | 97.7%97.6%–97.8% min–max of 3 seeds | efg | 2,111 / 2,163seed 0 run | Not measured | Not measured | — | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 97.0% (95%: 96.2%–97.6%) Logic 70.9% | — | Chat | 97.0%96.2%–97.6% | fg | 2,098 / 2,163 | Not measured | Not measured | — | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 96.9% (95%: 96.1%–97.6%) Logic 74.6% | — | Chat | 96.9%96.1%–97.6% | fg | 2,097 / 2,163 | Not measured | Not measured | — | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 96.8% (95%: 95.9%–97.4%) Logic 72.9% | — | Chat | 96.8%95.9%–97.4% | g | 2,093 / 2,163 | Not measured | Not measured | — | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Accuracy 94.7% (95%: 93.7%–95.6%) Logic 70.3% | — | Chat | 94.7%93.7%–95.6% | h | 2,049 / 2,163 | Not measured | Not measured | — | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Accuracy 94.6% (95%: 93.6%–95.5%) Logic 64.8% | — | Chat | 94.6%93.6%–95.5% | h | 2,047 / 2,163 | Not measured | Not measured | — | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Accuracy 94.0% (95%: 92.9%–94.9%) Logic 75.4% | — | Chat | 94.0%92.9%–94.9% | h | 2,033 / 2,163 | Not measured | Not measured | — | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 89.6% (95%: 88.2%–90.8%) Logic 65.1% | — | Chat | 89.6%88.2%–90.8% | i | 1,937 / 2,163 | Not measured | Not measured | — | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Accuracy 56.8% (95%: 54.7%–58.9%) Logic 44.1% | — | Decision | 56.8%54.7%–58.9% | j | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 69.18chosen-answer Brier 0.1846 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Accuracy 56.0% (95%: 53.9%–58.1%) Logic 42.3% | — | Decision | 56.0%53.9%–58.1% | j | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 67.89chosen-answer Brier 0.2021 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Accuracy 55.9% (95%: 53.8%–58.0%) Logic 44.7% | — | Decision | 55.9%53.8%–58.0% | j | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 67.89chosen-answer Brier 0.2015 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Accuracy 55.8% (95%: 53.7%–57.9%) Logic 45.9% | — | Decision | 55.8%53.7%–57.9% | j | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 68.25chosen-answer Brier 0.1934 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Accuracy 55.2% (95%: 53.1%–57.3%) Logic 43.1% | — | Decision | 55.2%53.1%–57.3% | j | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 67.61chosen-answer Brier 0.2002 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 51.5% (95%: 49.4%–53.6%) Logic 39.7% | — | Decision | 51.5%49.4%–53.6% | k | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 65.62chosen-answer Brier 0.2026 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 50.4% (95%: 48.3%–52.5%) Logic 40.4% | — | Decision | 50.4%48.3%–52.5% | k | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 64.11chosen-answer Brier 0.2221 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Accuracy 39.2% (95%: 37.1%–41.2%) Logic 32.1% | — | Decision | 39.2%37.1%–41.2% | l | 847 / 2,163 | No answers at this threshold | 0.651 | 58.63chosen-answer Brier 0.2190 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Accuracy 38.7% (95%: 36.6%–40.7%) Logic 30.8% | — | Decision | 38.7%36.6%–40.7% | l | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 58.95chosen-answer Brier 0.2075 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Accuracy 37.1% (95%: 35.1%–39.1%) Logic 30.7% | — | Decision | 37.1%35.1%–39.1% | l | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 57.72chosen-answer Brier 0.2163 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Accuracy 36.8% (95%: 34.7%–38.8%) Logic 32.3% | — | Decision | 36.8%34.7%–38.8% | l | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 55.70chosen-answer Brier 0.2536 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Accuracy 35.3% (95%: 33.3%–37.3%) Logic 31.7% | — | Decision | 35.3%33.3%–37.3% | lm | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 56.49chosen-answer Brier 0.2229 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Accuracy 32.1% (95%: 30.2%–34.1%) Logic 29.2% | — | Decision | 32.1%30.2%–34.1% | mn | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 53.21chosen-answer Brier 0.2566 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 29.7% (95%: 27.8%–31.6%) Logic 24.7% | — | Decision | 29.7%27.8%–31.6% | n | 642 / 2,163 | No answers at this threshold | 0.583 | 54.40chosen-answer Brier 0.2088 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Accuracy 28.7% (95%: 26.8%–30.6%) Logic 22.4% | — | Decision | 28.7%26.8%–30.6% | n | 620 / 2,163 | No answers at this threshold | 0.569 | 53.90chosen-answer Brier 0.2086 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Accuracy 28.2% (95%: 26.4%–30.2%) Logic 27.0% | — | Decision | 28.2%26.4%–30.2% | n | 611 / 2,163 | No answers at this threshold | 0.592 | 53.62chosen-answer Brier 0.2100 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 28.2% (95%: 26.3%–30.1%) Logic 26.7% | — | Decision | 28.2%26.3%–30.1% | n | 610 / 2,163 | No answers at this threshold | 0.596 | 53.60chosen-answer Brier 0.2100 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Accuracy 22.7% (95%: 20.9%–24.5%) Logic 23.6% | — | Decision | 22.7%20.9%–24.5% | o | 490 / 2,163 | No answers at this threshold | 0.501 | 52.01chosen-answer Brier 0.1863 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Accuracy 20.6% (95%: 18.9%–22.3%) Logic 23.2% | — | Decision | 20.6%18.9%–22.3% | o | 445 / 2,163 | No answers at this threshold | 0.500 | 50.98chosen-answer Brier 0.1862 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Google Liquid AI Alibaba eCloud Technology PelaAI newsbang Base Labs tattrongvu Patrick-18 StartLux Fastino Apertix AI Valen-Team Cloudflare EldanRing Blockbrain Labs Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B | — | Chat | 98.7%235/238 | 96.1%196/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 97.0%130/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | Chat | 97.5%232/238 | 97.1%198/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 96.3%129/134 | 99.1%109/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 99.6%225/226 | 98.8%254/257 | 99.7%343/344 |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B | — | Chat | 97.5%232/238 | 97.5%199/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 95.5%128/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 98.7%223/226 | 98.4%253/257 | 100.0%344/344 |
| Gemma 4 E4B on · chat Open · 8.0B | — | Chat | 97.5%232/238 | 94.1%192/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 94.8%127/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 97.1%102/105 | 99.1%224/226 | 98.8%254/257 | 100.0%344/344 |
| Gemma 4 E4B chat Open · 8.0B | — | Chat | 98.3%234/238 | 95.1%194/204 | 100.0%17/17 | 98.7%155/157 | 99.1%108/109 | 95.5%128/134 | 98.2%108/110 | 98.0%99/101 | 98.1%158/161 | 96.2%101/105 | 98.7%223/226 | 99.2%255/257 | 99.4%342/344 |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | Chat | 96.2%229/238 | 92.6%189/204 | 94.1%16/17 | 99.4%156/157 | 93.6%102/109 | 97.8%131/134 | 98.2%108/110 | 92.1%93/101 | 97.5%157/161 | 92.4%97/105 | 97.8%221/226 | 97.3%250/257 | 96.8%333/344 |
| Qwen3-14B off · chat Open · 15B | — | Chat | 94.5%225/238 | 83.3%170/204 | 100.0%17/17 | 96.8%152/157 | 99.1%108/109 | 94.8%127/134 | 98.2%108/110 | 95.0%96/101 | 91.3%147/161 | 100.0%105/105 | 99.1%224/226 | 98.4%253/257 | 98.8%340/344 |
| Erk-14B off · chat Open · 15B | — | Chat | 91.6%218/238 | 72.1%147/204 | 100.0%17/17 | 83.4%131/157 | 82.6%90/109 | 94.0%126/134 | 94.5%104/110 | 94.1%95/101 | 89.4%144/161 | 97.1%102/105 | 95.6%216/226 | 94.9%244/257 | 95.6%329/344 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 66.8%159/238 | 55.9%114/204 | 41.2%7/17 | 56.1%88/157 | 54.1%59/109 | 65.7%88/134 | 77.3%85/110 | 77.2%78/101 | 67.7%109/161 | 69.5%73/105 | 67.3%152/226 | 66.1%170/257 | 73.0%251/344 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 61.3%146/238 | 43.1%88/204 | 41.2%7/17 | 51.0%80/157 | 52.3%57/109 | 63.4%85/134 | 76.4%84/110 | 64.4%65/101 | 64.0%103/161 | 64.8%68/105 | 64.2%145/226 | 68.9%177/257 | 69.2%238/344 |
| Elephant-1 Open · 9.0B | — | Decision | 59.2%141/238 | 40.2%82/204 | 41.2%7/17 | 52.2%82/157 | 56.0%61/109 | 63.4%85/134 | 72.7%80/110 | 72.3%73/101 | 60.9%98/161 | 65.7%69/105 | 63.7%144/226 | 64.2%165/257 | 69.8%240/344 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 57.6%137/238 | 41.2%84/204 | 35.3%6/17 | 49.0%77/157 | 50.5%55/109 | 60.4%81/134 | 77.3%85/110 | 64.4%65/101 | 62.7%101/161 | 66.7%70/105 | 66.4%150/226 | 67.3%173/257 | 62.8%216/344 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 58.4%139/238 | 38.2%78/204 | 23.5%4/17 | 38.9%61/157 | 53.2%58/109 | 56.7%76/134 | 73.6%81/110 | 71.3%72/101 | 64.0%103/161 | 63.8%67/105 | 64.6%146/226 | 63.4%163/257 | 71.5%246/344 |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 59.7%142/238 | 42.2%86/204 | 17.6%3/17 | 51.0%80/157 | 53.2%58/109 | 61.2%82/134 | 72.7%80/110 | 63.4%64/101 | 66.5%107/161 | 61.0%64/105 | 61.5%139/226 | 66.1%170/257 | 63.4%218/344 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 56.3%134/238 | 39.7%81/204 | 35.3%6/17 | 42.7%67/157 | 56.0%61/109 | 53.7%72/134 | 72.7%80/110 | 70.3%71/101 | 60.2%97/161 | 61.0%64/105 | 64.2%145/226 | 60.7%156/257 | 69.2%238/344 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 59.2%141/238 | 38.2%78/204 | 35.3%6/17 | 46.5%73/157 | 49.5%54/109 | 54.5%73/134 | 70.9%78/110 | 65.3%66/101 | 64.6%104/161 | 62.9%66/105 | 60.6%137/226 | 62.3%160/257 | 66.3%228/344 |
| GLiDE 9B Open · 7.9B | — | Decision | 51.3%122/238 | 38.2%78/204 | 23.5%4/17 | 43.3%68/157 | 48.6%53/109 | 55.2%74/134 | 73.6%81/110 | 63.4%64/101 | 62.7%101/161 | 67.6%71/105 | 62.8%142/226 | 64.2%165/257 | 66.9%230/344 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 55.0%131/238 | 41.7%85/204 | 35.3%6/17 | 43.9%69/157 | 48.6%53/109 | 53.0%71/134 | 67.3%74/110 | 66.3%67/101 | 60.2%97/161 | 56.2%59/105 | 58.8%133/226 | 61.1%157/257 | 61.0%210/344 |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 52.9%126/238 | 34.8%71/204 | 47.1%8/17 | 46.5%73/157 | 45.9%50/109 | 53.0%71/134 | 61.8%68/110 | 65.3%66/101 | 54.7%88/161 | 57.1%60/105 | 58.4%132/226 | 65.0%167/257 | 63.1%217/344 |
| Clef-Flash Open · 9.0B | — | Decision | 49.6%118/238 | 33.3%68/204 | 52.9%9/17 | 42.0%66/157 | 43.1%47/109 | 42.5%57/134 | 57.3%63/110 | 59.4%60/101 | 57.8%93/161 | 53.3%56/105 | 49.1%111/226 | 55.3%142/257 | 54.4%187/344 |
| Winnow-12B Open · 12B | — | Decision | 51.3%122/238 | 31.9%65/204 | 41.2%7/17 | 35.7%56/157 | 38.5%42/109 | 47.0%63/134 | 61.8%68/110 | 67.3%68/101 | 52.2%84/161 | 45.7%48/105 | 45.6%103/226 | 46.3%119/257 | 49.1%169/344 |
| Cygnet Open · 12B | — | Decision | 43.7%104/238 | 30.9%63/204 | 41.2%7/17 | 33.1%52/157 | 39.4%43/109 | 36.6%49/134 | 50.9%56/110 | 59.4%60/101 | 48.4%78/161 | 45.7%48/105 | 49.6%112/226 | 47.9%123/257 | 47.7%164/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Accuracy 98.8% (95%: 98.3%–99.2%) Logic 87.7% | — | Chat | 98.8%98.3%–99.2% | a | 2,138 / 2,163 | Not measured | Not measured | — | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.5% (mean of 3 seeds; min–max 98.4%–98.8%) Logic 85.9% | — | Chat | 98.5%98.4%–98.8% min–max of 3 seeds | a | 2,136 / 2,163seed 0 run | Not measured | Not measured | — | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.5% (95%: 97.9%–99.0%) Logic 85.0% | — | Chat | 98.5%97.9%–99.0% | a | 2,131 / 2,163 | Not measured | Not measured | — | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.2% (95%: 97.6%–98.7%) Logic 82.6% | — | Chat | 98.2%97.6%–98.7% | a | 2,125 / 2,163 | Not measured | Not measured | — | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.1% (95%: 97.4%–98.6%) Logic 83.2% | — | Chat | 98.1%97.4%–98.6% | a | 2,122 / 2,163 | Not measured | Not measured | — | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Accuracy 96.3% (95%: 95.4%–97.0%) Logic 60.2% | — | Chat | 96.3%95.4%–97.0% | b | 2,082 / 2,163 | Not measured | Not measured | — | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Accuracy 95.8% (95%: 94.9%–96.6%) Logic 65.3% | — | Chat | 95.8%94.9%–96.6% | b | 2,072 / 2,163 | Not measured | Not measured | — | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Accuracy 90.8% (95%: 89.5%–91.9%) Logic 54.2% | — | Chat | 90.8%89.5%–91.9% | c | 1,963 / 2,163 | Not measured | Not measured | — | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Accuracy 66.3% (95%: 64.2%–68.2%) Logic 50.6% | — | Decision | 66.3%64.2%–68.2% | d | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 74.96chosen-answer Brier 0.1632 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 62.1% (95%: 60.0%–64.1%) Logic 46.4% | — | Decision | 62.1%60.0%–64.1% | e | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 72.28chosen-answer Brier 0.1752 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Accuracy 61.3% (95%: 59.3%–63.4%) Logic 46.4% | — | Decision | 61.3%59.3%–63.4% | ef | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 71.86chosen-answer Brier 0.1763 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 60.1% (95%: 58.0%–62.1%) Logic 43.9% | — | Decision | 60.1%58.0%–62.1% | efg | 1,300 / 2,163 | No answers at this threshold | 0.803 | 71.15chosen-answer Brier 0.1779 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 59.8% (95%: 57.7%–61.9%) Logic 48.2% | — | Decision | 59.8%57.7%–61.9% | efg | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 71.32chosen-answer Brier 0.1719 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 59.8% (95%: 57.7%–61.8%) Logic 44.3% | — | Decision | 59.8%57.7%–61.8% | efg | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 71.08chosen-answer Brier 0.1762 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Accuracy 58.8% (95%: 56.7%–60.9%) Logic 45.3% | — | Decision | 58.8%56.7%–60.9% | fgh | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 69.67chosen-answer Brier 0.1946 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Accuracy 58.4% (95%: 56.3%–60.5%) Logic 46.6% | — | Decision | 58.4%56.3%–60.5% | fgh | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 69.90chosen-answer Brier 0.1864 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Accuracy 57.9% (95%: 55.8%–60.0%) Logic 47.8% | — | Decision | 57.9%55.8%–60.0% | gh | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 69.69chosen-answer Brier 0.1855 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Accuracy 56.0% (95%: 53.9%–58.1%) Logic 43.3% | — | Decision | 56.0%53.9%–58.1% | h | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 67.15chosen-answer Brier 0.2174 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 55.3% (95%: 53.2%–57.4%) Logic 44.4% | — | Decision | 55.3%53.2%–57.4% | h | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 68.37chosen-answer Brier 0.1860 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Accuracy 49.8% (95%: 47.7%–51.9%) Logic 38.9% | — | Decision | 49.8%47.7%–51.9% | i | 1,077 / 2,163 | No answers at this threshold | 0.730 | 64.44chosen-answer Brier 0.2092 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Accuracy 46.9% (95%: 44.8%–49.0%) Logic 39.7% | — | Decision | 46.9%44.8%–49.0% | ij | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 63.00chosen-answer Brier 0.2087 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Accuracy 44.3% (95%: 42.3%–46.4%) Logic 39.3% | — | Decision | 44.3%42.3%–46.4% | j | 959 / 2,163 | No answers at this threshold | 0.735 | 61.87chosen-answer Brier 0.2060 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Google Alibaba Liquid AI MachinePulse PelaAI Poindexter Labs Sthānika AI Base Labs Perplexity Fastino vLLM Semantic Router Nace.AI Turion.AI Zefan Cai OmniJev Cloudflare Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B | — | Chat | 99.2%236/238 | 98.0%200/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 99.0%104/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| Qwen3.8-27B low · chat Open · 27B | — | Chat | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 98.7%155/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 99.0%104/105 | 99.1%224/226 | 99.6%256/257 | 99.4%342/344 |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | Chat | 98.7%235/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 96.3%105/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 99.6%225/226 | 98.4%253/257 | 99.7%343/344 |
| Karpo Decision 31B Open · 31B | — | Decision | 77.7%185/238 | 65.2%133/204 | 64.7%11/17 | 68.8%108/157 | 74.3%81/109 | 85.1%114/134 | 92.7%102/110 | 92.1%93/101 | 88.2%142/161 | 81.9%86/105 | 85.8%194/226 | 83.7%215/257 | 87.2%300/344 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 75.6%180/238 | 60.3%123/204 | 64.7%11/17 | 70.1%110/157 | 75.2%82/109 | 79.1%106/134 | 90.9%100/110 | 93.1%94/101 | 85.1%137/161 | 81.9%86/105 | 88.5%200/226 | 82.9%213/257 | 85.5%294/344 |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 76.9%183/238 | 58.8%120/204 | 52.9%9/17 | 63.7%100/157 | 70.6%77/109 | 76.1%102/134 | 88.2%97/110 | 89.1%90/101 | 82.0%132/161 | 79.0%83/105 | 85.0%192/226 | 80.2%206/257 | 82.3%283/344 |
| Copernicus-31B Open · 31B | — | Decision | 76.1%181/238 | 54.9%112/204 | 58.8%10/17 | 63.1%99/157 | 71.6%78/109 | 79.1%106/134 | 89.1%98/110 | 92.1%93/101 | 81.4%131/161 | 76.2%80/105 | 85.4%193/226 | 79.4%204/257 | 83.7%288/344 |
| Sieve 31B v3 Open · 31B | — | Decision | 75.6%180/238 | 55.9%114/204 | 52.9%9/17 | 63.7%100/157 | 69.7%76/109 | 81.3%109/134 | 90.9%100/110 | 86.1%87/101 | 80.1%129/161 | 80.0%84/105 | 82.7%187/226 | 79.4%204/257 | 84.0%289/344 |
| Mammoth-1 Open · 27B | — | Decision | 68.1%162/238 | 50.0%102/204 | 35.3%6/17 | 57.3%90/157 | 62.4%68/109 | 77.6%104/134 | 88.2%97/110 | 81.2%82/101 | 79.5%128/161 | 79.0%83/105 | 84.5%191/226 | 78.6%202/257 | 82.8%285/344 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 69.7%166/238 | 52.5%107/204 | 76.5%13/17 | 55.4%87/157 | 60.6%66/109 | 77.6%104/134 | 86.4%95/110 | 83.2%84/101 | 77.6%125/161 | 73.3%77/105 | 76.5%173/226 | 76.7%197/257 | 79.7%274/344 |
| GLiDE 27B Open · 27B | — | Decision | 69.3%165/238 | 52.0%106/204 | 52.9%9/17 | 61.8%97/157 | 60.6%66/109 | 73.1%98/134 | 90.9%100/110 | 81.2%82/101 | 77.6%125/161 | 70.5%74/105 | 77.4%175/226 | 73.9%190/257 | 78.5%270/344 |
| d3-ultra Open · 26B | — | Decision | 67.2%160/238 | 50.0%102/204 | 52.9%9/17 | 56.1%88/157 | 73.4%80/109 | 69.4%93/134 | 88.2%97/110 | 85.1%86/101 | 77.6%125/161 | 75.2%79/105 | 75.2%170/226 | 72.0%185/257 | 79.4%273/344 |
| d3 Open · 26B | — | Decision | 65.1%155/238 | 52.5%107/204 | 47.1%8/17 | 58.6%92/157 | 56.0%61/109 | 67.9%91/134 | 86.4%95/110 | 80.2%81/101 | 77.6%125/161 | 70.5%74/105 | 76.5%173/226 | 73.2%188/257 | 79.4%273/344 |
| Drex Max Open · 27B | — | Decision | 68.1%162/238 | 52.5%107/204 | 52.9%9/17 | 56.1%88/157 | 58.7%64/109 | 67.9%91/134 | 89.1%98/110 | 79.2%80/101 | 77.0%124/161 | 69.5%73/105 | 75.7%171/226 | 71.6%184/257 | 78.5%270/344 |
| GLiDE v1.2 Open · 27B | — | Decision | 67.2%160/238 | 52.9%108/204 | 52.9%9/17 | 56.7%89/157 | 60.6%66/109 | 73.9%99/134 | 88.2%97/110 | 78.2%79/101 | 72.0%116/161 | 71.4%75/105 | 75.2%170/226 | 70.4%181/257 | 76.2%262/344 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 66.0%157/238 | 50.5%103/204 | 52.9%9/17 | 55.4%87/157 | 60.6%66/109 | 72.4%97/134 | 83.6%92/110 | 76.2%77/101 | 74.5%120/161 | 70.5%74/105 | 77.4%175/226 | 70.8%182/257 | 78.5%270/344 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 68.1%162/238 | 48.5%99/204 | 58.8%10/17 | 56.7%89/157 | 61.5%67/109 | 67.9%91/134 | 82.7%91/110 | 80.2%81/101 | 75.2%121/161 | 66.7%70/105 | 77.9%176/226 | 70.4%181/257 | 75.9%261/344 |
| Turion Jev 31B Open · 31B | — | Decision | 70.2%167/238 | 41.2%84/204 | 41.2%7/17 | 56.1%88/157 | 56.9%62/109 | 76.1%102/134 | 81.8%90/110 | 84.2%85/101 | 76.4%123/161 | 71.4%75/105 | 73.0%165/226 | 73.2%188/257 | 74.7%257/344 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 55.5%132/238 | 40.7%83/204 | 29.4%5/17 | 49.7%78/157 | 51.4%56/109 | 57.5%77/134 | 70.0%77/110 | 65.3%66/101 | 64.0%103/161 | 61.9%65/105 | 62.4%141/226 | 64.6%166/257 | 66.0%227/344 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 48.7%116/238 | 36.3%74/204 | 47.1%8/17 | 48.4%76/157 | 38.5%42/109 | 47.0%63/134 | 73.6%81/110 | 68.3%69/101 | 62.1%100/161 | 53.3%56/105 | 56.2%127/226 | 61.5%158/257 | 61.0%210/344 |
| OneJev-27B Open · 27B | — | Decision | 52.9%126/238 | 32.8%67/204 | 47.1%8/17 | 43.9%69/157 | 38.5%42/109 | 47.0%63/134 | 67.3%74/110 | 65.3%66/101 | 60.2%97/161 | 55.2%58/105 | 54.0%122/226 | 59.9%154/257 | 63.1%217/344 |
| Clef Open · 27B | — | Decision | 54.6%130/238 | 36.3%74/204 | 52.9%9/17 | 38.9%61/157 | 40.4%44/109 | 36.6%49/134 | 59.1%65/110 | 57.4%58/101 | 57.8%93/161 | 49.5%52/105 | 53.1%120/226 | 54.1%139/257 | 61.3%211/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Accuracy 99.4% (95%: 99.0%–99.6%) Logic 93.3% | — | Chat | 99.4%99.0%–99.6% | a | 2,150 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Accuracy 99.2% (95%: 98.7%–99.5%) Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | a | 2,146 / 2,163 | Not measured | Not measured | — | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Accuracy 99.0% (95%: 98.5%–99.4%) Logic 86.7% | — | Chat | 99.0%98.5%–99.4% | a | 2,142 / 2,163 | Not measured | Not measured | — | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Accuracy 81.6% (95%: 79.9%–83.1%) Logic 60.7% | — | Decision | 81.6%79.9%–83.1% | b | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 85.99chosen-answer Brier 0.0958 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Accuracy 80.3% (95%: 78.5%–81.9%) Logic 58.9% | — | Decision | 80.3%78.5%–81.9% | b | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 85.05chosen-answer Brier 0.1016 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Accuracy 77.4% (95%: 75.6%–79.1%) Logic 59.0% | — | Decision | 77.4%75.6%–79.1% | c | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 83.18chosen-answer Brier 0.1103 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Accuracy 77.3% (95%: 75.5%–79.1%) Logic 58.7% | — | Decision | 77.3%75.5%–79.1% | c | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 83.06chosen-answer Brier 0.1122 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Accuracy 77.1% (95%: 75.3%–78.8%) Logic 57.9% | — | Decision | 77.1%75.3%–78.8% | cd | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 82.89chosen-answer Brier 0.1133 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Accuracy 74.0% (95%: 72.1%–75.8%) Logic 54.5% | — | Decision | 74.0%72.1%–75.8% | de | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 81.16chosen-answer Brier 0.1164 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Accuracy 72.5% (95%: 70.6%–74.3%) Logic 54.8% | — | Decision | 72.5%70.6%–74.3% | ef | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 79.32chosen-answer Brier 0.1385 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Accuracy 72.0% (95%: 70.1%–73.8%) Logic 58.4% | — | Decision | 72.0%70.1%–73.8% | efg | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 79.22chosen-answer Brier 0.1354 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 71.5% (95%: 69.6%–73.4%) Logic 54.6% | — | Decision | 71.5%69.6%–73.4% | efg | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 78.39chosen-answer Brier 0.1474 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 70.4% (95%: 68.5%–72.3%) Logic 54.5% | — | Decision | 70.4%68.5%–72.3% | fg | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 78.71chosen-answer Brier 0.1299 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Accuracy 70.3% (95%: 68.4%–72.2%) Logic 53.1% | — | Decision | 70.3%68.4%–72.2% | fg | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 77.10chosen-answer Brier 0.1611 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Accuracy 69.9% (95%: 67.9%–71.8%) Logic 55.2% | — | Decision | 69.9%67.9%–71.8% | fg | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 77.95chosen-answer Brier 0.1396 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Accuracy 69.8% (95%: 67.8%–71.7%) Logic 53.3% | — | Decision | 69.8%67.8%–71.7% | g | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 78.16chosen-answer Brier 0.1345 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Accuracy 69.3% (95%: 67.3%–71.2%) Logic 52.5% | with web | Decision | 69.3%67.3%–71.2% | g | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 77.94chosen-answer Brier 0.1342 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Accuracy 69.0% (95%: 67.0%–70.9%) Logic 52.3% | — | Decision | 69.0%67.0%–70.9% | g | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 77.52chosen-answer Brier 0.1399 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Accuracy 59.0% (95%: 56.9%–61.0%) Logic 41.9% | — | Decision | 59.0%56.9%–61.0% | h | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 70.98chosen-answer Brier 0.1703 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Accuracy 54.6% (95%: 52.4%–56.6%) Logic 42.6% | — | Decision | 54.6%52.4%–56.6% | i | 1,180 / 2,163 | No answers at this threshold | 0.764 | 64.53chosen-answer Brier 0.2549 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Accuracy 53.8% (95%: 51.7%–55.9%) Logic 40.6% | — | Decision | 53.8%51.7%–55.9% | i | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 67.54chosen-answer Brier 0.1869 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Accuracy 51.1% (95%: 49.0%–53.2%) Logic 40.8% | — | Decision | 51.1%49.0%–53.2% | i | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 65.20chosen-answer Brier 0.2070 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
DeepSeek Thinking Machines Xiaomi Z.ai MiniMax REAX Logits Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | Chat | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 99.0%100/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | Chat | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | Chat | 100.0%238/238 | 99.0%202/204 | 100.0%17/17 | 99.4%156/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 99.6%225/226 | 98.8%254/257 | 100.0%344/344 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active | — | Chat | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | Chat | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 98.1%154/157 | 98.2%107/109 | 97.8%131/134 | 98.2%108/110 | 99.0%100/101 | 98.1%158/161 | 100.0%105/105 | 100.0%226/226 | 98.4%253/257 | 99.7%343/344 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | Chat | 98.7%235/238 | 97.5%199/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 98.5%132/134 | 96.4%106/110 | 95.0%96/101 | 98.8%159/161 | 99.0%104/105 | 98.2%222/226 | 98.1%252/257 | 98.8%340/344 |
| REAX 1.3 Open · 57B | — | Decision | 76.9%183/238 | 57.8%118/204 | 64.7%11/17 | 61.8%97/157 | 76.1%83/109 | 76.1%102/134 | 92.7%102/110 | 84.2%85/101 | 83.9%135/161 | 79.0%83/105 | 85.4%193/226 | 77.8%200/257 | 85.8%295/344 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 71.4%170/238 | 51.5%105/204 | 64.7%11/17 | 59.9%94/157 | 75.2%82/109 | 75.4%101/134 | 90.0%99/110 | 85.1%86/101 | 81.4%131/161 | 75.2%79/105 | 81.4%184/226 | 78.6%202/257 | 82.3%283/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.2% | — | Chat | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | — | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Accuracy 99.6% (95%: 99.3%–99.8%) Logic 95.4% | — | Chat | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Accuracy 99.5% (95%: 99.2%–99.7%) Logic 95.6% | — | Chat | 99.5%99.2%–99.7% | a | 2,153 / 2,163 | Not measured | Not measured | — | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Accuracy 99.4% (95%: 99.0%–99.7%) Logic 95.1% | — | Chat | 99.4%99.0%–99.7% | ab | 2,151 / 2,163 | Not measured | Not measured | — | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Accuracy 99.3% (95%: 98.9%–99.6%) Logic running | — | Chat | 99.3%98.9%–99.6% | ab | 2,148 / 2,163 | Not measured | Not measured | — | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Accuracy 98.8% (95%: 98.2%–99.2%) Logic 93.2% | — | Chat | 98.8%98.2%–99.2% | bc | 2,137 / 2,163 | Not measured | Not measured | — | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Accuracy 98.2% (95%: 97.6%–98.7%) Logic 94.0% | — | Chat | 98.2%97.6%–98.7% | c | 2,125 / 2,163 | Not measured | Not measured | — | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Accuracy 78.0% (95%: 76.2%–79.7%) Logic 59.3% | — | Decision | 78.0%76.2%–79.7% | d | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 83.57chosen-answer Brier 0.1084 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Accuracy 75.2% (95%: 73.4%–77.0%) Logic 57.8% | — | Decision | 75.2%73.4%–77.0% | e | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 81.60chosen-answer Brier 0.1201 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Meta OpenAI Liquid AI Google Fastino Nace.AI Celeris Subquadratic TypeSafe Microsoft Inception Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 99.0%100/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| GPT-6 Luna chat Closed · Efficient | — | Chat | 90.8%216/238 | 81.4%166/204 | 82.4%14/17 | 94.3%148/157 | 89.9%98/109 | 96.3%129/134 | 96.4%106/110 | 93.1%94/101 | 93.2%150/161 | 98.1%103/105 | 96.9%219/226 | 96.9%249/257 | 97.1%334/344 |
| d1-ultra Closed · Efficient | — | Decision | 71.8%171/238 | 53.4%109/204 | 70.6%12/17 | 65.6%103/157 | 59.6%65/109 | 78.4%105/134 | 84.5%93/110 | 77.2%78/101 | 72.7%117/161 | 78.1%82/105 | 79.6%180/226 | 79.4%204/257 | 82.6%284/344 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 65.1%155/238 | 48.0%98/204 | 58.8%10/17 | 61.8%97/157 | 65.1%71/109 | 75.4%101/134 | 81.8%90/110 | 79.2%80/101 | 70.8%114/161 | 76.2%80/105 | 77.0%174/226 | 76.7%197/257 | 80.2%276/344 |
| GLiDE API Closed · Efficient | — | Decision | 67.2%160/238 | 54.4%111/204 | 47.1%8/17 | 56.7%89/157 | 59.6%65/109 | 71.6%96/134 | 88.2%97/110 | 80.2%81/101 | 73.9%119/161 | 75.2%79/105 | 74.3%168/226 | 75.1%193/257 | 77.0%265/344 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 69.7%166/238 | 52.9%108/204 | 58.8%10/17 | 54.8%86/157 | 56.0%61/109 | 67.9%91/134 | 88.2%97/110 | 80.2%81/101 | 75.8%122/161 | 72.4%76/105 | 76.1%172/226 | 70.4%181/257 | 77.0%265/344 |
| celeris-1-decision Closed · Efficient | — | Decision | 72.7%173/238 | 53.4%109/204 | 58.8%10/17 | 57.3%90/157 | 59.6%65/109 | 70.1%94/134 | 92.7%102/110 | 75.2%76/101 | 68.9%111/161 | 69.5%73/105 | 72.6%164/226 | 73.5%189/257 | 75.0%258/344 |
| celeris-1.1-decision Closed · Efficient | — | Decision | 71.4%170/238 | 53.4%109/204 | 52.9%9/17 | 58.6%92/157 | 59.6%65/109 | 70.1%94/134 | 92.7%102/110 | 74.3%75/101 | 68.9%111/161 | 70.5%74/105 | 72.1%163/226 | 73.5%189/257 | 75.0%258/344 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 65.1%155/238 | 51.5%105/204 | 52.9%9/17 | 52.9%83/157 | 67.0%73/109 | 70.1%94/134 | 82.7%91/110 | 76.2%77/101 | 74.5%120/161 | 72.4%76/105 | 74.8%169/226 | 68.1%175/257 | 74.4%256/344 |
| d1.1 Closed · Efficient | — | Decision | 63.0%150/238 | 44.1%90/204 | 41.2%7/17 | 50.3%79/157 | 56.0%61/109 | 63.4%85/134 | 72.7%80/110 | 73.3%74/101 | 64.6%104/161 | 66.7%70/105 | 72.1%163/226 | 75.9%195/257 | 73.3%252/344 |
| Jev 1.14 Closed · Efficient | — | Decision | 59.7%142/238 | 42.6%87/204 | 58.8%10/17 | 52.2%82/157 | 58.7%64/109 | 64.9%87/134 | 78.2%86/110 | 76.2%77/101 | 66.5%107/161 | 74.3%78/105 | 70.4%159/226 | 71.2%183/257 | 71.8%247/344 |
| Jev 1.13 Closed · Efficient | — | Decision | 56.7%135/238 | 47.5%97/204 | 41.2%7/17 | 51.0%80/157 | 55.0%60/109 | 63.4%85/134 | 78.2%86/110 | 76.2%77/101 | 64.0%103/161 | 74.3%78/105 | 69.9%158/226 | 70.0%180/257 | 69.8%240/344 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 61.3%146/238 | 40.2%82/204 | 23.5%4/17 | 48.4%76/157 | 52.3%57/109 | 53.7%72/134 | 70.0%77/110 | 63.4%64/101 | 60.9%98/161 | 61.9%65/105 | 66.4%150/226 | 63.8%164/257 | 66.6%229/344 |
| d1 Closed · Efficient | — | Decision | 52.1%124/238 | 39.7%81/204 | 35.3%6/17 | 40.8%64/157 | 42.2%46/109 | 51.5%69/134 | 59.1%65/110 | 63.4%64/101 | 60.9%98/161 | 61.9%65/105 | 58.0%131/226 | 61.9%159/257 | 67.2%231/344 |
| Mercury Decide Closed · Efficient | — | Decision | 56.7%135/238 | 36.3%74/204 | 41.2%7/17 | 38.9%61/157 | 44.0%48/109 | 44.0%59/134 | 67.3%74/110 | 75.2%76/101 | 57.1%92/161 | 56.2%59/105 | 50.9%115/226 | 58.8%151/257 | 59.9%206/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Accuracy 99.6% (95%: 99.2%–99.8%) Logic 95.5% | — | Chat | 99.6%99.2%–99.8% | a | 2,154 / 2,163 | Not measured | Not measured | — | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 93.7% (95%: 92.6%–94.6%) Logic 58.0% | — | Chat | 93.7%92.6%–94.6% | b | 2,026 / 2,163 | Not measured | Not measured | — | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Accuracy 74.1% (95%: 72.2%–75.9%) Logic 54.9% | — | Decision | 74.1%72.2%–75.9% | c | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 80.88chosen-answer Brier 0.1235 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Accuracy 71.3% (95%: 69.4%–73.2%) Logic 50.0% | — | Decision | 71.3%69.4%–73.2% | cd | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 78.84chosen-answer Brier 0.1366 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Accuracy 70.8% (95%: 68.8%–72.7%) Logic 54.5% | — | Decision | 70.8%68.8%–72.7% | d | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 78.56chosen-answer Brier 0.1367 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Accuracy 70.1% (95%: 68.1%–72.0%) Logic 54.8% | — | Decision | 70.1%68.1%–72.0% | d | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 78.01chosen-answer Brier 0.1407 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 70.0% (95%: 68.0%–71.9%) Logic 47.3% | — | Decision | 70.0%68.0%–71.9% | d | 1,514 / 2,163 | No answers at this threshold | 0.858 | 77.96chosen-answer Brier 0.1407 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 69.9% (95%: 67.9%–71.8%) Logic 47.3% | — | Decision | 69.9%67.9%–71.8% | d | 1,511 / 2,163 | No answers at this threshold | 0.859 | 77.91chosen-answer Brier 0.1403 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Accuracy 68.6% (95%: 66.6%–70.5%) Logic 52.4% | — | Decision | 68.6%66.6%–70.5% | d | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 76.15chosen-answer Brier 0.1626 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 65.2% (95%: 63.2%–67.2%) Logic 46.0% | — | Decision | 65.2%63.2%–67.2% | e | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 74.62chosen-answer Brier 0.1595 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 65.1% (95%: 63.1%–67.1%) Logic 48.2% | — | Decision | 65.1%63.1%–67.1% | e | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 74.56chosen-answer Brier 0.1602 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 64.1% (95%: 62.0%–66.1%) Logic 46.3% | — | Decision | 64.1%62.0%–66.1% | e | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 74.08chosen-answer Brier 0.1592 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Accuracy 59.4% (95%: 57.3%–61.4%) Logic 44.4% | — | Decision | 59.4%57.3%–61.4% | f | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 70.49chosen-answer Brier 0.1839 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 55.6% (95%: 53.5%–57.7%) Logic 40.8% | — | Decision | 55.6%53.5%–57.7% | g | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 68.16chosen-answer Brier 0.1930 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Accuracy 53.5% (95%: 51.4%–55.6%) Logic 45.1% | — | Decision | 53.5%51.4%–55.6% | g | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 66.02chosen-answer Brier 0.2146 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse | — | Chat | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.7% (95%: 99.3%–99.8%) Logic 95.5% | — | Chat | 99.7%99.3%–99.8% | a | 2,156 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship | — | Chat | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Claude Fable 5.1 chat Closed · Flagship | — | Chat | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 99.4%156/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.8% (95%: 99.5%–99.9%) Logic 96.2% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.7% (95%: 99.4%–99.9%) Logic 95.2% | — | Chat | 99.7%99.4%–99.9% | a | 2,157 / 2,163 | Not measured | Not measured | — | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Alibaba OpenAI Google xAI Liquid AI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed | — | Chat | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| GPT-6 Astra chat Closed · Undisclosed | — | Chat | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| GPT-6.1 Sol chat Closed · Undisclosed | — | Chat | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Gemini 4 Carbon chat Closed · Undisclosed | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 99.6%225/226 | 99.2%255/257 | 99.7%343/344 |
| Grok 4.7 chat Closed · Undisclosed | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | Chat | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 98.8%159/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| GPT-6.1 Terra chat Closed · Undisclosed | — | Chat | 99.6%237/238 | 98.5%201/204 | 100.0%17/17 | 99.4%156/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 98.1%158/161 | 100.0%105/105 | 99.6%225/226 | 98.8%254/257 | 99.7%343/344 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 67.2%160/238 | 46.1%94/204 | 47.1%8/17 | 54.8%86/157 | 59.6%65/109 | 75.4%101/134 | 84.5%93/110 | 82.2%83/101 | 75.8%122/161 | 69.5%73/105 | 75.2%170/226 | 73.2%188/257 | 77.9%268/344 |
| d1.2 Closed · Undisclosed | — | Decision | 62.6%149/238 | 40.2%82/204 | 41.2%7/17 | 52.9%83/157 | 60.6%66/109 | 65.7%88/134 | 75.5%83/110 | 78.2%79/101 | 66.5%107/161 | 67.6%71/105 | 72.1%163/226 | 73.2%188/257 | 73.0%251/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Accuracy 99.8% (95%: 99.5%–99.9%) Logic 96.3% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.7% | — | Chat | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.5% | — | Chat | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.6% (95%: 99.2%–99.8%) Logic 96.7% | — | Chat | 99.6%99.2%–99.8% | ab | 2,154 / 2,163 | Not measured | Not measured | — | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.5% (95%: 99.2%–99.7%) Logic running | — | Chat | 99.5%99.2%–99.7% | ab | 2,153 / 2,163 | Not measured | Not measured | — | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Accuracy 99.5% (95%: 99.1%–99.7%) Logic 94.0% | — | Chat | 99.5%99.1%–99.7% | ab | 2,152 / 2,163 | Not measured | Not measured | — | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.2% (95%: 98.7%–99.5%) Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | b | 2,146 / 2,163 | Not measured | Not measured | — | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 69.9% (95%: 67.9%–71.8%) Logic 51.6% | — | Decision | 69.9%67.9%–71.8% | c | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 78.23chosen-answer Brier 0.1340 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 65.5% (95%: 63.5%–67.5%) Logic 49.2% | — | Decision | 65.5%63.5%–67.5% | d | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 74.98chosen-answer Brier 0.1556 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
MachinePulse PelaAI REAX Poindexter Labs Sthānika AI Logits Liquid AI Base Labs Perplexity Fastino vLLM Semantic Router Google Nace.AI Celeris Turion.AI Subquadratic TypeSafe newsbang tattrongvu Microsoft Patrick-18 StartLux scar-ai Apertix AI Valen-Team Zefan Cai OmniJev Inception victorbennett Cloudflare quincyqiang EldanRing Blockbrain Labs Metask Lab Strands Agents Other Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | 77.7%185/238 | 65.2%133/204 | 64.7%11/17 | 68.8%108/157 | 74.3%81/109 | 85.1%114/134 | 92.7%102/110 | 92.1%93/101 | 88.2%142/161 | 81.9%86/105 | 85.8%194/226 | 83.7%215/257 | 87.2%300/344 |
| KnowLine-31B-Gen2 Open · 31B | — | 75.6%180/238 | 60.3%123/204 | 64.7%11/17 | 70.1%110/157 | 75.2%82/109 | 79.1%106/134 | 90.9%100/110 | 93.1%94/101 | 85.1%137/161 | 81.9%86/105 | 88.5%200/226 | 82.9%213/257 | 85.5%294/344 |
| REAX 1.3 Open · 57B | — | 76.9%183/238 | 57.8%118/204 | 64.7%11/17 | 61.8%97/157 | 76.1%83/109 | 76.1%102/134 | 92.7%102/110 | 84.2%85/101 | 83.9%135/161 | 79.0%83/105 | 85.4%193/226 | 77.8%200/257 | 85.8%295/344 |
| KnowLine-31B-Gen1.5 Open · 31B | — | 76.9%183/238 | 58.8%120/204 | 52.9%9/17 | 63.7%100/157 | 70.6%77/109 | 76.1%102/134 | 88.2%97/110 | 89.1%90/101 | 82.0%132/161 | 79.0%83/105 | 85.0%192/226 | 80.2%206/257 | 82.3%283/344 |
| Copernicus-31B Open · 31B | — | 76.1%181/238 | 54.9%112/204 | 58.8%10/17 | 63.1%99/157 | 71.6%78/109 | 79.1%106/134 | 89.1%98/110 | 92.1%93/101 | 81.4%131/161 | 76.2%80/105 | 85.4%193/226 | 79.4%204/257 | 83.7%288/344 |
| Sieve 31B v3 Open · 31B | — | 75.6%180/238 | 55.9%114/204 | 52.9%9/17 | 63.7%100/157 | 69.7%76/109 | 81.3%109/134 | 90.9%100/110 | 86.1%87/101 | 80.1%129/161 | 80.0%84/105 | 82.7%187/226 | 79.4%204/257 | 84.0%289/344 |
| Logits Sigma-1 Duo Open · 57B | — | 71.4%170/238 | 51.5%105/204 | 64.7%11/17 | 59.9%94/157 | 75.2%82/109 | 75.4%101/134 | 90.0%99/110 | 85.1%86/101 | 81.4%131/161 | 75.2%79/105 | 81.4%184/226 | 78.6%202/257 | 82.3%283/344 |
| d1-ultra Closed · Efficient | — | 71.8%171/238 | 53.4%109/204 | 70.6%12/17 | 65.6%103/157 | 59.6%65/109 | 78.4%105/134 | 84.5%93/110 | 77.2%78/101 | 72.7%117/161 | 78.1%82/105 | 79.6%180/226 | 79.4%204/257 | 82.6%284/344 |
| Mammoth-1 Open · 27B | — | 68.1%162/238 | 50.0%102/204 | 35.3%6/17 | 57.3%90/157 | 62.4%68/109 | 77.6%104/134 | 88.2%97/110 | 81.2%82/101 | 79.5%128/161 | 79.0%83/105 | 84.5%191/226 | 78.6%202/257 | 82.8%285/344 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 69.7%166/238 | 52.5%107/204 | 76.5%13/17 | 55.4%87/157 | 60.6%66/109 | 77.6%104/134 | 86.4%95/110 | 83.2%84/101 | 77.6%125/161 | 73.3%77/105 | 76.5%173/226 | 76.7%197/257 | 79.7%274/344 |
| GLiDE 27B Open · 27B | — | 69.3%165/238 | 52.0%106/204 | 52.9%9/17 | 61.8%97/157 | 60.6%66/109 | 73.1%98/134 | 90.9%100/110 | 81.2%82/101 | 77.6%125/161 | 70.5%74/105 | 77.4%175/226 | 73.9%190/257 | 78.5%270/344 |
| d3-ultra Open · 26B | — | 67.2%160/238 | 50.0%102/204 | 52.9%9/17 | 56.1%88/157 | 73.4%80/109 | 69.4%93/134 | 88.2%97/110 | 85.1%86/101 | 77.6%125/161 | 75.2%79/105 | 75.2%170/226 | 72.0%185/257 | 79.4%273/344 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 65.1%155/238 | 48.0%98/204 | 58.8%10/17 | 61.8%97/157 | 65.1%71/109 | 75.4%101/134 | 81.8%90/110 | 79.2%80/101 | 70.8%114/161 | 76.2%80/105 | 77.0%174/226 | 76.7%197/257 | 80.2%276/344 |
| GLiDE API Closed · Efficient | — | 67.2%160/238 | 54.4%111/204 | 47.1%8/17 | 56.7%89/157 | 59.6%65/109 | 71.6%96/134 | 88.2%97/110 | 80.2%81/101 | 73.9%119/161 | 75.2%79/105 | 74.3%168/226 | 75.1%193/257 | 77.0%265/344 |
| d3 Open · 26B | — | 65.1%155/238 | 52.5%107/204 | 47.1%8/17 | 58.6%92/157 | 56.0%61/109 | 67.9%91/134 | 86.4%95/110 | 80.2%81/101 | 77.6%125/161 | 70.5%74/105 | 76.5%173/226 | 73.2%188/257 | 79.4%273/344 |
| Drex Max Open · 27B | — | 68.1%162/238 | 52.5%107/204 | 52.9%9/17 | 56.1%88/157 | 58.7%64/109 | 67.9%91/134 | 89.1%98/110 | 79.2%80/101 | 77.0%124/161 | 69.5%73/105 | 75.7%171/226 | 71.6%184/257 | 78.5%270/344 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 69.7%166/238 | 52.9%108/204 | 58.8%10/17 | 54.8%86/157 | 56.0%61/109 | 67.9%91/134 | 88.2%97/110 | 80.2%81/101 | 75.8%122/161 | 72.4%76/105 | 76.1%172/226 | 70.4%181/257 | 77.0%265/344 |
| celeris-1-decision Closed · Efficient | — | 72.7%173/238 | 53.4%109/204 | 58.8%10/17 | 57.3%90/157 | 59.6%65/109 | 70.1%94/134 | 92.7%102/110 | 75.2%76/101 | 68.9%111/161 | 69.5%73/105 | 72.6%164/226 | 73.5%189/257 | 75.0%258/344 |
| GLiDE v1.2 Open · 27B | — | 67.2%160/238 | 52.9%108/204 | 52.9%9/17 | 56.7%89/157 | 60.6%66/109 | 73.9%99/134 | 88.2%97/110 | 78.2%79/101 | 72.0%116/161 | 71.4%75/105 | 75.2%170/226 | 70.4%181/257 | 76.2%262/344 |
| celeris-1.1-decision Closed · Efficient | — | 71.4%170/238 | 53.4%109/204 | 52.9%9/17 | 58.6%92/157 | 59.6%65/109 | 70.1%94/134 | 92.7%102/110 | 74.3%75/101 | 68.9%111/161 | 70.5%74/105 | 72.1%163/226 | 73.5%189/257 | 75.0%258/344 |
| d1.2-ultra Closed · Undisclosed | — | 67.2%160/238 | 46.1%94/204 | 47.1%8/17 | 54.8%86/157 | 59.6%65/109 | 75.4%101/134 | 84.5%93/110 | 82.2%83/101 | 75.8%122/161 | 69.5%73/105 | 75.2%170/226 | 73.2%188/257 | 77.9%268/344 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 66.0%157/238 | 50.5%103/204 | 52.9%9/17 | 55.4%87/157 | 60.6%66/109 | 72.4%97/134 | 83.6%92/110 | 76.2%77/101 | 74.5%120/161 | 70.5%74/105 | 77.4%175/226 | 70.8%182/257 | 78.5%270/344 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 68.1%162/238 | 48.5%99/204 | 58.8%10/17 | 56.7%89/157 | 61.5%67/109 | 67.9%91/134 | 82.7%91/110 | 80.2%81/101 | 75.2%121/161 | 66.7%70/105 | 77.9%176/226 | 70.4%181/257 | 75.9%261/344 |
| Turion Jev 31B Open · 31B | — | 70.2%167/238 | 41.2%84/204 | 41.2%7/17 | 56.1%88/157 | 56.9%62/109 | 76.1%102/134 | 81.8%90/110 | 84.2%85/101 | 76.4%123/161 | 71.4%75/105 | 73.0%165/226 | 73.2%188/257 | 74.7%257/344 |
| SubQ Snap 1.1 Closed · Efficient | — | 65.1%155/238 | 51.5%105/204 | 52.9%9/17 | 52.9%83/157 | 67.0%73/109 | 70.1%94/134 | 82.7%91/110 | 76.2%77/101 | 74.5%120/161 | 72.4%76/105 | 74.8%169/226 | 68.1%175/257 | 74.4%256/344 |
| KnowLine-12B-Gen1 Open · 12B | — | 66.8%159/238 | 55.9%114/204 | 41.2%7/17 | 56.1%88/157 | 54.1%59/109 | 65.7%88/134 | 77.3%85/110 | 77.2%78/101 | 67.7%109/161 | 69.5%73/105 | 67.3%152/226 | 66.1%170/257 | 73.0%251/344 |
| d1.2 Closed · Undisclosed | — | 62.6%149/238 | 40.2%82/204 | 41.2%7/17 | 52.9%83/157 | 60.6%66/109 | 65.7%88/134 | 75.5%83/110 | 78.2%79/101 | 66.5%107/161 | 67.6%71/105 | 72.1%163/226 | 73.2%188/257 | 73.0%251/344 |
| d1.1 Closed · Efficient | — | 63.0%150/238 | 44.1%90/204 | 41.2%7/17 | 50.3%79/157 | 56.0%61/109 | 63.4%85/134 | 72.7%80/110 | 73.3%74/101 | 64.6%104/161 | 66.7%70/105 | 72.1%163/226 | 75.9%195/257 | 73.3%252/344 |
| Jev 1.14 Closed · Efficient | — | 59.7%142/238 | 42.6%87/204 | 58.8%10/17 | 52.2%82/157 | 58.7%64/109 | 64.9%87/134 | 78.2%86/110 | 76.2%77/101 | 66.5%107/161 | 74.3%78/105 | 70.4%159/226 | 71.2%183/257 | 71.8%247/344 |
| Jev 1.13 Closed · Efficient | — | 56.7%135/238 | 47.5%97/204 | 41.2%7/17 | 51.0%80/157 | 55.0%60/109 | 63.4%85/134 | 78.2%86/110 | 76.2%77/101 | 64.0%103/161 | 74.3%78/105 | 69.9%158/226 | 70.0%180/257 | 69.8%240/344 |
| Adjudex 9B v5 Open · 9.0B | — | 61.3%146/238 | 43.1%88/204 | 41.2%7/17 | 51.0%80/157 | 52.3%57/109 | 63.4%85/134 | 76.4%84/110 | 64.4%65/101 | 64.0%103/161 | 64.8%68/105 | 64.2%145/226 | 68.9%177/257 | 69.2%238/344 |
| Elephant-1 Open · 9.0B | — | 59.2%141/238 | 40.2%82/204 | 41.2%7/17 | 52.2%82/157 | 56.0%61/109 | 63.4%85/134 | 72.7%80/110 | 72.3%73/101 | 60.9%98/161 | 65.7%69/105 | 63.7%144/226 | 64.2%165/257 | 69.8%240/344 |
| KnowLine-9B-Gen2 Open · 9.0B | — | 57.6%137/238 | 41.2%84/204 | 35.3%6/17 | 49.0%77/157 | 50.5%55/109 | 60.4%81/134 | 77.3%85/110 | 64.4%65/101 | 62.7%101/161 | 66.7%70/105 | 66.4%150/226 | 67.3%173/257 | 62.8%216/344 |
| Jeffan-9B-1211 Open · 9.0B | — | 58.4%139/238 | 38.2%78/204 | 23.5%4/17 | 38.9%61/157 | 53.2%58/109 | 56.7%76/134 | 73.6%81/110 | 71.3%72/101 | 64.0%103/161 | 63.8%67/105 | 64.6%146/226 | 63.4%163/257 | 71.5%246/344 |
| KnowLine-9B-Gen1 Open · 9.0B | — | 59.7%142/238 | 42.2%86/204 | 17.6%3/17 | 51.0%80/157 | 53.2%58/109 | 61.2%82/134 | 72.7%80/110 | 63.4%64/101 | 66.5%107/161 | 61.0%64/105 | 61.5%139/226 | 66.1%170/257 | 63.4%218/344 |
| Microsoft-Decision-1 Closed · Efficient | — | 61.3%146/238 | 40.2%82/204 | 23.5%4/17 | 48.4%76/157 | 52.3%57/109 | 53.7%72/134 | 70.0%77/110 | 63.4%64/101 | 60.9%98/161 | 61.9%65/105 | 66.4%150/226 | 63.8%164/257 | 66.6%229/344 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 55.5%132/238 | 40.7%83/204 | 29.4%5/17 | 49.7%78/157 | 51.4%56/109 | 57.5%77/134 | 70.0%77/110 | 65.3%66/101 | 64.0%103/161 | 61.9%65/105 | 62.4%141/226 | 64.6%166/257 | 66.0%227/344 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 56.3%134/238 | 39.7%81/204 | 35.3%6/17 | 42.7%67/157 | 56.0%61/109 | 53.7%72/134 | 72.7%80/110 | 70.3%71/101 | 60.2%97/161 | 61.0%64/105 | 64.2%145/226 | 60.7%156/257 | 69.2%238/344 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 59.2%141/238 | 38.2%78/204 | 35.3%6/17 | 46.5%73/157 | 49.5%54/109 | 54.5%73/134 | 70.9%78/110 | 65.3%66/101 | 64.6%104/161 | 62.9%66/105 | 60.6%137/226 | 62.3%160/257 | 66.3%228/344 |
| GLiDE 9B Open · 7.9B | — | 51.3%122/238 | 38.2%78/204 | 23.5%4/17 | 43.3%68/157 | 48.6%53/109 | 55.2%74/134 | 73.6%81/110 | 63.4%64/101 | 62.7%101/161 | 67.6%71/105 | 62.8%142/226 | 64.2%165/257 | 66.9%230/344 |
| vera-medium-omni Open · 4.2B | — | 55.9%133/238 | 36.3%74/204 | 35.3%6/17 | 42.7%67/157 | 48.6%53/109 | 50.7%68/134 | 70.9%78/110 | 67.3%68/101 | 62.7%101/161 | 63.8%67/105 | 58.4%132/226 | 61.1%157/257 | 65.4%225/344 |
| Cerno 9B v1.2 Open · 9.0B | — | 55.0%131/238 | 41.7%85/204 | 35.3%6/17 | 43.9%69/157 | 48.6%53/109 | 53.0%71/134 | 67.3%74/110 | 66.3%67/101 | 60.2%97/161 | 56.2%59/105 | 58.8%133/226 | 61.1%157/257 | 61.0%210/344 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 52.1%124/238 | 37.7%77/204 | 23.5%4/17 | 40.8%64/157 | 53.2%58/109 | 60.4%81/134 | 71.8%79/110 | 60.4%61/101 | 57.8%93/161 | 61.9%65/105 | 59.3%134/226 | 59.9%154/257 | 63.1%217/344 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 55.5%132/238 | 36.3%74/204 | 29.4%5/17 | 44.6%70/157 | 50.5%55/109 | 53.0%71/134 | 69.1%76/110 | 60.4%61/101 | 60.2%97/161 | 59.0%62/105 | 58.8%133/226 | 61.9%159/257 | 62.5%215/344 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 54.2%129/238 | 40.7%83/204 | 17.6%3/17 | 41.4%65/157 | 49.5%54/109 | 52.2%70/134 | 74.5%82/110 | 64.4%65/101 | 59.6%96/161 | 58.1%61/105 | 55.8%126/226 | 61.5%158/257 | 62.8%216/344 |
| d1 Closed · Efficient | — | 52.1%124/238 | 39.7%81/204 | 35.3%6/17 | 40.8%64/157 | 42.2%46/109 | 51.5%69/134 | 59.1%65/110 | 63.4%64/101 | 60.9%98/161 | 61.9%65/105 | 58.0%131/226 | 61.9%159/257 | 67.2%231/344 |
| Valen-Decision-1.5-9B Open · 9.0B | — | 52.9%126/238 | 34.8%71/204 | 47.1%8/17 | 46.5%73/157 | 45.9%50/109 | 53.0%71/134 | 61.8%68/110 | 65.3%66/101 | 54.7%88/161 | 57.1%60/105 | 58.4%132/226 | 65.0%167/257 | 63.1%217/344 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 51.7%123/238 | 39.2%80/204 | 29.4%5/17 | 40.8%64/157 | 52.3%57/109 | 57.5%77/134 | 70.9%78/110 | 63.4%64/101 | 56.5%91/161 | 57.1%60/105 | 59.7%135/226 | 61.9%159/257 | 58.7%202/344 |
| Open-Jev-27B-v1.1 Open · 27B | — | 48.7%116/238 | 36.3%74/204 | 47.1%8/17 | 48.4%76/157 | 38.5%42/109 | 47.0%63/134 | 73.6%81/110 | 68.3%69/101 | 62.1%100/161 | 53.3%56/105 | 56.2%127/226 | 61.5%158/257 | 61.0%210/344 |
| OneJev-27B Open · 27B | — | 52.9%126/238 | 32.8%67/204 | 47.1%8/17 | 43.9%69/157 | 38.5%42/109 | 47.0%63/134 | 67.3%74/110 | 65.3%66/101 | 60.2%97/161 | 55.2%58/105 | 54.0%122/226 | 59.9%154/257 | 63.1%217/344 |
| Mercury Decide Closed · Efficient | — | 56.7%135/238 | 36.3%74/204 | 41.2%7/17 | 38.9%61/157 | 44.0%48/109 | 44.0%59/134 | 67.3%74/110 | 75.2%76/101 | 57.1%92/161 | 56.2%59/105 | 50.9%115/226 | 58.8%151/257 | 59.9%206/344 |
| Verdin-4B-v1.1 Open · 4.0B | — | 55.9%133/238 | 34.3%70/204 | 17.6%3/17 | 33.8%53/157 | 54.1%59/109 | 44.0%59/134 | 64.5%71/110 | 60.4%61/101 | 50.3%81/161 | 45.7%48/105 | 55.3%125/226 | 59.9%154/257 | 57.3%197/344 |
| Clef Open · 27B | — | 54.6%130/238 | 36.3%74/204 | 52.9%9/17 | 38.9%61/157 | 40.4%44/109 | 36.6%49/134 | 59.1%65/110 | 57.4%58/101 | 57.8%93/161 | 49.5%52/105 | 53.1%120/226 | 54.1%139/257 | 61.3%211/344 |
| Chinese Decision Pointer 4B Open · 4.0B | — | 47.1%112/238 | 34.3%70/204 | 35.3%6/17 | 43.3%68/157 | 39.4%43/109 | 49.3%66/134 | 62.7%69/110 | 56.4%57/101 | 56.5%91/161 | 45.7%48/105 | 49.1%111/226 | 55.6%143/257 | 60.2%207/344 |
| Clef-Flash Open · 9.0B | — | 49.6%118/238 | 33.3%68/204 | 52.9%9/17 | 42.0%66/157 | 43.1%47/109 | 42.5%57/134 | 57.3%63/110 | 59.4%60/101 | 57.8%93/161 | 53.3%56/105 | 49.1%111/226 | 55.3%142/257 | 54.4%187/344 |
| Winnow-12B Open · 12B | — | 51.3%122/238 | 31.9%65/204 | 41.2%7/17 | 35.7%56/157 | 38.5%42/109 | 47.0%63/134 | 61.8%68/110 | 67.3%68/101 | 52.2%84/161 | 45.7%48/105 | 45.6%103/226 | 46.3%119/257 | 49.1%169/344 |
| Cygnet Open · 12B | — | 43.7%104/238 | 30.9%63/204 | 41.2%7/17 | 33.1%52/157 | 39.4%43/109 | 36.6%49/134 | 50.9%56/110 | 59.4%60/101 | 48.4%78/161 | 45.7%48/105 | 49.6%112/226 | 47.9%123/257 | 47.7%164/344 |
| Metask-Jev-4B Open · 4.5B | — | 40.3%96/238 | 31.4%64/204 | 11.8%2/17 | 30.6%48/157 | 32.1%35/109 | 32.1%43/134 | 39.1%43/110 | 42.6%43/101 | 43.5%70/161 | 34.3%36/105 | 38.1%86/226 | 47.9%123/257 | 45.9%158/344 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 33.6%80/238 | 25.0%51/204 | 35.3%6/17 | 32.5%51/157 | 34.9%38/109 | 32.8%44/134 | 50.0%55/110 | 54.5%55/101 | 46.0%74/161 | 46.7%49/105 | 40.3%91/226 | 43.6%112/257 | 37.8%130/344 |
| d1-2B Open · 1.9B | — | 29.4%70/238 | 33.3%68/204 | 35.3%6/17 | 37.6%59/157 | 35.8%39/109 | 34.3%46/134 | 46.4%51/110 | 39.6%40/101 | 32.9%53/161 | 39.0%41/105 | 41.6%94/226 | 40.5%104/257 | 38.1%131/344 |
| GLiDE 2B Open · 1.9B | — | 32.8%78/238 | 27.9%57/204 | 23.5%4/17 | 30.6%48/157 | 36.7%40/109 | 32.8%44/134 | 40.9%45/110 | 40.6%41/101 | 38.5%62/161 | 42.9%45/105 | 35.8%81/226 | 42.4%109/257 | 41.0%141/344 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 31.1%74/238 | 25.5%52/204 | 11.8%2/17 | 29.9%47/157 | 26.6%29/109 | 35.1%47/134 | 40.0%44/110 | 51.5%52/101 | 37.3%60/161 | 35.2%37/105 | 37.2%84/226 | 38.1%98/257 | 39.8%137/344 |
| GLiDE 0.8B Open · 0.8B | — | 30.3%72/238 | 27.0%55/204 | 29.4%5/17 | 29.9%47/157 | 33.9%37/109 | 26.1%35/134 | 26.4%29/110 | 34.7%35/101 | 36.0%58/161 | 23.8%25/105 | 29.6%67/226 | 34.6%89/257 | 40.7%140/344 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 27.7%66/238 | 20.6%42/204 | 29.4%5/17 | 23.6%37/157 | 35.8%39/109 | 29.1%39/134 | 38.2%42/110 | 42.6%43/101 | 36.0%58/161 | 27.6%29/105 | 30.5%69/226 | 28.4%73/257 | 29.1%100/344 |
| Strands Decider 2B Open · 2.0B | — | 29.8%71/238 | 24.0%49/204 | 23.5%4/17 | 25.5%40/157 | 30.3%33/109 | 27.6%37/134 | 30.9%34/110 | 37.6%38/101 | 31.7%51/161 | 25.7%27/105 | 29.6%67/226 | 25.3%65/257 | 30.2%104/344 |
| OneJev1.6-0.8B Open · 0.8B | — | 26.5%63/238 | 22.5%46/204 | 11.8%2/17 | 25.5%40/157 | 33.9%37/109 | 25.4%34/134 | 28.2%31/110 | 28.7%29/101 | 32.9%53/161 | 28.6%30/105 | 29.2%66/226 | 24.1%62/257 | 34.3%118/344 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 27.3%65/238 | 21.6%44/204 | 35.3%6/17 | 21.0%33/157 | 31.2%34/109 | 23.9%32/134 | 30.9%34/110 | 29.7%30/101 | 33.5%54/161 | 27.6%29/105 | 30.1%68/226 | 33.5%86/257 | 27.6%95/344 |
| GLiClass v4 Edge Open · 0.4B | — | 22.7%54/238 | 23.0%47/204 | 29.4%5/17 | 21.7%34/157 | 30.3%33/109 | 19.4%26/134 | 24.5%27/110 | 23.8%24/101 | 18.0%29/161 | 24.8%26/105 | 22.1%50/226 | 19.1%49/257 | 25.0%86/344 |
| GLiDE 0.4B Open · 0.4B | — | 19.3%46/238 | 17.6%36/204 | 5.9%1/17 | 23.6%37/157 | 25.7%28/109 | 18.7%25/134 | 10.9%12/110 | 22.8%23/101 | 23.6%38/161 | 19.0%20/105 | 22.6%51/226 | 18.3%47/257 | 23.5%81/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Accuracy 81.6% (95%: 79.9%–83.1%) Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 85.99chosen-answer Brier 0.0958 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Accuracy 80.3% (95%: 78.5%–81.9%) Logic 58.9% | — | 80.3%78.5%–81.9% | ab | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 85.05chosen-answer Brier 0.1016 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Accuracy 78.0% (95%: 76.2%–79.7%) Logic 59.3% | — | 78.0%76.2%–79.7% | bc | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 83.57chosen-answer Brier 0.1084 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Accuracy 77.4% (95%: 75.6%–79.1%) Logic 59.0% | — | 77.4%75.6%–79.1% | cd | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 83.18chosen-answer Brier 0.1103 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Accuracy 77.3% (95%: 75.5%–79.1%) Logic 58.7% | — | 77.3%75.5%–79.1% | cd | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 83.06chosen-answer Brier 0.1122 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Accuracy 77.1% (95%: 75.3%–78.8%) Logic 57.9% | — | 77.1%75.3%–78.8% | cd | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 82.89chosen-answer Brier 0.1133 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Accuracy 75.2% (95%: 73.4%–77.0%) Logic 57.8% | — | 75.2%73.4%–77.0% | de | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 81.60chosen-answer Brier 0.1201 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Accuracy 74.1% (95%: 72.2%–75.9%) Logic 54.9% | — | 74.1%72.2%–75.9% | defg | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 80.88chosen-answer Brier 0.1235 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Accuracy 74.0% (95%: 72.1%–75.8%) Logic 54.5% | — | 74.0%72.1%–75.8% | defh | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 81.16chosen-answer Brier 0.1164 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Accuracy 72.5% (95%: 70.6%–74.3%) Logic 54.8% | — | 72.5%70.6%–74.3% | efghi | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 79.32chosen-answer Brier 0.1385 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Accuracy 72.0% (95%: 70.1%–73.8%) Logic 58.4% | — | 72.0%70.1%–73.8% | efghij | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 79.22chosen-answer Brier 0.1354 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 71.5% (95%: 69.6%–73.4%) Logic 54.6% | — | 71.5%69.6%–73.4% | fghij | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 78.39chosen-answer Brier 0.1474 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Accuracy 71.3% (95%: 69.4%–73.2%) Logic 50.0% | — | 71.3%69.4%–73.2% | efghij | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 78.84chosen-answer Brier 0.1366 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Accuracy 70.8% (95%: 68.8%–72.7%) Logic 54.5% | — | 70.8%68.8%–72.7% | fghij | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 78.56chosen-answer Brier 0.1367 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 70.4% (95%: 68.5%–72.3%) Logic 54.5% | — | 70.4%68.5%–72.3% | gij | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 78.71chosen-answer Brier 0.1299 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Accuracy 70.3% (95%: 68.4%–72.2%) Logic 53.1% | — | 70.3%68.4%–72.2% | gijk | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 77.10chosen-answer Brier 0.1611 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Accuracy 70.1% (95%: 68.1%–72.0%) Logic 54.8% | — | 70.1%68.1%–72.0% | ijk | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 78.01chosen-answer Brier 0.1407 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 70.0% (95%: 68.0%–71.9%) Logic 47.3% | — | 70.0%68.0%–71.9% | hijk | 1,514 / 2,163 | No answers at this threshold | 0.858 | 77.96chosen-answer Brier 0.1407 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Accuracy 69.9% (95%: 67.9%–71.8%) Logic 55.2% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 77.95chosen-answer Brier 0.1396 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 69.9% (95%: 67.9%–71.8%) Logic 47.3% | — | 69.9%67.9%–71.8% | ijkl | 1,511 / 2,163 | No answers at this threshold | 0.859 | 77.91chosen-answer Brier 0.1403 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 69.9% (95%: 67.9%–71.8%) Logic 51.6% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 78.23chosen-answer Brier 0.1340 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Accuracy 69.8% (95%: 67.8%–71.7%) Logic 53.3% | — | 69.8%67.8%–71.7% | ijkl | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 78.16chosen-answer Brier 0.1345 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Accuracy 69.3% (95%: 67.3%–71.2%) Logic 52.5% | with web | 69.3%67.3%–71.2% | jklm | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 77.94chosen-answer Brier 0.1342 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Accuracy 69.0% (95%: 67.0%–70.9%) Logic 52.3% | — | 69.0%67.0%–70.9% | ijklmn | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 77.52chosen-answer Brier 0.1399 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Accuracy 68.6% (95%: 66.6%–70.5%) Logic 52.4% | — | 68.6%66.6%–70.5% | jklmn | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 76.15chosen-answer Brier 0.1626 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Accuracy 66.3% (95%: 64.2%–68.2%) Logic 50.6% | — | 66.3%64.2%–68.2% | klmno | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 74.96chosen-answer Brier 0.1632 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 65.5% (95%: 63.5%–67.5%) Logic 49.2% | — | 65.5%63.5%–67.5% | lmnop | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 74.98chosen-answer Brier 0.1556 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 65.2% (95%: 63.2%–67.2%) Logic 46.0% | — | 65.2%63.2%–67.2% | mnop | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 74.62chosen-answer Brier 0.1595 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 65.1% (95%: 63.1%–67.1%) Logic 48.2% | — | 65.1%63.1%–67.1% | nop | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 74.56chosen-answer Brier 0.1602 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 64.1% (95%: 62.0%–66.1%) Logic 46.3% | — | 64.1%62.0%–66.1% | opq | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 74.08chosen-answer Brier 0.1592 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 62.1% (95%: 60.0%–64.1%) Logic 46.4% | — | 62.1%60.0%–64.1% | opqr | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 72.28chosen-answer Brier 0.1752 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Accuracy 61.3% (95%: 59.3%–63.4%) Logic 46.4% | — | 61.3%59.3%–63.4% | pqr | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 71.86chosen-answer Brier 0.1763 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 60.1% (95%: 58.0%–62.1%) Logic 43.9% | — | 60.1%58.0%–62.1% | qrs | 1,300 / 2,163 | No answers at this threshold | 0.803 | 71.15chosen-answer Brier 0.1779 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 59.8% (95%: 57.7%–61.9%) Logic 48.2% | — | 59.8%57.7%–61.9% | qrs | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 71.32chosen-answer Brier 0.1719 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 59.8% (95%: 57.7%–61.8%) Logic 44.3% | — | 59.8%57.7%–61.8% | qrs | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 71.08chosen-answer Brier 0.1762 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Accuracy 59.4% (95%: 57.3%–61.4%) Logic 44.4% | — | 59.4%57.3%–61.4% | rst | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 70.49chosen-answer Brier 0.1839 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Accuracy 59.0% (95%: 56.9%–61.0%) Logic 41.9% | — | 59.0%56.9%–61.0% | rstu | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 70.98chosen-answer Brier 0.1703 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Accuracy 58.8% (95%: 56.7%–60.9%) Logic 45.3% | — | 58.8%56.7%–60.9% | rstu | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 69.67chosen-answer Brier 0.1946 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Accuracy 58.4% (95%: 56.3%–60.5%) Logic 46.6% | — | 58.4%56.3%–60.5% | rstuv | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 69.90chosen-answer Brier 0.1864 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Accuracy 57.9% (95%: 55.8%–60.0%) Logic 47.8% | — | 57.9%55.8%–60.0% | rstuvw | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 69.69chosen-answer Brier 0.1855 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Accuracy 56.8% (95%: 54.7%–58.9%) Logic 44.1% | — | 56.8%54.7%–58.9% | stuvw | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 69.18chosen-answer Brier 0.1846 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Accuracy 56.0% (95%: 53.9%–58.1%) Logic 43.3% | — | 56.0%53.9%–58.1% | stuvw | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 67.15chosen-answer Brier 0.2174 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Accuracy 56.0% (95%: 53.9%–58.1%) Logic 42.3% | — | 56.0%53.9%–58.1% | stuvw | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 67.89chosen-answer Brier 0.2021 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Accuracy 55.9% (95%: 53.8%–58.0%) Logic 44.7% | — | 55.9%53.8%–58.0% | stuvw | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 67.89chosen-answer Brier 0.2015 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Accuracy 55.8% (95%: 53.7%–57.9%) Logic 45.9% | — | 55.8%53.7%–57.9% | stuvwx | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 68.25chosen-answer Brier 0.1934 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 55.6% (95%: 53.5%–57.7%) Logic 40.8% | — | 55.6%53.5%–57.7% | stuvwxy | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 68.16chosen-answer Brier 0.1930 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 55.3% (95%: 53.2%–57.4%) Logic 44.4% | — | 55.3%53.2%–57.4% | tuvwxy | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 68.37chosen-answer Brier 0.1860 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Accuracy 55.2% (95%: 53.1%–57.3%) Logic 43.1% | — | 55.2%53.1%–57.3% | tuvwxy | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 67.61chosen-answer Brier 0.2002 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Accuracy 54.6% (95%: 52.4%–56.6%) Logic 42.6% | — | 54.6%52.4%–56.6% | uvwxyz | 1,180 / 2,163 | No answers at this threshold | 0.764 | 64.53chosen-answer Brier 0.2549 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Accuracy 53.8% (95%: 51.7%–55.9%) Logic 40.6% | — | 53.8%51.7%–55.9% | vwxyzA | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 67.54chosen-answer Brier 0.1869 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Accuracy 53.5% (95%: 51.4%–55.6%) Logic 45.1% | — | 53.5%51.4%–55.6% | wxyzA | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 66.02chosen-answer Brier 0.2146 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 51.5% (95%: 49.4%–53.6%) Logic 39.7% | — | 51.5%49.4%–53.6% | yzAB | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 65.62chosen-answer Brier 0.2026 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Accuracy 51.1% (95%: 49.0%–53.2%) Logic 40.8% | — | 51.1%49.0%–53.2% | xyzAB | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 65.20chosen-answer Brier 0.2070 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 50.4% (95%: 48.3%–52.5%) Logic 40.4% | — | 50.4%48.3%–52.5% | zAB | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 64.11chosen-answer Brier 0.2221 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Accuracy 49.8% (95%: 47.7%–51.9%) Logic 38.9% | — | 49.8%47.7%–51.9% | AB | 1,077 / 2,163 | No answers at this threshold | 0.730 | 64.44chosen-answer Brier 0.2092 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Accuracy 46.9% (95%: 44.8%–49.0%) Logic 39.7% | — | 46.9%44.8%–49.0% | BC | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 63.00chosen-answer Brier 0.2087 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Accuracy 44.3% (95%: 42.3%–46.4%) Logic 39.3% | — | 44.3%42.3%–46.4% | C | 959 / 2,163 | No answers at this threshold | 0.735 | 61.87chosen-answer Brier 0.2060 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Accuracy 39.2% (95%: 37.1%–41.2%) Logic 32.1% | — | 39.2%37.1%–41.2% | D | 847 / 2,163 | No answers at this threshold | 0.651 | 58.63chosen-answer Brier 0.2190 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Accuracy 38.7% (95%: 36.6%–40.7%) Logic 30.8% | — | 38.7%36.6%–40.7% | D | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 58.95chosen-answer Brier 0.2075 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Accuracy 37.1% (95%: 35.1%–39.1%) Logic 30.7% | — | 37.1%35.1%–39.1% | D | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 57.72chosen-answer Brier 0.2163 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Accuracy 36.8% (95%: 34.7%–38.8%) Logic 32.3% | — | 36.8%34.7%–38.8% | D | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 55.70chosen-answer Brier 0.2536 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Accuracy 35.3% (95%: 33.3%–37.3%) Logic 31.7% | — | 35.3%33.3%–37.3% | DE | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 56.49chosen-answer Brier 0.2229 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Accuracy 32.1% (95%: 30.2%–34.1%) Logic 29.2% | — | 32.1%30.2%–34.1% | EF | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 53.21chosen-answer Brier 0.2566 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 29.7% (95%: 27.8%–31.6%) Logic 24.7% | — | 29.7%27.8%–31.6% | F | 642 / 2,163 | No answers at this threshold | 0.583 | 54.40chosen-answer Brier 0.2088 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Accuracy 28.7% (95%: 26.8%–30.6%) Logic 22.4% | — | 28.7%26.8%–30.6% | F | 620 / 2,163 | No answers at this threshold | 0.569 | 53.90chosen-answer Brier 0.2086 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Accuracy 28.2% (95%: 26.4%–30.2%) Logic 27.0% | — | 28.2%26.4%–30.2% | F | 611 / 2,163 | No answers at this threshold | 0.592 | 53.62chosen-answer Brier 0.2100 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 28.2% (95%: 26.3%–30.1%) Logic 26.7% | — | 28.2%26.3%–30.1% | F | 610 / 2,163 | No answers at this threshold | 0.596 | 53.60chosen-answer Brier 0.2100 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Accuracy 22.7% (95%: 20.9%–24.5%) Logic 23.6% | — | 22.7%20.9%–24.5% | G | 490 / 2,163 | No answers at this threshold | 0.501 | 52.01chosen-answer Brier 0.1863 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Accuracy 20.6% (95%: 18.9%–22.3%) Logic 23.2% | — | 20.6%18.9%–22.3% | G | 445 / 2,163 | No answers at this threshold | 0.500 | 50.98chosen-answer Brier 0.1862 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
scar-ai PelaAI Perplexity StartLux victorbennett quincyqiang Metask Lab Liquid AI Fastino Strands Agents OmniJev Other Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| vera-medium-omni Open · 4.2B | — | 55.9%133/238 | 36.3%74/204 | 35.3%6/17 | 42.7%67/157 | 48.6%53/109 | 50.7%68/134 | 70.9%78/110 | 67.3%68/101 | 62.7%101/161 | 63.8%67/105 | 58.4%132/226 | 61.1%157/257 | 65.4%225/344 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 52.1%124/238 | 37.7%77/204 | 23.5%4/17 | 40.8%64/157 | 53.2%58/109 | 60.4%81/134 | 71.8%79/110 | 60.4%61/101 | 57.8%93/161 | 61.9%65/105 | 59.3%134/226 | 59.9%154/257 | 63.1%217/344 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 55.5%132/238 | 36.3%74/204 | 29.4%5/17 | 44.6%70/157 | 50.5%55/109 | 53.0%71/134 | 69.1%76/110 | 60.4%61/101 | 60.2%97/161 | 59.0%62/105 | 58.8%133/226 | 61.9%159/257 | 62.5%215/344 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 54.2%129/238 | 40.7%83/204 | 17.6%3/17 | 41.4%65/157 | 49.5%54/109 | 52.2%70/134 | 74.5%82/110 | 64.4%65/101 | 59.6%96/161 | 58.1%61/105 | 55.8%126/226 | 61.5%158/257 | 62.8%216/344 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 51.7%123/238 | 39.2%80/204 | 29.4%5/17 | 40.8%64/157 | 52.3%57/109 | 57.5%77/134 | 70.9%78/110 | 63.4%64/101 | 56.5%91/161 | 57.1%60/105 | 59.7%135/226 | 61.9%159/257 | 58.7%202/344 |
| Verdin-4B-v1.1 Open · 4.0B | — | 55.9%133/238 | 34.3%70/204 | 17.6%3/17 | 33.8%53/157 | 54.1%59/109 | 44.0%59/134 | 64.5%71/110 | 60.4%61/101 | 50.3%81/161 | 45.7%48/105 | 55.3%125/226 | 59.9%154/257 | 57.3%197/344 |
| Chinese Decision Pointer 4B Open · 4.0B | — | 47.1%112/238 | 34.3%70/204 | 35.3%6/17 | 43.3%68/157 | 39.4%43/109 | 49.3%66/134 | 62.7%69/110 | 56.4%57/101 | 56.5%91/161 | 45.7%48/105 | 49.1%111/226 | 55.6%143/257 | 60.2%207/344 |
| Metask-Jev-4B Open · 4.5B | — | 40.3%96/238 | 31.4%64/204 | 11.8%2/17 | 30.6%48/157 | 32.1%35/109 | 32.1%43/134 | 39.1%43/110 | 42.6%43/101 | 43.5%70/161 | 34.3%36/105 | 38.1%86/226 | 47.9%123/257 | 45.9%158/344 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 33.6%80/238 | 25.0%51/204 | 35.3%6/17 | 32.5%51/157 | 34.9%38/109 | 32.8%44/134 | 50.0%55/110 | 54.5%55/101 | 46.0%74/161 | 46.7%49/105 | 40.3%91/226 | 43.6%112/257 | 37.8%130/344 |
| d1-2B Open · 1.9B | — | 29.4%70/238 | 33.3%68/204 | 35.3%6/17 | 37.6%59/157 | 35.8%39/109 | 34.3%46/134 | 46.4%51/110 | 39.6%40/101 | 32.9%53/161 | 39.0%41/105 | 41.6%94/226 | 40.5%104/257 | 38.1%131/344 |
| GLiDE 2B Open · 1.9B | — | 32.8%78/238 | 27.9%57/204 | 23.5%4/17 | 30.6%48/157 | 36.7%40/109 | 32.8%44/134 | 40.9%45/110 | 40.6%41/101 | 38.5%62/161 | 42.9%45/105 | 35.8%81/226 | 42.4%109/257 | 41.0%141/344 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 31.1%74/238 | 25.5%52/204 | 11.8%2/17 | 29.9%47/157 | 26.6%29/109 | 35.1%47/134 | 40.0%44/110 | 51.5%52/101 | 37.3%60/161 | 35.2%37/105 | 37.2%84/226 | 38.1%98/257 | 39.8%137/344 |
| GLiDE 0.8B Open · 0.8B | — | 30.3%72/238 | 27.0%55/204 | 29.4%5/17 | 29.9%47/157 | 33.9%37/109 | 26.1%35/134 | 26.4%29/110 | 34.7%35/101 | 36.0%58/161 | 23.8%25/105 | 29.6%67/226 | 34.6%89/257 | 40.7%140/344 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 27.7%66/238 | 20.6%42/204 | 29.4%5/17 | 23.6%37/157 | 35.8%39/109 | 29.1%39/134 | 38.2%42/110 | 42.6%43/101 | 36.0%58/161 | 27.6%29/105 | 30.5%69/226 | 28.4%73/257 | 29.1%100/344 |
| Strands Decider 2B Open · 2.0B | — | 29.8%71/238 | 24.0%49/204 | 23.5%4/17 | 25.5%40/157 | 30.3%33/109 | 27.6%37/134 | 30.9%34/110 | 37.6%38/101 | 31.7%51/161 | 25.7%27/105 | 29.6%67/226 | 25.3%65/257 | 30.2%104/344 |
| OneJev1.6-0.8B Open · 0.8B | — | 26.5%63/238 | 22.5%46/204 | 11.8%2/17 | 25.5%40/157 | 33.9%37/109 | 25.4%34/134 | 28.2%31/110 | 28.7%29/101 | 32.9%53/161 | 28.6%30/105 | 29.2%66/226 | 24.1%62/257 | 34.3%118/344 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 27.3%65/238 | 21.6%44/204 | 35.3%6/17 | 21.0%33/157 | 31.2%34/109 | 23.9%32/134 | 30.9%34/110 | 29.7%30/101 | 33.5%54/161 | 27.6%29/105 | 30.1%68/226 | 33.5%86/257 | 27.6%95/344 |
| GLiClass v4 Edge Open · 0.4B | — | 22.7%54/238 | 23.0%47/204 | 29.4%5/17 | 21.7%34/157 | 30.3%33/109 | 19.4%26/134 | 24.5%27/110 | 23.8%24/101 | 18.0%29/161 | 24.8%26/105 | 22.1%50/226 | 19.1%49/257 | 25.0%86/344 |
| GLiDE 0.4B Open · 0.4B | — | 19.3%46/238 | 17.6%36/204 | 5.9%1/17 | 23.6%37/157 | 25.7%28/109 | 18.7%25/134 | 10.9%12/110 | 22.8%23/101 | 23.6%38/161 | 19.0%20/105 | 22.6%51/226 | 18.3%47/257 | 23.5%81/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Accuracy 56.8% (95%: 54.7%–58.9%) Logic 44.1% | — | 56.8%54.7%–58.9% | a | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 69.18chosen-answer Brier 0.1846 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Accuracy 56.0% (95%: 53.9%–58.1%) Logic 42.3% | — | 56.0%53.9%–58.1% | a | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 67.89chosen-answer Brier 0.2021 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Accuracy 55.9% (95%: 53.8%–58.0%) Logic 44.7% | — | 55.9%53.8%–58.0% | a | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 67.89chosen-answer Brier 0.2015 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Accuracy 55.8% (95%: 53.7%–57.9%) Logic 45.9% | — | 55.8%53.7%–57.9% | a | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 68.25chosen-answer Brier 0.1934 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Accuracy 55.2% (95%: 53.1%–57.3%) Logic 43.1% | — | 55.2%53.1%–57.3% | a | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 67.61chosen-answer Brier 0.2002 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 51.5% (95%: 49.4%–53.6%) Logic 39.7% | — | 51.5%49.4%–53.6% | b | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 65.62chosen-answer Brier 0.2026 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 50.4% (95%: 48.3%–52.5%) Logic 40.4% | — | 50.4%48.3%–52.5% | b | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 64.11chosen-answer Brier 0.2221 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Accuracy 39.2% (95%: 37.1%–41.2%) Logic 32.1% | — | 39.2%37.1%–41.2% | c | 847 / 2,163 | No answers at this threshold | 0.651 | 58.63chosen-answer Brier 0.2190 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Accuracy 38.7% (95%: 36.6%–40.7%) Logic 30.8% | — | 38.7%36.6%–40.7% | c | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 58.95chosen-answer Brier 0.2075 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Accuracy 37.1% (95%: 35.1%–39.1%) Logic 30.7% | — | 37.1%35.1%–39.1% | c | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 57.72chosen-answer Brier 0.2163 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Accuracy 36.8% (95%: 34.7%–38.8%) Logic 32.3% | — | 36.8%34.7%–38.8% | c | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 55.70chosen-answer Brier 0.2536 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Accuracy 35.3% (95%: 33.3%–37.3%) Logic 31.7% | — | 35.3%33.3%–37.3% | cd | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 56.49chosen-answer Brier 0.2229 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Accuracy 32.1% (95%: 30.2%–34.1%) Logic 29.2% | — | 32.1%30.2%–34.1% | de | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 53.21chosen-answer Brier 0.2566 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 29.7% (95%: 27.8%–31.6%) Logic 24.7% | — | 29.7%27.8%–31.6% | ef | 642 / 2,163 | No answers at this threshold | 0.583 | 54.40chosen-answer Brier 0.2088 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Accuracy 28.7% (95%: 26.8%–30.6%) Logic 22.4% | — | 28.7%26.8%–30.6% | ef | 620 / 2,163 | No answers at this threshold | 0.569 | 53.90chosen-answer Brier 0.2086 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Accuracy 28.2% (95%: 26.4%–30.2%) Logic 27.0% | — | 28.2%26.4%–30.2% | f | 611 / 2,163 | No answers at this threshold | 0.592 | 53.62chosen-answer Brier 0.2100 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 28.2% (95%: 26.3%–30.1%) Logic 26.7% | — | 28.2%26.3%–30.1% | f | 610 / 2,163 | No answers at this threshold | 0.596 | 53.60chosen-answer Brier 0.2100 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Accuracy 22.7% (95%: 20.9%–24.5%) Logic 23.6% | — | 22.7%20.9%–24.5% | g | 490 / 2,163 | No answers at this threshold | 0.501 | 52.01chosen-answer Brier 0.1863 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Accuracy 20.6% (95%: 18.9%–22.3%) Logic 23.2% | — | 20.6%18.9%–22.3% | g | 445 / 2,163 | No answers at this threshold | 0.500 | 50.98chosen-answer Brier 0.1862 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
PelaAI newsbang Base Labs tattrongvu Patrick-18 StartLux Fastino Apertix AI Valen-Team Cloudflare EldanRing Blockbrain Labs Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B | — | 66.8%159/238 | 55.9%114/204 | 41.2%7/17 | 56.1%88/157 | 54.1%59/109 | 65.7%88/134 | 77.3%85/110 | 77.2%78/101 | 67.7%109/161 | 69.5%73/105 | 67.3%152/226 | 66.1%170/257 | 73.0%251/344 |
| Adjudex 9B v5 Open · 9.0B | — | 61.3%146/238 | 43.1%88/204 | 41.2%7/17 | 51.0%80/157 | 52.3%57/109 | 63.4%85/134 | 76.4%84/110 | 64.4%65/101 | 64.0%103/161 | 64.8%68/105 | 64.2%145/226 | 68.9%177/257 | 69.2%238/344 |
| Elephant-1 Open · 9.0B | — | 59.2%141/238 | 40.2%82/204 | 41.2%7/17 | 52.2%82/157 | 56.0%61/109 | 63.4%85/134 | 72.7%80/110 | 72.3%73/101 | 60.9%98/161 | 65.7%69/105 | 63.7%144/226 | 64.2%165/257 | 69.8%240/344 |
| KnowLine-9B-Gen2 Open · 9.0B | — | 57.6%137/238 | 41.2%84/204 | 35.3%6/17 | 49.0%77/157 | 50.5%55/109 | 60.4%81/134 | 77.3%85/110 | 64.4%65/101 | 62.7%101/161 | 66.7%70/105 | 66.4%150/226 | 67.3%173/257 | 62.8%216/344 |
| Jeffan-9B-1211 Open · 9.0B | — | 58.4%139/238 | 38.2%78/204 | 23.5%4/17 | 38.9%61/157 | 53.2%58/109 | 56.7%76/134 | 73.6%81/110 | 71.3%72/101 | 64.0%103/161 | 63.8%67/105 | 64.6%146/226 | 63.4%163/257 | 71.5%246/344 |
| KnowLine-9B-Gen1 Open · 9.0B | — | 59.7%142/238 | 42.2%86/204 | 17.6%3/17 | 51.0%80/157 | 53.2%58/109 | 61.2%82/134 | 72.7%80/110 | 63.4%64/101 | 66.5%107/161 | 61.0%64/105 | 61.5%139/226 | 66.1%170/257 | 63.4%218/344 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 56.3%134/238 | 39.7%81/204 | 35.3%6/17 | 42.7%67/157 | 56.0%61/109 | 53.7%72/134 | 72.7%80/110 | 70.3%71/101 | 60.2%97/161 | 61.0%64/105 | 64.2%145/226 | 60.7%156/257 | 69.2%238/344 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 59.2%141/238 | 38.2%78/204 | 35.3%6/17 | 46.5%73/157 | 49.5%54/109 | 54.5%73/134 | 70.9%78/110 | 65.3%66/101 | 64.6%104/161 | 62.9%66/105 | 60.6%137/226 | 62.3%160/257 | 66.3%228/344 |
| GLiDE 9B Open · 7.9B | — | 51.3%122/238 | 38.2%78/204 | 23.5%4/17 | 43.3%68/157 | 48.6%53/109 | 55.2%74/134 | 73.6%81/110 | 63.4%64/101 | 62.7%101/161 | 67.6%71/105 | 62.8%142/226 | 64.2%165/257 | 66.9%230/344 |
| Cerno 9B v1.2 Open · 9.0B | — | 55.0%131/238 | 41.7%85/204 | 35.3%6/17 | 43.9%69/157 | 48.6%53/109 | 53.0%71/134 | 67.3%74/110 | 66.3%67/101 | 60.2%97/161 | 56.2%59/105 | 58.8%133/226 | 61.1%157/257 | 61.0%210/344 |
| Valen-Decision-1.5-9B Open · 9.0B | — | 52.9%126/238 | 34.8%71/204 | 47.1%8/17 | 46.5%73/157 | 45.9%50/109 | 53.0%71/134 | 61.8%68/110 | 65.3%66/101 | 54.7%88/161 | 57.1%60/105 | 58.4%132/226 | 65.0%167/257 | 63.1%217/344 |
| Clef-Flash Open · 9.0B | — | 49.6%118/238 | 33.3%68/204 | 52.9%9/17 | 42.0%66/157 | 43.1%47/109 | 42.5%57/134 | 57.3%63/110 | 59.4%60/101 | 57.8%93/161 | 53.3%56/105 | 49.1%111/226 | 55.3%142/257 | 54.4%187/344 |
| Winnow-12B Open · 12B | — | 51.3%122/238 | 31.9%65/204 | 41.2%7/17 | 35.7%56/157 | 38.5%42/109 | 47.0%63/134 | 61.8%68/110 | 67.3%68/101 | 52.2%84/161 | 45.7%48/105 | 45.6%103/226 | 46.3%119/257 | 49.1%169/344 |
| Cygnet Open · 12B | — | 43.7%104/238 | 30.9%63/204 | 41.2%7/17 | 33.1%52/157 | 39.4%43/109 | 36.6%49/134 | 50.9%56/110 | 59.4%60/101 | 48.4%78/161 | 45.7%48/105 | 49.6%112/226 | 47.9%123/257 | 47.7%164/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Accuracy 66.3% (95%: 64.2%–68.2%) Logic 50.6% | — | 66.3%64.2%–68.2% | a | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 74.96chosen-answer Brier 0.1632 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 62.1% (95%: 60.0%–64.1%) Logic 46.4% | — | 62.1%60.0%–64.1% | b | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 72.28chosen-answer Brier 0.1752 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Accuracy 61.3% (95%: 59.3%–63.4%) Logic 46.4% | — | 61.3%59.3%–63.4% | bc | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 71.86chosen-answer Brier 0.1763 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 60.1% (95%: 58.0%–62.1%) Logic 43.9% | — | 60.1%58.0%–62.1% | bcd | 1,300 / 2,163 | No answers at this threshold | 0.803 | 71.15chosen-answer Brier 0.1779 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 59.8% (95%: 57.7%–61.9%) Logic 48.2% | — | 59.8%57.7%–61.9% | bcd | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 71.32chosen-answer Brier 0.1719 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 59.8% (95%: 57.7%–61.8%) Logic 44.3% | — | 59.8%57.7%–61.8% | bcd | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 71.08chosen-answer Brier 0.1762 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Accuracy 58.8% (95%: 56.7%–60.9%) Logic 45.3% | — | 58.8%56.7%–60.9% | cde | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 69.67chosen-answer Brier 0.1946 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Accuracy 58.4% (95%: 56.3%–60.5%) Logic 46.6% | — | 58.4%56.3%–60.5% | cde | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 69.90chosen-answer Brier 0.1864 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Accuracy 57.9% (95%: 55.8%–60.0%) Logic 47.8% | — | 57.9%55.8%–60.0% | de | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 69.69chosen-answer Brier 0.1855 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Accuracy 56.0% (95%: 53.9%–58.1%) Logic 43.3% | — | 56.0%53.9%–58.1% | e | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 67.15chosen-answer Brier 0.2174 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 55.3% (95%: 53.2%–57.4%) Logic 44.4% | — | 55.3%53.2%–57.4% | e | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 68.37chosen-answer Brier 0.1860 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Accuracy 49.8% (95%: 47.7%–51.9%) Logic 38.9% | — | 49.8%47.7%–51.9% | f | 1,077 / 2,163 | No answers at this threshold | 0.730 | 64.44chosen-answer Brier 0.2092 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Accuracy 46.9% (95%: 44.8%–49.0%) Logic 39.7% | — | 46.9%44.8%–49.0% | fg | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 63.00chosen-answer Brier 0.2087 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Accuracy 44.3% (95%: 42.3%–46.4%) Logic 39.3% | — | 44.3%42.3%–46.4% | g | 959 / 2,163 | No answers at this threshold | 0.735 | 61.87chosen-answer Brier 0.2060 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
MachinePulse PelaAI Poindexter Labs Sthānika AI Base Labs Perplexity Fastino vLLM Semantic Router Nace.AI Turion.AI Zefan Cai OmniJev Cloudflare Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | 77.7%185/238 | 65.2%133/204 | 64.7%11/17 | 68.8%108/157 | 74.3%81/109 | 85.1%114/134 | 92.7%102/110 | 92.1%93/101 | 88.2%142/161 | 81.9%86/105 | 85.8%194/226 | 83.7%215/257 | 87.2%300/344 |
| KnowLine-31B-Gen2 Open · 31B | — | 75.6%180/238 | 60.3%123/204 | 64.7%11/17 | 70.1%110/157 | 75.2%82/109 | 79.1%106/134 | 90.9%100/110 | 93.1%94/101 | 85.1%137/161 | 81.9%86/105 | 88.5%200/226 | 82.9%213/257 | 85.5%294/344 |
| KnowLine-31B-Gen1.5 Open · 31B | — | 76.9%183/238 | 58.8%120/204 | 52.9%9/17 | 63.7%100/157 | 70.6%77/109 | 76.1%102/134 | 88.2%97/110 | 89.1%90/101 | 82.0%132/161 | 79.0%83/105 | 85.0%192/226 | 80.2%206/257 | 82.3%283/344 |
| Copernicus-31B Open · 31B | — | 76.1%181/238 | 54.9%112/204 | 58.8%10/17 | 63.1%99/157 | 71.6%78/109 | 79.1%106/134 | 89.1%98/110 | 92.1%93/101 | 81.4%131/161 | 76.2%80/105 | 85.4%193/226 | 79.4%204/257 | 83.7%288/344 |
| Sieve 31B v3 Open · 31B | — | 75.6%180/238 | 55.9%114/204 | 52.9%9/17 | 63.7%100/157 | 69.7%76/109 | 81.3%109/134 | 90.9%100/110 | 86.1%87/101 | 80.1%129/161 | 80.0%84/105 | 82.7%187/226 | 79.4%204/257 | 84.0%289/344 |
| Mammoth-1 Open · 27B | — | 68.1%162/238 | 50.0%102/204 | 35.3%6/17 | 57.3%90/157 | 62.4%68/109 | 77.6%104/134 | 88.2%97/110 | 81.2%82/101 | 79.5%128/161 | 79.0%83/105 | 84.5%191/226 | 78.6%202/257 | 82.8%285/344 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 69.7%166/238 | 52.5%107/204 | 76.5%13/17 | 55.4%87/157 | 60.6%66/109 | 77.6%104/134 | 86.4%95/110 | 83.2%84/101 | 77.6%125/161 | 73.3%77/105 | 76.5%173/226 | 76.7%197/257 | 79.7%274/344 |
| GLiDE 27B Open · 27B | — | 69.3%165/238 | 52.0%106/204 | 52.9%9/17 | 61.8%97/157 | 60.6%66/109 | 73.1%98/134 | 90.9%100/110 | 81.2%82/101 | 77.6%125/161 | 70.5%74/105 | 77.4%175/226 | 73.9%190/257 | 78.5%270/344 |
| d3-ultra Open · 26B | — | 67.2%160/238 | 50.0%102/204 | 52.9%9/17 | 56.1%88/157 | 73.4%80/109 | 69.4%93/134 | 88.2%97/110 | 85.1%86/101 | 77.6%125/161 | 75.2%79/105 | 75.2%170/226 | 72.0%185/257 | 79.4%273/344 |
| d3 Open · 26B | — | 65.1%155/238 | 52.5%107/204 | 47.1%8/17 | 58.6%92/157 | 56.0%61/109 | 67.9%91/134 | 86.4%95/110 | 80.2%81/101 | 77.6%125/161 | 70.5%74/105 | 76.5%173/226 | 73.2%188/257 | 79.4%273/344 |
| Drex Max Open · 27B | — | 68.1%162/238 | 52.5%107/204 | 52.9%9/17 | 56.1%88/157 | 58.7%64/109 | 67.9%91/134 | 89.1%98/110 | 79.2%80/101 | 77.0%124/161 | 69.5%73/105 | 75.7%171/226 | 71.6%184/257 | 78.5%270/344 |
| GLiDE v1.2 Open · 27B | — | 67.2%160/238 | 52.9%108/204 | 52.9%9/17 | 56.7%89/157 | 60.6%66/109 | 73.9%99/134 | 88.2%97/110 | 78.2%79/101 | 72.0%116/161 | 71.4%75/105 | 75.2%170/226 | 70.4%181/257 | 76.2%262/344 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 66.0%157/238 | 50.5%103/204 | 52.9%9/17 | 55.4%87/157 | 60.6%66/109 | 72.4%97/134 | 83.6%92/110 | 76.2%77/101 | 74.5%120/161 | 70.5%74/105 | 77.4%175/226 | 70.8%182/257 | 78.5%270/344 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 68.1%162/238 | 48.5%99/204 | 58.8%10/17 | 56.7%89/157 | 61.5%67/109 | 67.9%91/134 | 82.7%91/110 | 80.2%81/101 | 75.2%121/161 | 66.7%70/105 | 77.9%176/226 | 70.4%181/257 | 75.9%261/344 |
| Turion Jev 31B Open · 31B | — | 70.2%167/238 | 41.2%84/204 | 41.2%7/17 | 56.1%88/157 | 56.9%62/109 | 76.1%102/134 | 81.8%90/110 | 84.2%85/101 | 76.4%123/161 | 71.4%75/105 | 73.0%165/226 | 73.2%188/257 | 74.7%257/344 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 55.5%132/238 | 40.7%83/204 | 29.4%5/17 | 49.7%78/157 | 51.4%56/109 | 57.5%77/134 | 70.0%77/110 | 65.3%66/101 | 64.0%103/161 | 61.9%65/105 | 62.4%141/226 | 64.6%166/257 | 66.0%227/344 |
| Open-Jev-27B-v1.1 Open · 27B | — | 48.7%116/238 | 36.3%74/204 | 47.1%8/17 | 48.4%76/157 | 38.5%42/109 | 47.0%63/134 | 73.6%81/110 | 68.3%69/101 | 62.1%100/161 | 53.3%56/105 | 56.2%127/226 | 61.5%158/257 | 61.0%210/344 |
| OneJev-27B Open · 27B | — | 52.9%126/238 | 32.8%67/204 | 47.1%8/17 | 43.9%69/157 | 38.5%42/109 | 47.0%63/134 | 67.3%74/110 | 65.3%66/101 | 60.2%97/161 | 55.2%58/105 | 54.0%122/226 | 59.9%154/257 | 63.1%217/344 |
| Clef Open · 27B | — | 54.6%130/238 | 36.3%74/204 | 52.9%9/17 | 38.9%61/157 | 40.4%44/109 | 36.6%49/134 | 59.1%65/110 | 57.4%58/101 | 57.8%93/161 | 49.5%52/105 | 53.1%120/226 | 54.1%139/257 | 61.3%211/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Accuracy 81.6% (95%: 79.9%–83.1%) Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 85.99chosen-answer Brier 0.0958 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Accuracy 80.3% (95%: 78.5%–81.9%) Logic 58.9% | — | 80.3%78.5%–81.9% | a | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 85.05chosen-answer Brier 0.1016 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Accuracy 77.4% (95%: 75.6%–79.1%) Logic 59.0% | — | 77.4%75.6%–79.1% | b | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 83.18chosen-answer Brier 0.1103 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Accuracy 77.3% (95%: 75.5%–79.1%) Logic 58.7% | — | 77.3%75.5%–79.1% | b | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 83.06chosen-answer Brier 0.1122 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Accuracy 77.1% (95%: 75.3%–78.8%) Logic 57.9% | — | 77.1%75.3%–78.8% | bc | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 82.89chosen-answer Brier 0.1133 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Accuracy 74.0% (95%: 72.1%–75.8%) Logic 54.5% | — | 74.0%72.1%–75.8% | cd | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 81.16chosen-answer Brier 0.1164 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Accuracy 72.5% (95%: 70.6%–74.3%) Logic 54.8% | — | 72.5%70.6%–74.3% | de | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 79.32chosen-answer Brier 0.1385 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Accuracy 72.0% (95%: 70.1%–73.8%) Logic 58.4% | — | 72.0%70.1%–73.8% | def | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 79.22chosen-answer Brier 0.1354 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 71.5% (95%: 69.6%–73.4%) Logic 54.6% | — | 71.5%69.6%–73.4% | def | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 78.39chosen-answer Brier 0.1474 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 70.4% (95%: 68.5%–72.3%) Logic 54.5% | — | 70.4%68.5%–72.3% | ef | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 78.71chosen-answer Brier 0.1299 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Accuracy 70.3% (95%: 68.4%–72.2%) Logic 53.1% | — | 70.3%68.4%–72.2% | ef | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 77.10chosen-answer Brier 0.1611 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Accuracy 69.9% (95%: 67.9%–71.8%) Logic 55.2% | — | 69.9%67.9%–71.8% | ef | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 77.95chosen-answer Brier 0.1396 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Accuracy 69.8% (95%: 67.8%–71.7%) Logic 53.3% | — | 69.8%67.8%–71.7% | f | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 78.16chosen-answer Brier 0.1345 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Accuracy 69.3% (95%: 67.3%–71.2%) Logic 52.5% | with web | 69.3%67.3%–71.2% | f | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 77.94chosen-answer Brier 0.1342 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Accuracy 69.0% (95%: 67.0%–70.9%) Logic 52.3% | — | 69.0%67.0%–70.9% | f | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 77.52chosen-answer Brier 0.1399 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Accuracy 59.0% (95%: 56.9%–61.0%) Logic 41.9% | — | 59.0%56.9%–61.0% | g | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 70.98chosen-answer Brier 0.1703 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Accuracy 54.6% (95%: 52.4%–56.6%) Logic 42.6% | — | 54.6%52.4%–56.6% | h | 1,180 / 2,163 | No answers at this threshold | 0.764 | 64.53chosen-answer Brier 0.2549 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Accuracy 53.8% (95%: 51.7%–55.9%) Logic 40.6% | — | 53.8%51.7%–55.9% | h | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 67.54chosen-answer Brier 0.1869 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Accuracy 51.1% (95%: 49.0%–53.2%) Logic 40.8% | — | 51.1%49.0%–53.2% | h | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 65.20chosen-answer Brier 0.2070 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
REAX Logits Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 76.9%183/238 | 57.8%118/204 | 64.7%11/17 | 61.8%97/157 | 76.1%83/109 | 76.1%102/134 | 92.7%102/110 | 84.2%85/101 | 83.9%135/161 | 79.0%83/105 | 85.4%193/226 | 77.8%200/257 | 85.8%295/344 |
| Logits Sigma-1 Duo Open · 57B | — | 71.4%170/238 | 51.5%105/204 | 64.7%11/17 | 59.9%94/157 | 75.2%82/109 | 75.4%101/134 | 90.0%99/110 | 85.1%86/101 | 81.4%131/161 | 75.2%79/105 | 81.4%184/226 | 78.6%202/257 | 82.3%283/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Accuracy 78.0% (95%: 76.2%–79.7%) Logic 59.3% | — | 78.0%76.2%–79.7% | a | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 83.57chosen-answer Brier 0.1084 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Accuracy 75.2% (95%: 73.4%–77.0%) Logic 57.8% | — | 75.2%73.4%–77.0% | b | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 81.60chosen-answer Brier 0.1201 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Liquid AI Google Fastino Nace.AI Celeris Subquadratic TypeSafe Microsoft Inception Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient | — | 71.8%171/238 | 53.4%109/204 | 70.6%12/17 | 65.6%103/157 | 59.6%65/109 | 78.4%105/134 | 84.5%93/110 | 77.2%78/101 | 72.7%117/161 | 78.1%82/105 | 79.6%180/226 | 79.4%204/257 | 82.6%284/344 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 65.1%155/238 | 48.0%98/204 | 58.8%10/17 | 61.8%97/157 | 65.1%71/109 | 75.4%101/134 | 81.8%90/110 | 79.2%80/101 | 70.8%114/161 | 76.2%80/105 | 77.0%174/226 | 76.7%197/257 | 80.2%276/344 |
| GLiDE API Closed · Efficient | — | 67.2%160/238 | 54.4%111/204 | 47.1%8/17 | 56.7%89/157 | 59.6%65/109 | 71.6%96/134 | 88.2%97/110 | 80.2%81/101 | 73.9%119/161 | 75.2%79/105 | 74.3%168/226 | 75.1%193/257 | 77.0%265/344 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 69.7%166/238 | 52.9%108/204 | 58.8%10/17 | 54.8%86/157 | 56.0%61/109 | 67.9%91/134 | 88.2%97/110 | 80.2%81/101 | 75.8%122/161 | 72.4%76/105 | 76.1%172/226 | 70.4%181/257 | 77.0%265/344 |
| celeris-1-decision Closed · Efficient | — | 72.7%173/238 | 53.4%109/204 | 58.8%10/17 | 57.3%90/157 | 59.6%65/109 | 70.1%94/134 | 92.7%102/110 | 75.2%76/101 | 68.9%111/161 | 69.5%73/105 | 72.6%164/226 | 73.5%189/257 | 75.0%258/344 |
| celeris-1.1-decision Closed · Efficient | — | 71.4%170/238 | 53.4%109/204 | 52.9%9/17 | 58.6%92/157 | 59.6%65/109 | 70.1%94/134 | 92.7%102/110 | 74.3%75/101 | 68.9%111/161 | 70.5%74/105 | 72.1%163/226 | 73.5%189/257 | 75.0%258/344 |
| SubQ Snap 1.1 Closed · Efficient | — | 65.1%155/238 | 51.5%105/204 | 52.9%9/17 | 52.9%83/157 | 67.0%73/109 | 70.1%94/134 | 82.7%91/110 | 76.2%77/101 | 74.5%120/161 | 72.4%76/105 | 74.8%169/226 | 68.1%175/257 | 74.4%256/344 |
| d1.1 Closed · Efficient | — | 63.0%150/238 | 44.1%90/204 | 41.2%7/17 | 50.3%79/157 | 56.0%61/109 | 63.4%85/134 | 72.7%80/110 | 73.3%74/101 | 64.6%104/161 | 66.7%70/105 | 72.1%163/226 | 75.9%195/257 | 73.3%252/344 |
| Jev 1.14 Closed · Efficient | — | 59.7%142/238 | 42.6%87/204 | 58.8%10/17 | 52.2%82/157 | 58.7%64/109 | 64.9%87/134 | 78.2%86/110 | 76.2%77/101 | 66.5%107/161 | 74.3%78/105 | 70.4%159/226 | 71.2%183/257 | 71.8%247/344 |
| Jev 1.13 Closed · Efficient | — | 56.7%135/238 | 47.5%97/204 | 41.2%7/17 | 51.0%80/157 | 55.0%60/109 | 63.4%85/134 | 78.2%86/110 | 76.2%77/101 | 64.0%103/161 | 74.3%78/105 | 69.9%158/226 | 70.0%180/257 | 69.8%240/344 |
| Microsoft-Decision-1 Closed · Efficient | — | 61.3%146/238 | 40.2%82/204 | 23.5%4/17 | 48.4%76/157 | 52.3%57/109 | 53.7%72/134 | 70.0%77/110 | 63.4%64/101 | 60.9%98/161 | 61.9%65/105 | 66.4%150/226 | 63.8%164/257 | 66.6%229/344 |
| d1 Closed · Efficient | — | 52.1%124/238 | 39.7%81/204 | 35.3%6/17 | 40.8%64/157 | 42.2%46/109 | 51.5%69/134 | 59.1%65/110 | 63.4%64/101 | 60.9%98/161 | 61.9%65/105 | 58.0%131/226 | 61.9%159/257 | 67.2%231/344 |
| Mercury Decide Closed · Efficient | — | 56.7%135/238 | 36.3%74/204 | 41.2%7/17 | 38.9%61/157 | 44.0%48/109 | 44.0%59/134 | 67.3%74/110 | 75.2%76/101 | 57.1%92/161 | 56.2%59/105 | 50.9%115/226 | 58.8%151/257 | 59.9%206/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Accuracy 74.1% (95%: 72.2%–75.9%) Logic 54.9% | — | 74.1%72.2%–75.9% | a | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 80.88chosen-answer Brier 0.1235 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Accuracy 71.3% (95%: 69.4%–73.2%) Logic 50.0% | — | 71.3%69.4%–73.2% | ab | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 78.84chosen-answer Brier 0.1366 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Accuracy 70.8% (95%: 68.8%–72.7%) Logic 54.5% | — | 70.8%68.8%–72.7% | b | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 78.56chosen-answer Brier 0.1367 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Accuracy 70.1% (95%: 68.1%–72.0%) Logic 54.8% | — | 70.1%68.1%–72.0% | b | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 78.01chosen-answer Brier 0.1407 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 70.0% (95%: 68.0%–71.9%) Logic 47.3% | — | 70.0%68.0%–71.9% | b | 1,514 / 2,163 | No answers at this threshold | 0.858 | 77.96chosen-answer Brier 0.1407 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 69.9% (95%: 67.9%–71.8%) Logic 47.3% | — | 69.9%67.9%–71.8% | b | 1,511 / 2,163 | No answers at this threshold | 0.859 | 77.91chosen-answer Brier 0.1403 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Accuracy 68.6% (95%: 66.6%–70.5%) Logic 52.4% | — | 68.6%66.6%–70.5% | b | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 76.15chosen-answer Brier 0.1626 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 65.2% (95%: 63.2%–67.2%) Logic 46.0% | — | 65.2%63.2%–67.2% | c | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 74.62chosen-answer Brier 0.1595 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 65.1% (95%: 63.1%–67.1%) Logic 48.2% | — | 65.1%63.1%–67.1% | c | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 74.56chosen-answer Brier 0.1602 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 64.1% (95%: 62.0%–66.1%) Logic 46.3% | — | 64.1%62.0%–66.1% | c | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 74.08chosen-answer Brier 0.1592 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Accuracy 59.4% (95%: 57.3%–61.4%) Logic 44.4% | — | 59.4%57.3%–61.4% | d | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 70.49chosen-answer Brier 0.1839 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 55.6% (95%: 53.5%–57.7%) Logic 40.8% | — | 55.6%53.5%–57.7% | e | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 68.16chosen-answer Brier 0.1930 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Accuracy 53.5% (95%: 51.4%–55.6%) Logic 45.1% | — | 53.5%51.4%–55.6% | e | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 66.02chosen-answer Brier 0.2146 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
No workhorse decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
No flagship decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Liquid AI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed | — | 67.2%160/238 | 46.1%94/204 | 47.1%8/17 | 54.8%86/157 | 59.6%65/109 | 75.4%101/134 | 84.5%93/110 | 82.2%83/101 | 75.8%122/161 | 69.5%73/105 | 75.2%170/226 | 73.2%188/257 | 77.9%268/344 |
| d1.2 Closed · Undisclosed | — | 62.6%149/238 | 40.2%82/204 | 41.2%7/17 | 52.9%83/157 | 60.6%66/109 | 65.7%88/134 | 75.5%83/110 | 78.2%79/101 | 66.5%107/161 | 67.6%71/105 | 72.1%163/226 | 73.2%188/257 | 73.0%251/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 69.9% (95%: 67.9%–71.8%) Logic 51.6% | — | 69.9%67.9%–71.8% | a | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 78.23chosen-answer Brier 0.1340 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 65.5% (95%: 63.5%–67.5%) Logic 49.2% | — | 65.5%63.5%–67.5% | b | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 74.98chosen-answer Brier 0.1556 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Alibaba DeepSeek OpenAI Thinking Machines Google Meta xAI Xiaomi OpenBMB IBM Liquid AI Other Z.ai MiniMax KRAFTON eCloud Technology Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship | — | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Qwen4-Max high · chat Closed · Undisclosed | — | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| Claude Fable 5.1 chat Closed · Flagship | — | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 99.4%156/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| Claude Sonnet 5.5 chat Closed · Workhorse | — | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 99.0%100/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| GPT-6 Astra chat Closed · Undisclosed | — | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| GPT-6.1 Sol chat Closed · Undisclosed | — | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| Gemini 4 Carbon chat Closed · Undisclosed | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 99.6%225/226 | 99.2%255/257 | 99.7%343/344 |
| Gemini 4 Hydrogen chat | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Muse Ember max · chat Closed · Efficient | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 99.0%100/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Grok 4.7 chat Closed · Undisclosed | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 98.8%159/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | 100.0%238/238 | 99.0%202/204 | 100.0%17/17 | 99.4%156/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 99.6%225/226 | 98.8%254/257 | 100.0%344/344 |
| Qwen3.5-4B on · chat Open · 4.7B | — | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Gemma 4 31B chat Open · 31B | — | 99.2%236/238 | 98.0%200/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 99.0%104/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active | — | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| MiniCPM5-2B on · chat Open · 2.5B | — | 99.2%236/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 97.1%102/105 | 100.0%226/226 | 99.2%255/257 | 99.4%342/344 |
| GPT-6.1 Terra chat Closed · Undisclosed | — | 99.6%237/238 | 98.5%201/204 | 100.0%17/17 | 99.4%156/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 98.1%158/161 | 100.0%105/105 | 99.6%225/226 | 98.8%254/257 | 99.7%343/344 |
| Granite-4.2-3B high · chat Open · 3.0B | — | 99.6%237/238 | 98.5%201/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Qwen3.8-27B low · chat Open · 27B | — | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 98.7%155/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 99.0%104/105 | 99.1%224/226 | 99.6%256/257 | 99.4%342/344 |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | 98.7%235/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 96.3%105/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 99.6%225/226 | 98.4%253/257 | 99.7%343/344 |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B | — | 98.3%234/238 | 97.5%199/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 98.8%159/161 | 98.1%103/105 | 99.6%225/226 | 99.6%256/257 | 100.0%344/344 |
| LFM3-1.7B on · chat Open · 1.7B | — | 98.3%234/238 | 97.5%199/204 | 100.0%17/17 | 100.0%157/157 | 97.2%106/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 99.4%342/344 |
| Gemma 4 12B Unified chat Open · 12B | — | 98.7%235/238 | 96.1%196/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 97.0%130/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 98.1%154/157 | 98.2%107/109 | 97.8%131/134 | 98.2%108/110 | 99.0%100/101 | 98.1%158/161 | 100.0%105/105 | 100.0%226/226 | 98.4%253/257 | 99.7%343/344 |
| LFM3-1.7B off · chat Open · 1.7B | — | 98.7%235/238 | 96.1%196/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 99.0%100/101 | 100.0%161/161 | 98.1%103/105 | 99.1%224/226 | 98.8%254/257 | 99.1%341/344 |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | 97.5%232/238 | 97.1%198/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 96.3%129/134 | 99.1%109/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 99.6%225/226 | 98.8%254/257 | 99.7%343/344 |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B | — | 97.5%232/238 | 97.5%199/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 95.5%128/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 98.7%223/226 | 98.4%253/257 | 100.0%344/344 |
| Qwen3.5-4B off · chat Open · 4.7B | — | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 97.5%153/157 | 96.3%105/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 98.7%223/226 | 98.1%252/257 | 99.1%341/344 |
| Gemma 4 E4B on · chat Open · 8.0B | — | 97.5%232/238 | 94.1%192/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 94.8%127/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 97.1%102/105 | 99.1%224/226 | 98.8%254/257 | 100.0%344/344 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | 98.7%235/238 | 97.5%199/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 98.5%132/134 | 96.4%106/110 | 95.0%96/101 | 98.8%159/161 | 99.0%104/105 | 98.2%222/226 | 98.1%252/257 | 98.8%340/344 |
| MiniCPM5-2B off · chat Open · 2.5B | — | 98.7%235/238 | 96.6%197/204 | 88.2%15/17 | 98.1%154/157 | 98.2%107/109 | 98.5%132/134 | 97.3%107/110 | 99.0%100/101 | 97.5%157/161 | 97.1%102/105 | 99.1%224/226 | 98.4%253/257 | 99.7%343/344 |
| Gemma 4 E4B chat Open · 8.0B | — | 98.3%234/238 | 95.1%194/204 | 100.0%17/17 | 98.7%155/157 | 99.1%108/109 | 95.5%128/134 | 98.2%108/110 | 98.0%99/101 | 98.1%158/161 | 96.2%101/105 | 98.7%223/226 | 99.2%255/257 | 99.4%342/344 |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B | — | 97.5%232/238 | 93.6%191/204 | 100.0%17/17 | 98.1%154/157 | 96.3%105/109 | 96.3%129/134 | 97.3%107/110 | 97.0%98/101 | 98.8%159/161 | 97.1%102/105 | 99.1%224/226 | 99.2%255/257 | 99.7%343/344 |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | 97.9%233/238 | 94.1%192/204 | 94.1%16/17 | 99.4%156/157 | 97.2%106/109 | 95.5%128/134 | 95.5%105/110 | 99.0%100/101 | 96.9%156/161 | 97.1%102/105 | 98.2%222/226 | 98.4%253/257 | 99.4%342/344 |
| Raon-1 nano on · chat Open · 3.1B | — | 96.6%230/238 | 90.2%184/204 | 100.0%17/17 | 98.1%154/157 | 96.3%105/109 | 97.8%131/134 | 97.3%107/110 | 97.0%98/101 | 99.4%160/161 | 98.1%103/105 | 98.2%222/226 | 97.7%251/257 | 97.7%336/344 |
| Gemma 4 E2B on · chat Open · 5.1B | — | 94.5%225/238 | 91.2%186/204 | 94.1%16/17 | 98.1%154/157 | 97.2%106/109 | 94.8%127/134 | 97.3%107/110 | 97.0%98/101 | 98.8%159/161 | 96.2%101/105 | 99.1%224/226 | 98.1%252/257 | 99.4%342/344 |
| Gemma 4 E2B chat Open · 5.1B | — | 96.2%229/238 | 90.7%185/204 | 94.1%16/17 | 97.5%153/157 | 98.2%107/109 | 95.5%128/134 | 97.3%107/110 | 98.0%99/101 | 96.3%155/161 | 96.2%101/105 | 98.7%223/226 | 97.3%250/257 | 98.8%340/344 |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 96.2%229/238 | 92.6%189/204 | 94.1%16/17 | 99.4%156/157 | 93.6%102/109 | 97.8%131/134 | 98.2%108/110 | 92.1%93/101 | 97.5%157/161 | 92.4%97/105 | 97.8%221/226 | 97.3%250/257 | 96.8%333/344 |
| Qwen3-14B off · chat Open · 15B | — | 94.5%225/238 | 83.3%170/204 | 100.0%17/17 | 96.8%152/157 | 99.1%108/109 | 94.8%127/134 | 98.2%108/110 | 95.0%96/101 | 91.3%147/161 | 100.0%105/105 | 99.1%224/226 | 98.4%253/257 | 98.8%340/344 |
| Qwen3.5-2B on · chat Open · 2.0B | — | 95.4%227/238 | 79.9%163/204 | 88.2%15/17 | 98.1%154/157 | 89.9%98/109 | 91.8%123/134 | 97.3%107/110 | 96.0%97/101 | 95.7%154/161 | 93.3%98/105 | 98.2%222/226 | 97.3%250/257 | 99.1%341/344 |
| Qwen3.5-2B off · chat Open · 2.0B | — | 94.5%225/238 | 87.3%178/204 | 100.0%17/17 | 89.2%140/157 | 90.8%99/109 | 97.8%131/134 | 93.6%103/110 | 97.0%98/101 | 93.2%150/161 | 93.3%98/105 | 95.1%215/226 | 99.2%255/257 | 98.3%338/344 |
| LFM2.5-2.6B on · chat Open · 2.7B | — | 95.8%228/238 | 89.2%182/204 | 88.2%15/17 | 93.6%147/157 | 94.5%103/109 | 95.5%128/134 | 98.2%108/110 | 97.0%98/101 | 98.1%158/161 | 97.1%102/105 | 92.0%208/226 | 93.4%240/257 | 91.9%316/344 |
| GPT-6 Luna chat Closed · Efficient | — | 90.8%216/238 | 81.4%166/204 | 82.4%14/17 | 94.3%148/157 | 89.9%98/109 | 96.3%129/134 | 96.4%106/110 | 93.1%94/101 | 93.2%150/161 | 98.1%103/105 | 96.9%219/226 | 96.9%249/257 | 97.1%334/344 |
| Erk-14B off · chat Open · 15B | — | 91.6%218/238 | 72.1%147/204 | 100.0%17/17 | 83.4%131/157 | 82.6%90/109 | 94.0%126/134 | 94.5%104/110 | 94.1%95/101 | 89.4%144/161 | 97.1%102/105 | 95.6%216/226 | 94.9%244/257 | 95.6%329/344 |
| Raon-1 nano off · chat Open · 3.1B | — | 91.2%217/238 | 75.5%154/204 | 94.1%16/17 | 87.3%137/157 | 89.9%98/109 | 93.3%125/134 | 92.7%102/110 | 92.1%93/101 | 85.7%138/161 | 91.4%96/105 | 91.2%206/226 | 93.0%239/257 | 91.9%316/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC); Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)); Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC); Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test); GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.8% (95%: 99.5%–99.9%) Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Accuracy 99.8% (95%: 99.5%–99.9%) Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.7% (95%: 99.4%–99.9%) Logic 95.2% | — | 99.7%99.4%–99.9% | abc | 2,157 / 2,163 | Not measured | Not measured | — | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.7% (95%: 99.3%–99.8%) Logic 95.5% | — | 99.7%99.3%–99.8% | abd | 2,156 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.2% | — | 99.6%99.3%–99.8% | abcdef | 2,155 / 2,163 | Not measured | Not measured | — | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.7% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.5% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Accuracy 99.6% (95%: 99.3%–99.8%) Logic 95.4% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.6% (95%: 99.2%–99.8%) Logic 96.7% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.2%–99.8%) Logic 96.1% | — | 99.6%99.2%–99.8% | abcdef | 2,154 / 2,163 | Not measured | Not measured | — | 85.7%answered 99% | 4,400 ms | 6,680 ms | — |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Accuracy 99.6% (95%: 99.2%–99.8%) Logic 95.5% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | — | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.5% (95%: 99.2%–99.7%) Logic running | — | 99.5%99.2%–99.7% | abcdefg | 2,153 / 2,163 | Not measured | Not measured | — | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Accuracy 99.5% (95%: 99.2%–99.7%) Logic 95.6% | — | 99.5%99.2%–99.7% | abcdef | 2,153 / 2,163 | Not measured | Not measured | — | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Accuracy 99.5% (95%: 99.1%–99.7%) Logic 94.0% | — | 99.5%99.1%–99.7% | abcdefgh | 2,152 / 2,163 | Not measured | Not measured | — | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Accuracy 99.4% (95%: 99.0%–99.7%) Logic 95.1% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | — | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Accuracy 99.4% (95%: 99.0%–99.7%) Logic 92.4% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Accuracy 99.4% (95%: 99.0%–99.6%) Logic 93.3% | — | 99.4%99.0%–99.6% | abcdefgh | 2,150 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Accuracy 99.3% (95%: 98.9%–99.6%) Logic running | — | 99.3%98.9%–99.6% | abcdefghij | 2,148 / 2,163 | Not measured | Not measured | — | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Accuracy 99.2% (mean of 3 seeds; min–max 99.2%–99.3%) Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | abcdefghi | 2,147 / 2,163seed 0 run | Not measured | Not measured | — | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.2% (95%: 98.7%–99.5%) Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | — | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Accuracy 99.2% (95%: 98.7%–99.5%) Logic 91.6% | — | 99.2%98.7%–99.5% | abcdefghi | 2,146 / 2,163 | Not measured | Not measured | — | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Accuracy 99.2% (95%: 98.7%–99.5%) Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | — | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Accuracy 99.0% (95%: 98.5%–99.4%) Logic 86.7% | — | 99.0%98.5%–99.4% | bcdefghij | 2,142 / 2,163 | Not measured | Not measured | — | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Accuracy 98.9% (mean of 3 seeds; min–max 98.8%–99.0%) Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abcdefghij | 2,142 / 2,163seed 0 run | Not measured | Not measured | — | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Accuracy 98.9% (mean of 3 seeds; min–max 98.8%–99.1%) Logic 84.2% | — | 98.9%98.8%–99.1% min–max of 3 seeds | cefghijk | 2,139 / 2,163seed 0 run | Not measured | Not measured | — | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Accuracy 98.8% (95%: 98.3%–99.2%) Logic 87.7% | — | 98.8%98.3%–99.2% | dfghijkl | 2,138 / 2,163 | Not measured | Not measured | — | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Accuracy 98.8% (95%: 98.2%–99.2%) Logic 93.2% | — | 98.8%98.2%–99.2% | fghijkl | 2,137 / 2,163 | Not measured | Not measured | — | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Accuracy 98.7% (95%: 98.1%–99.1%) Logic 85.5% | — | 98.7%98.1%–99.1% | ghijkl | 2,135 / 2,163 | Not measured | Not measured | — | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.5% (mean of 3 seeds; min–max 98.4%–98.8%) Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | efghijkl | 2,136 / 2,163seed 0 run | Not measured | Not measured | — | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.5% (95%: 97.9%–99.0%) Logic 85.0% | — | 98.5%97.9%–99.0% | hijkl | 2,131 / 2,163 | Not measured | Not measured | — | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Accuracy 98.5% (95%: 97.9%–98.9%) Logic 90.1% | — | 98.5%97.9%–98.9% | hijklm | 2,130 / 2,163 | Not measured | Not measured | — | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.2% (95%: 97.6%–98.7%) Logic 82.6% | — | 98.2%97.6%–98.7% | ijklm | 2,125 / 2,163 | Not measured | Not measured | — | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Accuracy 98.2% (95%: 97.6%–98.7%) Logic 94.0% | — | 98.2%97.6%–98.7% | ijklmn | 2,125 / 2,163 | Not measured | Not measured | — | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Accuracy 98.2% (mean of 3 seeds; min–max 98.1%–98.3%) Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | ijklmn | 2,126 / 2,163seed 0 run | Not measured | Not measured | — | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.1% (95%: 97.4%–98.6%) Logic 83.2% | — | 98.1%97.4%–98.6% | jklmn | 2,122 / 2,163 | Not measured | Not measured | — | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 97.8% (95%: 97.1%–98.4%) Logic 75.9% | — | 97.8%97.1%–98.4% | klmno | 2,116 / 2,163 | Not measured | Not measured | — | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 97.7% (mean of 3 seeds; min–max 97.6%–97.8%) Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | lmnop | 2,111 / 2,163seed 0 run | Not measured | Not measured | — | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 97.0% (95%: 96.2%–97.6%) Logic 70.9% | — | 97.0%96.2%–97.6% | mnop | 2,098 / 2,163 | Not measured | Not measured | — | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 96.9% (95%: 96.1%–97.6%) Logic 74.6% | — | 96.9%96.1%–97.6% | mnop | 2,097 / 2,163 | Not measured | Not measured | — | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 96.8% (95%: 95.9%–97.4%) Logic 72.9% | — | 96.8%95.9%–97.4% | nopq | 2,093 / 2,163 | Not measured | Not measured | — | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Accuracy 96.3% (95%: 95.4%–97.0%) Logic 60.2% | — | 96.3%95.4%–97.0% | opqr | 2,082 / 2,163 | Not measured | Not measured | — | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Accuracy 95.8% (95%: 94.9%–96.6%) Logic 65.3% | — | 95.8%94.9%–96.6% | pqrs | 2,072 / 2,163 | Not measured | Not measured | — | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Accuracy 94.7% (95%: 93.7%–95.6%) Logic 70.3% | — | 94.7%93.7%–95.6% | qrs | 2,049 / 2,163 | Not measured | Not measured | — | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Accuracy 94.6% (95%: 93.6%–95.5%) Logic 64.8% | — | 94.6%93.6%–95.5% | rs | 2,047 / 2,163 | Not measured | Not measured | — | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Accuracy 94.0% (95%: 92.9%–94.9%) Logic 75.4% | — | 94.0%92.9%–94.9% | rs | 2,033 / 2,163 | Not measured | Not measured | — | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 93.7% (95%: 92.6%–94.6%) Logic 58.0% | — | 93.7%92.6%–94.6% | s | 2,026 / 2,163 | Not measured | Not measured | — | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Accuracy 90.8% (95%: 89.5%–91.9%) Logic 54.2% | — | 90.8%89.5%–91.9% | t | 1,963 / 2,163 | Not measured | Not measured | — | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 89.6% (95%: 88.2%–90.8%) Logic 65.1% | — | 89.6%88.2%–90.8% | t | 1,937 / 2,163 | Not measured | Not measured | — | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Alibaba OpenBMB IBM Other Google KRAFTON Liquid AI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B | — | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| MiniCPM5-2B on · chat Open · 2.5B | — | 99.2%236/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 97.1%102/105 | 100.0%226/226 | 99.2%255/257 | 99.4%342/344 |
| Granite-4.2-3B high · chat Open · 3.0B | — | 99.6%237/238 | 98.5%201/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B | — | 98.3%234/238 | 97.5%199/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 98.8%159/161 | 98.1%103/105 | 99.6%225/226 | 99.6%256/257 | 100.0%344/344 |
| LFM3-1.7B on · chat Open · 1.7B | — | 98.3%234/238 | 97.5%199/204 | 100.0%17/17 | 100.0%157/157 | 97.2%106/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 99.4%342/344 |
| LFM3-1.7B off · chat Open · 1.7B | — | 98.7%235/238 | 96.1%196/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 98.5%132/134 | 98.2%108/110 | 99.0%100/101 | 100.0%161/161 | 98.1%103/105 | 99.1%224/226 | 98.8%254/257 | 99.1%341/344 |
| Qwen3.5-4B off · chat Open · 4.7B | — | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 97.5%153/157 | 96.3%105/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 98.7%223/226 | 98.1%252/257 | 99.1%341/344 |
| MiniCPM5-2B off · chat Open · 2.5B | — | 98.7%235/238 | 96.6%197/204 | 88.2%15/17 | 98.1%154/157 | 98.2%107/109 | 98.5%132/134 | 97.3%107/110 | 99.0%100/101 | 97.5%157/161 | 97.1%102/105 | 99.1%224/226 | 98.4%253/257 | 99.7%343/344 |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B | — | 97.5%232/238 | 93.6%191/204 | 100.0%17/17 | 98.1%154/157 | 96.3%105/109 | 96.3%129/134 | 97.3%107/110 | 97.0%98/101 | 98.8%159/161 | 97.1%102/105 | 99.1%224/226 | 99.2%255/257 | 99.7%343/344 |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | 97.9%233/238 | 94.1%192/204 | 94.1%16/17 | 99.4%156/157 | 97.2%106/109 | 95.5%128/134 | 95.5%105/110 | 99.0%100/101 | 96.9%156/161 | 97.1%102/105 | 98.2%222/226 | 98.4%253/257 | 99.4%342/344 |
| Raon-1 nano on · chat Open · 3.1B | — | 96.6%230/238 | 90.2%184/204 | 100.0%17/17 | 98.1%154/157 | 96.3%105/109 | 97.8%131/134 | 97.3%107/110 | 97.0%98/101 | 99.4%160/161 | 98.1%103/105 | 98.2%222/226 | 97.7%251/257 | 97.7%336/344 |
| Gemma 4 E2B on · chat Open · 5.1B | — | 94.5%225/238 | 91.2%186/204 | 94.1%16/17 | 98.1%154/157 | 97.2%106/109 | 94.8%127/134 | 97.3%107/110 | 97.0%98/101 | 98.8%159/161 | 96.2%101/105 | 99.1%224/226 | 98.1%252/257 | 99.4%342/344 |
| Gemma 4 E2B chat Open · 5.1B | — | 96.2%229/238 | 90.7%185/204 | 94.1%16/17 | 97.5%153/157 | 98.2%107/109 | 95.5%128/134 | 97.3%107/110 | 98.0%99/101 | 96.3%155/161 | 96.2%101/105 | 98.7%223/226 | 97.3%250/257 | 98.8%340/344 |
| Qwen3.5-2B on · chat Open · 2.0B | — | 95.4%227/238 | 79.9%163/204 | 88.2%15/17 | 98.1%154/157 | 89.9%98/109 | 91.8%123/134 | 97.3%107/110 | 96.0%97/101 | 95.7%154/161 | 93.3%98/105 | 98.2%222/226 | 97.3%250/257 | 99.1%341/344 |
| Qwen3.5-2B off · chat Open · 2.0B | — | 94.5%225/238 | 87.3%178/204 | 100.0%17/17 | 89.2%140/157 | 90.8%99/109 | 97.8%131/134 | 93.6%103/110 | 97.0%98/101 | 93.2%150/161 | 93.3%98/105 | 95.1%215/226 | 99.2%255/257 | 98.3%338/344 |
| LFM2.5-2.6B on · chat Open · 2.7B | — | 95.8%228/238 | 89.2%182/204 | 88.2%15/17 | 93.6%147/157 | 94.5%103/109 | 95.5%128/134 | 98.2%108/110 | 97.0%98/101 | 98.1%158/161 | 97.1%102/105 | 92.0%208/226 | 93.4%240/257 | 91.9%316/344 |
| Raon-1 nano off · chat Open · 3.1B | — | 91.2%217/238 | 75.5%154/204 | 94.1%16/17 | 87.3%137/157 | 89.9%98/109 | 93.3%125/134 | 92.7%102/110 | 92.1%93/101 | 85.7%138/161 | 91.4%96/105 | 91.2%206/226 | 93.0%239/257 | 91.9%316/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Accuracy 99.4% (95%: 99.0%–99.7%) Logic 92.4% | — | 99.4%99.0%–99.7% | a | 2,151 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Accuracy 99.2% (mean of 3 seeds; min–max 99.2%–99.3%) Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | ab | 2,147 / 2,163seed 0 run | Not measured | Not measured | — | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Accuracy 99.2% (95%: 98.7%–99.5%) Logic 91.6% | — | 99.2%98.7%–99.5% | ab | 2,146 / 2,163 | Not measured | Not measured | — | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Accuracy 98.9% (mean of 3 seeds; min–max 98.8%–99.0%) Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abc | 2,142 / 2,163seed 0 run | Not measured | Not measured | — | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Accuracy 98.9% (mean of 3 seeds; min–max 98.8%–99.1%) Logic 84.2% | — | 98.9%98.8%–99.1% min–max of 3 seeds | abc | 2,139 / 2,163seed 0 run | Not measured | Not measured | — | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Accuracy 98.7% (95%: 98.1%–99.1%) Logic 85.5% | — | 98.7%98.1%–99.1% | bcd | 2,135 / 2,163 | Not measured | Not measured | — | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Accuracy 98.5% (95%: 97.9%–98.9%) Logic 90.1% | — | 98.5%97.9%–98.9% | bcde | 2,130 / 2,163 | Not measured | Not measured | — | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Accuracy 98.2% (mean of 3 seeds; min–max 98.1%–98.3%) Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | cdef | 2,126 / 2,163seed 0 run | Not measured | Not measured | — | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 97.8% (95%: 97.1%–98.4%) Logic 75.9% | — | 97.8%97.1%–98.4% | defg | 2,116 / 2,163 | Not measured | Not measured | — | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 97.7% (mean of 3 seeds; min–max 97.6%–97.8%) Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | efg | 2,111 / 2,163seed 0 run | Not measured | Not measured | — | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 97.0% (95%: 96.2%–97.6%) Logic 70.9% | — | 97.0%96.2%–97.6% | fg | 2,098 / 2,163 | Not measured | Not measured | — | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 96.9% (95%: 96.1%–97.6%) Logic 74.6% | — | 96.9%96.1%–97.6% | g | 2,097 / 2,163 | Not measured | Not measured | — | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 96.8% (95%: 95.9%–97.4%) Logic 72.9% | — | 96.8%95.9%–97.4% | g | 2,093 / 2,163 | Not measured | Not measured | — | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Accuracy 94.7% (95%: 93.7%–95.6%) Logic 70.3% | — | 94.7%93.7%–95.6% | h | 2,049 / 2,163 | Not measured | Not measured | — | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Accuracy 94.6% (95%: 93.6%–95.5%) Logic 64.8% | — | 94.6%93.6%–95.5% | h | 2,047 / 2,163 | Not measured | Not measured | — | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Accuracy 94.0% (95%: 92.9%–94.9%) Logic 75.4% | — | 94.0%92.9%–94.9% | h | 2,033 / 2,163 | Not measured | Not measured | — | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 89.6% (95%: 88.2%–90.8%) Logic 65.1% | — | 89.6%88.2%–90.8% | i | 1,937 / 2,163 | Not measured | Not measured | — | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Google Liquid AI Alibaba eCloud Technology Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B | — | 98.7%235/238 | 96.1%196/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 97.0%130/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 98.1%103/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | 97.5%232/238 | 97.1%198/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 96.3%129/134 | 99.1%109/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 99.6%225/226 | 98.8%254/257 | 99.7%343/344 |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B | — | 97.5%232/238 | 97.5%199/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 95.5%128/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 98.7%223/226 | 98.4%253/257 | 100.0%344/344 |
| Gemma 4 E4B on · chat Open · 8.0B | — | 97.5%232/238 | 94.1%192/204 | 100.0%17/17 | 100.0%157/157 | 98.2%107/109 | 94.8%127/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 97.1%102/105 | 99.1%224/226 | 98.8%254/257 | 100.0%344/344 |
| Gemma 4 E4B chat Open · 8.0B | — | 98.3%234/238 | 95.1%194/204 | 100.0%17/17 | 98.7%155/157 | 99.1%108/109 | 95.5%128/134 | 98.2%108/110 | 98.0%99/101 | 98.1%158/161 | 96.2%101/105 | 98.7%223/226 | 99.2%255/257 | 99.4%342/344 |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 96.2%229/238 | 92.6%189/204 | 94.1%16/17 | 99.4%156/157 | 93.6%102/109 | 97.8%131/134 | 98.2%108/110 | 92.1%93/101 | 97.5%157/161 | 92.4%97/105 | 97.8%221/226 | 97.3%250/257 | 96.8%333/344 |
| Qwen3-14B off · chat Open · 15B | — | 94.5%225/238 | 83.3%170/204 | 100.0%17/17 | 96.8%152/157 | 99.1%108/109 | 94.8%127/134 | 98.2%108/110 | 95.0%96/101 | 91.3%147/161 | 100.0%105/105 | 99.1%224/226 | 98.4%253/257 | 98.8%340/344 |
| Erk-14B off · chat Open · 15B | — | 91.6%218/238 | 72.1%147/204 | 100.0%17/17 | 83.4%131/157 | 82.6%90/109 | 94.0%126/134 | 94.5%104/110 | 94.1%95/101 | 89.4%144/161 | 97.1%102/105 | 95.6%216/226 | 94.9%244/257 | 95.6%329/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Accuracy 98.8% (95%: 98.3%–99.2%) Logic 87.7% | — | 98.8%98.3%–99.2% | a | 2,138 / 2,163 | Not measured | Not measured | — | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.5% (mean of 3 seeds; min–max 98.4%–98.8%) Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | a | 2,136 / 2,163seed 0 run | Not measured | Not measured | — | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.5% (95%: 97.9%–99.0%) Logic 85.0% | — | 98.5%97.9%–99.0% | a | 2,131 / 2,163 | Not measured | Not measured | — | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.2% (95%: 97.6%–98.7%) Logic 82.6% | — | 98.2%97.6%–98.7% | a | 2,125 / 2,163 | Not measured | Not measured | — | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 98.1% (95%: 97.4%–98.6%) Logic 83.2% | — | 98.1%97.4%–98.6% | a | 2,122 / 2,163 | Not measured | Not measured | — | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Accuracy 96.3% (95%: 95.4%–97.0%) Logic 60.2% | — | 96.3%95.4%–97.0% | b | 2,082 / 2,163 | Not measured | Not measured | — | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Accuracy 95.8% (95%: 94.9%–96.6%) Logic 65.3% | — | 95.8%94.9%–96.6% | b | 2,072 / 2,163 | Not measured | Not measured | — | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Accuracy 90.8% (95%: 89.5%–91.9%) Logic 54.2% | — | 90.8%89.5%–91.9% | c | 1,963 / 2,163 | Not measured | Not measured | — | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Google Alibaba Liquid AI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B | — | 99.2%236/238 | 98.0%200/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 99.0%104/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| Qwen3.8-27B low · chat Open · 27B | — | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 98.7%155/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 99.0%104/105 | 99.1%224/226 | 99.6%256/257 | 99.4%342/344 |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | 98.7%235/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 96.3%105/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 99.4%160/161 | 98.1%103/105 | 99.6%225/226 | 98.4%253/257 | 99.7%343/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Accuracy 99.4% (95%: 99.0%–99.6%) Logic 93.3% | — | 99.4%99.0%–99.6% | a | 2,150 / 2,163 | Not measured | Not measured | — | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Accuracy 99.2% (95%: 98.7%–99.5%) Logic 94.0% | — | 99.2%98.7%–99.5% | a | 2,146 / 2,163 | Not measured | Not measured | — | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Accuracy 99.0% (95%: 98.5%–99.4%) Logic 86.7% | — | 99.0%98.5%–99.4% | a | 2,142 / 2,163 | Not measured | Not measured | — | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
DeepSeek Thinking Machines Xiaomi Z.ai MiniMax Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 99.0%100/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | 100.0%238/238 | 99.0%202/204 | 100.0%17/17 | 99.4%156/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 99.6%225/226 | 98.8%254/257 | 100.0%344/344 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active | — | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 99.4%160/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | 98.7%235/238 | 98.0%200/204 | 100.0%17/17 | 98.1%154/157 | 98.2%107/109 | 97.8%131/134 | 98.2%108/110 | 99.0%100/101 | 98.1%158/161 | 100.0%105/105 | 100.0%226/226 | 98.4%253/257 | 99.7%343/344 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | 98.7%235/238 | 97.5%199/204 | 100.0%17/17 | 99.4%156/157 | 98.2%107/109 | 98.5%132/134 | 96.4%106/110 | 95.0%96/101 | 98.8%159/161 | 99.0%104/105 | 98.2%222/226 | 98.1%252/257 | 98.8%340/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.2% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | — | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Accuracy 99.6% (95%: 99.3%–99.8%) Logic 95.4% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Accuracy 99.5% (95%: 99.2%–99.7%) Logic 95.6% | — | 99.5%99.2%–99.7% | a | 2,153 / 2,163 | Not measured | Not measured | — | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Accuracy 99.4% (95%: 99.0%–99.7%) Logic 95.1% | — | 99.4%99.0%–99.7% | ab | 2,151 / 2,163 | Not measured | Not measured | — | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Accuracy 99.3% (95%: 98.9%–99.6%) Logic running | — | 99.3%98.9%–99.6% | ab | 2,148 / 2,163 | Not measured | Not measured | — | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Accuracy 98.8% (95%: 98.2%–99.2%) Logic 93.2% | — | 98.8%98.2%–99.2% | bc | 2,137 / 2,163 | Not measured | Not measured | — | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Accuracy 98.2% (95%: 97.6%–98.7%) Logic 94.0% | — | 98.2%97.6%–98.7% | c | 2,125 / 2,163 | Not measured | Not measured | — | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Meta OpenAI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 99.0%100/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| GPT-6 Luna chat Closed · Efficient | — | 90.8%216/238 | 81.4%166/204 | 82.4%14/17 | 94.3%148/157 | 89.9%98/109 | 96.3%129/134 | 96.4%106/110 | 93.1%94/101 | 93.2%150/161 | 98.1%103/105 | 96.9%219/226 | 96.9%249/257 | 97.1%334/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Accuracy 99.6% (95%: 99.2%–99.8%) Logic 95.5% | — | 99.6%99.2%–99.8% | a | 2,154 / 2,163 | Not measured | Not measured | — | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 93.7% (95%: 92.6%–94.6%) Logic 58.0% | — | 93.7%92.6%–94.6% | b | 2,026 / 2,163 | Not measured | Not measured | — | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse | — | 99.6%237/238 | 99.0%202/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.7% (95%: 99.3%–99.8%) Logic 95.5% | — | 99.7%99.3%–99.8% | a | 2,156 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship | — | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Claude Fable 5.1 chat Closed · Flagship | — | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 99.4%156/157 | 100.0%109/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.8% (95%: 99.5%–99.9%) Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.7% (95%: 99.4%–99.9%) Logic 95.2% | — | 99.7%99.4%–99.9% | a | 2,157 / 2,163 | Not measured | Not measured | — | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Alibaba OpenAI Google xAI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Basic concepts238 questions | Digits and place value204 questions | Number bases17 questions | Division and divisibility157 questions | Prime factorisation and GCD/LCM109 questions | Rational numbers134 questions | Decimals110 questions | Ordering101 questions | Simple inequalities161 questions | Absolute value105 questions | Exponents226 questions | Radicals257 questions | Factorisation344 questions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed | — | 100.0%238/238 | 100.0%204/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 100.0%344/344 |
| GPT-6 Astra chat Closed · Undisclosed | — | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| GPT-6.1 Sol chat Closed · Undisclosed | — | 100.0%238/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 100.0%344/344 |
| Gemini 4 Carbon chat Closed · Undisclosed | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 100.0%110/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 99.6%225/226 | 99.2%255/257 | 99.7%343/344 |
| Grok 4.7 chat Closed · Undisclosed | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 100.0%161/161 | 100.0%105/105 | 100.0%226/226 | 98.8%254/257 | 99.7%343/344 |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | 99.6%237/238 | 99.5%203/204 | 100.0%17/17 | 100.0%157/157 | 100.0%109/109 | 98.5%132/134 | 98.2%108/110 | 100.0%101/101 | 98.8%159/161 | 100.0%105/105 | 100.0%226/226 | 99.2%255/257 | 99.7%343/344 |
| GPT-6.1 Terra chat Closed · Undisclosed | — | 99.6%237/238 | 98.5%201/204 | 100.0%17/17 | 99.4%156/157 | 99.1%108/109 | 98.5%132/134 | 99.1%109/110 | 100.0%101/101 | 98.1%158/161 | 100.0%105/105 | 99.6%225/226 | 98.8%254/257 | 99.7%343/344 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Accuracy 99.8% (95%: 99.5%–99.9%) Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | — | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.7% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.6% (95%: 99.3%–99.8%) Logic 96.5% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | — | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.6% (95%: 99.2%–99.8%) Logic 96.7% | — | 99.6%99.2%–99.8% | ab | 2,154 / 2,163 | Not measured | Not measured | — | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 99.5% (95%: 99.2%–99.7%) Logic running | — | 99.5%99.2%–99.7% | ab | 2,153 / 2,163 | Not measured | Not measured | — | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Accuracy 99.5% (95%: 99.1%–99.7%) Logic 94.0% | — | 99.5%99.1%–99.7% | ab | 2,152 / 2,163 | Not measured | Not measured | — | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Accuracy 99.2% (95%: 98.7%–99.5%) Logic 94.0% | — | 99.2%98.7%–99.5% | b | 2,146 / 2,163 | Not measured | Not measured | — | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.