Logic · Accuracy
Accuracy on Logic
1,125 questions, asked in English. How to read this page · Method
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
OpenAI Google Alibaba Anthropic DeepSeek Thinking Machines Meta Xiaomi MiniMax Z.ai IBM OpenBMB Liquid AI Other KRAFTON MachinePulse REAX PelaAI Poindexter Labs Fastino Sthānika AI Logits Nace.AI Perplexity vLLM Semantic Router Base Labs eCloud Technology Subquadratic Turion.AI tattrongvu TypeSafe Celeris StartLux newsbang Patrick-18 Inception Microsoft Valen-Team scar-ai Apertix AI Zefan Cai Cloudflare OmniJev quincyqiang victorbennett EldanRing Blockbrain Labs Metask Lab Strands Agents Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed | — | Chat | 98.3%515/524 | 95.5%448/469 | 94.7%125/132 |
| Gemini 4 Carbon chat Closed · Undisclosed | — | Chat | 98.1%514/524 | 95.7%449/469 | 94.7%125/132 |
| GPT-6.1 Sol chat Closed · Undisclosed | — | Chat | 97.9%513/524 | 95.5%448/469 | 94.7%125/132 |
| Qwen4-Max high · chat Closed · Undisclosed | — | Chat | 97.9%513/524 | 95.3%447/469 | 93.2%123/132 |
| Claude Opus 5.5 chat Closed · Flagship | — | Chat | 98.3%515/524 | 94.7%444/469 | 93.2%123/132 |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | Chat | 98.3%515/524 | 94.7%444/469 | 93.2%123/132 |
| Gemini 4 Hydrogen chat | — | Chat | 97.9%513/524 | 94.5%443/469 | 94.7%125/132 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | Chat | 98.1%514/524 | 93.6%439/469 | 93.2%123/132 |
| Claude Sonnet 5.5 chat Closed · Workhorse | — | Chat | 97.5%511/524 | 93.6%439/469 | 93.9%124/132 |
| Muse Ember max · chat Closed · Efficient | — | Chat | 97.9%513/524 | 93.2%437/469 | 93.9%124/132 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | Chat | 97.9%513/524 | 92.8%435/469 | 94.7%125/132 |
| Claude Fable 5.1 chat Closed · Flagship | — | Chat | 97.7%512/524 | 93.0%436/469 | 93.2%123/132 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | Chat | 97.3%510/524 | 92.8%435/469 | 94.7%125/132 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active | — | Chat | 98.1%514/524 | 92.5%434/469 | 92.4%122/132 |
| Qwen3.8-27B low · chat Open · 27B | — | Chat | 97.7%512/524 | 90.0%422/469 | 93.9%124/132 |
| GPT-6.1 Terra chat Closed · Undisclosed | — | Chat | 97.3%510/524 | 90.4%424/469 | 93.2%123/132 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | Chat | 95.8%502/524 | 92.5%434/469 | 91.7%121/132 |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | Chat | 97.1%509/524 | 90.8%426/469 | 92.4%122/132 |
| Gemma 4 31B chat Open · 31B | — | Chat | 97.1%509/524 | 89.3%419/469 | 92.4%122/132 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | Chat | 95.4%500/524 | 90.6%425/469 | 93.2%123/132 |
| Qwen3.5-4B on · chat Open · 4.7B | — | Chat | 96.4%505/524 | 88.1%413/469 | 91.7%121/132 |
| Granite-4.2-3B high · chat Open · 3.0B | — | Chat | 94.8%497/524 | 87.4%410/469 | 93.9%124/132 |
| Qwen3.5-4B off · chat Open · 4.7B | — | Chat | 95.8%502/524 | 83.6%392/469 | 90.9%120/132 |
| MiniCPM5-2B on · chat Open · 2.5B | — | Chat | 95.4%500/524 | 79.7%374/469 | 93.9%124/132 |
| Gemma 4 12B Unified chat Open · 12B | — | Chat | 92.4%484/524 | 81.0%380/469 | 93.2%123/132 |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | Chat | 93.3%489/524 | 77.0%361/469 | 94.7%125/132 |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | Chat | 92.4%484/524 | 77.4%363/469 | 91.7%121/132 |
| LFM3-1.7B off · chat Open · 1.7B | — | Chat | 90.1%472/524 | 78.7%369/469 | 91.7%121/132 |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B | — | Chat | 90.5%474/524 | 76.8%360/469 | 92.4%122/132 |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B | — | Chat | 90.3%473/524 | 77.2%362/469 | 91.7%121/132 |
| LFM3-1.7B on · chat Open · 1.7B | — | Chat | 88.2%462/524 | 78.3%367/469 | 91.7%121/132 |
| MiniCPM5-2B off · chat Open · 2.5B | — | Chat | 90.6%475/524 | 75.1%352/469 | 87.1%115/132 |
| Gemma 4 E4B chat Open · 8.0B | — | Chat | 89.3%468/524 | 73.1%343/469 | 94.7%125/132 |
| Gemma 4 E4B on · chat Open · 8.0B | — | Chat | 88.7%465/524 | 73.1%343/469 | 91.7%121/132 |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | Chat | 85.1%446/524 | 62.0%291/469 | 89.4%118/132 |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B | — | Chat | 84.5%443/524 | 62.9%295/469 | 87.9%116/132 |
| LFM2.5-2.6B on · chat Open · 2.7B | — | Chat | 83.2%436/524 | 63.8%299/469 | 85.6%113/132 |
| Gemma 4 E2B on · chat Open · 5.1B | — | Chat | 82.3%431/524 | 61.6%289/469 | 90.2%119/132 |
| Gemma 4 E2B chat Open · 5.1B | — | Chat | 79.6%417/524 | 61.0%286/469 | 88.6%117/132 |
| Raon-1 nano on · chat Open · 3.1B | — | Chat | 82.4%432/524 | 53.3%250/469 | 87.9%116/132 |
| Qwen3.5-2B on · chat Open · 2.0B | — | Chat | 77.3%405/524 | 58.4%274/469 | 84.8%112/132 |
| Qwen3-14B off · chat Open · 15B | — | Chat | 74.6%391/524 | 48.2%226/469 | 89.4%118/132 |
| Raon-1 nano off · chat Open · 3.1B | — | Chat | 70.8%371/524 | 53.9%253/469 | 81.8%108/132 |
| Qwen3.5-2B off · chat Open · 2.0B | — | Chat | 72.5%380/524 | 51.6%242/469 | 81.1%107/132 |
| Karpo Decision 31B Open · 31B | — | Decision | 50.8%266/524 | 64.0%300/469 | 88.6%117/132 |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | Chat | 75.4%395/524 | 36.9%173/469 | 82.6%109/132 |
| REAX 1.3 Open · 57B | — | Decision | 47.5%249/524 | 64.4%302/469 | 87.9%116/132 |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 47.1%247/524 | 64.2%301/469 | 87.9%116/132 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 47.1%247/524 | 63.8%299/469 | 88.6%117/132 |
| Copernicus-31B Open · 31B | — | Decision | 47.1%247/524 | 63.8%299/469 | 86.4%114/132 |
| GLiDE 27B Open · 27B | — | Decision | 42.0%220/524 | 68.0%319/469 | 89.4%118/132 |
| GPT-6 Luna chat Closed · Efficient | — | Chat | 67.0%351/524 | 39.7%186/469 | 87.1%115/132 |
| Sieve 31B v3 Open · 31B | — | Decision | 45.8%240/524 | 62.9%295/469 | 87.9%116/132 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 44.7%234/524 | 63.8%299/469 | 88.6%117/132 |
| GLiDE v1.2 Open · 27B | — | Decision | 40.5%212/524 | 63.1%296/469 | 85.6%113/132 |
| d1-ultra Closed · Efficient | — | Decision | 44.7%234/524 | 57.4%269/469 | 87.1%115/132 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 41.4%217/524 | 60.8%285/469 | 86.4%114/132 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 41.4%217/524 | 59.9%281/469 | 89.4%118/132 |
| d3-ultra Open · 26B | — | Decision | 42.2%221/524 | 59.3%278/469 | 87.1%115/132 |
| GLiDE API Closed · Efficient | — | Decision | 38.7%203/524 | 63.1%296/469 | 86.4%114/132 |
| Mammoth-1 Open · 27B | — | Decision | 42.7%224/524 | 58.6%275/469 | 86.4%114/132 |
| d3 Open · 26B | — | Decision | 40.6%213/524 | 60.6%284/469 | 87.9%116/132 |
| Erk-14B off · chat Open · 15B | — | Chat | 63.7%334/524 | 35.6%167/469 | 82.6%109/132 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 39.9%209/524 | 58.4%274/469 | 88.6%117/132 |
| Drex Max Open · 27B | — | Decision | 42.4%222/524 | 55.7%261/469 | 86.4%114/132 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 39.1%205/524 | 58.0%272/469 | 86.4%114/132 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 37.2%195/524 | 60.3%283/469 | 84.1%111/132 |
| Turion Jev 31B Open · 31B | — | Decision | 36.6%192/524 | 60.1%282/469 | 86.4%114/132 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 38.2%200/524 | 57.4%269/469 | 84.1%111/132 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 36.6%192/524 | 56.7%266/469 | 84.1%111/132 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 39.3%206/524 | 53.1%249/469 | 81.8%108/132 |
| d1.2 Closed · Undisclosed | — | Decision | 39.7%208/524 | 51.6%242/469 | 78.0%103/132 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 34.2%179/524 | 53.7%252/469 | 84.1%111/132 |
| Jev 1.14 Closed · Efficient | — | Decision | 35.7%187/524 | 53.1%249/469 | 80.3%106/132 |
| GLiDE 9B Open · 7.9B | — | Decision | 32.1%168/524 | 55.7%261/469 | 82.6%109/132 |
| celeris-1-decision Closed · Efficient | — | Decision | 37.2%195/524 | 50.5%237/469 | 75.8%100/132 |
| celeris-1.1-decision Closed · Efficient | — | Decision | 37.8%198/524 | 49.5%232/469 | 77.3%102/132 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 32.3%169/524 | 52.5%246/469 | 82.6%109/132 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 34.0%178/524 | 49.9%234/469 | 83.3%110/132 |
| Elephant-1 Open · 9.0B | — | Decision | 33.2%174/524 | 51.0%239/469 | 82.6%109/132 |
| Jev 1.13 Closed · Efficient | — | Decision | 33.2%174/524 | 52.0%244/469 | 78.0%103/132 |
| d1.1 Closed · Efficient | — | Decision | 34.7%182/524 | 50.7%238/469 | 74.2%98/132 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 32.4%170/524 | 50.7%238/469 | 81.8%108/132 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 34.9%183/524 | 47.3%222/469 | 79.5%105/132 |
| Mercury Decide Closed · Efficient | — | Decision | 33.4%175/524 | 51.6%242/469 | 68.2%90/132 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 30.7%161/524 | 50.1%235/469 | 81.1%107/132 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 34.0%178/524 | 46.9%220/469 | 77.3%102/132 |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 34.2%179/524 | 45.8%215/469 | 79.5%105/132 |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 32.4%170/524 | 47.5%223/469 | 79.5%105/132 |
| vera-medium-omni Open · 4.2B | — | Decision | 31.9%167/524 | 48.6%228/469 | 76.5%101/132 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 33.4%175/524 | 45.2%212/469 | 81.1%107/132 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 31.1%163/524 | 48.0%225/469 | 75.0%99/132 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 31.3%164/524 | 46.1%216/469 | 79.5%105/132 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 31.1%163/524 | 47.1%221/469 | 72.0%95/132 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 30.2%158/524 | 47.5%223/469 | 72.0%95/132 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 34.2%179/524 | 41.4%194/469 | 74.2%98/132 |
| Clef Open · 27B | — | Decision | 30.2%158/524 | 46.1%216/469 | 64.4%85/132 |
| d1 Closed · Efficient | — | Decision | 29.0%152/524 | 44.3%208/469 | 75.0%99/132 |
| OneJev-27B Open · 27B | — | Decision | 30.0%157/524 | 45.0%211/469 | 67.4%89/132 |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 29.0%152/524 | 43.9%206/469 | 72.7%96/132 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 28.4%149/524 | 42.4%199/469 | 75.0%99/132 |
| Winnow-12B Open · 12B | — | Decision | 26.5%139/524 | 45.4%213/469 | 72.0%95/132 |
| Cygnet Open · 12B | — | Decision | 26.9%141/524 | 47.3%222/469 | 59.8%79/132 |
| Clef-Flash Open · 9.0B | — | Decision | 30.0%157/524 | 40.1%188/469 | 70.5%93/132 |
| GLiDE 2B Open · 1.9B | — | Decision | 26.5%139/524 | 33.0%155/469 | 52.3%69/132 |
| Metask-Jev-4B Open · 4.5B | — | Decision | 24.6%129/524 | 34.5%162/469 | 53.0%70/132 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 25.0%131/524 | 32.2%151/469 | 56.8%75/132 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 25.8%135/524 | 29.9%140/469 | 53.8%71/132 |
| d1-2B Open · 1.9B | — | Decision | 24.8%130/524 | 31.8%149/469 | 50.0%66/132 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 25.6%134/524 | 29.0%136/469 | 43.9%58/132 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 24.0%126/524 | 27.5%129/469 | 37.1%49/132 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 25.2%132/524 | 24.3%114/469 | 40.9%54/132 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 19.5%102/524 | 25.6%120/469 | 42.4%56/132 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 23.9%125/524 | 22.0%103/469 | 28.0%37/132 |
| GLiDE 0.4B Open · 0.4B | — | Decision | 20.8%109/524 | 24.1%113/469 | 29.5%39/132 |
| Strands Decider 2B Open · 2.0B | — | Decision | 19.3%101/524 | 24.1%113/469 | 28.8%38/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 1,125 of 1,125); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Not ranked (only complete runs are): Spark-X2.6-4B (high · chat) (not running now, 20 of 1,125 stored, last answer 13 Nov 02:18 UTC); Grok 4.7 (chat) (paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)); Claude Fable 5.2 (chat) (not running now, 147 of 1,125 stored, last answer 12 Nov 11:32 UTC); Granite-4.2-3B (off · chat) (still running, 259 of 1,125); GPT-6 Luna (max · chat) (paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)); DeepSeek-V4-Flash (chat) (sample: a random subset by design, 317 answers stored, not the full 1,125); Claude Haiku 5.5 (max · chat) (paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Accuracy 96.7% (95%: 95.5%–97.6%) Math 99.6% | — | Chat | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 56.4%answered 97% | 6,590 ms | 14,330 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Accuracy 96.7% (95%: 95.5%–97.6%) Math 99.6% | — | Chat | 96.7%95.5%–97.6% | ab | 1,088 / 1,125 | Not measured | Not measured | — | 45.5%answered 98% | 44,520 ms | 206,660 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Accuracy 96.5% (95%: 95.3%–97.5%) Math 99.6% | — | Chat | 96.5%95.3%–97.5% | a | 1,086 / 1,125 | Not measured | Not measured | — | 63.2%answered 97% | 7,070 ms | 16,200 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Accuracy 96.3% (95%: 95.0%–97.2%) Math 99.8% | — | Chat | 96.3%95.0%–97.2% | abc | 1,083 / 1,125 | Not measured | Not measured | — | 0.0%answered 100% | 11,570 ms | 38,640 ms | $5.690$5,690 |
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 96.2% (95%: 94.9%–97.2%) Math 99.8% | — | Chat | 96.2%94.9%–97.2% | abcde | 1,082 / 1,125 | Not measured | Not measured | — | 38.2%answered 98% | 5,830 ms | 12,240 ms | $19.540$19,540 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Accuracy 96.2% (95%: 94.9%–97.2%) Math 99.6% | — | Chat | 96.2%94.9%–97.2% | abcde | 1,082 / 1,125 | Not measured | Not measured | — | 3.8%answered 100% | 6,680 ms | 40,040 ms | $1.647$1,647 |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Accuracy 96.1% (95%: 94.8%–97.1%) Math 99.6% | — | Chat | 96.1%94.8%–97.1% | abdf | 1,081 / 1,125 | Not measured | Not measured | — | 76.1%answered 95% | 8,210 ms | 24,090 ms | — |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Accuracy 95.6% (95%: 94.3%–96.7%) Math 99.5% | — | Chat | 95.6%94.3%–96.7% | abcdefg | 1,076 / 1,125 | Not measured | Not measured | — | 37.3%answered 98% | 13,180 ms | 144,250 ms | $4.121$4,121 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 95.5% (95%: 94.1%–96.5%) Math 99.7% | — | Chat | 95.5%94.1%–96.5% | abcdefg | 1,074 / 1,125 | Not measured | Not measured | — | 41.3%answered 98% | 6,710 ms | 14,890 ms | $11.480$11,480 API list-price equivalent |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Accuracy 95.5% (95%: 94.1%–96.5%) Math 99.6% | — | Chat | 95.5%94.1%–96.5% | abcdefg | 1,074 / 1,125 | Not measured | Not measured | — | 22.6%answered 99% | 3,110 ms | 14,020 ms | $1.820$1,820 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Accuracy 95.4% (95%: 94.0%–96.5%) Math 99.6% | — | Chat | 95.4%94.0%–96.5% | abcdefg | 1,073 / 1,125 | Not measured | Not measured | — | 50.0%answered 97% | 13,290 ms | 78,510 ms | $5.489$5,489 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 95.2% (95%: 93.8%–96.3%) Math 99.7% | — | Chat | 95.2%93.8%–96.3% | abcdefgh | 1,071 / 1,125 | Not measured | Not measured | — | 8.3%answered 100% | 3,820 ms | 13,460 ms | $0.000$0.00 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Accuracy 95.1% (95%: 93.7%–96.2%) Math 99.4% | — | Chat | 95.1%93.7%–96.2% | abcdefgh | 1,070 / 1,125 | Not measured | Not measured | — | 23.3%answered 98% | 24,740 ms | 179,930 ms | $1.390$1,390 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active Qwen3.5-397B-A17B on · chat · Alibaba · chat model · reasoning on Parameters: 397B MoE · 17B active Accuracy 95.1% (95%: 93.7%–96.2%) Math running | — | Chat | 95.1%93.7%–96.2% | abcdefgh | 1,070 / 1,125 | Not measured | Not measured | — | 31.7%answered 98% | 84,650 ms | 167,520 ms | $13.426$13,426 |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Accuracy 94.0% (95%: 92.5%–95.3%) Math 99.2% | — | Chat | 94.0%92.5%–95.3% | bcdefghi | 1,058 / 1,125 | Not measured | Not measured | — | 39.1%answered 97% | 18,710 ms | 101,030 ms | $4.884$4,884estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Accuracy 94.0% (95%: 92.4%–95.2%) Math 99.2% | — | Chat | 94.0%92.4%–95.2% | ceghi | 1,057 / 1,125 | Not measured | Not measured | — | 61.1%answered 95% | 5,640 ms | 10,330 ms | $0.000$0.00 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Accuracy 94.0% (95%: 92.4%–95.2%) Math 98.2% | — | Chat | 94.0%92.4%–95.2% | bcdefghi | 1,057 / 1,125 | Not measured | Not measured | — | 38.3%answered 97% | 5,610 ms | 37,430 ms | $0.803$803 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Accuracy 94.0% (95%: 92.4%–95.2%) Math 99.5% | — | Chat | 94.0%92.4%–95.2% | defghi | 1,057 / 1,125 | Not measured | Not measured | — | 34.6%answered 98% | 17,550 ms | 73,220 ms | $1.376$1,376 |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Accuracy 93.3% (95%: 91.7%–94.6%) Math 99.4% | — | Chat | 93.3%91.7%–94.6% | fghij | 1,050 / 1,125 | Not measured | Not measured | — | 16.3%answered 99% | 18,290 ms | 39,790 ms | $1.244$1,244estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Accuracy 93.2% (95%: 91.5%–94.5%) Math 98.8% | — | Chat | 93.2%91.5%–94.5% | ghij | 1,048 / 1,125 | Not measured | Not measured | — | 4.2%answered 100% | 12,050 ms | 82,830 ms | $1.057$1,057 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Accuracy 92.4% (95%: 90.7%–93.8%) Math 99.4% | — | Chat | 92.4%90.7%–93.8% | hij | 1,039 / 1,125 | Not measured | Not measured | — | 36.1%answered 97% | 142,520 ms | 246,280 ms | $3.525$3,525estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Accuracy 91.6% (95%: 89.9%–93.1%) Math 99.2% | — | Chat | 91.6%89.9%–93.1% | ijk | 1,031 / 1,125 | Not measured | Not measured | — | 53.2%answered 94% | 221,350 ms | 1,229,550 ms | $10.893$10,893estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Accuracy 90.1% (95%: 88.3%–91.7%) Math 98.5% | — | Chat | 90.1%88.3%–91.7% | jkl | 1,014 / 1,125 | Not measured | Not measured | — | 33.6%answered 96% | 45,380 ms | 127,510 ms | $1.420$1,420estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Accuracy 88.9% (mean of 3 seeds; min–max 88.3%–89.6%) Math 99.2% | — | Chat | 88.9%88.3%–89.6% min–max of 3 seeds | klm | 998 / 1,125seed 0 run | Not measured | Not measured | — | 51.5%answered 92% | 101,100 ms | 333,427 ms | $3.335$3,335estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Accuracy 87.7% (95%: 85.7%–89.5%) Math 98.8% | — | Chat | 87.7%85.7%–89.5% | klmn | 987 / 1,125 | Not measured | Not measured | — | 15.5%answered 98% | 42,060 ms | 82,940 ms | $1.292$1,292estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Accuracy 86.7% (95%: 84.6%–88.5%) Math 99.0% | — | Chat | 86.7%84.6%–88.5% | lmno | 975 / 1,125 | Not measured | Not measured | — | 37.8%answered 95% | 228,980 ms | 776,930 ms | $7.562$7,562estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 85.9% (mean of 3 seeds; min–max 85.5%–86.1%) Math 98.5% | — | Chat | 85.9%85.5%–86.1% min–max of 3 seeds | lmno | 968 / 1,125seed 0 run | Not measured | Not measured | — | 55.9%answered 90% | 31,567 ms | 83,467 ms | $0.993$993estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Accuracy 85.5% (95%: 83.3%–87.4%) Math 98.7% | — | Chat | 85.5%83.3%–87.4% | lmno | 962 / 1,125 | Not measured | Not measured | — | 54.2%answered 90% | 163,870 ms | 656,750 ms | $0.487$487estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 85.0% (95%: 82.8%–86.9%) Math 98.5% | — | Chat | 85.0%82.8%–86.9% | mno | 956 / 1,125 | Not measured | Not measured | — | Not measured | 57,900 ms | 109,440 ms | $1.503$1,503estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Accuracy 84.6% (mean of 3 seeds; min–max 84.4%–85.0%) Math 98.9% | — | Chat | 84.6%84.4%–85.0% min–max of 3 seeds | mno | 956 / 1,125seed 0 run | Not measured | Not measured | — | Not measured | 27,950 ms | 107,497 ms | $1.040$1,040estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Accuracy 84.2% (mean of 3 seeds; min–max 83.9%–84.4%) Math 98.9% | — | Chat | 84.2%83.9%–84.4% min–max of 3 seeds | mno | 950 / 1,125seed 0 run | Not measured | Not measured | — | 43.6%answered 91% | 29,847 ms | 113,220 ms | $1.017$1,017estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Accuracy 83.6% (mean of 3 seeds; min–max 83.1%–83.9%) Math 98.2% | — | Chat | 83.6%83.1%–83.9% min–max of 3 seeds | no | 942 / 1,125seed 0 run | Not measured | Not measured | — | 43.9%answered 88% | 46,453 ms | 111,607 ms | $1.327$1,327estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 83.2% (95%: 80.9%–85.3%) Math 98.1% | — | Chat | 83.2%80.9%–85.3% | no | 936 / 1,125 | Not measured | Not measured | — | 52.4%answered 88% | 32,690 ms | 64,070 ms | $0.842$842estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 82.6% (95%: 80.3%–84.7%) Math 98.2% | — | Chat | 82.6%80.3%–84.7% | o | 929 / 1,125 | Not measured | Not measured | — | 45.1%answered 91% | 284,080 ms | 539,980 ms | $0.650$650estimate · DGX Spark |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 76.8% (mean of 3 seeds; min–max 76.0%–77.6%) Math 97.7% | — | Chat | 76.8%76.0%–77.6% min–max of 3 seeds | p | 855 / 1,125seed 0 run | Not measured | Not measured | — | 32.6%answered 91% | 21,223 ms | 45,077 ms | $0.574$574estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 75.9% (95%: 73.3%–78.3%) Math 97.8% | — | Chat | 75.9%73.3%–78.3% | p | 854 / 1,125 | Not measured | Not measured | — | Not measured | 41,350 ms | 70,680 ms | $1.193$1,193estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Accuracy 75.4% (95%: 72.8%–77.8%) Math 94.0% | — | Chat | 75.4%72.8%–77.8% | p | 848 / 1,125 | Not measured | Not measured | — | 44.3%answered 86% | 81,330 ms | 224,000 ms | $2.480$2,480estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 74.6% (95%: 72.0%–77.0%) Math 96.9% | — | Chat | 74.6%72.0%–77.0% | p | 839 / 1,125 | Not measured | Not measured | — | 26.4%answered 93% | 173,220 ms | 297,090 ms | $0.363$363estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 72.9% (95%: 70.2%–75.4%) Math 96.8% | — | Chat | 72.9%70.2%–75.4% | p | 820 / 1,125 | Not measured | Not measured | — | 34.0%answered 90% | 21,390 ms | 35,160 ms | $0.513$513estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 70.9% (95%: 68.2%–73.5%) Math 97.0% | — | Chat | 70.9%68.2%–73.5% | pq | 798 / 1,125 | Not measured | Not measured | — | 10.5%answered 95% | 493,230 ms† | 1,396,520 ms† | $15.330$15,330estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Accuracy 70.3% (95%: 67.6%–72.9%) Math 94.7% | — | Chat | 70.3%67.6%–72.9% | pq | 791 / 1,125 | Not measured | Not measured | — | 33.1%answered 91% | 186,330 ms | 272,210 ms | $4.330$4,330estimate · RTX PRO 6000 Blackwell |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Accuracy 65.3% (95%: 62.5%–68.1%) Math 95.8% | — | Chat | 65.3%62.5%–68.1% | qr | 735 / 1,125 | Not measured | Not measured | — | 16.1%answered 94% | 16,560 ms | 178,680 ms | $1.772$1,772estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 65.1% (95%: 62.2%–67.8%) Math 89.6% | — | Chat | 65.1%62.2%–67.8% | qrs | 732 / 1,125 | Not measured | Not measured | — | 35.7%answered 87% | 276,720 ms† | 965,630 ms† | $9.270$9,270estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Accuracy 64.8% (95%: 62.0%–67.5%) Math 94.6% | — | Chat | 64.8%62.0%–67.5% | qrs | 729 / 1,125 | Not measured | Not measured | — | 35.2%answered 86% | 71,290 ms | 113,470 ms | $1.605$1,605estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Accuracy 60.7% (95%: 57.8%–63.5%) Math 81.6% | — | Decision | 60.7%57.8%–63.5% | rstu | 683 / 1,125 | 169 right of 172 kept3 wrong · 953 skipped | 0.816 | 71.80chosen-answer Brier 0.1711 | 6.6%answered 97% | 173 ms | 211 ms | $0.128$128estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Accuracy 60.2% (95%: 57.3%–63.0%) Math 96.3% | — | Chat | 60.2%57.3%–63.0% | rstuvwxyzA | 677 / 1,125 | Not measured | Not measured | — | 58.3%answered 70% | 138,140 ms† | 403,380 ms† | $4.664$4,664estimate · RTX PRO 6000 Blackwell† |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Accuracy 59.3% (95%: 56.4%–62.1%) Math 78.0% | — | Decision | 59.3%56.4%–62.1% | rstv | 667 / 1,125 | 51 right of 51 kept0 mistakes · 1,074 skipped | 0.816 | 71.07chosen-answer Brier 0.1714 | 5.3%answered 98% | 215 ms | 278 ms | $0.160$160estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Accuracy 59.0% (95%: 56.1%–61.9%) Math 77.4% | — | Decision | 59.0%56.1%–61.9% | rstuvw | 664 / 1,125 | 89 right of 89 kept0 mistakes · 1,036 skipped | 0.804 | 70.72chosen-answer Brier 0.1757 | 0.4%answered 100% | 983 ms | 1,279 ms | $0.061$61.43estimate · DGX Spark |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Accuracy 58.9% (95%: 56.0%–61.8%) Math 80.3% | — | Decision | 58.9%56.0%–61.8% | rstuvw | 663 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.825 | 71.09chosen-answer Brier 0.1675 | 0.0%answered 100% | 151 ms | 204 ms | $0.112$112estimate · RTX PRO 6000 Blackwell |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Accuracy 58.7% (95%: 55.8%–61.5%) Math 77.3% | — | Decision | 58.7%55.8%–61.5% | rstuvwx | 660 / 1,125 | 66 right of 66 kept0 mistakes · 1,059 skipped | 0.801 | 70.48chosen-answer Brier 0.1771 | 1.9%answered 99% | 185 ms | 255 ms | $0.136$136estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Accuracy 58.4% (95%: 55.5%–61.2%) Math 72.0% | — | Decision | 58.4%55.5%–61.2% | rstuvwxy | 657 / 1,125 | 74 right of 75 kept1 wrong · 1,050 skipped | 0.822 | 70.67chosen-answer Brier 0.1707 | 44.3%answered 79% | 544 ms | 655 ms | $0.033$33.16estimate · DGX Spark |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 58.0% (95%: 55.0%–60.8%) Math 93.7% | — | Chat | 58.0%55.0%–60.8% | stuvwxyzAB | 652 / 1,125 | Not measured | Not measured | — | 55.1%answered 72% | 3,860 ms | 5,820 ms | $0.000$0.00 |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Accuracy 57.9% (95%: 55.0%–60.7%) Math 77.1% | — | Decision | 57.9%55.0%–60.7% | rstuvwxz | 651 / 1,125 | 79 right of 81 kept2 wrong · 1,044 skipped | 0.797 | 70.03chosen-answer Brier 0.1781 | 5.5%answered 98% | 1,050 ms | 1,335 ms | $0.063$62.95estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Accuracy 57.8% (95%: 54.9%–60.6%) Math 75.2% | — | Decision | 57.8%54.9%–60.6% | rstuvwxyzA | 650 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.796 | 69.90chosen-answer Brier 0.1798 | 7.1%answered 97% | 285 ms | 399 ms | $0.211$211estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Accuracy 55.2% (95%: 52.3%–58.1%) Math 69.9% | — | Decision | 55.2%52.3%–58.1% | tuvwxyzABC | 621 / 1,125 | 45 right of 45 kept0 mistakes · 1,080 skipped | 0.793 | 68.39chosen-answer Brier 0.1842 | 50.2%answered 73% | 111 ms | 140 ms | $0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Accuracy 54.9% (95%: 52.0%–57.8%) Math 74.1% | — | Decision | 54.9%52.0%–57.8% | tuvwxyzABC | 618 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.783 | 68.09chosen-answer Brier 0.1876 | 91.0%answered 35% | 182 ms | 425 ms | $0.013$12.64 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Accuracy 54.8% (95%: 51.8%–57.6%) Math 70.1% | — | Decision | 54.8%51.8%–57.6% | tuvwxyzABCD | 616 / 1,125 | 89 right of 90 kept1 wrong · 1,035 skipped | 0.793 | 68.07chosen-answer Brier 0.1861 | 54.1%answered 69% | 329 ms | 1,540 ms | $0.022$22.37 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Accuracy 54.8% (95%: 51.8%–57.6%) Math 72.5% | — | Decision | 54.8%51.8%–57.6% | tuvwxyzABCD | 616 / 1,125 | 62 right of 62 kept0 mistakes · 1,063 skipped | 0.803 | 67.98chosen-answer Brier 0.1881 | 18.1%answered 91% | 108 ms | 131 ms | $0.081$81.16estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 54.6% (95%: 51.7%–57.5%) Math 71.5% | — | Decision | 54.6%51.7%–57.5% | uwxyzABCDE | 614 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.778 | 66.46chosen-answer Brier 0.2166 | 10.1%answered 95% | 156 ms | 185 ms | $0.109$109estimate · RTX PRO 6000 Blackwell |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Accuracy 54.5% (95%: 51.6%–57.4%) Math 70.8% | — | Decision | 54.5%51.6%–57.4% | vwxyzABCD | 613 / 1,125 | 52 right of 52 kept0 mistakes · 1,073 skipped | 0.814 | 68.27chosen-answer Brier 0.1794 | 60.0%answered 67% | 271 ms | 291 ms | $0.052$52.03 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Accuracy 54.5% (95%: 51.6%–57.4%) Math 74.0% | — | Decision | 54.5%51.6%–57.4% | vwxyzABCDE | 613 / 1,125 | 106 right of 108 kept2 wrong · 1,017 skipped | 0.795 | 68.07chosen-answer Brier 0.1835 | 72.3%answered 58% | 221 ms | 244 ms | $0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 54.5% (95%: 51.6%–57.4%) Math 70.4% | — | Decision | 54.5%51.6%–57.4% | vwxyzABCD | 613 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.806 | 68.34chosen-answer Brier 0.1781 | 14.8%answered 93% | 76 ms | 98 ms | $0.057$57.18estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Accuracy 54.2% (95%: 51.3%–57.1%) Math 90.8% | — | Chat | 54.2%51.3%–57.1% | tuvwxyzABCDEFG | 610 / 1,125 | Not measured | Not measured | — | 30.7%answered 84% | 10,150 ms | 18,410 ms | $0.390$390estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Accuracy 53.3% (95%: 50.4%–56.2%) Math 69.8% | — | Decision | 53.3%50.4%–56.2% | wxyzABCDEF | 600 / 1,125 | 78 right of 80 kept2 wrong · 1,045 skipped | 0.798 | 67.64chosen-answer Brier 0.1805 | 14.4%answered 93% | 238 ms | 552 ms | $0.007$7.46 |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Accuracy 53.1% (95%: 50.1%–56.0%) Math 70.3% | — | Decision | 53.1%50.1%–56.0% | wxyzABCDEF | 597 / 1,125 | 147 right of 155 kept8 wrong · 970 skipped | 0.786 | 65.68chosen-answer Brier 0.2170 | 7.3%answered 97% | 106 ms | 143 ms | $0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Accuracy 52.5% (95%: 49.6%–55.4%) Math 69.3% | with web | Decision | 52.5%49.6%–55.4% | xyzABCDEFGH | 591 / 1,125 | 78 right of 79 kept1 wrong · 1,046 skipped | 0.800 | 67.24chosen-answer Brier 0.1806 | 37.7%answered 80% | 448 ms | 827 ms | $1.030$1,030 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Accuracy 52.4% (95%: 49.4%–55.3%) Math 68.6% | — | Decision | 52.4%49.4%–55.3% | zABCDEFGHI | 589 / 1,125 | 70 right of 72 kept2 wrong · 1,053 skipped | 0.787 | 65.94chosen-answer Brier 0.2047 | 65.7%answered 61% | 384 ms | 393 ms | $0.015$14.92 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Accuracy 52.3% (95%: 49.3%–55.2%) Math 69.0% | — | Decision | 52.3%49.3%–55.2% | yABCDEFGHI | 588 / 1,125 | 7 right of 7 kept0 mistakes · 1,118 skipped | 0.782 | 66.89chosen-answer Brier 0.1849 | 1.5%answered 99% | 385 ms | 533 ms | $0.025$24.91estimate · DGX Spark |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 51.6% (95%: 48.6%–54.5%) Math 69.9% | — | Decision | 51.6%48.6%–54.5% | BCDEFGHIJ | 580 / 1,125 | 13 right of 13 kept0 mistakes · 1,112 skipped | 0.800 | 66.77chosen-answer Brier 0.1802 | 94.5%answered 28% | 220 ms | 341 ms | $0.013$12.64 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Accuracy 50.6% (95%: 47.7%–53.5%) Math 66.3% | — | Decision | 50.6%47.7%–53.5% | BCDEFGHIJK | 569 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.774 | 65.90chosen-answer Brier 0.1878 | 8.8%answered 95% | 68 ms | 88 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Accuracy 50.0% (95%: 47.1%–53.0%) Math 71.3% | — | Decision | 50.0%47.1%–53.0% | CDEFGHIJKL | 563 / 1,125 | 43 right of 43 kept0 mistakes · 1,082 skipped | 0.768 | 65.37chosen-answer Brier 0.1930 | 2.0%answered 99% | 228 ms | 344 ms | $0.037$36.77 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 49.2% (95%: 46.2%–52.1%) Math 65.5% | — | Decision | 49.2%46.2%–52.1% | CDEFGHIJKLM | 553 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.742 | 64.51chosen-answer Brier 0.2013 | 78.8%answered 43% | 222 ms | 390 ms | $0.006$6.02 |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 48.2% (95%: 45.3%–51.1%) Math 59.8% | — | Decision | 48.2%45.3%–51.1% | EFGHIJKLMN | 542 / 1,125 | 10 right of 10 kept0 mistakes · 1,115 skipped | 0.753 | 64.33chosen-answer Brier 0.1952 | 23.4%answered 87% | 52 ms | 66 ms | $0.039$39.35estimate · RTX PRO 6000 Blackwell |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 48.2% (95%: 45.3%–51.1%) Math 65.1% | — | Decision | 48.2%45.3%–51.1% | DEFGHIJKLMN | 542 / 1,125 | 61 right of 61 kept0 mistakes · 1,064 skipped | 0.750 | 64.14chosen-answer Brier 0.1990 | 24.5%answered 86% | 166 ms | 193 ms | $0.026$25.64 |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Accuracy 47.8% (95%: 44.9%–50.7%) Math 57.9% | — | Decision | 47.8%44.9%–50.7% | FGHIJKLMN | 538 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.749 | 63.95chosen-answer Brier 0.1993 | 30.4%answered 82% | 208 ms | 251 ms | $0.013$13.29estimate · DGX Spark |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 47.3% (95%: 44.4%–50.2%) Math 70.0% | — | Decision | 47.3%44.4%–50.2% | FGHIJKLMNO | 532 / 1,125 | No answers at this threshold | 0.735 | 62.94chosen-answer Brier 0.2141 | 92.3%answered 28% | 390 ms | 394 ms | $0.015$15.40 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 47.3% (95%: 44.4%–50.2%) Math 69.9% | — | Decision | 47.3%44.4%–50.2% | FGHIJKLMNO | 532 / 1,125 | No answers at this threshold | 0.733 | 62.92chosen-answer Brier 0.2144 | 91.0%answered 29% | 383 ms | 390 ms | $0.002$1.54 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Accuracy 46.6% (95%: 43.7%–49.5%) Math 58.4% | — | Decision | 46.6%43.7%–49.5% | GHIJKLMNOP | 524 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.762 | 63.55chosen-answer Brier 0.1947 | 17.4%answered 90% | 160 ms | 175 ms | $0.010$10.04estimate · DGX Spark |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 46.4% (95%: 43.5%–49.3%) Math 62.1% | — | Decision | 46.4%43.5%–49.3% | GHIJKLMNOP | 522 / 1,125 | 54 right of 55 kept1 wrong · 1,070 skipped | 0.771 | 63.48chosen-answer Brier 0.1945 | 12.2%answered 93% | 41 ms | 46 ms | $0.031$31.36estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Accuracy 46.4% (95%: 43.5%–49.3%) Math 61.3% | — | Decision | 46.4%43.5%–49.3% | GHIJKLMNOP | 522 / 1,125 | 53 right of 53 kept0 mistakes · 1,072 skipped | 0.768 | 63.39chosen-answer Brier 0.1962 | 75.6%answered 47% | 79 ms | 98 ms | $0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 46.3% (95%: 43.4%–49.2%) Math 64.1% | — | Decision | 46.3%43.4%–49.2% | HIJKLMNOP | 521 / 1,125 | 65 right of 65 kept0 mistakes · 1,060 skipped | 0.764 | 63.36chosen-answer Brier 0.1959 | 15.8%answered 91% | 167 ms | 203 ms | $0.026$25.64 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 46.0% (95%: 43.2%–49.0%) Math 65.2% | — | Decision | 46.0%43.2%–49.0% | IJKLMNOPQR | 518 / 1,125 | 48 right of 48 kept0 mistakes · 1,077 skipped | 0.748 | 63.14chosen-answer Brier 0.1977 | 90.6%answered 28% | 176 ms | 272 ms | $0.006$6.02 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Accuracy 45.9% (95%: 43.0%–48.8%) Math 55.8% | — | Decision | 45.9%43.0%–48.8% | IJKLMNOPQS | 516 / 1,125 | 21 right of 21 kept0 mistakes · 1,104 skipped | 0.720 | 62.63chosen-answer Brier 0.2061 | 16.0%answered 91% | 73 ms | 84 ms | $0.005$4.62estimate · DGX Spark |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Accuracy 45.3% (95%: 42.4%–48.3%) Math 58.8% | — | Decision | 45.3%42.4%–48.3% | JKLMNOPQRST | 510 / 1,125 | 56 right of 56 kept0 mistakes · 1,069 skipped | 0.759 | 61.81chosen-answer Brier 0.2171 | 12.4%answered 93% | 47 ms | 53 ms | $0.034$33.81estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Accuracy 45.1% (95%: 42.2%–48.0%) Math 53.5% | — | Decision | 45.1%42.2%–48.0% | KLMNOPQRST | 507 / 1,125 | 84 right of 91 kept7 wrong · 1,034 skipped | 0.746 | 61.18chosen-answer Brier 0.2272 | 20.5%answered 88% | 301 ms | 360 ms | $0.008$7.52 |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Accuracy 44.7% (95%: 41.8%–47.6%) Math 55.9% | — | Decision | 44.7%41.8%–47.6% | KLMNOPQRST | 503 / 1,125 | 17 right of 17 kept0 mistakes · 1,108 skipped | 0.757 | 62.14chosen-answer Brier 0.2042 | 24.4%answered 85% | 33 ms | 41 ms | $0.026$26.44estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Accuracy 44.4% (95%: 41.6%–47.4%) Math 59.4% | — | Decision | 44.4%41.6%–47.4% | KLMNOPQRST | 500 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.712 | 61.53chosen-answer Brier 0.2139 | 19.8%answered 88% | 315 ms | 362 ms | $0.013$13.42 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 44.4% (95%: 41.5%–47.3%) Math 55.3% | — | Decision | 44.4%41.5%–47.3% | KLMNOPQRST | 499 / 1,125 | 35 right of 37 kept2 wrong · 1,088 skipped | 0.701 | 61.42chosen-answer Brier 0.2151 | 13.6%answered 92% | 60 ms | 67 ms | $0.042$42.42estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 44.3% (95%: 41.4%–47.2%) Math 59.8% | — | Decision | 44.3%41.4%–47.2% | KLMNOPQRST | 498 / 1,125 | 30 right of 30 kept0 mistakes · 1,095 skipped | 0.725 | 61.97chosen-answer Brier 0.2032 | 6.7%answered 96% | 50 ms | 61 ms | $0.036$36.27estimate · RTX PRO 6000 Blackwell |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Accuracy 44.1% (95%: 41.2%–47.0%) Math 56.8% | — | Decision | 44.1%41.2%–47.0% | LMNOPQRST | 496 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.739 | 61.90chosen-answer Brier 0.2028 | 13.6%answered 92% | 32 ms | 37 ms | $0.026$25.82estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 43.9% (95%: 41.0%–46.8%) Math 60.1% | — | Decision | 43.9%41.0%–46.8% | LMNOPQRST | 494 / 1,125 | No answers at this threshold | 0.730 | 61.90chosen-answer Brier 0.2011 | 19.5%answered 88% | 52 ms | 65 ms | $0.040$39.96estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Accuracy 43.3% (95%: 40.4%–46.2%) Math 56.0% | — | Decision | 43.3%40.4%–46.2% | LMNOPQRST | 487 / 1,125 | 26 right of 27 kept1 wrong · 1,098 skipped | 0.752 | 60.41chosen-answer Brier 0.2247 | 6.0%answered 97% | 209 ms | 269 ms | $0.013$13.35estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Accuracy 43.1% (95%: 40.2%–46.0%) Math 55.2% | — | Decision | 43.1%40.2%–46.0% | LMNOPQRST | 485 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 61.34chosen-answer Brier 0.2043 | 28.9%answered 82% | 45 ms | 55 ms | $0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Accuracy 42.6% (95%: 39.7%–45.5%) Math 54.6% | — | Decision | 42.6%39.7%–45.5% | MNOPQRST | 479 / 1,125 | No answers at this threshold | 0.733 | 59.78chosen-answer Brier 0.2302 | 28.2%answered 82% | 413 ms | 576 ms | $0.294$294estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Accuracy 42.3% (95%: 39.5%–45.2%) Math 56.0% | — | Decision | 42.3%39.5%–45.2% | NOPQRST | 476 / 1,125 | 17 right of 18 kept1 wrong · 1,107 skipped | 0.708 | 59.66chosen-answer Brier 0.2298 | 8.3%answered 95% | 96 ms | 115 ms | $0.006$5.89estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Accuracy 41.9% (95%: 39.0%–44.8%) Math 59.0% | — | Decision | 41.9%39.0%–44.8% | NOPQRST | 471 / 1,125 | 33 right of 33 kept0 mistakes · 1,092 skipped | 0.717 | 60.73chosen-answer Brier 0.2040 | 37.7%answered 75% | 345 ms | 504 ms | $0.018$18.41 |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Accuracy 40.8% (95%: 38.0%–43.7%) Math 51.1% | — | Decision | 40.8%38.0%–43.7% | OPQRST | 459 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.697 | 58.55chosen-answer Brier 0.2369 | 3.6%answered 98% | 196 ms | 222 ms | $0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 40.8% (95%: 38.0%–43.7%) Math 55.6% | — | Decision | 40.8%38.0%–43.7% | OPQRST | 459 / 1,125 | 55 right of 56 kept1 wrong · 1,069 skipped | 0.723 | 59.76chosen-answer Brier 0.2128 | 11.3%answered 93% | 188 ms | 290 ms | $0.006$6.02 |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Accuracy 40.6% (95%: 37.8%–43.5%) Math 53.8% | — | Decision | 40.6%37.8%–43.5% | OPQRST | 457 / 1,125 | 4 right of 4 kept0 mistakes · 1,121 skipped | 0.726 | 60.27chosen-answer Brier 0.2009 | 1.2%answered 99% | 108 ms | 136 ms | $0.086$86.07estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 40.4% (95%: 37.5%–43.3%) Math 50.4% | — | Decision | 40.4%37.5%–43.3% | PQRST | 454 / 1,125 | 39 right of 39 kept0 mistakes · 1,086 skipped | 0.724 | 58.90chosen-answer Brier 0.2256 | 14.6%answered 91% | 38 ms | 48 ms | $0.030$29.51estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 39.7% (95%: 36.9%–42.6%) Math 51.5% | — | Decision | 39.7%36.9%–42.6% | RT | 447 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 59.26chosen-answer Brier 0.2122 | 12.6%answered 92% | 45 ms | 51 ms | $0.033$33.20estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Accuracy 39.7% (95%: 36.9%–42.6%) Math 46.9% | — | Decision | 39.7%36.9%–42.6% | QRST | 447 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.727 | 58.43chosen-answer Brier 0.2287 | 29.7%answered 80% | 66 ms | 91 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Accuracy 39.3% (95%: 36.5%–42.2%) Math 44.3% | — | Decision | 39.3%36.5%–42.2% | STU | 442 / 1,125 | No answers at this threshold | 0.720 | 58.97chosen-answer Brier 0.2135 | 11.3%answered 93% | 40 ms | 48 ms | $0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Accuracy 38.9% (95%: 36.1%–41.8%) Math 49.8% | — | Decision | 38.9%36.1%–41.8% | TU | 438 / 1,125 | No answers at this threshold | 0.659 | 57.70chosen-answer Brier 0.2353 | 14.4%answered 91% | 110 ms | 137 ms | $0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Accuracy 32.3% (95%: 29.6%–35.1%) Math 36.8% | — | Decision | 32.3%29.6%–35.1% | UV | 363 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.632 | 54.39chosen-answer Brier 0.2349 | 32.9%answered 75% | 34 ms | 42 ms | $0.002$2.26estimate · DGX Spark |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Accuracy 32.1% (95%: 29.4%–34.9%) Math 39.2% | — | Decision | 32.1%29.4%–34.9% | UV | 361 / 1,125 | No answers at this threshold | 0.660 | 56.30chosen-answer Brier 0.1950 | 1.5%answered 99% | 123 ms | 152 ms | $0.091$90.99estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Accuracy 31.7% (95%: 29.1%–34.5%) Math 35.3% | — | Decision | 31.7%29.1%–34.5% | V | 357 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.601 | 55.15chosen-answer Brier 0.2143 | 15.7%answered 89% | 59 ms | 70 ms | $0.004$3.78estimate · DGX Spark |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Accuracy 30.8% (95%: 28.1%–33.5%) Math 38.7% | — | Decision | 30.8%28.1%–33.5% | VW | 346 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.640 | 55.51chosen-answer Brier 0.1975 | 31.4%answered 76% | 40 ms | 45 ms | $0.003$2.52estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Accuracy 30.7% (95%: 28.0%–33.4%) Math 37.1% | — | Decision | 30.7%28.0%–33.4% | VW | 345 / 1,125 | 3 right of 3 kept0 mistakes · 1,122 skipped | 0.620 | 55.05chosen-answer Brier 0.2058 | 86.5%answered 22% | 17 ms | 23 ms | $0.015$14.76estimate · RTX PRO 6000 Blackwell |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Accuracy 29.2% (95%: 26.6%–31.9%) Math 32.1% | — | Decision | 29.2%26.6%–31.9% | VWX | 328 / 1,125 | 9 right of 9 kept0 mistakes · 1,116 skipped | 0.617 | 53.53chosen-answer Brier 0.2210 | 26.1%answered 80% | 23 ms | 29 ms | $0.002$1.58estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Accuracy 27.0% (95%: 24.5%–29.7%) Math 28.2% | — | Decision | 27.0%24.5%–29.7% | VWX | 304 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.545 | 52.72chosen-answer Brier 0.2159 | 4.6%answered 97% | 16 ms | 19 ms | $0.001$1.16estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 26.7% (95%: 24.2%–29.3%) Math 28.2% | — | Decision | 26.7%24.2%–29.3% | VWX | 300 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.573 | 53.45chosen-answer Brier 0.1976 | 1.2%answered 99% | 31 ms | 39 ms | $0.002$2.00estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 24.7% (95%: 22.3%–27.3%) Math 29.7% | — | Decision | 24.7%22.3%–27.3% | WX | 278 / 1,125 | No answers at this threshold | 0.641 | 53.22chosen-answer Brier 0.1827 | 21.4%answered 83% | 27 ms | 31 ms | $0.002$1.79estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Accuracy 23.6% (95%: 21.2%–26.1%) Math 22.7% | — | Decision | 23.6%21.2%–26.1% | WX | 265 / 1,125 | No answers at this threshold | 0.527 | 51.99chosen-answer Brier 0.1958 | 12.4%answered 90% | 15 ms | 28 ms | $0.001$1.31estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Accuracy 23.2% (95%: 20.8%–25.8%) Math 20.6% | — | Decision | 23.2%20.8%–25.8% | X | 261 / 1,125 | No answers at this threshold | 0.556 | 52.19chosen-answer Brier 0.1883 | 76.1%answered 30% | 13 ms | 16 ms | $0.001$0.95estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Accuracy 22.4% (95%: 20.1%–24.9%) Math 28.7% | — | Decision | 22.4%20.1%–24.9% | X | 252 / 1,125 | No answers at this threshold | 0.568 | 51.86chosen-answer Brier 0.1868 | 8.3%answered 94% | 36 ms | 43 ms | $0.028$27.67estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Alibaba IBM OpenBMB Other Google Liquid AI KRAFTON StartLux Perplexity scar-ai PelaAI quincyqiang victorbennett Fastino Metask Lab OmniJev Strands Agents Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B | — | Chat | 96.4%505/524 | 88.1%413/469 | 91.7%121/132 |
| Granite-4.2-3B high · chat Open · 3.0B | — | Chat | 94.8%497/524 | 87.4%410/469 | 93.9%124/132 |
| Qwen3.5-4B off · chat Open · 4.7B | — | Chat | 95.8%502/524 | 83.6%392/469 | 90.9%120/132 |
| MiniCPM5-2B on · chat Open · 2.5B | — | Chat | 95.4%500/524 | 79.7%374/469 | 93.9%124/132 |
| LFM3-1.7B off · chat Open · 1.7B | — | Chat | 90.1%472/524 | 78.7%369/469 | 91.7%121/132 |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B | — | Chat | 90.3%473/524 | 77.2%362/469 | 91.7%121/132 |
| LFM3-1.7B on · chat Open · 1.7B | — | Chat | 88.2%462/524 | 78.3%367/469 | 91.7%121/132 |
| MiniCPM5-2B off · chat Open · 2.5B | — | Chat | 90.6%475/524 | 75.1%352/469 | 87.1%115/132 |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | Chat | 85.1%446/524 | 62.0%291/469 | 89.4%118/132 |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B | — | Chat | 84.5%443/524 | 62.9%295/469 | 87.9%116/132 |
| LFM2.5-2.6B on · chat Open · 2.7B | — | Chat | 83.2%436/524 | 63.8%299/469 | 85.6%113/132 |
| Gemma 4 E2B on · chat Open · 5.1B | — | Chat | 82.3%431/524 | 61.6%289/469 | 90.2%119/132 |
| Gemma 4 E2B chat Open · 5.1B | — | Chat | 79.6%417/524 | 61.0%286/469 | 88.6%117/132 |
| Raon-1 nano on · chat Open · 3.1B | — | Chat | 82.4%432/524 | 53.3%250/469 | 87.9%116/132 |
| Qwen3.5-2B on · chat Open · 2.0B | — | Chat | 77.3%405/524 | 58.4%274/469 | 84.8%112/132 |
| Raon-1 nano off · chat Open · 3.1B | — | Chat | 70.8%371/524 | 53.9%253/469 | 81.8%108/132 |
| Qwen3.5-2B off · chat Open · 2.0B | — | Chat | 72.5%380/524 | 51.6%242/469 | 81.1%107/132 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | Decision | 32.4%170/524 | 50.7%238/469 | 81.8%108/132 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | Decision | 30.7%161/524 | 50.1%235/469 | 81.1%107/132 |
| vera-medium-omni Open · 4.2B | — | Decision | 31.9%167/524 | 48.6%228/469 | 76.5%101/132 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | Decision | 31.3%164/524 | 46.1%216/469 | 79.5%105/132 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | Decision | 30.2%158/524 | 47.5%223/469 | 72.0%95/132 |
| Chinese Decision Pointer 4B Open · 4.0B | — | Decision | 29.0%152/524 | 43.9%206/469 | 72.7%96/132 |
| Verdin-4B-v1.1 Open · 4.0B | — | Decision | 28.4%149/524 | 42.4%199/469 | 75.0%99/132 |
| GLiDE 2B Open · 1.9B | — | Decision | 26.5%139/524 | 33.0%155/469 | 52.3%69/132 |
| Metask-Jev-4B Open · 4.5B | — | Decision | 24.6%129/524 | 34.5%162/469 | 53.0%70/132 |
| KnowLine-2B-Gen1 Open · 2.0B | — | Decision | 25.0%131/524 | 32.2%151/469 | 56.8%75/132 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | Decision | 25.8%135/524 | 29.9%140/469 | 53.8%71/132 |
| d1-2B Open · 1.9B | — | Decision | 24.8%130/524 | 31.8%149/469 | 50.0%66/132 |
| GLiDE 0.8B Open · 0.8B | — | Decision | 25.6%134/524 | 29.0%136/469 | 43.9%58/132 |
| OneJev1.6-0.8B Open · 0.8B | — | Decision | 24.0%126/524 | 27.5%129/469 | 37.1%49/132 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | Decision | 25.2%132/524 | 24.3%114/469 | 40.9%54/132 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | Decision | 19.5%102/524 | 25.6%120/469 | 42.4%56/132 |
| GLiClass v4 Edge Open · 0.4B | — | Decision | 23.9%125/524 | 22.0%103/469 | 28.0%37/132 |
| GLiDE 0.4B Open · 0.4B | — | Decision | 20.8%109/524 | 24.1%113/469 | 29.5%39/132 |
| Strands Decider 2B Open · 2.0B | — | Decision | 19.3%101/524 | 24.1%113/469 | 28.8%38/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Spark-X2.6-4B (high · chat) (not running now, 20 of 1,125 stored, last answer 13 Nov 02:18 UTC); Granite-4.2-3B (off · chat) (still running, 259 of 1,125).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Accuracy 92.4% (95%: 90.7%–93.8%) Math 99.4% | — | Chat | 92.4%90.7%–93.8% | a | 1,039 / 1,125 | Not measured | Not measured | — | 36.1%answered 97% | 142,520 ms | 246,280 ms | $3.525$3,525estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Accuracy 91.6% (95%: 89.9%–93.1%) Math 99.2% | — | Chat | 91.6%89.9%–93.1% | ab | 1,031 / 1,125 | Not measured | Not measured | — | 53.2%answered 94% | 221,350 ms | 1,229,550 ms | $10.893$10,893estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Accuracy 90.1% (95%: 88.3%–91.7%) Math 98.5% | — | Chat | 90.1%88.3%–91.7% | ab | 1,014 / 1,125 | Not measured | Not measured | — | 33.6%answered 96% | 45,380 ms | 127,510 ms | $1.420$1,420estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Accuracy 88.9% (mean of 3 seeds; min–max 88.3%–89.6%) Math 99.2% | — | Chat | 88.9%88.3%–89.6% min–max of 3 seeds | bc | 998 / 1,125seed 0 run | Not measured | Not measured | — | 51.5%answered 92% | 101,100 ms | 333,427 ms | $3.335$3,335estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Accuracy 85.5% (95%: 83.3%–87.4%) Math 98.7% | — | Chat | 85.5%83.3%–87.4% | cd | 962 / 1,125 | Not measured | Not measured | — | 54.2%answered 90% | 163,870 ms | 656,750 ms | $0.487$487estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Accuracy 84.6% (mean of 3 seeds; min–max 84.4%–85.0%) Math 98.9% | — | Chat | 84.6%84.4%–85.0% min–max of 3 seeds | cd | 956 / 1,125seed 0 run | Not measured | Not measured | — | Not measured | 27,950 ms | 107,497 ms | $1.040$1,040estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Accuracy 84.2% (mean of 3 seeds; min–max 83.9%–84.4%) Math 98.9% | — | Chat | 84.2%83.9%–84.4% min–max of 3 seeds | d | 950 / 1,125seed 0 run | Not measured | Not measured | — | 43.6%answered 91% | 29,847 ms | 113,220 ms | $1.017$1,017estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Accuracy 83.6% (mean of 3 seeds; min–max 83.1%–83.9%) Math 98.2% | — | Chat | 83.6%83.1%–83.9% min–max of 3 seeds | d | 942 / 1,125seed 0 run | Not measured | Not measured | — | 43.9%answered 88% | 46,453 ms | 111,607 ms | $1.327$1,327estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 76.8% (mean of 3 seeds; min–max 76.0%–77.6%) Math 97.7% | — | Chat | 76.8%76.0%–77.6% min–max of 3 seeds | e | 855 / 1,125seed 0 run | Not measured | Not measured | — | 32.6%answered 91% | 21,223 ms | 45,077 ms | $0.574$574estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 75.9% (95%: 73.3%–78.3%) Math 97.8% | — | Chat | 75.9%73.3%–78.3% | e | 854 / 1,125 | Not measured | Not measured | — | Not measured | 41,350 ms | 70,680 ms | $1.193$1,193estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Accuracy 75.4% (95%: 72.8%–77.8%) Math 94.0% | — | Chat | 75.4%72.8%–77.8% | ef | 848 / 1,125 | Not measured | Not measured | — | 44.3%answered 86% | 81,330 ms | 224,000 ms | $2.480$2,480estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 74.6% (95%: 72.0%–77.0%) Math 96.9% | — | Chat | 74.6%72.0%–77.0% | ef | 839 / 1,125 | Not measured | Not measured | — | 26.4%answered 93% | 173,220 ms | 297,090 ms | $0.363$363estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 72.9% (95%: 70.2%–75.4%) Math 96.8% | — | Chat | 72.9%70.2%–75.4% | ef | 820 / 1,125 | Not measured | Not measured | — | 34.0%answered 90% | 21,390 ms | 35,160 ms | $0.513$513estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 70.9% (95%: 68.2%–73.5%) Math 97.0% | — | Chat | 70.9%68.2%–73.5% | ef | 798 / 1,125 | Not measured | Not measured | — | 10.5%answered 95% | 493,230 ms† | 1,396,520 ms† | $15.330$15,330estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Accuracy 70.3% (95%: 67.6%–72.9%) Math 94.7% | — | Chat | 70.3%67.6%–72.9% | fg | 791 / 1,125 | Not measured | Not measured | — | 33.1%answered 91% | 186,330 ms | 272,210 ms | $4.330$4,330estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 65.1% (95%: 62.2%–67.8%) Math 89.6% | — | Chat | 65.1%62.2%–67.8% | gh | 732 / 1,125 | Not measured | Not measured | — | 35.7%answered 87% | 276,720 ms† | 965,630 ms† | $9.270$9,270estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Accuracy 64.8% (95%: 62.0%–67.5%) Math 94.6% | — | Chat | 64.8%62.0%–67.5% | h | 729 / 1,125 | Not measured | Not measured | — | 35.2%answered 86% | 71,290 ms | 113,470 ms | $1.605$1,605estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Accuracy 45.9% (95%: 43.0%–48.8%) Math 55.8% | — | Decision | 45.9%43.0%–48.8% | i | 516 / 1,125 | 21 right of 21 kept0 mistakes · 1,104 skipped | 0.720 | 62.63chosen-answer Brier 0.2061 | 16.0%answered 91% | 73 ms | 84 ms | $0.005$4.62estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Accuracy 44.7% (95%: 41.8%–47.6%) Math 55.9% | — | Decision | 44.7%41.8%–47.6% | ij | 503 / 1,125 | 17 right of 17 kept0 mistakes · 1,108 skipped | 0.757 | 62.14chosen-answer Brier 0.2042 | 24.4%answered 85% | 33 ms | 41 ms | $0.026$26.44estimate · RTX PRO 6000 Blackwell |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Accuracy 44.1% (95%: 41.2%–47.0%) Math 56.8% | — | Decision | 44.1%41.2%–47.0% | ijk | 496 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.739 | 61.90chosen-answer Brier 0.2028 | 13.6%answered 92% | 32 ms | 37 ms | $0.026$25.82estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Accuracy 43.1% (95%: 40.2%–46.0%) Math 55.2% | — | Decision | 43.1%40.2%–46.0% | ijk | 485 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 61.34chosen-answer Brier 0.2043 | 28.9%answered 82% | 45 ms | 55 ms | $0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Accuracy 42.3% (95%: 39.5%–45.2%) Math 56.0% | — | Decision | 42.3%39.5%–45.2% | ijk | 476 / 1,125 | 17 right of 18 kept1 wrong · 1,107 skipped | 0.708 | 59.66chosen-answer Brier 0.2298 | 8.3%answered 95% | 96 ms | 115 ms | $0.006$5.89estimate · DGX Spark |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 40.4% (95%: 37.5%–43.3%) Math 50.4% | — | Decision | 40.4%37.5%–43.3% | jk | 454 / 1,125 | 39 right of 39 kept0 mistakes · 1,086 skipped | 0.724 | 58.90chosen-answer Brier 0.2256 | 14.6%answered 91% | 38 ms | 48 ms | $0.030$29.51estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 39.7% (95%: 36.9%–42.6%) Math 51.5% | — | Decision | 39.7%36.9%–42.6% | k | 447 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 59.26chosen-answer Brier 0.2122 | 12.6%answered 92% | 45 ms | 51 ms | $0.033$33.20estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Accuracy 32.3% (95%: 29.6%–35.1%) Math 36.8% | — | Decision | 32.3%29.6%–35.1% | l | 363 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.632 | 54.39chosen-answer Brier 0.2349 | 32.9%answered 75% | 34 ms | 42 ms | $0.002$2.26estimate · DGX Spark |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Accuracy 32.1% (95%: 29.4%–34.9%) Math 39.2% | — | Decision | 32.1%29.4%–34.9% | l | 361 / 1,125 | No answers at this threshold | 0.660 | 56.30chosen-answer Brier 0.1950 | 1.5%answered 99% | 123 ms | 152 ms | $0.091$90.99estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Accuracy 31.7% (95%: 29.1%–34.5%) Math 35.3% | — | Decision | 31.7%29.1%–34.5% | l | 357 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.601 | 55.15chosen-answer Brier 0.2143 | 15.7%answered 89% | 59 ms | 70 ms | $0.004$3.78estimate · DGX Spark |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Accuracy 30.8% (95%: 28.1%–33.5%) Math 38.7% | — | Decision | 30.8%28.1%–33.5% | l | 346 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.640 | 55.51chosen-answer Brier 0.1975 | 31.4%answered 76% | 40 ms | 45 ms | $0.003$2.52estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Accuracy 30.7% (95%: 28.0%–33.4%) Math 37.1% | — | Decision | 30.7%28.0%–33.4% | lm | 345 / 1,125 | 3 right of 3 kept0 mistakes · 1,122 skipped | 0.620 | 55.05chosen-answer Brier 0.2058 | 86.5%answered 22% | 17 ms | 23 ms | $0.015$14.76estimate · RTX PRO 6000 Blackwell |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Accuracy 29.2% (95%: 26.6%–31.9%) Math 32.1% | — | Decision | 29.2%26.6%–31.9% | lmn | 328 / 1,125 | 9 right of 9 kept0 mistakes · 1,116 skipped | 0.617 | 53.53chosen-answer Brier 0.2210 | 26.1%answered 80% | 23 ms | 29 ms | $0.002$1.58estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Accuracy 27.0% (95%: 24.5%–29.7%) Math 28.2% | — | Decision | 27.0%24.5%–29.7% | lmno | 304 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.545 | 52.72chosen-answer Brier 0.2159 | 4.6%answered 97% | 16 ms | 19 ms | $0.001$1.16estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 26.7% (95%: 24.2%–29.3%) Math 28.2% | — | Decision | 26.7%24.2%–29.3% | lmno | 300 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.573 | 53.45chosen-answer Brier 0.1976 | 1.2%answered 99% | 31 ms | 39 ms | $0.002$2.00estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 24.7% (95%: 22.3%–27.3%) Math 29.7% | — | Decision | 24.7%22.3%–27.3% | mno | 278 / 1,125 | No answers at this threshold | 0.641 | 53.22chosen-answer Brier 0.1827 | 21.4%answered 83% | 27 ms | 31 ms | $0.002$1.79estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Accuracy 23.6% (95%: 21.2%–26.1%) Math 22.7% | — | Decision | 23.6%21.2%–26.1% | no | 265 / 1,125 | No answers at this threshold | 0.527 | 51.99chosen-answer Brier 0.1958 | 12.4%answered 90% | 15 ms | 28 ms | $0.001$1.31estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Accuracy 23.2% (95%: 20.8%–25.8%) Math 20.6% | — | Decision | 23.2%20.8%–25.8% | o | 261 / 1,125 | No answers at this threshold | 0.556 | 52.19chosen-answer Brier 0.1883 | 76.1%answered 30% | 13 ms | 16 ms | $0.001$0.95estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Accuracy 22.4% (95%: 20.1%–24.9%) Math 28.7% | — | Decision | 22.4%20.1%–24.9% | o | 252 / 1,125 | No answers at this threshold | 0.568 | 51.86chosen-answer Brier 0.1868 | 8.3%answered 94% | 36 ms | 43 ms | $0.028$27.67estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Google Alibaba Liquid AI eCloud Technology PelaAI tattrongvu Fastino StartLux newsbang Base Labs Patrick-18 Valen-Team Apertix AI EldanRing Blockbrain Labs Cloudflare Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B | — | Chat | 92.4%484/524 | 81.0%380/469 | 93.2%123/132 |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | Chat | 92.4%484/524 | 77.4%363/469 | 91.7%121/132 |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B | — | Chat | 90.5%474/524 | 76.8%360/469 | 92.4%122/132 |
| Gemma 4 E4B chat Open · 8.0B | — | Chat | 89.3%468/524 | 73.1%343/469 | 94.7%125/132 |
| Gemma 4 E4B on · chat Open · 8.0B | — | Chat | 88.7%465/524 | 73.1%343/469 | 91.7%121/132 |
| Qwen3-14B off · chat Open · 15B | — | Chat | 74.6%391/524 | 48.2%226/469 | 89.4%118/132 |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | Chat | 75.4%395/524 | 36.9%173/469 | 82.6%109/132 |
| Erk-14B off · chat Open · 15B | — | Chat | 63.7%334/524 | 35.6%167/469 | 82.6%109/132 |
| KnowLine-12B-Gen1 Open · 12B | — | Decision | 36.6%192/524 | 56.7%266/469 | 84.1%111/132 |
| Jeffan-9B-1211 Open · 9.0B | — | Decision | 34.2%179/524 | 53.7%252/469 | 84.1%111/132 |
| GLiDE 9B Open · 7.9B | — | Decision | 32.1%168/524 | 55.7%261/469 | 82.6%109/132 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | Decision | 32.3%169/524 | 52.5%246/469 | 82.6%109/132 |
| Adjudex 9B v5 Open · 9.0B | — | Decision | 34.0%178/524 | 49.9%234/469 | 83.3%110/132 |
| Elephant-1 Open · 9.0B | — | Decision | 33.2%174/524 | 51.0%239/469 | 82.6%109/132 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | Decision | 34.9%183/524 | 47.3%222/469 | 79.5%105/132 |
| Valen-Decision-1.5-9B Open · 9.0B | — | Decision | 34.2%179/524 | 45.8%215/469 | 79.5%105/132 |
| KnowLine-9B-Gen1 Open · 9.0B | — | Decision | 32.4%170/524 | 47.5%223/469 | 79.5%105/132 |
| KnowLine-9B-Gen2 Open · 9.0B | — | Decision | 33.4%175/524 | 45.2%212/469 | 81.1%107/132 |
| Cerno 9B v1.2 Open · 9.0B | — | Decision | 31.1%163/524 | 48.0%225/469 | 75.0%99/132 |
| Winnow-12B Open · 12B | — | Decision | 26.5%139/524 | 45.4%213/469 | 72.0%95/132 |
| Cygnet Open · 12B | — | Decision | 26.9%141/524 | 47.3%222/469 | 59.8%79/132 |
| Clef-Flash Open · 9.0B | — | Decision | 30.0%157/524 | 40.1%188/469 | 70.5%93/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Accuracy 87.7% (95%: 85.7%–89.5%) Math 98.8% | — | Chat | 87.7%85.7%–89.5% | a | 987 / 1,125 | Not measured | Not measured | — | 15.5%answered 98% | 42,060 ms | 82,940 ms | $1.292$1,292estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 85.9% (mean of 3 seeds; min–max 85.5%–86.1%) Math 98.5% | — | Chat | 85.9%85.5%–86.1% min–max of 3 seeds | ab | 968 / 1,125seed 0 run | Not measured | Not measured | — | 55.9%answered 90% | 31,567 ms | 83,467 ms | $0.993$993estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 85.0% (95%: 82.8%–86.9%) Math 98.5% | — | Chat | 85.0%82.8%–86.9% | ab | 956 / 1,125 | Not measured | Not measured | — | Not measured | 57,900 ms | 109,440 ms | $1.503$1,503estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 83.2% (95%: 80.9%–85.3%) Math 98.1% | — | Chat | 83.2%80.9%–85.3% | b | 936 / 1,125 | Not measured | Not measured | — | 52.4%answered 88% | 32,690 ms | 64,070 ms | $0.842$842estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 82.6% (95%: 80.3%–84.7%) Math 98.2% | — | Chat | 82.6%80.3%–84.7% | b | 929 / 1,125 | Not measured | Not measured | — | 45.1%answered 91% | 284,080 ms | 539,980 ms | $0.650$650estimate · DGX Spark |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Accuracy 65.3% (95%: 62.5%–68.1%) Math 95.8% | — | Chat | 65.3%62.5%–68.1% | c | 735 / 1,125 | Not measured | Not measured | — | 16.1%answered 94% | 16,560 ms | 178,680 ms | $1.772$1,772estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Accuracy 60.2% (95%: 57.3%–63.0%) Math 96.3% | — | Chat | 60.2%57.3%–63.0% | cd | 677 / 1,125 | Not measured | Not measured | — | 58.3%answered 70% | 138,140 ms† | 403,380 ms† | $4.664$4,664estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Accuracy 54.2% (95%: 51.3%–57.1%) Math 90.8% | — | Chat | 54.2%51.3%–57.1% | de | 610 / 1,125 | Not measured | Not measured | — | 30.7%answered 84% | 10,150 ms | 18,410 ms | $0.390$390estimate · RTX PRO 6000 Blackwell |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Accuracy 50.6% (95%: 47.7%–53.5%) Math 66.3% | — | Decision | 50.6%47.7%–53.5% | ef | 569 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.774 | 65.90chosen-answer Brier 0.1878 | 8.8%answered 95% | 68 ms | 88 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 48.2% (95%: 45.3%–51.1%) Math 59.8% | — | Decision | 48.2%45.3%–51.1% | efg | 542 / 1,125 | 10 right of 10 kept0 mistakes · 1,115 skipped | 0.753 | 64.33chosen-answer Brier 0.1952 | 23.4%answered 87% | 52 ms | 66 ms | $0.039$39.35estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Accuracy 47.8% (95%: 44.9%–50.7%) Math 57.9% | — | Decision | 47.8%44.9%–50.7% | efg | 538 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.749 | 63.95chosen-answer Brier 0.1993 | 30.4%answered 82% | 208 ms | 251 ms | $0.013$13.29estimate · DGX Spark |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Accuracy 46.6% (95%: 43.7%–49.5%) Math 58.4% | — | Decision | 46.6%43.7%–49.5% | fg | 524 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.762 | 63.55chosen-answer Brier 0.1947 | 17.4%answered 90% | 160 ms | 175 ms | $0.010$10.04estimate · DGX Spark |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 46.4% (95%: 43.5%–49.3%) Math 62.1% | — | Decision | 46.4%43.5%–49.3% | fg | 522 / 1,125 | 54 right of 55 kept1 wrong · 1,070 skipped | 0.771 | 63.48chosen-answer Brier 0.1945 | 12.2%answered 93% | 41 ms | 46 ms | $0.031$31.36estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Accuracy 46.4% (95%: 43.5%–49.3%) Math 61.3% | — | Decision | 46.4%43.5%–49.3% | fg | 522 / 1,125 | 53 right of 53 kept0 mistakes · 1,072 skipped | 0.768 | 63.39chosen-answer Brier 0.1962 | 75.6%answered 47% | 79 ms | 98 ms | $0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Accuracy 45.3% (95%: 42.4%–48.3%) Math 58.8% | — | Decision | 45.3%42.4%–48.3% | fgh | 510 / 1,125 | 56 right of 56 kept0 mistakes · 1,069 skipped | 0.759 | 61.81chosen-answer Brier 0.2171 | 12.4%answered 93% | 47 ms | 53 ms | $0.034$33.81estimate · RTX PRO 6000 Blackwell |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 44.4% (95%: 41.5%–47.3%) Math 55.3% | — | Decision | 44.4%41.5%–47.3% | ghi | 499 / 1,125 | 35 right of 37 kept2 wrong · 1,088 skipped | 0.701 | 61.42chosen-answer Brier 0.2151 | 13.6%answered 92% | 60 ms | 67 ms | $0.042$42.42estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 44.3% (95%: 41.4%–47.2%) Math 59.8% | — | Decision | 44.3%41.4%–47.2% | ghi | 498 / 1,125 | 30 right of 30 kept0 mistakes · 1,095 skipped | 0.725 | 61.97chosen-answer Brier 0.2032 | 6.7%answered 96% | 50 ms | 61 ms | $0.036$36.27estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 43.9% (95%: 41.0%–46.8%) Math 60.1% | — | Decision | 43.9%41.0%–46.8% | ghi | 494 / 1,125 | No answers at this threshold | 0.730 | 61.90chosen-answer Brier 0.2011 | 19.5%answered 88% | 52 ms | 65 ms | $0.040$39.96estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Accuracy 43.3% (95%: 40.4%–46.2%) Math 56.0% | — | Decision | 43.3%40.4%–46.2% | ghi | 487 / 1,125 | 26 right of 27 kept1 wrong · 1,098 skipped | 0.752 | 60.41chosen-answer Brier 0.2247 | 6.0%answered 97% | 209 ms | 269 ms | $0.013$13.35estimate · DGX Spark |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Accuracy 39.7% (95%: 36.9%–42.6%) Math 46.9% | — | Decision | 39.7%36.9%–42.6% | hi | 447 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.727 | 58.43chosen-answer Brier 0.2287 | 29.7%answered 80% | 66 ms | 91 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Accuracy 39.3% (95%: 36.5%–42.2%) Math 44.3% | — | Decision | 39.3%36.5%–42.2% | i | 442 / 1,125 | No answers at this threshold | 0.720 | 58.97chosen-answer Brier 0.2135 | 11.3%answered 93% | 40 ms | 48 ms | $0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Accuracy 38.9% (95%: 36.1%–41.8%) Math 49.8% | — | Decision | 38.9%36.1%–41.8% | i | 438 / 1,125 | No answers at this threshold | 0.659 | 57.70chosen-answer Brier 0.2353 | 14.4%answered 91% | 110 ms | 137 ms | $0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Alibaba Google Liquid AI MachinePulse PelaAI Poindexter Labs Fastino Sthānika AI Perplexity vLLM Semantic Router Base Labs Nace.AI Turion.AI Zefan Cai Cloudflare OmniJev Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B | — | Chat | 97.7%512/524 | 90.0%422/469 | 93.9%124/132 |
| Gemma 4 31B chat Open · 31B | — | Chat | 97.1%509/524 | 89.3%419/469 | 92.4%122/132 |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | Chat | 93.3%489/524 | 77.0%361/469 | 94.7%125/132 |
| Karpo Decision 31B Open · 31B | — | Decision | 50.8%266/524 | 64.0%300/469 | 88.6%117/132 |
| KnowLine-31B-Gen1.5 Open · 31B | — | Decision | 47.1%247/524 | 64.2%301/469 | 87.9%116/132 |
| KnowLine-31B-Gen2 Open · 31B | — | Decision | 47.1%247/524 | 63.8%299/469 | 88.6%117/132 |
| Copernicus-31B Open · 31B | — | Decision | 47.1%247/524 | 63.8%299/469 | 86.4%114/132 |
| GLiDE 27B Open · 27B | — | Decision | 42.0%220/524 | 68.0%319/469 | 89.4%118/132 |
| Sieve 31B v3 Open · 31B | — | Decision | 45.8%240/524 | 62.9%295/469 | 87.9%116/132 |
| GLiDE v1.2 Open · 27B | — | Decision | 40.5%212/524 | 63.1%296/469 | 85.6%113/132 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | Decision | 41.4%217/524 | 59.9%281/469 | 89.4%118/132 |
| d3-ultra Open · 26B | — | Decision | 42.2%221/524 | 59.3%278/469 | 87.1%115/132 |
| Mammoth-1 Open · 27B | — | Decision | 42.7%224/524 | 58.6%275/469 | 86.4%114/132 |
| d3 Open · 26B | — | Decision | 40.6%213/524 | 60.6%284/469 | 87.9%116/132 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | Decision | 39.9%209/524 | 58.4%274/469 | 88.6%117/132 |
| Drex Max Open · 27B | — | Decision | 42.4%222/524 | 55.7%261/469 | 86.4%114/132 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | Decision | 39.1%205/524 | 58.0%272/469 | 86.4%114/132 |
| Turion Jev 31B Open · 31B | — | Decision | 36.6%192/524 | 60.1%282/469 | 86.4%114/132 |
| Open-Jev-27B-v1.1 Open · 27B | — | Decision | 31.1%163/524 | 47.1%221/469 | 72.0%95/132 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | Decision | 34.2%179/524 | 41.4%194/469 | 74.2%98/132 |
| Clef Open · 27B | — | Decision | 30.2%158/524 | 46.1%216/469 | 64.4%85/132 |
| OneJev-27B Open · 27B | — | Decision | 30.0%157/524 | 45.0%211/469 | 67.4%89/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Accuracy 94.0% (95%: 92.5%–95.3%) Math 99.2% | — | Chat | 94.0%92.5%–95.3% | a | 1,058 / 1,125 | Not measured | Not measured | — | 39.1%answered 97% | 18,710 ms | 101,030 ms | $4.884$4,884estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Accuracy 93.3% (95%: 91.7%–94.6%) Math 99.4% | — | Chat | 93.3%91.7%–94.6% | a | 1,050 / 1,125 | Not measured | Not measured | — | 16.3%answered 99% | 18,290 ms | 39,790 ms | $1.244$1,244estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Accuracy 86.7% (95%: 84.6%–88.5%) Math 99.0% | — | Chat | 86.7%84.6%–88.5% | b | 975 / 1,125 | Not measured | Not measured | — | 37.8%answered 95% | 228,980 ms | 776,930 ms | $7.562$7,562estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Accuracy 60.7% (95%: 57.8%–63.5%) Math 81.6% | — | Decision | 60.7%57.8%–63.5% | c | 683 / 1,125 | 169 right of 172 kept3 wrong · 953 skipped | 0.816 | 71.80chosen-answer Brier 0.1711 | 6.6%answered 97% | 173 ms | 211 ms | $0.128$128estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Accuracy 59.0% (95%: 56.1%–61.9%) Math 77.4% | — | Decision | 59.0%56.1%–61.9% | cd | 664 / 1,125 | 89 right of 89 kept0 mistakes · 1,036 skipped | 0.804 | 70.72chosen-answer Brier 0.1757 | 0.4%answered 100% | 983 ms | 1,279 ms | $0.061$61.43estimate · DGX Spark |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Accuracy 58.9% (95%: 56.0%–61.8%) Math 80.3% | — | Decision | 58.9%56.0%–61.8% | cd | 663 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.825 | 71.09chosen-answer Brier 0.1675 | 0.0%answered 100% | 151 ms | 204 ms | $0.112$112estimate · RTX PRO 6000 Blackwell |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Accuracy 58.7% (95%: 55.8%–61.5%) Math 77.3% | — | Decision | 58.7%55.8%–61.5% | cd | 660 / 1,125 | 66 right of 66 kept0 mistakes · 1,059 skipped | 0.801 | 70.48chosen-answer Brier 0.1771 | 1.9%answered 99% | 185 ms | 255 ms | $0.136$136estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Accuracy 58.4% (95%: 55.5%–61.2%) Math 72.0% | — | Decision | 58.4%55.5%–61.2% | cd | 657 / 1,125 | 74 right of 75 kept1 wrong · 1,050 skipped | 0.822 | 70.67chosen-answer Brier 0.1707 | 44.3%answered 79% | 544 ms | 655 ms | $0.033$33.16estimate · DGX Spark |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Accuracy 57.9% (95%: 55.0%–60.7%) Math 77.1% | — | Decision | 57.9%55.0%–60.7% | cde | 651 / 1,125 | 79 right of 81 kept2 wrong · 1,044 skipped | 0.797 | 70.03chosen-answer Brier 0.1781 | 5.5%answered 98% | 1,050 ms | 1,335 ms | $0.063$62.95estimate · DGX Spark |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Accuracy 55.2% (95%: 52.3%–58.1%) Math 69.9% | — | Decision | 55.2%52.3%–58.1% | def | 621 / 1,125 | 45 right of 45 kept0 mistakes · 1,080 skipped | 0.793 | 68.39chosen-answer Brier 0.1842 | 50.2%answered 73% | 111 ms | 140 ms | $0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Accuracy 54.8% (95%: 51.8%–57.6%) Math 72.5% | — | Decision | 54.8%51.8%–57.6% | def | 616 / 1,125 | 62 right of 62 kept0 mistakes · 1,063 skipped | 0.803 | 67.98chosen-answer Brier 0.1881 | 18.1%answered 91% | 108 ms | 131 ms | $0.081$81.16estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 54.6% (95%: 51.7%–57.5%) Math 71.5% | — | Decision | 54.6%51.7%–57.5% | def | 614 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.778 | 66.46chosen-answer Brier 0.2166 | 10.1%answered 95% | 156 ms | 185 ms | $0.109$109estimate · RTX PRO 6000 Blackwell |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Accuracy 54.5% (95%: 51.6%–57.4%) Math 74.0% | — | Decision | 54.5%51.6%–57.4% | def | 613 / 1,125 | 106 right of 108 kept2 wrong · 1,017 skipped | 0.795 | 68.07chosen-answer Brier 0.1835 | 72.3%answered 58% | 221 ms | 244 ms | $0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 54.5% (95%: 51.6%–57.4%) Math 70.4% | — | Decision | 54.5%51.6%–57.4% | def | 613 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.806 | 68.34chosen-answer Brier 0.1781 | 14.8%answered 93% | 76 ms | 98 ms | $0.057$57.18estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Accuracy 53.3% (95%: 50.4%–56.2%) Math 69.8% | — | Decision | 53.3%50.4%–56.2% | ef | 600 / 1,125 | 78 right of 80 kept2 wrong · 1,045 skipped | 0.798 | 67.64chosen-answer Brier 0.1805 | 14.4%answered 93% | 238 ms | 552 ms | $0.007$7.46 |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Accuracy 53.1% (95%: 50.1%–56.0%) Math 70.3% | — | Decision | 53.1%50.1%–56.0% | ef | 597 / 1,125 | 147 right of 155 kept8 wrong · 970 skipped | 0.786 | 65.68chosen-answer Brier 0.2170 | 7.3%answered 97% | 106 ms | 143 ms | $0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Accuracy 52.5% (95%: 49.6%–55.4%) Math 69.3% | with web | Decision | 52.5%49.6%–55.4% | f | 591 / 1,125 | 78 right of 79 kept1 wrong · 1,046 skipped | 0.800 | 67.24chosen-answer Brier 0.1806 | 37.7%answered 80% | 448 ms | 827 ms | $1.030$1,030 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Accuracy 52.3% (95%: 49.3%–55.2%) Math 69.0% | — | Decision | 52.3%49.3%–55.2% | f | 588 / 1,125 | 7 right of 7 kept0 mistakes · 1,118 skipped | 0.782 | 66.89chosen-answer Brier 0.1849 | 1.5%answered 99% | 385 ms | 533 ms | $0.025$24.91estimate · DGX Spark |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Accuracy 42.6% (95%: 39.7%–45.5%) Math 54.6% | — | Decision | 42.6%39.7%–45.5% | g | 479 / 1,125 | No answers at this threshold | 0.733 | 59.78chosen-answer Brier 0.2302 | 28.2%answered 82% | 413 ms | 576 ms | $0.294$294estimate · RTX PRO 6000 Blackwell |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Accuracy 41.9% (95%: 39.0%–44.8%) Math 59.0% | — | Decision | 41.9%39.0%–44.8% | g | 471 / 1,125 | 33 right of 33 kept0 mistakes · 1,092 skipped | 0.717 | 60.73chosen-answer Brier 0.2040 | 37.7%answered 75% | 345 ms | 504 ms | $0.018$18.41 |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Accuracy 40.8% (95%: 38.0%–43.7%) Math 51.1% | — | Decision | 40.8%38.0%–43.7% | g | 459 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.697 | 58.55chosen-answer Brier 0.2369 | 3.6%answered 98% | 196 ms | 222 ms | $0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Accuracy 40.6% (95%: 37.8%–43.5%) Math 53.8% | — | Decision | 40.6%37.8%–43.5% | g | 457 / 1,125 | 4 right of 4 kept0 mistakes · 1,121 skipped | 0.726 | 60.27chosen-answer Brier 0.2009 | 1.2%answered 99% | 108 ms | 136 ms | $0.086$86.07estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
DeepSeek Thinking Machines Xiaomi Alibaba MiniMax Z.ai REAX Logits Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | Chat | 98.3%515/524 | 94.7%444/469 | 93.2%123/132 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | Chat | 98.1%514/524 | 93.6%439/469 | 93.2%123/132 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | Chat | 97.9%513/524 | 92.8%435/469 | 94.7%125/132 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | Chat | 97.3%510/524 | 92.8%435/469 | 94.7%125/132 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active | — | Chat | 98.1%514/524 | 92.5%434/469 | 92.4%122/132 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | Chat | 95.8%502/524 | 92.5%434/469 | 91.7%121/132 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | Chat | 95.4%500/524 | 90.6%425/469 | 93.2%123/132 |
| REAX 1.3 Open · 57B | — | Decision | 47.5%249/524 | 64.4%302/469 | 87.9%116/132 |
| Logits Sigma-1 Duo Open · 57B | — | Decision | 44.7%234/524 | 63.8%299/469 | 88.6%117/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): DeepSeek-V4-Flash (chat) (sample: a random subset by design, 317 answers stored, not the full 1,125).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Accuracy 96.2% (95%: 94.9%–97.2%) Math 99.6% | — | Chat | 96.2%94.9%–97.2% | a | 1,082 / 1,125 | Not measured | Not measured | — | 3.8%answered 100% | 6,680 ms | 40,040 ms | $1.647$1,647 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Accuracy 95.6% (95%: 94.3%–96.7%) Math 99.5% | — | Chat | 95.6%94.3%–96.7% | ab | 1,076 / 1,125 | Not measured | Not measured | — | 37.3%answered 98% | 13,180 ms | 144,250 ms | $4.121$4,121 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Accuracy 95.4% (95%: 94.0%–96.5%) Math 99.6% | — | Chat | 95.4%94.0%–96.5% | ab | 1,073 / 1,125 | Not measured | Not measured | — | 50.0%answered 97% | 13,290 ms | 78,510 ms | $5.489$5,489 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Accuracy 95.1% (95%: 93.7%–96.2%) Math 99.4% | — | Chat | 95.1%93.7%–96.2% | abc | 1,070 / 1,125 | Not measured | Not measured | — | 23.3%answered 98% | 24,740 ms | 179,930 ms | $1.390$1,390 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active Qwen3.5-397B-A17B on · chat · Alibaba · chat model · reasoning on Parameters: 397B MoE · 17B active Accuracy 95.1% (95%: 93.7%–96.2%) Math running | — | Chat | 95.1%93.7%–96.2% | abc | 1,070 / 1,125 | Not measured | Not measured | — | 31.7%answered 98% | 84,650 ms | 167,520 ms | $13.426$13,426 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Accuracy 94.0% (95%: 92.4%–95.2%) Math 98.2% | — | Chat | 94.0%92.4%–95.2% | bc | 1,057 / 1,125 | Not measured | Not measured | — | 38.3%answered 97% | 5,610 ms | 37,430 ms | $0.803$803 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Accuracy 93.2% (95%: 91.5%–94.5%) Math 98.8% | — | Chat | 93.2%91.5%–94.5% | c | 1,048 / 1,125 | Not measured | Not measured | — | 4.2%answered 100% | 12,050 ms | 82,830 ms | $1.057$1,057 |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Accuracy 59.3% (95%: 56.4%–62.1%) Math 78.0% | — | Decision | 59.3%56.4%–62.1% | d | 667 / 1,125 | 51 right of 51 kept0 mistakes · 1,074 skipped | 0.816 | 71.07chosen-answer Brier 0.1714 | 5.3%answered 98% | 215 ms | 278 ms | $0.160$160estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Accuracy 57.8% (95%: 54.9%–60.6%) Math 75.2% | — | Decision | 57.8%54.9%–60.6% | d | 650 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.796 | 69.90chosen-answer Brier 0.1798 | 7.1%answered 97% | 285 ms | 399 ms | $0.211$211estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Meta OpenAI Liquid AI Nace.AI Fastino Subquadratic Google TypeSafe Celeris Inception Microsoft Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient | — | Chat | 97.9%513/524 | 93.2%437/469 | 93.9%124/132 |
| GPT-6 Luna chat Closed · Efficient | — | Chat | 67.0%351/524 | 39.7%186/469 | 87.1%115/132 |
| d1-ultra Closed · Efficient | — | Decision | 44.7%234/524 | 57.4%269/469 | 87.1%115/132 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | Decision | 41.4%217/524 | 60.8%285/469 | 86.4%114/132 |
| GLiDE API Closed · Efficient | — | Decision | 38.7%203/524 | 63.1%296/469 | 86.4%114/132 |
| SubQ Snap 1.1 Closed · Efficient | — | Decision | 37.2%195/524 | 60.3%283/469 | 84.1%111/132 |
| Gemini Decision 1 20261111 Closed · Efficient | — | Decision | 39.3%206/524 | 53.1%249/469 | 81.8%108/132 |
| Jev 1.14 Closed · Efficient | — | Decision | 35.7%187/524 | 53.1%249/469 | 80.3%106/132 |
| celeris-1-decision Closed · Efficient | — | Decision | 37.2%195/524 | 50.5%237/469 | 75.8%100/132 |
| celeris-1.1-decision Closed · Efficient | — | Decision | 37.8%198/524 | 49.5%232/469 | 77.3%102/132 |
| Jev 1.13 Closed · Efficient | — | Decision | 33.2%174/524 | 52.0%244/469 | 78.0%103/132 |
| d1.1 Closed · Efficient | — | Decision | 34.7%182/524 | 50.7%238/469 | 74.2%98/132 |
| Mercury Decide Closed · Efficient | — | Decision | 33.4%175/524 | 51.6%242/469 | 68.2%90/132 |
| Microsoft-Decision-1 Closed · Efficient | — | Decision | 34.0%178/524 | 46.9%220/469 | 77.3%102/132 |
| d1 Closed · Efficient | — | Decision | 29.0%152/524 | 44.3%208/469 | 75.0%99/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 1,125 of 1,125); shown once.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)); Claude Haiku 5.5 (max · chat) (paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Accuracy 95.5% (95%: 94.1%–96.5%) Math 99.6% | — | Chat | 95.5%94.1%–96.5% | a | 1,074 / 1,125 | Not measured | Not measured | — | 22.6%answered 99% | 3,110 ms | 14,020 ms | $1.820$1,820 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 58.0% (95%: 55.0%–60.8%) Math 93.7% | — | Chat | 58.0%55.0%–60.8% | b | 652 / 1,125 | Not measured | Not measured | — | 55.1%answered 72% | 3,860 ms | 5,820 ms | $0.000$0.00 |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Accuracy 54.9% (95%: 52.0%–57.8%) Math 74.1% | — | Decision | 54.9%52.0%–57.8% | bc | 618 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.783 | 68.09chosen-answer Brier 0.1876 | 91.0%answered 35% | 182 ms | 425 ms | $0.013$12.64 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Accuracy 54.8% (95%: 51.8%–57.6%) Math 70.1% | — | Decision | 54.8%51.8%–57.6% | bc | 616 / 1,125 | 89 right of 90 kept1 wrong · 1,035 skipped | 0.793 | 68.07chosen-answer Brier 0.1861 | 54.1%answered 69% | 329 ms | 1,540 ms | $0.022$22.37 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Accuracy 54.5% (95%: 51.6%–57.4%) Math 70.8% | — | Decision | 54.5%51.6%–57.4% | bc | 613 / 1,125 | 52 right of 52 kept0 mistakes · 1,073 skipped | 0.814 | 68.27chosen-answer Brier 0.1794 | 60.0%answered 67% | 271 ms | 291 ms | $0.052$52.03 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Accuracy 52.4% (95%: 49.4%–55.3%) Math 68.6% | — | Decision | 52.4%49.4%–55.3% | bcd | 589 / 1,125 | 70 right of 72 kept2 wrong · 1,053 skipped | 0.787 | 65.94chosen-answer Brier 0.2047 | 65.7%answered 61% | 384 ms | 393 ms | $0.015$14.92 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Accuracy 50.0% (95%: 47.1%–53.0%) Math 71.3% | — | Decision | 50.0%47.1%–53.0% | cde | 563 / 1,125 | 43 right of 43 kept0 mistakes · 1,082 skipped | 0.768 | 65.37chosen-answer Brier 0.1930 | 2.0%answered 99% | 228 ms | 344 ms | $0.037$36.77 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 48.2% (95%: 45.3%–51.1%) Math 65.1% | — | Decision | 48.2%45.3%–51.1% | de | 542 / 1,125 | 61 right of 61 kept0 mistakes · 1,064 skipped | 0.750 | 64.14chosen-answer Brier 0.1990 | 24.5%answered 86% | 166 ms | 193 ms | $0.026$25.64 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 47.3% (95%: 44.4%–50.2%) Math 70.0% | — | Decision | 47.3%44.4%–50.2% | de | 532 / 1,125 | No answers at this threshold | 0.735 | 62.94chosen-answer Brier 0.2141 | 92.3%answered 28% | 390 ms | 394 ms | $0.015$15.40 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 47.3% (95%: 44.4%–50.2%) Math 69.9% | — | Decision | 47.3%44.4%–50.2% | de | 532 / 1,125 | No answers at this threshold | 0.733 | 62.92chosen-answer Brier 0.2144 | 91.0%answered 29% | 383 ms | 390 ms | $0.002$1.54 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 46.3% (95%: 43.4%–49.2%) Math 64.1% | — | Decision | 46.3%43.4%–49.2% | e | 521 / 1,125 | 65 right of 65 kept0 mistakes · 1,060 skipped | 0.764 | 63.36chosen-answer Brier 0.1959 | 15.8%answered 91% | 167 ms | 203 ms | $0.026$25.64 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 46.0% (95%: 43.2%–49.0%) Math 65.2% | — | Decision | 46.0%43.2%–49.0% | e | 518 / 1,125 | 48 right of 48 kept0 mistakes · 1,077 skipped | 0.748 | 63.14chosen-answer Brier 0.1977 | 90.6%answered 28% | 176 ms | 272 ms | $0.006$6.02 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Accuracy 45.1% (95%: 42.2%–48.0%) Math 53.5% | — | Decision | 45.1%42.2%–48.0% | ef | 507 / 1,125 | 84 right of 91 kept7 wrong · 1,034 skipped | 0.746 | 61.18chosen-answer Brier 0.2272 | 20.5%answered 88% | 301 ms | 360 ms | $0.008$7.52 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Accuracy 44.4% (95%: 41.6%–47.4%) Math 59.4% | — | Decision | 44.4%41.6%–47.4% | ef | 500 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.712 | 61.53chosen-answer Brier 0.2139 | 19.8%answered 88% | 315 ms | 362 ms | $0.013$13.42 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 40.8% (95%: 38.0%–43.7%) Math 55.6% | — | Decision | 40.8%38.0%–43.7% | f | 459 / 1,125 | 55 right of 56 kept1 wrong · 1,069 skipped | 0.723 | 59.76chosen-answer Brier 0.2128 | 11.3%answered 93% | 188 ms | 290 ms | $0.006$6.02 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse | — | Chat | 97.5%511/524 | 93.6%439/469 | 93.9%124/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 95.5% (95%: 94.1%–96.5%) Math 99.7% | — | Chat | 95.5%94.1%–96.5% | a | 1,074 / 1,125 | Not measured | Not measured | — | 41.3%answered 98% | 6,710 ms | 14,890 ms | $11.480$11,480 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship | — | Chat | 98.3%515/524 | 94.7%444/469 | 93.2%123/132 |
| Claude Fable 5.1 chat Closed · Flagship | — | Chat | 97.7%512/524 | 93.0%436/469 | 93.2%123/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 147 of 1,125 stored, last answer 12 Nov 11:32 UTC).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 96.2% (95%: 94.9%–97.2%) Math 99.8% | — | Chat | 96.2%94.9%–97.2% | a | 1,082 / 1,125 | Not measured | Not measured | — | 38.2%answered 98% | 5,830 ms | 12,240 ms | $19.540$19,540 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 95.2% (95%: 93.8%–96.3%) Math 99.7% | — | Chat | 95.2%93.8%–96.3% | b | 1,071 / 1,125 | Not measured | Not measured | — | 8.3%answered 100% | 3,820 ms | 13,460 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
OpenAI Google Alibaba Liquid AI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Models | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed | — | Chat | 98.3%515/524 | 95.5%448/469 | 94.7%125/132 |
| Gemini 4 Carbon chat Closed · Undisclosed | — | Chat | 98.1%514/524 | 95.7%449/469 | 94.7%125/132 |
| GPT-6.1 Sol chat Closed · Undisclosed | — | Chat | 97.9%513/524 | 95.5%448/469 | 94.7%125/132 |
| Qwen4-Max high · chat Closed · Undisclosed | — | Chat | 97.9%513/524 | 95.3%447/469 | 93.2%123/132 |
| GPT-6.1 Terra chat Closed · Undisclosed | — | Chat | 97.3%510/524 | 90.4%424/469 | 93.2%123/132 |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | Chat | 97.1%509/524 | 90.8%426/469 | 92.4%122/132 |
| d1.2-ultra Closed · Undisclosed | — | Decision | 38.2%200/524 | 57.4%269/469 | 84.1%111/132 |
| d1.2 Closed · Undisclosed | — | Decision | 39.7%208/524 | 51.6%242/469 | 78.0%103/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Grok 4.7 (chat) (paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Accuracy 96.7% (95%: 95.5%–97.6%) Math 99.6% | — | Chat | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 56.4%answered 97% | 6,590 ms | 14,330 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Accuracy 96.7% (95%: 95.5%–97.6%) Math 99.6% | — | Chat | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 45.5%answered 98% | 44,520 ms | 206,660 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Accuracy 96.5% (95%: 95.3%–97.5%) Math 99.6% | — | Chat | 96.5%95.3%–97.5% | a | 1,086 / 1,125 | Not measured | Not measured | — | 63.2%answered 97% | 7,070 ms | 16,200 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Accuracy 96.3% (95%: 95.0%–97.2%) Math 99.8% | — | Chat | 96.3%95.0%–97.2% | a | 1,083 / 1,125 | Not measured | Not measured | — | 0.0%answered 100% | 11,570 ms | 38,640 ms | $5.690$5,690 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Accuracy 94.0% (95%: 92.4%–95.2%) Math 99.2% | — | Chat | 94.0%92.4%–95.2% | b | 1,057 / 1,125 | Not measured | Not measured | — | 61.1%answered 95% | 5,640 ms | 10,330 ms | $0.000$0.00 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Accuracy 94.0% (95%: 92.4%–95.2%) Math 99.5% | — | Chat | 94.0%92.4%–95.2% | b | 1,057 / 1,125 | Not measured | Not measured | — | 34.6%answered 98% | 17,550 ms | 73,220 ms | $1.376$1,376 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 51.6% (95%: 48.6%–54.5%) Math 69.9% | — | Decision | 51.6%48.6%–54.5% | c | 580 / 1,125 | 13 right of 13 kept0 mistakes · 1,112 skipped | 0.800 | 66.77chosen-answer Brier 0.1802 | 94.5%answered 28% | 220 ms | 341 ms | $0.013$12.64 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 49.2% (95%: 46.2%–52.1%) Math 65.5% | — | Decision | 49.2%46.2%–52.1% | c | 553 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.742 | 64.51chosen-answer Brier 0.2013 | 78.8%answered 43% | 222 ms | 390 ms | $0.006$6.02 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
MachinePulse REAX PelaAI Poindexter Labs Fastino Sthānika AI Logits Liquid AI Nace.AI Perplexity vLLM Semantic Router Base Labs Subquadratic Turion.AI Google tattrongvu TypeSafe Celeris StartLux newsbang Patrick-18 Inception Microsoft Valen-Team scar-ai Apertix AI Zefan Cai Cloudflare OmniJev quincyqiang victorbennett EldanRing Blockbrain Labs Metask Lab Other Strands Agents Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | 50.8%266/524 | 64.0%300/469 | 88.6%117/132 |
| REAX 1.3 Open · 57B | — | 47.5%249/524 | 64.4%302/469 | 87.9%116/132 |
| KnowLine-31B-Gen1.5 Open · 31B | — | 47.1%247/524 | 64.2%301/469 | 87.9%116/132 |
| KnowLine-31B-Gen2 Open · 31B | — | 47.1%247/524 | 63.8%299/469 | 88.6%117/132 |
| Copernicus-31B Open · 31B | — | 47.1%247/524 | 63.8%299/469 | 86.4%114/132 |
| GLiDE 27B Open · 27B | — | 42.0%220/524 | 68.0%319/469 | 89.4%118/132 |
| Sieve 31B v3 Open · 31B | — | 45.8%240/524 | 62.9%295/469 | 87.9%116/132 |
| Logits Sigma-1 Duo Open · 57B | — | 44.7%234/524 | 63.8%299/469 | 88.6%117/132 |
| GLiDE v1.2 Open · 27B | — | 40.5%212/524 | 63.1%296/469 | 85.6%113/132 |
| d1-ultra Closed · Efficient | — | 44.7%234/524 | 57.4%269/469 | 87.1%115/132 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 41.4%217/524 | 60.8%285/469 | 86.4%114/132 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 41.4%217/524 | 59.9%281/469 | 89.4%118/132 |
| d3-ultra Open · 26B | — | 42.2%221/524 | 59.3%278/469 | 87.1%115/132 |
| GLiDE API Closed · Efficient | — | 38.7%203/524 | 63.1%296/469 | 86.4%114/132 |
| Mammoth-1 Open · 27B | — | 42.7%224/524 | 58.6%275/469 | 86.4%114/132 |
| d3 Open · 26B | — | 40.6%213/524 | 60.6%284/469 | 87.9%116/132 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 39.9%209/524 | 58.4%274/469 | 88.6%117/132 |
| Drex Max Open · 27B | — | 42.4%222/524 | 55.7%261/469 | 86.4%114/132 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 39.1%205/524 | 58.0%272/469 | 86.4%114/132 |
| SubQ Snap 1.1 Closed · Efficient | — | 37.2%195/524 | 60.3%283/469 | 84.1%111/132 |
| Turion Jev 31B Open · 31B | — | 36.6%192/524 | 60.1%282/469 | 86.4%114/132 |
| d1.2-ultra Closed · Undisclosed | — | 38.2%200/524 | 57.4%269/469 | 84.1%111/132 |
| KnowLine-12B-Gen1 Open · 12B | — | 36.6%192/524 | 56.7%266/469 | 84.1%111/132 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 39.3%206/524 | 53.1%249/469 | 81.8%108/132 |
| d1.2 Closed · Undisclosed | — | 39.7%208/524 | 51.6%242/469 | 78.0%103/132 |
| Jeffan-9B-1211 Open · 9.0B | — | 34.2%179/524 | 53.7%252/469 | 84.1%111/132 |
| Jev 1.14 Closed · Efficient | — | 35.7%187/524 | 53.1%249/469 | 80.3%106/132 |
| GLiDE 9B Open · 7.9B | — | 32.1%168/524 | 55.7%261/469 | 82.6%109/132 |
| celeris-1-decision Closed · Efficient | — | 37.2%195/524 | 50.5%237/469 | 75.8%100/132 |
| celeris-1.1-decision Closed · Efficient | — | 37.8%198/524 | 49.5%232/469 | 77.3%102/132 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 32.3%169/524 | 52.5%246/469 | 82.6%109/132 |
| Adjudex 9B v5 Open · 9.0B | — | 34.0%178/524 | 49.9%234/469 | 83.3%110/132 |
| Elephant-1 Open · 9.0B | — | 33.2%174/524 | 51.0%239/469 | 82.6%109/132 |
| Jev 1.13 Closed · Efficient | — | 33.2%174/524 | 52.0%244/469 | 78.0%103/132 |
| d1.1 Closed · Efficient | — | 34.7%182/524 | 50.7%238/469 | 74.2%98/132 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 32.4%170/524 | 50.7%238/469 | 81.8%108/132 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 34.9%183/524 | 47.3%222/469 | 79.5%105/132 |
| Mercury Decide Closed · Efficient | — | 33.4%175/524 | 51.6%242/469 | 68.2%90/132 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 30.7%161/524 | 50.1%235/469 | 81.1%107/132 |
| Microsoft-Decision-1 Closed · Efficient | — | 34.0%178/524 | 46.9%220/469 | 77.3%102/132 |
| Valen-Decision-1.5-9B Open · 9.0B | — | 34.2%179/524 | 45.8%215/469 | 79.5%105/132 |
| KnowLine-9B-Gen1 Open · 9.0B | — | 32.4%170/524 | 47.5%223/469 | 79.5%105/132 |
| vera-medium-omni Open · 4.2B | — | 31.9%167/524 | 48.6%228/469 | 76.5%101/132 |
| KnowLine-9B-Gen2 Open · 9.0B | — | 33.4%175/524 | 45.2%212/469 | 81.1%107/132 |
| Cerno 9B v1.2 Open · 9.0B | — | 31.1%163/524 | 48.0%225/469 | 75.0%99/132 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 31.3%164/524 | 46.1%216/469 | 79.5%105/132 |
| Open-Jev-27B-v1.1 Open · 27B | — | 31.1%163/524 | 47.1%221/469 | 72.0%95/132 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 30.2%158/524 | 47.5%223/469 | 72.0%95/132 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 34.2%179/524 | 41.4%194/469 | 74.2%98/132 |
| Clef Open · 27B | — | 30.2%158/524 | 46.1%216/469 | 64.4%85/132 |
| d1 Closed · Efficient | — | 29.0%152/524 | 44.3%208/469 | 75.0%99/132 |
| OneJev-27B Open · 27B | — | 30.0%157/524 | 45.0%211/469 | 67.4%89/132 |
| Chinese Decision Pointer 4B Open · 4.0B | — | 29.0%152/524 | 43.9%206/469 | 72.7%96/132 |
| Verdin-4B-v1.1 Open · 4.0B | — | 28.4%149/524 | 42.4%199/469 | 75.0%99/132 |
| Winnow-12B Open · 12B | — | 26.5%139/524 | 45.4%213/469 | 72.0%95/132 |
| Cygnet Open · 12B | — | 26.9%141/524 | 47.3%222/469 | 59.8%79/132 |
| Clef-Flash Open · 9.0B | — | 30.0%157/524 | 40.1%188/469 | 70.5%93/132 |
| GLiDE 2B Open · 1.9B | — | 26.5%139/524 | 33.0%155/469 | 52.3%69/132 |
| Metask-Jev-4B Open · 4.5B | — | 24.6%129/524 | 34.5%162/469 | 53.0%70/132 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 25.0%131/524 | 32.2%151/469 | 56.8%75/132 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 25.8%135/524 | 29.9%140/469 | 53.8%71/132 |
| d1-2B Open · 1.9B | — | 24.8%130/524 | 31.8%149/469 | 50.0%66/132 |
| GLiDE 0.8B Open · 0.8B | — | 25.6%134/524 | 29.0%136/469 | 43.9%58/132 |
| OneJev1.6-0.8B Open · 0.8B | — | 24.0%126/524 | 27.5%129/469 | 37.1%49/132 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 25.2%132/524 | 24.3%114/469 | 40.9%54/132 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 19.5%102/524 | 25.6%120/469 | 42.4%56/132 |
| GLiClass v4 Edge Open · 0.4B | — | 23.9%125/524 | 22.0%103/469 | 28.0%37/132 |
| GLiDE 0.4B Open · 0.4B | — | 20.8%109/524 | 24.1%113/469 | 29.5%39/132 |
| Strands Decider 2B Open · 2.0B | — | 19.3%101/524 | 24.1%113/469 | 28.8%38/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Decision style
k = 5 by our rule (methods agree, ARI ≥ 0.80); it is also the most stable.
Skill and “I don't know”, 5 styles
68 decision models, one dot each in its maker's colour: Logic accuracy across, the “I don't know” rate up. Each style is a shaded area and a dot shape; its name is computed from the data.
Karpo Decision 31B: Strong; Logic 60.7%, Math 81.6%, “I don't know” 1.52%
REAX 1.3: Strong; Logic 59.3%, Math 78.0%, “I don't know” 1.09%
KnowLine-31B-Gen1.5: Strong; Logic 59.0%, Math 77.4%, “I don't know” 0.21%
KnowLine-31B-Gen2: Strong; Logic 58.9%, Math 80.3%, “I don't know” 0.03%
Copernicus-31B: Strong; Logic 58.7%, Math 77.3%, “I don't know” 0.52%
GLiDE 27B: Strong; Logic 58.4%, Math 72.0%, “I don't know” 11.47%
Sieve 31B v3: Strong; Logic 57.9%, Math 77.1%, “I don't know” 1.13%
Logits Sigma-1 Duo: Strong; Logic 57.8%, Math 75.2%, “I don't know” 1.37%
GLiDE (v1.2): Strong; Logic 55.2%, Math 69.9%, “I don't know” 17.18%
d1-ultra: Strong, cautious; Logic 54.9%, Math 74.1%, “I don't know” 47.75%
Drex Max v1.1: Strong; Logic 54.8%, Math 70.1%, “I don't know” 20.77%
Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): Strong; Logic 54.8%, Math 72.5%, “I don't know” 4.93%
d3-ultra: Strong; Logic 54.6%, Math 71.5%, “I don't know” 2.65%
GLiDE (API): Strong; Logic 54.5%, Math 70.8%, “I don't know” 20.47%
Mammoth-1: Strong, cautious; Logic 54.5%, Math 74.0%, “I don't know” 30.63%; Ward puts it in Strong
d3: Strong; Logic 54.5%, Math 70.4%, “I don't know” 3.41%
Perplexity Decider 27B v1.2 (API): Strong; Logic 53.3%, Math 69.8%, “I don't know” 3.86%
Drex Max: Strong; Logic 53.1%, Math 70.3%, “I don't know” 2.07%
SubQ Snap 1.1: Strong, cautious; Logic 52.4%, Math 68.6%, “I don't know” 26.82%; Ward puts it in Strong
Turion Jev 31B: Strong; Logic 52.3%, Math 69.0%, “I don't know” 0.67%
d1.2-ultra: Strong, cautious; Logic 51.6%, Math 69.9%, “I don't know” 51.92%
KnowLine-12B-Gen1: Strong; Logic 50.6%, Math 66.3%, “I don't know” 1.79%
Gemini Decision 1 (20261111): Strong; Logic 50.0%, Math 71.3%, “I don't know” 1.13%
d1.2: Strong, cautious; Logic 49.2%, Math 65.5%, “I don't know” 34.91%
Jeffan-9B-1211: Middle; Logic 48.2%, Math 59.8%, “I don't know” 6.27%
Jev 1.14: Middle; Logic 48.2%, Math 65.1%, “I don't know” 6.66%
GLiDE 9B: Middle; Logic 47.8%, Math 57.9%, “I don't know” 13.53%
celeris-1-decision: Strong, cautious; Logic 47.3%, Math 70.0%, “I don't know” 49.82%
celeris-1.1-decision: Strong, cautious; Logic 47.3%, Math 69.9%, “I don't know” 49.24%
StartLux-Decision-v1.1-9B-Instant: Middle; Logic 46.6%, Math 58.4%, “I don't know” 4.59%
Adjudex 9B v5: Middle; Logic 46.4%, Math 62.1%, “I don't know” 3.13%
Elephant-1: Strong, cautious; Logic 46.4%, Math 61.3%, “I don't know” 39.72%
Jev 1.13: Middle; Logic 46.3%, Math 64.1%, “I don't know” 4.47%
d1.1: Strong, cautious; Logic 46.0%, Math 65.2%, “I don't know” 53.53%
StartLux-Decision-v1.1-4B-Instant: Middle; Logic 45.9%, Math 55.8%, “I don't know” 7.36%
Dexy-Decision-9B-V2: Middle; Logic 45.3%, Math 58.8%, “I don't know” 2.89%
Mercury Decide: Middle; Logic 45.1%, Math 53.5%, “I don't know” 5.14%
Perplexity Decider Flash v1.3: Middle; Logic 44.7%, Math 55.9%, “I don't know” 7.60%
Microsoft-Decision-1: Middle; Logic 44.4%, Math 59.4%, “I don't know” 5.50%
Valen-Decision-1.5-9B: Middle; Logic 44.4%, Math 55.3%, “I don't know” 3.59%
KnowLine-9B-Gen1: Middle; Logic 44.3%, Math 59.8%, “I don't know” 1.58%
vera-medium-omni: Middle; Logic 44.1%, Math 56.8%, “I don't know” 3.86%
KnowLine-9B-Gen2: Middle; Logic 43.9%, Math 60.1%, “I don't know” 4.99%
Cerno 9B v1.2: Middle; Logic 43.3%, Math 56.0%, “I don't know” 1.49%
Perplexity Decider Flash v1.2: Middle; Logic 43.1%, Math 55.2%, “I don't know” 10.01%
Open-Jev-27B-v1.1: Middle; Logic 42.6%, Math 54.6%, “I don't know” 7.57%
KnowLine 4B Gen5.5: Middle; Logic 42.3%, Math 56.0%, “I don't know” 2.19%
Drex 2.0: Middle; Logic 41.9%, Math 59.0%, “I don't know” 14.66%
Clef: Middle; Logic 40.8%, Math 51.1%, “I don't know” 1.25%
d1: Middle; Logic 40.8%, Math 55.6%, “I don't know” 3.35%
OneJev-27B: Middle; Logic 40.6%, Math 53.8%, “I don't know” 0.79%
Chinese Decision Pointer 4B: Middle; Logic 40.4%, Math 50.4%, “I don't know” 4.20%
Verdin-4B-v1.1: Middle; Logic 39.7%, Math 51.5%, “I don't know” 4.84%
Winnow-12B: Middle; Logic 39.7%, Math 46.9%, “I don't know” 10.19%
Cygnet: Middle; Logic 39.3%, Math 44.3%, “I don't know” 4.26%
Clef-Flash: Middle; Logic 38.9%, Math 49.8%, “I don't know” 3.80%
GLiDE 2B: Low-scoring; Logic 32.3%, Math 36.8%, “I don't know” 15.91%
Metask-Jev-4B: Low-scoring; Logic 32.1%, Math 39.2%, “I don't know” 1.28%
KnowLine-2B-Gen1: Low-scoring; Logic 31.7%, Math 35.3%, “I don't know” 4.65%
StartLux-Decision-v1.1-2B-Instant: Low-scoring; Logic 30.8%, Math 38.7%, “I don't know” 11.07%
d1-2B: Low-scoring, cautious; Logic 30.7%, Math 37.1%, “I don't know” 70.41%
GLiDE 0.8B: Low-scoring; Logic 29.2%, Math 32.1%, “I don't know” 16.76%
OneJev1.6-0.8B: Low-scoring; Logic 27.0%, Math 28.2%, “I don't know” 1.46%
KnowLine-0.8B-Gen1: Low-scoring; Logic 26.7%, Math 28.2%, “I don't know” 0.46%
StartLux-Decision-v1.1-0.8B-Instant: Low-scoring; Logic 24.7%, Math 29.7%, “I don't know” 11.13%
GLiClass v4 Edge: Low-scoring; Logic 23.6%, Math 22.7%, “I don't know” 12.65%
GLiDE 0.4B: Low-scoring, cautious; Logic 23.2%, Math 20.6%, “I don't know” 68.28%
Strands Decider 2B: Low-scoring; Logic 22.4%, Math 28.7%, “I don't know” 3.74%
Strong Strong, cautious Middle Low-scoring Low-scoring, cautious MachinePulse REAX PelaAI Poindexter Labs Fastino Sthānika AI Logits Liquid AI Nace.AI Perplexity vLLM Semantic Router Base Labs Subquadratic Turion.AI Google tattrongvu TypeSafe Celeris StartLux newsbang Patrick-18 Inception Microsoft Valen-Team scar-ai Apertix AI Zefan Cai Cloudflare OmniJev quincyqiang victorbennett EldanRing Blockbrain Labs Metask Lab Other Strands Agents
The groups come from the data, not from our rules; borderline models can switch groups.
Shaded areas only outline each style's dots; they are not confidence regions.
Styles (k = 5)
| Styleits members under it | Models | Logicmean accuracy | Mathmean accuracy | “I don't know”mean rate | Mean zLogic / Math / IDK |
|---|---|---|---|---|---|
| Strong | 19 | 55.7% | 73.2% | 5.07% | +1.08 / +1.00 / -0.44 |
| Karpo Decision 31B, REAX 1.3, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2, Copernicus-31B, GLiDE 27B, Sieve 31B v3, Logits Sigma-1 Duo, GLiDE v1.2, Drex Max v1.1, Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell, d3-ultra, GLiDE API, d3, Perplexity Decider 27B v1.2 API, Drex Max, Turion Jev 31B, KnowLine-12B-Gen1, Gemini Decision 1 20261111 | |||||
| Strong, cautious | 9 | 49.9% | 68.7% | 42.70% | +0.50 / +0.70 / +1.78 |
| d1-ultra, Mammoth-1, SubQ Snap 1.1, d1.2-ultra, d1.2, celeris-1-decision, celeris-1.1-decision, Elephant-1, d1.1 | |||||
| Middle | 28 | 43.5% | 56.0% | 5.35% | -0.15 / -0.14 / -0.42 |
| Jeffan-9B-1211, Jev 1.14, GLiDE 9B, StartLux-Decision-v1.1-9B-Instant, Adjudex 9B v5, Jev 1.13, StartLux-Decision-v1.1-4B-Instant, Dexy-Decision-9B-V2, Mercury Decide, Perplexity Decider Flash v1.3, Microsoft-Decision-1, Valen-Decision-1.5-9B, KnowLine-9B-Gen1, vera-medium-omni, KnowLine-9B-Gen2, Cerno 9B v1.2, Perplexity Decider Flash v1.2, Open-Jev-27B-v1.1, KnowLine 4B Gen5.5, Drex 2.0, Clef, d1, OneJev-27B, Chinese Decision Pointer 4B, Verdin-4B-v1.1, Winnow-12B, Cygnet, Clef-Flash | |||||
| Low-scoring | 10 | 28.0% | 31.9% | 7.91% | -1.72 / -1.74 / -0.27 |
| GLiDE 2B, Metask-Jev-4B, KnowLine-2B-Gen1, StartLux-Decision-v1.1-2B-Instant, GLiDE 0.8B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-0.8B-Instant, GLiClass v4 Edge, Strands Decider 2B | |||||
| Low-scoring, cautious | 2 | 26.9% | 28.8% | 69.34% | -1.83 / -1.94 / +3.34 |
| d1-2B, GLiDE 0.4B | |||||
Models (k = 5)
| Model | Stylek-means | Logic | Math | “I don't know” | Wardits style | Same partnersquestions resampled | Same partnersmodels resampled |
|---|---|---|---|---|---|---|---|
| Karpo Decision 31B | Strong | 60.7% | 81.6% | 1.52% | same | 99% | 96% |
| REAX 1.3 | Strong | 59.3% | 78.0% | 1.09% | same | 99% | 96% |
| KnowLine-31B-Gen1.5 | Strong | 59.0% | 77.4% | 0.21% | same | 99% | 95% |
| KnowLine-31B-Gen2 | Strong | 58.9% | 80.3% | 0.03% | same | 99% | 95% |
| Copernicus-31B | Strong | 58.7% | 77.3% | 0.52% | same | 99% | 96% |
| GLiDE 27B | Strong | 58.4% | 72.0% | 11.47% | same | 99% | 95% |
| Sieve 31B v3 | Strong | 57.9% | 77.1% | 1.13% | same | 99% | 95% |
| Logits Sigma-1 Duo | Strong | 57.8% | 75.2% | 1.37% | same | 99% | 96% |
| GLiDE v1.2 | Strong | 55.2% | 69.9% | 17.18% | same | 99% | 87% |
| Drex Max v1.1 | Strong | 54.8% | 70.1% | 20.77% | same | 99% | 79% |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell | Strong | 54.8% | 72.5% | 4.93% | same | 99% | 96% |
| d3-ultra | Strong | 54.6% | 71.5% | 2.65% | same | 99% | 96% |
| GLiDE API | Strong | 54.5% | 70.8% | 20.47% | same | 99% | 83% |
| d3 | Strong | 54.5% | 70.4% | 3.41% | same | 99% | 96% |
| Perplexity Decider 27B v1.2 API | Strong | 53.3% | 69.8% | 3.86% | same | 99% | 96% |
| Drex Max | Strong | 53.1% | 70.3% | 2.07% | same | 99% | 96% |
| Turion Jev 31B | Strong | 52.3% | 69.0% | 0.67% | same | 99% | 95% |
| KnowLine-12B-Gen1 | Strong | 50.6% | 66.3% | 1.79% | same | 94% | 92% |
| Gemini Decision 1 20261111 | Strong | 50.0% | 71.3% | 1.13% | same | 99% | 95% |
| d1-ultra | Strong, cautious | 54.9% | 74.1% | 47.75% | same | 100% | 96% |
| Mammoth-1 | Strong, cautious | 54.5% | 74.0% | 30.63% | Strong | 100% | 88% |
| SubQ Snap 1.1 | Strong, cautious | 52.4% | 68.6% | 26.82% | Strong | 100% | 83% |
| d1.2-ultra | Strong, cautious | 51.6% | 69.9% | 51.92% | same | 100% | 95% |
| d1.2 | Strong, cautious | 49.2% | 65.5% | 34.91% | same | 100% | 94% |
| celeris-1-decision | Strong, cautious | 47.3% | 70.0% | 49.82% | same | 100% | 95% |
| celeris-1.1-decision | Strong, cautious | 47.3% | 69.9% | 49.24% | same | 100% | 95% |
| Elephant-1 | Strong, cautious | 46.4% | 61.3% | 39.72% | same | 100% | 95% |
| d1.1 | Strong, cautious | 46.0% | 65.2% | 53.53% | same | 100% | 95% |
| Jeffan-9B-1211 | Middle | 48.2% | 59.8% | 6.27% | same | 99% | 98% |
| Jev 1.14 | Middle | 48.2% | 65.1% | 6.66% | same | 77% | 83% |
| GLiDE 9B | Middle | 47.8% | 57.9% | 13.53% | same | 99% | 98% |
| StartLux-Decision-v1.1-9B-Instant | Middle | 46.6% | 58.4% | 4.59% | same | 99% | 98% |
| Adjudex 9B v5 | Middle | 46.4% | 62.1% | 3.13% | same | 99% | 98% |
| Jev 1.13 | Middle | 46.3% | 64.1% | 4.47% | same | 98% | 98% |
| StartLux-Decision-v1.1-4B-Instant | Middle | 45.9% | 55.8% | 7.36% | same | 99% | 98% |
| Dexy-Decision-9B-V2 | Middle | 45.3% | 58.8% | 2.89% | same | 99% | 99% |
| Mercury Decide | Middle | 45.1% | 53.5% | 5.14% | same | 99% | 99% |
| Perplexity Decider Flash v1.3 | Middle | 44.7% | 55.9% | 7.60% | same | 99% | 98% |
| Microsoft-Decision-1 | Middle | 44.4% | 59.4% | 5.50% | same | 99% | 98% |
| Valen-Decision-1.5-9B | Middle | 44.4% | 55.3% | 3.59% | same | 99% | 99% |
| KnowLine-9B-Gen1 | Middle | 44.3% | 59.8% | 1.58% | same | 99% | 98% |
| vera-medium-omni | Middle | 44.1% | 56.8% | 3.86% | same | 99% | 98% |
| KnowLine-9B-Gen2 | Middle | 43.9% | 60.1% | 4.99% | same | 99% | 99% |
| Cerno 9B v1.2 | Middle | 43.3% | 56.0% | 1.49% | same | 99% | 98% |
| Perplexity Decider Flash v1.2 | Middle | 43.1% | 55.2% | 10.01% | same | 99% | 98% |
| Open-Jev-27B-v1.1 | Middle | 42.6% | 54.6% | 7.57% | same | 99% | 98% |
| KnowLine 4B Gen5.5 | Middle | 42.3% | 56.0% | 2.19% | same | 99% | 98% |
| Drex 2.0 | Middle | 41.9% | 59.0% | 14.66% | same | 99% | 98% |
| Clef | Middle | 40.8% | 51.1% | 1.25% | same | 99% | 97% |
| d1 | Middle | 40.8% | 55.6% | 3.35% | same | 99% | 98% |
| OneJev-27B | Middle | 40.6% | 53.8% | 0.79% | same | 99% | 98% |
| Chinese Decision Pointer 4B | Middle | 40.4% | 50.4% | 4.20% | same | 99% | 97% |
| Verdin-4B-v1.1 | Middle | 39.7% | 51.5% | 4.84% | same | 99% | 98% |
| Winnow-12B | Middle | 39.7% | 46.9% | 10.19% | same | 99% | 97% |
| Cygnet | Middle | 39.3% | 44.3% | 4.26% | same | 98% | 97% |
| Clef-Flash | Middle | 38.9% | 49.8% | 3.80% | same | 99% | 97% |
| GLiDE 2B | Low-scoring | 32.3% | 36.8% | 15.91% | same | 100% | 100% |
| Metask-Jev-4B | Low-scoring | 32.1% | 39.2% | 1.28% | same | 100% | 100% |
| KnowLine-2B-Gen1 | Low-scoring | 31.7% | 35.3% | 4.65% | same | 100% | 100% |
| StartLux-Decision-v1.1-2B-Instant | Low-scoring | 30.8% | 38.7% | 11.07% | same | 100% | 100% |
| GLiDE 0.8B | Low-scoring | 29.2% | 32.1% | 16.76% | same | 100% | 100% |
| OneJev1.6-0.8B | Low-scoring | 27.0% | 28.2% | 1.46% | same | 100% | 100% |
| KnowLine-0.8B-Gen1 | Low-scoring | 26.7% | 28.2% | 0.46% | same | 100% | 100% |
| StartLux-Decision-v1.1-0.8B-Instant | Low-scoring | 24.7% | 29.7% | 11.13% | same | 100% | 100% |
| GLiClass v4 Edge | Low-scoring | 23.6% | 22.7% | 12.65% | same | 100% | 100% |
| Strands Decider 2B | Low-scoring | 22.4% | 28.7% | 3.74% | same | 100% | 100% |
| d1-2B | Low-scoring, cautious | 30.7% | 37.1% | 70.41% | same | 100% | 99% |
| GLiDE 0.4B | Low-scoring, cautious | 23.2% | 20.6% | 68.28% | same | 100% | 100% |
How many styles
| k | k-means vs WardARI | Stability k-meansquestions resampled | Stability Wardquestions resampled | Stability k-meansmodels resampled | Stability Wardmodels resampled | Mean stability | Silhouettek-means | SilhouetteWard |
|---|---|---|---|---|---|---|---|---|
| 2 | 0.73 | 0.91 | 0.82 | 0.70 | 0.62 | 0.76 | 0.50 | 0.46 |
| 3 | 0.79 | 0.83 | 0.79 | 0.61 | 0.60 | 0.71 | 0.49 | 0.51 |
| 4 | 0.92 | 0.96 | 0.93 | 0.89 | 0.86 | 0.91 | 0.55 | 0.54 |
| 5 ✓ | 0.94 | 0.98 | 0.94 | 0.97 | 0.91 | 0.95 | 0.59 | 0.58 |
| 6 | 0.98 | 0.98 | 0.94 | 0.87 | 0.89 | 0.92 | 0.58 | 0.57 |
How the styles are found
Models. The Logic leaderboard's decision rows (one row per model by the site's rule) whose setup has complete English standard runs on Logic (1,125 questions) and Math (2,163) and a complete “I don't know” run on both: 68 models. Rows with web or on a sample are left out, as is any row missing a run (never filled in). Left out: Perplexity Decider 27B v1.2 (API · with web) (with web).
Numbers. Logic and Math accuracy: the standard runs, as on this site. “I don't know” rate: the answers that chose “I don't know” (option F) over all answers of the model's two “I don't know” runs, Logic and Math pooled (3,288 answers); not the non-hallucination rate F / (F + wrong). Each number is turned into a z-score over these models (mean 0, SD 1).
Clustering. k-means (best of 100 seeded restarts, seed 4317) and Ward clustering on the three z-scores for k = 2–6. Agreement: the adjusted Rand index (ARI) of the two (1 = the same groups; label-free). Stability: 500 bootstrap replicates (seed 4317) of each resampling unit, the questions (one resampled question set per benchmark, the same for every model and for both its runs) and the models (drawn with replacement; each model drawn is compared once). In every replicate the three numbers, the z-scores and both clusterings are refitted (k-means: 20 restarts) and compared with the full-data result by ARI on the models present. Same partners: per model, the mean share of its style partners it keeps in a replicate (Jaccard of the two partner sets, itself included), k-means at that k.
Choosing k. The rule for k is chosen by us (analyst-selected): among the k at which k-means and Ward agree (ARI ≥ 0.80), the one with the highest mean stability (the mean of the four bootstrap stabilities: k-means and Ward, questions and models resampled); a tie to the smaller k. If no k agrees, the k with the highest min(agreement, mean stability). Chosen: k = 5.
Names. A style's name from its mean z-scores (Logic, Math, IDK): skill = the mean of the two accuracy z-scores, Strong at or above +0.5, Low-scoring at or below −0.5, else Middle; IDK at or above +0.5 'cautious', at or below −0.5 'decisive'. Middle with an IDK word is that word alone ('Cautious', 'Decisive'). Styles are listed by mean skill, highest first.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 1,125 of 1,125); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Accuracy 60.7% (95%: 57.8%–63.5%) Math 81.6% | — | 60.7%57.8%–63.5% | ab | 683 / 1,125 | 169 right of 172 kept3 wrong · 953 skipped | 0.816 | 71.80chosen-answer Brier 0.1711 | 6.6%answered 97% | 173 ms | 211 ms | $0.128$128estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Accuracy 59.3% (95%: 56.4%–62.1%) Math 78.0% | — | 59.3%56.4%–62.1% | ac | 667 / 1,125 | 51 right of 51 kept0 mistakes · 1,074 skipped | 0.816 | 71.07chosen-answer Brier 0.1714 | 5.3%answered 98% | 215 ms | 278 ms | $0.160$160estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Accuracy 59.0% (95%: 56.1%–61.9%) Math 77.4% | — | 59.0%56.1%–61.9% | abcd | 664 / 1,125 | 89 right of 89 kept0 mistakes · 1,036 skipped | 0.804 | 70.72chosen-answer Brier 0.1757 | 0.4%answered 100% | 983 ms | 1,279 ms | $0.061$61.43estimate · DGX Spark |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Accuracy 58.9% (95%: 56.0%–61.8%) Math 80.3% | — | 58.9%56.0%–61.8% | abcd | 663 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.825 | 71.09chosen-answer Brier 0.1675 | 0.0%answered 100% | 151 ms | 204 ms | $0.112$112estimate · RTX PRO 6000 Blackwell |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Accuracy 58.7% (95%: 55.8%–61.5%) Math 77.3% | — | 58.7%55.8%–61.5% | abcd | 660 / 1,125 | 66 right of 66 kept0 mistakes · 1,059 skipped | 0.801 | 70.48chosen-answer Brier 0.1771 | 1.9%answered 99% | 185 ms | 255 ms | $0.136$136estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Accuracy 58.4% (95%: 55.5%–61.2%) Math 72.0% | — | 58.4%55.5%–61.2% | abcde | 657 / 1,125 | 74 right of 75 kept1 wrong · 1,050 skipped | 0.822 | 70.67chosen-answer Brier 0.1707 | 44.3%answered 79% | 544 ms | 655 ms | $0.033$33.16estimate · DGX Spark |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Accuracy 57.9% (95%: 55.0%–60.7%) Math 77.1% | — | 57.9%55.0%–60.7% | abcdf | 651 / 1,125 | 79 right of 81 kept2 wrong · 1,044 skipped | 0.797 | 70.03chosen-answer Brier 0.1781 | 5.5%answered 98% | 1,050 ms | 1,335 ms | $0.063$62.95estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Accuracy 57.8% (95%: 54.9%–60.6%) Math 75.2% | — | 57.8%54.9%–60.6% | abcdefg | 650 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.796 | 69.90chosen-answer Brier 0.1798 | 7.1%answered 97% | 285 ms | 399 ms | $0.211$211estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Accuracy 55.2% (95%: 52.3%–58.1%) Math 69.9% | — | 55.2%52.3%–58.1% | abcdefgh | 621 / 1,125 | 45 right of 45 kept0 mistakes · 1,080 skipped | 0.793 | 68.39chosen-answer Brier 0.1842 | 50.2%answered 73% | 111 ms | 140 ms | $0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Accuracy 54.9% (95%: 52.0%–57.8%) Math 74.1% | — | 54.9%52.0%–57.8% | abcdefgh | 618 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.783 | 68.09chosen-answer Brier 0.1876 | 91.0%answered 35% | 182 ms | 425 ms | $0.013$12.64 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Accuracy 54.8% (95%: 51.8%–57.6%) Math 70.1% | — | 54.8%51.8%–57.6% | abcdefghi | 616 / 1,125 | 89 right of 90 kept1 wrong · 1,035 skipped | 0.793 | 68.07chosen-answer Brier 0.1861 | 54.1%answered 69% | 329 ms | 1,540 ms | $0.022$22.37 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Accuracy 54.8% (95%: 51.8%–57.6%) Math 72.5% | — | 54.8%51.8%–57.6% | abcdefgh | 616 / 1,125 | 62 right of 62 kept0 mistakes · 1,063 skipped | 0.803 | 67.98chosen-answer Brier 0.1881 | 18.1%answered 91% | 108 ms | 131 ms | $0.081$81.16estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 54.6% (95%: 51.7%–57.5%) Math 71.5% | — | 54.6%51.7%–57.5% | bdefghij | 614 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.778 | 66.46chosen-answer Brier 0.2166 | 10.1%answered 95% | 156 ms | 185 ms | $0.109$109estimate · RTX PRO 6000 Blackwell |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Accuracy 54.5% (95%: 51.6%–57.4%) Math 70.8% | — | 54.5%51.6%–57.4% | cdefghi | 613 / 1,125 | 52 right of 52 kept0 mistakes · 1,073 skipped | 0.814 | 68.27chosen-answer Brier 0.1794 | 60.0%answered 67% | 271 ms | 291 ms | $0.052$52.03 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Accuracy 54.5% (95%: 51.6%–57.4%) Math 74.0% | — | 54.5%51.6%–57.4% | cdefghij | 613 / 1,125 | 106 right of 108 kept2 wrong · 1,017 skipped | 0.795 | 68.07chosen-answer Brier 0.1835 | 72.3%answered 58% | 221 ms | 244 ms | $0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 54.5% (95%: 51.6%–57.4%) Math 70.4% | — | 54.5%51.6%–57.4% | cdefgh | 613 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.806 | 68.34chosen-answer Brier 0.1781 | 14.8%answered 93% | 76 ms | 98 ms | $0.057$57.18estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Accuracy 53.3% (95%: 50.4%–56.2%) Math 69.8% | — | 53.3%50.4%–56.2% | defghijk | 600 / 1,125 | 78 right of 80 kept2 wrong · 1,045 skipped | 0.798 | 67.64chosen-answer Brier 0.1805 | 14.4%answered 93% | 238 ms | 552 ms | $0.007$7.46 |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Accuracy 53.1% (95%: 50.1%–56.0%) Math 70.3% | — | 53.1%50.1%–56.0% | defghijk | 597 / 1,125 | 147 right of 155 kept8 wrong · 970 skipped | 0.786 | 65.68chosen-answer Brier 0.2170 | 7.3%answered 97% | 106 ms | 143 ms | $0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Accuracy 52.5% (95%: 49.6%–55.4%) Math 69.3% | with web | 52.5%49.6%–55.4% | fghijkl | 591 / 1,125 | 78 right of 79 kept1 wrong · 1,046 skipped | 0.800 | 67.24chosen-answer Brier 0.1806 | 37.7%answered 80% | 448 ms | 827 ms | $1.030$1,030 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Accuracy 52.4% (95%: 49.4%–55.3%) Math 68.6% | — | 52.4%49.4%–55.3% | fghijklm | 589 / 1,125 | 70 right of 72 kept2 wrong · 1,053 skipped | 0.787 | 65.94chosen-answer Brier 0.2047 | 65.7%answered 61% | 384 ms | 393 ms | $0.015$14.92 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Accuracy 52.3% (95%: 49.3%–55.2%) Math 69.0% | — | 52.3%49.3%–55.2% | eghijklmn | 588 / 1,125 | 7 right of 7 kept0 mistakes · 1,118 skipped | 0.782 | 66.89chosen-answer Brier 0.1849 | 1.5%answered 99% | 385 ms | 533 ms | $0.025$24.91estimate · DGX Spark |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 51.6% (95%: 48.6%–54.5%) Math 69.9% | — | 51.6%48.6%–54.5% | hijklmno | 580 / 1,125 | 13 right of 13 kept0 mistakes · 1,112 skipped | 0.800 | 66.77chosen-answer Brier 0.1802 | 94.5%answered 28% | 220 ms | 341 ms | $0.013$12.64 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Accuracy 50.6% (95%: 47.7%–53.5%) Math 66.3% | — | 50.6%47.7%–53.5% | hijklmnop | 569 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.774 | 65.90chosen-answer Brier 0.1878 | 8.8%answered 95% | 68 ms | 88 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Accuracy 50.0% (95%: 47.1%–53.0%) Math 71.3% | — | 50.0%47.1%–53.0% | hijklmnopq | 563 / 1,125 | 43 right of 43 kept0 mistakes · 1,082 skipped | 0.768 | 65.37chosen-answer Brier 0.1930 | 2.0%answered 99% | 228 ms | 344 ms | $0.037$36.77 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 49.2% (95%: 46.2%–52.1%) Math 65.5% | — | 49.2%46.2%–52.1% | hijklmnopqr | 553 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.742 | 64.51chosen-answer Brier 0.2013 | 78.8%answered 43% | 222 ms | 390 ms | $0.006$6.02 |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 48.2% (95%: 45.3%–51.1%) Math 59.8% | — | 48.2%45.3%–51.1% | jklmnopqrs | 542 / 1,125 | 10 right of 10 kept0 mistakes · 1,115 skipped | 0.753 | 64.33chosen-answer Brier 0.1952 | 23.4%answered 87% | 52 ms | 66 ms | $0.039$39.35estimate · RTX PRO 6000 Blackwell |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 48.2% (95%: 45.3%–51.1%) Math 65.1% | — | 48.2%45.3%–51.1% | ijklmnopqrs | 542 / 1,125 | 61 right of 61 kept0 mistakes · 1,064 skipped | 0.750 | 64.14chosen-answer Brier 0.1990 | 24.5%answered 86% | 166 ms | 193 ms | $0.026$25.64 |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Accuracy 47.8% (95%: 44.9%–50.7%) Math 57.9% | — | 47.8%44.9%–50.7% | klmnopqrs | 538 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.749 | 63.95chosen-answer Brier 0.1993 | 30.4%answered 82% | 208 ms | 251 ms | $0.013$13.29estimate · DGX Spark |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 47.3% (95%: 44.4%–50.2%) Math 70.0% | — | 47.3%44.4%–50.2% | klmnopqrstu | 532 / 1,125 | No answers at this threshold | 0.735 | 62.94chosen-answer Brier 0.2141 | 92.3%answered 28% | 390 ms | 394 ms | $0.015$15.40 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 47.3% (95%: 44.4%–50.2%) Math 69.9% | — | 47.3%44.4%–50.2% | klmnopqrst | 532 / 1,125 | No answers at this threshold | 0.733 | 62.92chosen-answer Brier 0.2144 | 91.0%answered 29% | 383 ms | 390 ms | $0.002$1.54 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Accuracy 46.6% (95%: 43.7%–49.5%) Math 58.4% | — | 46.6%43.7%–49.5% | lmnopqrstuv | 524 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.762 | 63.55chosen-answer Brier 0.1947 | 17.4%answered 90% | 160 ms | 175 ms | $0.010$10.04estimate · DGX Spark |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 46.4% (95%: 43.5%–49.3%) Math 62.1% | — | 46.4%43.5%–49.3% | lmnopqrstuv | 522 / 1,125 | 54 right of 55 kept1 wrong · 1,070 skipped | 0.771 | 63.48chosen-answer Brier 0.1945 | 12.2%answered 93% | 41 ms | 46 ms | $0.031$31.36estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Accuracy 46.4% (95%: 43.5%–49.3%) Math 61.3% | — | 46.4%43.5%–49.3% | lmnopqrstuv | 522 / 1,125 | 53 right of 53 kept0 mistakes · 1,072 skipped | 0.768 | 63.39chosen-answer Brier 0.1962 | 75.6%answered 47% | 79 ms | 98 ms | $0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 46.3% (95%: 43.4%–49.2%) Math 64.1% | — | 46.3%43.4%–49.2% | lmnopqrstuv | 521 / 1,125 | 65 right of 65 kept0 mistakes · 1,060 skipped | 0.764 | 63.36chosen-answer Brier 0.1959 | 15.8%answered 91% | 167 ms | 203 ms | $0.026$25.64 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 46.0% (95%: 43.2%–49.0%) Math 65.2% | — | 46.0%43.2%–49.0% | mnopqrstuvw | 518 / 1,125 | 48 right of 48 kept0 mistakes · 1,077 skipped | 0.748 | 63.14chosen-answer Brier 0.1977 | 90.6%answered 28% | 176 ms | 272 ms | $0.006$6.02 |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Accuracy 45.9% (95%: 43.0%–48.8%) Math 55.8% | — | 45.9%43.0%–48.8% | nopqrstuvx | 516 / 1,125 | 21 right of 21 kept0 mistakes · 1,104 skipped | 0.720 | 62.63chosen-answer Brier 0.2061 | 16.0%answered 91% | 73 ms | 84 ms | $0.005$4.62estimate · DGX Spark |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Accuracy 45.3% (95%: 42.4%–48.3%) Math 58.8% | — | 45.3%42.4%–48.3% | opqrstuvwxy | 510 / 1,125 | 56 right of 56 kept0 mistakes · 1,069 skipped | 0.759 | 61.81chosen-answer Brier 0.2171 | 12.4%answered 93% | 47 ms | 53 ms | $0.034$33.81estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Accuracy 45.1% (95%: 42.2%–48.0%) Math 53.5% | — | 45.1%42.2%–48.0% | pqrstuvwxy | 507 / 1,125 | 84 right of 91 kept7 wrong · 1,034 skipped | 0.746 | 61.18chosen-answer Brier 0.2272 | 20.5%answered 88% | 301 ms | 360 ms | $0.008$7.52 |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Accuracy 44.7% (95%: 41.8%–47.6%) Math 55.9% | — | 44.7%41.8%–47.6% | pqrstuvwxy | 503 / 1,125 | 17 right of 17 kept0 mistakes · 1,108 skipped | 0.757 | 62.14chosen-answer Brier 0.2042 | 24.4%answered 85% | 33 ms | 41 ms | $0.026$26.44estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Accuracy 44.4% (95%: 41.6%–47.4%) Math 59.4% | — | 44.4%41.6%–47.4% | pqrstuvwxy | 500 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.712 | 61.53chosen-answer Brier 0.2139 | 19.8%answered 88% | 315 ms | 362 ms | $0.013$13.42 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 44.4% (95%: 41.5%–47.3%) Math 55.3% | — | 44.4%41.5%–47.3% | pqrstuvwxy | 499 / 1,125 | 35 right of 37 kept2 wrong · 1,088 skipped | 0.701 | 61.42chosen-answer Brier 0.2151 | 13.6%answered 92% | 60 ms | 67 ms | $0.042$42.42estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 44.3% (95%: 41.4%–47.2%) Math 59.8% | — | 44.3%41.4%–47.2% | pqrstuvwxy | 498 / 1,125 | 30 right of 30 kept0 mistakes · 1,095 skipped | 0.725 | 61.97chosen-answer Brier 0.2032 | 6.7%answered 96% | 50 ms | 61 ms | $0.036$36.27estimate · RTX PRO 6000 Blackwell |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Accuracy 44.1% (95%: 41.2%–47.0%) Math 56.8% | — | 44.1%41.2%–47.0% | qrstuvwxy | 496 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.739 | 61.90chosen-answer Brier 0.2028 | 13.6%answered 92% | 32 ms | 37 ms | $0.026$25.82estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 43.9% (95%: 41.0%–46.8%) Math 60.1% | — | 43.9%41.0%–46.8% | qrstuvwxy | 494 / 1,125 | No answers at this threshold | 0.730 | 61.90chosen-answer Brier 0.2011 | 19.5%answered 88% | 52 ms | 65 ms | $0.040$39.96estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Accuracy 43.3% (95%: 40.4%–46.2%) Math 56.0% | — | 43.3%40.4%–46.2% | rstuvwxy | 487 / 1,125 | 26 right of 27 kept1 wrong · 1,098 skipped | 0.752 | 60.41chosen-answer Brier 0.2247 | 6.0%answered 97% | 209 ms | 269 ms | $0.013$13.35estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Accuracy 43.1% (95%: 40.2%–46.0%) Math 55.2% | — | 43.1%40.2%–46.0% | qrstuvwxy | 485 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 61.34chosen-answer Brier 0.2043 | 28.9%answered 82% | 45 ms | 55 ms | $0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Accuracy 42.6% (95%: 39.7%–45.5%) Math 54.6% | — | 42.6%39.7%–45.5% | rstuvwxy | 479 / 1,125 | No answers at this threshold | 0.733 | 59.78chosen-answer Brier 0.2302 | 28.2%answered 82% | 413 ms | 576 ms | $0.294$294estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Accuracy 42.3% (95%: 39.5%–45.2%) Math 56.0% | — | 42.3%39.5%–45.2% | stuvwxy | 476 / 1,125 | 17 right of 18 kept1 wrong · 1,107 skipped | 0.708 | 59.66chosen-answer Brier 0.2298 | 8.3%answered 95% | 96 ms | 115 ms | $0.006$5.89estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Accuracy 41.9% (95%: 39.0%–44.8%) Math 59.0% | — | 41.9%39.0%–44.8% | stuvwxy | 471 / 1,125 | 33 right of 33 kept0 mistakes · 1,092 skipped | 0.717 | 60.73chosen-answer Brier 0.2040 | 37.7%answered 75% | 345 ms | 504 ms | $0.018$18.41 |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Accuracy 40.8% (95%: 38.0%–43.7%) Math 51.1% | — | 40.8%38.0%–43.7% | uvwxy | 459 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.697 | 58.55chosen-answer Brier 0.2369 | 3.6%answered 98% | 196 ms | 222 ms | $0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 40.8% (95%: 38.0%–43.7%) Math 55.6% | — | 40.8%38.0%–43.7% | tuvwxy | 459 / 1,125 | 55 right of 56 kept1 wrong · 1,069 skipped | 0.723 | 59.76chosen-answer Brier 0.2128 | 11.3%answered 93% | 188 ms | 290 ms | $0.006$6.02 |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Accuracy 40.6% (95%: 37.8%–43.5%) Math 53.8% | — | 40.6%37.8%–43.5% | tuvwxy | 457 / 1,125 | 4 right of 4 kept0 mistakes · 1,121 skipped | 0.726 | 60.27chosen-answer Brier 0.2009 | 1.2%answered 99% | 108 ms | 136 ms | $0.086$86.07estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 40.4% (95%: 37.5%–43.3%) Math 50.4% | — | 40.4%37.5%–43.3% | vwxy | 454 / 1,125 | 39 right of 39 kept0 mistakes · 1,086 skipped | 0.724 | 58.90chosen-answer Brier 0.2256 | 14.6%answered 91% | 38 ms | 48 ms | $0.030$29.51estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 39.7% (95%: 36.9%–42.6%) Math 51.5% | — | 39.7%36.9%–42.6% | wy | 447 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 59.26chosen-answer Brier 0.2122 | 12.6%answered 92% | 45 ms | 51 ms | $0.033$33.20estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Accuracy 39.7% (95%: 36.9%–42.6%) Math 46.9% | — | 39.7%36.9%–42.6% | xy | 447 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.727 | 58.43chosen-answer Brier 0.2287 | 29.7%answered 80% | 66 ms | 91 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Accuracy 39.3% (95%: 36.5%–42.2%) Math 44.3% | — | 39.3%36.5%–42.2% | xy | 442 / 1,125 | No answers at this threshold | 0.720 | 58.97chosen-answer Brier 0.2135 | 11.3%answered 93% | 40 ms | 48 ms | $0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Accuracy 38.9% (95%: 36.1%–41.8%) Math 49.8% | — | 38.9%36.1%–41.8% | yz | 438 / 1,125 | No answers at this threshold | 0.659 | 57.70chosen-answer Brier 0.2353 | 14.4%answered 91% | 110 ms | 137 ms | $0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Accuracy 32.3% (95%: 29.6%–35.1%) Math 36.8% | — | 32.3%29.6%–35.1% | zA | 363 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.632 | 54.39chosen-answer Brier 0.2349 | 32.9%answered 75% | 34 ms | 42 ms | $0.002$2.26estimate · DGX Spark |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Accuracy 32.1% (95%: 29.4%–34.9%) Math 39.2% | — | 32.1%29.4%–34.9% | zA | 361 / 1,125 | No answers at this threshold | 0.660 | 56.30chosen-answer Brier 0.1950 | 1.5%answered 99% | 123 ms | 152 ms | $0.091$90.99estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Accuracy 31.7% (95%: 29.1%–34.5%) Math 35.3% | — | 31.7%29.1%–34.5% | A | 357 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.601 | 55.15chosen-answer Brier 0.2143 | 15.7%answered 89% | 59 ms | 70 ms | $0.004$3.78estimate · DGX Spark |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Accuracy 30.8% (95%: 28.1%–33.5%) Math 38.7% | — | 30.8%28.1%–33.5% | AB | 346 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.640 | 55.51chosen-answer Brier 0.1975 | 31.4%answered 76% | 40 ms | 45 ms | $0.003$2.52estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Accuracy 30.7% (95%: 28.0%–33.4%) Math 37.1% | — | 30.7%28.0%–33.4% | AB | 345 / 1,125 | 3 right of 3 kept0 mistakes · 1,122 skipped | 0.620 | 55.05chosen-answer Brier 0.2058 | 86.5%answered 22% | 17 ms | 23 ms | $0.015$14.76estimate · RTX PRO 6000 Blackwell |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Accuracy 29.2% (95%: 26.6%–31.9%) Math 32.1% | — | 29.2%26.6%–31.9% | ABC | 328 / 1,125 | 9 right of 9 kept0 mistakes · 1,116 skipped | 0.617 | 53.53chosen-answer Brier 0.2210 | 26.1%answered 80% | 23 ms | 29 ms | $0.002$1.58estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Accuracy 27.0% (95%: 24.5%–29.7%) Math 28.2% | — | 27.0%24.5%–29.7% | ABC | 304 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.545 | 52.72chosen-answer Brier 0.2159 | 4.6%answered 97% | 16 ms | 19 ms | $0.001$1.16estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 26.7% (95%: 24.2%–29.3%) Math 28.2% | — | 26.7%24.2%–29.3% | ABC | 300 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.573 | 53.45chosen-answer Brier 0.1976 | 1.2%answered 99% | 31 ms | 39 ms | $0.002$2.00estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 24.7% (95%: 22.3%–27.3%) Math 29.7% | — | 24.7%22.3%–27.3% | BC | 278 / 1,125 | No answers at this threshold | 0.641 | 53.22chosen-answer Brier 0.1827 | 21.4%answered 83% | 27 ms | 31 ms | $0.002$1.79estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Accuracy 23.6% (95%: 21.2%–26.1%) Math 22.7% | — | 23.6%21.2%–26.1% | BC | 265 / 1,125 | No answers at this threshold | 0.527 | 51.99chosen-answer Brier 0.1958 | 12.4%answered 90% | 15 ms | 28 ms | $0.001$1.31estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Accuracy 23.2% (95%: 20.8%–25.8%) Math 20.6% | — | 23.2%20.8%–25.8% | C | 261 / 1,125 | No answers at this threshold | 0.556 | 52.19chosen-answer Brier 0.1883 | 76.1%answered 30% | 13 ms | 16 ms | $0.001$0.95estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Accuracy 22.4% (95%: 20.1%–24.9%) Math 28.7% | — | 22.4%20.1%–24.9% | C | 252 / 1,125 | No answers at this threshold | 0.568 | 51.86chosen-answer Brier 0.1868 | 8.3%answered 94% | 36 ms | 43 ms | $0.028$27.67estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
StartLux Perplexity scar-ai PelaAI quincyqiang victorbennett Fastino Metask Lab Liquid AI OmniJev Other Strands Agents Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B | — | 32.4%170/524 | 50.7%238/469 | 81.8%108/132 |
| Perplexity Decider Flash v1.3 Open · 4.0B | — | 30.7%161/524 | 50.1%235/469 | 81.1%107/132 |
| vera-medium-omni Open · 4.2B | — | 31.9%167/524 | 48.6%228/469 | 76.5%101/132 |
| Perplexity Decider Flash v1.2 Open · 4.0B | — | 31.3%164/524 | 46.1%216/469 | 79.5%105/132 |
| KnowLine 4B Gen5.5 Open · 4.0B | — | 30.2%158/524 | 47.5%223/469 | 72.0%95/132 |
| Chinese Decision Pointer 4B Open · 4.0B | — | 29.0%152/524 | 43.9%206/469 | 72.7%96/132 |
| Verdin-4B-v1.1 Open · 4.0B | — | 28.4%149/524 | 42.4%199/469 | 75.0%99/132 |
| GLiDE 2B Open · 1.9B | — | 26.5%139/524 | 33.0%155/469 | 52.3%69/132 |
| Metask-Jev-4B Open · 4.5B | — | 24.6%129/524 | 34.5%162/469 | 53.0%70/132 |
| KnowLine-2B-Gen1 Open · 2.0B | — | 25.0%131/524 | 32.2%151/469 | 56.8%75/132 |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B | — | 25.8%135/524 | 29.9%140/469 | 53.8%71/132 |
| d1-2B Open · 1.9B | — | 24.8%130/524 | 31.8%149/469 | 50.0%66/132 |
| GLiDE 0.8B Open · 0.8B | — | 25.6%134/524 | 29.0%136/469 | 43.9%58/132 |
| OneJev1.6-0.8B Open · 0.8B | — | 24.0%126/524 | 27.5%129/469 | 37.1%49/132 |
| KnowLine-0.8B-Gen1 Open · 0.8B | — | 25.2%132/524 | 24.3%114/469 | 40.9%54/132 |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B | — | 19.5%102/524 | 25.6%120/469 | 42.4%56/132 |
| GLiClass v4 Edge Open · 0.4B | — | 23.9%125/524 | 22.0%103/469 | 28.0%37/132 |
| GLiDE 0.4B Open · 0.4B | — | 20.8%109/524 | 24.1%113/469 | 29.5%39/132 |
| Strands Decider 2B Open · 2.0B | — | 19.3%101/524 | 24.1%113/469 | 28.8%38/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Accuracy 45.9% (95%: 43.0%–48.8%) Math 55.8% | — | 45.9%43.0%–48.8% | a | 516 / 1,125 | 21 right of 21 kept0 mistakes · 1,104 skipped | 0.720 | 62.63chosen-answer Brier 0.2061 | 16.0%answered 91% | 73 ms | 84 ms | $0.005$4.62estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Accuracy 44.7% (95%: 41.8%–47.6%) Math 55.9% | — | 44.7%41.8%–47.6% | ab | 503 / 1,125 | 17 right of 17 kept0 mistakes · 1,108 skipped | 0.757 | 62.14chosen-answer Brier 0.2042 | 24.4%answered 85% | 33 ms | 41 ms | $0.026$26.44estimate · RTX PRO 6000 Blackwell |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Accuracy 44.1% (95%: 41.2%–47.0%) Math 56.8% | — | 44.1%41.2%–47.0% | abc | 496 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.739 | 61.90chosen-answer Brier 0.2028 | 13.6%answered 92% | 32 ms | 37 ms | $0.026$25.82estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Accuracy 43.1% (95%: 40.2%–46.0%) Math 55.2% | — | 43.1%40.2%–46.0% | abc | 485 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 61.34chosen-answer Brier 0.2043 | 28.9%answered 82% | 45 ms | 55 ms | $0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Accuracy 42.3% (95%: 39.5%–45.2%) Math 56.0% | — | 42.3%39.5%–45.2% | abc | 476 / 1,125 | 17 right of 18 kept1 wrong · 1,107 skipped | 0.708 | 59.66chosen-answer Brier 0.2298 | 8.3%answered 95% | 96 ms | 115 ms | $0.006$5.89estimate · DGX Spark |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 40.4% (95%: 37.5%–43.3%) Math 50.4% | — | 40.4%37.5%–43.3% | bc | 454 / 1,125 | 39 right of 39 kept0 mistakes · 1,086 skipped | 0.724 | 58.90chosen-answer Brier 0.2256 | 14.6%answered 91% | 38 ms | 48 ms | $0.030$29.51estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Accuracy 39.7% (95%: 36.9%–42.6%) Math 51.5% | — | 39.7%36.9%–42.6% | c | 447 / 1,125 | 35 right of 35 kept0 mistakes · 1,090 skipped | 0.729 | 59.26chosen-answer Brier 0.2122 | 12.6%answered 92% | 45 ms | 51 ms | $0.033$33.20estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Accuracy 32.3% (95%: 29.6%–35.1%) Math 36.8% | — | 32.3%29.6%–35.1% | d | 363 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.632 | 54.39chosen-answer Brier 0.2349 | 32.9%answered 75% | 34 ms | 42 ms | $0.002$2.26estimate · DGX Spark |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Accuracy 32.1% (95%: 29.4%–34.9%) Math 39.2% | — | 32.1%29.4%–34.9% | d | 361 / 1,125 | No answers at this threshold | 0.660 | 56.30chosen-answer Brier 0.1950 | 1.5%answered 99% | 123 ms | 152 ms | $0.091$90.99estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Accuracy 31.7% (95%: 29.1%–34.5%) Math 35.3% | — | 31.7%29.1%–34.5% | d | 357 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.601 | 55.15chosen-answer Brier 0.2143 | 15.7%answered 89% | 59 ms | 70 ms | $0.004$3.78estimate · DGX Spark |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Accuracy 30.8% (95%: 28.1%–33.5%) Math 38.7% | — | 30.8%28.1%–33.5% | d | 346 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.640 | 55.51chosen-answer Brier 0.1975 | 31.4%answered 76% | 40 ms | 45 ms | $0.003$2.52estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Accuracy 30.7% (95%: 28.0%–33.4%) Math 37.1% | — | 30.7%28.0%–33.4% | d | 345 / 1,125 | 3 right of 3 kept0 mistakes · 1,122 skipped | 0.620 | 55.05chosen-answer Brier 0.2058 | 86.5%answered 22% | 17 ms | 23 ms | $0.015$14.76estimate · RTX PRO 6000 Blackwell |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Accuracy 29.2% (95%: 26.6%–31.9%) Math 32.1% | — | 29.2%26.6%–31.9% | de | 328 / 1,125 | 9 right of 9 kept0 mistakes · 1,116 skipped | 0.617 | 53.53chosen-answer Brier 0.2210 | 26.1%answered 80% | 23 ms | 29 ms | $0.002$1.58estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Accuracy 27.0% (95%: 24.5%–29.7%) Math 28.2% | — | 27.0%24.5%–29.7% | def | 304 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.545 | 52.72chosen-answer Brier 0.2159 | 4.6%answered 97% | 16 ms | 19 ms | $0.001$1.16estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 26.7% (95%: 24.2%–29.3%) Math 28.2% | — | 26.7%24.2%–29.3% | def | 300 / 1,125 | 2 right of 2 kept0 mistakes · 1,123 skipped | 0.573 | 53.45chosen-answer Brier 0.1976 | 1.2%answered 99% | 31 ms | 39 ms | $0.002$2.00estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Accuracy 24.7% (95%: 22.3%–27.3%) Math 29.7% | — | 24.7%22.3%–27.3% | ef | 278 / 1,125 | No answers at this threshold | 0.641 | 53.22chosen-answer Brier 0.1827 | 21.4%answered 83% | 27 ms | 31 ms | $0.002$1.79estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Accuracy 23.6% (95%: 21.2%–26.1%) Math 22.7% | — | 23.6%21.2%–26.1% | ef | 265 / 1,125 | No answers at this threshold | 0.527 | 51.99chosen-answer Brier 0.1958 | 12.4%answered 90% | 15 ms | 28 ms | $0.001$1.31estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Accuracy 23.2% (95%: 20.8%–25.8%) Math 20.6% | — | 23.2%20.8%–25.8% | f | 261 / 1,125 | No answers at this threshold | 0.556 | 52.19chosen-answer Brier 0.1883 | 76.1%answered 30% | 13 ms | 16 ms | $0.001$0.95estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Accuracy 22.4% (95%: 20.1%–24.9%) Math 28.7% | — | 22.4%20.1%–24.9% | f | 252 / 1,125 | No answers at this threshold | 0.568 | 51.86chosen-answer Brier 0.1868 | 8.3%answered 94% | 36 ms | 43 ms | $0.028$27.67estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
PelaAI tattrongvu Fastino StartLux newsbang Base Labs Patrick-18 Valen-Team Apertix AI EldanRing Blockbrain Labs Cloudflare Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B | — | 36.6%192/524 | 56.7%266/469 | 84.1%111/132 |
| Jeffan-9B-1211 Open · 9.0B | — | 34.2%179/524 | 53.7%252/469 | 84.1%111/132 |
| GLiDE 9B Open · 7.9B | — | 32.1%168/524 | 55.7%261/469 | 82.6%109/132 |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B | — | 32.3%169/524 | 52.5%246/469 | 82.6%109/132 |
| Adjudex 9B v5 Open · 9.0B | — | 34.0%178/524 | 49.9%234/469 | 83.3%110/132 |
| Elephant-1 Open · 9.0B | — | 33.2%174/524 | 51.0%239/469 | 82.6%109/132 |
| Dexy-Decision-9B-V2 Open · 9.0B | — | 34.9%183/524 | 47.3%222/469 | 79.5%105/132 |
| Valen-Decision-1.5-9B Open · 9.0B | — | 34.2%179/524 | 45.8%215/469 | 79.5%105/132 |
| KnowLine-9B-Gen1 Open · 9.0B | — | 32.4%170/524 | 47.5%223/469 | 79.5%105/132 |
| KnowLine-9B-Gen2 Open · 9.0B | — | 33.4%175/524 | 45.2%212/469 | 81.1%107/132 |
| Cerno 9B v1.2 Open · 9.0B | — | 31.1%163/524 | 48.0%225/469 | 75.0%99/132 |
| Winnow-12B Open · 12B | — | 26.5%139/524 | 45.4%213/469 | 72.0%95/132 |
| Cygnet Open · 12B | — | 26.9%141/524 | 47.3%222/469 | 59.8%79/132 |
| Clef-Flash Open · 9.0B | — | 30.0%157/524 | 40.1%188/469 | 70.5%93/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Accuracy 50.6% (95%: 47.7%–53.5%) Math 66.3% | — | 50.6%47.7%–53.5% | a | 569 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.774 | 65.90chosen-answer Brier 0.1878 | 8.8%answered 95% | 68 ms | 88 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 48.2% (95%: 45.3%–51.1%) Math 59.8% | — | 48.2%45.3%–51.1% | ab | 542 / 1,125 | 10 right of 10 kept0 mistakes · 1,115 skipped | 0.753 | 64.33chosen-answer Brier 0.1952 | 23.4%answered 87% | 52 ms | 66 ms | $0.039$39.35estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Accuracy 47.8% (95%: 44.9%–50.7%) Math 57.9% | — | 47.8%44.9%–50.7% | ab | 538 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.749 | 63.95chosen-answer Brier 0.1993 | 30.4%answered 82% | 208 ms | 251 ms | $0.013$13.29estimate · DGX Spark |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Accuracy 46.6% (95%: 43.7%–49.5%) Math 58.4% | — | 46.6%43.7%–49.5% | ab | 524 / 1,125 | 14 right of 14 kept0 mistakes · 1,111 skipped | 0.762 | 63.55chosen-answer Brier 0.1947 | 17.4%answered 90% | 160 ms | 175 ms | $0.010$10.04estimate · DGX Spark |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 46.4% (95%: 43.5%–49.3%) Math 62.1% | — | 46.4%43.5%–49.3% | ab | 522 / 1,125 | 54 right of 55 kept1 wrong · 1,070 skipped | 0.771 | 63.48chosen-answer Brier 0.1945 | 12.2%answered 93% | 41 ms | 46 ms | $0.031$31.36estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Accuracy 46.4% (95%: 43.5%–49.3%) Math 61.3% | — | 46.4%43.5%–49.3% | ab | 522 / 1,125 | 53 right of 53 kept0 mistakes · 1,072 skipped | 0.768 | 63.39chosen-answer Brier 0.1962 | 75.6%answered 47% | 79 ms | 98 ms | $0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Accuracy 45.3% (95%: 42.4%–48.3%) Math 58.8% | — | 45.3%42.4%–48.3% | ab | 510 / 1,125 | 56 right of 56 kept0 mistakes · 1,069 skipped | 0.759 | 61.81chosen-answer Brier 0.2171 | 12.4%answered 93% | 47 ms | 53 ms | $0.034$33.81estimate · RTX PRO 6000 Blackwell |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 44.4% (95%: 41.5%–47.3%) Math 55.3% | — | 44.4%41.5%–47.3% | bc | 499 / 1,125 | 35 right of 37 kept2 wrong · 1,088 skipped | 0.701 | 61.42chosen-answer Brier 0.2151 | 13.6%answered 92% | 60 ms | 67 ms | $0.042$42.42estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 44.3% (95%: 41.4%–47.2%) Math 59.8% | — | 44.3%41.4%–47.2% | bc | 498 / 1,125 | 30 right of 30 kept0 mistakes · 1,095 skipped | 0.725 | 61.97chosen-answer Brier 0.2032 | 6.7%answered 96% | 50 ms | 61 ms | $0.036$36.27estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Accuracy 43.9% (95%: 41.0%–46.8%) Math 60.1% | — | 43.9%41.0%–46.8% | bc | 494 / 1,125 | No answers at this threshold | 0.730 | 61.90chosen-answer Brier 0.2011 | 19.5%answered 88% | 52 ms | 65 ms | $0.040$39.96estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Accuracy 43.3% (95%: 40.4%–46.2%) Math 56.0% | — | 43.3%40.4%–46.2% | bc | 487 / 1,125 | 26 right of 27 kept1 wrong · 1,098 skipped | 0.752 | 60.41chosen-answer Brier 0.2247 | 6.0%answered 97% | 209 ms | 269 ms | $0.013$13.35estimate · DGX Spark |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Accuracy 39.7% (95%: 36.9%–42.6%) Math 46.9% | — | 39.7%36.9%–42.6% | c | 447 / 1,125 | 32 right of 32 kept0 mistakes · 1,093 skipped | 0.727 | 58.43chosen-answer Brier 0.2287 | 29.7%answered 80% | 66 ms | 91 ms | $0.052$51.64estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Accuracy 39.3% (95%: 36.5%–42.2%) Math 44.3% | — | 39.3%36.5%–42.2% | c | 442 / 1,125 | No answers at this threshold | 0.720 | 58.97chosen-answer Brier 0.2135 | 11.3%answered 93% | 40 ms | 48 ms | $0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Accuracy 38.9% (95%: 36.1%–41.8%) Math 49.8% | — | 38.9%36.1%–41.8% | c | 438 / 1,125 | No answers at this threshold | 0.659 | 57.70chosen-answer Brier 0.2353 | 14.4%answered 91% | 110 ms | 137 ms | $0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
MachinePulse PelaAI Poindexter Labs Fastino Sthānika AI Perplexity vLLM Semantic Router Base Labs Nace.AI Turion.AI Zefan Cai Cloudflare OmniJev Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| Karpo Decision 31B Open · 31B | — | 50.8%266/524 | 64.0%300/469 | 88.6%117/132 |
| KnowLine-31B-Gen1.5 Open · 31B | — | 47.1%247/524 | 64.2%301/469 | 87.9%116/132 |
| KnowLine-31B-Gen2 Open · 31B | — | 47.1%247/524 | 63.8%299/469 | 88.6%117/132 |
| Copernicus-31B Open · 31B | — | 47.1%247/524 | 63.8%299/469 | 86.4%114/132 |
| GLiDE 27B Open · 27B | — | 42.0%220/524 | 68.0%319/469 | 89.4%118/132 |
| Sieve 31B v3 Open · 31B | — | 45.8%240/524 | 62.9%295/469 | 87.9%116/132 |
| GLiDE v1.2 Open · 27B | — | 40.5%212/524 | 63.1%296/469 | 85.6%113/132 |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B | — | 41.4%217/524 | 59.9%281/469 | 89.4%118/132 |
| d3-ultra Open · 26B | — | 42.2%221/524 | 59.3%278/469 | 87.1%115/132 |
| Mammoth-1 Open · 27B | — | 42.7%224/524 | 58.6%275/469 | 86.4%114/132 |
| d3 Open · 26B | — | 40.6%213/524 | 60.6%284/469 | 87.9%116/132 |
| Perplexity Decider 27B v1.2 API Open · 27B | — | 39.9%209/524 | 58.4%274/469 | 88.6%117/132 |
| Drex Max Open · 27B | — | 42.4%222/524 | 55.7%261/469 | 86.4%114/132 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | with web | 39.1%205/524 | 58.0%272/469 | 86.4%114/132 |
| Turion Jev 31B Open · 31B | — | 36.6%192/524 | 60.1%282/469 | 86.4%114/132 |
| Open-Jev-27B-v1.1 Open · 27B | — | 31.1%163/524 | 47.1%221/469 | 72.0%95/132 |
| Drex 2.0 Open · 35B MoE · 3.0B active | — | 34.2%179/524 | 41.4%194/469 | 74.2%98/132 |
| Clef Open · 27B | — | 30.2%158/524 | 46.1%216/469 | 64.4%85/132 |
| OneJev-27B Open · 27B | — | 30.0%157/524 | 45.0%211/469 | 67.4%89/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Accuracy 60.7% (95%: 57.8%–63.5%) Math 81.6% | — | 60.7%57.8%–63.5% | a | 683 / 1,125 | 169 right of 172 kept3 wrong · 953 skipped | 0.816 | 71.80chosen-answer Brier 0.1711 | 6.6%answered 97% | 173 ms | 211 ms | $0.128$128estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Accuracy 59.0% (95%: 56.1%–61.9%) Math 77.4% | — | 59.0%56.1%–61.9% | ab | 664 / 1,125 | 89 right of 89 kept0 mistakes · 1,036 skipped | 0.804 | 70.72chosen-answer Brier 0.1757 | 0.4%answered 100% | 983 ms | 1,279 ms | $0.061$61.43estimate · DGX Spark |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Accuracy 58.9% (95%: 56.0%–61.8%) Math 80.3% | — | 58.9%56.0%–61.8% | ab | 663 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.825 | 71.09chosen-answer Brier 0.1675 | 0.0%answered 100% | 151 ms | 204 ms | $0.112$112estimate · RTX PRO 6000 Blackwell |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Accuracy 58.7% (95%: 55.8%–61.5%) Math 77.3% | — | 58.7%55.8%–61.5% | ab | 660 / 1,125 | 66 right of 66 kept0 mistakes · 1,059 skipped | 0.801 | 70.48chosen-answer Brier 0.1771 | 1.9%answered 99% | 185 ms | 255 ms | $0.136$136estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Accuracy 58.4% (95%: 55.5%–61.2%) Math 72.0% | — | 58.4%55.5%–61.2% | ab | 657 / 1,125 | 74 right of 75 kept1 wrong · 1,050 skipped | 0.822 | 70.67chosen-answer Brier 0.1707 | 44.3%answered 79% | 544 ms | 655 ms | $0.033$33.16estimate · DGX Spark |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Accuracy 57.9% (95%: 55.0%–60.7%) Math 77.1% | — | 57.9%55.0%–60.7% | abc | 651 / 1,125 | 79 right of 81 kept2 wrong · 1,044 skipped | 0.797 | 70.03chosen-answer Brier 0.1781 | 5.5%answered 98% | 1,050 ms | 1,335 ms | $0.063$62.95estimate · DGX Spark |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Accuracy 55.2% (95%: 52.3%–58.1%) Math 69.9% | — | 55.2%52.3%–58.1% | bcd | 621 / 1,125 | 45 right of 45 kept0 mistakes · 1,080 skipped | 0.793 | 68.39chosen-answer Brier 0.1842 | 50.2%answered 73% | 111 ms | 140 ms | $0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Accuracy 54.8% (95%: 51.8%–57.6%) Math 72.5% | — | 54.8%51.8%–57.6% | bcd | 616 / 1,125 | 62 right of 62 kept0 mistakes · 1,063 skipped | 0.803 | 67.98chosen-answer Brier 0.1881 | 18.1%answered 91% | 108 ms | 131 ms | $0.081$81.16estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 54.6% (95%: 51.7%–57.5%) Math 71.5% | — | 54.6%51.7%–57.5% | bcd | 614 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.778 | 66.46chosen-answer Brier 0.2166 | 10.1%answered 95% | 156 ms | 185 ms | $0.109$109estimate · RTX PRO 6000 Blackwell |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Accuracy 54.5% (95%: 51.6%–57.4%) Math 74.0% | — | 54.5%51.6%–57.4% | bcd | 613 / 1,125 | 106 right of 108 kept2 wrong · 1,017 skipped | 0.795 | 68.07chosen-answer Brier 0.1835 | 72.3%answered 58% | 221 ms | 244 ms | $0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Accuracy 54.5% (95%: 51.6%–57.4%) Math 70.4% | — | 54.5%51.6%–57.4% | bcd | 613 / 1,125 | 54 right of 54 kept0 mistakes · 1,071 skipped | 0.806 | 68.34chosen-answer Brier 0.1781 | 14.8%answered 93% | 76 ms | 98 ms | $0.057$57.18estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Accuracy 53.3% (95%: 50.4%–56.2%) Math 69.8% | — | 53.3%50.4%–56.2% | cd | 600 / 1,125 | 78 right of 80 kept2 wrong · 1,045 skipped | 0.798 | 67.64chosen-answer Brier 0.1805 | 14.4%answered 93% | 238 ms | 552 ms | $0.007$7.46 |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Accuracy 53.1% (95%: 50.1%–56.0%) Math 70.3% | — | 53.1%50.1%–56.0% | cd | 597 / 1,125 | 147 right of 155 kept8 wrong · 970 skipped | 0.786 | 65.68chosen-answer Brier 0.2170 | 7.3%answered 97% | 106 ms | 143 ms | $0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Accuracy 52.5% (95%: 49.6%–55.4%) Math 69.3% | with web | 52.5%49.6%–55.4% | d | 591 / 1,125 | 78 right of 79 kept1 wrong · 1,046 skipped | 0.800 | 67.24chosen-answer Brier 0.1806 | 37.7%answered 80% | 448 ms | 827 ms | $1.030$1,030 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Accuracy 52.3% (95%: 49.3%–55.2%) Math 69.0% | — | 52.3%49.3%–55.2% | d | 588 / 1,125 | 7 right of 7 kept0 mistakes · 1,118 skipped | 0.782 | 66.89chosen-answer Brier 0.1849 | 1.5%answered 99% | 385 ms | 533 ms | $0.025$24.91estimate · DGX Spark |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Accuracy 42.6% (95%: 39.7%–45.5%) Math 54.6% | — | 42.6%39.7%–45.5% | e | 479 / 1,125 | No answers at this threshold | 0.733 | 59.78chosen-answer Brier 0.2302 | 28.2%answered 82% | 413 ms | 576 ms | $0.294$294estimate · RTX PRO 6000 Blackwell |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Accuracy 41.9% (95%: 39.0%–44.8%) Math 59.0% | — | 41.9%39.0%–44.8% | e | 471 / 1,125 | 33 right of 33 kept0 mistakes · 1,092 skipped | 0.717 | 60.73chosen-answer Brier 0.2040 | 37.7%answered 75% | 345 ms | 504 ms | $0.018$18.41 |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Accuracy 40.8% (95%: 38.0%–43.7%) Math 51.1% | — | 40.8%38.0%–43.7% | e | 459 / 1,125 | 6 right of 6 kept0 mistakes · 1,119 skipped | 0.697 | 58.55chosen-answer Brier 0.2369 | 3.6%answered 98% | 196 ms | 222 ms | $0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Accuracy 40.6% (95%: 37.8%–43.5%) Math 53.8% | — | 40.6%37.8%–43.5% | e | 457 / 1,125 | 4 right of 4 kept0 mistakes · 1,121 skipped | 0.726 | 60.27chosen-answer Brier 0.2009 | 1.2%answered 99% | 108 ms | 136 ms | $0.086$86.07estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
REAX Logits Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| REAX 1.3 Open · 57B | — | 47.5%249/524 | 64.4%302/469 | 87.9%116/132 |
| Logits Sigma-1 Duo Open · 57B | — | 44.7%234/524 | 63.8%299/469 | 88.6%117/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Accuracy 59.3% (95%: 56.4%–62.1%) Math 78.0% | — | 59.3%56.4%–62.1% | a | 667 / 1,125 | 51 right of 51 kept0 mistakes · 1,074 skipped | 0.816 | 71.07chosen-answer Brier 0.1714 | 5.3%answered 98% | 215 ms | 278 ms | $0.160$160estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Accuracy 57.8% (95%: 54.9%–60.6%) Math 75.2% | — | 57.8%54.9%–60.6% | a | 650 / 1,125 | 8 right of 8 kept0 mistakes · 1,117 skipped | 0.796 | 69.90chosen-answer Brier 0.1798 | 7.1%answered 97% | 285 ms | 399 ms | $0.211$211estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Liquid AI Nace.AI Fastino Subquadratic Google TypeSafe Celeris Inception Microsoft Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| d1-ultra Closed · Efficient | — | 44.7%234/524 | 57.4%269/469 | 87.1%115/132 |
| Drex Max v1.1 Closed · Efficient (provisional) | — | 41.4%217/524 | 60.8%285/469 | 86.4%114/132 |
| GLiDE API Closed · Efficient | — | 38.7%203/524 | 63.1%296/469 | 86.4%114/132 |
| SubQ Snap 1.1 Closed · Efficient | — | 37.2%195/524 | 60.3%283/469 | 84.1%111/132 |
| Gemini Decision 1 20261111 Closed · Efficient | — | 39.3%206/524 | 53.1%249/469 | 81.8%108/132 |
| Jev 1.14 Closed · Efficient | — | 35.7%187/524 | 53.1%249/469 | 80.3%106/132 |
| celeris-1-decision Closed · Efficient | — | 37.2%195/524 | 50.5%237/469 | 75.8%100/132 |
| celeris-1.1-decision Closed · Efficient | — | 37.8%198/524 | 49.5%232/469 | 77.3%102/132 |
| Jev 1.13 Closed · Efficient | — | 33.2%174/524 | 52.0%244/469 | 78.0%103/132 |
| d1.1 Closed · Efficient | — | 34.7%182/524 | 50.7%238/469 | 74.2%98/132 |
| Mercury Decide Closed · Efficient | — | 33.4%175/524 | 51.6%242/469 | 68.2%90/132 |
| Microsoft-Decision-1 Closed · Efficient | — | 34.0%178/524 | 46.9%220/469 | 77.3%102/132 |
| d1 Closed · Efficient | — | 29.0%152/524 | 44.3%208/469 | 75.0%99/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 1,125 of 1,125); shown once.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Accuracy 54.9% (95%: 52.0%–57.8%) Math 74.1% | — | 54.9%52.0%–57.8% | a | 618 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.783 | 68.09chosen-answer Brier 0.1876 | 91.0%answered 35% | 182 ms | 425 ms | $0.013$12.64 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Accuracy 54.8% (95%: 51.8%–57.6%) Math 70.1% | — | 54.8%51.8%–57.6% | a | 616 / 1,125 | 89 right of 90 kept1 wrong · 1,035 skipped | 0.793 | 68.07chosen-answer Brier 0.1861 | 54.1%answered 69% | 329 ms | 1,540 ms | $0.022$22.37 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Accuracy 54.5% (95%: 51.6%–57.4%) Math 70.8% | — | 54.5%51.6%–57.4% | a | 613 / 1,125 | 52 right of 52 kept0 mistakes · 1,073 skipped | 0.814 | 68.27chosen-answer Brier 0.1794 | 60.0%answered 67% | 271 ms | 291 ms | $0.052$52.03 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Accuracy 52.4% (95%: 49.4%–55.3%) Math 68.6% | — | 52.4%49.4%–55.3% | ab | 589 / 1,125 | 70 right of 72 kept2 wrong · 1,053 skipped | 0.787 | 65.94chosen-answer Brier 0.2047 | 65.7%answered 61% | 384 ms | 393 ms | $0.015$14.92 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Accuracy 50.0% (95%: 47.1%–53.0%) Math 71.3% | — | 50.0%47.1%–53.0% | abc | 563 / 1,125 | 43 right of 43 kept0 mistakes · 1,082 skipped | 0.768 | 65.37chosen-answer Brier 0.1930 | 2.0%answered 99% | 228 ms | 344 ms | $0.037$36.77 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 48.2% (95%: 45.3%–51.1%) Math 65.1% | — | 48.2%45.3%–51.1% | bcd | 542 / 1,125 | 61 right of 61 kept0 mistakes · 1,064 skipped | 0.750 | 64.14chosen-answer Brier 0.1990 | 24.5%answered 86% | 166 ms | 193 ms | $0.026$25.64 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 47.3% (95%: 44.4%–50.2%) Math 70.0% | — | 47.3%44.4%–50.2% | bcd | 532 / 1,125 | No answers at this threshold | 0.735 | 62.94chosen-answer Brier 0.2141 | 92.3%answered 28% | 390 ms | 394 ms | $0.015$15.40 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Accuracy 47.3% (95%: 44.4%–50.2%) Math 69.9% | — | 47.3%44.4%–50.2% | bcd | 532 / 1,125 | No answers at this threshold | 0.733 | 62.92chosen-answer Brier 0.2144 | 91.0%answered 29% | 383 ms | 390 ms | $0.002$1.54 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Accuracy 46.3% (95%: 43.4%–49.2%) Math 64.1% | — | 46.3%43.4%–49.2% | cd | 521 / 1,125 | 65 right of 65 kept0 mistakes · 1,060 skipped | 0.764 | 63.36chosen-answer Brier 0.1959 | 15.8%answered 91% | 167 ms | 203 ms | $0.026$25.64 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 46.0% (95%: 43.2%–49.0%) Math 65.2% | — | 46.0%43.2%–49.0% | cd | 518 / 1,125 | 48 right of 48 kept0 mistakes · 1,077 skipped | 0.748 | 63.14chosen-answer Brier 0.1977 | 90.6%answered 28% | 176 ms | 272 ms | $0.006$6.02 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Accuracy 45.1% (95%: 42.2%–48.0%) Math 53.5% | — | 45.1%42.2%–48.0% | cde | 507 / 1,125 | 84 right of 91 kept7 wrong · 1,034 skipped | 0.746 | 61.18chosen-answer Brier 0.2272 | 20.5%answered 88% | 301 ms | 360 ms | $0.008$7.52 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Accuracy 44.4% (95%: 41.6%–47.4%) Math 59.4% | — | 44.4%41.6%–47.4% | de | 500 / 1,125 | 20 right of 20 kept0 mistakes · 1,105 skipped | 0.712 | 61.53chosen-answer Brier 0.2139 | 19.8%answered 88% | 315 ms | 362 ms | $0.013$13.42 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 40.8% (95%: 38.0%–43.7%) Math 55.6% | — | 40.8%38.0%–43.7% | e | 459 / 1,125 | 55 right of 56 kept1 wrong · 1,069 skipped | 0.723 | 59.76chosen-answer Brier 0.2128 | 11.3%answered 93% | 188 ms | 290 ms | $0.006$6.02 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
No workhorse decision-model setup has answers on Logic yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
No flagship decision-model setup has answers on Logic yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Liquid AI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed | — | 38.2%200/524 | 57.4%269/469 | 84.1%111/132 |
| d1.2 Closed · Undisclosed | — | 39.7%208/524 | 51.6%242/469 | 78.0%103/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 51.6% (95%: 48.6%–54.5%) Math 69.9% | — | 51.6%48.6%–54.5% | a | 580 / 1,125 | 13 right of 13 kept0 mistakes · 1,112 skipped | 0.800 | 66.77chosen-answer Brier 0.1802 | 94.5%answered 28% | 220 ms | 341 ms | $0.013$12.64 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Accuracy 49.2% (95%: 46.2%–52.1%) Math 65.5% | — | 49.2%46.2%–52.1% | a | 553 / 1,125 | 28 right of 28 kept0 mistakes · 1,097 skipped | 0.742 | 64.51chosen-answer Brier 0.2013 | 78.8%answered 43% | 222 ms | 390 ms | $0.006$6.02 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
OpenAI Google Alibaba Anthropic DeepSeek Thinking Machines Meta Xiaomi MiniMax Z.ai IBM OpenBMB Liquid AI Other KRAFTON eCloud Technology Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed | — | 98.3%515/524 | 95.5%448/469 | 94.7%125/132 |
| Gemini 4 Carbon chat Closed · Undisclosed | — | 98.1%514/524 | 95.7%449/469 | 94.7%125/132 |
| GPT-6.1 Sol chat Closed · Undisclosed | — | 97.9%513/524 | 95.5%448/469 | 94.7%125/132 |
| Qwen4-Max high · chat Closed · Undisclosed | — | 97.9%513/524 | 95.3%447/469 | 93.2%123/132 |
| Claude Opus 5.5 chat Closed · Flagship | — | 98.3%515/524 | 94.7%444/469 | 93.2%123/132 |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | 98.3%515/524 | 94.7%444/469 | 93.2%123/132 |
| Gemini 4 Hydrogen chat | — | 97.9%513/524 | 94.5%443/469 | 94.7%125/132 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | 98.1%514/524 | 93.6%439/469 | 93.2%123/132 |
| Claude Sonnet 5.5 chat Closed · Workhorse | — | 97.5%511/524 | 93.6%439/469 | 93.9%124/132 |
| Muse Ember max · chat Closed · Efficient | — | 97.9%513/524 | 93.2%437/469 | 93.9%124/132 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | 97.9%513/524 | 92.8%435/469 | 94.7%125/132 |
| Claude Fable 5.1 chat Closed · Flagship | — | 97.7%512/524 | 93.0%436/469 | 93.2%123/132 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | 97.3%510/524 | 92.8%435/469 | 94.7%125/132 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active | — | 98.1%514/524 | 92.5%434/469 | 92.4%122/132 |
| Qwen3.8-27B low · chat Open · 27B | — | 97.7%512/524 | 90.0%422/469 | 93.9%124/132 |
| GPT-6.1 Terra chat Closed · Undisclosed | — | 97.3%510/524 | 90.4%424/469 | 93.2%123/132 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | 95.8%502/524 | 92.5%434/469 | 91.7%121/132 |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | 97.1%509/524 | 90.8%426/469 | 92.4%122/132 |
| Gemma 4 31B chat Open · 31B | — | 97.1%509/524 | 89.3%419/469 | 92.4%122/132 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | 95.4%500/524 | 90.6%425/469 | 93.2%123/132 |
| Qwen3.5-4B on · chat Open · 4.7B | — | 96.4%505/524 | 88.1%413/469 | 91.7%121/132 |
| Granite-4.2-3B high · chat Open · 3.0B | — | 94.8%497/524 | 87.4%410/469 | 93.9%124/132 |
| Qwen3.5-4B off · chat Open · 4.7B | — | 95.8%502/524 | 83.6%392/469 | 90.9%120/132 |
| MiniCPM5-2B on · chat Open · 2.5B | — | 95.4%500/524 | 79.7%374/469 | 93.9%124/132 |
| Gemma 4 12B Unified chat Open · 12B | — | 92.4%484/524 | 81.0%380/469 | 93.2%123/132 |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | 93.3%489/524 | 77.0%361/469 | 94.7%125/132 |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | 92.4%484/524 | 77.4%363/469 | 91.7%121/132 |
| LFM3-1.7B off · chat Open · 1.7B | — | 90.1%472/524 | 78.7%369/469 | 91.7%121/132 |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B | — | 90.5%474/524 | 76.8%360/469 | 92.4%122/132 |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B | — | 90.3%473/524 | 77.2%362/469 | 91.7%121/132 |
| LFM3-1.7B on · chat Open · 1.7B | — | 88.2%462/524 | 78.3%367/469 | 91.7%121/132 |
| MiniCPM5-2B off · chat Open · 2.5B | — | 90.6%475/524 | 75.1%352/469 | 87.1%115/132 |
| Gemma 4 E4B chat Open · 8.0B | — | 89.3%468/524 | 73.1%343/469 | 94.7%125/132 |
| Gemma 4 E4B on · chat Open · 8.0B | — | 88.7%465/524 | 73.1%343/469 | 91.7%121/132 |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | 85.1%446/524 | 62.0%291/469 | 89.4%118/132 |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B | — | 84.5%443/524 | 62.9%295/469 | 87.9%116/132 |
| LFM2.5-2.6B on · chat Open · 2.7B | — | 83.2%436/524 | 63.8%299/469 | 85.6%113/132 |
| Gemma 4 E2B on · chat Open · 5.1B | — | 82.3%431/524 | 61.6%289/469 | 90.2%119/132 |
| Gemma 4 E2B chat Open · 5.1B | — | 79.6%417/524 | 61.0%286/469 | 88.6%117/132 |
| Raon-1 nano on · chat Open · 3.1B | — | 82.4%432/524 | 53.3%250/469 | 87.9%116/132 |
| Qwen3.5-2B on · chat Open · 2.0B | — | 77.3%405/524 | 58.4%274/469 | 84.8%112/132 |
| Qwen3-14B off · chat Open · 15B | — | 74.6%391/524 | 48.2%226/469 | 89.4%118/132 |
| Raon-1 nano off · chat Open · 3.1B | — | 70.8%371/524 | 53.9%253/469 | 81.8%108/132 |
| Qwen3.5-2B off · chat Open · 2.0B | — | 72.5%380/524 | 51.6%242/469 | 81.1%107/132 |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 75.4%395/524 | 36.9%173/469 | 82.6%109/132 |
| GPT-6 Luna chat Closed · Efficient | — | 67.0%351/524 | 39.7%186/469 | 87.1%115/132 |
| Erk-14B off · chat Open · 15B | — | 63.7%334/524 | 35.6%167/469 | 82.6%109/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Not ranked (only complete runs are): Spark-X2.6-4B (high · chat) (not running now, 20 of 1,125 stored, last answer 13 Nov 02:18 UTC); Grok 4.7 (chat) (paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)); Claude Fable 5.2 (chat) (not running now, 147 of 1,125 stored, last answer 12 Nov 11:32 UTC); Granite-4.2-3B (off · chat) (still running, 259 of 1,125); GPT-6 Luna (max · chat) (paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)); DeepSeek-V4-Flash (chat) (sample: a random subset by design, 317 answers stored, not the full 1,125); Claude Haiku 5.5 (max · chat) (paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Accuracy 96.7% (95%: 95.5%–97.6%) Math 99.6% | — | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 56.4%answered 97% | 6,590 ms | 14,330 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Accuracy 96.7% (95%: 95.5%–97.6%) Math 99.6% | — | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 45.5%answered 98% | 44,520 ms | 206,660 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Accuracy 96.5% (95%: 95.3%–97.5%) Math 99.6% | — | 96.5%95.3%–97.5% | a | 1,086 / 1,125 | Not measured | Not measured | — | 63.2%answered 97% | 7,070 ms | 16,200 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Accuracy 96.3% (95%: 95.0%–97.2%) Math 99.8% | — | 96.3%95.0%–97.2% | ab | 1,083 / 1,125 | Not measured | Not measured | — | 0.0%answered 100% | 11,570 ms | 38,640 ms | $5.690$5,690 |
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 96.2% (95%: 94.9%–97.2%) Math 99.8% | — | 96.2%94.9%–97.2% | abc | 1,082 / 1,125 | Not measured | Not measured | — | 38.2%answered 98% | 5,830 ms | 12,240 ms | $19.540$19,540 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Accuracy 96.2% (95%: 94.9%–97.2%) Math 99.6% | — | 96.2%94.9%–97.2% | abc | 1,082 / 1,125 | Not measured | Not measured | — | 3.8%answered 100% | 6,680 ms | 40,040 ms | $1.647$1,647 |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Accuracy 96.1% (95%: 94.8%–97.1%) Math 99.6% | — | 96.1%94.8%–97.1% | abcd | 1,081 / 1,125 | Not measured | Not measured | — | 76.1%answered 95% | 8,210 ms | 24,090 ms | — |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Accuracy 95.6% (95%: 94.3%–96.7%) Math 99.5% | — | 95.6%94.3%–96.7% | abcde | 1,076 / 1,125 | Not measured | Not measured | — | 37.3%answered 98% | 13,180 ms | 144,250 ms | $4.121$4,121 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 95.5% (95%: 94.1%–96.5%) Math 99.7% | — | 95.5%94.1%–96.5% | abcdef | 1,074 / 1,125 | Not measured | Not measured | — | 41.3%answered 98% | 6,710 ms | 14,890 ms | $11.480$11,480 API list-price equivalent |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Accuracy 95.5% (95%: 94.1%–96.5%) Math 99.6% | — | 95.5%94.1%–96.5% | abcdef | 1,074 / 1,125 | Not measured | Not measured | — | 22.6%answered 99% | 3,110 ms | 14,020 ms | $1.820$1,820 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Accuracy 95.4% (95%: 94.0%–96.5%) Math 99.6% | — | 95.4%94.0%–96.5% | abcdef | 1,073 / 1,125 | Not measured | Not measured | — | 50.0%answered 97% | 13,290 ms | 78,510 ms | $5.489$5,489 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 95.2% (95%: 93.8%–96.3%) Math 99.7% | — | 95.2%93.8%–96.3% | abcdef | 1,071 / 1,125 | Not measured | Not measured | — | 8.3%answered 100% | 3,820 ms | 13,460 ms | $0.000$0.00 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Accuracy 95.1% (95%: 93.7%–96.2%) Math 99.4% | — | 95.1%93.7%–96.2% | abcdef | 1,070 / 1,125 | Not measured | Not measured | — | 23.3%answered 98% | 24,740 ms | 179,930 ms | $1.390$1,390 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active Qwen3.5-397B-A17B on · chat · Alibaba · chat model · reasoning on Parameters: 397B MoE · 17B active Accuracy 95.1% (95%: 93.7%–96.2%) Math running | — | 95.1%93.7%–96.2% | abcdef | 1,070 / 1,125 | Not measured | Not measured | — | 31.7%answered 98% | 84,650 ms | 167,520 ms | $13.426$13,426 |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Accuracy 94.0% (95%: 92.5%–95.3%) Math 99.2% | — | 94.0%92.5%–95.3% | bcdefg | 1,058 / 1,125 | Not measured | Not measured | — | 39.1%answered 97% | 18,710 ms | 101,030 ms | $4.884$4,884estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Accuracy 94.0% (95%: 92.4%–95.2%) Math 99.2% | — | 94.0%92.4%–95.2% | efg | 1,057 / 1,125 | Not measured | Not measured | — | 61.1%answered 95% | 5,640 ms | 10,330 ms | $0.000$0.00 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Accuracy 94.0% (95%: 92.4%–95.2%) Math 98.2% | — | 94.0%92.4%–95.2% | cdefg | 1,057 / 1,125 | Not measured | Not measured | — | 38.3%answered 97% | 5,610 ms | 37,430 ms | $0.803$803 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Accuracy 94.0% (95%: 92.4%–95.2%) Math 99.5% | — | 94.0%92.4%–95.2% | defg | 1,057 / 1,125 | Not measured | Not measured | — | 34.6%answered 98% | 17,550 ms | 73,220 ms | $1.376$1,376 |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Accuracy 93.3% (95%: 91.7%–94.6%) Math 99.4% | — | 93.3%91.7%–94.6% | efgh | 1,050 / 1,125 | Not measured | Not measured | — | 16.3%answered 99% | 18,290 ms | 39,790 ms | $1.244$1,244estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Accuracy 93.2% (95%: 91.5%–94.5%) Math 98.8% | — | 93.2%91.5%–94.5% | fgh | 1,048 / 1,125 | Not measured | Not measured | — | 4.2%answered 100% | 12,050 ms | 82,830 ms | $1.057$1,057 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Accuracy 92.4% (95%: 90.7%–93.8%) Math 99.4% | — | 92.4%90.7%–93.8% | gh | 1,039 / 1,125 | Not measured | Not measured | — | 36.1%answered 97% | 142,520 ms | 246,280 ms | $3.525$3,525estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Accuracy 91.6% (95%: 89.9%–93.1%) Math 99.2% | — | 91.6%89.9%–93.1% | ghi | 1,031 / 1,125 | Not measured | Not measured | — | 53.2%answered 94% | 221,350 ms | 1,229,550 ms | $10.893$10,893estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Accuracy 90.1% (95%: 88.3%–91.7%) Math 98.5% | — | 90.1%88.3%–91.7% | hij | 1,014 / 1,125 | Not measured | Not measured | — | 33.6%answered 96% | 45,380 ms | 127,510 ms | $1.420$1,420estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Accuracy 88.9% (mean of 3 seeds; min–max 88.3%–89.6%) Math 99.2% | — | 88.9%88.3%–89.6% min–max of 3 seeds | ijk | 998 / 1,125seed 0 run | Not measured | Not measured | — | 51.5%answered 92% | 101,100 ms | 333,427 ms | $3.335$3,335estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Accuracy 87.7% (95%: 85.7%–89.5%) Math 98.8% | — | 87.7%85.7%–89.5% | jkl | 987 / 1,125 | Not measured | Not measured | — | 15.5%answered 98% | 42,060 ms | 82,940 ms | $1.292$1,292estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Accuracy 86.7% (95%: 84.6%–88.5%) Math 99.0% | — | 86.7%84.6%–88.5% | jklm | 975 / 1,125 | Not measured | Not measured | — | 37.8%answered 95% | 228,980 ms | 776,930 ms | $7.562$7,562estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 85.9% (mean of 3 seeds; min–max 85.5%–86.1%) Math 98.5% | — | 85.9%85.5%–86.1% min–max of 3 seeds | klm | 968 / 1,125seed 0 run | Not measured | Not measured | — | 55.9%answered 90% | 31,567 ms | 83,467 ms | $0.993$993estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Accuracy 85.5% (95%: 83.3%–87.4%) Math 98.7% | — | 85.5%83.3%–87.4% | klm | 962 / 1,125 | Not measured | Not measured | — | 54.2%answered 90% | 163,870 ms | 656,750 ms | $0.487$487estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 85.0% (95%: 82.8%–86.9%) Math 98.5% | — | 85.0%82.8%–86.9% | klm | 956 / 1,125 | Not measured | Not measured | — | Not measured | 57,900 ms | 109,440 ms | $1.503$1,503estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Accuracy 84.6% (mean of 3 seeds; min–max 84.4%–85.0%) Math 98.9% | — | 84.6%84.4%–85.0% min–max of 3 seeds | klm | 956 / 1,125seed 0 run | Not measured | Not measured | — | Not measured | 27,950 ms | 107,497 ms | $1.040$1,040estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Accuracy 84.2% (mean of 3 seeds; min–max 83.9%–84.4%) Math 98.9% | — | 84.2%83.9%–84.4% min–max of 3 seeds | klm | 950 / 1,125seed 0 run | Not measured | Not measured | — | 43.6%answered 91% | 29,847 ms | 113,220 ms | $1.017$1,017estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Accuracy 83.6% (mean of 3 seeds; min–max 83.1%–83.9%) Math 98.2% | — | 83.6%83.1%–83.9% min–max of 3 seeds | lm | 942 / 1,125seed 0 run | Not measured | Not measured | — | 43.9%answered 88% | 46,453 ms | 111,607 ms | $1.327$1,327estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 83.2% (95%: 80.9%–85.3%) Math 98.1% | — | 83.2%80.9%–85.3% | m | 936 / 1,125 | Not measured | Not measured | — | 52.4%answered 88% | 32,690 ms | 64,070 ms | $0.842$842estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 82.6% (95%: 80.3%–84.7%) Math 98.2% | — | 82.6%80.3%–84.7% | m | 929 / 1,125 | Not measured | Not measured | — | 45.1%answered 91% | 284,080 ms | 539,980 ms | $0.650$650estimate · DGX Spark |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 76.8% (mean of 3 seeds; min–max 76.0%–77.6%) Math 97.7% | — | 76.8%76.0%–77.6% min–max of 3 seeds | n | 855 / 1,125seed 0 run | Not measured | Not measured | — | 32.6%answered 91% | 21,223 ms | 45,077 ms | $0.574$574estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 75.9% (95%: 73.3%–78.3%) Math 97.8% | — | 75.9%73.3%–78.3% | n | 854 / 1,125 | Not measured | Not measured | — | Not measured | 41,350 ms | 70,680 ms | $1.193$1,193estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Accuracy 75.4% (95%: 72.8%–77.8%) Math 94.0% | — | 75.4%72.8%–77.8% | n | 848 / 1,125 | Not measured | Not measured | — | 44.3%answered 86% | 81,330 ms | 224,000 ms | $2.480$2,480estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 74.6% (95%: 72.0%–77.0%) Math 96.9% | — | 74.6%72.0%–77.0% | n | 839 / 1,125 | Not measured | Not measured | — | 26.4%answered 93% | 173,220 ms | 297,090 ms | $0.363$363estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 72.9% (95%: 70.2%–75.4%) Math 96.8% | — | 72.9%70.2%–75.4% | n | 820 / 1,125 | Not measured | Not measured | — | 34.0%answered 90% | 21,390 ms | 35,160 ms | $0.513$513estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 70.9% (95%: 68.2%–73.5%) Math 97.0% | — | 70.9%68.2%–73.5% | no | 798 / 1,125 | Not measured | Not measured | — | 10.5%answered 95% | 493,230 ms† | 1,396,520 ms† | $15.330$15,330estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Accuracy 70.3% (95%: 67.6%–72.9%) Math 94.7% | — | 70.3%67.6%–72.9% | nop | 791 / 1,125 | Not measured | Not measured | — | 33.1%answered 91% | 186,330 ms | 272,210 ms | $4.330$4,330estimate · RTX PRO 6000 Blackwell |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Accuracy 65.3% (95%: 62.5%–68.1%) Math 95.8% | — | 65.3%62.5%–68.1% | opq | 735 / 1,125 | Not measured | Not measured | — | 16.1%answered 94% | 16,560 ms | 178,680 ms | $1.772$1,772estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 65.1% (95%: 62.2%–67.8%) Math 89.6% | — | 65.1%62.2%–67.8% | opq | 732 / 1,125 | Not measured | Not measured | — | 35.7%answered 87% | 276,720 ms† | 965,630 ms† | $9.270$9,270estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Accuracy 64.8% (95%: 62.0%–67.5%) Math 94.6% | — | 64.8%62.0%–67.5% | pq | 729 / 1,125 | Not measured | Not measured | — | 35.2%answered 86% | 71,290 ms | 113,470 ms | $1.605$1,605estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Accuracy 60.2% (95%: 57.3%–63.0%) Math 96.3% | — | 60.2%57.3%–63.0% | qr | 677 / 1,125 | Not measured | Not measured | — | 58.3%answered 70% | 138,140 ms† | 403,380 ms† | $4.664$4,664estimate · RTX PRO 6000 Blackwell† |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 58.0% (95%: 55.0%–60.8%) Math 93.7% | — | 58.0%55.0%–60.8% | r | 652 / 1,125 | Not measured | Not measured | — | 55.1%answered 72% | 3,860 ms | 5,820 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Accuracy 54.2% (95%: 51.3%–57.1%) Math 90.8% | — | 54.2%51.3%–57.1% | r | 610 / 1,125 | Not measured | Not measured | — | 30.7%answered 84% | 10,150 ms | 18,410 ms | $0.390$390estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Alibaba IBM OpenBMB Other Google Liquid AI KRAFTON Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B | — | 96.4%505/524 | 88.1%413/469 | 91.7%121/132 |
| Granite-4.2-3B high · chat Open · 3.0B | — | 94.8%497/524 | 87.4%410/469 | 93.9%124/132 |
| Qwen3.5-4B off · chat Open · 4.7B | — | 95.8%502/524 | 83.6%392/469 | 90.9%120/132 |
| MiniCPM5-2B on · chat Open · 2.5B | — | 95.4%500/524 | 79.7%374/469 | 93.9%124/132 |
| LFM3-1.7B off · chat Open · 1.7B | — | 90.1%472/524 | 78.7%369/469 | 91.7%121/132 |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B | — | 90.3%473/524 | 77.2%362/469 | 91.7%121/132 |
| LFM3-1.7B on · chat Open · 1.7B | — | 88.2%462/524 | 78.3%367/469 | 91.7%121/132 |
| MiniCPM5-2B off · chat Open · 2.5B | — | 90.6%475/524 | 75.1%352/469 | 87.1%115/132 |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B | — | 85.1%446/524 | 62.0%291/469 | 89.4%118/132 |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B | — | 84.5%443/524 | 62.9%295/469 | 87.9%116/132 |
| LFM2.5-2.6B on · chat Open · 2.7B | — | 83.2%436/524 | 63.8%299/469 | 85.6%113/132 |
| Gemma 4 E2B on · chat Open · 5.1B | — | 82.3%431/524 | 61.6%289/469 | 90.2%119/132 |
| Gemma 4 E2B chat Open · 5.1B | — | 79.6%417/524 | 61.0%286/469 | 88.6%117/132 |
| Raon-1 nano on · chat Open · 3.1B | — | 82.4%432/524 | 53.3%250/469 | 87.9%116/132 |
| Qwen3.5-2B on · chat Open · 2.0B | — | 77.3%405/524 | 58.4%274/469 | 84.8%112/132 |
| Raon-1 nano off · chat Open · 3.1B | — | 70.8%371/524 | 53.9%253/469 | 81.8%108/132 |
| Qwen3.5-2B off · chat Open · 2.0B | — | 72.5%380/524 | 51.6%242/469 | 81.1%107/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Spark-X2.6-4B (high · chat) (not running now, 20 of 1,125 stored, last answer 13 Nov 02:18 UTC); Granite-4.2-3B (off · chat) (still running, 259 of 1,125).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Accuracy 92.4% (95%: 90.7%–93.8%) Math 99.4% | — | 92.4%90.7%–93.8% | a | 1,039 / 1,125 | Not measured | Not measured | — | 36.1%answered 97% | 142,520 ms | 246,280 ms | $3.525$3,525estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Accuracy 91.6% (95%: 89.9%–93.1%) Math 99.2% | — | 91.6%89.9%–93.1% | a | 1,031 / 1,125 | Not measured | Not measured | — | 53.2%answered 94% | 221,350 ms | 1,229,550 ms | $10.893$10,893estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Accuracy 90.1% (95%: 88.3%–91.7%) Math 98.5% | — | 90.1%88.3%–91.7% | ab | 1,014 / 1,125 | Not measured | Not measured | — | 33.6%answered 96% | 45,380 ms | 127,510 ms | $1.420$1,420estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Accuracy 88.9% (mean of 3 seeds; min–max 88.3%–89.6%) Math 99.2% | — | 88.9%88.3%–89.6% min–max of 3 seeds | bc | 998 / 1,125seed 0 run | Not measured | Not measured | — | 51.5%answered 92% | 101,100 ms | 333,427 ms | $3.335$3,335estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Accuracy 85.5% (95%: 83.3%–87.4%) Math 98.7% | — | 85.5%83.3%–87.4% | cd | 962 / 1,125 | Not measured | Not measured | — | 54.2%answered 90% | 163,870 ms | 656,750 ms | $0.487$487estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Accuracy 84.6% (mean of 3 seeds; min–max 84.4%–85.0%) Math 98.9% | — | 84.6%84.4%–85.0% min–max of 3 seeds | d | 956 / 1,125seed 0 run | Not measured | Not measured | — | Not measured | 27,950 ms | 107,497 ms | $1.040$1,040estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Accuracy 84.2% (mean of 3 seeds; min–max 83.9%–84.4%) Math 98.9% | — | 84.2%83.9%–84.4% min–max of 3 seeds | d | 950 / 1,125seed 0 run | Not measured | Not measured | — | 43.6%answered 91% | 29,847 ms | 113,220 ms | $1.017$1,017estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Accuracy 83.6% (mean of 3 seeds; min–max 83.1%–83.9%) Math 98.2% | — | 83.6%83.1%–83.9% min–max of 3 seeds | d | 942 / 1,125seed 0 run | Not measured | Not measured | — | 43.9%answered 88% | 46,453 ms | 111,607 ms | $1.327$1,327estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 76.8% (mean of 3 seeds; min–max 76.0%–77.6%) Math 97.7% | — | 76.8%76.0%–77.6% min–max of 3 seeds | e | 855 / 1,125seed 0 run | Not measured | Not measured | — | 32.6%answered 91% | 21,223 ms | 45,077 ms | $0.574$574estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 75.9% (95%: 73.3%–78.3%) Math 97.8% | — | 75.9%73.3%–78.3% | e | 854 / 1,125 | Not measured | Not measured | — | Not measured | 41,350 ms | 70,680 ms | $1.193$1,193estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Accuracy 75.4% (95%: 72.8%–77.8%) Math 94.0% | — | 75.4%72.8%–77.8% | ef | 848 / 1,125 | Not measured | Not measured | — | 44.3%answered 86% | 81,330 ms | 224,000 ms | $2.480$2,480estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 74.6% (95%: 72.0%–77.0%) Math 96.9% | — | 74.6%72.0%–77.0% | efg | 839 / 1,125 | Not measured | Not measured | — | 26.4%answered 93% | 173,220 ms | 297,090 ms | $0.363$363estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Accuracy 72.9% (95%: 70.2%–75.4%) Math 96.8% | — | 72.9%70.2%–75.4% | efg | 820 / 1,125 | Not measured | Not measured | — | 34.0%answered 90% | 21,390 ms | 35,160 ms | $0.513$513estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B Raon-1 nano on · chat · KRAFTON · chat model Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 70.9% (95%: 68.2%–73.5%) Math 97.0% | — | 70.9%68.2%–73.5% | fg | 798 / 1,125 | Not measured | Not measured | — | 10.5%answered 95% | 493,230 ms† | 1,396,520 ms† | $15.330$15,330estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Accuracy 70.3% (95%: 67.6%–72.9%) Math 94.7% | — | 70.3%67.6%–72.9% | g | 791 / 1,125 | Not measured | Not measured | — | 33.1%answered 91% | 186,330 ms | 272,210 ms | $4.330$4,330estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B Raon-1 nano off · chat · KRAFTON · chat model · reasoning off Parameters: 3.1B dense · 1.5B effective Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation) Accuracy 65.1% (95%: 62.2%–67.8%) Math 89.6% | — | 65.1%62.2%–67.8% | h | 732 / 1,125 | Not measured | Not measured | — | 35.7%answered 87% | 276,720 ms† | 965,630 ms† | $9.270$9,270estimate · RTX PRO 6000 Blackwell† |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Accuracy 64.8% (95%: 62.0%–67.5%) Math 94.6% | — | 64.8%62.0%–67.5% | h | 729 / 1,125 | Not measured | Not measured | — | 35.2%answered 86% | 71,290 ms | 113,470 ms | $1.605$1,605estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Google Alibaba Liquid AI eCloud Technology Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B | — | 92.4%484/524 | 81.0%380/469 | 93.2%123/132 |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B | — | 92.4%484/524 | 77.4%363/469 | 91.7%121/132 |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B | — | 90.5%474/524 | 76.8%360/469 | 92.4%122/132 |
| Gemma 4 E4B chat Open · 8.0B | — | 89.3%468/524 | 73.1%343/469 | 94.7%125/132 |
| Gemma 4 E4B on · chat Open · 8.0B | — | 88.7%465/524 | 73.1%343/469 | 91.7%121/132 |
| Qwen3-14B off · chat Open · 15B | — | 74.6%391/524 | 48.2%226/469 | 89.4%118/132 |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 75.4%395/524 | 36.9%173/469 | 82.6%109/132 |
| Erk-14B off · chat Open · 15B | — | 63.7%334/524 | 35.6%167/469 | 82.6%109/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Accuracy 87.7% (95%: 85.7%–89.5%) Math 98.8% | — | 87.7%85.7%–89.5% | a | 987 / 1,125 | Not measured | Not measured | — | 15.5%answered 98% | 42,060 ms | 82,940 ms | $1.292$1,292estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 85.9% (mean of 3 seeds; min–max 85.5%–86.1%) Math 98.5% | — | 85.9%85.5%–86.1% min–max of 3 seeds | ab | 968 / 1,125seed 0 run | Not measured | Not measured | — | 55.9%answered 90% | 31,567 ms | 83,467 ms | $0.993$993estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 85.0% (95%: 82.8%–86.9%) Math 98.5% | — | 85.0%82.8%–86.9% | abc | 956 / 1,125 | Not measured | Not measured | — | Not measured | 57,900 ms | 109,440 ms | $1.503$1,503estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 83.2% (95%: 80.9%–85.3%) Math 98.1% | — | 83.2%80.9%–85.3% | bc | 936 / 1,125 | Not measured | Not measured | — | 52.4%answered 88% | 32,690 ms | 64,070 ms | $0.842$842estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Accuracy 82.6% (95%: 80.3%–84.7%) Math 98.2% | — | 82.6%80.3%–84.7% | c | 929 / 1,125 | Not measured | Not measured | — | 45.1%answered 91% | 284,080 ms | 539,980 ms | $0.650$650estimate · DGX Spark |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Accuracy 65.3% (95%: 62.5%–68.1%) Math 95.8% | — | 65.3%62.5%–68.1% | d | 735 / 1,125 | Not measured | Not measured | — | 16.1%answered 94% | 16,560 ms | 178,680 ms | $1.772$1,772estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active LFM2.5-8B-A1B on · chat · Liquid AI · chat model Parameters: 8.3B MoE · 1.5B active Accuracy 60.2% (95%: 57.3%–63.0%) Math 96.3% | — | 60.2%57.3%–63.0% | e | 677 / 1,125 | Not measured | Not measured | — | 58.3%answered 70% | 138,140 ms† | 403,380 ms† | $4.664$4,664estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Accuracy 54.2% (95%: 51.3%–57.1%) Math 90.8% | — | 54.2%51.3%–57.1% | f | 610 / 1,125 | Not measured | Not measured | — | 30.7%answered 84% | 10,150 ms | 18,410 ms | $0.390$390estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Alibaba Google Liquid AI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B | — | 97.7%512/524 | 90.0%422/469 | 93.9%124/132 |
| Gemma 4 31B chat Open · 31B | — | 97.1%509/524 | 89.3%419/469 | 92.4%122/132 |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active | — | 93.3%489/524 | 77.0%361/469 | 94.7%125/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Accuracy 94.0% (95%: 92.5%–95.3%) Math 99.2% | — | 94.0%92.5%–95.3% | a | 1,058 / 1,125 | Not measured | Not measured | — | 39.1%answered 97% | 18,710 ms | 101,030 ms | $4.884$4,884estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Accuracy 93.3% (95%: 91.7%–94.6%) Math 99.4% | — | 93.3%91.7%–94.6% | a | 1,050 / 1,125 | Not measured | Not measured | — | 16.3%answered 99% | 18,290 ms | 39,790 ms | $1.244$1,244estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Accuracy 86.7% (95%: 84.6%–88.5%) Math 99.0% | — | 86.7%84.6%–88.5% | b | 975 / 1,125 | Not measured | Not measured | — | 37.8%answered 95% | 228,980 ms | 776,930 ms | $7.562$7,562estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
DeepSeek Thinking Machines Xiaomi Alibaba MiniMax Z.ai Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active | — | 98.3%515/524 | 94.7%444/469 | 93.2%123/132 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active | — | 98.1%514/524 | 93.6%439/469 | 93.2%123/132 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active | — | 97.9%513/524 | 92.8%435/469 | 94.7%125/132 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active | — | 97.3%510/524 | 92.8%435/469 | 94.7%125/132 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active | — | 98.1%514/524 | 92.5%434/469 | 92.4%122/132 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active | — | 95.8%502/524 | 92.5%434/469 | 91.7%121/132 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active | — | 95.4%500/524 | 90.6%425/469 | 93.2%123/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): DeepSeek-V4-Flash (chat) (sample: a random subset by design, 317 answers stored, not the full 1,125).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Accuracy 96.2% (95%: 94.9%–97.2%) Math 99.6% | — | 96.2%94.9%–97.2% | a | 1,082 / 1,125 | Not measured | Not measured | — | 3.8%answered 100% | 6,680 ms | 40,040 ms | $1.647$1,647 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Accuracy 95.6% (95%: 94.3%–96.7%) Math 99.5% | — | 95.6%94.3%–96.7% | ab | 1,076 / 1,125 | Not measured | Not measured | — | 37.3%answered 98% | 13,180 ms | 144,250 ms | $4.121$4,121 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Accuracy 95.4% (95%: 94.0%–96.5%) Math 99.6% | — | 95.4%94.0%–96.5% | ab | 1,073 / 1,125 | Not measured | Not measured | — | 50.0%answered 97% | 13,290 ms | 78,510 ms | $5.489$5,489 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Accuracy 95.1% (95%: 93.7%–96.2%) Math 99.4% | — | 95.1%93.7%–96.2% | abc | 1,070 / 1,125 | Not measured | Not measured | — | 23.3%answered 98% | 24,740 ms | 179,930 ms | $1.390$1,390 |
| Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active Qwen3.5-397B-A17B on · chat · Alibaba · chat model · reasoning on Parameters: 397B MoE · 17B active Accuracy 95.1% (95%: 93.7%–96.2%) Math running | — | 95.1%93.7%–96.2% | abc | 1,070 / 1,125 | Not measured | Not measured | — | 31.7%answered 98% | 84,650 ms | 167,520 ms | $13.426$13,426 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Accuracy 94.0% (95%: 92.4%–95.2%) Math 98.2% | — | 94.0%92.4%–95.2% | bc | 1,057 / 1,125 | Not measured | Not measured | — | 38.3%answered 97% | 5,610 ms | 37,430 ms | $0.803$803 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Accuracy 93.2% (95%: 91.5%–94.5%) Math 98.8% | — | 93.2%91.5%–94.5% | c | 1,048 / 1,125 | Not measured | Not measured | — | 4.2%answered 100% | 12,050 ms | 82,830 ms | $1.057$1,057 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Meta OpenAI Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient | — | 97.9%513/524 | 93.2%437/469 | 93.9%124/132 |
| GPT-6 Luna chat Closed · Efficient | — | 67.0%351/524 | 39.7%186/469 | 87.1%115/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)); Claude Haiku 5.5 (max · chat) (paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Accuracy 95.5% (95%: 94.1%–96.5%) Math 99.6% | — | 95.5%94.1%–96.5% | a | 1,074 / 1,125 | Not measured | Not measured | — | 22.6%answered 99% | 3,110 ms | 14,020 ms | $1.820$1,820 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Accuracy 58.0% (95%: 55.0%–60.8%) Math 93.7% | — | 58.0%55.0%–60.8% | b | 652 / 1,125 | Not measured | Not measured | — | 55.1%answered 72% | 3,860 ms | 5,820 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse | — | 97.5%511/524 | 93.6%439/469 | 93.9%124/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 95.5% (95%: 94.1%–96.5%) Math 99.7% | — | 95.5%94.1%–96.5% | a | 1,074 / 1,125 | Not measured | Not measured | — | 41.3%answered 98% | 6,710 ms | 14,890 ms | $11.480$11,480 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
Anthropic Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship | — | 98.3%515/524 | 94.7%444/469 | 93.2%123/132 |
| Claude Fable 5.1 chat Closed · Flagship | — | 97.7%512/524 | 93.0%436/469 | 93.2%123/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 147 of 1,125 stored, last answer 12 Nov 11:32 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 96.2% (95%: 94.9%–97.2%) Math 99.8% | — | 96.2%94.9%–97.2% | a | 1,082 / 1,125 | Not measured | Not measured | — | 38.2%answered 98% | 5,830 ms | 12,240 ms | $19.540$19,540 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Accuracy 95.2% (95%: 93.8%–96.3%) Math 99.7% | — | 95.2%93.8%–96.3% | b | 1,071 / 1,125 | Not measured | Not measured | — | 8.3%answered 100% | 3,820 ms | 13,460 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
Accuracy
Share of the questions answered right, with the Wilson 95% interval and the tie-group letters · higher is better; the line is chance (20%)
OpenAI Google Alibaba Chance, 20% (five options)Bars run to each accuracy; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful.
Accuracy by topic
| Setup | Web | Numerical reasoning524 questions | Verbal reasoning469 questions | Formal logic132 questions |
|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed | — | 98.3%515/524 | 95.5%448/469 | 94.7%125/132 |
| Gemini 4 Carbon chat Closed · Undisclosed | — | 98.1%514/524 | 95.7%449/469 | 94.7%125/132 |
| GPT-6.1 Sol chat Closed · Undisclosed | — | 97.9%513/524 | 95.5%448/469 | 94.7%125/132 |
| Qwen4-Max high · chat Closed · Undisclosed | — | 97.9%513/524 | 95.3%447/469 | 93.2%123/132 |
| GPT-6.1 Terra chat Closed · Undisclosed | — | 97.3%510/524 | 90.4%424/469 | 93.2%123/132 |
| Qwen3.8 Flash off · chat Closed · Undisclosed | — | 97.1%509/524 | 90.8%426/469 | 92.4%122/132 |
Share right of the questions each setup answered in each topic; under it right / answered.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Grok 4.7 (chat) (paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 1,125) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Decision Score (AA formula)0–100 · chosen-answer Brier | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Accuracy 96.7% (95%: 95.5%–97.6%) Math 99.6% | — | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 56.4%answered 97% | 6,590 ms | 14,330 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Accuracy 96.7% (95%: 95.5%–97.6%) Math 99.6% | — | 96.7%95.5%–97.6% | a | 1,088 / 1,125 | Not measured | Not measured | — | 45.5%answered 98% | 44,520 ms | 206,660 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Accuracy 96.5% (95%: 95.3%–97.5%) Math 99.6% | — | 96.5%95.3%–97.5% | a | 1,086 / 1,125 | Not measured | Not measured | — | 63.2%answered 97% | 7,070 ms | 16,200 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Accuracy 96.3% (95%: 95.0%–97.2%) Math 99.8% | — | 96.3%95.0%–97.2% | a | 1,083 / 1,125 | Not measured | Not measured | — | 0.0%answered 100% | 11,570 ms | 38,640 ms | $5.690$5,690 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Accuracy 94.0% (95%: 92.4%–95.2%) Math 99.2% | — | 94.0%92.4%–95.2% | b | 1,057 / 1,125 | Not measured | Not measured | — | 61.1%answered 95% | 5,640 ms | 10,330 ms | $0.000$0.00 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Accuracy 94.0% (95%: 92.4%–95.2%) Math 99.5% | — | 94.0%92.4%–95.2% | b | 1,057 / 1,125 | Not measured | Not measured | — | 34.6%answered 98% | 17,550 ms | 73,220 ms | $1.376$1,376 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.