Logic and Math · Latency
Latency on Logic and Math
Logic and Math pooled: 3,288 questions; only setups that finished both. How to read this page · Method
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents Blockbrain Labs newsbang tattrongvu Patrick-18 victorbennett EldanRing vLLM Semantic Router Valen-Team Base Labs Nace.AI Cloudflare Metask Lab Poindexter Labs MachinePulse Apertix AI TypeSafe REAX Logits Google Zefan Cai Inception Turion.AI Microsoft Celeris Subquadratic Sthānika AI Meta MiniMax DeepSeek Thinking Machines Anthropic OpenAI Z.ai Alibaba eCloud Technology OpenBMB Xiaomi IBM KRAFTON 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GLiDE 0.4B: 21.5% at a median 11 ms
GLiClass v4 Edge: 23.0% at a median 12 ms
OneJev1.6-0.8B: 27.8% at a median 15 ms
d1-2B: 34.9% at a median 17 ms
GLiDE 0.8B: 31.1% at a median 20 ms
StartLux-Decision-v1.1-0.8B-Instant: 28.0% at a median 24 ms
KnowLine-0.8B-Gen1: 27.7% at a median 26 ms
GLiDE 2B: 35.2% at a median 28 ms
Perplexity Decider Flash v1.3: 52.1% at a median 29 ms
Chinese Decision Pointer 4B: 47.0% at a median 30 ms
vera-medium-omni: 52.5% at a median 31 ms
Strands Decider 2B: 26.5% at a median 35 ms
StartLux-Decision-v1.1-2B-Instant: 35.9% at a median 36 ms
Cygnet: 42.6% at a median 38 ms
Adjudex 9B v5: 56.7% at a median 38 ms
KnowLine-9B-Gen1: 54.5% at a median 38 ms
Jeffan-9B-1211: 55.8% at a median 39 ms
Perplexity Decider Flash v1.2: 51.1% at a median 41 ms
KnowLine-9B-Gen2: 54.6% at a median 42 ms
Dexy-Decision-9B-V2: 54.2% at a median 46 ms
Verdin-4B-v1.1: 47.5% at a median 46 ms
KnowLine-12B-Gen1: 60.9% at a median 51 ms
KnowLine-2B-Gen1: 34.1% at a median 51 ms
Winnow-12B: 44.4% at a median 55 ms
d3: 65.0% at a median 57 ms
Valen-Decision-1.5-9B: 51.6% at a median 59 ms
StartLux-Decision-v1.1-4B-Instant: 52.4% at a median 68 ms
KnowLine 4B Gen5.5: 51.3% at a median 76 ms
Elephant-1: 56.2% at a median 80 ms
OneJev-27B: 49.3% at a median 80 ms
GLiDE (v1.2): 64.8% at a median 81 ms
Drex Max: 64.4% at a median 81 ms
Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): 66.4% at a median 97 ms
Clef-Flash: 46.1% at a median 103 ms
KnowLine-31B-Gen2: 73.0% at a median 116 ms
Metask-Jev-4B: 36.7% at a median 117 ms
Copernicus-31B: 71.0% at a median 128 ms
d3-ultra: 65.7% at a median 129 ms
StartLux-Decision-v1.1-9B-Instant: 54.4% at a median 142 ms
Mammoth-1: 67.3% at a median 150 ms
Karpo Decision 31B: 74.4% at a median 162 ms
Cerno 9B v1.2: 51.7% at a median 163 ms
Jev 1.14: 59.3% at a median 164 ms
Jev 1.13: 58.0% at a median 166 ms
d1.1: 58.6% at a median 174 ms
GLiDE 9B: 54.5% at a median 177 ms
d1-ultra: 67.5% at a median 180 ms
d1: 50.5% at a median 185 ms
Clef: 47.6% at a median 185 ms
REAX 1.3: 71.6% at a median 200 ms
Logits Sigma-1 Duo: 69.3% at a median 207 ms
d1.2: 59.9% at a median 208 ms
d1.2-ultra: 63.6% at a median 217 ms
Gemini Decision 1 (20261111): 64.1% at a median 227 ms
Perplexity Decider 27B v1.2 (API): 64.1% at a median 230 ms
Open-Jev-27B-v1.1: 50.5% at a median 250 ms
GLiDE (API): 65.2% at a median 266 ms
Mercury Decide: 50.6% at a median 293 ms
Turion Jev 31B: 63.3% at a median 301 ms
Microsoft-Decision-1: 54.3% at a median 312 ms
Drex Max v1.1: 64.8% at a median 327 ms
Drex 2.0: 53.1% at a median 339 ms
celeris-1.1-decision: 62.1% at a median 381 ms
SubQ Snap 1.1: 63.0% at a median 382 ms
celeris-1-decision: 62.2% at a median 388 ms
Perplexity Decider 27B v1.2 (API · with web): 63.6% at a median 430 ms
GLiDE 27B: 67.3% at a median 450 ms
Sieve 31B v3: 70.5% at a median 676 ms
KnowLine-31B-Gen1.5: 71.1% at a median 811 ms
Muse Ember (max · chat): 98.2% at a median 1,460 ms
MiniMax-M3.1-Flash (high · chat): 96.8% at a median 2,170 ms
DeepSeek-V4.1-Flash (high · chat · 1017): 98.4% at a median 2,270 ms
Inkling-Small (high · chat · Jul 2026): 98.2% at a median 2,750 ms
Inkling 2 Small (high · chat): 98.2% at a median 2,870 ms
Claude Opus 5.5 (chat): 98.5% at a median 3,600 ms
GPT-6 Luna (chat): 81.4% at a median 3,830 ms
Claude Sonnet 5.5 (chat): 98.2% at a median 4,030 ms
GPT-6.1 Terra (chat): 97.4% at a median 4,440 ms
GLM-5.3-Flash (high · chat): 96.9% at a median 4,640 ms
Gemini 4 Hydrogen (chat): 98.4% at a median 4,920 ms
Claude Fable 5.1 (chat): 98.2% at a median 4,940 ms
GPT-6.1 Sol (chat): 98.6% at a median 5,240 ms
GPT-6 Astra (chat): 98.6% at a median 5,580 ms
LFM3-1.7B (on · chat): 93.9% at a median 5,630 ms
LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell): 94.0% at a median 5,667 ms
Qwen4-Max (high · chat): 98.6% at a median 5,700 ms
Qwen3.8 Flash (off · chat): 97.6% at a median 6,780 ms
Erk-14B (off · chat): 78.3% at a median 8,740 ms
Gemma 4 E2B (on · chat · 32k cap · sampled): 90.5% at a median 8,873 ms
MiniCPM5-2B (off · chat): 93.2% at a median 9,065 ms
Qwen3-14B (off · chat): 85.4% at a median 9,380 ms
Gemma 4 E2B (chat): 88.6% at a median 9,935 ms
MiMo-V2.6-Flash (on · chat): 98.0% at a median 10,590 ms
Qwen3.8-27B (low · chat): 97.4% at a median 11,200 ms
Gemma 4 31B (chat): 97.3% at a median 12,575 ms
Gemma 4 E4B (chat): 93.0% at a median 13,150 ms
Gemma 4 E4B (on · chat · 32k cap · sampled): 94.2% at a median 13,945 ms
LFM2.5-2.6B (on · chat): 87.6% at a median 14,035 ms
Qwen3.5-4B (off · chat): 95.6% at a median 14,295 ms
Gemini 4 Carbon (chat): 98.6% at a median 16,350 ms
Qwen3.5-2B (off · chat): 84.4% at a median 17,805 ms
Gemma 4 E2B (on · chat · 32k cap): 90.3% at a median 19,100 ms
MiniCPM5-2B (on · chat): 95.7% at a median 19,260 ms
LFM2.5-8B-A1B (on · chat): 83.9% at a median 20,095 ms (mixed concurrency: This run changed the number of requests in flight part-way; see its record on Method)
Gemma 4 12B Unified (chat): 95.0% at a median 22,345 ms
Gemma 4 E4B (on · chat · 32k cap): 93.9% at a median 25,530 ms
Granite-4.2-3B (high · chat): 96.6% at a median 25,615 ms
LFM3-1.7B (off · chat): 94.2% at a median 29,040 ms
LFM2.5-24B-A2B (on · chat): 94.8% at a median 42,425 ms
Raon-1 nano (off · chat): 81.2% at a median 42,585 ms (other engine: Logic: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; Math: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Raon-1 nano (on · chat): 88.1% at a median 54,985 ms (other engine: Logic: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; Math: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Qwen3.5-4B (on · chat): 97.0% at a median 63,270 ms
Gemma 4 E2B (on · chat): 89.3% at a median 77,885 ms
Qwen3.5-2B (on · chat): 86.4% at a median 121,840 ms
Gemma 4 E4B (on · chat): 92.9% at a median 131,535 ms
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents Blockbrain Labs newsbang tattrongvu Patrick-18 victorbennett EldanRing vLLM Semantic Router Valen-Team Base Labs Nace.AI Cloudflare Metask Lab Poindexter Labs MachinePulse Apertix AI TypeSafe REAX Logits Google Zefan Cai Inception Turion.AI Microsoft Celeris Subquadratic Sthānika AI Meta MiniMax DeepSeek Thinking Machines Anthropic OpenAI Z.ai Alibaba eCloud Technology OpenBMB Xiaomi IBM KRAFTON
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 3,276 of 3,288); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 9 setups not yet: DeepSeek-V4-Flash (chat) (missing Logic); Qwen3.5-397B-A17B (on · chat) (missing Math); GPT-6 Luna (max · chat) (missing Logic and Math); Claude Haiku 5.5 (max · chat) (missing Logic and Math); Claude Fable 5.2 (chat) (missing Logic and Math); Grok 4.7 (chat) (missing Logic); Spark-X2.6-4B (high · chat) (missing Logic and Math); Granite-4.2-3B (off · chat) (missing Logic and Math); Gemini 4 Carbon (high · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,580 ms · p90 9,580 ms Logic 96.7% · Math 99.6% | — | Chat | 98.6%98.2%–99.0% | a | 3,243 / 3,288 | Not measured | Not measured | 60.9%answered 99% | 5,580 ms | 9,580 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Latency: median 16,350 ms · p90 90,630 ms Logic 96.7% · Math 99.6% | — | Chat | 98.6%98.1%–98.9% | a | 3,242 / 3,288 | Not measured | Not measured | 53.3%answered 99% | 16,350 ms | 90,630 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,240 ms · p90 10,410 ms Logic 96.5% · Math 99.6% | — | Chat | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | 66.2%answered 99% | 5,240 ms | 10,410 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Latency: median 5,700 ms · p90 19,750 ms Logic 96.3% · Math 99.8% | — | Chat | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | 10.7%answered 100% | 5,700 ms | 19,750 ms | $2.919$2,919 |
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,600 ms · p90 8,380 ms Logic 96.2% · Math 99.8% | — | Chat | 98.5%98.1%–98.9% | a | 3,240 / 3,288 | Not measured | Not measured | 41.3%answered 99% | 3,600 ms | 8,380 ms | $13.179$13,179 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Latency: median 2,270 ms · p90 14,870 ms Logic 96.2% · Math 99.6% | — | Chat | 98.4%98.0%–98.8% | ab | 3,237 / 3,288 | Not measured | Not measured | 6.3%answered 100% | 2,270 ms | 14,870 ms | $0.713$713 |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Latency: median 4,920 ms · p90 12,150 ms Logic 96.1% · Math 99.6% | — | Chat | 98.4%97.9%–98.8% | abc | 3,235 / 3,288 | Not measured | Not measured | 78.3%answered 98% | 4,920 ms | 12,150 ms | — |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 4,030 ms · p90 10,080 ms Logic 95.5% · Math 99.7% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,230 / 3,288 | Not measured | Not measured | 43.7%answered 99% | 4,030 ms | 10,080 ms | $7.042$7,042 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Latency: median 2,870 ms · p90 35,530 ms Logic 95.6% · Math 99.5% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,229 / 3,288 | Not measured | Not measured | 41.7%answered 99% | 2,870 ms | 35,530 ms | $1.830$1,830 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 4,940 ms · p90 8,620 ms Logic 95.2% · Math 99.7% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | 15.7%answered 100% | 4,940 ms | 8,620 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Latency: median 2,750 ms · p90 34,100 ms Logic 95.4% · Math 99.6% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | 50.6%answered 99% | 2,750 ms | 34,100 ms | $2.506$2,506 |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Latency: median 1,460 ms · p90 6,300 ms Logic 95.5% · Math 99.6% | — | Chat | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | 31.3%answered 99% | 1,460 ms | 6,300 ms | $0.863$863 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Latency: median 10,590 ms · p90 53,970 ms Logic 95.1% · Math 99.4% | — | Chat | 98.0%97.4%–98.4% | abcde | 3,221 / 3,288 | Not measured | Not measured | 27.6%answered 99% | 10,590 ms | 53,970 ms | $0.615$615 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Latency: median 6,780 ms · p90 36,290 ms Logic 94.0% · Math 99.5% | — | Chat | 97.6%97.0%–98.1% | bcdef | 3,209 / 3,288 | Not measured | Not measured | 31.3%answered 99% | 6,780 ms | 36,290 ms | $0.732$732 |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Latency: median 11,200 ms · p90 39,560 ms Logic 94.0% · Math 99.2% | — | Chat | 97.4%96.8%–97.9% | cdef | 3,204 / 3,288 | Not measured | Not measured | 33.9%answered 99% | 11,200 ms | 39,560 ms | $2.020$2,020estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,440 ms · p90 8,290 ms Logic 94.0% · Math 99.2% | — | Chat | 97.4%96.8%–97.9% | def | 3,203 / 3,288 | Not measured | Not measured | 65.0%answered 98% | 4,440 ms | 8,290 ms | $0.000$0.00 |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Latency: median 12,575 ms · p90 25,280 ms Logic 93.3% · Math 99.4% | — | Chat | 97.3%96.7%–97.8% | def | 3,200 / 3,288 | Not measured | Not measured | 17.0%answered 99% | 12,575 ms | 25,280 ms | $0.775$775estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Latency: median 63,270 ms · p90 193,950 ms Logic 92.4% · Math 99.4% | — | Chat | 97.0%96.4%–97.5% | ef | 3,190 / 3,288 | Not measured | Not measured | 34.2%answered 99% | 63,270 ms | 193,950 ms | $2.042$2,042estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Latency: median 4,640 ms · p90 25,900 ms Logic 93.2% · Math 98.8% | — | Chat | 96.9%96.2%–97.4% | efg | 3,185 / 3,288 | Not measured | Not measured | 7.4%answered 100% | 4,640 ms | 25,900 ms | $0.471$471 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Latency: median 2,170 ms · p90 15,760 ms Logic 94.0% · Math 98.2% | — | Chat | 96.8%96.1%–97.3% | fg | 3,182 / 3,288 | Not measured | Not measured | 38.9%answered 99% | 2,170 ms | 15,760 ms | $0.342$342 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Latency: median 25,615 ms · p90 523,100 ms Logic 91.6% · Math 99.2% | — | Chat | 96.6%96.0%–97.2% | fg | 3,177 / 3,288 | Not measured | Not measured | 54.7%answered 98% | 25,615 ms | 523,100 ms | $4.241$4,241estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Latency: median 19,260 ms · p90 194,917 ms Logic 88.9% · Math 99.2% | — | Chat | 95.7%95.4%–96.0% min–max of 3 seeds | gh | 3,145 / 3,288seed 0 run | Not measured | Not measured | 51.8%answered 97% | 19,260 ms | 194,917 ms | $1.410$1,410estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Latency: median 14,295 ms · p90 78,230 ms Logic 90.1% · Math 98.5% | — | Chat | 95.6%94.9%–96.3% | ghi | 3,144 / 3,288 | Not measured | Not measured | 34.0%answered 98% | 14,295 ms | 78,230 ms | $0.683$683estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Latency: median 22,345 ms · p90 53,390 ms Logic 87.7% · Math 98.8% | — | Chat | 95.0%94.2%–95.7% | hij | 3,125 / 3,288 | Not measured | Not measured | 15.3%answered 99% | 22,345 ms | 53,390 ms | $0.749$749estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Latency: median 42,425 ms · p90 432,510 ms Logic 86.7% · Math 99.0% | — | Chat | 94.8%94.0%–95.5% | hijk | 3,117 / 3,288 | Not measured | Not measured | 44.0%answered 97% | 42,425 ms | 432,510 ms | $3.205$3,205estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 13,945 ms · p90 51,987 ms Logic 85.9% · Math 98.5% | — | Chat | 94.2%94.0%–94.4% min–max of 3 seeds | hijkl | 3,104 / 3,288seed 0 run | Not measured | Not measured | 54.1%answered 96% | 13,945 ms | 51,987 ms | $0.549$549estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Latency: median 29,040 ms · p90 343,820 ms Logic 85.5% · Math 98.7% | — | Chat | 94.2%93.3%–94.9% | hijkl | 3,097 / 3,288 | Not measured | Not measured | 55.9%answered 96% | 29,040 ms | 343,820 ms | $0.206$206estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Latency: median 5,667 ms · p90 58,313 ms Logic 84.6% · Math 98.9% | — | Chat | 94.0%93.8%–94.2% min–max of 3 seeds | hijkl | 3,098 / 3,288seed 0 run | Not measured | Not measured | Not measured | 5,667 ms | 58,313 ms | $0.449$449estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Latency: median 5,630 ms · p90 63,067 ms Logic 84.2% · Math 98.9% | — | Chat | 93.9%93.89%–93.95% min–max of 3 seeds | ijkl | 3,089 / 3,288seed 0 run | Not measured | Not measured | 46.7%answered 96% | 5,630 ms | 63,067 ms | $0.435$435estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 25,530 ms · p90 77,380 ms Logic 85.0% · Math 98.5% | — | Chat | 93.9%93.0%–94.7% | jkl | 3,087 / 3,288 | Not measured | Not measured | Not measured | 25,530 ms | 77,380 ms | $0.855$855estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Latency: median 9,065 ms · p90 75,703 ms Logic 83.6% · Math 98.2% | — | Chat | 93.2%92.9%–93.3% min–max of 3 seeds | jkl | 3,068 / 3,288seed 0 run | Not measured | Not measured | 41.0%answered 95% | 9,065 ms | 75,703 ms | $0.598$598estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 13,150 ms · p90 44,990 ms Logic 83.2% · Math 98.1% | — | Chat | 93.0%92.1%–93.8% | kl | 3,058 / 3,288 | Not measured | Not measured | 51.0%answered 95% | 13,150 ms | 44,990 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 131,535 ms · p90 392,620 ms Logic 82.6% · Math 98.2% | — | Chat | 92.9%92.0%–93.7% | l | 3,054 / 3,288 | Not measured | Not measured | 43.7%answered 96% | 131,535 ms | 392,620 ms | $0.375$375estimate · DGX Spark |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 8,873 ms · p90 32,503 ms Logic 76.8% · Math 97.7% | — | Chat | 90.5%90.2%–90.9% min–max of 3 seeds | mn | 2,966 / 3,288seed 0 run | Not measured | Not measured | 31.0%answered 97% | 8,873 ms | 32,503 ms | $0.325$325estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 19,100 ms · p90 54,630 ms Logic 75.9% · Math 97.8% | — | Chat | 90.3%89.3%–91.3% | m | 2,970 / 3,288 | Not measured | Not measured | Not measured | 19,100 ms | 54,630 ms | $0.678$678estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 77,885 ms · p90 232,800 ms Logic 74.6% · Math 96.9% | — | Chat | 89.3%88.2%–90.3% | mno | 2,936 / 3,288 | Not measured | Not measured | 24.3%answered 97% | 77,885 ms | 232,800 ms | $0.215$215estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 9,935 ms · p90 27,690 ms Logic 72.9% · Math 96.8% | — | Chat | 88.6%87.5%–89.6% | mnop | 2,913 / 3,288 | Not measured | Not measured | 32.3%answered 96% | 9,935 ms | 27,690 ms | $0.313$313estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B | — | Chat | 88.1%86.9%–89.1% | nop | 2,896 / 3,288 | Not measured | Not measured | Not measured | 54,985 ms† | 987,660 ms† | $6.344$6,344estimate · RTX PRO 6000 Blackwell† |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Latency: median 14,035 ms · p90 142,970 ms Logic 75.4% · Math 94.0% | — | Chat | 87.6%86.5%–88.7% | opq | 2,881 / 3,288 | Not measured | Not measured | 40.6%answered 95% | 14,035 ms | 142,970 ms | $1.046$1,046estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Latency: median 121,840 ms · p90 241,590 ms Logic 70.3% · Math 94.7% | — | Chat | 86.4%85.2%–87.5% | pqr | 2,840 / 3,288 | Not measured | Not measured | 30.9%answered 97% | 121,840 ms | 241,590 ms | $3.106$3,106estimate · RTX PRO 6000 Blackwell |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Latency: median 9,380 ms · p90 35,110 ms Logic 65.3% · Math 95.8% | — | Chat | 85.4%84.1%–86.5% | qr | 2,807 / 3,288 | Not measured | Not measured | 20.9%answered 96% | 9,380 ms | 35,110 ms | $0.840$840estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Latency: median 17,805 ms · p90 92,100 ms Logic 64.8% · Math 94.6% | — | Chat | 84.4%83.1%–85.6% | r | 2,776 / 3,288 | Not measured | Not measured | 34.3%answered 94% | 17,805 ms | 92,100 ms | $0.788$788estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | Chat | 83.9%82.6%–85.1% | rs | 2,759 / 3,288 | Not measured | Not measured | 55.9%answered 88% | 20,095 ms† | 230,570 ms† | $1.913$1,913estimate · RTX PRO 6000 Blackwell† |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,830 ms · p90 5,300 ms Logic 58.0% · Math 93.7% | — | Chat | 81.4%80.1%–82.7% | s | 2,678 / 3,288 | Not measured | Not measured | Not measured | 3,830 ms | 5,300 ms | $0.000$0.00 |
| Raon-1 nano off · chat Open · 3.1B | — | Chat | 81.2%79.8%–82.5% | st | 2,669 / 3,288 | Not measured | Not measured | 31.1%answered 94% | 42,585 ms† | 593,950 ms† | $4.422$4,422estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Latency: median 8,740 ms · p90 16,420 ms Logic 54.2% · Math 90.8% | — | Chat | 78.3%76.8%–79.6% | t | 2,573 / 3,288 | Not measured | Not measured | 32.3%answered 92% | 8,740 ms | 16,420 ms | $0.308$308estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Latency: median 162 ms · p90 198 ms Logic 60.7% · Math 81.6% | — | Decision | 74.4%72.9%–75.9% | u | 2,447 / 3,288 | 1,006 right of 1,009 kept3 wrong · 2,279 skipped | 0.882 | 5.7%answered 98% | 162 ms | 198 ms | $0.118$118estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Latency: median 116 ms · p90 177 ms Logic 58.9% · Math 80.3% | — | Decision | 73.0%71.4%–74.5% | uv | 2,399 / 3,288 | 523 right of 524 kept1 wrong · 2,764 skipped | 0.882 | 0.1%answered 100% | 116 ms | 177 ms | $0.093$92.98estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Latency: median 200 ms · p90 260 ms Logic 59.3% · Math 78.0% | — | Decision | 71.6%70.0%–73.1% | v | 2,354 / 3,288 | 229 right of 229 kept0 mistakes · 3,059 skipped | 0.878 | 3.8%answered 99% | 200 ms | 260 ms | $0.148$148estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Latency: median 811 ms · p90 1,111 ms Logic 59.0% · Math 77.4% | — | Decision | 71.1%69.5%–72.6% | vw | 2,338 / 3,288 | 553 right of 554 kept1 wrong · 2,734 skipped | 0.870 | 0.7%answered 100% | 811 ms | 1,111 ms | $0.052$51.85estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Latency: median 128 ms · p90 218 ms Logic 58.7% · Math 77.3% | — | Decision | 71.0%69.4%–72.5% | vw | 2,333 / 3,288 | 384 right of 384 kept0 mistakes · 2,904 skipped | 0.873 | 1.8%answered 99% | 128 ms | 218 ms | $0.102$102estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Latency: median 676 ms · p90 1,173 ms Logic 57.9% · Math 77.1% | — | Decision | 70.5%68.9%–72.1% | vwx | 2,319 / 3,288 | 294 right of 296 kept2 wrong · 2,992 skipped | 0.865 | 3.8%answered 99% | 676 ms | 1,173 ms | $0.047$47.29estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Latency: median 207 ms · p90 324 ms Logic 57.8% · Math 75.2% | — | Decision | 69.3%67.7%–70.8% | wxy | 2,277 / 3,288 | 23 right of 23 kept0 mistakes · 3,265 skipped | 0.868 | 4.4%answered 99% | 207 ms | 324 ms | $0.165$165estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Latency: median 180 ms · p90 426 ms Logic 54.9% · Math 74.1% | — | Decision | 67.5%65.9%–69.1% | xyzA | 2,221 / 3,288 | 233 right of 233 kept0 mistakes · 3,055 skipped | 0.852 | 89.1%answered 52% | 180 ms | 426 ms | $0.008$8.37 |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Latency: median 450 ms · p90 587 ms Logic 58.4% · Math 72.0% | — | Decision | 67.3%65.7%–68.9% | xyz | 2,214 / 3,288 | 272 right of 274 kept2 wrong · 3,014 skipped | 0.852 | 31.9%answered 89% | 450 ms | 587 ms | $0.029$28.55estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Latency: median 150 ms · p90 232 ms Logic 54.5% · Math 74.0% | — | Decision | 67.3%65.7%–68.9% | yzA | 2,213 / 3,288 | 518 right of 520 kept2 wrong · 2,768 skipped | 0.868 | 71.4%answered 69% | 150 ms | 232 ms | $0.028$27.98 list$0.120$120estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Latency: median 97 ms · p90 125 ms Logic 54.8% · Math 72.5% | — | Decision | 66.4%64.8%–68.0% | zAB | 2,184 / 3,288 | 385 right of 385 kept0 mistakes · 2,903 skipped | 0.844 | 13.5%answered 95% | 97 ms | 125 ms | $0.071$71.10estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 129 ms · p90 168 ms Logic 54.6% · Math 71.5% | — | Decision | 65.7%64.1%–67.3% | zABC | 2,161 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.835 | 7.5%answered 97% | 129 ms | 168 ms | $0.096$95.71estimate · RTX PRO 6000 Blackwell |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Latency: median 266 ms · p90 284 ms Logic 54.5% · Math 70.8% | — | Decision | 65.2%63.6%–66.8% | zABC | 2,144 / 3,288 | 213 right of 213 kept0 mistakes · 3,075 skipped | 0.852 | 49.2%answered 80% | 266 ms | 284 ms | $0.036$36.42 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 57 ms · p90 86 ms Logic 54.5% · Math 70.4% | — | Decision | 65.0%63.3%–66.6% | zABC | 2,136 / 3,288 | 216 right of 217 kept1 wrong · 3,071 skipped | 0.858 | 9.3%answered 97% | 57 ms | 86 ms | $0.046$46.28estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Latency: median 327 ms · p90 518 ms Logic 54.8% · Math 70.1% | — | Decision | 64.8%63.2%–66.5% | zABC | 2,132 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.846 | 48.3%answered 79% | 327 ms | 518 ms | $0.016$16.13 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Latency: median 81 ms · p90 136 ms Logic 55.2% · Math 69.9% | — | Decision | 64.8%63.2%–66.5% | ABC | 2,132 / 3,288 | 176 right of 176 kept0 mistakes · 3,112 skipped | 0.843 | 42.3%answered 83% | 81 ms | 136 ms | $0.010$9.71 list$0.066$66.05estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Latency: median 81 ms · p90 130 ms Logic 53.1% · Math 70.3% | — | Decision | 64.4%62.8%–66.0% | zABC | 2,118 / 3,288 | 866 right of 884 kept18 wrong · 2,404 skipped | 0.841 | 5.7%answered 98% | 81 ms | 130 ms | $0.016$16.31 list$0.064$63.74estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Latency: median 230 ms · p90 439 ms Logic 53.3% · Math 69.8% | — | Decision | 64.1%62.5%–65.8% | BCD | 2,109 / 3,288 | 409 right of 411 kept2 wrong · 2,877 skipped | 0.849 | 10.3%answered 96% | 230 ms | 439 ms | $0.005$5.38 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Latency: median 227 ms · p90 347 ms Logic 50.0% · Math 71.3% | — | Decision | 64.1%62.4%–65.7% | zABCD | 2,106 / 3,288 | 257 right of 257 kept0 mistakes · 3,031 skipped | 0.838 | 3.0%answered 99% | 227 ms | 347 ms | $0.027$27.07 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 217 ms · p90 347 ms Logic 51.6% · Math 69.9% | — | Decision | 63.6%61.9%–65.2% | BCD | 2,091 / 3,288 | 128 right of 128 kept0 mistakes · 3,160 skipped | 0.856 | 90.2%answered 48% | 217 ms | 347 ms | $0.008$8.37 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Latency: median 430 ms · p90 707 ms Logic 52.5% · Math 69.3% | with web | Decision | 63.6%61.9%–65.2% | CD | 2,090 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.854 | 25.8%answered 90% | 430 ms | 707 ms | $1.025$1,025 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Latency: median 301 ms · p90 448 ms Logic 52.3% · Math 69.0% | — | Decision | 63.3%61.6%–64.9% | BCDE | 2,081 / 3,288 | 50 right of 50 kept0 mistakes · 3,238 skipped | 0.844 | 1.9%answered 99% | 301 ms | 448 ms | $0.020$20.01estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Latency: median 382 ms · p90 391 ms Logic 52.4% · Math 68.6% | — | Decision | 63.0%61.4%–64.7% | CDE | 2,072 / 3,288 | 430 right of 433 kept3 wrong · 2,855 skipped | 0.827 | 58.3%answered 73% | 382 ms | 391 ms | $0.011$10.75 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 388 ms · p90 392 ms Logic 47.3% · Math 70.0% | — | Decision | 62.2%60.6%–63.9% | CDEF | 2,046 / 3,288 | No answers at this threshold | 0.829 | 87.2%answered 50% | 388 ms | 392 ms | $0.011$11.25 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 381 ms · p90 388 ms Logic 47.3% · Math 69.9% | — | Decision | 62.1%60.5%–63.8% | CDEF | 2,043 / 3,288 | No answers at this threshold | 0.830 | 86.7%answered 51% | 381 ms | 388 ms | $0.001$1.13 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Latency: median 51 ms · p90 76 ms Logic 50.6% · Math 66.3% | — | Decision | 60.9%59.2%–62.5% | DEFG | 2,002 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.810 | 4.5%answered 98% | 51 ms | 76 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 208 ms · p90 390 ms Logic 49.2% · Math 65.5% | — | Decision | 59.9%58.2%–61.6% | EFGH | 1,970 / 3,288 | 169 right of 169 kept0 mistakes · 3,119 skipped | 0.812 | 66.9%answered 65% | 208 ms | 390 ms | $0.004$3.98 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 164 ms · p90 188 ms Logic 48.2% · Math 65.1% | — | Decision | 59.3%57.6%–61.0% | FGHI | 1,951 / 3,288 | 263 right of 263 kept0 mistakes · 3,025 skipped | 0.810 | 15.4%answered 93% | 164 ms | 188 ms | $0.021$21.27 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 174 ms · p90 285 ms Logic 46.0% · Math 65.2% | — | Decision | 58.6%56.9%–60.3% | FGHI | 1,928 / 3,288 | 231 right of 231 kept0 mistakes · 3,057 skipped | 0.811 | 86.0%answered 46% | 174 ms | 285 ms | $0.004$3.98 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 166 ms · p90 205 ms Logic 46.3% · Math 64.1% | — | Decision | 58.0%56.3%–59.7% | GHIJ | 1,907 / 3,288 | 313 right of 313 kept0 mistakes · 2,975 skipped | 0.817 | 10.2%answered 96% | 166 ms | 205 ms | $0.021$21.27 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 38 ms · p90 45 ms Logic 46.4% · Math 62.1% | — | Decision | 56.7%55.0%–58.4% | HIJK | 1,865 / 3,288 | 251 right of 253 kept2 wrong · 3,035 skipped | 0.802 | 7.0%answered 97% | 38 ms | 45 ms | $0.028$28.19estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Latency: median 80 ms · p90 94 ms Logic 46.4% · Math 61.3% | — | Decision | 56.2%54.5%–57.9% | IJK | 1,849 / 3,288 | 235 right of 236 kept1 wrong · 3,052 skipped | 0.797 | 69.4%answered 60% | 80 ms | 94 ms | $0.010$9.79 list$0.058$58.27estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 39 ms · p90 53 ms Logic 48.2% · Math 59.8% | — | Decision | 55.8%54.1%–57.5% | IJKL | 1,836 / 3,288 | 74 right of 74 kept0 mistakes · 3,214 skipped | 0.798 | 13.5%answered 94% | 39 ms | 53 ms | $0.032$32.40estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 42 ms · p90 57 ms Logic 43.9% · Math 60.1% | — | Decision | 54.6%52.9%–56.3% | JKLM | 1,794 / 3,288 | No answers at this threshold | 0.786 | 10.4%answered 95% | 42 ms | 57 ms | $0.034$34.08estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Latency: median 177 ms · p90 222 ms Logic 47.8% · Math 57.9% | — | Decision | 54.5%52.8%–56.2% | JKLM | 1,791 / 3,288 | 105 right of 105 kept0 mistakes · 3,183 skipped | 0.786 | 26.9%answered 86% | 177 ms | 222 ms | $0.011$11.45estimate · DGX Spark |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 38 ms · p90 55 ms Logic 44.3% · Math 59.8% | — | Decision | 54.5%52.8%–56.2% | JKLMN | 1,791 / 3,288 | 153 right of 153 kept0 mistakes · 3,135 skipped | 0.782 | 3.4%answered 98% | 38 ms | 55 ms | $0.030$29.87estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Latency: median 142 ms · p90 165 ms Logic 46.6% · Math 58.4% | — | Decision | 54.4%52.7%–56.1% | KLMNO | 1,788 / 3,288 | 63 right of 63 kept0 mistakes · 3,225 skipped | 0.779 | 9.6%answered 95% | 142 ms | 165 ms | $0.009$8.92estimate · DGX Spark |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Latency: median 312 ms · p90 360 ms Logic 44.4% · Math 59.4% | — | Decision | 54.3%52.6%–56.0% | KLMNOP | 1,784 / 3,288 | 47 right of 47 kept0 mistakes · 3,241 skipped | 0.769 | 11.4%answered 94% | 312 ms | 360 ms | $0.009$9.07 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Latency: median 46 ms · p90 51 ms Logic 45.3% · Math 58.8% | — | Decision | 54.2%52.5%–55.9% | KLMNOP | 1,782 / 3,288 | 263 right of 266 kept3 wrong · 3,022 skipped | 0.795 | 6.1%answered 97% | 46 ms | 51 ms | $0.033$32.61estimate · RTX PRO 6000 Blackwell |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Latency: median 339 ms · p90 472 ms Logic 41.9% · Math 59.0% | — | Decision | 53.1%51.4%–54.8% | KLMNOP | 1,747 / 3,288 | 207 right of 207 kept0 mistakes · 3,081 skipped | 0.789 | 28.1%answered 85% | 339 ms | 472 ms | $0.012$12.17 |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Latency: median 31 ms · p90 36 ms Logic 44.1% · Math 56.8% | — | Decision | 52.5%50.8%–54.2% | LMNOPQ | 1,725 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.777 | 7.9%answered 96% | 31 ms | 36 ms | $0.024$24.19estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Latency: median 68 ms · p90 80 ms Logic 45.9% · Math 55.8% | — | Decision | 52.4%50.7%–54.1% | LMNOPQ | 1,724 / 3,288 | 117 right of 117 kept0 mistakes · 3,171 skipped | 0.763 | 14.4%answered 93% | 68 ms | 80 ms | $0.004$4.28estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Latency: median 29 ms · p90 35 ms Logic 44.7% · Math 55.9% | — | Decision | 52.1%50.4%–53.8% | MNOPQ | 1,713 / 3,288 | 130 right of 131 kept1 wrong · 3,157 skipped | 0.782 | 15.0%answered 92% | 29 ms | 35 ms | $0.024$23.77estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Latency: median 163 ms · p90 240 ms Logic 43.3% · Math 56.0% | — | Decision | 51.7%50.0%–53.4% | MNOPQ | 1,699 / 3,288 | 270 right of 281 kept11 wrong · 3,007 skipped | 0.776 | 3.1%answered 99% | 163 ms | 240 ms | $0.011$10.82estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 59 ms · p90 65 ms Logic 44.4% · Math 55.3% | — | Decision | 51.6%49.9%–53.3% | MNOPQ | 1,696 / 3,288 | 165 right of 167 kept2 wrong · 3,121 skipped | 0.765 | 7.1%answered 96% | 59 ms | 65 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Latency: median 76 ms · p90 104 ms Logic 42.3% · Math 56.0% | — | Decision | 51.3%49.6%–53.0% | MNOPQR | 1,687 / 3,288 | 71 right of 72 kept1 wrong · 3,216 skipped | 0.752 | 4.4%answered 98% | 76 ms | 104 ms | $0.005$5.02estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Latency: median 41 ms · p90 51 ms Logic 43.1% · Math 55.2% | — | Decision | 51.1%49.4%–52.8% | MNOPQR | 1,680 / 3,288 | 185 right of 190 kept5 wrong · 3,098 skipped | 0.769 | 18.9%answered 90% | 41 ms | 51 ms | $0.003$2.72 list$0.032$31.55estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Latency: median 293 ms · p90 349 ms Logic 45.1% · Math 53.5% | — | Decision | 50.6%48.9%–52.3% | NOPQRS | 1,664 / 3,288 | 356 right of 372 kept16 wrong · 2,916 skipped | 0.777 | 10.1%answered 95% | 293 ms | 349 ms | $0.005$5.46 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 185 ms · p90 292 ms Logic 40.8% · Math 55.6% | — | Decision | 50.5%48.8%–52.3% | OPQRS | 1,662 / 3,288 | 207 right of 208 kept1 wrong · 3,080 skipped | 0.757 | 6.4%answered 97% | 185 ms | 292 ms | $0.004$3.98 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Latency: median 250 ms · p90 482 ms Logic 42.6% · Math 54.6% | — | Decision | 50.5%48.7%–52.2% | PQRS | 1,659 / 3,288 | No answers at this threshold | 0.759 | 14.5%answered 92% | 250 ms | 482 ms | $0.212$212estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Latency: median 80 ms · p90 134 ms Logic 40.6% · Math 53.8% | — | Decision | 49.3%47.6%–51.0% | QRST | 1,620 / 3,288 | 30 right of 30 kept0 mistakes · 3,258 skipped | 0.770 | 1.6%answered 99% | 80 ms | 134 ms | $0.069$69.42estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Latency: median 185 ms · p90 216 ms Logic 40.8% · Math 51.1% | — | Decision | 47.6%45.9%–49.3% | RSTU | 1,564 / 3,288 | 18 right of 18 kept0 mistakes · 3,270 skipped | 0.747 | 2.4%answered 99% | 185 ms | 216 ms | $0.081$81.46 list$0.135$135estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 46 ms · p90 52 ms Logic 39.7% · Math 51.5% | — | Decision | 47.5%45.8%–49.2% | STU | 1,561 / 3,288 | 161 right of 165 kept4 wrong · 3,123 skipped | 0.747 | 8.9%answered 95% | 46 ms | 52 ms | $0.033$33.45estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 30 ms · p90 42 ms Logic 40.4% · Math 50.4% | — | Decision | 47.0%45.3%–48.7% | STU | 1,545 / 3,288 | 188 right of 194 kept6 wrong · 3,094 skipped | 0.743 | 7.8%answered 96% | 30 ms | 42 ms | $0.025$25.03estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Latency: median 103 ms · p90 131 ms Logic 38.9% · Math 49.8% | — | Decision | 46.1%44.4%–47.8% | TUV | 1,515 / 3,288 | No answers at this threshold | 0.707 | 6.9%answered 96% | 103 ms | 131 ms | $0.013$12.90 list$0.075$75.10estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Latency: median 55 ms · p90 72 ms Logic 39.7% · Math 46.9% | — | Decision | 44.4%42.7%–46.1% | UV | 1,461 / 3,288 | 52 right of 53 kept1 wrong · 3,235 skipped | 0.737 | 17.1%answered 90% | 55 ms | 72 ms | $0.043$42.91estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Latency: median 38 ms · p90 44 ms Logic 39.3% · Math 44.3% | — | Decision | 42.6%40.9%–44.3% | V | 1,401 / 3,288 | No answers at this threshold | 0.729 | 7.3%answered 96% | 38 ms | 44 ms | $0.029$29.34 list$0.029$28.61estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Latency: median 117 ms · p90 146 ms Logic 32.1% · Math 39.2% | — | Decision | 36.7%35.1%–38.4% | W | 1,208 / 3,288 | No answers at this threshold | 0.659 | 2.0%answered 99% | 117 ms | 146 ms | $0.086$85.83estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Latency: median 36 ms · p90 42 ms Logic 30.8% · Math 38.7% | — | Decision | 35.9%34.3%–37.6% | W | 1,182 / 3,288 | 4 right of 4 kept0 mistakes · 3,284 skipped | 0.675 | 16.4%answered 89% | 36 ms | 42 ms | $0.002$2.36estimate · DGX Spark |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Latency: median 28 ms · p90 37 ms Logic 32.3% · Math 36.8% | — | Decision | 35.2%33.6%–36.9% | W | 1,158 / 3,288 | 29 right of 30 kept1 wrong · 3,258 skipped | 0.637 | 23.1%answered 84% | 28 ms | 37 ms | $0.002$1.94estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Latency: median 17 ms · p90 21 ms Logic 30.7% · Math 37.1% | — | Decision | 34.9%33.3%–36.5% | WX | 1,147 / 3,288 | 7 right of 7 kept0 mistakes · 3,281 skipped | 0.648 | 82.5%answered 30% | 17 ms | 21 ms | $0.014$14.30estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Latency: median 51 ms · p90 63 ms Logic 31.7% · Math 35.3% | — | Decision | 34.1%32.5%–35.7% | WX | 1,120 / 3,288 | 16 right of 16 kept0 mistakes · 3,272 skipped | 0.626 | 6.8%answered 95% | 51 ms | 63 ms | $0.003$3.40estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Latency: median 20 ms · p90 25 ms Logic 29.2% · Math 32.1% | — | Decision | 31.1%29.5%–32.7% | XY | 1,022 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.602 | 22.7%answered 83% | 20 ms | 25 ms | $0.001$1.38estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 24 ms · p90 29 ms Logic 24.7% · Math 29.7% | — | Decision | 28.0%26.5%–29.5% | YZ | 920 / 3,288 | No answers at this threshold | 0.605 | 14.9%answered 89% | 24 ms | 29 ms | $0.002$1.62estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Latency: median 15 ms · p90 17 ms Logic 27.0% · Math 28.2% | — | Decision | 27.8%26.3%–29.4% | YZ | 915 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.576 | 2.0%answered 99% | 15 ms | 17 ms | $0.001$1.11estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 26 ms · p90 35 ms Logic 26.7% · Math 28.2% | — | Decision | 27.7%26.2%–29.2% | YZ | 910 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.590 | 0.6%answered 100% | 26 ms | 35 ms | $0.002$1.80estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Latency: median 35 ms · p90 42 ms Logic 22.4% · Math 28.7% | — | Decision | 26.5%25.0%–28.1% | Zα | 872 / 3,288 | No answers at this threshold | 0.573 | 5.0%answered 96% | 35 ms | 42 ms | $0.027$26.72estimate · RTX PRO 6000 Blackwell |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Latency: median 12 ms · p90 23 ms Logic 23.6% · Math 22.7% | — | Decision | 23.0%21.6%–24.4% | αβ | 755 / 3,288 | No answers at this threshold | 0.511 | 15.6%answered 87% | 12 ms | 23 ms | $0.001$1.01estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Latency: median 11 ms · p90 14 ms Logic 23.2% · Math 20.6% | — | Decision | 21.5%20.1%–22.9% | β | 706 / 3,288 | No answers at this threshold | 0.520 | 73.9%answered 32% | 11 ms | 14 ms | $0.001$0.83estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents victorbennett Metask Lab Google OpenBMB Alibaba IBM KRAFTON 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GLiDE 0.4B: 21.5% at a median 11 ms
GLiClass v4 Edge: 23.0% at a median 12 ms
OneJev1.6-0.8B: 27.8% at a median 15 ms
d1-2B: 34.9% at a median 17 ms
GLiDE 0.8B: 31.1% at a median 20 ms
StartLux-Decision-v1.1-0.8B-Instant: 28.0% at a median 24 ms
KnowLine-0.8B-Gen1: 27.7% at a median 26 ms
GLiDE 2B: 35.2% at a median 28 ms
Perplexity Decider Flash v1.3: 52.1% at a median 29 ms
Chinese Decision Pointer 4B: 47.0% at a median 30 ms
vera-medium-omni: 52.5% at a median 31 ms
Strands Decider 2B: 26.5% at a median 35 ms
StartLux-Decision-v1.1-2B-Instant: 35.9% at a median 36 ms
Perplexity Decider Flash v1.2: 51.1% at a median 41 ms
Verdin-4B-v1.1: 47.5% at a median 46 ms
KnowLine-2B-Gen1: 34.1% at a median 51 ms
StartLux-Decision-v1.1-4B-Instant: 52.4% at a median 68 ms
KnowLine 4B Gen5.5: 51.3% at a median 76 ms
Metask-Jev-4B: 36.7% at a median 117 ms
LFM3-1.7B (on · chat): 93.9% at a median 5,630 ms
LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell): 94.0% at a median 5,667 ms
Gemma 4 E2B (on · chat · 32k cap · sampled): 90.5% at a median 8,873 ms
MiniCPM5-2B (off · chat): 93.2% at a median 9,065 ms
Gemma 4 E2B (chat): 88.6% at a median 9,935 ms
LFM2.5-2.6B (on · chat): 87.6% at a median 14,035 ms
Qwen3.5-4B (off · chat): 95.6% at a median 14,295 ms
Qwen3.5-2B (off · chat): 84.4% at a median 17,805 ms
Gemma 4 E2B (on · chat · 32k cap): 90.3% at a median 19,100 ms
MiniCPM5-2B (on · chat): 95.7% at a median 19,260 ms
Granite-4.2-3B (high · chat): 96.6% at a median 25,615 ms
LFM3-1.7B (off · chat): 94.2% at a median 29,040 ms
Raon-1 nano (off · chat): 81.2% at a median 42,585 ms (other engine: Logic: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; Math: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Raon-1 nano (on · chat): 88.1% at a median 54,985 ms (other engine: Logic: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; Math: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Qwen3.5-4B (on · chat): 97.0% at a median 63,270 ms
Gemma 4 E2B (on · chat): 89.3% at a median 77,885 ms
Qwen3.5-2B (on · chat): 86.4% at a median 121,840 ms
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents victorbennett Metask Lab Google OpenBMB Alibaba IBM KRAFTON
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: Spark-X2.6-4B (high · chat) (missing Logic and Math); Granite-4.2-3B (off · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Latency: median 63,270 ms · p90 193,950 ms Logic 92.4% · Math 99.4% | — | Chat | 97.0%96.4%–97.5% | a | 3,190 / 3,288 | Not measured | Not measured | 34.2%answered 99% | 63,270 ms | 193,950 ms | $2.042$2,042estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Latency: median 25,615 ms · p90 523,100 ms Logic 91.6% · Math 99.2% | — | Chat | 96.6%96.0%–97.2% | ab | 3,177 / 3,288 | Not measured | Not measured | 54.7%answered 98% | 25,615 ms | 523,100 ms | $4.241$4,241estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Latency: median 19,260 ms · p90 194,917 ms Logic 88.9% · Math 99.2% | — | Chat | 95.7%95.4%–96.0% min–max of 3 seeds | b | 3,145 / 3,288seed 0 run | Not measured | Not measured | 51.8%answered 97% | 19,260 ms | 194,917 ms | $1.410$1,410estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Latency: median 14,295 ms · p90 78,230 ms Logic 90.1% · Math 98.5% | — | Chat | 95.6%94.9%–96.3% | bc | 3,144 / 3,288 | Not measured | Not measured | 34.0%answered 98% | 14,295 ms | 78,230 ms | $0.683$683estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Latency: median 29,040 ms · p90 343,820 ms Logic 85.5% · Math 98.7% | — | Chat | 94.2%93.3%–94.9% | cd | 3,097 / 3,288 | Not measured | Not measured | 55.9%answered 96% | 29,040 ms | 343,820 ms | $0.206$206estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Latency: median 5,667 ms · p90 58,313 ms Logic 84.6% · Math 98.9% | — | Chat | 94.0%93.8%–94.2% min–max of 3 seeds | cd | 3,098 / 3,288seed 0 run | Not measured | Not measured | Not measured | 5,667 ms | 58,313 ms | $0.449$449estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Latency: median 5,630 ms · p90 63,067 ms Logic 84.2% · Math 98.9% | — | Chat | 93.9%93.89%–93.95% min–max of 3 seeds | d | 3,089 / 3,288seed 0 run | Not measured | Not measured | 46.7%answered 96% | 5,630 ms | 63,067 ms | $0.435$435estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Latency: median 9,065 ms · p90 75,703 ms Logic 83.6% · Math 98.2% | — | Chat | 93.2%92.9%–93.3% min–max of 3 seeds | d | 3,068 / 3,288seed 0 run | Not measured | Not measured | 41.0%answered 95% | 9,065 ms | 75,703 ms | $0.598$598estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 8,873 ms · p90 32,503 ms Logic 76.8% · Math 97.7% | — | Chat | 90.5%90.2%–90.9% min–max of 3 seeds | e | 2,966 / 3,288seed 0 run | Not measured | Not measured | 31.0%answered 97% | 8,873 ms | 32,503 ms | $0.325$325estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 19,100 ms · p90 54,630 ms Logic 75.9% · Math 97.8% | — | Chat | 90.3%89.3%–91.3% | e | 2,970 / 3,288 | Not measured | Not measured | Not measured | 19,100 ms | 54,630 ms | $0.678$678estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 77,885 ms · p90 232,800 ms Logic 74.6% · Math 96.9% | — | Chat | 89.3%88.2%–90.3% | ef | 2,936 / 3,288 | Not measured | Not measured | 24.3%answered 97% | 77,885 ms | 232,800 ms | $0.215$215estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 9,935 ms · p90 27,690 ms Logic 72.9% · Math 96.8% | — | Chat | 88.6%87.5%–89.6% | ef | 2,913 / 3,288 | Not measured | Not measured | 32.3%answered 96% | 9,935 ms | 27,690 ms | $0.313$313estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B | — | Chat | 88.1%86.9%–89.1% | fg | 2,896 / 3,288 | Not measured | Not measured | Not measured | 54,985 ms† | 987,660 ms† | $6.344$6,344estimate · RTX PRO 6000 Blackwell† |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Latency: median 14,035 ms · p90 142,970 ms Logic 75.4% · Math 94.0% | — | Chat | 87.6%86.5%–88.7% | fg | 2,881 / 3,288 | Not measured | Not measured | 40.6%answered 95% | 14,035 ms | 142,970 ms | $1.046$1,046estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Latency: median 121,840 ms · p90 241,590 ms Logic 70.3% · Math 94.7% | — | Chat | 86.4%85.2%–87.5% | gh | 2,840 / 3,288 | Not measured | Not measured | 30.9%answered 97% | 121,840 ms | 241,590 ms | $3.106$3,106estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Latency: median 17,805 ms · p90 92,100 ms Logic 64.8% · Math 94.6% | — | Chat | 84.4%83.1%–85.6% | h | 2,776 / 3,288 | Not measured | Not measured | 34.3%answered 94% | 17,805 ms | 92,100 ms | $0.788$788estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B | — | Chat | 81.2%79.8%–82.5% | i | 2,669 / 3,288 | Not measured | Not measured | 31.1%answered 94% | 42,585 ms† | 593,950 ms† | $4.422$4,422estimate · RTX PRO 6000 Blackwell† |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Latency: median 31 ms · p90 36 ms Logic 44.1% · Math 56.8% | — | Decision | 52.5%50.8%–54.2% | j | 1,725 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.777 | 7.9%answered 96% | 31 ms | 36 ms | $0.024$24.19estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Latency: median 68 ms · p90 80 ms Logic 45.9% · Math 55.8% | — | Decision | 52.4%50.7%–54.1% | j | 1,724 / 3,288 | 117 right of 117 kept0 mistakes · 3,171 skipped | 0.763 | 14.4%answered 93% | 68 ms | 80 ms | $0.004$4.28estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Latency: median 29 ms · p90 35 ms Logic 44.7% · Math 55.9% | — | Decision | 52.1%50.4%–53.8% | j | 1,713 / 3,288 | 130 right of 131 kept1 wrong · 3,157 skipped | 0.782 | 15.0%answered 92% | 29 ms | 35 ms | $0.024$23.77estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Latency: median 76 ms · p90 104 ms Logic 42.3% · Math 56.0% | — | Decision | 51.3%49.6%–53.0% | j | 1,687 / 3,288 | 71 right of 72 kept1 wrong · 3,216 skipped | 0.752 | 4.4%answered 98% | 76 ms | 104 ms | $0.005$5.02estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Latency: median 41 ms · p90 51 ms Logic 43.1% · Math 55.2% | — | Decision | 51.1%49.4%–52.8% | j | 1,680 / 3,288 | 185 right of 190 kept5 wrong · 3,098 skipped | 0.769 | 18.9%answered 90% | 41 ms | 51 ms | $0.003$2.72 list$0.032$31.55estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 46 ms · p90 52 ms Logic 39.7% · Math 51.5% | — | Decision | 47.5%45.8%–49.2% | k | 1,561 / 3,288 | 161 right of 165 kept4 wrong · 3,123 skipped | 0.747 | 8.9%answered 95% | 46 ms | 52 ms | $0.033$33.45estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 30 ms · p90 42 ms Logic 40.4% · Math 50.4% | — | Decision | 47.0%45.3%–48.7% | k | 1,545 / 3,288 | 188 right of 194 kept6 wrong · 3,094 skipped | 0.743 | 7.8%answered 96% | 30 ms | 42 ms | $0.025$25.03estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Latency: median 117 ms · p90 146 ms Logic 32.1% · Math 39.2% | — | Decision | 36.7%35.1%–38.4% | l | 1,208 / 3,288 | No answers at this threshold | 0.659 | 2.0%answered 99% | 117 ms | 146 ms | $0.086$85.83estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Latency: median 36 ms · p90 42 ms Logic 30.8% · Math 38.7% | — | Decision | 35.9%34.3%–37.6% | l | 1,182 / 3,288 | 4 right of 4 kept0 mistakes · 3,284 skipped | 0.675 | 16.4%answered 89% | 36 ms | 42 ms | $0.002$2.36estimate · DGX Spark |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Latency: median 28 ms · p90 37 ms Logic 32.3% · Math 36.8% | — | Decision | 35.2%33.6%–36.9% | l | 1,158 / 3,288 | 29 right of 30 kept1 wrong · 3,258 skipped | 0.637 | 23.1%answered 84% | 28 ms | 37 ms | $0.002$1.94estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Latency: median 17 ms · p90 21 ms Logic 30.7% · Math 37.1% | — | Decision | 34.9%33.3%–36.5% | l | 1,147 / 3,288 | 7 right of 7 kept0 mistakes · 3,281 skipped | 0.648 | 82.5%answered 30% | 17 ms | 21 ms | $0.014$14.30estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Latency: median 51 ms · p90 63 ms Logic 31.7% · Math 35.3% | — | Decision | 34.1%32.5%–35.7% | lm | 1,120 / 3,288 | 16 right of 16 kept0 mistakes · 3,272 skipped | 0.626 | 6.8%answered 95% | 51 ms | 63 ms | $0.003$3.40estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Latency: median 20 ms · p90 25 ms Logic 29.2% · Math 32.1% | — | Decision | 31.1%29.5%–32.7% | mn | 1,022 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.602 | 22.7%answered 83% | 20 ms | 25 ms | $0.001$1.38estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 24 ms · p90 29 ms Logic 24.7% · Math 29.7% | — | Decision | 28.0%26.5%–29.5% | no | 920 / 3,288 | No answers at this threshold | 0.605 | 14.9%answered 89% | 24 ms | 29 ms | $0.002$1.62estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Latency: median 15 ms · p90 17 ms Logic 27.0% · Math 28.2% | — | Decision | 27.8%26.3%–29.4% | o | 915 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.576 | 2.0%answered 99% | 15 ms | 17 ms | $0.001$1.11estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 26 ms · p90 35 ms Logic 26.7% · Math 28.2% | — | Decision | 27.7%26.2%–29.2% | o | 910 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.590 | 0.6%answered 100% | 26 ms | 35 ms | $0.002$1.80estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Latency: median 35 ms · p90 42 ms Logic 22.4% · Math 28.7% | — | Decision | 26.5%25.0%–28.1% | o | 872 / 3,288 | No answers at this threshold | 0.573 | 5.0%answered 96% | 35 ms | 42 ms | $0.027$26.72estimate · RTX PRO 6000 Blackwell |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Latency: median 12 ms · p90 23 ms Logic 23.6% · Math 22.7% | — | Decision | 23.0%21.6%–24.4% | p | 755 / 3,288 | No answers at this threshold | 0.511 | 15.6%answered 87% | 12 ms | 23 ms | $0.001$1.01estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Latency: median 11 ms · p90 14 ms Logic 23.2% · Math 20.6% | — | Decision | 21.5%20.1%–22.9% | p | 706 / 3,288 | No answers at this threshold | 0.520 | 73.9%answered 32% | 11 ms | 14 ms | $0.001$0.83estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Blockbrain Labs newsbang PelaAI tattrongvu Patrick-18 EldanRing Valen-Team Base Labs Cloudflare StartLux Apertix AI Fastino eCloud Technology Alibaba Google Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Cygnet: 42.6% at a median 38 ms
Adjudex 9B v5: 56.7% at a median 38 ms
KnowLine-9B-Gen1: 54.5% at a median 38 ms
Jeffan-9B-1211: 55.8% at a median 39 ms
KnowLine-9B-Gen2: 54.6% at a median 42 ms
Dexy-Decision-9B-V2: 54.2% at a median 46 ms
KnowLine-12B-Gen1: 60.9% at a median 51 ms
Winnow-12B: 44.4% at a median 55 ms
Valen-Decision-1.5-9B: 51.6% at a median 59 ms
Elephant-1: 56.2% at a median 80 ms
Clef-Flash: 46.1% at a median 103 ms
StartLux-Decision-v1.1-9B-Instant: 54.4% at a median 142 ms
Cerno 9B v1.2: 51.7% at a median 163 ms
GLiDE 9B: 54.5% at a median 177 ms
Erk-14B (off · chat): 78.3% at a median 8,740 ms
Qwen3-14B (off · chat): 85.4% at a median 9,380 ms
Gemma 4 E4B (chat): 93.0% at a median 13,150 ms
Gemma 4 E4B (on · chat · 32k cap · sampled): 94.2% at a median 13,945 ms
LFM2.5-8B-A1B (on · chat): 83.9% at a median 20,095 ms (mixed concurrency: This run changed the number of requests in flight part-way; see its record on Method)
Gemma 4 12B Unified (chat): 95.0% at a median 22,345 ms
Gemma 4 E4B (on · chat · 32k cap): 93.9% at a median 25,530 ms
Gemma 4 E4B (on · chat): 92.9% at a median 131,535 ms
Blockbrain Labs newsbang PelaAI tattrongvu Patrick-18 EldanRing Valen-Team Base Labs Cloudflare StartLux Apertix AI Fastino eCloud Technology Alibaba Google Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Latency: median 22,345 ms · p90 53,390 ms Logic 87.7% · Math 98.8% | — | Chat | 95.0%94.2%–95.7% | a | 3,125 / 3,288 | Not measured | Not measured | 15.3%answered 99% | 22,345 ms | 53,390 ms | $0.749$749estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 13,945 ms · p90 51,987 ms Logic 85.9% · Math 98.5% | — | Chat | 94.2%94.0%–94.4% min–max of 3 seeds | ab | 3,104 / 3,288seed 0 run | Not measured | Not measured | 54.1%answered 96% | 13,945 ms | 51,987 ms | $0.549$549estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 25,530 ms · p90 77,380 ms Logic 85.0% · Math 98.5% | — | Chat | 93.9%93.0%–94.7% | abc | 3,087 / 3,288 | Not measured | Not measured | Not measured | 25,530 ms | 77,380 ms | $0.855$855estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 13,150 ms · p90 44,990 ms Logic 83.2% · Math 98.1% | — | Chat | 93.0%92.1%–93.8% | bc | 3,058 / 3,288 | Not measured | Not measured | 51.0%answered 95% | 13,150 ms | 44,990 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 131,535 ms · p90 392,620 ms Logic 82.6% · Math 98.2% | — | Chat | 92.9%92.0%–93.7% | c | 3,054 / 3,288 | Not measured | Not measured | 43.7%answered 96% | 131,535 ms | 392,620 ms | $0.375$375estimate · DGX Spark |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Latency: median 9,380 ms · p90 35,110 ms Logic 65.3% · Math 95.8% | — | Chat | 85.4%84.1%–86.5% | d | 2,807 / 3,288 | Not measured | Not measured | 20.9%answered 96% | 9,380 ms | 35,110 ms | $0.840$840estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | Chat | 83.9%82.6%–85.1% | d | 2,759 / 3,288 | Not measured | Not measured | 55.9%answered 88% | 20,095 ms† | 230,570 ms† | $1.913$1,913estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Latency: median 8,740 ms · p90 16,420 ms Logic 54.2% · Math 90.8% | — | Chat | 78.3%76.8%–79.6% | e | 2,573 / 3,288 | Not measured | Not measured | 32.3%answered 92% | 8,740 ms | 16,420 ms | $0.308$308estimate · RTX PRO 6000 Blackwell |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Latency: median 51 ms · p90 76 ms Logic 50.6% · Math 66.3% | — | Decision | 60.9%59.2%–62.5% | f | 2,002 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.810 | 4.5%answered 98% | 51 ms | 76 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 38 ms · p90 45 ms Logic 46.4% · Math 62.1% | — | Decision | 56.7%55.0%–58.4% | g | 1,865 / 3,288 | 251 right of 253 kept2 wrong · 3,035 skipped | 0.802 | 7.0%answered 97% | 38 ms | 45 ms | $0.028$28.19estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Latency: median 80 ms · p90 94 ms Logic 46.4% · Math 61.3% | — | Decision | 56.2%54.5%–57.9% | g | 1,849 / 3,288 | 235 right of 236 kept1 wrong · 3,052 skipped | 0.797 | 69.4%answered 60% | 80 ms | 94 ms | $0.010$9.79 list$0.058$58.27estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 39 ms · p90 53 ms Logic 48.2% · Math 59.8% | — | Decision | 55.8%54.1%–57.5% | g | 1,836 / 3,288 | 74 right of 74 kept0 mistakes · 3,214 skipped | 0.798 | 13.5%answered 94% | 39 ms | 53 ms | $0.032$32.40estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 42 ms · p90 57 ms Logic 43.9% · Math 60.1% | — | Decision | 54.6%52.9%–56.3% | g | 1,794 / 3,288 | No answers at this threshold | 0.786 | 10.4%answered 95% | 42 ms | 57 ms | $0.034$34.08estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Latency: median 177 ms · p90 222 ms Logic 47.8% · Math 57.9% | — | Decision | 54.5%52.8%–56.2% | ghi | 1,791 / 3,288 | 105 right of 105 kept0 mistakes · 3,183 skipped | 0.786 | 26.9%answered 86% | 177 ms | 222 ms | $0.011$11.45estimate · DGX Spark |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 38 ms · p90 55 ms Logic 44.3% · Math 59.8% | — | Decision | 54.5%52.8%–56.2% | gh | 1,791 / 3,288 | 153 right of 153 kept0 mistakes · 3,135 skipped | 0.782 | 3.4%answered 98% | 38 ms | 55 ms | $0.030$29.87estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Latency: median 142 ms · p90 165 ms Logic 46.6% · Math 58.4% | — | Decision | 54.4%52.7%–56.1% | ghi | 1,788 / 3,288 | 63 right of 63 kept0 mistakes · 3,225 skipped | 0.779 | 9.6%answered 95% | 142 ms | 165 ms | $0.009$8.92estimate · DGX Spark |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Latency: median 46 ms · p90 51 ms Logic 45.3% · Math 58.8% | — | Decision | 54.2%52.5%–55.9% | ghi | 1,782 / 3,288 | 263 right of 266 kept3 wrong · 3,022 skipped | 0.795 | 6.1%answered 97% | 46 ms | 51 ms | $0.033$32.61estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Latency: median 163 ms · p90 240 ms Logic 43.3% · Math 56.0% | — | Decision | 51.7%50.0%–53.4% | hi | 1,699 / 3,288 | 270 right of 281 kept11 wrong · 3,007 skipped | 0.776 | 3.1%answered 99% | 163 ms | 240 ms | $0.011$10.82estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 59 ms · p90 65 ms Logic 44.4% · Math 55.3% | — | Decision | 51.6%49.9%–53.3% | i | 1,696 / 3,288 | 165 right of 167 kept2 wrong · 3,121 skipped | 0.765 | 7.1%answered 96% | 59 ms | 65 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Latency: median 103 ms · p90 131 ms Logic 38.9% · Math 49.8% | — | Decision | 46.1%44.4%–47.8% | j | 1,515 / 3,288 | No answers at this threshold | 0.707 | 6.9%answered 96% | 103 ms | 131 ms | $0.013$12.90 list$0.075$75.10estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Latency: median 55 ms · p90 72 ms Logic 39.7% · Math 46.9% | — | Decision | 44.4%42.7%–46.1% | jk | 1,461 / 3,288 | 52 right of 53 kept1 wrong · 3,235 skipped | 0.737 | 17.1%answered 90% | 55 ms | 72 ms | $0.043$42.91estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Latency: median 38 ms · p90 44 ms Logic 39.3% · Math 44.3% | — | Decision | 42.6%40.9%–44.3% | k | 1,401 / 3,288 | No answers at this threshold | 0.729 | 7.3%answered 96% | 38 ms | 44 ms | $0.029$29.34 list$0.029$28.61estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
vLLM Semantic Router OmniJev Fastino Nace.AI Perplexity PelaAI Poindexter Labs Base Labs MachinePulse Cloudflare Zefan Cai Turion.AI Sthānika AI Alibaba Google Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
d3: 65.0% at a median 57 ms
OneJev-27B: 49.3% at a median 80 ms
GLiDE (v1.2): 64.8% at a median 81 ms
Drex Max: 64.4% at a median 81 ms
Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): 66.4% at a median 97 ms
KnowLine-31B-Gen2: 73.0% at a median 116 ms
Copernicus-31B: 71.0% at a median 128 ms
d3-ultra: 65.7% at a median 129 ms
Mammoth-1: 67.3% at a median 150 ms
Karpo Decision 31B: 74.4% at a median 162 ms
Clef: 47.6% at a median 185 ms
Perplexity Decider 27B v1.2 (API): 64.1% at a median 230 ms
Open-Jev-27B-v1.1: 50.5% at a median 250 ms
Turion Jev 31B: 63.3% at a median 301 ms
Drex 2.0: 53.1% at a median 339 ms
Perplexity Decider 27B v1.2 (API · with web): 63.6% at a median 430 ms
GLiDE 27B: 67.3% at a median 450 ms
Sieve 31B v3: 70.5% at a median 676 ms
KnowLine-31B-Gen1.5: 71.1% at a median 811 ms
Qwen3.8-27B (low · chat): 97.4% at a median 11,200 ms
Gemma 4 31B (chat): 97.3% at a median 12,575 ms
LFM2.5-24B-A2B (on · chat): 94.8% at a median 42,425 ms
vLLM Semantic Router OmniJev Fastino Nace.AI Perplexity PelaAI Poindexter Labs Base Labs MachinePulse Cloudflare Zefan Cai Turion.AI Sthānika AI Alibaba Google Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Latency: median 11,200 ms · p90 39,560 ms Logic 94.0% · Math 99.2% | — | Chat | 97.4%96.8%–97.9% | a | 3,204 / 3,288 | Not measured | Not measured | 33.9%answered 99% | 11,200 ms | 39,560 ms | $2.020$2,020estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Latency: median 12,575 ms · p90 25,280 ms Logic 93.3% · Math 99.4% | — | Chat | 97.3%96.7%–97.8% | a | 3,200 / 3,288 | Not measured | Not measured | 17.0%answered 99% | 12,575 ms | 25,280 ms | $0.775$775estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Latency: median 42,425 ms · p90 432,510 ms Logic 86.7% · Math 99.0% | — | Chat | 94.8%94.0%–95.5% | b | 3,117 / 3,288 | Not measured | Not measured | 44.0%answered 97% | 42,425 ms | 432,510 ms | $3.205$3,205estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Latency: median 162 ms · p90 198 ms Logic 60.7% · Math 81.6% | — | Decision | 74.4%72.9%–75.9% | c | 2,447 / 3,288 | 1,006 right of 1,009 kept3 wrong · 2,279 skipped | 0.882 | 5.7%answered 98% | 162 ms | 198 ms | $0.118$118estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Latency: median 116 ms · p90 177 ms Logic 58.9% · Math 80.3% | — | Decision | 73.0%71.4%–74.5% | c | 2,399 / 3,288 | 523 right of 524 kept1 wrong · 2,764 skipped | 0.882 | 0.1%answered 100% | 116 ms | 177 ms | $0.093$92.98estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Latency: median 811 ms · p90 1,111 ms Logic 59.0% · Math 77.4% | — | Decision | 71.1%69.5%–72.6% | d | 2,338 / 3,288 | 553 right of 554 kept1 wrong · 2,734 skipped | 0.870 | 0.7%answered 100% | 811 ms | 1,111 ms | $0.052$51.85estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Latency: median 128 ms · p90 218 ms Logic 58.7% · Math 77.3% | — | Decision | 71.0%69.4%–72.5% | d | 2,333 / 3,288 | 384 right of 384 kept0 mistakes · 2,904 skipped | 0.873 | 1.8%answered 99% | 128 ms | 218 ms | $0.102$102estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Latency: median 676 ms · p90 1,173 ms Logic 57.9% · Math 77.1% | — | Decision | 70.5%68.9%–72.1% | d | 2,319 / 3,288 | 294 right of 296 kept2 wrong · 2,992 skipped | 0.865 | 3.8%answered 99% | 676 ms | 1,173 ms | $0.047$47.29estimate · DGX Spark |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Latency: median 450 ms · p90 587 ms Logic 58.4% · Math 72.0% | — | Decision | 67.3%65.7%–68.9% | e | 2,214 / 3,288 | 272 right of 274 kept2 wrong · 3,014 skipped | 0.852 | 31.9%answered 89% | 450 ms | 587 ms | $0.029$28.55estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Latency: median 150 ms · p90 232 ms Logic 54.5% · Math 74.0% | — | Decision | 67.3%65.7%–68.9% | ef | 2,213 / 3,288 | 518 right of 520 kept2 wrong · 2,768 skipped | 0.868 | 71.4%answered 69% | 150 ms | 232 ms | $0.028$27.98 list$0.120$120estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Latency: median 97 ms · p90 125 ms Logic 54.8% · Math 72.5% | — | Decision | 66.4%64.8%–68.0% | efg | 2,184 / 3,288 | 385 right of 385 kept0 mistakes · 2,903 skipped | 0.844 | 13.5%answered 95% | 97 ms | 125 ms | $0.071$71.10estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 129 ms · p90 168 ms Logic 54.6% · Math 71.5% | — | Decision | 65.7%64.1%–67.3% | efgh | 2,161 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.835 | 7.5%answered 97% | 129 ms | 168 ms | $0.096$95.71estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 57 ms · p90 86 ms Logic 54.5% · Math 70.4% | — | Decision | 65.0%63.3%–66.6% | fgh | 2,136 / 3,288 | 216 right of 217 kept1 wrong · 3,071 skipped | 0.858 | 9.3%answered 97% | 57 ms | 86 ms | $0.046$46.28estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Latency: median 81 ms · p90 136 ms Logic 55.2% · Math 69.9% | — | Decision | 64.8%63.2%–66.5% | fgh | 2,132 / 3,288 | 176 right of 176 kept0 mistakes · 3,112 skipped | 0.843 | 42.3%answered 83% | 81 ms | 136 ms | $0.010$9.71 list$0.066$66.05estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Latency: median 81 ms · p90 130 ms Logic 53.1% · Math 70.3% | — | Decision | 64.4%62.8%–66.0% | gh | 2,118 / 3,288 | 866 right of 884 kept18 wrong · 2,404 skipped | 0.841 | 5.7%answered 98% | 81 ms | 130 ms | $0.016$16.31 list$0.064$63.74estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Latency: median 230 ms · p90 439 ms Logic 53.3% · Math 69.8% | — | Decision | 64.1%62.5%–65.8% | h | 2,109 / 3,288 | 409 right of 411 kept2 wrong · 2,877 skipped | 0.849 | 10.3%answered 96% | 230 ms | 439 ms | $0.005$5.38 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Latency: median 430 ms · p90 707 ms Logic 52.5% · Math 69.3% | with web | Decision | 63.6%61.9%–65.2% | h | 2,090 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.854 | 25.8%answered 90% | 430 ms | 707 ms | $1.025$1,025 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Latency: median 301 ms · p90 448 ms Logic 52.3% · Math 69.0% | — | Decision | 63.3%61.6%–64.9% | h | 2,081 / 3,288 | 50 right of 50 kept0 mistakes · 3,238 skipped | 0.844 | 1.9%answered 99% | 301 ms | 448 ms | $0.020$20.01estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Latency: median 339 ms · p90 472 ms Logic 41.9% · Math 59.0% | — | Decision | 53.1%51.4%–54.8% | i | 1,747 / 3,288 | 207 right of 207 kept0 mistakes · 3,081 skipped | 0.789 | 28.1%answered 85% | 339 ms | 472 ms | $0.012$12.17 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Latency: median 250 ms · p90 482 ms Logic 42.6% · Math 54.6% | — | Decision | 50.5%48.7%–52.2% | ij | 1,659 / 3,288 | No answers at this threshold | 0.759 | 14.5%answered 92% | 250 ms | 482 ms | $0.212$212estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Latency: median 80 ms · p90 134 ms Logic 40.6% · Math 53.8% | — | Decision | 49.3%47.6%–51.0% | j | 1,620 / 3,288 | 30 right of 30 kept0 mistakes · 3,258 skipped | 0.770 | 1.6%answered 99% | 80 ms | 134 ms | $0.069$69.42estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Latency: median 185 ms · p90 216 ms Logic 40.8% · Math 51.1% | — | Decision | 47.6%45.9%–49.3% | j | 1,564 / 3,288 | 18 right of 18 kept0 mistakes · 3,270 skipped | 0.747 | 2.4%answered 99% | 185 ms | 216 ms | $0.081$81.46 list$0.135$135estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
REAX Logits MiniMax DeepSeek Thinking Machines Z.ai Xiaomi 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
REAX 1.3: 71.6% at a median 200 ms
Logits Sigma-1 Duo: 69.3% at a median 207 ms
MiniMax-M3.1-Flash (high · chat): 96.8% at a median 2,170 ms
DeepSeek-V4.1-Flash (high · chat · 1017): 98.4% at a median 2,270 ms
Inkling-Small (high · chat · Jul 2026): 98.2% at a median 2,750 ms
Inkling 2 Small (high · chat): 98.2% at a median 2,870 ms
GLM-5.3-Flash (high · chat): 96.9% at a median 4,640 ms
MiMo-V2.6-Flash (on · chat): 98.0% at a median 10,590 ms
REAX Logits MiniMax DeepSeek Thinking Machines Z.ai Xiaomi
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: DeepSeek-V4-Flash (chat) (missing Logic); Qwen3.5-397B-A17B (on · chat) (missing Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Latency: median 2,270 ms · p90 14,870 ms Logic 96.2% · Math 99.6% | — | Chat | 98.4%98.0%–98.8% | a | 3,237 / 3,288 | Not measured | Not measured | 6.3%answered 100% | 2,270 ms | 14,870 ms | $0.713$713 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Latency: median 2,870 ms · p90 35,530 ms Logic 95.6% · Math 99.5% | — | Chat | 98.2%97.7%–98.6% | a | 3,229 / 3,288 | Not measured | Not measured | 41.7%answered 99% | 2,870 ms | 35,530 ms | $1.830$1,830 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Latency: median 2,750 ms · p90 34,100 ms Logic 95.4% · Math 99.6% | — | Chat | 98.2%97.7%–98.6% | a | 3,228 / 3,288 | Not measured | Not measured | 50.6%answered 99% | 2,750 ms | 34,100 ms | $2.506$2,506 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Latency: median 10,590 ms · p90 53,970 ms Logic 95.1% · Math 99.4% | — | Chat | 98.0%97.4%–98.4% | a | 3,221 / 3,288 | Not measured | Not measured | 27.6%answered 99% | 10,590 ms | 53,970 ms | $0.615$615 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Latency: median 4,640 ms · p90 25,900 ms Logic 93.2% · Math 98.8% | — | Chat | 96.9%96.2%–97.4% | b | 3,185 / 3,288 | Not measured | Not measured | 7.4%answered 100% | 4,640 ms | 25,900 ms | $0.471$471 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Latency: median 2,170 ms · p90 15,760 ms Logic 94.0% · Math 98.2% | — | Chat | 96.8%96.1%–97.3% | b | 3,182 / 3,288 | Not measured | Not measured | 38.9%answered 99% | 2,170 ms | 15,760 ms | $0.342$342 |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Latency: median 200 ms · p90 260 ms Logic 59.3% · Math 78.0% | — | Decision | 71.6%70.0%–73.1% | c | 2,354 / 3,288 | 229 right of 229 kept0 mistakes · 3,059 skipped | 0.878 | 3.8%answered 99% | 200 ms | 260 ms | $0.148$148estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Latency: median 207 ms · p90 324 ms Logic 57.8% · Math 75.2% | — | Decision | 69.3%67.7%–70.8% | d | 2,277 / 3,288 | 23 right of 23 kept0 mistakes · 3,265 skipped | 0.868 | 4.4%answered 99% | 207 ms | 324 ms | $0.165$165estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
TypeSafe Liquid AI Google Fastino Inception Microsoft Nace.AI Celeris Subquadratic Meta OpenAI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Jev 1.14: 59.3% at a median 164 ms
Jev 1.13: 58.0% at a median 166 ms
d1.1: 58.6% at a median 174 ms
d1-ultra: 67.5% at a median 180 ms
d1: 50.5% at a median 185 ms
Gemini Decision 1 (20261111): 64.1% at a median 227 ms
GLiDE (API): 65.2% at a median 266 ms
Mercury Decide: 50.6% at a median 293 ms
Microsoft-Decision-1: 54.3% at a median 312 ms
Drex Max v1.1: 64.8% at a median 327 ms
celeris-1.1-decision: 62.1% at a median 381 ms
SubQ Snap 1.1: 63.0% at a median 382 ms
celeris-1-decision: 62.2% at a median 388 ms
Muse Ember (max · chat): 98.2% at a median 1,460 ms
GPT-6 Luna (chat): 81.4% at a median 3,830 ms
TypeSafe Liquid AI Google Fastino Inception Microsoft Nace.AI Celeris Subquadratic Meta OpenAI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 3,276 of 3,288); shown once.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: GPT-6 Luna (max · chat) (missing Logic and Math); Claude Haiku 5.5 (max · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Latency: median 1,460 ms · p90 6,300 ms Logic 95.5% · Math 99.6% | — | Chat | 98.2%97.7%–98.6% | a | 3,228 / 3,288 | Not measured | Not measured | 31.3%answered 99% | 1,460 ms | 6,300 ms | $0.863$863 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,830 ms · p90 5,300 ms Logic 58.0% · Math 93.7% | — | Chat | 81.4%80.1%–82.7% | b | 2,678 / 3,288 | Not measured | Not measured | Not measured | 3,830 ms | 5,300 ms | $0.000$0.00 |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Latency: median 180 ms · p90 426 ms Logic 54.9% · Math 74.1% | — | Decision | 67.5%65.9%–69.1% | c | 2,221 / 3,288 | 233 right of 233 kept0 mistakes · 3,055 skipped | 0.852 | 89.1%answered 52% | 180 ms | 426 ms | $0.008$8.37 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Latency: median 266 ms · p90 284 ms Logic 54.5% · Math 70.8% | — | Decision | 65.2%63.6%–66.8% | cd | 2,144 / 3,288 | 213 right of 213 kept0 mistakes · 3,075 skipped | 0.852 | 49.2%answered 80% | 266 ms | 284 ms | $0.036$36.42 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Latency: median 327 ms · p90 518 ms Logic 54.8% · Math 70.1% | — | Decision | 64.8%63.2%–66.5% | de | 2,132 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.846 | 48.3%answered 79% | 327 ms | 518 ms | $0.016$16.13 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Latency: median 227 ms · p90 347 ms Logic 50.0% · Math 71.3% | — | Decision | 64.1%62.4%–65.7% | de | 2,106 / 3,288 | 257 right of 257 kept0 mistakes · 3,031 skipped | 0.838 | 3.0%answered 99% | 227 ms | 347 ms | $0.027$27.07 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Latency: median 382 ms · p90 391 ms Logic 52.4% · Math 68.6% | — | Decision | 63.0%61.4%–64.7% | de | 2,072 / 3,288 | 430 right of 433 kept3 wrong · 2,855 skipped | 0.827 | 58.3%answered 73% | 382 ms | 391 ms | $0.011$10.75 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 388 ms · p90 392 ms Logic 47.3% · Math 70.0% | — | Decision | 62.2%60.6%–63.9% | e | 2,046 / 3,288 | No answers at this threshold | 0.829 | 87.2%answered 50% | 388 ms | 392 ms | $0.011$11.25 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 381 ms · p90 388 ms Logic 47.3% · Math 69.9% | — | Decision | 62.1%60.5%–63.8% | e | 2,043 / 3,288 | No answers at this threshold | 0.830 | 86.7%answered 51% | 381 ms | 388 ms | $0.001$1.13 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 164 ms · p90 188 ms Logic 48.2% · Math 65.1% | — | Decision | 59.3%57.6%–61.0% | f | 1,951 / 3,288 | 263 right of 263 kept0 mistakes · 3,025 skipped | 0.810 | 15.4%answered 93% | 164 ms | 188 ms | $0.021$21.27 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 174 ms · p90 285 ms Logic 46.0% · Math 65.2% | — | Decision | 58.6%56.9%–60.3% | f | 1,928 / 3,288 | 231 right of 231 kept0 mistakes · 3,057 skipped | 0.811 | 86.0%answered 46% | 174 ms | 285 ms | $0.004$3.98 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 166 ms · p90 205 ms Logic 46.3% · Math 64.1% | — | Decision | 58.0%56.3%–59.7% | f | 1,907 / 3,288 | 313 right of 313 kept0 mistakes · 2,975 skipped | 0.817 | 10.2%answered 96% | 166 ms | 205 ms | $0.021$21.27 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Latency: median 312 ms · p90 360 ms Logic 44.4% · Math 59.4% | — | Decision | 54.3%52.6%–56.0% | g | 1,784 / 3,288 | 47 right of 47 kept0 mistakes · 3,241 skipped | 0.769 | 11.4%answered 94% | 312 ms | 360 ms | $0.009$9.07 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Latency: median 293 ms · p90 349 ms Logic 45.1% · Math 53.5% | — | Decision | 50.6%48.9%–52.3% | h | 1,664 / 3,288 | 356 right of 372 kept16 wrong · 2,916 skipped | 0.777 | 10.1%answered 95% | 293 ms | 349 ms | $0.005$5.46 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 185 ms · p90 292 ms Logic 40.8% · Math 55.6% | — | Decision | 50.5%48.8%–52.3% | h | 1,662 / 3,288 | 207 right of 208 kept1 wrong · 3,080 skipped | 0.757 | 6.4%answered 97% | 185 ms | 292 ms | $0.004$3.98 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Anthropic 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Claude Sonnet 5.5 (chat): 98.2% at a median 4,030 ms
Anthropic
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 4,030 ms · p90 10,080 ms Logic 95.5% · Math 99.7% | — | Chat | 98.2%97.7%–98.6% | a | 3,230 / 3,288 | Not measured | Not measured | 43.7%answered 99% | 4,030 ms | 10,080 ms | $7.042$7,042 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Anthropic 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Claude Opus 5.5 (chat): 98.5% at a median 3,600 ms
Claude Fable 5.1 (chat): 98.2% at a median 4,940 ms
Anthropic
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 1 setup not yet: Claude Fable 5.2 (chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,600 ms · p90 8,380 ms Logic 96.2% · Math 99.8% | — | Chat | 98.5%98.1%–98.9% | a | 3,240 / 3,288 | Not measured | Not measured | 41.3%answered 99% | 3,600 ms | 8,380 ms | $13.179$13,179 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 4,940 ms · p90 8,620 ms Logic 95.2% · Math 99.7% | — | Chat | 98.2%97.7%–98.6% | b | 3,228 / 3,288 | Not measured | Not measured | 15.7%answered 100% | 4,940 ms | 8,620 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Liquid AI OpenAI Alibaba Google 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
d1.2: 59.9% at a median 208 ms
d1.2-ultra: 63.6% at a median 217 ms
GPT-6.1 Terra (chat): 97.4% at a median 4,440 ms
GPT-6.1 Sol (chat): 98.6% at a median 5,240 ms
GPT-6 Astra (chat): 98.6% at a median 5,580 ms
Qwen4-Max (high · chat): 98.6% at a median 5,700 ms
Qwen3.8 Flash (off · chat): 97.6% at a median 6,780 ms
Gemini 4 Carbon (chat): 98.6% at a median 16,350 ms
Liquid AI OpenAI Alibaba Google
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: Grok 4.7 (chat) (missing Logic); Gemini 4 Carbon (high · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,580 ms · p90 9,580 ms Logic 96.7% · Math 99.6% | — | Chat | 98.6%98.2%–99.0% | a | 3,243 / 3,288 | Not measured | Not measured | 60.9%answered 99% | 5,580 ms | 9,580 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Latency: median 16,350 ms · p90 90,630 ms Logic 96.7% · Math 99.6% | — | Chat | 98.6%98.1%–98.9% | a | 3,242 / 3,288 | Not measured | Not measured | 53.3%answered 99% | 16,350 ms | 90,630 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,240 ms · p90 10,410 ms Logic 96.5% · Math 99.6% | — | Chat | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | 66.2%answered 99% | 5,240 ms | 10,410 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Latency: median 5,700 ms · p90 19,750 ms Logic 96.3% · Math 99.8% | — | Chat | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | 10.7%answered 100% | 5,700 ms | 19,750 ms | $2.919$2,919 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Latency: median 6,780 ms · p90 36,290 ms Logic 94.0% · Math 99.5% | — | Chat | 97.6%97.0%–98.1% | b | 3,209 / 3,288 | Not measured | Not measured | 31.3%answered 99% | 6,780 ms | 36,290 ms | $0.732$732 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,440 ms · p90 8,290 ms Logic 94.0% · Math 99.2% | — | Chat | 97.4%96.8%–97.9% | b | 3,203 / 3,288 | Not measured | Not measured | 65.0%answered 98% | 4,440 ms | 8,290 ms | $0.000$0.00 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 217 ms · p90 347 ms Logic 51.6% · Math 69.9% | — | Decision | 63.6%61.9%–65.2% | c | 2,091 / 3,288 | 128 right of 128 kept0 mistakes · 3,160 skipped | 0.856 | 90.2%answered 48% | 217 ms | 347 ms | $0.008$8.37 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 208 ms · p90 390 ms Logic 49.2% · Math 65.5% | — | Decision | 59.9%58.2%–61.6% | d | 1,970 / 3,288 | 169 right of 169 kept0 mistakes · 3,119 skipped | 0.812 | 66.9%answered 65% | 208 ms | 390 ms | $0.004$3.98 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents Blockbrain Labs newsbang tattrongvu Patrick-18 victorbennett EldanRing vLLM Semantic Router Valen-Team Base Labs Nace.AI Cloudflare Metask Lab Poindexter Labs MachinePulse Apertix AI TypeSafe REAX Logits Google Zefan Cai Inception Turion.AI Microsoft Celeris Subquadratic Sthānika AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GLiDE 0.4B: 21.5% at a median 11 ms
GLiClass v4 Edge: 23.0% at a median 12 ms
OneJev1.6-0.8B: 27.8% at a median 15 ms
d1-2B: 34.9% at a median 17 ms
GLiDE 0.8B: 31.1% at a median 20 ms
StartLux-Decision-v1.1-0.8B-Instant: 28.0% at a median 24 ms
KnowLine-0.8B-Gen1: 27.7% at a median 26 ms
GLiDE 2B: 35.2% at a median 28 ms
Perplexity Decider Flash v1.3: 52.1% at a median 29 ms
Chinese Decision Pointer 4B: 47.0% at a median 30 ms
vera-medium-omni: 52.5% at a median 31 ms
Strands Decider 2B: 26.5% at a median 35 ms
StartLux-Decision-v1.1-2B-Instant: 35.9% at a median 36 ms
Cygnet: 42.6% at a median 38 ms
Adjudex 9B v5: 56.7% at a median 38 ms
KnowLine-9B-Gen1: 54.5% at a median 38 ms
Jeffan-9B-1211: 55.8% at a median 39 ms
Perplexity Decider Flash v1.2: 51.1% at a median 41 ms
KnowLine-9B-Gen2: 54.6% at a median 42 ms
Dexy-Decision-9B-V2: 54.2% at a median 46 ms
Verdin-4B-v1.1: 47.5% at a median 46 ms
KnowLine-12B-Gen1: 60.9% at a median 51 ms
KnowLine-2B-Gen1: 34.1% at a median 51 ms
Winnow-12B: 44.4% at a median 55 ms
d3: 65.0% at a median 57 ms
Valen-Decision-1.5-9B: 51.6% at a median 59 ms
StartLux-Decision-v1.1-4B-Instant: 52.4% at a median 68 ms
KnowLine 4B Gen5.5: 51.3% at a median 76 ms
Elephant-1: 56.2% at a median 80 ms
OneJev-27B: 49.3% at a median 80 ms
GLiDE (v1.2): 64.8% at a median 81 ms
Drex Max: 64.4% at a median 81 ms
Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): 66.4% at a median 97 ms
Clef-Flash: 46.1% at a median 103 ms
KnowLine-31B-Gen2: 73.0% at a median 116 ms
Metask-Jev-4B: 36.7% at a median 117 ms
Copernicus-31B: 71.0% at a median 128 ms
d3-ultra: 65.7% at a median 129 ms
StartLux-Decision-v1.1-9B-Instant: 54.4% at a median 142 ms
Mammoth-1: 67.3% at a median 150 ms
Karpo Decision 31B: 74.4% at a median 162 ms
Cerno 9B v1.2: 51.7% at a median 163 ms
Jev 1.14: 59.3% at a median 164 ms
Jev 1.13: 58.0% at a median 166 ms
d1.1: 58.6% at a median 174 ms
GLiDE 9B: 54.5% at a median 177 ms
d1-ultra: 67.5% at a median 180 ms
d1: 50.5% at a median 185 ms
Clef: 47.6% at a median 185 ms
REAX 1.3: 71.6% at a median 200 ms
Logits Sigma-1 Duo: 69.3% at a median 207 ms
d1.2: 59.9% at a median 208 ms
d1.2-ultra: 63.6% at a median 217 ms
Gemini Decision 1 (20261111): 64.1% at a median 227 ms
Perplexity Decider 27B v1.2 (API): 64.1% at a median 230 ms
Open-Jev-27B-v1.1: 50.5% at a median 250 ms
GLiDE (API): 65.2% at a median 266 ms
Mercury Decide: 50.6% at a median 293 ms
Turion Jev 31B: 63.3% at a median 301 ms
Microsoft-Decision-1: 54.3% at a median 312 ms
Drex Max v1.1: 64.8% at a median 327 ms
Drex 2.0: 53.1% at a median 339 ms
celeris-1.1-decision: 62.1% at a median 381 ms
SubQ Snap 1.1: 63.0% at a median 382 ms
celeris-1-decision: 62.2% at a median 388 ms
Perplexity Decider 27B v1.2 (API · with web): 63.6% at a median 430 ms
GLiDE 27B: 67.3% at a median 450 ms
Sieve 31B v3: 70.5% at a median 676 ms
KnowLine-31B-Gen1.5: 71.1% at a median 811 ms
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents Blockbrain Labs newsbang tattrongvu Patrick-18 victorbennett EldanRing vLLM Semantic Router Valen-Team Base Labs Nace.AI Cloudflare Metask Lab Poindexter Labs MachinePulse Apertix AI TypeSafe REAX Logits Google Zefan Cai Inception Turion.AI Microsoft Celeris Subquadratic Sthānika AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 3,276 of 3,288); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Latency: median 162 ms · p90 198 ms Logic 60.7% · Math 81.6% | — | 74.4%72.9%–75.9% | a | 2,447 / 3,288 | 1,006 right of 1,009 kept3 wrong · 2,279 skipped | 0.882 | 5.7%answered 98% | 162 ms | 198 ms | $0.118$118estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Latency: median 116 ms · p90 177 ms Logic 58.9% · Math 80.3% | — | 73.0%71.4%–74.5% | ab | 2,399 / 3,288 | 523 right of 524 kept1 wrong · 2,764 skipped | 0.882 | 0.1%answered 100% | 116 ms | 177 ms | $0.093$92.98estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Latency: median 200 ms · p90 260 ms Logic 59.3% · Math 78.0% | — | 71.6%70.0%–73.1% | b | 2,354 / 3,288 | 229 right of 229 kept0 mistakes · 3,059 skipped | 0.878 | 3.8%answered 99% | 200 ms | 260 ms | $0.148$148estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Latency: median 811 ms · p90 1,111 ms Logic 59.0% · Math 77.4% | — | 71.1%69.5%–72.6% | bc | 2,338 / 3,288 | 553 right of 554 kept1 wrong · 2,734 skipped | 0.870 | 0.7%answered 100% | 811 ms | 1,111 ms | $0.052$51.85estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Latency: median 128 ms · p90 218 ms Logic 58.7% · Math 77.3% | — | 71.0%69.4%–72.5% | bc | 2,333 / 3,288 | 384 right of 384 kept0 mistakes · 2,904 skipped | 0.873 | 1.8%answered 99% | 128 ms | 218 ms | $0.102$102estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Latency: median 676 ms · p90 1,173 ms Logic 57.9% · Math 77.1% | — | 70.5%68.9%–72.1% | bcd | 2,319 / 3,288 | 294 right of 296 kept2 wrong · 2,992 skipped | 0.865 | 3.8%answered 99% | 676 ms | 1,173 ms | $0.047$47.29estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Latency: median 207 ms · p90 324 ms Logic 57.8% · Math 75.2% | — | 69.3%67.7%–70.8% | cde | 2,277 / 3,288 | 23 right of 23 kept0 mistakes · 3,265 skipped | 0.868 | 4.4%answered 99% | 207 ms | 324 ms | $0.165$165estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Latency: median 180 ms · p90 426 ms Logic 54.9% · Math 74.1% | — | 67.5%65.9%–69.1% | defg | 2,221 / 3,288 | 233 right of 233 kept0 mistakes · 3,055 skipped | 0.852 | 89.1%answered 52% | 180 ms | 426 ms | $0.008$8.37 |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Latency: median 450 ms · p90 587 ms Logic 58.4% · Math 72.0% | — | 67.3%65.7%–68.9% | efh | 2,214 / 3,288 | 272 right of 274 kept2 wrong · 3,014 skipped | 0.852 | 31.9%answered 89% | 450 ms | 587 ms | $0.029$28.55estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Latency: median 150 ms · p90 232 ms Logic 54.5% · Math 74.0% | — | 67.3%65.7%–68.9% | efghi | 2,213 / 3,288 | 518 right of 520 kept2 wrong · 2,768 skipped | 0.868 | 71.4%answered 69% | 150 ms | 232 ms | $0.028$27.98 list$0.120$120estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Latency: median 97 ms · p90 125 ms Logic 54.8% · Math 72.5% | — | 66.4%64.8%–68.0% | fghij | 2,184 / 3,288 | 385 right of 385 kept0 mistakes · 2,903 skipped | 0.844 | 13.5%answered 95% | 97 ms | 125 ms | $0.071$71.10estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 129 ms · p90 168 ms Logic 54.6% · Math 71.5% | — | 65.7%64.1%–67.3% | fghijk | 2,161 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.835 | 7.5%answered 97% | 129 ms | 168 ms | $0.096$95.71estimate · RTX PRO 6000 Blackwell |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Latency: median 266 ms · p90 284 ms Logic 54.5% · Math 70.8% | — | 65.2%63.6%–66.8% | fghijkl | 2,144 / 3,288 | 213 right of 213 kept0 mistakes · 3,075 skipped | 0.852 | 49.2%answered 80% | 266 ms | 284 ms | $0.036$36.42 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 57 ms · p90 86 ms Logic 54.5% · Math 70.4% | — | 65.0%63.3%–66.6% | fghijkl | 2,136 / 3,288 | 216 right of 217 kept1 wrong · 3,071 skipped | 0.858 | 9.3%answered 97% | 57 ms | 86 ms | $0.046$46.28estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Latency: median 327 ms · p90 518 ms Logic 54.8% · Math 70.1% | — | 64.8%63.2%–66.5% | fghijkl | 2,132 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.846 | 48.3%answered 79% | 327 ms | 518 ms | $0.016$16.13 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Latency: median 81 ms · p90 136 ms Logic 55.2% · Math 69.9% | — | 64.8%63.2%–66.5% | gijkl | 2,132 / 3,288 | 176 right of 176 kept0 mistakes · 3,112 skipped | 0.843 | 42.3%answered 83% | 81 ms | 136 ms | $0.010$9.71 list$0.066$66.05estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Latency: median 81 ms · p90 130 ms Logic 53.1% · Math 70.3% | — | 64.4%62.8%–66.0% | gijkl | 2,118 / 3,288 | 866 right of 884 kept18 wrong · 2,404 skipped | 0.841 | 5.7%answered 98% | 81 ms | 130 ms | $0.016$16.31 list$0.064$63.74estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Latency: median 230 ms · p90 439 ms Logic 53.3% · Math 69.8% | — | 64.1%62.5%–65.8% | jklm | 2,109 / 3,288 | 409 right of 411 kept2 wrong · 2,877 skipped | 0.849 | 10.3%answered 96% | 230 ms | 439 ms | $0.005$5.38 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Latency: median 227 ms · p90 347 ms Logic 50.0% · Math 71.3% | — | 64.1%62.4%–65.7% | hijklm | 2,106 / 3,288 | 257 right of 257 kept0 mistakes · 3,031 skipped | 0.838 | 3.0%answered 99% | 227 ms | 347 ms | $0.027$27.07 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 217 ms · p90 347 ms Logic 51.6% · Math 69.9% | — | 63.6%61.9%–65.2% | jklm | 2,091 / 3,288 | 128 right of 128 kept0 mistakes · 3,160 skipped | 0.856 | 90.2%answered 48% | 217 ms | 347 ms | $0.008$8.37 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Latency: median 430 ms · p90 707 ms Logic 52.5% · Math 69.3% | with web | 63.6%61.9%–65.2% | klm | 2,090 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.854 | 25.8%answered 90% | 430 ms | 707 ms | $1.025$1,025 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Latency: median 301 ms · p90 448 ms Logic 52.3% · Math 69.0% | — | 63.3%61.6%–64.9% | jklmn | 2,081 / 3,288 | 50 right of 50 kept0 mistakes · 3,238 skipped | 0.844 | 1.9%answered 99% | 301 ms | 448 ms | $0.020$20.01estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Latency: median 382 ms · p90 391 ms Logic 52.4% · Math 68.6% | — | 63.0%61.4%–64.7% | klmn | 2,072 / 3,288 | 430 right of 433 kept3 wrong · 2,855 skipped | 0.827 | 58.3%answered 73% | 382 ms | 391 ms | $0.011$10.75 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 388 ms · p90 392 ms Logic 47.3% · Math 70.0% | — | 62.2%60.6%–63.9% | lmno | 2,046 / 3,288 | No answers at this threshold | 0.829 | 87.2%answered 50% | 388 ms | 392 ms | $0.011$11.25 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 381 ms · p90 388 ms Logic 47.3% · Math 69.9% | — | 62.1%60.5%–63.8% | lmnop | 2,043 / 3,288 | No answers at this threshold | 0.830 | 86.7%answered 51% | 381 ms | 388 ms | $0.001$1.13 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Latency: median 51 ms · p90 76 ms Logic 50.6% · Math 66.3% | — | 60.9%59.2%–62.5% | mnopq | 2,002 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.810 | 4.5%answered 98% | 51 ms | 76 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 208 ms · p90 390 ms Logic 49.2% · Math 65.5% | — | 59.9%58.2%–61.6% | nopqr | 1,970 / 3,288 | 169 right of 169 kept0 mistakes · 3,119 skipped | 0.812 | 66.9%answered 65% | 208 ms | 390 ms | $0.004$3.98 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 164 ms · p90 188 ms Logic 48.2% · Math 65.1% | — | 59.3%57.6%–61.0% | opqrs | 1,951 / 3,288 | 263 right of 263 kept0 mistakes · 3,025 skipped | 0.810 | 15.4%answered 93% | 164 ms | 188 ms | $0.021$21.27 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 174 ms · p90 285 ms Logic 46.0% · Math 65.2% | — | 58.6%56.9%–60.3% | pqrs | 1,928 / 3,288 | 231 right of 231 kept0 mistakes · 3,057 skipped | 0.811 | 86.0%answered 46% | 174 ms | 285 ms | $0.004$3.98 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 166 ms · p90 205 ms Logic 46.3% · Math 64.1% | — | 58.0%56.3%–59.7% | qrst | 1,907 / 3,288 | 313 right of 313 kept0 mistakes · 2,975 skipped | 0.817 | 10.2%answered 96% | 166 ms | 205 ms | $0.021$21.27 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 38 ms · p90 45 ms Logic 46.4% · Math 62.1% | — | 56.7%55.0%–58.4% | rstu | 1,865 / 3,288 | 251 right of 253 kept2 wrong · 3,035 skipped | 0.802 | 7.0%answered 97% | 38 ms | 45 ms | $0.028$28.19estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Latency: median 80 ms · p90 94 ms Logic 46.4% · Math 61.3% | — | 56.2%54.5%–57.9% | stu | 1,849 / 3,288 | 235 right of 236 kept1 wrong · 3,052 skipped | 0.797 | 69.4%answered 60% | 80 ms | 94 ms | $0.010$9.79 list$0.058$58.27estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 39 ms · p90 53 ms Logic 48.2% · Math 59.8% | — | 55.8%54.1%–57.5% | stu | 1,836 / 3,288 | 74 right of 74 kept0 mistakes · 3,214 skipped | 0.798 | 13.5%answered 94% | 39 ms | 53 ms | $0.032$32.40estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 42 ms · p90 57 ms Logic 43.9% · Math 60.1% | — | 54.6%52.9%–56.3% | tuv | 1,794 / 3,288 | No answers at this threshold | 0.786 | 10.4%answered 95% | 42 ms | 57 ms | $0.034$34.08estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Latency: median 177 ms · p90 222 ms Logic 47.8% · Math 57.9% | — | 54.5%52.8%–56.2% | uv | 1,791 / 3,288 | 105 right of 105 kept0 mistakes · 3,183 skipped | 0.786 | 26.9%answered 86% | 177 ms | 222 ms | $0.011$11.45estimate · DGX Spark |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 38 ms · p90 55 ms Logic 44.3% · Math 59.8% | — | 54.5%52.8%–56.2% | uv | 1,791 / 3,288 | 153 right of 153 kept0 mistakes · 3,135 skipped | 0.782 | 3.4%answered 98% | 38 ms | 55 ms | $0.030$29.87estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Latency: median 142 ms · p90 165 ms Logic 46.6% · Math 58.4% | — | 54.4%52.7%–56.1% | uv | 1,788 / 3,288 | 63 right of 63 kept0 mistakes · 3,225 skipped | 0.779 | 9.6%answered 95% | 142 ms | 165 ms | $0.009$8.92estimate · DGX Spark |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Latency: median 312 ms · p90 360 ms Logic 44.4% · Math 59.4% | — | 54.3%52.6%–56.0% | uvw | 1,784 / 3,288 | 47 right of 47 kept0 mistakes · 3,241 skipped | 0.769 | 11.4%answered 94% | 312 ms | 360 ms | $0.009$9.07 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Latency: median 46 ms · p90 51 ms Logic 45.3% · Math 58.8% | — | 54.2%52.5%–55.9% | uvw | 1,782 / 3,288 | 263 right of 266 kept3 wrong · 3,022 skipped | 0.795 | 6.1%answered 97% | 46 ms | 51 ms | $0.033$32.61estimate · RTX PRO 6000 Blackwell |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Latency: median 339 ms · p90 472 ms Logic 41.9% · Math 59.0% | — | 53.1%51.4%–54.8% | uvwx | 1,747 / 3,288 | 207 right of 207 kept0 mistakes · 3,081 skipped | 0.789 | 28.1%answered 85% | 339 ms | 472 ms | $0.012$12.17 |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Latency: median 31 ms · p90 36 ms Logic 44.1% · Math 56.8% | — | 52.5%50.8%–54.2% | vwxy | 1,725 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.777 | 7.9%answered 96% | 31 ms | 36 ms | $0.024$24.19estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Latency: median 68 ms · p90 80 ms Logic 45.9% · Math 55.8% | — | 52.4%50.7%–54.1% | vwxy | 1,724 / 3,288 | 117 right of 117 kept0 mistakes · 3,171 skipped | 0.763 | 14.4%answered 93% | 68 ms | 80 ms | $0.004$4.28estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Latency: median 29 ms · p90 35 ms Logic 44.7% · Math 55.9% | — | 52.1%50.4%–53.8% | vwxy | 1,713 / 3,288 | 130 right of 131 kept1 wrong · 3,157 skipped | 0.782 | 15.0%answered 92% | 29 ms | 35 ms | $0.024$23.77estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Latency: median 163 ms · p90 240 ms Logic 43.3% · Math 56.0% | — | 51.7%50.0%–53.4% | vwxy | 1,699 / 3,288 | 270 right of 281 kept11 wrong · 3,007 skipped | 0.776 | 3.1%answered 99% | 163 ms | 240 ms | $0.011$10.82estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 59 ms · p90 65 ms Logic 44.4% · Math 55.3% | — | 51.6%49.9%–53.3% | vwxy | 1,696 / 3,288 | 165 right of 167 kept2 wrong · 3,121 skipped | 0.765 | 7.1%answered 96% | 59 ms | 65 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Latency: median 76 ms · p90 104 ms Logic 42.3% · Math 56.0% | — | 51.3%49.6%–53.0% | vwxyz | 1,687 / 3,288 | 71 right of 72 kept1 wrong · 3,216 skipped | 0.752 | 4.4%answered 98% | 76 ms | 104 ms | $0.005$5.02estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Latency: median 41 ms · p90 51 ms Logic 43.1% · Math 55.2% | — | 51.1%49.4%–52.8% | vwxyz | 1,680 / 3,288 | 185 right of 190 kept5 wrong · 3,098 skipped | 0.769 | 18.9%answered 90% | 41 ms | 51 ms | $0.003$2.72 list$0.032$31.55estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Latency: median 293 ms · p90 349 ms Logic 45.1% · Math 53.5% | — | 50.6%48.9%–52.3% | wxyzA | 1,664 / 3,288 | 356 right of 372 kept16 wrong · 2,916 skipped | 0.777 | 10.1%answered 95% | 293 ms | 349 ms | $0.005$5.46 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 185 ms · p90 292 ms Logic 40.8% · Math 55.6% | — | 50.5%48.8%–52.3% | wxyzA | 1,662 / 3,288 | 207 right of 208 kept1 wrong · 3,080 skipped | 0.757 | 6.4%answered 97% | 185 ms | 292 ms | $0.004$3.98 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Latency: median 250 ms · p90 482 ms Logic 42.6% · Math 54.6% | — | 50.5%48.7%–52.2% | xyzA | 1,659 / 3,288 | No answers at this threshold | 0.759 | 14.5%answered 92% | 250 ms | 482 ms | $0.212$212estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Latency: median 80 ms · p90 134 ms Logic 40.6% · Math 53.8% | — | 49.3%47.6%–51.0% | yzAB | 1,620 / 3,288 | 30 right of 30 kept0 mistakes · 3,258 skipped | 0.770 | 1.6%answered 99% | 80 ms | 134 ms | $0.069$69.42estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Latency: median 185 ms · p90 216 ms Logic 40.8% · Math 51.1% | — | 47.6%45.9%–49.3% | zABC | 1,564 / 3,288 | 18 right of 18 kept0 mistakes · 3,270 skipped | 0.747 | 2.4%answered 99% | 185 ms | 216 ms | $0.081$81.46 list$0.135$135estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 46 ms · p90 52 ms Logic 39.7% · Math 51.5% | — | 47.5%45.8%–49.2% | ABC | 1,561 / 3,288 | 161 right of 165 kept4 wrong · 3,123 skipped | 0.747 | 8.9%answered 95% | 46 ms | 52 ms | $0.033$33.45estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 30 ms · p90 42 ms Logic 40.4% · Math 50.4% | — | 47.0%45.3%–48.7% | ABC | 1,545 / 3,288 | 188 right of 194 kept6 wrong · 3,094 skipped | 0.743 | 7.8%answered 96% | 30 ms | 42 ms | $0.025$25.03estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Latency: median 103 ms · p90 131 ms Logic 38.9% · Math 49.8% | — | 46.1%44.4%–47.8% | BCD | 1,515 / 3,288 | No answers at this threshold | 0.707 | 6.9%answered 96% | 103 ms | 131 ms | $0.013$12.90 list$0.075$75.10estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Latency: median 55 ms · p90 72 ms Logic 39.7% · Math 46.9% | — | 44.4%42.7%–46.1% | CD | 1,461 / 3,288 | 52 right of 53 kept1 wrong · 3,235 skipped | 0.737 | 17.1%answered 90% | 55 ms | 72 ms | $0.043$42.91estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Latency: median 38 ms · p90 44 ms Logic 39.3% · Math 44.3% | — | 42.6%40.9%–44.3% | D | 1,401 / 3,288 | No answers at this threshold | 0.729 | 7.3%answered 96% | 38 ms | 44 ms | $0.029$29.34 list$0.029$28.61estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Latency: median 117 ms · p90 146 ms Logic 32.1% · Math 39.2% | — | 36.7%35.1%–38.4% | E | 1,208 / 3,288 | No answers at this threshold | 0.659 | 2.0%answered 99% | 117 ms | 146 ms | $0.086$85.83estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Latency: median 36 ms · p90 42 ms Logic 30.8% · Math 38.7% | — | 35.9%34.3%–37.6% | E | 1,182 / 3,288 | 4 right of 4 kept0 mistakes · 3,284 skipped | 0.675 | 16.4%answered 89% | 36 ms | 42 ms | $0.002$2.36estimate · DGX Spark |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Latency: median 28 ms · p90 37 ms Logic 32.3% · Math 36.8% | — | 35.2%33.6%–36.9% | E | 1,158 / 3,288 | 29 right of 30 kept1 wrong · 3,258 skipped | 0.637 | 23.1%answered 84% | 28 ms | 37 ms | $0.002$1.94estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Latency: median 17 ms · p90 21 ms Logic 30.7% · Math 37.1% | — | 34.9%33.3%–36.5% | EF | 1,147 / 3,288 | 7 right of 7 kept0 mistakes · 3,281 skipped | 0.648 | 82.5%answered 30% | 17 ms | 21 ms | $0.014$14.30estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Latency: median 51 ms · p90 63 ms Logic 31.7% · Math 35.3% | — | 34.1%32.5%–35.7% | EF | 1,120 / 3,288 | 16 right of 16 kept0 mistakes · 3,272 skipped | 0.626 | 6.8%answered 95% | 51 ms | 63 ms | $0.003$3.40estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Latency: median 20 ms · p90 25 ms Logic 29.2% · Math 32.1% | — | 31.1%29.5%–32.7% | FG | 1,022 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.602 | 22.7%answered 83% | 20 ms | 25 ms | $0.001$1.38estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 24 ms · p90 29 ms Logic 24.7% · Math 29.7% | — | 28.0%26.5%–29.5% | GH | 920 / 3,288 | No answers at this threshold | 0.605 | 14.9%answered 89% | 24 ms | 29 ms | $0.002$1.62estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Latency: median 15 ms · p90 17 ms Logic 27.0% · Math 28.2% | — | 27.8%26.3%–29.4% | GH | 915 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.576 | 2.0%answered 99% | 15 ms | 17 ms | $0.001$1.11estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 26 ms · p90 35 ms Logic 26.7% · Math 28.2% | — | 27.7%26.2%–29.2% | GH | 910 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.590 | 0.6%answered 100% | 26 ms | 35 ms | $0.002$1.80estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Latency: median 35 ms · p90 42 ms Logic 22.4% · Math 28.7% | — | 26.5%25.0%–28.1% | HI | 872 / 3,288 | No answers at this threshold | 0.573 | 5.0%answered 96% | 35 ms | 42 ms | $0.027$26.72estimate · RTX PRO 6000 Blackwell |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Latency: median 12 ms · p90 23 ms Logic 23.6% · Math 22.7% | — | 23.0%21.6%–24.4% | IJ | 755 / 3,288 | No answers at this threshold | 0.511 | 15.6%answered 87% | 12 ms | 23 ms | $0.001$1.01estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Latency: median 11 ms · p90 14 ms Logic 23.2% · Math 20.6% | — | 21.5%20.1%–22.9% | J | 706 / 3,288 | No answers at this threshold | 0.520 | 73.9%answered 32% | 11 ms | 14 ms | $0.001$0.83estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents victorbennett Metask Lab 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GLiDE 0.4B: 21.5% at a median 11 ms
GLiClass v4 Edge: 23.0% at a median 12 ms
OneJev1.6-0.8B: 27.8% at a median 15 ms
d1-2B: 34.9% at a median 17 ms
GLiDE 0.8B: 31.1% at a median 20 ms
StartLux-Decision-v1.1-0.8B-Instant: 28.0% at a median 24 ms
KnowLine-0.8B-Gen1: 27.7% at a median 26 ms
GLiDE 2B: 35.2% at a median 28 ms
Perplexity Decider Flash v1.3: 52.1% at a median 29 ms
Chinese Decision Pointer 4B: 47.0% at a median 30 ms
vera-medium-omni: 52.5% at a median 31 ms
Strands Decider 2B: 26.5% at a median 35 ms
StartLux-Decision-v1.1-2B-Instant: 35.9% at a median 36 ms
Perplexity Decider Flash v1.2: 51.1% at a median 41 ms
Verdin-4B-v1.1: 47.5% at a median 46 ms
KnowLine-2B-Gen1: 34.1% at a median 51 ms
StartLux-Decision-v1.1-4B-Instant: 52.4% at a median 68 ms
KnowLine 4B Gen5.5: 51.3% at a median 76 ms
Metask-Jev-4B: 36.7% at a median 117 ms
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents victorbennett Metask Lab
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Latency: median 31 ms · p90 36 ms Logic 44.1% · Math 56.8% | — | 52.5%50.8%–54.2% | a | 1,725 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.777 | 7.9%answered 96% | 31 ms | 36 ms | $0.024$24.19estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Latency: median 68 ms · p90 80 ms Logic 45.9% · Math 55.8% | — | 52.4%50.7%–54.1% | a | 1,724 / 3,288 | 117 right of 117 kept0 mistakes · 3,171 skipped | 0.763 | 14.4%answered 93% | 68 ms | 80 ms | $0.004$4.28estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Latency: median 29 ms · p90 35 ms Logic 44.7% · Math 55.9% | — | 52.1%50.4%–53.8% | a | 1,713 / 3,288 | 130 right of 131 kept1 wrong · 3,157 skipped | 0.782 | 15.0%answered 92% | 29 ms | 35 ms | $0.024$23.77estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Latency: median 76 ms · p90 104 ms Logic 42.3% · Math 56.0% | — | 51.3%49.6%–53.0% | a | 1,687 / 3,288 | 71 right of 72 kept1 wrong · 3,216 skipped | 0.752 | 4.4%answered 98% | 76 ms | 104 ms | $0.005$5.02estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Latency: median 41 ms · p90 51 ms Logic 43.1% · Math 55.2% | — | 51.1%49.4%–52.8% | a | 1,680 / 3,288 | 185 right of 190 kept5 wrong · 3,098 skipped | 0.769 | 18.9%answered 90% | 41 ms | 51 ms | $0.003$2.72 list$0.032$31.55estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 46 ms · p90 52 ms Logic 39.7% · Math 51.5% | — | 47.5%45.8%–49.2% | b | 1,561 / 3,288 | 161 right of 165 kept4 wrong · 3,123 skipped | 0.747 | 8.9%answered 95% | 46 ms | 52 ms | $0.033$33.45estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 30 ms · p90 42 ms Logic 40.4% · Math 50.4% | — | 47.0%45.3%–48.7% | b | 1,545 / 3,288 | 188 right of 194 kept6 wrong · 3,094 skipped | 0.743 | 7.8%answered 96% | 30 ms | 42 ms | $0.025$25.03estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Latency: median 117 ms · p90 146 ms Logic 32.1% · Math 39.2% | — | 36.7%35.1%–38.4% | c | 1,208 / 3,288 | No answers at this threshold | 0.659 | 2.0%answered 99% | 117 ms | 146 ms | $0.086$85.83estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Latency: median 36 ms · p90 42 ms Logic 30.8% · Math 38.7% | — | 35.9%34.3%–37.6% | c | 1,182 / 3,288 | 4 right of 4 kept0 mistakes · 3,284 skipped | 0.675 | 16.4%answered 89% | 36 ms | 42 ms | $0.002$2.36estimate · DGX Spark |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Latency: median 28 ms · p90 37 ms Logic 32.3% · Math 36.8% | — | 35.2%33.6%–36.9% | c | 1,158 / 3,288 | 29 right of 30 kept1 wrong · 3,258 skipped | 0.637 | 23.1%answered 84% | 28 ms | 37 ms | $0.002$1.94estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Latency: median 17 ms · p90 21 ms Logic 30.7% · Math 37.1% | — | 34.9%33.3%–36.5% | c | 1,147 / 3,288 | 7 right of 7 kept0 mistakes · 3,281 skipped | 0.648 | 82.5%answered 30% | 17 ms | 21 ms | $0.014$14.30estimate · RTX PRO 6000 Blackwell |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Latency: median 51 ms · p90 63 ms Logic 31.7% · Math 35.3% | — | 34.1%32.5%–35.7% | cd | 1,120 / 3,288 | 16 right of 16 kept0 mistakes · 3,272 skipped | 0.626 | 6.8%answered 95% | 51 ms | 63 ms | $0.003$3.40estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Latency: median 20 ms · p90 25 ms Logic 29.2% · Math 32.1% | — | 31.1%29.5%–32.7% | d | 1,022 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.602 | 22.7%answered 83% | 20 ms | 25 ms | $0.001$1.38estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 24 ms · p90 29 ms Logic 24.7% · Math 29.7% | — | 28.0%26.5%–29.5% | e | 920 / 3,288 | No answers at this threshold | 0.605 | 14.9%answered 89% | 24 ms | 29 ms | $0.002$1.62estimate · DGX Spark |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Latency: median 15 ms · p90 17 ms Logic 27.0% · Math 28.2% | — | 27.8%26.3%–29.4% | e | 915 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.576 | 2.0%answered 99% | 15 ms | 17 ms | $0.001$1.11estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 26 ms · p90 35 ms Logic 26.7% · Math 28.2% | — | 27.7%26.2%–29.2% | e | 910 / 3,288 | 2 right of 2 kept0 mistakes · 3,286 skipped | 0.590 | 0.6%answered 100% | 26 ms | 35 ms | $0.002$1.80estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Latency: median 35 ms · p90 42 ms Logic 22.4% · Math 28.7% | — | 26.5%25.0%–28.1% | e | 872 / 3,288 | No answers at this threshold | 0.573 | 5.0%answered 96% | 35 ms | 42 ms | $0.027$26.72estimate · RTX PRO 6000 Blackwell |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Latency: median 12 ms · p90 23 ms Logic 23.6% · Math 22.7% | — | 23.0%21.6%–24.4% | f | 755 / 3,288 | No answers at this threshold | 0.511 | 15.6%answered 87% | 12 ms | 23 ms | $0.001$1.01estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Latency: median 11 ms · p90 14 ms Logic 23.2% · Math 20.6% | — | 21.5%20.1%–22.9% | f | 706 / 3,288 | No answers at this threshold | 0.520 | 73.9%answered 32% | 11 ms | 14 ms | $0.001$0.83estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Blockbrain Labs newsbang PelaAI tattrongvu Patrick-18 EldanRing Valen-Team Base Labs Cloudflare StartLux Apertix AI Fastino 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Cygnet: 42.6% at a median 38 ms
Adjudex 9B v5: 56.7% at a median 38 ms
KnowLine-9B-Gen1: 54.5% at a median 38 ms
Jeffan-9B-1211: 55.8% at a median 39 ms
KnowLine-9B-Gen2: 54.6% at a median 42 ms
Dexy-Decision-9B-V2: 54.2% at a median 46 ms
KnowLine-12B-Gen1: 60.9% at a median 51 ms
Winnow-12B: 44.4% at a median 55 ms
Valen-Decision-1.5-9B: 51.6% at a median 59 ms
Elephant-1: 56.2% at a median 80 ms
Clef-Flash: 46.1% at a median 103 ms
StartLux-Decision-v1.1-9B-Instant: 54.4% at a median 142 ms
Cerno 9B v1.2: 51.7% at a median 163 ms
GLiDE 9B: 54.5% at a median 177 ms
Blockbrain Labs newsbang PelaAI tattrongvu Patrick-18 EldanRing Valen-Team Base Labs Cloudflare StartLux Apertix AI Fastino
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Latency: median 51 ms · p90 76 ms Logic 50.6% · Math 66.3% | — | 60.9%59.2%–62.5% | a | 2,002 / 3,288 | 13 right of 13 kept0 mistakes · 3,275 skipped | 0.810 | 4.5%answered 98% | 51 ms | 76 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 38 ms · p90 45 ms Logic 46.4% · Math 62.1% | — | 56.7%55.0%–58.4% | b | 1,865 / 3,288 | 251 right of 253 kept2 wrong · 3,035 skipped | 0.802 | 7.0%answered 97% | 38 ms | 45 ms | $0.028$28.19estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Latency: median 80 ms · p90 94 ms Logic 46.4% · Math 61.3% | — | 56.2%54.5%–57.9% | b | 1,849 / 3,288 | 235 right of 236 kept1 wrong · 3,052 skipped | 0.797 | 69.4%answered 60% | 80 ms | 94 ms | $0.010$9.79 list$0.058$58.27estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 39 ms · p90 53 ms Logic 48.2% · Math 59.8% | — | 55.8%54.1%–57.5% | b | 1,836 / 3,288 | 74 right of 74 kept0 mistakes · 3,214 skipped | 0.798 | 13.5%answered 94% | 39 ms | 53 ms | $0.032$32.40estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 42 ms · p90 57 ms Logic 43.9% · Math 60.1% | — | 54.6%52.9%–56.3% | b | 1,794 / 3,288 | No answers at this threshold | 0.786 | 10.4%answered 95% | 42 ms | 57 ms | $0.034$34.08estimate · RTX PRO 6000 Blackwell |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Latency: median 177 ms · p90 222 ms Logic 47.8% · Math 57.9% | — | 54.5%52.8%–56.2% | bc | 1,791 / 3,288 | 105 right of 105 kept0 mistakes · 3,183 skipped | 0.786 | 26.9%answered 86% | 177 ms | 222 ms | $0.011$11.45estimate · DGX Spark |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 38 ms · p90 55 ms Logic 44.3% · Math 59.8% | — | 54.5%52.8%–56.2% | b | 1,791 / 3,288 | 153 right of 153 kept0 mistakes · 3,135 skipped | 0.782 | 3.4%answered 98% | 38 ms | 55 ms | $0.030$29.87estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Latency: median 142 ms · p90 165 ms Logic 46.6% · Math 58.4% | — | 54.4%52.7%–56.1% | bc | 1,788 / 3,288 | 63 right of 63 kept0 mistakes · 3,225 skipped | 0.779 | 9.6%answered 95% | 142 ms | 165 ms | $0.009$8.92estimate · DGX Spark |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Latency: median 46 ms · p90 51 ms Logic 45.3% · Math 58.8% | — | 54.2%52.5%–55.9% | bc | 1,782 / 3,288 | 263 right of 266 kept3 wrong · 3,022 skipped | 0.795 | 6.1%answered 97% | 46 ms | 51 ms | $0.033$32.61estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Latency: median 163 ms · p90 240 ms Logic 43.3% · Math 56.0% | — | 51.7%50.0%–53.4% | c | 1,699 / 3,288 | 270 right of 281 kept11 wrong · 3,007 skipped | 0.776 | 3.1%answered 99% | 163 ms | 240 ms | $0.011$10.82estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 59 ms · p90 65 ms Logic 44.4% · Math 55.3% | — | 51.6%49.9%–53.3% | c | 1,696 / 3,288 | 165 right of 167 kept2 wrong · 3,121 skipped | 0.765 | 7.1%answered 96% | 59 ms | 65 ms | $0.041$41.44estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Latency: median 103 ms · p90 131 ms Logic 38.9% · Math 49.8% | — | 46.1%44.4%–47.8% | d | 1,515 / 3,288 | No answers at this threshold | 0.707 | 6.9%answered 96% | 103 ms | 131 ms | $0.013$12.90 list$0.075$75.10estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Latency: median 55 ms · p90 72 ms Logic 39.7% · Math 46.9% | — | 44.4%42.7%–46.1% | de | 1,461 / 3,288 | 52 right of 53 kept1 wrong · 3,235 skipped | 0.737 | 17.1%answered 90% | 55 ms | 72 ms | $0.043$42.91estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Latency: median 38 ms · p90 44 ms Logic 39.3% · Math 44.3% | — | 42.6%40.9%–44.3% | e | 1,401 / 3,288 | No answers at this threshold | 0.729 | 7.3%answered 96% | 38 ms | 44 ms | $0.029$29.34 list$0.029$28.61estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
vLLM Semantic Router OmniJev Fastino Nace.AI Perplexity PelaAI Poindexter Labs Base Labs MachinePulse Cloudflare Zefan Cai Turion.AI Sthānika AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
d3: 65.0% at a median 57 ms
OneJev-27B: 49.3% at a median 80 ms
GLiDE (v1.2): 64.8% at a median 81 ms
Drex Max: 64.4% at a median 81 ms
Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): 66.4% at a median 97 ms
KnowLine-31B-Gen2: 73.0% at a median 116 ms
Copernicus-31B: 71.0% at a median 128 ms
d3-ultra: 65.7% at a median 129 ms
Mammoth-1: 67.3% at a median 150 ms
Karpo Decision 31B: 74.4% at a median 162 ms
Clef: 47.6% at a median 185 ms
Perplexity Decider 27B v1.2 (API): 64.1% at a median 230 ms
Open-Jev-27B-v1.1: 50.5% at a median 250 ms
Turion Jev 31B: 63.3% at a median 301 ms
Drex 2.0: 53.1% at a median 339 ms
Perplexity Decider 27B v1.2 (API · with web): 63.6% at a median 430 ms
GLiDE 27B: 67.3% at a median 450 ms
Sieve 31B v3: 70.5% at a median 676 ms
KnowLine-31B-Gen1.5: 71.1% at a median 811 ms
vLLM Semantic Router OmniJev Fastino Nace.AI Perplexity PelaAI Poindexter Labs Base Labs MachinePulse Cloudflare Zefan Cai Turion.AI Sthānika AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Latency: median 162 ms · p90 198 ms Logic 60.7% · Math 81.6% | — | 74.4%72.9%–75.9% | a | 2,447 / 3,288 | 1,006 right of 1,009 kept3 wrong · 2,279 skipped | 0.882 | 5.7%answered 98% | 162 ms | 198 ms | $0.118$118estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Latency: median 116 ms · p90 177 ms Logic 58.9% · Math 80.3% | — | 73.0%71.4%–74.5% | a | 2,399 / 3,288 | 523 right of 524 kept1 wrong · 2,764 skipped | 0.882 | 0.1%answered 100% | 116 ms | 177 ms | $0.093$92.98estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Latency: median 811 ms · p90 1,111 ms Logic 59.0% · Math 77.4% | — | 71.1%69.5%–72.6% | b | 2,338 / 3,288 | 553 right of 554 kept1 wrong · 2,734 skipped | 0.870 | 0.7%answered 100% | 811 ms | 1,111 ms | $0.052$51.85estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Latency: median 128 ms · p90 218 ms Logic 58.7% · Math 77.3% | — | 71.0%69.4%–72.5% | b | 2,333 / 3,288 | 384 right of 384 kept0 mistakes · 2,904 skipped | 0.873 | 1.8%answered 99% | 128 ms | 218 ms | $0.102$102estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Latency: median 676 ms · p90 1,173 ms Logic 57.9% · Math 77.1% | — | 70.5%68.9%–72.1% | b | 2,319 / 3,288 | 294 right of 296 kept2 wrong · 2,992 skipped | 0.865 | 3.8%answered 99% | 676 ms | 1,173 ms | $0.047$47.29estimate · DGX Spark |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Latency: median 450 ms · p90 587 ms Logic 58.4% · Math 72.0% | — | 67.3%65.7%–68.9% | c | 2,214 / 3,288 | 272 right of 274 kept2 wrong · 3,014 skipped | 0.852 | 31.9%answered 89% | 450 ms | 587 ms | $0.029$28.55estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Latency: median 150 ms · p90 232 ms Logic 54.5% · Math 74.0% | — | 67.3%65.7%–68.9% | cd | 2,213 / 3,288 | 518 right of 520 kept2 wrong · 2,768 skipped | 0.868 | 71.4%answered 69% | 150 ms | 232 ms | $0.028$27.98 list$0.120$120estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Latency: median 97 ms · p90 125 ms Logic 54.8% · Math 72.5% | — | 66.4%64.8%–68.0% | cde | 2,184 / 3,288 | 385 right of 385 kept0 mistakes · 2,903 skipped | 0.844 | 13.5%answered 95% | 97 ms | 125 ms | $0.071$71.10estimate · RTX PRO 6000 Blackwell |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 129 ms · p90 168 ms Logic 54.6% · Math 71.5% | — | 65.7%64.1%–67.3% | cdef | 2,161 / 3,288 | 39 right of 39 kept0 mistakes · 3,249 skipped | 0.835 | 7.5%answered 97% | 129 ms | 168 ms | $0.096$95.71estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 57 ms · p90 86 ms Logic 54.5% · Math 70.4% | — | 65.0%63.3%–66.6% | def | 2,136 / 3,288 | 216 right of 217 kept1 wrong · 3,071 skipped | 0.858 | 9.3%answered 97% | 57 ms | 86 ms | $0.046$46.28estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Latency: median 81 ms · p90 136 ms Logic 55.2% · Math 69.9% | — | 64.8%63.2%–66.5% | def | 2,132 / 3,288 | 176 right of 176 kept0 mistakes · 3,112 skipped | 0.843 | 42.3%answered 83% | 81 ms | 136 ms | $0.010$9.71 list$0.066$66.05estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Latency: median 81 ms · p90 130 ms Logic 53.1% · Math 70.3% | — | 64.4%62.8%–66.0% | ef | 2,118 / 3,288 | 866 right of 884 kept18 wrong · 2,404 skipped | 0.841 | 5.7%answered 98% | 81 ms | 130 ms | $0.016$16.31 list$0.064$63.74estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Latency: median 230 ms · p90 439 ms Logic 53.3% · Math 69.8% | — | 64.1%62.5%–65.8% | f | 2,109 / 3,288 | 409 right of 411 kept2 wrong · 2,877 skipped | 0.849 | 10.3%answered 96% | 230 ms | 439 ms | $0.005$5.38 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Latency: median 430 ms · p90 707 ms Logic 52.5% · Math 69.3% | with web | 63.6%61.9%–65.2% | f | 2,090 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.854 | 25.8%answered 90% | 430 ms | 707 ms | $1.025$1,025 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Latency: median 301 ms · p90 448 ms Logic 52.3% · Math 69.0% | — | 63.3%61.6%–64.9% | f | 2,081 / 3,288 | 50 right of 50 kept0 mistakes · 3,238 skipped | 0.844 | 1.9%answered 99% | 301 ms | 448 ms | $0.020$20.01estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Latency: median 339 ms · p90 472 ms Logic 41.9% · Math 59.0% | — | 53.1%51.4%–54.8% | g | 1,747 / 3,288 | 207 right of 207 kept0 mistakes · 3,081 skipped | 0.789 | 28.1%answered 85% | 339 ms | 472 ms | $0.012$12.17 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Latency: median 250 ms · p90 482 ms Logic 42.6% · Math 54.6% | — | 50.5%48.7%–52.2% | gh | 1,659 / 3,288 | No answers at this threshold | 0.759 | 14.5%answered 92% | 250 ms | 482 ms | $0.212$212estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Latency: median 80 ms · p90 134 ms Logic 40.6% · Math 53.8% | — | 49.3%47.6%–51.0% | h | 1,620 / 3,288 | 30 right of 30 kept0 mistakes · 3,258 skipped | 0.770 | 1.6%answered 99% | 80 ms | 134 ms | $0.069$69.42estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Latency: median 185 ms · p90 216 ms Logic 40.8% · Math 51.1% | — | 47.6%45.9%–49.3% | h | 1,564 / 3,288 | 18 right of 18 kept0 mistakes · 3,270 skipped | 0.747 | 2.4%answered 99% | 185 ms | 216 ms | $0.081$81.46 list$0.135$135estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
REAX Logits 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
REAX 1.3: 71.6% at a median 200 ms
Logits Sigma-1 Duo: 69.3% at a median 207 ms
REAX Logits
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Latency: median 200 ms · p90 260 ms Logic 59.3% · Math 78.0% | — | 71.6%70.0%–73.1% | a | 2,354 / 3,288 | 229 right of 229 kept0 mistakes · 3,059 skipped | 0.878 | 3.8%answered 99% | 200 ms | 260 ms | $0.148$148estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Latency: median 207 ms · p90 324 ms Logic 57.8% · Math 75.2% | — | 69.3%67.7%–70.8% | b | 2,277 / 3,288 | 23 right of 23 kept0 mistakes · 3,265 skipped | 0.868 | 4.4%answered 99% | 207 ms | 324 ms | $0.165$165estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
TypeSafe Liquid AI Google Fastino Inception Microsoft Nace.AI Celeris Subquadratic 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Jev 1.14: 59.3% at a median 164 ms
Jev 1.13: 58.0% at a median 166 ms
d1.1: 58.6% at a median 174 ms
d1-ultra: 67.5% at a median 180 ms
d1: 50.5% at a median 185 ms
Gemini Decision 1 (20261111): 64.1% at a median 227 ms
GLiDE (API): 65.2% at a median 266 ms
Mercury Decide: 50.6% at a median 293 ms
Microsoft-Decision-1: 54.3% at a median 312 ms
Drex Max v1.1: 64.8% at a median 327 ms
celeris-1.1-decision: 62.1% at a median 381 ms
SubQ Snap 1.1: 63.0% at a median 382 ms
celeris-1-decision: 62.2% at a median 388 ms
TypeSafe Liquid AI Google Fastino Inception Microsoft Nace.AI Celeris Subquadratic
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 3,276 of 3,288); shown once.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Latency: median 180 ms · p90 426 ms Logic 54.9% · Math 74.1% | — | 67.5%65.9%–69.1% | a | 2,221 / 3,288 | 233 right of 233 kept0 mistakes · 3,055 skipped | 0.852 | 89.1%answered 52% | 180 ms | 426 ms | $0.008$8.37 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Latency: median 266 ms · p90 284 ms Logic 54.5% · Math 70.8% | — | 65.2%63.6%–66.8% | ab | 2,144 / 3,288 | 213 right of 213 kept0 mistakes · 3,075 skipped | 0.852 | 49.2%answered 80% | 266 ms | 284 ms | $0.036$36.42 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Latency: median 327 ms · p90 518 ms Logic 54.8% · Math 70.1% | — | 64.8%63.2%–66.5% | bc | 2,132 / 3,288 | 404 right of 405 kept1 wrong · 2,883 skipped | 0.846 | 48.3%answered 79% | 327 ms | 518 ms | $0.016$16.13 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Latency: median 227 ms · p90 347 ms Logic 50.0% · Math 71.3% | — | 64.1%62.4%–65.7% | bc | 2,106 / 3,288 | 257 right of 257 kept0 mistakes · 3,031 skipped | 0.838 | 3.0%answered 99% | 227 ms | 347 ms | $0.027$27.07 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Latency: median 382 ms · p90 391 ms Logic 52.4% · Math 68.6% | — | 63.0%61.4%–64.7% | bc | 2,072 / 3,288 | 430 right of 433 kept3 wrong · 2,855 skipped | 0.827 | 58.3%answered 73% | 382 ms | 391 ms | $0.011$10.75 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 388 ms · p90 392 ms Logic 47.3% · Math 70.0% | — | 62.2%60.6%–63.9% | c | 2,046 / 3,288 | No answers at this threshold | 0.829 | 87.2%answered 50% | 388 ms | 392 ms | $0.011$11.25 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 381 ms · p90 388 ms Logic 47.3% · Math 69.9% | — | 62.1%60.5%–63.8% | c | 2,043 / 3,288 | No answers at this threshold | 0.830 | 86.7%answered 51% | 381 ms | 388 ms | $0.001$1.13 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 164 ms · p90 188 ms Logic 48.2% · Math 65.1% | — | 59.3%57.6%–61.0% | d | 1,951 / 3,288 | 263 right of 263 kept0 mistakes · 3,025 skipped | 0.810 | 15.4%answered 93% | 164 ms | 188 ms | $0.021$21.27 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 174 ms · p90 285 ms Logic 46.0% · Math 65.2% | — | 58.6%56.9%–60.3% | d | 1,928 / 3,288 | 231 right of 231 kept0 mistakes · 3,057 skipped | 0.811 | 86.0%answered 46% | 174 ms | 285 ms | $0.004$3.98 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 166 ms · p90 205 ms Logic 46.3% · Math 64.1% | — | 58.0%56.3%–59.7% | d | 1,907 / 3,288 | 313 right of 313 kept0 mistakes · 2,975 skipped | 0.817 | 10.2%answered 96% | 166 ms | 205 ms | $0.021$21.27 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Latency: median 312 ms · p90 360 ms Logic 44.4% · Math 59.4% | — | 54.3%52.6%–56.0% | e | 1,784 / 3,288 | 47 right of 47 kept0 mistakes · 3,241 skipped | 0.769 | 11.4%answered 94% | 312 ms | 360 ms | $0.009$9.07 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Latency: median 293 ms · p90 349 ms Logic 45.1% · Math 53.5% | — | 50.6%48.9%–52.3% | f | 1,664 / 3,288 | 356 right of 372 kept16 wrong · 2,916 skipped | 0.777 | 10.1%answered 95% | 293 ms | 349 ms | $0.005$5.46 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 185 ms · p90 292 ms Logic 40.8% · Math 55.6% | — | 50.5%48.8%–52.3% | f | 1,662 / 3,288 | 207 right of 208 kept1 wrong · 3,080 skipped | 0.757 | 6.4%answered 97% | 185 ms | 292 ms | $0.004$3.98 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
No workhorse decision-model setup has answers on Logic and Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
No flagship decision-model setup has answers on Logic and Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
d1.2: 59.9% at a median 208 ms
d1.2-ultra: 63.6% at a median 217 ms
Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 217 ms · p90 347 ms Logic 51.6% · Math 69.9% | — | 63.6%61.9%–65.2% | a | 2,091 / 3,288 | 128 right of 128 kept0 mistakes · 3,160 skipped | 0.856 | 90.2%answered 48% | 217 ms | 347 ms | $0.008$8.37 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 208 ms · p90 390 ms Logic 49.2% · Math 65.5% | — | 59.9%58.2%–61.6% | b | 1,970 / 3,288 | 169 right of 169 kept0 mistakes · 3,119 skipped | 0.812 | 66.9%answered 65% | 208 ms | 390 ms | $0.004$3.98 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Meta MiniMax DeepSeek Thinking Machines Anthropic OpenAI Z.ai Google Other Alibaba eCloud Technology OpenBMB Xiaomi Liquid AI IBM KRAFTON 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Muse Ember (max · chat): 98.2% at a median 1,460 ms
MiniMax-M3.1-Flash (high · chat): 96.8% at a median 2,170 ms
DeepSeek-V4.1-Flash (high · chat · 1017): 98.4% at a median 2,270 ms
Inkling-Small (high · chat · Jul 2026): 98.2% at a median 2,750 ms
Inkling 2 Small (high · chat): 98.2% at a median 2,870 ms
Claude Opus 5.5 (chat): 98.5% at a median 3,600 ms
GPT-6 Luna (chat): 81.4% at a median 3,830 ms
Claude Sonnet 5.5 (chat): 98.2% at a median 4,030 ms
GPT-6.1 Terra (chat): 97.4% at a median 4,440 ms
GLM-5.3-Flash (high · chat): 96.9% at a median 4,640 ms
Gemini 4 Hydrogen (chat): 98.4% at a median 4,920 ms
Claude Fable 5.1 (chat): 98.2% at a median 4,940 ms
GPT-6.1 Sol (chat): 98.6% at a median 5,240 ms
GPT-6 Astra (chat): 98.6% at a median 5,580 ms
LFM3-1.7B (on · chat): 93.9% at a median 5,630 ms
LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell): 94.0% at a median 5,667 ms
Qwen4-Max (high · chat): 98.6% at a median 5,700 ms
Qwen3.8 Flash (off · chat): 97.6% at a median 6,780 ms
Erk-14B (off · chat): 78.3% at a median 8,740 ms
Gemma 4 E2B (on · chat · 32k cap · sampled): 90.5% at a median 8,873 ms
MiniCPM5-2B (off · chat): 93.2% at a median 9,065 ms
Qwen3-14B (off · chat): 85.4% at a median 9,380 ms
Gemma 4 E2B (chat): 88.6% at a median 9,935 ms
MiMo-V2.6-Flash (on · chat): 98.0% at a median 10,590 ms
Qwen3.8-27B (low · chat): 97.4% at a median 11,200 ms
Gemma 4 31B (chat): 97.3% at a median 12,575 ms
Gemma 4 E4B (chat): 93.0% at a median 13,150 ms
Gemma 4 E4B (on · chat · 32k cap · sampled): 94.2% at a median 13,945 ms
LFM2.5-2.6B (on · chat): 87.6% at a median 14,035 ms
Qwen3.5-4B (off · chat): 95.6% at a median 14,295 ms
Gemini 4 Carbon (chat): 98.6% at a median 16,350 ms
Qwen3.5-2B (off · chat): 84.4% at a median 17,805 ms
Gemma 4 E2B (on · chat · 32k cap): 90.3% at a median 19,100 ms
MiniCPM5-2B (on · chat): 95.7% at a median 19,260 ms
LFM2.5-8B-A1B (on · chat): 83.9% at a median 20,095 ms (mixed concurrency: This run changed the number of requests in flight part-way; see its record on Method)
Gemma 4 12B Unified (chat): 95.0% at a median 22,345 ms
Gemma 4 E4B (on · chat · 32k cap): 93.9% at a median 25,530 ms
Granite-4.2-3B (high · chat): 96.6% at a median 25,615 ms
LFM3-1.7B (off · chat): 94.2% at a median 29,040 ms
LFM2.5-24B-A2B (on · chat): 94.8% at a median 42,425 ms
Raon-1 nano (off · chat): 81.2% at a median 42,585 ms (other engine: Logic: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; Math: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Raon-1 nano (on · chat): 88.1% at a median 54,985 ms (other engine: Logic: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; Math: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Qwen3.5-4B (on · chat): 97.0% at a median 63,270 ms
Gemma 4 E2B (on · chat): 89.3% at a median 77,885 ms
Qwen3.5-2B (on · chat): 86.4% at a median 121,840 ms
Gemma 4 E4B (on · chat): 92.9% at a median 131,535 ms
Meta MiniMax DeepSeek Thinking Machines Anthropic OpenAI Z.ai Google Other Alibaba eCloud Technology OpenBMB Xiaomi Liquid AI IBM KRAFTON
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 9 setups not yet: DeepSeek-V4-Flash (chat) (missing Logic); Qwen3.5-397B-A17B (on · chat) (missing Math); GPT-6 Luna (max · chat) (missing Logic and Math); Claude Haiku 5.5 (max · chat) (missing Logic and Math); Claude Fable 5.2 (chat) (missing Logic and Math); Grok 4.7 (chat) (missing Logic); Spark-X2.6-4B (high · chat) (missing Logic and Math); Granite-4.2-3B (off · chat) (missing Logic and Math); Gemini 4 Carbon (high · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,580 ms · p90 9,580 ms Logic 96.7% · Math 99.6% | — | 98.6%98.2%–99.0% | a | 3,243 / 3,288 | Not measured | Not measured | 60.9%answered 99% | 5,580 ms | 9,580 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Latency: median 16,350 ms · p90 90,630 ms Logic 96.7% · Math 99.6% | — | 98.6%98.1%–98.9% | a | 3,242 / 3,288 | Not measured | Not measured | 53.3%answered 99% | 16,350 ms | 90,630 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,240 ms · p90 10,410 ms Logic 96.5% · Math 99.6% | — | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | 66.2%answered 99% | 5,240 ms | 10,410 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Latency: median 5,700 ms · p90 19,750 ms Logic 96.3% · Math 99.8% | — | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | 10.7%answered 100% | 5,700 ms | 19,750 ms | $2.919$2,919 |
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,600 ms · p90 8,380 ms Logic 96.2% · Math 99.8% | — | 98.5%98.1%–98.9% | a | 3,240 / 3,288 | Not measured | Not measured | 41.3%answered 99% | 3,600 ms | 8,380 ms | $13.179$13,179 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Latency: median 2,270 ms · p90 14,870 ms Logic 96.2% · Math 99.6% | — | 98.4%98.0%–98.8% | a | 3,237 / 3,288 | Not measured | Not measured | 6.3%answered 100% | 2,270 ms | 14,870 ms | $0.713$713 |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Latency: median 4,920 ms · p90 12,150 ms Logic 96.1% · Math 99.6% | — | 98.4%97.9%–98.8% | ab | 3,235 / 3,288 | Not measured | Not measured | 78.3%answered 98% | 4,920 ms | 12,150 ms | — |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 4,030 ms · p90 10,080 ms Logic 95.5% · Math 99.7% | — | 98.2%97.7%–98.6% | abc | 3,230 / 3,288 | Not measured | Not measured | 43.7%answered 99% | 4,030 ms | 10,080 ms | $7.042$7,042 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Latency: median 2,870 ms · p90 35,530 ms Logic 95.6% · Math 99.5% | — | 98.2%97.7%–98.6% | abcd | 3,229 / 3,288 | Not measured | Not measured | 41.7%answered 99% | 2,870 ms | 35,530 ms | $1.830$1,830 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 4,940 ms · p90 8,620 ms Logic 95.2% · Math 99.7% | — | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | 15.7%answered 100% | 4,940 ms | 8,620 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Latency: median 2,750 ms · p90 34,100 ms Logic 95.4% · Math 99.6% | — | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | 50.6%answered 99% | 2,750 ms | 34,100 ms | $2.506$2,506 |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Latency: median 1,460 ms · p90 6,300 ms Logic 95.5% · Math 99.6% | — | 98.2%97.7%–98.6% | abcd | 3,228 / 3,288 | Not measured | Not measured | 31.3%answered 99% | 1,460 ms | 6,300 ms | $0.863$863 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Latency: median 10,590 ms · p90 53,970 ms Logic 95.1% · Math 99.4% | — | 98.0%97.4%–98.4% | abcde | 3,221 / 3,288 | Not measured | Not measured | 27.6%answered 99% | 10,590 ms | 53,970 ms | $0.615$615 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Latency: median 6,780 ms · p90 36,290 ms Logic 94.0% · Math 99.5% | — | 97.6%97.0%–98.1% | bcdef | 3,209 / 3,288 | Not measured | Not measured | 31.3%answered 99% | 6,780 ms | 36,290 ms | $0.732$732 |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Latency: median 11,200 ms · p90 39,560 ms Logic 94.0% · Math 99.2% | — | 97.4%96.8%–97.9% | cdef | 3,204 / 3,288 | Not measured | Not measured | 33.9%answered 99% | 11,200 ms | 39,560 ms | $2.020$2,020estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,440 ms · p90 8,290 ms Logic 94.0% · Math 99.2% | — | 97.4%96.8%–97.9% | cdef | 3,203 / 3,288 | Not measured | Not measured | 65.0%answered 98% | 4,440 ms | 8,290 ms | $0.000$0.00 |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Latency: median 12,575 ms · p90 25,280 ms Logic 93.3% · Math 99.4% | — | 97.3%96.7%–97.8% | def | 3,200 / 3,288 | Not measured | Not measured | 17.0%answered 99% | 12,575 ms | 25,280 ms | $0.775$775estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Latency: median 63,270 ms · p90 193,950 ms Logic 92.4% · Math 99.4% | — | 97.0%96.4%–97.5% | ef | 3,190 / 3,288 | Not measured | Not measured | 34.2%answered 99% | 63,270 ms | 193,950 ms | $2.042$2,042estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Latency: median 4,640 ms · p90 25,900 ms Logic 93.2% · Math 98.8% | — | 96.9%96.2%–97.4% | fg | 3,185 / 3,288 | Not measured | Not measured | 7.4%answered 100% | 4,640 ms | 25,900 ms | $0.471$471 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Latency: median 2,170 ms · p90 15,760 ms Logic 94.0% · Math 98.2% | — | 96.8%96.1%–97.3% | fg | 3,182 / 3,288 | Not measured | Not measured | 38.9%answered 99% | 2,170 ms | 15,760 ms | $0.342$342 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Latency: median 25,615 ms · p90 523,100 ms Logic 91.6% · Math 99.2% | — | 96.6%96.0%–97.2% | fg | 3,177 / 3,288 | Not measured | Not measured | 54.7%answered 98% | 25,615 ms | 523,100 ms | $4.241$4,241estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Latency: median 19,260 ms · p90 194,917 ms Logic 88.9% · Math 99.2% | — | 95.7%95.4%–96.0% min–max of 3 seeds | gh | 3,145 / 3,288seed 0 run | Not measured | Not measured | 51.8%answered 97% | 19,260 ms | 194,917 ms | $1.410$1,410estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Latency: median 14,295 ms · p90 78,230 ms Logic 90.1% · Math 98.5% | — | 95.6%94.9%–96.3% | gh | 3,144 / 3,288 | Not measured | Not measured | 34.0%answered 98% | 14,295 ms | 78,230 ms | $0.683$683estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Latency: median 22,345 ms · p90 53,390 ms Logic 87.7% · Math 98.8% | — | 95.0%94.2%–95.7% | hi | 3,125 / 3,288 | Not measured | Not measured | 15.3%answered 99% | 22,345 ms | 53,390 ms | $0.749$749estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Latency: median 42,425 ms · p90 432,510 ms Logic 86.7% · Math 99.0% | — | 94.8%94.0%–95.5% | hij | 3,117 / 3,288 | Not measured | Not measured | 44.0%answered 97% | 42,425 ms | 432,510 ms | $3.205$3,205estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 13,945 ms · p90 51,987 ms Logic 85.9% · Math 98.5% | — | 94.2%94.0%–94.4% min–max of 3 seeds | hijk | 3,104 / 3,288seed 0 run | Not measured | Not measured | 54.1%answered 96% | 13,945 ms | 51,987 ms | $0.549$549estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Latency: median 29,040 ms · p90 343,820 ms Logic 85.5% · Math 98.7% | — | 94.2%93.3%–94.9% | hijk | 3,097 / 3,288 | Not measured | Not measured | 55.9%answered 96% | 29,040 ms | 343,820 ms | $0.206$206estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Latency: median 5,667 ms · p90 58,313 ms Logic 84.6% · Math 98.9% | — | 94.0%93.8%–94.2% min–max of 3 seeds | hijk | 3,098 / 3,288seed 0 run | Not measured | Not measured | Not measured | 5,667 ms | 58,313 ms | $0.449$449estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Latency: median 5,630 ms · p90 63,067 ms Logic 84.2% · Math 98.9% | — | 93.9%93.89%–93.95% min–max of 3 seeds | ijk | 3,089 / 3,288seed 0 run | Not measured | Not measured | 46.7%answered 96% | 5,630 ms | 63,067 ms | $0.435$435estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 25,530 ms · p90 77,380 ms Logic 85.0% · Math 98.5% | — | 93.9%93.0%–94.7% | ijk | 3,087 / 3,288 | Not measured | Not measured | Not measured | 25,530 ms | 77,380 ms | $0.855$855estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Latency: median 9,065 ms · p90 75,703 ms Logic 83.6% · Math 98.2% | — | 93.2%92.9%–93.3% min–max of 3 seeds | jk | 3,068 / 3,288seed 0 run | Not measured | Not measured | 41.0%answered 95% | 9,065 ms | 75,703 ms | $0.598$598estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 13,150 ms · p90 44,990 ms Logic 83.2% · Math 98.1% | — | 93.0%92.1%–93.8% | k | 3,058 / 3,288 | Not measured | Not measured | 51.0%answered 95% | 13,150 ms | 44,990 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 131,535 ms · p90 392,620 ms Logic 82.6% · Math 98.2% | — | 92.9%92.0%–93.7% | k | 3,054 / 3,288 | Not measured | Not measured | 43.7%answered 96% | 131,535 ms | 392,620 ms | $0.375$375estimate · DGX Spark |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 8,873 ms · p90 32,503 ms Logic 76.8% · Math 97.7% | — | 90.5%90.2%–90.9% min–max of 3 seeds | l | 2,966 / 3,288seed 0 run | Not measured | Not measured | 31.0%answered 97% | 8,873 ms | 32,503 ms | $0.325$325estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 19,100 ms · p90 54,630 ms Logic 75.9% · Math 97.8% | — | 90.3%89.3%–91.3% | l | 2,970 / 3,288 | Not measured | Not measured | Not measured | 19,100 ms | 54,630 ms | $0.678$678estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 77,885 ms · p90 232,800 ms Logic 74.6% · Math 96.9% | — | 89.3%88.2%–90.3% | lm | 2,936 / 3,288 | Not measured | Not measured | 24.3%answered 97% | 77,885 ms | 232,800 ms | $0.215$215estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 9,935 ms · p90 27,690 ms Logic 72.9% · Math 96.8% | — | 88.6%87.5%–89.6% | lmn | 2,913 / 3,288 | Not measured | Not measured | 32.3%answered 96% | 9,935 ms | 27,690 ms | $0.313$313estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B | — | 88.1%86.9%–89.1% | mn | 2,896 / 3,288 | Not measured | Not measured | Not measured | 54,985 ms† | 987,660 ms† | $6.344$6,344estimate · RTX PRO 6000 Blackwell† |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Latency: median 14,035 ms · p90 142,970 ms Logic 75.4% · Math 94.0% | — | 87.6%86.5%–88.7% | mno | 2,881 / 3,288 | Not measured | Not measured | 40.6%answered 95% | 14,035 ms | 142,970 ms | $1.046$1,046estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Latency: median 121,840 ms · p90 241,590 ms Logic 70.3% · Math 94.7% | — | 86.4%85.2%–87.5% | nop | 2,840 / 3,288 | Not measured | Not measured | 30.9%answered 97% | 121,840 ms | 241,590 ms | $3.106$3,106estimate · RTX PRO 6000 Blackwell |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Latency: median 9,380 ms · p90 35,110 ms Logic 65.3% · Math 95.8% | — | 85.4%84.1%–86.5% | op | 2,807 / 3,288 | Not measured | Not measured | 20.9%answered 96% | 9,380 ms | 35,110 ms | $0.840$840estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Latency: median 17,805 ms · p90 92,100 ms Logic 64.8% · Math 94.6% | — | 84.4%83.1%–85.6% | p | 2,776 / 3,288 | Not measured | Not measured | 34.3%answered 94% | 17,805 ms | 92,100 ms | $0.788$788estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 83.9%82.6%–85.1% | pq | 2,759 / 3,288 | Not measured | Not measured | 55.9%answered 88% | 20,095 ms† | 230,570 ms† | $1.913$1,913estimate · RTX PRO 6000 Blackwell† |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,830 ms · p90 5,300 ms Logic 58.0% · Math 93.7% | — | 81.4%80.1%–82.7% | q | 2,678 / 3,288 | Not measured | Not measured | Not measured | 3,830 ms | 5,300 ms | $0.000$0.00 |
| Raon-1 nano off · chat Open · 3.1B | — | 81.2%79.8%–82.5% | qr | 2,669 / 3,288 | Not measured | Not measured | 31.1%answered 94% | 42,585 ms† | 593,950 ms† | $4.422$4,422estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Latency: median 8,740 ms · p90 16,420 ms Logic 54.2% · Math 90.8% | — | 78.3%76.8%–79.6% | r | 2,573 / 3,288 | Not measured | Not measured | 32.3%answered 92% | 8,740 ms | 16,420 ms | $0.308$308estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Other Google OpenBMB Liquid AI Alibaba IBM KRAFTON 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
LFM3-1.7B (on · chat): 93.9% at a median 5,630 ms
LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell): 94.0% at a median 5,667 ms
Gemma 4 E2B (on · chat · 32k cap · sampled): 90.5% at a median 8,873 ms
MiniCPM5-2B (off · chat): 93.2% at a median 9,065 ms
Gemma 4 E2B (chat): 88.6% at a median 9,935 ms
LFM2.5-2.6B (on · chat): 87.6% at a median 14,035 ms
Qwen3.5-4B (off · chat): 95.6% at a median 14,295 ms
Qwen3.5-2B (off · chat): 84.4% at a median 17,805 ms
Gemma 4 E2B (on · chat · 32k cap): 90.3% at a median 19,100 ms
MiniCPM5-2B (on · chat): 95.7% at a median 19,260 ms
Granite-4.2-3B (high · chat): 96.6% at a median 25,615 ms
LFM3-1.7B (off · chat): 94.2% at a median 29,040 ms
Raon-1 nano (off · chat): 81.2% at a median 42,585 ms (other engine: Logic: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; Math: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Raon-1 nano (on · chat): 88.1% at a median 54,985 ms (other engine: Logic: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; Math: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Qwen3.5-4B (on · chat): 97.0% at a median 63,270 ms
Gemma 4 E2B (on · chat): 89.3% at a median 77,885 ms
Qwen3.5-2B (on · chat): 86.4% at a median 121,840 ms
Other Google OpenBMB Liquid AI Alibaba IBM KRAFTON
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: Spark-X2.6-4B (high · chat) (missing Logic and Math); Granite-4.2-3B (off · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Latency: median 63,270 ms · p90 193,950 ms Logic 92.4% · Math 99.4% | — | 97.0%96.4%–97.5% | a | 3,190 / 3,288 | Not measured | Not measured | 34.2%answered 99% | 63,270 ms | 193,950 ms | $2.042$2,042estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Latency: median 25,615 ms · p90 523,100 ms Logic 91.6% · Math 99.2% | — | 96.6%96.0%–97.2% | ab | 3,177 / 3,288 | Not measured | Not measured | 54.7%answered 98% | 25,615 ms | 523,100 ms | $4.241$4,241estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Latency: median 19,260 ms · p90 194,917 ms Logic 88.9% · Math 99.2% | — | 95.7%95.4%–96.0% min–max of 3 seeds | b | 3,145 / 3,288seed 0 run | Not measured | Not measured | 51.8%answered 97% | 19,260 ms | 194,917 ms | $1.410$1,410estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Latency: median 14,295 ms · p90 78,230 ms Logic 90.1% · Math 98.5% | — | 95.6%94.9%–96.3% | b | 3,144 / 3,288 | Not measured | Not measured | 34.0%answered 98% | 14,295 ms | 78,230 ms | $0.683$683estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Latency: median 29,040 ms · p90 343,820 ms Logic 85.5% · Math 98.7% | — | 94.2%93.3%–94.9% | c | 3,097 / 3,288 | Not measured | Not measured | 55.9%answered 96% | 29,040 ms | 343,820 ms | $0.206$206estimate · DGX Spark |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Latency: median 5,667 ms · p90 58,313 ms Logic 84.6% · Math 98.9% | — | 94.0%93.8%–94.2% min–max of 3 seeds | c | 3,098 / 3,288seed 0 run | Not measured | Not measured | Not measured | 5,667 ms | 58,313 ms | $0.449$449estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Latency: median 5,630 ms · p90 63,067 ms Logic 84.2% · Math 98.9% | — | 93.9%93.89%–93.95% min–max of 3 seeds | c | 3,089 / 3,288seed 0 run | Not measured | Not measured | 46.7%answered 96% | 5,630 ms | 63,067 ms | $0.435$435estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Latency: median 9,065 ms · p90 75,703 ms Logic 83.6% · Math 98.2% | — | 93.2%92.9%–93.3% min–max of 3 seeds | c | 3,068 / 3,288seed 0 run | Not measured | Not measured | 41.0%answered 95% | 9,065 ms | 75,703 ms | $0.598$598estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 8,873 ms · p90 32,503 ms Logic 76.8% · Math 97.7% | — | 90.5%90.2%–90.9% min–max of 3 seeds | de | 2,966 / 3,288seed 0 run | Not measured | Not measured | 31.0%answered 97% | 8,873 ms | 32,503 ms | $0.325$325estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 19,100 ms · p90 54,630 ms Logic 75.9% · Math 97.8% | — | 90.3%89.3%–91.3% | d | 2,970 / 3,288 | Not measured | Not measured | Not measured | 19,100 ms | 54,630 ms | $0.678$678estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 77,885 ms · p90 232,800 ms Logic 74.6% · Math 96.9% | — | 89.3%88.2%–90.3% | def | 2,936 / 3,288 | Not measured | Not measured | 24.3%answered 97% | 77,885 ms | 232,800 ms | $0.215$215estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 9,935 ms · p90 27,690 ms Logic 72.9% · Math 96.8% | — | 88.6%87.5%–89.6% | ef | 2,913 / 3,288 | Not measured | Not measured | 32.3%answered 96% | 9,935 ms | 27,690 ms | $0.313$313estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B | — | 88.1%86.9%–89.1% | fg | 2,896 / 3,288 | Not measured | Not measured | Not measured | 54,985 ms† | 987,660 ms† | $6.344$6,344estimate · RTX PRO 6000 Blackwell† |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Latency: median 14,035 ms · p90 142,970 ms Logic 75.4% · Math 94.0% | — | 87.6%86.5%–88.7% | fg | 2,881 / 3,288 | Not measured | Not measured | 40.6%answered 95% | 14,035 ms | 142,970 ms | $1.046$1,046estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Latency: median 121,840 ms · p90 241,590 ms Logic 70.3% · Math 94.7% | — | 86.4%85.2%–87.5% | gh | 2,840 / 3,288 | Not measured | Not measured | 30.9%answered 97% | 121,840 ms | 241,590 ms | $3.106$3,106estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Latency: median 17,805 ms · p90 92,100 ms Logic 64.8% · Math 94.6% | — | 84.4%83.1%–85.6% | h | 2,776 / 3,288 | Not measured | Not measured | 34.3%answered 94% | 17,805 ms | 92,100 ms | $0.788$788estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B | — | 81.2%79.8%–82.5% | i | 2,669 / 3,288 | Not measured | Not measured | 31.1%answered 94% | 42,585 ms† | 593,950 ms† | $4.422$4,422estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
eCloud Technology Alibaba Google Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Erk-14B (off · chat): 78.3% at a median 8,740 ms
Qwen3-14B (off · chat): 85.4% at a median 9,380 ms
Gemma 4 E4B (chat): 93.0% at a median 13,150 ms
Gemma 4 E4B (on · chat · 32k cap · sampled): 94.2% at a median 13,945 ms
LFM2.5-8B-A1B (on · chat): 83.9% at a median 20,095 ms (mixed concurrency: This run changed the number of requests in flight part-way; see its record on Method)
Gemma 4 12B Unified (chat): 95.0% at a median 22,345 ms
Gemma 4 E4B (on · chat · 32k cap): 93.9% at a median 25,530 ms
Gemma 4 E4B (on · chat): 92.9% at a median 131,535 ms
eCloud Technology Alibaba Google Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Latency: median 22,345 ms · p90 53,390 ms Logic 87.7% · Math 98.8% | — | 95.0%94.2%–95.7% | a | 3,125 / 3,288 | Not measured | Not measured | 15.3%answered 99% | 22,345 ms | 53,390 ms | $0.749$749estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 13,945 ms · p90 51,987 ms Logic 85.9% · Math 98.5% | — | 94.2%94.0%–94.4% min–max of 3 seeds | ab | 3,104 / 3,288seed 0 run | Not measured | Not measured | 54.1%answered 96% | 13,945 ms | 51,987 ms | $0.549$549estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 25,530 ms · p90 77,380 ms Logic 85.0% · Math 98.5% | — | 93.9%93.0%–94.7% | bc | 3,087 / 3,288 | Not measured | Not measured | Not measured | 25,530 ms | 77,380 ms | $0.855$855estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 13,150 ms · p90 44,990 ms Logic 83.2% · Math 98.1% | — | 93.0%92.1%–93.8% | c | 3,058 / 3,288 | Not measured | Not measured | 51.0%answered 95% | 13,150 ms | 44,990 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 131,535 ms · p90 392,620 ms Logic 82.6% · Math 98.2% | — | 92.9%92.0%–93.7% | c | 3,054 / 3,288 | Not measured | Not measured | 43.7%answered 96% | 131,535 ms | 392,620 ms | $0.375$375estimate · DGX Spark |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Latency: median 9,380 ms · p90 35,110 ms Logic 65.3% · Math 95.8% | — | 85.4%84.1%–86.5% | d | 2,807 / 3,288 | Not measured | Not measured | 20.9%answered 96% | 9,380 ms | 35,110 ms | $0.840$840estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 83.9%82.6%–85.1% | d | 2,759 / 3,288 | Not measured | Not measured | 55.9%answered 88% | 20,095 ms† | 230,570 ms† | $1.913$1,913estimate · RTX PRO 6000 Blackwell† |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Latency: median 8,740 ms · p90 16,420 ms Logic 54.2% · Math 90.8% | — | 78.3%76.8%–79.6% | e | 2,573 / 3,288 | Not measured | Not measured | 32.3%answered 92% | 8,740 ms | 16,420 ms | $0.308$308estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Alibaba Google Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Qwen3.8-27B (low · chat): 97.4% at a median 11,200 ms
Gemma 4 31B (chat): 97.3% at a median 12,575 ms
LFM2.5-24B-A2B (on · chat): 94.8% at a median 42,425 ms
Alibaba Google Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Latency: median 11,200 ms · p90 39,560 ms Logic 94.0% · Math 99.2% | — | 97.4%96.8%–97.9% | a | 3,204 / 3,288 | Not measured | Not measured | 33.9%answered 99% | 11,200 ms | 39,560 ms | $2.020$2,020estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Latency: median 12,575 ms · p90 25,280 ms Logic 93.3% · Math 99.4% | — | 97.3%96.7%–97.8% | a | 3,200 / 3,288 | Not measured | Not measured | 17.0%answered 99% | 12,575 ms | 25,280 ms | $0.775$775estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Latency: median 42,425 ms · p90 432,510 ms Logic 86.7% · Math 99.0% | — | 94.8%94.0%–95.5% | b | 3,117 / 3,288 | Not measured | Not measured | 44.0%answered 97% | 42,425 ms | 432,510 ms | $3.205$3,205estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
MiniMax DeepSeek Thinking Machines Z.ai Xiaomi 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
MiniMax-M3.1-Flash (high · chat): 96.8% at a median 2,170 ms
DeepSeek-V4.1-Flash (high · chat · 1017): 98.4% at a median 2,270 ms
Inkling-Small (high · chat · Jul 2026): 98.2% at a median 2,750 ms
Inkling 2 Small (high · chat): 98.2% at a median 2,870 ms
GLM-5.3-Flash (high · chat): 96.9% at a median 4,640 ms
MiMo-V2.6-Flash (on · chat): 98.0% at a median 10,590 ms
MiniMax DeepSeek Thinking Machines Z.ai Xiaomi
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: DeepSeek-V4-Flash (chat) (missing Logic); Qwen3.5-397B-A17B (on · chat) (missing Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Latency: median 2,270 ms · p90 14,870 ms Logic 96.2% · Math 99.6% | — | 98.4%98.0%–98.8% | a | 3,237 / 3,288 | Not measured | Not measured | 6.3%answered 100% | 2,270 ms | 14,870 ms | $0.713$713 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Latency: median 2,870 ms · p90 35,530 ms Logic 95.6% · Math 99.5% | — | 98.2%97.7%–98.6% | a | 3,229 / 3,288 | Not measured | Not measured | 41.7%answered 99% | 2,870 ms | 35,530 ms | $1.830$1,830 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Latency: median 2,750 ms · p90 34,100 ms Logic 95.4% · Math 99.6% | — | 98.2%97.7%–98.6% | a | 3,228 / 3,288 | Not measured | Not measured | 50.6%answered 99% | 2,750 ms | 34,100 ms | $2.506$2,506 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Latency: median 10,590 ms · p90 53,970 ms Logic 95.1% · Math 99.4% | — | 98.0%97.4%–98.4% | a | 3,221 / 3,288 | Not measured | Not measured | 27.6%answered 99% | 10,590 ms | 53,970 ms | $0.615$615 |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Latency: median 4,640 ms · p90 25,900 ms Logic 93.2% · Math 98.8% | — | 96.9%96.2%–97.4% | b | 3,185 / 3,288 | Not measured | Not measured | 7.4%answered 100% | 4,640 ms | 25,900 ms | $0.471$471 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Latency: median 2,170 ms · p90 15,760 ms Logic 94.0% · Math 98.2% | — | 96.8%96.1%–97.3% | b | 3,182 / 3,288 | Not measured | Not measured | 38.9%answered 99% | 2,170 ms | 15,760 ms | $0.342$342 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Meta OpenAI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Muse Ember (max · chat): 98.2% at a median 1,460 ms
GPT-6 Luna (chat): 81.4% at a median 3,830 ms
Meta OpenAI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: GPT-6 Luna (max · chat) (missing Logic and Math); Claude Haiku 5.5 (max · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Latency: median 1,460 ms · p90 6,300 ms Logic 95.5% · Math 99.6% | — | 98.2%97.7%–98.6% | a | 3,228 / 3,288 | Not measured | Not measured | 31.3%answered 99% | 1,460 ms | 6,300 ms | $0.863$863 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,830 ms · p90 5,300 ms Logic 58.0% · Math 93.7% | — | 81.4%80.1%–82.7% | b | 2,678 / 3,288 | Not measured | Not measured | Not measured | 3,830 ms | 5,300 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Anthropic 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Claude Sonnet 5.5 (chat): 98.2% at a median 4,030 ms
Anthropic
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 4,030 ms · p90 10,080 ms Logic 95.5% · Math 99.7% | — | 98.2%97.7%–98.6% | a | 3,230 / 3,288 | Not measured | Not measured | 43.7%answered 99% | 4,030 ms | 10,080 ms | $7.042$7,042 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Anthropic 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Claude Opus 5.5 (chat): 98.5% at a median 3,600 ms
Claude Fable 5.1 (chat): 98.2% at a median 4,940 ms
Anthropic
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 1 setup not yet: Claude Fable 5.2 (chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,600 ms · p90 8,380 ms Logic 96.2% · Math 99.8% | — | 98.5%98.1%–98.9% | a | 3,240 / 3,288 | Not measured | Not measured | 41.3%answered 99% | 3,600 ms | 8,380 ms | $13.179$13,179 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 4,940 ms · p90 8,620 ms Logic 95.2% · Math 99.7% | — | 98.2%97.7%–98.6% | b | 3,228 / 3,288 | Not measured | Not measured | 15.7%answered 100% | 4,940 ms | 8,620 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
OpenAI Alibaba Google 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GPT-6.1 Terra (chat): 97.4% at a median 4,440 ms
GPT-6.1 Sol (chat): 98.6% at a median 5,240 ms
GPT-6 Astra (chat): 98.6% at a median 5,580 ms
Qwen4-Max (high · chat): 98.6% at a median 5,700 ms
Qwen3.8 Flash (off · chat): 97.6% at a median 6,780 ms
Gemini 4 Carbon (chat): 98.6% at a median 16,350 ms
OpenAI Alibaba Google
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not in Logic and Math yet
Logic and Math pools only setups with a complete English run on both benchmarks. 2 setups not yet: Grok 4.7 (chat) (missing Logic); Gemini 4 Carbon (high · chat) (missing Logic and Math).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 3,288) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,580 ms · p90 9,580 ms Logic 96.7% · Math 99.6% | — | 98.6%98.2%–99.0% | a | 3,243 / 3,288 | Not measured | Not measured | 60.9%answered 99% | 5,580 ms | 9,580 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Latency: median 16,350 ms · p90 90,630 ms Logic 96.7% · Math 99.6% | — | 98.6%98.1%–98.9% | a | 3,242 / 3,288 | Not measured | Not measured | 53.3%answered 99% | 16,350 ms | 90,630 ms | — |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,240 ms · p90 10,410 ms Logic 96.5% · Math 99.6% | — | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | 66.2%answered 99% | 5,240 ms | 10,410 ms | $0.000$0.00 |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Latency: median 5,700 ms · p90 19,750 ms Logic 96.3% · Math 99.8% | — | 98.6%98.1%–98.9% | a | 3,241 / 3,288 | Not measured | Not measured | 10.7%answered 100% | 5,700 ms | 19,750 ms | $2.919$2,919 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Latency: median 6,780 ms · p90 36,290 ms Logic 94.0% · Math 99.5% | — | 97.6%97.0%–98.1% | b | 3,209 / 3,288 | Not measured | Not measured | 31.3%answered 99% | 6,780 ms | 36,290 ms | $0.732$732 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,440 ms · p90 8,290 ms Logic 94.0% · Math 99.2% | — | 97.4%96.8%–97.9% | b | 3,203 / 3,288 | Not measured | Not measured | 65.0%answered 98% | 4,440 ms | 8,290 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.