Math · Latency
Latency on Math
2,163 questions, asked in English. How to read this page · Method
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents newsbang tattrongvu Blockbrain Labs Patrick-18 victorbennett EldanRing vLLM Semantic Router Valen-Team Nace.AI Base Labs Cloudflare Metask Lab Poindexter Labs Apertix AI MachinePulse TypeSafe REAX Logits Google Zefan Cai Turion.AI Inception Microsoft Subquadratic Celeris Sthānika AI Meta MiniMax DeepSeek Thinking Machines Anthropic Z.ai OpenAI Alibaba OpenBMB eCloud Technology Xiaomi xAI IBM KRAFTON 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GLiDE 0.4B: 20.6% at a median 11 ms
GLiClass v4 Edge: 22.7% at a median 12 ms
OneJev1.6-0.8B: 28.2% at a median 15 ms
d1-2B: 37.1% at a median 16 ms
GLiDE 0.8B: 32.1% at a median 20 ms
StartLux-Decision-v1.1-0.8B-Instant: 29.7% at a median 24 ms
KnowLine-0.8B-Gen1: 28.2% at a median 25 ms
GLiDE 2B: 36.8% at a median 28 ms
Perplexity Decider Flash v1.3: 55.9% at a median 28 ms
Chinese Decision Pointer 4B: 50.4% at a median 28 ms
vera-medium-omni: 56.8% at a median 31 ms
Strands Decider 2B: 28.7% at a median 35 ms
StartLux-Decision-v1.1-2B-Instant: 38.7% at a median 36 ms
Adjudex 9B v5: 62.1% at a median 36 ms
Jeffan-9B-1211: 59.8% at a median 37 ms
KnowLine-9B-Gen1: 59.8% at a median 37 ms
Cygnet: 44.3% at a median 37 ms
Perplexity Decider Flash v1.2: 55.2% at a median 40 ms
KnowLine-9B-Gen2: 60.1% at a median 41 ms
Dexy-Decision-9B-V2: 58.8% at a median 45 ms
Verdin-4B-v1.1: 51.5% at a median 47 ms
KnowLine-12B-Gen1: 66.3% at a median 49 ms
KnowLine-2B-Gen1: 35.3% at a median 50 ms
Winnow-12B: 46.9% at a median 52 ms
d3: 70.4% at a median 55 ms
Valen-Decision-1.5-9B: 55.3% at a median 58 ms
StartLux-Decision-v1.1-4B-Instant: 55.8% at a median 67 ms
KnowLine 4B Gen5.5: 56.0% at a median 74 ms
Drex Max: 70.3% at a median 76 ms
GLiDE (v1.2): 69.9% at a median 79 ms
OneJev-27B: 53.8% at a median 79 ms
Elephant-1: 61.3% at a median 80 ms
Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): 72.5% at a median 91 ms
Clef-Flash: 49.8% at a median 99 ms
Metask-Jev-4B: 39.2% at a median 113 ms
KnowLine-31B-Gen2: 80.3% at a median 114 ms
Copernicus-31B: 77.3% at a median 118 ms
d3-ultra: 71.5% at a median 118 ms
Mammoth-1: 74.0% at a median 140 ms
StartLux-Decision-v1.1-9B-Instant: 58.4% at a median 140 ms
Cerno 9B v1.2: 56.0% at a median 153 ms
Karpo Decision 31B: 81.6% at a median 155 ms
Jev 1.14: 65.1% at a median 163 ms
Jev 1.13: 64.1% at a median 166 ms
d1.1: 65.2% at a median 172 ms
GLiDE 9B: 57.9% at a median 174 ms
d1-ultra: 74.1% at a median 179 ms
Clef: 51.1% at a median 181 ms
d1: 55.6% at a median 184 ms
REAX 1.3: 78.0% at a median 189 ms
Logits Sigma-1 Duo: 75.2% at a median 196 ms
d1.2: 65.5% at a median 204 ms
d1.2-ultra: 69.9% at a median 215 ms
Gemini Decision 1 (20261111): 71.3% at a median 226 ms
Perplexity Decider 27B v1.2 (API): 69.8% at a median 226 ms
Open-Jev-27B-v1.1: 54.6% at a median 233 ms
GLiDE (API): 70.8% at a median 264 ms
Turion Jev 31B: 69.0% at a median 278 ms
Mercury Decide: 53.5% at a median 288 ms
Microsoft-Decision-1: 59.4% at a median 310 ms
Drex Max v1.1: 70.1% at a median 326 ms
Drex 2.0: 59.0% at a median 334 ms
SubQ Snap 1.1: 68.6% at a median 381 ms
celeris-1.1-decision: 69.9% at a median 381 ms
celeris-1-decision: 70.0% at a median 387 ms
Perplexity Decider 27B v1.2 (API · with web): 69.3% at a median 422 ms
GLiDE 27B: 72.0% at a median 425 ms
Sieve 31B v3: 77.1% at a median 637 ms
KnowLine-31B-Gen1.5: 77.4% at a median 764 ms
Muse Ember (max · chat): 99.6% at a median 1,260 ms
MiniMax-M3.1-Flash (high · chat): 98.2% at a median 1,670 ms
DeepSeek-V4.1-Flash (high · chat · 1017): 99.6% at a median 1,910 ms
Inkling-Small (high · chat · Jul 2026): 99.6% at a median 2,030 ms
Inkling 2 Small (high · chat): 99.5% at a median 2,110 ms
Claude Opus 5.5 (chat): 99.8% at a median 3,200 ms
Claude Sonnet 5.5 (chat): 99.7% at a median 3,460 ms
GLM-5.3-Flash (high · chat): 98.8% at a median 3,800 ms
GPT-6 Luna (chat): 93.7% at a median 3,830 ms
LFM3-1.7B (on · chat): 98.9% at a median 3,893 ms
LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell): 98.9% at a median 3,990 ms
GPT-6.1 Terra (chat): 99.2% at a median 4,080 ms
Gemini 4 Hydrogen (chat): 99.6% at a median 4,400 ms
Qwen4-Max (high · chat): 99.8% at a median 4,710 ms
GPT-6.1 Sol (chat): 99.6% at a median 4,810 ms
Claude Fable 5.1 (chat): 99.7% at a median 5,030 ms
GPT-6 Astra (chat): 99.6% at a median 5,280 ms
Qwen3.8 Flash (off · chat): 99.5% at a median 5,740 ms
MiniCPM5-2B (off · chat): 98.2% at a median 6,353 ms
Gemma 4 E2B (on · chat · 32k cap · sampled): 97.7% at a median 7,137 ms
Erk-14B (off · chat): 90.8% at a median 7,890 ms
Qwen3-14B (off · chat): 95.8% at a median 8,040 ms
Gemma 4 E2B (chat): 96.8% at a median 8,120 ms
DeepSeek-V4-Flash (chat): 99.3% at a median 8,130 ms
MiMo-V2.6-Flash (on · chat): 99.4% at a median 8,200 ms
LFM2.5-2.6B (on · chat): 94.0% at a median 9,860 ms
Qwen3.8-27B (low · chat): 99.2% at a median 10,020 ms
Gemma 4 E4B (chat): 98.1% at a median 10,770 ms
Qwen3.5-4B (off · chat): 98.5% at a median 10,960 ms
Grok 4.7 (chat): 99.5% at a median 11,100 ms
Gemma 4 31B (chat): 99.4% at a median 11,110 ms
Gemma 4 E4B (on · chat · 32k cap · sampled): 98.5% at a median 11,190 ms
Qwen3.5-2B (off · chat): 94.6% at a median 11,200 ms
MiniCPM5-2B (on · chat): 99.2% at a median 12,537 ms
Gemini 4 Carbon (chat): 99.6% at a median 13,410 ms
LFM2.5-8B-A1B (on · chat): 96.3% at a median 14,530 ms (mixed concurrency: This run changed the number of requests in flight part-way; see its record on Method)
Gemma 4 E2B (on · chat · 32k cap): 97.8% at a median 14,870 ms
Granite-4.2-3B (high · chat): 99.2% at a median 15,340 ms
Gemma 4 12B Unified (chat): 98.8% at a median 19,100 ms
LFM3-1.7B (off · chat): 98.7% at a median 19,570 ms
Gemma 4 E4B (on · chat · 32k cap): 98.5% at a median 20,640 ms
Raon-1 nano (off · chat): 89.6% at a median 25,550 ms (other engine: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
LFM2.5-24B-A2B (on · chat): 99.0% at a median 27,950 ms
Raon-1 nano (on · chat): 97.0% at a median 34,320 ms (other engine: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Qwen3.5-4B (on · chat): 99.4% at a median 46,630 ms
Gemma 4 E2B (on · chat): 96.9% at a median 64,440 ms
Qwen3.5-2B (on · chat): 94.7% at a median 98,390 ms
Gemma 4 E4B (on · chat): 98.2% at a median 108,720 ms
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents newsbang tattrongvu Blockbrain Labs Patrick-18 victorbennett EldanRing vLLM Semantic Router Valen-Team Nace.AI Base Labs Cloudflare Metask Lab Poindexter Labs Apertix AI MachinePulse TypeSafe REAX Logits Google Zefan Cai Turion.AI Inception Microsoft Subquadratic Celeris Sthānika AI Meta MiniMax DeepSeek Thinking Machines Anthropic Z.ai OpenAI Alibaba OpenBMB eCloud Technology Xiaomi xAI IBM KRAFTON
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC); Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)); Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC); Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test); GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,200 ms · p90 4,790 ms Logic 96.2% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Latency: median 4,710 ms · p90 9,130 ms Logic 96.3% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 5,030 ms · p90 6,490 ms Logic 95.2% | — | Chat | 99.7%99.4%–99.9% | ab | 2,157 / 2,163 | Not measured | Not measured | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,460 ms · p90 5,970 ms Logic 95.5% | — | Chat | 99.7%99.3%–99.8% | abcd | 2,156 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Latency: median 1,910 ms · p90 3,570 ms Logic 96.2% | — | Chat | 99.6%99.3%–99.8% | abcdefg | 2,155 / 2,163 | Not measured | Not measured | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,280 ms · p90 7,260 ms Logic 96.7% | — | Chat | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,810 ms · p90 6,560 ms Logic 96.5% | — | Chat | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Latency: median 2,030 ms · p90 6,330 ms Logic 95.4% | — | Chat | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Latency: median 13,410 ms · p90 27,800 ms Logic 96.7% | — | Chat | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Latency: median 4,400 ms · p90 6,680 ms Logic 96.1% | — | Chat | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | 85.7%answered 99% | 4,400 ms | 6,680 ms | — |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Latency: median 1,260 ms · p90 2,270 ms Logic 95.5% | — | Chat | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 11,100 ms · p90 66,530 ms Logic running | — | Chat | 99.5%99.2%–99.7% | abcdefg | 2,153 / 2,163 | Not measured | Not measured | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Latency: median 2,110 ms · p90 6,000 ms Logic 95.6% | — | Chat | 99.5%99.2%–99.7% | abcdef | 2,153 / 2,163 | Not measured | Not measured | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Latency: median 5,740 ms · p90 11,800 ms Logic 94.0% | — | Chat | 99.5%99.1%–99.7% | abcdefgh | 2,152 / 2,163 | Not measured | Not measured | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Latency: median 8,200 ms · p90 18,090 ms Logic 95.1% | — | Chat | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Latency: median 46,630 ms · p90 96,460 ms Logic 92.4% | — | Chat | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Latency: median 11,110 ms · p90 16,690 ms Logic 93.3% | — | Chat | 99.4%99.0%–99.6% | abcdefghi | 2,150 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Latency: median 8,130 ms · p90 16,920 ms Logic running | — | Chat | 99.3%98.9%–99.6% | abcdefghijk | 2,148 / 2,163 | Not measured | Not measured | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Latency: median 12,537 ms · p90 32,720 ms Logic 88.9% | — | Chat | 99.2%99.2%–99.3% min–max of 3 seeds | abcdefghij | 2,147 / 2,163seed 0 run | Not measured | Not measured | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,080 ms · p90 5,870 ms Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | abcdefghijk | 2,146 / 2,163 | Not measured | Not measured | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Latency: median 15,340 ms · p90 53,280 ms Logic 91.6% | — | Chat | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Latency: median 10,020 ms · p90 20,630 ms Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | abcdefghijk | 2,146 / 2,163 | Not measured | Not measured | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Latency: median 27,950 ms · p90 60,210 ms Logic 86.7% | — | Chat | 99.0%98.5%–99.4% | bcdefghijk | 2,142 / 2,163 | Not measured | Not measured | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Latency: median 3,990 ms · p90 10,603 ms Logic 84.6% | — | Chat | 98.9%98.8%–99.0% min–max of 3 seeds | abcdefghijkl | 2,142 / 2,163seed 0 run | Not measured | Not measured | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Latency: median 3,893 ms · p90 10,353 ms Logic 84.2% | — | Chat | 98.9%98.8%–99.1% min–max of 3 seeds | bcdefghijkl | 2,139 / 2,163seed 0 run | Not measured | Not measured | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Latency: median 19,100 ms · p90 28,120 ms Logic 87.7% | — | Chat | 98.8%98.3%–99.2% | cdefghijklm | 2,138 / 2,163 | Not measured | Not measured | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Latency: median 3,800 ms · p90 7,970 ms Logic 93.2% | — | Chat | 98.8%98.2%–99.2% | dfghijklm | 2,137 / 2,163 | Not measured | Not measured | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Latency: median 19,570 ms · p90 53,410 ms Logic 85.5% | — | Chat | 98.7%98.1%–99.1% | ghijklm | 2,135 / 2,163 | Not measured | Not measured | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 11,190 ms · p90 22,357 ms Logic 85.9% | — | Chat | 98.5%98.4%–98.8% min–max of 3 seeds | efghijklm | 2,136 / 2,163seed 0 run | Not measured | Not measured | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 20,640 ms · p90 34,280 ms Logic 85.0% | — | Chat | 98.5%97.9%–99.0% | hijklmn | 2,131 / 2,163 | Not measured | Not measured | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Latency: median 10,960 ms · p90 19,970 ms Logic 90.1% | — | Chat | 98.5%97.9%–98.9% | hijklmno | 2,130 / 2,163 | Not measured | Not measured | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 108,720 ms · p90 175,470 ms Logic 82.6% | — | Chat | 98.2%97.6%–98.7% | jklmno | 2,125 / 2,163 | Not measured | Not measured | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Latency: median 1,670 ms · p90 7,470 ms Logic 94.0% | — | Chat | 98.2%97.6%–98.7% | ijklmnop | 2,125 / 2,163 | Not measured | Not measured | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Latency: median 6,353 ms · p90 15,697 ms Logic 83.6% | — | Chat | 98.2%98.1%–98.3% min–max of 3 seeds | jklmnop | 2,126 / 2,163seed 0 run | Not measured | Not measured | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 10,770 ms · p90 17,700 ms Logic 83.2% | — | Chat | 98.1%97.4%–98.6% | klmnop | 2,122 / 2,163 | Not measured | Not measured | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 14,870 ms · p90 26,040 ms Logic 75.9% | — | Chat | 97.8%97.1%–98.4% | lmnopq | 2,116 / 2,163 | Not measured | Not measured | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 7,137 ms · p90 14,787 ms Logic 76.8% | — | Chat | 97.7%97.6%–97.8% min–max of 3 seeds | mnopqr | 2,111 / 2,163seed 0 run | Not measured | Not measured | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B | — | Chat | 97.0%96.2%–97.6% | nopqr | 2,098 / 2,163 | Not measured | Not measured | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 64,440 ms · p90 106,260 ms Logic 74.6% | — | Chat | 96.9%96.1%–97.6% | opqr | 2,097 / 2,163 | Not measured | Not measured | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 8,120 ms · p90 13,650 ms Logic 72.9% | — | Chat | 96.8%95.9%–97.4% | pqrs | 2,093 / 2,163 | Not measured | Not measured | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | Chat | 96.3%95.4%–97.0% | qrst | 2,082 / 2,163 | Not measured | Not measured | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Latency: median 8,040 ms · p90 18,980 ms Logic 65.3% | — | Chat | 95.8%94.9%–96.6% | rstu | 2,072 / 2,163 | Not measured | Not measured | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Latency: median 98,390 ms · p90 194,980 ms Logic 70.3% | — | Chat | 94.7%93.7%–95.6% | stu | 2,049 / 2,163 | Not measured | Not measured | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Latency: median 11,200 ms · p90 32,140 ms Logic 64.8% | — | Chat | 94.6%93.6%–95.5% | stu | 2,047 / 2,163 | Not measured | Not measured | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Latency: median 9,860 ms · p90 21,240 ms Logic 75.4% | — | Chat | 94.0%92.9%–94.9% | tu | 2,033 / 2,163 | Not measured | Not measured | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,830 ms · p90 5,000 ms Logic 58.0% | — | Chat | 93.7%92.6%–94.6% | u | 2,026 / 2,163 | Not measured | Not measured | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Latency: median 7,890 ms · p90 14,890 ms Logic 54.2% | — | Chat | 90.8%89.5%–91.9% | v | 1,963 / 2,163 | Not measured | Not measured | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B | — | Chat | 89.6%88.2%–90.8% | v | 1,937 / 2,163 | Not measured | Not measured | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Latency: median 155 ms · p90 186 ms Logic 60.7% | — | Decision | 81.6%79.9%–83.1% | w | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Latency: median 114 ms · p90 120 ms Logic 58.9% | — | Decision | 80.3%78.5%–81.9% | wx | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Latency: median 189 ms · p90 232 ms Logic 59.3% | — | Decision | 78.0%76.2%–79.7% | xy | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Latency: median 764 ms · p90 867 ms Logic 59.0% | — | Decision | 77.4%75.6%–79.1% | yz | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Latency: median 118 ms · p90 142 ms Logic 58.7% | — | Decision | 77.3%75.5%–79.1% | yz | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Latency: median 637 ms · p90 735 ms Logic 57.9% | — | Decision | 77.1%75.3%–78.8% | yz | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Latency: median 196 ms · p90 221 ms Logic 57.8% | — | Decision | 75.2%73.4%–77.0% | zA | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Latency: median 179 ms · p90 426 ms Logic 54.9% | — | Decision | 74.1%72.2%–75.9% | yzAB | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Latency: median 140 ms · p90 173 ms Logic 54.5% | — | Decision | 74.0%72.1%–75.8% | zAB | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Latency: median 91 ms · p90 103 ms Logic 54.8% | — | Decision | 72.5%70.6%–74.3% | ABC | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Latency: median 425 ms · p90 461 ms Logic 58.4% | — | Decision | 72.0%70.1%–73.8% | ABCD | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 118 ms · p90 148 ms Logic 54.6% | — | Decision | 71.5%69.6%–73.4% | BCD | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Latency: median 226 ms · p90 351 ms Logic 50.0% | — | Decision | 71.3%69.4%–73.2% | ABCD | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Latency: median 264 ms · p90 279 ms Logic 54.5% | — | Decision | 70.8%68.8%–72.7% | BCD | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 55 ms · p90 62 ms Logic 54.5% | — | Decision | 70.4%68.5%–72.3% | BCDE | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Latency: median 76 ms · p90 89 ms Logic 53.1% | — | Decision | 70.3%68.4%–72.2% | BCDE | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Latency: median 326 ms · p90 473 ms Logic 54.8% | — | Decision | 70.1%68.1%–72.0% | BCDE | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 387 ms · p90 391 ms Logic 47.3% | — | Decision | 70.0%68.0%–71.9% | BCDEF | 1,514 / 2,163 | No answers at this threshold | 0.858 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Latency: median 79 ms · p90 84 ms Logic 55.2% | — | Decision | 69.9%67.9%–71.8% | CDEF | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 381 ms · p90 386 ms Logic 47.3% | — | Decision | 69.9%67.9%–71.8% | BCDEF | 1,511 / 2,163 | No answers at this threshold | 0.859 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 215 ms · p90 347 ms Logic 51.6% | — | Decision | 69.9%67.9%–71.8% | CDE | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Latency: median 226 ms · p90 368 ms Logic 53.3% | — | Decision | 69.8%67.8%–71.7% | CDEF | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Latency: median 422 ms · p90 641 ms Logic 52.5% | with web | Decision | 69.3%67.3%–71.2% | CDEFG | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Latency: median 278 ms · p90 329 ms Logic 52.3% | — | Decision | 69.0%67.0%–70.9% | CDEFG | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Latency: median 381 ms · p90 390 ms Logic 52.4% | — | Decision | 68.6%66.6%–70.5% | DEFG | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Latency: median 49 ms · p90 55 ms Logic 50.6% | — | Decision | 66.3%64.2%–68.2% | EFGH | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 204 ms · p90 386 ms Logic 49.2% | — | Decision | 65.5%63.5%–67.5% | FGHI | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 172 ms · p90 290 ms Logic 46.0% | — | Decision | 65.2%63.2%–67.2% | GHI | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 163 ms · p90 185 ms Logic 48.2% | — | Decision | 65.1%63.1%–67.1% | GHI | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 166 ms · p90 206 ms Logic 46.3% | — | Decision | 64.1%62.0%–66.1% | HIJ | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 36 ms · p90 44 ms Logic 46.4% | — | Decision | 62.1%60.0%–64.1% | HIJK | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Latency: median 80 ms · p90 92 ms Logic 46.4% | — | Decision | 61.3%59.3%–63.4% | IJK | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 41 ms · p90 46 ms Logic 43.9% | — | Decision | 60.1%58.0%–62.1% | JKL | 1,300 / 2,163 | No answers at this threshold | 0.803 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 37 ms · p90 51 ms Logic 48.2% | — | Decision | 59.8%57.7%–61.9% | JKL | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 37 ms · p90 43 ms Logic 44.3% | — | Decision | 59.8%57.7%–61.8% | JKL | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Latency: median 310 ms · p90 359 ms Logic 44.4% | — | Decision | 59.4%57.3%–61.4% | KLM | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Latency: median 334 ms · p90 432 ms Logic 41.9% | — | Decision | 59.0%56.9%–61.0% | KLM | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Latency: median 45 ms · p90 50 ms Logic 45.3% | — | Decision | 58.8%56.7%–60.9% | KLM | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Latency: median 140 ms · p90 146 ms Logic 46.6% | — | Decision | 58.4%56.3%–60.5% | KLMN | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Latency: median 174 ms · p90 182 ms Logic 47.8% | — | Decision | 57.9%55.8%–60.0% | KLMN | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Latency: median 31 ms · p90 35 ms Logic 44.1% | — | Decision | 56.8%54.7%–58.9% | LMN | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Latency: median 153 ms · p90 175 ms Logic 43.3% | — | Decision | 56.0%53.9%–58.1% | LMNOP | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Latency: median 74 ms · p90 84 ms Logic 42.3% | — | Decision | 56.0%53.9%–58.1% | LMNO | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Latency: median 28 ms · p90 31 ms Logic 44.7% | — | Decision | 55.9%53.8%–58.0% | LMNO | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Latency: median 67 ms · p90 69 ms Logic 45.9% | — | Decision | 55.8%53.7%–57.9% | LMNO | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 184 ms · p90 297 ms Logic 40.8% | — | Decision | 55.6%53.5%–57.7% | LMNOP | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 58 ms · p90 64 ms Logic 44.4% | — | Decision | 55.3%53.2%–57.4% | MNOP | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Latency: median 40 ms · p90 47 ms Logic 43.1% | — | Decision | 55.2%53.1%–57.3% | MNOP | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Latency: median 233 ms · p90 272 ms Logic 42.6% | — | Decision | 54.6%52.4%–56.6% | MNOPQ | 1,180 / 2,163 | No answers at this threshold | 0.764 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Latency: median 79 ms · p90 106 ms Logic 40.6% | — | Decision | 53.8%51.7%–55.9% | NOPQ | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Latency: median 288 ms · p90 342 ms Logic 45.1% | — | Decision | 53.5%51.4%–55.6% | NOPQ | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 47 ms · p90 52 ms Logic 39.7% | — | Decision | 51.5%49.4%–53.6% | PQR | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Latency: median 181 ms · p90 210 ms Logic 40.8% | — | Decision | 51.1%49.0%–53.2% | OPQR | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 28 ms · p90 32 ms Logic 40.4% | — | Decision | 50.4%48.3%–52.5% | QR | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Latency: median 99 ms · p90 127 ms Logic 38.9% | — | Decision | 49.8%47.7%–51.9% | QR | 1,077 / 2,163 | No answers at this threshold | 0.730 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Latency: median 52 ms · p90 61 ms Logic 39.7% | — | Decision | 46.9%44.8%–49.0% | RS | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Latency: median 37 ms · p90 41 ms Logic 39.3% | — | Decision | 44.3%42.3%–46.4% | S | 959 / 2,163 | No answers at this threshold | 0.735 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Latency: median 113 ms · p90 144 ms Logic 32.1% | — | Decision | 39.2%37.1%–41.2% | T | 847 / 2,163 | No answers at this threshold | 0.651 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Latency: median 36 ms · p90 37 ms Logic 30.8% | — | Decision | 38.7%36.6%–40.7% | T | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Latency: median 16 ms · p90 21 ms Logic 30.7% | — | Decision | 37.1%35.1%–39.1% | TU | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Latency: median 28 ms · p90 30 ms Logic 32.3% | — | Decision | 36.8%34.7%–38.8% | T | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Latency: median 50 ms · p90 54 ms Logic 31.7% | — | Decision | 35.3%33.3%–37.3% | TU | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Latency: median 20 ms · p90 21 ms Logic 29.2% | — | Decision | 32.1%30.2%–34.1% | UV | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 24 ms · p90 25 ms Logic 24.7% | — | Decision | 29.7%27.8%–31.6% | V | 642 / 2,163 | No answers at this threshold | 0.583 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Latency: median 35 ms · p90 41 ms Logic 22.4% | — | Decision | 28.7%26.8%–30.6% | V | 620 / 2,163 | No answers at this threshold | 0.569 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Latency: median 15 ms · p90 17 ms Logic 27.0% | — | Decision | 28.2%26.4%–30.2% | V | 611 / 2,163 | No answers at this threshold | 0.592 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 25 ms · p90 28 ms Logic 26.7% | — | Decision | 28.2%26.3%–30.1% | V | 610 / 2,163 | No answers at this threshold | 0.596 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Latency: median 12 ms · p90 13 ms Logic 23.6% | — | Decision | 22.7%20.9%–24.5% | W | 490 / 2,163 | No answers at this threshold | 0.501 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Latency: median 11 ms · p90 12 ms Logic 23.2% | — | Decision | 20.6%18.9%–22.3% | W | 445 / 2,163 | No answers at this threshold | 0.500 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents victorbennett Metask Lab OpenBMB Google Alibaba IBM KRAFTON 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GLiDE 0.4B: 20.6% at a median 11 ms
GLiClass v4 Edge: 22.7% at a median 12 ms
OneJev1.6-0.8B: 28.2% at a median 15 ms
d1-2B: 37.1% at a median 16 ms
GLiDE 0.8B: 32.1% at a median 20 ms
StartLux-Decision-v1.1-0.8B-Instant: 29.7% at a median 24 ms
KnowLine-0.8B-Gen1: 28.2% at a median 25 ms
GLiDE 2B: 36.8% at a median 28 ms
Perplexity Decider Flash v1.3: 55.9% at a median 28 ms
Chinese Decision Pointer 4B: 50.4% at a median 28 ms
vera-medium-omni: 56.8% at a median 31 ms
Strands Decider 2B: 28.7% at a median 35 ms
StartLux-Decision-v1.1-2B-Instant: 38.7% at a median 36 ms
Perplexity Decider Flash v1.2: 55.2% at a median 40 ms
Verdin-4B-v1.1: 51.5% at a median 47 ms
KnowLine-2B-Gen1: 35.3% at a median 50 ms
StartLux-Decision-v1.1-4B-Instant: 55.8% at a median 67 ms
KnowLine 4B Gen5.5: 56.0% at a median 74 ms
Metask-Jev-4B: 39.2% at a median 113 ms
LFM3-1.7B (on · chat): 98.9% at a median 3,893 ms
LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell): 98.9% at a median 3,990 ms
MiniCPM5-2B (off · chat): 98.2% at a median 6,353 ms
Gemma 4 E2B (on · chat · 32k cap · sampled): 97.7% at a median 7,137 ms
Gemma 4 E2B (chat): 96.8% at a median 8,120 ms
LFM2.5-2.6B (on · chat): 94.0% at a median 9,860 ms
Qwen3.5-4B (off · chat): 98.5% at a median 10,960 ms
Qwen3.5-2B (off · chat): 94.6% at a median 11,200 ms
MiniCPM5-2B (on · chat): 99.2% at a median 12,537 ms
Gemma 4 E2B (on · chat · 32k cap): 97.8% at a median 14,870 ms
Granite-4.2-3B (high · chat): 99.2% at a median 15,340 ms
LFM3-1.7B (off · chat): 98.7% at a median 19,570 ms
Raon-1 nano (off · chat): 89.6% at a median 25,550 ms (other engine: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Raon-1 nano (on · chat): 97.0% at a median 34,320 ms (other engine: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Qwen3.5-4B (on · chat): 99.4% at a median 46,630 ms
Gemma 4 E2B (on · chat): 96.9% at a median 64,440 ms
Qwen3.5-2B (on · chat): 94.7% at a median 98,390 ms
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents victorbennett Metask Lab OpenBMB Google Alibaba IBM KRAFTON
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Latency: median 46,630 ms · p90 96,460 ms Logic 92.4% | — | Chat | 99.4%99.0%–99.7% | a | 2,151 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Latency: median 12,537 ms · p90 32,720 ms Logic 88.9% | — | Chat | 99.2%99.2%–99.3% min–max of 3 seeds | ab | 2,147 / 2,163seed 0 run | Not measured | Not measured | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Latency: median 15,340 ms · p90 53,280 ms Logic 91.6% | — | Chat | 99.2%98.7%–99.5% | ab | 2,146 / 2,163 | Not measured | Not measured | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Latency: median 3,990 ms · p90 10,603 ms Logic 84.6% | — | Chat | 98.9%98.8%–99.0% min–max of 3 seeds | abc | 2,142 / 2,163seed 0 run | Not measured | Not measured | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Latency: median 3,893 ms · p90 10,353 ms Logic 84.2% | — | Chat | 98.9%98.8%–99.1% min–max of 3 seeds | abcd | 2,139 / 2,163seed 0 run | Not measured | Not measured | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Latency: median 19,570 ms · p90 53,410 ms Logic 85.5% | — | Chat | 98.7%98.1%–99.1% | abcde | 2,135 / 2,163 | Not measured | Not measured | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Latency: median 10,960 ms · p90 19,970 ms Logic 90.1% | — | Chat | 98.5%97.9%–98.9% | bcde | 2,130 / 2,163 | Not measured | Not measured | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Latency: median 6,353 ms · p90 15,697 ms Logic 83.6% | — | Chat | 98.2%98.1%–98.3% min–max of 3 seeds | cdef | 2,126 / 2,163seed 0 run | Not measured | Not measured | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 14,870 ms · p90 26,040 ms Logic 75.9% | — | Chat | 97.8%97.1%–98.4% | defg | 2,116 / 2,163 | Not measured | Not measured | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 7,137 ms · p90 14,787 ms Logic 76.8% | — | Chat | 97.7%97.6%–97.8% min–max of 3 seeds | efg | 2,111 / 2,163seed 0 run | Not measured | Not measured | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B | — | Chat | 97.0%96.2%–97.6% | fg | 2,098 / 2,163 | Not measured | Not measured | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 64,440 ms · p90 106,260 ms Logic 74.6% | — | Chat | 96.9%96.1%–97.6% | fg | 2,097 / 2,163 | Not measured | Not measured | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 8,120 ms · p90 13,650 ms Logic 72.9% | — | Chat | 96.8%95.9%–97.4% | g | 2,093 / 2,163 | Not measured | Not measured | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Latency: median 98,390 ms · p90 194,980 ms Logic 70.3% | — | Chat | 94.7%93.7%–95.6% | h | 2,049 / 2,163 | Not measured | Not measured | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Latency: median 11,200 ms · p90 32,140 ms Logic 64.8% | — | Chat | 94.6%93.6%–95.5% | h | 2,047 / 2,163 | Not measured | Not measured | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Latency: median 9,860 ms · p90 21,240 ms Logic 75.4% | — | Chat | 94.0%92.9%–94.9% | h | 2,033 / 2,163 | Not measured | Not measured | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B | — | Chat | 89.6%88.2%–90.8% | i | 1,937 / 2,163 | Not measured | Not measured | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Latency: median 31 ms · p90 35 ms Logic 44.1% | — | Decision | 56.8%54.7%–58.9% | j | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Latency: median 74 ms · p90 84 ms Logic 42.3% | — | Decision | 56.0%53.9%–58.1% | j | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Latency: median 28 ms · p90 31 ms Logic 44.7% | — | Decision | 55.9%53.8%–58.0% | j | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Latency: median 67 ms · p90 69 ms Logic 45.9% | — | Decision | 55.8%53.7%–57.9% | j | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Latency: median 40 ms · p90 47 ms Logic 43.1% | — | Decision | 55.2%53.1%–57.3% | j | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 47 ms · p90 52 ms Logic 39.7% | — | Decision | 51.5%49.4%–53.6% | k | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 28 ms · p90 32 ms Logic 40.4% | — | Decision | 50.4%48.3%–52.5% | k | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Latency: median 113 ms · p90 144 ms Logic 32.1% | — | Decision | 39.2%37.1%–41.2% | l | 847 / 2,163 | No answers at this threshold | 0.651 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Latency: median 36 ms · p90 37 ms Logic 30.8% | — | Decision | 38.7%36.6%–40.7% | l | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Latency: median 16 ms · p90 21 ms Logic 30.7% | — | Decision | 37.1%35.1%–39.1% | l | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Latency: median 28 ms · p90 30 ms Logic 32.3% | — | Decision | 36.8%34.7%–38.8% | l | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Latency: median 50 ms · p90 54 ms Logic 31.7% | — | Decision | 35.3%33.3%–37.3% | lm | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Latency: median 20 ms · p90 21 ms Logic 29.2% | — | Decision | 32.1%30.2%–34.1% | mn | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 24 ms · p90 25 ms Logic 24.7% | — | Decision | 29.7%27.8%–31.6% | n | 642 / 2,163 | No answers at this threshold | 0.583 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Latency: median 35 ms · p90 41 ms Logic 22.4% | — | Decision | 28.7%26.8%–30.6% | n | 620 / 2,163 | No answers at this threshold | 0.569 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Latency: median 15 ms · p90 17 ms Logic 27.0% | — | Decision | 28.2%26.4%–30.2% | n | 611 / 2,163 | No answers at this threshold | 0.592 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 25 ms · p90 28 ms Logic 26.7% | — | Decision | 28.2%26.3%–30.1% | n | 610 / 2,163 | No answers at this threshold | 0.596 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Latency: median 12 ms · p90 13 ms Logic 23.6% | — | Decision | 22.7%20.9%–24.5% | o | 490 / 2,163 | No answers at this threshold | 0.501 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Latency: median 11 ms · p90 12 ms Logic 23.2% | — | Decision | 20.6%18.9%–22.3% | o | 445 / 2,163 | No answers at this threshold | 0.500 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
newsbang tattrongvu PelaAI Blockbrain Labs Patrick-18 EldanRing Valen-Team Base Labs Cloudflare StartLux Apertix AI Fastino eCloud Technology Alibaba Google Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Adjudex 9B v5: 62.1% at a median 36 ms
Jeffan-9B-1211: 59.8% at a median 37 ms
KnowLine-9B-Gen1: 59.8% at a median 37 ms
Cygnet: 44.3% at a median 37 ms
KnowLine-9B-Gen2: 60.1% at a median 41 ms
Dexy-Decision-9B-V2: 58.8% at a median 45 ms
KnowLine-12B-Gen1: 66.3% at a median 49 ms
Winnow-12B: 46.9% at a median 52 ms
Valen-Decision-1.5-9B: 55.3% at a median 58 ms
Elephant-1: 61.3% at a median 80 ms
Clef-Flash: 49.8% at a median 99 ms
StartLux-Decision-v1.1-9B-Instant: 58.4% at a median 140 ms
Cerno 9B v1.2: 56.0% at a median 153 ms
GLiDE 9B: 57.9% at a median 174 ms
Erk-14B (off · chat): 90.8% at a median 7,890 ms
Qwen3-14B (off · chat): 95.8% at a median 8,040 ms
Gemma 4 E4B (chat): 98.1% at a median 10,770 ms
Gemma 4 E4B (on · chat · 32k cap · sampled): 98.5% at a median 11,190 ms
LFM2.5-8B-A1B (on · chat): 96.3% at a median 14,530 ms (mixed concurrency: This run changed the number of requests in flight part-way; see its record on Method)
Gemma 4 12B Unified (chat): 98.8% at a median 19,100 ms
Gemma 4 E4B (on · chat · 32k cap): 98.5% at a median 20,640 ms
Gemma 4 E4B (on · chat): 98.2% at a median 108,720 ms
newsbang tattrongvu PelaAI Blockbrain Labs Patrick-18 EldanRing Valen-Team Base Labs Cloudflare StartLux Apertix AI Fastino eCloud Technology Alibaba Google Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Latency: median 19,100 ms · p90 28,120 ms Logic 87.7% | — | Chat | 98.8%98.3%–99.2% | a | 2,138 / 2,163 | Not measured | Not measured | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 11,190 ms · p90 22,357 ms Logic 85.9% | — | Chat | 98.5%98.4%–98.8% min–max of 3 seeds | a | 2,136 / 2,163seed 0 run | Not measured | Not measured | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 20,640 ms · p90 34,280 ms Logic 85.0% | — | Chat | 98.5%97.9%–99.0% | a | 2,131 / 2,163 | Not measured | Not measured | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 108,720 ms · p90 175,470 ms Logic 82.6% | — | Chat | 98.2%97.6%–98.7% | a | 2,125 / 2,163 | Not measured | Not measured | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 10,770 ms · p90 17,700 ms Logic 83.2% | — | Chat | 98.1%97.4%–98.6% | a | 2,122 / 2,163 | Not measured | Not measured | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | Chat | 96.3%95.4%–97.0% | b | 2,082 / 2,163 | Not measured | Not measured | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Latency: median 8,040 ms · p90 18,980 ms Logic 65.3% | — | Chat | 95.8%94.9%–96.6% | b | 2,072 / 2,163 | Not measured | Not measured | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Latency: median 7,890 ms · p90 14,890 ms Logic 54.2% | — | Chat | 90.8%89.5%–91.9% | c | 1,963 / 2,163 | Not measured | Not measured | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Latency: median 49 ms · p90 55 ms Logic 50.6% | — | Decision | 66.3%64.2%–68.2% | d | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 36 ms · p90 44 ms Logic 46.4% | — | Decision | 62.1%60.0%–64.1% | e | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Latency: median 80 ms · p90 92 ms Logic 46.4% | — | Decision | 61.3%59.3%–63.4% | ef | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 41 ms · p90 46 ms Logic 43.9% | — | Decision | 60.1%58.0%–62.1% | efg | 1,300 / 2,163 | No answers at this threshold | 0.803 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 37 ms · p90 51 ms Logic 48.2% | — | Decision | 59.8%57.7%–61.9% | efg | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 37 ms · p90 43 ms Logic 44.3% | — | Decision | 59.8%57.7%–61.8% | efg | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Latency: median 45 ms · p90 50 ms Logic 45.3% | — | Decision | 58.8%56.7%–60.9% | fgh | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Latency: median 140 ms · p90 146 ms Logic 46.6% | — | Decision | 58.4%56.3%–60.5% | fgh | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Latency: median 174 ms · p90 182 ms Logic 47.8% | — | Decision | 57.9%55.8%–60.0% | gh | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Latency: median 153 ms · p90 175 ms Logic 43.3% | — | Decision | 56.0%53.9%–58.1% | h | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 58 ms · p90 64 ms Logic 44.4% | — | Decision | 55.3%53.2%–57.4% | h | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Latency: median 99 ms · p90 127 ms Logic 38.9% | — | Decision | 49.8%47.7%–51.9% | i | 1,077 / 2,163 | No answers at this threshold | 0.730 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Latency: median 52 ms · p90 61 ms Logic 39.7% | — | Decision | 46.9%44.8%–49.0% | ij | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Latency: median 37 ms · p90 41 ms Logic 39.3% | — | Decision | 44.3%42.3%–46.4% | j | 959 / 2,163 | No answers at this threshold | 0.735 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
vLLM Semantic Router Nace.AI Fastino OmniJev Perplexity PelaAI Poindexter Labs Base Labs MachinePulse Cloudflare Zefan Cai Turion.AI Sthānika AI Alibaba Google Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
d3: 70.4% at a median 55 ms
Drex Max: 70.3% at a median 76 ms
GLiDE (v1.2): 69.9% at a median 79 ms
OneJev-27B: 53.8% at a median 79 ms
Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): 72.5% at a median 91 ms
KnowLine-31B-Gen2: 80.3% at a median 114 ms
Copernicus-31B: 77.3% at a median 118 ms
d3-ultra: 71.5% at a median 118 ms
Mammoth-1: 74.0% at a median 140 ms
Karpo Decision 31B: 81.6% at a median 155 ms
Clef: 51.1% at a median 181 ms
Perplexity Decider 27B v1.2 (API): 69.8% at a median 226 ms
Open-Jev-27B-v1.1: 54.6% at a median 233 ms
Turion Jev 31B: 69.0% at a median 278 ms
Drex 2.0: 59.0% at a median 334 ms
Perplexity Decider 27B v1.2 (API · with web): 69.3% at a median 422 ms
GLiDE 27B: 72.0% at a median 425 ms
Sieve 31B v3: 77.1% at a median 637 ms
KnowLine-31B-Gen1.5: 77.4% at a median 764 ms
Qwen3.8-27B (low · chat): 99.2% at a median 10,020 ms
Gemma 4 31B (chat): 99.4% at a median 11,110 ms
LFM2.5-24B-A2B (on · chat): 99.0% at a median 27,950 ms
vLLM Semantic Router Nace.AI Fastino OmniJev Perplexity PelaAI Poindexter Labs Base Labs MachinePulse Cloudflare Zefan Cai Turion.AI Sthānika AI Alibaba Google Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Latency: median 11,110 ms · p90 16,690 ms Logic 93.3% | — | Chat | 99.4%99.0%–99.6% | a | 2,150 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Latency: median 10,020 ms · p90 20,630 ms Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | a | 2,146 / 2,163 | Not measured | Not measured | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Latency: median 27,950 ms · p90 60,210 ms Logic 86.7% | — | Chat | 99.0%98.5%–99.4% | a | 2,142 / 2,163 | Not measured | Not measured | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Latency: median 155 ms · p90 186 ms Logic 60.7% | — | Decision | 81.6%79.9%–83.1% | b | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Latency: median 114 ms · p90 120 ms Logic 58.9% | — | Decision | 80.3%78.5%–81.9% | b | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Latency: median 764 ms · p90 867 ms Logic 59.0% | — | Decision | 77.4%75.6%–79.1% | c | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Latency: median 118 ms · p90 142 ms Logic 58.7% | — | Decision | 77.3%75.5%–79.1% | c | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Latency: median 637 ms · p90 735 ms Logic 57.9% | — | Decision | 77.1%75.3%–78.8% | cd | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Latency: median 140 ms · p90 173 ms Logic 54.5% | — | Decision | 74.0%72.1%–75.8% | de | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Latency: median 91 ms · p90 103 ms Logic 54.8% | — | Decision | 72.5%70.6%–74.3% | ef | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Latency: median 425 ms · p90 461 ms Logic 58.4% | — | Decision | 72.0%70.1%–73.8% | efg | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 118 ms · p90 148 ms Logic 54.6% | — | Decision | 71.5%69.6%–73.4% | efg | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 55 ms · p90 62 ms Logic 54.5% | — | Decision | 70.4%68.5%–72.3% | fg | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Latency: median 76 ms · p90 89 ms Logic 53.1% | — | Decision | 70.3%68.4%–72.2% | fg | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Latency: median 79 ms · p90 84 ms Logic 55.2% | — | Decision | 69.9%67.9%–71.8% | fg | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Latency: median 226 ms · p90 368 ms Logic 53.3% | — | Decision | 69.8%67.8%–71.7% | g | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Latency: median 422 ms · p90 641 ms Logic 52.5% | with web | Decision | 69.3%67.3%–71.2% | g | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Latency: median 278 ms · p90 329 ms Logic 52.3% | — | Decision | 69.0%67.0%–70.9% | g | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Latency: median 334 ms · p90 432 ms Logic 41.9% | — | Decision | 59.0%56.9%–61.0% | h | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Latency: median 233 ms · p90 272 ms Logic 42.6% | — | Decision | 54.6%52.4%–56.6% | i | 1,180 / 2,163 | No answers at this threshold | 0.764 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Latency: median 79 ms · p90 106 ms Logic 40.6% | — | Decision | 53.8%51.7%–55.9% | i | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Latency: median 181 ms · p90 210 ms Logic 40.8% | — | Decision | 51.1%49.0%–53.2% | i | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
REAX Logits MiniMax DeepSeek Thinking Machines Z.ai Xiaomi 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
REAX 1.3: 78.0% at a median 189 ms
Logits Sigma-1 Duo: 75.2% at a median 196 ms
MiniMax-M3.1-Flash (high · chat): 98.2% at a median 1,670 ms
DeepSeek-V4.1-Flash (high · chat · 1017): 99.6% at a median 1,910 ms
Inkling-Small (high · chat · Jul 2026): 99.6% at a median 2,030 ms
Inkling 2 Small (high · chat): 99.5% at a median 2,110 ms
GLM-5.3-Flash (high · chat): 98.8% at a median 3,800 ms
DeepSeek-V4-Flash (chat): 99.3% at a median 8,130 ms
MiMo-V2.6-Flash (on · chat): 99.4% at a median 8,200 ms
REAX Logits MiniMax DeepSeek Thinking Machines Z.ai Xiaomi
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Latency: median 1,910 ms · p90 3,570 ms Logic 96.2% | — | Chat | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Latency: median 2,030 ms · p90 6,330 ms Logic 95.4% | — | Chat | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Latency: median 2,110 ms · p90 6,000 ms Logic 95.6% | — | Chat | 99.5%99.2%–99.7% | a | 2,153 / 2,163 | Not measured | Not measured | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Latency: median 8,200 ms · p90 18,090 ms Logic 95.1% | — | Chat | 99.4%99.0%–99.7% | ab | 2,151 / 2,163 | Not measured | Not measured | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Latency: median 8,130 ms · p90 16,920 ms Logic running | — | Chat | 99.3%98.9%–99.6% | ab | 2,148 / 2,163 | Not measured | Not measured | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Latency: median 3,800 ms · p90 7,970 ms Logic 93.2% | — | Chat | 98.8%98.2%–99.2% | bc | 2,137 / 2,163 | Not measured | Not measured | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Latency: median 1,670 ms · p90 7,470 ms Logic 94.0% | — | Chat | 98.2%97.6%–98.7% | c | 2,125 / 2,163 | Not measured | Not measured | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Latency: median 189 ms · p90 232 ms Logic 59.3% | — | Decision | 78.0%76.2%–79.7% | d | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Latency: median 196 ms · p90 221 ms Logic 57.8% | — | Decision | 75.2%73.4%–77.0% | e | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
TypeSafe Liquid AI Google Fastino Inception Microsoft Nace.AI Subquadratic Celeris Meta OpenAI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Jev 1.14: 65.1% at a median 163 ms
Jev 1.13: 64.1% at a median 166 ms
d1.1: 65.2% at a median 172 ms
d1-ultra: 74.1% at a median 179 ms
d1: 55.6% at a median 184 ms
Gemini Decision 1 (20261111): 71.3% at a median 226 ms
GLiDE (API): 70.8% at a median 264 ms
Mercury Decide: 53.5% at a median 288 ms
Microsoft-Decision-1: 59.4% at a median 310 ms
Drex Max v1.1: 70.1% at a median 326 ms
SubQ Snap 1.1: 68.6% at a median 381 ms
celeris-1.1-decision: 69.9% at a median 381 ms
celeris-1-decision: 70.0% at a median 387 ms
Muse Ember (max · chat): 99.6% at a median 1,260 ms
GPT-6 Luna (chat): 93.7% at a median 3,830 ms
TypeSafe Liquid AI Google Fastino Inception Microsoft Nace.AI Subquadratic Celeris Meta OpenAI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Latency: median 1,260 ms · p90 2,270 ms Logic 95.5% | — | Chat | 99.6%99.2%–99.8% | a | 2,154 / 2,163 | Not measured | Not measured | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,830 ms · p90 5,000 ms Logic 58.0% | — | Chat | 93.7%92.6%–94.6% | b | 2,026 / 2,163 | Not measured | Not measured | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Latency: median 179 ms · p90 426 ms Logic 54.9% | — | Decision | 74.1%72.2%–75.9% | c | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Latency: median 226 ms · p90 351 ms Logic 50.0% | — | Decision | 71.3%69.4%–73.2% | cd | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Latency: median 264 ms · p90 279 ms Logic 54.5% | — | Decision | 70.8%68.8%–72.7% | d | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Latency: median 326 ms · p90 473 ms Logic 54.8% | — | Decision | 70.1%68.1%–72.0% | d | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 387 ms · p90 391 ms Logic 47.3% | — | Decision | 70.0%68.0%–71.9% | d | 1,514 / 2,163 | No answers at this threshold | 0.858 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 381 ms · p90 386 ms Logic 47.3% | — | Decision | 69.9%67.9%–71.8% | d | 1,511 / 2,163 | No answers at this threshold | 0.859 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Latency: median 381 ms · p90 390 ms Logic 52.4% | — | Decision | 68.6%66.6%–70.5% | d | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 172 ms · p90 290 ms Logic 46.0% | — | Decision | 65.2%63.2%–67.2% | e | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 163 ms · p90 185 ms Logic 48.2% | — | Decision | 65.1%63.1%–67.1% | e | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 166 ms · p90 206 ms Logic 46.3% | — | Decision | 64.1%62.0%–66.1% | e | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Latency: median 310 ms · p90 359 ms Logic 44.4% | — | Decision | 59.4%57.3%–61.4% | f | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 184 ms · p90 297 ms Logic 40.8% | — | Decision | 55.6%53.5%–57.7% | g | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Latency: median 288 ms · p90 342 ms Logic 45.1% | — | Decision | 53.5%51.4%–55.6% | g | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Anthropic 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Claude Sonnet 5.5 (chat): 99.7% at a median 3,460 ms
Anthropic
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,460 ms · p90 5,970 ms Logic 95.5% | — | Chat | 99.7%99.3%–99.8% | a | 2,156 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Anthropic 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Claude Opus 5.5 (chat): 99.8% at a median 3,200 ms
Claude Fable 5.1 (chat): 99.7% at a median 5,030 ms
Anthropic
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,200 ms · p90 4,790 ms Logic 96.2% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 5,030 ms · p90 6,490 ms Logic 95.2% | — | Chat | 99.7%99.4%–99.9% | a | 2,157 / 2,163 | Not measured | Not measured | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Liquid AI OpenAI Alibaba xAI Google 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
d1.2: 65.5% at a median 204 ms
d1.2-ultra: 69.9% at a median 215 ms
GPT-6.1 Terra (chat): 99.2% at a median 4,080 ms
Qwen4-Max (high · chat): 99.8% at a median 4,710 ms
GPT-6.1 Sol (chat): 99.6% at a median 4,810 ms
GPT-6 Astra (chat): 99.6% at a median 5,280 ms
Qwen3.8 Flash (off · chat): 99.5% at a median 5,740 ms
Grok 4.7 (chat): 99.5% at a median 11,100 ms
Gemini 4 Carbon (chat): 99.6% at a median 13,410 ms
Liquid AI OpenAI Alibaba xAI Google
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)).
Every setup, every measure
| Setup | Web | Models | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Latency: median 4,710 ms · p90 9,130 ms Logic 96.3% | — | Chat | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,280 ms · p90 7,260 ms Logic 96.7% | — | Chat | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,810 ms · p90 6,560 ms Logic 96.5% | — | Chat | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Latency: median 13,410 ms · p90 27,800 ms Logic 96.7% | — | Chat | 99.6%99.2%–99.8% | ab | 2,154 / 2,163 | Not measured | Not measured | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 11,100 ms · p90 66,530 ms Logic running | — | Chat | 99.5%99.2%–99.7% | ab | 2,153 / 2,163 | Not measured | Not measured | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Latency: median 5,740 ms · p90 11,800 ms Logic 94.0% | — | Chat | 99.5%99.1%–99.7% | ab | 2,152 / 2,163 | Not measured | Not measured | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,080 ms · p90 5,870 ms Logic 94.0% | — | Chat | 99.2%98.7%–99.5% | b | 2,146 / 2,163 | Not measured | Not measured | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 215 ms · p90 347 ms Logic 51.6% | — | Decision | 69.9%67.9%–71.8% | c | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 204 ms · p90 386 ms Logic 49.2% | — | Decision | 65.5%63.5%–67.5% | d | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents newsbang tattrongvu Blockbrain Labs Patrick-18 victorbennett EldanRing vLLM Semantic Router Valen-Team Nace.AI Base Labs Cloudflare Metask Lab Poindexter Labs Apertix AI MachinePulse TypeSafe REAX Logits Google Zefan Cai Turion.AI Inception Microsoft Subquadratic Celeris Sthānika AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GLiDE 0.4B: 20.6% at a median 11 ms
GLiClass v4 Edge: 22.7% at a median 12 ms
OneJev1.6-0.8B: 28.2% at a median 15 ms
d1-2B: 37.1% at a median 16 ms
GLiDE 0.8B: 32.1% at a median 20 ms
StartLux-Decision-v1.1-0.8B-Instant: 29.7% at a median 24 ms
KnowLine-0.8B-Gen1: 28.2% at a median 25 ms
GLiDE 2B: 36.8% at a median 28 ms
Perplexity Decider Flash v1.3: 55.9% at a median 28 ms
Chinese Decision Pointer 4B: 50.4% at a median 28 ms
vera-medium-omni: 56.8% at a median 31 ms
Strands Decider 2B: 28.7% at a median 35 ms
StartLux-Decision-v1.1-2B-Instant: 38.7% at a median 36 ms
Adjudex 9B v5: 62.1% at a median 36 ms
Jeffan-9B-1211: 59.8% at a median 37 ms
KnowLine-9B-Gen1: 59.8% at a median 37 ms
Cygnet: 44.3% at a median 37 ms
Perplexity Decider Flash v1.2: 55.2% at a median 40 ms
KnowLine-9B-Gen2: 60.1% at a median 41 ms
Dexy-Decision-9B-V2: 58.8% at a median 45 ms
Verdin-4B-v1.1: 51.5% at a median 47 ms
KnowLine-12B-Gen1: 66.3% at a median 49 ms
KnowLine-2B-Gen1: 35.3% at a median 50 ms
Winnow-12B: 46.9% at a median 52 ms
d3: 70.4% at a median 55 ms
Valen-Decision-1.5-9B: 55.3% at a median 58 ms
StartLux-Decision-v1.1-4B-Instant: 55.8% at a median 67 ms
KnowLine 4B Gen5.5: 56.0% at a median 74 ms
Drex Max: 70.3% at a median 76 ms
GLiDE (v1.2): 69.9% at a median 79 ms
OneJev-27B: 53.8% at a median 79 ms
Elephant-1: 61.3% at a median 80 ms
Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): 72.5% at a median 91 ms
Clef-Flash: 49.8% at a median 99 ms
Metask-Jev-4B: 39.2% at a median 113 ms
KnowLine-31B-Gen2: 80.3% at a median 114 ms
Copernicus-31B: 77.3% at a median 118 ms
d3-ultra: 71.5% at a median 118 ms
Mammoth-1: 74.0% at a median 140 ms
StartLux-Decision-v1.1-9B-Instant: 58.4% at a median 140 ms
Cerno 9B v1.2: 56.0% at a median 153 ms
Karpo Decision 31B: 81.6% at a median 155 ms
Jev 1.14: 65.1% at a median 163 ms
Jev 1.13: 64.1% at a median 166 ms
d1.1: 65.2% at a median 172 ms
GLiDE 9B: 57.9% at a median 174 ms
d1-ultra: 74.1% at a median 179 ms
Clef: 51.1% at a median 181 ms
d1: 55.6% at a median 184 ms
REAX 1.3: 78.0% at a median 189 ms
Logits Sigma-1 Duo: 75.2% at a median 196 ms
d1.2: 65.5% at a median 204 ms
d1.2-ultra: 69.9% at a median 215 ms
Gemini Decision 1 (20261111): 71.3% at a median 226 ms
Perplexity Decider 27B v1.2 (API): 69.8% at a median 226 ms
Open-Jev-27B-v1.1: 54.6% at a median 233 ms
GLiDE (API): 70.8% at a median 264 ms
Turion Jev 31B: 69.0% at a median 278 ms
Mercury Decide: 53.5% at a median 288 ms
Microsoft-Decision-1: 59.4% at a median 310 ms
Drex Max v1.1: 70.1% at a median 326 ms
Drex 2.0: 59.0% at a median 334 ms
SubQ Snap 1.1: 68.6% at a median 381 ms
celeris-1.1-decision: 69.9% at a median 381 ms
celeris-1-decision: 70.0% at a median 387 ms
Perplexity Decider 27B v1.2 (API · with web): 69.3% at a median 422 ms
GLiDE 27B: 72.0% at a median 425 ms
Sieve 31B v3: 77.1% at a median 637 ms
KnowLine-31B-Gen1.5: 77.4% at a median 764 ms
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents newsbang tattrongvu Blockbrain Labs Patrick-18 victorbennett EldanRing vLLM Semantic Router Valen-Team Nace.AI Base Labs Cloudflare Metask Lab Poindexter Labs Apertix AI MachinePulse TypeSafe REAX Logits Google Zefan Cai Turion.AI Inception Microsoft Subquadratic Celeris Sthānika AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Latency: median 155 ms · p90 186 ms Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Latency: median 114 ms · p90 120 ms Logic 58.9% | — | 80.3%78.5%–81.9% | ab | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Latency: median 189 ms · p90 232 ms Logic 59.3% | — | 78.0%76.2%–79.7% | bc | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Latency: median 764 ms · p90 867 ms Logic 59.0% | — | 77.4%75.6%–79.1% | cd | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Latency: median 118 ms · p90 142 ms Logic 58.7% | — | 77.3%75.5%–79.1% | cd | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Latency: median 637 ms · p90 735 ms Logic 57.9% | — | 77.1%75.3%–78.8% | cd | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Latency: median 196 ms · p90 221 ms Logic 57.8% | — | 75.2%73.4%–77.0% | de | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Latency: median 179 ms · p90 426 ms Logic 54.9% | — | 74.1%72.2%–75.9% | defg | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Latency: median 140 ms · p90 173 ms Logic 54.5% | — | 74.0%72.1%–75.8% | defh | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Latency: median 91 ms · p90 103 ms Logic 54.8% | — | 72.5%70.6%–74.3% | efghi | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Latency: median 425 ms · p90 461 ms Logic 58.4% | — | 72.0%70.1%–73.8% | efghij | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 118 ms · p90 148 ms Logic 54.6% | — | 71.5%69.6%–73.4% | fghij | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Latency: median 226 ms · p90 351 ms Logic 50.0% | — | 71.3%69.4%–73.2% | efghij | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Latency: median 264 ms · p90 279 ms Logic 54.5% | — | 70.8%68.8%–72.7% | fghij | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 55 ms · p90 62 ms Logic 54.5% | — | 70.4%68.5%–72.3% | gij | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Latency: median 76 ms · p90 89 ms Logic 53.1% | — | 70.3%68.4%–72.2% | gijk | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Latency: median 326 ms · p90 473 ms Logic 54.8% | — | 70.1%68.1%–72.0% | ijk | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 387 ms · p90 391 ms Logic 47.3% | — | 70.0%68.0%–71.9% | hijk | 1,514 / 2,163 | No answers at this threshold | 0.858 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Latency: median 79 ms · p90 84 ms Logic 55.2% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 381 ms · p90 386 ms Logic 47.3% | — | 69.9%67.9%–71.8% | ijkl | 1,511 / 2,163 | No answers at this threshold | 0.859 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 215 ms · p90 347 ms Logic 51.6% | — | 69.9%67.9%–71.8% | ijk | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Latency: median 226 ms · p90 368 ms Logic 53.3% | — | 69.8%67.8%–71.7% | ijkl | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Latency: median 422 ms · p90 641 ms Logic 52.5% | with web | 69.3%67.3%–71.2% | jklm | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Latency: median 278 ms · p90 329 ms Logic 52.3% | — | 69.0%67.0%–70.9% | ijklmn | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Latency: median 381 ms · p90 390 ms Logic 52.4% | — | 68.6%66.6%–70.5% | jklmn | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Latency: median 49 ms · p90 55 ms Logic 50.6% | — | 66.3%64.2%–68.2% | klmno | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 204 ms · p90 386 ms Logic 49.2% | — | 65.5%63.5%–67.5% | lmnop | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 172 ms · p90 290 ms Logic 46.0% | — | 65.2%63.2%–67.2% | mnop | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 163 ms · p90 185 ms Logic 48.2% | — | 65.1%63.1%–67.1% | nop | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 166 ms · p90 206 ms Logic 46.3% | — | 64.1%62.0%–66.1% | opq | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 36 ms · p90 44 ms Logic 46.4% | — | 62.1%60.0%–64.1% | opqr | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Latency: median 80 ms · p90 92 ms Logic 46.4% | — | 61.3%59.3%–63.4% | pqr | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 41 ms · p90 46 ms Logic 43.9% | — | 60.1%58.0%–62.1% | qrs | 1,300 / 2,163 | No answers at this threshold | 0.803 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 37 ms · p90 51 ms Logic 48.2% | — | 59.8%57.7%–61.9% | qrs | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 37 ms · p90 43 ms Logic 44.3% | — | 59.8%57.7%–61.8% | qrs | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Latency: median 310 ms · p90 359 ms Logic 44.4% | — | 59.4%57.3%–61.4% | rst | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Latency: median 334 ms · p90 432 ms Logic 41.9% | — | 59.0%56.9%–61.0% | rstu | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Latency: median 45 ms · p90 50 ms Logic 45.3% | — | 58.8%56.7%–60.9% | rstu | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Latency: median 140 ms · p90 146 ms Logic 46.6% | — | 58.4%56.3%–60.5% | rstuv | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Latency: median 174 ms · p90 182 ms Logic 47.8% | — | 57.9%55.8%–60.0% | rstuvw | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Latency: median 31 ms · p90 35 ms Logic 44.1% | — | 56.8%54.7%–58.9% | stuvw | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Latency: median 153 ms · p90 175 ms Logic 43.3% | — | 56.0%53.9%–58.1% | stuvw | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Latency: median 74 ms · p90 84 ms Logic 42.3% | — | 56.0%53.9%–58.1% | stuvw | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Latency: median 28 ms · p90 31 ms Logic 44.7% | — | 55.9%53.8%–58.0% | stuvw | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Latency: median 67 ms · p90 69 ms Logic 45.9% | — | 55.8%53.7%–57.9% | stuvwx | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 184 ms · p90 297 ms Logic 40.8% | — | 55.6%53.5%–57.7% | stuvwxy | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 58 ms · p90 64 ms Logic 44.4% | — | 55.3%53.2%–57.4% | tuvwxy | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Latency: median 40 ms · p90 47 ms Logic 43.1% | — | 55.2%53.1%–57.3% | tuvwxy | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Latency: median 233 ms · p90 272 ms Logic 42.6% | — | 54.6%52.4%–56.6% | uvwxyz | 1,180 / 2,163 | No answers at this threshold | 0.764 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Latency: median 79 ms · p90 106 ms Logic 40.6% | — | 53.8%51.7%–55.9% | vwxyzA | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Latency: median 288 ms · p90 342 ms Logic 45.1% | — | 53.5%51.4%–55.6% | wxyzA | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 47 ms · p90 52 ms Logic 39.7% | — | 51.5%49.4%–53.6% | yzAB | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Latency: median 181 ms · p90 210 ms Logic 40.8% | — | 51.1%49.0%–53.2% | xyzAB | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 28 ms · p90 32 ms Logic 40.4% | — | 50.4%48.3%–52.5% | zAB | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Latency: median 99 ms · p90 127 ms Logic 38.9% | — | 49.8%47.7%–51.9% | AB | 1,077 / 2,163 | No answers at this threshold | 0.730 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Latency: median 52 ms · p90 61 ms Logic 39.7% | — | 46.9%44.8%–49.0% | BC | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Latency: median 37 ms · p90 41 ms Logic 39.3% | — | 44.3%42.3%–46.4% | C | 959 / 2,163 | No answers at this threshold | 0.735 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Latency: median 113 ms · p90 144 ms Logic 32.1% | — | 39.2%37.1%–41.2% | D | 847 / 2,163 | No answers at this threshold | 0.651 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Latency: median 36 ms · p90 37 ms Logic 30.8% | — | 38.7%36.6%–40.7% | D | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Latency: median 16 ms · p90 21 ms Logic 30.7% | — | 37.1%35.1%–39.1% | D | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Latency: median 28 ms · p90 30 ms Logic 32.3% | — | 36.8%34.7%–38.8% | D | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Latency: median 50 ms · p90 54 ms Logic 31.7% | — | 35.3%33.3%–37.3% | DE | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Latency: median 20 ms · p90 21 ms Logic 29.2% | — | 32.1%30.2%–34.1% | EF | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 24 ms · p90 25 ms Logic 24.7% | — | 29.7%27.8%–31.6% | F | 642 / 2,163 | No answers at this threshold | 0.583 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Latency: median 35 ms · p90 41 ms Logic 22.4% | — | 28.7%26.8%–30.6% | F | 620 / 2,163 | No answers at this threshold | 0.569 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Latency: median 15 ms · p90 17 ms Logic 27.0% | — | 28.2%26.4%–30.2% | F | 611 / 2,163 | No answers at this threshold | 0.592 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 25 ms · p90 28 ms Logic 26.7% | — | 28.2%26.3%–30.1% | F | 610 / 2,163 | No answers at this threshold | 0.596 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Latency: median 12 ms · p90 13 ms Logic 23.6% | — | 22.7%20.9%–24.5% | G | 490 / 2,163 | No answers at this threshold | 0.501 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Latency: median 11 ms · p90 12 ms Logic 23.2% | — | 20.6%18.9%–22.3% | G | 445 / 2,163 | No answers at this threshold | 0.500 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents victorbennett Metask Lab 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GLiDE 0.4B: 20.6% at a median 11 ms
GLiClass v4 Edge: 22.7% at a median 12 ms
OneJev1.6-0.8B: 28.2% at a median 15 ms
d1-2B: 37.1% at a median 16 ms
GLiDE 0.8B: 32.1% at a median 20 ms
StartLux-Decision-v1.1-0.8B-Instant: 29.7% at a median 24 ms
KnowLine-0.8B-Gen1: 28.2% at a median 25 ms
GLiDE 2B: 36.8% at a median 28 ms
Perplexity Decider Flash v1.3: 55.9% at a median 28 ms
Chinese Decision Pointer 4B: 50.4% at a median 28 ms
vera-medium-omni: 56.8% at a median 31 ms
Strands Decider 2B: 28.7% at a median 35 ms
StartLux-Decision-v1.1-2B-Instant: 38.7% at a median 36 ms
Perplexity Decider Flash v1.2: 55.2% at a median 40 ms
Verdin-4B-v1.1: 51.5% at a median 47 ms
KnowLine-2B-Gen1: 35.3% at a median 50 ms
StartLux-Decision-v1.1-4B-Instant: 55.8% at a median 67 ms
KnowLine 4B Gen5.5: 56.0% at a median 74 ms
Metask-Jev-4B: 39.2% at a median 113 ms
Fastino Other OmniJev Liquid AI StartLux PelaAI Perplexity quincyqiang scar-ai Strands Agents victorbennett Metask Lab
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| vera-medium-omni Open · 4.2B vera-medium-omni · scar-ai · decision model Parameters: 4.2B dense Base model: Qwen3.5-4B Latency: median 31 ms · p90 35 ms Logic 44.1% | — | 56.8%54.7%–58.9% | a | 1,229 / 2,163 | 33 right of 33 kept0 mistakes · 2,130 skipped | 0.789 | 3.8%answered 98% | 31 ms | 35 ms | $0.023$23.34estimate · RTX PRO 6000 Blackwell |
| KnowLine 4B Gen5.5 Open · 4.0B KnowLine 4B Gen5.5 · PelaAI · decision model Parameters: 4B dense Latency: median 74 ms · p90 84 ms Logic 42.3% | — | 56.0%53.9%–58.1% | a | 1,211 / 2,163 | 54 right of 54 kept0 mistakes · 2,109 skipped | 0.765 | 1.6%answered 99% | 74 ms | 84 ms | $0.005$4.56estimate · DGX Spark |
| Perplexity Decider Flash v1.3 Open · 4.0B Perplexity Decider Flash v1.3 · Perplexity · decision model Parameters: 4B dense Latency: median 28 ms · p90 31 ms Logic 44.7% | — | 55.9%53.8%–58.0% | a | 1,210 / 2,163 | 113 right of 114 kept1 wrong · 2,049 skipped | 0.787 | 8.4%answered 96% | 28 ms | 31 ms | $0.022$22.38estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-4B-Instant Open · 4.0B StartLux-Decision-v1.1-4B-Instant · StartLux · decision model Parameters: 4B dense Latency: median 67 ms · p90 69 ms Logic 45.9% | — | 55.8%53.7%–57.9% | a | 1,208 / 2,163 | 96 right of 96 kept0 mistakes · 2,067 skipped | 0.779 | 13.4%answered 94% | 67 ms | 69 ms | $0.004$4.10estimate · DGX Spark |
| Perplexity Decider Flash v1.2 Open · 4.0B Perplexity Decider Flash v1.2 · Perplexity · decision model Parameters: 4B dense Latency: median 40 ms · p90 47 ms Logic 43.1% | — | 55.2%53.1%–57.3% | a | 1,195 / 2,163 | 150 right of 155 kept5 wrong · 2,008 skipped | 0.781 | 12.0%answered 94% | 40 ms | 47 ms | $0.002$2.18 list$0.030$30.06estimate · RTX PRO 6000 Blackwell |
| Verdin-4B-v1.1 Open · 4.0B Verdin-4B-v1.1 · victorbennett · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 47 ms · p90 52 ms Logic 39.7% | — | 51.5%49.4%–53.6% | b | 1,114 / 2,163 | 126 right of 130 kept4 wrong · 2,033 skipped | 0.752 | 6.4%answered 97% | 47 ms | 52 ms | $0.034$33.58estimate · RTX PRO 6000 Blackwell |
| Chinese Decision Pointer 4B Open · 4.0B Chinese Decision Pointer 4B · quincyqiang · decision model Parameters: 4B dense Base model: Qwen3.5-4B Latency: median 28 ms · p90 32 ms Logic 40.4% | — | 50.4%48.3%–52.5% | b | 1,091 / 2,163 | 149 right of 155 kept6 wrong · 2,008 skipped | 0.746 | 3.4%answered 98% | 28 ms | 32 ms | $0.023$22.70estimate · RTX PRO 6000 Blackwell |
| Metask-Jev-4B Open · 4.5B Metask-Jev-4B · Metask Lab · decision model Parameters: 4.54B dense Latency: median 113 ms · p90 144 ms Logic 32.1% | — | 39.2%37.1%–41.2% | c | 847 / 2,163 | No answers at this threshold | 0.651 | 2.2%answered 99% | 113 ms | 144 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-2B-Instant Open · 2.0B StartLux-Decision-v1.1-2B-Instant · StartLux · decision model Parameters: 2B dense Latency: median 36 ms · p90 37 ms Logic 30.8% | — | 38.7%36.6%–40.7% | c | 836 / 2,163 | 2 right of 2 kept0 mistakes · 2,161 skipped | 0.685 | 7.2%answered 95% | 36 ms | 37 ms | $0.002$2.27estimate · DGX Spark |
| d1-2B Open · 1.9B d1-2B · Liquid AI · decision model Parameters: 1.92B dense Base model: LFM3-VL-2B Latency: median 16 ms · p90 21 ms Logic 30.7% | — | 37.1%35.1%–39.1% | c | 802 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.656 | 80.2%answered 34% | 16 ms | 21 ms | $0.014$14.07estimate · RTX PRO 6000 Blackwell |
| GLiDE 2B Open · 1.9B GLiDE 2B · Fastino · decision model Parameters: 1.88B dense Latency: median 28 ms · p90 30 ms Logic 32.3% | — | 36.8%34.7%–38.8% | c | 795 / 2,163 | 15 right of 16 kept1 wrong · 2,147 skipped | 0.635 | 17.3%answered 89% | 28 ms | 30 ms | $0.002$1.78estimate · DGX Spark |
| KnowLine-2B-Gen1 Open · 2.0B KnowLine-2B-Gen1 · PelaAI · decision model Parameters: 2B dense Base model: Qwen3.5-2B Latency: median 50 ms · p90 54 ms Logic 31.7% | — | 35.3%33.3%–37.3% | cd | 763 / 2,163 | 8 right of 8 kept0 mistakes · 2,155 skipped | 0.636 | 1.7%answered 99% | 50 ms | 54 ms | $0.003$3.20estimate · DGX Spark |
| GLiDE 0.8B Open · 0.8B GLiDE 0.8B · Fastino · decision model Parameters: 0.75B dense Latency: median 20 ms · p90 21 ms Logic 29.2% | — | 32.1%30.2%–34.1% | de | 694 / 2,163 | 4 right of 4 kept0 mistakes · 2,159 skipped | 0.589 | 20.8%answered 85% | 20 ms | 21 ms | $0.001$1.28estimate · DGX Spark |
| StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 24 ms · p90 25 ms Logic 24.7% | — | 29.7%27.8%–31.6% | ef | 642 / 2,163 | No answers at this threshold | 0.583 | 11.3%answered 92% | 24 ms | 25 ms | $0.002$1.53estimate · DGX Spark |
| Strands Decider 2B Open · 2.0B Strands Decider 2B · Strands Agents · decision model Parameters: 2B dense Latency: median 35 ms · p90 41 ms Logic 22.4% | — | 28.7%26.8%–30.6% | ef | 620 / 2,163 | No answers at this threshold | 0.569 | 3.2%answered 98% | 35 ms | 41 ms | $0.026$26.22estimate · RTX PRO 6000 Blackwell |
| OneJev1.6-0.8B Open · 0.8B OneJev1.6-0.8B · OmniJev · decision model Parameters: 0.8B dense Latency: median 15 ms · p90 17 ms Logic 27.0% | — | 28.2%26.4%–30.2% | f | 611 / 2,163 | No answers at this threshold | 0.592 | 0.6%answered 100% | 15 ms | 17 ms | $0.001$1.09estimate · DGX Spark |
| KnowLine-0.8B-Gen1 Open · 0.8B KnowLine-0.8B-Gen1 · PelaAI · decision model Parameters: 0.8B dense Base model: Qwen3.5-0.8B Latency: median 25 ms · p90 28 ms Logic 26.7% | — | 28.2%26.3%–30.1% | f | 610 / 2,163 | No answers at this threshold | 0.596 | 0.3%answered 100% | 25 ms | 28 ms | $0.002$1.69estimate · DGX Spark |
| GLiClass v4 Edge Open · 0.4B GLiClass v4 Edge · decision model Parameters: 0.4B dense Base model: DeBERTa-v3-large Latency: median 12 ms · p90 13 ms Logic 23.6% | — | 22.7%20.9%–24.5% | g | 490 / 2,163 | No answers at this threshold | 0.501 | 17.2%answered 86% | 12 ms | 13 ms | $0.001$0.85estimate · DGX Spark |
| GLiDE 0.4B Open · 0.4B GLiDE 0.4B · Fastino · decision model Parameters: 0.40B dense Latency: median 11 ms · p90 12 ms Logic 23.2% | — | 20.6%18.9%–22.3% | g | 445 / 2,163 | No answers at this threshold | 0.500 | 72.8%answered 33% | 11 ms | 12 ms | $0.001$0.77estimate · DGX Spark |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
newsbang tattrongvu PelaAI Blockbrain Labs Patrick-18 EldanRing Valen-Team Base Labs Cloudflare StartLux Apertix AI Fastino 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Adjudex 9B v5: 62.1% at a median 36 ms
Jeffan-9B-1211: 59.8% at a median 37 ms
KnowLine-9B-Gen1: 59.8% at a median 37 ms
Cygnet: 44.3% at a median 37 ms
KnowLine-9B-Gen2: 60.1% at a median 41 ms
Dexy-Decision-9B-V2: 58.8% at a median 45 ms
KnowLine-12B-Gen1: 66.3% at a median 49 ms
Winnow-12B: 46.9% at a median 52 ms
Valen-Decision-1.5-9B: 55.3% at a median 58 ms
Elephant-1: 61.3% at a median 80 ms
Clef-Flash: 49.8% at a median 99 ms
StartLux-Decision-v1.1-9B-Instant: 58.4% at a median 140 ms
Cerno 9B v1.2: 56.0% at a median 153 ms
GLiDE 9B: 57.9% at a median 174 ms
newsbang tattrongvu PelaAI Blockbrain Labs Patrick-18 EldanRing Valen-Team Base Labs Cloudflare StartLux Apertix AI Fastino
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-12B-Gen1 Open · 12B KnowLine-12B-Gen1 · PelaAI · decision model Parameters: 12B dense Base model: Gemma 4 12B Latency: median 49 ms · p90 55 ms Logic 50.6% | — | 66.3%64.2%–68.2% | a | 1,433 / 2,163 | 7 right of 7 kept0 mistakes · 2,156 skipped | 0.819 | 1.1%answered 100% | 49 ms | 55 ms | $0.036$36.13estimate · RTX PRO 6000 Blackwell |
| Adjudex 9B v5 Open · 9.0B Adjudex 9B v5 · newsbang · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 36 ms · p90 44 ms Logic 46.4% | — | 62.1%60.0%–64.1% | b | 1,343 / 2,163 | 197 right of 198 kept1 wrong · 1,965 skipped | 0.806 | 3.1%answered 99% | 36 ms | 44 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Elephant-1 Open · 9.0B Elephant-1 · Base Labs · decision model Parameters: 9B dense Latency: median 80 ms · p90 92 ms Logic 46.4% | — | 61.3%59.3%–63.4% | bc | 1,327 / 2,163 | 182 right of 183 kept1 wrong · 1,980 skipped | 0.803 | 65.0%answered 67% | 80 ms | 92 ms | $0.008$7.90 list$0.058$58.20estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen2 Open · 9.0B KnowLine-9B-Gen2 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 41 ms · p90 46 ms Logic 43.9% | — | 60.1%58.0%–62.1% | bcd | 1,300 / 2,163 | No answers at this threshold | 0.803 | 3.2%answered 99% | 41 ms | 46 ms | $0.031$31.02estimate · RTX PRO 6000 Blackwell |
| Jeffan-9B-1211 Open · 9.0B Jeffan-9B-1211 · tattrongvu · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 37 ms · p90 51 ms Logic 48.2% | — | 59.8%57.7%–61.9% | bcd | 1,294 / 2,163 | 64 right of 64 kept0 mistakes · 2,099 skipped | 0.814 | 6.6%answered 97% | 37 ms | 51 ms | $0.029$28.78estimate · RTX PRO 6000 Blackwell |
| KnowLine-9B-Gen1 Open · 9.0B KnowLine-9B-Gen1 · PelaAI · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 37 ms · p90 43 ms Logic 44.3% | — | 59.8%57.7%–61.8% | bcd | 1,293 / 2,163 | 123 right of 123 kept0 mistakes · 2,040 skipped | 0.799 | 0.9%answered 100% | 37 ms | 43 ms | $0.027$26.54estimate · RTX PRO 6000 Blackwell |
| Dexy-Decision-9B-V2 Open · 9.0B Dexy-Decision-9B-V2 · Patrick-18 · decision model Parameters: 9B dense Base model: vllm-sr/d3-flash Latency: median 45 ms · p90 50 ms Logic 45.3% | — | 58.8%56.7%–60.9% | cde | 1,272 / 2,163 | 207 right of 210 kept3 wrong · 1,953 skipped | 0.804 | 1.7%answered 99% | 45 ms | 50 ms | $0.032$31.98estimate · RTX PRO 6000 Blackwell |
| StartLux-Decision-v1.1-9B-Instant Open · 9.0B StartLux-Decision-v1.1-9B-Instant · StartLux · decision model Parameters: 9B dense Latency: median 140 ms · p90 146 ms Logic 46.6% | — | 58.4%56.3%–60.5% | cde | 1,264 / 2,163 | 49 right of 49 kept0 mistakes · 2,114 skipped | 0.781 | 4.2%answered 98% | 140 ms | 146 ms | $0.008$8.34estimate · DGX Spark |
| GLiDE 9B Open · 7.9B GLiDE 9B · Fastino · decision model Parameters: 7.94B dense Latency: median 174 ms · p90 182 ms Logic 47.8% | — | 57.9%55.8%–60.0% | de | 1,253 / 2,163 | 73 right of 73 kept0 mistakes · 2,090 skipped | 0.798 | 24.6%answered 89% | 174 ms | 182 ms | $0.010$10.49estimate · DGX Spark |
| Cerno 9B v1.2 Open · 9.0B Cerno 9B v1.2 · Apertix AI · decision model Parameters: 9B dense Latency: median 153 ms · p90 175 ms Logic 43.3% | — | 56.0%53.9%–58.1% | e | 1,212 / 2,163 | 244 right of 254 kept10 wrong · 1,909 skipped | 0.780 | 1.2%answered 99% | 153 ms | 175 ms | $0.010$9.51estimate · DGX Spark |
| Valen-Decision-1.5-9B Open · 9.0B Valen-Decision-1.5-9B · Valen-Team · decision model Parameters: 9B dense Base model: Qwen3.5-9B Latency: median 58 ms · p90 64 ms Logic 44.4% | — | 55.3%53.2%–57.4% | e | 1,197 / 2,163 | 130 right of 130 kept0 mistakes · 2,033 skipped | 0.790 | 2.6%answered 99% | 58 ms | 64 ms | $0.041$40.93estimate · RTX PRO 6000 Blackwell |
| Clef-Flash Open · 9.0B Clef-Flash · Cloudflare · decision model Parameters: 9B dense Latency: median 99 ms · p90 127 ms Logic 38.9% | — | 49.8%47.7%–51.9% | f | 1,077 / 2,163 | No answers at this threshold | 0.730 | 2.0%answered 99% | 99 ms | 127 ms | $0.011$10.85 list$0.072$71.95estimate · RTX PRO 6000 Blackwell |
| Winnow-12B Open · 12B Winnow-12B · EldanRing · decision model Parameters: 12B dense Latency: median 52 ms · p90 61 ms Logic 39.7% | — | 46.9%44.8%–49.0% | fg | 1,014 / 2,163 | 20 right of 21 kept1 wrong · 2,142 skipped | 0.743 | 9.2%answered 95% | 52 ms | 61 ms | $0.038$38.37estimate · RTX PRO 6000 Blackwell |
| Cygnet Open · 12B Cygnet · Blockbrain Labs · decision model Parameters: 11.95B dense Base model: Gemma 4 12B IT Latency: median 37 ms · p90 41 ms Logic 39.3% | — | 44.3%42.3%–46.4% | g | 959 / 2,163 | No answers at this threshold | 0.735 | 4.9%answered 97% | 37 ms | 41 ms | $0.024$23.94 list$0.028$27.50estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
vLLM Semantic Router Nace.AI Fastino OmniJev Perplexity PelaAI Poindexter Labs Base Labs MachinePulse Cloudflare Zefan Cai Turion.AI Sthānika AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
d3: 70.4% at a median 55 ms
Drex Max: 70.3% at a median 76 ms
GLiDE (v1.2): 69.9% at a median 79 ms
OneJev-27B: 53.8% at a median 79 ms
Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): 72.5% at a median 91 ms
KnowLine-31B-Gen2: 80.3% at a median 114 ms
Copernicus-31B: 77.3% at a median 118 ms
d3-ultra: 71.5% at a median 118 ms
Mammoth-1: 74.0% at a median 140 ms
Karpo Decision 31B: 81.6% at a median 155 ms
Clef: 51.1% at a median 181 ms
Perplexity Decider 27B v1.2 (API): 69.8% at a median 226 ms
Open-Jev-27B-v1.1: 54.6% at a median 233 ms
Turion Jev 31B: 69.0% at a median 278 ms
Drex 2.0: 59.0% at a median 334 ms
Perplexity Decider 27B v1.2 (API · with web): 69.3% at a median 422 ms
GLiDE 27B: 72.0% at a median 425 ms
Sieve 31B v3: 77.1% at a median 637 ms
KnowLine-31B-Gen1.5: 77.4% at a median 764 ms
vLLM Semantic Router Nace.AI Fastino OmniJev Perplexity PelaAI Poindexter Labs Base Labs MachinePulse Cloudflare Zefan Cai Turion.AI Sthānika AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Karpo Decision 31B Open · 31B Karpo Decision 31B · MachinePulse · decision model Parameters: 31B dense Latency: median 155 ms · p90 186 ms Logic 60.7% | — | 81.6%79.9%–83.1% | a | 1,764 / 2,163 | 837 right of 837 kept0 mistakes · 1,326 skipped | 0.903 | 4.8%answered 99% | 155 ms | 186 ms | $0.113$113estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen2 Open · 31B KnowLine-31B-Gen2 · PelaAI · decision model Parameters: 31B dense Base model: Gemma 4 31B Latency: median 114 ms · p90 120 ms Logic 58.9% | — | 80.3%78.5%–81.9% | a | 1,736 / 2,163 | 469 right of 470 kept1 wrong · 1,693 skipped | 0.898 | 0.2%answered 100% | 114 ms | 120 ms | $0.083$83.14estimate · RTX PRO 6000 Blackwell |
| KnowLine-31B-Gen1.5 Open · 31B KnowLine-31B-Gen1.5 · PelaAI · decision model Parameters: 31B dense Latency: median 764 ms · p90 867 ms Logic 59.0% | — | 77.4%75.6%–79.1% | b | 1,674 / 2,163 | 464 right of 465 kept1 wrong · 1,698 skipped | 0.895 | 1.1%answered 100% | 764 ms | 867 ms | $0.047$46.87estimate · DGX Spark |
| Copernicus-31B Open · 31B Copernicus-31B · Poindexter Labs · decision model Parameters: 31B dense Latency: median 118 ms · p90 142 ms Logic 58.7% | — | 77.3%75.5%–79.1% | b | 1,673 / 2,163 | 318 right of 318 kept0 mistakes · 1,845 skipped | 0.899 | 1.7%answered 100% | 118 ms | 142 ms | $0.084$84.42estimate · RTX PRO 6000 Blackwell |
| Sieve 31B v3 Open · 31B Sieve 31B v3 · Sthānika AI · decision model Parameters: 31B dense Latency: median 637 ms · p90 735 ms Logic 57.9% | — | 77.1%75.3%–78.8% | bc | 1,668 / 2,163 | 215 right of 215 kept0 mistakes · 1,948 skipped | 0.891 | 2.1%answered 100% | 637 ms | 735 ms | $0.039$39.14estimate · DGX Spark |
| Mammoth-1 Open · 27B Mammoth-1 · Base Labs · decision model Parameters: 27B dense Latency: median 140 ms · p90 173 ms Logic 54.5% | — | 74.0%72.1%–75.8% | cd | 1,600 / 2,163 | 412 right of 412 kept0 mistakes · 1,751 skipped | 0.895 | 70.6%answered 75% | 140 ms | 173 ms | $0.023$22.56 list$0.104$104estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model Parameters: 27B dense Latency: median 91 ms · p90 103 ms Logic 54.8% | — | 72.5%70.6%–74.3% | de | 1,568 / 2,163 | 323 right of 323 kept0 mistakes · 1,840 skipped | 0.856 | 9.4%answered 97% | 91 ms | 103 ms | $0.066$65.87estimate · RTX PRO 6000 Blackwell |
| GLiDE 27B Open · 27B GLiDE 27B · Fastino · decision model Parameters: 27B dense Latency: median 425 ms · p90 461 ms Logic 58.4% | — | 72.0%70.1%–73.8% | def | 1,557 / 2,163 | 198 right of 199 kept1 wrong · 1,964 skipped | 0.864 | 21.6%answered 94% | 425 ms | 461 ms | $0.026$26.16estimate · DGX Spark |
| d3-ultra Open · 26B d3-ultra · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 118 ms · p90 148 ms Logic 54.6% | — | 71.5%69.6%–73.4% | def | 1,547 / 2,163 | 11 right of 11 kept0 mistakes · 2,152 skipped | 0.857 | 5.3%answered 98% | 118 ms | 148 ms | $0.089$88.58estimate · RTX PRO 6000 Blackwell |
| d3 Open · 26B d3 · vLLM Semantic Router · decision model Parameters: 26.09B dense Latency: median 55 ms · p90 62 ms Logic 54.5% | — | 70.4%68.5%–72.3% | ef | 1,523 / 2,163 | 162 right of 163 kept1 wrong · 2,000 skipped | 0.878 | 4.7%answered 99% | 55 ms | 62 ms | $0.041$40.61estimate · RTX PRO 6000 Blackwell |
| Drex Max Open · 27B Drex Max · Nace.AI · decision model Parameters: 27B dense Latency: median 76 ms · p90 89 ms Logic 53.1% | — | 70.3%68.4%–72.2% | ef | 1,521 / 2,163 | 719 right of 729 kept10 wrong · 1,434 skipped | 0.861 | 4.5%answered 99% | 76 ms | 89 ms | $0.013$13.06 list$0.055$54.68estimate · RTX PRO 6000 Blackwell |
| GLiDE v1.2 Open · 27B GLiDE v1.2 · Fastino · decision model Parameters: 27B dense Latency: median 79 ms · p90 84 ms Logic 55.2% | — | 69.9%67.9%–71.8% | ef | 1,511 / 2,163 | 131 right of 131 kept0 mistakes · 2,032 skipped | 0.862 | 35.8%answered 88% | 79 ms | 84 ms | $0.008$7.55 list$0.058$57.88estimate · RTX PRO 6000 Blackwell |
| Perplexity Decider 27B v1.2 API Open · 27B Perplexity Decider 27B v1.2 API · Perplexity · decision model Parameters: 27B dense Latency: median 226 ms · p90 368 ms Logic 53.3% | — | 69.8%67.8%–71.7% | f | 1,509 / 2,163 | 331 right of 331 kept0 mistakes · 1,832 skipped | 0.869 | 7.0%answered 98% | 226 ms | 368 ms | $0.004$4.29 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model Parameters: 27B dense Latency: median 422 ms · p90 641 ms Logic 52.5% | with web | 69.3%67.3%–71.2% | f | 1,499 / 2,163 | 326 right of 326 kept0 mistakes · 1,837 skipped | 0.873 | 15.4%answered 95% | 422 ms | 641 ms | $1.023$1,023 |
| Turion Jev 31B Open · 31B Turion Jev 31B · Turion.AI · decision model Parameters: 31B dense Latency: median 278 ms · p90 329 ms Logic 52.3% | — | 69.0%67.0%–70.9% | f | 1,493 / 2,163 | 43 right of 43 kept0 mistakes · 2,120 skipped | 0.864 | 2.2%answered 99% | 278 ms | 329 ms | $0.017$17.46estimate · DGX Spark |
| Drex 2.0 Open · 35B MoE · 3.0B active Drex 2.0 · Nace.AI · decision model Parameters: 35B MoE · 3B active Latency: median 334 ms · p90 432 ms Logic 41.9% | — | 59.0%56.9%–61.0% | g | 1,276 / 2,163 | 174 right of 174 kept0 mistakes · 1,989 skipped | 0.813 | 20.7%answered 91% | 334 ms | 432 ms | $0.009$8.92 |
| Open-Jev-27B-v1.1 Open · 27B Open-Jev-27B-v1.1 · Zefan Cai · decision model Parameters: 27B dense Base model: Qwen3.8-27B + adapter Latency: median 233 ms · p90 272 ms Logic 42.6% | — | 54.6%52.4%–56.6% | h | 1,180 / 2,163 | No answers at this threshold | 0.764 | 5.0%answered 98% | 233 ms | 272 ms | $0.169$169estimate · RTX PRO 6000 Blackwell |
| OneJev-27B Open · 27B OneJev-27B · OmniJev · decision model Parameters: 27B dense Latency: median 79 ms · p90 106 ms Logic 40.6% | — | 53.8%51.7%–55.9% | h | 1,163 / 2,163 | 26 right of 26 kept0 mistakes · 2,137 skipped | 0.783 | 1.8%answered 99% | 79 ms | 106 ms | $0.061$60.76estimate · RTX PRO 6000 Blackwell |
| Clef Open · 27B Clef · Cloudflare · decision model Parameters: 27B dense Latency: median 181 ms · p90 210 ms Logic 40.8% | — | 51.1%49.0%–53.2% | h | 1,105 / 2,163 | 12 right of 12 kept0 mistakes · 2,151 skipped | 0.767 | 1.6%answered 99% | 181 ms | 210 ms | $0.069$68.54 list$0.131$131estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
REAX Logits 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
REAX 1.3: 78.0% at a median 189 ms
Logits Sigma-1 Duo: 75.2% at a median 196 ms
REAX Logits
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| REAX 1.3 Open · 57B REAX 1.3 · REAX · decision model Parameters: 57.36B dense Latency: median 189 ms · p90 232 ms Logic 59.3% | — | 78.0%76.2%–79.7% | a | 1,687 / 2,163 | 178 right of 178 kept0 mistakes · 1,985 skipped | 0.903 | 2.3%answered 99% | 189 ms | 232 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| Logits Sigma-1 Duo Open · 57B Logits Sigma-1 Duo · Logits · decision model Parameters: 57.36B dense Latency: median 196 ms · p90 221 ms Logic 57.8% | — | 75.2%73.4%–77.0% | b | 1,627 / 2,163 | 15 right of 15 kept0 mistakes · 2,148 skipped | 0.899 | 2.1%answered 99% | 196 ms | 221 ms | $0.141$141estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
TypeSafe Liquid AI Google Fastino Inception Microsoft Nace.AI Subquadratic Celeris 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Jev 1.14: 65.1% at a median 163 ms
Jev 1.13: 64.1% at a median 166 ms
d1.1: 65.2% at a median 172 ms
d1-ultra: 74.1% at a median 179 ms
d1: 55.6% at a median 184 ms
Gemini Decision 1 (20261111): 71.3% at a median 226 ms
GLiDE (API): 70.8% at a median 264 ms
Mercury Decide: 53.5% at a median 288 ms
Microsoft-Decision-1: 59.4% at a median 310 ms
Drex Max v1.1: 70.1% at a median 326 ms
SubQ Snap 1.1: 68.6% at a median 381 ms
celeris-1.1-decision: 69.9% at a median 381 ms
celeris-1-decision: 70.0% at a median 387 ms
TypeSafe Liquid AI Google Fastino Inception Microsoft Nace.AI Subquadratic Celeris
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 2,151 of 2,163); shown once.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| d1-ultra Closed · Efficient d1-ultra · Liquid AI · decision model Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) Latency: median 179 ms · p90 426 ms Logic 54.9% | — | 74.1%72.2%–75.9% | a | 1,603 / 2,163 | 213 right of 213 kept0 mistakes · 1,950 skipped | 0.876 | 87.4%answered 61% | 179 ms | 426 ms | $0.006$6.14 |
| Gemini Decision 1 20261111 Closed · Efficient Gemini Decision 1 20261111 · Google · decision model Parameters: not disclosed by maker (closed weights) Latency: median 226 ms · p90 351 ms Logic 50.0% | — | 71.3%69.4%–73.2% | ab | 1,543 / 2,163 | 214 right of 214 kept0 mistakes · 1,949 skipped | 0.859 | 3.9%answered 99% | 226 ms | 351 ms | $0.022$22.02 |
| GLiDE API Closed · Efficient GLiDE API · Fastino · decision model Parameters: not disclosed by maker Latency: median 264 ms · p90 279 ms Logic 54.5% | — | 70.8%68.8%–72.7% | b | 1,531 / 2,163 | 161 right of 161 kept0 mistakes · 2,002 skipped | 0.865 | 40.6%answered 86% | 264 ms | 279 ms | $0.028$28.30 |
| Drex Max v1.1 Closed · Efficient (provisional) Drex Max v1.1 · Nace.AI · decision model Parameters: not disclosed by maker Latency: median 326 ms · p90 473 ms Logic 54.8% | — | 70.1%68.1%–72.0% | b | 1,516 / 2,163 | 315 right of 315 kept0 mistakes · 1,848 skipped | 0.867 | 43.5%answered 84% | 326 ms | 473 ms | $0.013$12.88 |
| celeris-1-decision Closed · Efficient celeris-1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 387 ms · p90 391 ms Logic 47.3% | — | 70.0%68.0%–71.9% | b | 1,514 / 2,163 | No answers at this threshold | 0.858 | 82.7%answered 62% | 387 ms | 391 ms | $0.009$9.10 |
| celeris-1.1-decision Closed · Efficient celeris-1.1-decision · Celeris · decision model Parameters: not disclosed by maker Latency: median 381 ms · p90 386 ms Logic 47.3% | — | 69.9%67.9%–71.8% | b | 1,511 / 2,163 | No answers at this threshold | 0.859 | 82.9%answered 62% | 381 ms | 386 ms | $0.001$0.91 |
| SubQ Snap 1.1 Closed · Efficient SubQ Snap 1.1 · Subquadratic · decision model Parameters: not disclosed by maker Shown once with SubQ Snap Preview: the same answer to every question Latency: median 381 ms · p90 390 ms Logic 52.4% | — | 68.6%66.6%–70.5% | b | 1,483 / 2,163 | 360 right of 361 kept1 wrong · 1,802 skipped | 0.839 | 52.5%answered 80% | 381 ms | 390 ms | $0.009$8.59 |
| d1.1 Closed · Efficient d1.1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 172 ms · p90 290 ms Logic 46.0% | — | 65.2%63.2%–67.2% | c | 1,410 / 2,163 | 183 right of 183 kept0 mistakes · 1,980 skipped | 0.829 | 82.5%answered 56% | 172 ms | 290 ms | $0.003$2.92 |
| Jev 1.14 Closed · Efficient Jev 1.14 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 163 ms · p90 185 ms Logic 48.2% | — | 65.1%63.1%–67.1% | c | 1,409 / 2,163 | 202 right of 202 kept0 mistakes · 1,961 skipped | 0.829 | 7.7%answered 97% | 163 ms | 185 ms | $0.019$18.99 |
| Jev 1.13 Closed · Efficient Jev 1.13 · TypeSafe · decision model Parameters: not disclosed by maker Latency: median 166 ms · p90 206 ms Logic 46.3% | — | 64.1%62.0%–66.1% | c | 1,386 / 2,163 | 248 right of 248 kept0 mistakes · 1,915 skipped | 0.834 | 5.7%answered 98% | 166 ms | 206 ms | $0.019$18.99 |
| Microsoft-Decision-1 Closed · Efficient Microsoft-Decision-1 · Microsoft · decision model Parameters: not disclosed by maker Base model: Qwen3.5-9B Latency: median 310 ms · p90 359 ms Logic 44.4% | — | 59.4%57.3%–61.4% | d | 1,284 / 2,163 | 27 right of 27 kept0 mistakes · 2,136 skipped | 0.787 | 5.1%answered 98% | 310 ms | 359 ms | $0.007$6.81 |
| d1 Closed · Efficient d1 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 184 ms · p90 297 ms Logic 40.8% | — | 55.6%53.5%–57.7% | e | 1,203 / 2,163 | 152 right of 152 kept0 mistakes · 2,011 skipped | 0.767 | 2.9%answered 99% | 184 ms | 297 ms | $0.003$2.92 |
| Mercury Decide Closed · Efficient Mercury Decide · Inception · decision model Parameters: not disclosed by maker Latency: median 288 ms · p90 342 ms Logic 45.1% | — | 53.5%51.4%–55.6% | e | 1,157 / 2,163 | 272 right of 281 kept9 wrong · 1,882 skipped | 0.789 | 3.2%answered 99% | 288 ms | 342 ms | $0.004$4.38 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
No workhorse decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
No flagship decision-model setup has answers on Math yet. Decision-model runs appear here as soon as the first one stores answers.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
d1.2: 65.5% at a median 204 ms
d1.2-ultra: 69.9% at a median 215 ms
Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| d1.2-ultra Closed · Undisclosed d1.2-ultra · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 215 ms · p90 347 ms Logic 51.6% | — | 69.9%67.9%–71.8% | a | 1,511 / 2,163 | 115 right of 115 kept0 mistakes · 2,048 skipped | 0.876 | 86.7%answered 58% | 215 ms | 347 ms | $0.006$6.14 |
| d1.2 Closed · Undisclosed d1.2 · Liquid AI · decision model Parameters: not disclosed by maker Latency: median 204 ms · p90 386 ms Logic 49.2% | — | 65.5%63.5%–67.5% | b | 1,417 / 2,163 | 141 right of 141 kept0 mistakes · 2,022 skipped | 0.838 | 56.3%answered 76% | 204 ms | 386 ms | $0.003$2.92 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Meta MiniMax DeepSeek Thinking Machines Anthropic Z.ai OpenAI Other Google Alibaba OpenBMB eCloud Technology Xiaomi Liquid AI xAI IBM KRAFTON 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Muse Ember (max · chat): 99.6% at a median 1,260 ms
MiniMax-M3.1-Flash (high · chat): 98.2% at a median 1,670 ms
DeepSeek-V4.1-Flash (high · chat · 1017): 99.6% at a median 1,910 ms
Inkling-Small (high · chat · Jul 2026): 99.6% at a median 2,030 ms
Inkling 2 Small (high · chat): 99.5% at a median 2,110 ms
Claude Opus 5.5 (chat): 99.8% at a median 3,200 ms
Claude Sonnet 5.5 (chat): 99.7% at a median 3,460 ms
GLM-5.3-Flash (high · chat): 98.8% at a median 3,800 ms
GPT-6 Luna (chat): 93.7% at a median 3,830 ms
LFM3-1.7B (on · chat): 98.9% at a median 3,893 ms
LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell): 98.9% at a median 3,990 ms
GPT-6.1 Terra (chat): 99.2% at a median 4,080 ms
Gemini 4 Hydrogen (chat): 99.6% at a median 4,400 ms
Qwen4-Max (high · chat): 99.8% at a median 4,710 ms
GPT-6.1 Sol (chat): 99.6% at a median 4,810 ms
Claude Fable 5.1 (chat): 99.7% at a median 5,030 ms
GPT-6 Astra (chat): 99.6% at a median 5,280 ms
Qwen3.8 Flash (off · chat): 99.5% at a median 5,740 ms
MiniCPM5-2B (off · chat): 98.2% at a median 6,353 ms
Gemma 4 E2B (on · chat · 32k cap · sampled): 97.7% at a median 7,137 ms
Erk-14B (off · chat): 90.8% at a median 7,890 ms
Qwen3-14B (off · chat): 95.8% at a median 8,040 ms
Gemma 4 E2B (chat): 96.8% at a median 8,120 ms
DeepSeek-V4-Flash (chat): 99.3% at a median 8,130 ms
MiMo-V2.6-Flash (on · chat): 99.4% at a median 8,200 ms
LFM2.5-2.6B (on · chat): 94.0% at a median 9,860 ms
Qwen3.8-27B (low · chat): 99.2% at a median 10,020 ms
Gemma 4 E4B (chat): 98.1% at a median 10,770 ms
Qwen3.5-4B (off · chat): 98.5% at a median 10,960 ms
Grok 4.7 (chat): 99.5% at a median 11,100 ms
Gemma 4 31B (chat): 99.4% at a median 11,110 ms
Gemma 4 E4B (on · chat · 32k cap · sampled): 98.5% at a median 11,190 ms
Qwen3.5-2B (off · chat): 94.6% at a median 11,200 ms
MiniCPM5-2B (on · chat): 99.2% at a median 12,537 ms
Gemini 4 Carbon (chat): 99.6% at a median 13,410 ms
LFM2.5-8B-A1B (on · chat): 96.3% at a median 14,530 ms (mixed concurrency: This run changed the number of requests in flight part-way; see its record on Method)
Gemma 4 E2B (on · chat · 32k cap): 97.8% at a median 14,870 ms
Granite-4.2-3B (high · chat): 99.2% at a median 15,340 ms
Gemma 4 12B Unified (chat): 98.8% at a median 19,100 ms
LFM3-1.7B (off · chat): 98.7% at a median 19,570 ms
Gemma 4 E4B (on · chat · 32k cap): 98.5% at a median 20,640 ms
Raon-1 nano (off · chat): 89.6% at a median 25,550 ms (other engine: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
LFM2.5-24B-A2B (on · chat): 99.0% at a median 27,950 ms
Raon-1 nano (on · chat): 97.0% at a median 34,320 ms (other engine: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Qwen3.5-4B (on · chat): 99.4% at a median 46,630 ms
Gemma 4 E2B (on · chat): 96.9% at a median 64,440 ms
Qwen3.5-2B (on · chat): 94.7% at a median 98,390 ms
Gemma 4 E4B (on · chat): 98.2% at a median 108,720 ms
Meta MiniMax DeepSeek Thinking Machines Anthropic Z.ai OpenAI Other Google Alibaba OpenBMB eCloud Technology Xiaomi Liquid AI xAI IBM KRAFTON
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC); Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)); Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC); Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test); GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,200 ms · p90 4,790 ms Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Latency: median 4,710 ms · p90 9,130 ms Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 5,030 ms · p90 6,490 ms Logic 95.2% | — | 99.7%99.4%–99.9% | abc | 2,157 / 2,163 | Not measured | Not measured | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,460 ms · p90 5,970 ms Logic 95.5% | — | 99.7%99.3%–99.8% | abd | 2,156 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Latency: median 1,910 ms · p90 3,570 ms Logic 96.2% | — | 99.6%99.3%–99.8% | abcdef | 2,155 / 2,163 | Not measured | Not measured | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,280 ms · p90 7,260 ms Logic 96.7% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,810 ms · p90 6,560 ms Logic 96.5% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Latency: median 2,030 ms · p90 6,330 ms Logic 95.4% | — | 99.6%99.3%–99.8% | abce | 2,155 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Latency: median 13,410 ms · p90 27,800 ms Logic 96.7% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Gemini 4 Hydrogen chat Gemini 4 Hydrogen chat · Google · chat model Parameters: not on record yet Latency: median 4,400 ms · p90 6,680 ms Logic 96.1% | — | 99.6%99.2%–99.8% | abcdef | 2,154 / 2,163 | Not measured | Not measured | 85.7%answered 99% | 4,400 ms | 6,680 ms | — |
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Latency: median 1,260 ms · p90 2,270 ms Logic 95.5% | — | 99.6%99.2%–99.8% | abcdefg | 2,154 / 2,163 | Not measured | Not measured | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 11,100 ms · p90 66,530 ms Logic running | — | 99.5%99.2%–99.7% | abcdefg | 2,153 / 2,163 | Not measured | Not measured | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Latency: median 2,110 ms · p90 6,000 ms Logic 95.6% | — | 99.5%99.2%–99.7% | abcdef | 2,153 / 2,163 | Not measured | Not measured | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Latency: median 5,740 ms · p90 11,800 ms Logic 94.0% | — | 99.5%99.1%–99.7% | abcdefgh | 2,152 / 2,163 | Not measured | Not measured | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Latency: median 8,200 ms · p90 18,090 ms Logic 95.1% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Latency: median 46,630 ms · p90 96,460 ms Logic 92.4% | — | 99.4%99.0%–99.7% | abcdefgh | 2,151 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Latency: median 11,110 ms · p90 16,690 ms Logic 93.3% | — | 99.4%99.0%–99.6% | abcdefgh | 2,150 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Latency: median 8,130 ms · p90 16,920 ms Logic running | — | 99.3%98.9%–99.6% | abcdefghij | 2,148 / 2,163 | Not measured | Not measured | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Latency: median 12,537 ms · p90 32,720 ms Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | abcdefghi | 2,147 / 2,163seed 0 run | Not measured | Not measured | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,080 ms · p90 5,870 ms Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Latency: median 15,340 ms · p90 53,280 ms Logic 91.6% | — | 99.2%98.7%–99.5% | abcdefghi | 2,146 / 2,163 | Not measured | Not measured | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Latency: median 10,020 ms · p90 20,630 ms Logic 94.0% | — | 99.2%98.7%–99.5% | abcdefghij | 2,146 / 2,163 | Not measured | Not measured | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Latency: median 27,950 ms · p90 60,210 ms Logic 86.7% | — | 99.0%98.5%–99.4% | bcdefghij | 2,142 / 2,163 | Not measured | Not measured | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Latency: median 3,990 ms · p90 10,603 ms Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abcdefghij | 2,142 / 2,163seed 0 run | Not measured | Not measured | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Latency: median 3,893 ms · p90 10,353 ms Logic 84.2% | — | 98.9%98.8%–99.1% min–max of 3 seeds | cefghijk | 2,139 / 2,163seed 0 run | Not measured | Not measured | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Latency: median 19,100 ms · p90 28,120 ms Logic 87.7% | — | 98.8%98.3%–99.2% | dfghijkl | 2,138 / 2,163 | Not measured | Not measured | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Latency: median 3,800 ms · p90 7,970 ms Logic 93.2% | — | 98.8%98.2%–99.2% | fghijkl | 2,137 / 2,163 | Not measured | Not measured | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Latency: median 19,570 ms · p90 53,410 ms Logic 85.5% | — | 98.7%98.1%–99.1% | ghijkl | 2,135 / 2,163 | Not measured | Not measured | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 11,190 ms · p90 22,357 ms Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | efghijkl | 2,136 / 2,163seed 0 run | Not measured | Not measured | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 20,640 ms · p90 34,280 ms Logic 85.0% | — | 98.5%97.9%–99.0% | hijkl | 2,131 / 2,163 | Not measured | Not measured | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Latency: median 10,960 ms · p90 19,970 ms Logic 90.1% | — | 98.5%97.9%–98.9% | hijklm | 2,130 / 2,163 | Not measured | Not measured | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 108,720 ms · p90 175,470 ms Logic 82.6% | — | 98.2%97.6%–98.7% | ijklm | 2,125 / 2,163 | Not measured | Not measured | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Latency: median 1,670 ms · p90 7,470 ms Logic 94.0% | — | 98.2%97.6%–98.7% | ijklmn | 2,125 / 2,163 | Not measured | Not measured | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Latency: median 6,353 ms · p90 15,697 ms Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | ijklmn | 2,126 / 2,163seed 0 run | Not measured | Not measured | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 10,770 ms · p90 17,700 ms Logic 83.2% | — | 98.1%97.4%–98.6% | jklmn | 2,122 / 2,163 | Not measured | Not measured | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 14,870 ms · p90 26,040 ms Logic 75.9% | — | 97.8%97.1%–98.4% | klmno | 2,116 / 2,163 | Not measured | Not measured | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 7,137 ms · p90 14,787 ms Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | lmnop | 2,111 / 2,163seed 0 run | Not measured | Not measured | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B | — | 97.0%96.2%–97.6% | mnop | 2,098 / 2,163 | Not measured | Not measured | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 64,440 ms · p90 106,260 ms Logic 74.6% | — | 96.9%96.1%–97.6% | mnop | 2,097 / 2,163 | Not measured | Not measured | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 8,120 ms · p90 13,650 ms Logic 72.9% | — | 96.8%95.9%–97.4% | nopq | 2,093 / 2,163 | Not measured | Not measured | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 96.3%95.4%–97.0% | opqr | 2,082 / 2,163 | Not measured | Not measured | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Latency: median 8,040 ms · p90 18,980 ms Logic 65.3% | — | 95.8%94.9%–96.6% | pqrs | 2,072 / 2,163 | Not measured | Not measured | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Latency: median 98,390 ms · p90 194,980 ms Logic 70.3% | — | 94.7%93.7%–95.6% | qrs | 2,049 / 2,163 | Not measured | Not measured | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Latency: median 11,200 ms · p90 32,140 ms Logic 64.8% | — | 94.6%93.6%–95.5% | rs | 2,047 / 2,163 | Not measured | Not measured | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Latency: median 9,860 ms · p90 21,240 ms Logic 75.4% | — | 94.0%92.9%–94.9% | rs | 2,033 / 2,163 | Not measured | Not measured | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,830 ms · p90 5,000 ms Logic 58.0% | — | 93.7%92.6%–94.6% | s | 2,026 / 2,163 | Not measured | Not measured | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Latency: median 7,890 ms · p90 14,890 ms Logic 54.2% | — | 90.8%89.5%–91.9% | t | 1,963 / 2,163 | Not measured | Not measured | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B | — | 89.6%88.2%–90.8% | t | 1,937 / 2,163 | Not measured | Not measured | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Other OpenBMB Google Liquid AI Alibaba IBM KRAFTON 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
LFM3-1.7B (on · chat): 98.9% at a median 3,893 ms
LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell): 98.9% at a median 3,990 ms
MiniCPM5-2B (off · chat): 98.2% at a median 6,353 ms
Gemma 4 E2B (on · chat · 32k cap · sampled): 97.7% at a median 7,137 ms
Gemma 4 E2B (chat): 96.8% at a median 8,120 ms
LFM2.5-2.6B (on · chat): 94.0% at a median 9,860 ms
Qwen3.5-4B (off · chat): 98.5% at a median 10,960 ms
Qwen3.5-2B (off · chat): 94.6% at a median 11,200 ms
MiniCPM5-2B (on · chat): 99.2% at a median 12,537 ms
Gemma 4 E2B (on · chat · 32k cap): 97.8% at a median 14,870 ms
Granite-4.2-3B (high · chat): 99.2% at a median 15,340 ms
LFM3-1.7B (off · chat): 98.7% at a median 19,570 ms
Raon-1 nano (off · chat): 89.6% at a median 25,550 ms (other engine: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Raon-1 nano (on · chat): 97.0% at a median 34,320 ms (other engine: served on KRAFTON's vLLM fork 13a5dd87, not stock vLLM: cost and speed not comparable; see its record on Method)
Qwen3.5-4B (on · chat): 99.4% at a median 46,630 ms
Gemma 4 E2B (on · chat): 96.9% at a median 64,440 ms
Qwen3.5-2B (on · chat): 94.7% at a median 98,390 ms
Other OpenBMB Google Liquid AI Alibaba IBM KRAFTON
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Granite-4.2-3B (off · chat) (not running now, 25 of 2,163 stored, last answer 16 Nov 20:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B on · chat Open · 4.7B Qwen3.5-4B on · chat · Alibaba · chat model Parameters: 4.66B dense Latency: median 46,630 ms · p90 96,460 ms Logic 92.4% | — | 99.4%99.0%–99.7% | a | 2,151 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 46,630 ms | 96,460 ms | $1.271$1,271estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B on · chat Open · 2.5B MiniCPM5-2B on · chat · OpenBMB · chat model Parameters: 2.52B dense Latency: median 12,537 ms · p90 32,720 ms Logic 88.9% | — | 99.2%99.2%–99.3% min–max of 3 seeds | ab | 2,147 / 2,163seed 0 run | Not measured | Not measured | 53.6%answered 99% | 12,537 ms | 32,720 ms | $0.409$409estimate · RTX PRO 6000 Blackwell |
| Granite-4.2-3B high · chat Open · 3.0B Granite-4.2-3B high · chat · IBM · chat model · reasoning high Parameters: 3B dense Base model: Granite-4.1-3B-Base Latency: median 15,340 ms · p90 53,280 ms Logic 91.6% | — | 99.2%98.7%–99.5% | ab | 2,146 / 2,163 | Not measured | Not measured | 61.5%answered 99% | 15,340 ms | 53,280 ms | $0.781$781estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell Open · 1.7B LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell · chat model · reasoning off Parameters: 1.73B dense Latency: median 3,990 ms · p90 10,603 ms Logic 84.6% | — | 98.9%98.8%–99.0% min–max of 3 seeds | abc | 2,142 / 2,163seed 0 run | Not measured | Not measured | Not measured | 3,990 ms | 10,603 ms | $0.142$142estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B on · chat Open · 1.7B LFM3-1.7B on · chat · chat model Parameters: 1.73B dense Latency: median 3,893 ms · p90 10,353 ms Logic 84.2% | — | 98.9%98.8%–99.1% min–max of 3 seeds | abc | 2,139 / 2,163seed 0 run | Not measured | Not measured | 67.6%answered 99% | 3,893 ms | 10,353 ms | $0.132$132estimate · RTX PRO 6000 Blackwell |
| LFM3-1.7B off · chat Open · 1.7B LFM3-1.7B off · chat · chat model · reasoning off Parameters: 1.73B dense Latency: median 19,570 ms · p90 53,410 ms Logic 85.5% | — | 98.7%98.1%–99.1% | bcd | 2,135 / 2,163 | Not measured | Not measured | 67.7%answered 99% | 19,570 ms | 53,410 ms | $0.060$59.94estimate · DGX Spark |
| Qwen3.5-4B off · chat Open · 4.7B Qwen3.5-4B off · chat · Alibaba · chat model · reasoning off Parameters: 4.66B dense Latency: median 10,960 ms · p90 19,970 ms Logic 90.1% | — | 98.5%97.9%–98.9% | bcde | 2,130 / 2,163 | Not measured | Not measured | 36.0%answered 100% | 10,960 ms | 19,970 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| MiniCPM5-2B off · chat Open · 2.5B MiniCPM5-2B off · chat · OpenBMB · chat model · reasoning off Parameters: 2.52B dense Latency: median 6,353 ms · p90 15,697 ms Logic 83.6% | — | 98.2%98.1%–98.3% min–max of 3 seeds | cdef | 2,126 / 2,163seed 0 run | Not measured | Not measured | 31.0%answered 99% | 6,353 ms | 15,697 ms | $0.219$219estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap Open · 5.1B Gemma 4 E2B on · chat · 32k cap · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 14,870 ms · p90 26,040 ms Logic 75.9% | — | 97.8%97.1%–98.4% | defg | 2,116 / 2,163 | Not measured | Not measured | Not measured | 14,870 ms | 26,040 ms | $0.410$410estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E2B on · chat · 32k cap · sampled Open · 5.1B Gemma 4 E2B on · chat · 32k cap · sampled · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 7,137 ms · p90 14,787 ms Logic 76.8% | — | 97.7%97.6%–97.8% min–max of 3 seeds | efg | 2,111 / 2,163seed 0 run | Not measured | Not measured | 23.0%answered 99% | 7,137 ms | 14,787 ms | $0.196$196estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano on · chat Open · 3.1B | — | 97.0%96.2%–97.6% | fg | 2,098 / 2,163 | Not measured | Not measured | Not measured | 34,320 ms† | 108,010 ms† | $1.670$1,670estimate · RTX PRO 6000 Blackwell† |
| Gemma 4 E2B on · chat Open · 5.1B Gemma 4 E2B on · chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 64,440 ms · p90 106,260 ms Logic 74.6% | — | 96.9%96.1%–97.6% | g | 2,097 / 2,163 | Not measured | Not measured | 16.5%answered 99% | 64,440 ms | 106,260 ms | $0.138$138estimate · DGX Spark |
| Gemma 4 E2B chat Open · 5.1B Gemma 4 E2B chat · Google · chat model Parameters: 5.1B dense · 2.3B effective Latency: median 8,120 ms · p90 13,650 ms Logic 72.9% | — | 96.8%95.9%–97.4% | g | 2,093 / 2,163 | Not measured | Not measured | 25.3%answered 99% | 8,120 ms | 13,650 ms | $0.208$208estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B on · chat Open · 2.0B Qwen3.5-2B on · chat · Alibaba · chat model Parameters: 2B dense Latency: median 98,390 ms · p90 194,980 ms Logic 70.3% | — | 94.7%93.7%–95.6% | h | 2,049 / 2,163 | Not measured | Not measured | 20.3%answered 99% | 98,390 ms | 194,980 ms | $2.470$2,470estimate · RTX PRO 6000 Blackwell |
| Qwen3.5-2B off · chat Open · 2.0B Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off Parameters: 2B dense Latency: median 11,200 ms · p90 32,140 ms Logic 64.8% | — | 94.6%93.6%–95.5% | h | 2,047 / 2,163 | Not measured | Not measured | 31.3%answered 98% | 11,200 ms | 32,140 ms | $0.363$363estimate · RTX PRO 6000 Blackwell |
| LFM2.5-2.6B on · chat Open · 2.7B LFM2.5-2.6B on · chat · Liquid AI · chat model Parameters: 2.69B dense Latency: median 9,860 ms · p90 21,240 ms Logic 75.4% | — | 94.0%92.9%–94.9% | h | 2,033 / 2,163 | Not measured | Not measured | 26.4%answered 99% | 9,860 ms | 21,240 ms | $0.300$300estimate · RTX PRO 6000 Blackwell |
| Raon-1 nano off · chat Open · 3.1B | — | 89.6%88.2%–90.8% | i | 1,937 / 2,163 | Not measured | Not measured | 22.1%answered 98% | 25,550 ms† | 134,840 ms† | $1.901$1,901estimate · RTX PRO 6000 Blackwell† |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
eCloud Technology Alibaba Google Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Erk-14B (off · chat): 90.8% at a median 7,890 ms
Qwen3-14B (off · chat): 95.8% at a median 8,040 ms
Gemma 4 E4B (chat): 98.1% at a median 10,770 ms
Gemma 4 E4B (on · chat · 32k cap · sampled): 98.5% at a median 11,190 ms
LFM2.5-8B-A1B (on · chat): 96.3% at a median 14,530 ms (mixed concurrency: This run changed the number of requests in flight part-way; see its record on Method)
Gemma 4 12B Unified (chat): 98.8% at a median 19,100 ms
Gemma 4 E4B (on · chat · 32k cap): 98.5% at a median 20,640 ms
Gemma 4 E4B (on · chat): 98.2% at a median 108,720 ms
eCloud Technology Alibaba Google Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95%; min–max for 3-seed rows | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 12B Unified chat Open · 12B Gemma 4 12B Unified chat · Google · chat model Parameters: 11.95B dense Latency: median 19,100 ms · p90 28,120 ms Logic 87.7% | — | 98.8%98.3%–99.2% | a | 2,138 / 2,163 | Not measured | Not measured | 14.3%answered 100% | 19,100 ms | 28,120 ms | $0.467$467estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap · sampled Open · 8.0B Gemma 4 E4B on · chat · 32k cap · sampled · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 11,190 ms · p90 22,357 ms Logic 85.9% | — | 98.5%98.4%–98.8% min–max of 3 seeds | a | 2,136 / 2,163seed 0 run | Not measured | Not measured | 45.5%answered 99% | 11,190 ms | 22,357 ms | $0.318$318estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat · 32k cap Open · 8.0B Gemma 4 E4B on · chat · 32k cap · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 20,640 ms · p90 34,280 ms Logic 85.0% | — | 98.5%97.9%–99.0% | a | 2,131 / 2,163 | Not measured | Not measured | Not measured | 20,640 ms | 34,280 ms | $0.518$518estimate · RTX PRO 6000 Blackwell |
| Gemma 4 E4B on · chat Open · 8.0B Gemma 4 E4B on · chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 108,720 ms · p90 175,470 ms Logic 82.6% | — | 98.2%97.6%–98.7% | a | 2,125 / 2,163 | Not measured | Not measured | 37.0%answered 99% | 108,720 ms | 175,470 ms | $0.233$233estimate · DGX Spark |
| Gemma 4 E4B chat Open · 8.0B Gemma 4 E4B chat · Google · chat model Parameters: 8B dense · 4.5B effective Latency: median 10,770 ms · p90 17,700 ms Logic 83.2% | — | 98.1%97.4%–98.6% | a | 2,122 / 2,163 | Not measured | Not measured | 44.0%answered 99% | 10,770 ms | 17,700 ms | $0.271$271estimate · RTX PRO 6000 Blackwell |
| LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active | — | 96.3%95.4%–97.0% | b | 2,082 / 2,163 | Not measured | Not measured | 45.2%answered 97% | 14,530 ms† | 33,190 ms† | $0.482$482estimate · RTX PRO 6000 Blackwell† |
| Qwen3-14B off · chat Open · 15B Qwen3-14B off · chat · Alibaba · chat model · reasoning off Parameters: 14.8B dense Latency: median 8,040 ms · p90 18,980 ms Logic 65.3% | — | 95.8%94.9%–96.6% | b | 2,072 / 2,163 | Not measured | Not measured | 39.3%answered 98% | 8,040 ms | 18,980 ms | $0.355$355estimate · RTX PRO 6000 Blackwell |
| Erk-14B off · chat Open · 15B Erk-14B off · chat · eCloud Technology · chat model · reasoning off Parameters: 14.77B dense Latency: median 7,890 ms · p90 14,890 ms Logic 54.2% | — | 90.8%89.5%–91.9% | c | 1,963 / 2,163 | Not measured | Not measured | 35.9%answered 96% | 7,890 ms | 14,890 ms | $0.266$266estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Alibaba Google Liquid AI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Qwen3.8-27B (low · chat): 99.2% at a median 10,020 ms
Gemma 4 31B (chat): 99.4% at a median 11,110 ms
LFM2.5-24B-A2B (on · chat): 99.0% at a median 27,950 ms
Alibaba Google Liquid AI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B chat Open · 31B Gemma 4 31B chat · Google · chat model Parameters: 30.7B dense Latency: median 11,110 ms · p90 16,690 ms Logic 93.3% | — | 99.4%99.0%–99.6% | a | 2,150 / 2,163 | Not measured | Not measured | 21.4%answered 100% | 11,110 ms | 16,690 ms | $0.531$531estimate · RTX PRO 6000 Blackwell |
| Qwen3.8-27B low · chat Open · 27B Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low Parameters: 27B dense Latency: median 10,020 ms · p90 20,630 ms Logic 94.0% | — | 99.2%98.7%–99.5% | a | 2,146 / 2,163 | Not measured | Not measured | 13.0%answered 100% | 10,020 ms | 20,630 ms | $0.530$530estimate · RTX PRO 6000 Blackwell |
| LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active LFM2.5-24B-A2B on · chat · Liquid AI · chat model Parameters: 24.0B MoE · 2.45B active Latency: median 27,950 ms · p90 60,210 ms Logic 86.7% | — | 99.0%98.5%–99.4% | a | 2,142 / 2,163 | Not measured | Not measured | 71.4%answered 99% | 27,950 ms | 60,210 ms | $0.939$939estimate · RTX PRO 6000 Blackwell |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
MiniMax DeepSeek Thinking Machines Z.ai Xiaomi 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
MiniMax-M3.1-Flash (high · chat): 98.2% at a median 1,670 ms
DeepSeek-V4.1-Flash (high · chat · 1017): 99.6% at a median 1,910 ms
Inkling-Small (high · chat · Jul 2026): 99.6% at a median 2,030 ms
Inkling 2 Small (high · chat): 99.5% at a median 2,110 ms
GLM-5.3-Flash (high · chat): 98.8% at a median 3,800 ms
DeepSeek-V4-Flash (chat): 99.3% at a median 8,130 ms
MiMo-V2.6-Flash (on · chat): 99.4% at a median 8,200 ms
MiniMax DeepSeek Thinking Machines Z.ai Xiaomi
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Qwen3.5-397B-A17B (on · chat) (Logic only; on this test only its 25-question smoke test).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high Parameters: 552B MoE · 16B active Latency: median 1,910 ms · p90 3,570 ms Logic 96.2% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | 20.0%answered 100% | 1,910 ms | 3,570 ms | $0.227$227 |
| Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high Parameters: 276B MoE · 12B active Latency: median 2,030 ms · p90 6,330 ms Logic 95.4% | — | 99.6%99.3%–99.8% | a | 2,155 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 2,030 ms | 6,330 ms | $0.954$954 |
| Inkling 2 Small high · chat Open · 594B MoE · 17B active Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high Parameters: 594B MoE · 17B active Latency: median 2,110 ms · p90 6,000 ms Logic 95.6% | — | 99.5%99.2%–99.7% | a | 2,153 / 2,163 | Not measured | Not measured | 61.5%answered 100% | 2,110 ms | 6,000 ms | $0.638$638 |
| MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on Parameters: 309B MoE · 15B active Latency: median 8,200 ms · p90 18,090 ms Logic 95.1% | — | 99.4%99.0%–99.7% | ab | 2,151 / 2,163 | Not measured | Not measured | 50.0%answered 100% | 8,200 ms | 18,090 ms | $0.212$212 |
| DeepSeek-V4-Flash chat Open · 285B MoE · 13B active DeepSeek-V4-Flash chat · DeepSeek · chat model Parameters: 285B MoE · 13B active Latency: median 8,130 ms · p90 16,920 ms Logic running | — | 99.3%98.9%–99.6% | ab | 2,148 / 2,163 | Not measured | Not measured | Not measured | 8,130 ms | 16,920 ms | $0.295$295estimate · DGX Spark |
| GLM-5.3-Flash high · chat Open · 320B MoE · 18B active GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high Parameters: 320B MoE · 18B active Latency: median 3,800 ms · p90 7,970 ms Logic 93.2% | — | 98.8%98.2%–99.2% | bc | 2,137 / 2,163 | Not measured | Not measured | 17.4%answered 100% | 3,800 ms | 7,970 ms | $0.166$166 |
| MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high Parameters: 428B MoE · 23B active Latency: median 1,670 ms · p90 7,470 ms Logic 94.0% | — | 98.2%97.6%–98.7% | c | 2,125 / 2,163 | Not measured | Not measured | 40.6%answered 99% | 1,670 ms | 7,470 ms | $0.103$103 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Meta OpenAI 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Muse Ember (max · chat): 99.6% at a median 1,260 ms
GPT-6 Luna (chat): 93.7% at a median 3,830 ms
Meta OpenAI
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): GPT-6 Luna (max · chat) (paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Muse Ember max · chat Closed · Efficient Muse Ember max · chat · Meta · chat model · reasoning max Parameters: not disclosed by maker (closed weights) Latency: median 1,260 ms · p90 2,270 ms Logic 95.5% | — | 99.6%99.2%–99.8% | a | 2,154 / 2,163 | Not measured | Not measured | 64.3%answered 100% | 1,260 ms | 2,270 ms | $0.365$365 |
| GPT-6 Luna chat Closed · Efficient GPT-6 Luna chat · OpenAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,830 ms · p90 5,000 ms Logic 58.0% | — | 93.7%92.6%–94.6% | b | 2,026 / 2,163 | Not measured | Not measured | 68.9%answered 95% | 3,830 ms | 5,000 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Anthropic 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Claude Sonnet 5.5 (chat): 99.7% at a median 3,460 ms
Anthropic
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 chat Closed · Workhorse Claude Sonnet 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,460 ms · p90 5,970 ms Logic 95.5% | — | 99.7%99.3%–99.8% | a | 2,156 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,460 ms | 5,970 ms | $4.734$4,734 API list-price equivalent |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
Anthropic 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
Claude Opus 5.5 (chat): 99.8% at a median 3,200 ms
Claude Fable 5.1 (chat): 99.7% at a median 5,030 ms
Anthropic
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Claude Fable 5.2 (chat) (not running now, 5 of 2,163 stored, last answer 11 Nov 12:02 UTC).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 chat Closed · Flagship Claude Opus 5.5 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 3,200 ms · p90 4,790 ms Logic 96.2% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 62.5%answered 100% | 3,200 ms | 4,790 ms | $9.871$9,871 API list-price equivalent |
| Claude Fable 5.1 chat Closed · Flagship Claude Fable 5.1 chat · Anthropic · chat model Parameters: not disclosed by maker (closed weights) Latency: median 5,030 ms · p90 6,490 ms Logic 95.2% | — | 99.7%99.4%–99.9% | a | 2,157 / 2,163 | Not measured | Not measured | 60.0%answered 100% | 5,030 ms | 6,490 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
Latency
Milliseconds from request to reply per question (log scale): the median (family colour) and the 90th percentile (grey, nearest rank); beside them questions per minute, wall clock from the run's first to last answer, and requests in flight on average over its active time · lower latency is better
OpenAI Alibaba xAI Google 90th percentile
Accuracy across latency
Each setup's accuracy against its median latency (log scale); the dotted line joins the setups no other is both faster and more accurate than; shaded: faster and more accurate than the median setup
GPT-6.1 Terra (chat): 99.2% at a median 4,080 ms
Qwen4-Max (high · chat): 99.8% at a median 4,710 ms
GPT-6.1 Sol (chat): 99.6% at a median 4,810 ms
GPT-6 Astra (chat): 99.6% at a median 5,280 ms
Qwen3.8 Flash (off · chat): 99.5% at a median 5,740 ms
Grok 4.7 (chat): 99.5% at a median 11,100 ms
Gemini 4 Carbon (chat): 99.6% at a median 13,410 ms
OpenAI Alibaba xAI Google
Latency is the wall time of each request from our machine; runs that shared a GPU or ran with several requests in flight slow each other down, so throughput (questions / min) depends on the run's concurrency. In flight: requests in flight on average, the sum of request latencies over the run's active time (its first to last answer with gaps of more than 10 minutes left out), e.g. “≈ 6” is six at a time.
Median latency is per request; with several requests in flight a run finishes faster than requests × median.
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
Not ranked (only complete runs are): Gemini 4 Carbon (high · chat) (paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)).
Every setup, every measure
| Setup | Web | Accuracy% of N (N ≤ 2,163) · Wilson 95% | Tie groupHolm · McNemar | Right / Nquestions | Kept at ≥99%right / wrong | AUROCright vs wrong | Non-hallucinationof answers not right, % “I don't know” · higher is better | Median latencyms | p90ms, nearest rank | Cost$ per 1,000 qper 1M q |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen4-Max high · chat Closed · Undisclosed Qwen4-Max high · chat · Alibaba · chat model · reasoning high Parameters: not disclosed by maker (closed weights) Latency: median 4,710 ms · p90 9,130 ms Logic 96.3% | — | 99.8%99.5%–99.9% | a | 2,158 / 2,163 | Not measured | Not measured | 54.5%answered 100% | 4,710 ms | 9,130 ms | $1.478$1,478 |
| GPT-6 Astra chat Closed · Undisclosed GPT-6 Astra chat · OpenAI · chat model Parameters: not on record yet Latency: median 5,280 ms · p90 7,260 ms Logic 96.7% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 5,280 ms | 7,260 ms | $0.000$0.00 |
| GPT-6.1 Sol chat Closed · Undisclosed GPT-6.1 Sol chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,810 ms · p90 6,560 ms Logic 96.5% | — | 99.6%99.3%–99.8% | ab | 2,155 / 2,163 | Not measured | Not measured | 78.6%answered 99% | 4,810 ms | 6,560 ms | $0.000$0.00 |
| Gemini 4 Carbon chat Closed · Undisclosed Gemini 4 Carbon chat · Google · chat model Parameters: not disclosed by maker (closed weights) Latency: median 13,410 ms · p90 27,800 ms Logic 96.7% | — | 99.6%99.2%–99.8% | ab | 2,154 / 2,163 | Not measured | Not measured | 75.0%answered 99% | 13,410 ms | 27,800 ms | — |
| Grok 4.7 chat Closed · Undisclosed Grok 4.7 chat · xAI · chat model Parameters: not disclosed by maker (closed weights) Latency: median 11,100 ms · p90 66,530 ms Logic running | — | 99.5%99.2%–99.7% | ab | 2,153 / 2,163 | Not measured | Not measured | 97.5%answered 98% | 11,100 ms | 66,530 ms | $12.061$12,061 API list-price equivalent |
| Qwen3.8 Flash off · chat Closed · Undisclosed Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off Parameters: not disclosed by maker (closed weights) Latency: median 5,740 ms · p90 11,800 ms Logic 94.0% | — | 99.5%99.1%–99.7% | ab | 2,152 / 2,163 | Not measured | Not measured | 16.7%answered 100% | 5,740 ms | 11,800 ms | $0.397$397 |
| GPT-6.1 Terra chat Closed · Undisclosed GPT-6.1 Terra chat · OpenAI · chat model Parameters: not on record yet Latency: median 4,080 ms · p90 5,870 ms Logic 94.0% | — | 99.2%98.7%–99.5% | b | 2,146 / 2,163 | Not measured | Not measured | 77.8%answered 99% | 4,080 ms | 5,870 ms | $0.000$0.00 |
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.