Logic & Math bench

Leaderboard

Class:
TierModelLogic accuracy 1,125 questions, 95% interval; min–max for 3-seed rowsMath of 2,163 questionsNon-hallucination of answers not right, % “I don't know”“I don't know” share declinedCalibration error, points; decision models$ per 1,000 q$ per 1M q list price or GPU timeGPU time s per 1,000 q, our hardwareLatency median, p90 under it
1GPT-6 Astra chat Closed · Undisclosed

GPT-6 Astra chat · OpenAI · chat model

Parameters: not on record yet · Closed; no list price published: Undisclosed

Logic 96.7% (95%: 95.5%–97.6%) · Math 99.6%

Non-hallucination 56.4% · “I don't know” on 2.8% of questions

Runs on an API · cost $0.000$0.00 per 1,000 questionsper 1M questions · 6.59 s per question (median latency)

Its run record (Method) →

96.7%95.5%–97.6%99.6%56.4%2.8%–$0.000$0.00–6.59 sp90 14.3 s
1Gemini 4 Carbon chat Closed · Undisclosed

Gemini 4 Carbon chat · Google · chat model

Parameters: not disclosed by maker (closed weights) · Closed; no list price published: Undisclosed

Logic 96.7% (95%: 95.5%–97.6%) · Math 99.6%

Non-hallucination 45.5% · “I don't know” on 1.8% of questions

Runs on an API · 44.5 s per question (median latency)

Its run record (Method) →

96.7%95.5%–97.6%99.6%45.5%1.8%–––44.5 sp90 206.7 s
1GPT-6.1 Sol chat Closed · Undisclosed

GPT-6.1 Sol chat · OpenAI · chat model

Parameters: not on record yet · Closed; no list price published: Undisclosed

Logic 96.5% (95%: 95.3%–97.5%) · Math 99.6%

Non-hallucination 63.2% · “I don't know” on 3.2% of questions

Runs on an API · cost $0.000$0.00 per 1,000 questionsper 1M questions · 7.07 s per question (median latency)

Its run record (Method) →

96.5%95.3%–97.5%99.6%63.2%3.2%–$0.000$0.00–7.07 sp90 16.2 s
1Qwen4-Max high · chat Closed · Undisclosed

Qwen4-Max high · chat · Alibaba · chat model · reasoning high

Parameters: not disclosed by maker (closed weights) · Closed; no list price published: Undisclosed

Logic 96.3% (95%: 95.0%–97.2%) · Math 99.8%

Non-hallucination 0.0% · “I don't know” on 0.0% of questions

Runs on an API · cost $5.690$5,690 per 1,000 questionsper 1M questions · 11.6 s per question (median latency)

Its run record (Method) →

96.3%95.0%–97.2%99.8%0.0%0.0%–$5.690$5,690–11.6 sp90 38.6 s
1Claude Opus 5.5 chat Closed · Flagship

Claude Opus 5.5 chat · Anthropic · chat model

Parameters: not disclosed by maker (closed weights) · Closed; list price $4.00 per 1M input tokens: Flagship (source: https://platform.claude.com/docs/en/about-claude/pricing)

Logic 96.2% (95%: 94.9%–97.2%) · Math 99.8%

Non-hallucination 38.2% · “I don't know” on 1.9% of questions

Runs on an API · cost $19.540$19,540 API list-price equivalent per 1,000 questionsper 1M questions · 5.83 s per question (median latency)

Its run record (Method) →

96.2%94.9%–97.2%99.8%38.2%1.9%–$19.540$19,540 API list-price equivalent–5.83 sp90 12.2 s
1DeepSeek-V4.1-Flash high · chat · 1017 Open · 552B MoE · 16B active

DeepSeek-V4.1-Flash high · chat · 1017 · DeepSeek · chat model · reasoning high

Parameters: 552B MoE · 16B active · Mixture of experts: 552B total, about 16B active per step. Size class uses 552B.

Logic 96.2% (95%: 94.9%–97.2%) · Math 99.6%

Non-hallucination 3.8% · “I don't know” on 0.2% of questions

Runs on an API · cost $1.647$1,647 per 1,000 questionsper 1M questions · 6.68 s per question (median latency)

Its run record (Method) →

96.2%94.9%–97.2%99.6%3.8%0.2%–$1.647$1,647–6.68 sp90 40.0 s
1Gemini 4 Hydrogen chat

Gemini 4 Hydrogen chat · Google · chat model

Parameters: not on record yet

Logic 96.1% (95%: 94.8%–97.1%) · Math 99.6%

Non-hallucination 76.1% · “I don't know” on 4.8% of questions

Runs on an API · 8.21 s per question (median latency)

Its run record (Method) →

96.1%94.8%–97.1%99.6%76.1%4.8%–––8.21 sp90 24.1 s
1Inkling 2 Small high · chat Open · 594B MoE · 17B active

Inkling 2 Small high · chat · Thinking Machines · chat model · reasoning high

Parameters: 594B MoE · 17B active · Mixture of experts: 594B total, about 17B active per step. Size class uses 594B.

Logic 95.6% (95%: 94.3%–96.7%) · Math 99.5%

Non-hallucination 37.3% · “I don't know” on 2.0% of questions

Runs on an API · cost $4.121$4,121 per 1,000 questionsper 1M questions · 13.2 s per question (median latency)

Its run record (Method) →

95.6%94.3%–96.7%99.5%37.3%2.0%–$4.121$4,121–13.2 sp90 144.2 s
1Claude Sonnet 5.5 chat Closed · Workhorse

Claude Sonnet 5.5 chat · Anthropic · chat model

Parameters: not disclosed by maker (closed weights) · Closed; list price $2.00 per 1M input tokens: Workhorse (source: https://platform.claude.com/docs/en/about-claude/pricing)

Logic 95.5% (95%: 94.1%–96.5%) · Math 99.7%

Non-hallucination 41.3% · “I don't know” on 2.3% of questions

Runs on an API · cost $11.480$11,480 API list-price equivalent per 1,000 questionsper 1M questions · 6.71 s per question (median latency)

Its run record (Method) →

95.5%94.1%–96.5%99.7%41.3%2.3%–$11.480$11,480 API list-price equivalent–6.71 sp90 14.9 s
1Muse Ember max · chat Closed · Efficient

Muse Ember max · chat · Meta · chat model · reasoning max

Parameters: not disclosed by maker (closed weights) · Closed; list price $0.10 per 1M input tokens: Efficient (source: https://openrouter.ai/meta/muse-ember)

Logic 95.5% (95%: 94.1%–96.5%) · Math 99.6%

Non-hallucination 22.6% · “I don't know” on 1.1% of questions

Runs on an API · cost $1.820$1,820 per 1,000 questionsper 1M questions · 3.11 s per question (median latency)

Its run record (Method) →

95.5%94.1%–96.5%99.6%22.6%1.1%–$1.820$1,820–3.11 sp90 14.0 s
1Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active

Inkling-Small high · chat · Jul 2026 · Thinking Machines · chat model · reasoning high

Parameters: 276B MoE · 12B active · Mixture of experts: 276B total, about 12B active per step. Size class uses 276B.

Logic 95.4% (95%: 94.0%–96.5%) · Math 99.6%

Non-hallucination 50.0% · “I don't know” on 3.2% of questions

Runs on an API · cost $5.489$5,489 per 1,000 questionsper 1M questions · 13.3 s per question (median latency)

Its run record (Method) →

95.4%94.0%–96.5%99.6%50.0%3.2%–$5.489$5,489–13.3 sp90 78.5 s
1Claude Fable 5.1 chat Closed · Flagship

Claude Fable 5.1 chat · Anthropic · chat model

Parameters: not disclosed by maker (closed weights) · Closed; list price $10.00 per 1M input tokens: Flagship (source: https://platform.claude.com/docs/en/about-claude/pricing)

Logic 95.2% (95%: 93.8%–96.3%) · Math 99.7%

Non-hallucination 8.3% · “I don't know” on 0.4% of questions

Runs on an API · cost $0.000$0.00 per 1,000 questionsper 1M questions · 3.82 s per question (median latency)

Its run record (Method) →

95.2%93.8%–96.3%99.7%8.3%0.4%–$0.000$0.00–3.82 sp90 13.5 s
1MiMo-V2.6-Flash on · chat Open · 309B MoE · 15B active

MiMo-V2.6-Flash on · chat · Xiaomi · chat model · reasoning on

Parameters: 309B MoE · 15B active · Mixture of experts: 309B total, about 15B active per step. Size class uses 309B.

Logic 95.1% (95%: 93.7%–96.2%) · Math 99.4%

Non-hallucination 23.3% · “I don't know” on 1.5% of questions

Runs on an API · cost $1.390$1,390 per 1,000 questionsper 1M questions · 24.7 s per question (median latency)

Its run record (Method) →

95.1%93.7%–96.2%99.4%23.3%1.5%–$1.390$1,390–24.7 sp90 179.9 s
1Qwen3.5-397B-A17B on · chat Open · 397B MoE · 17B active

Qwen3.5-397B-A17B on · chat · Alibaba · chat model · reasoning on

Parameters: 397B MoE · 17B active · Mixture of experts: 397B total, about 17B active per step. Size class uses 397B.

Logic 95.1% (95%: 93.7%–96.2%) · Math running

Non-hallucination 31.7% · “I don't know” on 1.8% of questions

Runs on an API · cost $13.426$13,426 per 1,000 questionsper 1M questions · 84.7 s per question (median latency)

Its run record (Method) →

95.1%93.7%–96.2%running31.7%1.8%–$13.426$13,426–84.7 sp90 167.5 s
2Qwen3.8-27B low · chat Open · 27B

Qwen3.8-27B low · chat · Alibaba · chat model · reasoning low

Parameters: 27B dense

Logic 94.0% (95%: 92.5%–95.3%) · Math 99.2%

Non-hallucination 39.1% · “I don't know” on 3.2% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $4.884$4,884estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 18.7 s per question (median latency)

Its run record (Method) →

94.0%92.5%–95.3%99.2%39.1%3.2%–$4.884$4,884estimate · RTX PRO 6000 Blackwell7,061 sRTX PRO 600018.7 sp90 101.0 s
2GPT-6.1 Terra chat Closed · Undisclosed

GPT-6.1 Terra chat · OpenAI · chat model

Parameters: not on record yet · Closed; no list price published: Undisclosed

Logic 94.0% (95%: 92.4%–95.2%) · Math 99.2%

Non-hallucination 61.1% · “I don't know” on 4.9% of questions

Runs on an API · cost $0.000$0.00 per 1,000 questionsper 1M questions · 5.64 s per question (median latency)

Its run record (Method) →

94.0%92.4%–95.2%99.2%61.1%4.9%–$0.000$0.00–5.64 sp90 10.3 s
2MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active

MiniMax-M3.1-Flash high · chat · MiniMax · chat model · reasoning high

Parameters: 428B MoE · 23B active · Mixture of experts: 428B total, about 23B active per step. Size class uses 428B.

Logic 94.0% (95%: 92.4%–95.2%) · Math 98.2%

Non-hallucination 38.3% · “I don't know” on 3.2% of questions

Runs on an API · cost $0.803$803 per 1,000 questionsper 1M questions · 5.61 s per question (median latency)

Its run record (Method) →

94.0%92.4%–95.2%98.2%38.3%3.2%–$0.803$803–5.61 sp90 37.4 s
2Qwen3.8 Flash off · chat Closed · Undisclosed

Qwen3.8 Flash off · chat · Alibaba · chat model · reasoning off

Parameters: not disclosed by maker (closed weights) · Closed; no list price published: Undisclosed

Logic 94.0% (95%: 92.4%–95.2%) · Math 99.5%

Non-hallucination 34.6% · “I don't know” on 2.5% of questions

Runs on an API · cost $1.376$1,376 per 1,000 questionsper 1M questions · 17.6 s per question (median latency)

Its run record (Method) →

94.0%92.4%–95.2%99.5%34.6%2.5%–$1.376$1,376–17.6 sp90 73.2 s
2Gemma 4 31B chat Open · 31B

Gemma 4 31B chat · Google · chat model

Parameters: 30.7B dense

Logic 93.3% (95%: 91.7%–94.6%) · Math 99.4%

Non-hallucination 16.3% · “I don't know” on 1.4% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $1.244$1,244estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 18.3 s per question (median latency)

Its run record (Method) →

93.3%91.7%–94.6%99.4%16.3%1.4%–$1.244$1,244estimate · RTX PRO 6000 Blackwell1,798 sRTX PRO 600018.3 sp90 39.8 s
2GLM-5.3-Flash high · chat Open · 320B MoE · 18B active

GLM-5.3-Flash high · chat · Z.ai · chat model · reasoning high

Parameters: 320B MoE · 18B active · Mixture of experts: 320B total, about 18B active per step. Size class uses 320B.

Logic 93.2% (95%: 91.5%–94.5%) · Math 98.8%

Non-hallucination 4.2% · “I don't know” on 0.3% of questions

Runs on an API · cost $1.057$1,057 per 1,000 questionsper 1M questions · 12.1 s per question (median latency)

Its run record (Method) →

93.2%91.5%–94.5%98.8%4.2%0.3%–$1.057$1,057–12.1 sp90 82.8 s
2Qwen3.5-4B on · chat Open · 4.7B

Qwen3.5-4B on · chat · Alibaba · chat model

Parameters: 4.66B dense

Logic 92.4% (95%: 90.7%–93.8%) · Math 99.4%

Non-hallucination 36.1% · “I don't know” on 3.1% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $3.525$3,525estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 142.5 s per question (median latency)

Other settings: Qwen3.5-4B (off · chat)

Its run record (Method) →

92.4%90.7%–93.8%99.4%36.1%3.1%–$3.525$3,525estimate · RTX PRO 6000 Blackwell5,096 sRTX PRO 6000142.5 sp90 246.3 s
2Granite-4.2-3B high · chat Open · 3.0B

Granite-4.2-3B high · chat · IBM · chat model · reasoning high

Parameters: 3B dense

Base model: Granite-4.1-3B-Base

Logic 91.6% (95%: 89.9%–93.1%) · Math 99.2%

Non-hallucination 53.2% · “I don't know” on 5.9% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $10.893$10,893estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 221.3 s per question (median latency)

Its run record (Method) →

91.6%89.9%–93.1%99.2%53.2%5.9%–$10.893$10,893estimate · RTX PRO 6000 Blackwell15,748 sRTX PRO 6000221.3 sp90 1229.5 s
3MiniCPM5-2B on · chat Open · 2.5B

MiniCPM5-2B on · chat · OpenBMB · chat model

Parameters: 2.52B dense

Logic 88.9% (mean of 3 seeds; min–max 88.3%–89.6%) · Math 99.2%

Seed runs: seed 0: 88.7% (998 of 1,125, run minicpm5-2b_en) · seed 1: 88.3% (993 of 1,125, run minicpm5-2b-seed1_en) · seed 2: 89.6% (1,008 of 1,125, run minicpm5-2b-seed2_en). Tiers, tie letters, topics, confidence, “I don't know” and the answer links are from the seed-0 run minicpm5-2b_en, not the mean

Non-hallucination 51.5% · “I don't know” on 7.6% of questions · seed 0's run only (one run, not a 3-seed mean)

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $3.335$3,335estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 101.1 s per question (median latency)

Other settings: MiniCPM5-2B (off · chat)

Its run record (Method) →

88.9%88.3%–89.6% min–max of 3 seeds99.2%51.5%seed 0 only7.6%seed 0 only–$3.335$3,335estimate · RTX PRO 6000 Blackwellmean of 3 seeds4,821 sRTX PRO 6000mean of 3 seeds101.1 sp90 333.4 smean of 3 seeds
3Gemma 4 12B Unified chat Open · 12B

Gemma 4 12B Unified chat · Google · chat model

Parameters: 11.95B dense

Logic 87.7% (95%: 85.7%–89.5%) · Math 98.8%

Non-hallucination 15.5% · “I don't know” on 2.0% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $1.292$1,292estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 42.1 s per question (median latency)

Its run record (Method) →

87.7%85.7%–89.5%98.8%15.5%2.0%–$1.292$1,292estimate · RTX PRO 6000 Blackwell1,868 sRTX PRO 600042.1 sp90 82.9 s
3LFM2.5-24B-A2B on · chat Open · 24B MoE · 2.5B active

LFM2.5-24B-A2B on · chat · Liquid AI · chat model

Parameters: 24.0B MoE · 2.45B active · Mixture of experts: 24.0B total, about 2.45B active per step. Size class uses 24.0B.

Logic 86.7% (95%: 84.6%–88.5%) · Math 99.0%

Non-hallucination 37.8% · “I don't know” on 5.2% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $7.562$7,562estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 229.0 s per question (median latency)

Its run record (Method) →

86.7%84.6%–88.5%99.0%37.8%5.2%–$7.562$7,562estimate · RTX PRO 6000 Blackwell10,932 sRTX PRO 6000229.0 sp90 776.9 s
3LFM3-1.7B off · chat Open · 1.7B

LFM3-1.7B off · chat · chat model · reasoning off

Parameters: 1.73B dense

Logic 85.5% (95%: 83.3%–87.4%) · Math 98.7%

Non-hallucination 54.2% · “I don't know” on 10.4% of questions

Runs on NVIDIA DGX Spark (GB10) · cost $0.487$487estimate · DGX Spark per 1,000 questionsper 1M questions · 163.9 s per question (median latency)

Other settings: LFM3-1.7B (on · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell)

Its run record (Method) →

85.5%83.3%–87.4%98.7%54.2%10.4%–$0.487$487estimate · DGX Spark8,234 sDGX Spark163.9 sp90 656.8 s
–DeepSeek-V4-Flash chat Open · 285B MoE · 13B active sample

DeepSeek-V4-Flash chat · DeepSeek · chat model

Parameters: 285B MoE · 13B active · Mixture of experts: 285B total, about 13B active per step. Size class uses 285B.

Not tiered: sample (317 questions)

Logic 83.6% (95%: 79.1%–87.3%) · Math 99.3%

Runs on NVIDIA DGX Spark (GB10) · cost $1.154$1,154estimate · DGX Spark per 1,000 questionsper 1M questions · 24.8 s per question (median latency)

Its run record (Method) →

83.6%79.1%–87.3%99.3%–––$1.154$1,154estimate · DGX Spark19,527 sDGX Spark24.8 sp90 91.1 s
4Gemma 4 E4B chat Open · 8.0B

Gemma 4 E4B chat · Google · chat model

Parameters: 8B dense · 4.5B effective

Logic 83.2% (95%: 80.9%–85.3%) · Math 98.1%

Non-hallucination 52.4% · “I don't know” on 11.7% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.842$842estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 32.7 s per question (median latency)

Other settings: Gemma 4 E4B (on · chat), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E4B (on · chat · 32k cap · sampled)

Its run record (Method) →

83.2%80.9%–85.3%98.1%52.4%11.7%–$0.842$842estimate · RTX PRO 6000 Blackwell1,218 sRTX PRO 600032.7 sp90 64.1 s
5LFM2.5-2.6B on · chat Open · 2.7B

LFM2.5-2.6B on · chat · Liquid AI · chat model

Parameters: 2.69B dense

Logic 75.4% (95%: 72.8%–77.8%) · Math 94.0%

Non-hallucination 44.3% · “I don't know” on 13.6% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $2.480$2,480estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 81.3 s per question (median latency)

Its run record (Method) →

75.4%72.8%–77.8%94.0%44.3%13.6%–$2.480$2,480estimate · RTX PRO 6000 Blackwell3,585 sRTX PRO 600081.3 sp90 224.0 s
5Gemma 4 E2B chat Open · 5.1B

Gemma 4 E2B chat · Google · chat model

Parameters: 5.1B dense · 2.3B effective

Logic 72.9% (95%: 70.2%–75.4%) · Math 96.8%

Non-hallucination 34.0% · “I don't know” on 9.8% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.513$513estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 21.4 s per question (median latency)

Other settings: Gemma 4 E2B (on · chat), Gemma 4 E2B (on · chat · 32k cap), Gemma 4 E2B (on · chat · 32k cap · sampled)

Its run record (Method) →

72.9%70.2%–75.4%96.8%34.0%9.8%–$0.513$513estimate · RTX PRO 6000 Blackwell742 sRTX PRO 600021.4 sp90 35.2 s
6Qwen3-14B off · chat Open · 15B

Qwen3-14B off · chat · Alibaba · chat model · reasoning off

Parameters: 14.8B dense

Logic 65.3% (95%: 62.5%–68.1%) · Math 95.8%

Non-hallucination 16.1% · “I don't know” on 6.3% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $1.772$1,772estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 16.6 s per question (median latency)

Its run record (Method) →

65.3%62.5%–68.1%95.8%16.1%6.3%–$1.772$1,772estimate · RTX PRO 6000 Blackwell2,562 sRTX PRO 600016.6 sp90 178.7 s
6Raon-1 nano off · chat Open · 3.1B

Raon-1 nano off · chat · KRAFTON · chat model · reasoning off

Parameters: 3.1B dense · 1.5B effective

Base model: Raon-1 mini preview (MoE-to-Dense pruning and distillation)

Logic 65.1% (95%: 62.2%–67.8%) · Math 89.6%

Non-hallucination 35.7% · “I don't know” on 12.8% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $9.270$9,270estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions† · 276.7 s† per question (median latency)

Other settings: Raon-1 nano (on · chat)

Its run record (Method) →

65.1%62.2%–67.8%89.6%35.7%12.8%–$9.270$9,270estimate · RTX PRO 6000 Blackwell†13,403 sRTX PRO 6000276.7 s†p90 965.6 s
6Qwen3.5-2B off · chat Open · 2.0B

Qwen3.5-2B off · chat · Alibaba · chat model · reasoning off

Parameters: 2B dense

Logic 64.8% (95%: 62.0%–67.5%) · Math 94.6%

Non-hallucination 35.2% · “I don't know” on 14.0% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $1.605$1,605estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 71.3 s per question (median latency)

Other settings: Qwen3.5-2B (on · chat)

Its run record (Method) →

64.8%62.0%–67.5%94.6%35.2%14.0%–$1.605$1,605estimate · RTX PRO 6000 Blackwell2,320 sRTX PRO 600071.3 sp90 113.5 s
6Karpo Decision 31B Open · 31B

Karpo Decision 31B · MachinePulse · decision model

Parameters: 31B dense

Logic 60.7% (95%: 57.8%–63.5%) · Math 81.6%

Non-hallucination 6.6% · “I don't know” on 2.7% of questions

Confidence: 172 answers kept at ≥99%, 169 right · 3 wrong · AUROC 0.816

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.128$128estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.173 s per question (median latency)

Its run record (Method) →

60.7%57.8%–63.5%81.6%6.6%2.7%5.7$0.128$128estimate · RTX PRO 6000 Blackwell186 sRTX PRO 60000.173 sp90 0.211 s
6LFM2.5-8B-A1B on · chat Open · 8.3B MoE · 1.5B active

LFM2.5-8B-A1B on · chat · Liquid AI · chat model

Parameters: 8.3B MoE · 1.5B active · Mixture of experts: 8.3B total, about 1.5B active per step. Size class uses 8.3B.

Logic 60.2% (95%: 57.3%–63.0%) · Math 96.3%

Non-hallucination 58.3% · “I don't know” on 29.8% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $4.664$4,664estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions† · 138.1 s† per question (median latency)

Its run record (Method) →

60.2%57.3%–63.0%96.3%58.3%29.8%–$4.664$4,664estimate · RTX PRO 6000 Blackwell†6,743 sRTX PRO 6000138.1 s†p90 403.4 s
6REAX 1.3 Open · 57B

REAX 1.3 · REAX · decision model

Parameters: 57.36B dense · Two models run together, option probabilities averaged 50/50: 57.36B in all, all used on every question.

Logic 59.3% (95%: 56.4%–62.1%) · Math 78.0%

Non-hallucination 5.3% · “I don't know” on 2.2% of questions

Confidence: 51 answers kept at ≥99%, 51 right · 0 wrong · AUROC 0.816

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.160$160estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.215 s per question (median latency)

Its run record (Method) →

59.3%56.4%–62.1%78.0%5.3%2.2%3.7$0.160$160estimate · RTX PRO 6000 Blackwell232 sRTX PRO 60000.215 sp90 0.278 s
6KnowLine-31B-Gen1.5 Open · 31B

KnowLine-31B-Gen1.5 · PelaAI · decision model

Parameters: 31B dense

Logic 59.0% (95%: 56.1%–61.9%) · Math 77.4%

Non-hallucination 0.4% · “I don't know” on 0.2% of questions

Confidence: 89 answers kept at ≥99%, 89 right · 0 wrong · AUROC 0.804

Runs on NVIDIA DGX Spark (GB10) · cost $0.061$61.43estimate · DGX Spark per 1,000 questionsper 1M questions · 0.983 s per question (median latency)

Its run record (Method) →

59.0%56.1%–61.9%77.4%0.4%0.2%2.6$0.061$61.43estimate · DGX Spark1,039 sDGX Spark0.983 sp90 1.28 s
6KnowLine-31B-Gen2 Open · 31B

KnowLine-31B-Gen2 · PelaAI · decision model

Parameters: 31B dense

Base model: Gemma 4 31B

Logic 58.9% (95%: 56.0%–61.8%) · Math 80.3%

Non-hallucination 0.0% · “I don't know” on 0.0% of questions

Confidence: 54 answers kept at ≥99%, 54 right · 0 wrong · AUROC 0.825

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.112$112estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.151 s per question (median latency)

Its run record (Method) →

58.9%56.0%–61.8%80.3%0.0%0.0%3.7$0.112$112estimate · RTX PRO 6000 Blackwell162 sRTX PRO 60000.151 sp90 0.204 s
6Copernicus-31B Open · 31B

Copernicus-31B · Poindexter Labs · decision model

Parameters: 31B dense

Logic 58.7% (95%: 55.8%–61.5%) · Math 77.3%

Non-hallucination 1.9% · “I don't know” on 0.8% of questions

Confidence: 66 answers kept at ≥99%, 66 right · 0 wrong · AUROC 0.801

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.136$136estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.185 s per question (median latency)

Its run record (Method) →

58.7%55.8%–61.5%77.3%1.9%0.8%3.9$0.136$136estimate · RTX PRO 6000 Blackwell196 sRTX PRO 60000.185 sp90 0.255 s
6GLiDE 27B Open · 27B

GLiDE 27B · Fastino · decision model

Parameters: 27B dense

Logic 58.4% (95%: 55.5%–61.2%) · Math 72.0%

Non-hallucination 44.3% · “I don't know” on 21.1% of questions

Confidence: 75 answers kept at ≥99%, 74 right · 1 wrong · AUROC 0.822

Runs on NVIDIA DGX Spark (GB10) · cost $0.033$33.16estimate · DGX Spark per 1,000 questionsper 1M questions · 0.544 s per question (median latency)

Its run record (Method) →

58.4%55.5%–61.2%72.0%44.3%21.1%4.4$0.033$33.16estimate · DGX Spark561 sDGX Spark0.544 sp90 0.655 s
7GPT-6 Luna chat Closed · Efficient

GPT-6 Luna chat · OpenAI · chat model

Parameters: not disclosed by maker (closed weights) · Closed; list price $0.10 per 1M input tokens: Efficient (source: https://openrouter.ai/openai/gpt-6-luna)

Logic 58.0% (95%: 55.0%–60.8%) · Math 93.7%

Non-hallucination 55.1% · “I don't know” on 28.1% of questions

Runs on an API · cost $0.000$0.00 per 1,000 questionsper 1M questions · 3.86 s per question (median latency)

Its run record (Method) →

58.0%55.0%–60.8%93.7%55.1%28.1%–$0.000$0.00–3.86 sp90 5.82 s
7Sieve 31B v3 Open · 31B

Sieve 31B v3 · Sthānika AI · decision model

Parameters: 31B dense

Logic 57.9% (95%: 55.0%–60.7%) · Math 77.1%

Non-hallucination 5.5% · “I don't know” on 2.4% of questions

Confidence: 81 answers kept at ≥99%, 79 right · 2 wrong · AUROC 0.797

Runs on NVIDIA DGX Spark (GB10) · cost $0.063$62.95estimate · DGX Spark per 1,000 questionsper 1M questions · 1.05 s per question (median latency)

Its run record (Method) →

57.9%55.0%–60.7%77.1%5.5%2.4%2.1$0.063$62.95estimate · DGX Spark1,065 sDGX Spark1.05 sp90 1.33 s
7Logits Sigma-1 Duo Open · 57B

Logits Sigma-1 Duo · Logits · decision model

Parameters: 57.36B dense

Logic 57.8% (95%: 54.9%–60.6%) · Math 75.2%

Non-hallucination 7.1% · “I don't know” on 3.0% of questions

Confidence: 8 answers kept at ≥99%, 8 right · 0 wrong · AUROC 0.796

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.211$211estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.285 s per question (median latency)

Its run record (Method) →

57.8%54.9%–60.6%75.2%7.1%3.0%5.0$0.211$211estimate · RTX PRO 6000 Blackwell306 sRTX PRO 60000.285 sp90 0.399 s
7GLiDE v1.2 Open · 27B

GLiDE v1.2 · Fastino · decision model

Parameters: 27B dense

Logic 55.2% (95%: 52.3%–58.1%) · Math 69.9%

Non-hallucination 50.2% · “I don't know” on 26.8% of questions

Confidence: 45 answers kept at ≥99%, 45 right · 0 wrong · AUROC 0.793

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.111 s per question (median latency)

Its run record (Method) →

55.2%52.3%–58.1%69.9%50.2%26.8%5.9$0.014$13.88 list$0.082$81.77estimate · RTX PRO 6000 Blackwell118 sRTX PRO 60000.111 sp90 0.140 s
7d1-ultra Closed · Efficient

d1-ultra · Liquid AI · decision model

Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) · Closed; list price $0.042 per 1M input tokens: Efficient (source: https://www.liquid.ai/blog/d1-ultra)

Logic 54.9% (95%: 52.0%–57.8%) · Math 74.1%

Non-hallucination 91.0% · “I don't know” on 64.9% of questions

Confidence: 20 answers kept at ≥99%, 20 right · 0 wrong · AUROC 0.783

Runs on an API · cost $0.013$12.64 per 1,000 questionsper 1M questions · 0.182 s per question (median latency)

Its run record (Method) →

54.9%52.0%–57.8%74.1%91.0%64.9%3.2$0.013$12.64–0.182 sp90 0.425 s
7Drex Max v1.1 Closed · Efficient (provisional)

Drex Max v1.1 · Nace.AI · decision model

Parameters: not disclosed by maker · Closed; introductory list price (the only one published) $0.03 per 1M input tokens: Efficient (provisional) (source: https://nace.ai/drex)

Logic 54.8% (95%: 51.8%–57.6%) · Math 70.1%

Non-hallucination 54.1% · “I don't know” on 30.8% of questions

Confidence: 90 answers kept at ≥99%, 89 right · 1 wrong · AUROC 0.793

Runs on an API · cost $0.022$22.37 per 1,000 questionsper 1M questions · 0.329 s per question (median latency)

Its run record (Method) →

54.8%51.8%–57.6%70.1%54.1%30.8%6.1$0.022$22.37–0.329 sp90 1.54 s
7Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell Open · 27B

Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell · Perplexity · decision model

Parameters: 27B dense

Logic 54.8% (95%: 51.8%–57.6%) · Math 72.5%

Non-hallucination 18.1% · “I don't know” on 9.1% of questions

Confidence: 62 answers kept at ≥99%, 62 right · 0 wrong · AUROC 0.803

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.081$81.16estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.108 s per question (median latency)

Its run record (Method) →

54.8%51.8%–57.6%72.5%18.1%9.1%9.0$0.081$81.16estimate · RTX PRO 6000 Blackwell117 sRTX PRO 60000.108 sp90 0.131 s
7d3-ultra Open · 26B

d3-ultra · vLLM Semantic Router · decision model

Parameters: 26.09B dense

Logic 54.6% (95%: 51.7%–57.5%) · Math 71.5%

Non-hallucination 10.1% · “I don't know” on 4.7% of questions

Confidence: 28 answers kept at ≥99%, 28 right · 0 wrong · AUROC 0.778

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.109$109estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.156 s per question (median latency)

Its run record (Method) →

54.6%51.7%–57.5%71.5%10.1%4.7%15.1$0.109$109estimate · RTX PRO 6000 Blackwell158 sRTX PRO 60000.156 sp90 0.185 s
7GLiDE API Closed · Efficient

GLiDE API · Fastino · decision model

Parameters: not disclosed by maker · Closed; list price $0.15 per 1M input tokens: Efficient (source: https://docs.fastino.ai/pricing)

Logic 54.5% (95%: 51.6%–57.4%) · Math 70.8%

Non-hallucination 60.0% · “I don't know” on 32.5% of questions

Confidence: 52 answers kept at ≥99%, 52 right · 0 wrong · AUROC 0.814

Runs on an API · cost $0.052$52.03 per 1,000 questionsper 1M questions · 0.271 s per question (median latency)

Its run record (Method) →

54.5%51.6%–57.4%70.8%60.0%32.5%6.6$0.052$52.03–0.271 sp90 0.291 s
7Mammoth-1 Open · 27B

Mammoth-1 · Base Labs · decision model

Parameters: 27B dense

Logic 54.5% (95%: 51.6%–57.4%) · Math 74.0%

Non-hallucination 72.3% · “I don't know” on 42.4% of questions

Confidence: 108 answers kept at ≥99%, 106 right · 2 wrong · AUROC 0.795

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.221 s per question (median latency)

Its run record (Method) →

54.5%51.6%–57.4%74.0%72.3%42.4%5.3$0.038$38.39 list$0.151$151estimate · RTX PRO 6000 Blackwell218 sRTX PRO 60000.221 sp90 0.244 s
7d3 Open · 26B

d3 · vLLM Semantic Router · decision model

Parameters: 26.09B dense

Logic 54.5% (95%: 51.6%–57.4%) · Math 70.4%

Non-hallucination 14.8% · “I don't know” on 7.2% of questions

Confidence: 54 answers kept at ≥99%, 54 right · 0 wrong · AUROC 0.806

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.057$57.18estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.076 s per question (median latency)

Its run record (Method) →

54.5%51.6%–57.4%70.4%14.8%7.2%4.6$0.057$57.18estimate · RTX PRO 6000 Blackwell83 sRTX PRO 60000.076 sp90 0.098 s
7Erk-14B off · chat Open · 15B

Erk-14B off · chat · eCloud Technology · chat model · reasoning off

Parameters: 14.77B dense

Logic 54.2% (95%: 51.3%–57.1%) · Math 90.8%

Non-hallucination 30.7% · “I don't know” on 16.2% of questions

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.390$390estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 10.2 s per question (median latency)

Its run record (Method) →

54.2%51.3%–57.1%90.8%30.7%16.2%–$0.390$390estimate · RTX PRO 6000 Blackwell564 sRTX PRO 600010.2 sp90 18.4 s
7Perplexity Decider 27B v1.2 API Open · 27B

Perplexity Decider 27B v1.2 API · Perplexity · decision model

Parameters: 27B dense

Logic 53.3% (95%: 50.4%–56.2%) · Math 69.8%

Non-hallucination 14.4% · “I don't know” on 7.1% of questions

Confidence: 80 answers kept at ≥99%, 78 right · 2 wrong · AUROC 0.798

Runs on an API · cost $0.007$7.46 per 1,000 questionsper 1M questions · 0.238 s per question (median latency)

Other settings: Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell)

Its run record (Method) →

53.3%50.4%–56.2%69.8%14.4%7.1%3.4$0.007$7.46–0.238 sp90 0.552 s
7Drex Max Open · 27B

Drex Max · Nace.AI · decision model

Parameters: 27B dense

Logic 53.1% (95%: 50.1%–56.0%) · Math 70.3%

Non-hallucination 7.3% · “I don't know” on 3.5% of questions

Confidence: 155 answers kept at ≥99%, 147 right · 8 wrong · AUROC 0.786

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.106 s per question (median latency)

Its run record (Method) →

53.1%50.1%–56.0%70.3%7.3%3.5%16.2$0.023$22.55 list$0.081$81.16estimate · RTX PRO 6000 Blackwell117 sRTX PRO 60000.106 sp90 0.143 s
–Perplexity Decider 27B v1.2 API · with web Open · 27B

Perplexity Decider 27B v1.2 API · with web · Perplexity · decision model

Parameters: 27B dense

Not tiered: with web

Logic 52.5% (95%: 49.6%–55.4%) · Math 69.3%

Non-hallucination 37.7% · “I don't know” on 19.9% of questions

Confidence: 79 answers kept at ≥99%, 78 right · 1 wrong · AUROC 0.800

Runs on an API · cost $1.030$1,030 per 1,000 questionsper 1M questions · 0.448 s per question (median latency)

Its run record (Method) →

52.5%49.6%–55.4%69.3%37.7%19.9%4.2$1.030$1,030–0.448 sp90 0.827 s
7SubQ Snap 1.1 Closed · Efficient

SubQ Snap 1.1 · Subquadratic · decision model

Parameters: not disclosed by maker · Closed; list price $0.04 per 1M input tokens: Efficient (source: https://docs.subq.ai/using-the-api/models/)

Logic 52.4% (95%: 49.4%–55.3%) · Math 68.6%

Non-hallucination 65.7% · “I don't know” on 39.0% of questions

Confidence: 72 answers kept at ≥99%, 70 right · 2 wrong · AUROC 0.787

Runs on an API · cost $0.015$14.92 per 1,000 questionsper 1M questions · 0.384 s per question (median latency)

Shown once with SubQ Snap Preview: the same answer to every question

Its run record (Method) →

52.4%49.4%–55.3%68.6%65.7%39.0%12.9$0.015$14.92–0.384 sp90 0.393 s
7Turion Jev 31B Open · 31B

Turion Jev 31B · Turion.AI · decision model

Parameters: 31B dense

Logic 52.3% (95%: 49.3%–55.2%) · Math 69.0%

Non-hallucination 1.5% · “I don't know” on 0.7% of questions

Confidence: 7 answers kept at ≥99%, 7 right · 0 wrong · AUROC 0.782

Runs on NVIDIA DGX Spark (GB10) · cost $0.025$24.91estimate · DGX Spark per 1,000 questionsper 1M questions · 0.385 s per question (median latency)

Its run record (Method) →

52.3%49.3%–55.2%69.0%1.5%0.7%2.9$0.025$24.91estimate · DGX Spark421 sDGX Spark0.385 sp90 0.533 s
7d1.2-ultra Closed · Undisclosed

d1.2-ultra · Liquid AI · decision model

Parameters: not disclosed by maker · Closed; no list price published: Undisclosed

Logic 51.6% (95%: 48.6%–54.5%) · Math 69.9%

Non-hallucination 94.5% · “I don't know” on 71.8% of questions

Confidence: 13 answers kept at ≥99%, 13 right · 0 wrong · AUROC 0.800

Runs on an API · cost $0.013$12.64 per 1,000 questionsper 1M questions · 0.220 s per question (median latency)

Its run record (Method) →

51.6%48.6%–54.5%69.9%94.5%71.8%3.0$0.013$12.64–0.220 sp90 0.341 s
7KnowLine-12B-Gen1 Open · 12B

KnowLine-12B-Gen1 · PelaAI · decision model

Parameters: 12B dense

Base model: Gemma 4 12B

Logic 50.6% (95%: 47.7%–53.5%) · Math 66.3%

Non-hallucination 8.8% · “I don't know” on 4.5% of questions

Confidence: 6 answers kept at ≥99%, 6 right · 0 wrong · AUROC 0.774

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.052$51.64estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.068 s per question (median latency)

Its run record (Method) →

50.6%47.7%–53.5%66.3%8.8%4.5%4.8$0.052$51.64estimate · RTX PRO 6000 Blackwell75 sRTX PRO 60000.068 sp90 0.088 s
8Gemini Decision 1 20261111 Closed · Efficient

Gemini Decision 1 20261111 · Google · decision model

Parameters: not disclosed by maker (closed weights) · Closed; list price $0.10 per 1M input tokens: Efficient (source: https://openrouter.ai/google/gemini-decision-1)

Logic 50.0% (95%: 47.1%–53.0%) · Math 71.3%

Non-hallucination 2.0% · “I don't know” on 1.0% of questions

Confidence: 43 answers kept at ≥99%, 43 right · 0 wrong · AUROC 0.768

Runs on an API · cost $0.037$36.77 per 1,000 questionsper 1M questions · 0.228 s per question (median latency)

Its run record (Method) →

50.0%47.1%–53.0%71.3%2.0%1.0%3.1$0.037$36.77–0.228 sp90 0.344 s
8d1.2 Closed · Undisclosed

d1.2 · Liquid AI · decision model

Parameters: not disclosed by maker · Closed; no list price published: Undisclosed

Logic 49.2% (95%: 46.2%–52.1%) · Math 65.5%

Non-hallucination 78.8% · “I don't know” on 56.6% of questions

Confidence: 28 answers kept at ≥99%, 28 right · 0 wrong · AUROC 0.742

Runs on an API · cost $0.006$6.02 per 1,000 questionsper 1M questions · 0.222 s per question (median latency)

Its run record (Method) →

49.2%46.2%–52.1%65.5%78.8%56.6%2.6$0.006$6.02–0.222 sp90 0.390 s
8Jeffan-9B-1211 Open · 9.0B

Jeffan-9B-1211 · tattrongvu · decision model

Parameters: 9B dense

Base model: Qwen3.5-9B

Logic 48.2% (95%: 45.3%–51.1%) · Math 59.8%

Non-hallucination 23.4% · “I don't know” on 13.1% of questions

Confidence: 10 answers kept at ≥99%, 10 right · 0 wrong · AUROC 0.753

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.039$39.35estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.052 s per question (median latency)

Its run record (Method) →

48.2%45.3%–51.1%59.8%23.4%13.1%4.0$0.039$39.35estimate · RTX PRO 6000 Blackwell57 sRTX PRO 60000.052 sp90 0.066 s
8Jev 1.14 Closed · Efficient

Jev 1.14 · TypeSafe · decision model

Parameters: not disclosed by maker · Closed; list price $0.042 per 1M input tokens: Efficient (source: https://docs.typesafe.ai/models)

Logic 48.2% (95%: 45.3%–51.1%) · Math 65.1%

Non-hallucination 24.5% · “I don't know” on 14.1% of questions

Confidence: 61 answers kept at ≥99%, 61 right · 0 wrong · AUROC 0.750

Runs on an API · cost $0.026$25.64 per 1,000 questionsper 1M questions · 0.166 s per question (median latency)

Its run record (Method) →

48.2%45.3%–51.1%65.1%24.5%14.1%3.3$0.026$25.64–0.166 sp90 0.193 s
8GLiDE 9B Open · 7.9B

GLiDE 9B · Fastino · decision model

Parameters: 7.94B dense

Logic 47.8% (95%: 44.9%–50.7%) · Math 57.9%

Non-hallucination 30.4% · “I don't know” on 17.7% of questions

Confidence: 32 answers kept at ≥99%, 32 right · 0 wrong · AUROC 0.749

Runs on NVIDIA DGX Spark (GB10) · cost $0.013$13.29estimate · DGX Spark per 1,000 questionsper 1M questions · 0.208 s per question (median latency)

Its run record (Method) →

47.8%44.9%–50.7%57.9%30.4%17.7%6.7$0.013$13.29estimate · DGX Spark225 sDGX Spark0.208 sp90 0.251 s
8celeris-1-decision Closed · Efficient

celeris-1-decision · Celeris · decision model

Parameters: not disclosed by maker · Closed; list price $0.04 per 1M input tokens: Efficient (source: https://docs.celeris.ai/pricing)

Logic 47.3% (95%: 44.4%–50.2%) · Math 70.0%

Non-hallucination 92.3% · “I don't know” on 72.2% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.735

Runs on an API · cost $0.015$15.40 per 1,000 questionsper 1M questions · 0.390 s per question (median latency)

Its run record (Method) →

47.3%44.4%–50.2%70.0%92.3%72.2%10.7$0.015$15.40–0.390 sp90 0.394 s
8celeris-1.1-decision Closed · Efficient

celeris-1.1-decision · Celeris · decision model

Parameters: not disclosed by maker · Closed; list price $0.004 per 1M input tokens: Efficient (source: https://docs.celeris.ai/pricing)

Logic 47.3% (95%: 44.4%–50.2%) · Math 69.9%

Non-hallucination 91.0% · “I don't know” on 71.0% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.733

Runs on an API · cost $0.002$1.54 per 1,000 questionsper 1M questions · 0.383 s per question (median latency)

Its run record (Method) →

47.3%44.4%–50.2%69.9%91.0%71.0%9.8$0.002$1.54–0.383 sp90 0.390 s
8StartLux-Decision-v1.1-9B-Instant Open · 9.0B

StartLux-Decision-v1.1-9B-Instant · StartLux · decision model

Parameters: 9B dense

Logic 46.6% (95%: 43.7%–49.5%) · Math 58.4%

Non-hallucination 17.4% · “I don't know” on 10.0% of questions

Confidence: 14 answers kept at ≥99%, 14 right · 0 wrong · AUROC 0.762

Runs on NVIDIA DGX Spark (GB10) · cost $0.010$10.04estimate · DGX Spark per 1,000 questionsper 1M questions · 0.160 s per question (median latency)

Its run record (Method) →

46.6%43.7%–49.5%58.4%17.4%10.0%3.8$0.010$10.04estimate · DGX Spark170 sDGX Spark0.160 sp90 0.175 s
8Adjudex 9B v5 Open · 9.0B

Adjudex 9B v5 · newsbang · decision model

Parameters: 9B dense

Base model: Qwen3.5-9B

Logic 46.4% (95%: 43.5%–49.3%) · Math 62.1%

Non-hallucination 12.2% · “I don't know” on 6.8% of questions

Confidence: 55 answers kept at ≥99%, 54 right · 1 wrong · AUROC 0.771

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.031$31.36estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.041 s per question (median latency)

Its run record (Method) →

46.4%43.5%–49.3%62.1%12.2%6.8%7.2$0.031$31.36estimate · RTX PRO 6000 Blackwell45 sRTX PRO 60000.041 sp90 0.046 s
8Elephant-1 Open · 9.0B

Elephant-1 · Base Labs · decision model

Parameters: 9B dense

Logic 46.4% (95%: 43.5%–49.3%) · Math 61.3%

Non-hallucination 75.6% · “I don't know” on 52.9% of questions

Confidence: 53 answers kept at ≥99%, 53 right · 0 wrong · AUROC 0.768

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.079 s per question (median latency)

Its run record (Method) →

46.4%43.5%–49.3%61.3%75.6%52.9%8.6$0.013$13.44 list$0.058$58.41estimate · RTX PRO 6000 Blackwell84 sRTX PRO 60000.079 sp90 0.098 s
8Jev 1.13 Closed · Efficient

Jev 1.13 · TypeSafe · decision model

Parameters: not disclosed by maker · Closed; list price $0.042 per 1M input tokens: Efficient (source: https://typesafe.ai/blog/introducing-system-one-models-and-jev)

Logic 46.3% (95%: 43.4%–49.2%) · Math 64.1%

Non-hallucination 15.8% · “I don't know” on 9.1% of questions

Confidence: 65 answers kept at ≥99%, 65 right · 0 wrong · AUROC 0.764

Runs on an API · cost $0.026$25.64 per 1,000 questionsper 1M questions · 0.167 s per question (median latency)

Its run record (Method) →

46.3%43.4%–49.2%64.1%15.8%9.1%5.9$0.026$25.64–0.167 sp90 0.203 s
8d1.1 Closed · Efficient

d1.1 · Liquid AI · decision model

Parameters: not disclosed by maker · Closed; list price $0.02 per 1M input tokens: Efficient (source: https://www.liquid.ai/pricing)

Logic 46.0% (95%: 43.2%–49.0%) · Math 65.2%

Non-hallucination 90.6% · “I don't know” on 71.6% of questions

Confidence: 48 answers kept at ≥99%, 48 right · 0 wrong · AUROC 0.748

Runs on an API · cost $0.006$6.02 per 1,000 questionsper 1M questions · 0.176 s per question (median latency)

Its run record (Method) →

46.0%43.2%–49.0%65.2%90.6%71.6%4.4$0.006$6.02–0.176 sp90 0.272 s
8StartLux-Decision-v1.1-4B-Instant Open · 4.0B

StartLux-Decision-v1.1-4B-Instant · StartLux · decision model

Parameters: 4B dense

Logic 45.9% (95%: 43.0%–48.8%) · Math 55.8%

Non-hallucination 16.0% · “I don't know” on 9.2% of questions

Confidence: 21 answers kept at ≥99%, 21 right · 0 wrong · AUROC 0.720

Runs on NVIDIA DGX Spark (GB10) · cost $0.005$4.62estimate · DGX Spark per 1,000 questionsper 1M questions · 0.073 s per question (median latency)

Its run record (Method) →

45.9%43.0%–48.8%55.8%16.0%9.2%3.3$0.005$4.62estimate · DGX Spark78 sDGX Spark0.073 sp90 0.084 s
8Dexy-Decision-9B-V2 Open · 9.0B

Dexy-Decision-9B-V2 · Patrick-18 · decision model

Parameters: 9B dense

Base model: vllm-sr/d3-flash

Logic 45.3% (95%: 42.4%–48.3%) · Math 58.8%

Non-hallucination 12.4% · “I don't know” on 7.1% of questions

Confidence: 56 answers kept at ≥99%, 56 right · 0 wrong · AUROC 0.759

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.034$33.81estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.047 s per question (median latency)

Its run record (Method) →

45.3%42.4%–48.3%58.8%12.4%7.1%14.9$0.034$33.81estimate · RTX PRO 6000 Blackwell49 sRTX PRO 60000.047 sp90 0.053 s
8Mercury Decide Closed · Efficient

Mercury Decide · Inception · decision model

Parameters: not disclosed by maker · Closed; list price $0.04 per 1M input tokens: Efficient (source: https://openrouter.ai/inception/mercury-decide)

Logic 45.1% (95%: 42.2%–48.0%) · Math 53.5%

Non-hallucination 20.5% · “I don't know” on 12.2% of questions

Confidence: 91 answers kept at ≥99%, 84 right · 7 wrong · AUROC 0.746

Runs on an API · cost $0.008$7.52 per 1,000 questionsper 1M questions · 0.301 s per question (median latency)

Its run record (Method) →

45.1%42.2%–48.0%53.5%20.5%12.2%15.4$0.008$7.52–0.301 sp90 0.360 s
8Perplexity Decider Flash v1.3 Open · 4.0B

Perplexity Decider Flash v1.3 · Perplexity · decision model

Parameters: 4B dense

Logic 44.7% (95%: 41.8%–47.6%) · Math 55.9%

Non-hallucination 24.4% · “I don't know” on 14.9% of questions

Confidence: 17 answers kept at ≥99%, 17 right · 0 wrong · AUROC 0.757

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.026$26.44estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.033 s per question (median latency)

Its run record (Method) →

44.7%41.8%–47.6%55.9%24.4%14.9%10.1$0.026$26.44estimate · RTX PRO 6000 Blackwell38 sRTX PRO 60000.033 sp90 0.041 s
8Microsoft-Decision-1 Closed · Efficient

Microsoft-Decision-1 · Microsoft · decision model

Parameters: not disclosed by maker · Closed; list price $0.042 per 1M input tokens: Efficient (source: https://openrouter.ai/microsoft/microsoft-decision-1)

Base model: Qwen3.5-9B

Logic 44.4% (95%: 41.6%–47.4%) · Math 59.4%

Non-hallucination 19.8% · “I don't know” on 12.0% of questions

Confidence: 20 answers kept at ≥99%, 20 right · 0 wrong · AUROC 0.712

Runs on an API · cost $0.013$13.42 per 1,000 questionsper 1M questions · 0.315 s per question (median latency)

Its run record (Method) →

44.4%41.6%–47.4%59.4%19.8%12.0%8.0$0.013$13.42–0.315 sp90 0.362 s
8Valen-Decision-1.5-9B Open · 9.0B

Valen-Decision-1.5-9B · Valen-Team · decision model

Parameters: 9B dense

Base model: Qwen3.5-9B

Logic 44.4% (95%: 41.5%–47.3%) · Math 55.3%

Non-hallucination 13.6% · “I don't know” on 8.2% of questions

Confidence: 37 answers kept at ≥99%, 35 right · 2 wrong · AUROC 0.701

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.042$42.42estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.060 s per question (median latency)

Its run record (Method) →

44.4%41.5%–47.3%55.3%13.6%8.2%5.8$0.042$42.42estimate · RTX PRO 6000 Blackwell61 sRTX PRO 60000.060 sp90 0.067 s
8KnowLine-9B-Gen1 Open · 9.0B

KnowLine-9B-Gen1 · PelaAI · decision model

Parameters: 9B dense

Base model: Qwen3.5-9B

Logic 44.3% (95%: 41.4%–47.2%) · Math 59.8%

Non-hallucination 6.7% · “I don't know” on 3.9% of questions

Confidence: 30 answers kept at ≥99%, 30 right · 0 wrong · AUROC 0.725

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.036$36.27estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.050 s per question (median latency)

Its run record (Method) →

44.3%41.4%–47.2%59.8%6.7%3.9%3.5$0.036$36.27estimate · RTX PRO 6000 Blackwell52 sRTX PRO 60000.050 sp90 0.061 s
8vera-medium-omni Open · 4.2B

vera-medium-omni · scar-ai · decision model

Parameters: 4.2B dense

Base model: Qwen3.5-4B

Logic 44.1% (95%: 41.2%–47.0%) · Math 56.8%

Non-hallucination 13.6% · “I don't know” on 8.1% of questions

Confidence: 6 answers kept at ≥99%, 6 right · 0 wrong · AUROC 0.739

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.026$25.82estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.032 s per question (median latency)

Its run record (Method) →

44.1%41.2%–47.0%56.8%13.6%8.1%4.7$0.026$25.82estimate · RTX PRO 6000 Blackwell37 sRTX PRO 60000.032 sp90 0.037 s
8KnowLine-9B-Gen2 Open · 9.0B

KnowLine-9B-Gen2 · PelaAI · decision model

Parameters: 9B dense

Base model: Qwen3.5-9B

Logic 43.9% (95%: 41.0%–46.8%) · Math 60.1%

Non-hallucination 19.5% · “I don't know” on 12.1% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.730

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.040$39.96estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.052 s per question (median latency)

Its run record (Method) →

43.9%41.0%–46.8%60.1%19.5%12.1%3.2$0.040$39.96estimate · RTX PRO 6000 Blackwell58 sRTX PRO 60000.052 sp90 0.065 s
8Cerno 9B v1.2 Open · 9.0B

Cerno 9B v1.2 · Apertix AI · decision model

Parameters: 9B dense

Logic 43.3% (95%: 40.4%–46.2%) · Math 56.0%

Non-hallucination 6.0% · “I don't know” on 3.4% of questions

Confidence: 27 answers kept at ≥99%, 26 right · 1 wrong · AUROC 0.752

Runs on NVIDIA DGX Spark (GB10) · cost $0.013$13.35estimate · DGX Spark per 1,000 questionsper 1M questions · 0.209 s per question (median latency)

Its run record (Method) →

43.3%40.4%–46.2%56.0%6.0%3.4%17.0$0.013$13.35estimate · DGX Spark226 sDGX Spark0.209 sp90 0.269 s
8Perplexity Decider Flash v1.2 Open · 4.0B

Perplexity Decider Flash v1.2 · Perplexity · decision model

Parameters: 4B dense

Logic 43.1% (95%: 40.2%–46.0%) · Math 55.2%

Non-hallucination 28.9% · “I don't know” on 18.3% of questions

Confidence: 35 answers kept at ≥99%, 35 right · 0 wrong · AUROC 0.729

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.045 s per question (median latency)

Its run record (Method) →

43.1%40.2%–46.0%55.2%28.9%18.3%6.0$0.004$3.76 list$0.034$34.43estimate · RTX PRO 6000 Blackwell50 sRTX PRO 60000.045 sp90 0.055 s
9Open-Jev-27B-v1.1 Open · 27B

Open-Jev-27B-v1.1 · Zefan Cai · decision model

Parameters: 27B dense

Base model: Qwen3.8-27B + adapter

Logic 42.6% (95%: 39.7%–45.5%) · Math 54.6%

Non-hallucination 28.2% · “I don't know” on 17.7% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.733

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.294$294estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.413 s per question (median latency)

Its run record (Method) →

42.6%39.7%–45.5%54.6%28.2%17.7%12.9$0.294$294estimate · RTX PRO 6000 Blackwell426 sRTX PRO 60000.413 sp90 0.576 s
9KnowLine 4B Gen5.5 Open · 4.0B

KnowLine 4B Gen5.5 · PelaAI · decision model

Parameters: 4B dense

Logic 42.3% (95%: 39.5%–45.2%) · Math 56.0%

Non-hallucination 8.3% · “I don't know” on 5.0% of questions

Confidence: 18 answers kept at ≥99%, 17 right · 1 wrong · AUROC 0.708

Runs on NVIDIA DGX Spark (GB10) · cost $0.006$5.89estimate · DGX Spark per 1,000 questionsper 1M questions · 0.096 s per question (median latency)

Its run record (Method) →

42.3%39.5%–45.2%56.0%8.3%5.0%13.7$0.006$5.89estimate · DGX Spark100 sDGX Spark0.096 sp90 0.115 s
9Drex 2.0 Open · 35B MoE · 3.0B active

Drex 2.0 · Nace.AI · decision model

Parameters: 35B MoE · 3B active · Mixture of experts: 35B total, about 3B active per step. Size class uses 35B.

Logic 41.9% (95%: 39.0%–44.8%) · Math 59.0%

Non-hallucination 37.7% · “I don't know” on 25.1% of questions

Confidence: 33 answers kept at ≥99%, 33 right · 0 wrong · AUROC 0.717

Runs on an API · cost $0.018$18.41 per 1,000 questionsper 1M questions · 0.345 s per question (median latency)

Its run record (Method) →

41.9%39.0%–44.8%59.0%37.7%25.1%4.3$0.018$18.41–0.345 sp90 0.504 s
9Clef Open · 27B

Clef · Cloudflare · decision model

Parameters: 27B dense

Logic 40.8% (95%: 38.0%–43.7%) · Math 51.1%

Non-hallucination 3.6% · “I don't know” on 2.1% of questions

Confidence: 6 answers kept at ≥99%, 6 right · 0 wrong · AUROC 0.697

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.196 s per question (median latency)

Its run record (Method) →

40.8%38.0%–43.7%51.1%3.6%2.1%15.7$0.106$106 list$0.141$141estimate · RTX PRO 6000 Blackwell204 sRTX PRO 60000.196 sp90 0.222 s
9d1 Closed · Efficient

d1 · Liquid AI · decision model

Parameters: not disclosed by maker · Closed; list price $0.02 per 1M input tokens: Efficient (source: https://www.liquid.ai/pricing)

Logic 40.8% (95%: 38.0%–43.7%) · Math 55.6%

Non-hallucination 11.3% · “I don't know” on 7.2% of questions

Confidence: 56 answers kept at ≥99%, 55 right · 1 wrong · AUROC 0.723

Runs on an API · cost $0.006$6.02 per 1,000 questionsper 1M questions · 0.188 s per question (median latency)

Its run record (Method) →

40.8%38.0%–43.7%55.6%11.3%7.2%10.9$0.006$6.02–0.188 sp90 0.290 s
9OneJev-27B Open · 27B

OneJev-27B · OmniJev · decision model

Parameters: 27B dense

Logic 40.6% (95%: 37.8%–43.5%) · Math 53.8%

Non-hallucination 1.2% · “I don't know” on 0.7% of questions

Confidence: 4 answers kept at ≥99%, 4 right · 0 wrong · AUROC 0.726

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.086$86.07estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.108 s per question (median latency)

Its run record (Method) →

40.6%37.8%–43.5%53.8%1.2%0.7%4.6$0.086$86.07estimate · RTX PRO 6000 Blackwell124 sRTX PRO 60000.108 sp90 0.136 s
9Chinese Decision Pointer 4B Open · 4.0B

Chinese Decision Pointer 4B · quincyqiang · decision model

Parameters: 4B dense

Base model: Qwen3.5-4B

Logic 40.4% (95%: 37.5%–43.3%) · Math 50.4%

Non-hallucination 14.6% · “I don't know” on 9.0% of questions

Confidence: 39 answers kept at ≥99%, 39 right · 0 wrong · AUROC 0.724

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.030$29.51estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.038 s per question (median latency)

Its run record (Method) →

40.4%37.5%–43.3%50.4%14.6%9.0%15.5$0.030$29.51estimate · RTX PRO 6000 Blackwell43 sRTX PRO 60000.038 sp90 0.048 s
9Verdin-4B-v1.1 Open · 4.0B

Verdin-4B-v1.1 · victorbennett · decision model

Parameters: 4B dense

Base model: Qwen3.5-4B

Logic 39.7% (95%: 36.9%–42.6%) · Math 51.5%

Non-hallucination 12.6% · “I don't know” on 7.9% of questions

Confidence: 35 answers kept at ≥99%, 35 right · 0 wrong · AUROC 0.729

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.033$33.20estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.045 s per question (median latency)

Its run record (Method) →

39.7%36.9%–42.6%51.5%12.6%7.9%10.8$0.033$33.20estimate · RTX PRO 6000 Blackwell48 sRTX PRO 60000.045 sp90 0.051 s
9Winnow-12B Open · 12B

Winnow-12B · EldanRing · decision model

Parameters: 12B dense

Logic 39.7% (95%: 36.9%–42.6%) · Math 46.9%

Non-hallucination 29.7% · “I don't know” on 19.9% of questions

Confidence: 32 answers kept at ≥99%, 32 right · 0 wrong · AUROC 0.727

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.052$51.64estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.066 s per question (median latency)

Its run record (Method) →

39.7%36.9%–42.6%46.9%29.7%19.9%14.4$0.052$51.64estimate · RTX PRO 6000 Blackwell75 sRTX PRO 60000.066 sp90 0.091 s
9Cygnet Open · 12B

Cygnet · Blockbrain Labs · decision model

Parameters: 11.95B dense

Base model: Gemma 4 12B IT

Logic 39.3% (95%: 36.5%–42.2%) · Math 44.3%

Non-hallucination 11.3% · “I don't know” on 7.1% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.720

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.040 s per question (median latency)

Its run record (Method) →

39.3%36.5%–42.2%44.3%11.3%7.1%11.2$0.040$39.71 list$0.031$30.74estimate · RTX PRO 6000 Blackwell44 sRTX PRO 60000.040 sp90 0.048 s
9Clef-Flash Open · 9.0B

Clef-Flash · Cloudflare · decision model

Parameters: 9B dense

Logic 38.9% (95%: 36.1%–41.8%) · Math 49.8%

Non-hallucination 14.4% · “I don't know” on 9.2% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.659

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.110 s per question (median latency)

Its run record (Method) →

38.9%36.1%–41.8%49.8%14.4%9.2%11.5$0.017$16.83 list$0.081$81.16estimate · RTX PRO 6000 Blackwell117 sRTX PRO 60000.110 sp90 0.137 s
10GLiDE 2B Open · 1.9B

GLiDE 2B · Fastino · decision model

Parameters: 1.88B dense

Logic 32.3% (95%: 29.6%–35.1%) · Math 36.8%

Non-hallucination 32.9% · “I don't know” on 24.6% of questions

Confidence: 14 answers kept at ≥99%, 14 right · 0 wrong · AUROC 0.632

Runs on NVIDIA DGX Spark (GB10) · cost $0.002$2.26estimate · DGX Spark per 1,000 questionsper 1M questions · 0.034 s per question (median latency)

Its run record (Method) →

32.3%29.6%–35.1%36.8%32.9%24.6%16.2$0.002$2.26estimate · DGX Spark38 sDGX Spark0.034 sp90 0.042 s
10Metask-Jev-4B Open · 4.5B

Metask-Jev-4B · Metask Lab · decision model

Parameters: 4.54B dense

Logic 32.1% (95%: 29.4%–34.9%) · Math 39.2%

Non-hallucination 1.5% · “I don't know” on 1.1% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.660

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.091$90.99estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.123 s per question (median latency)

Its run record (Method) →

32.1%29.4%–34.9%39.2%1.5%1.1%2.5$0.091$90.99estimate · RTX PRO 6000 Blackwell132 sRTX PRO 60000.123 sp90 0.152 s
10KnowLine-2B-Gen1 Open · 2.0B

KnowLine-2B-Gen1 · PelaAI · decision model

Parameters: 2B dense

Base model: Qwen3.5-2B

Logic 31.7% (95%: 29.1%–34.5%) · Math 35.3%

Non-hallucination 15.7% · “I don't know” on 11.5% of questions

Confidence: 8 answers kept at ≥99%, 8 right · 0 wrong · AUROC 0.601

Runs on NVIDIA DGX Spark (GB10) · cost $0.004$3.78estimate · DGX Spark per 1,000 questionsper 1M questions · 0.059 s per question (median latency)

Its run record (Method) →

31.7%29.1%–34.5%35.3%15.7%11.5%8.3$0.004$3.78estimate · DGX Spark64 sDGX Spark0.059 sp90 0.070 s
10StartLux-Decision-v1.1-2B-Instant Open · 2.0B

StartLux-Decision-v1.1-2B-Instant · StartLux · decision model

Parameters: 2B dense

Logic 30.8% (95%: 28.1%–33.5%) · Math 38.7%

Non-hallucination 31.4% · “I don't know” on 23.6% of questions

Confidence: 2 answers kept at ≥99%, 2 right · 0 wrong · AUROC 0.640

Runs on NVIDIA DGX Spark (GB10) · cost $0.003$2.52estimate · DGX Spark per 1,000 questionsper 1M questions · 0.040 s per question (median latency)

Its run record (Method) →

30.8%28.1%–33.5%38.7%31.4%23.6%5.3$0.003$2.52estimate · DGX Spark43 sDGX Spark0.040 sp90 0.045 s
10d1-2B Open · 1.9B

d1-2B · Liquid AI · decision model

Parameters: 1.92B dense

Base model: LFM3-VL-2B

Logic 30.7% (95%: 28.0%–33.4%) · Math 37.1%

Non-hallucination 86.5% · “I don't know” on 78.3% of questions

Confidence: 3 answers kept at ≥99%, 3 right · 0 wrong · AUROC 0.620

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.015$14.76estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.017 s per question (median latency)

Its run record (Method) →

30.7%28.0%–33.4%37.1%86.5%78.3%6.9$0.015$14.76estimate · RTX PRO 6000 Blackwell21 sRTX PRO 60000.017 sp90 0.023 s
10GLiDE 0.8B Open · 0.8B

GLiDE 0.8B · Fastino · decision model

Parameters: 0.75B dense

Logic 29.2% (95%: 26.6%–31.9%) · Math 32.1%

Non-hallucination 26.1% · “I don't know” on 20.0% of questions

Confidence: 9 answers kept at ≥99%, 9 right · 0 wrong · AUROC 0.617

Runs on NVIDIA DGX Spark (GB10) · cost $0.002$1.58estimate · DGX Spark per 1,000 questionsper 1M questions · 0.023 s per question (median latency)

Its run record (Method) →

29.2%26.6%–31.9%32.1%26.1%20.0%14.0$0.002$1.58estimate · DGX Spark27 sDGX Spark0.023 sp90 0.029 s
10OneJev1.6-0.8B Open · 0.8B

OneJev1.6-0.8B · OmniJev · decision model

Parameters: 0.8B dense

Logic 27.0% (95%: 24.5%–29.7%) · Math 28.2%

Non-hallucination 4.6% · “I don't know” on 3.4% of questions

Confidence: 2 answers kept at ≥99%, 2 right · 0 wrong · AUROC 0.545

Runs on NVIDIA DGX Spark (GB10) · cost $0.001$1.16estimate · DGX Spark per 1,000 questionsper 1M questions · 0.016 s per question (median latency)

Its run record (Method) →

27.0%24.5%–29.7%28.2%4.6%3.4%11.2$0.001$1.16estimate · DGX Spark20 sDGX Spark0.016 sp90 0.019 s
10KnowLine-0.8B-Gen1 Open · 0.8B

KnowLine-0.8B-Gen1 · PelaAI · decision model

Parameters: 0.8B dense

Base model: Qwen3.5-0.8B

Logic 26.7% (95%: 24.2%–29.3%) · Math 28.2%

Non-hallucination 1.2% · “I don't know” on 0.9% of questions

Confidence: 2 answers kept at ≥99%, 2 right · 0 wrong · AUROC 0.573

Runs on NVIDIA DGX Spark (GB10) · cost $0.002$2.00estimate · DGX Spark per 1,000 questionsper 1M questions · 0.031 s per question (median latency)

Its run record (Method) →

26.7%24.2%–29.3%28.2%1.2%0.9%8.4$0.002$2.00estimate · DGX Spark34 sDGX Spark0.031 sp90 0.039 s
11StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B

StartLux-Decision-v1.1-0.8B-Instant · StartLux · decision model

Parameters: 0.8B dense

Base model: Qwen3.5-0.8B

Logic 24.7% (95%: 22.3%–27.3%) · Math 29.7%

Non-hallucination 21.4% · “I don't know” on 16.8% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.641

Runs on NVIDIA DGX Spark (GB10) · cost $0.002$1.79estimate · DGX Spark per 1,000 questionsper 1M questions · 0.027 s per question (median latency)

Its run record (Method) →

24.7%22.3%–27.3%29.7%21.4%16.8%7.8$0.002$1.79estimate · DGX Spark30 sDGX Spark0.027 sp90 0.031 s
11GLiClass v4 Edge Open · 0.4B

GLiClass v4 Edge · decision model

Parameters: 0.4B dense

Base model: DeBERTa-v3-large

Logic 23.6% (95%: 21.2%–26.1%) · Math 22.7%

Non-hallucination 12.4% · “I don't know” on 9.7% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.527

Runs on NVIDIA DGX Spark (GB10) · cost $0.001$1.31estimate · DGX Spark per 1,000 questionsper 1M questions · 0.015 s per question (median latency)

Its run record (Method) →

23.6%21.2%–26.1%22.7%12.4%9.7%8.7$0.001$1.31estimate · DGX Spark22 sDGX Spark0.015 sp90 0.028 s
11GLiDE 0.4B Open · 0.4B

GLiDE 0.4B · Fastino · decision model

Parameters: 0.40B dense

Logic 23.2% (95%: 20.8%–25.8%) · Math 20.6%

Non-hallucination 76.1% · “I don't know” on 69.8% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.556

Runs on NVIDIA DGX Spark (GB10) · cost $0.001$0.95estimate · DGX Spark per 1,000 questionsper 1M questions · 0.013 s per question (median latency)

Its run record (Method) →

23.2%20.8%–25.8%20.6%76.1%69.8%8.8$0.001$0.95estimate · DGX Spark16 sDGX Spark0.013 sp90 0.016 s
11Strands Decider 2B Open · 2.0B

Strands Decider 2B · Strands Agents · decision model

Parameters: 2B dense

Logic 22.4% (95%: 20.1%–24.9%) · Math 28.7%

Non-hallucination 8.3% · “I don't know” on 6.5% of questions

Confidence, kept at ≥99%: No answers at this threshold · AUROC 0.568

Runs on NVIDIA RTX PRO 6000 Blackwell (96 GB) · cost $0.028$27.67estimate · RTX PRO 6000 Blackwell per 1,000 questionsper 1M questions · 0.036 s per question (median latency)

Its run record (Method) →

22.4%20.1%–24.9%28.7%8.3%6.5%12.0$0.028$27.67estimate · RTX PRO 6000 Blackwell40 sRTX PRO 60000.036 sp90 0.043 s

updated 16 Nov 2026 22:07 UTC

Not ranked (only complete runs are): Spark-X2.6-4B (high · chat) (not running now, 20 of 1,125 stored, last answer 13 Nov 02:18 UTC); Grok 4.7 (chat) (paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)); Claude Fable 5.2 (chat) (not running now, 147 of 1,125 stored, last answer 12 Nov 11:32 UTC); Granite-4.2-3B (off · chat) (still running, 259 of 1,125); GPT-6 Luna (max · chat) (paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)); DeepSeek-V4-Flash (chat) (sample: a random subset by design, 317 answers stored, not the full 1,125); Claude Haiku 5.5 (max · chat) (paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)).

Score: the share of the 1,125 Logic questions answered right, the whisker its 95% interval. # is the tier: rows in one tier are not told apart by the paired test (a line separates tiers); – : with web or sample, not tiered.

One row per model: its highest reasoning setting (off < low < medium < high), a model with one setup that setup; a tie by the earliest run, then the name. Perplexity Decider 27B v1.2: its API setup on every part (its other setup is under other settings). The model's other settings are under its row; All setups lists every setup.

# is the tier: rows in accuracy order, the top row starts Tier 1, a row joins the tier unless the paired test behind the tie letters (exact McNemar, Holm-corrected over all pairs of tiered rows in view, p < 0.05) separates it from the tier's top row; otherwise it starts the next tier. – : with web or sample, not tiered.

Paired comparisons

PairBenchmarkQuestionsFirstSecondDifference (pts)pHolm pResult
LFM3-1.7B (on) vs MiniCPM5-2B (on)Logic1,12584.2%88.9%-4.6< 0.0001< 0.0001MiniCPM5-2B (on) ahead
LFM3-1.7B (on) vs MiniCPM5-2B (on)Math2,16398.9%99.2%-0.30.0430.171no clear difference
LFM3-1.7B (on) vs MiniCPM5-2B (on)English reading54771.1%78.1%-7.0< 0.0001< 0.0001MiniCPM5-2B (on) ahead
LFM3-1.7B (off) vs MiniCPM5-2B (off)Logic1,12584.6%83.6%+1.00.2380.238no clear difference
LFM3-1.7B (off) vs MiniCPM5-2B (off)Math2,16398.9%98.2%+0.80.00020.0017LFM3-1.7B (off) ahead
LFM3-1.7B (off) vs MiniCPM5-2B (off)English reading54771.8%67.8%+4.00.0220.109no clear difference
LFM3-1.7B (on) vs Gemma 4 E4B (on, 32k, sampled)Logic1,12584.2%85.9%-1.70.0510.171no clear difference
LFM3-1.7B (on) vs Gemma 4 E4B (on, 32k, sampled)Math2,16398.9%98.5%+0.40.0500.171no clear difference
LFM3-1.7B (on) vs Gemma 4 E2B (on, 32k, sampled)Logic1,12584.2%76.8%+7.5< 0.0001< 0.0001LFM3-1.7B (on) ahead
LFM3-1.7B (on) vs Gemma 4 E2B (on, 32k, sampled)Math2,16398.9%97.7%+1.3< 0.0001< 0.0001LFM3-1.7B (on) ahead
LFM3-1.7B (on) vs Claude Opus 5.5Math2,16398.9%99.8%-0.8< 0.0001< 0.0001Claude Opus 5.5 ahead
LFM3-1.7B (on) vs Claude Sonnet 5.5Math2,16398.9%99.7%-0.7< 0.0001< 0.0001Claude Sonnet 5.5 ahead
LFM3-1.7B (on) vs Grok 4.7Math2,16398.9%99.5%-0.60.00010.0011Grok 4.7 ahead
LFM3-1.7B (on) vs Muse EmberMath2,16398.9%99.6%-0.60.00010.0011Muse Ember ahead
LFM3-1.7B (on) vs Gemma 4 31BMath2,16398.9%99.4%-0.50.00500.030Gemma 4 31B ahead

Each question's score is the mean over a setup's runs (one run: 0 or 1; three seeds: 0, ⅓, ⅔ or 1); only questions every run of both setups answered count. The difference is the first setup's mean score minus the second's, in points. p: two-sided sign-flip (paired permutation) test on the sum of the per-question differences themselves (each flip keeps its size, e.g. ⅓ or 1), 200,000 random flips with a fixed seed. Holm: corrected over all 15 planned tests in this table, a pending one counted as p = 1 until its runs are in. 'Ahead' when the Holm-corrected p is below 0.05. LFM3-1.7B against the large models is its thinking-on 3-seed setup. English reading: Reading's 547 English questions (its per-question results, turkce #10695). A 3-seed row's questions per minute, cost and GPU time elsewhere on the page are the mean of its three runs' own values (each run's questions over its own wall time), not pooled.

Chat and decision models on Logic

  1. Chat models
  2. GPT-6 Astra chat

    GPT-6 Astra (chat): 96.7%; 95% interval 95.5% to 97.6%

    96.7% (95%–98%)
  3. Gemini 4 Carbon chat

    Gemini 4 Carbon (chat): 96.7%; 95% interval 95.5% to 97.6%

    96.7% (95%–98%)
  4. GPT-6.1 Sol chat

    GPT-6.1 Sol (chat): 96.5%; 95% interval 95.3% to 97.5%

    96.5% (95%–97%)
  5. Qwen4-Max high · chat

    Qwen4-Max (high · chat): 96.3%; 95% interval 95.0% to 97.2%

    96.3% (95%–97%)
  6. Claude Opus 5.5 chat

    Claude Opus 5.5 (chat): 96.2%; 95% interval 94.9% to 97.2%

    96.2% (95%–97%)
  7. DeepSeek-V4.1-Flash high · chat · 1017

    DeepSeek-V4.1-Flash (high · chat · 1017): 96.2%; 95% interval 94.9% to 97.2%

    96.2% (95%–97%)
  8. Gemini 4 Hydrogen chat

    Gemini 4 Hydrogen (chat): 96.1%; 95% interval 94.8% to 97.1%

    96.1% (95%–97%)
  9. Inkling 2 Small high · chat

    Inkling 2 Small (high · chat): 95.6%; 95% interval 94.3% to 96.7%

    95.6% (94%–97%)
  10. Claude Sonnet 5.5 chat

    Claude Sonnet 5.5 (chat): 95.5%; 95% interval 94.1% to 96.5%

    95.5% (94%–97%)
  11. Muse Ember max · chat

    Muse Ember (max · chat): 95.5%; 95% interval 94.1% to 96.5%

    95.5% (94%–97%)
  12. Inkling-Small high · chat · Jul 2026

    Inkling-Small (high · chat · Jul 2026): 95.4%; 95% interval 94.0% to 96.5%

    95.4% (94%–96%)
  13. Claude Fable 5.1 chat

    Claude Fable 5.1 (chat): 95.2%; 95% interval 93.8% to 96.3%

    95.2% (94%–96%)
  14. MiMo-V2.6-Flash on · chat

    MiMo-V2.6-Flash (on · chat): 95.1%; 95% interval 93.7% to 96.2%

    95.1% (94%–96%)
  15. Qwen3.5-397B-A17B on · chat

    Qwen3.5-397B-A17B (on · chat): 95.1%; 95% interval 93.7% to 96.2%

    95.1% (94%–96%)
  16. Qwen3.8-27B low · chat

    Qwen3.8-27B (low · chat): 94.0%; 95% interval 92.5% to 95.3%

    94.0% (93%–95%)
  17. GPT-6.1 Terra chat

    GPT-6.1 Terra (chat): 94.0%; 95% interval 92.4% to 95.2%

    94.0% (92%–95%)
  18. MiniMax-M3.1-Flash high · chat

    MiniMax-M3.1-Flash (high · chat): 94.0%; 95% interval 92.4% to 95.2%

    94.0% (92%–95%)
  19. Qwen3.8 Flash off · chat

    Qwen3.8 Flash (off · chat): 94.0%; 95% interval 92.4% to 95.2%

    94.0% (92%–95%)
  20. Gemma 4 31B chat

    Gemma 4 31B (chat): 93.3%; 95% interval 91.7% to 94.6%

    93.3% (92%–95%)
  21. GLM-5.3-Flash high · chat

    GLM-5.3-Flash (high · chat): 93.2%; 95% interval 91.5% to 94.5%

    93.2% (92%–94%)
  22. Qwen3.5-4B on · chat

    Qwen3.5-4B (on · chat): 92.4%; 95% interval 90.7% to 93.8%

    92.4% (91%–94%)
  23. Granite-4.2-3B high · chat

    Granite-4.2-3B (high · chat): 91.6%; 95% interval 89.9% to 93.1%

    91.6% (90%–93%)
  24. MiniCPM5-2B on · chat

    MiniCPM5-2B (on · chat): 88.9% (mean of 3 seeds); min–max of 3 seeds 88.3% to 89.6%

    88.9% (88%–90%)
  25. Gemma 4 12B Unified chat

    Gemma 4 12B Unified (chat): 87.7%; 95% interval 85.7% to 89.5%

    87.7% (86%–90%)
  26. LFM2.5-24B-A2B on · chat

    LFM2.5-24B-A2B (on · chat): 86.7%; 95% interval 84.6% to 88.5%

    86.7% (85%–89%)
  27. LFM3-1.7B off · chat

    LFM3-1.7B (off · chat): 85.5%; 95% interval 83.3% to 87.4%

    85.5% (83%–87%)
  28. DeepSeek-V4-Flash chat

    DeepSeek-V4-Flash (chat): 83.6%; 95% interval 79.1% to 87.3%

    83.6% (79%–87%)
  29. Gemma 4 E4B chat

    Gemma 4 E4B (chat): 83.2%; 95% interval 80.9% to 85.3%

    83.2% (81%–85%)
  30. LFM2.5-2.6B on · chat

    LFM2.5-2.6B (on · chat): 75.4%; 95% interval 72.8% to 77.8%

    75.4% (73%–78%)
  31. Gemma 4 E2B chat

    Gemma 4 E2B (chat): 72.9%; 95% interval 70.2% to 75.4%

    72.9% (70%–75%)
  32. Qwen3-14B off · chat

    Qwen3-14B (off · chat): 65.3%; 95% interval 62.5% to 68.1%

    65.3% (63%–68%)
  33. Raon-1 nano off · chat

    Raon-1 nano (off · chat): 65.1%; 95% interval 62.2% to 67.8%

    65.1% (62%–68%)
  34. Qwen3.5-2B off · chat

    Qwen3.5-2B (off · chat): 64.8%; 95% interval 62.0% to 67.5%

    64.8% (62%–68%)
  35. LFM2.5-8B-A1B on · chat

    LFM2.5-8B-A1B (on · chat): 60.2%; 95% interval 57.3% to 63.0%

    60.2% (57%–63%)
  36. GPT-6 Luna chat

    GPT-6 Luna (chat): 58.0%; 95% interval 55.0% to 60.8%

    58.0% (55%–61%)
  37. Erk-14B off · chat

    Erk-14B (off · chat): 54.2%; 95% interval 51.3% to 57.1%

    54.2% (51%–57%)
  38. Decision models, the 5 highest
  39. Karpo Decision 31B

    Karpo Decision 31B: 60.7%; 95% interval 57.8% to 63.5%

    60.7% (58%–64%)
  40. REAX 1.3

    REAX 1.3: 59.3%; 95% interval 56.4% to 62.1%

    59.3% (56%–62%)
  41. KnowLine-31B-Gen1.5

    KnowLine-31B-Gen1.5: 59.0%; 95% interval 56.1% to 61.9%

    59.0% (56%–62%)
  42. KnowLine-31B-Gen2

    KnowLine-31B-Gen2: 58.9%; 95% interval 56.0% to 61.8%

    58.9% (56%–62%)
  43. Copernicus-31B

    Copernicus-31B: 58.7%; 95% interval 55.8% to 61.5%

    58.7% (56%–62%)

OpenAI Google Alibaba Anthropic DeepSeek Thinking Machines Meta Xiaomi MiniMax Z.ai IBM OpenBMB Liquid AI Other KRAFTON eCloud Technology MachinePulse REAX PelaAI Poindexter Labs Chance, 20% (five options)

Each bar is a leaderboard row on Logic with its 95% interval: the chat models and the highest decision models. A chat model writes a reply that ends in a letter; a decision model scores the five options and picks one.

Logic and Math, setup by setup

10%10%30%30%50%50%70%70%90%90%y = x: the same accuracy on bothLogic accuracyMath accuracy

GPT-6 Astra (chat): Logic 96.7% (95.5%–97.6%), Math 99.6% (99.3%–99.8%)

Gemini 4 Carbon (chat): Logic 96.7% (95.5%–97.6%), Math 99.6% (99.2%–99.8%)

GPT-6.1 Sol (chat): Logic 96.5% (95.3%–97.5%), Math 99.6% (99.3%–99.8%)

Qwen4-Max (high · chat): Logic 96.3% (95.0%–97.2%), Math 99.8% (99.5%–99.9%)

Claude Opus 5.5 (chat): Logic 96.2% (94.9%–97.2%), Math 99.8% (99.5%–99.9%)

DeepSeek-V4.1-Flash (high · chat · 1017): Logic 96.2% (94.9%–97.2%), Math 99.6% (99.3%–99.8%)

Gemini 4 Hydrogen (chat): Logic 96.1% (94.8%–97.1%), Math 99.6% (99.2%–99.8%)

Claude Sonnet 5.5 (chat): Logic 95.5% (94.1%–96.5%), Math 99.7% (99.3%–99.8%)

Inkling 2 Small (high · chat): Logic 95.6% (94.3%–96.7%), Math 99.5% (99.2%–99.7%)

Claude Fable 5.1 (chat): Logic 95.2% (93.8%–96.3%), Math 99.7% (99.4%–99.9%)

Inkling-Small (high · chat · Jul 2026): Logic 95.4% (94.0%–96.5%), Math 99.6% (99.3%–99.8%)

Muse Ember (max · chat): Logic 95.5% (94.1%–96.5%), Math 99.6% (99.2%–99.8%)

MiMo-V2.6-Flash (on · chat): Logic 95.1% (93.7%–96.2%), Math 99.4% (99.0%–99.7%)

Qwen3.8 Flash (off · chat): Logic 94.0% (92.4%–95.2%), Math 99.5% (99.1%–99.7%)

Qwen3.8-27B (low · chat): Logic 94.0% (92.5%–95.3%), Math 99.2% (98.7%–99.5%)

GPT-6.1 Terra (chat): Logic 94.0% (92.4%–95.2%), Math 99.2% (98.7%–99.5%)

Gemma 4 31B (chat): Logic 93.3% (91.7%–94.6%), Math 99.4% (99.0%–99.6%)

Qwen3.5-4B (on · chat): Logic 92.4% (90.7%–93.8%), Math 99.4% (99.0%–99.7%)

GLM-5.3-Flash (high · chat): Logic 93.2% (91.5%–94.5%), Math 98.8% (98.2%–99.2%)

MiniMax-M3.1-Flash (high · chat): Logic 94.0% (92.4%–95.2%), Math 98.2% (97.6%–98.7%)

Granite-4.2-3B (high · chat): Logic 91.6% (89.9%–93.1%), Math 99.2% (98.7%–99.5%)

MiniCPM5-2B (on · chat): Logic 88.7% (86.7%–90.4%), Math 99.3% (98.8%–99.5%)

Qwen3.5-4B (off · chat): Logic 90.1% (88.3%–91.7%), Math 98.5% (97.9%–98.9%)

Gemma 4 12B Unified (chat): Logic 87.7% (85.7%–89.5%), Math 98.8% (98.3%–99.2%)

LFM2.5-24B-A2B (on · chat): Logic 86.7% (84.6%–88.5%), Math 99.0% (98.5%–99.4%)

Gemma 4 E4B (on · chat · 32k cap · sampled): Logic 86.0% (83.9%–87.9%), Math 98.8% (98.2%–99.1%)

LFM3-1.7B (off · chat): Logic 85.5% (83.3%–87.4%), Math 98.7% (98.1%–99.1%)

LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell): Logic 85.0% (82.8%–86.9%), Math 99.0% (98.5%–99.4%)

LFM3-1.7B (on · chat): Logic 84.4% (82.2%–86.4%), Math 98.9% (98.4%–99.3%)

Gemma 4 E4B (on · chat · 32k cap): Logic 85.0% (82.8%–86.9%), Math 98.5% (97.9%–99.0%)

MiniCPM5-2B (off · chat): Logic 83.7% (81.5%–85.8%), Math 98.3% (97.7%–98.8%)

Gemma 4 E4B (chat): Logic 83.2% (80.9%–85.3%), Math 98.1% (97.4%–98.6%)

Gemma 4 E4B (on · chat): Logic 82.6% (80.3%–84.7%), Math 98.2% (97.6%–98.7%)

Gemma 4 E2B (on · chat · 32k cap · sampled): Logic 76.0% (73.4%–78.4%), Math 97.6% (96.9%–98.2%)

Gemma 4 E2B (on · chat · 32k cap): Logic 75.9% (73.3%–78.3%), Math 97.8% (97.1%–98.4%)

Gemma 4 E2B (on · chat): Logic 74.6% (72.0%–77.0%), Math 96.9% (96.1%–97.6%)

Gemma 4 E2B (chat): Logic 72.9% (70.2%–75.4%), Math 96.8% (95.9%–97.4%)

Raon-1 nano (on · chat): Logic 70.9% (68.2%–73.5%), Math 97.0% (96.2%–97.6%)

LFM2.5-2.6B (on · chat): Logic 75.4% (72.8%–77.8%), Math 94.0% (92.9%–94.9%)

Qwen3.5-2B (on · chat): Logic 70.3% (67.6%–72.9%), Math 94.7% (93.7%–95.6%)

Qwen3-14B (off · chat): Logic 65.3% (62.5%–68.1%), Math 95.8% (94.9%–96.6%)

Qwen3.5-2B (off · chat): Logic 64.8% (62.0%–67.5%), Math 94.6% (93.6%–95.5%)

LFM2.5-8B-A1B (on · chat): Logic 60.2% (57.3%–63.0%), Math 96.3% (95.4%–97.0%)

GPT-6 Luna (chat): Logic 58.0% (55.0%–60.8%), Math 93.7% (92.6%–94.6%)

Raon-1 nano (off · chat): Logic 65.1% (62.2%–67.8%), Math 89.6% (88.2%–90.8%)

Erk-14B (off · chat): Logic 54.2% (51.3%–57.1%), Math 90.8% (89.5%–91.9%)

Karpo Decision 31B: Logic 60.7% (57.8%–63.5%), Math 81.6% (79.9%–83.1%)

KnowLine-31B-Gen2: Logic 58.9% (56.0%–61.8%), Math 80.3% (78.5%–81.9%)

REAX 1.3: Logic 59.3% (56.4%–62.1%), Math 78.0% (76.2%–79.7%)

KnowLine-31B-Gen1.5: Logic 59.0% (56.1%–61.9%), Math 77.4% (75.6%–79.1%)

Copernicus-31B: Logic 58.7% (55.8%–61.5%), Math 77.3% (75.5%–79.1%)

Sieve 31B v3: Logic 57.9% (55.0%–60.7%), Math 77.1% (75.3%–78.8%)

Logits Sigma-1 Duo: Logic 57.8% (54.9%–60.6%), Math 75.2% (73.4%–77.0%)

d1-ultra: Logic 54.9% (52.0%–57.8%), Math 74.1% (72.2%–75.9%)

GLiDE 27B: Logic 58.4% (55.5%–61.2%), Math 72.0% (70.1%–73.8%)

Mammoth-1: Logic 54.5% (51.6%–57.4%), Math 74.0% (72.1%–75.8%)

Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell): Logic 54.8% (51.8%–57.6%), Math 72.5% (70.6%–74.3%)

d3-ultra: Logic 54.6% (51.7%–57.5%), Math 71.5% (69.6%–73.4%)

GLiDE (API): Logic 54.5% (51.6%–57.4%), Math 70.8% (68.8%–72.7%)

d3: Logic 54.5% (51.6%–57.4%), Math 70.4% (68.5%–72.3%)

Drex Max v1.1: Logic 54.8% (51.8%–57.6%), Math 70.1% (68.1%–72.0%)

GLiDE (v1.2): Logic 55.2% (52.3%–58.1%), Math 69.9% (67.9%–71.8%)

Drex Max: Logic 53.1% (50.1%–56.0%), Math 70.3% (68.4%–72.2%)

Perplexity Decider 27B v1.2 (API): Logic 53.3% (50.4%–56.2%), Math 69.8% (67.8%–71.7%)

Gemini Decision 1 (20261111): Logic 50.0% (47.1%–53.0%), Math 71.3% (69.4%–73.2%)

d1.2-ultra: Logic 51.6% (48.6%–54.5%), Math 69.9% (67.9%–71.8%)

Perplexity Decider 27B v1.2 (API · with web): Logic 52.5% (49.6%–55.4%), Math 69.3% (67.3%–71.2%)

Turion Jev 31B: Logic 52.3% (49.3%–55.2%), Math 69.0% (67.0%–70.9%)

SubQ Snap 1.1: Logic 52.4% (49.4%–55.3%), Math 68.6% (66.6%–70.5%)

celeris-1-decision: Logic 47.3% (44.4%–50.2%), Math 70.0% (68.0%–71.9%)

celeris-1.1-decision: Logic 47.3% (44.4%–50.2%), Math 69.9% (67.9%–71.8%)

KnowLine-12B-Gen1: Logic 50.6% (47.7%–53.5%), Math 66.3% (64.2%–68.2%)

d1.2: Logic 49.2% (46.2%–52.1%), Math 65.5% (63.5%–67.5%)

Jev 1.14: Logic 48.2% (45.3%–51.1%), Math 65.1% (63.1%–67.1%)

d1.1: Logic 46.0% (43.2%–49.0%), Math 65.2% (63.2%–67.2%)

Jev 1.13: Logic 46.3% (43.4%–49.2%), Math 64.1% (62.0%–66.1%)

Adjudex 9B v5: Logic 46.4% (43.5%–49.3%), Math 62.1% (60.0%–64.1%)

Elephant-1: Logic 46.4% (43.5%–49.3%), Math 61.3% (59.3%–63.4%)

Jeffan-9B-1211: Logic 48.2% (45.3%–51.1%), Math 59.8% (57.7%–61.9%)

KnowLine-9B-Gen2: Logic 43.9% (41.0%–46.8%), Math 60.1% (58.0%–62.1%)

GLiDE 9B: Logic 47.8% (44.9%–50.7%), Math 57.9% (55.8%–60.0%)

KnowLine-9B-Gen1: Logic 44.3% (41.4%–47.2%), Math 59.8% (57.7%–61.8%)

StartLux-Decision-v1.1-9B-Instant: Logic 46.6% (43.7%–49.5%), Math 58.4% (56.3%–60.5%)

Microsoft-Decision-1: Logic 44.4% (41.6%–47.4%), Math 59.4% (57.3%–61.4%)

Dexy-Decision-9B-V2: Logic 45.3% (42.4%–48.3%), Math 58.8% (56.7%–60.9%)

Drex 2.0: Logic 41.9% (39.0%–44.8%), Math 59.0% (56.9%–61.0%)

vera-medium-omni: Logic 44.1% (41.2%–47.0%), Math 56.8% (54.7%–58.9%)

StartLux-Decision-v1.1-4B-Instant: Logic 45.9% (43.0%–48.8%), Math 55.8% (53.7%–57.9%)

Perplexity Decider Flash v1.3: Logic 44.7% (41.8%–47.6%), Math 55.9% (53.8%–58.0%)

Cerno 9B v1.2: Logic 43.3% (40.4%–46.2%), Math 56.0% (53.9%–58.1%)

Valen-Decision-1.5-9B: Logic 44.4% (41.5%–47.3%), Math 55.3% (53.2%–57.4%)

KnowLine 4B Gen5.5: Logic 42.3% (39.5%–45.2%), Math 56.0% (53.9%–58.1%)

Perplexity Decider Flash v1.2: Logic 43.1% (40.2%–46.0%), Math 55.2% (53.1%–57.3%)

Mercury Decide: Logic 45.1% (42.2%–48.0%), Math 53.5% (51.4%–55.6%)

d1: Logic 40.8% (38.0%–43.7%), Math 55.6% (53.5%–57.7%)

Open-Jev-27B-v1.1: Logic 42.6% (39.7%–45.5%), Math 54.6% (52.4%–56.6%)

OneJev-27B: Logic 40.6% (37.8%–43.5%), Math 53.8% (51.7%–55.9%)

Clef: Logic 40.8% (38.0%–43.7%), Math 51.1% (49.0%–53.2%)

Verdin-4B-v1.1: Logic 39.7% (36.9%–42.6%), Math 51.5% (49.4%–53.6%)

Chinese Decision Pointer 4B: Logic 40.4% (37.5%–43.3%), Math 50.4% (48.3%–52.5%)

Clef-Flash: Logic 38.9% (36.1%–41.8%), Math 49.8% (47.7%–51.9%)

Winnow-12B: Logic 39.7% (36.9%–42.6%), Math 46.9% (44.8%–49.0%)

Cygnet: Logic 39.3% (36.5%–42.2%), Math 44.3% (42.3%–46.4%)

Metask-Jev-4B: Logic 32.1% (29.4%–34.9%), Math 39.2% (37.1%–41.2%)

StartLux-Decision-v1.1-2B-Instant: Logic 30.8% (28.1%–33.5%), Math 38.7% (36.6%–40.7%)

GLiDE 2B: Logic 32.3% (29.6%–35.1%), Math 36.8% (34.7%–38.8%)

d1-2B: Logic 30.7% (28.0%–33.4%), Math 37.1% (35.1%–39.1%)

KnowLine-2B-Gen1: Logic 31.7% (29.1%–34.5%), Math 35.3% (33.3%–37.3%)

GLiDE 0.8B: Logic 29.2% (26.6%–31.9%), Math 32.1% (30.2%–34.1%)

StartLux-Decision-v1.1-0.8B-Instant: Logic 24.7% (22.3%–27.3%), Math 29.7% (27.8%–31.6%)

OneJev1.6-0.8B: Logic 27.0% (24.5%–29.7%), Math 28.2% (26.4%–30.2%)

KnowLine-0.8B-Gen1: Logic 26.7% (24.2%–29.3%), Math 28.2% (26.3%–30.1%)

Strands Decider 2B: Logic 22.4% (20.1%–24.9%), Math 28.7% (26.8%–30.6%)

GLiClass v4 Edge: Logic 23.6% (21.2%–26.1%), Math 22.7% (20.9%–24.5%)

GLiDE 0.4B: Logic 23.2% (20.8%–25.8%), Math 20.6% (18.9%–22.3%)

Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): Logic 53.2% (50.3%–56.1%), Math 70.3% (68.4%–72.2%)

OpenAI Google Alibaba Anthropic DeepSeek Thinking Machines Meta Xiaomi Z.ai MiniMax IBM OpenBMB Liquid AI Other KRAFTON eCloud Technology MachinePulse PelaAI REAX Poindexter Labs Sthānika AI Logits Fastino Base Labs Perplexity vLLM Semantic Router Nace.AI Turion.AI Subquadratic Celeris TypeSafe newsbang tattrongvu StartLux Microsoft Patrick-18 scar-ai Apertix AI Valen-Team Inception Zefan Cai OmniJev Cloudflare victorbennett quincyqiang EldanRing Blockbrain Labs Metask Lab Strands Agents

One dot per setup that has finished both benchmarks (116 setups, chat models included): across, its Logic accuracy; up, its Math accuracy; the whiskers are 95% intervals. The count is of setups, not models (Perplexity Decider 27B v1.2's two setups count twice). Above the dashed line a setup does better on Math.

Numerical reasoning is the decision models' lowest-scoring logic topic: median 34.0%, against verbal reasoning 50.5%, formal logic 79.5%

  1. Numerical reasoning

    Numerical reasoning: median decision setup 34.0%, median chat setup 94.8%

    34.0%94.8%
  2. Verbal reasoning

    Verbal reasoning: median decision setup 50.5%, median chat setup 81.0%

    50.5%81.0%
  3. Formal logic

    Formal logic: median decision setup 79.5%, median chat setup 92.4%

    79.5%92.4%

Median of the 69 decision setups Median of the 47 chat setups

The median accuracy of the finished setups on each logic topic, decision models and chat models apart; it is the lowest topic for 67 of the 69 decision setups. Medians over finished runs only.

How the scores work

More in Method →