Answer Check

Answer Check

2,069 pairs of a machine-written English summary, a question and its reference answer · 3 labels · Score 0–100, chance = 0

Decision

Models
Class
Measure
(accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.Expected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.Of the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.Client wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.Input tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.Measured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.

Score higher is better

View
  1. Karpo Decision 31B90.0
  2. Copernicus-31B89.0
  3. KnowLine-31B-Gen1.588.8
  4. Sieve 31B v388.1
  5. KnowLine-31B-Gen287.7
  6. REAX 1.387.0
  7. d1.2-ultra86.4
  8. Turion Jev 31B86.4
  9. Inkling 2 Small thinking on (no off mode) · chat86.2
  10. Perplexity Decider 27B v1.386.2
  11. GLiDE 27B86.1
  12. Logits Sigma-1 Duo85.1
  13. GLiDE API85.1
  14. GLiDE v1.284.9
  15. SubQ Snap 1.184.9
  16. d1-ultra84.7
  17. Microsoft-Decision-183.9
  18. Gemini 4 Hydrogen thinking on (no off mode) · chat83.7
  19. vera-medium-omni83.7
  20. Jev 1.1483.6
  21. Jeffan-9B-121183.0
  22. Jev 1.1383.0
  23. d1.182.5
  24. GLiDE 9B82.2
  25. Cerno 9B v1.281.9
  26. StartLux-Decision-v1.1-4B-Instant81.7
  27. Dexy-Decision-9B-V281.1
  28. KnowLine-12B-Gen180.9
  29. KnowLine 4B Gen5.580.6
  30. Adjudex 9B v580.4
  31. MiniCPM5-2B thinking on · chat0.1% invalid, mean of 3 seeds79.2
  32. d179.2
  33. Perplexity Decider Flash v1.378.8
  34. StartLux-Decision-v1.1-9B-Instant78.8
  35. Chinese Decision Pointer 4B78.1
  36. Gemma 4 E4B thinking on · chat2 invalid77.2
  37. Perplexity Decider Flash v1.277.2
  38. Gemini Decision 1 2026111177.0
  39. Gemma 4 12B Unified thinking off · chat77.0
  40. Elephant-176.9
  41. Winnow-12B76.6
  42. KnowLine-9B-Gen176.1
  43. Granite-4.2-3B thinking on · chat75.9
  44. LFM3-1.7B thinking on · chat0.1% invalid, mean of 3 seeds75.8
  45. Verdin-4B-v1.175.4
  46. KnowLine-9B-Gen275.3
  47. Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat1.5% invalid, mean of 3 seeds75.1
  48. LFM3-1.7B thinking off · chat0.1% invalid, mean of 3 seeds74.9
  49. Valen-Decision-1.5-9B74.3
  50. Cygnet72.9
  51. Gemma 4 E2B thinking on · chat71.1
  52. Qwen3.5-2B thinking on · chat78 invalid70.3
  53. Clef-Flash69.8
  54. Gemma 4 12B Unified thinking on · chat296 invalid67.7
  55. MiniCPM5-2B thinking off · chat0.2% invalid, mean of 3 seeds67.0
  56. Qwen3.5-4B thinking on · chat179 invalid66.2
  57. StartLux-Decision-v1.1-2B-Instant66.0
  58. GLiDE 2B65.6
  59. Qwen3.5-4B thinking off · chat1.0% invalid, mean of 3 seeds64.6
  60. Metask-Jev-4B61.7
  61. KnowLine-2B-Gen161.1
  62. GLiDE 0.8B60.8
  63. Strands Decider 2B60.3
  64. LFM2.5-8B-A1B thinking on (no off mode) · chat57.7
  65. Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat0.03% invalid, mean of 3 seeds55.2
  66. Gemma 4 E2B thinking off · chat54.8
  67. Qwen3.5-2B thinking off · chat2.0% invalid, mean of 3 seeds50.2
  68. OneJev1.6-0.8B50.0
  69. d1-2B49.6
  70. StartLux-Decision-v1.1-0.8B-Instant48.9
  71. GLiDE 0.4B44.4
  72. GLiClass v4 Edge44.3
  73. Decider 143.3
  74. Raon-1 nano thinking off · chat0.03% invalid, mean of 3 seeds40.6
  75. KnowLine-0.8B-Gen138.5

No row of this kind yet.

  • MachinePulse
  • Poindexter Labs
  • PelaAI
  • Sthānika AI
  • REAX
  • Liquid AI
  • Turion.AI
  • Thinking Machines
  • Perplexity
  • Fastino
  • Logits
  • Subquadratic
  • Microsoft
  • Google
  • scar-ai
  • TypeSafe
  • tattrongvu
  • Apertix AI
  • StartLux
  • Patrick-18
  • newsbang
  • OpenBMB
  • quincyqiang
  • Base Labs
  • EldanRing
  • IBM
  • victorbennett
  • Valen-Team
  • Blockbrain Labs
  • Alibaba
  • Cloudflare
  • Metask Lab
  • Strands Agents
  • OmniJev
  • Knowledgator
  • meraGPT
  • KRAFTON
The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –.Model(accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairsExpected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, pointsOf the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarmsClient wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under itInput tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairsMeasured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs
1Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details90.093.3%2.880.8%1.7% false alarms0.22 sp90 0.31 s1 at a timeOur GPU230 sRTX PRO 6000
2Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details89.092.7%11.081.5%2.5% false alarms0.52 sp90 0.83 s1 at a timeOur GPU577 sRTX PRO 6000
3KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails88.892.6%5.783.6%3.0% false alarms2.54 sp90 3.88 s1 at a timeOur GPU2,613 sDGX Spark
4Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails88.192.1%7.079.3%2.1% false alarms2.67 sp90 3.93 s1 at a timeOur GPU2,824 sDGX Spark
5KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details87.791.8%7.286.3%3.6% false alarms0.39 sp90 0.61 s1 at a timeOur GPU449 sRTX PRO 6000
6REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details87.091.3%4.867.5%1.2% false alarms0.68 sp90 1.05 s1 at a timeOur GPU745 sRTX PRO 6000
7d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details86.491.0%8.885.4%4.4% false alarms0.23 sp90 0.35 s8 at a time$0.035API
8Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails86.491.0%6.766.3%1.7% false alarms1.12 sp90 1.96 s1 at a timeOur GPU1,329 sDGX Spark
Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details86.290.8%No probabilities73.6%2.4% false alarms2.70 sp90 16.4 s1 at a time$1.49API
9Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details86.290.8%2.882.7%3.5% false alarms0.22 sp90 0.34 s1 at a timeOur GPU245 sRTX PRO 6000
10GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails86.190.7%3.674.8%1.5% false alarms1.03 sp90 1.95 s1 at a timeOur GPU1,205 sDGX Spark
11Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details85.190.1%7.063.8%0.9% false alarms0.64 sp90 1.01 s1 at a timeOur GPU723 sRTX PRO 6000
12GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails85.190.0%3.068.7%1.7% false alarms0.30 sp90 0.56 s8 at a timeNot verifiedAPI
13GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details84.989.9%4.769.3%1.8% false alarms0.23 sp90 0.36 s1 at a timeOur GPU254 sRTX PRO 6000
14SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails84.989.9%2.378.1%4.1% false alarms0.39 sp90 0.41 s8 at a timeNot verifiedAPI
15d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details84.789.8%11.480.8%4.5% false alarms0.48 sp90 0.90 s8 at a time$0.035API
16Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details83.989.3%2.482.4%6.5% false alarms0.32 sp90 0.38 s8 at a time$0.036API
Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails83.789.1%No probabilities68.7%2.2% false alarms6.37 sp90 19.2 s1 at a timeNot verifiedAPI
17vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details83.789.1%12.476.0%5.7% false alarms0.06 sp90 0.06 s1 at a timeOur GPU60 sRTX PRO 6000
18Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details83.689.1%2.872.0%3.5% false alarms0.18 sp90 0.20 s8 at a time$0.049API
19Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details83.088.6%3.359.9%2.5% false alarms0.09 sp90 0.14 s1 at a timeOur GPU100 sRTX PRO 6000
20Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details83.088.6%1.979.9%6.0% false alarms0.17 sp90 0.19 s8 at a time$0.049API
21d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details82.588.4%8.770.5%4.1% false alarms0.25 sp90 0.45 s8 at a time$0.017API
22GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details82.288.2%7.269.9%3.8% false alarms0.08 sp90 0.11 s1 at a timeOur GPU90 sRTX PRO 6000
23Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details81.987.9%2.965.3%3.0% false alarms0.11 sp90 0.18 s1 at a timeOur GPU126 sRTX PRO 6000
24StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails81.787.8%8.389.4%8.3% false alarms0.12 sp90 0.19 s1 at a timeOur GPU140 sDGX Spark
25Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details81.187.4%2.564.7%3.9% false alarms0.06 sp90 0.09 s1 at a timeOur GPU72 sRTX PRO 6000
26KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details80.987.2%6.861.1%3.6% false alarms0.16 sp90 0.25 s1 at a timeOur GPU195 sRTX PRO 6000
27KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details80.687.1%4.562.0%3.4% false alarms0.04 sp90 0.06 s1 at a timeOur GPU49 sRTX PRO 6000
28Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details80.487.0%4.965.0%4.9% false alarms0.07 sp90 0.09 s1 at a timeOur GPU71 sRTX PRO 6000
MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details79.279.0–79.6 · min–max of 3 seeds86.2%86.0%–86.4%min–max of 3 seedsNo probabilities77.4%6.5% false alarmsmean of 3 seeds5.00 sp90 8.72 s30 at a time3 seeds pooledOur GPU197 sRTX PRO 6000mean of 3 seeds
29d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails79.286.1%3.166.3%5.3% false alarms0.27 sp90 0.60 s8 at a timeNot verifiedAPI
30Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details78.885.9%4.267.5%5.6% false alarms0.06 sp90 0.08 s1 at a timeOur GPU61 sRTX PRO 6000
31StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails78.885.9%8.288.8%9.5% false alarms0.23 sp90 0.36 s1 at a timeOur GPU268 sDGX Spark
32Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details78.185.4%6.769.0%6.0% false alarms0.07 sp90 0.11 s1 at a timeOur GPU77 sRTX PRO 6000
Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details77.284.8%No probabilities67.5%4.5% false alarms9.38 sp90 11.9 s30 at a timeOur GPU326 sRTX PRO 6000
33Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details77.284.8%2.847.7%4.0% false alarms0.06 sp90 0.09 s1 at a timeOur GPU65 sRTX PRO 6000
34Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details77.084.7%3.951.1%1.8% false alarms0.20 sp90 0.29 s8 at a time$0.093API
Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details77.084.7%No probabilities52.0%2.6% false alarms5.12 sp90 15.7 s64 at a timeOur GPU121 sRTX PRO 6000
35Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details76.984.6%3.741.0%1.9% false alarms0.09 sp90 0.11 s1 at a timeOur GPU98 sRTX PRO 6000
36Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details76.684.4%8.351.7%3.8% false alarms0.12 sp90 0.18 s1 at a timeOur GPU132 sRTX PRO 6000
37KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details76.184.1%3.650.8%3.8% false alarms0.10 sp90 0.14 s1 at a timeOur GPU105 sRTX PRO 6000
Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details75.983.9%No probabilities79.0%9.0% false alarms19.6 sp90 64.5 s30 at a timeOur GPU1,050 sRTX PRO 6000
LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details75.874.9–76.5 · min–max of 3 seeds83.9%83.3%–84.3%min–max of 3 seedsNo probabilities63.3%4.1% false alarmsmean of 3 seeds5.57 sp90 16.4 s30 at a time3 seeds pooledOur GPU268 sRTX PRO 6000mean of 3 seeds
38Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details75.483.6%3.446.8%3.7% false alarms0.05 sp90 0.07 s1 at a timeOur GPU54 sRTX PRO 6000
39KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details75.383.5%4.535.9%1.4% false alarms0.10 sp90 0.16 s1 at a timeOur GPU128 sRTX PRO 6000
Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details75.174.1–75.9 · min–max of 3 seeds83.4%82.7%–84.0%min–max of 3 seedsNo probabilities64.3%4.3% false alarmsmean of 3 seeds6.13 sp90 9.70 s30 at a time3 seeds pooledOur GPU234 sRTX PRO 6000mean of 3 seeds
LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details74.973.5–75.6 · min–max of 3 seeds83.3%82.3%–83.8%min–max of 3 seedsNo probabilities62.6%3.9% false alarmsmean of 3 seeds7.62 sp90 22.3 s64 at a time3 seeds pooledOur GPU180 sRTX PRO 6000mean of 3 seeds
40Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details74.382.8%3.239.8%2.8% false alarms0.07 sp90 0.10 s1 at a timeOur GPU78 sRTX PRO 6000
41Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details72.981.9%6.529.2%2.1% false alarms0.09 sp90 0.13 s1 at a timeOur GPU93 sRTX PRO 6000
Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details71.180.8%No probabilities49.5%5.1% false alarms4.02 sp90 5.85 s64 at a timeOur GPU69 sRTX PRO 6000
Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details70.380.2%No probabilities55.3%4.3% false alarms36.8 sp90 66.9 s64 at a timeOur GPU636 sRTX PRO 6000
42Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details69.879.8%5.937.7%3.9% false alarms0.15 sp90 0.18 s1 at a timeOur GPU156 sRTX PRO 6000
Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details67.778.4%No probabilities38.0%0.7% false alarms32.6 sp90 1640.2 s64 at a timeOur GPU4,327 sRTX PRO 6000
MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details67.066.9–67.0 · min–max of 3 seeds78.0%77.96%–78.01%min–max of 3 seedsNo probabilities28.8%2.7% false alarmsmean of 3 seeds0.86 sp90 3.48 s64 at a time3 seeds pooledOur GPU51 sRTX PRO 6000mean of 3 seeds
Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details66.277.5%No probabilities73.2%6.2% false alarms37.8 sp90 96.1 s64 at a timeOur GPU741 sRTX PRO 6000
43StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails66.077.3%7.267.5%14.0% false alarms0.06 sp90 0.09 s1 at a timeOur GPU78 sDGX Spark
44GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details65.677.1%4.060.8%10.8% false alarms0.04 sp90 0.04 s1 at a timeOur GPU44 sRTX PRO 6000
Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details64.664.5–64.8 · min–max of 3 seeds76.4%76.3%–76.6%min–max of 3 seedsNo probabilities61.2%7.6% false alarmsmean of 3 seeds3.49 sp90 20.4 s64 at a time3 seeds pooledOur GPU148 sRTX PRO 6000mean of 3 seeds
45Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details61.774.5%6.823.1%3.4% false alarms0.12 sp90 0.19 s1 at a timeOur GPU134 sRTX PRO 6000
46KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails61.174.0%2.716.4%2.7% false alarms0.11 sp90 0.17 s1 at a timeOur GPU130 sDGX Spark
47GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails60.873.9%3.831.3%6.6% false alarms0.05 sp90 0.07 s1 at a timeOur GPU63 sDGX Spark
48Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details60.373.6%5.04.6%0.8% false alarms0.04 sp90 0.05 s1 at a timeOur GPU43 sRTX PRO 6000
LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details57.771.8%No probabilities8.8%0.8% false alarms9.58 sp90 15.2 s30 at a timeOur GPU352 sRTX PRO 6000
Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details55.254.5–55.8 · min–max of 3 seeds70.1%69.7%–70.6%min–max of 3 seedsNo probabilities7.7%1.6% false alarmsmean of 3 seeds0.18 sp90 5.42 s64 at a time3 seeds pooledOur GPU33 sRTX PRO 6000mean of 3 seeds
Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details54.869.8%No probabilities6.1%1.3% false alarms0.19 sp90 3.31 s64 at a timeOur GPU18 sRTX PRO 6000
Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details50.249.3–50.8 · min–max of 3 seeds66.8%66.2%–67.2%min–max of 3 seedsNo probabilities34.4%7.8% false alarmsmean of 3 seeds1.53 sp90 13.7 s64 at a time3 seeds pooledOur GPU92 sRTX PRO 6000mean of 3 seeds
49OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails50.066.7%4.133.4%14.4% false alarms0.03 sp90 0.04 s1 at a timeOur GPU43 sDGX Spark
50d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details49.666.4%7.285.7%32.2% false alarms0.02 sp90 0.03 s1 at a timeOur GPU25 sRTX PRO 6000
51StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails48.965.9%5.672.3%26.5% false alarms0.04 sp90 0.06 s1 at a timeOur GPU53 sDGX Spark
52GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails44.462.9%7.731.0%16.8% false alarms0.03 sp90 0.04 s1 at a timeOur GPU41 sDGX Spark
53GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails44.362.9%9.834.9%10.6% false alarms0.07 sp90 0.18 s1 at a timeOur GPU128 sDGX Spark
54Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details43.362.2%5.931.9%17.6% false alarms0.65 sp90 0.75 s8 at a time$0.035API
Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details40.640.6–40.8 · min–max of 3 seeds60.4%60.4%–60.5%min–max of 3 seedsNo probabilities19.9%2.0% false alarmsmean of 3 seeds3.03 sp90 7.76 s30 at a time3 seeds pooledOur GPU209 sRTX PRO 6000mean of 3 seeds
55KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails38.559.0%14.610.0%3.6% false alarms0.07 sp90 0.10 s1 at a timeOur GPU81 sDGX Spark

No row of this kind yet.

2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology

Accuracy higher is better

View
  1. Karpo Decision 31B93.3%
  2. Copernicus-31B92.7%
  3. KnowLine-31B-Gen1.592.6%
  4. Sieve 31B v392.1%
  5. KnowLine-31B-Gen291.8%
  6. REAX 1.391.3%
  7. d1.2-ultra91.0%
  8. Turion Jev 31B91.0%
  9. Inkling 2 Small thinking on (no off mode) · chat90.8%
  10. Perplexity Decider 27B v1.390.8%
  11. GLiDE 27B90.7%
  12. Logits Sigma-1 Duo90.1%
  13. GLiDE API90.0%
  14. GLiDE v1.289.9%
  15. SubQ Snap 1.189.9%
  16. d1-ultra89.8%
  17. Microsoft-Decision-189.3%
  18. Gemini 4 Hydrogen thinking on (no off mode) · chat89.1%
  19. vera-medium-omni89.1%
  20. Jev 1.1489.1%
  21. Jeffan-9B-121188.6%
  22. Jev 1.1388.6%
  23. d1.188.4%
  24. GLiDE 9B88.2%
  25. Cerno 9B v1.287.9%
  26. StartLux-Decision-v1.1-4B-Instant87.8%
  27. Dexy-Decision-9B-V287.4%
  28. KnowLine-12B-Gen187.2%
  29. KnowLine 4B Gen5.587.1%
  30. Adjudex 9B v587.0%
  31. MiniCPM5-2B thinking on · chat0.1% invalid, mean of 3 seeds86.2%
  32. d186.1%
  33. Perplexity Decider Flash v1.385.9%
  34. StartLux-Decision-v1.1-9B-Instant85.9%
  35. Chinese Decision Pointer 4B85.4%
  36. Gemma 4 E4B thinking on · chat2 invalid84.8%
  37. Perplexity Decider Flash v1.284.8%
  38. Gemini Decision 1 2026111184.7%
  39. Gemma 4 12B Unified thinking off · chat84.7%
  40. Elephant-184.6%
  41. Winnow-12B84.4%
  42. KnowLine-9B-Gen184.1%
  43. Granite-4.2-3B thinking on · chat83.9%
  44. LFM3-1.7B thinking on · chat0.1% invalid, mean of 3 seeds83.9%
  45. Verdin-4B-v1.183.6%
  46. KnowLine-9B-Gen283.5%
  47. Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat1.5% invalid, mean of 3 seeds83.4%
  48. LFM3-1.7B thinking off · chat0.1% invalid, mean of 3 seeds83.3%
  49. Valen-Decision-1.5-9B82.8%
  50. Cygnet81.9%
  51. Gemma 4 E2B thinking on · chat80.8%
  52. Qwen3.5-2B thinking on · chat78 invalid80.2%
  53. Clef-Flash79.8%
  54. Gemma 4 12B Unified thinking on · chat296 invalid78.4%
  55. MiniCPM5-2B thinking off · chat0.2% invalid, mean of 3 seeds78.0%
  56. Qwen3.5-4B thinking on · chat179 invalid77.5%
  57. StartLux-Decision-v1.1-2B-Instant77.3%
  58. GLiDE 2B77.1%
  59. Qwen3.5-4B thinking off · chat1.0% invalid, mean of 3 seeds76.4%
  60. Metask-Jev-4B74.5%
  61. KnowLine-2B-Gen174.0%
  62. GLiDE 0.8B73.9%
  63. Strands Decider 2B73.6%
  64. LFM2.5-8B-A1B thinking on (no off mode) · chat71.8%
  65. Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat0.03% invalid, mean of 3 seeds70.1%
  66. Gemma 4 E2B thinking off · chat69.8%
  67. Qwen3.5-2B thinking off · chat2.0% invalid, mean of 3 seeds66.8%
  68. OneJev1.6-0.8B66.7%
  69. d1-2B66.4%
  70. StartLux-Decision-v1.1-0.8B-Instant65.9%
  71. GLiDE 0.4B62.9%
  72. GLiClass v4 Edge62.9%
  73. Decider 162.2%
  74. Raon-1 nano thinking off · chat0.03% invalid, mean of 3 seeds60.4%
  75. KnowLine-0.8B-Gen159.0%

No row of this kind yet.

  • MachinePulse
  • Poindexter Labs
  • PelaAI
  • Sthānika AI
  • REAX
  • Liquid AI
  • Turion.AI
  • Thinking Machines
  • Perplexity
  • Fastino
  • Logits
  • Subquadratic
  • Microsoft
  • Google
  • scar-ai
  • TypeSafe
  • tattrongvu
  • Apertix AI
  • StartLux
  • Patrick-18
  • newsbang
  • OpenBMB
  • quincyqiang
  • Base Labs
  • EldanRing
  • IBM
  • victorbennett
  • Valen-Team
  • Blockbrain Labs
  • Alibaba
  • Cloudflare
  • Metask Lab
  • Strands Agents
  • OmniJev
  • Knowledgator
  • meraGPT
  • KRAFTON
The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –.ModelCorrect recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairs(accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100Expected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, pointsOf the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarmsClient wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under itInput tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairsMeasured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs
1Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details93.3%90.02.880.8%1.7% false alarms0.22 sp90 0.31 s1 at a timeOur GPU230 sRTX PRO 6000
2Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details92.7%89.011.081.5%2.5% false alarms0.52 sp90 0.83 s1 at a timeOur GPU577 sRTX PRO 6000
3KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails92.6%88.85.783.6%3.0% false alarms2.54 sp90 3.88 s1 at a timeOur GPU2,613 sDGX Spark
4Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails92.1%88.17.079.3%2.1% false alarms2.67 sp90 3.93 s1 at a timeOur GPU2,824 sDGX Spark
5KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details91.8%87.77.286.3%3.6% false alarms0.39 sp90 0.61 s1 at a timeOur GPU449 sRTX PRO 6000
6REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details91.3%87.04.867.5%1.2% false alarms0.68 sp90 1.05 s1 at a timeOur GPU745 sRTX PRO 6000
7d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details91.0%86.48.885.4%4.4% false alarms0.23 sp90 0.35 s8 at a time$0.035API
8Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails91.0%86.46.766.3%1.7% false alarms1.12 sp90 1.96 s1 at a timeOur GPU1,329 sDGX Spark
Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details90.8%86.2No probabilities73.6%2.4% false alarms2.70 sp90 16.4 s1 at a time$1.49API
9Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details90.8%86.22.882.7%3.5% false alarms0.22 sp90 0.34 s1 at a timeOur GPU245 sRTX PRO 6000
10GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails90.7%86.13.674.8%1.5% false alarms1.03 sp90 1.95 s1 at a timeOur GPU1,205 sDGX Spark
11Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details90.1%85.17.063.8%0.9% false alarms0.64 sp90 1.01 s1 at a timeOur GPU723 sRTX PRO 6000
12GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails90.0%85.13.068.7%1.7% false alarms0.30 sp90 0.56 s8 at a timeNot verifiedAPI
13GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details89.9%84.94.769.3%1.8% false alarms0.23 sp90 0.36 s1 at a timeOur GPU254 sRTX PRO 6000
14SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails89.9%84.92.378.1%4.1% false alarms0.39 sp90 0.41 s8 at a timeNot verifiedAPI
15d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details89.8%84.711.480.8%4.5% false alarms0.48 sp90 0.90 s8 at a time$0.035API
16Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details89.3%83.92.482.4%6.5% false alarms0.32 sp90 0.38 s8 at a time$0.036API
Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails89.1%83.7No probabilities68.7%2.2% false alarms6.37 sp90 19.2 s1 at a timeNot verifiedAPI
17vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details89.1%83.712.476.0%5.7% false alarms0.06 sp90 0.06 s1 at a timeOur GPU60 sRTX PRO 6000
18Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details89.1%83.62.872.0%3.5% false alarms0.18 sp90 0.20 s8 at a time$0.049API
19Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details88.6%83.03.359.9%2.5% false alarms0.09 sp90 0.14 s1 at a timeOur GPU100 sRTX PRO 6000
20Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details88.6%83.01.979.9%6.0% false alarms0.17 sp90 0.19 s8 at a time$0.049API
21d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details88.4%82.58.770.5%4.1% false alarms0.25 sp90 0.45 s8 at a time$0.017API
22GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details88.2%82.27.269.9%3.8% false alarms0.08 sp90 0.11 s1 at a timeOur GPU90 sRTX PRO 6000
23Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details87.9%81.92.965.3%3.0% false alarms0.11 sp90 0.18 s1 at a timeOur GPU126 sRTX PRO 6000
24StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails87.8%81.78.389.4%8.3% false alarms0.12 sp90 0.19 s1 at a timeOur GPU140 sDGX Spark
25Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details87.4%81.12.564.7%3.9% false alarms0.06 sp90 0.09 s1 at a timeOur GPU72 sRTX PRO 6000
26KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details87.2%80.96.861.1%3.6% false alarms0.16 sp90 0.25 s1 at a timeOur GPU195 sRTX PRO 6000
27KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details87.1%80.64.562.0%3.4% false alarms0.04 sp90 0.06 s1 at a timeOur GPU49 sRTX PRO 6000
28Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details87.0%80.44.965.0%4.9% false alarms0.07 sp90 0.09 s1 at a timeOur GPU71 sRTX PRO 6000
MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details86.2%86.0%–86.4% · min–max of 3 seeds79.279.0–79.6min–max of 3 seedsNo probabilities77.4%6.5% false alarmsmean of 3 seeds5.00 sp90 8.72 s30 at a time3 seeds pooledOur GPU197 sRTX PRO 6000mean of 3 seeds
29d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails86.1%79.23.166.3%5.3% false alarms0.27 sp90 0.60 s8 at a timeNot verifiedAPI
30Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details85.9%78.84.267.5%5.6% false alarms0.06 sp90 0.08 s1 at a timeOur GPU61 sRTX PRO 6000
31StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails85.9%78.88.288.8%9.5% false alarms0.23 sp90 0.36 s1 at a timeOur GPU268 sDGX Spark
32Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details85.4%78.16.769.0%6.0% false alarms0.07 sp90 0.11 s1 at a timeOur GPU77 sRTX PRO 6000
Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details84.8%77.2No probabilities67.5%4.5% false alarms9.38 sp90 11.9 s30 at a timeOur GPU326 sRTX PRO 6000
33Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details84.8%77.22.847.7%4.0% false alarms0.06 sp90 0.09 s1 at a timeOur GPU65 sRTX PRO 6000
34Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details84.7%77.03.951.1%1.8% false alarms0.20 sp90 0.29 s8 at a time$0.093API
Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details84.7%77.0No probabilities52.0%2.6% false alarms5.12 sp90 15.7 s64 at a timeOur GPU121 sRTX PRO 6000
35Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details84.6%76.93.741.0%1.9% false alarms0.09 sp90 0.11 s1 at a timeOur GPU98 sRTX PRO 6000
36Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details84.4%76.68.351.7%3.8% false alarms0.12 sp90 0.18 s1 at a timeOur GPU132 sRTX PRO 6000
37KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details84.1%76.13.650.8%3.8% false alarms0.10 sp90 0.14 s1 at a timeOur GPU105 sRTX PRO 6000
Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details83.9%75.9No probabilities79.0%9.0% false alarms19.6 sp90 64.5 s30 at a timeOur GPU1,050 sRTX PRO 6000
LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details83.9%83.3%–84.3% · min–max of 3 seeds75.874.9–76.5min–max of 3 seedsNo probabilities63.3%4.1% false alarmsmean of 3 seeds5.57 sp90 16.4 s30 at a time3 seeds pooledOur GPU268 sRTX PRO 6000mean of 3 seeds
38Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details83.6%75.43.446.8%3.7% false alarms0.05 sp90 0.07 s1 at a timeOur GPU54 sRTX PRO 6000
39KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details83.5%75.34.535.9%1.4% false alarms0.10 sp90 0.16 s1 at a timeOur GPU128 sRTX PRO 6000
Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details83.4%82.7%–84.0% · min–max of 3 seeds75.174.1–75.9min–max of 3 seedsNo probabilities64.3%4.3% false alarmsmean of 3 seeds6.13 sp90 9.70 s30 at a time3 seeds pooledOur GPU234 sRTX PRO 6000mean of 3 seeds
LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details83.3%82.3%–83.8% · min–max of 3 seeds74.973.5–75.6min–max of 3 seedsNo probabilities62.6%3.9% false alarmsmean of 3 seeds7.62 sp90 22.3 s64 at a time3 seeds pooledOur GPU180 sRTX PRO 6000mean of 3 seeds
40Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details82.8%74.33.239.8%2.8% false alarms0.07 sp90 0.10 s1 at a timeOur GPU78 sRTX PRO 6000
41Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details81.9%72.96.529.2%2.1% false alarms0.09 sp90 0.13 s1 at a timeOur GPU93 sRTX PRO 6000
Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details80.8%71.1No probabilities49.5%5.1% false alarms4.02 sp90 5.85 s64 at a timeOur GPU69 sRTX PRO 6000
Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details80.2%70.3No probabilities55.3%4.3% false alarms36.8 sp90 66.9 s64 at a timeOur GPU636 sRTX PRO 6000
42Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details79.8%69.85.937.7%3.9% false alarms0.15 sp90 0.18 s1 at a timeOur GPU156 sRTX PRO 6000
Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details78.4%67.7No probabilities38.0%0.7% false alarms32.6 sp90 1640.2 s64 at a timeOur GPU4,327 sRTX PRO 6000
MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details78.0%77.96%–78.01% · min–max of 3 seeds67.066.9–67.0min–max of 3 seedsNo probabilities28.8%2.7% false alarmsmean of 3 seeds0.86 sp90 3.48 s64 at a time3 seeds pooledOur GPU51 sRTX PRO 6000mean of 3 seeds
Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details77.5%66.2No probabilities73.2%6.2% false alarms37.8 sp90 96.1 s64 at a timeOur GPU741 sRTX PRO 6000
43StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails77.3%66.07.267.5%14.0% false alarms0.06 sp90 0.09 s1 at a timeOur GPU78 sDGX Spark
44GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details77.1%65.64.060.8%10.8% false alarms0.04 sp90 0.04 s1 at a timeOur GPU44 sRTX PRO 6000
Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details76.4%76.3%–76.6% · min–max of 3 seeds64.664.5–64.8min–max of 3 seedsNo probabilities61.2%7.6% false alarmsmean of 3 seeds3.49 sp90 20.4 s64 at a time3 seeds pooledOur GPU148 sRTX PRO 6000mean of 3 seeds
45Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details74.5%61.76.823.1%3.4% false alarms0.12 sp90 0.19 s1 at a timeOur GPU134 sRTX PRO 6000
46KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails74.0%61.12.716.4%2.7% false alarms0.11 sp90 0.17 s1 at a timeOur GPU130 sDGX Spark
47GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails73.9%60.83.831.3%6.6% false alarms0.05 sp90 0.07 s1 at a timeOur GPU63 sDGX Spark
48Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details73.6%60.35.04.6%0.8% false alarms0.04 sp90 0.05 s1 at a timeOur GPU43 sRTX PRO 6000
LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details71.8%57.7No probabilities8.8%0.8% false alarms9.58 sp90 15.2 s30 at a timeOur GPU352 sRTX PRO 6000
Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details70.1%69.7%–70.6% · min–max of 3 seeds55.254.5–55.8min–max of 3 seedsNo probabilities7.7%1.6% false alarmsmean of 3 seeds0.18 sp90 5.42 s64 at a time3 seeds pooledOur GPU33 sRTX PRO 6000mean of 3 seeds
Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details69.8%54.8No probabilities6.1%1.3% false alarms0.19 sp90 3.31 s64 at a timeOur GPU18 sRTX PRO 6000
Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details66.8%66.2%–67.2% · min–max of 3 seeds50.249.3–50.8min–max of 3 seedsNo probabilities34.4%7.8% false alarmsmean of 3 seeds1.53 sp90 13.7 s64 at a time3 seeds pooledOur GPU92 sRTX PRO 6000mean of 3 seeds
49OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails66.7%50.04.133.4%14.4% false alarms0.03 sp90 0.04 s1 at a timeOur GPU43 sDGX Spark
50d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details66.4%49.67.285.7%32.2% false alarms0.02 sp90 0.03 s1 at a timeOur GPU25 sRTX PRO 6000
51StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails65.9%48.95.672.3%26.5% false alarms0.04 sp90 0.06 s1 at a timeOur GPU53 sDGX Spark
52GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails62.9%44.47.731.0%16.8% false alarms0.03 sp90 0.04 s1 at a timeOur GPU41 sDGX Spark
53GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails62.9%44.39.834.9%10.6% false alarms0.07 sp90 0.18 s1 at a timeOur GPU128 sDGX Spark
54Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details62.2%43.35.931.9%17.6% false alarms0.65 sp90 0.75 s8 at a time$0.035API
Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details60.4%60.4%–60.5% · min–max of 3 seeds40.640.6–40.8min–max of 3 seedsNo probabilities19.9%2.0% false alarmsmean of 3 seeds3.03 sp90 7.76 s30 at a time3 seeds pooledOur GPU209 sRTX PRO 6000mean of 3 seeds
55KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails59.0%38.514.610.0%3.6% false alarms0.07 sp90 0.10 s1 at a timeOur GPU81 sDGX Spark

No row of this kind yet.

2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology

Calibration lower is better

View
  1. Jev 1.131.9
  2. SubQ Snap 1.12.3
  3. Microsoft-Decision-12.4
  4. Dexy-Decision-9B-V22.5
  5. KnowLine-2B-Gen12.7
  6. Perplexity Decider 27B v1.32.8
  7. Perplexity Decider Flash v1.22.8
  8. Karpo Decision 31B2.8
  9. Jev 1.142.8
  10. Cerno 9B v1.22.9
  11. GLiDE API3.0
  12. d13.1
  13. Valen-Decision-1.5-9B3.2
  14. Jeffan-9B-12113.3
  15. Verdin-4B-v1.13.4
  16. KnowLine-9B-Gen13.6
  17. GLiDE 27B3.6
  18. Elephant-13.7
  19. GLiDE 0.8B3.8
  20. Gemini Decision 1 202611113.9
  21. GLiDE 2B4.0
  22. OneJev1.6-0.8B4.1
  23. Perplexity Decider Flash v1.34.2
  24. KnowLine-9B-Gen24.5
  25. KnowLine 4B Gen5.54.5
  26. GLiDE v1.24.7
  27. REAX 1.34.8
  28. Adjudex 9B v54.9
  29. Strands Decider 2B5.0
  30. StartLux-Decision-v1.1-0.8B-Instant5.6
  31. KnowLine-31B-Gen1.55.7
  32. Clef-Flash5.9
  33. Decider 15.9
  34. Cygnet6.5
  35. Chinese Decision Pointer 4B6.7
  36. Turion Jev 31B6.7
  37. Metask-Jev-4B6.8
  38. KnowLine-12B-Gen16.8
  39. Sieve 31B v37.0
  40. Logits Sigma-1 Duo7.0
  41. KnowLine-31B-Gen27.2
  42. d1-2B7.2
  43. StartLux-Decision-v1.1-2B-Instant7.2
  44. GLiDE 9B7.2
  45. GLiDE 0.4B7.7
  46. StartLux-Decision-v1.1-9B-Instant8.2
  47. Winnow-12B8.3
  48. StartLux-Decision-v1.1-4B-Instant8.3
  49. d1.18.7
  50. d1.2-ultra8.8
  51. GLiClass v4 Edge9.8
  52. Copernicus-31B11.0
  53. d1-ultra11.4
  54. vera-medium-omni12.4
  55. KnowLine-0.8B-Gen114.6
  56. Inkling 2 Small thinking on (no off mode) · chatNo probabilities
  57. Gemini 4 Hydrogen thinking on (no off mode) · chatNo probabilities
  58. MiniCPM5-2B thinking on · chat0.1% invalid, mean of 3 seedsNo probabilities
  59. Gemma 4 E4B thinking on · chat2 invalidNo probabilities
  60. Gemma 4 12B Unified thinking off · chatNo probabilities
  61. Granite-4.2-3B thinking on · chatNo probabilities
  62. LFM3-1.7B thinking on · chat0.1% invalid, mean of 3 seedsNo probabilities
  63. Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat1.5% invalid, mean of 3 seedsNo probabilities
  64. LFM3-1.7B thinking off · chat0.1% invalid, mean of 3 seedsNo probabilities
  65. Gemma 4 E2B thinking on · chatNo probabilities
  66. Qwen3.5-2B thinking on · chat78 invalidNo probabilities
  67. Gemma 4 12B Unified thinking on · chat296 invalidNo probabilities
  68. MiniCPM5-2B thinking off · chat0.2% invalid, mean of 3 seedsNo probabilities
  69. Qwen3.5-4B thinking on · chat179 invalidNo probabilities
  70. Qwen3.5-4B thinking off · chat1.0% invalid, mean of 3 seedsNo probabilities
  71. LFM2.5-8B-A1B thinking on (no off mode) · chatNo probabilities
  72. Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat0.03% invalid, mean of 3 seedsNo probabilities
  73. Gemma 4 E2B thinking off · chatNo probabilities
  74. Qwen3.5-2B thinking off · chat2.0% invalid, mean of 3 seedsNo probabilities
  75. Raon-1 nano thinking off · chat0.03% invalid, mean of 3 seedsNo probabilities

No row of this kind yet.

  • TypeSafe
  • Subquadratic
  • Microsoft
  • Patrick-18
  • PelaAI
  • Perplexity
  • MachinePulse
  • Apertix AI
  • Fastino
  • Liquid AI
  • Valen-Team
  • tattrongvu
  • victorbennett
  • Base Labs
  • Google
  • OmniJev
  • REAX
  • newsbang
  • Strands Agents
  • StartLux
  • Cloudflare
  • meraGPT
  • Blockbrain Labs
  • quincyqiang
  • Turion.AI
  • Metask Lab
  • Sthānika AI
  • Logits
  • EldanRing
  • Knowledgator
  • Poindexter Labs
  • scar-ai
  • Thinking Machines
  • OpenBMB
  • IBM
  • Alibaba
  • KRAFTON
The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –.ModelExpected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, points(accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairsOf the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarmsClient wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under itInput tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairsMeasured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs
1Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details1.983.088.6%79.9%6.0% false alarms0.17 sp90 0.19 s8 at a time$0.049API
2SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails2.384.989.9%78.1%4.1% false alarms0.39 sp90 0.41 s8 at a timeNot verifiedAPI
3Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details2.483.989.3%82.4%6.5% false alarms0.32 sp90 0.38 s8 at a time$0.036API
4Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details2.581.187.4%64.7%3.9% false alarms0.06 sp90 0.09 s1 at a timeOur GPU72 sRTX PRO 6000
5KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails2.761.174.0%16.4%2.7% false alarms0.11 sp90 0.17 s1 at a timeOur GPU130 sDGX Spark
6Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details2.886.290.8%82.7%3.5% false alarms0.22 sp90 0.34 s1 at a timeOur GPU245 sRTX PRO 6000
7Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details2.877.284.8%47.7%4.0% false alarms0.06 sp90 0.09 s1 at a timeOur GPU65 sRTX PRO 6000
8Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details2.890.093.3%80.8%1.7% false alarms0.22 sp90 0.31 s1 at a timeOur GPU230 sRTX PRO 6000
9Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details2.883.689.1%72.0%3.5% false alarms0.18 sp90 0.20 s8 at a time$0.049API
10Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details2.981.987.9%65.3%3.0% false alarms0.11 sp90 0.18 s1 at a timeOur GPU126 sRTX PRO 6000
11GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails3.085.190.0%68.7%1.7% false alarms0.30 sp90 0.56 s8 at a timeNot verifiedAPI
12d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails3.179.286.1%66.3%5.3% false alarms0.27 sp90 0.60 s8 at a timeNot verifiedAPI
13Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details3.274.382.8%39.8%2.8% false alarms0.07 sp90 0.10 s1 at a timeOur GPU78 sRTX PRO 6000
14Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details3.383.088.6%59.9%2.5% false alarms0.09 sp90 0.14 s1 at a timeOur GPU100 sRTX PRO 6000
15Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details3.475.483.6%46.8%3.7% false alarms0.05 sp90 0.07 s1 at a timeOur GPU54 sRTX PRO 6000
16KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details3.676.184.1%50.8%3.8% false alarms0.10 sp90 0.14 s1 at a timeOur GPU105 sRTX PRO 6000
17GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails3.686.190.7%74.8%1.5% false alarms1.03 sp90 1.95 s1 at a timeOur GPU1,205 sDGX Spark
18Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details3.776.984.6%41.0%1.9% false alarms0.09 sp90 0.11 s1 at a timeOur GPU98 sRTX PRO 6000
19GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails3.860.873.9%31.3%6.6% false alarms0.05 sp90 0.07 s1 at a timeOur GPU63 sDGX Spark
20Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details3.977.084.7%51.1%1.8% false alarms0.20 sp90 0.29 s8 at a time$0.093API
21GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details4.065.677.1%60.8%10.8% false alarms0.04 sp90 0.04 s1 at a timeOur GPU44 sRTX PRO 6000
22OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails4.150.066.7%33.4%14.4% false alarms0.03 sp90 0.04 s1 at a timeOur GPU43 sDGX Spark
23Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details4.278.885.9%67.5%5.6% false alarms0.06 sp90 0.08 s1 at a timeOur GPU61 sRTX PRO 6000
24KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details4.575.383.5%35.9%1.4% false alarms0.10 sp90 0.16 s1 at a timeOur GPU128 sRTX PRO 6000
25KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details4.580.687.1%62.0%3.4% false alarms0.04 sp90 0.06 s1 at a timeOur GPU49 sRTX PRO 6000
26GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details4.784.989.9%69.3%1.8% false alarms0.23 sp90 0.36 s1 at a timeOur GPU254 sRTX PRO 6000
27REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details4.887.091.3%67.5%1.2% false alarms0.68 sp90 1.05 s1 at a timeOur GPU745 sRTX PRO 6000
28Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details4.980.487.0%65.0%4.9% false alarms0.07 sp90 0.09 s1 at a timeOur GPU71 sRTX PRO 6000
29Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details5.060.373.6%4.6%0.8% false alarms0.04 sp90 0.05 s1 at a timeOur GPU43 sRTX PRO 6000
30StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails5.648.965.9%72.3%26.5% false alarms0.04 sp90 0.06 s1 at a timeOur GPU53 sDGX Spark
31KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails5.788.892.6%83.6%3.0% false alarms2.54 sp90 3.88 s1 at a timeOur GPU2,613 sDGX Spark
32Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details5.969.879.8%37.7%3.9% false alarms0.15 sp90 0.18 s1 at a timeOur GPU156 sRTX PRO 6000
33Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details5.943.362.2%31.9%17.6% false alarms0.65 sp90 0.75 s8 at a time$0.035API
34Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details6.572.981.9%29.2%2.1% false alarms0.09 sp90 0.13 s1 at a timeOur GPU93 sRTX PRO 6000
35Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details6.778.185.4%69.0%6.0% false alarms0.07 sp90 0.11 s1 at a timeOur GPU77 sRTX PRO 6000
36Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails6.786.491.0%66.3%1.7% false alarms1.12 sp90 1.96 s1 at a timeOur GPU1,329 sDGX Spark
37Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details6.861.774.5%23.1%3.4% false alarms0.12 sp90 0.19 s1 at a timeOur GPU134 sRTX PRO 6000
38KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details6.880.987.2%61.1%3.6% false alarms0.16 sp90 0.25 s1 at a timeOur GPU195 sRTX PRO 6000
39Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails7.088.192.1%79.3%2.1% false alarms2.67 sp90 3.93 s1 at a timeOur GPU2,824 sDGX Spark
40Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details7.085.190.1%63.8%0.9% false alarms0.64 sp90 1.01 s1 at a timeOur GPU723 sRTX PRO 6000
41KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details7.287.791.8%86.3%3.6% false alarms0.39 sp90 0.61 s1 at a timeOur GPU449 sRTX PRO 6000
42d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details7.249.666.4%85.7%32.2% false alarms0.02 sp90 0.03 s1 at a timeOur GPU25 sRTX PRO 6000
43StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails7.266.077.3%67.5%14.0% false alarms0.06 sp90 0.09 s1 at a timeOur GPU78 sDGX Spark
44GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details7.282.288.2%69.9%3.8% false alarms0.08 sp90 0.11 s1 at a timeOur GPU90 sRTX PRO 6000
45GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails7.744.462.9%31.0%16.8% false alarms0.03 sp90 0.04 s1 at a timeOur GPU41 sDGX Spark
46StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails8.278.885.9%88.8%9.5% false alarms0.23 sp90 0.36 s1 at a timeOur GPU268 sDGX Spark
47Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details8.376.684.4%51.7%3.8% false alarms0.12 sp90 0.18 s1 at a timeOur GPU132 sRTX PRO 6000
48StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails8.381.787.8%89.4%8.3% false alarms0.12 sp90 0.19 s1 at a timeOur GPU140 sDGX Spark
49d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details8.782.588.4%70.5%4.1% false alarms0.25 sp90 0.45 s8 at a time$0.017API
50d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details8.886.491.0%85.4%4.4% false alarms0.23 sp90 0.35 s8 at a time$0.035API
51GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails9.844.362.9%34.9%10.6% false alarms0.07 sp90 0.18 s1 at a timeOur GPU128 sDGX Spark
52Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details11.089.092.7%81.5%2.5% false alarms0.52 sp90 0.83 s1 at a timeOur GPU577 sRTX PRO 6000
53d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details11.484.789.8%80.8%4.5% false alarms0.48 sp90 0.90 s8 at a time$0.035API
54vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details12.483.789.1%76.0%5.7% false alarms0.06 sp90 0.06 s1 at a timeOur GPU60 sRTX PRO 6000
55KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails14.638.559.0%10.0%3.6% false alarms0.07 sp90 0.10 s1 at a timeOur GPU81 sDGX Spark
Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000DetailsNo probabilities86.290.8%73.6%2.4% false alarms2.70 sp90 16.4 s1 at a time$1.49API
Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetailsNo probabilities83.789.1%68.7%2.2% false alarms6.37 sp90 19.2 s1 at a timeNot verifiedAPI
MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000DetailsNo probabilities79.279.0–79.6min–max of 3 seeds86.2%86.0%–86.4%min–max of 3 seeds77.4%6.5% false alarmsmean of 3 seeds5.00 sp90 8.72 s30 at a time3 seeds pooledOur GPU197 sRTX PRO 6000mean of 3 seeds
Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000DetailsNo probabilities77.284.8%67.5%4.5% false alarms9.38 sp90 11.9 s30 at a timeOur GPU326 sRTX PRO 6000
Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000DetailsNo probabilities77.084.7%52.0%2.6% false alarms5.12 sp90 15.7 s64 at a timeOur GPU121 sRTX PRO 6000
Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000DetailsNo probabilities75.983.9%79.0%9.0% false alarms19.6 sp90 64.5 s30 at a timeOur GPU1,050 sRTX PRO 6000
LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000DetailsNo probabilities75.874.9–76.5min–max of 3 seeds83.9%83.3%–84.3%min–max of 3 seeds63.3%4.1% false alarmsmean of 3 seeds5.57 sp90 16.4 s30 at a time3 seeds pooledOur GPU268 sRTX PRO 6000mean of 3 seeds
Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000DetailsNo probabilities75.174.1–75.9min–max of 3 seeds83.4%82.7%–84.0%min–max of 3 seeds64.3%4.3% false alarmsmean of 3 seeds6.13 sp90 9.70 s30 at a time3 seeds pooledOur GPU234 sRTX PRO 6000mean of 3 seeds
LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000DetailsNo probabilities74.973.5–75.6min–max of 3 seeds83.3%82.3%–83.8%min–max of 3 seeds62.6%3.9% false alarmsmean of 3 seeds7.62 sp90 22.3 s64 at a time3 seeds pooledOur GPU180 sRTX PRO 6000mean of 3 seeds
Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000DetailsNo probabilities71.180.8%49.5%5.1% false alarms4.02 sp90 5.85 s64 at a timeOur GPU69 sRTX PRO 6000
Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000DetailsNo probabilities70.380.2%55.3%4.3% false alarms36.8 sp90 66.9 s64 at a timeOur GPU636 sRTX PRO 6000
Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000DetailsNo probabilities67.778.4%38.0%0.7% false alarms32.6 sp90 1640.2 s64 at a timeOur GPU4,327 sRTX PRO 6000
MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000DetailsNo probabilities67.066.9–67.0min–max of 3 seeds78.0%77.96%–78.01%min–max of 3 seeds28.8%2.7% false alarmsmean of 3 seeds0.86 sp90 3.48 s64 at a time3 seeds pooledOur GPU51 sRTX PRO 6000mean of 3 seeds
Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000DetailsNo probabilities66.277.5%73.2%6.2% false alarms37.8 sp90 96.1 s64 at a timeOur GPU741 sRTX PRO 6000
Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000DetailsNo probabilities64.664.5–64.8min–max of 3 seeds76.4%76.3%–76.6%min–max of 3 seeds61.2%7.6% false alarmsmean of 3 seeds3.49 sp90 20.4 s64 at a time3 seeds pooledOur GPU148 sRTX PRO 6000mean of 3 seeds
LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000DetailsNo probabilities57.771.8%8.8%0.8% false alarms9.58 sp90 15.2 s30 at a timeOur GPU352 sRTX PRO 6000
Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000DetailsNo probabilities55.254.5–55.8min–max of 3 seeds70.1%69.7%–70.6%min–max of 3 seeds7.7%1.6% false alarmsmean of 3 seeds0.18 sp90 5.42 s64 at a time3 seeds pooledOur GPU33 sRTX PRO 6000mean of 3 seeds
Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000DetailsNo probabilities54.869.8%6.1%1.3% false alarms0.19 sp90 3.31 s64 at a timeOur GPU18 sRTX PRO 6000
Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000DetailsNo probabilities50.249.3–50.8min–max of 3 seeds66.8%66.2%–67.2%min–max of 3 seeds34.4%7.8% false alarmsmean of 3 seeds1.53 sp90 13.7 s64 at a time3 seeds pooledOur GPU92 sRTX PRO 6000mean of 3 seeds
Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000DetailsNo probabilities40.640.6–40.8min–max of 3 seeds60.4%60.4%–60.5%min–max of 3 seeds19.9%2.0% false alarmsmean of 3 seeds3.03 sp90 7.76 s30 at a time3 seeds pooledOur GPU209 sRTX PRO 6000mean of 3 seeds

No row of this kind yet.

2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology

Wrong answers higher is better

View
  1. StartLux-Decision-v1.1-4B-Instant89.4%
  2. StartLux-Decision-v1.1-9B-Instant88.8%
  3. KnowLine-31B-Gen286.3%
  4. d1-2B85.7%
  5. d1.2-ultra85.4%
  6. KnowLine-31B-Gen1.583.6%
  7. Perplexity Decider 27B v1.382.7%
  8. Microsoft-Decision-182.4%
  9. Copernicus-31B81.5%
  10. d1-ultra80.8%
  11. Karpo Decision 31B80.8%
  12. Jev 1.1379.9%
  13. Sieve 31B v379.3%
  14. Granite-4.2-3B thinking on · chat79.0%
  15. SubQ Snap 1.178.1%
  16. MiniCPM5-2B thinking on · chat0.1% invalid, mean of 3 seeds77.4%
  17. vera-medium-omni76.0%
  18. GLiDE 27B74.8%
  19. Inkling 2 Small thinking on (no off mode) · chat73.6%
  20. Qwen3.5-4B thinking on · chat179 invalid73.2%
  21. StartLux-Decision-v1.1-0.8B-Instant72.3%
  22. Jev 1.1472.0%
  23. d1.170.5%
  24. GLiDE 9B69.9%
  25. GLiDE v1.269.3%
  26. Chinese Decision Pointer 4B69.0%
  27. Gemini 4 Hydrogen thinking on (no off mode) · chat68.7%
  28. GLiDE API68.7%
  29. Gemma 4 E4B thinking on · chat2 invalid67.5%
  30. Perplexity Decider Flash v1.367.5%
  31. REAX 1.367.5%
  32. StartLux-Decision-v1.1-2B-Instant67.5%
  33. d166.3%
  34. Turion Jev 31B66.3%
  35. Cerno 9B v1.265.3%
  36. Adjudex 9B v565.0%
  37. Dexy-Decision-9B-V264.7%
  38. Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat1.5% invalid, mean of 3 seeds64.3%
  39. Logits Sigma-1 Duo63.8%
  40. LFM3-1.7B thinking on · chat0.1% invalid, mean of 3 seeds63.3%
  41. LFM3-1.7B thinking off · chat0.1% invalid, mean of 3 seeds62.6%
  42. KnowLine 4B Gen5.562.0%
  43. Qwen3.5-4B thinking off · chat1.0% invalid, mean of 3 seeds61.2%
  44. KnowLine-12B-Gen161.1%
  45. GLiDE 2B60.8%
  46. Jeffan-9B-121159.9%
  47. Qwen3.5-2B thinking on · chat78 invalid55.3%
  48. Gemma 4 12B Unified thinking off · chat52.0%
  49. Winnow-12B51.7%
  50. Gemini Decision 1 2026111151.1%
  51. KnowLine-9B-Gen150.8%
  52. Gemma 4 E2B thinking on · chat49.5%
  53. Perplexity Decider Flash v1.247.7%
  54. Verdin-4B-v1.146.8%
  55. Elephant-141.0%
  56. Valen-Decision-1.5-9B39.8%
  57. Gemma 4 12B Unified thinking on · chat296 invalid38.0%
  58. Clef-Flash37.7%
  59. KnowLine-9B-Gen235.9%
  60. GLiClass v4 Edge34.9%
  61. Qwen3.5-2B thinking off · chat2.0% invalid, mean of 3 seeds34.4%
  62. OneJev1.6-0.8B33.4%
  63. Decider 131.9%
  64. GLiDE 0.8B31.3%
  65. GLiDE 0.4B31.0%
  66. Cygnet29.2%
  67. MiniCPM5-2B thinking off · chat0.2% invalid, mean of 3 seeds28.8%
  68. Metask-Jev-4B23.1%
  69. Raon-1 nano thinking off · chat0.03% invalid, mean of 3 seeds19.9%
  70. KnowLine-2B-Gen116.4%
  71. KnowLine-0.8B-Gen110.0%
  72. LFM2.5-8B-A1B thinking on (no off mode) · chat8.8%
  73. Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat0.03% invalid, mean of 3 seeds7.7%
  74. Gemma 4 E2B thinking off · chat6.1%
  75. Strands Decider 2B4.6%

No row of this kind yet.

  • StartLux
  • PelaAI
  • Liquid AI
  • Perplexity
  • Microsoft
  • Poindexter Labs
  • MachinePulse
  • TypeSafe
  • Sthānika AI
  • IBM
  • Subquadratic
  • OpenBMB
  • scar-ai
  • Fastino
  • Thinking Machines
  • Alibaba
  • quincyqiang
  • Google
  • REAX
  • Turion.AI
  • Apertix AI
  • newsbang
  • Patrick-18
  • Logits
  • tattrongvu
  • EldanRing
  • victorbennett
  • Base Labs
  • Valen-Team
  • Cloudflare
  • Knowledgator
  • OmniJev
  • meraGPT
  • Blockbrain Labs
  • Metask Lab
  • KRAFTON
  • Strands Agents
The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –.ModelOf the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarms(accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairsExpected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, pointsClient wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under itInput tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairsMeasured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs
1StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails89.4%8.3% false alarms81.787.8%8.30.12 sp90 0.19 s1 at a timeOur GPU140 sDGX Spark
2StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails88.8%9.5% false alarms78.885.9%8.20.23 sp90 0.36 s1 at a timeOur GPU268 sDGX Spark
3KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details86.3%3.6% false alarms87.791.8%7.20.39 sp90 0.61 s1 at a timeOur GPU449 sRTX PRO 6000
4d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details85.7%32.2% false alarms49.666.4%7.20.02 sp90 0.03 s1 at a timeOur GPU25 sRTX PRO 6000
5d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details85.4%4.4% false alarms86.491.0%8.80.23 sp90 0.35 s8 at a time$0.035API
6KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails83.6%3.0% false alarms88.892.6%5.72.54 sp90 3.88 s1 at a timeOur GPU2,613 sDGX Spark
7Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details82.7%3.5% false alarms86.290.8%2.80.22 sp90 0.34 s1 at a timeOur GPU245 sRTX PRO 6000
8Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details82.4%6.5% false alarms83.989.3%2.40.32 sp90 0.38 s8 at a time$0.036API
9Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details81.5%2.5% false alarms89.092.7%11.00.52 sp90 0.83 s1 at a timeOur GPU577 sRTX PRO 6000
10d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details80.8%4.5% false alarms84.789.8%11.40.48 sp90 0.90 s8 at a time$0.035API
11Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details80.8%1.7% false alarms90.093.3%2.80.22 sp90 0.31 s1 at a timeOur GPU230 sRTX PRO 6000
12Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details79.9%6.0% false alarms83.088.6%1.90.17 sp90 0.19 s8 at a time$0.049API
13Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails79.3%2.1% false alarms88.192.1%7.02.67 sp90 3.93 s1 at a timeOur GPU2,824 sDGX Spark
Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details79.0%9.0% false alarms75.983.9%No probabilities19.6 sp90 64.5 s30 at a timeOur GPU1,050 sRTX PRO 6000
14SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails78.1%4.1% false alarms84.989.9%2.30.39 sp90 0.41 s8 at a timeNot verifiedAPI
MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details77.4%6.5% false alarms · mean of 3 seeds79.279.0–79.6min–max of 3 seeds86.2%86.0%–86.4%min–max of 3 seedsNo probabilities5.00 sp90 8.72 s30 at a time3 seeds pooledOur GPU197 sRTX PRO 6000mean of 3 seeds
15vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details76.0%5.7% false alarms83.789.1%12.40.06 sp90 0.06 s1 at a timeOur GPU60 sRTX PRO 6000
16GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails74.8%1.5% false alarms86.190.7%3.61.03 sp90 1.95 s1 at a timeOur GPU1,205 sDGX Spark
Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details73.6%2.4% false alarms86.290.8%No probabilities2.70 sp90 16.4 s1 at a time$1.49API
Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details73.2%6.2% false alarms66.277.5%No probabilities37.8 sp90 96.1 s64 at a timeOur GPU741 sRTX PRO 6000
17StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails72.3%26.5% false alarms48.965.9%5.60.04 sp90 0.06 s1 at a timeOur GPU53 sDGX Spark
18Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details72.0%3.5% false alarms83.689.1%2.80.18 sp90 0.20 s8 at a time$0.049API
19d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details70.5%4.1% false alarms82.588.4%8.70.25 sp90 0.45 s8 at a time$0.017API
20GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details69.9%3.8% false alarms82.288.2%7.20.08 sp90 0.11 s1 at a timeOur GPU90 sRTX PRO 6000
21GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details69.3%1.8% false alarms84.989.9%4.70.23 sp90 0.36 s1 at a timeOur GPU254 sRTX PRO 6000
22Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details69.0%6.0% false alarms78.185.4%6.70.07 sp90 0.11 s1 at a timeOur GPU77 sRTX PRO 6000
Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails68.7%2.2% false alarms83.789.1%No probabilities6.37 sp90 19.2 s1 at a timeNot verifiedAPI
23GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails68.7%1.7% false alarms85.190.0%3.00.30 sp90 0.56 s8 at a timeNot verifiedAPI
Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details67.5%4.5% false alarms77.284.8%No probabilities9.38 sp90 11.9 s30 at a timeOur GPU326 sRTX PRO 6000
24Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details67.5%5.6% false alarms78.885.9%4.20.06 sp90 0.08 s1 at a timeOur GPU61 sRTX PRO 6000
25REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details67.5%1.2% false alarms87.091.3%4.80.68 sp90 1.05 s1 at a timeOur GPU745 sRTX PRO 6000
26StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails67.5%14.0% false alarms66.077.3%7.20.06 sp90 0.09 s1 at a timeOur GPU78 sDGX Spark
27d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails66.3%5.3% false alarms79.286.1%3.10.27 sp90 0.60 s8 at a timeNot verifiedAPI
28Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails66.3%1.7% false alarms86.491.0%6.71.12 sp90 1.96 s1 at a timeOur GPU1,329 sDGX Spark
29Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details65.3%3.0% false alarms81.987.9%2.90.11 sp90 0.18 s1 at a timeOur GPU126 sRTX PRO 6000
30Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details65.0%4.9% false alarms80.487.0%4.90.07 sp90 0.09 s1 at a timeOur GPU71 sRTX PRO 6000
31Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details64.7%3.9% false alarms81.187.4%2.50.06 sp90 0.09 s1 at a timeOur GPU72 sRTX PRO 6000
Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details64.3%4.3% false alarms · mean of 3 seeds75.174.1–75.9min–max of 3 seeds83.4%82.7%–84.0%min–max of 3 seedsNo probabilities6.13 sp90 9.70 s30 at a time3 seeds pooledOur GPU234 sRTX PRO 6000mean of 3 seeds
32Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details63.8%0.9% false alarms85.190.1%7.00.64 sp90 1.01 s1 at a timeOur GPU723 sRTX PRO 6000
LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details63.3%4.1% false alarms · mean of 3 seeds75.874.9–76.5min–max of 3 seeds83.9%83.3%–84.3%min–max of 3 seedsNo probabilities5.57 sp90 16.4 s30 at a time3 seeds pooledOur GPU268 sRTX PRO 6000mean of 3 seeds
LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details62.6%3.9% false alarms · mean of 3 seeds74.973.5–75.6min–max of 3 seeds83.3%82.3%–83.8%min–max of 3 seedsNo probabilities7.62 sp90 22.3 s64 at a time3 seeds pooledOur GPU180 sRTX PRO 6000mean of 3 seeds
33KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details62.0%3.4% false alarms80.687.1%4.50.04 sp90 0.06 s1 at a timeOur GPU49 sRTX PRO 6000
Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details61.2%7.6% false alarms · mean of 3 seeds64.664.5–64.8min–max of 3 seeds76.4%76.3%–76.6%min–max of 3 seedsNo probabilities3.49 sp90 20.4 s64 at a time3 seeds pooledOur GPU148 sRTX PRO 6000mean of 3 seeds
34KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details61.1%3.6% false alarms80.987.2%6.80.16 sp90 0.25 s1 at a timeOur GPU195 sRTX PRO 6000
35GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details60.8%10.8% false alarms65.677.1%4.00.04 sp90 0.04 s1 at a timeOur GPU44 sRTX PRO 6000
36Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details59.9%2.5% false alarms83.088.6%3.30.09 sp90 0.14 s1 at a timeOur GPU100 sRTX PRO 6000
Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details55.3%4.3% false alarms70.380.2%No probabilities36.8 sp90 66.9 s64 at a timeOur GPU636 sRTX PRO 6000
Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details52.0%2.6% false alarms77.084.7%No probabilities5.12 sp90 15.7 s64 at a timeOur GPU121 sRTX PRO 6000
37Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details51.7%3.8% false alarms76.684.4%8.30.12 sp90 0.18 s1 at a timeOur GPU132 sRTX PRO 6000
38Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details51.1%1.8% false alarms77.084.7%3.90.20 sp90 0.29 s8 at a time$0.093API
39KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details50.8%3.8% false alarms76.184.1%3.60.10 sp90 0.14 s1 at a timeOur GPU105 sRTX PRO 6000
Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details49.5%5.1% false alarms71.180.8%No probabilities4.02 sp90 5.85 s64 at a timeOur GPU69 sRTX PRO 6000
40Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details47.7%4.0% false alarms77.284.8%2.80.06 sp90 0.09 s1 at a timeOur GPU65 sRTX PRO 6000
41Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details46.8%3.7% false alarms75.483.6%3.40.05 sp90 0.07 s1 at a timeOur GPU54 sRTX PRO 6000
42Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details41.0%1.9% false alarms76.984.6%3.70.09 sp90 0.11 s1 at a timeOur GPU98 sRTX PRO 6000
43Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details39.8%2.8% false alarms74.382.8%3.20.07 sp90 0.10 s1 at a timeOur GPU78 sRTX PRO 6000
Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details38.0%0.7% false alarms67.778.4%No probabilities32.6 sp90 1640.2 s64 at a timeOur GPU4,327 sRTX PRO 6000
44Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details37.7%3.9% false alarms69.879.8%5.90.15 sp90 0.18 s1 at a timeOur GPU156 sRTX PRO 6000
45KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details35.9%1.4% false alarms75.383.5%4.50.10 sp90 0.16 s1 at a timeOur GPU128 sRTX PRO 6000
46GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails34.9%10.6% false alarms44.362.9%9.80.07 sp90 0.18 s1 at a timeOur GPU128 sDGX Spark
Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details34.4%7.8% false alarms · mean of 3 seeds50.249.3–50.8min–max of 3 seeds66.8%66.2%–67.2%min–max of 3 seedsNo probabilities1.53 sp90 13.7 s64 at a time3 seeds pooledOur GPU92 sRTX PRO 6000mean of 3 seeds
47OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails33.4%14.4% false alarms50.066.7%4.10.03 sp90 0.04 s1 at a timeOur GPU43 sDGX Spark
48Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details31.9%17.6% false alarms43.362.2%5.90.65 sp90 0.75 s8 at a time$0.035API
49GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails31.3%6.6% false alarms60.873.9%3.80.05 sp90 0.07 s1 at a timeOur GPU63 sDGX Spark
50GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails31.0%16.8% false alarms44.462.9%7.70.03 sp90 0.04 s1 at a timeOur GPU41 sDGX Spark
51Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details29.2%2.1% false alarms72.981.9%6.50.09 sp90 0.13 s1 at a timeOur GPU93 sRTX PRO 6000
MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details28.8%2.7% false alarms · mean of 3 seeds67.066.9–67.0min–max of 3 seeds78.0%77.96%–78.01%min–max of 3 seedsNo probabilities0.86 sp90 3.48 s64 at a time3 seeds pooledOur GPU51 sRTX PRO 6000mean of 3 seeds
52Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details23.1%3.4% false alarms61.774.5%6.80.12 sp90 0.19 s1 at a timeOur GPU134 sRTX PRO 6000
Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details19.9%2.0% false alarms · mean of 3 seeds40.640.6–40.8min–max of 3 seeds60.4%60.4%–60.5%min–max of 3 seedsNo probabilities3.03 sp90 7.76 s30 at a time3 seeds pooledOur GPU209 sRTX PRO 6000mean of 3 seeds
53KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails16.4%2.7% false alarms61.174.0%2.70.11 sp90 0.17 s1 at a timeOur GPU130 sDGX Spark
54KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails10.0%3.6% false alarms38.559.0%14.60.07 sp90 0.10 s1 at a timeOur GPU81 sDGX Spark
LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details8.8%0.8% false alarms57.771.8%No probabilities9.58 sp90 15.2 s30 at a timeOur GPU352 sRTX PRO 6000
Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details7.7%1.6% false alarms · mean of 3 seeds55.254.5–55.8min–max of 3 seeds70.1%69.7%–70.6%min–max of 3 seedsNo probabilities0.18 sp90 5.42 s64 at a time3 seeds pooledOur GPU33 sRTX PRO 6000mean of 3 seeds
Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details6.1%1.3% false alarms54.869.8%No probabilities0.19 sp90 3.31 s64 at a timeOur GPU18 sRTX PRO 6000
55Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details4.6%0.8% false alarms60.373.6%5.00.04 sp90 0.05 s1 at a timeOur GPU43 sRTX PRO 6000

No row of this kind yet.

2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology

Latency lower is better

View
  1. d1-2B0.02 s
  2. GLiDE 0.4B0.03 s
  3. OneJev1.6-0.8B0.03 s
  4. GLiDE 2B0.04 s
  5. Strands Decider 2B0.04 s
  6. StartLux-Decision-v1.1-0.8B-Instant0.04 s
  7. KnowLine 4B Gen5.50.04 s
  8. GLiDE 0.8B0.05 s
  9. Verdin-4B-v1.10.05 s
  10. Perplexity Decider Flash v1.30.06 s
  11. vera-medium-omni0.06 s
  12. Perplexity Decider Flash v1.20.06 s
  13. Dexy-Decision-9B-V20.06 s
  14. StartLux-Decision-v1.1-2B-Instant0.06 s
  15. Adjudex 9B v50.07 s
  16. KnowLine-0.8B-Gen10.07 s
  17. Chinese Decision Pointer 4B0.07 s
  18. Valen-Decision-1.5-9B0.07 s
  19. GLiClass v4 Edge0.07 s
  20. GLiDE 9B0.08 s
  21. Cygnet0.09 s
  22. Elephant-10.09 s
  23. Jeffan-9B-12110.09 s
  24. KnowLine-9B-Gen10.10 s
  25. KnowLine-9B-Gen20.10 s
  26. KnowLine-2B-Gen10.11 s
  27. Cerno 9B v1.20.11 s
  28. Metask-Jev-4B0.12 s
  29. StartLux-Decision-v1.1-4B-Instant0.12 s
  30. Winnow-12B0.12 s
  31. Clef-Flash0.15 s
  32. KnowLine-12B-Gen10.16 s
  33. Jev 1.130.17 s
  34. Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat0.03% invalid, mean of 3 seeds0.18 s
  35. Jev 1.140.18 s
  36. Gemma 4 E2B thinking off · chat0.19 s
  37. Gemini Decision 1 202611110.20 s
  38. Karpo Decision 31B0.22 s
  39. Perplexity Decider 27B v1.30.22 s
  40. d1.2-ultra0.23 s
  41. StartLux-Decision-v1.1-9B-Instant0.23 s
  42. GLiDE v1.20.23 s
  43. d1.10.25 s
  44. d10.27 s
  45. GLiDE API0.30 s
  46. Microsoft-Decision-10.32 s
  47. KnowLine-31B-Gen20.39 s
  48. SubQ Snap 1.10.39 s
  49. d1-ultra0.48 s
  50. Copernicus-31B0.52 s
  51. Logits Sigma-1 Duo0.64 s
  52. Decider 10.65 s
  53. REAX 1.30.68 s
  54. MiniCPM5-2B thinking off · chat0.2% invalid, mean of 3 seeds0.86 s
  55. GLiDE 27B1.03 s
  56. Turion Jev 31B1.12 s
  57. Qwen3.5-2B thinking off · chat2.0% invalid, mean of 3 seeds1.53 s
  58. KnowLine-31B-Gen1.52.54 s
  59. Sieve 31B v32.67 s
  60. Inkling 2 Small thinking on (no off mode) · chat2.70 s
  61. Raon-1 nano thinking off · chat0.03% invalid, mean of 3 seeds3.03 s
  62. Qwen3.5-4B thinking off · chat1.0% invalid, mean of 3 seeds3.49 s
  63. Gemma 4 E2B thinking on · chat4.02 s
  64. MiniCPM5-2B thinking on · chat0.1% invalid, mean of 3 seeds5.00 s
  65. Gemma 4 12B Unified thinking off · chat5.12 s
  66. LFM3-1.7B thinking on · chat0.1% invalid, mean of 3 seeds5.57 s
  67. Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat1.5% invalid, mean of 3 seeds6.13 s
  68. Gemini 4 Hydrogen thinking on (no off mode) · chat6.37 s
  69. LFM3-1.7B thinking off · chat0.1% invalid, mean of 3 seeds7.62 s
  70. Gemma 4 E4B thinking on · chat2 invalid9.38 s
  71. LFM2.5-8B-A1B thinking on (no off mode) · chat9.58 s
  72. Granite-4.2-3B thinking on · chat19.6 s
  73. Gemma 4 12B Unified thinking on · chat296 invalid32.6 s
  74. Qwen3.5-2B thinking on · chat78 invalid36.8 s
  75. Qwen3.5-4B thinking on · chat179 invalid37.8 s

No row of this kind yet.

  • Liquid AI
  • Fastino
  • OmniJev
  • Strands Agents
  • StartLux
  • PelaAI
  • victorbennett
  • Perplexity
  • scar-ai
  • Patrick-18
  • newsbang
  • quincyqiang
  • Valen-Team
  • Knowledgator
  • Blockbrain Labs
  • Base Labs
  • tattrongvu
  • Apertix AI
  • Metask Lab
  • EldanRing
  • Cloudflare
  • TypeSafe
  • Google
  • MachinePulse
  • Microsoft
  • Subquadratic
  • Poindexter Labs
  • Logits
  • meraGPT
  • REAX
  • OpenBMB
  • Turion.AI
  • Alibaba
  • Sthānika AI
  • Thinking Machines
  • KRAFTON
  • IBM
The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –.ModelClient wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under it(accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairsExpected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, pointsOf the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarmsInput tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairsMeasured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs
1d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details0.02 sp90 0.03 s · 1 at a time49.666.4%7.285.7%32.2% false alarmsOur GPU25 sRTX PRO 6000
2GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails0.03 sp90 0.04 s · 1 at a time44.462.9%7.731.0%16.8% false alarmsOur GPU41 sDGX Spark
3OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails0.03 sp90 0.04 s · 1 at a time50.066.7%4.133.4%14.4% false alarmsOur GPU43 sDGX Spark
4GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details0.04 sp90 0.04 s · 1 at a time65.677.1%4.060.8%10.8% false alarmsOur GPU44 sRTX PRO 6000
5Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details0.04 sp90 0.05 s · 1 at a time60.373.6%5.04.6%0.8% false alarmsOur GPU43 sRTX PRO 6000
6StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails0.04 sp90 0.06 s · 1 at a time48.965.9%5.672.3%26.5% false alarmsOur GPU53 sDGX Spark
7KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details0.04 sp90 0.06 s · 1 at a time80.687.1%4.562.0%3.4% false alarmsOur GPU49 sRTX PRO 6000
8GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails0.05 sp90 0.07 s · 1 at a time60.873.9%3.831.3%6.6% false alarmsOur GPU63 sDGX Spark
9Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details0.05 sp90 0.07 s · 1 at a time75.483.6%3.446.8%3.7% false alarmsOur GPU54 sRTX PRO 6000
10Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details0.06 sp90 0.08 s · 1 at a time78.885.9%4.267.5%5.6% false alarmsOur GPU61 sRTX PRO 6000
11vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details0.06 sp90 0.06 s · 1 at a time83.789.1%12.476.0%5.7% false alarmsOur GPU60 sRTX PRO 6000
12Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details0.06 sp90 0.09 s · 1 at a time77.284.8%2.847.7%4.0% false alarmsOur GPU65 sRTX PRO 6000
13Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details0.06 sp90 0.09 s · 1 at a time81.187.4%2.564.7%3.9% false alarmsOur GPU72 sRTX PRO 6000
14StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails0.06 sp90 0.09 s · 1 at a time66.077.3%7.267.5%14.0% false alarmsOur GPU78 sDGX Spark
15Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details0.07 sp90 0.09 s · 1 at a time80.487.0%4.965.0%4.9% false alarmsOur GPU71 sRTX PRO 6000
16KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails0.07 sp90 0.10 s · 1 at a time38.559.0%14.610.0%3.6% false alarmsOur GPU81 sDGX Spark
17Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details0.07 sp90 0.11 s · 1 at a time78.185.4%6.769.0%6.0% false alarmsOur GPU77 sRTX PRO 6000
18Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details0.07 sp90 0.10 s · 1 at a time74.382.8%3.239.8%2.8% false alarmsOur GPU78 sRTX PRO 6000
19GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails0.07 sp90 0.18 s · 1 at a time44.362.9%9.834.9%10.6% false alarmsOur GPU128 sDGX Spark
20GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details0.08 sp90 0.11 s · 1 at a time82.288.2%7.269.9%3.8% false alarmsOur GPU90 sRTX PRO 6000
21Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details0.09 sp90 0.13 s · 1 at a time72.981.9%6.529.2%2.1% false alarmsOur GPU93 sRTX PRO 6000
22Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details0.09 sp90 0.11 s · 1 at a time76.984.6%3.741.0%1.9% false alarmsOur GPU98 sRTX PRO 6000
23Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details0.09 sp90 0.14 s · 1 at a time83.088.6%3.359.9%2.5% false alarmsOur GPU100 sRTX PRO 6000
24KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details0.10 sp90 0.14 s · 1 at a time76.184.1%3.650.8%3.8% false alarmsOur GPU105 sRTX PRO 6000
25KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details0.10 sp90 0.16 s · 1 at a time75.383.5%4.535.9%1.4% false alarmsOur GPU128 sRTX PRO 6000
26KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails0.11 sp90 0.17 s · 1 at a time61.174.0%2.716.4%2.7% false alarmsOur GPU130 sDGX Spark
27Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details0.11 sp90 0.18 s · 1 at a time81.987.9%2.965.3%3.0% false alarmsOur GPU126 sRTX PRO 6000
28Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details0.12 sp90 0.19 s · 1 at a time61.774.5%6.823.1%3.4% false alarmsOur GPU134 sRTX PRO 6000
29StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails0.12 sp90 0.19 s · 1 at a time81.787.8%8.389.4%8.3% false alarmsOur GPU140 sDGX Spark
30Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details0.12 sp90 0.18 s · 1 at a time76.684.4%8.351.7%3.8% false alarmsOur GPU132 sRTX PRO 6000
31Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details0.15 sp90 0.18 s · 1 at a time69.879.8%5.937.7%3.9% false alarmsOur GPU156 sRTX PRO 6000
32KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details0.16 sp90 0.25 s · 1 at a time80.987.2%6.861.1%3.6% false alarmsOur GPU195 sRTX PRO 6000
33Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details0.17 sp90 0.19 s · 8 at a time83.088.6%1.979.9%6.0% false alarms$0.049API
Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details0.18 sp90 5.42 s · 64 at a time · 3 seeds pooled55.254.5–55.8min–max of 3 seeds70.1%69.7%–70.6%min–max of 3 seedsNo probabilities7.7%1.6% false alarmsmean of 3 seedsOur GPU33 sRTX PRO 6000mean of 3 seeds
34Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details0.18 sp90 0.20 s · 8 at a time83.689.1%2.872.0%3.5% false alarms$0.049API
Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details0.19 sp90 3.31 s · 64 at a time54.869.8%No probabilities6.1%1.3% false alarmsOur GPU18 sRTX PRO 6000
35Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details0.20 sp90 0.29 s · 8 at a time77.084.7%3.951.1%1.8% false alarms$0.093API
36Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details0.22 sp90 0.31 s · 1 at a time90.093.3%2.880.8%1.7% false alarmsOur GPU230 sRTX PRO 6000
37Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details0.22 sp90 0.34 s · 1 at a time86.290.8%2.882.7%3.5% false alarmsOur GPU245 sRTX PRO 6000
38d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details0.23 sp90 0.35 s · 8 at a time86.491.0%8.885.4%4.4% false alarms$0.035API
39StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails0.23 sp90 0.36 s · 1 at a time78.885.9%8.288.8%9.5% false alarmsOur GPU268 sDGX Spark
40GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details0.23 sp90 0.36 s · 1 at a time84.989.9%4.769.3%1.8% false alarmsOur GPU254 sRTX PRO 6000
41d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details0.25 sp90 0.45 s · 8 at a time82.588.4%8.770.5%4.1% false alarms$0.017API
42d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails0.27 sp90 0.60 s · 8 at a time79.286.1%3.166.3%5.3% false alarmsNot verifiedAPI
43GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails0.30 sp90 0.56 s · 8 at a time85.190.0%3.068.7%1.7% false alarmsNot verifiedAPI
44Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details0.32 sp90 0.38 s · 8 at a time83.989.3%2.482.4%6.5% false alarms$0.036API
45KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details0.39 sp90 0.61 s · 1 at a time87.791.8%7.286.3%3.6% false alarmsOur GPU449 sRTX PRO 6000
46SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails0.39 sp90 0.41 s · 8 at a time84.989.9%2.378.1%4.1% false alarmsNot verifiedAPI
47d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details0.48 sp90 0.90 s · 8 at a time84.789.8%11.480.8%4.5% false alarms$0.035API
48Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details0.52 sp90 0.83 s · 1 at a time89.092.7%11.081.5%2.5% false alarmsOur GPU577 sRTX PRO 6000
49Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details0.64 sp90 1.01 s · 1 at a time85.190.1%7.063.8%0.9% false alarmsOur GPU723 sRTX PRO 6000
50Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details0.65 sp90 0.75 s · 8 at a time43.362.2%5.931.9%17.6% false alarms$0.035API
51REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details0.68 sp90 1.05 s · 1 at a time87.091.3%4.867.5%1.2% false alarmsOur GPU745 sRTX PRO 6000
MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details0.86 sp90 3.48 s · 64 at a time · 3 seeds pooled67.066.9–67.0min–max of 3 seeds78.0%77.96%–78.01%min–max of 3 seedsNo probabilities28.8%2.7% false alarmsmean of 3 seedsOur GPU51 sRTX PRO 6000mean of 3 seeds
52GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails1.03 sp90 1.95 s · 1 at a time86.190.7%3.674.8%1.5% false alarmsOur GPU1,205 sDGX Spark
53Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails1.12 sp90 1.96 s · 1 at a time86.491.0%6.766.3%1.7% false alarmsOur GPU1,329 sDGX Spark
Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details1.53 sp90 13.7 s · 64 at a time · 3 seeds pooled50.249.3–50.8min–max of 3 seeds66.8%66.2%–67.2%min–max of 3 seedsNo probabilities34.4%7.8% false alarmsmean of 3 seedsOur GPU92 sRTX PRO 6000mean of 3 seeds
54KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails2.54 sp90 3.88 s · 1 at a time88.892.6%5.783.6%3.0% false alarmsOur GPU2,613 sDGX Spark
55Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails2.67 sp90 3.93 s · 1 at a time88.192.1%7.079.3%2.1% false alarmsOur GPU2,824 sDGX Spark
Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details2.70 sp90 16.4 s · 1 at a time86.290.8%No probabilities73.6%2.4% false alarms$1.49API
Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details3.03 sp90 7.76 s · 30 at a time · 3 seeds pooled40.640.6–40.8min–max of 3 seeds60.4%60.4%–60.5%min–max of 3 seedsNo probabilities19.9%2.0% false alarmsmean of 3 seedsOur GPU209 sRTX PRO 6000mean of 3 seeds
Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details3.49 sp90 20.4 s · 64 at a time · 3 seeds pooled64.664.5–64.8min–max of 3 seeds76.4%76.3%–76.6%min–max of 3 seedsNo probabilities61.2%7.6% false alarmsmean of 3 seedsOur GPU148 sRTX PRO 6000mean of 3 seeds
Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details4.02 sp90 5.85 s · 64 at a time71.180.8%No probabilities49.5%5.1% false alarmsOur GPU69 sRTX PRO 6000
MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details5.00 sp90 8.72 s · 30 at a time · 3 seeds pooled79.279.0–79.6min–max of 3 seeds86.2%86.0%–86.4%min–max of 3 seedsNo probabilities77.4%6.5% false alarmsmean of 3 seedsOur GPU197 sRTX PRO 6000mean of 3 seeds
Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details5.12 sp90 15.7 s · 64 at a time77.084.7%No probabilities52.0%2.6% false alarmsOur GPU121 sRTX PRO 6000
LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details5.57 sp90 16.4 s · 30 at a time · 3 seeds pooled75.874.9–76.5min–max of 3 seeds83.9%83.3%–84.3%min–max of 3 seedsNo probabilities63.3%4.1% false alarmsmean of 3 seedsOur GPU268 sRTX PRO 6000mean of 3 seeds
Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details6.13 sp90 9.70 s · 30 at a time · 3 seeds pooled75.174.1–75.9min–max of 3 seeds83.4%82.7%–84.0%min–max of 3 seedsNo probabilities64.3%4.3% false alarmsmean of 3 seedsOur GPU234 sRTX PRO 6000mean of 3 seeds
Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails6.37 sp90 19.2 s · 1 at a time83.789.1%No probabilities68.7%2.2% false alarmsNot verifiedAPI
LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details7.62 sp90 22.3 s · 64 at a time · 3 seeds pooled74.973.5–75.6min–max of 3 seeds83.3%82.3%–83.8%min–max of 3 seedsNo probabilities62.6%3.9% false alarmsmean of 3 seedsOur GPU180 sRTX PRO 6000mean of 3 seeds
Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details9.38 sp90 11.9 s · 30 at a time77.284.8%No probabilities67.5%4.5% false alarmsOur GPU326 sRTX PRO 6000
LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details9.58 sp90 15.2 s · 30 at a time57.771.8%No probabilities8.8%0.8% false alarmsOur GPU352 sRTX PRO 6000
Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details19.6 sp90 64.5 s · 30 at a time75.983.9%No probabilities79.0%9.0% false alarmsOur GPU1,050 sRTX PRO 6000
Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details32.6 sp90 1640.2 s · 64 at a time67.778.4%No probabilities38.0%0.7% false alarmsOur GPU4,327 sRTX PRO 6000
Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details36.8 sp90 66.9 s · 64 at a time70.380.2%No probabilities55.3%4.3% false alarmsOur GPU636 sRTX PRO 6000
Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details37.8 sp90 96.1 s · 64 at a time66.277.5%No probabilities73.2%6.2% false alarmsOur GPU741 sRTX PRO 6000

No row of this kind yet.

2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology

List price lower is better

View
  1. d1.1$0.017
  2. d1-ultra$0.035
  3. d1.2-ultra$0.035
  4. Decider 1$0.035
  5. Microsoft-Decision-1$0.036
  6. Jev 1.13$0.049
  7. Jev 1.14$0.049
  8. Gemini Decision 1 20261111$0.093
  9. Inkling 2 Small thinking on (no off mode) · chat$1.49
  10. Karpo Decision 31BOur GPU
  11. Copernicus-31BOur GPU
  12. KnowLine-31B-Gen1.5Our GPU
  13. Sieve 31B v3Our GPU
  14. KnowLine-31B-Gen2Our GPU
  15. REAX 1.3Our GPU
  16. Turion Jev 31BOur GPU
  17. Perplexity Decider 27B v1.3Our GPU
  18. GLiDE 27BOur GPU
  19. Logits Sigma-1 DuoOur GPU
  20. GLiDE APINot verified
  21. GLiDE v1.2Our GPU
  22. SubQ Snap 1.1Not verified
  23. Gemini 4 Hydrogen thinking on (no off mode) · chatNot verified
  24. vera-medium-omniOur GPU
  25. Jeffan-9B-1211Our GPU
  26. GLiDE 9BOur GPU
  27. Cerno 9B v1.2Our GPU
  28. StartLux-Decision-v1.1-4B-InstantOur GPU
  29. Dexy-Decision-9B-V2Our GPU
  30. KnowLine-12B-Gen1Our GPU
  31. KnowLine 4B Gen5.5Our GPU
  32. Adjudex 9B v5Our GPU
  33. MiniCPM5-2B thinking on · chat0.1% invalid, mean of 3 seedsOur GPU
  34. d1Not verified
  35. Perplexity Decider Flash v1.3Our GPU
  36. StartLux-Decision-v1.1-9B-InstantOur GPU
  37. Chinese Decision Pointer 4BOur GPU
  38. Gemma 4 E4B thinking on · chat2 invalidOur GPU
  39. Perplexity Decider Flash v1.2Our GPU
  40. Gemma 4 12B Unified thinking off · chatOur GPU
  41. Elephant-1Our GPU
  42. Winnow-12BOur GPU
  43. KnowLine-9B-Gen1Our GPU
  44. Granite-4.2-3B thinking on · chatOur GPU
  45. LFM3-1.7B thinking on · chat0.1% invalid, mean of 3 seedsOur GPU
  46. Verdin-4B-v1.1Our GPU
  47. KnowLine-9B-Gen2Our GPU
  48. Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat1.5% invalid, mean of 3 seedsOur GPU
  49. LFM3-1.7B thinking off · chat0.1% invalid, mean of 3 seedsOur GPU
  50. Valen-Decision-1.5-9BOur GPU
  51. CygnetOur GPU
  52. Gemma 4 E2B thinking on · chatOur GPU
  53. Qwen3.5-2B thinking on · chat78 invalidOur GPU
  54. Clef-FlashOur GPU
  55. Gemma 4 12B Unified thinking on · chat296 invalidOur GPU
  56. MiniCPM5-2B thinking off · chat0.2% invalid, mean of 3 seedsOur GPU
  57. Qwen3.5-4B thinking on · chat179 invalidOur GPU
  58. StartLux-Decision-v1.1-2B-InstantOur GPU
  59. GLiDE 2BOur GPU
  60. Qwen3.5-4B thinking off · chat1.0% invalid, mean of 3 seedsOur GPU
  61. Metask-Jev-4BOur GPU
  62. KnowLine-2B-Gen1Our GPU
  63. GLiDE 0.8BOur GPU
  64. Strands Decider 2BOur GPU
  65. LFM2.5-8B-A1B thinking on (no off mode) · chatOur GPU
  66. Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat0.03% invalid, mean of 3 seedsOur GPU
  67. Gemma 4 E2B thinking off · chatOur GPU
  68. Qwen3.5-2B thinking off · chat2.0% invalid, mean of 3 seedsOur GPU
  69. OneJev1.6-0.8BOur GPU
  70. d1-2BOur GPU
  71. StartLux-Decision-v1.1-0.8B-InstantOur GPU
  72. GLiDE 0.4BOur GPU
  73. GLiClass v4 EdgeOur GPU
  74. Raon-1 nano thinking off · chat0.03% invalid, mean of 3 seedsOur GPU
  75. KnowLine-0.8B-Gen1Our GPU

No row of this kind yet.

  • Liquid AI
  • meraGPT
  • Microsoft
  • TypeSafe
  • Google
  • Thinking Machines
  • MachinePulse
  • Poindexter Labs
  • PelaAI
  • Sthānika AI
  • REAX
  • Turion.AI
  • Perplexity
  • Fastino
  • Logits
  • Subquadratic
  • scar-ai
  • tattrongvu
  • Apertix AI
  • StartLux
  • Patrick-18
  • newsbang
  • OpenBMB
  • quincyqiang
  • Base Labs
  • EldanRing
  • IBM
  • victorbennett
  • Valen-Team
  • Blockbrain Labs
  • Alibaba
  • Cloudflare
  • Metask Lab
  • Strands Agents
  • OmniJev
  • Knowledgator
  • KRAFTON
The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –.ModelInput tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairs(accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairsExpected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, pointsOf the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarmsClient wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under itMeasured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs
1d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details$0.01782.588.4%8.770.5%4.1% false alarms0.25 sp90 0.45 s8 at a timeAPI
2d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details$0.03584.789.8%11.480.8%4.5% false alarms0.48 sp90 0.90 s8 at a timeAPI
3d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details$0.03586.491.0%8.885.4%4.4% false alarms0.23 sp90 0.35 s8 at a timeAPI
4Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details$0.03543.362.2%5.931.9%17.6% false alarms0.65 sp90 0.75 s8 at a timeAPI
5Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details$0.03683.989.3%2.482.4%6.5% false alarms0.32 sp90 0.38 s8 at a timeAPI
6Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details$0.04983.088.6%1.979.9%6.0% false alarms0.17 sp90 0.19 s8 at a timeAPI
7Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details$0.04983.689.1%2.872.0%3.5% false alarms0.18 sp90 0.20 s8 at a timeAPI
8Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details$0.09377.084.7%3.951.1%1.8% false alarms0.20 sp90 0.29 s8 at a timeAPI
Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details$1.4986.290.8%No probabilities73.6%2.4% false alarms2.70 sp90 16.4 s1 at a timeAPI
–Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000DetailsOur GPU90.093.3%2.880.8%1.7% false alarms0.22 sp90 0.31 s1 at a time230 sRTX PRO 6000
–Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000DetailsOur GPU89.092.7%11.081.5%2.5% false alarms0.52 sp90 0.83 s1 at a time577 sRTX PRO 6000
–KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetailsOur GPU88.892.6%5.783.6%3.0% false alarms2.54 sp90 3.88 s1 at a time2,613 sDGX Spark
–Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetailsOur GPU88.192.1%7.079.3%2.1% false alarms2.67 sp90 3.93 s1 at a time2,824 sDGX Spark
–KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000DetailsOur GPU87.791.8%7.286.3%3.6% false alarms0.39 sp90 0.61 s1 at a time449 sRTX PRO 6000
–REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000DetailsOur GPU87.091.3%4.867.5%1.2% false alarms0.68 sp90 1.05 s1 at a time745 sRTX PRO 6000
–Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetailsOur GPU86.491.0%6.766.3%1.7% false alarms1.12 sp90 1.96 s1 at a time1,329 sDGX Spark
–Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000DetailsOur GPU86.290.8%2.882.7%3.5% false alarms0.22 sp90 0.34 s1 at a time245 sRTX PRO 6000
–GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetailsOur GPU86.190.7%3.674.8%1.5% false alarms1.03 sp90 1.95 s1 at a time1,205 sDGX Spark
–Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000DetailsOur GPU85.190.1%7.063.8%0.9% false alarms0.64 sp90 1.01 s1 at a time723 sRTX PRO 6000
–GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetailsNot verified85.190.0%3.068.7%1.7% false alarms0.30 sp90 0.56 s8 at a timeAPI
–GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000DetailsOur GPU84.989.9%4.769.3%1.8% false alarms0.23 sp90 0.36 s1 at a time254 sRTX PRO 6000
–SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetailsNot verified84.989.9%2.378.1%4.1% false alarms0.39 sp90 0.41 s8 at a timeAPI
Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetailsNot verified83.789.1%No probabilities68.7%2.2% false alarms6.37 sp90 19.2 s1 at a timeAPI
–vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000DetailsOur GPU83.789.1%12.476.0%5.7% false alarms0.06 sp90 0.06 s1 at a time60 sRTX PRO 6000
–Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000DetailsOur GPU83.088.6%3.359.9%2.5% false alarms0.09 sp90 0.14 s1 at a time100 sRTX PRO 6000
–GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000DetailsOur GPU82.288.2%7.269.9%3.8% false alarms0.08 sp90 0.11 s1 at a time90 sRTX PRO 6000
–Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000DetailsOur GPU81.987.9%2.965.3%3.0% false alarms0.11 sp90 0.18 s1 at a time126 sRTX PRO 6000
–StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetailsOur GPU81.787.8%8.389.4%8.3% false alarms0.12 sp90 0.19 s1 at a time140 sDGX Spark
–Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000DetailsOur GPU81.187.4%2.564.7%3.9% false alarms0.06 sp90 0.09 s1 at a time72 sRTX PRO 6000
–KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000DetailsOur GPU80.987.2%6.861.1%3.6% false alarms0.16 sp90 0.25 s1 at a time195 sRTX PRO 6000
–KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000DetailsOur GPU80.687.1%4.562.0%3.4% false alarms0.04 sp90 0.06 s1 at a time49 sRTX PRO 6000
–Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000DetailsOur GPU80.487.0%4.965.0%4.9% false alarms0.07 sp90 0.09 s1 at a time71 sRTX PRO 6000
MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000DetailsOur GPU79.279.0–79.6min–max of 3 seeds86.2%86.0%–86.4%min–max of 3 seedsNo probabilities77.4%6.5% false alarmsmean of 3 seeds5.00 sp90 8.72 s30 at a time3 seeds pooled197 sRTX PRO 6000mean of 3 seeds
–d1 Closed · EfficientLatency 0.27 sList price not verifiedDetailsNot verified79.286.1%3.166.3%5.3% false alarms0.27 sp90 0.60 s8 at a timeAPI
–Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000DetailsOur GPU78.885.9%4.267.5%5.6% false alarms0.06 sp90 0.08 s1 at a time61 sRTX PRO 6000
–StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetailsOur GPU78.885.9%8.288.8%9.5% false alarms0.23 sp90 0.36 s1 at a time268 sDGX Spark
–Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000DetailsOur GPU78.185.4%6.769.0%6.0% false alarms0.07 sp90 0.11 s1 at a time77 sRTX PRO 6000
Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000DetailsOur GPU77.284.8%No probabilities67.5%4.5% false alarms9.38 sp90 11.9 s30 at a time326 sRTX PRO 6000
–Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000DetailsOur GPU77.284.8%2.847.7%4.0% false alarms0.06 sp90 0.09 s1 at a time65 sRTX PRO 6000
Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000DetailsOur GPU77.084.7%No probabilities52.0%2.6% false alarms5.12 sp90 15.7 s64 at a time121 sRTX PRO 6000
–Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000DetailsOur GPU76.984.6%3.741.0%1.9% false alarms0.09 sp90 0.11 s1 at a time98 sRTX PRO 6000
–Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000DetailsOur GPU76.684.4%8.351.7%3.8% false alarms0.12 sp90 0.18 s1 at a time132 sRTX PRO 6000
–KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000DetailsOur GPU76.184.1%3.650.8%3.8% false alarms0.10 sp90 0.14 s1 at a time105 sRTX PRO 6000
Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000DetailsOur GPU75.983.9%No probabilities79.0%9.0% false alarms19.6 sp90 64.5 s30 at a time1,050 sRTX PRO 6000
LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000DetailsOur GPU75.874.9–76.5min–max of 3 seeds83.9%83.3%–84.3%min–max of 3 seedsNo probabilities63.3%4.1% false alarmsmean of 3 seeds5.57 sp90 16.4 s30 at a time3 seeds pooled268 sRTX PRO 6000mean of 3 seeds
–Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000DetailsOur GPU75.483.6%3.446.8%3.7% false alarms0.05 sp90 0.07 s1 at a time54 sRTX PRO 6000
–KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000DetailsOur GPU75.383.5%4.535.9%1.4% false alarms0.10 sp90 0.16 s1 at a time128 sRTX PRO 6000
Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000DetailsOur GPU75.174.1–75.9min–max of 3 seeds83.4%82.7%–84.0%min–max of 3 seedsNo probabilities64.3%4.3% false alarmsmean of 3 seeds6.13 sp90 9.70 s30 at a time3 seeds pooled234 sRTX PRO 6000mean of 3 seeds
LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000DetailsOur GPU74.973.5–75.6min–max of 3 seeds83.3%82.3%–83.8%min–max of 3 seedsNo probabilities62.6%3.9% false alarmsmean of 3 seeds7.62 sp90 22.3 s64 at a time3 seeds pooled180 sRTX PRO 6000mean of 3 seeds
–Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000DetailsOur GPU74.382.8%3.239.8%2.8% false alarms0.07 sp90 0.10 s1 at a time78 sRTX PRO 6000
–Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000DetailsOur GPU72.981.9%6.529.2%2.1% false alarms0.09 sp90 0.13 s1 at a time93 sRTX PRO 6000
Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000DetailsOur GPU71.180.8%No probabilities49.5%5.1% false alarms4.02 sp90 5.85 s64 at a time69 sRTX PRO 6000
Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000DetailsOur GPU70.380.2%No probabilities55.3%4.3% false alarms36.8 sp90 66.9 s64 at a time636 sRTX PRO 6000
–Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000DetailsOur GPU69.879.8%5.937.7%3.9% false alarms0.15 sp90 0.18 s1 at a time156 sRTX PRO 6000
Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000DetailsOur GPU67.778.4%No probabilities38.0%0.7% false alarms32.6 sp90 1640.2 s64 at a time4,327 sRTX PRO 6000
MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000DetailsOur GPU67.066.9–67.0min–max of 3 seeds78.0%77.96%–78.01%min–max of 3 seedsNo probabilities28.8%2.7% false alarmsmean of 3 seeds0.86 sp90 3.48 s64 at a time3 seeds pooled51 sRTX PRO 6000mean of 3 seeds
Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000DetailsOur GPU66.277.5%No probabilities73.2%6.2% false alarms37.8 sp90 96.1 s64 at a time741 sRTX PRO 6000
–StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetailsOur GPU66.077.3%7.267.5%14.0% false alarms0.06 sp90 0.09 s1 at a time78 sDGX Spark
–GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000DetailsOur GPU65.677.1%4.060.8%10.8% false alarms0.04 sp90 0.04 s1 at a time44 sRTX PRO 6000
Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000DetailsOur GPU64.664.5–64.8min–max of 3 seeds76.4%76.3%–76.6%min–max of 3 seedsNo probabilities61.2%7.6% false alarmsmean of 3 seeds3.49 sp90 20.4 s64 at a time3 seeds pooled148 sRTX PRO 6000mean of 3 seeds
–Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000DetailsOur GPU61.774.5%6.823.1%3.4% false alarms0.12 sp90 0.19 s1 at a time134 sRTX PRO 6000
–KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetailsOur GPU61.174.0%2.716.4%2.7% false alarms0.11 sp90 0.17 s1 at a time130 sDGX Spark
–GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetailsOur GPU60.873.9%3.831.3%6.6% false alarms0.05 sp90 0.07 s1 at a time63 sDGX Spark
–Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000DetailsOur GPU60.373.6%5.04.6%0.8% false alarms0.04 sp90 0.05 s1 at a time43 sRTX PRO 6000
LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000DetailsOur GPU57.771.8%No probabilities8.8%0.8% false alarms9.58 sp90 15.2 s30 at a time352 sRTX PRO 6000
Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000DetailsOur GPU55.254.5–55.8min–max of 3 seeds70.1%69.7%–70.6%min–max of 3 seedsNo probabilities7.7%1.6% false alarmsmean of 3 seeds0.18 sp90 5.42 s64 at a time3 seeds pooled33 sRTX PRO 6000mean of 3 seeds
Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000DetailsOur GPU54.869.8%No probabilities6.1%1.3% false alarms0.19 sp90 3.31 s64 at a time18 sRTX PRO 6000
Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000DetailsOur GPU50.249.3–50.8min–max of 3 seeds66.8%66.2%–67.2%min–max of 3 seedsNo probabilities34.4%7.8% false alarmsmean of 3 seeds1.53 sp90 13.7 s64 at a time3 seeds pooled92 sRTX PRO 6000mean of 3 seeds
–OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetailsOur GPU50.066.7%4.133.4%14.4% false alarms0.03 sp90 0.04 s1 at a time43 sDGX Spark
–d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000DetailsOur GPU49.666.4%7.285.7%32.2% false alarms0.02 sp90 0.03 s1 at a time25 sRTX PRO 6000
–StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetailsOur GPU48.965.9%5.672.3%26.5% false alarms0.04 sp90 0.06 s1 at a time53 sDGX Spark
–GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetailsOur GPU44.462.9%7.731.0%16.8% false alarms0.03 sp90 0.04 s1 at a time41 sDGX Spark
–GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetailsOur GPU44.362.9%9.834.9%10.6% false alarms0.07 sp90 0.18 s1 at a time128 sDGX Spark
Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000DetailsOur GPU40.640.6–40.8min–max of 3 seeds60.4%60.4%–60.5%min–max of 3 seedsNo probabilities19.9%2.0% false alarmsmean of 3 seeds3.03 sp90 7.76 s30 at a time3 seeds pooled209 sRTX PRO 6000mean of 3 seeds
–KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetailsOur GPU38.559.0%14.610.0%3.6% false alarms0.07 sp90 0.10 s1 at a time81 sDGX Spark

No row of this kind yet.

2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology

GPU time lower is better

View
  1. Gemma 4 E2B thinking off · chat18 s
  2. d1-2B25 s
  3. Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat0.03% invalid, mean of 3 seeds33 s
  4. GLiDE 0.4B41 s
  5. OneJev1.6-0.8B43 s
  6. Strands Decider 2B43 s
  7. GLiDE 2B44 s
  8. KnowLine 4B Gen5.549 s
  9. MiniCPM5-2B thinking off · chat0.2% invalid, mean of 3 seeds51 s
  10. StartLux-Decision-v1.1-0.8B-Instant53 s
  11. Verdin-4B-v1.154 s
  12. vera-medium-omni60 s
  13. Perplexity Decider Flash v1.361 s
  14. GLiDE 0.8B63 s
  15. Perplexity Decider Flash v1.265 s
  16. Gemma 4 E2B thinking on · chat69 s
  17. Adjudex 9B v571 s
  18. Dexy-Decision-9B-V272 s
  19. Chinese Decision Pointer 4B77 s
  20. StartLux-Decision-v1.1-2B-Instant78 s
  21. Valen-Decision-1.5-9B78 s
  22. KnowLine-0.8B-Gen181 s
  23. GLiDE 9B90 s
  24. Qwen3.5-2B thinking off · chat2.0% invalid, mean of 3 seeds92 s
  25. Cygnet93 s
  26. Elephant-198 s
  27. Jeffan-9B-1211100 s
  28. KnowLine-9B-Gen1105 s
  29. Gemma 4 12B Unified thinking off · chat121 s
  30. Cerno 9B v1.2126 s
  31. GLiClass v4 Edge128 s
  32. KnowLine-9B-Gen2128 s
  33. KnowLine-2B-Gen1130 s
  34. Winnow-12B132 s
  35. Metask-Jev-4B134 s
  36. StartLux-Decision-v1.1-4B-Instant140 s
  37. Qwen3.5-4B thinking off · chat1.0% invalid, mean of 3 seeds148 s
  38. Clef-Flash156 s
  39. LFM3-1.7B thinking off · chat0.1% invalid, mean of 3 seeds180 s
  40. KnowLine-12B-Gen1195 s
  41. MiniCPM5-2B thinking on · chat0.1% invalid, mean of 3 seeds197 s
  42. Raon-1 nano thinking off · chat0.03% invalid, mean of 3 seeds209 s
  43. Karpo Decision 31B230 s
  44. Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat1.5% invalid, mean of 3 seeds234 s
  45. Perplexity Decider 27B v1.3245 s
  46. GLiDE v1.2254 s
  47. StartLux-Decision-v1.1-9B-Instant268 s
  48. LFM3-1.7B thinking on · chat0.1% invalid, mean of 3 seeds268 s
  49. Gemma 4 E4B thinking on · chat2 invalid326 s
  50. LFM2.5-8B-A1B thinking on (no off mode) · chat352 s
  51. KnowLine-31B-Gen2449 s
  52. Copernicus-31B577 s
  53. Qwen3.5-2B thinking on · chat78 invalid636 s
  54. Logits Sigma-1 Duo723 s
  55. Qwen3.5-4B thinking on · chat179 invalid741 s
  56. REAX 1.3745 s
  57. Granite-4.2-3B thinking on · chat1,050 s
  58. GLiDE 27B1,205 s
  59. Turion Jev 31B1,329 s
  60. KnowLine-31B-Gen1.52,613 s
  61. Sieve 31B v32,824 s
  62. Gemma 4 12B Unified thinking on · chat296 invalid4,327 s
  63. d1.2-ultraAPI
  64. Inkling 2 Small thinking on (no off mode) · chatAPI
  65. GLiDE APIAPI
  66. SubQ Snap 1.1API
  67. d1-ultraAPI
  68. Microsoft-Decision-1API
  69. Gemini 4 Hydrogen thinking on (no off mode) · chatAPI
  70. Jev 1.14API
  71. Jev 1.13API
  72. d1.1API
  73. d1API
  74. Gemini Decision 1 20261111API
  75. Decider 1API

No row of this kind yet.

  • Google
  • Liquid AI
  • Fastino
  • OmniJev
  • Strands Agents
  • PelaAI
  • OpenBMB
  • StartLux
  • victorbennett
  • scar-ai
  • Perplexity
  • newsbang
  • Patrick-18
  • quincyqiang
  • Valen-Team
  • Alibaba
  • Blockbrain Labs
  • Base Labs
  • tattrongvu
  • Apertix AI
  • Knowledgator
  • EldanRing
  • Metask Lab
  • Cloudflare
  • KRAFTON
  • MachinePulse
  • Poindexter Labs
  • Logits
  • REAX
  • IBM
  • Turion.AI
  • Sthānika AI
  • Thinking Machines
  • Subquadratic
  • Microsoft
  • TypeSafe
  • meraGPT
The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –.ModelMeasured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs(accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairsExpected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, pointsOf the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarmsClient wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under itInput tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairs
Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details18 sRTX PRO 600054.869.8%No probabilities6.1%1.3% false alarms0.19 sp90 3.31 s64 at a timeOur GPU
1d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details25 sRTX PRO 600049.666.4%7.285.7%32.2% false alarms0.02 sp90 0.03 s1 at a timeOur GPU
Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details33 sRTX PRO 6000 · mean of 3 seeds55.254.5–55.8min–max of 3 seeds70.1%69.7%–70.6%min–max of 3 seedsNo probabilities7.7%1.6% false alarmsmean of 3 seeds0.18 sp90 5.42 s64 at a time3 seeds pooledOur GPU
2GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails41 sDGX Spark44.462.9%7.731.0%16.8% false alarms0.03 sp90 0.04 s1 at a timeOur GPU
3OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails43 sDGX Spark50.066.7%4.133.4%14.4% false alarms0.03 sp90 0.04 s1 at a timeOur GPU
4Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details43 sRTX PRO 600060.373.6%5.04.6%0.8% false alarms0.04 sp90 0.05 s1 at a timeOur GPU
5GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details44 sRTX PRO 600065.677.1%4.060.8%10.8% false alarms0.04 sp90 0.04 s1 at a timeOur GPU
6KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details49 sRTX PRO 600080.687.1%4.562.0%3.4% false alarms0.04 sp90 0.06 s1 at a timeOur GPU
MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details51 sRTX PRO 6000 · mean of 3 seeds67.066.9–67.0min–max of 3 seeds78.0%77.96%–78.01%min–max of 3 seedsNo probabilities28.8%2.7% false alarmsmean of 3 seeds0.86 sp90 3.48 s64 at a time3 seeds pooledOur GPU
7StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails53 sDGX Spark48.965.9%5.672.3%26.5% false alarms0.04 sp90 0.06 s1 at a timeOur GPU
8Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details54 sRTX PRO 600075.483.6%3.446.8%3.7% false alarms0.05 sp90 0.07 s1 at a timeOur GPU
9vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details60 sRTX PRO 600083.789.1%12.476.0%5.7% false alarms0.06 sp90 0.06 s1 at a timeOur GPU
10Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details61 sRTX PRO 600078.885.9%4.267.5%5.6% false alarms0.06 sp90 0.08 s1 at a timeOur GPU
11GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails63 sDGX Spark60.873.9%3.831.3%6.6% false alarms0.05 sp90 0.07 s1 at a timeOur GPU
12Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details65 sRTX PRO 600077.284.8%2.847.7%4.0% false alarms0.06 sp90 0.09 s1 at a timeOur GPU
Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details69 sRTX PRO 600071.180.8%No probabilities49.5%5.1% false alarms4.02 sp90 5.85 s64 at a timeOur GPU
13Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details71 sRTX PRO 600080.487.0%4.965.0%4.9% false alarms0.07 sp90 0.09 s1 at a timeOur GPU
14Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details72 sRTX PRO 600081.187.4%2.564.7%3.9% false alarms0.06 sp90 0.09 s1 at a timeOur GPU
15Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details77 sRTX PRO 600078.185.4%6.769.0%6.0% false alarms0.07 sp90 0.11 s1 at a timeOur GPU
16StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails78 sDGX Spark66.077.3%7.267.5%14.0% false alarms0.06 sp90 0.09 s1 at a timeOur GPU
17Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details78 sRTX PRO 600074.382.8%3.239.8%2.8% false alarms0.07 sp90 0.10 s1 at a timeOur GPU
18KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails81 sDGX Spark38.559.0%14.610.0%3.6% false alarms0.07 sp90 0.10 s1 at a timeOur GPU
19GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details90 sRTX PRO 600082.288.2%7.269.9%3.8% false alarms0.08 sp90 0.11 s1 at a timeOur GPU
Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details92 sRTX PRO 6000 · mean of 3 seeds50.249.3–50.8min–max of 3 seeds66.8%66.2%–67.2%min–max of 3 seedsNo probabilities34.4%7.8% false alarmsmean of 3 seeds1.53 sp90 13.7 s64 at a time3 seeds pooledOur GPU
20Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details93 sRTX PRO 600072.981.9%6.529.2%2.1% false alarms0.09 sp90 0.13 s1 at a timeOur GPU
21Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details98 sRTX PRO 600076.984.6%3.741.0%1.9% false alarms0.09 sp90 0.11 s1 at a timeOur GPU
22Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details100 sRTX PRO 600083.088.6%3.359.9%2.5% false alarms0.09 sp90 0.14 s1 at a timeOur GPU
23KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details105 sRTX PRO 600076.184.1%3.650.8%3.8% false alarms0.10 sp90 0.14 s1 at a timeOur GPU
Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details121 sRTX PRO 600077.084.7%No probabilities52.0%2.6% false alarms5.12 sp90 15.7 s64 at a timeOur GPU
24Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details126 sRTX PRO 600081.987.9%2.965.3%3.0% false alarms0.11 sp90 0.18 s1 at a timeOur GPU
25GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails128 sDGX Spark44.362.9%9.834.9%10.6% false alarms0.07 sp90 0.18 s1 at a timeOur GPU
26KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details128 sRTX PRO 600075.383.5%4.535.9%1.4% false alarms0.10 sp90 0.16 s1 at a timeOur GPU
27KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails130 sDGX Spark61.174.0%2.716.4%2.7% false alarms0.11 sp90 0.17 s1 at a timeOur GPU
28Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details132 sRTX PRO 600076.684.4%8.351.7%3.8% false alarms0.12 sp90 0.18 s1 at a timeOur GPU
29Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details134 sRTX PRO 600061.774.5%6.823.1%3.4% false alarms0.12 sp90 0.19 s1 at a timeOur GPU
30StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails140 sDGX Spark81.787.8%8.389.4%8.3% false alarms0.12 sp90 0.19 s1 at a timeOur GPU
Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details148 sRTX PRO 6000 · mean of 3 seeds64.664.5–64.8min–max of 3 seeds76.4%76.3%–76.6%min–max of 3 seedsNo probabilities61.2%7.6% false alarmsmean of 3 seeds3.49 sp90 20.4 s64 at a time3 seeds pooledOur GPU
31Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details156 sRTX PRO 600069.879.8%5.937.7%3.9% false alarms0.15 sp90 0.18 s1 at a timeOur GPU
LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details180 sRTX PRO 6000 · mean of 3 seeds74.973.5–75.6min–max of 3 seeds83.3%82.3%–83.8%min–max of 3 seedsNo probabilities62.6%3.9% false alarmsmean of 3 seeds7.62 sp90 22.3 s64 at a time3 seeds pooledOur GPU
32KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details195 sRTX PRO 600080.987.2%6.861.1%3.6% false alarms0.16 sp90 0.25 s1 at a timeOur GPU
MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details197 sRTX PRO 6000 · mean of 3 seeds79.279.0–79.6min–max of 3 seeds86.2%86.0%–86.4%min–max of 3 seedsNo probabilities77.4%6.5% false alarmsmean of 3 seeds5.00 sp90 8.72 s30 at a time3 seeds pooledOur GPU
Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details209 sRTX PRO 6000 · mean of 3 seeds40.640.6–40.8min–max of 3 seeds60.4%60.4%–60.5%min–max of 3 seedsNo probabilities19.9%2.0% false alarmsmean of 3 seeds3.03 sp90 7.76 s30 at a time3 seeds pooledOur GPU
33Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details230 sRTX PRO 600090.093.3%2.880.8%1.7% false alarms0.22 sp90 0.31 s1 at a timeOur GPU
Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details234 sRTX PRO 6000 · mean of 3 seeds75.174.1–75.9min–max of 3 seeds83.4%82.7%–84.0%min–max of 3 seedsNo probabilities64.3%4.3% false alarmsmean of 3 seeds6.13 sp90 9.70 s30 at a time3 seeds pooledOur GPU
34Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details245 sRTX PRO 600086.290.8%2.882.7%3.5% false alarms0.22 sp90 0.34 s1 at a timeOur GPU
35GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details254 sRTX PRO 600084.989.9%4.769.3%1.8% false alarms0.23 sp90 0.36 s1 at a timeOur GPU
36StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails268 sDGX Spark78.885.9%8.288.8%9.5% false alarms0.23 sp90 0.36 s1 at a timeOur GPU
LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details268 sRTX PRO 6000 · mean of 3 seeds75.874.9–76.5min–max of 3 seeds83.9%83.3%–84.3%min–max of 3 seedsNo probabilities63.3%4.1% false alarmsmean of 3 seeds5.57 sp90 16.4 s30 at a time3 seeds pooledOur GPU
Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details326 sRTX PRO 600077.284.8%No probabilities67.5%4.5% false alarms9.38 sp90 11.9 s30 at a timeOur GPU
LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details352 sRTX PRO 600057.771.8%No probabilities8.8%0.8% false alarms9.58 sp90 15.2 s30 at a timeOur GPU
37KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details449 sRTX PRO 600087.791.8%7.286.3%3.6% false alarms0.39 sp90 0.61 s1 at a timeOur GPU
38Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details577 sRTX PRO 600089.092.7%11.081.5%2.5% false alarms0.52 sp90 0.83 s1 at a timeOur GPU
Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details636 sRTX PRO 600070.380.2%No probabilities55.3%4.3% false alarms36.8 sp90 66.9 s64 at a timeOur GPU
39Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details723 sRTX PRO 600085.190.1%7.063.8%0.9% false alarms0.64 sp90 1.01 s1 at a timeOur GPU
Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details741 sRTX PRO 600066.277.5%No probabilities73.2%6.2% false alarms37.8 sp90 96.1 s64 at a timeOur GPU
40REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details745 sRTX PRO 600087.091.3%4.867.5%1.2% false alarms0.68 sp90 1.05 s1 at a timeOur GPU
Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details1,050 sRTX PRO 600075.983.9%No probabilities79.0%9.0% false alarms19.6 sp90 64.5 s30 at a timeOur GPU
41GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails1,205 sDGX Spark86.190.7%3.674.8%1.5% false alarms1.03 sp90 1.95 s1 at a timeOur GPU
42Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails1,329 sDGX Spark86.491.0%6.766.3%1.7% false alarms1.12 sp90 1.96 s1 at a timeOur GPU
43KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails2,613 sDGX Spark88.892.6%5.783.6%3.0% false alarms2.54 sp90 3.88 s1 at a timeOur GPU
44Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails2,824 sDGX Spark88.192.1%7.079.3%2.1% false alarms2.67 sp90 3.93 s1 at a timeOur GPU
Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details4,327 sRTX PRO 600067.778.4%No probabilities38.0%0.7% false alarms32.6 sp90 1640.2 s64 at a timeOur GPU
–d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000DetailsAPI86.491.0%8.885.4%4.4% false alarms0.23 sp90 0.35 s8 at a time$0.035
Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000DetailsAPI86.290.8%No probabilities73.6%2.4% false alarms2.70 sp90 16.4 s1 at a time$1.49
–GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetailsAPI85.190.0%3.068.7%1.7% false alarms0.30 sp90 0.56 s8 at a timeNot verified
–SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetailsAPI84.989.9%2.378.1%4.1% false alarms0.39 sp90 0.41 s8 at a timeNot verified
–d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000DetailsAPI84.789.8%11.480.8%4.5% false alarms0.48 sp90 0.90 s8 at a time$0.035
–Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000DetailsAPI83.989.3%2.482.4%6.5% false alarms0.32 sp90 0.38 s8 at a time$0.036
Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetailsAPI83.789.1%No probabilities68.7%2.2% false alarms6.37 sp90 19.2 s1 at a timeNot verified
–Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000DetailsAPI83.689.1%2.872.0%3.5% false alarms0.18 sp90 0.20 s8 at a time$0.049
–Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000DetailsAPI83.088.6%1.979.9%6.0% false alarms0.17 sp90 0.19 s8 at a time$0.049
–d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000DetailsAPI82.588.4%8.770.5%4.1% false alarms0.25 sp90 0.45 s8 at a time$0.017
–d1 Closed · EfficientLatency 0.27 sList price not verifiedDetailsAPI79.286.1%3.166.3%5.3% false alarms0.27 sp90 0.60 s8 at a timeNot verified
–Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000DetailsAPI77.084.7%3.951.1%1.8% false alarms0.20 sp90 0.29 s8 at a time$0.093
–Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000DetailsAPI43.362.2%5.931.9%17.6% false alarms0.65 sp90 0.75 s8 at a time$0.035

No row of this kind yet.

2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology

Details of each setup

Karpo Decision 31B · MachinePulse

Open · 31B

MachinePulseAI/Karpo Decision 31B@d69cdd57, its card's karpo-decision package (machinepulse-ai/karpo-decision@ac40d399) System One server, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 98.0% of 942, a conflicting one 80.8% of 329, none 93.0% of 798

Copernicus-31B · Poindexter Labs

Open · 31B

Poindexter-Labs/Copernicus-31B@be715259, its card's copernicus_server.py (Transformers backend, the card's defaults), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 98.8% of 942, a conflicting one 81.5% of 329, none 90.0% of 798

KnowLine-31B-Gen1.5 · PelaAI

Open · 31B

PelaAI/KnowLine-31B-Gen1.5@8e56aa1b, the card's serve_knowline.sh (SGLang), BF16, on one DGX Spark (ours) as for its published runs, self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 1 published-cohort pair was sent in test requests whose outputs were not kept, so its scored row comes from a later request

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 97.2% of 942, a conflicting one 83.6% of 329, none 90.7% of 798

Sieve 31B v3 · Sthānika AI

Open · 31B

sthanika-ai/Sieve-31B-v3@8fcee858, sieve-serve 0.5.1 (text only, design 'letters'), BF16, on one DGX Spark (ours) as for its published runs, self-hosted; the server reports no input-token count; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 98.6% of 942, a conflicting one 79.3% of 329, none 89.6% of 798

KnowLine-31B-Gen2 · PelaAI

Open · 31B

PelaAI/KnowLine-31B-Gen2@fd99a9ed, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card names 0.5.21, which we do not have), BF16, SHA256SUMS checked, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); the run overlapped a host GPU recovery warning; its serving process continued, and the run completed without interruption or resume (producer report)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 96.8% of 942, a conflicting one 86.3% of 329, none 88.1% of 798

REAX 1.3 · REAX

Open · 57B

decisionmodels/REAX-1.3@fa4bf7ee, REAX Engine reax13 server on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 96.2% of 942, a conflicting one 67.5% of 329, none 95.4% of 798

d1.2-ultra · Liquid AI

Closed · Efficient

Liquid AI decisions API (api.liquid.ai/decisions, model d1.2-ultra); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 91.9% of 942, a conflicting one 85.4% of 329, none 92.1% of 798

List price $0.042 per 1M input tokens, its source (2026-11-16)

Turion Jev 31B · Turion.AI

Open · 31B

turion-ai/turion-jev-31b@866c0fc6, the card's turion_jev.serve on vLLM 0.32.0 (BF16 weights, FP8 at load, text only), on one DGX Spark (ours) as for its published runs, self-hosted; vLLM memory share 0.80 instead of the card's 0.92 (KV cache only); one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 96.8% of 942, a conflicting one 66.3% of 329, none 94.2% of 798

Inkling 2 Small · Thinking Machines

594B MoE · 17B active

OpenRouter thinkingmachines/inkling-2-small (paid per token; providers with no data retention and no training on inputs), reasoning at the route's default; billed per token at OpenRouter's list price; one question per request, 1 request at a time (latency not comparable across different concurrency); list prices $0.30 per 1M input and $1.00 per 1M output tokens (OpenRouter, 16 Nov); answered by OpenRouter's providers Together 1,104, Cloudflare 317, PrimeIntellect 242, Nebius 201, DeepInfra 179, Modal 26; OpenRouter's reported charges for the 2,069 billed requests of this model: $2.8921 (actual billing, distinct from the list-price figure); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 97.2% of 942, a conflicting one 73.6% of 329, none 90.3% of 798

List price $0.3 per 1M input tokens, its source (2026-11-16)

Perplexity Decider 27B v1.3 · Perplexity

Open · 27B

perplexity-ai/pplx-decider-v1.3@9c2ebb27, the runner's pplx_decider_server.py with the checkpoint's own pplx_decider package (as for its published v1.2 runs), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 95.1% of 942, a conflicting one 82.7% of 329, none 89.1% of 798

GLiDE 27B · Fastino

Open · 27B

fastino/GLiDE-27B@4d32743a, GLiDE server (glide.server), BF16, GLIDE_MAX_BATCH=1, on one DGX Spark (ours) as for its published runs, self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 97.1% of 942, a conflicting one 74.8% of 329, none 89.7% of 798

Logits Sigma-1 Duo · Logits

Open · 57B

logitshq/Logits-Sigma-1-Duo@a0c1b6e8 (weights and code identical to d9641cd8, the revision its card's run names; two members averaged per option, FP8 weights), its card's serving image (code/Dockerfile: torch 2.15, transformers 5.22.1, causal-conv1d 1.7.0, kit decision-index@25c6ec7) plus only gcc, libc6-dev and python3-dev, which its Triton kernels need to compile a launcher at first use and the image lacks, with the engine and option its serve.sh fixes (FP8 matmul on its own Triton kernel: its cuBLAS block-scaled path was not taken on this GPU, a fallback the engine makes itself), behind di_serve.py, our System One wrapper of the engine, on one RTX PRO 6000 (ours), self-hosted; text only (the load report lists FP8 scales missing for member A's vision tower, which text requests do not use; not checked on images); one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 94.4% of 942, a conflicting one 63.8% of 329, none 95.9% of 798

GLiDE · Fastino

Closed · Efficient

Fastino decisions API (api.fastino.ai, model fastino/GLiDE); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 96.4% of 942, a conflicting one 68.7% of 329, none 91.3% of 798

GLiDE · Fastino

Open · 27B

fastino/GLiDE@a7e4589, GLiDE server (Fastino's release code), BF16, GLIDE_MAX_BATCH=1, on one RTX PRO 6000 (ours) as the backgammon league's pinned server, self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 96.6% of 942, a conflicting one 69.3% of 329, none 90.6% of 798

SubQ Snap 1.1 · Subquadratic

Closed · Efficient

SubQ decisions API (api.subq.ai, model subq-snap-1.1); its probabilities were byte-identical to SubQ Snap Preview's on all 2,069 pairs; one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 91.9% of 942, a conflicting one 78.1% of 329, none 92.5% of 798

d1-ultra · Liquid AI

Closed · Efficient

Liquid AI decisions API (api.liquid.ai); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 95.7% of 942, a conflicting one 80.8% of 329, none 86.6% of 798

List price $0.042 per 1M input tokens, its source (2026-10-16)

Microsoft-Decision-1 · Microsoft

Closed · Efficient

OpenRouter decisions API (openrouter.ai/api, model microsoft/microsoft-decision-1); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 89.5% of 942, a conflicting one 82.4% of 329, none 91.8% of 798

List price $0.042 per 1M input tokens, its source (2026-10-17)

Gemini 4 Hydrogen · Google

Closed · Workhorse

Google Antigravity agent harness through Can's agbench bridge (Google AI Ultra subscription), thinking high, tool descriptions present and tools denied, a fresh conversation per request; $0 billed (the subscription's quota); one question per request, 1 request at a time (latency not comparable across different concurrency); the pass stopped once when the bridge's safety switch tripped on a false positive (fixed by its owner) and resumed in the same order by a recorded decision; the one request in flight at the stop was sent again, so its recorded outcome is not guaranteed to be the model's first output; thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 88.4% of 942, a conflicting one 68.7% of 329, none 98.4% of 798

vera-medium-omni · scar-ai

Open · 4.2B

scar-ai/vera-medium-omni@1f23a6cf, its card's own server (vera-serve from vera-s1 0.1.8, PyTorch engine), BF16, on one RTX PRO 6000 (ours), self-hosted; server probabilities are rounded to four decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 92.8% of 942, a conflicting one 76.0% of 329, none 90.2% of 798

Jev 1.14 · TypeSafe

Closed · Efficient

TypeSafe decisions API (api.typesafe.ai, model jev-1.14.0); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 94.1% of 942, a conflicting one 72.0% of 329, none 90.2% of 798

List price $0.042 per 1M input tokens, its source (2026-10-16)

Jeffan-9B-1211 · tattrongvu

Open · 9.0B

tattrongvu/Jeffan-9B-1211@862d1a7a, its card's Decision Index engine (jeffan_engine:DecisionEngine, transformers 5.17.0, flash-linear-attention 0.5.2, kit decision-index@f8552a1, options max_len and budget 65,536 as its card; without the card's optional causal-conv1d, so its PyTorch reference convolution) behind di_serve.py, our System One wrapper of the engine, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 91.7% of 942, a conflicting one 59.9% of 329, none 96.9% of 798

Jev 1.13 · TypeSafe

Closed · Efficient

TypeSafe decisions API (api.typesafe.ai, model jev-1.13.0); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 94.3% of 942, a conflicting one 79.9% of 329, none 85.6% of 798

List price $0.042 per 1M input tokens, its source (2026-10-08)

d1.1 · Liquid AI

Closed · Efficient

Liquid AI decisions API (api.liquid.ai/decisions, model d1.1); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 92.5% of 942, a conflicting one 70.5% of 329, none 90.8% of 798

List price $0.02 per 1M input tokens, its source (2026-10-24)

GLiDE 9B · Fastino

Open · 7.9B

fastino/GLiDE-9B@ab54df5a, GLiDE server (python -m glide.server) in vLLM v0.30.0 image with flash-linear-attention 0.5.2, as for GLiDE 2B, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 94.6% of 942, a conflicting one 69.9% of 329, none 88.1% of 798

Cerno 9B v1.2 · Apertix AI

Open · 9.0B

apertixai/cerno-9b-v1.2@f02a632f, its card's Decision Index engine (CernoEngine, transformers 5.17.0, flash-linear-attention 0.5.2, kit decision-index@9eb2dbe2, the checkpoint's decider_config.json settings, loaded by Hub id and that revision as its card runs it, from the local copy; causal-conv1d not installed, so its PyTorch reference convolution) behind di_serve.py, our System One wrapper of the engine, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 91.2% of 942, a conflicting one 65.3% of 329, none 93.4% of 798

StartLux-Decision-v1.1-4B-Instant · StartLux

Open · 4.0B

startlux-models/StartLux-Decision-v1.1-4B-Instant@416088f7, its package's own server (startlux-decide serve --launch-vllm, StartLux-Decision@a99977d, vLLM 0.31.0) as for its published Reading and Logic & Math runs, BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 90.2% of 942, a conflicting one 89.4% of 329, none 84.2% of 798

Dexy-Decision-9B-V2 · Patrick-18

Open · 9.0B

Patrick-18/Dexy-Decision-9B-V2@4f434e2b, its card's Transformers service (d3_server.py, in the environment its requirements.txt pins: torch 2.14.0, transformers 5.19.0), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 92.7% of 942, a conflicting one 64.7% of 329, none 90.5% of 798

KnowLine-12B-Gen1 · PelaAI

Open · 12B

PelaAI/KnowLine-12B-Gen1@f26fbc2d, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card names 0.5.21, which we do not have), BF16, SHA256SUMS checked, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 93.3% of 942, a conflicting one 61.1% of 329, none 90.8% of 798

KnowLine 4B Gen5.5 · PelaAI

Open · 4.0B

PelaAI/KnowLine-4B-Gen5.5@779944d2 (the site's pin; no longer in the maker's repository, copied from the snapshot its published runs used, SHA256SUMS checked), its card's serve_knowline.sh (SGLang with FP8 at load, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23, on one RTX PRO 6000 (ours), self-hosted; its published runs were on a DGX Spark; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 90.5% of 942, a conflicting one 62.0% of 329, none 93.4% of 798

Adjudex 9B v5 · newsbang

Open · 9.0B

newsbang/adjudex-9b-v5@ffdcb825, its card's autojev server (the Perplexity Decider v1.1 source, in the environment of its requirements), default settings, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 92.2% of 942, a conflicting one 65.0% of 329, none 89.7% of 798

MiniCPM5-2B · OpenBMB

Open · 2.5B

openbmb/MiniCPM5-2B@42433e90, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.1% (per seed in seed_runs; cut off at the token limit per seed: 0, 0, 0); the three runs are fresh runs (14 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the runs they replace and the server settings recorded for them; the earlier runs stay in history; gpu_hours is the mean of the three runs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0.1% of 2,069 on average (seed 0: 3; seed 1: 0; seed 2: 1), counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label, the mean of its seeds: the reference answer 89.8% of 942, a conflicting one 77.4% of 329, none 85.4% of 798

Seeds: seed 0: grading:chat-minicpm5-2b-seed0-r2-v1, 86.0% right, 3 invalid, 0 cut off, 250 wrong answers caught; seed 1: grading:chat-minicpm5-2b-seed1-r2-v1, 86.1% right, 0 invalid, 0 cut off, 258 wrong answers caught; seed 2: grading:chat-minicpm5-2b-seed2-r2-v1, 86.4% right, 1 invalid, 0 cut off, 256 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval

Measures: mean of 3 seeds; latency over all requests of the three runs

Outside the three: its first run, grading:chat-minicpm5-2b-v1, asked 64 requests at a time, where the three seed runs were asked 30 at a time; not ranked, kept in the file published before the seeds

d1 · Liquid AI

Closed · Efficient

Liquid AI decisions API (api.liquid.ai/decisions, model d1); billed 'd1' route, the site's 'd1:free' was withdrawn (HTTP 404, 13 Nov); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 91.1% of 942, a conflicting one 66.3% of 329, none 88.5% of 798

Perplexity Decider Flash v1.3 · Perplexity

Open · 4.0B

perplexity-ai/pplx-decider-flash-v1.3@d06fb3f2, the runner's pplx_decider_server.py with the checkpoint's own pplx_decider package (as for its published v1.2 runs), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 93.7% of 942, a conflicting one 67.5% of 329, none 84.2% of 798

StartLux-Decision-v1.1-9B-Instant · StartLux

Open · 9.0B

startlux-models/StartLux-Decision-v1.1-9B-Instant@18a17f73, its package's own server (startlux-decide serve --launch-vllm, StartLux-Decision@a99977d, vLLM 0.31.0), BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 90.3% of 942, a conflicting one 88.8% of 329, none 79.5% of 798

Chinese Decision Pointer 4B · quincyqiang

Open · 4.0B

quincyqiang/chinese-decision-pointer-4b@889f23e2, its card's Decision Index engine (yanqiangmiffy/chinese-decision@c16450b, ChineseDecisionEngine with debias 'reverse' as its card's run; transformers 5.17.0, kit decision-index@9eb2dbe2, as the card names no kit commit; causal-conv1d not installed, so its PyTorch reference convolution) behind di_serve.py, our System One wrapper of the engine, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 90.7% of 942, a conflicting one 69.0% of 329, none 86.0% of 798

Gemma 4 E4B · Google

Open · 8.0B

google/gemma-4-E4B-it@4ecc2ba6, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 2 of 2069 replies (0.10%) had no answer marker and count wrong (0 of them cut off at the 32,768-token limit); every reply kept; an earlier attempt of this same run stopped part-way (510 recorded pairs and an unknown number of further requests possibly answered but not recorded) and is kept apart, not merged; this attempt started again from the first pair, so its outcomes are this attempt's recorded outputs, not guaranteed to be the model's first outputs on those pairs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 2 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 91.2% of 942, a conflicting one 67.5% of 329, none 84.3% of 798

Perplexity Decider Flash v1.2 · Perplexity

Open · 4.0B

perplexity-ai/pplx-decider-flash-v1.2@6d771ad7, the runner's pplx_decider_server.py with the checkpoint's own pplx_decider package (as for its published v1.2 runs), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 94.1% of 942, a conflicting one 47.7% of 329, none 89.1% of 798

Gemini Decision 1 · Google

Closed · Efficient

Google Decisions API (generativelanguage.googleapis.com, model gemini-decision-1); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 96.5% of 942, a conflicting one 51.1% of 329, none 84.6% of 798

List price $0.1 per 1M input tokens, its source (2026-11-13)

Gemma 4 12B Unified · Google

Open · 12B

google/gemma-4-12B-it@43c9d014, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking off; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 97.4% of 942, a conflicting one 52.0% of 329, none 83.2% of 798

Elephant-1 · Base Labs

Open · 9.0B

baselabs/elephant-1@b3f8a934, the runner's baselabs_server.py in the card's setup.sh environment (OneJev 81ce62f1 with the checkpoint's serve.patch), on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 93.7% of 942, a conflicting one 41.0% of 329, none 91.7% of 798

Winnow-12B · EldanRing

Open · 12B

EldanRing/Winnow-12B@b2b14213, its Q8_0 GGUF served by winnow-inference@d4631fb, the server its Logic & Math runs use, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 94.3% of 942, a conflicting one 51.7% of 329, none 86.2% of 798

KnowLine-9B-Gen1 · PelaAI

Open · 9.0B

PelaAI/KnowLine-9B-Gen1@325378d1, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card's release launcher names 0.5.21, which we do not have), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 84.4% of 942, a conflicting one 50.8% of 329, none 97.5% of 798

Granite-4.2-3B · IBM

Open · 3.0B

ibm-granite/granite-4.2-3b@4e9c275e, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 88.3% of 942, a conflicting one 79.0% of 329, none 80.7% of 798

LFM3-1.7B · Liquid AI

Open · 1.7B

LiquidAI/LFM3-1.7B@0dc4268b, BF16, cu135-nightly-be47189f37f6162991366ff6b91539f6e9d1f432 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.1% (per seed in seed_runs; cut off at the token limit per seed: 2, 1, 1); the three runs are fresh runs (14 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the runs they replace and the server settings recorded for them; seed 0 is the rerun made to diagnose the missing answer lines; the earlier runs stay in history; gpu_hours is the mean of the three runs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0.1% of 2,069 on average (seed 0: 2; seed 1: 1; seed 2: 1), counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label, the mean of its seeds: the reference answer 91.8% of 942, a conflicting one 63.3% of 329, none 83.0% of 798

Seeds: seed 0: grading:chat-lfm3-1.7b-seed0-replay-v1, 84.0% right, 2 invalid, 2 cut off, 215 wrong answers caught; seed 1: grading:chat-lfm3-1.7b-seed1-r2-v1, 83.3% right, 1 invalid, 1 cut off, 194 wrong answers caught; seed 2: grading:chat-lfm3-1.7b-seed2-r2-v1, 84.3% right, 1 invalid, 1 cut off, 216 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval

Measures: mean of 3 seeds; latency over all requests of the three runs

Outside the three: its first run, grading:chat-lfm3-1.7b-v1, asked 64 requests at a time, where the three seed runs were asked 30 at a time; not ranked, kept in the file published before the seeds

Verdin-4B-v1.1 · victorbennett

Open · 4.0B

victorbennett/Verdin-4B-v1.1@45830d21, its card's own server (python -m verdin.server, in the package versions its card lists: torch 2.8.0 cu128, transformers 5.17.0, flash-linear-attention 0.5.2, triton 3.8.0; no causal-conv1d, so its own torch conv path), BF16, on one RTX PRO 6000 (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 95.0% of 942, a conflicting one 46.8% of 329, none 85.2% of 798

KnowLine-9B-Gen2 · PelaAI

Open · 9.0B

PelaAI/KnowLine-9B-Gen2@8ec8eb60, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card names 0.5.21, which we do not have), BF16, SHA256SUMS checked, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 89.3% of 942, a conflicting one 35.9% of 329, none 96.4% of 798

Gemma 4 E4B · Google

Open · 8.0B

google/gemma-4-E4B-it@4ecc2ba6, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 1.5% (per seed in seed_runs; cut off at the token limit per seed: 0, 0, 0); three new runs (16 Nov, every reply kept), seeds 0, 1, 2 at Gemma's own sampling (its generation_config.json: temperature 1.0, top_p 0.95, top_k 64; Ümit #10554), with the weights, template and token limit of the temperature-0 row, which stays as its own row; their server settings recorded; gpu_hours is the mean of the three runs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 1.5% of 2,069 on average (seed 0: 34; seed 1: 31; seed 2: 30), counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label, the mean of its seeds: the reference answer 91.0% of 942, a conflicting one 64.3% of 329, none 82.2% of 798

Seeds: seed 0: grading:chat-gemma-4-e4b-think-sampled-seed0-v1, 84.0% right, 34 invalid, 0 cut off, 216 wrong answers caught; seed 1: grading:chat-gemma-4-e4b-think-sampled-seed1-v1, 82.7% right, 31 invalid, 0 cut off, 208 wrong answers caught; seed 2: grading:chat-gemma-4-e4b-think-sampled-seed2-v1, 83.5% right, 30 invalid, 0 cut off, 211 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval

Measures: mean of 3 seeds; latency over all requests of the three runs

LFM3-1.7B · Liquid AI

Open · 1.7B

LiquidAI/LFM3-1.7B@0dc4268b, BF16, cu135-nightly-be47189f37f6162991366ff6b91539f6e9d1f432 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.1% (per seed in seed_runs; cut off at the token limit per seed: 2, 2, 1); seed 0 is the published single run (14 Nov, every reply kept); seeds 1 and 2 are new runs (15 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0.1% of 2,069 on average (seed 0: 2; seed 1: 2; seed 2: 1), counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label, the mean of its seeds: the reference answer 92.1% of 942, a conflicting one 62.6% of 329, none 81.3% of 798

Seeds: seed 0: grading:chat-lfm3-1.7b-nothink-r2-v1, 83.8% right, 2 invalid, 2 cut off, 212 wrong answers caught; seed 1: grading:chat-lfm3-1.7b-nothink-seed1-r2-v1, 83.7% right, 2 invalid, 2 cut off, 209 wrong answers caught; seed 2: grading:chat-lfm3-1.7b-nothink-seed2-r2-v1, 82.3% right, 1 invalid, 1 cut off, 197 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval

Measures: mean of 3 seeds; latency over all requests of the three runs

Valen-Decision-1.5-9B · Valen-Team

Open · 9.0B

Valen-Team/Valen-Decision-1.5-9B@5a512117, its card's vLLM deployment (valen-vllm, the Valen plugin at Valen-Team/Valen@773d3a64 in the official vllm/vllm-openai:v0.33.0 image), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 93.4% of 942, a conflicting one 39.8% of 329, none 88.1% of 798

Cygnet · Blockbrain Labs

Open · 12B

no weights of its own: google/gemma-4-12B-it@707f0a3b, unchanged, on vLLM 0.30.0 behind cygnet-recipe's shim (blockbrain-ai/cygnet-recipe@3cf591c, temperature 3.4), the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 94.4% of 942, a conflicting one 29.2% of 329, none 89.0% of 798

Gemma 4 E2B · Google

Open · 5.1B

google/gemma-4-E2B-it@4adf1574, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 89.6% of 942, a conflicting one 49.5% of 329, none 83.2% of 798

Qwen3.5-2B · Alibaba

Open · 2.0B

Qwen/Qwen3.5-2B@15852e8c, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 78 of 2069 replies (4%) had no answer marker and count wrong (5 of them cut off at the 32,768-token limit); a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history; an earlier attempt of this same run stopped part-way (1,210 recorded pairs and 64 further requests possibly answered but not recorded) and is kept apart, not merged; this attempt started again from the first pair, so its outcomes are this attempt's recorded outputs, not guaranteed to be the model's first outputs on those pairs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 78 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 86.5% of 942, a conflicting one 55.3% of 329, none 83.1% of 798

Clef-Flash · Cloudflare

Open · 9.0B

Cloudflare/clef-flash@fde727a2, the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; server probabilities are rounded to four decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 92.6% of 942, a conflicting one 37.7% of 329, none 82.2% of 798

Gemma 4 12B Unified · Google

Open · 12B

google/gemma-4-12B-it@43c9d014, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 296 of 2069 replies (14%) had no answer marker and count wrong (295 of them cut off at the 32,768-token limit); a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 296 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 93.4% of 942, a conflicting one 38.0% of 329, none 77.4% of 798

MiniCPM5-2B · OpenBMB

Open · 2.5B

openbmb/MiniCPM5-2B@42433e90, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.2% (per seed in seed_runs; cut off at the token limit per seed: 1, 0, 0); seed 0 is the published single run (14 Nov, every reply kept); seeds 1 and 2 are new runs (15 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs (GPU seconds per seed: 219, 76, 76, at the same recorded settings; the cause of the difference is not known)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0.2% of 2,069 on average (seed 0: 6; seed 1: 5; seed 2: 4), counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label, the mean of its seeds: the reference answer 87.9% of 942, a conflicting one 28.8% of 329, none 86.6% of 798

Seeds: seed 0: grading:chat-minicpm5-2b-nothink-r2-v1, 78.0% right, 6 invalid, 1 cut off, 93 wrong answers caught; seed 1: grading:chat-minicpm5-2b-nothink-seed1-r2-v1, 78.0% right, 5 invalid, 0 cut off, 94 wrong answers caught; seed 2: grading:chat-minicpm5-2b-nothink-seed2-r2-v1, 78.0% right, 4 invalid, 0 cut off, 97 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval

Measures: mean of 3 seeds; latency over all requests of the three runs

Qwen3.5-4B · Alibaba

Open · 4.0B

Qwen/Qwen3.5-4B@851bf6e8, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 179 of 2069 replies (9%) had no answer marker and count wrong (0 of them cut off at the 32,768-token limit); a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 179 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 87.7% of 942, a conflicting one 73.2% of 329, none 67.2% of 798

StartLux-Decision-v1.1-2B-Instant · StartLux

Open · 2.0B

startlux-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd14, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 88.8% of 942, a conflicting one 67.5% of 329, none 67.9% of 798

GLiDE 2B · Fastino

Open · 1.9B

fastino/GLiDE-2B@00f122c1, GLiDE server (python -m glide.server) in vLLM v0.30.0 image with flash-linear-attention 0.5.2 on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 89.7% of 942, a conflicting one 60.8% of 329, none 68.9% of 798

Qwen3.5-4B · Alibaba

Open · 4.0B

Qwen/Qwen3.5-4B@851bf6e8, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 1.0% (per seed in seed_runs; cut off at the token limit per seed: 0, 1, 0); seed 0 is the published single run (every reply kept); seeds 1 and 2 are new runs (16 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 1.0% of 2,069 on average (seed 0: 20; seed 1: 25; seed 2: 15), counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label, the mean of its seeds: the reference answer 94.4% of 942, a conflicting one 61.2% of 329, none 61.5% of 798

Seeds: seed 0: grading:chat-qwen3.5-4b-nothink-v1, 76.4% right, 20 invalid, 0 cut off, 196 wrong answers caught; seed 1: grading:chat-qwen3.5-4b-nothink-seed1-v1, 76.3% right, 25 invalid, 1 cut off, 202 wrong answers caught; seed 2: grading:chat-qwen3.5-4b-nothink-seed2-v1, 76.6% right, 15 invalid, 0 cut off, 206 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval

Measures: mean of 3 seeds; latency over all requests of the three runs

Metask-Jev-4B · Metask Lab

Open · 4.5B

wayfind/metask-jev-4b-policy-mix@ea20fe85, the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 96.2% of 942, a conflicting one 23.1% of 329, none 70.0% of 798

KnowLine-2B-Gen1 · PelaAI

Open · 2.0B

PelaAI/KnowLine-2B-Gen1@53245d47, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 92.6% of 942, a conflicting one 16.4% of 329, none 75.9% of 798

GLiDE 0.8B · Fastino

Open · 0.8B

fastino/GLiDE-0.8B@9eacdeec, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 89.9% of 942, a conflicting one 31.3% of 329, none 72.4% of 798

Strands Decider 2B · Strands Agents

Open · 2.0B

StrandsAgents/strands-decider-2B-hobson-v21@2b52a623 (a LoRA and head on Qwen3.5-2B-Base), the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; server probabilities are rounded to four decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 96.4% of 942, a conflicting one 4.6% of 329, none 75.1% of 798

LFM2.5-8B-A1B · Liquid AI

Open · 8.3B MoE · 1.5B active

LiquidAI/LFM2.5-8B-A1B@dd83f5cb, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept; an earlier attempt of this same run stopped part-way (45 recorded pairs and an unknown number of further requests possibly answered but not recorded) and is kept apart, not merged; this attempt started again from the first pair, so its outcomes are this attempt's recorded outputs, not guaranteed to be the model's first outputs on those pairs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 79.5% of 942, a conflicting one 8.8% of 329, none 88.7% of 798

Gemma 4 E2B · Google

Open · 5.1B

google/gemma-4-E2B-it@4adf1574, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.03% (per seed in seed_runs; cut off at the token limit per seed: 0, 0, 0); three new runs (16 Nov, every reply kept), seeds 0, 1, 2 at Gemma's own sampling (its generation_config.json: temperature 1.0, top_p 0.95, top_k 64; Ümit #10554), with the weights, template and token limit of the temperature-0 row, which stays as its own row; their server settings recorded; gpu_hours is the mean of the three runs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0.03% of 2,069 on average (seed 0: 1; seed 1: 0; seed 2: 1), counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label, the mean of its seeds: the reference answer 94.3% of 942, a conflicting one 7.7% of 329, none 67.3% of 798

Seeds: seed 0: grading:chat-gemma-4-e2b-sampled-seed0-v1, 69.7% right, 1 invalid, 0 cut off, 22 wrong answers caught; seed 1: grading:chat-gemma-4-e2b-sampled-seed1-v1, 70.6% right, 0 invalid, 0 cut off, 29 wrong answers caught; seed 2: grading:chat-gemma-4-e2b-sampled-seed2-v1, 70.1% right, 1 invalid, 0 cut off, 25 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval

Measures: mean of 3 seeds; latency over all requests of the three runs

Gemma 4 E2B · Google

Open · 5.1B

google/gemma-4-E2B-it@4adf1574, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking off; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label: the reference answer 94.7% of 942, a conflicting one 6.1% of 329, none 66.8% of 798

Qwen3.5-2B · Alibaba

Open · 2.3B

Qwen/Qwen3.5-2B@15852e8c, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 2.0% (per seed in seed_runs; cut off at the token limit per seed: 1, 0, 0); seed 0 is the published single run (every reply kept); seeds 1 and 2 are new runs (16 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 2.0% of 2,069 on average (seed 0: 37; seed 1: 45; seed 2: 43), counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label, the mean of its seeds: the reference answer 80.4% of 942, a conflicting one 34.4% of 329, none 64.2% of 798

Seeds: seed 0: grading:chat-qwen3.5-2b-nothink-v1, 66.2% right, 37 invalid, 1 cut off, 108 wrong answers caught; seed 1: grading:chat-qwen3.5-2b-nothink-seed1-v1, 67.1% right, 45 invalid, 0 cut off, 116 wrong answers caught; seed 2: grading:chat-qwen3.5-2b-nothink-seed2-v1, 67.2% right, 43 invalid, 0 cut off, 116 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval

Measures: mean of 3 seeds; latency over all requests of the three runs

OneJev1.6-0.8B · OmniJev

Open · 0.8B

OmniJev/OneJev1.6-0.8B@bcd88a38, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; server probabilities are rounded to six decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 82.3% of 942, a conflicting one 33.4% of 329, none 62.0% of 798

d1-2B · Liquid AI

Open · 1.9B

LiquidAI/d1-2B@10e7671d, d1_server (Liquid's model code) on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 84.3% of 942, a conflicting one 85.7% of 329, none 37.3% of 798

StartLux-Decision-v1.1-0.8B-Instant · StartLux

Open · 0.8B

startlux-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae64, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 83.9% of 942, a conflicting one 72.3% of 329, none 42.1% of 798

GLiDE 0.4B · Fastino

Open · 0.4B

fastino/GLiDE-0.4B@6bda7ab9, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 84.0% of 942, a conflicting one 31.0% of 329, none 51.2% of 798

GLiClass v4 Edge · Knowledgator

Open · 0.4B

knowledgator/gliclass-v4-edge@885af1af, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 92.5% of 942, a conflicting one 34.9% of 329, none 39.5% of 798

Decider 1 · meraGPT

Closed · Efficient

meraGPT decisions API (meragpt.com, model decider-1); one question per request, 8 requests at a time (latency not comparable across different concurrency)

Hardware: the maker's API (its latency includes the network)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 70.2% of 942, a conflicting one 31.9% of 329, none 65.3% of 798

List price $0.04 per 1M input tokens, its source (2026-10-19)

Raon-1 nano · KRAFTON

Open · 3.1B

KRAFTON/Raon-1-nano@fcf2d1d5, BF16, KRAFTON vLLM fork krafton-ai/vllm@13a5dd87 (not a vLLM release) on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.03% (per seed in seed_runs; cut off at the token limit per seed: 0, 1, 0); seed 0 is the published single run (every reply kept); seeds 1 and 2 are new runs (16 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs

Hardware: one RTX PRO 6000 Blackwell (96 GB), ours

Invalid replies: 0.03% of 2,069 on average (seed 0: 0; seed 1: 1; seed 2: 1), counted wrong

Option probabilities: none (a chat model's label), so no calibration error and no confidence

Recall by label, the mean of its seeds: the reference answer 64.9% of 942, a conflicting one 19.9% of 329, none 71.9% of 798

Seeds: seed 0: grading:chat-raon-1-nano-nothink-v1, 60.5% right, 0 invalid, 0 cut off, 65 wrong answers caught; seed 1: grading:chat-raon-1-nano-nothink-seed1-v1, 60.4% right, 1 invalid, 1 cut off, 68 wrong answers caught; seed 2: grading:chat-raon-1-nano-nothink-seed2-v1, 60.4% right, 1 invalid, 0 cut off, 63 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval

Measures: mean of 3 seeds; latency over all requests of the three runs

KnowLine-0.8B-Gen1 · PelaAI

Open · 0.8B

PelaAI/KnowLine-0.8B-Gen1@6b27831f, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)

Hardware: one NVIDIA DGX Spark (GB10)

Invalid replies: 0 of 2,069, counted wrong

Option probabilities: yes, so its calibration error is measured

Recall by label: the reference answer 98.0% of 942, a conflicting one 10.0% of 329, none 33.2% of 798