Answer Check
Answer Check
2,069 pairs of a machine-written English summary, a question and its reference answer · 3 labels · Score 0–100, chance = 0
Score higher is better
No row of this kind yet.
- MachinePulse
- Poindexter Labs
- PelaAI
- Sthānika AI
- REAX
- Liquid AI
- Turion.AI
- Thinking Machines
- Perplexity
- Fastino
- Logits
- Subquadratic
- Microsoft
- scar-ai
- TypeSafe
- tattrongvu
- Apertix AI
- StartLux
- Patrick-18
- newsbang
- OpenBMB
- quincyqiang
- Base Labs
- EldanRing
- IBM
- victorbennett
- Valen-Team
- Blockbrain Labs
- Alibaba
- Cloudflare
- Metask Lab
- Strands Agents
- OmniJev
- Knowledgator
- meraGPT
- KRAFTON
| The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –. | Model | (accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100 | Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairs | Expected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, points | Of the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarms | Client wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under it | Input tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairs | Measured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs |
|---|---|---|---|---|---|---|---|---|
| 1 | Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details | 90.0 | 93.3% | 2.8 | 80.8%1.7% false alarms | 0.22 sp90 0.31 s1 at a time | Our GPU | 230 sRTX PRO 6000 |
| 2 | Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details | 89.0 | 92.7% | 11.0 | 81.5%2.5% false alarms | 0.52 sp90 0.83 s1 at a time | Our GPU | 577 sRTX PRO 6000 |
| 3 | KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails | 88.8 | 92.6% | 5.7 | 83.6%3.0% false alarms | 2.54 sp90 3.88 s1 at a time | Our GPU | 2,613 sDGX Spark |
| 4 | Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails | 88.1 | 92.1% | 7.0 | 79.3%2.1% false alarms | 2.67 sp90 3.93 s1 at a time | Our GPU | 2,824 sDGX Spark |
| 5 | KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details | 87.7 | 91.8% | 7.2 | 86.3%3.6% false alarms | 0.39 sp90 0.61 s1 at a time | Our GPU | 449 sRTX PRO 6000 |
| 6 | REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details | 87.0 | 91.3% | 4.8 | 67.5%1.2% false alarms | 0.68 sp90 1.05 s1 at a time | Our GPU | 745 sRTX PRO 6000 |
| 7 | d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details | 86.4 | 91.0% | 8.8 | 85.4%4.4% false alarms | 0.23 sp90 0.35 s8 at a time | $0.035 | API |
| 8 | Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails | 86.4 | 91.0% | 6.7 | 66.3%1.7% false alarms | 1.12 sp90 1.96 s1 at a time | Our GPU | 1,329 sDGX Spark |
| Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details | 86.2 | 90.8% | No probabilities | 73.6%2.4% false alarms | 2.70 sp90 16.4 s1 at a time | $1.49 | API | |
| 9 | Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details | 86.2 | 90.8% | 2.8 | 82.7%3.5% false alarms | 0.22 sp90 0.34 s1 at a time | Our GPU | 245 sRTX PRO 6000 |
| 10 | GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails | 86.1 | 90.7% | 3.6 | 74.8%1.5% false alarms | 1.03 sp90 1.95 s1 at a time | Our GPU | 1,205 sDGX Spark |
| 11 | Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details | 85.1 | 90.1% | 7.0 | 63.8%0.9% false alarms | 0.64 sp90 1.01 s1 at a time | Our GPU | 723 sRTX PRO 6000 |
| 12 | GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails | 85.1 | 90.0% | 3.0 | 68.7%1.7% false alarms | 0.30 sp90 0.56 s8 at a time | Not verified | API |
| 13 | GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details | 84.9 | 89.9% | 4.7 | 69.3%1.8% false alarms | 0.23 sp90 0.36 s1 at a time | Our GPU | 254 sRTX PRO 6000 |
| 14 | SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails | 84.9 | 89.9% | 2.3 | 78.1%4.1% false alarms | 0.39 sp90 0.41 s8 at a time | Not verified | API |
| 15 | d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details | 84.7 | 89.8% | 11.4 | 80.8%4.5% false alarms | 0.48 sp90 0.90 s8 at a time | $0.035 | API |
| 16 | Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details | 83.9 | 89.3% | 2.4 | 82.4%6.5% false alarms | 0.32 sp90 0.38 s8 at a time | $0.036 | API |
| Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails | 83.7 | 89.1% | No probabilities | 68.7%2.2% false alarms | 6.37 sp90 19.2 s1 at a time | Not verified | API | |
| 17 | vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details | 83.7 | 89.1% | 12.4 | 76.0%5.7% false alarms | 0.06 sp90 0.06 s1 at a time | Our GPU | 60 sRTX PRO 6000 |
| 18 | Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details | 83.6 | 89.1% | 2.8 | 72.0%3.5% false alarms | 0.18 sp90 0.20 s8 at a time | $0.049 | API |
| 19 | Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details | 83.0 | 88.6% | 3.3 | 59.9%2.5% false alarms | 0.09 sp90 0.14 s1 at a time | Our GPU | 100 sRTX PRO 6000 |
| 20 | Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details | 83.0 | 88.6% | 1.9 | 79.9%6.0% false alarms | 0.17 sp90 0.19 s8 at a time | $0.049 | API |
| 21 | d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details | 82.5 | 88.4% | 8.7 | 70.5%4.1% false alarms | 0.25 sp90 0.45 s8 at a time | $0.017 | API |
| 22 | GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details | 82.2 | 88.2% | 7.2 | 69.9%3.8% false alarms | 0.08 sp90 0.11 s1 at a time | Our GPU | 90 sRTX PRO 6000 |
| 23 | Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details | 81.9 | 87.9% | 2.9 | 65.3%3.0% false alarms | 0.11 sp90 0.18 s1 at a time | Our GPU | 126 sRTX PRO 6000 |
| 24 | StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails | 81.7 | 87.8% | 8.3 | 89.4%8.3% false alarms | 0.12 sp90 0.19 s1 at a time | Our GPU | 140 sDGX Spark |
| 25 | Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details | 81.1 | 87.4% | 2.5 | 64.7%3.9% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU | 72 sRTX PRO 6000 |
| 26 | KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details | 80.9 | 87.2% | 6.8 | 61.1%3.6% false alarms | 0.16 sp90 0.25 s1 at a time | Our GPU | 195 sRTX PRO 6000 |
| 27 | KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details | 80.6 | 87.1% | 4.5 | 62.0%3.4% false alarms | 0.04 sp90 0.06 s1 at a time | Our GPU | 49 sRTX PRO 6000 |
| 28 | Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details | 80.4 | 87.0% | 4.9 | 65.0%4.9% false alarms | 0.07 sp90 0.09 s1 at a time | Our GPU | 71 sRTX PRO 6000 |
| MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details | 79.279.0–79.6 · min–max of 3 seeds | 86.2%86.0%–86.4%min–max of 3 seeds | No probabilities | 77.4%6.5% false alarmsmean of 3 seeds | 5.00 sp90 8.72 s30 at a time3 seeds pooled | Our GPU | 197 sRTX PRO 6000mean of 3 seeds | |
| 29 | d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails | 79.2 | 86.1% | 3.1 | 66.3%5.3% false alarms | 0.27 sp90 0.60 s8 at a time | Not verified | API |
| 30 | Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details | 78.8 | 85.9% | 4.2 | 67.5%5.6% false alarms | 0.06 sp90 0.08 s1 at a time | Our GPU | 61 sRTX PRO 6000 |
| 31 | StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails | 78.8 | 85.9% | 8.2 | 88.8%9.5% false alarms | 0.23 sp90 0.36 s1 at a time | Our GPU | 268 sDGX Spark |
| 32 | Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details | 78.1 | 85.4% | 6.7 | 69.0%6.0% false alarms | 0.07 sp90 0.11 s1 at a time | Our GPU | 77 sRTX PRO 6000 |
| Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details | 77.2 | 84.8% | No probabilities | 67.5%4.5% false alarms | 9.38 sp90 11.9 s30 at a time | Our GPU | 326 sRTX PRO 6000 | |
| 33 | Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details | 77.2 | 84.8% | 2.8 | 47.7%4.0% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU | 65 sRTX PRO 6000 |
| 34 | Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details | 77.0 | 84.7% | 3.9 | 51.1%1.8% false alarms | 0.20 sp90 0.29 s8 at a time | $0.093 | API |
| Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details | 77.0 | 84.7% | No probabilities | 52.0%2.6% false alarms | 5.12 sp90 15.7 s64 at a time | Our GPU | 121 sRTX PRO 6000 | |
| 35 | Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details | 76.9 | 84.6% | 3.7 | 41.0%1.9% false alarms | 0.09 sp90 0.11 s1 at a time | Our GPU | 98 sRTX PRO 6000 |
| 36 | Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details | 76.6 | 84.4% | 8.3 | 51.7%3.8% false alarms | 0.12 sp90 0.18 s1 at a time | Our GPU | 132 sRTX PRO 6000 |
| 37 | KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details | 76.1 | 84.1% | 3.6 | 50.8%3.8% false alarms | 0.10 sp90 0.14 s1 at a time | Our GPU | 105 sRTX PRO 6000 |
| Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details | 75.9 | 83.9% | No probabilities | 79.0%9.0% false alarms | 19.6 sp90 64.5 s30 at a time | Our GPU | 1,050 sRTX PRO 6000 | |
| LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details | 75.874.9–76.5 · min–max of 3 seeds | 83.9%83.3%–84.3%min–max of 3 seeds | No probabilities | 63.3%4.1% false alarmsmean of 3 seeds | 5.57 sp90 16.4 s30 at a time3 seeds pooled | Our GPU | 268 sRTX PRO 6000mean of 3 seeds | |
| 38 | Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details | 75.4 | 83.6% | 3.4 | 46.8%3.7% false alarms | 0.05 sp90 0.07 s1 at a time | Our GPU | 54 sRTX PRO 6000 |
| 39 | KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details | 75.3 | 83.5% | 4.5 | 35.9%1.4% false alarms | 0.10 sp90 0.16 s1 at a time | Our GPU | 128 sRTX PRO 6000 |
| Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details | 75.174.1–75.9 · min–max of 3 seeds | 83.4%82.7%–84.0%min–max of 3 seeds | No probabilities | 64.3%4.3% false alarmsmean of 3 seeds | 6.13 sp90 9.70 s30 at a time3 seeds pooled | Our GPU | 234 sRTX PRO 6000mean of 3 seeds | |
| LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details | 74.973.5–75.6 · min–max of 3 seeds | 83.3%82.3%–83.8%min–max of 3 seeds | No probabilities | 62.6%3.9% false alarmsmean of 3 seeds | 7.62 sp90 22.3 s64 at a time3 seeds pooled | Our GPU | 180 sRTX PRO 6000mean of 3 seeds | |
| 40 | Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details | 74.3 | 82.8% | 3.2 | 39.8%2.8% false alarms | 0.07 sp90 0.10 s1 at a time | Our GPU | 78 sRTX PRO 6000 |
| 41 | Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details | 72.9 | 81.9% | 6.5 | 29.2%2.1% false alarms | 0.09 sp90 0.13 s1 at a time | Our GPU | 93 sRTX PRO 6000 |
| Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details | 71.1 | 80.8% | No probabilities | 49.5%5.1% false alarms | 4.02 sp90 5.85 s64 at a time | Our GPU | 69 sRTX PRO 6000 | |
| Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details | 70.3 | 80.2% | No probabilities | 55.3%4.3% false alarms | 36.8 sp90 66.9 s64 at a time | Our GPU | 636 sRTX PRO 6000 | |
| 42 | Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details | 69.8 | 79.8% | 5.9 | 37.7%3.9% false alarms | 0.15 sp90 0.18 s1 at a time | Our GPU | 156 sRTX PRO 6000 |
| Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details | 67.7 | 78.4% | No probabilities | 38.0%0.7% false alarms | 32.6 sp90 1640.2 s64 at a time | Our GPU | 4,327 sRTX PRO 6000 | |
| MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details | 67.066.9–67.0 · min–max of 3 seeds | 78.0%77.96%–78.01%min–max of 3 seeds | No probabilities | 28.8%2.7% false alarmsmean of 3 seeds | 0.86 sp90 3.48 s64 at a time3 seeds pooled | Our GPU | 51 sRTX PRO 6000mean of 3 seeds | |
| Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details | 66.2 | 77.5% | No probabilities | 73.2%6.2% false alarms | 37.8 sp90 96.1 s64 at a time | Our GPU | 741 sRTX PRO 6000 | |
| 43 | StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails | 66.0 | 77.3% | 7.2 | 67.5%14.0% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU | 78 sDGX Spark |
| 44 | GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details | 65.6 | 77.1% | 4.0 | 60.8%10.8% false alarms | 0.04 sp90 0.04 s1 at a time | Our GPU | 44 sRTX PRO 6000 |
| Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details | 64.664.5–64.8 · min–max of 3 seeds | 76.4%76.3%–76.6%min–max of 3 seeds | No probabilities | 61.2%7.6% false alarmsmean of 3 seeds | 3.49 sp90 20.4 s64 at a time3 seeds pooled | Our GPU | 148 sRTX PRO 6000mean of 3 seeds | |
| 45 | Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details | 61.7 | 74.5% | 6.8 | 23.1%3.4% false alarms | 0.12 sp90 0.19 s1 at a time | Our GPU | 134 sRTX PRO 6000 |
| 46 | KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails | 61.1 | 74.0% | 2.7 | 16.4%2.7% false alarms | 0.11 sp90 0.17 s1 at a time | Our GPU | 130 sDGX Spark |
| 47 | GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails | 60.8 | 73.9% | 3.8 | 31.3%6.6% false alarms | 0.05 sp90 0.07 s1 at a time | Our GPU | 63 sDGX Spark |
| 48 | Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details | 60.3 | 73.6% | 5.0 | 4.6%0.8% false alarms | 0.04 sp90 0.05 s1 at a time | Our GPU | 43 sRTX PRO 6000 |
| LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details | 57.7 | 71.8% | No probabilities | 8.8%0.8% false alarms | 9.58 sp90 15.2 s30 at a time | Our GPU | 352 sRTX PRO 6000 | |
| Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details | 55.254.5–55.8 · min–max of 3 seeds | 70.1%69.7%–70.6%min–max of 3 seeds | No probabilities | 7.7%1.6% false alarmsmean of 3 seeds | 0.18 sp90 5.42 s64 at a time3 seeds pooled | Our GPU | 33 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details | 54.8 | 69.8% | No probabilities | 6.1%1.3% false alarms | 0.19 sp90 3.31 s64 at a time | Our GPU | 18 sRTX PRO 6000 | |
| Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details | 50.249.3–50.8 · min–max of 3 seeds | 66.8%66.2%–67.2%min–max of 3 seeds | No probabilities | 34.4%7.8% false alarmsmean of 3 seeds | 1.53 sp90 13.7 s64 at a time3 seeds pooled | Our GPU | 92 sRTX PRO 6000mean of 3 seeds | |
| 49 | OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails | 50.0 | 66.7% | 4.1 | 33.4%14.4% false alarms | 0.03 sp90 0.04 s1 at a time | Our GPU | 43 sDGX Spark |
| 50 | d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details | 49.6 | 66.4% | 7.2 | 85.7%32.2% false alarms | 0.02 sp90 0.03 s1 at a time | Our GPU | 25 sRTX PRO 6000 |
| 51 | StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails | 48.9 | 65.9% | 5.6 | 72.3%26.5% false alarms | 0.04 sp90 0.06 s1 at a time | Our GPU | 53 sDGX Spark |
| 52 | GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails | 44.4 | 62.9% | 7.7 | 31.0%16.8% false alarms | 0.03 sp90 0.04 s1 at a time | Our GPU | 41 sDGX Spark |
| 53 | GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails | 44.3 | 62.9% | 9.8 | 34.9%10.6% false alarms | 0.07 sp90 0.18 s1 at a time | Our GPU | 128 sDGX Spark |
| 54 | Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details | 43.3 | 62.2% | 5.9 | 31.9%17.6% false alarms | 0.65 sp90 0.75 s8 at a time | $0.035 | API |
| Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details | 40.640.6–40.8 · min–max of 3 seeds | 60.4%60.4%–60.5%min–max of 3 seeds | No probabilities | 19.9%2.0% false alarmsmean of 3 seeds | 3.03 sp90 7.76 s30 at a time3 seeds pooled | Our GPU | 209 sRTX PRO 6000mean of 3 seeds | |
| 55 | KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails | 38.5 | 59.0% | 14.6 | 10.0%3.6% false alarms | 0.07 sp90 0.10 s1 at a time | Our GPU | 81 sDGX Spark |
No row of this kind yet.
2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology
Accuracy higher is better
No row of this kind yet.
- MachinePulse
- Poindexter Labs
- PelaAI
- Sthānika AI
- REAX
- Liquid AI
- Turion.AI
- Thinking Machines
- Perplexity
- Fastino
- Logits
- Subquadratic
- Microsoft
- scar-ai
- TypeSafe
- tattrongvu
- Apertix AI
- StartLux
- Patrick-18
- newsbang
- OpenBMB
- quincyqiang
- Base Labs
- EldanRing
- IBM
- victorbennett
- Valen-Team
- Blockbrain Labs
- Alibaba
- Cloudflare
- Metask Lab
- Strands Agents
- OmniJev
- Knowledgator
- meraGPT
- KRAFTON
| The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –. | Model | Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairs | (accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100 | Expected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, points | Of the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarms | Client wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under it | Input tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairs | Measured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs |
|---|---|---|---|---|---|---|---|---|
| 1 | Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details | 93.3% | 90.0 | 2.8 | 80.8%1.7% false alarms | 0.22 sp90 0.31 s1 at a time | Our GPU | 230 sRTX PRO 6000 |
| 2 | Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details | 92.7% | 89.0 | 11.0 | 81.5%2.5% false alarms | 0.52 sp90 0.83 s1 at a time | Our GPU | 577 sRTX PRO 6000 |
| 3 | KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails | 92.6% | 88.8 | 5.7 | 83.6%3.0% false alarms | 2.54 sp90 3.88 s1 at a time | Our GPU | 2,613 sDGX Spark |
| 4 | Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails | 92.1% | 88.1 | 7.0 | 79.3%2.1% false alarms | 2.67 sp90 3.93 s1 at a time | Our GPU | 2,824 sDGX Spark |
| 5 | KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details | 91.8% | 87.7 | 7.2 | 86.3%3.6% false alarms | 0.39 sp90 0.61 s1 at a time | Our GPU | 449 sRTX PRO 6000 |
| 6 | REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details | 91.3% | 87.0 | 4.8 | 67.5%1.2% false alarms | 0.68 sp90 1.05 s1 at a time | Our GPU | 745 sRTX PRO 6000 |
| 7 | d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details | 91.0% | 86.4 | 8.8 | 85.4%4.4% false alarms | 0.23 sp90 0.35 s8 at a time | $0.035 | API |
| 8 | Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails | 91.0% | 86.4 | 6.7 | 66.3%1.7% false alarms | 1.12 sp90 1.96 s1 at a time | Our GPU | 1,329 sDGX Spark |
| Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details | 90.8% | 86.2 | No probabilities | 73.6%2.4% false alarms | 2.70 sp90 16.4 s1 at a time | $1.49 | API | |
| 9 | Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details | 90.8% | 86.2 | 2.8 | 82.7%3.5% false alarms | 0.22 sp90 0.34 s1 at a time | Our GPU | 245 sRTX PRO 6000 |
| 10 | GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails | 90.7% | 86.1 | 3.6 | 74.8%1.5% false alarms | 1.03 sp90 1.95 s1 at a time | Our GPU | 1,205 sDGX Spark |
| 11 | Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details | 90.1% | 85.1 | 7.0 | 63.8%0.9% false alarms | 0.64 sp90 1.01 s1 at a time | Our GPU | 723 sRTX PRO 6000 |
| 12 | GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails | 90.0% | 85.1 | 3.0 | 68.7%1.7% false alarms | 0.30 sp90 0.56 s8 at a time | Not verified | API |
| 13 | GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details | 89.9% | 84.9 | 4.7 | 69.3%1.8% false alarms | 0.23 sp90 0.36 s1 at a time | Our GPU | 254 sRTX PRO 6000 |
| 14 | SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails | 89.9% | 84.9 | 2.3 | 78.1%4.1% false alarms | 0.39 sp90 0.41 s8 at a time | Not verified | API |
| 15 | d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details | 89.8% | 84.7 | 11.4 | 80.8%4.5% false alarms | 0.48 sp90 0.90 s8 at a time | $0.035 | API |
| 16 | Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details | 89.3% | 83.9 | 2.4 | 82.4%6.5% false alarms | 0.32 sp90 0.38 s8 at a time | $0.036 | API |
| Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails | 89.1% | 83.7 | No probabilities | 68.7%2.2% false alarms | 6.37 sp90 19.2 s1 at a time | Not verified | API | |
| 17 | vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details | 89.1% | 83.7 | 12.4 | 76.0%5.7% false alarms | 0.06 sp90 0.06 s1 at a time | Our GPU | 60 sRTX PRO 6000 |
| 18 | Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details | 89.1% | 83.6 | 2.8 | 72.0%3.5% false alarms | 0.18 sp90 0.20 s8 at a time | $0.049 | API |
| 19 | Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details | 88.6% | 83.0 | 3.3 | 59.9%2.5% false alarms | 0.09 sp90 0.14 s1 at a time | Our GPU | 100 sRTX PRO 6000 |
| 20 | Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details | 88.6% | 83.0 | 1.9 | 79.9%6.0% false alarms | 0.17 sp90 0.19 s8 at a time | $0.049 | API |
| 21 | d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details | 88.4% | 82.5 | 8.7 | 70.5%4.1% false alarms | 0.25 sp90 0.45 s8 at a time | $0.017 | API |
| 22 | GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details | 88.2% | 82.2 | 7.2 | 69.9%3.8% false alarms | 0.08 sp90 0.11 s1 at a time | Our GPU | 90 sRTX PRO 6000 |
| 23 | Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details | 87.9% | 81.9 | 2.9 | 65.3%3.0% false alarms | 0.11 sp90 0.18 s1 at a time | Our GPU | 126 sRTX PRO 6000 |
| 24 | StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails | 87.8% | 81.7 | 8.3 | 89.4%8.3% false alarms | 0.12 sp90 0.19 s1 at a time | Our GPU | 140 sDGX Spark |
| 25 | Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details | 87.4% | 81.1 | 2.5 | 64.7%3.9% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU | 72 sRTX PRO 6000 |
| 26 | KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details | 87.2% | 80.9 | 6.8 | 61.1%3.6% false alarms | 0.16 sp90 0.25 s1 at a time | Our GPU | 195 sRTX PRO 6000 |
| 27 | KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details | 87.1% | 80.6 | 4.5 | 62.0%3.4% false alarms | 0.04 sp90 0.06 s1 at a time | Our GPU | 49 sRTX PRO 6000 |
| 28 | Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details | 87.0% | 80.4 | 4.9 | 65.0%4.9% false alarms | 0.07 sp90 0.09 s1 at a time | Our GPU | 71 sRTX PRO 6000 |
| MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details | 86.2%86.0%–86.4% · min–max of 3 seeds | 79.279.0–79.6min–max of 3 seeds | No probabilities | 77.4%6.5% false alarmsmean of 3 seeds | 5.00 sp90 8.72 s30 at a time3 seeds pooled | Our GPU | 197 sRTX PRO 6000mean of 3 seeds | |
| 29 | d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails | 86.1% | 79.2 | 3.1 | 66.3%5.3% false alarms | 0.27 sp90 0.60 s8 at a time | Not verified | API |
| 30 | Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details | 85.9% | 78.8 | 4.2 | 67.5%5.6% false alarms | 0.06 sp90 0.08 s1 at a time | Our GPU | 61 sRTX PRO 6000 |
| 31 | StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails | 85.9% | 78.8 | 8.2 | 88.8%9.5% false alarms | 0.23 sp90 0.36 s1 at a time | Our GPU | 268 sDGX Spark |
| 32 | Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details | 85.4% | 78.1 | 6.7 | 69.0%6.0% false alarms | 0.07 sp90 0.11 s1 at a time | Our GPU | 77 sRTX PRO 6000 |
| Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details | 84.8% | 77.2 | No probabilities | 67.5%4.5% false alarms | 9.38 sp90 11.9 s30 at a time | Our GPU | 326 sRTX PRO 6000 | |
| 33 | Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details | 84.8% | 77.2 | 2.8 | 47.7%4.0% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU | 65 sRTX PRO 6000 |
| 34 | Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details | 84.7% | 77.0 | 3.9 | 51.1%1.8% false alarms | 0.20 sp90 0.29 s8 at a time | $0.093 | API |
| Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details | 84.7% | 77.0 | No probabilities | 52.0%2.6% false alarms | 5.12 sp90 15.7 s64 at a time | Our GPU | 121 sRTX PRO 6000 | |
| 35 | Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details | 84.6% | 76.9 | 3.7 | 41.0%1.9% false alarms | 0.09 sp90 0.11 s1 at a time | Our GPU | 98 sRTX PRO 6000 |
| 36 | Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details | 84.4% | 76.6 | 8.3 | 51.7%3.8% false alarms | 0.12 sp90 0.18 s1 at a time | Our GPU | 132 sRTX PRO 6000 |
| 37 | KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details | 84.1% | 76.1 | 3.6 | 50.8%3.8% false alarms | 0.10 sp90 0.14 s1 at a time | Our GPU | 105 sRTX PRO 6000 |
| Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details | 83.9% | 75.9 | No probabilities | 79.0%9.0% false alarms | 19.6 sp90 64.5 s30 at a time | Our GPU | 1,050 sRTX PRO 6000 | |
| LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details | 83.9%83.3%–84.3% · min–max of 3 seeds | 75.874.9–76.5min–max of 3 seeds | No probabilities | 63.3%4.1% false alarmsmean of 3 seeds | 5.57 sp90 16.4 s30 at a time3 seeds pooled | Our GPU | 268 sRTX PRO 6000mean of 3 seeds | |
| 38 | Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details | 83.6% | 75.4 | 3.4 | 46.8%3.7% false alarms | 0.05 sp90 0.07 s1 at a time | Our GPU | 54 sRTX PRO 6000 |
| 39 | KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details | 83.5% | 75.3 | 4.5 | 35.9%1.4% false alarms | 0.10 sp90 0.16 s1 at a time | Our GPU | 128 sRTX PRO 6000 |
| Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details | 83.4%82.7%–84.0% · min–max of 3 seeds | 75.174.1–75.9min–max of 3 seeds | No probabilities | 64.3%4.3% false alarmsmean of 3 seeds | 6.13 sp90 9.70 s30 at a time3 seeds pooled | Our GPU | 234 sRTX PRO 6000mean of 3 seeds | |
| LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details | 83.3%82.3%–83.8% · min–max of 3 seeds | 74.973.5–75.6min–max of 3 seeds | No probabilities | 62.6%3.9% false alarmsmean of 3 seeds | 7.62 sp90 22.3 s64 at a time3 seeds pooled | Our GPU | 180 sRTX PRO 6000mean of 3 seeds | |
| 40 | Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details | 82.8% | 74.3 | 3.2 | 39.8%2.8% false alarms | 0.07 sp90 0.10 s1 at a time | Our GPU | 78 sRTX PRO 6000 |
| 41 | Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details | 81.9% | 72.9 | 6.5 | 29.2%2.1% false alarms | 0.09 sp90 0.13 s1 at a time | Our GPU | 93 sRTX PRO 6000 |
| Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details | 80.8% | 71.1 | No probabilities | 49.5%5.1% false alarms | 4.02 sp90 5.85 s64 at a time | Our GPU | 69 sRTX PRO 6000 | |
| Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details | 80.2% | 70.3 | No probabilities | 55.3%4.3% false alarms | 36.8 sp90 66.9 s64 at a time | Our GPU | 636 sRTX PRO 6000 | |
| 42 | Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details | 79.8% | 69.8 | 5.9 | 37.7%3.9% false alarms | 0.15 sp90 0.18 s1 at a time | Our GPU | 156 sRTX PRO 6000 |
| Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details | 78.4% | 67.7 | No probabilities | 38.0%0.7% false alarms | 32.6 sp90 1640.2 s64 at a time | Our GPU | 4,327 sRTX PRO 6000 | |
| MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details | 78.0%77.96%–78.01% · min–max of 3 seeds | 67.066.9–67.0min–max of 3 seeds | No probabilities | 28.8%2.7% false alarmsmean of 3 seeds | 0.86 sp90 3.48 s64 at a time3 seeds pooled | Our GPU | 51 sRTX PRO 6000mean of 3 seeds | |
| Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details | 77.5% | 66.2 | No probabilities | 73.2%6.2% false alarms | 37.8 sp90 96.1 s64 at a time | Our GPU | 741 sRTX PRO 6000 | |
| 43 | StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails | 77.3% | 66.0 | 7.2 | 67.5%14.0% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU | 78 sDGX Spark |
| 44 | GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details | 77.1% | 65.6 | 4.0 | 60.8%10.8% false alarms | 0.04 sp90 0.04 s1 at a time | Our GPU | 44 sRTX PRO 6000 |
| Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details | 76.4%76.3%–76.6% · min–max of 3 seeds | 64.664.5–64.8min–max of 3 seeds | No probabilities | 61.2%7.6% false alarmsmean of 3 seeds | 3.49 sp90 20.4 s64 at a time3 seeds pooled | Our GPU | 148 sRTX PRO 6000mean of 3 seeds | |
| 45 | Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details | 74.5% | 61.7 | 6.8 | 23.1%3.4% false alarms | 0.12 sp90 0.19 s1 at a time | Our GPU | 134 sRTX PRO 6000 |
| 46 | KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails | 74.0% | 61.1 | 2.7 | 16.4%2.7% false alarms | 0.11 sp90 0.17 s1 at a time | Our GPU | 130 sDGX Spark |
| 47 | GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails | 73.9% | 60.8 | 3.8 | 31.3%6.6% false alarms | 0.05 sp90 0.07 s1 at a time | Our GPU | 63 sDGX Spark |
| 48 | Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details | 73.6% | 60.3 | 5.0 | 4.6%0.8% false alarms | 0.04 sp90 0.05 s1 at a time | Our GPU | 43 sRTX PRO 6000 |
| LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details | 71.8% | 57.7 | No probabilities | 8.8%0.8% false alarms | 9.58 sp90 15.2 s30 at a time | Our GPU | 352 sRTX PRO 6000 | |
| Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details | 70.1%69.7%–70.6% · min–max of 3 seeds | 55.254.5–55.8min–max of 3 seeds | No probabilities | 7.7%1.6% false alarmsmean of 3 seeds | 0.18 sp90 5.42 s64 at a time3 seeds pooled | Our GPU | 33 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details | 69.8% | 54.8 | No probabilities | 6.1%1.3% false alarms | 0.19 sp90 3.31 s64 at a time | Our GPU | 18 sRTX PRO 6000 | |
| Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details | 66.8%66.2%–67.2% · min–max of 3 seeds | 50.249.3–50.8min–max of 3 seeds | No probabilities | 34.4%7.8% false alarmsmean of 3 seeds | 1.53 sp90 13.7 s64 at a time3 seeds pooled | Our GPU | 92 sRTX PRO 6000mean of 3 seeds | |
| 49 | OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails | 66.7% | 50.0 | 4.1 | 33.4%14.4% false alarms | 0.03 sp90 0.04 s1 at a time | Our GPU | 43 sDGX Spark |
| 50 | d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details | 66.4% | 49.6 | 7.2 | 85.7%32.2% false alarms | 0.02 sp90 0.03 s1 at a time | Our GPU | 25 sRTX PRO 6000 |
| 51 | StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails | 65.9% | 48.9 | 5.6 | 72.3%26.5% false alarms | 0.04 sp90 0.06 s1 at a time | Our GPU | 53 sDGX Spark |
| 52 | GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails | 62.9% | 44.4 | 7.7 | 31.0%16.8% false alarms | 0.03 sp90 0.04 s1 at a time | Our GPU | 41 sDGX Spark |
| 53 | GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails | 62.9% | 44.3 | 9.8 | 34.9%10.6% false alarms | 0.07 sp90 0.18 s1 at a time | Our GPU | 128 sDGX Spark |
| 54 | Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details | 62.2% | 43.3 | 5.9 | 31.9%17.6% false alarms | 0.65 sp90 0.75 s8 at a time | $0.035 | API |
| Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details | 60.4%60.4%–60.5% · min–max of 3 seeds | 40.640.6–40.8min–max of 3 seeds | No probabilities | 19.9%2.0% false alarmsmean of 3 seeds | 3.03 sp90 7.76 s30 at a time3 seeds pooled | Our GPU | 209 sRTX PRO 6000mean of 3 seeds | |
| 55 | KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails | 59.0% | 38.5 | 14.6 | 10.0%3.6% false alarms | 0.07 sp90 0.10 s1 at a time | Our GPU | 81 sDGX Spark |
No row of this kind yet.
2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology
Calibration lower is better
No row of this kind yet.
- TypeSafe
- Subquadratic
- Microsoft
- Patrick-18
- PelaAI
- Perplexity
- MachinePulse
- Apertix AI
- Fastino
- Liquid AI
- Valen-Team
- tattrongvu
- victorbennett
- Base Labs
- OmniJev
- REAX
- newsbang
- Strands Agents
- StartLux
- Cloudflare
- meraGPT
- Blockbrain Labs
- quincyqiang
- Turion.AI
- Metask Lab
- Sthānika AI
- Logits
- EldanRing
- Knowledgator
- Poindexter Labs
- scar-ai
- Thinking Machines
- OpenBMB
- IBM
- Alibaba
- KRAFTON
| The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –. | Model | Expected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, points | (accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100 | Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairs | Of the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarms | Client wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under it | Input tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairs | Measured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs |
|---|---|---|---|---|---|---|---|---|
| 1 | Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details | 1.9 | 83.0 | 88.6% | 79.9%6.0% false alarms | 0.17 sp90 0.19 s8 at a time | $0.049 | API |
| 2 | SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails | 2.3 | 84.9 | 89.9% | 78.1%4.1% false alarms | 0.39 sp90 0.41 s8 at a time | Not verified | API |
| 3 | Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details | 2.4 | 83.9 | 89.3% | 82.4%6.5% false alarms | 0.32 sp90 0.38 s8 at a time | $0.036 | API |
| 4 | Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details | 2.5 | 81.1 | 87.4% | 64.7%3.9% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU | 72 sRTX PRO 6000 |
| 5 | KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails | 2.7 | 61.1 | 74.0% | 16.4%2.7% false alarms | 0.11 sp90 0.17 s1 at a time | Our GPU | 130 sDGX Spark |
| 6 | Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details | 2.8 | 86.2 | 90.8% | 82.7%3.5% false alarms | 0.22 sp90 0.34 s1 at a time | Our GPU | 245 sRTX PRO 6000 |
| 7 | Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details | 2.8 | 77.2 | 84.8% | 47.7%4.0% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU | 65 sRTX PRO 6000 |
| 8 | Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details | 2.8 | 90.0 | 93.3% | 80.8%1.7% false alarms | 0.22 sp90 0.31 s1 at a time | Our GPU | 230 sRTX PRO 6000 |
| 9 | Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details | 2.8 | 83.6 | 89.1% | 72.0%3.5% false alarms | 0.18 sp90 0.20 s8 at a time | $0.049 | API |
| 10 | Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details | 2.9 | 81.9 | 87.9% | 65.3%3.0% false alarms | 0.11 sp90 0.18 s1 at a time | Our GPU | 126 sRTX PRO 6000 |
| 11 | GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails | 3.0 | 85.1 | 90.0% | 68.7%1.7% false alarms | 0.30 sp90 0.56 s8 at a time | Not verified | API |
| 12 | d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails | 3.1 | 79.2 | 86.1% | 66.3%5.3% false alarms | 0.27 sp90 0.60 s8 at a time | Not verified | API |
| 13 | Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details | 3.2 | 74.3 | 82.8% | 39.8%2.8% false alarms | 0.07 sp90 0.10 s1 at a time | Our GPU | 78 sRTX PRO 6000 |
| 14 | Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details | 3.3 | 83.0 | 88.6% | 59.9%2.5% false alarms | 0.09 sp90 0.14 s1 at a time | Our GPU | 100 sRTX PRO 6000 |
| 15 | Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details | 3.4 | 75.4 | 83.6% | 46.8%3.7% false alarms | 0.05 sp90 0.07 s1 at a time | Our GPU | 54 sRTX PRO 6000 |
| 16 | KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details | 3.6 | 76.1 | 84.1% | 50.8%3.8% false alarms | 0.10 sp90 0.14 s1 at a time | Our GPU | 105 sRTX PRO 6000 |
| 17 | GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails | 3.6 | 86.1 | 90.7% | 74.8%1.5% false alarms | 1.03 sp90 1.95 s1 at a time | Our GPU | 1,205 sDGX Spark |
| 18 | Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details | 3.7 | 76.9 | 84.6% | 41.0%1.9% false alarms | 0.09 sp90 0.11 s1 at a time | Our GPU | 98 sRTX PRO 6000 |
| 19 | GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails | 3.8 | 60.8 | 73.9% | 31.3%6.6% false alarms | 0.05 sp90 0.07 s1 at a time | Our GPU | 63 sDGX Spark |
| 20 | Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details | 3.9 | 77.0 | 84.7% | 51.1%1.8% false alarms | 0.20 sp90 0.29 s8 at a time | $0.093 | API |
| 21 | GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details | 4.0 | 65.6 | 77.1% | 60.8%10.8% false alarms | 0.04 sp90 0.04 s1 at a time | Our GPU | 44 sRTX PRO 6000 |
| 22 | OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails | 4.1 | 50.0 | 66.7% | 33.4%14.4% false alarms | 0.03 sp90 0.04 s1 at a time | Our GPU | 43 sDGX Spark |
| 23 | Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details | 4.2 | 78.8 | 85.9% | 67.5%5.6% false alarms | 0.06 sp90 0.08 s1 at a time | Our GPU | 61 sRTX PRO 6000 |
| 24 | KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details | 4.5 | 75.3 | 83.5% | 35.9%1.4% false alarms | 0.10 sp90 0.16 s1 at a time | Our GPU | 128 sRTX PRO 6000 |
| 25 | KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details | 4.5 | 80.6 | 87.1% | 62.0%3.4% false alarms | 0.04 sp90 0.06 s1 at a time | Our GPU | 49 sRTX PRO 6000 |
| 26 | GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details | 4.7 | 84.9 | 89.9% | 69.3%1.8% false alarms | 0.23 sp90 0.36 s1 at a time | Our GPU | 254 sRTX PRO 6000 |
| 27 | REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details | 4.8 | 87.0 | 91.3% | 67.5%1.2% false alarms | 0.68 sp90 1.05 s1 at a time | Our GPU | 745 sRTX PRO 6000 |
| 28 | Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details | 4.9 | 80.4 | 87.0% | 65.0%4.9% false alarms | 0.07 sp90 0.09 s1 at a time | Our GPU | 71 sRTX PRO 6000 |
| 29 | Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details | 5.0 | 60.3 | 73.6% | 4.6%0.8% false alarms | 0.04 sp90 0.05 s1 at a time | Our GPU | 43 sRTX PRO 6000 |
| 30 | StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails | 5.6 | 48.9 | 65.9% | 72.3%26.5% false alarms | 0.04 sp90 0.06 s1 at a time | Our GPU | 53 sDGX Spark |
| 31 | KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails | 5.7 | 88.8 | 92.6% | 83.6%3.0% false alarms | 2.54 sp90 3.88 s1 at a time | Our GPU | 2,613 sDGX Spark |
| 32 | Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details | 5.9 | 69.8 | 79.8% | 37.7%3.9% false alarms | 0.15 sp90 0.18 s1 at a time | Our GPU | 156 sRTX PRO 6000 |
| 33 | Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details | 5.9 | 43.3 | 62.2% | 31.9%17.6% false alarms | 0.65 sp90 0.75 s8 at a time | $0.035 | API |
| 34 | Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details | 6.5 | 72.9 | 81.9% | 29.2%2.1% false alarms | 0.09 sp90 0.13 s1 at a time | Our GPU | 93 sRTX PRO 6000 |
| 35 | Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details | 6.7 | 78.1 | 85.4% | 69.0%6.0% false alarms | 0.07 sp90 0.11 s1 at a time | Our GPU | 77 sRTX PRO 6000 |
| 36 | Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails | 6.7 | 86.4 | 91.0% | 66.3%1.7% false alarms | 1.12 sp90 1.96 s1 at a time | Our GPU | 1,329 sDGX Spark |
| 37 | Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details | 6.8 | 61.7 | 74.5% | 23.1%3.4% false alarms | 0.12 sp90 0.19 s1 at a time | Our GPU | 134 sRTX PRO 6000 |
| 38 | KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details | 6.8 | 80.9 | 87.2% | 61.1%3.6% false alarms | 0.16 sp90 0.25 s1 at a time | Our GPU | 195 sRTX PRO 6000 |
| 39 | Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails | 7.0 | 88.1 | 92.1% | 79.3%2.1% false alarms | 2.67 sp90 3.93 s1 at a time | Our GPU | 2,824 sDGX Spark |
| 40 | Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details | 7.0 | 85.1 | 90.1% | 63.8%0.9% false alarms | 0.64 sp90 1.01 s1 at a time | Our GPU | 723 sRTX PRO 6000 |
| 41 | KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details | 7.2 | 87.7 | 91.8% | 86.3%3.6% false alarms | 0.39 sp90 0.61 s1 at a time | Our GPU | 449 sRTX PRO 6000 |
| 42 | d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details | 7.2 | 49.6 | 66.4% | 85.7%32.2% false alarms | 0.02 sp90 0.03 s1 at a time | Our GPU | 25 sRTX PRO 6000 |
| 43 | StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails | 7.2 | 66.0 | 77.3% | 67.5%14.0% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU | 78 sDGX Spark |
| 44 | GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details | 7.2 | 82.2 | 88.2% | 69.9%3.8% false alarms | 0.08 sp90 0.11 s1 at a time | Our GPU | 90 sRTX PRO 6000 |
| 45 | GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails | 7.7 | 44.4 | 62.9% | 31.0%16.8% false alarms | 0.03 sp90 0.04 s1 at a time | Our GPU | 41 sDGX Spark |
| 46 | StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails | 8.2 | 78.8 | 85.9% | 88.8%9.5% false alarms | 0.23 sp90 0.36 s1 at a time | Our GPU | 268 sDGX Spark |
| 47 | Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details | 8.3 | 76.6 | 84.4% | 51.7%3.8% false alarms | 0.12 sp90 0.18 s1 at a time | Our GPU | 132 sRTX PRO 6000 |
| 48 | StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails | 8.3 | 81.7 | 87.8% | 89.4%8.3% false alarms | 0.12 sp90 0.19 s1 at a time | Our GPU | 140 sDGX Spark |
| 49 | d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details | 8.7 | 82.5 | 88.4% | 70.5%4.1% false alarms | 0.25 sp90 0.45 s8 at a time | $0.017 | API |
| 50 | d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details | 8.8 | 86.4 | 91.0% | 85.4%4.4% false alarms | 0.23 sp90 0.35 s8 at a time | $0.035 | API |
| 51 | GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails | 9.8 | 44.3 | 62.9% | 34.9%10.6% false alarms | 0.07 sp90 0.18 s1 at a time | Our GPU | 128 sDGX Spark |
| 52 | Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details | 11.0 | 89.0 | 92.7% | 81.5%2.5% false alarms | 0.52 sp90 0.83 s1 at a time | Our GPU | 577 sRTX PRO 6000 |
| 53 | d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details | 11.4 | 84.7 | 89.8% | 80.8%4.5% false alarms | 0.48 sp90 0.90 s8 at a time | $0.035 | API |
| 54 | vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details | 12.4 | 83.7 | 89.1% | 76.0%5.7% false alarms | 0.06 sp90 0.06 s1 at a time | Our GPU | 60 sRTX PRO 6000 |
| 55 | KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails | 14.6 | 38.5 | 59.0% | 10.0%3.6% false alarms | 0.07 sp90 0.10 s1 at a time | Our GPU | 81 sDGX Spark |
| Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details | No probabilities | 86.2 | 90.8% | 73.6%2.4% false alarms | 2.70 sp90 16.4 s1 at a time | $1.49 | API | |
| Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails | No probabilities | 83.7 | 89.1% | 68.7%2.2% false alarms | 6.37 sp90 19.2 s1 at a time | Not verified | API | |
| MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details | No probabilities | 79.279.0–79.6min–max of 3 seeds | 86.2%86.0%–86.4%min–max of 3 seeds | 77.4%6.5% false alarmsmean of 3 seeds | 5.00 sp90 8.72 s30 at a time3 seeds pooled | Our GPU | 197 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details | No probabilities | 77.2 | 84.8% | 67.5%4.5% false alarms | 9.38 sp90 11.9 s30 at a time | Our GPU | 326 sRTX PRO 6000 | |
| Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details | No probabilities | 77.0 | 84.7% | 52.0%2.6% false alarms | 5.12 sp90 15.7 s64 at a time | Our GPU | 121 sRTX PRO 6000 | |
| Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details | No probabilities | 75.9 | 83.9% | 79.0%9.0% false alarms | 19.6 sp90 64.5 s30 at a time | Our GPU | 1,050 sRTX PRO 6000 | |
| LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details | No probabilities | 75.874.9–76.5min–max of 3 seeds | 83.9%83.3%–84.3%min–max of 3 seeds | 63.3%4.1% false alarmsmean of 3 seeds | 5.57 sp90 16.4 s30 at a time3 seeds pooled | Our GPU | 268 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details | No probabilities | 75.174.1–75.9min–max of 3 seeds | 83.4%82.7%–84.0%min–max of 3 seeds | 64.3%4.3% false alarmsmean of 3 seeds | 6.13 sp90 9.70 s30 at a time3 seeds pooled | Our GPU | 234 sRTX PRO 6000mean of 3 seeds | |
| LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details | No probabilities | 74.973.5–75.6min–max of 3 seeds | 83.3%82.3%–83.8%min–max of 3 seeds | 62.6%3.9% false alarmsmean of 3 seeds | 7.62 sp90 22.3 s64 at a time3 seeds pooled | Our GPU | 180 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details | No probabilities | 71.1 | 80.8% | 49.5%5.1% false alarms | 4.02 sp90 5.85 s64 at a time | Our GPU | 69 sRTX PRO 6000 | |
| Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details | No probabilities | 70.3 | 80.2% | 55.3%4.3% false alarms | 36.8 sp90 66.9 s64 at a time | Our GPU | 636 sRTX PRO 6000 | |
| Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details | No probabilities | 67.7 | 78.4% | 38.0%0.7% false alarms | 32.6 sp90 1640.2 s64 at a time | Our GPU | 4,327 sRTX PRO 6000 | |
| MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details | No probabilities | 67.066.9–67.0min–max of 3 seeds | 78.0%77.96%–78.01%min–max of 3 seeds | 28.8%2.7% false alarmsmean of 3 seeds | 0.86 sp90 3.48 s64 at a time3 seeds pooled | Our GPU | 51 sRTX PRO 6000mean of 3 seeds | |
| Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details | No probabilities | 66.2 | 77.5% | 73.2%6.2% false alarms | 37.8 sp90 96.1 s64 at a time | Our GPU | 741 sRTX PRO 6000 | |
| Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details | No probabilities | 64.664.5–64.8min–max of 3 seeds | 76.4%76.3%–76.6%min–max of 3 seeds | 61.2%7.6% false alarmsmean of 3 seeds | 3.49 sp90 20.4 s64 at a time3 seeds pooled | Our GPU | 148 sRTX PRO 6000mean of 3 seeds | |
| LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details | No probabilities | 57.7 | 71.8% | 8.8%0.8% false alarms | 9.58 sp90 15.2 s30 at a time | Our GPU | 352 sRTX PRO 6000 | |
| Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details | No probabilities | 55.254.5–55.8min–max of 3 seeds | 70.1%69.7%–70.6%min–max of 3 seeds | 7.7%1.6% false alarmsmean of 3 seeds | 0.18 sp90 5.42 s64 at a time3 seeds pooled | Our GPU | 33 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details | No probabilities | 54.8 | 69.8% | 6.1%1.3% false alarms | 0.19 sp90 3.31 s64 at a time | Our GPU | 18 sRTX PRO 6000 | |
| Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details | No probabilities | 50.249.3–50.8min–max of 3 seeds | 66.8%66.2%–67.2%min–max of 3 seeds | 34.4%7.8% false alarmsmean of 3 seeds | 1.53 sp90 13.7 s64 at a time3 seeds pooled | Our GPU | 92 sRTX PRO 6000mean of 3 seeds | |
| Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details | No probabilities | 40.640.6–40.8min–max of 3 seeds | 60.4%60.4%–60.5%min–max of 3 seeds | 19.9%2.0% false alarmsmean of 3 seeds | 3.03 sp90 7.76 s30 at a time3 seeds pooled | Our GPU | 209 sRTX PRO 6000mean of 3 seeds |
No row of this kind yet.
2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology
Wrong answers higher is better
No row of this kind yet.
- StartLux
- PelaAI
- Liquid AI
- Perplexity
- Microsoft
- Poindexter Labs
- MachinePulse
- TypeSafe
- Sthānika AI
- IBM
- Subquadratic
- OpenBMB
- scar-ai
- Fastino
- Thinking Machines
- Alibaba
- quincyqiang
- REAX
- Turion.AI
- Apertix AI
- newsbang
- Patrick-18
- Logits
- tattrongvu
- EldanRing
- victorbennett
- Base Labs
- Valen-Team
- Cloudflare
- Knowledgator
- OmniJev
- meraGPT
- Blockbrain Labs
- Metask Lab
- KRAFTON
- Strands Agents
| The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –. | Model | Of the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarms | (accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100 | Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairs | Expected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, points | Client wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under it | Input tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairs | Measured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs |
|---|---|---|---|---|---|---|---|---|
| 1 | StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails | 89.4%8.3% false alarms | 81.7 | 87.8% | 8.3 | 0.12 sp90 0.19 s1 at a time | Our GPU | 140 sDGX Spark |
| 2 | StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails | 88.8%9.5% false alarms | 78.8 | 85.9% | 8.2 | 0.23 sp90 0.36 s1 at a time | Our GPU | 268 sDGX Spark |
| 3 | KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details | 86.3%3.6% false alarms | 87.7 | 91.8% | 7.2 | 0.39 sp90 0.61 s1 at a time | Our GPU | 449 sRTX PRO 6000 |
| 4 | d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details | 85.7%32.2% false alarms | 49.6 | 66.4% | 7.2 | 0.02 sp90 0.03 s1 at a time | Our GPU | 25 sRTX PRO 6000 |
| 5 | d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details | 85.4%4.4% false alarms | 86.4 | 91.0% | 8.8 | 0.23 sp90 0.35 s8 at a time | $0.035 | API |
| 6 | KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails | 83.6%3.0% false alarms | 88.8 | 92.6% | 5.7 | 2.54 sp90 3.88 s1 at a time | Our GPU | 2,613 sDGX Spark |
| 7 | Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details | 82.7%3.5% false alarms | 86.2 | 90.8% | 2.8 | 0.22 sp90 0.34 s1 at a time | Our GPU | 245 sRTX PRO 6000 |
| 8 | Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details | 82.4%6.5% false alarms | 83.9 | 89.3% | 2.4 | 0.32 sp90 0.38 s8 at a time | $0.036 | API |
| 9 | Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details | 81.5%2.5% false alarms | 89.0 | 92.7% | 11.0 | 0.52 sp90 0.83 s1 at a time | Our GPU | 577 sRTX PRO 6000 |
| 10 | d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details | 80.8%4.5% false alarms | 84.7 | 89.8% | 11.4 | 0.48 sp90 0.90 s8 at a time | $0.035 | API |
| 11 | Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details | 80.8%1.7% false alarms | 90.0 | 93.3% | 2.8 | 0.22 sp90 0.31 s1 at a time | Our GPU | 230 sRTX PRO 6000 |
| 12 | Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details | 79.9%6.0% false alarms | 83.0 | 88.6% | 1.9 | 0.17 sp90 0.19 s8 at a time | $0.049 | API |
| 13 | Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails | 79.3%2.1% false alarms | 88.1 | 92.1% | 7.0 | 2.67 sp90 3.93 s1 at a time | Our GPU | 2,824 sDGX Spark |
| Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details | 79.0%9.0% false alarms | 75.9 | 83.9% | No probabilities | 19.6 sp90 64.5 s30 at a time | Our GPU | 1,050 sRTX PRO 6000 | |
| 14 | SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails | 78.1%4.1% false alarms | 84.9 | 89.9% | 2.3 | 0.39 sp90 0.41 s8 at a time | Not verified | API |
| MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details | 77.4%6.5% false alarms · mean of 3 seeds | 79.279.0–79.6min–max of 3 seeds | 86.2%86.0%–86.4%min–max of 3 seeds | No probabilities | 5.00 sp90 8.72 s30 at a time3 seeds pooled | Our GPU | 197 sRTX PRO 6000mean of 3 seeds | |
| 15 | vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details | 76.0%5.7% false alarms | 83.7 | 89.1% | 12.4 | 0.06 sp90 0.06 s1 at a time | Our GPU | 60 sRTX PRO 6000 |
| 16 | GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails | 74.8%1.5% false alarms | 86.1 | 90.7% | 3.6 | 1.03 sp90 1.95 s1 at a time | Our GPU | 1,205 sDGX Spark |
| Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details | 73.6%2.4% false alarms | 86.2 | 90.8% | No probabilities | 2.70 sp90 16.4 s1 at a time | $1.49 | API | |
| Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details | 73.2%6.2% false alarms | 66.2 | 77.5% | No probabilities | 37.8 sp90 96.1 s64 at a time | Our GPU | 741 sRTX PRO 6000 | |
| 17 | StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails | 72.3%26.5% false alarms | 48.9 | 65.9% | 5.6 | 0.04 sp90 0.06 s1 at a time | Our GPU | 53 sDGX Spark |
| 18 | Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details | 72.0%3.5% false alarms | 83.6 | 89.1% | 2.8 | 0.18 sp90 0.20 s8 at a time | $0.049 | API |
| 19 | d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details | 70.5%4.1% false alarms | 82.5 | 88.4% | 8.7 | 0.25 sp90 0.45 s8 at a time | $0.017 | API |
| 20 | GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details | 69.9%3.8% false alarms | 82.2 | 88.2% | 7.2 | 0.08 sp90 0.11 s1 at a time | Our GPU | 90 sRTX PRO 6000 |
| 21 | GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details | 69.3%1.8% false alarms | 84.9 | 89.9% | 4.7 | 0.23 sp90 0.36 s1 at a time | Our GPU | 254 sRTX PRO 6000 |
| 22 | Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details | 69.0%6.0% false alarms | 78.1 | 85.4% | 6.7 | 0.07 sp90 0.11 s1 at a time | Our GPU | 77 sRTX PRO 6000 |
| Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails | 68.7%2.2% false alarms | 83.7 | 89.1% | No probabilities | 6.37 sp90 19.2 s1 at a time | Not verified | API | |
| 23 | GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails | 68.7%1.7% false alarms | 85.1 | 90.0% | 3.0 | 0.30 sp90 0.56 s8 at a time | Not verified | API |
| Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details | 67.5%4.5% false alarms | 77.2 | 84.8% | No probabilities | 9.38 sp90 11.9 s30 at a time | Our GPU | 326 sRTX PRO 6000 | |
| 24 | Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details | 67.5%5.6% false alarms | 78.8 | 85.9% | 4.2 | 0.06 sp90 0.08 s1 at a time | Our GPU | 61 sRTX PRO 6000 |
| 25 | REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details | 67.5%1.2% false alarms | 87.0 | 91.3% | 4.8 | 0.68 sp90 1.05 s1 at a time | Our GPU | 745 sRTX PRO 6000 |
| 26 | StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails | 67.5%14.0% false alarms | 66.0 | 77.3% | 7.2 | 0.06 sp90 0.09 s1 at a time | Our GPU | 78 sDGX Spark |
| 27 | d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails | 66.3%5.3% false alarms | 79.2 | 86.1% | 3.1 | 0.27 sp90 0.60 s8 at a time | Not verified | API |
| 28 | Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails | 66.3%1.7% false alarms | 86.4 | 91.0% | 6.7 | 1.12 sp90 1.96 s1 at a time | Our GPU | 1,329 sDGX Spark |
| 29 | Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details | 65.3%3.0% false alarms | 81.9 | 87.9% | 2.9 | 0.11 sp90 0.18 s1 at a time | Our GPU | 126 sRTX PRO 6000 |
| 30 | Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details | 65.0%4.9% false alarms | 80.4 | 87.0% | 4.9 | 0.07 sp90 0.09 s1 at a time | Our GPU | 71 sRTX PRO 6000 |
| 31 | Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details | 64.7%3.9% false alarms | 81.1 | 87.4% | 2.5 | 0.06 sp90 0.09 s1 at a time | Our GPU | 72 sRTX PRO 6000 |
| Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details | 64.3%4.3% false alarms · mean of 3 seeds | 75.174.1–75.9min–max of 3 seeds | 83.4%82.7%–84.0%min–max of 3 seeds | No probabilities | 6.13 sp90 9.70 s30 at a time3 seeds pooled | Our GPU | 234 sRTX PRO 6000mean of 3 seeds | |
| 32 | Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details | 63.8%0.9% false alarms | 85.1 | 90.1% | 7.0 | 0.64 sp90 1.01 s1 at a time | Our GPU | 723 sRTX PRO 6000 |
| LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details | 63.3%4.1% false alarms · mean of 3 seeds | 75.874.9–76.5min–max of 3 seeds | 83.9%83.3%–84.3%min–max of 3 seeds | No probabilities | 5.57 sp90 16.4 s30 at a time3 seeds pooled | Our GPU | 268 sRTX PRO 6000mean of 3 seeds | |
| LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details | 62.6%3.9% false alarms · mean of 3 seeds | 74.973.5–75.6min–max of 3 seeds | 83.3%82.3%–83.8%min–max of 3 seeds | No probabilities | 7.62 sp90 22.3 s64 at a time3 seeds pooled | Our GPU | 180 sRTX PRO 6000mean of 3 seeds | |
| 33 | KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details | 62.0%3.4% false alarms | 80.6 | 87.1% | 4.5 | 0.04 sp90 0.06 s1 at a time | Our GPU | 49 sRTX PRO 6000 |
| Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details | 61.2%7.6% false alarms · mean of 3 seeds | 64.664.5–64.8min–max of 3 seeds | 76.4%76.3%–76.6%min–max of 3 seeds | No probabilities | 3.49 sp90 20.4 s64 at a time3 seeds pooled | Our GPU | 148 sRTX PRO 6000mean of 3 seeds | |
| 34 | KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details | 61.1%3.6% false alarms | 80.9 | 87.2% | 6.8 | 0.16 sp90 0.25 s1 at a time | Our GPU | 195 sRTX PRO 6000 |
| 35 | GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details | 60.8%10.8% false alarms | 65.6 | 77.1% | 4.0 | 0.04 sp90 0.04 s1 at a time | Our GPU | 44 sRTX PRO 6000 |
| 36 | Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details | 59.9%2.5% false alarms | 83.0 | 88.6% | 3.3 | 0.09 sp90 0.14 s1 at a time | Our GPU | 100 sRTX PRO 6000 |
| Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details | 55.3%4.3% false alarms | 70.3 | 80.2% | No probabilities | 36.8 sp90 66.9 s64 at a time | Our GPU | 636 sRTX PRO 6000 | |
| Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details | 52.0%2.6% false alarms | 77.0 | 84.7% | No probabilities | 5.12 sp90 15.7 s64 at a time | Our GPU | 121 sRTX PRO 6000 | |
| 37 | Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details | 51.7%3.8% false alarms | 76.6 | 84.4% | 8.3 | 0.12 sp90 0.18 s1 at a time | Our GPU | 132 sRTX PRO 6000 |
| 38 | Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details | 51.1%1.8% false alarms | 77.0 | 84.7% | 3.9 | 0.20 sp90 0.29 s8 at a time | $0.093 | API |
| 39 | KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details | 50.8%3.8% false alarms | 76.1 | 84.1% | 3.6 | 0.10 sp90 0.14 s1 at a time | Our GPU | 105 sRTX PRO 6000 |
| Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details | 49.5%5.1% false alarms | 71.1 | 80.8% | No probabilities | 4.02 sp90 5.85 s64 at a time | Our GPU | 69 sRTX PRO 6000 | |
| 40 | Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details | 47.7%4.0% false alarms | 77.2 | 84.8% | 2.8 | 0.06 sp90 0.09 s1 at a time | Our GPU | 65 sRTX PRO 6000 |
| 41 | Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details | 46.8%3.7% false alarms | 75.4 | 83.6% | 3.4 | 0.05 sp90 0.07 s1 at a time | Our GPU | 54 sRTX PRO 6000 |
| 42 | Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details | 41.0%1.9% false alarms | 76.9 | 84.6% | 3.7 | 0.09 sp90 0.11 s1 at a time | Our GPU | 98 sRTX PRO 6000 |
| 43 | Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details | 39.8%2.8% false alarms | 74.3 | 82.8% | 3.2 | 0.07 sp90 0.10 s1 at a time | Our GPU | 78 sRTX PRO 6000 |
| Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details | 38.0%0.7% false alarms | 67.7 | 78.4% | No probabilities | 32.6 sp90 1640.2 s64 at a time | Our GPU | 4,327 sRTX PRO 6000 | |
| 44 | Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details | 37.7%3.9% false alarms | 69.8 | 79.8% | 5.9 | 0.15 sp90 0.18 s1 at a time | Our GPU | 156 sRTX PRO 6000 |
| 45 | KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details | 35.9%1.4% false alarms | 75.3 | 83.5% | 4.5 | 0.10 sp90 0.16 s1 at a time | Our GPU | 128 sRTX PRO 6000 |
| 46 | GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails | 34.9%10.6% false alarms | 44.3 | 62.9% | 9.8 | 0.07 sp90 0.18 s1 at a time | Our GPU | 128 sDGX Spark |
| Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details | 34.4%7.8% false alarms · mean of 3 seeds | 50.249.3–50.8min–max of 3 seeds | 66.8%66.2%–67.2%min–max of 3 seeds | No probabilities | 1.53 sp90 13.7 s64 at a time3 seeds pooled | Our GPU | 92 sRTX PRO 6000mean of 3 seeds | |
| 47 | OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails | 33.4%14.4% false alarms | 50.0 | 66.7% | 4.1 | 0.03 sp90 0.04 s1 at a time | Our GPU | 43 sDGX Spark |
| 48 | Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details | 31.9%17.6% false alarms | 43.3 | 62.2% | 5.9 | 0.65 sp90 0.75 s8 at a time | $0.035 | API |
| 49 | GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails | 31.3%6.6% false alarms | 60.8 | 73.9% | 3.8 | 0.05 sp90 0.07 s1 at a time | Our GPU | 63 sDGX Spark |
| 50 | GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails | 31.0%16.8% false alarms | 44.4 | 62.9% | 7.7 | 0.03 sp90 0.04 s1 at a time | Our GPU | 41 sDGX Spark |
| 51 | Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details | 29.2%2.1% false alarms | 72.9 | 81.9% | 6.5 | 0.09 sp90 0.13 s1 at a time | Our GPU | 93 sRTX PRO 6000 |
| MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details | 28.8%2.7% false alarms · mean of 3 seeds | 67.066.9–67.0min–max of 3 seeds | 78.0%77.96%–78.01%min–max of 3 seeds | No probabilities | 0.86 sp90 3.48 s64 at a time3 seeds pooled | Our GPU | 51 sRTX PRO 6000mean of 3 seeds | |
| 52 | Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details | 23.1%3.4% false alarms | 61.7 | 74.5% | 6.8 | 0.12 sp90 0.19 s1 at a time | Our GPU | 134 sRTX PRO 6000 |
| Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details | 19.9%2.0% false alarms · mean of 3 seeds | 40.640.6–40.8min–max of 3 seeds | 60.4%60.4%–60.5%min–max of 3 seeds | No probabilities | 3.03 sp90 7.76 s30 at a time3 seeds pooled | Our GPU | 209 sRTX PRO 6000mean of 3 seeds | |
| 53 | KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails | 16.4%2.7% false alarms | 61.1 | 74.0% | 2.7 | 0.11 sp90 0.17 s1 at a time | Our GPU | 130 sDGX Spark |
| 54 | KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails | 10.0%3.6% false alarms | 38.5 | 59.0% | 14.6 | 0.07 sp90 0.10 s1 at a time | Our GPU | 81 sDGX Spark |
| LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details | 8.8%0.8% false alarms | 57.7 | 71.8% | No probabilities | 9.58 sp90 15.2 s30 at a time | Our GPU | 352 sRTX PRO 6000 | |
| Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details | 7.7%1.6% false alarms · mean of 3 seeds | 55.254.5–55.8min–max of 3 seeds | 70.1%69.7%–70.6%min–max of 3 seeds | No probabilities | 0.18 sp90 5.42 s64 at a time3 seeds pooled | Our GPU | 33 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details | 6.1%1.3% false alarms | 54.8 | 69.8% | No probabilities | 0.19 sp90 3.31 s64 at a time | Our GPU | 18 sRTX PRO 6000 | |
| 55 | Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details | 4.6%0.8% false alarms | 60.3 | 73.6% | 5.0 | 0.04 sp90 0.05 s1 at a time | Our GPU | 43 sRTX PRO 6000 |
No row of this kind yet.
2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology
Latency lower is better
No row of this kind yet.
- Liquid AI
- Fastino
- OmniJev
- Strands Agents
- StartLux
- PelaAI
- victorbennett
- Perplexity
- scar-ai
- Patrick-18
- newsbang
- quincyqiang
- Valen-Team
- Knowledgator
- Blockbrain Labs
- Base Labs
- tattrongvu
- Apertix AI
- Metask Lab
- EldanRing
- Cloudflare
- TypeSafe
- MachinePulse
- Microsoft
- Subquadratic
- Poindexter Labs
- Logits
- meraGPT
- REAX
- OpenBMB
- Turion.AI
- Alibaba
- Sthānika AI
- Thinking Machines
- KRAFTON
- IBM
| The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –. | Model | Client wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under it | (accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100 | Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairs | Expected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, points | Of the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarms | Input tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairs | Measured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs |
|---|---|---|---|---|---|---|---|---|
| 1 | d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details | 0.02 sp90 0.03 s · 1 at a time | 49.6 | 66.4% | 7.2 | 85.7%32.2% false alarms | Our GPU | 25 sRTX PRO 6000 |
| 2 | GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails | 0.03 sp90 0.04 s · 1 at a time | 44.4 | 62.9% | 7.7 | 31.0%16.8% false alarms | Our GPU | 41 sDGX Spark |
| 3 | OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails | 0.03 sp90 0.04 s · 1 at a time | 50.0 | 66.7% | 4.1 | 33.4%14.4% false alarms | Our GPU | 43 sDGX Spark |
| 4 | GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details | 0.04 sp90 0.04 s · 1 at a time | 65.6 | 77.1% | 4.0 | 60.8%10.8% false alarms | Our GPU | 44 sRTX PRO 6000 |
| 5 | Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details | 0.04 sp90 0.05 s · 1 at a time | 60.3 | 73.6% | 5.0 | 4.6%0.8% false alarms | Our GPU | 43 sRTX PRO 6000 |
| 6 | StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails | 0.04 sp90 0.06 s · 1 at a time | 48.9 | 65.9% | 5.6 | 72.3%26.5% false alarms | Our GPU | 53 sDGX Spark |
| 7 | KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details | 0.04 sp90 0.06 s · 1 at a time | 80.6 | 87.1% | 4.5 | 62.0%3.4% false alarms | Our GPU | 49 sRTX PRO 6000 |
| 8 | GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails | 0.05 sp90 0.07 s · 1 at a time | 60.8 | 73.9% | 3.8 | 31.3%6.6% false alarms | Our GPU | 63 sDGX Spark |
| 9 | Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details | 0.05 sp90 0.07 s · 1 at a time | 75.4 | 83.6% | 3.4 | 46.8%3.7% false alarms | Our GPU | 54 sRTX PRO 6000 |
| 10 | Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details | 0.06 sp90 0.08 s · 1 at a time | 78.8 | 85.9% | 4.2 | 67.5%5.6% false alarms | Our GPU | 61 sRTX PRO 6000 |
| 11 | vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details | 0.06 sp90 0.06 s · 1 at a time | 83.7 | 89.1% | 12.4 | 76.0%5.7% false alarms | Our GPU | 60 sRTX PRO 6000 |
| 12 | Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details | 0.06 sp90 0.09 s · 1 at a time | 77.2 | 84.8% | 2.8 | 47.7%4.0% false alarms | Our GPU | 65 sRTX PRO 6000 |
| 13 | Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details | 0.06 sp90 0.09 s · 1 at a time | 81.1 | 87.4% | 2.5 | 64.7%3.9% false alarms | Our GPU | 72 sRTX PRO 6000 |
| 14 | StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails | 0.06 sp90 0.09 s · 1 at a time | 66.0 | 77.3% | 7.2 | 67.5%14.0% false alarms | Our GPU | 78 sDGX Spark |
| 15 | Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details | 0.07 sp90 0.09 s · 1 at a time | 80.4 | 87.0% | 4.9 | 65.0%4.9% false alarms | Our GPU | 71 sRTX PRO 6000 |
| 16 | KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails | 0.07 sp90 0.10 s · 1 at a time | 38.5 | 59.0% | 14.6 | 10.0%3.6% false alarms | Our GPU | 81 sDGX Spark |
| 17 | Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details | 0.07 sp90 0.11 s · 1 at a time | 78.1 | 85.4% | 6.7 | 69.0%6.0% false alarms | Our GPU | 77 sRTX PRO 6000 |
| 18 | Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details | 0.07 sp90 0.10 s · 1 at a time | 74.3 | 82.8% | 3.2 | 39.8%2.8% false alarms | Our GPU | 78 sRTX PRO 6000 |
| 19 | GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails | 0.07 sp90 0.18 s · 1 at a time | 44.3 | 62.9% | 9.8 | 34.9%10.6% false alarms | Our GPU | 128 sDGX Spark |
| 20 | GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details | 0.08 sp90 0.11 s · 1 at a time | 82.2 | 88.2% | 7.2 | 69.9%3.8% false alarms | Our GPU | 90 sRTX PRO 6000 |
| 21 | Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details | 0.09 sp90 0.13 s · 1 at a time | 72.9 | 81.9% | 6.5 | 29.2%2.1% false alarms | Our GPU | 93 sRTX PRO 6000 |
| 22 | Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details | 0.09 sp90 0.11 s · 1 at a time | 76.9 | 84.6% | 3.7 | 41.0%1.9% false alarms | Our GPU | 98 sRTX PRO 6000 |
| 23 | Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details | 0.09 sp90 0.14 s · 1 at a time | 83.0 | 88.6% | 3.3 | 59.9%2.5% false alarms | Our GPU | 100 sRTX PRO 6000 |
| 24 | KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details | 0.10 sp90 0.14 s · 1 at a time | 76.1 | 84.1% | 3.6 | 50.8%3.8% false alarms | Our GPU | 105 sRTX PRO 6000 |
| 25 | KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details | 0.10 sp90 0.16 s · 1 at a time | 75.3 | 83.5% | 4.5 | 35.9%1.4% false alarms | Our GPU | 128 sRTX PRO 6000 |
| 26 | KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails | 0.11 sp90 0.17 s · 1 at a time | 61.1 | 74.0% | 2.7 | 16.4%2.7% false alarms | Our GPU | 130 sDGX Spark |
| 27 | Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details | 0.11 sp90 0.18 s · 1 at a time | 81.9 | 87.9% | 2.9 | 65.3%3.0% false alarms | Our GPU | 126 sRTX PRO 6000 |
| 28 | Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details | 0.12 sp90 0.19 s · 1 at a time | 61.7 | 74.5% | 6.8 | 23.1%3.4% false alarms | Our GPU | 134 sRTX PRO 6000 |
| 29 | StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails | 0.12 sp90 0.19 s · 1 at a time | 81.7 | 87.8% | 8.3 | 89.4%8.3% false alarms | Our GPU | 140 sDGX Spark |
| 30 | Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details | 0.12 sp90 0.18 s · 1 at a time | 76.6 | 84.4% | 8.3 | 51.7%3.8% false alarms | Our GPU | 132 sRTX PRO 6000 |
| 31 | Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details | 0.15 sp90 0.18 s · 1 at a time | 69.8 | 79.8% | 5.9 | 37.7%3.9% false alarms | Our GPU | 156 sRTX PRO 6000 |
| 32 | KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details | 0.16 sp90 0.25 s · 1 at a time | 80.9 | 87.2% | 6.8 | 61.1%3.6% false alarms | Our GPU | 195 sRTX PRO 6000 |
| 33 | Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details | 0.17 sp90 0.19 s · 8 at a time | 83.0 | 88.6% | 1.9 | 79.9%6.0% false alarms | $0.049 | API |
| Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details | 0.18 sp90 5.42 s · 64 at a time · 3 seeds pooled | 55.254.5–55.8min–max of 3 seeds | 70.1%69.7%–70.6%min–max of 3 seeds | No probabilities | 7.7%1.6% false alarmsmean of 3 seeds | Our GPU | 33 sRTX PRO 6000mean of 3 seeds | |
| 34 | Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details | 0.18 sp90 0.20 s · 8 at a time | 83.6 | 89.1% | 2.8 | 72.0%3.5% false alarms | $0.049 | API |
| Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details | 0.19 sp90 3.31 s · 64 at a time | 54.8 | 69.8% | No probabilities | 6.1%1.3% false alarms | Our GPU | 18 sRTX PRO 6000 | |
| 35 | Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details | 0.20 sp90 0.29 s · 8 at a time | 77.0 | 84.7% | 3.9 | 51.1%1.8% false alarms | $0.093 | API |
| 36 | Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details | 0.22 sp90 0.31 s · 1 at a time | 90.0 | 93.3% | 2.8 | 80.8%1.7% false alarms | Our GPU | 230 sRTX PRO 6000 |
| 37 | Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details | 0.22 sp90 0.34 s · 1 at a time | 86.2 | 90.8% | 2.8 | 82.7%3.5% false alarms | Our GPU | 245 sRTX PRO 6000 |
| 38 | d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details | 0.23 sp90 0.35 s · 8 at a time | 86.4 | 91.0% | 8.8 | 85.4%4.4% false alarms | $0.035 | API |
| 39 | StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails | 0.23 sp90 0.36 s · 1 at a time | 78.8 | 85.9% | 8.2 | 88.8%9.5% false alarms | Our GPU | 268 sDGX Spark |
| 40 | GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details | 0.23 sp90 0.36 s · 1 at a time | 84.9 | 89.9% | 4.7 | 69.3%1.8% false alarms | Our GPU | 254 sRTX PRO 6000 |
| 41 | d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details | 0.25 sp90 0.45 s · 8 at a time | 82.5 | 88.4% | 8.7 | 70.5%4.1% false alarms | $0.017 | API |
| 42 | d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails | 0.27 sp90 0.60 s · 8 at a time | 79.2 | 86.1% | 3.1 | 66.3%5.3% false alarms | Not verified | API |
| 43 | GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails | 0.30 sp90 0.56 s · 8 at a time | 85.1 | 90.0% | 3.0 | 68.7%1.7% false alarms | Not verified | API |
| 44 | Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details | 0.32 sp90 0.38 s · 8 at a time | 83.9 | 89.3% | 2.4 | 82.4%6.5% false alarms | $0.036 | API |
| 45 | KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details | 0.39 sp90 0.61 s · 1 at a time | 87.7 | 91.8% | 7.2 | 86.3%3.6% false alarms | Our GPU | 449 sRTX PRO 6000 |
| 46 | SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails | 0.39 sp90 0.41 s · 8 at a time | 84.9 | 89.9% | 2.3 | 78.1%4.1% false alarms | Not verified | API |
| 47 | d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details | 0.48 sp90 0.90 s · 8 at a time | 84.7 | 89.8% | 11.4 | 80.8%4.5% false alarms | $0.035 | API |
| 48 | Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details | 0.52 sp90 0.83 s · 1 at a time | 89.0 | 92.7% | 11.0 | 81.5%2.5% false alarms | Our GPU | 577 sRTX PRO 6000 |
| 49 | Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details | 0.64 sp90 1.01 s · 1 at a time | 85.1 | 90.1% | 7.0 | 63.8%0.9% false alarms | Our GPU | 723 sRTX PRO 6000 |
| 50 | Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details | 0.65 sp90 0.75 s · 8 at a time | 43.3 | 62.2% | 5.9 | 31.9%17.6% false alarms | $0.035 | API |
| 51 | REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details | 0.68 sp90 1.05 s · 1 at a time | 87.0 | 91.3% | 4.8 | 67.5%1.2% false alarms | Our GPU | 745 sRTX PRO 6000 |
| MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details | 0.86 sp90 3.48 s · 64 at a time · 3 seeds pooled | 67.066.9–67.0min–max of 3 seeds | 78.0%77.96%–78.01%min–max of 3 seeds | No probabilities | 28.8%2.7% false alarmsmean of 3 seeds | Our GPU | 51 sRTX PRO 6000mean of 3 seeds | |
| 52 | GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails | 1.03 sp90 1.95 s · 1 at a time | 86.1 | 90.7% | 3.6 | 74.8%1.5% false alarms | Our GPU | 1,205 sDGX Spark |
| 53 | Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails | 1.12 sp90 1.96 s · 1 at a time | 86.4 | 91.0% | 6.7 | 66.3%1.7% false alarms | Our GPU | 1,329 sDGX Spark |
| Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details | 1.53 sp90 13.7 s · 64 at a time · 3 seeds pooled | 50.249.3–50.8min–max of 3 seeds | 66.8%66.2%–67.2%min–max of 3 seeds | No probabilities | 34.4%7.8% false alarmsmean of 3 seeds | Our GPU | 92 sRTX PRO 6000mean of 3 seeds | |
| 54 | KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails | 2.54 sp90 3.88 s · 1 at a time | 88.8 | 92.6% | 5.7 | 83.6%3.0% false alarms | Our GPU | 2,613 sDGX Spark |
| 55 | Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails | 2.67 sp90 3.93 s · 1 at a time | 88.1 | 92.1% | 7.0 | 79.3%2.1% false alarms | Our GPU | 2,824 sDGX Spark |
| Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details | 2.70 sp90 16.4 s · 1 at a time | 86.2 | 90.8% | No probabilities | 73.6%2.4% false alarms | $1.49 | API | |
| Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details | 3.03 sp90 7.76 s · 30 at a time · 3 seeds pooled | 40.640.6–40.8min–max of 3 seeds | 60.4%60.4%–60.5%min–max of 3 seeds | No probabilities | 19.9%2.0% false alarmsmean of 3 seeds | Our GPU | 209 sRTX PRO 6000mean of 3 seeds | |
| Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details | 3.49 sp90 20.4 s · 64 at a time · 3 seeds pooled | 64.664.5–64.8min–max of 3 seeds | 76.4%76.3%–76.6%min–max of 3 seeds | No probabilities | 61.2%7.6% false alarmsmean of 3 seeds | Our GPU | 148 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details | 4.02 sp90 5.85 s · 64 at a time | 71.1 | 80.8% | No probabilities | 49.5%5.1% false alarms | Our GPU | 69 sRTX PRO 6000 | |
| MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details | 5.00 sp90 8.72 s · 30 at a time · 3 seeds pooled | 79.279.0–79.6min–max of 3 seeds | 86.2%86.0%–86.4%min–max of 3 seeds | No probabilities | 77.4%6.5% false alarmsmean of 3 seeds | Our GPU | 197 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details | 5.12 sp90 15.7 s · 64 at a time | 77.0 | 84.7% | No probabilities | 52.0%2.6% false alarms | Our GPU | 121 sRTX PRO 6000 | |
| LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details | 5.57 sp90 16.4 s · 30 at a time · 3 seeds pooled | 75.874.9–76.5min–max of 3 seeds | 83.9%83.3%–84.3%min–max of 3 seeds | No probabilities | 63.3%4.1% false alarmsmean of 3 seeds | Our GPU | 268 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details | 6.13 sp90 9.70 s · 30 at a time · 3 seeds pooled | 75.174.1–75.9min–max of 3 seeds | 83.4%82.7%–84.0%min–max of 3 seeds | No probabilities | 64.3%4.3% false alarmsmean of 3 seeds | Our GPU | 234 sRTX PRO 6000mean of 3 seeds | |
| Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails | 6.37 sp90 19.2 s · 1 at a time | 83.7 | 89.1% | No probabilities | 68.7%2.2% false alarms | Not verified | API | |
| LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details | 7.62 sp90 22.3 s · 64 at a time · 3 seeds pooled | 74.973.5–75.6min–max of 3 seeds | 83.3%82.3%–83.8%min–max of 3 seeds | No probabilities | 62.6%3.9% false alarmsmean of 3 seeds | Our GPU | 180 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details | 9.38 sp90 11.9 s · 30 at a time | 77.2 | 84.8% | No probabilities | 67.5%4.5% false alarms | Our GPU | 326 sRTX PRO 6000 | |
| LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details | 9.58 sp90 15.2 s · 30 at a time | 57.7 | 71.8% | No probabilities | 8.8%0.8% false alarms | Our GPU | 352 sRTX PRO 6000 | |
| Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details | 19.6 sp90 64.5 s · 30 at a time | 75.9 | 83.9% | No probabilities | 79.0%9.0% false alarms | Our GPU | 1,050 sRTX PRO 6000 | |
| Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details | 32.6 sp90 1640.2 s · 64 at a time | 67.7 | 78.4% | No probabilities | 38.0%0.7% false alarms | Our GPU | 4,327 sRTX PRO 6000 | |
| Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details | 36.8 sp90 66.9 s · 64 at a time | 70.3 | 80.2% | No probabilities | 55.3%4.3% false alarms | Our GPU | 636 sRTX PRO 6000 | |
| Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details | 37.8 sp90 96.1 s · 64 at a time | 66.2 | 77.5% | No probabilities | 73.2%6.2% false alarms | Our GPU | 741 sRTX PRO 6000 |
No row of this kind yet.
2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology
List price lower is better
No row of this kind yet.
- Liquid AI
- meraGPT
- Microsoft
- TypeSafe
- Thinking Machines
- MachinePulse
- Poindexter Labs
- PelaAI
- Sthānika AI
- REAX
- Turion.AI
- Perplexity
- Fastino
- Logits
- Subquadratic
- scar-ai
- tattrongvu
- Apertix AI
- StartLux
- Patrick-18
- newsbang
- OpenBMB
- quincyqiang
- Base Labs
- EldanRing
- IBM
- victorbennett
- Valen-Team
- Blockbrain Labs
- Alibaba
- Cloudflare
- Metask Lab
- Strands Agents
- OmniJev
- Knowledgator
- KRAFTON
| The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –. | Model | Input tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairs | (accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100 | Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairs | Expected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, points | Of the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarms | Client wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under it | Measured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs |
|---|---|---|---|---|---|---|---|---|
| 1 | d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details | $0.017 | 82.5 | 88.4% | 8.7 | 70.5%4.1% false alarms | 0.25 sp90 0.45 s8 at a time | API |
| 2 | d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details | $0.035 | 84.7 | 89.8% | 11.4 | 80.8%4.5% false alarms | 0.48 sp90 0.90 s8 at a time | API |
| 3 | d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details | $0.035 | 86.4 | 91.0% | 8.8 | 85.4%4.4% false alarms | 0.23 sp90 0.35 s8 at a time | API |
| 4 | Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details | $0.035 | 43.3 | 62.2% | 5.9 | 31.9%17.6% false alarms | 0.65 sp90 0.75 s8 at a time | API |
| 5 | Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details | $0.036 | 83.9 | 89.3% | 2.4 | 82.4%6.5% false alarms | 0.32 sp90 0.38 s8 at a time | API |
| 6 | Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details | $0.049 | 83.0 | 88.6% | 1.9 | 79.9%6.0% false alarms | 0.17 sp90 0.19 s8 at a time | API |
| 7 | Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details | $0.049 | 83.6 | 89.1% | 2.8 | 72.0%3.5% false alarms | 0.18 sp90 0.20 s8 at a time | API |
| 8 | Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details | $0.093 | 77.0 | 84.7% | 3.9 | 51.1%1.8% false alarms | 0.20 sp90 0.29 s8 at a time | API |
| Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details | $1.49 | 86.2 | 90.8% | No probabilities | 73.6%2.4% false alarms | 2.70 sp90 16.4 s1 at a time | API | |
| – | Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details | Our GPU | 90.0 | 93.3% | 2.8 | 80.8%1.7% false alarms | 0.22 sp90 0.31 s1 at a time | 230 sRTX PRO 6000 |
| – | Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details | Our GPU | 89.0 | 92.7% | 11.0 | 81.5%2.5% false alarms | 0.52 sp90 0.83 s1 at a time | 577 sRTX PRO 6000 |
| – | KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails | Our GPU | 88.8 | 92.6% | 5.7 | 83.6%3.0% false alarms | 2.54 sp90 3.88 s1 at a time | 2,613 sDGX Spark |
| – | Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails | Our GPU | 88.1 | 92.1% | 7.0 | 79.3%2.1% false alarms | 2.67 sp90 3.93 s1 at a time | 2,824 sDGX Spark |
| – | KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details | Our GPU | 87.7 | 91.8% | 7.2 | 86.3%3.6% false alarms | 0.39 sp90 0.61 s1 at a time | 449 sRTX PRO 6000 |
| – | REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details | Our GPU | 87.0 | 91.3% | 4.8 | 67.5%1.2% false alarms | 0.68 sp90 1.05 s1 at a time | 745 sRTX PRO 6000 |
| – | Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails | Our GPU | 86.4 | 91.0% | 6.7 | 66.3%1.7% false alarms | 1.12 sp90 1.96 s1 at a time | 1,329 sDGX Spark |
| – | Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details | Our GPU | 86.2 | 90.8% | 2.8 | 82.7%3.5% false alarms | 0.22 sp90 0.34 s1 at a time | 245 sRTX PRO 6000 |
| – | GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails | Our GPU | 86.1 | 90.7% | 3.6 | 74.8%1.5% false alarms | 1.03 sp90 1.95 s1 at a time | 1,205 sDGX Spark |
| – | Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details | Our GPU | 85.1 | 90.1% | 7.0 | 63.8%0.9% false alarms | 0.64 sp90 1.01 s1 at a time | 723 sRTX PRO 6000 |
| – | GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails | Not verified | 85.1 | 90.0% | 3.0 | 68.7%1.7% false alarms | 0.30 sp90 0.56 s8 at a time | API |
| – | GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details | Our GPU | 84.9 | 89.9% | 4.7 | 69.3%1.8% false alarms | 0.23 sp90 0.36 s1 at a time | 254 sRTX PRO 6000 |
| – | SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails | Not verified | 84.9 | 89.9% | 2.3 | 78.1%4.1% false alarms | 0.39 sp90 0.41 s8 at a time | API |
| Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails | Not verified | 83.7 | 89.1% | No probabilities | 68.7%2.2% false alarms | 6.37 sp90 19.2 s1 at a time | API | |
| – | vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details | Our GPU | 83.7 | 89.1% | 12.4 | 76.0%5.7% false alarms | 0.06 sp90 0.06 s1 at a time | 60 sRTX PRO 6000 |
| – | Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details | Our GPU | 83.0 | 88.6% | 3.3 | 59.9%2.5% false alarms | 0.09 sp90 0.14 s1 at a time | 100 sRTX PRO 6000 |
| – | GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details | Our GPU | 82.2 | 88.2% | 7.2 | 69.9%3.8% false alarms | 0.08 sp90 0.11 s1 at a time | 90 sRTX PRO 6000 |
| – | Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details | Our GPU | 81.9 | 87.9% | 2.9 | 65.3%3.0% false alarms | 0.11 sp90 0.18 s1 at a time | 126 sRTX PRO 6000 |
| – | StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails | Our GPU | 81.7 | 87.8% | 8.3 | 89.4%8.3% false alarms | 0.12 sp90 0.19 s1 at a time | 140 sDGX Spark |
| – | Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details | Our GPU | 81.1 | 87.4% | 2.5 | 64.7%3.9% false alarms | 0.06 sp90 0.09 s1 at a time | 72 sRTX PRO 6000 |
| – | KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details | Our GPU | 80.9 | 87.2% | 6.8 | 61.1%3.6% false alarms | 0.16 sp90 0.25 s1 at a time | 195 sRTX PRO 6000 |
| – | KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details | Our GPU | 80.6 | 87.1% | 4.5 | 62.0%3.4% false alarms | 0.04 sp90 0.06 s1 at a time | 49 sRTX PRO 6000 |
| – | Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details | Our GPU | 80.4 | 87.0% | 4.9 | 65.0%4.9% false alarms | 0.07 sp90 0.09 s1 at a time | 71 sRTX PRO 6000 |
| MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details | Our GPU | 79.279.0–79.6min–max of 3 seeds | 86.2%86.0%–86.4%min–max of 3 seeds | No probabilities | 77.4%6.5% false alarmsmean of 3 seeds | 5.00 sp90 8.72 s30 at a time3 seeds pooled | 197 sRTX PRO 6000mean of 3 seeds | |
| – | d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails | Not verified | 79.2 | 86.1% | 3.1 | 66.3%5.3% false alarms | 0.27 sp90 0.60 s8 at a time | API |
| – | Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details | Our GPU | 78.8 | 85.9% | 4.2 | 67.5%5.6% false alarms | 0.06 sp90 0.08 s1 at a time | 61 sRTX PRO 6000 |
| – | StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails | Our GPU | 78.8 | 85.9% | 8.2 | 88.8%9.5% false alarms | 0.23 sp90 0.36 s1 at a time | 268 sDGX Spark |
| – | Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details | Our GPU | 78.1 | 85.4% | 6.7 | 69.0%6.0% false alarms | 0.07 sp90 0.11 s1 at a time | 77 sRTX PRO 6000 |
| Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details | Our GPU | 77.2 | 84.8% | No probabilities | 67.5%4.5% false alarms | 9.38 sp90 11.9 s30 at a time | 326 sRTX PRO 6000 | |
| – | Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details | Our GPU | 77.2 | 84.8% | 2.8 | 47.7%4.0% false alarms | 0.06 sp90 0.09 s1 at a time | 65 sRTX PRO 6000 |
| Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details | Our GPU | 77.0 | 84.7% | No probabilities | 52.0%2.6% false alarms | 5.12 sp90 15.7 s64 at a time | 121 sRTX PRO 6000 | |
| – | Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details | Our GPU | 76.9 | 84.6% | 3.7 | 41.0%1.9% false alarms | 0.09 sp90 0.11 s1 at a time | 98 sRTX PRO 6000 |
| – | Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details | Our GPU | 76.6 | 84.4% | 8.3 | 51.7%3.8% false alarms | 0.12 sp90 0.18 s1 at a time | 132 sRTX PRO 6000 |
| – | KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details | Our GPU | 76.1 | 84.1% | 3.6 | 50.8%3.8% false alarms | 0.10 sp90 0.14 s1 at a time | 105 sRTX PRO 6000 |
| Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details | Our GPU | 75.9 | 83.9% | No probabilities | 79.0%9.0% false alarms | 19.6 sp90 64.5 s30 at a time | 1,050 sRTX PRO 6000 | |
| LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details | Our GPU | 75.874.9–76.5min–max of 3 seeds | 83.9%83.3%–84.3%min–max of 3 seeds | No probabilities | 63.3%4.1% false alarmsmean of 3 seeds | 5.57 sp90 16.4 s30 at a time3 seeds pooled | 268 sRTX PRO 6000mean of 3 seeds | |
| – | Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details | Our GPU | 75.4 | 83.6% | 3.4 | 46.8%3.7% false alarms | 0.05 sp90 0.07 s1 at a time | 54 sRTX PRO 6000 |
| – | KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details | Our GPU | 75.3 | 83.5% | 4.5 | 35.9%1.4% false alarms | 0.10 sp90 0.16 s1 at a time | 128 sRTX PRO 6000 |
| Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details | Our GPU | 75.174.1–75.9min–max of 3 seeds | 83.4%82.7%–84.0%min–max of 3 seeds | No probabilities | 64.3%4.3% false alarmsmean of 3 seeds | 6.13 sp90 9.70 s30 at a time3 seeds pooled | 234 sRTX PRO 6000mean of 3 seeds | |
| LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details | Our GPU | 74.973.5–75.6min–max of 3 seeds | 83.3%82.3%–83.8%min–max of 3 seeds | No probabilities | 62.6%3.9% false alarmsmean of 3 seeds | 7.62 sp90 22.3 s64 at a time3 seeds pooled | 180 sRTX PRO 6000mean of 3 seeds | |
| – | Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details | Our GPU | 74.3 | 82.8% | 3.2 | 39.8%2.8% false alarms | 0.07 sp90 0.10 s1 at a time | 78 sRTX PRO 6000 |
| – | Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details | Our GPU | 72.9 | 81.9% | 6.5 | 29.2%2.1% false alarms | 0.09 sp90 0.13 s1 at a time | 93 sRTX PRO 6000 |
| Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details | Our GPU | 71.1 | 80.8% | No probabilities | 49.5%5.1% false alarms | 4.02 sp90 5.85 s64 at a time | 69 sRTX PRO 6000 | |
| Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details | Our GPU | 70.3 | 80.2% | No probabilities | 55.3%4.3% false alarms | 36.8 sp90 66.9 s64 at a time | 636 sRTX PRO 6000 | |
| – | Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details | Our GPU | 69.8 | 79.8% | 5.9 | 37.7%3.9% false alarms | 0.15 sp90 0.18 s1 at a time | 156 sRTX PRO 6000 |
| Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details | Our GPU | 67.7 | 78.4% | No probabilities | 38.0%0.7% false alarms | 32.6 sp90 1640.2 s64 at a time | 4,327 sRTX PRO 6000 | |
| MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details | Our GPU | 67.066.9–67.0min–max of 3 seeds | 78.0%77.96%–78.01%min–max of 3 seeds | No probabilities | 28.8%2.7% false alarmsmean of 3 seeds | 0.86 sp90 3.48 s64 at a time3 seeds pooled | 51 sRTX PRO 6000mean of 3 seeds | |
| Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details | Our GPU | 66.2 | 77.5% | No probabilities | 73.2%6.2% false alarms | 37.8 sp90 96.1 s64 at a time | 741 sRTX PRO 6000 | |
| – | StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails | Our GPU | 66.0 | 77.3% | 7.2 | 67.5%14.0% false alarms | 0.06 sp90 0.09 s1 at a time | 78 sDGX Spark |
| – | GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details | Our GPU | 65.6 | 77.1% | 4.0 | 60.8%10.8% false alarms | 0.04 sp90 0.04 s1 at a time | 44 sRTX PRO 6000 |
| Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details | Our GPU | 64.664.5–64.8min–max of 3 seeds | 76.4%76.3%–76.6%min–max of 3 seeds | No probabilities | 61.2%7.6% false alarmsmean of 3 seeds | 3.49 sp90 20.4 s64 at a time3 seeds pooled | 148 sRTX PRO 6000mean of 3 seeds | |
| – | Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details | Our GPU | 61.7 | 74.5% | 6.8 | 23.1%3.4% false alarms | 0.12 sp90 0.19 s1 at a time | 134 sRTX PRO 6000 |
| – | KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails | Our GPU | 61.1 | 74.0% | 2.7 | 16.4%2.7% false alarms | 0.11 sp90 0.17 s1 at a time | 130 sDGX Spark |
| – | GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails | Our GPU | 60.8 | 73.9% | 3.8 | 31.3%6.6% false alarms | 0.05 sp90 0.07 s1 at a time | 63 sDGX Spark |
| – | Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details | Our GPU | 60.3 | 73.6% | 5.0 | 4.6%0.8% false alarms | 0.04 sp90 0.05 s1 at a time | 43 sRTX PRO 6000 |
| LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details | Our GPU | 57.7 | 71.8% | No probabilities | 8.8%0.8% false alarms | 9.58 sp90 15.2 s30 at a time | 352 sRTX PRO 6000 | |
| Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details | Our GPU | 55.254.5–55.8min–max of 3 seeds | 70.1%69.7%–70.6%min–max of 3 seeds | No probabilities | 7.7%1.6% false alarmsmean of 3 seeds | 0.18 sp90 5.42 s64 at a time3 seeds pooled | 33 sRTX PRO 6000mean of 3 seeds | |
| Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details | Our GPU | 54.8 | 69.8% | No probabilities | 6.1%1.3% false alarms | 0.19 sp90 3.31 s64 at a time | 18 sRTX PRO 6000 | |
| Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details | Our GPU | 50.249.3–50.8min–max of 3 seeds | 66.8%66.2%–67.2%min–max of 3 seeds | No probabilities | 34.4%7.8% false alarmsmean of 3 seeds | 1.53 sp90 13.7 s64 at a time3 seeds pooled | 92 sRTX PRO 6000mean of 3 seeds | |
| – | OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails | Our GPU | 50.0 | 66.7% | 4.1 | 33.4%14.4% false alarms | 0.03 sp90 0.04 s1 at a time | 43 sDGX Spark |
| – | d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details | Our GPU | 49.6 | 66.4% | 7.2 | 85.7%32.2% false alarms | 0.02 sp90 0.03 s1 at a time | 25 sRTX PRO 6000 |
| – | StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails | Our GPU | 48.9 | 65.9% | 5.6 | 72.3%26.5% false alarms | 0.04 sp90 0.06 s1 at a time | 53 sDGX Spark |
| – | GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails | Our GPU | 44.4 | 62.9% | 7.7 | 31.0%16.8% false alarms | 0.03 sp90 0.04 s1 at a time | 41 sDGX Spark |
| – | GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails | Our GPU | 44.3 | 62.9% | 9.8 | 34.9%10.6% false alarms | 0.07 sp90 0.18 s1 at a time | 128 sDGX Spark |
| Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details | Our GPU | 40.640.6–40.8min–max of 3 seeds | 60.4%60.4%–60.5%min–max of 3 seeds | No probabilities | 19.9%2.0% false alarmsmean of 3 seeds | 3.03 sp90 7.76 s30 at a time3 seeds pooled | 209 sRTX PRO 6000mean of 3 seeds | |
| – | KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails | Our GPU | 38.5 | 59.0% | 14.6 | 10.0%3.6% false alarms | 0.07 sp90 0.10 s1 at a time | 81 sDGX Spark |
No row of this kind yet.
2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology
GPU time lower is better
No row of this kind yet.
- Liquid AI
- Fastino
- OmniJev
- Strands Agents
- PelaAI
- OpenBMB
- StartLux
- victorbennett
- scar-ai
- Perplexity
- newsbang
- Patrick-18
- quincyqiang
- Valen-Team
- Alibaba
- Blockbrain Labs
- Base Labs
- tattrongvu
- Apertix AI
- Knowledgator
- EldanRing
- Metask Lab
- Cloudflare
- KRAFTON
- MachinePulse
- Poindexter Labs
- Logits
- REAX
- IBM
- Turion.AI
- Sthānika AI
- Thinking Machines
- Subquadratic
- Microsoft
- TypeSafe
- meraGPT
| The place by the chosen measure among the models in view, 1, 2, 3; a model without that measure is listed after them with –. | Model | Measured wall time of the whole run on the row's own GPU, named under the value (our RTX PRO 6000 or a DGX Spark), model load excluded, over all the pairs it answered, per 1,000 pairs: a rate, not API dollars. Seconds on different hardware are not like for like, even at the same requests in flight. Lower is better.s per 1,000 pairs | (accuracy − 1/3) / (2/3) × 100: 0 = one of the three labels at random on every pair, 100 = every pair right; below chance reads below 0. Higher is better.0–100 | Correct recorded labels out of all 2,069 pairs; a reply without a usable label counts wrong. Higher is better.of 2,069 pairs | Expected calibration error over 10 equal-width bins of the chosen label's probability, in points. Only where the run gives option probabilities: it is not a confidence, and none is derived for chat models. Lower is better.error, points | Of the 329 pairs whose summary states a conflicting answer, the share labelled so (recall); under it, false alarms: 'conflicting' given on the 1,740 pairs that state the answer or give none. An invalid reply counts as missed, never as a correct rejection. Higher is better.caught · false alarms | Client wall time per request: the median, the 90th percentile under it, and how many requests ran at once. Runs at different concurrency are not speeds under the same load, and neither are runs on different hardware (our RTX PRO 6000, a DGX Spark or the maker's API, named in the model's box), even at the same concurrency. Lower is better.median, p90 under it | Input tokens billed × the maker's list price, per 1,000 pairs (the price, its source and date in the model's box); a list price, not an invoice. Not verified: no list price of that model at that date could be checked; never a zero. Lower is better.$ per 1,000 pairs |
|---|---|---|---|---|---|---|---|---|
| Gemma 4 E2B thinking off · chat Open · 5.1BLatency 0.19 sGPU time 18 s/1,000 on RTX PRO 6000Details | 18 sRTX PRO 6000 | 54.8 | 69.8% | No probabilities | 6.1%1.3% false alarms | 0.19 sp90 3.31 s64 at a time | Our GPU | |
| 1 | d1-2B Open · 1.9BLatency 0.02 sGPU time 25 s/1,000 on RTX PRO 6000Details | 25 sRTX PRO 6000 | 49.6 | 66.4% | 7.2 | 85.7%32.2% false alarms | 0.02 sp90 0.03 s1 at a time | Our GPU |
| Gemma 4 E2B thinking off · sampled (temperature 1.0) · chat 0.03% invalid, mean of 3 seeds Open · 5.1BLatency 0.18 sGPU time 33 s/1,000 on RTX PRO 6000Details | 33 sRTX PRO 6000 · mean of 3 seeds | 55.254.5–55.8min–max of 3 seeds | 70.1%69.7%–70.6%min–max of 3 seeds | No probabilities | 7.7%1.6% false alarmsmean of 3 seeds | 0.18 sp90 5.42 s64 at a time3 seeds pooled | Our GPU | |
| 2 | GLiDE 0.4B Open · 0.4BLatency 0.03 sGPU time 41 s/1,000 on DGX SparkDetails | 41 sDGX Spark | 44.4 | 62.9% | 7.7 | 31.0%16.8% false alarms | 0.03 sp90 0.04 s1 at a time | Our GPU |
| 3 | OneJev1.6-0.8B Open · 0.8BLatency 0.03 sGPU time 43 s/1,000 on DGX SparkDetails | 43 sDGX Spark | 50.0 | 66.7% | 4.1 | 33.4%14.4% false alarms | 0.03 sp90 0.04 s1 at a time | Our GPU |
| 4 | Strands Decider 2B Open · 2.0BLatency 0.04 sGPU time 43 s/1,000 on RTX PRO 6000Details | 43 sRTX PRO 6000 | 60.3 | 73.6% | 5.0 | 4.6%0.8% false alarms | 0.04 sp90 0.05 s1 at a time | Our GPU |
| 5 | GLiDE 2B Open · 1.9BLatency 0.04 sGPU time 44 s/1,000 on RTX PRO 6000Details | 44 sRTX PRO 6000 | 65.6 | 77.1% | 4.0 | 60.8%10.8% false alarms | 0.04 sp90 0.04 s1 at a time | Our GPU |
| 6 | KnowLine 4B Gen5.5 Open · 4.0BLatency 0.04 sGPU time 49 s/1,000 on RTX PRO 6000Details | 49 sRTX PRO 6000 | 80.6 | 87.1% | 4.5 | 62.0%3.4% false alarms | 0.04 sp90 0.06 s1 at a time | Our GPU |
| MiniCPM5-2B thinking off · chat 0.2% invalid, mean of 3 seeds Open · 2.5BLatency 0.86 sGPU time 51 s/1,000 on RTX PRO 6000Details | 51 sRTX PRO 6000 · mean of 3 seeds | 67.066.9–67.0min–max of 3 seeds | 78.0%77.96%–78.01%min–max of 3 seeds | No probabilities | 28.8%2.7% false alarmsmean of 3 seeds | 0.86 sp90 3.48 s64 at a time3 seeds pooled | Our GPU | |
| 7 | StartLux-Decision-v1.1-0.8B-Instant Open · 0.8BLatency 0.04 sGPU time 53 s/1,000 on DGX SparkDetails | 53 sDGX Spark | 48.9 | 65.9% | 5.6 | 72.3%26.5% false alarms | 0.04 sp90 0.06 s1 at a time | Our GPU |
| 8 | Verdin-4B-v1.1 Open · 4.0BLatency 0.05 sGPU time 54 s/1,000 on RTX PRO 6000Details | 54 sRTX PRO 6000 | 75.4 | 83.6% | 3.4 | 46.8%3.7% false alarms | 0.05 sp90 0.07 s1 at a time | Our GPU |
| 9 | vera-medium-omni Open · 4.2BLatency 0.06 sGPU time 60 s/1,000 on RTX PRO 6000Details | 60 sRTX PRO 6000 | 83.7 | 89.1% | 12.4 | 76.0%5.7% false alarms | 0.06 sp90 0.06 s1 at a time | Our GPU |
| 10 | Perplexity Decider Flash v1.3 Open · 4.0BLatency 0.06 sGPU time 61 s/1,000 on RTX PRO 6000Details | 61 sRTX PRO 6000 | 78.8 | 85.9% | 4.2 | 67.5%5.6% false alarms | 0.06 sp90 0.08 s1 at a time | Our GPU |
| 11 | GLiDE 0.8B Open · 0.8BLatency 0.05 sGPU time 63 s/1,000 on DGX SparkDetails | 63 sDGX Spark | 60.8 | 73.9% | 3.8 | 31.3%6.6% false alarms | 0.05 sp90 0.07 s1 at a time | Our GPU |
| 12 | Perplexity Decider Flash v1.2 Open · 4.0BLatency 0.06 sGPU time 65 s/1,000 on RTX PRO 6000Details | 65 sRTX PRO 6000 | 77.2 | 84.8% | 2.8 | 47.7%4.0% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU |
| Gemma 4 E2B thinking on · chat Open · 5.1BLatency 4.02 sGPU time 69 s/1,000 on RTX PRO 6000Details | 69 sRTX PRO 6000 | 71.1 | 80.8% | No probabilities | 49.5%5.1% false alarms | 4.02 sp90 5.85 s64 at a time | Our GPU | |
| 13 | Adjudex 9B v5 Open · 9.0BLatency 0.07 sGPU time 71 s/1,000 on RTX PRO 6000Details | 71 sRTX PRO 6000 | 80.4 | 87.0% | 4.9 | 65.0%4.9% false alarms | 0.07 sp90 0.09 s1 at a time | Our GPU |
| 14 | Dexy-Decision-9B-V2 Open · 9.0BLatency 0.06 sGPU time 72 s/1,000 on RTX PRO 6000Details | 72 sRTX PRO 6000 | 81.1 | 87.4% | 2.5 | 64.7%3.9% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU |
| 15 | Chinese Decision Pointer 4B Open · 4.0BLatency 0.07 sGPU time 77 s/1,000 on RTX PRO 6000Details | 77 sRTX PRO 6000 | 78.1 | 85.4% | 6.7 | 69.0%6.0% false alarms | 0.07 sp90 0.11 s1 at a time | Our GPU |
| 16 | StartLux-Decision-v1.1-2B-Instant Open · 2.0BLatency 0.06 sGPU time 78 s/1,000 on DGX SparkDetails | 78 sDGX Spark | 66.0 | 77.3% | 7.2 | 67.5%14.0% false alarms | 0.06 sp90 0.09 s1 at a time | Our GPU |
| 17 | Valen-Decision-1.5-9B Open · 9.0BLatency 0.07 sGPU time 78 s/1,000 on RTX PRO 6000Details | 78 sRTX PRO 6000 | 74.3 | 82.8% | 3.2 | 39.8%2.8% false alarms | 0.07 sp90 0.10 s1 at a time | Our GPU |
| 18 | KnowLine-0.8B-Gen1 Open · 0.8BLatency 0.07 sGPU time 81 s/1,000 on DGX SparkDetails | 81 sDGX Spark | 38.5 | 59.0% | 14.6 | 10.0%3.6% false alarms | 0.07 sp90 0.10 s1 at a time | Our GPU |
| 19 | GLiDE 9B Open · 7.9BLatency 0.08 sGPU time 90 s/1,000 on RTX PRO 6000Details | 90 sRTX PRO 6000 | 82.2 | 88.2% | 7.2 | 69.9%3.8% false alarms | 0.08 sp90 0.11 s1 at a time | Our GPU |
| Qwen3.5-2B thinking off · chat 2.0% invalid, mean of 3 seeds Open · 2.3BLatency 1.53 sGPU time 92 s/1,000 on RTX PRO 6000Details | 92 sRTX PRO 6000 · mean of 3 seeds | 50.249.3–50.8min–max of 3 seeds | 66.8%66.2%–67.2%min–max of 3 seeds | No probabilities | 34.4%7.8% false alarmsmean of 3 seeds | 1.53 sp90 13.7 s64 at a time3 seeds pooled | Our GPU | |
| 20 | Cygnet Open · 12BLatency 0.09 sGPU time 93 s/1,000 on RTX PRO 6000Details | 93 sRTX PRO 6000 | 72.9 | 81.9% | 6.5 | 29.2%2.1% false alarms | 0.09 sp90 0.13 s1 at a time | Our GPU |
| 21 | Elephant-1 Open · 9.0BLatency 0.09 sGPU time 98 s/1,000 on RTX PRO 6000Details | 98 sRTX PRO 6000 | 76.9 | 84.6% | 3.7 | 41.0%1.9% false alarms | 0.09 sp90 0.11 s1 at a time | Our GPU |
| 22 | Jeffan-9B-1211 Open · 9.0BLatency 0.09 sGPU time 100 s/1,000 on RTX PRO 6000Details | 100 sRTX PRO 6000 | 83.0 | 88.6% | 3.3 | 59.9%2.5% false alarms | 0.09 sp90 0.14 s1 at a time | Our GPU |
| 23 | KnowLine-9B-Gen1 Open · 9.0BLatency 0.10 sGPU time 105 s/1,000 on RTX PRO 6000Details | 105 sRTX PRO 6000 | 76.1 | 84.1% | 3.6 | 50.8%3.8% false alarms | 0.10 sp90 0.14 s1 at a time | Our GPU |
| Gemma 4 12B Unified thinking off · chat Open · 12BLatency 5.12 sGPU time 121 s/1,000 on RTX PRO 6000Details | 121 sRTX PRO 6000 | 77.0 | 84.7% | No probabilities | 52.0%2.6% false alarms | 5.12 sp90 15.7 s64 at a time | Our GPU | |
| 24 | Cerno 9B v1.2 Open · 9.0BLatency 0.11 sGPU time 126 s/1,000 on RTX PRO 6000Details | 126 sRTX PRO 6000 | 81.9 | 87.9% | 2.9 | 65.3%3.0% false alarms | 0.11 sp90 0.18 s1 at a time | Our GPU |
| 25 | GLiClass v4 Edge Open · 0.4BLatency 0.07 sGPU time 128 s/1,000 on DGX SparkDetails | 128 sDGX Spark | 44.3 | 62.9% | 9.8 | 34.9%10.6% false alarms | 0.07 sp90 0.18 s1 at a time | Our GPU |
| 26 | KnowLine-9B-Gen2 Open · 9.0BLatency 0.10 sGPU time 128 s/1,000 on RTX PRO 6000Details | 128 sRTX PRO 6000 | 75.3 | 83.5% | 4.5 | 35.9%1.4% false alarms | 0.10 sp90 0.16 s1 at a time | Our GPU |
| 27 | KnowLine-2B-Gen1 Open · 2.0BLatency 0.11 sGPU time 130 s/1,000 on DGX SparkDetails | 130 sDGX Spark | 61.1 | 74.0% | 2.7 | 16.4%2.7% false alarms | 0.11 sp90 0.17 s1 at a time | Our GPU |
| 28 | Winnow-12B Open · 12BLatency 0.12 sGPU time 132 s/1,000 on RTX PRO 6000Details | 132 sRTX PRO 6000 | 76.6 | 84.4% | 8.3 | 51.7%3.8% false alarms | 0.12 sp90 0.18 s1 at a time | Our GPU |
| 29 | Metask-Jev-4B Open · 4.5BLatency 0.12 sGPU time 134 s/1,000 on RTX PRO 6000Details | 134 sRTX PRO 6000 | 61.7 | 74.5% | 6.8 | 23.1%3.4% false alarms | 0.12 sp90 0.19 s1 at a time | Our GPU |
| 30 | StartLux-Decision-v1.1-4B-Instant Open · 4.0BLatency 0.12 sGPU time 140 s/1,000 on DGX SparkDetails | 140 sDGX Spark | 81.7 | 87.8% | 8.3 | 89.4%8.3% false alarms | 0.12 sp90 0.19 s1 at a time | Our GPU |
| Qwen3.5-4B thinking off · chat 1.0% invalid, mean of 3 seeds Open · 4.0BLatency 3.49 sGPU time 148 s/1,000 on RTX PRO 6000Details | 148 sRTX PRO 6000 · mean of 3 seeds | 64.664.5–64.8min–max of 3 seeds | 76.4%76.3%–76.6%min–max of 3 seeds | No probabilities | 61.2%7.6% false alarmsmean of 3 seeds | 3.49 sp90 20.4 s64 at a time3 seeds pooled | Our GPU | |
| 31 | Clef-Flash Open · 9.0BLatency 0.15 sGPU time 156 s/1,000 on RTX PRO 6000Details | 156 sRTX PRO 6000 | 69.8 | 79.8% | 5.9 | 37.7%3.9% false alarms | 0.15 sp90 0.18 s1 at a time | Our GPU |
| LFM3-1.7B thinking off · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 7.62 sGPU time 180 s/1,000 on RTX PRO 6000Details | 180 sRTX PRO 6000 · mean of 3 seeds | 74.973.5–75.6min–max of 3 seeds | 83.3%82.3%–83.8%min–max of 3 seeds | No probabilities | 62.6%3.9% false alarmsmean of 3 seeds | 7.62 sp90 22.3 s64 at a time3 seeds pooled | Our GPU | |
| 32 | KnowLine-12B-Gen1 Open · 12BLatency 0.16 sGPU time 195 s/1,000 on RTX PRO 6000Details | 195 sRTX PRO 6000 | 80.9 | 87.2% | 6.8 | 61.1%3.6% false alarms | 0.16 sp90 0.25 s1 at a time | Our GPU |
| MiniCPM5-2B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 2.5BLatency 5.00 sGPU time 197 s/1,000 on RTX PRO 6000Details | 197 sRTX PRO 6000 · mean of 3 seeds | 79.279.0–79.6min–max of 3 seeds | 86.2%86.0%–86.4%min–max of 3 seeds | No probabilities | 77.4%6.5% false alarmsmean of 3 seeds | 5.00 sp90 8.72 s30 at a time3 seeds pooled | Our GPU | |
| Raon-1 nano thinking off · chat 0.03% invalid, mean of 3 seeds Open · 3.1BLatency 3.03 sGPU time 209 s/1,000 on RTX PRO 6000Details | 209 sRTX PRO 6000 · mean of 3 seeds | 40.640.6–40.8min–max of 3 seeds | 60.4%60.4%–60.5%min–max of 3 seeds | No probabilities | 19.9%2.0% false alarmsmean of 3 seeds | 3.03 sp90 7.76 s30 at a time3 seeds pooled | Our GPU | |
| 33 | Karpo Decision 31B Open · 31BLatency 0.22 sGPU time 230 s/1,000 on RTX PRO 6000Details | 230 sRTX PRO 6000 | 90.0 | 93.3% | 2.8 | 80.8%1.7% false alarms | 0.22 sp90 0.31 s1 at a time | Our GPU |
| Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat 1.5% invalid, mean of 3 seeds Open · 8.0BLatency 6.13 sGPU time 234 s/1,000 on RTX PRO 6000Details | 234 sRTX PRO 6000 · mean of 3 seeds | 75.174.1–75.9min–max of 3 seeds | 83.4%82.7%–84.0%min–max of 3 seeds | No probabilities | 64.3%4.3% false alarmsmean of 3 seeds | 6.13 sp90 9.70 s30 at a time3 seeds pooled | Our GPU | |
| 34 | Perplexity Decider 27B v1.3 Open · 27BLatency 0.22 sGPU time 245 s/1,000 on RTX PRO 6000Details | 245 sRTX PRO 6000 | 86.2 | 90.8% | 2.8 | 82.7%3.5% false alarms | 0.22 sp90 0.34 s1 at a time | Our GPU |
| 35 | GLiDE v1.2 Open · 27BLatency 0.23 sGPU time 254 s/1,000 on RTX PRO 6000Details | 254 sRTX PRO 6000 | 84.9 | 89.9% | 4.7 | 69.3%1.8% false alarms | 0.23 sp90 0.36 s1 at a time | Our GPU |
| 36 | StartLux-Decision-v1.1-9B-Instant Open · 9.0BLatency 0.23 sGPU time 268 s/1,000 on DGX SparkDetails | 268 sDGX Spark | 78.8 | 85.9% | 8.2 | 88.8%9.5% false alarms | 0.23 sp90 0.36 s1 at a time | Our GPU |
| LFM3-1.7B thinking on · chat 0.1% invalid, mean of 3 seeds Open · 1.7BLatency 5.57 sGPU time 268 s/1,000 on RTX PRO 6000Details | 268 sRTX PRO 6000 · mean of 3 seeds | 75.874.9–76.5min–max of 3 seeds | 83.9%83.3%–84.3%min–max of 3 seeds | No probabilities | 63.3%4.1% false alarmsmean of 3 seeds | 5.57 sp90 16.4 s30 at a time3 seeds pooled | Our GPU | |
| Gemma 4 E4B thinking on · chat 2 invalid Open · 8.0BLatency 9.38 sGPU time 326 s/1,000 on RTX PRO 6000Details | 326 sRTX PRO 6000 | 77.2 | 84.8% | No probabilities | 67.5%4.5% false alarms | 9.38 sp90 11.9 s30 at a time | Our GPU | |
| LFM2.5-8B-A1B thinking on (no off mode) · chat Open · 8.3B MoE · 1.5B activeLatency 9.58 sGPU time 352 s/1,000 on RTX PRO 6000Details | 352 sRTX PRO 6000 | 57.7 | 71.8% | No probabilities | 8.8%0.8% false alarms | 9.58 sp90 15.2 s30 at a time | Our GPU | |
| 37 | KnowLine-31B-Gen2 Open · 31BLatency 0.39 sGPU time 449 s/1,000 on RTX PRO 6000Details | 449 sRTX PRO 6000 | 87.7 | 91.8% | 7.2 | 86.3%3.6% false alarms | 0.39 sp90 0.61 s1 at a time | Our GPU |
| 38 | Copernicus-31B Open · 31BLatency 0.52 sGPU time 577 s/1,000 on RTX PRO 6000Details | 577 sRTX PRO 6000 | 89.0 | 92.7% | 11.0 | 81.5%2.5% false alarms | 0.52 sp90 0.83 s1 at a time | Our GPU |
| Qwen3.5-2B thinking on · chat 78 invalid Open · 2.0BLatency 36.8 sGPU time 636 s/1,000 on RTX PRO 6000Details | 636 sRTX PRO 6000 | 70.3 | 80.2% | No probabilities | 55.3%4.3% false alarms | 36.8 sp90 66.9 s64 at a time | Our GPU | |
| 39 | Logits Sigma-1 Duo Open · 57BLatency 0.64 sGPU time 723 s/1,000 on RTX PRO 6000Details | 723 sRTX PRO 6000 | 85.1 | 90.1% | 7.0 | 63.8%0.9% false alarms | 0.64 sp90 1.01 s1 at a time | Our GPU |
| Qwen3.5-4B thinking on · chat 179 invalid Open · 4.0BLatency 37.8 sGPU time 741 s/1,000 on RTX PRO 6000Details | 741 sRTX PRO 6000 | 66.2 | 77.5% | No probabilities | 73.2%6.2% false alarms | 37.8 sp90 96.1 s64 at a time | Our GPU | |
| 40 | REAX 1.3 Open · 57BLatency 0.68 sGPU time 745 s/1,000 on RTX PRO 6000Details | 745 sRTX PRO 6000 | 87.0 | 91.3% | 4.8 | 67.5%1.2% false alarms | 0.68 sp90 1.05 s1 at a time | Our GPU |
| Granite-4.2-3B thinking on · chat Open · 3.0BLatency 19.6 sGPU time 1,050 s/1,000 on RTX PRO 6000Details | 1,050 sRTX PRO 6000 | 75.9 | 83.9% | No probabilities | 79.0%9.0% false alarms | 19.6 sp90 64.5 s30 at a time | Our GPU | |
| 41 | GLiDE 27B Open · 27BLatency 1.03 sGPU time 1,205 s/1,000 on DGX SparkDetails | 1,205 sDGX Spark | 86.1 | 90.7% | 3.6 | 74.8%1.5% false alarms | 1.03 sp90 1.95 s1 at a time | Our GPU |
| 42 | Turion Jev 31B Open · 31BLatency 1.12 sGPU time 1,329 s/1,000 on DGX SparkDetails | 1,329 sDGX Spark | 86.4 | 91.0% | 6.7 | 66.3%1.7% false alarms | 1.12 sp90 1.96 s1 at a time | Our GPU |
| 43 | KnowLine-31B-Gen1.5 Open · 31BLatency 2.54 sGPU time 2,613 s/1,000 on DGX SparkDetails | 2,613 sDGX Spark | 88.8 | 92.6% | 5.7 | 83.6%3.0% false alarms | 2.54 sp90 3.88 s1 at a time | Our GPU |
| 44 | Sieve 31B v3 Open · 31BLatency 2.67 sGPU time 2,824 s/1,000 on DGX SparkDetails | 2,824 sDGX Spark | 88.1 | 92.1% | 7.0 | 79.3%2.1% false alarms | 2.67 sp90 3.93 s1 at a time | Our GPU |
| Gemma 4 12B Unified thinking on · chat 296 invalid Open · 12BLatency 32.6 sGPU time 4,327 s/1,000 on RTX PRO 6000Details | 4,327 sRTX PRO 6000 | 67.7 | 78.4% | No probabilities | 38.0%0.7% false alarms | 32.6 sp90 1640.2 s64 at a time | Our GPU | |
| – | d1.2-ultra Closed · EfficientLatency 0.23 sList price $0.035/1,000Details | API | 86.4 | 91.0% | 8.8 | 85.4%4.4% false alarms | 0.23 sp90 0.35 s8 at a time | $0.035 |
| Inkling 2 Small thinking on (no off mode) · chat 594B MoE · 17B activeLatency 2.70 sList price $1.49/1,000Details | API | 86.2 | 90.8% | No probabilities | 73.6%2.4% false alarms | 2.70 sp90 16.4 s1 at a time | $1.49 | |
| – | GLiDE API Closed · EfficientLatency 0.30 sList price not verifiedDetails | API | 85.1 | 90.0% | 3.0 | 68.7%1.7% false alarms | 0.30 sp90 0.56 s8 at a time | Not verified |
| – | SubQ Snap 1.1 Closed · EfficientLatency 0.39 sList price not verifiedDetails | API | 84.9 | 89.9% | 2.3 | 78.1%4.1% false alarms | 0.39 sp90 0.41 s8 at a time | Not verified |
| – | d1-ultra Closed · EfficientLatency 0.48 sList price $0.035/1,000Details | API | 84.7 | 89.8% | 11.4 | 80.8%4.5% false alarms | 0.48 sp90 0.90 s8 at a time | $0.035 |
| – | Microsoft-Decision-1 Closed · EfficientLatency 0.32 sList price $0.036/1,000Details | API | 83.9 | 89.3% | 2.4 | 82.4%6.5% false alarms | 0.32 sp90 0.38 s8 at a time | $0.036 |
| Gemini 4 Hydrogen thinking on (no off mode) · chat Closed · WorkhorseLatency 6.37 sList price not verifiedDetails | API | 83.7 | 89.1% | No probabilities | 68.7%2.2% false alarms | 6.37 sp90 19.2 s1 at a time | Not verified | |
| – | Jev 1.14 Closed · EfficientLatency 0.18 sList price $0.049/1,000Details | API | 83.6 | 89.1% | 2.8 | 72.0%3.5% false alarms | 0.18 sp90 0.20 s8 at a time | $0.049 |
| – | Jev 1.13 Closed · EfficientLatency 0.17 sList price $0.049/1,000Details | API | 83.0 | 88.6% | 1.9 | 79.9%6.0% false alarms | 0.17 sp90 0.19 s8 at a time | $0.049 |
| – | d1.1 Closed · EfficientLatency 0.25 sList price $0.017/1,000Details | API | 82.5 | 88.4% | 8.7 | 70.5%4.1% false alarms | 0.25 sp90 0.45 s8 at a time | $0.017 |
| – | d1 Closed · EfficientLatency 0.27 sList price not verifiedDetails | API | 79.2 | 86.1% | 3.1 | 66.3%5.3% false alarms | 0.27 sp90 0.60 s8 at a time | Not verified |
| – | Gemini Decision 1 20261111 Closed · EfficientLatency 0.20 sList price $0.093/1,000Details | API | 77.0 | 84.7% | 3.9 | 51.1%1.8% false alarms | 0.20 sp90 0.29 s8 at a time | $0.093 |
| – | Decider 1 Closed · EfficientLatency 0.65 sList price $0.035/1,000Details | API | 43.3 | 62.2% | 5.9 | 31.9%17.6% false alarms | 0.65 sp90 0.75 s8 at a time | $0.035 |
No row of this kind yet.
2,069 pairs per model · updated 16 November 2026 · bench-scores.json · metrics.json · Methodology
Details of each setup
Karpo Decision 31B · MachinePulse
Open · 31B
MachinePulseAI/Karpo Decision 31B@d69cdd57, its card's karpo-decision package (machinepulse-ai/karpo-decision@ac40d399) System One server, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 98.0% of 942, a conflicting one 80.8% of 329, none 93.0% of 798
Copernicus-31B · Poindexter Labs
Open · 31B
Poindexter-Labs/Copernicus-31B@be715259, its card's copernicus_server.py (Transformers backend, the card's defaults), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 98.8% of 942, a conflicting one 81.5% of 329, none 90.0% of 798
KnowLine-31B-Gen1.5 · PelaAI
Open · 31B
PelaAI/KnowLine-31B-Gen1.5@8e56aa1b, the card's serve_knowline.sh (SGLang), BF16, on one DGX Spark (ours) as for its published runs, self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 1 published-cohort pair was sent in test requests whose outputs were not kept, so its scored row comes from a later request
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 97.2% of 942, a conflicting one 83.6% of 329, none 90.7% of 798
Sieve 31B v3 · Sthānika AI
Open · 31B
sthanika-ai/Sieve-31B-v3@8fcee858, sieve-serve 0.5.1 (text only, design 'letters'), BF16, on one DGX Spark (ours) as for its published runs, self-hosted; the server reports no input-token count; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 98.6% of 942, a conflicting one 79.3% of 329, none 89.6% of 798
KnowLine-31B-Gen2 · PelaAI
Open · 31B
PelaAI/KnowLine-31B-Gen2@fd99a9ed, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card names 0.5.21, which we do not have), BF16, SHA256SUMS checked, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); the run overlapped a host GPU recovery warning; its serving process continued, and the run completed without interruption or resume (producer report)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 96.8% of 942, a conflicting one 86.3% of 329, none 88.1% of 798
REAX 1.3 · REAX
Open · 57B
decisionmodels/REAX-1.3@fa4bf7ee, REAX Engine reax13 server on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 96.2% of 942, a conflicting one 67.5% of 329, none 95.4% of 798
d1.2-ultra · Liquid AI
Closed · Efficient
Liquid AI decisions API (api.liquid.ai/decisions, model d1.2-ultra); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 91.9% of 942, a conflicting one 85.4% of 329, none 92.1% of 798
List price $0.042 per 1M input tokens, its source (2026-11-16)
Turion Jev 31B · Turion.AI
Open · 31B
turion-ai/turion-jev-31b@866c0fc6, the card's turion_jev.serve on vLLM 0.32.0 (BF16 weights, FP8 at load, text only), on one DGX Spark (ours) as for its published runs, self-hosted; vLLM memory share 0.80 instead of the card's 0.92 (KV cache only); one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 96.8% of 942, a conflicting one 66.3% of 329, none 94.2% of 798
Inkling 2 Small · Thinking Machines
594B MoE · 17B active
OpenRouter thinkingmachines/inkling-2-small (paid per token; providers with no data retention and no training on inputs), reasoning at the route's default; billed per token at OpenRouter's list price; one question per request, 1 request at a time (latency not comparable across different concurrency); list prices $0.30 per 1M input and $1.00 per 1M output tokens (OpenRouter, 16 Nov); answered by OpenRouter's providers Together 1,104, Cloudflare 317, PrimeIntellect 242, Nebius 201, DeepInfra 179, Modal 26; OpenRouter's reported charges for the 2,069 billed requests of this model: $2.8921 (actual billing, distinct from the list-price figure); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 97.2% of 942, a conflicting one 73.6% of 329, none 90.3% of 798
List price $0.3 per 1M input tokens, its source (2026-11-16)
Perplexity Decider 27B v1.3 · Perplexity
Open · 27B
perplexity-ai/pplx-decider-v1.3@9c2ebb27, the runner's pplx_decider_server.py with the checkpoint's own pplx_decider package (as for its published v1.2 runs), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 95.1% of 942, a conflicting one 82.7% of 329, none 89.1% of 798
GLiDE 27B · Fastino
Open · 27B
fastino/GLiDE-27B@4d32743a, GLiDE server (glide.server), BF16, GLIDE_MAX_BATCH=1, on one DGX Spark (ours) as for its published runs, self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 97.1% of 942, a conflicting one 74.8% of 329, none 89.7% of 798
Logits Sigma-1 Duo · Logits
Open · 57B
logitshq/Logits-Sigma-1-Duo@a0c1b6e8 (weights and code identical to d9641cd8, the revision its card's run names; two members averaged per option, FP8 weights), its card's serving image (code/Dockerfile: torch 2.15, transformers 5.22.1, causal-conv1d 1.7.0, kit decision-index@25c6ec7) plus only gcc, libc6-dev and python3-dev, which its Triton kernels need to compile a launcher at first use and the image lacks, with the engine and option its serve.sh fixes (FP8 matmul on its own Triton kernel: its cuBLAS block-scaled path was not taken on this GPU, a fallback the engine makes itself), behind di_serve.py, our System One wrapper of the engine, on one RTX PRO 6000 (ours), self-hosted; text only (the load report lists FP8 scales missing for member A's vision tower, which text requests do not use; not checked on images); one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 94.4% of 942, a conflicting one 63.8% of 329, none 95.9% of 798
GLiDE · Fastino
Closed · Efficient
Fastino decisions API (api.fastino.ai, model fastino/GLiDE); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 96.4% of 942, a conflicting one 68.7% of 329, none 91.3% of 798
GLiDE · Fastino
Open · 27B
fastino/GLiDE@a7e4589, GLiDE server (Fastino's release code), BF16, GLIDE_MAX_BATCH=1, on one RTX PRO 6000 (ours) as the backgammon league's pinned server, self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 96.6% of 942, a conflicting one 69.3% of 329, none 90.6% of 798
SubQ Snap 1.1 · Subquadratic
Closed · Efficient
SubQ decisions API (api.subq.ai, model subq-snap-1.1); its probabilities were byte-identical to SubQ Snap Preview's on all 2,069 pairs; one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 91.9% of 942, a conflicting one 78.1% of 329, none 92.5% of 798
d1-ultra · Liquid AI
Closed · Efficient
Liquid AI decisions API (api.liquid.ai); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 95.7% of 942, a conflicting one 80.8% of 329, none 86.6% of 798
List price $0.042 per 1M input tokens, its source (2026-10-16)
Microsoft-Decision-1 · Microsoft
Closed · Efficient
OpenRouter decisions API (openrouter.ai/api, model microsoft/microsoft-decision-1); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 89.5% of 942, a conflicting one 82.4% of 329, none 91.8% of 798
List price $0.042 per 1M input tokens, its source (2026-10-17)
Gemini 4 Hydrogen · Google
Closed · Workhorse
Google Antigravity agent harness through Can's agbench bridge (Google AI Ultra subscription), thinking high, tool descriptions present and tools denied, a fresh conversation per request; $0 billed (the subscription's quota); one question per request, 1 request at a time (latency not comparable across different concurrency); the pass stopped once when the bridge's safety switch tripped on a false positive (fixed by its owner) and resumed in the same order by a recorded decision; the one request in flight at the stop was sent again, so its recorded outcome is not guaranteed to be the model's first output; thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 88.4% of 942, a conflicting one 68.7% of 329, none 98.4% of 798
vera-medium-omni · scar-ai
Open · 4.2B
scar-ai/vera-medium-omni@1f23a6cf, its card's own server (vera-serve from vera-s1 0.1.8, PyTorch engine), BF16, on one RTX PRO 6000 (ours), self-hosted; server probabilities are rounded to four decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 92.8% of 942, a conflicting one 76.0% of 329, none 90.2% of 798
Jev 1.14 · TypeSafe
Closed · Efficient
TypeSafe decisions API (api.typesafe.ai, model jev-1.14.0); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 94.1% of 942, a conflicting one 72.0% of 329, none 90.2% of 798
List price $0.042 per 1M input tokens, its source (2026-10-16)
Jeffan-9B-1211 · tattrongvu
Open · 9.0B
tattrongvu/Jeffan-9B-1211@862d1a7a, its card's Decision Index engine (jeffan_engine:DecisionEngine, transformers 5.17.0, flash-linear-attention 0.5.2, kit decision-index@f8552a1, options max_len and budget 65,536 as its card; without the card's optional causal-conv1d, so its PyTorch reference convolution) behind di_serve.py, our System One wrapper of the engine, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 91.7% of 942, a conflicting one 59.9% of 329, none 96.9% of 798
Jev 1.13 · TypeSafe
Closed · Efficient
TypeSafe decisions API (api.typesafe.ai, model jev-1.13.0); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 94.3% of 942, a conflicting one 79.9% of 329, none 85.6% of 798
List price $0.042 per 1M input tokens, its source (2026-10-08)
d1.1 · Liquid AI
Closed · Efficient
Liquid AI decisions API (api.liquid.ai/decisions, model d1.1); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 92.5% of 942, a conflicting one 70.5% of 329, none 90.8% of 798
List price $0.02 per 1M input tokens, its source (2026-10-24)
GLiDE 9B · Fastino
Open · 7.9B
fastino/GLiDE-9B@ab54df5a, GLiDE server (python -m glide.server) in vLLM v0.30.0 image with flash-linear-attention 0.5.2, as for GLiDE 2B, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 94.6% of 942, a conflicting one 69.9% of 329, none 88.1% of 798
Cerno 9B v1.2 · Apertix AI
Open · 9.0B
apertixai/cerno-9b-v1.2@f02a632f, its card's Decision Index engine (CernoEngine, transformers 5.17.0, flash-linear-attention 0.5.2, kit decision-index@9eb2dbe2, the checkpoint's decider_config.json settings, loaded by Hub id and that revision as its card runs it, from the local copy; causal-conv1d not installed, so its PyTorch reference convolution) behind di_serve.py, our System One wrapper of the engine, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 91.2% of 942, a conflicting one 65.3% of 329, none 93.4% of 798
StartLux-Decision-v1.1-4B-Instant · StartLux
Open · 4.0B
startlux-models/StartLux-Decision-v1.1-4B-Instant@416088f7, its package's own server (startlux-decide serve --launch-vllm, StartLux-Decision@a99977d, vLLM 0.31.0) as for its published Reading and Logic & Math runs, BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 90.2% of 942, a conflicting one 89.4% of 329, none 84.2% of 798
Dexy-Decision-9B-V2 · Patrick-18
Open · 9.0B
Patrick-18/Dexy-Decision-9B-V2@4f434e2b, its card's Transformers service (d3_server.py, in the environment its requirements.txt pins: torch 2.14.0, transformers 5.19.0), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 92.7% of 942, a conflicting one 64.7% of 329, none 90.5% of 798
KnowLine-12B-Gen1 · PelaAI
Open · 12B
PelaAI/KnowLine-12B-Gen1@f26fbc2d, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card names 0.5.21, which we do not have), BF16, SHA256SUMS checked, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 93.3% of 942, a conflicting one 61.1% of 329, none 90.8% of 798
KnowLine 4B Gen5.5 · PelaAI
Open · 4.0B
PelaAI/KnowLine-4B-Gen5.5@779944d2 (the site's pin; no longer in the maker's repository, copied from the snapshot its published runs used, SHA256SUMS checked), its card's serve_knowline.sh (SGLang with FP8 at load, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23, on one RTX PRO 6000 (ours), self-hosted; its published runs were on a DGX Spark; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 90.5% of 942, a conflicting one 62.0% of 329, none 93.4% of 798
Adjudex 9B v5 · newsbang
Open · 9.0B
newsbang/adjudex-9b-v5@ffdcb825, its card's autojev server (the Perplexity Decider v1.1 source, in the environment of its requirements), default settings, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 92.2% of 942, a conflicting one 65.0% of 329, none 89.7% of 798
MiniCPM5-2B · OpenBMB
Open · 2.5B
openbmb/MiniCPM5-2B@42433e90, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.1% (per seed in seed_runs; cut off at the token limit per seed: 0, 0, 0); the three runs are fresh runs (14 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the runs they replace and the server settings recorded for them; the earlier runs stay in history; gpu_hours is the mean of the three runs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0.1% of 2,069 on average (seed 0: 3; seed 1: 0; seed 2: 1), counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label, the mean of its seeds: the reference answer 89.8% of 942, a conflicting one 77.4% of 329, none 85.4% of 798
Seeds: seed 0: grading:chat-minicpm5-2b-seed0-r2-v1, 86.0% right, 3 invalid, 0 cut off, 250 wrong answers caught; seed 1: grading:chat-minicpm5-2b-seed1-r2-v1, 86.1% right, 0 invalid, 0 cut off, 258 wrong answers caught; seed 2: grading:chat-minicpm5-2b-seed2-r2-v1, 86.4% right, 1 invalid, 0 cut off, 256 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval
Measures: mean of 3 seeds; latency over all requests of the three runs
Outside the three: its first run, grading:chat-minicpm5-2b-v1, asked 64 requests at a time, where the three seed runs were asked 30 at a time; not ranked, kept in the file published before the seeds
d1 · Liquid AI
Closed · Efficient
Liquid AI decisions API (api.liquid.ai/decisions, model d1); billed 'd1' route, the site's 'd1:free' was withdrawn (HTTP 404, 13 Nov); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 91.1% of 942, a conflicting one 66.3% of 329, none 88.5% of 798
Perplexity Decider Flash v1.3 · Perplexity
Open · 4.0B
perplexity-ai/pplx-decider-flash-v1.3@d06fb3f2, the runner's pplx_decider_server.py with the checkpoint's own pplx_decider package (as for its published v1.2 runs), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 93.7% of 942, a conflicting one 67.5% of 329, none 84.2% of 798
StartLux-Decision-v1.1-9B-Instant · StartLux
Open · 9.0B
startlux-models/StartLux-Decision-v1.1-9B-Instant@18a17f73, its package's own server (startlux-decide serve --launch-vllm, StartLux-Decision@a99977d, vLLM 0.31.0), BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 90.3% of 942, a conflicting one 88.8% of 329, none 79.5% of 798
Chinese Decision Pointer 4B · quincyqiang
Open · 4.0B
quincyqiang/chinese-decision-pointer-4b@889f23e2, its card's Decision Index engine (yanqiangmiffy/chinese-decision@c16450b, ChineseDecisionEngine with debias 'reverse' as its card's run; transformers 5.17.0, kit decision-index@9eb2dbe2, as the card names no kit commit; causal-conv1d not installed, so its PyTorch reference convolution) behind di_serve.py, our System One wrapper of the engine, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 90.7% of 942, a conflicting one 69.0% of 329, none 86.0% of 798
Gemma 4 E4B · Google
Open · 8.0B
google/gemma-4-E4B-it@4ecc2ba6, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 2 of 2069 replies (0.10%) had no answer marker and count wrong (0 of them cut off at the 32,768-token limit); every reply kept; an earlier attempt of this same run stopped part-way (510 recorded pairs and an unknown number of further requests possibly answered but not recorded) and is kept apart, not merged; this attempt started again from the first pair, so its outcomes are this attempt's recorded outputs, not guaranteed to be the model's first outputs on those pairs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 2 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 91.2% of 942, a conflicting one 67.5% of 329, none 84.3% of 798
Perplexity Decider Flash v1.2 · Perplexity
Open · 4.0B
perplexity-ai/pplx-decider-flash-v1.2@6d771ad7, the runner's pplx_decider_server.py with the checkpoint's own pplx_decider package (as for its published v1.2 runs), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 94.1% of 942, a conflicting one 47.7% of 329, none 89.1% of 798
Gemini Decision 1 · Google
Closed · Efficient
Google Decisions API (generativelanguage.googleapis.com, model gemini-decision-1); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 96.5% of 942, a conflicting one 51.1% of 329, none 84.6% of 798
List price $0.1 per 1M input tokens, its source (2026-11-13)
Gemma 4 12B Unified · Google
Open · 12B
google/gemma-4-12B-it@43c9d014, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking off; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 97.4% of 942, a conflicting one 52.0% of 329, none 83.2% of 798
Elephant-1 · Base Labs
Open · 9.0B
baselabs/elephant-1@b3f8a934, the runner's baselabs_server.py in the card's setup.sh environment (OneJev 81ce62f1 with the checkpoint's serve.patch), on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 93.7% of 942, a conflicting one 41.0% of 329, none 91.7% of 798
Winnow-12B · EldanRing
Open · 12B
EldanRing/Winnow-12B@b2b14213, its Q8_0 GGUF served by winnow-inference@d4631fb, the server its Logic & Math runs use, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 94.3% of 942, a conflicting one 51.7% of 329, none 86.2% of 798
KnowLine-9B-Gen1 · PelaAI
Open · 9.0B
PelaAI/KnowLine-9B-Gen1@325378d1, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card's release launcher names 0.5.21, which we do not have), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 84.4% of 942, a conflicting one 50.8% of 329, none 97.5% of 798
Granite-4.2-3B · IBM
Open · 3.0B
ibm-granite/granite-4.2-3b@4e9c275e, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 88.3% of 942, a conflicting one 79.0% of 329, none 80.7% of 798
LFM3-1.7B · Liquid AI
Open · 1.7B
LiquidAI/LFM3-1.7B@0dc4268b, BF16, cu135-nightly-be47189f37f6162991366ff6b91539f6e9d1f432 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.1% (per seed in seed_runs; cut off at the token limit per seed: 2, 1, 1); the three runs are fresh runs (14 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the runs they replace and the server settings recorded for them; seed 0 is the rerun made to diagnose the missing answer lines; the earlier runs stay in history; gpu_hours is the mean of the three runs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0.1% of 2,069 on average (seed 0: 2; seed 1: 1; seed 2: 1), counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label, the mean of its seeds: the reference answer 91.8% of 942, a conflicting one 63.3% of 329, none 83.0% of 798
Seeds: seed 0: grading:chat-lfm3-1.7b-seed0-replay-v1, 84.0% right, 2 invalid, 2 cut off, 215 wrong answers caught; seed 1: grading:chat-lfm3-1.7b-seed1-r2-v1, 83.3% right, 1 invalid, 1 cut off, 194 wrong answers caught; seed 2: grading:chat-lfm3-1.7b-seed2-r2-v1, 84.3% right, 1 invalid, 1 cut off, 216 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval
Measures: mean of 3 seeds; latency over all requests of the three runs
Outside the three: its first run, grading:chat-lfm3-1.7b-v1, asked 64 requests at a time, where the three seed runs were asked 30 at a time; not ranked, kept in the file published before the seeds
Verdin-4B-v1.1 · victorbennett
Open · 4.0B
victorbennett/Verdin-4B-v1.1@45830d21, its card's own server (python -m verdin.server, in the package versions its card lists: torch 2.8.0 cu128, transformers 5.17.0, flash-linear-attention 0.5.2, triton 3.8.0; no causal-conv1d, so its own torch conv path), BF16, on one RTX PRO 6000 (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 95.0% of 942, a conflicting one 46.8% of 329, none 85.2% of 798
KnowLine-9B-Gen2 · PelaAI
Open · 9.0B
PelaAI/KnowLine-9B-Gen2@8ec8eb60, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card names 0.5.21, which we do not have), BF16, SHA256SUMS checked, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 89.3% of 942, a conflicting one 35.9% of 329, none 96.4% of 798
Gemma 4 E4B · Google
Open · 8.0B
google/gemma-4-E4B-it@4ecc2ba6, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 1.5% (per seed in seed_runs; cut off at the token limit per seed: 0, 0, 0); three new runs (16 Nov, every reply kept), seeds 0, 1, 2 at Gemma's own sampling (its generation_config.json: temperature 1.0, top_p 0.95, top_k 64; Ümit #10554), with the weights, template and token limit of the temperature-0 row, which stays as its own row; their server settings recorded; gpu_hours is the mean of the three runs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 1.5% of 2,069 on average (seed 0: 34; seed 1: 31; seed 2: 30), counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label, the mean of its seeds: the reference answer 91.0% of 942, a conflicting one 64.3% of 329, none 82.2% of 798
Seeds: seed 0: grading:chat-gemma-4-e4b-think-sampled-seed0-v1, 84.0% right, 34 invalid, 0 cut off, 216 wrong answers caught; seed 1: grading:chat-gemma-4-e4b-think-sampled-seed1-v1, 82.7% right, 31 invalid, 0 cut off, 208 wrong answers caught; seed 2: grading:chat-gemma-4-e4b-think-sampled-seed2-v1, 83.5% right, 30 invalid, 0 cut off, 211 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval
Measures: mean of 3 seeds; latency over all requests of the three runs
LFM3-1.7B · Liquid AI
Open · 1.7B
LiquidAI/LFM3-1.7B@0dc4268b, BF16, cu135-nightly-be47189f37f6162991366ff6b91539f6e9d1f432 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.1% (per seed in seed_runs; cut off at the token limit per seed: 2, 2, 1); seed 0 is the published single run (14 Nov, every reply kept); seeds 1 and 2 are new runs (15 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0.1% of 2,069 on average (seed 0: 2; seed 1: 2; seed 2: 1), counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label, the mean of its seeds: the reference answer 92.1% of 942, a conflicting one 62.6% of 329, none 81.3% of 798
Seeds: seed 0: grading:chat-lfm3-1.7b-nothink-r2-v1, 83.8% right, 2 invalid, 2 cut off, 212 wrong answers caught; seed 1: grading:chat-lfm3-1.7b-nothink-seed1-r2-v1, 83.7% right, 2 invalid, 2 cut off, 209 wrong answers caught; seed 2: grading:chat-lfm3-1.7b-nothink-seed2-r2-v1, 82.3% right, 1 invalid, 1 cut off, 197 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval
Measures: mean of 3 seeds; latency over all requests of the three runs
Valen-Decision-1.5-9B · Valen-Team
Open · 9.0B
Valen-Team/Valen-Decision-1.5-9B@5a512117, its card's vLLM deployment (valen-vllm, the Valen plugin at Valen-Team/Valen@773d3a64 in the official vllm/vllm-openai:v0.33.0 image), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 93.4% of 942, a conflicting one 39.8% of 329, none 88.1% of 798
Cygnet · Blockbrain Labs
Open · 12B
no weights of its own: google/gemma-4-12B-it@707f0a3b, unchanged, on vLLM 0.30.0 behind cygnet-recipe's shim (blockbrain-ai/cygnet-recipe@3cf591c, temperature 3.4), the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 94.4% of 942, a conflicting one 29.2% of 329, none 89.0% of 798
Gemma 4 E2B · Google
Open · 5.1B
google/gemma-4-E2B-it@4adf1574, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 89.6% of 942, a conflicting one 49.5% of 329, none 83.2% of 798
Qwen3.5-2B · Alibaba
Open · 2.0B
Qwen/Qwen3.5-2B@15852e8c, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 78 of 2069 replies (4%) had no answer marker and count wrong (5 of them cut off at the 32,768-token limit); a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history; an earlier attempt of this same run stopped part-way (1,210 recorded pairs and 64 further requests possibly answered but not recorded) and is kept apart, not merged; this attempt started again from the first pair, so its outcomes are this attempt's recorded outputs, not guaranteed to be the model's first outputs on those pairs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 78 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 86.5% of 942, a conflicting one 55.3% of 329, none 83.1% of 798
Clef-Flash · Cloudflare
Open · 9.0B
Cloudflare/clef-flash@fde727a2, the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; server probabilities are rounded to four decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 92.6% of 942, a conflicting one 37.7% of 329, none 82.2% of 798
Gemma 4 12B Unified · Google
Open · 12B
google/gemma-4-12B-it@43c9d014, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 296 of 2069 replies (14%) had no answer marker and count wrong (295 of them cut off at the 32,768-token limit); a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 296 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 93.4% of 942, a conflicting one 38.0% of 329, none 77.4% of 798
MiniCPM5-2B · OpenBMB
Open · 2.5B
openbmb/MiniCPM5-2B@42433e90, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.2% (per seed in seed_runs; cut off at the token limit per seed: 1, 0, 0); seed 0 is the published single run (14 Nov, every reply kept); seeds 1 and 2 are new runs (15 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs (GPU seconds per seed: 219, 76, 76, at the same recorded settings; the cause of the difference is not known)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0.2% of 2,069 on average (seed 0: 6; seed 1: 5; seed 2: 4), counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label, the mean of its seeds: the reference answer 87.9% of 942, a conflicting one 28.8% of 329, none 86.6% of 798
Seeds: seed 0: grading:chat-minicpm5-2b-nothink-r2-v1, 78.0% right, 6 invalid, 1 cut off, 93 wrong answers caught; seed 1: grading:chat-minicpm5-2b-nothink-seed1-r2-v1, 78.0% right, 5 invalid, 0 cut off, 94 wrong answers caught; seed 2: grading:chat-minicpm5-2b-nothink-seed2-r2-v1, 78.0% right, 4 invalid, 0 cut off, 97 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval
Measures: mean of 3 seeds; latency over all requests of the three runs
Qwen3.5-4B · Alibaba
Open · 4.0B
Qwen/Qwen3.5-4B@851bf6e8, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 179 of 2069 replies (9%) had no answer marker and count wrong (0 of them cut off at the 32,768-token limit); a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 179 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 87.7% of 942, a conflicting one 73.2% of 329, none 67.2% of 798
StartLux-Decision-v1.1-2B-Instant · StartLux
Open · 2.0B
startlux-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd14, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 88.8% of 942, a conflicting one 67.5% of 329, none 67.9% of 798
GLiDE 2B · Fastino
Open · 1.9B
fastino/GLiDE-2B@00f122c1, GLiDE server (python -m glide.server) in vLLM v0.30.0 image with flash-linear-attention 0.5.2 on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 89.7% of 942, a conflicting one 60.8% of 329, none 68.9% of 798
Qwen3.5-4B · Alibaba
Open · 4.0B
Qwen/Qwen3.5-4B@851bf6e8, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 1.0% (per seed in seed_runs; cut off at the token limit per seed: 0, 1, 0); seed 0 is the published single run (every reply kept); seeds 1 and 2 are new runs (16 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 1.0% of 2,069 on average (seed 0: 20; seed 1: 25; seed 2: 15), counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label, the mean of its seeds: the reference answer 94.4% of 942, a conflicting one 61.2% of 329, none 61.5% of 798
Seeds: seed 0: grading:chat-qwen3.5-4b-nothink-v1, 76.4% right, 20 invalid, 0 cut off, 196 wrong answers caught; seed 1: grading:chat-qwen3.5-4b-nothink-seed1-v1, 76.3% right, 25 invalid, 1 cut off, 202 wrong answers caught; seed 2: grading:chat-qwen3.5-4b-nothink-seed2-v1, 76.6% right, 15 invalid, 0 cut off, 206 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval
Measures: mean of 3 seeds; latency over all requests of the three runs
Metask-Jev-4B · Metask Lab
Open · 4.5B
wayfind/metask-jev-4b-policy-mix@ea20fe85, the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 96.2% of 942, a conflicting one 23.1% of 329, none 70.0% of 798
KnowLine-2B-Gen1 · PelaAI
Open · 2.0B
PelaAI/KnowLine-2B-Gen1@53245d47, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 92.6% of 942, a conflicting one 16.4% of 329, none 75.9% of 798
GLiDE 0.8B · Fastino
Open · 0.8B
fastino/GLiDE-0.8B@9eacdeec, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 89.9% of 942, a conflicting one 31.3% of 329, none 72.4% of 798
Strands Decider 2B · Strands Agents
Open · 2.0B
StrandsAgents/strands-decider-2B-hobson-v21@2b52a623 (a LoRA and head on Qwen3.5-2B-Base), the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; server probabilities are rounded to four decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 96.4% of 942, a conflicting one 4.6% of 329, none 75.1% of 798
LFM2.5-8B-A1B · Liquid AI
Open · 8.3B MoE · 1.5B active
LiquidAI/LFM2.5-8B-A1B@dd83f5cb, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept; an earlier attempt of this same run stopped part-way (45 recorded pairs and an unknown number of further requests possibly answered but not recorded) and is kept apart, not merged; this attempt started again from the first pair, so its outcomes are this attempt's recorded outputs, not guaranteed to be the model's first outputs on those pairs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 79.5% of 942, a conflicting one 8.8% of 329, none 88.7% of 798
Gemma 4 E2B · Google
Open · 5.1B
google/gemma-4-E2B-it@4adf1574, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.03% (per seed in seed_runs; cut off at the token limit per seed: 0, 0, 0); three new runs (16 Nov, every reply kept), seeds 0, 1, 2 at Gemma's own sampling (its generation_config.json: temperature 1.0, top_p 0.95, top_k 64; Ümit #10554), with the weights, template and token limit of the temperature-0 row, which stays as its own row; their server settings recorded; gpu_hours is the mean of the three runs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0.03% of 2,069 on average (seed 0: 1; seed 1: 0; seed 2: 1), counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label, the mean of its seeds: the reference answer 94.3% of 942, a conflicting one 7.7% of 329, none 67.3% of 798
Seeds: seed 0: grading:chat-gemma-4-e2b-sampled-seed0-v1, 69.7% right, 1 invalid, 0 cut off, 22 wrong answers caught; seed 1: grading:chat-gemma-4-e2b-sampled-seed1-v1, 70.6% right, 0 invalid, 0 cut off, 29 wrong answers caught; seed 2: grading:chat-gemma-4-e2b-sampled-seed2-v1, 70.1% right, 1 invalid, 0 cut off, 25 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval
Measures: mean of 3 seeds; latency over all requests of the three runs
Gemma 4 E2B · Google
Open · 5.1B
google/gemma-4-E2B-it@4adf1574, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking off; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label: the reference answer 94.7% of 942, a conflicting one 6.1% of 329, none 66.8% of 798
Qwen3.5-2B · Alibaba
Open · 2.3B
Qwen/Qwen3.5-2B@15852e8c, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 2.0% (per seed in seed_runs; cut off at the token limit per seed: 1, 0, 0); seed 0 is the published single run (every reply kept); seeds 1 and 2 are new runs (16 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 2.0% of 2,069 on average (seed 0: 37; seed 1: 45; seed 2: 43), counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label, the mean of its seeds: the reference answer 80.4% of 942, a conflicting one 34.4% of 329, none 64.2% of 798
Seeds: seed 0: grading:chat-qwen3.5-2b-nothink-v1, 66.2% right, 37 invalid, 1 cut off, 108 wrong answers caught; seed 1: grading:chat-qwen3.5-2b-nothink-seed1-v1, 67.1% right, 45 invalid, 0 cut off, 116 wrong answers caught; seed 2: grading:chat-qwen3.5-2b-nothink-seed2-v1, 67.2% right, 43 invalid, 0 cut off, 116 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval
Measures: mean of 3 seeds; latency over all requests of the three runs
OneJev1.6-0.8B · OmniJev
Open · 0.8B
OmniJev/OneJev1.6-0.8B@bcd88a38, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; server probabilities are rounded to six decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 82.3% of 942, a conflicting one 33.4% of 329, none 62.0% of 798
d1-2B · Liquid AI
Open · 1.9B
LiquidAI/d1-2B@10e7671d, d1_server (Liquid's model code) on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 84.3% of 942, a conflicting one 85.7% of 329, none 37.3% of 798
StartLux-Decision-v1.1-0.8B-Instant · StartLux
Open · 0.8B
startlux-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae64, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 83.9% of 942, a conflicting one 72.3% of 329, none 42.1% of 798
GLiDE 0.4B · Fastino
Open · 0.4B
fastino/GLiDE-0.4B@6bda7ab9, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 84.0% of 942, a conflicting one 31.0% of 329, none 51.2% of 798
GLiClass v4 Edge · Knowledgator
Open · 0.4B
knowledgator/gliclass-v4-edge@885af1af, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 92.5% of 942, a conflicting one 34.9% of 329, none 39.5% of 798
Decider 1 · meraGPT
Closed · Efficient
meraGPT decisions API (meragpt.com, model decider-1); one question per request, 8 requests at a time (latency not comparable across different concurrency)
Hardware: the maker's API (its latency includes the network)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 70.2% of 942, a conflicting one 31.9% of 329, none 65.3% of 798
List price $0.04 per 1M input tokens, its source (2026-10-19)
Raon-1 nano · KRAFTON
Open · 3.1B
KRAFTON/Raon-1-nano@fcf2d1d5, BF16, KRAFTON vLLM fork krafton-ai/vllm@13a5dd87 (not a vLLM release) on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.03% (per seed in seed_runs; cut off at the token limit per seed: 0, 1, 0); seed 0 is the published single run (every reply kept); seeds 1 and 2 are new runs (16 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs
Hardware: one RTX PRO 6000 Blackwell (96 GB), ours
Invalid replies: 0.03% of 2,069 on average (seed 0: 0; seed 1: 1; seed 2: 1), counted wrong
Option probabilities: none (a chat model's label), so no calibration error and no confidence
Recall by label, the mean of its seeds: the reference answer 64.9% of 942, a conflicting one 19.9% of 329, none 71.9% of 798
Seeds: seed 0: grading:chat-raon-1-nano-nothink-v1, 60.5% right, 0 invalid, 0 cut off, 65 wrong answers caught; seed 1: grading:chat-raon-1-nano-nothink-seed1-v1, 60.4% right, 1 invalid, 1 cut off, 68 wrong answers caught; seed 2: grading:chat-raon-1-nano-nothink-seed2-v1, 60.4% right, 1 invalid, 0 cut off, 63 wrong answers caught. Its score is their mean; the lowest and highest beside it are a range, not a confidence interval
Measures: mean of 3 seeds; latency over all requests of the three runs
KnowLine-0.8B-Gen1 · PelaAI
Open · 0.8B
PelaAI/KnowLine-0.8B-Gen1@6b27831f, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency)
Hardware: one NVIDIA DGX Spark (GB10)
Invalid replies: 0 of 2,069, counted wrong
Option probabilities: yes, so its calibration error is measured
Recall by label: the reference answer 98.0% of 942, a conflicting one 10.0% of 329, none 33.2% of 798