When should you trust a decision model?
Our reading comprehension and grammar test: 2,198 questions answered by 37 decision models, with chat models for comparison. A decision model scores each answer option and picks one. Grammar questions test whether a model knows a rule and can apply it. Every answer is public.
The questions are in native Turkish (not translated).
74 of 74 setups
Part
Questions per request
Accuracy
Share of the questions answered right · higher is better
Accuracy against cost or latency
Up: accuracy, higher is better · across: cost or latency, lower is better
Calibration
How far a model’s confidence is from how often it is right, as shipped · lower is better
Latency
Median time per request · lower is better
Cost
Estimated dollars per 1,000 questions · lower is better
Cost of every setup
Every setup we ran, whatever the filters, the model selection or the charts’ default · cheapest first, with accuracy, latency, the confidence cutoff and hallucination beside the cost · lower cost is better
Info The tables, the summary and the notes behind the charts
Leaderboard
Swipe for more columns →
Decision models
| Model | Accuracy (%) | Reading (%) | Grammar (%) | Median latency per request (s) | $ per 1,000 questions |
|---|---|---|---|---|---|
| Liquid d1-ultrad1-ultra1 question per request · Decision model1 per request | 77.175.3 to 78.8 | 92.1 | 70.7 | 0.223 | 0.014 |
| Fastino GLiDE v1.2GLiDE v1.2local · 1 question per request · Decision model1 per request · local | 68.566.5 to 70.4 | 89.6 | 59.5 | 0.413 | 0.014§ |
| Fastino GLiDEGLiDE1 question per request · Decision model1 per request | 67.965.9 to 69.8 | 87.8 | 59.4 | 2.09 | 0.160 |
| vLLM-SR d3d31 question per request · Decision model1 per request | 67.865.8 to 69.7 | 89.2 | 58.7 | 0.092 | Not measured |
| vLLM-SR d3-ultrad3-ultratransformers, rev 52ba9e7 · 1 question per request · Decision model1 per request · transformers | 67.865.8 to 69.7 | 89.8 | 58.4 | 0.333 | Not measured |
| Drex Max v11 question per request · Decision model1 per request | 67.765.8 to 69.7 | 89.6 | 58.4 | 0.082 | 0.023§ |
| Drex Max 1.11 question per request · Decision model1 per request | 67.465.4 to 69.3 | 88.6 | 58.4 | 0.313 | 0.023 |
| Mammoth-11 question per request · Decision model1 per request | 67.165.1 to 69.0 | 90.4 | 57.2 | 0.169 | 0.040§ |
| Perplexity Decider 27B v1.2Decider v1.2local · 1 question per request · Decision model1 per request · local | 67.165.1 to 69.0 | 89.2 | 57.7 | 0.580 | 0.008§ |
| Perplexity Decider 27B v1.1Decider v1.11 question per request · Decision model1 per request | 66.964.9 to 68.8 | 89.0 | 57.5 | 0.233 | 0.008 |
| SubQ Snap PreviewSnap Preview1 question per request · Decision model1 per request | 65.263.2 to 67.2 | 87.7 | 55.7 | 0.389 | 0.019 |
| SubQ Snap 1.1Snap 1.11 question per request · Decision model1 per request | 63.961.8 to 65.9 | 87.0 | 54.0 | 0.378 | 0.015 |
| GPT-6 Luna (Decisions API)Decisions API, beta · 1 question per request · Decision model1 per request | 63.661.6 to 65.6 | 85.5 | 54.3 | 0.191 | 0.040 |
| Liquid d1.1d1.11 question per request · Decision model1 per request | 61.559.5 to 63.5 | 84.6 | 51.7 | 0.195 | 0.007 |
| Celeris-1 Decision1 question per request · Decision model1 per request | 60.558.4 to 62.5 | 85.2 | 49.9 | 0.401 | 0.015 |
| Perplexity Decider 27B v1Decider v11 question per request · Decision model1 per request | 60.458.3 to 62.4 | 86.0 | 49.5 | 0.216 | 0.016 |
| OneJev 27BOneJev1 question per request · Decision model1 per request | 59.657.5 to 61.6 | 85.2 | 48.7 | 0.125 | Not measured |
| Jev 1.14.0Jev 1.141 question per request · Decision model1 per request | 59.257.1 to 61.2 | 85.1 | 48.2 | 0.211 | 0.029 |
| Inception Mercury DecideMercury Decidevia OpenRouter · 1 question per request · Decision model1 per request · via OpenRouter | 58.356.2 to 60.3 | 84.5 | 47.1 | 0.313 | 0.008 |
| Jev 1.13.0Jev 1.131 question per request · Decision model1 per request | 57.555.4 to 59.6 | 83.7 | 46.4 | 0.243 | 0.029 |
| Clef1 question per request · Decision model1 per request | 57.355.2 to 59.3 | 85.1 | 45.5 | 0.205 | 0.110§ |
| Liquid d1d11 question per request · Decision model1 per request | 57.054.9 to 59.1 | 82.8 | 46.0 | 0.305 | 0.007 |
| Drex 2.01 question per request · Decision model1 per request | 55.753.6 to 57.8 | 81.4 | 44.8 | 0.325 | 0.019 |
| Cygnet1 question per request · Decision model1 per request | 55.753.6 to 57.8 | 82.8 | 44.2 | 0.036 | Not measured |
| Open-Jev 27B v1.1Open-Jev1 question per request · Decision model1 per request | 55.753.6 to 57.8 | 82.9 | 44.1 | 0.386 | Not measured |
| Winnow-12B1 question per request · Decision model1 per request | 55.553.4 to 57.5 | 82.0 | 44.2 | 0.059 | Not measured |
| Elephant-11 question per request · Decision model1 per request | 52.450.3 to 54.4 | 80.5 | 40.4 | 0.074 | 0.014§ |
| Perplexity Decider Flash v1.2Decider Flash v1.2local · 1 question per request · Decision model1 per request · local | 47.845.7 to 49.9 | 72.7 | 37.2 | 0.132 | 0.004§ |
| Microsoft-Decision-1via OpenRouter · 1 question per request · Decision model1 per request · via OpenRouter | 46.544.4 to 48.5 | 73.5 | 35.0 | 0.286 | 0.014 |
| Clef-flash1 question per request · Decision model1 per request | 45.743.6 to 47.8 | 73.0 | 34.1 | 0.075 | 0.017§ |
| Metask-Jev 4B1 question per request · Decision model1 per request | 40.838.7 to 42.8 | 62.8 | 31.4 | 0.090 | Not measured |
| Liquid d1-2B (open weights)d1-2B1 question per request · Decision model1 per request | 29.427.5 to 31.3 | 40.9 | 24.5 | 0.019 | Not measured |
| Strands Decider 2B1 question per request · Decision model1 per request | 27.525.7 to 29.4 | 36.9 | 23.5 | 0.026 | Not measured |
| meraGPT Decider 11 question per request · Decision model1 per request | 27.125.3 to 29.0 | 34.6 | 23.9 | 0.382 | 0.015 |
| GLiNER2.5-multi-Decide1 question per request · Decision model1 per request | 20.218.6 to 21.9 | 21.0 | 19.8 | 0.009 | Not measured |
| Laya1 question per request · Decision model1 per request | 19.017.4 to 20.7 | 20.9 | 18.2 | 0.011 | Not measured |
| Laya multilingual1 question per request · Decision model1 per request | 17.916.3 to 19.5 | 17.1 | 18.2 | 0.009 | Not measured |
Other models: chat and general models
| Model | Accuracy (%) | Reading (%) | Grammar (%) | Median latency per request (s) | $ per 1,000 questions |
|---|---|---|---|---|---|
| Gemini 3.8 Flashhigh reasoning · 16 questions per request · Chat model16 per request · high reasoning · chat model | 98.197.4 to 98.6 | 97.1 | 98.5 | 53.7 | 2.419 |
| GPT-6 Astrahigh reasoning · 16 questions per request · Chat model16 per request · high reasoning · chat model | 98.097.3 to 98.5 | 97.1 | 98.3 | 21.7 | 6.066 |
| GPT-6.1 Solhigh reasoning · 16 questions per request · Chat model16 per request · high reasoning · chat model | 97.897.1 to 98.3 | 96.5 | 98.3 | 24.7 | 1.622 |
| GPT-6 Astralow reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 97.696.9 to 98.2 | 97.0 | 97.9 | 13.0* | 5.378 |
| GPT-6 Astralow reasoning · 1 question per request · Chat model1 per request · low reasoning · chat model | 97.496.7 to 98.0 | 97.3 | 97.5 | 6.41 | 26.738 |
| Gemini 3.8 Flashlow reasoning · 1 question per request · Chat model1 per request · low reasoning · chat model | 97.196.3 to 97.8 | 96.2 | 97.5 | 2.72 | 0.963 |
| GPT-6.1 Sollow reasoning · 1 question per request · Chat model1 per request · low reasoning · chat model | 96.795.9 to 97.4 | 96.0 | 97.0 | 6.61 | 5.569 |
| GPT-6.1 Sollow reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 96.795.8 to 97.4 | 96.2 | 96.9 | 13.3 | 1.130 |
| Gemini 4 Argonlow reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 96.495.5 to 97.1 | 97.0 | 96.1 | 5.46 | 1.786 |
| Claude Fable 5.1high reasoning · 16 questions per request · Chat model16 per request · high reasoning · chat model | 96.195.2 to 96.9 | 95.6 | 96.4 | 30.3 | 17.042 |
| Claude Opus 5.5high reasoning · 16 questions per request · Chat model16 per request · high reasoning · chat model | 95.094.0 to 95.8 | 95.6 | 94.7 | 22.1 | 6.509 |
| GPT-6.1 Terrahigh reasoning · 16 questions per request · Chat model16 per request · high reasoning · chat model | 94.393.2 to 95.2 | 96.2 | 93.5 | 7.10 | 0.515 |
| Claude Opus 5.5low reasoning · 1 question per request · Chat model1 per request · low reasoning · chat model | 93.392.2 to 94.3 | 95.7 | 92.3 | 3.71 | 11.610 |
| Gemini 3.8 Flashlow reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 93.392.1 to 94.2 | 95.1 | 92.5 | 2.19 | 0.269 |
| Grok 4.7high reasoning · 16 questions per request · Chat model16 per request · high reasoning · chat model | 92.791.6 to 93.7 | 95.0 | 91.8 | 401.9 | 14.292 |
| DeepSeek-V4.1-Flash, first releasereasoning on · 16 questions per request · Chat model16 per request · reasoning on · chat model | 92.090.8 to 93.1 | 93.4 | 91.4 | 365.1 | Not measured |
| GPT-6.1 Terralow reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 90.989.6 to 92.0 | 95.9 | 88.7 | 4.24 | 0.426 |
| Claude Opus 5.5low reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 90.489.1 to 91.6 | 94.8 | 88.6 | 13.4* | 4.931 |
| Claude Sonnet 5.5high reasoning · 1 question per request · Chat model1 per request · high reasoning · chat model | 89.287.8 to 90.4 | 94.2 | 87.0 | 4.64 | 7.748 |
| Claude Sonnet 5.5high reasoning · 16 questions per request · Chat model16 per request · high reasoning · chat model | 85.083.5 to 86.5 | 85.2 | 85.0 | 20.5* | 3.654 |
| Kimi K3.1 Prolow reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 82.881.1 to 84.3 | 93.8 | 78.1 | 44.9 | 2.048 |
| GLM-5.3-Flashreasoning on · 16 questions per request · Chat model16 per request · reasoning on · chat model | 78.877.0 to 80.5 | 89.3 | 74.3 | 102.6 | Not measured |
| GLM-5.3low reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 74.572.7 to 76.3 | 87.7 | 68.9 | 17.1 | 0.814 |
| Qwen3.8-27Blow reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 73.972.1 to 75.7 | 89.3 | 67.4 | 648.4 | 2.931§ |
| Gemma 4 31Breasoning off · 16 questions per request · Chat model16 per request · reasoning off · chat model | 73.171.2 to 74.9 | 90.9 | 65.6 | 3.97 | Not measured |
| Kimi K3.1 Proreasoning none · 16 questions per request · Chat model16 per request · reasoning none · chat model | 72.770.8 to 74.5 | 92.5 | 64.3 | 5.75 | 0.853 |
| GPT-6 Luna (chat API)low reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 68.366.3 to 70.2 | 87.3 | 60.2 | 7.71 | 0.048 |
| Gemma 4 31Breasoning off · 1 question per request · Chat model1 per request · reasoning off · chat model | 66.864.8 to 68.7 | 89.0 | 57.3 | 0.298 | Not measured |
| DeepSeek-V4.1-Flash, first releasereasoning off · 16 questions per request · Chat model16 per request · reasoning off · chat model | 65.963.9 to 67.9 | 87.8 | 56.6 | 4.36 | Not measured |
| Claude Sonnet 5.5low reasoning · 16 questions per request · Chat model16 per request · low reasoning · chat model | 61.459.3 to 63.4 | 82.9 | 52.2 | 3.87* | 1.896 |
| GLM-5.3-Flashreasoning off · 16 questions per request · Chat model16 per request · reasoning off · chat model | 57.855.7 to 59.8 | 82.2 | 47.4 | 1.07 | Not measured |
| Qwen3.8-27Bvote share · reasoning off · 1 question per request · Chat model1 per request · reasoning off · vote share · chat model | 54.352.2 to 56.4 | 84.0 | 41.7 | 1.62 | 0.825§ |
| Qwen3.8-27Bstated confidence · reasoning off · 1 question per request · Chat model1 per request · reasoning off · stated confidence · chat model | 54.252.1 to 56.3 | 83.4 | 41.8 | 0.433 | 0.234§ |
| Qwen3.8-27Byes/no scoring · 1 question per request · Chat model1 per request · yes/no scoring · chat model | 51.949.8 to 54.0 | 80.5 | 39.8 | 0.280 | 0.704§ |
| Qwen3-14Breasoning off · 16 questions per request · Chat model16 per request · reasoning off · chat model | 45.743.6 to 47.8 | 72.1 | 34.5 | 2.85 | Not measured |
| Erk-14Breasoning off · 16 questions per request · Chat model16 per request · reasoning off · chat model | 45.042.9 to 47.1 | 72.3 | 33.4 | 2.39 | Not measured |
| GLiNER2.5 Multi (base)1 question per request · Chat model1 per request · chat model | 17.515.9 to 19.1 | 17.2 | 17.6 | 0.032 | Not measured |
Not measured: cost for setups on our own hardware whose maker lists no price for the same model. § Estimated at the maker's list price for the same model; our hardware cost is not measured. * Measured on a sample (200 questions).
Results 0bdaa447, updated 3 November 2026; answer key version 2.
- 2,198 questions
- 20 topics
- 37 decision models
- 74 setups
Results 0bdaa447, updated 3 November 2026; answer key version 2.
Among decision models, none led on every measure:
- Accuracy: Liquid d1-ultra, clearly ahead at 77.1%; then GLiDE v1.2, GLiDE, vLLM-SR d3 and d3-ultra (tied), Drex Max v1 and 1.1, Mammoth-1, and Deciders v1.1 and v1.2, at 66.9% to 68.5%.
- Calibration gap, as shipped: Liquid d1.1 had the smallest gap among decision models.
- Better than a flat guess, after refit: Open-Jev 27B v1.1 (31.9%), then SubQ Snap 1.1 and Perplexity Decider 27B v1.1 (31.5% each). We did not test whether the gaps between them are real.
- Estimated saving: d1-ultra, the biggest in all 16 LLM setups.
- Reading vs grammar: decision models got up to 92.1% of the reading questions right, but at most 70.7% of the grammar questions.