All benches
Ranking
Combined 3 benchmark groups Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
OpenAIAnthropicMetaGoogleAlibabaPelaAIREAXSthānika AILiquid AITurion.AIBase LabsNace.AIFastinoPerplexitySubquadraticCelerisTypeSafeCloudflareInceptionApertix AIStartLuxBlockbrain LabsEldanRingZefan CaiOmniJevKnowledgator
No combined score for GLiDE · API: Fastino's API served a changed model between our benches' runs (Reading 6 Oct, Logic & Math 7 Nov, Backgammon 20 Oct to 7 Nov), so its bench scores are not one model's. (the rule)
Decision Index 4 benches Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
GoogleAlibabaOpenBMBIBMLiquid AIMachinePulsePelaAIPoindexter LabsREAXSthānika AILogitsFastinoPerplexityTurion.AISubquadraticTypeSafetattrongvunewsbangMicrosoftStartLuxBase LabsPatrick-18scar-aiApertix AIValen-TeamKRAFTONvictorbennettquincyqiangEldanRingCloudflareBlockbrain LabsMetask LabOmniJevStrands AgentsKnowledgator
No Decision Index for GLiDE · API: Fastino's API served a changed model between our benches' runs (Reading 6 Oct, Logic & Math 7 Nov, Backgammon 20 Oct to 7 Nov), so its bench scores are not one model's. (the rule)
Turkish Reading Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
AnthropicGoogleOpenAIxAIAlibabaDeepSeekMoonshotMachinePulsePelaAIPoindexter LabsTurion.AIREAXLiquid AILogitsSthānika AIFastinoBase LabsMetavLLM Semantic RouterNace.AIZ.aiPerplexitySubquadraticThinking MachinesXiaomiOmniJevCelerisCloudflareTypeSafeInceptiontattrongvuMiniMaxZefan CaiBlockbrain LabsEldanRingnewsbangApertix AIPatrick-18StartLuxscar-aiValen-TeamMicrosofteCloud TechnologyvictorbennettquincyqiangMetask LabStrands AgentsmeraGPTKnowledgatorConvai InnovationsOpenBMB
Turkish Grammar Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
GoogleAnthropicOpenAIxAIDeepSeekAlibabaMetaMoonshotPelaAIPoindexter LabsThinking MachinesTurion.AIZ.aiREAXMachinePulseLiquid AILogitsSthānika AIXiaomiFastinoPerplexityMiniMaxvLLM Semantic RouterNace.AIBase LabsSubquadraticCelerisOmniJevTypeSafeInceptiontattrongvuCloudflareBlockbrain LabsEldanRingZefan CaiApertix AInewsbangPatrick-18StartLuxscar-aiquincyqiangvictorbennettValen-TeamMicrosofteCloud TechnologyMetask LabmeraGPTStrands AgentsKnowledgatorConvai InnovationsOpenBMB
English Reading Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
AnthropicGoogleOpenAIAlibabaPelaAIxAIPoindexter LabsDeepSeekREAXMoonshotMachinePulseLogitsFastinoSthānika AITurion.AILiquid AIPerplexityBase LabsvLLM Semantic RouterNace.AIZ.aiMetaXiaomiThinking MachinestattrongvuStartLuxCloudflareOmniJevTypeSafeApertix AISubquadraticnewsbangInceptionMicrosoftZefan CaiMiniMaxCelerisBlockbrain LabsEldanRingValen-TeamPatrick-18scar-aivictorbennettquincyqiangOpenBMBMetask LabIBMeCloud TechnologyKnowledgatormeraGPTKRAFTONStrands AgentsConvai Innovations
Vision Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
MetaGoogleLiquid AIFastinoPelaAIvLLM Semantic RouterNace.AISubquadraticTypeSafeOmniJevOpenAIStartLux
Logic Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
OpenAIGoogleAlibabaAnthropicDeepSeekThinking MachinesMetaXiaomiMiniMaxZ.aiIBMOpenBMBLiquid AIKRAFTONMachinePulseREAXPelaAIPoindexter LabsFastinoSthānika AILogitsNace.AIPerplexityvLLM Semantic RouterBase LabseCloud TechnologySubquadraticTurion.AItattrongvuTypeSafeCelerisStartLuxnewsbangPatrick-18InceptionMicrosoftValen-Teamscar-aiApertix AIZefan CaiCloudflareOmniJevquincyqiangvictorbennettEldanRingBlockbrain LabsMetask LabKnowledgatorStrands Agents
Math Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
AnthropicAlibabaDeepSeekOpenAIThinking MachinesGoogleMetaxAIXiaomiOpenBMBIBMLiquid AIZ.aiMiniMaxeCloud TechnologyKRAFTONMachinePulsePelaAIREAXPoindexter LabsSthānika AILogitsBase LabsPerplexityFastinovLLM Semantic RouterNace.AICelerisTurion.AISubquadraticTypeSafenewsbangtattrongvuMicrosoftPatrick-18StartLuxscar-aiApertix AIValen-TeamZefan CaiOmniJevInceptionvictorbennettCloudflarequincyqiangEldanRingBlockbrain LabsMetask LabStrands AgentsKnowledgator
Backgammon Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
OpenAIGoogleAnthropicxAIDeepSeekMetaPelaAISthānika AIREAXBase LabsMoonshotTurion.AIAlibabaNace.AILiquid AIZ.aiTypeSafeFastinoCelerisPerplexityCloudflareBlockbrain LabsStartLuxSubquadraticInceptionApertix AIEldanRingZefan CaiOmniJevKnowledgator
Answer Check Score 0–100, chance = 0 · higher is better ↑
No model of this type and class has a score here.
MachinePulsePoindexter LabsPelaAISthānika AIREAXTurion.AILiquid AIThinking MachinesPerplexityFastinoLogitsSubquadraticMicrosoftGooglescar-aiTypeSafetattrongvuApertix AIStartLuxPatrick-18newsbangOpenBMBquincyqiangBase LabsEldanRingIBMvictorbennettValen-TeamBlockbrain LabsAlibabaCloudflareMetask LabStrands AgentsOmniJevKnowledgatormeraGPTKRAFTON
| Rank | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra high · chat | 96.1TR 97.4 L&M 98.3 BG 92.5 | Not applicable | 96.4 | 97.9 | 95.2 | Not run | 95.9 | 99.5 | 92.5 | Not run | ||
| GPT-6.1 Sol high · chat | 95.7TR 97.2 L&M 98.2 BG 91.6 | Not applicable | 95.6 | 97.9 | 94.5 | Not run | 95.7 | 99.5 | 91.6 | Not run | ||
| Claude Fable 5.1 high · chat | 93.7TR 95.2 L&M 97.7 BG 88.1 | Not applicable | 94.5 | 95.5 | 95.0 | Not run | 94.0 | 99.7 | 88.1 | Not run | ||
| Claude Opus 5.5 high · chat | 93.2TR 93.7 L&M 98.2 BG 87.7 | Not applicable | 94.5 | 93.4 | 93.1 | Not run | 95.2 | 99.7 | 87.7 | Not run | ||
| Claude Sonnet 5.5 high · chat | 88.3TR 81.3 L&M 97.8 BG 85.8 | Not applicable | 81.5 | 81.2 | 82.6 | Not run | 94.3 | 99.6 | 85.8 | Not run | ||
| Muse Ember max · chat | 87.7TR 82.0 L&M 97.7 BG 83.4 | Not applicable | 87.8 | 79.6 | 87.0 | 82.8 | 94.3 | 99.5 | 83.4 | Not run | ||
| Gemma 4 31B reasoning off · chat | Open · 31B | 82.1TR 66.4 L&M 96.7 BG 83.4 | Not applicable | 88.6 | 57.0 | 88.1 | Not run | 91.7 | 99.2 | 83.4 | Not run | |
| Qwen3.8-27B low · chat | Open · 27B | 80.2TR 68.3 L&M 96.8 BG 75.4 | Not applicable | 86.7 | 60.5 | 86.3 | Not run | 92.6 | 99.0 | 75.4 | Not run | |
| Qwen3.8 Flash no reasoning · chat | 74.1TR 53.8 L&M 97.0 BG 71.5 | Not applicable | 84.0 | 41.0 | 83.8 | Not run | 92.4 | 99.4 | 71.5 | Not run | ||
| 1 | KnowLine-31B-Gen1.5 LoRA, merged | Open · 31B | 73.2TR 77.3 L&M 63.9 BG 78.3 | 75.3EN 92.0 Logic 48.8 Math 71.7 AC 88.8 | 91.0 | 71.4 | 92.0 | 37.3 | 48.8 | 71.7 | 78.3 | 88.8 |
| 2 | REAX 1.3 blend | Open · 57B | 71.3TR 72.4 L&M 64.5 BG 77.1 | 74.8EN 90.6 Logic 49.1 Math 72.5 AC 87.0 | 90.5 | 64.7 | 90.6 | Not run | 49.1 | 72.5 | 77.1 | 87.0 |
| 3 | Sieve 31B v3 full fine-tune | Open · 31B | 69.4TR 67.7 L&M 63.2 BG 77.3 | 73.8EN 88.3 Logic 47.3 Math 71.4 AC 88.1 | 89.7 | 58.3 | 88.3 | Not run | 47.3 | 71.4 | 77.3 | 88.1 |
| 4 | d1-ultra | 68.3TR 71.3 L&M 59.4 BG 74.3 | 71.1EN 88.3 Logic 43.7 Math 67.6 AC 84.7 | 90.1 | 63.4 | 88.3 | 37.6 | 43.7 | 67.6 | 74.3 | 84.7 | |
| 5 | Turion Jev 31B LoRA, merged | Open · 31B | 68.2TR 75.1 L&M 54.1 BG 75.5 | 69.1EN 88.3 Logic 40.3 Math 61.3 AC 86.4 | 90.9 | 68.5 | 88.3 | Not run | 40.3 | 61.3 | 75.5 | 86.4 |
| 6 | Gemini Decision 1 20261111 | 68.0TR 71.9 L&M 55.1 BG 76.9 | 66.9EN 88.8 Logic 37.6 Math 64.2 AC 77.0 | 90.1 | 64.2 | 88.8 | 58.1 | 37.6 | 64.2 | 76.9 | 77.0 | |
| 7 | Mammoth-1 post-trained | Open · 27B | 64.8TR 58.9 L&M 59.1 BG 76.4 | Not applicable | 88.0 | 46.5 | 87.7 | Not run | 43.1 | 67.5 | 76.4 | Not run |
| 8 | Drex Max fine-tune | Open · 27B | 63.4TR 59.7 L&M 55.5 BG 74.9 | Not applicable | 87.0 | 48.0 | 87.0 | 33.5 | 41.3 | 62.9 | 74.9 | Not run |
| 9 | Drex Max v1.1 | 63.2TR 59.2 L&M 56.1 BG 74.5 | Not applicable | 85.7 | 48.0 | 85.6 | 31.4 | 43.4 | 62.6 | 74.5 | Not run | |
| 10 | GLiDE v1.2 | Open · 27B | 62.6TR 60.6 L&M 56.1 BG 71.1 | 69.8EN 87.9 Logic 44.0 Math 62.3 AC 84.9 | 87.0 | 49.3 | 87.9 | Not run | 44.0 | 62.3 | 71.1 | 84.9 |
| 11 | Perplexity Decider 27B v1.2 DGX Sparksetups differ | Open · 27B | 61.2TR 58.8 L&M 55.2 BG 69.5 | Not applicable | 86.5 | 47.1 | 85.8 | Not run | 41.7 | 62.2 | 69.5 | Not run |
| 12 | SubQ Snap 1.1 | 58.5TR 54.8 L&M 53.8 BG 66.8 | 67.1EN 82.4 Logic 40.4 Math 60.7 AC 84.9 | 83.8 | 42.5 | 82.4 | 32.2 | 40.4 | 60.7 | 66.8 | 84.9 | |
| 13 | celeris-1-decision | 58.0TR 50.6 L&M 52.8 BG 70.6 | Not applicable | 81.5 | 37.4 | 80.1 | Not run | 34.1 | 62.5 | 70.6 | Not run | |
| 14 | Jev 1.14 | 57.1TR 49.0 L&M 49.2 BG 73.1 | 64.5EN 82.6 Logic 35.2 Math 56.4 AC 83.6 | 81.3 | 35.2 | 82.6 | 29.1 | 35.2 | 56.4 | 73.1 | 83.6 | |
| 15 | Jev 1.13 | 55.3TR 46.9 L&M 47.5 BG 71.6 | 63.3EN 82.4 Logic 32.9 Math 55.1 AC 83.0 | 79.6 | 33.0 | 82.4 | Not run | 32.9 | 55.1 | 71.6 | 83.0 | |
| 16 | Drex 2.0 | Open · 35B MoE · 3.0B active | 51.4TR 44.7 L&M 41.4 BG 68.3 | Not applicable | 76.8 | 31.0 | 82.4 | Not run | 27.3 | 48.7 | 68.3 | Not run |
| 17 | Clef post-trained | Open · 27B | 50.0TR 46.6 L&M 34.5 BG 69.0 | Not applicable | 81.3 | 31.8 | 83.3 | Not run | 26.0 | 38.9 | 69.0 | Not run |
| 18 | Elephant-1 post-trained | Open · 9.0B | 49.7TR 40.5 L&M 45.3 BG 63.2 | 60.8EN 81.7 Logic 33.0 Math 51.7 AC 76.9 | 75.6 | 25.5 | 81.7 | Not run | 33.0 | 51.7 | 63.2 | 76.9 |
| 19 | Mercury Decide | 48.8TR 47.9 L&M 38.3 BG 60.4 | Not applicable | 80.6 | 33.9 | 82.2 | Not run | 31.3 | 41.9 | 60.4 | Not run | |
| 20 | Cerno 9B v1.2 full fine-tune | Open · 9.0B | 47.5TR 43.1 L&M 39.6 BG 60.0 | 59.6EN 82.4 Logic 29.1 Math 45.0 AC 81.9 | 74.7 | 29.6 | 82.4 | Not run | 29.1 | 45.0 | 60.0 | 81.9 |
| 21 | StartLux-Decision-v1.1-4B-Instant fine-tune | Open · 4.0B | 47.4TR 34.8 L&M 40.5 BG 66.8 | 59.3EN 78.5 Logic 32.3 Math 44.8 AC 81.7 | 67.6 | 20.9 | 78.5 | Not run | 32.3 | 44.8 | 66.8 | 81.7 |
| 22 | Cygnet unchanged basesetups differ | Open · 12B | 47.0TR 44.6 L&M 28.3 BG 68.2 | 51.8EN 79.9 Logic 24.1 Math 30.4 AC 72.9 | 78.5 | 30.2 | 79.9 | Not run | 24.1 | 30.4 | 68.2 | 72.9 |
| 23 | Winnow-12B LoRA, merged | Open · 12B | 44.6TR 44.3 L&M 30.5 BG 58.9 | 53.7EN 79.9 Logic 24.7 Math 33.6 AC 76.6 | 77.5 | 30.2 | 79.9 | Not run | 24.7 | 33.6 | 58.9 | 76.6 |
| 24 | KnowLine 4B Gen5.5 LoRA, merged | Open · 4.0B | 43.5TR 35.6 L&M 39.1 BG 55.8 | 57.4EN 76.0 Logic 27.9 Math 45.0 AC 80.6 | 68.6 | 21.6 | 76.0 | 12.7 | 27.9 | 45.0 | 55.8 | 80.6 |
| 25 | Open-Jev-27B-v1.1 LoRA + head | Open · 27B | 42.8TR 44.6 L&M 38.1 BG 45.9 | Not applicable | 78.7 | 30.1 | 81.7 | Not run | 28.2 | 43.2 | 45.9 | Not run |
| 26 | Perplexity Decider Flash v1.2 | Open · 4.0B | 41.4TR 34.7 L&M 38.9 BG 50.8 | 56.9EN 77.4 Logic 28.9 Math 44.1 AC 77.2 | 65.9 | 21.4 | 77.4 | Not run | 28.9 | 44.1 | 50.8 | 77.2 |
| 27 | StartLux-Decision-v1.1-2B-Instant fine-tune | Open · 2.0B | 32.1TR 17.8 L&M 19.9 BG 58.7 | 41.2EN 62.1 Logic 13.4 Math 23.3 AC 66.0 | 41.1 | 7.9 | 62.1 | 4.3 | 13.4 | 23.3 | 58.7 | 66.0 |
| 28 | KnowLine-2B-Gen1 LoRA, merged | Open · 2.0B | 31.2TR 21.0 L&M 17.6 BG 55.0 | 39.7EN 64.1 Logic 14.7 Math 19.1 AC 61.1 | 42.8 | 11.7 | 64.1 | Not run | 14.7 | 19.1 | 55.0 | 61.1 |
| 29 | d1-2B | Open · 1.9B | 30.5TR 11.7 L&M 18.6 BG 61.1 | 35.3EN 57.0 Logic 13.3 Math 21.3 AC 49.6 | 26.1 | 5.6 | 57.0 | 4.2 | 13.3 | 21.3 | 61.1 | 49.6 |
| 30 | GLiDE 2B | Open · 1.9B | 27.5TR 19.1 L&M 19.0 BG 44.4 | 41.1EN 62.5 Logic 15.3 Math 20.9 AC 65.6 | 39.0 | 10.7 | 62.5 | Not run | 15.3 | 20.9 | 44.4 | 65.6 |
| 31 | StartLux-Decision-v1.1-0.8B-Instant fine-tune | Open · 0.8B | 22.4TR 10.7 L&M 10.0 BG 46.6 | 29.1EN 49.5 Logic 5.9 Math 12.1 AC 48.9 | 18.8 | 7.3 | 49.5 | 5.3 | 5.9 | 12.1 | 46.6 | 48.9 |
| 32 | KnowLine-0.8B-Gen1 LoRA, merged | Open · 0.8B | 19.7TR 8.9 L&M 9.6 BG 40.6 | 26.4EN 48.6 Logic 8.3 Math 10.3 AC 38.5 | 22.8 | 3.0 | 48.6 | 4.8 | 8.3 | 10.3 | 40.6 | 38.5 |
| 33 | GLiDE 0.8B | Open · 0.8B | 19.7TR 11.6 L&M 13.9 BG 33.6 | 34.5EN 50.9 Logic 11.4 Math 15.1 AC 60.8 | 28.0 | 4.6 | 50.9 | Not run | 11.4 | 15.1 | 33.6 | 60.8 |
| 34 | OneJev1.6-0.8B full fine-tune | Open · 0.8B | 17.7TR 11.4 L&M 9.8 BG 31.9 | 29.6EN 49.3 Logic 8.8 Math 10.3 AC 50.0 | 24.5 | 5.8 | 49.3 | 8.1 | 8.8 | 10.3 | 31.9 | 50.0 |
| 35 | GLiDE 0.4B | Open · 0.4B | 3.7TR 2.8 L&M 1.8 BG 6.5 | 24.6EN 49.3 Logic 4.0 Math 0.7 AC 44.4 | 5.7 | 1.6 | 49.3 | Not run | 4.0 | 0.7 | 6.5 | 44.4 |
| 36 | GLiClass v4 Edge fine-tune | Open · 0.4B | −0.8TR 4.2 L&M 3.7 BG −10.2 | 25.0EN 47.9 Logic 4.4 Math 3.3 AC 44.3 | 7.4 | 2.8 | 47.9 | Not run | 4.4 | 3.3 | −10.2 | 44.3 |
| Adjudex 9B v5 | Open · 9.0B | Not applicable | 62.1EN 82.2 Logic 33.0 Math 52.6 AC 80.4 | 76.0 | 26.7 | 82.2 | Not run | 33.0 | 52.6 | Not run | 80.4 | |
| Chinese Decision Pointer 4B LoRA, merged | Open · 4.0B | Not applicable | 54.3EN 75.5 Logic 25.4 Math 38.0 AC 78.1 | 63.6 | 20.5 | 75.5 | Not run | 25.4 | 38.0 | Not run | 78.1 | |
| Claude Fable 5.2 max · chat | Not applicable | Not applicable | 97.1 | 97.9 | 97.5 | Not run | Not run | Not run | Not run | Not run | ||
| Claude Haiku 5.5 max · chat | Not applicable | Not applicable | 83.6 | 30.4 | 90.9 | Not run | Not run | Not run | Not run | Not run | ||
| Clef-Flash post-trained | Open · 9.0B | Not applicable | 52.1EN 77.8 Logic 23.7 Math 37.2 AC 69.8 | 66.3 | 17.6 | 77.8 | Not run | 23.7 | 37.2 | Not run | 69.8 | |
| Copernicus-31B | Open · 31B | Not applicable | 75.0EN 91.1 Logic 48.3 Math 71.7 AC 89.0 | 91.4 | 71.2 | 91.1 | Not run | 48.3 | 71.7 | Not run | 89.0 | |
| Decider 1 | Not applicable | Not applicable | 18.3 | 4.9 | 44.7 | Not run | Not run | Not run | Not run | 43.3 | ||
| DeepSeek-V4-Flash chat | 285B MoE · 13B active | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | 99.1 | Not run | Not run | |
| DeepSeek-V4.1-Flash reasoning on · chat · first release | Open · 552B MoE · 16B active | Not applicable | Not applicable | 91.8 | 89.3 | Not run | Not run | Not run | Not run | Not run | Not run | |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017setups differ | Open · 552B MoE · 16B active | Not applicable | Not applicable | 92.6 | 88.5 | 90.6 | Not run | 95.2 | 99.5 | Not run | Not run | |
| DeepSeek-V4.1-Flash low · API + NVIDIA RTX PRO 6000 Blackwell · first release | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | 83.9 | Not run | ||
| Dexy-Decision-9B-V2 fine-tune | Open · 9.0B | Not applicable | 60.2EN 79.4 Logic 31.7 Math 48.5 AC 81.1 | 74.5 | 24.6 | 79.4 | Not run | 31.7 | 48.5 | Not run | 81.1 | |
| Erk-14B reasoning off · chat | Open · 14B | Not applicable | Not applicable | 65.3 | 16.7 | 65.0 | Not run | 42.8 | 88.4 | Not run | Not run | |
| GLM-5.3 low · chat | Not applicable | Not applicable | 84.6 | 61.2 | 84.7 | Not run | Not run | Not run | 73.8 | Not run | ||
| GLM-5.3-Flash reasoning on · chatsetups differ | Open · 320B MoE · 18B active | Not applicable | Not applicable | 86.7 | 67.9 | 87.0 | Not run | 91.4 | 98.5 | Not run | Not run | |
| GLiDE API | Not applicable | Not applicable | 84.8 | 49.3 | Not run | Not run | 43.1 | 63.5 | 71.9 | 85.1 | ||
| GLiDE 27B | Open · 27B | Not applicable | 71.9EN 88.6 Logic 48.0 Math 65.0 AC 86.1 | 88.6 | 53.4 | 88.6 | 37.4 | 48.0 | 65.0 | Not run | 86.1 | |
| GLiDE 9B | Open · 7.9B | Not applicable | 61.8EN 82.9 Logic 34.8 Math 47.4 AC 82.2 | 75.2 | 25.1 | 82.9 | Not run | 34.8 | 47.4 | Not run | 82.2 | |
| GLiNER2.5 Multi | Open · 0.3B | Not applicable | Not applicable | −3.5 | −3.0 | −4.7 | Not run | Not run | Not run | Not run | Not run | |
| GLiNER2.5-multi-Decide fine-tune | Open · 0.3B | Not applicable | Not applicable | 1.3 | −0.2 | 1.5 | Not run | Not run | Not run | Not run | Not run | |
| GPT-6 Luna low · chat | Not applicable | Not applicable | 84.2 | 50.2 | 82.9 | Not run | 47.4 | 92.1 | Not run | Not run | ||
| GPT-6 Luna Decisions API | Not applicable | Not applicable | 81.9 | 42.9 | 80.1 | 15.8 | Not run | Not run | 64.2 | Not run | ||
| GPT-6.1 Terra high · chat | Not applicable | Not applicable | 95.2 | 91.8 | 95.2 | Not run | 92.4 | 99.0 | Not run | Not run | ||
| Gemini 3.8 Flash high · chat | Not applicable | Not applicable | 96.4 | 98.1 | 95.0 | Not run | Not run | Not run | 87.3 | Not run | ||
| Gemini 4 Argon low · chat | Not applicable | Not applicable | 96.2 | 95.1 | 95.0 | Not run | Not run | Not run | 89.3 | Not run | ||
| Gemini 4 Carbon high · chat | Not applicable | Not applicable | 97.0 | 98.5 | 92.9 | Not run | 95.9 | 99.5 | Not run | Not run | ||
| Gemini 4 Hydrogen high · chat | Not applicable | 93.5EN 95.9 Logic 95.1 Math 99.5 AC 83.7 | 96.6 | 98.1 | 95.9 | Not run | 95.1 | 99.5 | Not run | 83.7 | ||
| Gemma 4 12B Unified default reasoning · chat | Open · 12B | Not applicable | 85.9EN 83.5 Logic 84.7 Math 98.6 AC 77.0 | Not run | Not run | 83.5 | Not run | 84.7 | 98.6 | Not run | 77.0 | |
| Gemma 4 12B Unified thinking on · chat | Open · 12B | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | Not run | 67.7 | |
| Gemma 4 E2B reasoning on · chat | Open · 5.1B | Not applicable | 75.9EN 70.3 Logic 66.1 Math 96.0 AC 71.1 | Not run | Not run | 70.3 | Not run | 66.1 | 96.0 | Not run | 71.1 | |
| Gemma 4 E2B default reasoning · chat | Open · 5.1B | Not applicable | Not applicable | Not run | Not run | 68.2 | Not run | Not run | Not run | Not run | 55.2 | |
| Gemma 4 E2B thinking off · chat | Open · 5.1B | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | Not run | 54.8 | |
| Gemma 4 E4B default reasoning · chat | Open · 8.0B | Not applicable | 83.0EN 78.1 Logic 79.0 Math 97.6 AC 77.2 | 72.0 | 36.3 | 78.1 | 28.6 | 79.0 | 97.6 | Not run | 77.2 | |
| Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat | Open · 8.0B | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | Not run | 75.1 | |
| Granite-4.2-3B reasoning on · chat | Open · 3.0B | Not applicable | 83.1EN 68.0 Logic 89.6 Math 99.0 AC 75.9 | Not run | Not run | 68.0 | Not run | 89.6 | 99.0 | Not run | 75.9 | |
| Grok 4.7 high · chat | Not applicable | Not applicable | 93.7 | 89.7 | 91.8 | Not run | Not run | 99.4 | 84.0 | Not run | ||
| Inkling 2 Small high · chat | 594B MoE · 17B active | Not applicable | Not applicable | Not run | Not run | Not run | Not run | 94.6 | 99.4 | Not run | 86.2 | |
| Inkling-Small high · chat · Jul 2026 | Open · 276B MoE · 12B active | Not applicable | Not applicable | 84.0 | 71.1 | 85.4 | Not run | 94.2 | 99.5 | Not run | Not run | |
| Jeffan-9B-1211 | Open · 9.0B | Not applicable | 63.1EN 84.5 Logic 35.2 Math 49.8 AC 83.0 | 78.7 | 32.5 | 84.5 | Not run | 35.2 | 49.8 | Not run | 83.0 | |
| Karpo Decision 31B | Open · 31B | Not applicable | 77.0EN 90.2 Logic 50.9 Math 76.9 AC 90.0 | 91.8 | 63.9 | 90.2 | Not run | 50.9 | 76.9 | Not run | 90.0 | |
| Kimi K3.1 Pro low · chat | Open · 2.8T MoE · 104B active | Not applicable | Not applicable | 92.2 | 72.6 | 90.4 | Not run | Not run | Not run | 76.2 | Not run | |
| KnowLine-12B-Gen1 | Open · 12B | Not applicable | 65.1EN 83.3 Logic 38.2 Math 57.8 AC 80.9 | 82.5 | 40.6 | 83.3 | Not run | 38.2 | 57.8 | Not run | 80.9 | |
| KnowLine-31B-Gen2 | Open · 31B | Not applicable | 75.5EN 90.4 Logic 48.7 Math 75.3 AC 87.7 | 91.8 | 70.0 | 90.4 | Not run | 48.7 | 75.3 | Not run | 87.7 | |
| KnowLine-9B-Gen1 | Open · 9.0B | Not applicable | 59.7EN 82.4 Logic 30.3 Math 49.7 AC 76.1 | 74.8 | 29.7 | 82.4 | Not run | 30.3 | 49.7 | Not run | 76.1 | |
| KnowLine-9B-Gen2 | Open · 9.0B | Not applicable | 59.2EN 81.5 Logic 29.9 Math 50.1 AC 75.3 | 74.5 | 27.6 | 81.5 | Not run | 29.9 | 50.1 | Not run | 75.3 | |
| LFM2.5-2.6B default reasoning · chat | Open · 2.7B | Not applicable | Not applicable | 31.8 | 11.2 | 67.3 | Not run | 69.2 | 92.5 | Not run | Not run | |
| LFM2.5-24B-A2B default reasoning · chat | Open · 24B MoE · 2.5B active | Not applicable | Not applicable | Not run | Not run | 78.3 | Not run | 83.3 | 98.8 | 60.6 | Not run | |
| LFM2.5-8B-A1B default reasoning · chat | Open · 8.3B MoE · 1.5B active | Not applicable | 66.6EN 63.0 Logic 50.2 Math 95.3 AC 57.7 | 7.0 | −1.2 | 63.0 | Not run | 50.2 | 95.3 | Not run | 57.7 | |
| LFM3-1.7B reasoning on · chat | Open · 1.7B | Not applicable | 80.3EN 64.7 Logic 81.9 Math 98.7 AC 75.8 | 30.3 | 9.2 | 64.7 | Not run | 81.9 | 98.7 | Not run | 75.8 | |
| LFM3-1.7B reasoning off · chat | Open · 1.7B | Not applicable | Not applicable | 21.5 | 5.7 | 63.9 | Not run | Not run | Not run | Not run | 74.9 | |
| Laya | Open · 0.4B | Not applicable | Not applicable | 1.1 | −2.2 | 3.6 | Not run | Not run | Not run | Not run | Not run | |
| Laya Multilingual | Open · 0.3B | Not applicable | Not applicable | −3.7 | −2.2 | −5.8 | Not run | Not run | Not run | Not run | Not run | |
| Logits Sigma-1 Duo blend | Open · 57B | Not applicable | 72.5EN 88.8 Logic 47.2 Math 69.0 AC 85.1 | 89.9 | 59.8 | 88.8 | Not run | 47.2 | 69.0 | Not run | 85.1 | |
| Metask-Jev-4B LoRA, merged | Open · 4.5B | Not applicable | 43.1EN 71.4 Logic 15.1 Math 23.9 AC 61.7 | 53.5 | 14.2 | 71.4 | Not run | 15.1 | 23.9 | Not run | 61.7 | |
| MiMo-V2.6-Flash high · chat | Open · 309B MoE · 15B active | Not applicable | Not applicable | 83.2 | 57.4 | 86.3 | Not run | 93.9 | 99.3 | Not run | Not run | |
| Microsoft-Decision-1 | Not applicable | 61.5EN 82.2 Logic 30.6 Math 49.2 AC 83.9 | 66.8 | 18.7 | 82.2 | Not run | 30.6 | 49.2 | Not run | 83.9 | ||
| MiniCPM5-2B reasoning off · chat | Open · 2.5B | Not applicable | 84.3EN 72.7 Logic 86.1 Math 99.0 AC 79.2 | −16.8 | −18.5 | 72.7 | Not run | 86.1 | 99.0 | Not run | 79.2 | |
| MiniCPM5-2B reasoning off · chat | Open · 2.5B | Not applicable | Not applicable | Not run | Not run | 59.7 | Not run | Not run | Not run | Not run | 67.0 | |
| MiniMax-M3.1-Flash high · chat | Open · 428B MoE · 23B active | Not applicable | Not applicable | 78.7 | 49.3 | 80.6 | Not run | 92.4 | 97.8 | Not run | Not run | |
| OneJev-27B full fine-tune | Open · 27B | Not applicable | Not applicable | 81.5 | 35.9 | 82.9 | 26.6 | 25.8 | 42.2 | Not run | Not run | |
| Perplexity Decider 27B v1 | Open · 27B | Not applicable | Not applicable | 82.5 | 36.9 | 85.6 | Not run | Not run | Not run | Not run | Not run | |
| Perplexity Decider 27B v1.1setups differ | Open · 27B | Not applicable | Not applicable | 86.3 | 46.8 | Not run | Not run | Not run | Not run | 68.0 | Not run | |
| Perplexity Decider 27B v1.2 API · with web | Open · 27B | Not applicable | Not applicable | 86.1 | 50.2 | 88.1 | Not run | 40.7 | 61.6 | Not run | Not run | |
| Perplexity Decider 27B v1.3 | Open · 27B | Not applicable | 70.6EN 87.0 Logic 43.4 Math 65.6 AC 86.2 | 86.3 | 49.6 | 87.0 | Not run | 43.4 | 65.6 | Not run | 86.2 | |
| Perplexity Decider Flash v1.3 | Open · 4.0B | Not applicable | 58.6EN 79.7 Logic 30.9 Math 44.9 AC 78.8 | 68.0 | 24.0 | 79.7 | Not run | 30.9 | 44.9 | Not run | 78.8 | |
| Qwen3-14B reasoning off · chat | Open · 15B | Not applicable | Not applicable | 65.1 | 18.1 | 74.6 | Not run | 56.7 | 94.7 | Not run | Not run | |
| Qwen3.5-2B reasoning off · chat | Open · 2.0B | Not applicable | 72.4EN 69.8 Logic 56.0 Math 93.3 AC 70.3 | 17.9 | 2.0 | 69.8 | Not run | 56.0 | 93.3 | Not run | 70.3 | |
| Qwen3.5-2B reasoning off · chat | Open · 2.3B | Not applicable | Not applicable | Not run | Not run | 59.1 | Not run | Not run | Not run | Not run | 50.2 | |
| Qwen3.5-397B-A17B high · chat | Open · 397B MoE · 17B active | Not applicable | Not applicable | 88.8 | 74.0 | 89.3 | Not run | 93.9 | Not run | Not run | Not run | |
| Qwen3.5-4B reasoning on · chat | Open · 4.0B | Not applicable | 84.9EN 83.8 Logic 90.4 Math 99.3 AC 66.2 | 73.9 | 37.9 | 83.8 | Not run | 90.4 | 99.3 | Not run | 66.2 | |
| Qwen3.5-4B reasoning off · chat | Open · 4.0B | Not applicable | Not applicable | 60.6 | 20.9 | 75.5 | Not run | Not run | Not run | Not run | 64.6 | |
| Qwen4-Max high · chat | Not applicable | Not applicable | 93.3 | 88.1 | 93.1 | Not run | 95.3 | 99.7 | Not run | Not run | ||
| Raon-1 nano reasoning off · chat | Open · 3.1B | Not applicable | 56.5EN 42.2 Logic 56.3 Math 86.9 AC 40.6 | Not run | Not run | 42.2 | Not run | 56.3 | 86.9 | Not run | 40.6 | |
| StartLux-Decision-v1.1-9B-Instant | Open · 9.0B | Not applicable | 60.9EN 83.5 Logic 33.2 Math 48.0 AC 78.8 | 73.5 | 24.4 | 83.5 | Not run | 33.2 | 48.0 | Not run | 78.8 | |
| Strands Decider 2B LoRA + head | Open · 2.0B | Not applicable | 26.5EN 31.9 Logic 3.0 Math 10.8 AC 60.3 | 21.1 | 4.3 | 31.9 | Not run | 3.0 | 10.8 | Not run | 60.3 | |
| SubQ Snap Preview | Not applicable | Not applicable | 84.6 | 44.6 | 82.4 | Not run | Not run | Not run | 63.8 | Not run | ||
| Valen-Decision-1.5-9B | Open · 9.0B | Not applicable | 57.1EN 79.7 Logic 30.4 Math 44.2 AC 74.3 | 68.8 | 18.8 | 79.7 | Not run | 30.4 | 44.2 | Not run | 74.3 | |
| Verdin-4B-v1.1 LoRA, merged | Open · 4.0B | Not applicable | 54.3EN 77.8 Logic 24.7 Math 39.4 AC 75.4 | 64.0 | 20.5 | 77.8 | Not run | 24.7 | 39.4 | Not run | 75.4 | |
| celeris-1.1-decision | Not applicable | Not applicable | Not run | Not run | 79.7 | Not run | 34.1 | 62.3 | Not run | Not run | ||
| d1 | Not applicable | 58.3EN 83.5 Logic 26.0 Math 44.5 AC 79.2 | 78.5 | 32.6 | 83.5 | Not run | 26.0 | 44.5 | Not run | 79.2 | ||
| d1-Omni-2B NVIDIA RTX PRO 6000 Blackwell | Open · 2.2B | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | 58.9 | Not run | |
| d1.1 | Not applicable | 64.2EN 85.4 Logic 32.6 Math 56.5 AC 82.5 | 80.8 | 39.6 | 85.4 | Not run | 32.6 | 56.5 | Not run | 82.5 | ||
| d1.2 | Not applicable | Not applicable | 82.5 | 39.3 | 86.3 | 33.7 | 36.4 | 56.9 | Not run | Not run | ||
| d1.2-ultra | Not applicable | 69.0EN 87.9 Logic 39.4 Math 62.3 AC 86.4 | 87.2 | 48.6 | 87.9 | 31.7 | 39.4 | 62.3 | Not run | 86.4 | ||
| d3 fine-tune | Open · 26B | Not applicable | Not applicable | 86.5 | 48.4 | 87.2 | 35.6 | 43.1 | 63.0 | Not run | Not run | |
| d3-ultra fine-tune | Open · 26B | Not applicable | Not applicable | 87.2 | 48.0 | 86.7 | 31.8 | 43.2 | 64.4 | Not run | Not run | |
| vera-medium-omni | Open · 4.2B | Not applicable | 59.8EN 79.4 Logic 30.1 Math 46.0 AC 83.7 | 69.3 | 22.3 | 79.4 | Not run | 30.1 | 46.0 | Not run | 83.7 |
Perplexity Decider 27B v1.2 · DGX Spark, different setups of one model: Logic and Math: a hosted API; the other benches: our hardware.
Cygnet, different setups of one model: Logic and Math: a hosted API; the other benches: our hardware.
DeepSeek-V4.1-Flash · reasoning on · chat · 1017, different setups of one model: Logic and Math: a hosted API; the other benches: our hardware.
GLM-5.3-Flash · reasoning on · chat, different setups of one model: Logic and Math: a hosted API; the other benches: our hardware.
Perplexity Decider 27B v1.1, different setups of one model: Backgammon: our hardware; the other benches: a hosted API.
How it's scored
- Every bench on one scale: 0 = random guessing, 100 = perfect; higher is better ↑.
- Turkish Reading (656 questions) and Turkish Grammar (1,542), 5 options each, English Reading (547 of the reading questions translated into English, 5 options) and Logic and Math: accuracy corrected for chance, (accuracy − 1/k) / (1 − 1/k) × 100, k the number of options. Vision: the same on Reading's 763 questions that depend on the printed page, read from the page image alone.
- Backgammon: 0 = the error rate of a player that picks a random legal move, 100 = no error.
- Answer Check (2,069 pairs of a machine-written English summary, a question and its reference answer; 3 labels: the summary states that answer, a conflicting one or none): the same correction with k = 3; its reference labels come from two model judges, not people (how it was built).
- Combined = the plain mean of Reading · Turkish (all 2,198 Turkish questions), Logic & Math (pooled) and Backgammon, equal weights, as before the splits, only for a model that ran all three, one model on every bench (the rule).
- Decision Index = the plain mean of English Reading, Logic, Math and Answer Check, equal weights, only for a model on all four, one model on every bench, as for Combined (the rule).
- Each bar is a model's score in its maker's colour, the maker's mark before its name; the small tag beside the name says how the setup ran where its file gives one (setting · route · snapshot), then the training method where our records state it. A bar, like a dot on the charts, opens the model's page. Jev 1.14 is the reference bar: TypeSafe's hosted Jev 1.14, the model the index's entries reproduce: a reference bar, with no index score of its own here.
- Class: the Reading site's Leaderboard's, by total parameters, else by list price (the classes).
- Learning from experience ("did it improve?", points gained, higher is better ↑) is a column of its own, outside the combined score.
- Lower is better ↓ for calibration error (ECE, in points: how far the stated probabilities are from the hit rate) and latency. Close scores read tied, never with an interval: the rule.
- The table under the chart: Not run, the model is not on that bench; Not applicable, the score does not apply (Combined without all three groups), the reason on hover or focus. Rank is the place by the chosen score among the models in view, 1, 2, 3 without repeats; models with the same score keep the table's order, which claims no difference (no bench's file carries an all-pairs test, so no tiers). A column head sorts by it; again, the other way.
- Full results adds under each score its calibration error (ECE, points), learning from experience and latency: Reading · Turkish requests on one pinned machine (spark2, a DGX Spark (GB10)), one request at a time, median and 95th percentile.
Methodology: every rule, the sources and how models are matched across benches.
Head to head
| Bench | KnowLine-31B-Gen1.5 | REAX 1.3 | Difference |
|---|---|---|---|
| Combined ↑ | 73.2 | 71.3 | KnowLine-31B-Gen1.5 ahead by 1.8 |
| Decision Index ↑ | 75.3 | 74.8 | tied (0.5) |
| Turkish Reading ↑ | 91.0 | 90.5 | tied (0.6) |
| Turkish Grammar ↑ | 71.4 | 64.7 | KnowLine-31B-Gen1.5 ahead by 6.6 |
| English Reading ↑ | 92.0 | 90.6 | tied (1.4) |
| Vision ↑ | 37.3 | Not run | only KnowLine-31B-Gen1.5 ran it |
| Logic ↑ | 48.8 | 49.1 | tied (−0.3) |
| Math ↑ | 71.7 | 72.5 | tied (−0.8) |
| Backgammon ↑ | 78.3 | 77.1 | tied (1.3) |
| Answer Check ↑ | 88.8 | 87.0 | tied (1.9) |
Full results
| Model | Size | Combined ↑ | Decision Index ↑ | Turkish Reading ↑ | Turkish Grammar ↑ | English Reading ↑ | Vision ↑ | Logic ↑ | Math ↑ | Backgammon ↑ | Answer Check ↑ | Learning from experience ↑points gained | Latency ↓median, p95 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| KnowLine-31B-Gen1.5 LoRA, mergedPelaAI | 31.3B | 73.2TR 77.3 L&M 63.9 BG 78.3 | 75.3EN 92.0 Logic 48.8 Math 71.7 AC 88.8 | 91.0ECE 3.4 ↓ | 71.4ECE 3.9 ↓ | 92.0ECE 5.0 ↓ | 37.3ECE 6.1 ↓ | 48.8ECE 2.6 ↓ | 71.7ECE 3.6 ↓ | 78.3 | 88.8ECE 5.7 ↓ | – | 1,051 msp95 1,533 ms |
| REAX 1.3 blendREAX | 57.4B | 71.3TR 72.4 L&M 64.5 BG 77.1 | 74.8EN 90.6 Logic 49.1 Math 72.5 AC 87.0 | 90.5ECE 2.1 ↓ | 64.7ECE 5.1 ↓ | 90.6ECE 3.6 ↓ | Not run | 49.1ECE 3.7 ↓ | 72.5ECE 7.1 ↓ | 77.1 | 87.0ECE 4.8 ↓ | – | not timed yet |
| Sieve 31B v3 full fine-tuneSthānika AI | 31.3B | 69.4TR 67.7 L&M 63.2 BG 77.3 | 73.8EN 88.3 Logic 47.3 Math 71.4 AC 88.1 | 89.7ECE 3.5 ↓ | 58.3ECE 3.3 ↓ | 88.3ECE 4.3 ↓ | Not run | 47.3ECE 2.1 ↓ | 71.4ECE 4.3 ↓ | 77.3 | 88.1ECE 7.0 ↓ | – | not timed yet |
| d1-ultraLiquid AI | Not disclosed | 68.3TR 71.3 L&M 59.4 BG 74.3 | 71.1EN 88.3 Logic 43.7 Math 67.6 AC 84.7 | 90.1ECE 3.6 ↓ | 63.4ECE 4.2 ↓ | 88.3ECE 4.4 ↓ | 37.6ECE 3.5 ↓ | 43.7ECE 3.2 ↓ | 67.6ECE 2.3 ↓ | 74.3 | 84.7ECE 11.4 ↓ | – | not timed yet |
| Turion Jev 31B LoRA, mergedTurion.AI | 31.3B | 68.2TR 75.1 L&M 54.1 BG 75.5 | 69.1EN 88.3 Logic 40.3 Math 61.3 AC 86.4 | 90.9ECE 9.5 ↓ | 68.5ECE 12.0 ↓ | 88.3ECE 11.6 ↓ | Not run | 40.3ECE 2.9 ↓ | 61.3ECE 4.7 ↓ | 75.5 | 86.4ECE 6.7 ↓ | – | not timed yet |
| Gemini Decision 1 20261111Google | Not disclosed | 68.0TR 71.9 L&M 55.1 BG 76.9 | 66.9EN 88.8 Logic 37.6 Math 64.2 AC 77.0 | 90.1ECE 2.8 ↓ | 64.2ECE 2.9 ↓ | 88.8ECE 4.5 ↓ | 58.1ECE 5.2 ↓ | 37.6ECE 3.1 ↓ | 64.2ECE 2.5 ↓ | 76.9 | 77.0ECE 3.9 ↓ | – | not timed yet |
| Mammoth-1 post-trainedBase Labs | 27.4B | 64.8TR 58.9 L&M 59.1 BG 76.4 | Not applicable | 88.0ECE 2.0 ↓ | 46.5ECE 7.9 ↓ | 87.7ECE 2.4 ↓ | Not run | 43.1ECE 5.3 ↓ | 67.5ECE 3.0 ↓ | 76.4 | Not run | – | not timed yet |
| Drex Max fine-tuneNace.AI | 26.1B | 63.4TR 59.7 L&M 55.5 BG 74.9 | Not applicable | 87.0ECE 3.4 ↓ | 48.0ECE 8.0 ↓ | 87.0ECE 2.6 ↓ | 33.5ECE 9.3 ↓ | 41.3ECE 16.2 ↓ | 62.9ECE 11.8 ↓ | 74.9 | Not run | – | not timed yet |
| Drex Max v1.1Nace.AI | Not disclosed | 63.2TR 59.2 L&M 56.1 BG 74.5 | Not applicable | 85.7ECE 3.4 ↓ | 48.0ECE 2.7 ↓ | 85.6ECE 3.8 ↓ | 31.4ECE 5.2 ↓ | 43.4ECE 6.1 ↓ | 62.6ECE 5.7 ↓ | 74.5 | Not run | – | not timed yet |
| GLiDE v1.2Fastino | 26.1B | 62.6TR 60.6 L&M 56.1 BG 71.1 | 69.8EN 87.9 Logic 44.0 Math 62.3 AC 84.9 | 87.0ECE 3.5 ↓ | 49.3ECE 3.0 ↓ | 87.9ECE 2.7 ↓ | Not run | 44.0ECE 5.9 ↓ | 62.3ECE 5.2 ↓ | 71.1 | 84.9ECE 4.7 ↓ | – | not timed yet |
| Perplexity Decider 27B v1.2 DGX SparkPerplexity | 26.1B | 61.2TR 58.8 L&M 55.2 BG 69.5 | Not applicable | 86.5ECE 3.1 ↓ | 47.1ECE 6.2 ↓ | 85.8ECE 3.7 ↓ | Not run | 41.7ECE 3.4 ↓ | 62.2ECE 3.2 ↓ | 69.5 | Not run | – | not timed yet |
| SubQ Snap 1.1Subquadratic | Not disclosed | 58.5TR 54.8 L&M 53.8 BG 66.8 | 67.1EN 82.4 Logic 40.4 Math 60.7 AC 84.9 | 83.8ECE 3.6 ↓ | 42.5ECE 14.5 ↓ | 82.4ECE 3.2 ↓ | 32.2ECE 11.7 ↓ | 40.4ECE 12.9 ↓ | 60.7ECE 9.5 ↓ | 66.8 | 84.9ECE 2.3 ↓ | – | not timed yet |
| celeris-1-decisionCeleris | Not disclosed | 58.0TR 50.6 L&M 52.8 BG 70.6 | Not applicable | 81.5ECE 6.2 ↓ | 37.4ECE 5.3 ↓ | 80.1ECE 5.2 ↓ | Not run | 34.1ECE 10.7 ↓ | 62.5ECE 4.3 ↓ | 70.6 | Not run | – | not timed yet |
| Jev 1.14TypeSafe | Not disclosed | 57.1TR 49.0 L&M 49.2 BG 73.1 | 64.5EN 82.6 Logic 35.2 Math 56.4 AC 83.6 | 81.3ECE 2.2 ↓ | 35.2ECE 4.0 ↓ | 82.6ECE 2.2 ↓ | 29.1ECE 3.7 ↓ | 35.2ECE 3.3 ↓ | 56.4ECE 2.8 ↓ | 73.1 | 83.6ECE 2.8 ↓ | – | not timed yet |
| Jev 1.13TypeSafe | Not disclosed | 55.3TR 46.9 L&M 47.5 BG 71.6 | 63.3EN 82.4 Logic 32.9 Math 55.1 AC 83.0 | 79.6ECE 3.8 ↓ | 33.0ECE 6.5 ↓ | 82.4ECE 3.1 ↓ | Not run | 32.9ECE 5.9 ↓ | 55.1ECE 2.6 ↓ | 71.6 | 83.0ECE 1.9 ↓ | – | not timed yet |
| Drex 2.0Nace.AI | 35.1B3B active | 51.4TR 44.7 L&M 41.4 BG 68.3 | Not applicable | 76.8ECE 10.1 ↓ | 31.0ECE 2.0 ↓ | 82.4ECE 6.9 ↓ | Not run | 27.3ECE 4.3 ↓ | 48.7ECE 1.9 ↓ | 68.3 | Not run | – | not timed yet |
| Clef post-trainedCloudflare | 27.5B | 50.0TR 46.6 L&M 34.5 BG 69.0 | Not applicable | 81.3ECE 4.0 ↓ | 31.8ECE 15.1 ↓ | 83.3ECE 3.1 ↓ | Not run | 26.0ECE 15.7 ↓ | 38.9ECE 11.0 ↓ | 69.0 | Not run | – | not timed yet |
| Elephant-1 post-trainedBase Labs | 9.4B | 49.7TR 40.5 L&M 45.3 BG 63.2 | 60.8EN 81.7 Logic 33.0 Math 51.7 AC 76.9 | 75.6ECE 4.3 ↓ | 25.5ECE 15.4 ↓ | 81.7ECE 2.7 ↓ | Not run | 33.0ECE 8.6 ↓ | 51.7ECE 4.3 ↓ | 63.2 | 76.9ECE 3.7 ↓ | – | not timed yet |
| Mercury DecideInception | Not disclosed | 48.8TR 47.9 L&M 38.3 BG 60.4 | Not applicable | 80.6ECE 7.5 ↓ | 33.9ECE 19.0 ↓ | 82.2ECE 6.6 ↓ | Not run | 31.3ECE 15.4 ↓ | 41.9ECE 15.8 ↓ | 60.4 | Not run | – | not timed yet |
| Cerno 9B v1.2 full fine-tuneApertix AI | 8.9B | 47.5TR 43.1 L&M 39.6 BG 60.0 | 59.6EN 82.4 Logic 29.1 Math 45.0 AC 81.9 | 74.7ECE 3.0 ↓ | 29.6ECE 14.2 ↓ | 82.4ECE 4.2 ↓ | Not run | 29.1ECE 17.0 ↓ | 45.0ECE 15.7 ↓ | 60.0 | 81.9ECE 2.9 ↓ | – | not timed yet |
| StartLux-Decision-v1.1-4B-Instant fine-tuneStartLux | 4.7B | 47.4TR 34.8 L&M 40.5 BG 66.8 | 59.3EN 78.5 Logic 32.3 Math 44.8 AC 81.7 | 67.6ECE 2.6 ↓ | 20.9ECE 10.3 ↓ | 78.5ECE 3.2 ↓ | Not run | 32.3ECE 3.3 ↓ | 44.8ECE 6.7 ↓ | 66.8 | 81.7ECE 8.3 ↓ | – | not timed yet |
| Cygnet unchanged baseBlockbrain Labs | 12B | 47.0TR 44.6 L&M 28.3 BG 68.2 | 51.8EN 79.9 Logic 24.1 Math 30.4 AC 72.9 | 78.5ECE 4.1 ↓ | 30.2ECE 18.9 ↓ | 79.9ECE 4.3 ↓ | Not run | 24.1ECE 11.2 ↓ | 30.4ECE 5.9 ↓ | 68.2 | 72.9ECE 6.5 ↓ | – | not timed yet |
| Winnow-12B LoRA, mergedEldanRing | 11.9B | 44.6TR 44.3 L&M 30.5 BG 58.9 | 53.7EN 79.9 Logic 24.7 Math 33.6 AC 76.6 | 77.5ECE 9.3 ↓ | 30.2ECE 24.1 ↓ | 79.9ECE 8.7 ↓ | Not run | 24.7ECE 14.4 ↓ | 33.6ECE 8.4 ↓ | 58.9 | 76.6ECE 8.3 ↓ | – | not timed yet |
| KnowLine 4B Gen5.5 LoRA, mergedPelaAI | 4.7B | 43.5TR 35.6 L&M 39.1 BG 55.8 | 57.4EN 76.0 Logic 27.9 Math 45.0 AC 80.6 | 68.6ECE 6.2 ↓ | 21.6ECE 16.6 ↓ | 76.0ECE 5.8 ↓ | 12.7ECE 11.0 ↓ | 27.9ECE 13.7 ↓ | 45.0ECE 8.6 ↓ | 55.8 | 80.6ECE 4.5 ↓ | – | 99 msp95 149 ms |
| Open-Jev-27B-v1.1 LoRA + headZefan Cai | 27.8B | 42.8TR 44.6 L&M 38.1 BG 45.9 | Not applicable | 78.7ECE 43.6 ↓ | 30.1ECE 16.4 ↓ | 81.7ECE 45.3 ↓ | Not run | 28.2ECE 12.9 ↓ | 43.2ECE 21.1 ↓ | 45.9 | Not run | – | not timed yet |
| Perplexity Decider Flash v1.2Perplexity | 4.5B | 41.4TR 34.7 L&M 38.9 BG 50.8 | 56.9EN 77.4 Logic 28.9 Math 44.1 AC 77.2 | 65.9ECE 3.7 ↓ | 21.4ECE 11.0 ↓ | 77.4ECE 4.1 ↓ | Not run | 28.9ECE 6.0 ↓ | 44.1ECE 10.4 ↓ | 50.8 | 77.2ECE 2.8 ↓ | – | not timed yet |
| StartLux-Decision-v1.1-2B-Instant fine-tuneStartLux | 2.3B | 32.1TR 17.8 L&M 19.9 BG 58.7 | 41.2EN 62.1 Logic 13.4 Math 23.3 AC 66.0 | 41.1ECE 3.3 ↓ | 7.9ECE 9.4 ↓ | 62.1ECE 8.2 ↓ | 4.3ECE 3.6 ↓ | 13.4ECE 5.3 ↓ | 23.3ECE 2.6 ↓ | 58.7 | 66.0ECE 7.2 ↓ | – | 40 msp95 48 ms |
| KnowLine-2B-Gen1 LoRA, mergedPelaAI | 2.3B | 31.2TR 21.0 L&M 17.6 BG 55.0 | 39.7EN 64.1 Logic 14.7 Math 19.1 AC 61.1 | 42.8ECE 1.9 ↓ | 11.7ECE 11.4 ↓ | 64.1ECE 5.1 ↓ | Not run | 14.7ECE 8.3 ↓ | 19.1ECE 9.4 ↓ | 55.0 | 61.1ECE 2.7 ↓ | – | 62 msp95 87 ms |
| d1-2BLiquid AI | 1.9B | 30.5TR 11.7 L&M 18.6 BG 61.1 | 35.3EN 57.0 Logic 13.3 Math 21.3 AC 49.6 | 26.1ECE 4.4 ↓ | 5.6ECE 6.7 ↓ | 57.0ECE 7.9 ↓ | 4.2ECE 1.1 ↓ | 13.3ECE 6.9 ↓ | 21.3ECE 5.6 ↓ | 61.1 | 49.6ECE 7.2 ↓ | – | not timed yet |
| GLiDE 2BFastino | 1.9B | 27.5TR 19.1 L&M 19.0 BG 44.4 | 41.1EN 62.5 Logic 15.3 Math 20.9 AC 65.6 | 39.0ECE 16.5 ↓ | 10.7ECE 22.5 ↓ | 62.5ECE 8.2 ↓ | Not run | 15.3ECE 16.2 ↓ | 20.9ECE 17.2 ↓ | 44.4 | 65.6ECE 4.0 ↓ | – | 34 msp95 47 ms |
| StartLux-Decision-v1.1-0.8B-Instant fine-tuneStartLux | 870M | 22.4TR 10.7 L&M 10.0 BG 46.6 | 29.1EN 49.5 Logic 5.9 Math 12.1 AC 48.9 | 18.8ECE 5.0 ↓ | 7.3ECE 4.6 ↓ | 49.5ECE 8.7 ↓ | 5.3ECE 1.7 ↓ | 5.9ECE 7.8 ↓ | 12.1ECE 4.8 ↓ | 46.6 | 48.9ECE 5.6 ↓ | – | 28 msp95 33 ms |
| KnowLine-0.8B-Gen1 LoRA, mergedPelaAI | 870M | 19.7TR 8.9 L&M 9.6 BG 40.6 | 26.4EN 48.6 Logic 8.3 Math 10.3 AC 38.5 | 22.8ECE 5.5 ↓ | 3.0ECE 10.8 ↓ | 48.6ECE 8.1 ↓ | 4.8ECE 1.5 ↓ | 8.3ECE 8.4 ↓ | 10.3ECE 10.5 ↓ | 40.6 | 38.5ECE 14.6 ↓ | – | 33 msp95 50 ms |
| GLiDE 0.8BFastino | 750M | 19.7TR 11.6 L&M 13.9 BG 33.6 | 34.5EN 50.9 Logic 11.4 Math 15.1 AC 60.8 | 28.0ECE 21.8 ↓ | 4.6ECE 26.1 ↓ | 50.9ECE 12.5 ↓ | Not run | 11.4ECE 14.0 ↓ | 15.1ECE 18.8 ↓ | 33.6 | 60.8ECE 3.8 ↓ | – | 22 msp95 31 ms |
| OneJev1.6-0.8B full fine-tuneOmniJev | 850M | 17.7TR 11.4 L&M 9.8 BG 31.9 | 29.6EN 49.3 Logic 8.8 Math 10.3 AC 50.0 | 24.5ECE 7.3 ↓ | 5.8ECE 10.3 ↓ | 49.3ECE 3.7 ↓ | 8.1ECE 4.2 ↓ | 8.8ECE 11.2 ↓ | 10.3ECE 10.0 ↓ | 31.9 | 50.0ECE 4.1 ↓ | – | 17 msp95 23 ms |
| GLiDE 0.4BFastino | 400M | 3.7TR 2.8 L&M 1.8 BG 6.5 | 24.6EN 49.3 Logic 4.0 Math 0.7 AC 44.4 | 5.7ECE 7.2 ↓ | 1.6ECE 8.3 ↓ | 49.3ECE 4.8 ↓ | Not run | 4.0ECE 8.8 ↓ | 0.7ECE 12.2 ↓ | 6.5 | 44.4ECE 7.7 ↓ | – | 16 msp95 24 ms |
| GLiClass v4 Edge fine-tuneKnowledgator | 440M | −0.8TR 4.2 L&M 3.7 BG −10.2 | 25.0EN 47.9 Logic 4.4 Math 3.3 AC 44.3 | 7.4ECE 11.9 ↓ | 2.8ECE 10.0 ↓ | 47.9ECE 3.8 ↓ | Not run | 4.4ECE 8.7 ↓ | 3.3ECE 7.0 ↓ | −10.2 | 44.3ECE 9.8 ↓ | – | 25 msp95 53 ms |
| Adjudex 9B v5newsbang | 8.4B | Not applicable | 62.1EN 82.2 Logic 33.0 Math 52.6 AC 80.4 | 76.0ECE 2.5 ↓ | 26.7ECE 12.2 ↓ | 82.2ECE 3.4 ↓ | Not run | 33.0ECE 7.2 ↓ | 52.6ECE 4.8 ↓ | Not run | 80.4ECE 4.9 ↓ | – | not timed yet |
| Chinese Decision Pointer 4B LoRA, mergedquincyqiang | 4.2B | Not applicable | 54.3EN 75.5 Logic 25.4 Math 38.0 AC 78.1 | 63.6ECE 3.3 ↓ | 20.5ECE 15.4 ↓ | 75.5ECE 3.5 ↓ | Not run | 25.4ECE 15.5 ↓ | 38.0ECE 13.7 ↓ | Not run | 78.1ECE 6.7 ↓ | – | not timed yet |
| Clef-Flash post-trainedCloudflare | 9.5B | Not applicable | 52.1EN 77.8 Logic 23.7 Math 37.2 AC 69.8 | 66.3ECE 5.6 ↓ | 17.6ECE 20.3 ↓ | 77.8ECE 4.0 ↓ | Not run | 23.7ECE 11.5 ↓ | 37.2ECE 2.7 ↓ | Not run | 69.8ECE 5.9 ↓ | – | not timed yet |
| Copernicus-31BPoindexter Labs | 31.3B | Not applicable | 75.0EN 91.1 Logic 48.3 Math 71.7 AC 89.0 | 91.4ECE 6.3 ↓ | 71.2ECE 10.5 ↓ | 91.1ECE 7.8 ↓ | Not run | 48.3ECE 3.9 ↓ | 71.7ECE 6.5 ↓ | Not run | 89.0ECE 11.0 ↓ | – | not timed yet |
| Decider 1meraGPT | Not disclosed | Not applicable | Not applicable | 18.3ECE 11.8 ↓ | 4.9ECE 15.4 ↓ | 44.7ECE 5.3 ↓ | Not run | Not run | Not run | Not run | 43.3ECE 5.9 ↓ | – | not timed yet |
| Dexy-Decision-9B-V2 fine-tunePatrick-18 | 8.4B | Not applicable | 60.2EN 79.4 Logic 31.7 Math 48.5 AC 81.1 | 74.5ECE 3.2 ↓ | 24.6ECE 12.7 ↓ | 79.4ECE 4.2 ↓ | Not run | 31.7ECE 14.9 ↓ | 48.5ECE 12.1 ↓ | Not run | 81.1ECE 2.5 ↓ | – | not timed yet |
| GLiDE APIFastino | Not disclosed | Not applicablenot one model | Not applicable | 84.8ECE 5.8 ↓ | 49.3ECE 4.1 ↓ | Not run | Not run | 43.1ECE 6.6 ↓ | 63.5ECE 5.0 ↓ | 71.9 | 85.1ECE 3.0 ↓ | – | not timed yet |
| GLiDE 27BFastino | 26.1B | Not applicable | 71.9EN 88.6 Logic 48.0 Math 65.0 AC 86.1 | 88.6ECE 2.4 ↓ | 53.4ECE 2.7 ↓ | 88.6ECE 1.7 ↓ | 37.4ECE 4.3 ↓ | 48.0ECE 4.4 ↓ | 65.0ECE 4.1 ↓ | Not run | 86.1ECE 3.6 ↓ | – | 542 msp95 739 ms |
| GLiDE 9BFastino | 7.9B | Not applicable | 61.8EN 82.9 Logic 34.8 Math 47.4 AC 82.2 | 75.2ECE 3.7 ↓ | 25.1ECE 12.0 ↓ | 82.9ECE 5.6 ↓ | Not run | 34.8ECE 6.7 ↓ | 47.4ECE 7.1 ↓ | Not run | 82.2ECE 7.2 ↓ | – | 207 msp95 259 ms |
| GLiNER2.5-multi-Decide fine-tuneFastino | 290M | Not applicable | Not applicable | 1.3ECE 24.1 ↓ | −0.2ECE 22.7 ↓ | 1.5ECE 27.2 ↓ | Not run | Not run | Not run | Not run | Not run | – | not timed yet |
| GPT-6 Luna Decisions APIOpenAI | Not disclosed | Not applicable | Not applicable | 81.9ECE 3.8 ↓ | 42.9ECE 3.0 ↓ | 80.1ECE 3.2 ↓ | 15.8ECE 5.7 ↓ | Not run | Not run | 64.2 | Not run | – | not timed yet |
| Jeffan-9B-1211tattrongvu | 8.9B | Not applicable | 63.1EN 84.5 Logic 35.2 Math 49.8 AC 83.0 | 78.7ECE 11.4 ↓ | 32.5ECE 3.0 ↓ | 84.5ECE 8.9 ↓ | Not run | 35.2ECE 4.0 ↓ | 49.8ECE 3.8 ↓ | Not run | 83.0ECE 3.3 ↓ | – | not timed yet |
| Karpo Decision 31BMachinePulse | 31.3B | Not applicable | 77.0EN 90.2 Logic 50.9 Math 76.9 AC 90.0 | 91.8ECE 1.2 ↓ | 63.9ECE 3.4 ↓ | 90.2ECE 2.4 ↓ | Not run | 50.9ECE 5.7 ↓ | 76.9ECE 1.2 ↓ | Not run | 90.0ECE 2.8 ↓ | – | not timed yet |
| KnowLine-12B-Gen1PelaAI | 12B | Not applicable | 65.1EN 83.3 Logic 38.2 Math 57.8 AC 80.9 | 82.5ECE 11.4 ↓ | 40.6ECE 4.3 ↓ | 83.3ECE 11.1 ↓ | Not run | 38.2ECE 4.8 ↓ | 57.8ECE 4.0 ↓ | Not run | 80.9ECE 6.8 ↓ | – | not timed yet |
| KnowLine-31B-Gen2PelaAI | 31.3B | Not applicable | 75.5EN 90.4 Logic 48.7 Math 75.3 AC 87.7 | 91.8ECE 6.3 ↓ | 70.0ECE 5.7 ↓ | 90.4ECE 7.7 ↓ | Not run | 48.7ECE 3.7 ↓ | 75.3ECE 3.6 ↓ | Not run | 87.7ECE 7.2 ↓ | – | not timed yet |
| KnowLine-9B-Gen1PelaAI | 9.7B | Not applicable | 59.7EN 82.4 Logic 30.3 Math 49.7 AC 76.1 | 74.8ECE 1.7 ↓ | 29.7ECE 6.8 ↓ | 82.4ECE 3.5 ↓ | Not run | 30.3ECE 3.5 ↓ | 49.7ECE 3.1 ↓ | Not run | 76.1ECE 3.6 ↓ | – | not timed yet |
| KnowLine-9B-Gen2PelaAI | 9.7B | Not applicable | 59.2EN 81.5 Logic 29.9 Math 50.1 AC 75.3 | 74.5ECE 8.2 ↓ | 27.6ECE 5.1 ↓ | 81.5ECE 8.7 ↓ | Not run | 29.9ECE 3.2 ↓ | 50.1ECE 4.8 ↓ | Not run | 75.3ECE 4.5 ↓ | – | not timed yet |
| LayaConvai Innovations | 420M | Not applicable | Not applicable | 1.1ECE 9.2 ↓ | −2.2ECE 14.9 ↓ | 3.6ECE 12.1 ↓ | Not run | Not run | Not run | Not run | Not run | – | not timed yet |
| Laya MultilingualConvai Innovations | 320M | Not applicable | Not applicable | −3.7ECE 33.1 ↓ | −2.2ECE 26.0 ↓ | −5.8ECE 44.4 ↓ | Not run | Not run | Not run | Not run | Not run | – | not timed yet |
| Logits Sigma-1 Duo blendLogits | 57.4B | Not applicable | 72.5EN 88.8 Logic 47.2 Math 69.0 AC 85.1 | 89.9ECE 5.3 ↓ | 59.8ECE 5.9 ↓ | 88.8ECE 8.0 ↓ | Not run | 47.2ECE 5.0 ↓ | 69.0ECE 7.5 ↓ | Not run | 85.1ECE 7.0 ↓ | – | not timed yet |
| Metask-Jev-4B LoRA, mergedMetask Lab | 4.5B | Not applicable | 43.1EN 71.4 Logic 15.1 Math 23.9 AC 61.7 | 53.5ECE 5.3 ↓ | 14.2ECE 6.5 ↓ | 71.4ECE 4.4 ↓ | Not run | 15.1ECE 2.5 ↓ | 23.9ECE 3.3 ↓ | Not run | 61.7ECE 6.8 ↓ | – | not timed yet |
| Microsoft-Decision-1Microsoft | Not disclosed | Not applicable | 61.5EN 82.2 Logic 30.6 Math 49.2 AC 83.9 | 66.8ECE 8.9 ↓ | 18.7ECE 24.4 ↓ | 82.2ECE 2.8 ↓ | Not run | 30.6ECE 8.0 ↓ | 49.2ECE 4.1 ↓ | Not run | 83.9ECE 2.4 ↓ | – | not timed yet |
| OneJev-27B full fine-tuneOmniJev | 27.4B | Not applicable | Not applicable | 81.5ECE 9.9 ↓ | 35.9ECE 3.7 ↓ | 82.9ECE 9.7 ↓ | 26.6ECE 4.9 ↓ | 25.8ECE 4.6 ↓ | 42.2ECE 3.6 ↓ | Not run | Not run | – | not timed yet |
| Perplexity Decider 27B v1Perplexity | 26.1B | Not applicable | Not applicable | 82.5ECE 5.0 ↓ | 36.9ECE 5.0 ↓ | 85.6ECE 2.6 ↓ | Not run | Not run | Not run | Not run | Not run | – | not timed yet |
| Perplexity Decider 27B v1.1Perplexity | 26.1B | Not applicable | Not applicable | 86.3ECE 3.0 ↓ | 46.8ECE 5.5 ↓ | Not run | Not run | Not run | Not run | 68.0 | Not run | – | not timed yet |
| Perplexity Decider 27B v1.2 API · with webPerplexity | 26.1B | Not applicable | Not applicable | 86.1ECE 2.7 ↓ | 50.2ECE 4.9 ↓ | 88.1ECE 3.6 ↓ | Not run | 40.7ECE 4.2 ↓ | 61.6ECE 3.7 ↓ | Not run | Not run | – | not timed yet |
| Perplexity Decider 27B v1.3Perplexity | 26.1B | Not applicable | 70.6EN 87.0 Logic 43.4 Math 65.6 AC 86.2 | 86.3ECE 2.4 ↓ | 49.6ECE 6.1 ↓ | 87.0ECE 3.4 ↓ | Not run | 43.4ECE 9.0 ↓ | 65.6ECE 4.5 ↓ | Not run | 86.2ECE 2.8 ↓ | – | not timed yet |
| Perplexity Decider Flash v1.3Perplexity | 4.5B | Not applicable | 58.6EN 79.7 Logic 30.9 Math 44.9 AC 78.8 | 68.0ECE 6.2 ↓ | 24.0ECE 15.3 ↓ | 79.7ECE 2.7 ↓ | Not run | 30.9ECE 10.1 ↓ | 44.9ECE 11.7 ↓ | Not run | 78.8ECE 4.2 ↓ | – | not timed yet |
| StartLux-Decision-v1.1-9B-InstantStartLux | 9.7B | Not applicable | 60.9EN 83.5 Logic 33.2 Math 48.0 AC 78.8 | 73.5ECE 3.6 ↓ | 24.4ECE 11.3 ↓ | 83.5ECE 6.6 ↓ | Not run | 33.2ECE 3.8 ↓ | 48.0ECE 2.7 ↓ | Not run | 78.8ECE 8.2 ↓ | – | not timed yet |
| Strands Decider 2B LoRA + headStrands Agents | 2.3B | Not applicable | 26.5EN 31.9 Logic 3.0 Math 10.8 AC 60.3 | 21.1ECE 10.2 ↓ | 4.3ECE 13.2 ↓ | 31.9ECE 10.5 ↓ | Not run | 3.0ECE 12.0 ↓ | 10.8ECE 8.8 ↓ | Not run | 60.3ECE 5.0 ↓ | – | not timed yet |
| SubQ Snap PreviewSubquadratic | Not disclosed | Not applicable | Not applicable | 84.6ECE 4.2 ↓ | 44.6ECE 6.0 ↓ | 82.4ECE 3.2 ↓ | Not run | Not run | Not run | 63.8 | Not run | – | not timed yet |
| Valen-Decision-1.5-9BValen-Team | 8.4B | Not applicable | 57.1EN 79.7 Logic 30.4 Math 44.2 AC 74.3 | 68.8ECE 9.7 ↓ | 18.8ECE 6.9 ↓ | 79.7ECE 8.4 ↓ | Not run | 30.4ECE 5.8 ↓ | 44.2ECE 4.1 ↓ | Not run | 74.3ECE 3.2 ↓ | – | not timed yet |
| Verdin-4B-v1.1 LoRA, mergedvictorbennett | 4.2B | Not applicable | 54.3EN 77.8 Logic 24.7 Math 39.4 AC 75.4 | 64.0ECE 3.9 ↓ | 20.5ECE 12.8 ↓ | 77.8ECE 3.5 ↓ | Not run | 24.7ECE 10.8 ↓ | 39.4ECE 5.7 ↓ | Not run | 75.4ECE 3.4 ↓ | – | not timed yet |
| celeris-1.1-decisionCeleris | Not disclosed | Not applicable | Not applicable | Not run | Not run | 79.7ECE 5.0 ↓ | Not run | 34.1ECE 9.8 ↓ | 62.3ECE 4.5 ↓ | Not run | Not run | – | not timed yet |
| d1Liquid AI | Not disclosed | Not applicable | 58.3EN 83.5 Logic 26.0 Math 44.5 AC 79.2 | 78.5ECE 2.8 ↓ | 32.6ECE 13.5 ↓ | 83.5ECE 2.6 ↓ | Not run | 26.0ECE 10.9 ↓ | 44.5ECE 3.3 ↓ | Not run | 79.2ECE 3.1 ↓ | – | not timed yet |
| d1-Omni-2B NVIDIA RTX PRO 6000 BlackwellLiquid AI | 2.2B | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | 58.9 | Not run | – | not timed yet |
| d1.1Liquid AI | Not disclosed | Not applicable | 64.2EN 85.4 Logic 32.6 Math 56.5 AC 82.5 | 80.8ECE 4.2 ↓ | 39.6ECE 2.2 ↓ | 85.4ECE 4.2 ↓ | Not run | 32.6ECE 4.4 ↓ | 56.5ECE 2.2 ↓ | Not run | 82.5ECE 8.7 ↓ | – | not timed yet |
| d1.2Liquid AI | Not disclosed | Not applicable | Not applicable | 82.5ECE 6.0 ↓ | 39.3ECE 2.5 ↓ | 86.3ECE 8.3 ↓ | 33.7ECE 5.6 ↓ | 36.4ECE 2.6 ↓ | 56.9ECE 3.2 ↓ | Not run | Not run | – | not timed yet |
| d1.2-ultraLiquid AI | Not disclosed | Not applicable | 69.0EN 87.9 Logic 39.4 Math 62.3 AC 86.4 | 87.2ECE 5.6 ↓ | 48.6ECE 3.1 ↓ | 87.9ECE 8.5 ↓ | 31.7ECE 2.8 ↓ | 39.4ECE 3.0 ↓ | 62.3ECE 4.6 ↓ | Not run | 86.4ECE 8.8 ↓ | – | not timed yet |
| d3 fine-tunevLLM Semantic Router | 26.1B | Not applicable | Not applicable | 86.5ECE 3.5 ↓ | 48.4ECE 2.4 ↓ | 87.2ECE 3.6 ↓ | 35.6ECE 3.7 ↓ | 43.1ECE 4.6 ↓ | 63.0ECE 4.5 ↓ | Not run | Not run | – | not timed yet |
| d3-ultra fine-tunevLLM Semantic Router | 26.1B | Not applicable | Not applicable | 87.2ECE 3.8 ↓ | 48.0ECE 16.3 ↓ | 86.7ECE 2.8 ↓ | 31.8ECE 18.8 ↓ | 43.2ECE 15.1 ↓ | 64.4ECE 8.6 ↓ | Not run | Not run | – | not timed yet |
| vera-medium-omniscar-ai | 4.2B | Not applicable | 59.8EN 79.4 Logic 30.1 Math 46.0 AC 83.7 | 69.3ECE 9.1 ↓ | 22.3ECE 8.0 ↓ | 79.4ECE 9.9 ↓ | Not run | 30.1ECE 4.7 ↓ | 46.0ECE 3.0 ↓ | Not run | 83.7ECE 12.4 ↓ | – | not timed yet |
| Model | Size | Combined ↑ | Decision Index ↑ | Turkish Reading ↑ | Turkish Grammar ↑ | English Reading ↑ | Vision ↑ | Logic ↑ | Math ↑ | Backgammon ↑ | Answer Check ↑ | Learning from experience ↑points gained | Latency ↓median, p95 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6 Astra high · chatOpenAI | Not disclosed | 96.1TR 97.4 L&M 98.3 BG 92.5 | Not applicable | 96.4ECE 1.0 ↓ | 97.9ECE 0.7 ↓ | 95.2ECE 1.8 ↓ | Not run | 95.9 | 99.5 | 92.5 | Not run | – | not timed yet |
| GPT-6.1 Sol high · chatOpenAI | Not disclosed | 95.7TR 97.2 L&M 98.2 BG 91.6 | Not applicable | 95.6ECE 1.8 ↓ | 97.9ECE 0.6 ↓ | 94.5ECE 2.4 ↓ | Not run | 95.7 | 99.5 | 91.6 | Not run | – | not timed yet |
| Claude Fable 5.1 high · chatAnthropic | Not disclosed | 93.7TR 95.2 L&M 97.7 BG 88.1 | Not applicable | 94.5ECE 10.9 ↓ | 95.5ECE 12.5 ↓ | 95.0ECE 11.8 ↓ | Not run | 94.0 | 99.7 | 88.1 | Not run | Backgammon 9.4 | not timed yet |
| Claude Opus 5.5 high · chatAnthropic | Not disclosed | 93.2TR 93.7 L&M 98.2 BG 87.7 | Not applicable | 94.5ECE 14.3 ↓ | 93.4ECE 14.9 ↓ | 93.1ECE 13.3 ↓ | Not run | 95.2 | 99.7 | 87.7 | Not run | Backgammon 6.7 | not timed yet |
| Claude Sonnet 5.5 high · chatAnthropic | Not disclosed | 88.3TR 81.3 L&M 97.8 BG 85.8 | Not applicable | 81.5ECE 13.8 ↓ | 81.2ECE 16.9 ↓ | 82.6ECE 13.9 ↓ | Not run | 94.3 | 99.6 | 85.8 | Not run | Backgammon 7.0 | not timed yet |
| Muse Ember max · chatMeta | Not disclosed | 87.7TR 82.0 L&M 97.7 BG 83.4 | Not applicable | 87.8ECE 7.5 ↓ | 79.6ECE 6.3 ↓ | 87.0ECE 7.0 ↓ | 82.8 | 94.3 | 99.5 | 83.4 | Not run | – | not timed yet |
| Gemma 4 31B reasoning off · chatGoogle | 31.3B | 82.1TR 66.4 L&M 96.7 BG 83.4 | Not applicable | 88.6ECE 8.1 ↓ | 57.0ECE 31.0 ↓ | 88.1ECE 8.5 ↓ | Not run | 91.7 | 99.2 | 83.4 | Not run | Backgammon 8.5 | not timed yet |
| Qwen3.8-27B low · chatAlibaba | 27.8B | 80.2TR 68.3 L&M 96.8 BG 75.4 | Not applicable | 86.7ECE 10.7 ↓ | 60.5ECE 29.4 ↓ | 86.3ECE 11.0 ↓ | Not run | 92.6 | 99.0 | 75.4 | Not run | Backgammon 19.0 | not timed yet |
| Qwen3.8 Flash no reasoning · chatAlibaba | Not disclosed | 74.1TR 53.8 L&M 97.0 BG 71.5 | Not applicable | 84.0ECE 5.7 ↓ | 41.0ECE 38.5 ↓ | 83.8ECE 5.5 ↓ | Not run | 92.4 | 99.4 | 71.5 | Not run | Backgammon 2.7 | not timed yet |
| Claude Fable 5.2 max · chatAnthropic | Not disclosed | Not applicable | Not applicable | 97.1ECE 8.2 ↓ | 97.9ECE 7.2 ↓ | 97.5ECE 10.7 ↓ | Not run | Not run | Not run | Not run | Not run | – | not timed yet |
| Claude Haiku 5.5 max · chatAnthropic | Not disclosed | Not applicable | Not applicable | 83.6ECE 18.9 ↓ | 30.4ECE 16.4 ↓ | 90.9ECE 18.4 ↓ | Not run | Not run | Not run | Not run | Not run | – | not timed yet |
| DeepSeek-V4-Flash chatDeepSeek | 285B13B active | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | 99.1 | Not run | Not run | – | not timed yet |
| DeepSeek-V4.1-Flash reasoning on · chat · first releaseDeepSeek | 552B16B active | Not applicable | Not applicable | 91.8ECE 6.6 ↓ | 89.3ECE 8.6 ↓ | Not run | Not run | Not run | Not run | Not run | Not run | – | not timed yet |
| DeepSeek-V4.1-Flash reasoning on · chat · 1017DeepSeek | 552B16B active | Not applicable | Not applicable | 92.6ECE 6.0 ↓ | 88.5ECE 7.3 ↓ | 90.6ECE 7.5 ↓ | Not run | 95.2 | 99.5 | Not run | Not run | – | not timed yet |
| DeepSeek-V4.1-Flash low · API + NVIDIA RTX PRO 6000 Blackwell · first releaseDeepSeek | Not disclosed | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | 83.9 | Not run | Backgammon 8.2 | not timed yet |
| Erk-14B reasoning off · chateCloud Technology | 14.8B | Not applicable | Not applicable | 65.3ECE 6.7 ↓ | 16.7ECE 16.3 ↓ | 65.0ECE 10.9 ↓ | Not run | 42.8 | 88.4 | Not run | Not run | – | not timed yet |
| GLM-5.3 low · chatZ.ai | 753.3B | Not applicable | Not applicable | 84.6ECE 8.6 ↓ | 61.2ECE 5.5 ↓ | 84.7ECE 3.6 ↓ | Not run | Not run | Not run | 73.8 | Not run | Backgammon 17.6 | not timed yet |
| GLM-5.3-Flash reasoning on · chatZ.ai | 320B18B active | Not applicable | Not applicable | 86.7ECE 5.5 ↓ | 67.9ECE 4.4 ↓ | 87.0ECE 7.0 ↓ | Not run | 91.4 | 98.5 | Not run | Not run | – | not timed yet |
| GLiNER2.5 MultiFastino | 290M | Not applicable | Not applicable | −3.5ECE 35.1 ↓ | −3.0ECE 35.8 ↓ | −4.7ECE 36.4 ↓ | Not run | Not run | Not run | Not run | Not run | – | not timed yet |
| GPT-6 Luna low · chatOpenAI | Not disclosed | Not applicable | Not applicable | 84.2ECE 9.0 ↓ | 50.2ECE 32.7 ↓ | 82.9ECE 10.1 ↓ | Not run | 47.4 | 92.1 | Not run | Not run | – | not timed yet |
| GPT-6.1 Terra high · chatOpenAI | Not disclosed | Not applicable | Not applicable | 95.2ECE 1.3 ↓ | 91.8ECE 3.5 ↓ | 95.2ECE 1.3 ↓ | Not run | 92.4 | 99.0 | Not run | Not run | – | not timed yet |
| Gemini 3.8 Flash high · chatGoogle | Not disclosed | Not applicable | Not applicable | 96.4ECE 1.3 ↓ | 98.1ECE 1.0 ↓ | 95.0ECE 1.7 ↓ | Not run | Not run | Not run | 87.3 | Not run | Backgammon 8.2 | not timed yet |
| Gemini 4 Argon low · chatGoogle | Not disclosed | Not applicable | Not applicable | 96.2ECE 1.1 ↓ | 95.1ECE 2.7 ↓ | 95.0ECE 1.5 ↓ | Not run | Not run | Not run | 89.3 | Not run | Backgammon 8.5 | not timed yet |
| Gemini 4 Carbon high · chatGoogle | Not disclosed | Not applicable | Not applicable | 97.0ECE 0.7 ↓ | 98.5ECE 0.3 ↓ | 92.9ECE 3.1 ↓ | Not run | 95.9 | 99.5 | Not run | Not run | – | not timed yet |
| Gemini 4 Hydrogen high · chatGoogle | Not disclosed | Not applicable | 93.5EN 95.9 Logic 95.1 Math 99.5 AC 83.7 | 96.6ECE 2.3 ↓ | 98.1ECE 0.9 ↓ | 95.9ECE 3.1 ↓ | Not run | 95.1 | 99.5 | Not run | 83.7 | – | not timed yet |
| Gemma 4 12B Unified default reasoning · chatGoogle | 12B | Not applicable | 85.9EN 83.5 Logic 84.7 Math 98.6 AC 77.0 | Not run | Not run | 83.5 | Not run | 84.7 | 98.6 | Not run | 77.0 | – | not timed yet |
| Gemma 4 12B Unified thinking on · chatGoogle | 11.9B | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | Not run | 67.7 | – | not timed yet |
| Gemma 4 E2B reasoning on · chatGoogle | 5.1B | Not applicable | 75.9EN 70.3 Logic 66.1 Math 96.0 AC 71.1 | Not run | Not run | 70.3 | Not run | 66.1 | 96.0 | Not run | 71.1 | – | not timed yet |
| Gemma 4 E2B default reasoning · chatGoogle | 5.1B | Not applicable | Not applicable | Not run | Not run | 68.2 | Not run | Not run | Not run | Not run | 55.2 | – | not timed yet |
| Gemma 4 E2B thinking off · chatGoogle | 5.1B | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | Not run | 54.8 | – | not timed yet |
| Gemma 4 E4B default reasoning · chatGoogle | 8B | Not applicable | 83.0EN 78.1 Logic 79.0 Math 97.6 AC 77.2 | 72.0 | 36.3 | 78.1 | 28.6 | 79.0 | 97.6 | Not run | 77.2 | – | not timed yet |
| Gemma 4 E4B thinking on · sampled (temperature 1.0) · chatGoogle | 8B | Not applicable | Not applicable | Not run | Not run | Not run | Not run | Not run | Not run | Not run | 75.1 | – | not timed yet |
| Granite-4.2-3B reasoning on · chatIBM | 3.7B | Not applicable | 83.1EN 68.0 Logic 89.6 Math 99.0 AC 75.9 | Not run | Not run | 68.0 | Not run | 89.6 | 99.0 | Not run | 75.9 | – | not timed yet |
| Grok 4.7 high · chatxAI | Not disclosed | Not applicable | Not applicable | 93.7ECE 10.5 ↓ | 89.7ECE 12.4 ↓ | 91.8ECE 11.1 ↓ | Not run | Not run | 99.4 | 84.0 | Not run | – | not timed yet |
| Inkling 2 Small high · chatThinking Machines | 594B17B active | Not applicable | Not applicable | Not run | Not run | Not run | Not run | 94.6 | 99.4 | Not run | 86.2 | – | not timed yet |
| Inkling-Small high · chat · Jul 2026Thinking Machines | 266B12B active | Not applicable | Not applicable | 84.0ECE 3.2 ↓ | 71.1ECE 9.9 ↓ | 85.4ECE 2.6 ↓ | Not run | 94.2 | 99.5 | Not run | Not run | – | not timed yet |
| Kimi K3.1 Pro low · chatMoonshot | 2.8T104B active | Not applicable | Not applicable | 92.2ECE 4.3 ↓ | 72.6ECE 16.2 ↓ | 90.4ECE 5.7 ↓ | Not run | Not run | Not run | 76.2 | Not run | Backgammon 4.8 | not timed yet |
| LFM2.5-2.6B default reasoning · chatLiquid AI | 2.7B | Not applicable | Not applicable | 31.8 | 11.2 | 67.3 | Not run | 69.2 | 92.5 | Not run | Not run | – | not timed yet |
| LFM2.5-24B-A2B default reasoning · chatLiquid AI | 24B2.5B active | Not applicable | Not applicable | Not run | Not run | 78.3 | Not run | 83.3 | 98.8 | 60.6 | Not run | – | not timed yet |
| LFM2.5-8B-A1B default reasoning · chatLiquid AI | 8.5B1.7B active | Not applicable | 66.6EN 63.0 Logic 50.2 Math 95.3 AC 57.7 | 7.0 | −1.2 | 63.0 | Not run | 50.2 | 95.3 | Not run | 57.7 | – | not timed yet |
| LFM3-1.7B reasoning on · chatLiquid AI | 1.7B | Not applicable | 80.3EN 64.7 Logic 81.9 Math 98.7 AC 75.8 | 30.3 | 9.2 | 64.7 | Not run | 81.9 | 98.7 | Not run | 75.8 | – | not timed yet |
| LFM3-1.7B reasoning off · chatLiquid AI | 1.7B | Not applicable | Not applicable | 21.5 | 5.7 | 63.9 | Not run | Not run | Not run | Not run | 74.9 | – | not timed yet |
| MiMo-V2.6-Flash high · chatXiaomi | 309B15B active | Not applicable | Not applicable | 83.2ECE 3.7 ↓ | 57.4ECE 10.3 ↓ | 86.3ECE 6.5 ↓ | Not run | 93.9 | 99.3 | Not run | Not run | – | not timed yet |
| MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | Not applicable | 84.3EN 72.7 Logic 86.1 Math 99.0 AC 79.2 | −16.8 | −18.5 | 72.7 | Not run | 86.1 | 99.0 | Not run | 79.2 | – | not timed yet |
| MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | Not applicable | Not applicable | Not run | Not run | 59.7 | Not run | Not run | Not run | Not run | 67.0 | – | not timed yet |
| MiniMax-M3.1-Flash high · chatMiniMax | 428B23B active | Not applicable | Not applicable | 78.7ECE 9.5 ↓ | 49.3ECE 26.1 ↓ | 80.6ECE 9.0 ↓ | Not run | 92.4 | 97.8 | Not run | Not run | – | not timed yet |
| Qwen3-14B reasoning off · chatAlibaba | 14.8B | Not applicable | Not applicable | 65.1ECE 23.2 ↓ | 18.1ECE 48.6 ↓ | 74.6ECE 17.8 ↓ | Not run | 56.7 | 94.7 | Not run | Not run | – | not timed yet |
| Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | Not applicable | 72.4EN 69.8 Logic 56.0 Math 93.3 AC 70.3 | 17.9 | 2.0 | 69.8 | Not run | 56.0 | 93.3 | Not run | 70.3 | – | not timed yet |
| Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | Not applicable | Not applicable | Not run | Not run | 59.1 | Not run | Not run | Not run | Not run | 50.2 | – | not timed yet |
| Qwen3.5-397B-A17B high · chatAlibaba | 397B17B active | Not applicable | Not applicable | 88.8ECE 2.2 ↓ | 74.0ECE 9.1 ↓ | 89.3ECE 3.3 ↓ | Not run | 93.9 | Not run | Not run | Not run | – | not timed yet |
| Qwen3.5-4B reasoning on · chatAlibaba | 4.7B | Not applicable | 84.9EN 83.8 Logic 90.4 Math 99.3 AC 66.2 | 73.9 | 37.9 | 83.8 | Not run | 90.4 | 99.3 | Not run | 66.2 | – | not timed yet |
| Qwen3.5-4B reasoning off · chatAlibaba | 4.7B | Not applicable | Not applicable | 60.6 | 20.9 | 75.5 | Not run | Not run | Not run | Not run | 64.6 | – | not timed yet |
| Qwen4-Max high · chatAlibaba | Not disclosed | Not applicable | Not applicable | 93.3ECE 7.0 ↓ | 88.1ECE 8.3 ↓ | 93.1ECE 8.5 ↓ | Not run | 95.3 | 99.7 | Not run | Not run | – | not timed yet |
| Raon-1 nano reasoning off · chatKRAFTON | 3.5B | Not applicable | 56.5EN 42.2 Logic 56.3 Math 86.9 AC 40.6 | Not run | Not run | 42.2 | Not run | 56.3 | 86.9 | Not run | 40.6 | – | not timed yet |
Size, calibration and latency against score
Size against score
Combined: 28 decision models; 8 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
Combined: 36 decision models
Latency ↓ against score
Combined: 11 decision models; 25 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Size against score
Decision Index: 44 decision models; 9 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
Decision Index: 53 decision models
Latency ↓ against score
Decision Index: 13 decision models; 40 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Size against score
Turkish Reading: 59 decision models; 17 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
Turkish Reading: 76 decision models
Latency ↓ against score
Turkish Reading: 13 decision models; 63 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Size against score
Turkish Grammar: 59 decision models; 17 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
Turkish Grammar: 76 decision models
Latency ↓ against score
Turkish Grammar: 13 decision models; 63 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Size against score
English Reading: 58 decision models; 17 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
English Reading: 75 decision models
Latency ↓ against score
English Reading: 13 decision models; 62 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Size against score
Vision: 12 decision models; 8 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
Vision: 20 decision models
Latency ↓ against score
Vision: 7 decision models; 13 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Size against score
Logic: 54 decision models; 15 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
Logic: 69 decision models
Latency ↓ against score
Logic: 13 decision models; 56 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Size against score
Math: 54 decision models; 15 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
Math: 69 decision models
Latency ↓ against score
Math: 13 decision models; 56 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Size against score
Backgammon: 30 decision models; 11 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
Models without a calibration error are left out
No points yet.
Latency ↓ against score
Backgammon: 11 decision models; 30 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Size against score
Answer Check: 44 decision models; 11 left out (models whose maker discloses no size are left out)
Calibration error ↓ against score
Answer Check: 55 decision models
Latency ↓ against score
Answer Check: 13 decision models; 42 left out (latency: Reading · Turkish requests on spark2, a DGX Spark (GB10))
Their score against ours
Jev Decision Index 0.4.1, Full score (6 November 2026, source): a sanity check, never our score; its 15 open decision models joined to ours by Hugging Face repo.
12 of the index's 15 models have a score here. Of their 66 pairs, 61 keep the index's order, 3 are reversed and 2 are tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑Combined, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 | 74.0 | 71.3 | 2 |
| 2 | GLiDE 27Bfastino/GLiDE-27B | 73.4 | not on this bench yet | – |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 | 71.9 | 73.2 | 1 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 | 71.1 | 69.4 | 3 |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b | 70.5 | 68.2 | 4 |
| 6 | d3-ultravllm-sr/d3-ultra | 70.2 | not on this bench yet | – |
| 7 | GLiDE 9Bfastino/GLiDE-9B | 62.5 | not on this bench yet | – |
| 8 | Elephant-1baselabs/elephant-1 | 60.3 | 49.7 | 5 |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 | 59.0 | 47.5 | 6 |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 | 56.0 | 43.5 | 8 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant | 54.8 | 47.4 | 7 |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 | 54.3 | 41.4 | 9 |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B | 36.4 | 19.7 | 10 |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B | 33.8 | 3.7 | 12 |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B | 29.6 | 17.7 | 11 |
| ref. | Jev 1.14reference, hosted | no index score here | 57.1 | – |
14 of the index's 15 models have a score here. Of their 91 pairs, 80 keep the index's order, 4 are reversed and 7 are tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑Decision Index, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 | 74.0 | 74.8 | 2 |
| 2 | GLiDE 27Bfastino/GLiDE-27B | 73.4 | 71.9 | 4 |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 | 71.9 | 75.3 | 1 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 | 71.1 | 73.8 | 3 |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b | 70.5 | 69.1 | 5 |
| 6 | d3-ultravllm-sr/d3-ultra | 70.2 | not on this bench yet | – |
| 7 | GLiDE 9Bfastino/GLiDE-9B | 62.5 | 61.8 | 6 |
| 8 | Elephant-1baselabs/elephant-1 | 60.3 | 60.8 | 7 |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 | 59.0 | 59.6 | 8 |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 | 56.0 | 57.4 | 10 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant | 54.8 | 59.3 | 9 |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 | 54.3 | 56.9 | 11 |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B | 36.4 | 34.5 | 12 |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B | 33.8 | 24.6 | 14 |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B | 29.6 | 29.6 | 13 |
| ref. | Jev 1.14reference, hosted | no index score here | 64.5 | – |
15 of the index's 15 models have a score here. Of their 105 pairs, 82 keep the index's order, 1 is reversed and 22 are tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑Turkish Reading, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 @ fa4bf7e | 74.0 | 90.5 | 3 |
| 2 | GLiDE 27Bfastino/GLiDE-27B @ 4d32743 | 73.4 | 88.6 | 5 |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 @ 8e56aa1 | 71.9 | 91.0 | 1 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 @ 8fcee85 | 71.1 | 89.7 | 4 |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b @ 866c0fc | 70.5 | 90.9 | 2 |
| 6 | d3-ultravllm-sr/d3-ultra @ 52ba9e7 | 70.2 | 87.2 | 6 |
| 7 | GLiDE 9Bfastino/GLiDE-9B @ ab54df5 | 62.5 | 75.2 | 8 |
| 8 | Elephant-1baselabs/elephant-1 @ b3f8a93 | 60.3 | 75.6 | 7 |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 @ f02a632 | 59.0 | 74.7 | 9 |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 @ 779944d | 56.0 | 68.6 | 10 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant @ 416088f | 54.8 | 67.6 | 11 |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 @ 6d771ad | 54.3 | 65.9 | 12 |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B @ 9eacdee | 36.4 | 28.0 | 13 |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B @ 6bda7ab | 33.8 | 5.7 | 15 |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B @ bcd88a3 | 29.6 | 24.5 | 14 |
| ref. | Jev 1.14reference, hosted | no index score here | 81.3 | – |
15 of the index's 15 models have a score here. Of their 105 pairs, 91 keep the index's order, 9 are reversed and 5 are tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑Turkish Grammar, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 @ fa4bf7e | 74.0 | 64.7 | 3 |
| 2 | GLiDE 27Bfastino/GLiDE-27B @ 4d32743 | 73.4 | 53.4 | 5 |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 @ 8e56aa1 | 71.9 | 71.4 | 1 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 @ 8fcee85 | 71.1 | 58.3 | 4 |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b @ 866c0fc | 70.5 | 68.5 | 2 |
| 6 | d3-ultravllm-sr/d3-ultra @ 52ba9e7 | 70.2 | 48.0 | 6 |
| 7 | GLiDE 9Bfastino/GLiDE-9B @ ab54df5 | 62.5 | 25.1 | 9 |
| 8 | Elephant-1baselabs/elephant-1 @ b3f8a93 | 60.3 | 25.5 | 8 |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 @ f02a632 | 59.0 | 29.6 | 7 |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 @ 779944d | 56.0 | 21.6 | 10 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant @ 416088f | 54.8 | 20.9 | 12 |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 @ 6d771ad | 54.3 | 21.4 | 11 |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B @ 9eacdee | 36.4 | 4.6 | 14 |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B @ 6bda7ab | 33.8 | 1.6 | 15 |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B @ bcd88a3 | 29.6 | 5.8 | 13 |
| ref. | Jev 1.14reference, hosted | no index score here | 35.2 | – |
15 of the index's 15 models have a score here. Of their 105 pairs, 79 keep the index's order, 0 are reversed and 26 are tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑English Reading, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 @ fa4bf7e | 74.0 | 90.6 | 2 |
| 2 | GLiDE 27Bfastino/GLiDE-27B @ 4d32743 | 73.4 | 88.6 | 3 |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 @ 8e56aa1 | 71.9 | 92.0 | 1 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 @ 8fcee85 | 71.1 | 88.3 | 4 |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b @ 866c0fc | 70.5 | 88.3 | 5 |
| 6 | d3-ultravllm-sr/d3-ultra @ 52ba9e7 | 70.2 | 86.7 | 6 |
| 7 | GLiDE 9Bfastino/GLiDE-9B @ ab54df5 | 62.5 | 82.9 | 7 |
| 8 | Elephant-1baselabs/elephant-1 @ b3f8a93 | 60.3 | 81.7 | 9 |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 @ f02a632 | 59.0 | 82.4 | 8 |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 @ 779944d | 56.0 | 76.0 | 12 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant @ 416088f | 54.8 | 78.5 | 10 |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 @ 6d771ad | 54.3 | 77.4 | 11 |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B @ 9eacdee | 36.4 | 50.9 | 13 |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B @ 6bda7ab | 33.8 | 49.3 | 14 |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B @ bcd88a3 | 29.6 | 49.3 | 15 |
| ref. | Jev 1.14reference, hosted | no index score here | 82.6 | – |
5 of the index's 15 models have a score here. Of their 10 pairs, 9 keep the index's order, 0 are reversed and 1 is tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑Vision, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 | 74.0 | not on this bench yet | – |
| 2 | GLiDE 27Bfastino/GLiDE-27B @ 4d32743 | 73.4 | 37.4 | 1 |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 @ 8e56aa1 | 71.9 | 37.3 | 2 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 | 71.1 | not on this bench yet | – |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b | 70.5 | not on this bench yet | – |
| 6 | d3-ultravllm-sr/d3-ultra @ 52ba9e7 | 70.2 | 31.8 | 3 |
| 7 | GLiDE 9Bfastino/GLiDE-9B | 62.5 | not on this bench yet | – |
| 8 | Elephant-1baselabs/elephant-1 | 60.3 | not on this bench yet | – |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 | 59.0 | not on this bench yet | – |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 @ 779944d | 56.0 | 12.7 | 4 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant | 54.8 | not on this bench yet | – |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 | 54.3 | not on this bench yet | – |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B | 36.4 | not on this bench yet | – |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B | 33.8 | not on this bench yet | – |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B @ bcd88a3 | 29.6 | 8.1 | 5 |
| ref. | Jev 1.14reference, hosted | no index score here | 29.1 | – |
15 of the index's 15 models have a score here. Of their 105 pairs, 88 keep the index's order, 3 are reversed and 14 are tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑Logic, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 @ fa4bf7e | 74.0 | 49.1 | 1 |
| 2 | GLiDE 27Bfastino/GLiDE-27B @ 4d32743 | 73.4 | 48.0 | 3 |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 @ 8e56aa1 | 71.9 | 48.8 | 2 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 @ 8fcee85 | 71.1 | 47.3 | 4 |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b @ 866c0fc | 70.5 | 40.3 | 6 |
| 6 | d3-ultravllm-sr/d3-ultra @ 52ba9e7 | 70.2 | 43.2 | 5 |
| 7 | GLiDE 9Bfastino/GLiDE-9B @ ab54df5 | 62.5 | 34.8 | 7 |
| 8 | Elephant-1baselabs/elephant-1 @ b3f8a93 | 60.3 | 33.0 | 8 |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 @ f02a632 | 59.0 | 29.1 | 10 |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 @ 779944d | 56.0 | 27.9 | 12 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant @ 416088f | 54.8 | 32.3 | 9 |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 @ 6d771ad | 54.3 | 28.9 | 11 |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B @ 9eacdee | 36.4 | 11.4 | 13 |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B @ 6bda7ab | 33.8 | 4.0 | 15 |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B @ bcd88a3 | 29.6 | 8.8 | 14 |
| ref. | Jev 1.14reference, hosted | no index score here | 35.2 | – |
15 of the index's 15 models have a score here. Of their 105 pairs, 90 keep the index's order, 5 are reversed and 10 are tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑Math, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 @ fa4bf7e | 74.0 | 72.5 | 1 |
| 2 | GLiDE 27Bfastino/GLiDE-27B @ 4d32743 | 73.4 | 65.0 | 4 |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 @ 8e56aa1 | 71.9 | 71.7 | 2 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 @ 8fcee85 | 71.1 | 71.4 | 3 |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b @ 866c0fc | 70.5 | 61.3 | 6 |
| 6 | d3-ultravllm-sr/d3-ultra @ 52ba9e7 | 70.2 | 64.4 | 5 |
| 7 | GLiDE 9Bfastino/GLiDE-9B @ ab54df5 | 62.5 | 47.4 | 8 |
| 8 | Elephant-1baselabs/elephant-1 @ b3f8a93 | 60.3 | 51.7 | 7 |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 @ f02a632 | 59.0 | 45.0 | 9 |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 @ 779944d | 56.0 | 45.0 | 10 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant @ 416088f | 54.8 | 44.8 | 11 |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 @ 6d771ad | 54.3 | 44.1 | 12 |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B @ 9eacdee | 36.4 | 15.1 | 13 |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B @ 6bda7ab | 33.8 | 0.7 | 15 |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B @ bcd88a3 | 29.6 | 10.3 | 14 |
| ref. | Jev 1.14reference, hosted | no index score here | 56.4 | – |
12 of the index's 15 models have a score here. Of their 66 pairs, 56 keep the index's order, 4 are reversed and 6 are tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑Backgammon, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 @ fa4bf7e | 74.0 | 77.1 | 3 |
| 2 | GLiDE 27Bfastino/GLiDE-27B | 73.4 | not on this bench yet | – |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 @ 8e56aa1 | 71.9 | 78.3 | 1 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 @ 8fcee85 | 71.1 | 77.3 | 2 |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b @ 866c0fc | 70.5 | 75.5 | 4 |
| 6 | d3-ultravllm-sr/d3-ultra | 70.2 | not on this bench yet | – |
| 7 | GLiDE 9Bfastino/GLiDE-9B | 62.5 | not on this bench yet | – |
| 8 | Elephant-1baselabs/elephant-1 @ b3f8a93 | 60.3 | 63.2 | 6 |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 @ f02a632 | 59.0 | 60.0 | 7 |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 @ 779944d | 56.0 | 55.8 | 8 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant @ 416088f | 54.8 | 66.8 | 5 |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 | 54.3 | 50.8 | 9 |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B @ 9eacdee | 36.4 | 33.6 | 10 |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B @ 6bda7ab | 33.8 | 6.5 | 12 |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B @ bcd88a3 | 29.6 | 31.9 | 11 |
| ref. | Jev 1.14reference, hosted | no index score here | 73.1 | – |
14 of the index's 15 models have a score here. Of their 91 pairs, 71 keep the index's order, 5 are reversed and 15 are tied on our score.
| Index rank | Model | Their score ↑Jev Decision Index 0.4.1, Full score | Our score ↑Answer Check, score 0–100, chance = 0 | Our rankof these |
|---|---|---|---|---|
| 1 | REAX 1.3decisionmodels/REAX-1.3 @ fa4bf7e | 74.0 | 87.0 | 3 |
| 2 | GLiDE 27Bfastino/GLiDE-27B @ 4d32743 | 73.4 | 86.1 | 5 |
| 3 | KnowLine-31B-Gen1.5PelaAI/KnowLine-31B-Gen1.5 @ 8e56aa1 | 71.9 | 88.8 | 1 |
| 4 | Sieve 31B v3sthanika-ai/Sieve-31B-v3 @ 8fcee85 | 71.1 | 88.1 | 2 |
| 5 | Turion Jev 31Bturion-ai/turion-jev-31b @ 866c0fc | 70.5 | 86.4 | 4 |
| 6 | d3-ultravllm-sr/d3-ultra | 70.2 | not on this bench yet | – |
| 7 | GLiDE 9Bfastino/GLiDE-9B @ ab54df5 | 62.5 | 82.2 | 6 |
| 8 | Elephant-1baselabs/elephant-1 @ b3f8a93 | 60.3 | 76.9 | 11 |
| 9 | Cerno 9B v1.2apertixai/cerno-9b-v1.2 @ f02a632 | 59.0 | 81.9 | 7 |
| 10 | KnowLine 4B Gen5.5PelaAI/KnowLine-4B-Gen5.5 @ 779944d | 56.0 | 80.6 | 9 |
| 11 | StartLux-Decision-v1.1-4B-Instantstartlux-models/StartLux-Decision-v1.1-4B-Instant @ 416088f | 54.8 | 81.7 | 8 |
| 12 | Perplexity Decider Flash v1.2perplexity-ai/pplx-decider-flash-v1.2 @ 6d771ad | 54.3 | 77.2 | 10 |
| 13 | GLiDE 0.8Bfastino/GLiDE-0.8B @ 9eacdee | 36.4 | 60.8 | 12 |
| 14 | GLiDE 0.4Bfastino/GLiDE-0.4B @ 6bda7ab | 33.8 | 44.4 | 14 |
| 15 | OneJev1.6-0.8BOmniJev/OneJev1.6-0.8B @ bcd88a3 | 29.6 | 50.0 | 13 |
| ref. | Jev 1.14reference, hosted | no index score here | 83.6 | – |
All benches
Turkish Reading
Decision models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Karpo Decision 31BMachinePulse | 31.3B | 91.8 | 93.4%656 questions | 1.2 | d69cdd5 |
| 2 | KnowLine-31B-Gen2PelaAI | 31.3B | 91.8tied with Karpo Decision 31B | 93.4%656 questions | 6.3 | fd99a9e |
| 3 | Copernicus-31BPoindexter Labs | 31.3B | 91.4tied with KnowLine-31B-Gen2 | 93.1%656 questions | 6.3 | be71525 |
| 4 | KnowLine-31B-Gen1.5 LoRA, mergedPelaAI | 31.3B | 91.0tied with Copernicus-31B | 92.8%656 questions | 3.4 | 8e56aa1 |
| 5 | Turion Jev 31B LoRA, mergedTurion.AI | 31.3B | 90.9tied with KnowLine-31B-Gen1.5 | 92.7%656 questions | 9.5 | 866c0fc |
| 6 | REAX 1.3 blendREAX | 57.4B | 90.5tied with Turion Jev 31B | 92.4%656 questions | 2.1 | fa4bf7e |
| 7 | Gemini Decision 1 20261111Google | Not disclosed | 90.1tied with REAX 1.3 | 92.1%656 questions | 2.8 | no repo |
| 8 | d1-ultraLiquid AI | Not disclosed | 90.1tied with Gemini Decision 1 · 20261111 | 92.1%656 questions | 3.6 | no repo |
| 9 | Logits Sigma-1 Duo blendLogits | 57.4B | 89.9tied with d1-ultra | 91.9%656 questions | 5.3 | a0c1b6e |
| 10 | Sieve 31B v3 full fine-tuneSthānika AI | 31.3B | 89.7tied with Logits Sigma-1 Duo | 91.8%656 questions | 3.5 | 8fcee85 |
| 11 | GLiDE 27BFastino | 26.1B | 88.6tied with Sieve 31B v3 | 90.9%656 questions | 2.4 | 4d32743 |
| 12 | Mammoth-1 post-trainedBase Labs | 27.4B | 88.0tied with GLiDE 27B | 90.4%656 questions | 2.0 | 61cec44 |
| 13 | d1.2-ultraLiquid AI | Not disclosed | 87.2tied with Mammoth-1 | 89.8%656 questions | 5.6 | no repo |
| 14 | d3-ultra fine-tunevLLM Semantic Router | 26.1B | 87.2tied with d1.2-ultra | 89.8%656 questions | 3.8 | 52ba9e7 |
| 15 | Drex Max fine-tuneNace.AI | 26.1B | 87.0tied with d3-ultra | 89.6%656 questions | 3.4 | cac65e1 |
| 16 | GLiDE v1.2Fastino | 26.1B | 87.0tied with Drex Max | 89.6%656 questions | 3.5 | a7e4589 |
| 17 | Perplexity Decider 27B v1.2 DGX SparkPerplexity | 26.1B | 86.5tied with GLiDE · v1.2 | 89.2%656 questions | 3.1 | 225c913 |
| 18 | d3 fine-tunevLLM Semantic Router | 26.1B | 86.5tied with Perplexity Decider 27B v1.2 · DGX Spark | 89.2%656 questions | 3.5 | 20f37b9 |
| 19 | Perplexity Decider 27B v1.1Perplexity | 26.1B | 86.3tied with d3 | 89.0%656 questions | 3.0 | no repo |
| 20 | Perplexity Decider 27B v1.3Perplexity | 26.1B | 86.3tied with Perplexity Decider 27B v1.1 | 89.0%656 questions | 2.4 | 9c2ebb2 |
| 21 | Perplexity Decider 27B v1.2 API · with webPerplexity | 26.1B | 86.1tied with Perplexity Decider 27B v1.3 | 88.9%656 questions | 2.7 | no repo |
| 22 | Drex Max v1.1Nace.AI | Not disclosed | 85.7tied with Perplexity Decider 27B v1.2 · API · with web | 88.6%656 questions | 3.4 | no repo |
| 23 | GLiDE APIFastino | Not disclosed | 84.8tied with Drex Max v1.1 | 87.8%656 questions | 5.8 | no repo |
| 24 | SubQ Snap PreviewSubquadratic | Not disclosed | 84.6tied with GLiDE · API | 87.7%656 questions | 4.2 | no repo |
| 25 | SubQ Snap 1.1Subquadratic | Not disclosed | 83.8tied with SubQ Snap Preview | 87.0%656 questions | 3.6 | no repo |
| 26 | KnowLine-12B-Gen1PelaAI | 12B | 82.5tied with SubQ Snap 1.1 | 86.0%656 questions | 11.4 | f26fbc2 |
| 27 | Perplexity Decider 27B v1Perplexity | 26.1B | 82.5tied with KnowLine-12B-Gen1 | 86.0%656 questions | 5.0 | no repo |
| 28 | d1.2Liquid AI | Not disclosed | 82.5tied with Perplexity Decider 27B v1 | 86.0%656 questions | 6.0 | no repo |
| 29 | GPT-6 Luna Decisions APIOpenAI | Not disclosed | 81.9tied with d1.2 | 85.5%656 questions | 3.8 | no repo |
| 30 | OneJev-27B full fine-tuneOmniJev | 27.4B | 81.5tied with GPT-6 Luna · Decisions API | 85.2%656 questions | 9.9 | a5e5f55 |
| 31 | celeris-1-decisionCeleris | Not disclosed | 81.5tied with OneJev-27B | 85.2%656 questions | 6.2 | no repo |
| 32 | Clef post-trainedCloudflare | 27.5B | 81.3tied with celeris-1-decision | 85.1%656 questions | 4.0 | 2f3de3d |
| 33 | Jev 1.14TypeSafe | Not disclosed | 81.3tied with Clef | 85.1%656 questions | 2.2 | no repo |
| 34 | d1.1Liquid AI | Not disclosed | 80.8tied with Jev 1.14 | 84.6%656 questions | 4.2 | no repo |
| 35 | Mercury DecideInception | Not disclosed | 80.6tied with d1.1 | 84.5%656 questions | 7.5 | no repo |
| 36 | Jev 1.13TypeSafe | Not disclosed | 79.6tied with Mercury Decide | 83.7%656 questions | 3.8 | no repo |
| 37 | Jeffan-9B-1211tattrongvu | 8.9B | 78.7tied with Jev 1.13 | 82.9%656 questions | 11.4 | 862d1a7 |
| 38 | Open-Jev-27B-v1.1 LoRA + headZefan Cai | 27.8B | 78.7tied with Jeffan-9B-1211 | 82.9%656 questions | 43.6 | 28cf730 |
| 39 | Cygnet unchanged baseBlockbrain Labs | 12B | 78.5tied with Open-Jev-27B-v1.1 | 82.8%656 questions | 4.1 | no repo |
| 40 | d1Liquid AI | Not disclosed | 78.5tied with Cygnet | 82.8%656 questions | 2.8 | no repo |
| 41 | Winnow-12B LoRA, mergedEldanRing | 11.9B | 77.5tied with d1 | 82.0%656 questions | 9.3 | b2b1421 |
| 42 | Drex 2.0Nace.AI | 35.1B3B active | 76.8tied with Winnow-12B | 81.4%656 questions | 10.1 | no repo |
| 43 | Adjudex 9B v5newsbang | 8.4B | 76.0tied with Drex 2.0 | 80.8%656 questions | 2.5 | ffdcb82 |
| 44 | Elephant-1 post-trainedBase Labs | 9.4B | 75.6tied with Adjudex 9B v5 | 80.5%656 questions | 4.3 | b3f8a93 |
| 45 | GLiDE 9BFastino | 7.9B | 75.2tied with Elephant-1 | 80.2%656 questions | 3.7 | ab54df5 |
| 46 | KnowLine-9B-Gen1PelaAI | 9.7B | 74.8tied with GLiDE 9B | 79.9%656 questions | 1.7 | 325378d |
| 47 | Cerno 9B v1.2 full fine-tuneApertix AI | 8.9B | 74.7tied with KnowLine-9B-Gen1 | 79.7%656 questions | 3.0 | f02a632 |
| 48 | Dexy-Decision-9B-V2 fine-tunePatrick-18 | 8.4B | 74.5tied with Cerno 9B v1.2 | 79.6%656 questions | 3.2 | 4f434e2 |
| 49 | KnowLine-9B-Gen2PelaAI | 9.7B | 74.5tied with Dexy-Decision-9B-V2 | 79.6%656 questions | 8.2 | 8ec8eb6 |
| 50 | StartLux-Decision-v1.1-9B-InstantStartLux | 9.7B | 73.5tied with KnowLine-9B-Gen2 | 78.8%656 questions | 3.6 | 18a17f7 |
| 51 | vera-medium-omniscar-ai | 4.2B | 69.3 | 75.5%656 questions | 9.1 | 1f23a6c |
| 52 | Valen-Decision-1.5-9BValen-Team | 8.4B | 68.8tied with vera-medium-omni | 75.0%656 questions | 9.7 | 5a51211 |
| 53 | KnowLine 4B Gen5.5 LoRA, mergedPelaAI | 4.7B | 68.6tied with Valen-Decision-1.5-9B | 74.8%656 questions | 6.2 | 779944d |
| 54 | Perplexity Decider Flash v1.3Perplexity | 4.5B | 68.0tied with KnowLine 4B Gen5.5 | 74.4%656 questions | 6.2 | d06fb3f |
| 55 | StartLux-Decision-v1.1-4B-Instant fine-tuneStartLux | 4.7B | 67.6tied with Perplexity Decider Flash v1.3 | 74.1%656 questions | 2.6 | 416088f |
| 56 | Microsoft-Decision-1Microsoft | Not disclosed | 66.8tied with StartLux-Decision-v1.1-4B-Instant | 73.5%656 questions | 8.9 | no repo |
| 57 | Clef-Flash post-trainedCloudflare | 9.5B | 66.3tied with Microsoft-Decision-1 | 73.0%656 questions | 5.6 | fde727a |
| 58 | Perplexity Decider Flash v1.2Perplexity | 4.5B | 65.9tied with Clef-Flash | 72.7%656 questions | 3.7 | 6d771ad |
| 59 | Verdin-4B-v1.1 LoRA, mergedvictorbennett | 4.2B | 64.0tied with Perplexity Decider Flash v1.2 | 71.2%656 questions | 3.9 | 45830d2 |
| 60 | Chinese Decision Pointer 4B LoRA, mergedquincyqiang | 4.2B | 63.6tied with Verdin-4B-v1.1 | 70.9%656 questions | 3.3 | 889f23e |
| 61 | Metask-Jev-4B LoRA, mergedMetask Lab | 4.5B | 53.5 | 62.8%656 questions | 5.3 | ea20fe8 |
| 62 | KnowLine-2B-Gen1 LoRA, mergedPelaAI | 2.3B | 42.8 | 54.3%656 questions | 1.9 | 53245d4 |
| 63 | StartLux-Decision-v1.1-2B-Instant fine-tuneStartLux | 2.3B | 41.1tied with KnowLine-2B-Gen1 | 52.9%656 questions | 3.3 | f1e6cd1 |
| 64 | GLiDE 2BFastino | 1.9B | 39.0tied with StartLux-Decision-v1.1-2B-Instant | 51.2%656 questions | 16.5 | 00f122c |
| 65 | GLiDE 0.8BFastino | 750M | 28.0 | 42.4%656 questions | 21.8 | 9eacdee |
| 66 | d1-2BLiquid AI | 1.9B | 26.1tied with GLiDE 0.8B | 40.9%656 questions | 4.4 | 10e7671 |
| 67 | OneJev1.6-0.8B full fine-tuneOmniJev | 850M | 24.5tied with d1-2B | 39.6%656 questions | 7.3 | bcd88a3 |
| 68 | KnowLine-0.8B-Gen1 LoRA, mergedPelaAI | 870M | 22.8tied with OneJev1.6-0.8B | 38.3%656 questions | 5.5 | 6b27831 |
| 69 | Strands Decider 2B LoRA + headStrands Agents | 2.3B | 21.1tied with KnowLine-0.8B-Gen1 | 36.9%656 questions | 10.2 | 2b52a62 |
| 70 | StartLux-Decision-v1.1-0.8B-Instant fine-tuneStartLux | 870M | 18.8tied with Strands Decider 2B | 35.1%656 questions | 5.0 | 6496ae6 |
| 71 | Decider 1meraGPT | Not disclosed | 18.3tied with StartLux-Decision-v1.1-0.8B-Instant | 34.6%656 questions | 11.8 | no repo |
| 72 | GLiClass v4 Edge fine-tuneKnowledgator | 440M | 7.4 | 25.9%656 questions | 11.9 | 885af1a |
| 73 | GLiDE 0.4BFastino | 400M | 5.7tied with GLiClass v4 Edge | 24.5%656 questions | 7.2 | 6bda7ab |
| 74 | GLiNER2.5-multi-Decide fine-tuneFastino | 290M | 1.3 | 21.0%656 questions | 24.1 | a35a0cd |
| 75 | LayaConvai Innovations | 420M | 1.1tied with GLiNER2.5-multi-Decide | 20.9%656 questions | 9.2 | c5d7873 |
| 76 | Laya MultilingualConvai Innovations | 320M | −3.7 | 17.1%656 questions | 33.1 | c5d7873 |
Chat models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.2 max · chatAnthropic | Not disclosed | 97.1 | 97.7%656 questions | 8.2 | no repo |
| 2 | Gemini 4 Carbon high · chatGoogle | Not disclosed | 97.0tied with Claude Fable 5.2 · max · chat | 97.6%656 questions | 0.7 | no repo |
| 3 | Gemini 4 Hydrogen high · chatGoogle | Not disclosed | 96.6tied with Gemini 4 Carbon · high · chat | 97.3%656 questions | 2.3 | no repo |
| 4 | GPT-6 Astra high · chatOpenAI | Not disclosed | 96.4tied with Gemini 4 Hydrogen · high · chat | 97.1%656 questions | 1.0 | no repo |
| 5 | Gemini 3.8 Flash high · chatGoogle | Not disclosed | 96.4tied with GPT-6 Astra · high · chat | 97.1%656 questions | 1.3 | no repo |
| 6 | Gemini 4 Argon low · chatGoogle | Not disclosed | 96.2tied with Gemini 3.8 Flash · high · chat | 97.0%656 questions | 1.1 | no repo |
| 7 | GPT-6.1 Sol high · chatOpenAI | Not disclosed | 95.6tied with Gemini 4 Argon · low · chat | 96.5%656 questions | 1.8 | no repo |
| 8 | GPT-6.1 Terra high · chatOpenAI | Not disclosed | 95.2tied with GPT-6.1 Sol · high · chat | 96.2%656 questions | 1.3 | no repo |
| 9 | Claude Fable 5.1 high · chatAnthropic | Not disclosed | 94.5tied with GPT-6.1 Terra · high · chat | 95.6%656 questions | 10.9 | no repo |
| 10 | Claude Opus 5.5 high · chatAnthropic | Not disclosed | 94.5tied with Claude Fable 5.1 · high · chat | 95.6%656 questions | 14.3 | no repo |
| 11 | Grok 4.7 high · chatxAI | Not disclosed | 93.7tied with Claude Opus 5.5 · high · chat | 95.0%656 questions | 10.5 | no repo |
| 12 | Qwen4-Max high · chatAlibaba | Not disclosed | 93.3tied with Grok 4.7 · high · chat | 94.7%656 questions | 7.0 | no repo |
| 13 | DeepSeek-V4.1-Flash reasoning on · chat · 1017DeepSeek | 552B16B active | 92.6tied with Qwen4-Max · high · chat | 94.1%656 questions | 6.0 | no repo |
| 14 | Kimi K3.1 Pro low · chatMoonshot | 2.8T104B active | 92.2tied with DeepSeek-V4.1-Flash · reasoning on · chat · 1017 | 93.8%656 questions | 4.3 | no repo |
| 15 | DeepSeek-V4.1-Flash reasoning on · chat · first releaseDeepSeek | 552B16B active | 91.8tied with Kimi K3.1 Pro · low · chat | 93.4%656 questions | 6.6 | no repo |
| 16 | Qwen3.5-397B-A17B high · chatAlibaba | 397B17B active | 88.8tied with DeepSeek-V4.1-Flash · reasoning on · chat · first release | 91.0%656 questions | 2.2 | no repo |
| 17 | Gemma 4 31B reasoning off · chatGoogle | 31.3B | 88.6tied with Qwen3.5-397B-A17B · high · chat | 90.9%656 questions | 8.1 | 842da37 |
| 18 | Muse Ember max · chatMeta | Not disclosed | 87.8tied with Gemma 4 31B · reasoning off · chat | 90.2%656 questions | 7.5 | no repo |
| 19 | GLM-5.3-Flash reasoning on · chatZ.ai | 320B18B active | 86.7tied with Muse Ember · max · chat | 89.3%656 questions | 5.5 | no repo |
| 20 | Qwen3.8-27B low · chatAlibaba | 27.8B | 86.7tied with GLM-5.3-Flash · reasoning on · chat | 89.3%656 questions | 10.7 | 1d4bf0f |
| 21 | GLM-5.3 low · chatZ.ai | 753.3B | 84.6tied with Qwen3.8-27B · low · chat | 87.7%656 questions | 8.6 | no repo |
| 22 | GPT-6 Luna low · chatOpenAI | Not disclosed | 84.2tied with GLM-5.3 · low · chat | 87.3%656 questions | 9.0 | no repo |
| 23 | Inkling-Small high · chat · Jul 2026Thinking Machines | 266B12B active | 84.0tied with GPT-6 Luna · low · chat | 87.2%656 questions | 3.2 | no repo |
| 24 | Qwen3.8 Flash no reasoning · chatAlibaba | Not disclosed | 84.0tied with Inkling-Small · high · chat · Jul 2026 | 87.2%656 questions | 5.7 | no repo |
| 25 | Claude Haiku 5.5 max · chatAnthropic | Not disclosed | 83.6tied with Qwen3.8 Flash · no reasoning · chat | 86.9%656 questions | 18.9 | no repo |
| 26 | MiMo-V2.6-Flash high · chatXiaomi | 309B15B active | 83.2tied with Claude Haiku 5.5 · max · chat | 86.6%656 questions | 3.7 | no repo |
| 27 | Claude Sonnet 5.5 high · chatAnthropic | Not disclosed | 81.5tied with MiMo-V2.6-Flash · high · chat | 85.2%656 questions | 13.8 | no repo |
| 28 | MiniMax-M3.1-Flash high · chatMiniMax | 428B23B active | 78.7tied with Claude Sonnet 5.5 · high · chat | 82.9%656 questions | 9.5 | no repo |
| 29 | Qwen3.5-4B reasoning on · chatAlibaba | 4.7B | 73.9 | 79.1%656 questions | – | 851bf6e |
| 30 | Gemma 4 E4B default reasoning · chatGoogle | 8B | 72.0tied with Qwen3.5-4B · reasoning on · chat | 77.6%656 questions | – | 4ecc2ba |
| 31 | Erk-14B reasoning off · chateCloud Technology | 14.8B | 65.3 | 72.3%656 questions | 6.7 | 8584b14 |
| 32 | Qwen3-14B reasoning off · chatAlibaba | 14.8B | 65.1tied with Erk-14B · reasoning off · chat | 72.1%656 questions | 23.2 | 40c0698 |
| 33 | Qwen3.5-4B reasoning off · chatAlibaba | 4.7B | 60.6 | 68.4%656 questions | – | 851bf6e |
| 34 | LFM2.5-2.6B default reasoning · chatLiquid AI | 2.7B | 31.8 | 45.4%656 questions | – | 7141b56 |
| 35 | LFM3-1.7B reasoning on · chatLiquid AI | 1.7B | 30.3tied with LFM2.5-2.6B · default reasoning · chat | 44.2%656 questions | – | 0dc4268 |
| 36 | LFM3-1.7B reasoning off · chatLiquid AI | 1.7B | 21.5 | 37.2%656 questions | – | 0dc4268 |
| 37 | Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | 17.9tied with LFM3-1.7B · reasoning off · chat | 34.3%656 questions | – | 15852e8 |
| 38 | LFM2.5-8B-A1B default reasoning · chatLiquid AI | 8.5B1.7B active | 7.0 | 25.6%656 questions | – | dd83f5c |
| 39 | GLiNER2.5 MultiFastino | 290M | −3.5 | 17.2%656 questions | 35.1 | 2ca71aa |
| 40 | MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | −16.8 | 6.6%656 questions | – | 42433e9 |
656 questions per model · updated 16 November 2026 · 0 = chance: 1 of 5 options right on every question (accuracy 20%); 100 = every one of the 656 Turkish reading questions (units 1 to 6) right (accuracy 100%) · feed: bench-scores.json
Turkish Grammar
Decision models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | KnowLine-31B-Gen1.5 LoRA, mergedPelaAI | 31.3B | 71.4 | 77.1%1,542 questions | 3.9 | 8e56aa1 |
| 2 | Copernicus-31BPoindexter Labs | 31.3B | 71.2tied with KnowLine-31B-Gen1.5 | 77.0%1,542 questions | 10.5 | be71525 |
| 3 | KnowLine-31B-Gen2PelaAI | 31.3B | 70.0tied with Copernicus-31B | 76.0%1,542 questions | 5.7 | fd99a9e |
| 4 | Turion Jev 31B LoRA, mergedTurion.AI | 31.3B | 68.5tied with KnowLine-31B-Gen2 | 74.8%1,542 questions | 12.0 | 866c0fc |
| 5 | REAX 1.3 blendREAX | 57.4B | 64.7 | 71.8%1,542 questions | 5.1 | fa4bf7e |
| 6 | Gemini Decision 1 20261111Google | Not disclosed | 64.2tied with REAX 1.3 | 71.3%1,542 questions | 2.9 | no repo |
| 7 | Karpo Decision 31BMachinePulse | 31.3B | 63.9tied with Gemini Decision 1 · 20261111 | 71.1%1,542 questions | 3.4 | d69cdd5 |
| 8 | d1-ultraLiquid AI | Not disclosed | 63.4tied with Karpo Decision 31B | 70.7%1,542 questions | 4.2 | no repo |
| 9 | Logits Sigma-1 Duo blendLogits | 57.4B | 59.8 | 67.8%1,542 questions | 5.9 | a0c1b6e |
| 10 | Sieve 31B v3 full fine-tuneSthānika AI | 31.3B | 58.3tied with Logits Sigma-1 Duo | 66.7%1,542 questions | 3.3 | 8fcee85 |
| 11 | GLiDE 27BFastino | 26.1B | 53.4 | 62.7%1,542 questions | 2.7 | 4d32743 |
| 12 | Perplexity Decider 27B v1.2 API · with webPerplexity | 26.1B | 50.2 | 60.2%1,542 questions | 4.9 | no repo |
| 13 | Perplexity Decider 27B v1.3Perplexity | 26.1B | 49.6tied with Perplexity Decider 27B v1.2 · API · with web | 59.7%1,542 questions | 6.1 | 9c2ebb2 |
| 14 | GLiDE v1.2Fastino | 26.1B | 49.3tied with Perplexity Decider 27B v1.3 | 59.5%1,542 questions | 3.0 | a7e4589 |
| 15 | GLiDE APIFastino | Not disclosed | 49.3tied with GLiDE · v1.2 | 59.4%1,542 questions | 4.1 | no repo |
| 16 | d1.2-ultraLiquid AI | Not disclosed | 48.6tied with GLiDE · API | 58.9%1,542 questions | 3.1 | no repo |
| 17 | d3 fine-tunevLLM Semantic Router | 26.1B | 48.4tied with d1.2-ultra | 58.7%1,542 questions | 2.4 | 20f37b9 |
| 18 | Drex Max fine-tuneNace.AI | 26.1B | 48.0tied with d3 | 58.4%1,542 questions | 8.0 | cac65e1 |
| 19 | d3-ultra fine-tunevLLM Semantic Router | 26.1B | 48.0tied with Drex Max | 58.4%1,542 questions | 16.3 | 52ba9e7 |
| 20 | Drex Max v1.1Nace.AI | Not disclosed | 48.0tied with d3-ultra | 58.4%1,542 questions | 2.7 | no repo |
| 21 | Perplexity Decider 27B v1.2 DGX SparkPerplexity | 26.1B | 47.1tied with Drex Max v1.1 | 57.7%1,542 questions | 6.2 | 225c913 |
| 22 | Perplexity Decider 27B v1.1Perplexity | 26.1B | 46.8tied with Perplexity Decider 27B v1.2 · DGX Spark | 57.5%1,542 questions | 5.5 | no repo |
| 23 | Mammoth-1 post-trainedBase Labs | 27.4B | 46.5tied with Perplexity Decider 27B v1.1 | 57.2%1,542 questions | 7.9 | 61cec44 |
| 24 | SubQ Snap PreviewSubquadratic | Not disclosed | 44.6tied with Mammoth-1 | 55.7%1,542 questions | 6.0 | no repo |
| 25 | GPT-6 Luna Decisions APIOpenAI | Not disclosed | 42.9tied with SubQ Snap Preview | 54.3%1,542 questions | 3.0 | no repo |
| 26 | SubQ Snap 1.1Subquadratic | Not disclosed | 42.5tied with GPT-6 Luna · Decisions API | 54.0%1,542 questions | 14.5 | no repo |
| 27 | KnowLine-12B-Gen1PelaAI | 12B | 40.6tied with SubQ Snap 1.1 | 52.5%1,542 questions | 4.3 | f26fbc2 |
| 28 | d1.1Liquid AI | Not disclosed | 39.6tied with KnowLine-12B-Gen1 | 51.7%1,542 questions | 2.2 | no repo |
| 29 | d1.2Liquid AI | Not disclosed | 39.3tied with d1.1 | 51.4%1,542 questions | 2.5 | no repo |
| 30 | celeris-1-decisionCeleris | Not disclosed | 37.4tied with d1.2 | 49.9%1,542 questions | 5.3 | no repo |
| 31 | Perplexity Decider 27B v1Perplexity | 26.1B | 36.9tied with celeris-1-decision | 49.5%1,542 questions | 5.0 | no repo |
| 32 | OneJev-27B full fine-tuneOmniJev | 27.4B | 35.9tied with Perplexity Decider 27B v1 | 48.7%1,542 questions | 3.7 | a5e5f55 |
| 33 | Jev 1.14TypeSafe | Not disclosed | 35.2tied with OneJev-27B | 48.2%1,542 questions | 4.0 | no repo |
| 34 | Mercury DecideInception | Not disclosed | 33.9tied with Jev 1.14 | 47.1%1,542 questions | 19.0 | no repo |
| 35 | Jev 1.13TypeSafe | Not disclosed | 33.0tied with Mercury Decide | 46.4%1,542 questions | 6.5 | no repo |
| 36 | d1Liquid AI | Not disclosed | 32.6tied with Jev 1.13 | 46.0%1,542 questions | 13.5 | no repo |
| 37 | Jeffan-9B-1211tattrongvu | 8.9B | 32.5tied with d1 | 46.0%1,542 questions | 3.0 | 862d1a7 |
| 38 | Clef post-trainedCloudflare | 27.5B | 31.8tied with Jeffan-9B-1211 | 45.5%1,542 questions | 15.1 | 2f3de3d |
| 39 | Drex 2.0Nace.AI | 35.1B3B active | 31.0tied with Clef | 44.8%1,542 questions | 2.0 | no repo |
| 40 | Cygnet unchanged baseBlockbrain Labs | 12B | 30.2tied with Drex 2.0 | 44.2%1,542 questions | 18.9 | no repo |
| 41 | Winnow-12B LoRA, mergedEldanRing | 11.9B | 30.2tied with Cygnet | 44.2%1,542 questions | 24.1 | b2b1421 |
| 42 | Open-Jev-27B-v1.1 LoRA + headZefan Cai | 27.8B | 30.1tied with Winnow-12B | 44.1%1,542 questions | 16.4 | 28cf730 |
| 43 | KnowLine-9B-Gen1PelaAI | 9.7B | 29.7tied with Open-Jev-27B-v1.1 | 43.8%1,542 questions | 6.8 | 325378d |
| 44 | Cerno 9B v1.2 full fine-tuneApertix AI | 8.9B | 29.6tied with KnowLine-9B-Gen1 | 43.7%1,542 questions | 14.2 | f02a632 |
| 45 | KnowLine-9B-Gen2PelaAI | 9.7B | 27.6tied with Cerno 9B v1.2 | 42.1%1,542 questions | 5.1 | 8ec8eb6 |
| 46 | Adjudex 9B v5newsbang | 8.4B | 26.7tied with KnowLine-9B-Gen2 | 41.4%1,542 questions | 12.2 | ffdcb82 |
| 47 | Elephant-1 post-trainedBase Labs | 9.4B | 25.5tied with Adjudex 9B v5 | 40.4%1,542 questions | 15.4 | b3f8a93 |
| 48 | GLiDE 9BFastino | 7.9B | 25.1tied with Elephant-1 | 40.1%1,542 questions | 12.0 | ab54df5 |
| 49 | Dexy-Decision-9B-V2 fine-tunePatrick-18 | 8.4B | 24.6tied with GLiDE 9B | 39.7%1,542 questions | 12.7 | 4f434e2 |
| 50 | StartLux-Decision-v1.1-9B-InstantStartLux | 9.7B | 24.4tied with Dexy-Decision-9B-V2 | 39.5%1,542 questions | 11.3 | 18a17f7 |
| 51 | Perplexity Decider Flash v1.3Perplexity | 4.5B | 24.0tied with StartLux-Decision-v1.1-9B-Instant | 39.2%1,542 questions | 15.3 | d06fb3f |
| 52 | vera-medium-omniscar-ai | 4.2B | 22.3tied with Perplexity Decider Flash v1.3 | 37.9%1,542 questions | 8.0 | 1f23a6c |
| 53 | KnowLine 4B Gen5.5 LoRA, mergedPelaAI | 4.7B | 21.6tied with vera-medium-omni | 37.3%1,542 questions | 16.6 | 779944d |
| 54 | Perplexity Decider Flash v1.2Perplexity | 4.5B | 21.4tied with KnowLine 4B Gen5.5 | 37.2%1,542 questions | 11.0 | 6d771ad |
| 55 | StartLux-Decision-v1.1-4B-Instant fine-tuneStartLux | 4.7B | 20.9tied with Perplexity Decider Flash v1.2 | 36.7%1,542 questions | 10.3 | 416088f |
| 56 | Chinese Decision Pointer 4B LoRA, mergedquincyqiang | 4.2B | 20.5tied with StartLux-Decision-v1.1-4B-Instant | 36.4%1,542 questions | 15.4 | 889f23e |
| 57 | Verdin-4B-v1.1 LoRA, mergedvictorbennett | 4.2B | 20.5tied with Chinese Decision Pointer 4B | 36.4%1,542 questions | 12.8 | 45830d2 |
| 58 | Valen-Decision-1.5-9BValen-Team | 8.4B | 18.8tied with Verdin-4B-v1.1 | 35.0%1,542 questions | 6.9 | 5a51211 |
| 59 | Microsoft-Decision-1Microsoft | Not disclosed | 18.7tied with Valen-Decision-1.5-9B | 35.0%1,542 questions | 24.4 | no repo |
| 60 | Clef-Flash post-trainedCloudflare | 9.5B | 17.6tied with Microsoft-Decision-1 | 34.1%1,542 questions | 20.3 | fde727a |
| 61 | Metask-Jev-4B LoRA, mergedMetask Lab | 4.5B | 14.2 | 31.4%1,542 questions | 6.5 | ea20fe8 |
| 62 | KnowLine-2B-Gen1 LoRA, mergedPelaAI | 2.3B | 11.7tied with Metask-Jev-4B | 29.4%1,542 questions | 11.4 | 53245d4 |
| 63 | GLiDE 2BFastino | 1.9B | 10.7tied with KnowLine-2B-Gen1 | 28.5%1,542 questions | 22.5 | 00f122c |
| 64 | StartLux-Decision-v1.1-2B-Instant fine-tuneStartLux | 2.3B | 7.9 | 26.3%1,542 questions | 9.4 | f1e6cd1 |
| 65 | StartLux-Decision-v1.1-0.8B-Instant fine-tuneStartLux | 870M | 7.3tied with StartLux-Decision-v1.1-2B-Instant | 25.8%1,542 questions | 4.6 | 6496ae6 |
| 66 | OneJev1.6-0.8B full fine-tuneOmniJev | 850M | 5.8tied with StartLux-Decision-v1.1-0.8B-Instant | 24.6%1,542 questions | 10.3 | bcd88a3 |
| 67 | d1-2BLiquid AI | 1.9B | 5.6tied with OneJev1.6-0.8B | 24.5%1,542 questions | 6.7 | 10e7671 |
| 68 | Decider 1meraGPT | Not disclosed | 4.9tied with d1-2B | 23.9%1,542 questions | 15.4 | no repo |
| 69 | GLiDE 0.8BFastino | 750M | 4.6tied with Decider 1 | 23.7%1,542 questions | 26.1 | 9eacdee |
| 70 | Strands Decider 2B LoRA + headStrands Agents | 2.3B | 4.3tied with GLiDE 0.8B | 23.5%1,542 questions | 13.2 | 2b52a62 |
| 71 | KnowLine-0.8B-Gen1 LoRA, mergedPelaAI | 870M | 3.0tied with Strands Decider 2B | 22.4%1,542 questions | 10.8 | 6b27831 |
| 72 | GLiClass v4 Edge fine-tuneKnowledgator | 440M | 2.8tied with KnowLine-0.8B-Gen1 | 22.2%1,542 questions | 10.0 | 885af1a |
| 73 | GLiDE 0.4BFastino | 400M | 1.6tied with GLiClass v4 Edge | 21.3%1,542 questions | 8.3 | 6bda7ab |
| 74 | GLiNER2.5-multi-Decide fine-tuneFastino | 290M | −0.2tied with GLiDE 0.4B | 19.8%1,542 questions | 22.7 | a35a0cd |
| 75 | LayaConvai Innovations | 420M | −2.2tied with GLiNER2.5-multi-Decide | 18.2%1,542 questions | 14.9 | c5d7873 |
| 76 | Laya MultilingualConvai Innovations | 320M | −2.2tied with Laya | 18.2%1,542 questions | 26.0 | c5d7873 |
Chat models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Gemini 4 Carbon high · chatGoogle | Not disclosed | 98.5 | 98.8%1,542 questions | 0.3 | no repo |
| 2 | Gemini 3.8 Flash high · chatGoogle | Not disclosed | 98.1tied with Gemini 4 Carbon · high · chat | 98.5%1,542 questions | 1.0 | no repo |
| 3 | Gemini 4 Hydrogen high · chatGoogle | Not disclosed | 98.1tied with Gemini 3.8 Flash · high · chat | 98.4%1,542 questions | 0.9 | no repo |
| 4 | Claude Fable 5.2 max · chatAnthropic | Not disclosed | 97.9tied with Gemini 4 Hydrogen · high · chat | 98.3%1,542 questions | 7.2 | no repo |
| 5 | GPT-6 Astra high · chatOpenAI | Not disclosed | 97.9tied with Claude Fable 5.2 · max · chat | 98.3%1,542 questions | 0.7 | no repo |
| 6 | GPT-6.1 Sol high · chatOpenAI | Not disclosed | 97.9tied with GPT-6 Astra · high · chat | 98.3%1,542 questions | 0.6 | no repo |
| 7 | Claude Fable 5.1 high · chatAnthropic | Not disclosed | 95.5tied with GPT-6.1 Sol · high · chat | 96.4%1,542 questions | 12.5 | no repo |
| 8 | Gemini 4 Argon low · chatGoogle | Not disclosed | 95.1tied with Claude Fable 5.1 · high · chat | 96.1%1,542 questions | 2.7 | no repo |
| 9 | Claude Opus 5.5 high · chatAnthropic | Not disclosed | 93.4tied with Gemini 4 Argon · low · chat | 94.7%1,542 questions | 14.9 | no repo |
| 10 | GPT-6.1 Terra high · chatOpenAI | Not disclosed | 91.8tied with Claude Opus 5.5 · high · chat | 93.5%1,542 questions | 3.5 | no repo |
| 11 | Grok 4.7 high · chatxAI | Not disclosed | 89.7tied with GPT-6.1 Terra · high · chat | 91.8%1,542 questions | 12.4 | no repo |
| 12 | DeepSeek-V4.1-Flash reasoning on · chat · first releaseDeepSeek | 552B16B active | 89.3tied with Grok 4.7 · high · chat | 91.4%1,542 questions | 8.6 | no repo |
| 13 | DeepSeek-V4.1-Flash reasoning on · chat · 1017DeepSeek | 552B16B active | 88.5tied with DeepSeek-V4.1-Flash · reasoning on · chat · first release | 90.8%1,542 questions | 7.3 | no repo |
| 14 | Qwen4-Max high · chatAlibaba | Not disclosed | 88.1tied with DeepSeek-V4.1-Flash · reasoning on · chat · 1017 | 90.5%1,542 questions | 8.3 | no repo |
| 15 | Claude Sonnet 5.5 high · chatAnthropic | Not disclosed | 81.2 | 85.0%1,542 questions | 16.9 | no repo |
| 16 | Muse Ember max · chatMeta | Not disclosed | 79.6tied with Claude Sonnet 5.5 · high · chat | 83.7%1,542 questions | 6.3 | no repo |
| 17 | Qwen3.5-397B-A17B high · chatAlibaba | 397B17B active | 74.0 | 79.2%1,542 questions | 9.1 | no repo |
| 18 | Kimi K3.1 Pro low · chatMoonshot | 2.8T104B active | 72.6tied with Qwen3.5-397B-A17B · high · chat | 78.1%1,542 questions | 16.2 | no repo |
| 19 | Inkling-Small high · chat · Jul 2026Thinking Machines | 266B12B active | 71.1tied with Kimi K3.1 Pro · low · chat | 76.9%1,542 questions | 9.9 | no repo |
| 20 | GLM-5.3-Flash reasoning on · chatZ.ai | 320B18B active | 67.9 | 74.3%1,542 questions | 4.4 | no repo |
| 21 | GLM-5.3 low · chatZ.ai | 753.3B | 61.2 | 68.9%1,542 questions | 5.5 | no repo |
| 22 | Qwen3.8-27B low · chatAlibaba | 27.8B | 60.5tied with GLM-5.3 · low · chat | 68.4%1,542 questions | 29.4 | 1d4bf0f |
| 23 | MiMo-V2.6-Flash high · chatXiaomi | 309B15B active | 57.4 | 66.0%1,542 questions | 10.3 | no repo |
| 24 | Gemma 4 31B reasoning off · chatGoogle | 31.3B | 57.0tied with MiMo-V2.6-Flash · high · chat | 65.6%1,542 questions | 31.0 | 842da37 |
| 25 | GPT-6 Luna low · chatOpenAI | Not disclosed | 50.2 | 60.2%1,542 questions | 32.7 | no repo |
| 26 | MiniMax-M3.1-Flash high · chatMiniMax | 428B23B active | 49.3tied with GPT-6 Luna · low · chat | 59.5%1,542 questions | 26.1 | no repo |
| 27 | Qwen3.8 Flash no reasoning · chatAlibaba | Not disclosed | 41.0 | 52.8%1,542 questions | 38.5 | no repo |
| 28 | Qwen3.5-4B reasoning on · chatAlibaba | 4.7B | 37.9 | 50.3%1,542 questions | – | 851bf6e |
| 29 | Gemma 4 E4B default reasoning · chatGoogle | 8B | 36.3tied with Qwen3.5-4B · reasoning on · chat | 49.0%1,542 questions | – | 4ecc2ba |
| 30 | Claude Haiku 5.5 max · chatAnthropic | Not disclosed | 30.4 | 44.4%1,542 questions | 16.4 | no repo |
| 31 | Qwen3.5-4B reasoning off · chatAlibaba | 4.7B | 20.9 | 36.7%1,542 questions | – | 851bf6e |
| 32 | Qwen3-14B reasoning off · chatAlibaba | 14.8B | 18.1 | 34.5%1,542 questions | 48.6 | 40c0698 |
| 33 | Erk-14B reasoning off · chateCloud Technology | 14.8B | 16.7tied with Qwen3-14B · reasoning off · chat | 33.4%1,542 questions | 16.3 | 8584b14 |
| 34 | LFM2.5-2.6B default reasoning · chatLiquid AI | 2.7B | 11.2 | 29.0%1,542 questions | – | 7141b56 |
| 35 | LFM3-1.7B reasoning on · chatLiquid AI | 1.7B | 9.2tied with LFM2.5-2.6B · default reasoning · chat | 27.4%1,542 questions | – | 0dc4268 |
| 36 | LFM3-1.7B reasoning off · chatLiquid AI | 1.7B | 5.7 | 24.6%1,542 questions | – | 0dc4268 |
| 37 | Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | 2.0 | 21.6%1,542 questions | – | 15852e8 |
| 38 | LFM2.5-8B-A1B default reasoning · chatLiquid AI | 8.5B1.7B active | −1.2 | 19.0%1,542 questions | – | dd83f5c |
| 39 | GLiNER2.5 MultiFastino | 290M | −3.0tied with LFM2.5-8B-A1B · default reasoning · chat | 17.6%1,542 questions | 35.8 | 2ca71aa |
| 40 | MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | −18.5 | 5.2%1,542 questions | – | 42433e9 |
1,542 questions per model · updated 16 November 2026 · 0 = chance: 1 of 5 options right on every question (accuracy 20%); 100 = every one of the 1,542 Turkish grammar questions (units 7 to 20) right (accuracy 100%) · feed: bench-scores.json
Reading · English
Decision models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | KnowLine-31B-Gen1.5 LoRA, mergedPelaAI | 31.3B | 92.0 | 93.6%547 questions | 5.0 | 8e56aa1 |
| 2 | Copernicus-31BPoindexter Labs | 31.3B | 91.1tied with KnowLine-31B-Gen1.5 | 92.9%547 questions | 7.8 | be71525 |
| 3 | REAX 1.3 blendREAX | 57.4B | 90.6tied with Copernicus-31B | 92.5%547 questions | 3.6 | fa4bf7e |
| 4 | KnowLine-31B-Gen2PelaAI | 31.3B | 90.4tied with REAX 1.3 | 92.3%547 questions | 7.7 | fd99a9e |
| 5 | Karpo Decision 31BMachinePulse | 31.3B | 90.2tied with KnowLine-31B-Gen2 | 92.1%547 questions | 2.4 | d69cdd5 |
| 6 | Gemini Decision 1 20261111Google | Not disclosed | 88.8tied with Karpo Decision 31B | 91.0%547 questions | 4.5 | no repo |
| 7 | Logits Sigma-1 Duo blendLogits | 57.4B | 88.8tied with Gemini Decision 1 · 20261111 | 91.0%547 questions | 8.0 | a0c1b6e |
| 8 | GLiDE 27BFastino | 26.1B | 88.6tied with Logits Sigma-1 Duo | 90.9%547 questions | 1.7 | 4d32743 |
| 9 | Sieve 31B v3 full fine-tuneSthānika AI | 31.3B | 88.3tied with GLiDE 27B | 90.7%547 questions | 4.3 | 8fcee85 |
| 10 | Turion Jev 31B LoRA, mergedTurion.AI | 31.3B | 88.3tied with Sieve 31B v3 | 90.7%547 questions | 11.6 | 866c0fc |
| 11 | d1-ultraLiquid AI | Not disclosed | 88.3tied with Turion Jev 31B | 90.7%547 questions | 4.4 | no repo |
| 12 | Perplexity Decider 27B v1.2 API · with webPerplexity | 26.1B | 88.1tied with d1-ultra | 90.5%547 questions | 3.6 | no repo |
| 13 | GLiDE v1.2Fastino | 26.1B | 87.9tied with Perplexity Decider 27B v1.2 · API · with web | 90.3%547 questions | 2.7 | a7e4589 |
| 14 | d1.2-ultraLiquid AI | Not disclosed | 87.9tied with GLiDE · v1.2 | 90.3%547 questions | 8.5 | no repo |
| 15 | Mammoth-1 post-trainedBase Labs | 27.4B | 87.7tied with d1.2-ultra | 90.1%547 questions | 2.4 | 61cec44 |
| 16 | d3 fine-tunevLLM Semantic Router | 26.1B | 87.2tied with Mammoth-1 | 89.8%547 questions | 3.6 | 20f37b9 |
| 17 | Drex Max fine-tuneNace.AI | 26.1B | 87.0tied with d3 | 89.6%547 questions | 2.6 | cac65e1 |
| 18 | Perplexity Decider 27B v1.3Perplexity | 26.1B | 87.0tied with Drex Max | 89.6%547 questions | 3.4 | 9c2ebb2 |
| 19 | d3-ultra fine-tunevLLM Semantic Router | 26.1B | 86.7tied with Perplexity Decider 27B v1.3 | 89.4%547 questions | 2.8 | 52ba9e7 |
| 20 | d1.2Liquid AI | Not disclosed | 86.3tied with d3-ultra | 89.0%547 questions | 8.3 | no repo |
| 21 | Perplexity Decider 27B v1.2 DGX SparkPerplexity | 26.1B | 85.8tied with d1.2 | 88.7%547 questions | 3.7 | 225c913 |
| 22 | Drex Max v1.1Nace.AI | Not disclosed | 85.6tied with Perplexity Decider 27B v1.2 · DGX Spark | 88.5%547 questions | 3.8 | no repo |
| 23 | Perplexity Decider 27B v1Perplexity | 26.1B | 85.6tied with Drex Max v1.1 | 88.5%547 questions | 2.6 | no repo |
| 24 | d1.1Liquid AI | Not disclosed | 85.4tied with Perplexity Decider 27B v1 | 88.3%547 questions | 4.2 | no repo |
| 25 | Jeffan-9B-1211tattrongvu | 8.9B | 84.5tied with d1.1 | 87.6%547 questions | 8.9 | 862d1a7 |
| 26 | StartLux-Decision-v1.1-9B-InstantStartLux | 9.7B | 83.5tied with Jeffan-9B-1211 | 86.8%547 questions | 6.6 | 18a17f7 |
| 27 | d1Liquid AI | Not disclosed | 83.5tied with StartLux-Decision-v1.1-9B-Instant | 86.8%547 questions | 2.6 | no repo |
| 28 | Clef post-trainedCloudflare | 27.5B | 83.3tied with d1 | 86.7%547 questions | 3.1 | 2f3de3d |
| 29 | KnowLine-12B-Gen1PelaAI | 12B | 83.3tied with Clef | 86.7%547 questions | 11.1 | f26fbc2 |
| 30 | GLiDE 9BFastino | 7.9B | 82.9tied with KnowLine-12B-Gen1 | 86.3%547 questions | 5.6 | ab54df5 |
| 31 | OneJev-27B full fine-tuneOmniJev | 27.4B | 82.9tied with GLiDE 9B | 86.3%547 questions | 9.7 | a5e5f55 |
| 32 | Jev 1.14TypeSafe | Not disclosed | 82.6tied with OneJev-27B | 86.1%547 questions | 2.2 | no repo |
| 33 | Cerno 9B v1.2 full fine-tuneApertix AI | 8.9B | 82.4tied with Jev 1.14 | 85.9%547 questions | 4.2 | f02a632 |
| 34 | Drex 2.0Nace.AI | 35.1B3B active | 82.4tied with Cerno 9B v1.2 | 85.9%547 questions | 6.9 | no repo |
| 35 | Jev 1.13TypeSafe | Not disclosed | 82.4tied with Drex 2.0 | 85.9%547 questions | 3.1 | no repo |
| 36 | KnowLine-9B-Gen1PelaAI | 9.7B | 82.4tied with Jev 1.13 | 85.9%547 questions | 3.5 | 325378d |
| 37 | SubQ Snap 1.1Subquadratic | Not disclosed | 82.4tied with KnowLine-9B-Gen1 | 85.9%547 questions | 3.2 | no repo |
| 38 | SubQ Snap PreviewSubquadratic | Not disclosed | 82.4tied with SubQ Snap 1.1 | 85.9%547 questions | 3.2 | no repo |
| 39 | Adjudex 9B v5newsbang | 8.4B | 82.2tied with SubQ Snap Preview | 85.7%547 questions | 3.4 | ffdcb82 |
| 40 | Mercury DecideInception | Not disclosed | 82.2tied with Adjudex 9B v5 | 85.7%547 questions | 6.6 | no repo |
| 41 | Microsoft-Decision-1Microsoft | Not disclosed | 82.2tied with Mercury Decide | 85.7%547 questions | 2.8 | no repo |
| 42 | Elephant-1 post-trainedBase Labs | 9.4B | 81.7tied with Microsoft-Decision-1 | 85.4%547 questions | 2.7 | b3f8a93 |
| 43 | Open-Jev-27B-v1.1 LoRA + headZefan Cai | 27.8B | 81.7tied with Elephant-1 | 85.4%547 questions | 45.3 | 28cf730 |
| 44 | KnowLine-9B-Gen2PelaAI | 9.7B | 81.5tied with Open-Jev-27B-v1.1 | 85.2%547 questions | 8.7 | 8ec8eb6 |
| 45 | GPT-6 Luna Decisions APIOpenAI | Not disclosed | 80.1tied with KnowLine-9B-Gen2 | 84.1%547 questions | 3.2 | no repo |
| 46 | celeris-1-decisionCeleris | Not disclosed | 80.1tied with GPT-6 Luna · Decisions API | 84.1%547 questions | 5.2 | no repo |
| 47 | Cygnet unchanged baseBlockbrain Labs | 12B | 79.9tied with celeris-1-decision | 83.9%547 questions | 4.3 | no repo |
| 48 | Winnow-12B LoRA, mergedEldanRing | 11.9B | 79.9tied with Cygnet | 83.9%547 questions | 8.7 | b2b1421 |
| 49 | Perplexity Decider Flash v1.3Perplexity | 4.5B | 79.7tied with Winnow-12B | 83.7%547 questions | 2.7 | d06fb3f |
| 50 | Valen-Decision-1.5-9BValen-Team | 8.4B | 79.7tied with Perplexity Decider Flash v1.3 | 83.7%547 questions | 8.4 | 5a51211 |
| 51 | celeris-1.1-decisionCeleris | Not disclosed | 79.7tied with Valen-Decision-1.5-9B | 83.7%547 questions | 5.0 | no repo |
| 52 | Dexy-Decision-9B-V2 fine-tunePatrick-18 | 8.4B | 79.4tied with celeris-1.1-decision | 83.5%547 questions | 4.2 | 4f434e2 |
| 53 | vera-medium-omniscar-ai | 4.2B | 79.4tied with Dexy-Decision-9B-V2 | 83.5%547 questions | 9.9 | 1f23a6c |
| 54 | StartLux-Decision-v1.1-4B-Instant fine-tuneStartLux | 4.7B | 78.5tied with vera-medium-omni | 82.8%547 questions | 3.2 | 416088f |
| 55 | Clef-Flash post-trainedCloudflare | 9.5B | 77.8tied with StartLux-Decision-v1.1-4B-Instant | 82.3%547 questions | 4.0 | fde727a |
| 56 | Verdin-4B-v1.1 LoRA, mergedvictorbennett | 4.2B | 77.8tied with Clef-Flash | 82.3%547 questions | 3.5 | 45830d2 |
| 57 | Perplexity Decider Flash v1.2Perplexity | 4.5B | 77.4tied with Verdin-4B-v1.1 | 81.9%547 questions | 4.1 | 6d771ad |
| 58 | KnowLine 4B Gen5.5 LoRA, mergedPelaAI | 4.7B | 76.0tied with Perplexity Decider Flash v1.2 | 80.8%547 questions | 5.8 | 779944d |
| 59 | Chinese Decision Pointer 4B LoRA, mergedquincyqiang | 4.2B | 75.5tied with KnowLine 4B Gen5.5 | 80.4%547 questions | 3.5 | 889f23e |
| 60 | Metask-Jev-4B LoRA, mergedMetask Lab | 4.5B | 71.4tied with Chinese Decision Pointer 4B | 77.1%547 questions | 4.4 | ea20fe8 |
| 61 | KnowLine-2B-Gen1 LoRA, mergedPelaAI | 2.3B | 64.1 | 71.3%547 questions | 5.1 | 53245d4 |
| 62 | GLiDE 2BFastino | 1.9B | 62.5tied with KnowLine-2B-Gen1 | 70.0%547 questions | 8.2 | 00f122c |
| 63 | StartLux-Decision-v1.1-2B-Instant fine-tuneStartLux | 2.3B | 62.1tied with GLiDE 2B | 69.7%547 questions | 8.2 | f1e6cd1 |
| 64 | d1-2BLiquid AI | 1.9B | 57.0 | 65.6%547 questions | 7.9 | 10e7671 |
| 65 | GLiDE 0.8BFastino | 750M | 50.9 | 60.7%547 questions | 12.5 | 9eacdee |
| 66 | StartLux-Decision-v1.1-0.8B-Instant fine-tuneStartLux | 870M | 49.5tied with GLiDE 0.8B | 59.6%547 questions | 8.7 | 6496ae6 |
| 67 | GLiDE 0.4BFastino | 400M | 49.3tied with StartLux-Decision-v1.1-0.8B-Instant | 59.4%547 questions | 4.8 | 6bda7ab |
| 68 | OneJev1.6-0.8B full fine-tuneOmniJev | 850M | 49.3tied with GLiDE 0.4B | 59.4%547 questions | 3.7 | bcd88a3 |
| 69 | KnowLine-0.8B-Gen1 LoRA, mergedPelaAI | 870M | 48.6tied with OneJev1.6-0.8B | 58.9%547 questions | 8.1 | 6b27831 |
| 70 | GLiClass v4 Edge fine-tuneKnowledgator | 440M | 47.9tied with KnowLine-0.8B-Gen1 | 58.3%547 questions | 3.8 | 885af1a |
| 71 | Decider 1meraGPT | Not disclosed | 44.7tied with GLiClass v4 Edge | 55.8%547 questions | 5.3 | no repo |
| 72 | Strands Decider 2B LoRA + headStrands Agents | 2.3B | 31.9 | 45.5%547 questions | 10.5 | 2b52a62 |
| 73 | LayaConvai Innovations | 420M | 3.6 | 22.9%547 questions | 12.1 | c5d7873 |
| 74 | GLiNER2.5-multi-Decide fine-tuneFastino | 290M | 1.5tied with Laya | 21.2%547 questions | 27.2 | a35a0cd |
| 75 | Laya MultilingualConvai Innovations | 320M | −5.8 | 15.4%547 questions | 44.4 | c5d7873 |
Chat models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.2 max · chatAnthropic | Not disclosed | 97.5 | 98.0%547 questions | 10.7 | no repo |
| 2 | Gemini 4 Hydrogen high · chatGoogle | Not disclosed | 95.9tied with Claude Fable 5.2 · max · chat | 96.7%547 questions | 3.1 | no repo |
| 3 | GPT-6 Astra high · chatOpenAI | Not disclosed | 95.2tied with Gemini 4 Hydrogen · high · chat | 96.2%547 questions | 1.8 | no repo |
| 4 | GPT-6.1 Terra high · chatOpenAI | Not disclosed | 95.2tied with GPT-6 Astra · high · chat | 96.2%547 questions | 1.3 | no repo |
| 5 | Claude Fable 5.1 high · chatAnthropic | Not disclosed | 95.0tied with GPT-6.1 Terra · high · chat | 96.0%547 questions | 11.8 | no repo |
| 6 | Gemini 3.8 Flash high · chatGoogle | Not disclosed | 95.0tied with Claude Fable 5.1 · high · chat | 96.0%547 questions | 1.7 | no repo |
| 7 | Gemini 4 Argon low · chatGoogle | Not disclosed | 95.0tied with Gemini 3.8 Flash · high · chat | 96.0%547 questions | 1.5 | no repo |
| 8 | GPT-6.1 Sol high · chatOpenAI | Not disclosed | 94.5tied with Gemini 4 Argon · low · chat | 95.6%547 questions | 2.4 | no repo |
| 9 | Claude Opus 5.5 high · chatAnthropic | Not disclosed | 93.1tied with GPT-6.1 Sol · high · chat | 94.5%547 questions | 13.3 | no repo |
| 10 | Qwen4-Max high · chatAlibaba | Not disclosed | 93.1tied with Claude Opus 5.5 · high · chat | 94.5%547 questions | 8.5 | no repo |
| 11 | Gemini 4 Carbon high · chatGoogle | Not disclosed | 92.9tied with Qwen4-Max · high · chat | 94.3%547 questions | 3.1 | no repo |
| 12 | Grok 4.7 high · chatxAI | Not disclosed | 91.8tied with Gemini 4 Carbon · high · chat | 93.4%547 questions | 11.1 | no repo |
| 13 | Claude Haiku 5.5 max · chatAnthropic | Not disclosed | 90.9tied with Grok 4.7 · high · chat | 92.7%547 questions | 18.4 | no repo |
| 14 | DeepSeek-V4.1-Flash reasoning on · chat · 1017DeepSeek | 552B16B active | 90.6tied with Claude Haiku 5.5 · max · chat | 92.5%547 questions | 7.5 | no repo |
| 15 | Kimi K3.1 Pro low · chatMoonshot | 2.8T104B active | 90.4tied with DeepSeek-V4.1-Flash · reasoning on · chat · 1017 | 92.3%547 questions | 5.7 | no repo |
| 16 | Qwen3.5-397B-A17B high · chatAlibaba | 397B17B active | 89.3tied with Kimi K3.1 Pro · low · chat | 91.4%547 questions | 3.3 | no repo |
| 17 | Gemma 4 31B reasoning off · chatGoogle | 31.3B | 88.1tied with Qwen3.5-397B-A17B · high · chat | 90.5%547 questions | 8.5 | 842da37 |
| 18 | GLM-5.3-Flash reasoning on · chatZ.ai | 320B18B active | 87.0tied with Gemma 4 31B · reasoning off · chat | 89.6%547 questions | 7.0 | no repo |
| 19 | Muse Ember max · chatMeta | Not disclosed | 87.0tied with GLM-5.3-Flash · reasoning on · chat | 89.6%547 questions | 7.0 | no repo |
| 20 | MiMo-V2.6-Flash high · chatXiaomi | 309B15B active | 86.3tied with Muse Ember · max · chat | 89.0%547 questions | 6.5 | no repo |
| 21 | Qwen3.8-27B low · chatAlibaba | 27.8B | 86.3tied with MiMo-V2.6-Flash · high · chat | 89.0%547 questions | 11.0 | 1d4bf0f |
| 22 | Inkling-Small high · chat · Jul 2026Thinking Machines | 266B12B active | 85.4tied with Qwen3.8-27B · low · chat | 88.3%547 questions | 2.6 | no repo |
| 23 | GLM-5.3 low · chatZ.ai | 753.3B | 84.7tied with Inkling-Small · high · chat · Jul 2026 | 87.8%547 questions | 3.6 | no repo |
| 24 | Qwen3.5-4B reasoning on · chatAlibaba | 4.7B | 83.8tied with GLM-5.3 · low · chat | 87.0%547 questions | – | 851bf6e |
| 25 | Qwen3.8 Flash no reasoning · chatAlibaba | Not disclosed | 83.8tied with Qwen3.5-4B · reasoning on · chat | 87.0%547 questions | 5.5 | no repo |
| 26 | Gemma 4 12B Unified default reasoning · chatGoogle | 12B | 83.5tied with Qwen3.8 Flash · no reasoning · chat | 86.8%547 questions | – | 43c9d01 |
| 27 | GPT-6 Luna low · chatOpenAI | Not disclosed | 82.9tied with Gemma 4 12B Unified · default reasoning · chat | 86.3%547 questions | 10.1 | no repo |
| 28 | Claude Sonnet 5.5 high · chatAnthropic | Not disclosed | 82.6tied with GPT-6 Luna · low · chat | 86.1%547 questions | 13.9 | no repo |
| 29 | MiniMax-M3.1-Flash high · chatMiniMax | 428B23B active | 80.6tied with Claude Sonnet 5.5 · high · chat | 84.5%547 questions | 9.0 | no repo |
| 30 | LFM2.5-24B-A2B default reasoning · chatLiquid AI | 24B2.5B active | 78.3tied with MiniMax-M3.1-Flash · high · chat | 82.6%547 questions | – | 1769516 |
| 31 | Gemma 4 E4B default reasoning · chatGoogle | 8B | 78.1tied with LFM2.5-24B-A2B · default reasoning · chat | 82.4%547 questions | – | 4ecc2ba |
| 32 | Qwen3.5-4B reasoning off · chatAlibaba | 4.7B | 75.5tied with Gemma 4 E4B · default reasoning · chat | 80.4%547 questions | – | 851bf6e |
| 33 | Qwen3-14B reasoning off · chatAlibaba | 14.8B | 74.6tied with Qwen3.5-4B · reasoning off · chat | 79.7%547 questions | 17.8 | 40c0698 |
| 34 | MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | 72.7tied with Qwen3-14B · reasoning off · chat | 78.1%547 questions | – | 42433e9 |
| 35 | Gemma 4 E2B reasoning on · chatGoogle | 5.1B | 70.3tied with MiniCPM5-2B · reasoning off · chat | 76.2%547 questions | – | 4adf157 |
| 36 | Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | 69.8tied with Gemma 4 E2B · reasoning on · chat | 75.9%547 questions | – | 15852e8 |
| 37 | Gemma 4 E2B default reasoning · chatGoogle | 5.1B | 68.2tied with Qwen3.5-2B · reasoning off · chat | 74.6%547 questions | – | 4adf157 |
| 38 | Granite-4.2-3B reasoning on · chatIBM | 3.7B | 68.0tied with Gemma 4 E2B · default reasoning · chat | 74.4%547 questions | – | 4e9c275 |
| 39 | LFM2.5-2.6B default reasoning · chatLiquid AI | 2.7B | 67.3tied with Granite-4.2-3B · reasoning on · chat | 73.9%547 questions | – | 7141b56 |
| 40 | Erk-14B reasoning off · chateCloud Technology | 14.8B | 65.0tied with LFM2.5-2.6B · default reasoning · chat | 72.0%547 questions | 10.9 | 8584b14 |
| 41 | LFM3-1.7B reasoning on · chatLiquid AI | 1.7B | 64.7tied with Erk-14B · reasoning off · chat | 71.8%547 questions | – | 0dc4268 |
| 42 | LFM3-1.7B reasoning off · chatLiquid AI | 1.7B | 63.9tied with LFM3-1.7B · reasoning on · chat | 71.1%547 questions | – | 0dc4268 |
| 43 | LFM2.5-8B-A1B default reasoning · chatLiquid AI | 8.5B1.7B active | 63.0tied with LFM3-1.7B · reasoning off · chat | 70.4%547 questions | – | dd83f5c |
| 44 | MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | 59.7tied with LFM2.5-8B-A1B · default reasoning · chat | 67.8%547 questions | – | 42433e9 |
| 45 | Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | 59.1tied with MiniCPM5-2B · reasoning off · chat | 67.3%547 questions | – | 15852e8 |
| 46 | Raon-1 nano reasoning off · chatKRAFTON | 3.5B | 42.2 | 53.7%547 questions | – | fcf2d1d |
| 47 | GLiNER2.5 MultiFastino | 290M | −4.7 | 16.3%547 questions | 36.4 | 2ca71aa |
547 questions per model · updated 16 November 2026 · 0 = chance: 1 of 5 options right on every question (accuracy 20%); 100 = every one of the 547 kept reading questions, translated into English, right (accuracy 100%) · feed: bench-scores.json
Reading · Vision
Decision models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Gemini Decision 1 20261111Google | Not disclosed | 58.1 | 66.4%763 questions | 5.2 | no repo |
| 2 | d1-ultraLiquid AI | Not disclosed | 37.6 | 50.1%763 questions | 3.5 | no repo |
| 3 | GLiDE 27BFastino | 26.1B | 37.4tied with d1-ultra | 49.9%763 questions | 4.3 | 4d32743 |
| 4 | KnowLine-31B-Gen1.5 LoRA, mergedPelaAI | 31.3B | 37.3tied with GLiDE 27B | 49.8%763 questions | 6.1 | 8e56aa1 |
| 5 | d3 fine-tunevLLM Semantic Router | 26.1B | 35.6tied with KnowLine-31B-Gen1.5 | 48.5%763 questions | 3.7 | 20f37b9 |
| 6 | d1.2Liquid AI | Not disclosed | 33.7tied with d3 | 46.9%763 questions | 5.6 | no repo |
| 7 | Drex Max fine-tuneNace.AI | 26.1B | 33.5tied with d1.2 | 46.8%763 questions | 9.3 | cac65e1 |
| 8 | SubQ Snap 1.1Subquadratic | Not disclosed | 32.2tied with Drex Max | 45.7%763 questions | 11.7 | no repo |
| 9 | d3-ultra fine-tunevLLM Semantic Router | 26.1B | 31.8tied with SubQ Snap 1.1 | 45.5%763 questions | 18.8 | 52ba9e7 |
| 10 | d1.2-ultraLiquid AI | Not disclosed | 31.7tied with d3-ultra | 45.3%763 questions | 2.8 | no repo |
| 11 | Drex Max v1.1Nace.AI | Not disclosed | 31.4tied with d1.2-ultra | 45.1%763 questions | 5.2 | no repo |
| 12 | Jev 1.14TypeSafe | Not disclosed | 29.1tied with Drex Max v1.1 | 43.3%763 questions | 3.7 | no repo |
| 13 | OneJev-27B full fine-tuneOmniJev | 27.4B | 26.6tied with Jev 1.14 | 41.3%763 questions | 4.9 | a5e5f55 |
| 14 | GPT-6 Luna Decisions APIOpenAI | Not disclosed | 15.8 | 32.6%763 questions | 5.7 | no repo |
| 15 | KnowLine 4B Gen5.5 LoRA, mergedPelaAI | 4.7B | 12.7tied with GPT-6 Luna · Decisions API | 30.1%763 questions | 11.0 | 779944d |
| 16 | OneJev1.6-0.8B full fine-tuneOmniJev | 850M | 8.1 | 26.5%763 questions | 4.2 | bcd88a3 |
| 17 | StartLux-Decision-v1.1-0.8B-Instant fine-tuneStartLux | 870M | 5.3tied with OneJev1.6-0.8B | 24.2%763 questions | 1.7 | 6496ae6 |
| 18 | KnowLine-0.8B-Gen1 LoRA, mergedPelaAI | 870M | 4.8tied with StartLux-Decision-v1.1-0.8B-Instant | 23.9%763 questions | 1.5 | 6b27831 |
| 19 | StartLux-Decision-v1.1-2B-Instant fine-tuneStartLux | 2.3B | 4.3tied with KnowLine-0.8B-Gen1 | 23.5%763 questions | 3.6 | f1e6cd1 |
| 20 | d1-2BLiquid AI | 1.9B | 4.2tied with StartLux-Decision-v1.1-2B-Instant | 23.3%763 questions | 1.1 | 10e7671 |
Chat models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Muse Ember max · chatMeta | Not disclosed | 82.8 | 86.2%763 questions | – | no repo |
| 2 | Gemma 4 E4B default reasoning · chatGoogle | 8B | 28.6 | 42.9%763 questions | – | 4ecc2ba |
763 questions per model · updated 16 November 2026 · 0 = chance: 1 of each question's options right on every question (accuracy 20% with 5 options); 100 = every one of the 763 questions that depend on the printed page right, read from the page image alone (accuracy 100%) · feed: bench-scores.json
Logic
Decision models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Karpo Decision 31BMachinePulse | 31.3B | 50.9 | 60.7%1,125 questions | 5.7 | d69cdd5 |
| 2 | REAX 1.3 blendREAX | 57.4B | 49.1tied with Karpo Decision 31B | 59.3%1,125 questions | 3.7 | fa4bf7e |
| 3 | KnowLine-31B-Gen1.5 LoRA, mergedPelaAI | 31.3B | 48.8tied with REAX 1.3 | 59.0%1,125 questions | 2.6 | 8e56aa1 |
| 4 | KnowLine-31B-Gen2PelaAI | 31.3B | 48.7tied with KnowLine-31B-Gen1.5 | 58.9%1,125 questions | 3.7 | fd99a9e |
| 5 | Copernicus-31BPoindexter Labs | 31.3B | 48.3tied with KnowLine-31B-Gen2 | 58.7%1,125 questions | 3.9 | be71525 |
| 6 | GLiDE 27BFastino | 26.1B | 48.0tied with Copernicus-31B | 58.4%1,125 questions | 4.4 | 4d32743 |
| 7 | Sieve 31B v3 full fine-tuneSthānika AI | 31.3B | 47.3tied with GLiDE 27B | 57.9%1,125 questions | 2.1 | 8fcee85 |
| 8 | Logits Sigma-1 Duo blendLogits | 57.4B | 47.2tied with Sieve 31B v3 | 57.8%1,125 questions | 5.0 | a0c1b6e |
| 9 | GLiDE v1.2Fastino | 26.1B | 44.0 | 55.2%1,125 questions | 5.9 | a7e4589 |
| 10 | d1-ultraLiquid AI | Not disclosed | 43.7tied with GLiDE · v1.2 | 54.9%1,125 questions | 3.2 | no repo |
| 11 | Drex Max v1.1Nace.AI | Not disclosed | 43.4tied with d1-ultra | 54.8%1,125 questions | 6.1 | no repo |
| 12 | Perplexity Decider 27B v1.3Perplexity | 26.1B | 43.4tied with Drex Max v1.1 | 54.8%1,125 questions | 9.0 | 9c2ebb2 |
| 13 | d3-ultra fine-tunevLLM Semantic Router | 26.1B | 43.2tied with Perplexity Decider 27B v1.3 | 54.6%1,125 questions | 15.1 | 52ba9e7 |
| 14 | GLiDE APIFastino | Not disclosed | 43.1tied with d3-ultra | 54.5%1,125 questions | 6.6 | no repo |
| 15 | Mammoth-1 post-trainedBase Labs | 27.4B | 43.1tied with GLiDE · API | 54.5%1,125 questions | 5.3 | 61cec44 |
| 16 | d3 fine-tunevLLM Semantic Router | 26.1B | 43.1tied with Mammoth-1 | 54.5%1,125 questions | 4.6 | 20f37b9 |
| 17 | Perplexity Decider 27B v1.2 DGX SparkPerplexity | 26.1B | 41.7tied with d3 | 53.3%1,125 questions | 3.4 | no repo |
| 18 | Drex Max fine-tuneNace.AI | 26.1B | 41.3tied with Perplexity Decider 27B v1.2 · DGX Spark | 53.1%1,125 questions | 16.2 | cac65e1 |
| 19 | Perplexity Decider 27B v1.2 API · with webPerplexity | 26.1B | 40.7tied with Drex Max | 52.5%1,125 questions | 4.2 | no repo |
| 20 | SubQ Snap 1.1Subquadratic | Not disclosed | 40.4tied with Perplexity Decider 27B v1.2 · API · with web | 52.4%1,125 questions | 12.9 | no repo |
| 21 | Turion Jev 31B LoRA, mergedTurion.AI | 31.3B | 40.3tied with SubQ Snap 1.1 | 52.3%1,125 questions | 2.9 | 866c0fc |
| 22 | d1.2-ultraLiquid AI | Not disclosed | 39.4tied with Turion Jev 31B | 51.6%1,125 questions | 3.0 | no repo |
| 23 | KnowLine-12B-Gen1PelaAI | 12B | 38.2tied with d1.2-ultra | 50.6%1,125 questions | 4.8 | f26fbc2 |
| 24 | Gemini Decision 1 20261111Google | Not disclosed | 37.6tied with KnowLine-12B-Gen1 | 50.0%1,125 questions | 3.1 | no repo |
| 25 | d1.2Liquid AI | Not disclosed | 36.4tied with Gemini Decision 1 · 20261111 | 49.2%1,125 questions | 2.6 | no repo |
| 26 | Jeffan-9B-1211tattrongvu | 8.9B | 35.2tied with d1.2 | 48.2%1,125 questions | 4.0 | 862d1a7 |
| 27 | Jev 1.14TypeSafe | Not disclosed | 35.2tied with Jeffan-9B-1211 | 48.2%1,125 questions | 3.3 | no repo |
| 28 | GLiDE 9BFastino | 7.9B | 34.8tied with Jev 1.14 | 47.8%1,125 questions | 6.7 | ab54df5 |
| 29 | celeris-1-decisionCeleris | Not disclosed | 34.1tied with GLiDE 9B | 47.3%1,125 questions | 10.7 | no repo |
| 30 | celeris-1.1-decisionCeleris | Not disclosed | 34.1tied with celeris-1-decision | 47.3%1,125 questions | 9.8 | no repo |
| 31 | StartLux-Decision-v1.1-9B-InstantStartLux | 9.7B | 33.2tied with celeris-1.1-decision | 46.6%1,125 questions | 3.8 | 18a17f7 |
| 32 | Adjudex 9B v5newsbang | 8.4B | 33.0tied with StartLux-Decision-v1.1-9B-Instant | 46.4%1,125 questions | 7.2 | ffdcb82 |
| 33 | Elephant-1 post-trainedBase Labs | 9.4B | 33.0tied with Adjudex 9B v5 | 46.4%1,125 questions | 8.6 | b3f8a93 |
| 34 | Jev 1.13TypeSafe | Not disclosed | 32.9tied with Elephant-1 | 46.3%1,125 questions | 5.9 | no repo |
| 35 | d1.1Liquid AI | Not disclosed | 32.6tied with Jev 1.13 | 46.0%1,125 questions | 4.4 | no repo |
| 36 | StartLux-Decision-v1.1-4B-Instant fine-tuneStartLux | 4.7B | 32.3tied with d1.1 | 45.9%1,125 questions | 3.3 | 416088f |
| 37 | Dexy-Decision-9B-V2 fine-tunePatrick-18 | 8.4B | 31.7tied with StartLux-Decision-v1.1-4B-Instant | 45.3%1,125 questions | 14.9 | 4f434e2 |
| 38 | Mercury DecideInception | Not disclosed | 31.3tied with Dexy-Decision-9B-V2 | 45.1%1,125 questions | 15.4 | no repo |
| 39 | Perplexity Decider Flash v1.3Perplexity | 4.5B | 30.9tied with Mercury Decide | 44.7%1,125 questions | 10.1 | d06fb3f |
| 40 | Microsoft-Decision-1Microsoft | Not disclosed | 30.6tied with Perplexity Decider Flash v1.3 | 44.4%1,125 questions | 8.0 | no repo |
| 41 | Valen-Decision-1.5-9BValen-Team | 8.4B | 30.4tied with Microsoft-Decision-1 | 44.4%1,125 questions | 5.8 | 5a51211 |
| 42 | KnowLine-9B-Gen1PelaAI | 9.7B | 30.3tied with Valen-Decision-1.5-9B | 44.3%1,125 questions | 3.5 | 325378d |
| 43 | vera-medium-omniscar-ai | 4.2B | 30.1tied with KnowLine-9B-Gen1 | 44.1%1,125 questions | 4.7 | 1f23a6c |
| 44 | KnowLine-9B-Gen2PelaAI | 9.7B | 29.9tied with vera-medium-omni | 43.9%1,125 questions | 3.2 | 8ec8eb6 |
| 45 | Cerno 9B v1.2 full fine-tuneApertix AI | 8.9B | 29.1tied with KnowLine-9B-Gen2 | 43.3%1,125 questions | 17.0 | f02a632 |
| 46 | Perplexity Decider Flash v1.2Perplexity | 4.5B | 28.9tied with Cerno 9B v1.2 | 43.1%1,125 questions | 6.0 | 6d771ad |
| 47 | Open-Jev-27B-v1.1 LoRA + headZefan Cai | 27.8B | 28.2tied with Perplexity Decider Flash v1.2 | 42.6%1,125 questions | 12.9 | 28cf730 |
| 48 | KnowLine 4B Gen5.5 LoRA, mergedPelaAI | 4.7B | 27.9tied with Open-Jev-27B-v1.1 | 42.3%1,125 questions | 13.7 | 779944d |
| 49 | Drex 2.0Nace.AI | 35.1B3B active | 27.3tied with KnowLine 4B Gen5.5 | 41.9%1,125 questions | 4.3 | no repo |
| 50 | Clef post-trainedCloudflare | 27.5B | 26.0tied with Drex 2.0 | 40.8%1,125 questions | 15.7 | 2f3de3d |
| 51 | d1Liquid AI | Not disclosed | 26.0tied with Clef | 40.8%1,125 questions | 10.9 | no repo |
| 52 | OneJev-27B full fine-tuneOmniJev | 27.4B | 25.8tied with d1 | 40.6%1,125 questions | 4.6 | a5e5f55 |
| 53 | Chinese Decision Pointer 4B LoRA, mergedquincyqiang | 4.2B | 25.4tied with OneJev-27B | 40.4%1,125 questions | 15.5 | 889f23e |
| 54 | Verdin-4B-v1.1 LoRA, mergedvictorbennett | 4.2B | 24.7tied with Chinese Decision Pointer 4B | 39.7%1,125 questions | 10.8 | 45830d2 |
| 55 | Winnow-12B LoRA, mergedEldanRing | 11.9B | 24.7tied with Verdin-4B-v1.1 | 39.7%1,125 questions | 14.4 | b2b1421 |
| 56 | Cygnet unchanged baseBlockbrain Labs | 12B | 24.1tied with Winnow-12B | 39.3%1,125 questions | 11.2 | no repo |
| 57 | Clef-Flash post-trainedCloudflare | 9.5B | 23.7tied with Cygnet | 38.9%1,125 questions | 11.5 | fde727a |
| 58 | GLiDE 2BFastino | 1.9B | 15.3 | 32.3%1,125 questions | 16.2 | 00f122c |
| 59 | Metask-Jev-4B LoRA, mergedMetask Lab | 4.5B | 15.1tied with GLiDE 2B | 32.1%1,125 questions | 2.5 | ea20fe8 |
| 60 | KnowLine-2B-Gen1 LoRA, mergedPelaAI | 2.3B | 14.7tied with Metask-Jev-4B | 31.7%1,125 questions | 8.3 | 53245d4 |
| 61 | StartLux-Decision-v1.1-2B-Instant fine-tuneStartLux | 2.3B | 13.4tied with KnowLine-2B-Gen1 | 30.8%1,125 questions | 5.3 | f1e6cd1 |
| 62 | d1-2BLiquid AI | 1.9B | 13.3tied with StartLux-Decision-v1.1-2B-Instant | 30.7%1,125 questions | 6.9 | 10e7671 |
| 63 | GLiDE 0.8BFastino | 750M | 11.4tied with d1-2B | 29.2%1,125 questions | 14.0 | 9eacdee |
| 64 | OneJev1.6-0.8B full fine-tuneOmniJev | 850M | 8.8tied with GLiDE 0.8B | 27.0%1,125 questions | 11.2 | bcd88a3 |
| 65 | KnowLine-0.8B-Gen1 LoRA, mergedPelaAI | 870M | 8.3tied with OneJev1.6-0.8B | 26.7%1,125 questions | 8.4 | 6b27831 |
| 66 | StartLux-Decision-v1.1-0.8B-Instant fine-tuneStartLux | 870M | 5.9tied with KnowLine-0.8B-Gen1 | 24.7%1,125 questions | 7.8 | 6496ae6 |
| 67 | GLiClass v4 Edge fine-tuneKnowledgator | 440M | 4.4tied with StartLux-Decision-v1.1-0.8B-Instant | 23.6%1,125 questions | 8.7 | 885af1a |
| 68 | GLiDE 0.4BFastino | 400M | 4.0tied with GLiClass v4 Edge | 23.2%1,125 questions | 8.8 | 6bda7ab |
| 69 | Strands Decider 2B LoRA + headStrands Agents | 2.3B | 3.0tied with GLiDE 0.4B | 22.4%1,125 questions | 12.0 | 2b52a62 |
Chat models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra high · chatOpenAI | Not disclosed | 95.9 | 96.7%1,125 questions | – | no repo |
| 2 | Gemini 4 Carbon high · chatGoogle | Not disclosed | 95.9tied with GPT-6 Astra · high · chat | 96.7%1,125 questions | – | no repo |
| 3 | GPT-6.1 Sol high · chatOpenAI | Not disclosed | 95.7tied with Gemini 4 Carbon · high · chat | 96.5%1,125 questions | – | no repo |
| 4 | Qwen4-Max high · chatAlibaba | Not disclosed | 95.3tied with GPT-6.1 Sol · high · chat | 96.3%1,125 questions | – | no repo |
| 5 | Claude Opus 5.5 high · chatAnthropic | Not disclosed | 95.2tied with Qwen4-Max · high · chat | 96.2%1,125 questions | – | no repo |
| 6 | DeepSeek-V4.1-Flash reasoning on · chat · 1017DeepSeek | 552B16B active | 95.2tied with Claude Opus 5.5 · high · chat | 96.2%1,125 questions | – | no repo |
| 7 | Gemini 4 Hydrogen high · chatGoogle | Not disclosed | 95.1tied with DeepSeek-V4.1-Flash · reasoning on · chat · 1017 | 96.1%1,125 questions | – | no repo |
| 8 | Inkling 2 Small high · chatThinking Machines | 594B17B active | 94.6tied with Gemini 4 Hydrogen · high · chat | 95.6%1,125 questions | – | no repo |
| 9 | Claude Sonnet 5.5 high · chatAnthropic | Not disclosed | 94.3tied with Inkling 2 Small · high · chat | 95.5%1,125 questions | – | no repo |
| 10 | Muse Ember max · chatMeta | Not disclosed | 94.3tied with Claude Sonnet 5.5 · high · chat | 95.5%1,125 questions | – | no repo |
| 11 | Inkling-Small high · chat · Jul 2026Thinking Machines | 266B12B active | 94.2tied with Muse Ember · max · chat | 95.4%1,125 questions | – | no repo |
| 12 | Claude Fable 5.1 high · chatAnthropic | Not disclosed | 94.0tied with Inkling-Small · high · chat · Jul 2026 | 95.2%1,125 questions | – | no repo |
| 13 | MiMo-V2.6-Flash high · chatXiaomi | 309B15B active | 93.9tied with Claude Fable 5.1 · high · chat | 95.1%1,125 questions | – | no repo |
| 14 | Qwen3.5-397B-A17B high · chatAlibaba | 397B17B active | 93.9tied with MiMo-V2.6-Flash · high · chat | 95.1%1,125 questions | – | no repo |
| 15 | Qwen3.8-27B low · chatAlibaba | 27.8B | 92.6tied with Qwen3.5-397B-A17B · high · chat | 94.0%1,125 questions | – | 1d4bf0f |
| 16 | GPT-6.1 Terra high · chatOpenAI | Not disclosed | 92.4tied with Qwen3.8-27B · low · chat | 94.0%1,125 questions | – | no repo |
| 17 | MiniMax-M3.1-Flash high · chatMiniMax | 428B23B active | 92.4tied with GPT-6.1 Terra · high · chat | 94.0%1,125 questions | – | no repo |
| 18 | Qwen3.8 Flash no reasoning · chatAlibaba | Not disclosed | 92.4tied with MiniMax-M3.1-Flash · high · chat | 94.0%1,125 questions | – | no repo |
| 19 | Gemma 4 31B reasoning off · chatGoogle | 31.3B | 91.7tied with Qwen3.8 Flash · no reasoning · chat | 93.3%1,125 questions | – | 842da37 |
| 20 | GLM-5.3-Flash reasoning on · chatZ.ai | 320B18B active | 91.4tied with Gemma 4 31B · reasoning off · chat | 93.2%1,125 questions | – | no repo |
| 21 | Qwen3.5-4B reasoning on · chatAlibaba | 4.7B | 90.4tied with GLM-5.3-Flash · reasoning on · chat | 92.4%1,125 questions | – | 851bf6e |
| 22 | Granite-4.2-3B reasoning on · chatIBM | 3.7B | 89.6tied with Qwen3.5-4B · reasoning on · chat | 91.6%1,125 questions | – | 4e9c275 |
| 23 | MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | 86.1 | 88.9%1,125 questions | – | 42433e9 |
| 24 | Gemma 4 12B Unified default reasoning · chatGoogle | 12B | 84.7tied with MiniCPM5-2B · reasoning off · chat | 87.7%1,125 questions | – | 43c9d01 |
| 25 | LFM2.5-24B-A2B default reasoning · chatLiquid AI | 24B2.5B active | 83.3tied with Gemma 4 12B Unified · default reasoning · chat | 86.7%1,125 questions | – | 1769516 |
| 26 | LFM3-1.7B reasoning on · chatLiquid AI | 1.7B | 81.9tied with LFM2.5-24B-A2B · default reasoning · chat | 85.5%1,125 questions | – | 0dc4268 |
| 27 | Gemma 4 E4B default reasoning · chatGoogle | 8B | 79.0tied with LFM3-1.7B · reasoning on · chat | 83.2%1,125 questions | – | 4ecc2ba |
| 28 | LFM2.5-2.6B default reasoning · chatLiquid AI | 2.7B | 69.2 | 75.4%1,125 questions | – | 7141b56 |
| 29 | Gemma 4 E2B reasoning on · chatGoogle | 5.1B | 66.1 | 72.9%1,125 questions | – | 4adf157 |
| 30 | Qwen3-14B reasoning off · chatAlibaba | 14.8B | 56.7 | 65.3%1,125 questions | – | 40c0698 |
| 31 | Raon-1 nano reasoning off · chatKRAFTON | 3.5B | 56.3tied with Qwen3-14B · reasoning off · chat | 65.1%1,125 questions | – | fcf2d1d |
| 32 | Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | 56.0tied with Raon-1 nano · reasoning off · chat | 64.8%1,125 questions | – | 15852e8 |
| 33 | LFM2.5-8B-A1B default reasoning · chatLiquid AI | 8.5B1.7B active | 50.2 | 60.2%1,125 questions | – | dd83f5c |
| 34 | GPT-6 Luna low · chatOpenAI | Not disclosed | 47.4tied with LFM2.5-8B-A1B · default reasoning · chat | 58.0%1,125 questions | – | no repo |
| 35 | Erk-14B reasoning off · chateCloud Technology | 14.8B | 42.8 | 54.2%1,125 questions | – | 8584b14 |
1,125 questions per model · updated 16 November 2026 · 0 = chance: 1 of 5 options right on every question (accuracy 20%); 100 = every one of the 1,125 Logic questions right (accuracy 100%) · feed: bench-scores.json
Math
Decision models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Karpo Decision 31BMachinePulse | 31.3B | 76.9 | 81.6%2,163 questions | 1.2 | d69cdd5 |
| 2 | KnowLine-31B-Gen2PelaAI | 31.3B | 75.3tied with Karpo Decision 31B | 80.3%2,163 questions | 3.6 | fd99a9e |
| 3 | REAX 1.3 blendREAX | 57.4B | 72.5 | 78.0%2,163 questions | 7.1 | fa4bf7e |
| 4 | KnowLine-31B-Gen1.5 LoRA, mergedPelaAI | 31.3B | 71.7tied with REAX 1.3 | 77.4%2,163 questions | 3.6 | 8e56aa1 |
| 5 | Copernicus-31BPoindexter Labs | 31.3B | 71.7tied with KnowLine-31B-Gen1.5 | 77.3%2,163 questions | 6.5 | be71525 |
| 6 | Sieve 31B v3 full fine-tuneSthānika AI | 31.3B | 71.4tied with Copernicus-31B | 77.1%2,163 questions | 4.3 | 8fcee85 |
| 7 | Logits Sigma-1 Duo blendLogits | 57.4B | 69.0 | 75.2%2,163 questions | 7.5 | a0c1b6e |
| 8 | d1-ultraLiquid AI | Not disclosed | 67.6tied with Logits Sigma-1 Duo | 74.1%2,163 questions | 2.3 | no repo |
| 9 | Mammoth-1 post-trainedBase Labs | 27.4B | 67.5tied with d1-ultra | 74.0%2,163 questions | 3.0 | 61cec44 |
| 10 | Perplexity Decider 27B v1.3Perplexity | 26.1B | 65.6tied with Mammoth-1 | 72.5%2,163 questions | 4.5 | 9c2ebb2 |
| 11 | GLiDE 27BFastino | 26.1B | 65.0tied with Perplexity Decider 27B v1.3 | 72.0%2,163 questions | 4.1 | 4d32743 |
| 12 | d3-ultra fine-tunevLLM Semantic Router | 26.1B | 64.4tied with GLiDE 27B | 71.5%2,163 questions | 8.6 | 52ba9e7 |
| 13 | Gemini Decision 1 20261111Google | Not disclosed | 64.2tied with d3-ultra | 71.3%2,163 questions | 2.5 | no repo |
| 14 | GLiDE APIFastino | Not disclosed | 63.5tied with Gemini Decision 1 · 20261111 | 70.8%2,163 questions | 5.0 | no repo |
| 15 | d3 fine-tunevLLM Semantic Router | 26.1B | 63.0tied with GLiDE · API | 70.4%2,163 questions | 4.5 | 20f37b9 |
| 16 | Drex Max fine-tuneNace.AI | 26.1B | 62.9tied with d3 | 70.3%2,163 questions | 11.8 | cac65e1 |
| 17 | Drex Max v1.1Nace.AI | Not disclosed | 62.6tied with Drex Max | 70.1%2,163 questions | 5.7 | no repo |
| 18 | celeris-1-decisionCeleris | Not disclosed | 62.5tied with Drex Max v1.1 | 70.0%2,163 questions | 4.3 | no repo |
| 19 | GLiDE v1.2Fastino | 26.1B | 62.3tied with celeris-1-decision | 69.9%2,163 questions | 5.2 | a7e4589 |
| 20 | celeris-1.1-decisionCeleris | Not disclosed | 62.3tied with GLiDE · v1.2 | 69.9%2,163 questions | 4.5 | no repo |
| 21 | d1.2-ultraLiquid AI | Not disclosed | 62.3tied with celeris-1.1-decision | 69.9%2,163 questions | 4.6 | no repo |
| 22 | Perplexity Decider 27B v1.2 DGX SparkPerplexity | 26.1B | 62.2tied with d1.2-ultra | 69.8%2,163 questions | 3.2 | no repo |
| 23 | Perplexity Decider 27B v1.2 API · with webPerplexity | 26.1B | 61.6tied with Perplexity Decider 27B v1.2 · DGX Spark | 69.3%2,163 questions | 3.7 | no repo |
| 24 | Turion Jev 31B LoRA, mergedTurion.AI | 31.3B | 61.3tied with Perplexity Decider 27B v1.2 · API · with web | 69.0%2,163 questions | 4.7 | 866c0fc |
| 25 | SubQ Snap 1.1Subquadratic | Not disclosed | 60.7tied with Turion Jev 31B | 68.6%2,163 questions | 9.5 | no repo |
| 26 | KnowLine-12B-Gen1PelaAI | 12B | 57.8 | 66.3%2,163 questions | 4.0 | f26fbc2 |
| 27 | d1.2Liquid AI | Not disclosed | 56.9tied with KnowLine-12B-Gen1 | 65.5%2,163 questions | 3.2 | no repo |
| 28 | d1.1Liquid AI | Not disclosed | 56.5tied with d1.2 | 65.2%2,163 questions | 2.2 | no repo |
| 29 | Jev 1.14TypeSafe | Not disclosed | 56.4tied with d1.1 | 65.1%2,163 questions | 2.8 | no repo |
| 30 | Jev 1.13TypeSafe | Not disclosed | 55.1tied with Jev 1.14 | 64.1%2,163 questions | 2.6 | no repo |
| 31 | Adjudex 9B v5newsbang | 8.4B | 52.6 | 62.1%2,163 questions | 4.8 | ffdcb82 |
| 32 | Elephant-1 post-trainedBase Labs | 9.4B | 51.7tied with Adjudex 9B v5 | 61.3%2,163 questions | 4.3 | b3f8a93 |
| 33 | KnowLine-9B-Gen2PelaAI | 9.7B | 50.1tied with Elephant-1 | 60.1%2,163 questions | 4.8 | 8ec8eb6 |
| 34 | Jeffan-9B-1211tattrongvu | 8.9B | 49.8tied with KnowLine-9B-Gen2 | 59.8%2,163 questions | 3.8 | 862d1a7 |
| 35 | KnowLine-9B-Gen1PelaAI | 9.7B | 49.7tied with Jeffan-9B-1211 | 59.8%2,163 questions | 3.1 | 325378d |
| 36 | Microsoft-Decision-1Microsoft | Not disclosed | 49.2tied with KnowLine-9B-Gen1 | 59.4%2,163 questions | 4.1 | no repo |
| 37 | Drex 2.0Nace.AI | 35.1B3B active | 48.7tied with Microsoft-Decision-1 | 59.0%2,163 questions | 1.9 | no repo |
| 38 | Dexy-Decision-9B-V2 fine-tunePatrick-18 | 8.4B | 48.5tied with Drex 2.0 | 58.8%2,163 questions | 12.1 | 4f434e2 |
| 39 | StartLux-Decision-v1.1-9B-InstantStartLux | 9.7B | 48.0tied with Dexy-Decision-9B-V2 | 58.4%2,163 questions | 2.7 | 18a17f7 |
| 40 | GLiDE 9BFastino | 7.9B | 47.4tied with StartLux-Decision-v1.1-9B-Instant | 57.9%2,163 questions | 7.1 | ab54df5 |
| 41 | vera-medium-omniscar-ai | 4.2B | 46.0tied with GLiDE 9B | 56.8%2,163 questions | 3.0 | 1f23a6c |
| 42 | Cerno 9B v1.2 full fine-tuneApertix AI | 8.9B | 45.0tied with vera-medium-omni | 56.0%2,163 questions | 15.7 | f02a632 |
| 43 | KnowLine 4B Gen5.5 LoRA, mergedPelaAI | 4.7B | 45.0tied with Cerno 9B v1.2 | 56.0%2,163 questions | 8.6 | 779944d |
| 44 | Perplexity Decider Flash v1.3Perplexity | 4.5B | 44.9tied with KnowLine 4B Gen5.5 | 55.9%2,163 questions | 11.7 | d06fb3f |
| 45 | StartLux-Decision-v1.1-4B-Instant fine-tuneStartLux | 4.7B | 44.8tied with Perplexity Decider Flash v1.3 | 55.8%2,163 questions | 6.7 | 416088f |
| 46 | d1Liquid AI | Not disclosed | 44.5tied with StartLux-Decision-v1.1-4B-Instant | 55.6%2,163 questions | 3.3 | no repo |
| 47 | Valen-Decision-1.5-9BValen-Team | 8.4B | 44.2tied with d1 | 55.3%2,163 questions | 4.1 | 5a51211 |
| 48 | Perplexity Decider Flash v1.2Perplexity | 4.5B | 44.1tied with Valen-Decision-1.5-9B | 55.2%2,163 questions | 10.4 | 6d771ad |
| 49 | Open-Jev-27B-v1.1 LoRA + headZefan Cai | 27.8B | 43.2tied with Perplexity Decider Flash v1.2 | 54.6%2,163 questions | 21.1 | 28cf730 |
| 50 | OneJev-27B full fine-tuneOmniJev | 27.4B | 42.2tied with Open-Jev-27B-v1.1 | 53.8%2,163 questions | 3.6 | a5e5f55 |
| 51 | Mercury DecideInception | Not disclosed | 41.9tied with OneJev-27B | 53.5%2,163 questions | 15.8 | no repo |
| 52 | Verdin-4B-v1.1 LoRA, mergedvictorbennett | 4.2B | 39.4 | 51.5%2,163 questions | 5.7 | 45830d2 |
| 53 | Clef post-trainedCloudflare | 27.5B | 38.9tied with Verdin-4B-v1.1 | 51.1%2,163 questions | 11.0 | 2f3de3d |
| 54 | Chinese Decision Pointer 4B LoRA, mergedquincyqiang | 4.2B | 38.0tied with Clef | 50.4%2,163 questions | 13.7 | 889f23e |
| 55 | Clef-Flash post-trainedCloudflare | 9.5B | 37.2tied with Chinese Decision Pointer 4B | 49.8%2,163 questions | 2.7 | fde727a |
| 56 | Winnow-12B LoRA, mergedEldanRing | 11.9B | 33.6 | 46.9%2,163 questions | 8.4 | b2b1421 |
| 57 | Cygnet unchanged baseBlockbrain Labs | 12B | 30.4 | 44.3%2,163 questions | 5.9 | no repo |
| 58 | Metask-Jev-4B LoRA, mergedMetask Lab | 4.5B | 23.9 | 39.2%2,163 questions | 3.3 | ea20fe8 |
| 59 | StartLux-Decision-v1.1-2B-Instant fine-tuneStartLux | 2.3B | 23.3tied with Metask-Jev-4B | 38.7%2,163 questions | 2.6 | f1e6cd1 |
| 60 | d1-2BLiquid AI | 1.9B | 21.3tied with StartLux-Decision-v1.1-2B-Instant | 37.1%2,163 questions | 5.6 | 10e7671 |
| 61 | GLiDE 2BFastino | 1.9B | 20.9tied with d1-2B | 36.8%2,163 questions | 17.2 | 00f122c |
| 62 | KnowLine-2B-Gen1 LoRA, mergedPelaAI | 2.3B | 19.1tied with GLiDE 2B | 35.3%2,163 questions | 9.4 | 53245d4 |
| 63 | GLiDE 0.8BFastino | 750M | 15.1 | 32.1%2,163 questions | 18.8 | 9eacdee |
| 64 | StartLux-Decision-v1.1-0.8B-Instant fine-tuneStartLux | 870M | 12.1 | 29.7%2,163 questions | 4.8 | 6496ae6 |
| 65 | Strands Decider 2B LoRA + headStrands Agents | 2.3B | 10.8tied with StartLux-Decision-v1.1-0.8B-Instant | 28.7%2,163 questions | 8.8 | 2b52a62 |
| 66 | OneJev1.6-0.8B full fine-tuneOmniJev | 850M | 10.3tied with Strands Decider 2B | 28.2%2,163 questions | 10.0 | bcd88a3 |
| 67 | KnowLine-0.8B-Gen1 LoRA, mergedPelaAI | 870M | 10.3tied with OneJev1.6-0.8B | 28.2%2,163 questions | 10.5 | 6b27831 |
| 68 | GLiClass v4 Edge fine-tuneKnowledgator | 440M | 3.3 | 22.7%2,163 questions | 7.0 | 885af1a |
| 69 | GLiDE 0.4BFastino | 400M | 0.7 | 20.6%2,163 questions | 12.2 | 6bda7ab |
Chat models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 high · chatAnthropic | Not disclosed | 99.7 | 99.8%2,163 questions | – | no repo |
| 2 | Qwen4-Max high · chatAlibaba | Not disclosed | 99.7tied with Claude Opus 5.5 · high · chat | 99.8%2,163 questions | – | no repo |
| 3 | Claude Fable 5.1 high · chatAnthropic | Not disclosed | 99.7tied with Qwen4-Max · high · chat | 99.7%2,163 questions | – | no repo |
| 4 | Claude Sonnet 5.5 high · chatAnthropic | Not disclosed | 99.6tied with Claude Fable 5.1 · high · chat | 99.7%2,163 questions | – | no repo |
| 5 | DeepSeek-V4.1-Flash reasoning on · chat · 1017DeepSeek | 552B16B active | 99.5tied with Claude Sonnet 5.5 · high · chat | 99.6%2,163 questions | – | no repo |
| 6 | GPT-6 Astra high · chatOpenAI | Not disclosed | 99.5tied with DeepSeek-V4.1-Flash · reasoning on · chat · 1017 | 99.6%2,163 questions | – | no repo |
| 7 | GPT-6.1 Sol high · chatOpenAI | Not disclosed | 99.5tied with GPT-6 Astra · high · chat | 99.6%2,163 questions | – | no repo |
| 8 | Inkling-Small high · chat · Jul 2026Thinking Machines | 266B12B active | 99.5tied with GPT-6.1 Sol · high · chat | 99.6%2,163 questions | – | no repo |
| 9 | Gemini 4 Carbon high · chatGoogle | Not disclosed | 99.5tied with Inkling-Small · high · chat · Jul 2026 | 99.6%2,163 questions | – | no repo |
| 10 | Gemini 4 Hydrogen high · chatGoogle | Not disclosed | 99.5tied with Gemini 4 Carbon · high · chat | 99.6%2,163 questions | – | no repo |
| 11 | Muse Ember max · chatMeta | Not disclosed | 99.5tied with Gemini 4 Hydrogen · high · chat | 99.6%2,163 questions | – | no repo |
| 12 | Grok 4.7 high · chatxAI | Not disclosed | 99.4tied with Muse Ember · max · chat | 99.5%2,163 questions | – | no repo |
| 13 | Inkling 2 Small high · chatThinking Machines | 594B17B active | 99.4tied with Grok 4.7 · high · chat | 99.5%2,163 questions | – | no repo |
| 14 | Qwen3.8 Flash no reasoning · chatAlibaba | Not disclosed | 99.4tied with Inkling 2 Small · high · chat | 99.5%2,163 questions | – | no repo |
| 15 | MiMo-V2.6-Flash high · chatXiaomi | 309B15B active | 99.3tied with Qwen3.8 Flash · no reasoning · chat | 99.4%2,163 questions | – | no repo |
| 16 | Qwen3.5-4B reasoning on · chatAlibaba | 4.7B | 99.3tied with MiMo-V2.6-Flash · high · chat | 99.4%2,163 questions | – | 851bf6e |
| 17 | Gemma 4 31B reasoning off · chatGoogle | 31.3B | 99.2tied with Qwen3.5-4B · reasoning on · chat | 99.4%2,163 questions | – | 842da37 |
| 18 | DeepSeek-V4-Flash chatDeepSeek | 285B13B active | 99.1tied with Gemma 4 31B · reasoning off · chat | 99.3%2,163 questions | – | no repo |
| 19 | MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | 99.0tied with DeepSeek-V4-Flash · chat | 99.2%2,163 questions | – | 42433e9 |
| 20 | GPT-6.1 Terra high · chatOpenAI | Not disclosed | 99.0tied with MiniCPM5-2B · reasoning off · chat | 99.2%2,163 questions | – | no repo |
| 21 | Granite-4.2-3B reasoning on · chatIBM | 3.7B | 99.0tied with GPT-6.1 Terra · high · chat | 99.2%2,163 questions | – | 4e9c275 |
| 22 | Qwen3.8-27B low · chatAlibaba | 27.8B | 99.0tied with Granite-4.2-3B · reasoning on · chat | 99.2%2,163 questions | – | 1d4bf0f |
| 23 | LFM2.5-24B-A2B default reasoning · chatLiquid AI | 24B2.5B active | 98.8tied with Qwen3.8-27B · low · chat | 99.0%2,163 questions | – | 1769516 |
| 24 | LFM3-1.7B reasoning on · chatLiquid AI | 1.7B | 98.7tied with LFM2.5-24B-A2B · default reasoning · chat | 98.9%2,163 questions | – | 0dc4268 |
| 25 | Gemma 4 12B Unified default reasoning · chatGoogle | 12B | 98.6tied with LFM3-1.7B · reasoning on · chat | 98.8%2,163 questions | – | 43c9d01 |
| 26 | GLM-5.3-Flash reasoning on · chatZ.ai | 320B18B active | 98.5tied with Gemma 4 12B Unified · default reasoning · chat | 98.8%2,163 questions | – | no repo |
| 27 | MiniMax-M3.1-Flash high · chatMiniMax | 428B23B active | 97.8tied with GLM-5.3-Flash · reasoning on · chat | 98.2%2,163 questions | – | no repo |
| 28 | Gemma 4 E4B default reasoning · chatGoogle | 8B | 97.6tied with MiniMax-M3.1-Flash · high · chat | 98.1%2,163 questions | – | 4ecc2ba |
| 29 | Gemma 4 E2B reasoning on · chatGoogle | 5.1B | 96.0tied with Gemma 4 E4B · default reasoning · chat | 96.8%2,163 questions | – | 4adf157 |
| 30 | LFM2.5-8B-A1B default reasoning · chatLiquid AI | 8.5B1.7B active | 95.3tied with Gemma 4 E2B · reasoning on · chat | 96.3%2,163 questions | – | dd83f5c |
| 31 | Qwen3-14B reasoning off · chatAlibaba | 14.8B | 94.7tied with LFM2.5-8B-A1B · default reasoning · chat | 95.8%2,163 questions | – | 40c0698 |
| 32 | Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | 93.3tied with Qwen3-14B · reasoning off · chat | 94.6%2,163 questions | – | 15852e8 |
| 33 | LFM2.5-2.6B default reasoning · chatLiquid AI | 2.7B | 92.5tied with Qwen3.5-2B · reasoning off · chat | 94.0%2,163 questions | – | 7141b56 |
| 34 | GPT-6 Luna low · chatOpenAI | Not disclosed | 92.1tied with LFM2.5-2.6B · default reasoning · chat | 93.7%2,163 questions | – | no repo |
| 35 | Erk-14B reasoning off · chateCloud Technology | 14.8B | 88.4 | 90.8%2,163 questions | – | 8584b14 |
| 36 | Raon-1 nano reasoning off · chatKRAFTON | 3.5B | 86.9tied with Erk-14B · reasoning off · chat | 89.6%2,163 questions | – | fcf2d1d |
2,163 questions per model · updated 16 November 2026 · 0 = chance: 1 of 5 options right on every question (accuracy 20%); 100 = every one of the 2,163 Math questions right (accuracy 100%) · feed: bench-scores.json
Backgammon
Decision models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Error rate (pr, lower is better) | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | KnowLine-31B-Gen1.5 LoRA, mergedPelaAI | 31.3B | 78.3 | 26.21521,928 decisions | – | 8e56aa1 |
| 2 | Sieve 31B v3 full fine-tuneSthānika AI | 31.3B | 77.3tied with KnowLine-31B-Gen1.5 | 27.49141,740 decisions | – | 8fcee85 |
| 3 | REAX 1.3 blendREAX | 57.4B | 77.1tied with Sieve 31B v3 | 27.77231,998 decisions | – | fa4bf7e |
| 4 | Gemini Decision 1 20261111Google | Not disclosed | 76.9tied with REAX 1.3 | 27.96861,704 decisions | – | no repo |
| 5 | Mammoth-1 post-trainedBase Labs | 27.4B | 76.4tied with Gemini Decision 1 · 20261111 | 28.54322,021 decisions | – | 61cec44 |
| 6 | Turion Jev 31B LoRA, mergedTurion.AI | 31.3B | 75.5tied with Mammoth-1 | 29.69571,855 decisions | – | 866c0fc |
| 7 | Drex Max fine-tuneNace.AI | 26.1B | 74.9tied with Turion Jev 31B | 30.39942,148 decisions | – | cac65e1 |
| 8 | Drex Max v1.1Nace.AI | Not disclosed | 74.5tied with Drex Max | 30.93832,154 decisions | – | no repo |
| 9 | d1-ultraLiquid AI | Not disclosed | 74.3tied with Drex Max v1.1 | 30.33742,071 decisions | – | no repo |
| 10 | Jev 1.14TypeSafe | Not disclosed | 73.1tied with d1-ultra | 31.73451,912 decisions | – | no repo |
| 11 | GLiDE APIFastino | Not disclosed | 71.9tied with Jev 1.14 | 33.18262,159 decisions | – | no repo |
| 12 | Jev 1.13TypeSafe | Not disclosed | 71.6tied with GLiDE · API | 34.40941,911 decisions | – | no repo |
| 13 | GLiDE v1.2Fastino | 26.1B | 71.1tied with Jev 1.13 | 35.00462,003 decisions | – | a7e4589 |
| 14 | celeris-1-decisionCeleris | Not disclosed | 70.6tied with GLiDE · v1.2 | 35.64281,922 decisions | – | no repo |
| 15 | Perplexity Decider 27B v1.2 DGX SparkPerplexity | 26.1B | 69.5tied with celeris-1-decision | 36.93342,028 decisions | – | 225c913 |
| 16 | Clef post-trainedCloudflare | 27.5B | 69.0tied with Perplexity Decider 27B v1.2 · DGX Spark | 36.53641,978 decisions | – | no repo |
| 17 | Drex 2.0Nace.AI | 35.1B3B active | 68.3tied with Clef | 38.441,937 decisions | – | no repo |
| 18 | Cygnet unchanged baseBlockbrain Labs | 12B | 68.2tied with Drex 2.0 | 38.55032,016 decisions | – | no repo |
| 19 | Perplexity Decider 27B v1.1Perplexity | 26.1B | 68.0tied with Cygnet | 38.78292,052 decisions | – | no repo |
| 20 | StartLux-Decision-v1.1-4B-Instant fine-tuneStartLux | 4.7B | 66.8tied with Perplexity Decider 27B v1.1 | 40.14491,989 decisions | – | 416088f |
| 21 | SubQ Snap 1.1Subquadratic | Not disclosed | 66.8tied with StartLux-Decision-v1.1-4B-Instant | 40.19972,326 decisions | – | no repo |
| 22 | GPT-6 Luna Decisions APIOpenAI | Not disclosed | 64.2 | 43.31071,962 decisions | – | no repo |
| 23 | SubQ Snap PreviewSubquadratic | Not disclosed | 63.8tied with GPT-6 Luna · Decisions API | 43.87062,200 decisions | – | no repo |
| 24 | Elephant-1 post-trainedBase Labs | 9.4B | 63.2tied with SubQ Snap Preview | 44.56162,044 decisions | – | b3f8a93 |
| 25 | d1-2BLiquid AI | 1.9B | 61.1tied with Elephant-1 | 47.10251,980 decisions | – | 10e7671 |
| 26 | Mercury DecideInception | Not disclosed | 60.4tied with d1-2B | 47.9481,853 decisions | – | no repo |
| 27 | Cerno 9B v1.2 full fine-tuneApertix AI | 8.9B | 60.0tied with Mercury Decide | 48.46732,108 decisions | – | f02a632 |
| 28 | Winnow-12B LoRA, mergedEldanRing | 11.9B | 58.9tied with Cerno 9B v1.2 | 49.72772,015 decisions | – | no repo |
| 29 | d1-Omni-2B NVIDIA RTX PRO 6000 BlackwellLiquid AI | 2.2B | 58.9tied with Winnow-12B | 49.79611,710 decisions | – | 81222da |
| 30 | StartLux-Decision-v1.1-2B-Instant fine-tuneStartLux | 2.3B | 58.7tied with d1-Omni-2B · NVIDIA RTX PRO 6000 Blackwell | 50.04171,771 decisions | – | f1e6cd1 |
| 31 | KnowLine 4B Gen5.5 LoRA, mergedPelaAI | 4.7B | 55.8 | 53.58962,153 decisions | – | 779944d |
| 32 | KnowLine-2B-Gen1 LoRA, mergedPelaAI | 2.3B | 55.0tied with KnowLine 4B Gen5.5 | 54.54791,955 decisions | – | 53245d4 |
| 33 | Perplexity Decider Flash v1.2Perplexity | 4.5B | 50.8 | 59.64222,006 decisions | – | no repo |
| 34 | StartLux-Decision-v1.1-0.8B-Instant fine-tuneStartLux | 870M | 46.6 | 64.71331,882 decisions | – | 6496ae6 |
| 35 | Open-Jev-27B-v1.1 LoRA + headZefan Cai | 27.8B | 45.9tied with StartLux-Decision-v1.1-0.8B-Instant | 65.55712,002 decisions | – | no repo |
| 36 | GLiDE 2BFastino | 1.9B | 44.4tied with Open-Jev-27B-v1.1 | 67.35151,676 decisions | – | 00f122c |
| 37 | KnowLine-0.8B-Gen1 LoRA, mergedPelaAI | 870M | 40.6 | 71.94121,728 decisions | – | 6b27831 |
| 38 | GLiDE 0.8BFastino | 750M | 33.6 | 80.36731,832 decisions | – | 9eacdee |
| 39 | OneJev1.6-0.8B full fine-tuneOmniJev | 850M | 31.9tied with GLiDE 0.8B | 82.51931,763 decisions | – | bcd88a3 |
| 40 | GLiDE 0.4BFastino | 400M | 6.5 | 113.2191,355 decisions | – | 6bda7ab |
| 41 | GLiClass v4 Edge fine-tuneKnowledgator | 440M | −10.2 | 133.5221,405 decisions | – | 885af1a |
Chat models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Error rate (pr, lower is better) | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra high · chatOpenAI | Not disclosed | 92.5 | 8.828561,489 decisions | – | no repo |
| 2 | GPT-6.1 Sol high · chatOpenAI | Not disclosed | 91.6tied with GPT-6 Astra · high · chat | 9.899021,389 decisions | – | no repo |
| 3 | Gemini 4 Argon low · chatGoogle | Not disclosed | 89.3tied with GPT-6.1 Sol · high · chat | 12.57231,332 decisions | – | no repo |
| 4 | Claude Fable 5.1 high · chatAnthropic | Not disclosed | 88.1tied with Gemini 4 Argon · low · chat | 13.9551,286 decisions | – | no repo |
| 5 | Claude Opus 5.5 high · chatAnthropic | Not disclosed | 87.7tied with Claude Fable 5.1 · high · chat | 14.44361,477 decisions | – | no repo |
| 6 | Gemini 3.8 Flash high · chatGoogle | Not disclosed | 87.3tied with Claude Opus 5.5 · high · chat | 14.94642,168 decisions | – | no repo |
| 7 | Claude Sonnet 5.5 high · chatAnthropic | Not disclosed | 85.8tied with Gemini 3.8 Flash · high · chat | 16.71942,164 decisions | – | no repo |
| 8 | Grok 4.7 high · chatxAI | Not disclosed | 84.0tied with Claude Sonnet 5.5 · high · chat | 18.78421,337 decisions | – | no repo |
| 9 | DeepSeek-V4.1-Flash low · API + NVIDIA RTX PRO 6000 Blackwell · first releaseDeepSeek | Not disclosed | 83.9tied with Grok 4.7 · high · chat | 18.99182,188 decisions | – | no repo |
| 10 | Gemma 4 31B reasoning off · chatGoogle | 31.3B | 83.4tied with DeepSeek-V4.1-Flash · low · API + NVIDIA RTX PRO 6000 Blackwell · first release | 19.57512,122 decisions | – | no repo |
| 11 | Muse Ember max · chatMeta | Not disclosed | 83.4tied with Gemma 4 31B · reasoning off · chat | 20.10641,667 decisions | – | no repo |
| 12 | Kimi K3.1 Pro low · chatMoonshot | 2.8T104B active | 76.2 | 28.04551,167 decisions | – | no repo |
| 13 | Qwen3.8-27B low · chatAlibaba | 27.8B | 75.4tied with Kimi K3.1 Pro · low · chat | 29.03262,108 decisions | – | no repo |
| 14 | GLM-5.3 low · chatZ.ai | 753.3B | 73.8tied with Qwen3.8-27B · low · chat | 30.86621,989 decisions | – | no repo |
| 15 | Qwen3.8 Flash no reasoning · chatAlibaba | Not disclosed | 71.5 | 33.57772,027 decisions | – | no repo |
| 16 | LFM2.5-24B-A2B default reasoning · chatLiquid AI | 24B2.5B active | 60.6 | 47.68632,193 decisions | – | no repo |
1,167 to 2,326 decisions per model · updated 16 November 2026 · 0 = 0 = a random legal play's error rate in the same league (League 1: 121.1, League 2: 117.9, League 3: 117.6; late entries: the league whose field they played); below 0 = worse than a random legal play. score = 100 × (1 − error rate / Random's error rate in that league); error rate (PR) = 500 × mean GNUbg equity loss per decision; 100 = 100 = error rate 0: the engine's best play on every decision · feed: bench-scores.json
Answer Check · does the text give this answer?
Decision models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Karpo Decision 31BMachinePulse | 31.3B | 90.0 | 93.3%2,069 pairs | 2.8 | d69cdd5 |
| 2 | Copernicus-31BPoindexter Labs | 31.3B | 89.0tied with Karpo Decision 31B | 92.7%2,069 pairs | 11.0 | be71525 |
| 3 | KnowLine-31B-Gen1.5 LoRA, mergedPelaAI | 31.3B | 88.8tied with Copernicus-31B | 92.6%2,069 pairs | 5.7 | 8e56aa1 |
| 4 | Sieve 31B v3 full fine-tuneSthānika AI | 31.3B | 88.1tied with KnowLine-31B-Gen1.5 | 92.1%2,069 pairs | 7.0 | 8fcee85 |
| 5 | KnowLine-31B-Gen2PelaAI | 31.3B | 87.7tied with Sieve 31B v3 | 91.8%2,069 pairs | 7.2 | fd99a9e |
| 6 | REAX 1.3 blendREAX | 57.4B | 87.0tied with KnowLine-31B-Gen2 | 91.3%2,069 pairs | 4.8 | fa4bf7e |
| 7 | Turion Jev 31B LoRA, mergedTurion.AI | 31.3B | 86.4tied with REAX 1.3 | 91.0%2,069 pairs | 6.7 | 866c0fc |
| 8 | d1.2-ultraLiquid AI | Not disclosed | 86.4tied with Turion Jev 31B | 91.0%2,069 pairs | 8.8 | no repo |
| 9 | Perplexity Decider 27B v1.3Perplexity | 26.1B | 86.2tied with d1.2-ultra | 90.8%2,069 pairs | 2.8 | 9c2ebb2 |
| 10 | GLiDE 27BFastino | 26.1B | 86.1tied with Perplexity Decider 27B v1.3 | 90.7%2,069 pairs | 3.6 | 4d32743 |
| 11 | Logits Sigma-1 Duo blendLogits | 57.4B | 85.1tied with GLiDE 27B | 90.1%2,069 pairs | 7.0 | a0c1b6e |
| 12 | GLiDE APIFastino | Not disclosed | 85.1tied with Logits Sigma-1 Duo | 90.0%2,069 pairs | 3.0 | no repo |
| 13 | GLiDE v1.2Fastino | 26.1B | 84.9tied with GLiDE · API | 89.9%2,069 pairs | 4.7 | a7e4589 |
| 14 | SubQ Snap 1.1Subquadratic | Not disclosed | 84.9tied with GLiDE · v1.2 | 89.9%2,069 pairs | 2.3 | no repo |
| 15 | d1-ultraLiquid AI | Not disclosed | 84.7tied with SubQ Snap 1.1 | 89.8%2,069 pairs | 11.4 | no repo |
| 16 | Microsoft-Decision-1Microsoft | Not disclosed | 83.9tied with d1-ultra | 89.3%2,069 pairs | 2.4 | no repo |
| 17 | vera-medium-omniscar-ai | 4.2B | 83.7tied with Microsoft-Decision-1 | 89.1%2,069 pairs | 12.4 | 1f23a6c |
| 18 | Jev 1.14TypeSafe | Not disclosed | 83.6tied with vera-medium-omni | 89.1%2,069 pairs | 2.8 | no repo |
| 19 | Jeffan-9B-1211tattrongvu | 8.9B | 83.0tied with Jev 1.14 | 88.6%2,069 pairs | 3.3 | 862d1a7 |
| 20 | Jev 1.13TypeSafe | Not disclosed | 83.0tied with Jeffan-9B-1211 | 88.6%2,069 pairs | 1.9 | no repo |
| 21 | d1.1Liquid AI | Not disclosed | 82.5tied with Jev 1.13 | 88.4%2,069 pairs | 8.7 | no repo |
| 22 | GLiDE 9BFastino | 7.9B | 82.2tied with d1.1 | 88.2%2,069 pairs | 7.2 | ab54df5 |
| 23 | Cerno 9B v1.2 full fine-tuneApertix AI | 8.9B | 81.9tied with GLiDE 9B | 87.9%2,069 pairs | 2.9 | f02a632 |
| 24 | StartLux-Decision-v1.1-4B-Instant fine-tuneStartLux | 4.7B | 81.7tied with Cerno 9B v1.2 | 87.8%2,069 pairs | 8.3 | 416088f |
| 25 | Dexy-Decision-9B-V2 fine-tunePatrick-18 | 8.4B | 81.1tied with StartLux-Decision-v1.1-4B-Instant | 87.4%2,069 pairs | 2.5 | 4f434e2 |
| 26 | KnowLine-12B-Gen1PelaAI | 12B | 80.9tied with Dexy-Decision-9B-V2 | 87.2%2,069 pairs | 6.8 | f26fbc2 |
| 27 | KnowLine 4B Gen5.5 LoRA, mergedPelaAI | 4.7B | 80.6tied with KnowLine-12B-Gen1 | 87.1%2,069 pairs | 4.5 | 779944d |
| 28 | Adjudex 9B v5newsbang | 8.4B | 80.4tied with KnowLine 4B Gen5.5 | 87.0%2,069 pairs | 4.9 | ffdcb82 |
| 29 | d1Liquid AI | Not disclosed | 79.2tied with Adjudex 9B v5 | 86.1%2,069 pairs | 3.1 | no repo |
| 30 | Perplexity Decider Flash v1.3Perplexity | 4.5B | 78.8tied with d1 | 85.9%2,069 pairs | 4.2 | d06fb3f |
| 31 | StartLux-Decision-v1.1-9B-InstantStartLux | 9.7B | 78.8tied with Perplexity Decider Flash v1.3 | 85.9%2,069 pairs | 8.2 | 18a17f7 |
| 32 | Chinese Decision Pointer 4B LoRA, mergedquincyqiang | 4.2B | 78.1tied with StartLux-Decision-v1.1-9B-Instant | 85.4%2,069 pairs | 6.7 | 889f23e |
| 33 | Perplexity Decider Flash v1.2Perplexity | 4.5B | 77.2tied with Chinese Decision Pointer 4B | 84.8%2,069 pairs | 2.8 | 6d771ad |
| 34 | Gemini Decision 1 20261111Google | Not disclosed | 77.0tied with Perplexity Decider Flash v1.2 | 84.7%2,069 pairs | 3.9 | no repo |
| 35 | Elephant-1 post-trainedBase Labs | 9.4B | 76.9tied with Gemini Decision 1 · 20261111 | 84.6%2,069 pairs | 3.7 | b3f8a93 |
| 36 | Winnow-12B LoRA, mergedEldanRing | 11.9B | 76.6tied with Elephant-1 | 84.4%2,069 pairs | 8.3 | b2b1421 |
| 37 | KnowLine-9B-Gen1PelaAI | 9.7B | 76.1tied with Winnow-12B | 84.1%2,069 pairs | 3.6 | 325378d |
| 38 | Verdin-4B-v1.1 LoRA, mergedvictorbennett | 4.2B | 75.4tied with KnowLine-9B-Gen1 | 83.6%2,069 pairs | 3.4 | 45830d2 |
| 39 | KnowLine-9B-Gen2PelaAI | 9.7B | 75.3tied with Verdin-4B-v1.1 | 83.5%2,069 pairs | 4.5 | 8ec8eb6 |
| 40 | Valen-Decision-1.5-9BValen-Team | 8.4B | 74.3tied with KnowLine-9B-Gen2 | 82.8%2,069 pairs | 3.2 | 5a51211 |
| 41 | Cygnet unchanged baseBlockbrain Labs | 12B | 72.9tied with Valen-Decision-1.5-9B | 81.9%2,069 pairs | 6.5 | no repo |
| 42 | Clef-Flash post-trainedCloudflare | 9.5B | 69.8 | 79.8%2,069 pairs | 5.9 | fde727a |
| 43 | StartLux-Decision-v1.1-2B-Instant fine-tuneStartLux | 2.3B | 66.0 | 77.3%2,069 pairs | 7.2 | f1e6cd1 |
| 44 | GLiDE 2BFastino | 1.9B | 65.6tied with StartLux-Decision-v1.1-2B-Instant | 77.1%2,069 pairs | 4.0 | 00f122c |
| 45 | Metask-Jev-4B LoRA, mergedMetask Lab | 4.5B | 61.7 | 74.5%2,069 pairs | 6.8 | ea20fe8 |
| 46 | KnowLine-2B-Gen1 LoRA, mergedPelaAI | 2.3B | 61.1tied with Metask-Jev-4B | 74.0%2,069 pairs | 2.7 | 53245d4 |
| 47 | GLiDE 0.8BFastino | 750M | 60.8tied with KnowLine-2B-Gen1 | 73.9%2,069 pairs | 3.8 | 9eacdee |
| 48 | Strands Decider 2B LoRA + headStrands Agents | 2.3B | 60.3tied with GLiDE 0.8B | 73.6%2,069 pairs | 5.0 | 2b52a62 |
| 49 | OneJev1.6-0.8B full fine-tuneOmniJev | 850M | 50.0 | 66.7%2,069 pairs | 4.1 | bcd88a3 |
| 50 | d1-2BLiquid AI | 1.9B | 49.6tied with OneJev1.6-0.8B | 66.4%2,069 pairs | 7.2 | 10e7671 |
| 51 | StartLux-Decision-v1.1-0.8B-Instant fine-tuneStartLux | 870M | 48.9tied with d1-2B | 65.9%2,069 pairs | 5.6 | 6496ae6 |
| 52 | GLiDE 0.4BFastino | 400M | 44.4 | 62.9%2,069 pairs | 7.7 | 6bda7ab |
| 53 | GLiClass v4 Edge fine-tuneKnowledgator | 440M | 44.3tied with GLiDE 0.4B | 62.9%2,069 pairs | 9.8 | 885af1a |
| 54 | Decider 1meraGPT | Not disclosed | 43.3tied with GLiClass v4 Edge | 62.2%2,069 pairs | 5.9 | no repo |
| 55 | KnowLine-0.8B-Gen1 LoRA, mergedPelaAI | 870M | 38.5 | 59.0%2,069 pairs | 14.6 | 6b27831 |
Chat models
| Rank | Model | Size | Score ↑0–100, chance = 0 | Accuracy | Calibration error ↓ECE, points | Commit |
|---|---|---|---|---|---|---|
| 1 | Inkling 2 Small high · chatThinking Machines | 594B17B active | 86.2 | 90.8%2,069 pairs | – | no repo |
| 2 | Gemini 4 Hydrogen high · chatGoogle | Not disclosed | 83.7 | 89.1%2,069 pairs | – | no repo |
| 3 | MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | 79.2 | 86.2%2,069 pairs | – | 42433e9 |
| 4 | Gemma 4 E4B default reasoning · chatGoogle | 8B | 77.2tied with MiniCPM5-2B · reasoning off · chat | 84.8%2,069 pairs | – | 4ecc2ba |
| 5 | Gemma 4 12B Unified default reasoning · chatGoogle | 12B | 77.0tied with Gemma 4 E4B · default reasoning · chat | 84.7%2,069 pairs | – | 43c9d01 |
| 6 | Granite-4.2-3B reasoning on · chatIBM | 3.7B | 75.9tied with Gemma 4 12B Unified · default reasoning · chat | 83.9%2,069 pairs | – | 4e9c275 |
| 7 | LFM3-1.7B reasoning on · chatLiquid AI | 1.7B | 75.8tied with Granite-4.2-3B · reasoning on · chat | 83.9%2,069 pairs | – | 0dc4268 |
| 8 | Gemma 4 E4B thinking on · sampled (temperature 1.0) · chatGoogle | 8B | 75.1tied with LFM3-1.7B · reasoning on · chat | 83.4%2,069 pairs | – | 4ecc2ba |
| 9 | LFM3-1.7B reasoning off · chatLiquid AI | 1.7B | 74.9tied with Gemma 4 E4B · thinking on · sampled (temperature 1.0) · chat | 83.3%2,069 pairs | – | 0dc4268 |
| 10 | Gemma 4 E2B reasoning on · chatGoogle | 5.1B | 71.1 | 80.8%2,069 pairs | – | 4adf157 |
| 11 | Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | 70.3tied with Gemma 4 E2B · reasoning on · chat | 80.2%2,069 pairs | – | 15852e8 |
| 12 | Gemma 4 12B Unified thinking on · chatGoogle | 11.9B | 67.7 | 78.4%2,069 pairs | – | 43c9d01 |
| 13 | MiniCPM5-2B reasoning off · chatOpenBMB | 2.5B | 67.0tied with Gemma 4 12B Unified · thinking on · chat | 78.0%2,069 pairs | – | 42433e9 |
| 14 | Qwen3.5-4B reasoning on · chatAlibaba | 4.7B | 66.2tied with MiniCPM5-2B · reasoning off · chat | 77.5%2,069 pairs | – | 851bf6e |
| 15 | Qwen3.5-4B reasoning off · chatAlibaba | 4.7B | 64.6tied with Qwen3.5-4B · reasoning on · chat | 76.4%2,069 pairs | – | 851bf6e |
| 16 | LFM2.5-8B-A1B default reasoning · chatLiquid AI | 8.5B1.7B active | 57.7 | 71.8%2,069 pairs | – | dd83f5c |
| 17 | Gemma 4 E2B default reasoning · chatGoogle | 5.1B | 55.2 | 70.1%2,069 pairs | – | 4adf157 |
| 18 | Gemma 4 E2B thinking off · chatGoogle | 5.1B | 54.8tied with Gemma 4 E2B · default reasoning · chat | 69.8%2,069 pairs | – | 4adf157 |
| 19 | Qwen3.5-2B reasoning off · chatAlibaba | 2.3B | 50.2 | 66.8%2,069 pairs | – | 15852e8 |
| 20 | Raon-1 nano reasoning off · chatKRAFTON | 3.5B | 40.6 | 60.4%2,069 pairs | – | fcf2d1d |
2,069 pairs per model · updated 16 November 2026 · 0 = uniform chance: one of the three labels at random on every pair (accuracy 1/3); 100 = every one of the 2,069 pairs labelled right (accuracy 100%) · feed: bench-scores.json
Learning from experience
| Model | Points gained ↑ |
|---|---|
| Qwen3.8-27B low · chatAlibaba | Backgammon 19.0 |
| GLM-5.3 low · chatZ.ai | Backgammon 17.6 |
| Claude Fable 5.1 high · chatAnthropic | Backgammon 9.4 |
| Gemini 4 Argon low · chatGoogle | Backgammon 8.5 |
| Gemma 4 31B reasoning off · chatGoogle | Backgammon 8.5 |
| DeepSeek-V4.1-Flash low · API + NVIDIA RTX PRO 6000 Blackwell · first releaseDeepSeek | Backgammon 8.2 |
| Gemini 3.8 Flash high · chatGoogle | Backgammon 8.2 |
| Claude Sonnet 5.5 high · chatAnthropic | Backgammon 7.0 |
| Claude Opus 5.5 high · chatAnthropic | Backgammon 6.7 |
| Kimi K3.1 Pro low · chatMoonshot | Backgammon 4.8 |
| Qwen3.8 Flash no reasoning · chatAlibaba | Backgammon 2.7 |
Gemini 4 Carbon high · chat
- Maker: Google
- Kind: chat model · class: Flagship
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 98.1 | 1 of 40tied with Claude Fable 5.2 · max · chat, Gemini 3.8 Flash · high · chat, Gemini 4 Hydrogen · high · chat and 2 more | 0.4 | – | text:1108Google Antigravity through Can's agbench bridge (Google AI Ultra subscription), one request at a time, thinking high; each request a new Antigravity agent conversation, so Antigravity's own agent instructions and tool declarations (about 13,000 tokens) come before our prompt, with every tool call denied and, as the bridge's owner reports, no workspace files and no earlier conversation; the bridge applies no response format, so the prompt adds a paragraph stating the answer layout |
| Turkish Reading | 97.0 | 2 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat and 8 more | 0.7 | – | text:1108Google Antigravity through Can's agbench bridge (Google AI Ultra subscription), one request at a time, thinking high; each request a new Antigravity agent conversation, so Antigravity's own agent instructions and tool declarations (about 13,000 tokens) come before our prompt, with every tool call denied and, as the bridge's owner reports, no workspace files and no earlier conversation; the bridge applies no response format, so the prompt adds a paragraph stating the answer layout |
| Turkish Grammar | 98.5 | 1 of 40tied with Gemini 3.8 Flash · high · chat, Gemini 4 Hydrogen · high · chat, Claude Fable 5.2 · max · chat and 2 more | 0.3 | – | text:1108Google Antigravity through Can's agbench bridge (Google AI Ultra subscription), one request at a time, thinking high; each request a new Antigravity agent conversation, so Antigravity's own agent instructions and tool declarations (about 13,000 tokens) come before our prompt, with every tool call denied and, as the bridge's owner reports, no workspace files and no earlier conversation; the bridge applies no response format, so the prompt adds a paragraph stating the answer layout |
| English Reading | 92.9 | 11 of 47tied with Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat, GPT-6.1 Terra · high · chat and 11 more | 3.1 | – | english:1055Google Antigravity through Can's agbench bridge (Google AI Ultra subscription), one request at a time, thinking high; each request a new Antigravity agent conversation, so Antigravity's own agent instructions and tool declarations (about 13,000 tokens) come before our prompt, with every tool call denied and, as the bridge's owner reports, no workspace files and no earlier conversation; the bridge applies no response format, so the prompt adds a paragraph stating the answer layout; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 95.9 | 2 of 35tied with GPT-6 Astra · high · chat, GPT-6.1 Sol · high · chat, Qwen4-Max · high · chat and 10 more | – | – | logic:gemini-4-carbon-sub_en, logic:gemini-4-carbon-sub_en_dontknowhosted API (the maker's service) |
| Math | 99.5 | 9 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:gemini-4-carbon-sub_en, math:gemini-4-carbon-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 98.3 | 2 of 34tied with GPT-6 Astra · high · chat, GPT-6.1 Sol · high · chat, Qwen4-Max · high · chat and 13 more | – | – | logic:gemini-4-carbon-sub_en, math:gemini-4-carbon-sub_en, logic:gemini-4-carbon-sub_en_dontknow, math:gemini-4-carbon-sub_en_dontknowhosted API (the maker's service) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Claude Fable 5.2 max · chat
- Maker: Anthropic
- Kind: chat model · class: Flagship
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 97.7 | 2 of 40tied with Gemini 4 Carbon · high · chat, Gemini 3.8 Flash · high · chat, Gemini 4 Hydrogen · high · chat and 2 more | 7.5 | – | text:980Claude Code CLI, Claude subscription |
| Turkish Reading | 97.1 | 1 of 40tied with Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat and 8 more | 8.2 | – | text:980Claude Code CLI, Claude subscription |
| Turkish Grammar | 97.9 | 4 of 40tied with Gemini 4 Carbon · high · chat, Gemini 3.8 Flash · high · chat, Gemini 4 Hydrogen · high · chat and 3 more | 7.2 | – | text:980Claude Code CLI, Claude subscription |
| English Reading | 97.5 | 1 of 47tied with Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat, GPT-6.1 Terra · high · chat and 4 more | 10.7 | – | english:971Claude Code CLI, Claude subscription; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Gemini 3.8 Flash high · chat
- Maker: Google
- Kind: chat model · class: Workhorse
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 8.2 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 97.6 | 3 of 40tied with Gemini 4 Carbon · high · chat, Claude Fable 5.2 · max · chat, Gemini 4 Hydrogen · high · chat and 2 more | 1.1 | – | text:140, text:123, text:188Gemini API |
| Turkish Reading | 96.4 | 5 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat and 9 more | 1.3 | – | text:140, text:123, text:188Gemini API |
| Turkish Grammar | 98.1 | 2 of 40tied with Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat, Claude Fable 5.2 · max · chat and 2 more | 1.0 | – | text:140, text:123, text:188Gemini API |
| English Reading | 95.0 | 6 of 47tied with Claude Fable 5.2 · max · chat, Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat and 9 more | 1.7 | – | english:583Gemini API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 87.3 | 6 of 16tied with Gemini 4 Argon · low · chat, Claude Fable 5.1 · high · chat, Claude Opus 5.5 · high · chat and 1 more | – | – | backgammon:gemini-3-8-flash-highLeague 2; calibration definition pending Ümit; hosted (API or subscription); setting: high; route: Google Antigravity subscription; improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Gemini 4 Hydrogen high · chat
- Maker: Google
- Kind: chat model · class: Workhorse
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 93.5EN 95.9 Logic 95.1 Math 99.5 AC 83.7 | 1 of 11 | – | – | – |
| Reading · Turkish (in Combined) | 97.6 | 4 of 40tied with Gemini 4 Carbon · high · chat, Claude Fable 5.2 · max · chat, Gemini 3.8 Flash · high · chat and 2 more | 1.2 | – | text:1114Google Antigravity through Can's agbench bridge (Google AI Ultra subscription), one request at a time, thinking high; each request a new Antigravity agent conversation, so Antigravity's own agent instructions and tool declarations (about 13,000 tokens) come before our prompt, with every tool call denied and, as the bridge's owner reports, no workspace files and no earlier conversation; the bridge applies no response format, so the prompt adds a paragraph stating the answer layout |
| Turkish Reading | 96.6 | 3 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, GPT-6 Astra · high · chat and 8 more | 2.3 | – | text:1114Google Antigravity through Can's agbench bridge (Google AI Ultra subscription), one request at a time, thinking high; each request a new Antigravity agent conversation, so Antigravity's own agent instructions and tool declarations (about 13,000 tokens) come before our prompt, with every tool call denied and, as the bridge's owner reports, no workspace files and no earlier conversation; the bridge applies no response format, so the prompt adds a paragraph stating the answer layout |
| Turkish Grammar | 98.1 | 3 of 40tied with Gemini 4 Carbon · high · chat, Gemini 3.8 Flash · high · chat, Claude Fable 5.2 · max · chat and 2 more | 0.9 | – | text:1114Google Antigravity through Can's agbench bridge (Google AI Ultra subscription), one request at a time, thinking high; each request a new Antigravity agent conversation, so Antigravity's own agent instructions and tool declarations (about 13,000 tokens) come before our prompt, with every tool call denied and, as the bridge's owner reports, no workspace files and no earlier conversation; the bridge applies no response format, so the prompt adds a paragraph stating the answer layout |
| English Reading | 95.9 | 2 of 47tied with Claude Fable 5.2 · max · chat, GPT-6 Astra · high · chat, GPT-6.1 Terra · high · chat and 8 more | 3.1 | – | english:1107Google Antigravity through Can's agbench bridge (Google AI Ultra subscription), one request at a time, thinking high; each request a new Antigravity agent conversation, so Antigravity's own agent instructions and tool declarations (about 13,000 tokens) come before our prompt, with every tool call denied and, as the bridge's owner reports, no workspace files and no earlier conversation; the bridge applies no response format, so the prompt adds a paragraph stating the answer layout; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 95.1 | 7 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 14 more | – | – | logic:gemini-4-hydrogen-sub_en, logic:gemini-4-hydrogen-sub_en_dontknowhosted API (the maker's service) |
| Math | 99.5 | 10 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:gemini-4-hydrogen-sub_en, math:gemini-4-hydrogen-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 98.0 | 7 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 14 more | – | – | logic:gemini-4-hydrogen-sub_en, math:gemini-4-hydrogen-sub_en, logic:gemini-4-hydrogen-sub_en_dontknow, math:gemini-4-hydrogen-sub_en_dontknowhosted API (the maker's service) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 83.7 | 2 of 20 | – | – | grading:chat-gemini-4-hydrogen-v1Google Antigravity agent harness through Can's agbench bridge (Google AI Ultra subscription), thinking high, tool descriptions present and tools denied, a fresh conversation per request; $0 billed (the subscription's quota); one question per request, 1 request at a time (latency not comparable across different concurrency); the pass stopped once when the bridge's safety switch tripped on a false positive (fixed by its owner) and resumed in the same order by a recorded decision; the one request in flight at the stop was sent again, so its recorded outcome is not guaranteed to be the model's first output; thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept |
GPT-6 Astra high · chat
- Maker: OpenAI
- Kind: chat model · class: Flagship
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 96.1TR 97.4 L&M 98.3 BG 92.5 | 1 of 9tied with GPT-6.1 Sol · high · chat | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 97.4 | 5 of 40tied with Gemini 4 Carbon · high · chat, Claude Fable 5.2 · max · chat, Gemini 3.8 Flash · high · chat and 3 more | 0.7 | – | text:277, text:610, text:48, text:191Codex CLI, ChatGPT subscription |
| Turkish Reading | 96.4 | 4 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat and 9 more | 1.0 | – | text:277, text:610, text:48, text:191Codex CLI, ChatGPT subscription |
| Turkish Grammar | 97.9 | 5 of 40tied with Gemini 4 Carbon · high · chat, Gemini 3.8 Flash · high · chat, Gemini 4 Hydrogen · high · chat and 3 more | 0.7 | – | text:277, text:610, text:48, text:191Codex CLI, ChatGPT subscription |
| English Reading | 95.2 | 3 of 47tied with Claude Fable 5.2 · max · chat, Gemini 4 Hydrogen · high · chat, GPT-6.1 Terra · high · chat and 8 more | 1.8 | – | english:615Codex CLI, ChatGPT subscription; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 95.9 | 1 of 35tied with Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat, Qwen4-Max · high · chat and 10 more | – | – | logic:gpt-6-astra-sub_en, logic:gpt-6-astra-sub_en_dontknowhosted API (the maker's service) |
| Math | 99.5 | 6 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:gpt-6-astra-sub_en, math:gpt-6-astra-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 98.3 | 1 of 34tied with Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat, Qwen4-Max · high · chat and 13 more | – | – | logic:gpt-6-astra-sub_en, math:gpt-6-astra-sub_en, logic:gpt-6-astra-sub_en_dontknow, math:gpt-6-astra-sub_en_dontknowhosted API (the maker's service) |
| Backgammon | 92.5 | 1 of 16tied with GPT-6.1 Sol · high · chat | – | – | backgammon:gpt-6-astra-highLeague 3; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 3's field (different positions from the league's); scored against League 3's Random; ChatGPT subscription (Codex backend). Late entry to the original League 3 table (not in standings).; hosted (API or subscription); setting: high |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
GPT-6.1 Sol high · chat
- Maker: OpenAI
- Kind: chat model · class: Workhorse
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 95.7TR 97.2 L&M 98.2 BG 91.6 | 2 of 9tied with GPT-6 Astra · high · chat | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 97.2 | 6 of 40tied with Gemini 4 Carbon · high · chat, Claude Fable 5.2 · max · chat, Gemini 3.8 Flash · high · chat and 4 more | 1.0 | – | text:276, text:124, text:190Codex CLI, ChatGPT subscription |
| Turkish Reading | 95.6 | 7 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat and 11 more | 1.8 | – | text:276, text:124, text:190Codex CLI, ChatGPT subscription |
| Turkish Grammar | 97.9 | 6 of 40tied with Gemini 4 Carbon · high · chat, Gemini 3.8 Flash · high · chat, Gemini 4 Hydrogen · high · chat and 3 more | 0.6 | – | text:276, text:124, text:190Codex CLI, ChatGPT subscription |
| English Reading | 94.5 | 8 of 47tied with Claude Fable 5.2 · max · chat, Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat and 11 more | 2.4 | – | english:618Codex CLI, ChatGPT subscription; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 95.7 | 3 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, Qwen4-Max · high · chat and 10 more | – | – | logic:gpt-6.1-sol-sub_en, logic:gpt-6.1-sol-sub_en_dontknowhosted API (the maker's service) |
| Math | 99.5 | 7 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:gpt-6.1-sol-sub_en, math:gpt-6.1-sol-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 98.2 | 3 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, Qwen4-Max · high · chat and 13 more | – | – | logic:gpt-6.1-sol-sub_en, math:gpt-6.1-sol-sub_en, logic:gpt-6.1-sol-sub_en_dontknow, math:gpt-6.1-sol-sub_en_dontknowhosted API (the maker's service) |
| Backgammon | 91.6 | 2 of 16tied with GPT-6 Astra · high · chat, Gemini 4 Argon · low · chat | – | – | backgammon:gpt-6-1-sol-highLeague 3; calibration definition pending Ümit; hosted (API or subscription); setting: high |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Gemini 4 Argon low · chat
- Maker: Google
- Kind: chat model · class: Flagship
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 8.5 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 95.5 | 7 of 40tied with GPT-6 Astra · high · chat, GPT-6.1 Sol · high · chat, Claude Fable 5.1 · high · chat and 1 more | 2.2 | – | text:557Gemini API |
| Turkish Reading | 96.2 | 6 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat and 9 more | 1.1 | – | text:557Gemini API |
| Turkish Grammar | 95.1 | 8 of 40tied with Claude Fable 5.1 · high · chat, Claude Opus 5.5 · high · chat | 2.7 | – | text:557Gemini API |
| English Reading | 95.0 | 7 of 47tied with Claude Fable 5.2 · max · chat, Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat and 9 more | 1.5 | – | english:593Gemini API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 89.3 | 3 of 16tied with GPT-6.1 Sol · high · chat, Claude Fable 5.1 · high · chat, Claude Opus 5.5 · high · chat and 1 more | – | – | backgammon:gemini-4-argon-onLeague 3; calibration definition pending Ümit; hosted (API or subscription); setting: high; route: Google Antigravity subscription; improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory); improvement_points from Run 2 after QC; original run +6.48, superseded |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Claude Fable 5.1 high · chat
- Maker: Anthropic
- Kind: chat model · class: Flagship
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 9.4 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 93.7TR 95.2 L&M 97.7 BG 88.1 | 3 of 9tied with Claude Opus 5.5 · high · chat | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 95.2 | 8 of 40tied with GPT-6.1 Sol · high · chat, Gemini 4 Argon · low · chat, Claude Opus 5.5 · high · chat | 12.0 | – | text:310Claude Code CLI, Claude subscription |
| Turkish Reading | 94.5 | 9 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat and 11 more | 10.9 | – | text:310Claude Code CLI, Claude subscription |
| Turkish Grammar | 95.5 | 7 of 40tied with Claude Fable 5.2 · max · chat, GPT-6 Astra · high · chat, GPT-6.1 Sol · high · chat and 2 more | 12.5 | – | text:310Claude Code CLI, Claude subscription |
| English Reading | 95.0 | 5 of 47tied with Claude Fable 5.2 · max · chat, Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat and 9 more | 11.8 | – | english:580Claude Code CLI, Claude subscription; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 94.0 | 12 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 16 more | – | – | logic:claude-fable-5.1-sub_en, logic:claude-fable-5.1-sub_en_dontknowhosted API (the maker's service) |
| Math | 99.7 | 3 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Sonnet 5.5 · high · chat and 24 more | – | – | math:claude-fable-5.1-sub_en, math:claude-fable-5.1-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 97.7 | 10 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 15 more | – | – | logic:claude-fable-5.1-sub_en, math:claude-fable-5.1-sub_en, logic:claude-fable-5.1-sub_en_dontknow, math:claude-fable-5.1-sub_en_dontknowhosted API (the maker's service) |
| Backgammon | 88.1 | 4 of 16tied with Gemini 4 Argon · low · chat, Claude Opus 5.5 · high · chat, Gemini 3.8 Flash · high · chat and 1 more | – | – | backgammon:claude-fable-5-1-onLeague 3; calibration definition pending Ümit; hosted (API or subscription); setting: high; improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Claude Opus 5.5 high · chat
- Maker: Anthropic
- Kind: chat model · class: Flagship
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 6.7 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 93.2TR 93.7 L&M 98.2 BG 87.7 | 4 of 9tied with Claude Fable 5.1 · high · chat | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 93.7 | 9 of 40tied with Gemini 4 Argon · low · chat, Claude Fable 5.1 · high · chat, GPT-6.1 Terra · high · chat | 14.6 | – | text:278, text:47, text:192Claude Code CLI, Claude subscription |
| Turkish Reading | 94.5 | 10 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat and 11 more | 14.3 | – | text:278, text:47, text:192Claude Code CLI, Claude subscription |
| Turkish Grammar | 93.4 | 9 of 40tied with Claude Fable 5.1 · high · chat, Gemini 4 Argon · low · chat, GPT-6.1 Terra · high · chat | 14.9 | – | text:278, text:47, text:192Claude Code CLI, Claude subscription |
| English Reading | 93.1 | 9 of 47tied with Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat, GPT-6.1 Terra · high · chat and 11 more | 13.3 | – | english:587Claude Code CLI, Claude subscription; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 95.2 | 5 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 14 more | – | – | logic:claude-opus-5.5-sub_en, logic:claude-opus-5.5-sub_en_dontknowhosted API (the maker's service) |
| Math | 99.7 | 1 of 36tied with Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat, Claude Sonnet 5.5 · high · chat and 24 more | – | – | math:claude-opus-5.5-sub_en, math:claude-opus-5.5-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 98.2 | 5 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 13 more | – | – | logic:claude-opus-5.5-sub_en, math:claude-opus-5.5-sub_en, logic:claude-opus-5.5-sub_en_dontknow, math:claude-opus-5.5-sub_en_dontknowhosted API (the maker's service) |
| Backgammon | 87.7 | 5 of 16tied with Gemini 4 Argon · low · chat, Claude Fable 5.1 · high · chat, Gemini 3.8 Flash · high · chat and 1 more | – | – | backgammon:claude-opus-5-5-onLeague 3; calibration definition pending Ümit; hosted (API or subscription); setting: high; improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
GPT-6.1 Terra high · chat
- Maker: OpenAI
- Kind: chat model · class: Workhorse
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 92.8 | 10 of 40tied with Claude Opus 5.5 · high · chat, Grok 4.7 · high · chat | 2.8 | – | text:275, text:274Codex CLI, ChatGPT subscription |
| Turkish Reading | 95.2 | 8 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat and 11 more | 1.3 | – | text:275, text:274Codex CLI, ChatGPT subscription |
| Turkish Grammar | 91.8 | 10 of 40tied with Claude Opus 5.5 · high · chat, Grok 4.7 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · first release | 3.5 | – | text:275, text:274Codex CLI, ChatGPT subscription |
| English Reading | 95.2 | 4 of 47tied with Claude Fable 5.2 · max · chat, Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat and 8 more | 1.3 | – | english:620Codex CLI, ChatGPT subscription; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 92.4 | 16 of 35tied with Qwen4-Max · high · chat, Claude Opus 5.5 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 and 15 more | – | – | logic:gpt-6.1-terra-sub_en, logic:gpt-6.1-terra-sub_en_dontknowhosted API (the maker's service) |
| Math | 99.0 | 20 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:gpt-6.1-terra-sub_en, math:gpt-6.1-terra-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 96.8 | 16 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 17 more | – | – | logic:gpt-6.1-terra-sub_en, math:gpt-6.1-terra-sub_en, logic:gpt-6.1-terra-sub_en_dontknow, math:gpt-6.1-terra-sub_en_dontknowhosted API (the maker's service) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Grok 4.7 high · chat
- Maker: xAI
- Kind: chat model · class: Workhorse
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic and Answer Check | ||||
| Reading · Turkish (in Combined) | 90.9 | 11 of 40tied with GPT-6.1 Terra · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · first release, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 and 1 more | 11.9 | – | text:311xAI Responses API, SuperGrok subscription |
| Turkish Reading | 93.7 | 11 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat and 11 more | 10.5 | – | text:311xAI Responses API, SuperGrok subscription |
| Turkish Grammar | 89.7 | 11 of 40tied with GPT-6.1 Terra · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · first release, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 and 1 more | 12.4 | – | text:311xAI Responses API, SuperGrok subscription |
| English Reading | 91.8 | 12 of 47tied with Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat, GPT-6.1 Terra · high · chat and 12 more | 11.1 | – | english:579xAI Responses API, SuperGrok subscription; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | 99.4 | 12 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:grok-4.7-sub_en, math:grok-4.7-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 84.0 | 8 of 16tied with Claude Sonnet 5.5 · high · chat, DeepSeek-V4.1-Flash · low · API + NVIDIA RTX PRO 6000 Blackwell · first release, Gemma 4 31B · reasoning off · chat and 1 more | – | – | backgammon:grok-4-7-onLeague 3; calibration definition pending Ümit; hosted (API or subscription); setting: high |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
DeepSeek-V4.1-Flash reasoning on · chat · first release
- Maker: DeepSeek
- Kind: chat model · class: Large
- Size: 552B (16B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on English Reading, Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 90.0 | 12 of 40tied with Grok 4.7 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · 1017, Qwen4-Max · high · chat | 8.0 | – | text:121, text:120SGLang, self-hosted, published mixed precision; first-release weights (revision fb2764a), not the 1017 update |
| Turkish Reading | 91.8 | 15 of 40tied with GPT-6.1 Sol · high · chat, GPT-6.1 Terra · high · chat, Claude Fable 5.1 · high · chat and 7 more | 6.6 | – | text:121, text:120SGLang, self-hosted, published mixed precision; first-release weights (revision fb2764a), not the 1017 update |
| Turkish Grammar | 89.3 | 12 of 40tied with GPT-6.1 Terra · high · chat, Grok 4.7 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 and 1 more | 8.6 | – | text:121, text:120SGLang, self-hosted, published mixed precision; first-release weights (revision fb2764a), not the 1017 update |
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
DeepSeek-V4.1-Flash reasoning on · chat · 1017
- Maker: DeepSeek
- Kind: chat model · class: Large
- Size: 552B (16B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Setups: Logic and Math: a hosted API; the other benches: our hardware.
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 89.7 | 13 of 40tied with Grok 4.7 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · first release, Qwen4-Max · high · chat | 6.9 | – | text:585, text:558SGLang, self-hosted, published mixed precision; 1017 weights (revision 1bae0e1) |
| Turkish Reading | 92.6 | 13 of 40tied with GPT-6 Astra · high · chat, Gemini 3.8 Flash · high · chat, Gemini 4 Argon · low · chat and 9 more | 6.0 | – | text:585, text:558SGLang, self-hosted, published mixed precision; 1017 weights (revision 1bae0e1) |
| Turkish Grammar | 88.5 | 13 of 40tied with Grok 4.7 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · first release, Qwen4-Max · high · chat | 7.3 | – | text:585, text:558SGLang, self-hosted, published mixed precision; 1017 weights (revision 1bae0e1) |
| English Reading | 90.6 | 14 of 47tied with GPT-6.1 Sol · high · chat, Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat and 8 more | 7.5 | – | english:808SGLang, self-hosted, published mixed precision; 1017 weights (revision 1bae0e1); the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 95.2 | 6 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 14 more | – | – | logic:deepseek-v4.1-flash-1017_en, logic:deepseek-v4.1-flash-1017_en_dontknowhosted API via OpenRouter |
| Math | 99.5 | 5 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:deepseek-v4.1-flash-1017_en, math:deepseek-v4.1-flash-1017_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 98.1 | 6 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 13 more | – | – | logic:deepseek-v4.1-flash-1017_en, math:deepseek-v4.1-flash-1017_en, logic:deepseek-v4.1-flash-1017_en_dontknow, math:deepseek-v4.1-flash-1017_en_dontknowhosted API via OpenRouter |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Qwen4-Max high · chat
- Maker: Alibaba
- Kind: chat model · class: Workhorse
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 89.6 | 14 of 40tied with Grok 4.7 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · first release, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 | 7.8 | – | text:1336OpenRouter chat API, Alibaba's endpoint (the maker's), fallbacks off, reasoning effort high, output cap 32,768 tokens, reasoning included; the answer schema |
| Turkish Reading | 93.3 | 12 of 40tied with Claude Fable 5.2 · max · chat, Gemini 4 Carbon · high · chat, Gemini 4 Hydrogen · high · chat and 11 more | 7.0 | – | text:1336OpenRouter chat API, Alibaba's endpoint (the maker's), fallbacks off, reasoning effort high, output cap 32,768 tokens, reasoning included; the answer schema |
| Turkish Grammar | 88.1 | 14 of 40tied with Grok 4.7 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · first release, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 | 8.3 | – | text:1336OpenRouter chat API, Alibaba's endpoint (the maker's), fallbacks off, reasoning effort high, output cap 32,768 tokens, reasoning included; the answer schema |
| English Reading | 93.1 | 10 of 47tied with Gemini 4 Hydrogen · high · chat, GPT-6 Astra · high · chat, GPT-6.1 Terra · high · chat and 11 more | 8.5 | – | english:1334OpenRouter chat API, Alibaba's endpoint (the maker's), fallbacks off, reasoning effort high, output cap 32,768 tokens, reasoning included; the answer schema; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 95.3 | 4 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 14 more | – | – | logic:qwen4-max_en, logic:qwen4-max_en_dontknowhosted API via OpenRouter |
| Math | 99.7 | 2 of 36tied with Claude Opus 5.5 · high · chat, Claude Fable 5.1 · high · chat, Claude Sonnet 5.5 · high · chat and 24 more | – | – | math:qwen4-max_en, math:qwen4-max_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 98.2 | 4 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 13 more | – | – | logic:qwen4-max_en, math:qwen4-max_en, logic:qwen4-max_en_dontknow, math:qwen4-max_en_dontknowhosted API via OpenRouter |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Muse Ember max · chat
- Maker: Meta
- Kind: chat model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 87.7TR 82.0 L&M 97.7 BG 83.4 | 6 of 9tied with Claude Sonnet 5.5 · high · chat | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 82.0 | 15 of 40tied with Claude Sonnet 5.5 · high · chat | 6.2 | – | text:939OpenRouter chat API, Meta's own endpoint, fallbacks off, output cap 65,536 tokens, reasoning max |
| Turkish Reading | 87.8 | 18 of 40tied with Qwen3.5-397B-A17B · high · chat, Gemma 4 31B · reasoning off · chat, GLM-5.3-Flash · reasoning on · chat and 5 more | 7.5 | – | text:939OpenRouter chat API, Meta's own endpoint, fallbacks off, output cap 65,536 tokens, reasoning max |
| Turkish Grammar | 79.6 | 16 of 40tied with Claude Sonnet 5.5 · high · chat | 6.3 | – | text:939OpenRouter chat API, Meta's own endpoint, fallbacks off, output cap 65,536 tokens, reasoning max |
| English Reading | 87.0 | 19 of 47tied with Claude Haiku 5.5 · max · chat, DeepSeek-V4.1-Flash · reasoning on · chat · 1017, Kimi K3.1 Pro · low · chat and 11 more | 7.0 | – | english:942OpenRouter chat API, Meta's own endpoint, fallbacks off, output cap 65,536 tokens, reasoning max; the kept reading questions in English |
| Vision | 82.8 | 1 of 2 | – | – | vision:128, vision:129, vision:130, vision:131, vision:132, vision:133, vision:134, vision:135, vision:136, vision:137, vision:138, vision:139, vision:140, vision:141, vision:142, vision:143, vision:144, vision:145, vision:146, vision:147, vision:148, vision:149, vision:150, vision:151, vision:152, vision:153, vision:154, vision:155, vision:156, vision:157, vision:158, vision:159, vision:160, vision:161, vision:162, vision:163, vision:164, vision:165, vision:166, vision:167, vision:168, vision:169OpenRouter chat API, Meta's own endpoint, fallbacks off, output cap 65,536 tokens, reasoning max; the page image only (condition B) |
| Logic | 94.3 | 10 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 16 more | – | – | logic:muse-ember_en, logic:muse-ember_en_dontknowhosted API via OpenRouter |
| Math | 99.5 | 11 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:muse-ember_en, math:muse-ember_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 97.7 | 12 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 15 more | – | – | logic:muse-ember_en, math:muse-ember_en, logic:muse-ember_en_dontknow, math:muse-ember_en_dontknowhosted API via OpenRouter |
| Backgammon | 83.4 | 11 of 16tied with Claude Sonnet 5.5 · high · chat, Grok 4.7 · high · chat, DeepSeek-V4.1-Flash · low · API + NVIDIA RTX PRO 6000 Blackwell · first release and 1 more | – | – | backgammon:muse-ember-maxLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Meta's Muse Ember through OpenRouter, pinned to Meta (fallbacks off) for every decision; never on our GPUs. Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 10–11 Nov 2026. It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #932 stopped before any decision, unbilled; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Every request sent reasoning_effort "max"; 21.76 M of its 22.09 M output tokens were reasoning. The endpoint returned no token probabilities, so it answers with one letter, read from the reply; 46 of its 1,713 answers named no option and a random legal play was made. A quality check (12 Nov) found that 1 of those 46 was an OpenRouter provider error (no answer) scored as invalid, in match #926; the other 45 were real length cuts. A full Run 2 of match #926 (same seed, fixed runner 175141f, its own database) played 6 games to 0 against Random with 0 provider errors; the published numbers keep the original match. Each decision is billed at Meta's list price from the tokens OpenRouter reported.; hosted (API or subscription) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Claude Sonnet 5.5 high · chat
- Maker: Anthropic
- Kind: chat model · class: Workhorse
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 7.0 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 88.3TR 81.3 L&M 97.8 BG 85.8 | 5 of 9tied with Muse Ember · max · chat | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 81.3 | 16 of 40tied with Muse Ember · max · chat | 16.0 | – | text:55, text:193, text:54Claude Code CLI, Claude subscription |
| Turkish Reading | 81.5 | 27 of 40tied with GLM-5.3 · low · chat, GPT-6 Luna · low · chat, Inkling-Small · high · chat · Jul 2026 and 4 more | 13.8 | – | text:55, text:193, text:54Claude Code CLI, Claude subscription |
| Turkish Grammar | 81.2 | 15 of 40tied with Muse Ember · max · chat | 16.9 | – | text:55, text:193, text:54Claude Code CLI, Claude subscription |
| English Reading | 82.6 | 28 of 47tied with MiMo-V2.6-Flash · high · chat, Qwen3.8-27B · low · chat, Inkling-Small · high · chat · Jul 2026 and 6 more | 13.9 | – | english:590Claude Code CLI, Claude subscription; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 94.3 | 9 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 16 more | – | – | logic:claude-sonnet-5.5-sub_en, logic:claude-sonnet-5.5-sub_en_dontknowhosted API (the maker's service) |
| Math | 99.6 | 4 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:claude-sonnet-5.5-sub_en, math:claude-sonnet-5.5-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 97.8 | 8 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 15 more | – | – | logic:claude-sonnet-5.5-sub_en, math:claude-sonnet-5.5-sub_en, logic:claude-sonnet-5.5-sub_en_dontknow, math:claude-sonnet-5.5-sub_en_dontknowhosted API (the maker's service) |
| Backgammon | 85.8 | 7 of 16tied with Claude Fable 5.1 · high · chat, Claude Opus 5.5 · high · chat, Gemini 3.8 Flash · high · chat and 3 more | – | – | backgammon:claude-sonnet-5-5-highLeague 2; calibration definition pending Ümit; hosted (API or subscription); setting: high; improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Kimi K3.1 Pro low · chat
- Maker: Moonshot
- Kind: chat model · class: Large
- Size: 2.8T (104B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 4.8 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 78.4 | 17 of 40tied with Qwen3.5-397B-A17B · high · chat | 12.6 | – | text:320, text:315OpenRouter chat API, Moonshot AI's endpoint (mxfp4), fallbacks off |
| Turkish Reading | 92.2 | 14 of 40tied with GPT-6.1 Sol · high · chat, GPT-6.1 Terra · high · chat, Claude Fable 5.1 · high · chat and 7 more | 4.3 | – | text:320, text:315OpenRouter chat API, Moonshot AI's endpoint (mxfp4), fallbacks off |
| Turkish Grammar | 72.6 | 18 of 40tied with Qwen3.5-397B-A17B · high · chat, Inkling-Small · high · chat · Jul 2026 | 16.2 | – | text:320, text:315OpenRouter chat API, Moonshot AI's endpoint (mxfp4), fallbacks off |
| English Reading | 90.4 | 15 of 47tied with GPT-6.1 Sol · high · chat, Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat and 10 more | 5.7 | – | english:584OpenRouter chat API, Moonshot AI's endpoint (mxfp4), fallbacks off; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 76.2 | 12 of 16tied with Qwen3.8-27B · low · chat, GLM-5.3 · low · chat | – | – | backgammon:kimi-k3-1-pro-offLeague 3; calibration definition pending Ümit; hosted (API or subscription); setting: off; improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Qwen3.5-397B-A17B high · chat
- Maker: Alibaba
- Kind: chat model · class: Large
- Size: 397B (17B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 78.4 | 18 of 40tied with Kimi K3.1 Pro · low · chat | 6.2 | – | text:870OpenRouter chat API, DeepInfra's endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included |
| Turkish Reading | 88.8 | 16 of 40tied with DeepSeek-V4.1-Flash · reasoning on · chat · 1017, Kimi K3.1 Pro · low · chat, DeepSeek-V4.1-Flash · reasoning on · chat · first release and 4 more | 2.2 | – | text:870OpenRouter chat API, DeepInfra's endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included |
| Turkish Grammar | 74.0 | 17 of 40tied with Kimi K3.1 Pro · low · chat | 9.1 | – | text:870OpenRouter chat API, DeepInfra's endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included |
| English Reading | 89.3 | 16 of 47tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Gemini 4 Carbon · high · chat and 10 more | 3.3 | – | english:871OpenRouter chat API, DeepInfra's endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 93.9 | 14 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 16 more | – | – | logic:qwen3.5-397b-a17b_en, logic:qwen3.5-397b-a17b_en_dontknowhosted API via OpenRouter |
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Copernicus-31B
- Maker: Poindexter Labs
- Kind: decision model · class: Medium
- Size: 31.3B
- Repo: Poindexter-Labs/Copernicus-31B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 75.0EN 91.1 Logic 48.3 Math 71.7 AC 89.0 | 4 of 53tied with KnowLine-31B-Gen2, KnowLine-31B-Gen1.5, REAX 1.3 and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 77.3 | 1 of 76tied with KnowLine-31B-Gen1.5, KnowLine-31B-Gen2, Turion Jev 31B | 9.1 | be71525 | text:1165Copernicus server (the card's copernicus_server.py, Transformers), self-hosted, BF16 |
| Turkish Reading | 91.4 | 3 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, KnowLine-31B-Gen1.5 and 8 more | 6.3 | be71525 | text:1165Copernicus server (the card's copernicus_server.py, Transformers), self-hosted, BF16 |
| Turkish Grammar | 71.2 | 2 of 76tied with KnowLine-31B-Gen1.5, KnowLine-31B-Gen2 | 10.5 | be71525 | text:1165Copernicus server (the card's copernicus_server.py, Transformers), self-hosted, BF16 |
| English Reading | 91.1 | 2 of 75tied with KnowLine-31B-Gen1.5, REAX 1.3, KnowLine-31B-Gen2 and 14 more | 7.8 | be71525 | english:1168Copernicus server (the card's copernicus_server.py, Transformers), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 48.3 | 5 of 69tied with Karpo Decision 31B, REAX 1.3, KnowLine-31B-Gen1.5 and 4 more | 3.9 | be71525 | logic:copernicus-31b_en, logic:copernicus-31b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 71.7 | 5 of 69tied with REAX 1.3, KnowLine-31B-Gen1.5, Sieve 31B v3 | 6.5 | be71525 | math:copernicus-31b_en, math:copernicus-31b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 63.7 | 5 of 69tied with REAX 1.3, KnowLine-31B-Gen1.5, Sieve 31B v3 | 5.0 | be71525 | logic:copernicus-31b_en, math:copernicus-31b_en, logic:copernicus-31b_en_dontknow, math:copernicus-31b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 89.0 | 2 of 55tied with Karpo Decision 31B, KnowLine-31B-Gen1.5, Sieve 31B v3 and 2 more | 11.0 | be71525 | grading:site-copernicus-31b-v1Poindexter-Labs/Copernicus-31B@be715259, its card's copernicus_server.py (Transformers backend, the card's defaults), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
KnowLine-31B-Gen1.5 LoRA, merged
- Maker: PelaAI · training: LoRA, merged
- Kind: decision model · class: Medium
- Size: 31.3B
- Repo: PelaAI/KnowLine-31B-Gen1.5
- Latency ↓: median 1,051 ms, p95 1,533 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 957)
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 71.9 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 73.2TR 77.3 L&M 63.9 BG 78.3 | 1 of 36 | – | – | – |
| Decision Index4 benches | 75.3EN 92.0 Logic 48.8 Math 71.7 AC 88.8 | 3 of 53tied with KnowLine-31B-Gen2, Copernicus-31B, REAX 1.3 | – | – | – |
| Reading · Turkish (in Combined) | 77.3 | 2 of 76tied with Copernicus-31B, KnowLine-31B-Gen2, Turion Jev 31B | 3.4 | 8e56aa1 | text:957KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16 |
| Turkish Reading | 91.0 | 4 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, Copernicus-31B and 10 more | 3.4 | 8e56aa1 | text:957KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16 |
| Turkish Grammar | 71.4 | 1 of 76tied with Copernicus-31B, KnowLine-31B-Gen2 | 3.9 | 8e56aa1 | text:957KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16 |
| English Reading | 92.0 | 1 of 75tied with Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2 and 10 more | 5.0 | 8e56aa1 | english:953KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16; the kept reading questions in English |
| Vision | 37.3 | 4 of 20tied with d1-ultra, GLiDE 27B, d3 and 1 more | 6.1 | 8e56aa1 | vision:86KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16; the page image only (condition B) |
| Logic | 48.8 | 3 of 69tied with Karpo Decision 31B, REAX 1.3, KnowLine-31B-Gen2 and 4 more | 2.6 | 8e56aa1 | logic:knowline-31b_en, logic:knowline-31b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 71.7 | 4 of 69tied with REAX 1.3, Copernicus-31B, Sieve 31B v3 | 3.6 | 8e56aa1 | math:knowline-31b_en, math:knowline-31b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 63.9 | 4 of 69tied with REAX 1.3, Copernicus-31B, Sieve 31B v3 | 2.6 | 8e56aa1 | logic:knowline-31b_en, math:knowline-31b_en, logic:knowline-31b_en_dontknow, math:knowline-31b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 78.3 | 1 of 41tied with Sieve 31B v3, REAX 1.3, Gemini Decision 1 · 20261111 and 1 more | – | 8e56aa1 | backgammon:knowline-31b-gen1-5League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 10–11 Nov 2026. It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Each option's probability is a softmax over the option labels, stored as returned. Its first run played 4 matches at once (#933–936) on a server meant for one request at a time; those matches were voided (rows kept, left out of every number) and the pairs replayed from the start, one match at a time.; run on our GPUs |
| Answer Check | 88.8 | 3 of 55tied with Karpo Decision 31B, Copernicus-31B, Sieve 31B v3 and 2 more | 5.7 | 8e56aa1 | grading:site-knowline-31b-gen1.5-v1PelaAI/KnowLine-31B-Gen1.5@8e56aa1b, the card's serve_knowline.sh (SGLang), BF16, on one DGX Spark (ours) as for its published runs, self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 1 published-cohort pair was sent in test requests whose outputs were not kept, so its scored row comes from a later request |
KnowLine-31B-Gen2
- Maker: PelaAI
- Kind: decision model · class: Medium
- Size: 31.3B
- Repo: PelaAI/KnowLine-31B-Gen2
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 75.5EN 90.4 Logic 48.7 Math 75.3 AC 87.7 | 2 of 53tied with Karpo Decision 31B, KnowLine-31B-Gen1.5, Copernicus-31B and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 76.5 | 3 of 76tied with Copernicus-31B, KnowLine-31B-Gen1.5, Turion Jev 31B | 5.6 | fd99a9e | text:1314KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| Turkish Reading | 91.8 | 2 of 76tied with Karpo Decision 31B, Copernicus-31B, KnowLine-31B-Gen1.5 and 8 more | 6.3 | fd99a9e | text:1314KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| Turkish Grammar | 70.0 | 3 of 76tied with KnowLine-31B-Gen1.5, Copernicus-31B, Turion Jev 31B | 5.7 | fd99a9e | text:1314KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| English Reading | 90.4 | 4 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 16 more | 7.7 | fd99a9e | english:1327KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 48.7 | 4 of 69tied with Karpo Decision 31B, REAX 1.3, KnowLine-31B-Gen1.5 and 4 more | 3.7 | fd99a9e | logic:knowline-31b-gen2_en, logic:knowline-31b-gen2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 75.3 | 2 of 69tied with Karpo Decision 31B | 3.6 | fd99a9e | math:knowline-31b-gen2_en, math:knowline-31b-gen2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 66.2 | 2 of 69tied with REAX 1.3 | 2.4 | fd99a9e | logic:knowline-31b-gen2_en, math:knowline-31b-gen2_en, logic:knowline-31b-gen2_en_dontknow, math:knowline-31b-gen2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 87.7 | 5 of 55tied with Copernicus-31B, KnowLine-31B-Gen1.5, Sieve 31B v3 and 5 more | 7.2 | fd99a9e | grading:site-knowline-31b-gen2-v1PelaAI/KnowLine-31B-Gen2@fd99a9ed, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card names 0.5.21, which we do not have), BF16, SHA256SUMS checked, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); the run overlapped a host GPU recovery warning; its serving process continued, and the run completed without interruption or resume (producer report) |
Turion Jev 31B LoRA, merged
- Maker: Turion.AI · training: LoRA, merged
- Kind: decision model · class: Medium
- Size: 31.3B
- Repo: turion-ai/turion-jev-31b
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 70.5 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 68.2TR 75.1 L&M 54.1 BG 75.5 | 5 of 36tied with Sieve 31B v3, d1-ultra, Gemini Decision 1 · 20261111 | – | – | – |
| Decision Index4 benches | 69.1EN 88.3 Logic 40.3 Math 61.3 AC 86.4 | 12 of 53tied with Perplexity Decider 27B v1.3, GLiDE · v1.2, d1.2-ultra | – | – | – |
| Reading · Turkish (in Combined) | 75.1 | 4 of 76tied with Copernicus-31B, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2 | 11.1 | 866c0fc | text:1273Turion Jev server (turion_jev.serve on vLLM, the maker's release code), self-hosted, FP8 at load |
| Turkish Reading | 90.9 | 5 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, Copernicus-31B and 12 more | 9.5 | 866c0fc | text:1273Turion Jev server (turion_jev.serve on vLLM, the maker's release code), self-hosted, FP8 at load |
| Turkish Grammar | 68.5 | 4 of 76tied with KnowLine-31B-Gen2 | 12.0 | 866c0fc | text:1273Turion Jev server (turion_jev.serve on vLLM, the maker's release code), self-hosted, FP8 at load |
| English Reading | 88.3 | 10 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 21 more | 11.6 | 866c0fc | english:1075Turion Jev server (turion_jev.serve on vLLM, the maker's release code), self-hosted, FP8 at load; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 40.3 | 21 of 69tied with d3-ultra, GLiDE · API, Mammoth-1 and 8 more | 2.9 | 866c0fc | logic:turion-jev-31b_en, logic:turion-jev-31b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 61.3 | 24 of 69tied with d3, Drex Max, Drex Max v1.1 and 7 more | 4.7 | 866c0fc | math:turion-jev-31b_en, math:turion-jev-31b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 54.1 | 22 of 69tied with Drex Max, Perplexity Decider 27B v1.2 · DGX Spark, Gemini Decision 1 · 20261111 and 5 more | 3.5 | 866c0fc | logic:turion-jev-31b_en, math:turion-jev-31b_en, logic:turion-jev-31b_en_dontknow, math:turion-jev-31b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 75.5 | 6 of 41tied with Sieve 31B v3, REAX 1.3, Gemini Decision 1 · 20261111 and 4 more | – | 866c0fc | backgammon:turion-jev-31bLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #46. turion-ai/turion-jev-31b @ 866c0fc on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), FP8 at load, through the server its package ships (python -m turion_jev.serve, vLLM 0.32.0 in process, --quantization fp8 --no-recheck), one request at a time; never on a hosted API. First stage 13 Nov 2026: its 9 hosted and program opponents (#1241–#1249, runner a127626 for #1241, then a72223d); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1403–#1410, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1250 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). BF16 weights quantised to FP8 at load and re-scoring off (--no-recheck), as in the card's run; probabilities rounded to 6 places. Its server was stopped after an OOM kill on spark2 (13 Nov 12:33 UTC, 3 failed health checks); from 14:29 UTC it ran through logic's launcher turion_serve_headroom.py --gpu-mem 0.80 (logic #6191), which sets only vLLM's memory share (the KV cache size) and then runs turion_jev.serve unchanged, pinned by its sha256. Never truncates: a prompt over 65,536 tokens is refused (422); nothing refused. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 86.4 | 7 of 55tied with Sieve 31B v3, KnowLine-31B-Gen2, REAX 1.3 and 8 more | 6.7 | 866c0fc | grading:site-turion-jev-31b-v1turion-ai/turion-jev-31b@866c0fc6, the card's turion_jev.serve on vLLM 0.32.0 (BF16 weights, FP8 at load, text only), on one DGX Spark (ours) as for its published runs, self-hosted; vLLM memory share 0.80 instead of the card's 0.92 (KV cache only); one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
Inkling-Small high · chat · Jul 2026
- Maker: Thinking Machines
- Kind: chat model · class: Large
- Size: 266B (12B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 75.0 | 19 of 40tied with GLM-5.3-Flash · reasoning on · chat | 7.9 | – | text:866OpenRouter chat API, DeepInfra's endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included; no response format (the endpoint has no structured outputs), so the prompt adds a paragraph stating the answer layout |
| Turkish Reading | 84.0 | 23 of 40tied with Muse Ember · max · chat, GLM-5.3-Flash · reasoning on · chat, Qwen3.8-27B · low · chat and 6 more | 3.2 | – | text:866OpenRouter chat API, DeepInfra's endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included; no response format (the endpoint has no structured outputs), so the prompt adds a paragraph stating the answer layout |
| Turkish Grammar | 71.1 | 19 of 40tied with Kimi K3.1 Pro · low · chat | 9.9 | – | text:866OpenRouter chat API, DeepInfra's endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included; no response format (the endpoint has no structured outputs), so the prompt adds a paragraph stating the answer layout |
| English Reading | 85.4 | 22 of 47tied with Qwen3.5-397B-A17B · high · chat, Gemma 4 31B · reasoning off · chat, GLM-5.3-Flash · reasoning on · chat and 9 more | 2.6 | – | english:867OpenRouter chat API, DeepInfra's endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included; no response format (the endpoint has no structured outputs), so the prompt adds a paragraph stating the answer layout; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 94.2 | 11 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 16 more | – | – | logic:inkling-small-64k_en, logic:inkling-small-64k_en_dontknowhosted API via OpenRouter |
| Math | 99.5 | 8 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:inkling-small-64k_en, math:inkling-small-64k_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 97.7 | 11 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 15 more | – | – | logic:inkling-small-64k_en, math:inkling-small-64k_en, logic:inkling-small-64k_en_dontknow, math:inkling-small-64k_en_dontknowhosted API via OpenRouter |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
GLM-5.3-Flash reasoning on · chat
- Maker: Z.ai
- Kind: chat model · class: Large
- Size: 320B (18B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Setups: Logic and Math: a hosted API; the other benches: our hardware.
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 73.5 | 20 of 40tied with Inkling-Small · high · chat · Jul 2026 | 4.5 | – | text:160, text:159TensorFold, self-hosted, EXL3 4-bit experts |
| Turkish Reading | 86.7 | 19 of 40tied with Qwen3.5-397B-A17B · high · chat, Gemma 4 31B · reasoning off · chat, Muse Ember · max · chat and 7 more | 5.5 | – | text:160, text:159TensorFold, self-hosted, EXL3 4-bit experts |
| Turkish Grammar | 67.9 | 20 of 40 | 4.4 | – | text:160, text:159TensorFold, self-hosted, EXL3 4-bit experts |
| English Reading | 87.0 | 18 of 47tied with Claude Haiku 5.5 · max · chat, DeepSeek-V4.1-Flash · reasoning on · chat · 1017, Kimi K3.1 Pro · low · chat and 11 more | 7.0 | – | english:823TensorFold, self-hosted, EXL3 4-bit experts; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 91.4 | 20 of 35tied with Claude Sonnet 5.5 · high · chat, Muse Ember · max · chat, Inkling-Small · high · chat · Jul 2026 and 10 more | – | – | logic:glm-5.3-flash_en, logic:glm-5.3-flash_en_dontknowhosted API via OpenRouter |
| Math | 98.5 | 26 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:glm-5.3-flash_en, math:glm-5.3-flash_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 96.1 | 19 of 34tied with Claude Sonnet 5.5 · high · chat, Inkling 2 Small · high · chat, Claude Fable 5.1 · high · chat and 11 more | – | – | logic:glm-5.3-flash_en, math:glm-5.3-flash_en, logic:glm-5.3-flash_en_dontknow, math:glm-5.3-flash_en_dontknowhosted API via OpenRouter |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
REAX 1.3 blend
- Maker: REAX · training: blend
- Kind: decision model · class: Large
- Size: 57.4B
- Repo: decisionmodels/REAX-1.3
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 74.0 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 71.3TR 72.4 L&M 64.5 BG 77.1 | 2 of 36 | – | – | – |
| Decision Index4 benches | 74.8EN 90.6 Logic 49.1 Math 72.5 AC 87.0 | 5 of 53tied with KnowLine-31B-Gen2, KnowLine-31B-Gen1.5, Copernicus-31B and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 72.4 | 5 of 76tied with Karpo Decision 31B, Gemini Decision 1 · 20261111, d1-ultra | 4.1 | fa4bf7e | text:827REAX Engine server (reax13 mode, the maker's release code), self-hosted, FP8 |
| Turkish Reading | 90.5 | 6 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, Copernicus-31B and 12 more | 2.1 | fa4bf7e | text:827REAX Engine server (reax13 mode, the maker's release code), self-hosted, FP8 |
| Turkish Grammar | 64.7 | 5 of 76tied with Gemini Decision 1 · 20261111, Karpo Decision 31B, d1-ultra | 5.1 | fa4bf7e | text:827REAX Engine server (reax13 mode, the maker's release code), self-hosted, FP8 |
| English Reading | 90.6 | 3 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, KnowLine-31B-Gen2 and 15 more | 3.6 | fa4bf7e | english:829REAX Engine server (reax13 mode, the maker's release code), self-hosted, FP8; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 49.1 | 2 of 69tied with Karpo Decision 31B, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2 and 4 more | 3.7 | fa4bf7e | logic:reax13_en, logic:reax13_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); weights byte-identical to decisionmodels/REAX-1.3 at this commit; served by the REAX engine (github.com/reaxlabs/reax-engine ff8f595) |
| Math | 72.5 | 3 of 69tied with KnowLine-31B-Gen1.5, Copernicus-31B, Sieve 31B v3 | 7.1 | fa4bf7e | math:reax13_en, math:reax13_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); weights byte-identical to decisionmodels/REAX-1.3 at this commit; served by the REAX engine (github.com/reaxlabs/reax-engine ff8f595) |
| Logic & Math (in Combined) | 64.5 | 3 of 69tied with KnowLine-31B-Gen2, KnowLine-31B-Gen1.5, Copernicus-31B and 1 more | 4.7 | fa4bf7e | logic:reax13_en, math:reax13_en, logic:reax13_en_dontknow, math:reax13_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); weights byte-identical to decisionmodels/REAX-1.3 at this commit; served by the REAX engine (github.com/reaxlabs/reax-engine ff8f595) |
| Backgammon | 77.1 | 3 of 41tied with KnowLine-31B-Gen1.5, Sieve 31B v3, Gemini Decision 1 · 20261111 and 3 more | – | fa4bf7e | backgammon:reax-1-3League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 18 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 7 Nov 2026. Each option's probability is the 50/50 mean of the two parts' probabilities, stored as returned; in the smoke test before the run every probability sum was 1.000000. Its pair with GPT-6 Luna was not played: GPT-6 Luna's API was out of credit; the pair is outside the requested scope (Ümit, 12 Nov). Run note: the server's container ran without Docker's no-new-privileges setting, approved for this run only (the host's Docker refused every container with it); every other protection was kept: all capabilities dropped, read-only root filesystem, non-root user, weights mounted read-only, no outside network.; run on our GPUs |
| Answer Check | 87.0 | 6 of 55tied with Copernicus-31B, KnowLine-31B-Gen1.5, Sieve 31B v3 and 9 more | 4.8 | fa4bf7e | grading:site-reax-1.3-v1decisionmodels/REAX-1.3@fa4bf7ee, REAX Engine reax13 server on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
Karpo Decision 31B
- Maker: MachinePulse
- Kind: decision model · class: Medium
- Size: 31.3B
- Repo: MachinePulseAI/Karpo Decision 31B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 77.0EN 90.2 Logic 50.9 Math 76.9 AC 90.0 | 1 of 53tied with KnowLine-31B-Gen2 | – | – | – |
| Reading · Turkish (in Combined) | 72.2 | 6 of 76tied with REAX 1.3, Gemini Decision 1 · 20261111, d1-ultra | 2.3 | d69cdd5 | text:1156karpo-decision server (the System One server of the card's karpo-decision package, machinepulse-ai/karpo-decision at ac40d39), self-hosted, BF16 |
| Turkish Reading | 91.8 | 1 of 76tied with KnowLine-31B-Gen2, Copernicus-31B, KnowLine-31B-Gen1.5 and 8 more | 1.2 | d69cdd5 | text:1156karpo-decision server (the System One server of the card's karpo-decision package, machinepulse-ai/karpo-decision at ac40d39), self-hosted, BF16 |
| Turkish Grammar | 63.9 | 7 of 76tied with REAX 1.3, Gemini Decision 1 · 20261111, d1-ultra | 3.4 | d69cdd5 | text:1156karpo-decision server (the System One server of the card's karpo-decision package, machinepulse-ai/karpo-decision at ac40d39), self-hosted, BF16 |
| English Reading | 90.2 | 5 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 16 more | 2.4 | d69cdd5 | english:1159karpo-decision server (the System One server of the card's karpo-decision package, machinepulse-ai/karpo-decision at ac40d39), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 50.9 | 1 of 69tied with REAX 1.3, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2 and 2 more | 5.7 | d69cdd5 | logic:karpo-decision-31b_en, logic:karpo-decision-31b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 76.9 | 1 of 69tied with KnowLine-31B-Gen2 | 1.2 | d69cdd5 | math:karpo-decision-31b_en, math:karpo-decision-31b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 68.0 | 1 of 69 | 2.7 | d69cdd5 | logic:karpo-decision-31b_en, math:karpo-decision-31b_en, logic:karpo-decision-31b_en_dontknow, math:karpo-decision-31b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 90.0 | 1 of 55tied with Copernicus-31B, KnowLine-31B-Gen1.5, Sieve 31B v3 | 2.8 | d69cdd5 | grading:site-karpo-decision-31b-v1MachinePulseAI/Karpo Decision 31B@d69cdd57, its card's karpo-decision package (machinepulse-ai/karpo-decision@ac40d399) System One server, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
Gemini Decision 1 20261111
- Maker: Google
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 68.0TR 71.9 L&M 55.1 BG 76.9 | 6 of 36tied with d1-ultra, Turion Jev 31B | – | – | – |
| Decision Index4 benches | 66.9EN 88.8 Logic 37.6 Math 64.2 AC 77.0 | 15 of 53tied with SubQ Snap 1.1 | – | – | – |
| Reading · Turkish (in Combined) | 71.9 | 7 of 76tied with REAX 1.3, Karpo Decision 31B, d1-ultra | 2.0 | – | text:1021OpenRouter decisions API, Google AI Studio endpoint |
| Turkish Reading | 90.1 | 7 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, Copernicus-31B and 16 more | 2.8 | – | text:1021OpenRouter decisions API, Google AI Studio endpoint |
| Turkish Grammar | 64.2 | 6 of 76tied with REAX 1.3, Karpo Decision 31B, d1-ultra | 2.9 | – | text:1021OpenRouter decisions API, Google AI Studio endpoint |
| English Reading | 88.8 | 6 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 20 more | 4.5 | – | english:1009OpenRouter decisions API, Google AI Studio endpoint; the kept reading questions in English |
| Vision | 58.1 | 1 of 20 | 5.2 | – | vision:116OpenRouter decisions API, Google AI Studio endpoint; the page image only (condition B) |
| Logic | 37.6 | 24 of 69tied with SubQ Snap 1.1, Turion Jev 31B, d1.2-ultra and 5 more | 3.1 | – | logic:gemini-decision-1_en, logic:gemini-decision-1_en_dontknowhosted API via OpenRouter |
| Math | 64.2 | 13 of 69tied with Perplexity Decider 27B v1.3, GLiDE 27B, d3-ultra and 9 more | 2.5 | – | math:gemini-decision-1_en, math:gemini-decision-1_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 55.1 | 19 of 69tied with GLiDE · API, d3, Drex Max v1.1 and 7 more | 2.0 | – | logic:gemini-decision-1_en, math:gemini-decision-1_en, logic:gemini-decision-1_en_dontknow, math:gemini-decision-1_en_dontknowhosted API via OpenRouter |
| Backgammon | 76.9 | 4 of 41tied with KnowLine-31B-Gen1.5, Sieve 31B v3, REAX 1.3 and 3 more | – | – | backgammon:gemini-decision-1League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Google's Gemini Decision 1 through OpenRouter, pinned to Google AI Studio (fallbacks off) for every decision; never on our GPUs. Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 11 Nov 2026. It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1016 aborted; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Two runner builds: dab6da5 for its first 9 matches (#993–#1013, before 21:46:31 UTC), with no probability-map check, and 38024d6 (map-check v2) from 21:46:31 UTC for its last 8 (#1053–#1063). Under map-check v2 a returned probability map is used only if it covers exactly the options shown, every value is finite and in [0, 1], and the values sum to 1 within 1e-6; this sum rule is new (no frozen sum tolerance existed), was chosen on 11 Nov 2026 and applies to this entry only; a failed map still plays the returned choice and stores no probabilities. An offline audit of the first matches under the same rule found no failing map. Option order: we send the options in our seeded order, as for every setup, while Google's docs say the API sorts the option keys alphabetically (UTF-8 byte order; an exact tie goes to the alphabetically first key), so the order the model sees is probably not ours (Google's docs; not verified by us). Its cost is the billed cost of each decision; the entry, with its paid opponents, billed $1.1819 in all.; hosted (API or subscription) |
| Answer Check | 77.0 | 34 of 55tied with d1, Perplexity Decider Flash v1.3, StartLux-Decision-v1.1-9B-Instant and 7 more | 3.9 | – | grading:gemini-v1Google Decisions API (generativelanguage.googleapis.com, model gemini-decision-1); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
d1-ultra
- Maker: Liquid AI
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 68.3TR 71.3 L&M 59.4 BG 74.3 | 4 of 36tied with Sieve 31B v3, Turion Jev 31B, Gemini Decision 1 · 20261111 | – | – | – |
| Decision Index4 benches | 71.1EN 88.3 Logic 43.7 Math 67.6 AC 84.7 | 9 of 53tied with Logits Sigma-1 Duo, GLiDE 27B, Perplexity Decider 27B v1.3 and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 71.3 | 8 of 76tied with REAX 1.3, Karpo Decision 31B, Gemini Decision 1 · 20261111 | 3.6 | – | text:199Liquid AI decisions API |
| Turkish Reading | 90.1 | 8 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, Copernicus-31B and 16 more | 3.6 | – | text:199Liquid AI decisions API |
| Turkish Grammar | 63.4 | 8 of 76tied with REAX 1.3, Gemini Decision 1 · 20261111, Karpo Decision 31B | 4.2 | – | text:199Liquid AI decisions API |
| English Reading | 88.3 | 11 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 21 more | 4.4 | – | english:411, english:418, english:422, english:426, english:428Liquid AI decisions API; the kept reading questions in English |
| Vision | 37.6 | 2 of 20tied with GLiDE 27B, KnowLine-31B-Gen1.5, d3 | 3.5 | – | vision:2, vision:5Liquid AI decisions API; the page image only (condition B) |
| Logic | 43.7 | 10 of 69tied with GLiDE · v1.2, Drex Max v1.1, Perplexity Decider 27B v1.3 and 6 more | 3.2 | – | logic:d1-ultra_en, logic:d1-ultra_en_dontknowhosted API (the maker's service) |
| Math | 67.6 | 8 of 69tied with Logits Sigma-1 Duo, Mammoth-1, Perplexity Decider 27B v1.3 | 2.3 | – | math:d1-ultra_en, math:d1-ultra_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 59.4 | 8 of 69tied with GLiDE 27B, Mammoth-1, Perplexity Decider 27B v1.3 | 2.1 | – | logic:d1-ultra_en, math:d1-ultra_en, logic:d1-ultra_en_dontknow, math:d1-ultra_en_dontknowhosted API (the maker's service) |
| Backgammon | 74.3 | 9 of 41tied with Mammoth-1, Turion Jev 31B, Drex Max and 2 more | – | – | backgammon:liquid-d1-ultraLeague 2; calibration definition pending Ümit; hosted (API or subscription) |
| Answer Check | 84.7 | 15 of 55tied with Turion Jev 31B, d1.2-ultra, Perplexity Decider 27B v1.3 and 11 more | 11.4 | – | grading:d1-ultra-v1Liquid AI decisions API (api.liquid.ai); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
Logits Sigma-1 Duo blend
- Maker: Logits · training: blend
- Kind: decision model · class: Large
- Size: 57.4B
- Repo: logitshq/Logits-Sigma-1-Duo
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 72.5EN 88.8 Logic 47.2 Math 69.0 AC 85.1 | 7 of 53tied with Sieve 31B v3, GLiDE 27B, d1-ultra | – | – | – |
| Reading · Turkish (in Combined) | 68.8 | 9 of 76tied with Sieve 31B v3 | 5.7 | a0c1b6e | text:1221Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16 |
| Turkish Reading | 89.9 | 9 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, Copernicus-31B and 17 more | 5.3 | a0c1b6e | text:1221Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16 |
| Turkish Grammar | 59.8 | 9 of 76tied with Sieve 31B v3 | 5.9 | a0c1b6e | text:1221Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16 |
| English Reading | 88.8 | 7 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 20 more | 8.0 | a0c1b6e | english:1229Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 47.2 | 8 of 69tied with REAX 1.3, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2 and 3 more | 5.0 | a0c1b6e | logic:logits-sigma-1-duo_en, logic:logits-sigma-1-duo_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 69.0 | 7 of 69tied with d1-ultra, Mammoth-1 | 7.5 | a0c1b6e | math:logits-sigma-1-duo_en, math:logits-sigma-1-duo_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 61.6 | 7 of 69tied with Sieve 31B v3 | 6.0 | a0c1b6e | logic:logits-sigma-1-duo_en, math:logits-sigma-1-duo_en, logic:logits-sigma-1-duo_en_dontknow, math:logits-sigma-1-duo_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 85.1 | 11 of 55tied with REAX 1.3, Turion Jev 31B, d1.2-ultra and 11 more | 7.0 | a0c1b6e | grading:site-logits-sigma-1-duo-v1logitshq/Logits-Sigma-1-Duo@a0c1b6e8 (weights and code identical to d9641cd8, the revision its card's run names; two members averaged per option, FP8 weights), its card's serving image (code/Dockerfile: torch 2.15, transformers 5.22.1, causal-conv1d 1.7.0, kit decision-index@25c6ec7) plus only gcc, libc6-dev and python3-dev, which its Triton kernels need to compile a launcher at first use and the image lacks, with the engine and option its serve.sh fixes (FP8 matmul on its own Triton kernel: its cuBLAS block-scaled path was not taken on this GPU, a fallback the engine makes itself), behind di_serve.py, our System One wrapper of the engine, on one RTX PRO 6000 (ours), self-hosted; text only (the load report lists FP8 scales missing for member A's vision tower, which text requests do not use; not checked on images); one question per request, 1 request at a time (latency not comparable across different concurrency) |
Qwen3.8-27B low · chat
- Maker: Alibaba
- Kind: chat model · class: Medium
- Size: 27.8B
- Repo: Qwen/Qwen3.8-27B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 19.0 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 80.2TR 68.3 L&M 96.8 BG 75.4 | 8 of 9 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 68.3 | 21 of 40tied with GLM-5.3 · low · chat, Gemma 4 31B · reasoning off · chat | 23.7 | 1d4bf0f | text:318, text:50, text:53, text:83vLLM, self-hosted |
| Turkish Reading | 86.7 | 20 of 40tied with Qwen3.5-397B-A17B · high · chat, Gemma 4 31B · reasoning off · chat, Muse Ember · max · chat and 7 more | 10.7 | 1d4bf0f | text:318, text:50, text:53, text:83vLLM, self-hosted |
| Turkish Grammar | 60.5 | 22 of 40tied with GLM-5.3 · low · chat | 29.4 | 1d4bf0f | text:318, text:50, text:53, text:83vLLM, self-hosted |
| English Reading | 86.3 | 21 of 47tied with Kimi K3.1 Pro · low · chat, Qwen3.5-397B-A17B · high · chat, Gemma 4 31B · reasoning off · chat and 10 more | 11.0 | 1d4bf0f | english:667vLLM, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 92.6 | 15 of 35tied with Qwen4-Max · high · chat, Claude Opus 5.5 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 and 14 more | – | 1d4bf0f | logic:qwen3.8-27b-bf16_en_low, logic:qwen3.8-27b-bf16_en_low_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 99.0 | 22 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | 1d4bf0f | math:qwen3.8-27b-bf16_en_low, math:qwen3.8-27b-bf16_en_low_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 96.8 | 15 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 17 more | – | 1d4bf0f | logic:qwen3.8-27b-bf16_en_low, math:qwen3.8-27b-bf16_en_low, logic:qwen3.8-27b-bf16_en_low_dontknow, math:qwen3.8-27b-bf16_en_low_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | 75.4 | 13 of 16tied with Kimi K3.1 Pro · low · chat, GLM-5.3 · low · chat | – | – | backgammon:qwen3-8-27b-onLeague 2; calibration definition pending Ümit; run on our GPUs; setting: low; improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
GLM-5.3 low · chat
- Maker: Z.ai
- Kind: chat model · class: Workhorse
- Size: 753.3B
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 17.6 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 68.2 | 22 of 40tied with Qwen3.8-27B · low · chat, Gemma 4 31B · reasoning off · chat | 2.6 | – | text:314OpenRouter chat API, Fireworks endpoint, fallbacks off |
| Turkish Reading | 84.6 | 21 of 40tied with Muse Ember · max · chat, GLM-5.3-Flash · reasoning on · chat, Qwen3.8-27B · low · chat and 6 more | 8.6 | – | text:314OpenRouter chat API, Fireworks endpoint, fallbacks off |
| Turkish Grammar | 61.2 | 21 of 40tied with Qwen3.8-27B · low · chat | 5.5 | – | text:314OpenRouter chat API, Fireworks endpoint, fallbacks off |
| English Reading | 84.7 | 23 of 47tied with Gemma 4 31B · reasoning off · chat, GLM-5.3-Flash · reasoning on · chat, Muse Ember · max · chat and 9 more | 3.6 | – | english:577OpenRouter chat API, Fireworks endpoint, fallbacks off; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 73.8 | 14 of 16tied with Kimi K3.1 Pro · low · chat, Qwen3.8-27B · low · chat | – | – | backgammon:glm-5-3-onLeague 2; calibration definition pending Ümit; hosted (API or subscription); setting: low; improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Sieve 31B v3 full fine-tune
- Maker: Sthānika AI · training: full fine-tune
- Kind: decision model · class: Medium
- Size: 31.3B
- Repo: sthanika-ai/Sieve-31B-v3
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 71.1 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 69.4TR 67.7 L&M 63.2 BG 77.3 | 3 of 36tied with d1-ultra, Turion Jev 31B | – | – | – |
| Decision Index4 benches | 73.8EN 88.3 Logic 47.3 Math 71.4 AC 88.1 | 6 of 53tied with Copernicus-31B, REAX 1.3, Logits Sigma-1 Duo | – | – | – |
| Reading · Turkish (in Combined) | 67.7 | 10 of 76tied with Logits Sigma-1 Duo | 2.9 | 8fcee85 | text:1241Sieve server (sieve-serve, the maker's release code), self-hosted, BF16 |
| Turkish Reading | 89.7 | 10 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, Copernicus-31B and 17 more | 3.5 | 8fcee85 | text:1241Sieve server (sieve-serve, the maker's release code), self-hosted, BF16 |
| Turkish Grammar | 58.3 | 10 of 76tied with Logits Sigma-1 Duo | 3.3 | 8fcee85 | text:1241Sieve server (sieve-serve, the maker's release code), self-hosted, BF16 |
| English Reading | 88.3 | 9 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 21 more | 4.3 | 8fcee85 | english:1064Sieve server (sieve-serve, the maker's release code), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 47.3 | 7 of 69tied with REAX 1.3, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2 and 3 more | 2.1 | 8fcee85 | logic:sieve-31b_en, logic:sieve-31b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 71.4 | 6 of 69tied with REAX 1.3, KnowLine-31B-Gen1.5, Copernicus-31B | 4.3 | 8fcee85 | math:sieve-31b_en, math:sieve-31b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 63.2 | 6 of 69tied with REAX 1.3, KnowLine-31B-Gen1.5, Copernicus-31B and 1 more | 2.6 | 8fcee85 | logic:sieve-31b_en, math:sieve-31b_en, logic:sieve-31b_en_dontknow, math:sieve-31b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 77.3 | 2 of 41tied with KnowLine-31B-Gen1.5, REAX 1.3, Gemini Decision 1 · 20261111 and 2 more | – | 8fcee85 | backgammon:sieve-31b-v3League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #44. sthanika-ai/Sieve-31B-v3 @ 8fcee85 on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through the server its card names (sieve-serve, sieve-decisions 0.5.1 at github.com/sthanika-ai/Sieve 1e64657, defaults), one request at a time; never on a hosted API. First stage 12 Nov 2026: its 9 hosted and program opponents (#1221–#1229, runner a127626); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1411–#1418, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1230 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Serving policy one read and 2 option orders averaged; probabilities rounded to 4 places. Its replies carry no input-token count, so the runner records 0 input tokens (per-token figures unknown, not zero). Never truncates (no token limit of its own; a request it cannot answer is refused, 422); nothing refused. Its first stage was stopped once after 12 h with pairs open (13 Nov 08:47 UTC) and finished at 11:29 UTC. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 88.1 | 4 of 55tied with Karpo Decision 31B, Copernicus-31B, KnowLine-31B-Gen1.5 and 6 more | 7.0 | 8fcee85 | grading:site-sieve-31b-v3-v1sthanika-ai/Sieve-31B-v3@8fcee858, sieve-serve 0.5.1 (text only, design 'letters'), BF16, on one DGX Spark (ours) as for its published runs, self-hosted; the server reports no input-token count; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
Gemma 4 31B reasoning off · chat
- Maker: Google
- Kind: chat model · class: Medium
- Size: 31.3B
- Repo: google/gemma-4-31B-it
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 8.5 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 82.1TR 66.4 L&M 96.7 BG 83.4 | 7 of 9 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 66.4 | 23 of 40tied with Qwen3.8-27B · low · chat, GLM-5.3 · low · chat, MiMo-V2.6-Flash · high · chat | 24.2 | 842da37 | text:119, text:189vLLM, self-hosted, FP8 |
| Turkish Reading | 88.6 | 17 of 40tied with Kimi K3.1 Pro · low · chat, DeepSeek-V4.1-Flash · reasoning on · chat · first release, Qwen3.5-397B-A17B · high · chat and 3 more | 8.1 | 842da37 | text:119, text:189vLLM, self-hosted, FP8 |
| Turkish Grammar | 57.0 | 24 of 40tied with MiMo-V2.6-Flash · high · chat | 31.0 | 842da37 | text:119, text:189vLLM, self-hosted, FP8 |
| English Reading | 88.1 | 17 of 47tied with Grok 4.7 · high · chat, Claude Haiku 5.5 · max · chat, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 and 8 more | 8.5 | 842da37 | english:774vLLM, self-hosted, FP8; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 91.7 | 19 of 35tied with Inkling 2 Small · high · chat, Claude Sonnet 5.5 · high · chat, Muse Ember · max · chat and 11 more | – | 842da37 | logic:gemma-4-31b-it_en_t0, logic:gemma-4-31b-it_en_t0_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); BF16 weights quantized to FP8 by vLLM at load |
| Math | 99.2 | 17 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | 842da37 | math:gemma-4-31b-it_en_t0, math:gemma-4-31b-it_en_t0_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); BF16 weights quantized to FP8 by vLLM at load |
| Logic & Math (in Combined) | 96.7 | 17 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 17 more | – | 842da37 | logic:gemma-4-31b-it_en_t0, math:gemma-4-31b-it_en_t0, logic:gemma-4-31b-it_en_t0_dontknow, math:gemma-4-31b-it_en_t0_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); BF16 weights quantized to FP8 by vLLM at load |
| Backgammon | 83.4 | 10 of 16tied with Grok 4.7 · high · chat, DeepSeek-V4.1-Flash · low · API + NVIDIA RTX PRO 6000 Blackwell · first release, Muse Ember · max · chat | – | – | backgammon:gemma-4-31b-onLeague 2; calibration definition pending Ümit; run on our GPUs; setting: on; improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
MiMo-V2.6-Flash high · chat
- Maker: Xiaomi
- Kind: chat model · class: Large
- Size: 309B (15B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 65.1 | 24 of 40tied with Gemma 4 31B · reasoning off · chat | 7.1 | – | text:877OpenRouter chat API, Xiaomi's own endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included |
| Turkish Reading | 83.2 | 26 of 40tied with GLM-5.3-Flash · reasoning on · chat, Qwen3.8-27B · low · chat, GLM-5.3 · low · chat and 5 more | 3.7 | – | text:877OpenRouter chat API, Xiaomi's own endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included |
| Turkish Grammar | 57.4 | 23 of 40tied with Gemma 4 31B · reasoning off · chat | 10.3 | – | text:877OpenRouter chat API, Xiaomi's own endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included |
| English Reading | 86.3 | 20 of 47tied with Kimi K3.1 Pro · low · chat, Qwen3.5-397B-A17B · high · chat, Gemma 4 31B · reasoning off · chat and 10 more | 6.5 | – | english:878OpenRouter chat API, Xiaomi's own endpoint (fp8), fallbacks off, output cap 65,536 tokens, reasoning included; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 93.9 | 13 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 16 more | – | – | logic:mimo-v2.6-flash-64k_en, logic:mimo-v2.6-flash-64k_en_dontknowhosted API via OpenRouter |
| Math | 99.3 | 15 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:mimo-v2.6-flash-64k_en, math:mimo-v2.6-flash-64k_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 97.5 | 13 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 17 more | – | – | logic:mimo-v2.6-flash-64k_en, math:mimo-v2.6-flash-64k_en, logic:mimo-v2.6-flash-64k_en_dontknow, math:mimo-v2.6-flash-64k_en_dontknowhosted API via OpenRouter |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
GLiDE 27B
- Maker: Fastino
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: fastino/GLiDE-27B
- Latency ↓: median 542 ms, p95 739 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 899)
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 73.4 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 71.9EN 88.6 Logic 48.0 Math 65.0 AC 86.1 | 8 of 53tied with Logits Sigma-1 Duo, d1-ultra, Perplexity Decider 27B v1.3 | – | – | – |
| Reading · Turkish (in Combined) | 63.9 | 11 of 76 | 2.3 | 4d32743 | text:899GLiDE server (Fastino's release code), self-hosted |
| Turkish Reading | 88.6 | 11 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, Copernicus-31B and 19 more | 2.4 | 4d32743 | text:899GLiDE server (Fastino's release code), self-hosted |
| Turkish Grammar | 53.4 | 11 of 76 | 2.7 | 4d32743 | text:899GLiDE server (Fastino's release code), self-hosted |
| English Reading | 88.6 | 8 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 21 more | 1.7 | 4d32743 | english:901GLiDE server (Fastino's release code), self-hosted; the kept reading questions in English |
| Vision | 37.4 | 3 of 20tied with d1-ultra, KnowLine-31B-Gen1.5, d3 | 4.3 | 4d32743 | vision:68GLiDE server (Fastino's release code), self-hosted; the page image only (condition B) |
| Logic | 48.0 | 6 of 69tied with Karpo Decision 31B, REAX 1.3, KnowLine-31B-Gen1.5 and 4 more | 4.4 | 4d32743 | logic:glide-27b_en, logic:glide-27b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 65.0 | 11 of 69tied with Perplexity Decider 27B v1.3, d3-ultra, Gemini Decision 1 · 20261111 and 3 more | 4.1 | 4d32743 | math:glide-27b_en, math:glide-27b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 59.2 | 9 of 69tied with d1-ultra, Mammoth-1, Perplexity Decider 27B v1.3 | 4.1 | 4d32743 | logic:glide-27b_en, math:glide-27b_en, logic:glide-27b_en_dontknow, math:glide-27b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 86.1 | 10 of 55tied with Sieve 31B v3, KnowLine-31B-Gen2, REAX 1.3 and 9 more | 3.6 | 4d32743 | grading:site-glide-27b-v1fastino/GLiDE-27B@4d32743a, GLiDE server (glide.server), BF16, GLIDE_MAX_BATCH=1, on one DGX Spark (ours) as for its published runs, self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
Perplexity Decider 27B v1.2 API · with web
- Maker: Perplexity
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 60.9 | 12 of 76tied with GLiDE · v1.2, Perplexity Decider 27B v1.3, d1.2-ultra and 7 more | 3.6 | – | text:640Perplexity Decisions API |
| Turkish Reading | 86.1 | 21 of 76tied with Logits Sigma-1 Duo, Sieve 31B v3, GLiDE 27B and 16 more | 2.7 | – | text:640Perplexity Decisions API |
| Turkish Grammar | 50.2 | 12 of 76tied with Perplexity Decider 27B v1.3, GLiDE · v1.2, GLiDE · API and 5 more | 4.9 | – | text:640Perplexity Decisions API |
| English Reading | 88.1 | 12 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 21 more | 3.6 | – | english:639Perplexity Decisions API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 40.7 | 19 of 69tied with Drex Max v1.1, Perplexity Decider 27B v1.3, d3-ultra and 9 more | 4.2 | – | logic:decider-v1.2-web_en, logic:decider-v1.2-web_en_dontknowhosted API (the maker's service) |
| Math | 61.6 | 23 of 69tied with GLiDE · API, d3, Drex Max and 8 more | 3.7 | – | math:decider-v1.2-web_en, math:decider-v1.2-web_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 54.5 | 21 of 69tied with Drex Max v1.1, GLiDE · v1.2, Drex Max and 6 more | 3.7 | – | logic:decider-v1.2-web_en, math:decider-v1.2-web_en, logic:decider-v1.2-web_en_dontknow, math:decider-v1.2-web_en_dontknowhosted API (the maker's service) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
GLiDE v1.2
- Maker: Fastino
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: fastino/GLiDE
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 62.6TR 60.6 L&M 56.1 BG 71.1 | 10 of 36tied with Drex Max, Drex Max v1.1 | – | – | – |
| Decision Index4 benches | 69.8EN 87.9 Logic 44.0 Math 62.3 AC 84.9 | 11 of 53tied with d1-ultra, Perplexity Decider 27B v1.3, Turion Jev 31B and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 60.6 | 13 of 76tied with Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, d1.2-ultra and 8 more | 2.9 | a7e4589 | text:248GLiDE server (Fastino's release code), self-hosted |
| Turkish Reading | 87.0 | 16 of 76tied with Turion Jev 31B, REAX 1.3, Gemini Decision 1 · 20261111 and 17 more | 3.5 | a7e4589 | text:248GLiDE server (Fastino's release code), self-hosted |
| Turkish Grammar | 49.3 | 14 of 76tied with Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, GLiDE · API and 7 more | 3.0 | a7e4589 | text:248GLiDE server (Fastino's release code), self-hosted |
| English Reading | 87.9 | 13 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 21 more | 2.7 | a7e4589 | english:521GLiDE server (Fastino's release code), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 44.0 | 9 of 69tied with d1-ultra, Drex Max v1.1, Perplexity Decider 27B v1.3 and 6 more | 5.9 | a7e4589 | logic:glide-v1.2_en, logic:glide-v1.2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 62.3 | 19 of 69tied with d3-ultra, Gemini Decision 1 · 20261111, GLiDE · API and 10 more | 5.2 | a7e4589 | math:glide-v1.2_en, math:glide-v1.2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 56.1 | 16 of 69tied with d3-ultra, GLiDE · API, d3 and 6 more | 5.1 | a7e4589 | logic:glide-v1.2_en, math:glide-v1.2_en, logic:glide-v1.2_en_dontknow, math:glide-v1.2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | 71.1 | 13 of 41tied with Jev 1.14, GLiDE · API, Jev 1.13 and 3 more | – | a7e4589 | backgammon:fastino-glide-v1-2-localLeague 1; calibration definition pending Ümit; run on our GPUs |
| Answer Check | 84.9 | 13 of 55tied with REAX 1.3, Turion Jev 31B, d1.2-ultra and 11 more | 4.7 | a7e4589 | grading:site-fastino-glide-v1.2-v1fastino/GLiDE@a7e4589, GLiDE server (Fastino's release code), BF16, GLIDE_MAX_BATCH=1, on one RTX PRO 6000 (ours) as the backgammon league's pinned server, self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 2 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
Perplexity Decider 27B v1.3
- Maker: Perplexity
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: perplexity-ai/pplx-decider-v1.3
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 70.6EN 87.0 Logic 43.4 Math 65.6 AC 86.2 | 10 of 53tied with GLiDE 27B, d1-ultra, GLiDE · v1.2 and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 60.5 | 14 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, d1.2-ultra and 8 more | 5.0 | 9c2ebb2 | text:1134Perplexity Decider server, self-hosted |
| Turkish Reading | 86.3 | 20 of 76tied with Gemini Decision 1 · 20261111, d1-ultra, Logits Sigma-1 Duo and 18 more | 2.4 | 9c2ebb2 | text:1134Perplexity Decider server, self-hosted |
| Turkish Grammar | 49.6 | 13 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, GLiDE · API and 6 more | 6.1 | 9c2ebb2 | text:1134Perplexity Decider server, self-hosted |
| English Reading | 87.0 | 18 of 75tied with Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2 and 26 more | 3.4 | 9c2ebb2 | english:1137Perplexity Decider server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 43.4 | 12 of 69tied with GLiDE · v1.2, d1-ultra, Drex Max v1.1 and 7 more | 9.0 | 9c2ebb2 | logic:decider-v1.3-local_en, logic:decider-v1.3-local_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 65.6 | 10 of 69tied with d1-ultra, Mammoth-1, GLiDE 27B and 3 more | 4.5 | 9c2ebb2 | math:decider-v1.3-local_en, math:decider-v1.3-local_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 58.0 | 11 of 69tied with d1-ultra, GLiDE 27B, Mammoth-1 and 2 more | 5.9 | 9c2ebb2 | logic:decider-v1.3-local_en, math:decider-v1.3-local_en, logic:decider-v1.3-local_en_dontknow, math:decider-v1.3-local_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 86.2 | 9 of 55tied with Sieve 31B v3, KnowLine-31B-Gen2, REAX 1.3 and 8 more | 2.8 | 9c2ebb2 | grading:site-perplexity-decider-27b-v1.3-v1perplexity-ai/pplx-decider-v1.3@9c2ebb27, the runner's pplx_decider_server.py with the checkpoint's own pplx_decider package (as for its published v1.2 runs), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
GPT-6 Luna low · chat
- Maker: OpenAI
- Kind: chat model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 60.4 | 25 of 40 | 25.7 | – | text:133Codex CLI, ChatGPT subscription |
| Turkish Reading | 84.2 | 22 of 40tied with Muse Ember · max · chat, GLM-5.3-Flash · reasoning on · chat, Qwen3.8-27B · low · chat and 6 more | 9.0 | – | text:133Codex CLI, ChatGPT subscription |
| Turkish Grammar | 50.2 | 25 of 40tied with MiniMax-M3.1-Flash · high · chat | 32.7 | – | text:133Codex CLI, ChatGPT subscription |
| English Reading | 82.9 | 27 of 47tied with GLM-5.3-Flash · reasoning on · chat, Muse Ember · max · chat, MiMo-V2.6-Flash · high · chat and 8 more | 10.1 | – | english:617Codex CLI, ChatGPT subscription; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 47.4 | 34 of 35tied with LFM2.5-8B-A1B · default reasoning · chat | – | – | logic:gpt-6-luna-low-sub_en, logic:gpt-6-luna-low-sub_en_dontknowhosted API (the maker's service) |
| Math | 92.1 | 34 of 36tied with Qwen3.5-2B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat | – | – | math:gpt-6-luna-low-sub_en, math:gpt-6-luna-low-sub_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 76.8 | 32 of 34tied with Raon-1 nano · reasoning off · chat | – | – | logic:gpt-6-luna-low-sub_en, math:gpt-6-luna-low-sub_enhosted API (the maker's service); no complete 'I don't know' run |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
d1.2-ultra
- Maker: Liquid AI
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 69.0EN 87.9 Logic 39.4 Math 62.3 AC 86.4 | 13 of 53tied with GLiDE · v1.2, Turion Jev 31B | – | – | – |
| Reading · Turkish (in Combined) | 60.1 | 15 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, Perplexity Decider 27B v1.3 and 8 more | 3.4 | – | text:1347Liquid AI decisions API |
| Turkish Reading | 87.2 | 13 of 76tied with KnowLine-31B-Gen1.5, Turion Jev 31B, REAX 1.3 and 18 more | 5.6 | – | text:1347Liquid AI decisions API |
| Turkish Grammar | 48.6 | 16 of 76tied with Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, GLiDE · v1.2 and 8 more | 3.1 | – | text:1347Liquid AI decisions API |
| English Reading | 87.9 | 14 of 75tied with KnowLine-31B-Gen1.5, Copernicus-31B, REAX 1.3 and 21 more | 8.5 | – | english:1359Liquid AI decisions API; the kept reading questions in English |
| Vision | 31.7 | 10 of 20tied with d1.2, Drex Max, SubQ Snap 1.1 and 3 more | 2.8 | – | vision:570Liquid AI decisions API; the page image only (condition B) |
| Logic | 39.4 | 22 of 69tied with Perplexity Decider 27B v1.2 · DGX Spark, Drex Max, Perplexity Decider 27B v1.2 · API · with web and 4 more | 3.0 | – | logic:d1.2-ultra_en, logic:d1.2-ultra_en_dontknowhosted API (the maker's service) |
| Math | 62.3 | 21 of 69tied with d3-ultra, Gemini Decision 1 · 20261111, GLiDE · API and 10 more | 4.6 | – | math:d1.2-ultra_en, math:d1.2-ultra_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 54.5 | 20 of 69tied with d3, Drex Max v1.1, GLiDE · v1.2 and 7 more | 3.2 | – | logic:d1.2-ultra_en, math:d1.2-ultra_en, logic:d1.2-ultra_en_dontknow, math:d1.2-ultra_en_dontknowhosted API (the maker's service) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 86.4 | 8 of 55tied with Sieve 31B v3, KnowLine-31B-Gen2, REAX 1.3 and 8 more | 8.8 | – | grading:site-liquid-d1.2-ultra-v1Liquid AI decisions API (api.liquid.ai/decisions, model d1.2-ultra); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
GLiDE API
- Maker: Fastino
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableNo combined score: Fastino's API served a changed model between our benches' runs (Reading 6 Oct, Logic & Math 7 Nov, Backgammon 20 Oct to 7 Nov), so its bench scores are not one model's. | ||||
| Decision Index4 benches | Not applicableNo Decision Index: Fastino's API served a changed model between our benches' runs (Reading 6 Oct, Logic & Math 7 Nov, Backgammon 20 Oct to 7 Nov), so its bench scores are not one model's. | ||||
| Reading · Turkish (in Combined) | 59.8 | 16 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, Perplexity Decider 27B v1.3 and 8 more | 4.5 | – | text:129Fastino System One API |
| Turkish Reading | 84.8 | 23 of 76tied with GLiDE 27B, Mammoth-1, d1.2-ultra and 19 more | 5.8 | – | text:129Fastino System One API |
| Turkish Grammar | 49.3 | 15 of 76tied with Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, GLiDE · v1.2 and 7 more | 4.1 | – | text:129Fastino System One API |
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 43.1 | 14 of 69tied with GLiDE · v1.2, d1-ultra, Drex Max v1.1 and 9 more | 6.6 | – | logic:glide_en, logic:glide_en_dontknowhosted API (the maker's service) |
| Math | 63.5 | 14 of 69tied with Perplexity Decider 27B v1.3, GLiDE 27B, d3-ultra and 10 more | 5.0 | – | math:glide_en, math:glide_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 56.5 | 13 of 69tied with Perplexity Decider 27B v1.3, d3-ultra, d3 and 5 more | 5.3 | – | logic:glide_en, math:glide_en, logic:glide_en_dontknow, math:glide_en_dontknowhosted API (the maker's service) |
| Backgammon | 71.9 | 11 of 41tied with Jev 1.14, Jev 1.13, GLiDE · v1.2 and 1 more | – | – | backgammon:fastino-glideLeague 2; calibration definition pending Ümit; hosted (API or subscription) |
| Answer Check | 85.1 | 12 of 55tied with REAX 1.3, Turion Jev 31B, d1.2-ultra and 11 more | 3.0 | – | grading:site-fastino-glide-v1Fastino decisions API (api.fastino.ai, model fastino/GLiDE); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
d3 fine-tune
- Maker: vLLM Semantic Router · training: fine-tune
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: vllm-sr/d3
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 59.7 | 17 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, Perplexity Decider 27B v1.3 and 8 more | 2.6 | 20f37b9 | text:259vLLM System One endpoint (nightly), self-hosted, BF16 |
| Turkish Reading | 86.5 | 18 of 76tied with Gemini Decision 1 · 20261111, d1-ultra, Logits Sigma-1 Duo and 15 more | 3.5 | 20f37b9 | text:259vLLM System One endpoint (nightly), self-hosted, BF16 |
| Turkish Grammar | 48.4 | 17 of 76tied with Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, GLiDE · v1.2 and 8 more | 2.4 | 20f37b9 | text:259vLLM System One endpoint (nightly), self-hosted, BF16 |
| English Reading | 87.2 | 16 of 75tied with Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2 and 24 more | 3.6 | 20f37b9 | english:554vLLM System One endpoint (nightly), self-hosted, BF16; the kept reading questions in English |
| Vision | 35.6 | 5 of 20tied with d1-ultra, GLiDE 27B, KnowLine-31B-Gen1.5 and 3 more | 3.7 | 20f37b9 | vision:41vLLM System One endpoint (nightly), self-hosted, BF16; the page image only (condition B) |
| Logic | 43.1 | 16 of 69tied with GLiDE · v1.2, d1-ultra, Drex Max v1.1 and 9 more | 4.6 | 20f37b9 | logic:d3_en, logic:d3_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 63.0 | 15 of 69tied with GLiDE 27B, d3-ultra, Gemini Decision 1 · 20261111 and 10 more | 4.5 | 20f37b9 | math:d3_en, math:d3_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 56.2 | 14 of 69tied with d3-ultra, GLiDE · API, Drex Max v1.1 and 5 more | 4.4 | 20f37b9 | logic:d3_en, math:d3_en, logic:d3_en_dontknow, math:d3_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
d3-ultra fine-tune
- Maker: vLLM Semantic Router · training: fine-tune
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: vllm-sr/d3-ultra
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 70.2 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 59.7 | 18 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, Perplexity Decider 27B v1.3 and 8 more | 12.5 | 52ba9e7 | text:269d3 package server (transformers, the release code), self-hosted, BF16 |
| Turkish Reading | 87.2 | 14 of 76tied with KnowLine-31B-Gen1.5, Turion Jev 31B, REAX 1.3 and 18 more | 3.8 | 52ba9e7 | text:269d3 package server (transformers, the release code), self-hosted, BF16 |
| Turkish Grammar | 48.0 | 19 of 76tied with Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, GLiDE · v1.2 and 8 more | 16.3 | 52ba9e7 | text:269d3 package server (transformers, the release code), self-hosted, BF16 |
| English Reading | 86.7 | 19 of 75tied with REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B and 26 more | 2.8 | 52ba9e7 | english:550d3 package server (transformers, the release code), self-hosted, BF16; the kept reading questions in English |
| Vision | 31.8 | 9 of 20tied with d1.2, Drex Max, SubQ Snap 1.1 and 3 more | 18.8 | 52ba9e7 | vision:29d3 package server (transformers, the release code), self-hosted, BF16; the page image only (condition B) |
| Logic | 43.2 | 13 of 69tied with GLiDE · v1.2, d1-ultra, Drex Max v1.1 and 9 more | 15.1 | 52ba9e7 | logic:d3-ultra_en, logic:d3-ultra_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 64.4 | 12 of 69tied with Perplexity Decider 27B v1.3, GLiDE 27B, Gemini Decision 1 · 20261111 and 8 more | 8.6 | 52ba9e7 | math:d3-ultra_en, math:d3-ultra_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 57.2 | 12 of 69tied with Perplexity Decider 27B v1.3, GLiDE · API, d3 and 3 more | 10.1 | 52ba9e7 | logic:d3-ultra_en, math:d3-ultra_en, logic:d3-ultra_en_dontknow, math:d3-ultra_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Drex Max fine-tune
- Maker: Nace.AI · training: fine-tune
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: nace-ai/drex-max-v1
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 63.4TR 59.7 L&M 55.5 BG 74.9 | 8 of 36tied with Drex Max v1.1, GLiDE · v1.2 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 59.7 | 19 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, Perplexity Decider 27B v1.3 and 8 more | 6.3 | cac65e1 | text:322Drex Max server (Nace's loader, drex_max), self-hosted, BF16 |
| Turkish Reading | 87.0 | 15 of 76tied with Turion Jev 31B, REAX 1.3, Gemini Decision 1 · 20261111 and 17 more | 3.4 | cac65e1 | text:322Drex Max server (Nace's loader, drex_max), self-hosted, BF16 |
| Turkish Grammar | 48.0 | 18 of 76tied with Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, GLiDE · v1.2 and 8 more | 8.0 | cac65e1 | text:322Drex Max server (Nace's loader, drex_max), self-hosted, BF16 |
| English Reading | 87.0 | 17 of 75tied with Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2 and 26 more | 2.6 | cac65e1 | english:552Drex Max server (Nace's loader, drex_max), self-hosted, BF16; the kept reading questions in English |
| Vision | 33.5 | 7 of 20tied with d3, d1.2, SubQ Snap 1.1 and 3 more | 9.3 | cac65e1 | vision:56Drex Max server (Nace's loader, drex_max), self-hosted, BF16; the page image only (condition B) |
| Logic | 41.3 | 18 of 69tied with GLiDE · v1.2, d1-ultra, Drex Max v1.1 and 10 more | 16.2 | cac65e1 | logic:drex-max-v1_en, logic:drex-max-v1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); served with Nace's drex-decision-models code at e206b27 |
| Math | 62.9 | 16 of 69tied with GLiDE 27B, d3-ultra, Gemini Decision 1 · 20261111 and 10 more | 11.8 | cac65e1 | math:drex-max-v1_en, math:drex-max-v1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); served with Nace's drex-decision-models code at e206b27 |
| Logic & Math (in Combined) | 55.5 | 17 of 69tied with d3-ultra, GLiDE · API, d3 and 7 more | 13.3 | cac65e1 | logic:drex-max-v1_en, math:drex-max-v1_en, logic:drex-max-v1_en_dontknow, math:drex-max-v1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); served with Nace's drex-decision-models code at e206b27 |
| Backgammon | 74.9 | 7 of 41tied with REAX 1.3, Gemini Decision 1 · 20261111, Mammoth-1 and 4 more | – | cac65e1 | backgammon:drex-max-v1League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 18 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 6 Nov 2026. Probabilities are stored as returned. Its pair with GPT-6 Luna was not played: GPT-6 Luna's API was out of credit; the pair is outside the requested scope (Ümit, 12 Nov).; run on our GPUs |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Drex Max v1.1
- Maker: Nace.AI
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 63.2TR 59.2 L&M 56.1 BG 74.5 | 9 of 36tied with Drex Max, GLiDE · v1.2 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 59.2 | 20 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, Perplexity Decider 27B v1.3 and 8 more | 2.9 | – | text:326Nace.AI System One API |
| Turkish Reading | 85.7 | 22 of 76tied with GLiDE 27B, Mammoth-1, d1.2-ultra and 15 more | 3.4 | – | text:326Nace.AI System One API |
| Turkish Grammar | 48.0 | 20 of 76tied with Perplexity Decider 27B v1.2 · API · with web, Perplexity Decider 27B v1.3, GLiDE · v1.2 and 8 more | 2.7 | – | text:326Nace.AI System One API |
| English Reading | 85.6 | 22 of 75tied with Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B and 35 more | 3.8 | – | english:399Nace.AI System One API; the kept reading questions in English |
| Vision | 31.4 | 11 of 20tied with d1.2, Drex Max, SubQ Snap 1.1 and 3 more | 5.2 | – | vision:58Nace.AI System One API; the page image only (condition B) |
| Logic | 43.4 | 11 of 69tied with GLiDE · v1.2, d1-ultra, Perplexity Decider 27B v1.3 and 7 more | 6.1 | – | logic:drex-max-1.1_en, logic:drex-max-1.1_en_dontknowhosted API (the maker's service) |
| Math | 62.6 | 17 of 69tied with d3-ultra, Gemini Decision 1 · 20261111, GLiDE · API and 10 more | 5.7 | – | math:drex-max-1.1_en, math:drex-max-1.1_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 56.1 | 15 of 69tied with d3-ultra, GLiDE · API, d3 and 6 more | 5.8 | – | logic:drex-max-1.1_en, math:drex-max-1.1_en, logic:drex-max-1.1_en_dontknow, math:drex-max-1.1_en_dontknowhosted API (the maker's service) |
| Backgammon | 74.5 | 8 of 41tied with Mammoth-1, Turion Jev 31B, Drex Max and 2 more | – | – | backgammon:drex-max-1-1League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 18 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 6 Nov 2026. Probabilities are stored as returned. Its pair with GPT-6 Luna was not played: GPT-6 Luna's API was out of credit; the pair is outside the requested scope (Ümit, 12 Nov).; hosted (API or subscription) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Mammoth-1 post-trained
- Maker: Base Labs · training: post-trained
- Kind: decision model · class: Medium
- Size: 27.4B
- Repo: baselabs/mammoth-1
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 64.8TR 58.9 L&M 59.1 BG 76.4 | 7 of 36 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 58.9 | 21 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, Perplexity Decider 27B v1.3 and 8 more | 5.6 | 61cec44 | text:374Base Labs server (qev, the model card's loader), self-hosted, BF16 |
| Turkish Reading | 88.0 | 12 of 76tied with Karpo Decision 31B, KnowLine-31B-Gen2, Copernicus-31B and 20 more | 2.0 | 61cec44 | text:374Base Labs server (qev, the model card's loader), self-hosted, BF16 |
| Turkish Grammar | 46.5 | 23 of 76tied with d1.2-ultra, d3, Drex Max and 5 more | 7.9 | 61cec44 | text:374Base Labs server (qev, the model card's loader), self-hosted, BF16 |
| English Reading | 87.7 | 15 of 75tied with Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2 and 22 more | 2.4 | 61cec44 | english:503Base Labs server (qev, the model card's loader), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 43.1 | 15 of 69tied with GLiDE · v1.2, d1-ultra, Drex Max v1.1 and 9 more | 5.3 | 61cec44 | logic:mammoth-1_en, logic:mammoth-1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 67.5 | 9 of 69tied with Logits Sigma-1 Duo, d1-ultra, Perplexity Decider 27B v1.3 | 3.0 | 61cec44 | math:mammoth-1_en, math:mammoth-1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 59.1 | 10 of 69tied with d1-ultra, GLiDE 27B, Perplexity Decider 27B v1.3 | 3.2 | 61cec44 | logic:mammoth-1_en, math:mammoth-1_en, logic:mammoth-1_en_dontknow, math:mammoth-1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | 76.4 | 5 of 41tied with KnowLine-31B-Gen1.5, Sieve 31B v3, REAX 1.3 and 5 more | – | 61cec44 | backgammon:base-labs-mammoth-1League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 2 Nov 2026. Probabilities come rounded to 6 decimals by the loader and are stored as returned.; run on our GPUs |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Perplexity Decider 27B v1.2 DGX Spark
- Maker: Perplexity
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: perplexity-ai/pplx-decider-v1.2
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Setups: Logic and Math: a hosted API; the other benches: our hardware.
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 61.2TR 58.8 L&M 55.2 BG 69.5 | 11 of 36 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 58.8 | 22 of 76tied with Perplexity Decider 27B v1.2 · API · with web, GLiDE · v1.2, Perplexity Decider 27B v1.3 and 8 more | 4.4 | 225c913 | text:217, text:633Perplexity Decider server, self-hosted |
| Turkish Reading | 86.5 | 17 of 76tied with Gemini Decision 1 · 20261111, d1-ultra, Logits Sigma-1 Duo and 15 more | 3.1 | 225c913 | text:217, text:633Perplexity Decider server, self-hosted |
| Turkish Grammar | 47.1 | 21 of 76tied with Perplexity Decider 27B v1.3, GLiDE · v1.2, GLiDE · API and 8 more | 6.2 | 225c913 | text:217, text:633Perplexity Decider server, self-hosted |
| English Reading | 85.8 | 21 of 75tied with Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B and 34 more | 3.7 | 225c913 | english:527Perplexity Decider server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 41.7 | 17 of 69tied with GLiDE · v1.2, d1-ultra, Drex Max v1.1 and 10 more | 3.4 | – | logic:decider-v1.2_en, logic:decider-v1.2_en_dontknowhosted API (the maker's service); other setups of this model under its row, not in this entry: Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) |
| Math | 62.2 | 22 of 69tied with Gemini Decision 1 · 20261111, GLiDE · API, d3 and 9 more | 3.2 | – | math:decider-v1.2_en, math:decider-v1.2_en_dontknowhosted API (the maker's service); other setups of this model under its row, not in this entry: Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) |
| Logic & Math (in Combined) | 55.2 | 18 of 69tied with GLiDE · API, d3, Drex Max v1.1 and 7 more | 3.1 | – | logic:decider-v1.2_en, math:decider-v1.2_en, logic:decider-v1.2_en_dontknow, math:decider-v1.2_en_dontknowhosted API (the maker's service); other setups of this model under its row, not in this entry: Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) |
| Backgammon | 69.5 | 15 of 41tied with Jev 1.13, GLiDE · v1.2, celeris-1-decision and 4 more | – | 225c913 | backgammon:decider-v1-2-27bLeague 1; calibration definition pending Ümit; run on our GPUs |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Perplexity Decider 27B v1.1
- Maker: Perplexity
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Setups: Backgammon: our hardware; the other benches: a hosted API.
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on English Reading, Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 58.6 | 23 of 76tied with GLiDE · v1.2, Perplexity Decider 27B v1.3, d1.2-ultra and 8 more | 3.8 | – | text:145Perplexity Decisions API |
| Turkish Reading | 86.3 | 19 of 76tied with Gemini Decision 1 · 20261111, d1-ultra, Logits Sigma-1 Duo and 18 more | 3.0 | – | text:145Perplexity Decisions API |
| Turkish Grammar | 46.8 | 22 of 76tied with GLiDE · v1.2, GLiDE · API, d1.2-ultra and 7 more | 5.5 | – | text:145Perplexity Decisions API |
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 68.0 | 19 of 41tied with Perplexity Decider 27B v1.2 · DGX Spark, Clef, Drex 2.0 and 3 more | – | – | backgammon:decider-v1-1-27bLeague 1; calibration definition pending Ümit; run on our GPUs; route: our GPUs (released checkpoint); the other benches' row is the Perplexity Decisions API |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
MiniMax-M3.1-Flash high · chat
- Maker: MiniMax
- Kind: chat model · class: Large
- Size: 428B (23B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 58.1 | 26 of 40 | 21.1 | – | text:862OpenRouter chat API, MiniMax's own endpoint, fallbacks off, output cap 65,536 tokens, reasoning included; JSON mode without the answer schema (the endpoint has no structured outputs), so the prompt adds a paragraph stating the answer layout |
| Turkish Reading | 78.7 | 28 of 40tied with Claude Sonnet 5.5 · high · chat | 9.5 | – | text:862OpenRouter chat API, MiniMax's own endpoint, fallbacks off, output cap 65,536 tokens, reasoning included; JSON mode without the answer schema (the endpoint has no structured outputs), so the prompt adds a paragraph stating the answer layout |
| Turkish Grammar | 49.3 | 26 of 40tied with GPT-6 Luna · low · chat | 26.1 | – | text:862OpenRouter chat API, MiniMax's own endpoint, fallbacks off, output cap 65,536 tokens, reasoning included; JSON mode without the answer schema (the endpoint has no structured outputs), so the prompt adds a paragraph stating the answer layout |
| English Reading | 80.6 | 29 of 47tied with GLM-5.3 · low · chat, Qwen3.5-4B · reasoning on · chat, Qwen3.8 Flash · no reasoning · chat and 5 more | 9.0 | – | english:863OpenRouter chat API, MiniMax's own endpoint, fallbacks off, output cap 65,536 tokens, reasoning included; JSON mode without the answer schema (the endpoint has no structured outputs), so the prompt adds a paragraph stating the answer layout; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 92.4 | 17 of 35tied with Qwen4-Max · high · chat, Claude Opus 5.5 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 and 15 more | – | – | logic:minimax-m3.1-flash_en, logic:minimax-m3.1-flash_en_dontknowhosted API via OpenRouter |
| Math | 97.8 | 27 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 25 more | – | – | math:minimax-m3.1-flash_en, math:minimax-m3.1-flash_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 96.0 | 20 of 34tied with MiMo-V2.6-Flash · high · chat, Qwen3.8 Flash · no reasoning · chat, Qwen3.8-27B · low · chat and 6 more | – | – | logic:minimax-m3.1-flash_en, math:minimax-m3.1-flash_en, logic:minimax-m3.1-flash_en_dontknow, math:minimax-m3.1-flash_en_dontknowhosted API via OpenRouter |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
SubQ Snap Preview
- Maker: Subquadratic
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 56.6 | 24 of 76tied with Perplexity Decider 27B v1.1, SubQ Snap 1.1, GPT-6 Luna · Decisions API | 4.9 | – | text:177SubQ Decisions API (public preview) |
| Turkish Reading | 84.6 | 24 of 76tied with Mammoth-1, d1.2-ultra, d3-ultra and 19 more | 4.2 | – | text:177SubQ Decisions API (public preview) |
| Turkish Grammar | 44.6 | 24 of 76tied with Perplexity Decider 27B v1.2 · DGX Spark, Perplexity Decider 27B v1.1, Mammoth-1 and 2 more | 6.0 | – | text:177SubQ Decisions API (public preview) |
| English Reading | 82.4 | 38 of 75tied with d1.2, Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1 and 31 more | 3.2 | – | english:595, english:597, english:598, english:599, english:600SubQ Decisions API (public preview); the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 63.8 | 23 of 41tied with GPT-6 Luna · Decisions API, Elephant-1 | – | – | backgammon:subq-snap-previewLeague 1; calibration definition pending Ümit; hosted (API or subscription) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
SubQ Snap 1.1
- Maker: Subquadratic
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 58.5TR 54.8 L&M 53.8 BG 66.8 | 12 of 36tied with celeris-1-decision | – | – | – |
| Decision Index4 benches | 67.1EN 82.4 Logic 40.4 Math 60.7 AC 84.9 | 14 of 53tied with Gemini Decision 1 · 20261111 | – | – | – |
| Reading · Turkish (in Combined) | 54.8 | 25 of 76tied with SubQ Snap Preview, GPT-6 Luna · Decisions API, KnowLine-12B-Gen1 | 11.2 | – | text:346SubQ Decisions API (public preview) |
| Turkish Reading | 83.8 | 25 of 76tied with d1.2-ultra, d3-ultra, Drex Max and 19 more | 3.6 | – | text:346SubQ Decisions API (public preview) |
| Turkish Grammar | 42.5 | 26 of 76tied with SubQ Snap Preview, GPT-6 Luna · Decisions API, KnowLine-12B-Gen1 | 14.5 | – | text:346SubQ Decisions API (public preview) |
| English Reading | 82.4 | 37 of 75tied with d1.2, Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1 and 31 more | 3.2 | – | english:413SubQ Decisions API (public preview); the kept reading questions in English |
| Vision | 32.2 | 8 of 20tied with d3, d1.2, Drex Max and 4 more | 11.7 | – | vision:62SubQ Decisions API (public preview); the page image only (condition B) |
| Logic | 40.4 | 20 of 69tied with d3-ultra, GLiDE · API, Mammoth-1 and 8 more | 12.9 | – | logic:subq-1.1_en, logic:subq-1.1_en_dontknowhosted API (the maker's service); shown as one row with SubQ Snap Preview (the same answer to every question); every number from SubQ Snap 1.1 |
| Math | 60.7 | 25 of 69tied with Drex Max v1.1, celeris-1-decision, GLiDE · v1.2 and 5 more | 9.5 | – | math:subq-1.1_en, math:subq-1.1_en_dontknowhosted API (the maker's service); shown as one row with SubQ Snap Preview (the same answer to every question); every number from SubQ Snap 1.1 |
| Logic & Math (in Combined) | 53.8 | 23 of 69tied with Perplexity Decider 27B v1.2 · DGX Spark, Gemini Decision 1 · 20261111, d1.2-ultra and 4 more | 10.5 | – | logic:subq-1.1_en, math:subq-1.1_en, logic:subq-1.1_en_dontknow, math:subq-1.1_en_dontknowhosted API (the maker's service); shown as one row with SubQ Snap Preview (the same answer to every question); every number from SubQ Snap 1.1 |
| Backgammon | 66.8 | 21 of 41tied with Clef, Drex 2.0, Cygnet and 2 more | – | – | backgammon:subq-snap-1-1League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 2 Nov 2026. Probabilities arrive as 32-bit floats and are stored as returned (they sum to 1 only within 32-bit rounding).; hosted (API or subscription) |
| Answer Check | 84.9 | 14 of 55tied with REAX 1.3, Turion Jev 31B, d1.2-ultra and 11 more | 2.3 | – | grading:site-subq-snap-1.1-v1SubQ decisions API (api.subq.ai, model subq-snap-1.1); its probabilities were byte-identical to SubQ Snap Preview's on all 2,069 pairs; one question per request, 8 requests at a time (latency not comparable across different concurrency) |
GPT-6 Luna Decisions API
- Maker: OpenAI
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 54.5 | 26 of 76tied with SubQ Snap Preview, SubQ Snap 1.1, KnowLine-12B-Gen1 | 2.4 | – | text:143OpenAI Decisions API (public beta) |
| Turkish Reading | 81.9 | 29 of 76tied with Drex Max v1.1, GLiDE · API, SubQ Snap Preview and 15 more | 3.8 | – | text:143OpenAI Decisions API (public beta) |
| Turkish Grammar | 42.9 | 25 of 76tied with SubQ Snap Preview, SubQ Snap 1.1, KnowLine-12B-Gen1 | 3.0 | – | text:143OpenAI Decisions API (public beta) |
| English Reading | 80.1 | 45 of 75tied with StartLux-Decision-v1.1-9B-Instant, d1, Clef and 29 more | 3.2 | – | english:397OpenAI Decisions API (public beta); the kept reading questions in English |
| Vision | 15.8 | 14 of 20tied with KnowLine 4B Gen5.5 | 5.7 | – | vision:3, vision:4OpenAI Decisions API (public beta); the page image only (condition B) |
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 64.2 | 22 of 41tied with SubQ Snap Preview, Elephant-1 | – | – | backgammon:gpt-6-lunaLeague 1; calibration definition pending Ümit; hosted (API or subscription) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Qwen3.8 Flash no reasoning · chat
- Maker: Alibaba
- Kind: chat model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 2.7 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 74.1TR 53.8 L&M 97.0 BG 71.5 | 9 of 9 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 53.8 | 27 of 40 | 28.7 | – | text:985OpenRouter chat API, Alibaba's endpoint, fallbacks off, reasoning off |
| Turkish Reading | 84.0 | 24 of 40tied with Muse Ember · max · chat, GLM-5.3-Flash · reasoning on · chat, Qwen3.8-27B · low · chat and 6 more | 5.7 | – | text:985OpenRouter chat API, Alibaba's endpoint, fallbacks off, reasoning off |
| Turkish Grammar | 41.0 | 27 of 40 | 38.5 | – | text:985OpenRouter chat API, Alibaba's endpoint, fallbacks off, reasoning off |
| English Reading | 83.8 | 25 of 47tied with GLM-5.3-Flash · reasoning on · chat, Muse Ember · max · chat, MiMo-V2.6-Flash · high · chat and 8 more | 5.5 | – | english:986OpenRouter chat API, Alibaba's endpoint, fallbacks off, reasoning off; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 92.4 | 18 of 35tied with Qwen4-Max · high · chat, Claude Opus 5.5 · high · chat, DeepSeek-V4.1-Flash · reasoning on · chat · 1017 and 15 more | – | – | logic:qwen3.8-flash_en, logic:qwen3.8-flash_en_dontknowhosted API via OpenRouter |
| Math | 99.4 | 14 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:qwen3.8-flash_en, math:qwen3.8-flash_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 97.0 | 14 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 17 more | – | – | logic:qwen3.8-flash_en, math:qwen3.8-flash_en, logic:qwen3.8-flash_en_dontknow, math:qwen3.8-flash_en_dontknowhosted API via OpenRouter |
| Backgammon | 71.5 | 15 of 16 | – | – | backgammon:qwen3-8-flash-offLeague 2; calibration definition pending Ümit; hosted (API or subscription); improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
KnowLine-12B-Gen1
- Maker: PelaAI
- Kind: decision model · class: Small
- Size: 12B
- Repo: PelaAI/KnowLine-12B-Gen1
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 65.1EN 83.3 Logic 38.2 Math 57.8 AC 80.9 | 16 of 53tied with Jev 1.14, d1.1 | – | – | – |
| Reading · Turkish (in Combined) | 53.1 | 27 of 76tied with SubQ Snap 1.1, GPT-6 Luna · Decisions API, d1.2 and 1 more | 6.2 | f26fbc2 | text:1309KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| Turkish Reading | 82.5 | 26 of 76tied with Perplexity Decider 27B v1.1, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1.2 · API · with web and 16 more | 11.4 | f26fbc2 | text:1309KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| Turkish Grammar | 40.6 | 27 of 76tied with GPT-6 Luna · Decisions API, SubQ Snap 1.1, d1.1 and 1 more | 4.3 | f26fbc2 | text:1309KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| English Reading | 83.3 | 29 of 75tied with d3, Drex Max, Perplexity Decider 27B v1.3 and 34 more | 11.1 | f26fbc2 | english:1313KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 38.2 | 23 of 69tied with Perplexity Decider 27B v1.2 · API · with web, SubQ Snap 1.1, Turion Jev 31B and 3 more | 4.8 | f26fbc2 | logic:knowline-12b-gen1_en, logic:knowline-12b-gen1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 57.8 | 26 of 69tied with d1.2, d1.1, Jev 1.14 | 4.0 | f26fbc2 | math:knowline-12b-gen1_en, math:knowline-12b-gen1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 51.1 | 26 of 69tied with celeris-1-decision, celeris-1.1-decision, d1.2 | 2.9 | f26fbc2 | logic:knowline-12b-gen1_en, math:knowline-12b-gen1_en, logic:knowline-12b-gen1_en_dontknow, math:knowline-12b-gen1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 80.9 | 26 of 55tied with Jeffan-9B-1211, Jev 1.13, d1.1 and 9 more | 6.8 | f26fbc2 | grading:site-knowline-12b-gen1-v1PelaAI/KnowLine-12B-Gen1@f26fbc2d, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card names 0.5.21, which we do not have), BF16, SHA256SUMS checked, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
d1.2
- Maker: Liquid AI
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 52.2 | 28 of 76tied with KnowLine-12B-Gen1, d1.1, celeris-1-decision and 1 more | 3.3 | – | text:1346Liquid AI decisions API |
| Turkish Reading | 82.5 | 28 of 76tied with Perplexity Decider 27B v1.1, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1.2 · API · with web and 16 more | 6.0 | – | text:1346Liquid AI decisions API |
| Turkish Grammar | 39.3 | 29 of 76tied with KnowLine-12B-Gen1, d1.1, celeris-1-decision and 1 more | 2.5 | – | text:1346Liquid AI decisions API |
| English Reading | 86.3 | 20 of 75tied with KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 · 20261111 and 34 more | 8.3 | – | english:1357Liquid AI decisions API; the kept reading questions in English |
| Vision | 33.7 | 6 of 20tied with KnowLine-31B-Gen1.5, d3, Drex Max and 4 more | 5.6 | – | vision:571Liquid AI decisions API; the page image only (condition B) |
| Logic | 36.4 | 25 of 69tied with KnowLine-12B-Gen1, Gemini Decision 1 · 20261111, Jeffan-9B-1211 and 4 more | 2.6 | – | logic:d1.2_en, logic:d1.2_en_dontknowhosted API (the maker's service) |
| Math | 56.9 | 27 of 69tied with KnowLine-12B-Gen1, d1.1, Jev 1.14 and 1 more | 3.2 | – | math:d1.2_en, math:d1.2_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 49.9 | 27 of 69tied with KnowLine-12B-Gen1, Jev 1.14, d1.1 | 2.5 | – | logic:d1.2_en, math:d1.2_en, logic:d1.2_en_dontknow, math:d1.2_en_dontknowhosted API (the maker's service) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
d1.1
- Maker: Liquid AI
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 64.2EN 85.4 Logic 32.6 Math 56.5 AC 82.5 | 18 of 53tied with KnowLine-12B-Gen1, Jev 1.14, Jev 1.13 and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 51.9 | 29 of 76tied with KnowLine-12B-Gen1, d1.2, celeris-1-decision and 1 more | 2.0 | – | text:254Liquid AI decisions API |
| Turkish Reading | 80.8 | 34 of 76tied with SubQ Snap Preview, SubQ Snap 1.1, KnowLine-12B-Gen1 and 14 more | 4.2 | – | text:254Liquid AI decisions API |
| Turkish Grammar | 39.6 | 28 of 76tied with KnowLine-12B-Gen1, d1.2, celeris-1-decision | 2.2 | – | text:254Liquid AI decisions API |
| English Reading | 85.4 | 24 of 75tied with Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B and 35 more | 4.2 | – | english:565, english:568, english:570, english:573, english:575Liquid AI decisions API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 32.6 | 35 of 69tied with Jeffan-9B-1211, Jev 1.14, GLiDE 9B and 15 more | 4.4 | – | logic:d1.1_en, logic:d1.1_en_dontknowhosted API (the maker's service) |
| Math | 56.5 | 28 of 69tied with KnowLine-12B-Gen1, d1.2, Jev 1.14 and 1 more | 2.2 | – | math:d1.1_en, math:d1.1_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 48.3 | 29 of 69tied with d1.2, Jev 1.14, Jev 1.13 | 0.8 | – | logic:d1.1_en, math:d1.1_en, logic:d1.1_en_dontknow, math:d1.1_en_dontknowhosted API (the maker's service) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 82.5 | 21 of 55tied with d1-ultra, Microsoft-Decision-1, vera-medium-omni and 10 more | 8.7 | – | grading:site-liquid-d1.1-v1Liquid AI decisions API (api.liquid.ai/decisions, model d1.1); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
celeris-1-decision
- Maker: Celeris
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 58.0TR 50.6 L&M 52.8 BG 70.6 | 13 of 36tied with SubQ Snap 1.1, Jev 1.14 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 50.6 | 30 of 76tied with d1.2, d1.1, Perplexity Decider 27B v1 and 2 more | 4.3 | – | text:150Celeris System One API |
| Turkish Reading | 81.5 | 31 of 76tied with GLiDE · API, SubQ Snap Preview, SubQ Snap 1.1 and 14 more | 6.2 | – | text:150Celeris System One API |
| Turkish Grammar | 37.4 | 30 of 76tied with d1.1, d1.2, Perplexity Decider 27B v1 and 2 more | 5.3 | – | text:150Celeris System One API |
| English Reading | 80.1 | 46 of 75tied with StartLux-Decision-v1.1-9B-Instant, d1, Clef and 29 more | 5.2 | – | english:546Celeris System One API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 34.1 | 29 of 69tied with d1.2, Jeffan-9B-1211, Jev 1.14 and 10 more | 10.7 | – | logic:celeris_en, logic:celeris_en_dontknowhosted API (the maker's service) |
| Math | 62.5 | 18 of 69tied with d3-ultra, Gemini Decision 1 · 20261111, GLiDE · API and 10 more | 4.3 | – | math:celeris_en, math:celeris_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 52.8 | 24 of 69tied with d1.2-ultra, Perplexity Decider 27B v1.2 · API · with web, Turion Jev 31B and 3 more | 5.8 | – | logic:celeris_en, math:celeris_en, logic:celeris_en_dontknow, math:celeris_en_dontknowhosted API (the maker's service) |
| Backgammon | 70.6 | 14 of 41tied with GLiDE · API, Jev 1.13, GLiDE · v1.2 and 2 more | – | – | backgammon:celeris-1-decisionLeague 1; calibration definition pending Ümit; hosted (API or subscription) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Perplexity Decider 27B v1
- Maker: Perplexity
- Kind: decision model · class: Medium
- Size: 26.1B
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 50.5 | 31 of 76tied with d1.2, d1.1, celeris-1-decision and 2 more | 5.0 | – | text:44Perplexity Decisions API |
| Turkish Reading | 82.5 | 27 of 76tied with Perplexity Decider 27B v1.1, Perplexity Decider 27B v1.3, Perplexity Decider 27B v1.2 · API · with web and 16 more | 5.0 | – | text:44Perplexity Decisions API |
| Turkish Grammar | 36.9 | 31 of 76tied with d1.2, celeris-1-decision, OneJev-27B and 1 more | 5.0 | – | text:44Perplexity Decisions API |
| English Reading | 85.6 | 23 of 75tied with Gemini Decision 1 · 20261111, Logits Sigma-1 Duo, GLiDE 27B and 35 more | 2.6 | – | english:626, english:627, english:628, english:629, english:630Perplexity Decisions API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
OneJev-27B full fine-tune
- Maker: OmniJev · training: full fine-tune
- Kind: decision model · class: Medium
- Size: 27.4B
- Repo: OmniJev/OneJev-27B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 49.5 | 32 of 76tied with celeris-1-decision, Perplexity Decider 27B v1, Jev 1.14 and 1 more | 5.5 | a5e5f55 | text:264OneJev server (qev, OmniJev's release code), self-hosted, BF16 |
| Turkish Reading | 81.5 | 30 of 76tied with GLiDE · API, SubQ Snap Preview, SubQ Snap 1.1 and 14 more | 9.9 | a5e5f55 | text:264OneJev server (qev, OmniJev's release code), self-hosted, BF16 |
| Turkish Grammar | 35.9 | 32 of 76tied with celeris-1-decision, Perplexity Decider 27B v1, Jev 1.14 and 1 more | 3.7 | a5e5f55 | text:264OneJev server (qev, OmniJev's release code), self-hosted, BF16 |
| English Reading | 82.9 | 31 of 75tied with Drex Max, Perplexity Decider 27B v1.3, d3-ultra and 33 more | 9.7 | a5e5f55 | english:778OneJev server (qev, OmniJev's release code), self-hosted, BF16; the kept reading questions in English |
| Vision | 26.6 | 13 of 20tied with Jev 1.14 | 4.9 | a5e5f55 | vision:28, vision:35OneJev server (qev, OmniJev's release code), self-hosted, BF16; the page image only (condition B) |
| Logic | 25.8 | 52 of 69tied with Open-Jev-27B-v1.1, KnowLine 4B Gen5.5, Drex 2.0 and 7 more | 4.6 | a5e5f55 | logic:onejev_en, logic:onejev_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 42.2 | 50 of 69tied with Valen-Decision-1.5-9B, Perplexity Decider Flash v1.2, Open-Jev-27B-v1.1 and 1 more | 3.6 | a5e5f55 | math:onejev_en, math:onejev_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 36.6 | 51 of 69tied with Mercury Decide, d1, Open-Jev-27B-v1.1 | 3.1 | a5e5f55 | logic:onejev_en, math:onejev_en, logic:onejev_en_dontknow, math:onejev_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Jev 1.14
- Maker: TypeSafe
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 57.1TR 49.0 L&M 49.2 BG 73.1 | 14 of 36tied with celeris-1-decision | – | – | – |
| Decision Index4 benches | 64.5EN 82.6 Logic 35.2 Math 56.4 AC 83.6 | 17 of 53tied with KnowLine-12B-Gen1, d1.1, Jev 1.13 and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 49.0 | 33 of 76tied with celeris-1-decision, Perplexity Decider 27B v1, OneJev-27B and 2 more | 2.5 | – | text:194TypeSafe System One API |
| Turkish Reading | 81.3 | 33 of 76tied with GLiDE · API, SubQ Snap Preview, SubQ Snap 1.1 and 15 more | 2.2 | – | text:194TypeSafe System One API |
| Turkish Grammar | 35.2 | 33 of 76tied with celeris-1-decision, Perplexity Decider 27B v1, OneJev-27B and 2 more | 4.0 | – | text:194TypeSafe System One API |
| English Reading | 82.6 | 32 of 75tied with d3-ultra, d1.2, Perplexity Decider 27B v1.2 · DGX Spark and 32 more | 2.2 | – | english:410, english:416, english:421, english:423, english:425TypeSafe System One API; the kept reading questions in English |
| Vision | 29.1 | 12 of 20tied with SubQ Snap 1.1, d3-ultra, d1.2-ultra and 2 more | 3.7 | – | vision:1, vision:6TypeSafe System One API; the page image only (condition B) |
| Logic | 35.2 | 27 of 69tied with Gemini Decision 1 · 20261111, d1.2, Jeffan-9B-1211 and 9 more | 3.3 | – | logic:jev-1.14_en, logic:jev-1.14_en_dontknowhosted API (the maker's service) |
| Math | 56.4 | 29 of 69tied with KnowLine-12B-Gen1, d1.2, d1.1 and 1 more | 2.8 | – | math:jev-1.14_en, math:jev-1.14_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 49.2 | 28 of 69tied with d1.2, d1.1, Jev 1.13 | 1.8 | – | logic:jev-1.14_en, math:jev-1.14_en, logic:jev-1.14_en_dontknow, math:jev-1.14_en_dontknowhosted API (the maker's service) |
| Backgammon | 73.1 | 10 of 41tied with Drex Max, Drex Max v1.1, d1-ultra and 3 more | – | – | backgammon:jev-1-14-0League 2; calibration definition pending Ümit; hosted (API or subscription) |
| Answer Check | 83.6 | 18 of 55tied with Logits Sigma-1 Duo, GLiDE · API, GLiDE · v1.2 and 10 more | 2.8 | – | grading:site-jev-1.14.0-v1TypeSafe decisions API (api.typesafe.ai, model jev-1.14.0); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
Qwen3.5-4B reasoning on · chat
- Maker: Alibaba
- Kind: chat model · class: Tiny
- Size: 4.7B
- Repo: Qwen/Qwen3.5-4B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 84.9EN 83.8 Logic 90.4 Math 99.3 AC 66.2 | 3 of 11tied with Gemma 4 12B Unified · default reasoning · chat, MiniCPM5-2B · reasoning off · chat | – | – | – |
| Reading · Turkish (in Combined) | 48.6 | 28 of 40tied with Gemma 4 E4B · default reasoning · chat | – | 851bf6e | text:1149our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-4B@851bf6e), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking on; output cap 32,768 |
| Turkish Reading | 73.9 | 29 of 40tied with Gemma 4 E4B · default reasoning · chat | – | 851bf6e | text:1149our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-4B@851bf6e), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking on; output cap 32,768 |
| Turkish Grammar | 37.9 | 28 of 40tied with Gemma 4 E4B · default reasoning · chat | – | 851bf6e | text:1149our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-4B@851bf6e), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking on; output cap 32,768 |
| English Reading | 83.8 | 24 of 47tied with GLM-5.3-Flash · reasoning on · chat, Muse Ember · max · chat, MiMo-V2.6-Flash · high · chat and 8 more | – | 851bf6e | english:1072our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-4B@851bf6e), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking on; output cap 32,768; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 90.4 | 21 of 35tied with Qwen3.8-27B · low · chat, GPT-6.1 Terra · high · chat, MiniMax-M3.1-Flash · high · chat and 4 more | – | 851bf6e | logic:qwen3.5-4b_en, logic:qwen3.5-4b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Qwen3.5-4B (off · chat) |
| Math | 99.3 | 16 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | 851bf6e | math:qwen3.5-4b_en, math:qwen3.5-4b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Qwen3.5-4B (off · chat) |
| Logic & Math (in Combined) | 96.3 | 18 of 34tied with Gemini 4 Hydrogen · high · chat, Claude Sonnet 5.5 · high · chat, Inkling 2 Small · high · chat and 12 more | – | 851bf6e | logic:qwen3.5-4b_en, math:qwen3.5-4b_en, logic:qwen3.5-4b_en_dontknow, math:qwen3.5-4b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Qwen3.5-4B (off · chat) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 66.2 | 14 of 20tied with Gemma 4 12B Unified · thinking on · chat, MiniCPM5-2B · reasoning off · chat, Qwen3.5-4B · reasoning off · chat | – | 851bf6e | grading:chat-qwen3.5-4b-r2-v1Qwen/Qwen3.5-4B@851bf6e8, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 179 of 2069 replies (9%) had no answer marker and count wrong (0 of them cut off at the 32,768-token limit); a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history |
Mercury Decide
- Maker: Inception
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 48.8TR 47.9 L&M 38.3 BG 60.4 | 19 of 36tied with Clef, Elephant-1 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 47.9 | 34 of 76tied with OneJev-27B, Jev 1.14, Jev 1.13 and 3 more | 15.5 | – | text:204OpenRouter System One API |
| Turkish Reading | 80.6 | 35 of 76tied with SubQ Snap 1.1, KnowLine-12B-Gen1, Perplexity Decider 27B v1 and 14 more | 7.5 | – | text:204OpenRouter System One API |
| Turkish Grammar | 33.9 | 34 of 76tied with OneJev-27B, Jev 1.14, Jev 1.13 and 3 more | 19.0 | – | text:204OpenRouter System One API |
| English Reading | 82.2 | 40 of 75tied with d1.2, Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1 and 31 more | 6.6 | – | english:398, english:400, english:401, english:402, english:403OpenRouter System One API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 31.3 | 38 of 69tied with celeris-1-decision, celeris-1.1-decision, StartLux-Decision-v1.1-9B-Instant and 14 more | 15.4 | – | logic:mercury-decide_en, logic:mercury-decide_en_dontknowhosted API via OpenRouter |
| Math | 41.9 | 51 of 69tied with Open-Jev-27B-v1.1, OneJev-27B | 15.8 | – | math:mercury-decide_en, math:mercury-decide_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 38.3 | 48 of 69tied with Cerno 9B v1.2, Valen-Decision-1.5-9B, KnowLine 4B Gen5.5 and 4 more | 15.7 | – | logic:mercury-decide_en, math:mercury-decide_en, logic:mercury-decide_en_dontknow, math:mercury-decide_en_dontknowhosted API via OpenRouter |
| Backgammon | 60.4 | 26 of 41tied with d1-2B, Cerno 9B v1.2, Winnow-12B and 2 more | – | – | backgammon:mercury-decideLeague 1; calibration definition pending Ümit; hosted (API or subscription) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Gemma 4 E4B default reasoning · chat
- Maker: Google
- Kind: chat model · class: Small
- Size: 8B
- Repo: google/gemma-4-E4B-it
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 83.0EN 78.1 Logic 79.0 Math 97.6 AC 77.2 | 6 of 11tied with MiniCPM5-2B · reasoning off · chat, Granite-4.2-3B · reasoning on · chat | – | – | – |
| Reading · Turkish (in Combined) | 46.9 | 29 of 40tied with Qwen3.5-4B · reasoning on · chat, Claude Haiku 5.5 · max · chat | – | 4ecc2ba | text:1040our GPU (logic's server, vLLM v0.30.0 BF16, revision google/gemma-4-E4B-it@4ecc2ba), one question per request, plain answer line CEVAP: X (no answer schema); sampling temperature 0.0; output cap 8,192 |
| Turkish Reading | 72.0 | 30 of 40tied with Qwen3.5-4B · reasoning on · chat | – | 4ecc2ba | text:1040our GPU (logic's server, vLLM v0.30.0 BF16, revision google/gemma-4-E4B-it@4ecc2ba), one question per request, plain answer line CEVAP: X (no answer schema); sampling temperature 0.0; output cap 8,192 |
| Turkish Grammar | 36.3 | 29 of 40tied with Qwen3.5-4B · reasoning on · chat | – | 4ecc2ba | text:1040our GPU (logic's server, vLLM v0.30.0 BF16, revision google/gemma-4-E4B-it@4ecc2ba), one question per request, plain answer line CEVAP: X (no answer schema); sampling temperature 0.0; output cap 8,192 |
| English Reading | 78.1 | 31 of 47tied with MiniMax-M3.1-Flash · high · chat, LFM2.5-24B-A2B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat and 1 more | – | 4ecc2ba | english:889our GPU (logic's server, vLLM v0.30.0 BF16, revision google/gemma-4-E4B-it@4ecc2ba), one question per request, plain answer line CEVAP: X (no answer schema); sampling temperature 0.0; output cap 8,192; the kept reading questions in English |
| Vision | 28.6 | 2 of 2 | – | 4ecc2ba | vision:465our GPU (logic's server, vLLM v0.30.0 BF16, revision google/gemma-4-E4B-it@4ecc2ba), one question per request, plain answer line CEVAP: X (no answer schema); sampling temperature 0.0; output cap 8,192; the page image only (condition B) |
| Logic | 79.0 | 27 of 35tied with LFM3-1.7B · reasoning on · chat | – | 4ecc2ba | logic:gemma-4-e4b-it_en, logic:gemma-4-e4b-it_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Gemma 4 E4B (on · chat), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E4B (on · chat · 32k cap · sampled); 4.5B effective parameters |
| Math | 97.6 | 28 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 25 more | – | 4ecc2ba | math:gemma-4-e4b-it_en, math:gemma-4-e4b-it_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Gemma 4 E4B (on · chat), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E4B (on · chat · 32k cap · sampled); 4.5B effective parameters |
| Logic & Math (in Combined) | 91.3 | 26 of 34tied with LFM3-1.7B · reasoning on · chat | – | 4ecc2ba | logic:gemma-4-e4b-it_en, math:gemma-4-e4b-it_en, logic:gemma-4-e4b-it_en_dontknow, math:gemma-4-e4b-it_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Gemma 4 E4B (on · chat), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E4B (on · chat · 32k cap · sampled); 4.5B effective parameters |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 77.2 | 4 of 20tied with MiniCPM5-2B · reasoning off · chat, Gemma 4 12B Unified · default reasoning · chat, Granite-4.2-3B · reasoning on · chat and 2 more | – | 4ecc2ba | grading:chat-gemma-4-e4b-think-v1google/gemma-4-E4B-it@4ecc2ba6, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 2 of 2069 replies (0.10%) had no answer marker and count wrong (0 of them cut off at the 32,768-token limit); every reply kept; an earlier attempt of this same run stopped part-way (510 recorded pairs and an unknown number of further requests possibly answered but not recorded) and is kept apart, not merged; this attempt started again from the first pair, so its outcomes are this attempt's recorded outputs, not guaranteed to be the model's first outputs on those pairs |
Jev 1.13
- Maker: TypeSafe
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 55.3TR 46.9 L&M 47.5 BG 71.6 | 15 of 36 | – | – | – |
| Decision Index4 benches | 63.3EN 82.4 Logic 32.9 Math 55.1 AC 83.0 | 19 of 53tied with Jev 1.14, d1.1, Jeffan-9B-1211 and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 46.9 | 35 of 76tied with Jev 1.14, Mercury Decide, Clef and 2 more | 5.1 | – | text:49TypeSafe System One API |
| Turkish Reading | 79.6 | 36 of 76tied with KnowLine-12B-Gen1, Perplexity Decider 27B v1, d1.2 and 14 more | 3.8 | – | text:49TypeSafe System One API |
| Turkish Grammar | 33.0 | 35 of 76tied with Jev 1.14, Mercury Decide, d1 and 3 more | 6.5 | – | text:49TypeSafe System One API |
| English Reading | 82.4 | 35 of 75tied with d1.2, Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1 and 31 more | 3.1 | – | english:564, english:567, english:569, english:572, english:574TypeSafe System One API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 32.9 | 34 of 69tied with Jeffan-9B-1211, Jev 1.14, GLiDE 9B and 14 more | 5.9 | – | logic:jev-1.13_en, logic:jev-1.13_en_dontknowhosted API (the maker's service) |
| Math | 55.1 | 30 of 69tied with d1.2, d1.1, Jev 1.14 | 2.6 | – | math:jev-1.13_en, math:jev-1.13_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 47.5 | 30 of 69tied with Jev 1.14, d1.1, Adjudex 9B v5 | 3.2 | – | logic:jev-1.13_en, math:jev-1.13_en, logic:jev-1.13_en_dontknow, math:jev-1.13_en_dontknowhosted API (the maker's service) |
| Backgammon | 71.6 | 12 of 41tied with Jev 1.14, GLiDE · API, GLiDE · v1.2 and 2 more | – | – | backgammon:jev-1-13-0League 1; calibration definition pending Ümit; hosted (API or subscription) |
| Answer Check | 83.0 | 20 of 55tied with Logits Sigma-1 Duo, GLiDE · API, GLiDE · v1.2 and 12 more | 1.9 | – | grading:site-jev-1.13.0-v1TypeSafe decisions API (api.typesafe.ai, model jev-1.13.0); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
Clef post-trained
- Maker: Cloudflare · training: post-trained
- Kind: decision model · class: Medium
- Size: 27.5B
- Repo: Cloudflare/clef
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 50.0TR 46.6 L&M 34.5 BG 69.0 | 17 of 36tied with Elephant-1, Mercury Decide | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 46.6 | 36 of 76tied with Mercury Decide, Jev 1.13, Jeffan-9B-1211 and 4 more | 10.7 | 2f3de3d | text:125Clef server, self-hosted |
| Turkish Reading | 81.3 | 32 of 76tied with GLiDE · API, SubQ Snap Preview, SubQ Snap 1.1 and 15 more | 4.0 | 2f3de3d | text:125Clef server, self-hosted |
| Turkish Grammar | 31.8 | 38 of 76tied with Mercury Decide, Jev 1.13, d1 and 7 more | 15.1 | 2f3de3d | text:125Clef server, self-hosted |
| English Reading | 83.3 | 28 of 75tied with d3, Drex Max, Perplexity Decider 27B v1.3 and 34 more | 3.1 | 2f3de3d | english:613Clef server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 26.0 | 50 of 69tied with Perplexity Decider Flash v1.2, Open-Jev-27B-v1.1, KnowLine 4B Gen5.5 and 8 more | 15.7 | 2f3de3d | logic:clef_en, logic:clef_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 38.9 | 53 of 69tied with Verdin-4B-v1.1, Chinese Decision Pointer 4B, Clef-Flash | 11.0 | 2f3de3d | math:clef_en, math:clef_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 34.5 | 52 of 69tied with Verdin-4B-v1.1, Chinese Decision Pointer 4B | 12.6 | 2f3de3d | logic:clef_en, math:clef_en, logic:clef_en_dontknow, math:clef_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | 69.0 | 16 of 41tied with GLiDE · v1.2, celeris-1-decision, Perplexity Decider 27B v1.2 · DGX Spark and 5 more | – | – | backgammon:clefLeague 2; calibration definition pending Ümit; run on our GPUs |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Claude Haiku 5.5 max · chat
- Maker: Anthropic
- Kind: chat model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 46.3 | 30 of 40tied with Gemma 4 E4B · default reasoning · chat | 17.5 | – | text:948OpenRouter chat API, Anthropic's own endpoint, fallbacks off, output cap 128,000 tokens, reasoning max |
| Turkish Reading | 83.6 | 25 of 40tied with GLM-5.3-Flash · reasoning on · chat, Qwen3.8-27B · low · chat, GLM-5.3 · low · chat and 5 more | 18.9 | – | text:948OpenRouter chat API, Anthropic's own endpoint, fallbacks off, output cap 128,000 tokens, reasoning max |
| Turkish Grammar | 30.4 | 30 of 40 | 16.4 | – | text:948OpenRouter chat API, Anthropic's own endpoint, fallbacks off, output cap 128,000 tokens, reasoning max |
| English Reading | 90.9 | 13 of 47tied with Claude Fable 5.1 · high · chat, Gemini 3.8 Flash · high · chat, Gemini 4 Argon · low · chat and 11 more | 18.4 | – | english:949OpenRouter chat API, Anthropic's own endpoint, fallbacks off, output cap 128,000 tokens, reasoning max; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Jeffan-9B-1211
- Maker: tattrongvu
- Kind: decision model · class: Small
- Size: 8.9B
- Repo: tattrongvu/Jeffan-9B-1211
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 63.1EN 84.5 Logic 35.2 Math 49.8 AC 83.0 | 20 of 53tied with Jev 1.14, d1.1, Jev 1.13 and 2 more | – | – | – |
| Reading · Turkish (in Combined) | 46.3 | 37 of 76tied with Mercury Decide, Jev 1.13, Clef and 5 more | 4.6 | 862d1a7 | text:1192Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16 |
| Turkish Reading | 78.7 | 37 of 76tied with KnowLine-12B-Gen1, Perplexity Decider 27B v1, d1.2 and 17 more | 11.4 | 862d1a7 | text:1192Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16 |
| Turkish Grammar | 32.5 | 37 of 76tied with Mercury Decide, Jev 1.13, d1 and 5 more | 3.0 | 862d1a7 | text:1192Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16 |
| English Reading | 84.5 | 25 of 75tied with GLiDE 27B, Sieve 31B v3, Turion Jev 31B and 33 more | 8.9 | 862d1a7 | english:1195Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 35.2 | 26 of 69tied with Gemini Decision 1 · 20261111, d1.2, Jev 1.14 and 9 more | 4.0 | 862d1a7 | logic:jeffan-9b_en, logic:jeffan-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 49.8 | 34 of 69tied with Elephant-1, KnowLine-9B-Gen2, KnowLine-9B-Gen1 and 4 more | 3.8 | 862d1a7 | math:jeffan-9b_en, math:jeffan-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 44.8 | 33 of 69tied with Adjudex 9B v5, Elephant-1, KnowLine-9B-Gen2 and 2 more | 3.6 | 862d1a7 | logic:jeffan-9b_en, math:jeffan-9b_en, logic:jeffan-9b_en_dontknow, math:jeffan-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 83.0 | 19 of 55tied with Logits Sigma-1 Duo, GLiDE · API, GLiDE · v1.2 and 12 more | 3.3 | 862d1a7 | grading:site-jeffan-9b-1211-v1tattrongvu/Jeffan-9B-1211@862d1a7a, its card's Decision Index engine (jeffan_engine:DecisionEngine, transformers 5.17.0, flash-linear-attention 0.5.2, kit decision-index@f8552a1, options max_len and budget 65,536 as its card; without the card's optional causal-conv1d, so its PyTorch reference convolution) behind di_serve.py, our System One wrapper of the engine, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
d1
- Maker: Liquid AI
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 58.3EN 83.5 Logic 26.0 Math 44.5 AC 79.2 | 33 of 53tied with vera-medium-omni, KnowLine-9B-Gen1, Cerno 9B v1.2 and 6 more | – | – | – |
| Reading · Turkish (in Combined) | 46.3 | 38 of 76tied with Mercury Decide, Jev 1.13, Clef and 5 more | 10.2 | – | text:52Liquid AI decisions API |
| Turkish Reading | 78.5 | 40 of 76tied with GPT-6 Luna · Decisions API, OneJev-27B, celeris-1-decision and 15 more | 2.8 | – | text:52Liquid AI decisions API |
| Turkish Grammar | 32.6 | 36 of 76tied with Mercury Decide, Jev 1.13, Jeffan-9B-1211 and 5 more | 13.5 | – | text:52Liquid AI decisions API |
| English Reading | 83.5 | 27 of 75tied with Mammoth-1, d3, Drex Max and 35 more | 2.6 | – | english:430, english:432, english:433, english:434, english:435Liquid AI decisions API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 26.0 | 51 of 69tied with Perplexity Decider Flash v1.2, Open-Jev-27B-v1.1, KnowLine 4B Gen5.5 and 8 more | 10.9 | – | logic:d1_en, logic:d1_en_dontknowhosted API (the maker's service) |
| Math | 44.5 | 46 of 69tied with vera-medium-omni, Cerno 9B v1.2, KnowLine 4B Gen5.5 and 5 more | 3.3 | – | math:d1_en, math:d1_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 38.2 | 49 of 69tied with Cerno 9B v1.2, Valen-Decision-1.5-9B, KnowLine 4B Gen5.5 and 4 more | 5.8 | – | logic:d1_en, math:d1_en, logic:d1_en_dontknow, math:d1_en_dontknowhosted API (the maker's service) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 79.2 | 29 of 55tied with Dexy-Decision-9B-V2, KnowLine-12B-Gen1, KnowLine 4B Gen5.5 and 6 more | 3.1 | – | grading:site-liquid-d1-v1Liquid AI decisions API (api.liquid.ai/decisions, model d1); billed 'd1' route, the site's 'd1:free' was withdrawn (HTTP 404, 13 Nov); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
Drex 2.0
- Maker: Nace.AI
- Kind: decision model · class: Medium
- Size: 35.1B (3B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 51.4TR 44.7 L&M 41.4 BG 68.3 | 16 of 36 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 44.7 | 39 of 76tied with Clef, Jeffan-9B-1211, d1 and 5 more | 4.2 | – | text:334Nace.AI System One API |
| Turkish Reading | 76.8 | 42 of 76tied with Mercury Decide, Jev 1.13, Jeffan-9B-1211 and 12 more | 10.1 | – | text:334Nace.AI System One API |
| Turkish Grammar | 31.0 | 39 of 76tied with Jev 1.13, d1, Jeffan-9B-1211 and 6 more | 2.0 | – | text:334Nace.AI System One API |
| English Reading | 82.4 | 34 of 75tied with d1.2, Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1 and 31 more | 6.9 | – | english:566, english:571, english:576, english:581, english:586Nace.AI System One API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 27.3 | 49 of 69tied with vera-medium-omni, KnowLine-9B-Gen2, Cerno 9B v1.2 and 9 more | 4.3 | – | logic:drex-2.0_en, logic:drex-2.0_en_dontknowhosted API (the maker's service) |
| Math | 48.7 | 37 of 69tied with KnowLine-9B-Gen2, Jeffan-9B-1211, KnowLine-9B-Gen1 and 4 more | 1.9 | – | math:drex-2.0_en, math:drex-2.0_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 41.4 | 40 of 69tied with GLiDE 9B, KnowLine-9B-Gen1, StartLux-Decision-v1.1-9B-Instant and 5 more | 1.1 | – | logic:drex-2.0_en, math:drex-2.0_en, logic:drex-2.0_en_dontknow, math:drex-2.0_en_dontknowhosted API (the maker's service) |
| Backgammon | 68.3 | 17 of 41tied with Perplexity Decider 27B v1.2 · DGX Spark, Clef, Cygnet and 3 more | – | – | backgammon:drex-2-0League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 18 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 6 Nov 2026. Probabilities come rounded to 4 decimals by Nace's API and are stored as returned. Its pair with GPT-6 Luna was not played: GPT-6 Luna's API was out of credit; the pair is outside the requested scope (Ümit, 12 Nov).; hosted (API or subscription) |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Cygnet unchanged base
- Maker: Blockbrain Labs · training: unchanged base
- Kind: decision model · class: Small
- Size: 12B
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Setups: Logic and Math: a hosted API; the other benches: our hardware.
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 47.0TR 44.6 L&M 28.3 BG 68.2 | 22 of 36tied with Cerno 9B v1.2, StartLux-Decision-v1.1-4B-Instant | – | – | – |
| Decision Index4 benches | 51.8EN 79.9 Logic 24.1 Math 30.4 AC 72.9 | 41 of 53tied with Clef-Flash | – | – | – |
| Reading · Turkish (in Combined) | 44.6 | 40 of 76tied with Clef, Jeffan-9B-1211, d1 and 5 more | 13.8 | – | text:127Cygnet decision server on vLLM, self-hosted |
| Turkish Reading | 78.5 | 39 of 76tied with GPT-6 Luna · Decisions API, OneJev-27B, celeris-1-decision and 15 more | 4.1 | – | text:127Cygnet decision server on vLLM, self-hosted |
| Turkish Grammar | 30.2 | 40 of 76tied with d1, Jeffan-9B-1211, Clef and 5 more | 18.9 | – | text:127Cygnet decision server on vLLM, self-hosted |
| English Reading | 79.9 | 47 of 75tied with StartLux-Decision-v1.1-9B-Instant, d1, Clef and 29 more | 4.3 | – | english:753Cygnet decision server on vLLM, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 24.1 | 56 of 69tied with Clef, d1, OneJev-27B and 4 more | 11.2 | – | logic:cygnet_en, logic:cygnet_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); no weights of its own: google/gemma-4-12B-it at 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7, unchanged, behind cygnet-recipe's shim (github.com/blockbrain-ai/cygnet-recipe 3cf591c, temperature 3.4); repo left empty so it is not matched with Gemma 4 12B Unified |
| Math | 30.4 | 57 of 69 | 5.9 | – | math:cygnet_en, math:cygnet_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); no weights of its own: google/gemma-4-12B-it at 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7, unchanged, behind cygnet-recipe's shim (github.com/blockbrain-ai/cygnet-recipe 3cf591c, temperature 3.4); repo left empty so it is not matched with Gemma 4 12B Unified |
| Logic & Math (in Combined) | 28.3 | 57 of 69 | 7.4 | – | logic:cygnet_en, math:cygnet_en, logic:cygnet_en_dontknow, math:cygnet_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); no weights of its own: google/gemma-4-12B-it at 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7, unchanged, behind cygnet-recipe's shim (github.com/blockbrain-ai/cygnet-recipe 3cf591c, temperature 3.4); repo left empty so it is not matched with Gemma 4 12B Unified |
| Backgammon | 68.2 | 18 of 41tied with Perplexity Decider 27B v1.2 · DGX Spark, Clef, Drex 2.0 and 3 more | – | – | backgammon:cygnet-gemma-4-12bLeague 1; calibration definition pending Ümit; run on our GPUs |
| Answer Check | 72.9 | 41 of 55tied with Valen-Decision-1.5-9B | 6.5 | – | grading:site-cygnet-v1no weights of its own: google/gemma-4-12B-it@707f0a3b, unchanged, on vLLM 0.30.0 behind cygnet-recipe's shim (blockbrain-ai/cygnet-recipe@3cf591c, temperature 3.4), the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
Open-Jev-27B-v1.1 LoRA + head
- Maker: Zefan Cai · training: LoRA + head
- Kind: decision model · class: Medium
- Size: 27.8B
- Repo: ZefanCai/Open-Jev-27B-v1.1
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 42.8TR 44.6 L&M 38.1 BG 45.9 | 25 of 36tied with KnowLine 4B Gen5.5 | – | – | – |
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 44.6 | 41 of 76tied with Clef, Jeffan-9B-1211, d1 and 5 more | 24.5 | 28cf730 | text:45Open-Jev server, self-hosted |
| Turkish Reading | 78.7 | 38 of 76tied with KnowLine-12B-Gen1, Perplexity Decider 27B v1, d1.2 and 17 more | 43.6 | 28cf730 | text:45Open-Jev server, self-hosted |
| Turkish Grammar | 30.1 | 42 of 76tied with d1, Jeffan-9B-1211, Clef and 6 more | 16.4 | 28cf730 | text:45Open-Jev server, self-hosted |
| English Reading | 81.7 | 43 of 75tied with Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1, Perplexity Decider 27B v1 and 32 more | 45.3 | 28cf730 | english:760Open-Jev server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 28.2 | 47 of 69tied with Perplexity Decider Flash v1.3, Microsoft-Decision-1, Valen-Decision-1.5-9B and 11 more | 12.9 | 28cf730 | logic:open-jev_en, logic:open-jev_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); the repository holds an adapter and head on Qwen3.8-27B |
| Math | 43.2 | 49 of 69tied with Cerno 9B v1.2, KnowLine 4B Gen5.5, Perplexity Decider Flash v1.3 and 6 more | 21.1 | 28cf730 | math:open-jev_en, math:open-jev_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); the repository holds an adapter and head on Qwen3.8-27B |
| Logic & Math (in Combined) | 38.1 | 50 of 69tied with Cerno 9B v1.2, Valen-Decision-1.5-9B, KnowLine 4B Gen5.5 and 4 more | 18.3 | 28cf730 | logic:open-jev_en, math:open-jev_en, logic:open-jev_en_dontknow, math:open-jev_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); the repository holds an adapter and head on Qwen3.8-27B |
| Backgammon | 45.9 | 35 of 41tied with StartLux-Decision-v1.1-0.8B-Instant, GLiDE 2B | – | – | backgammon:open-jev-27b-v1-1League 1; calibration definition pending Ümit; run on our GPUs |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Winnow-12B LoRA, merged
- Maker: EldanRing · training: LoRA, merged
- Kind: decision model · class: Small
- Size: 11.9B
- Repo: EldanRing/Winnow-12B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 44.6TR 44.3 L&M 30.5 BG 58.9 | 23 of 36tied with KnowLine 4B Gen5.5 | – | – | – |
| Decision Index4 benches | 53.7EN 79.9 Logic 24.7 Math 33.6 AC 76.6 | 39 of 53tied with Verdin-4B-v1.1, Chinese Decision Pointer 4B | – | – | – |
| Reading · Turkish (in Combined) | 44.3 | 42 of 76tied with Jeffan-9B-1211, d1, Drex 2.0 and 4 more | 19.7 | b2b1421 | text:128Winnow server (llama.cpp), self-hosted |
| Turkish Reading | 77.5 | 41 of 76tied with Clef, Jev 1.14, d1.1 and 14 more | 9.3 | b2b1421 | text:128Winnow server (llama.cpp), self-hosted |
| Turkish Grammar | 30.2 | 41 of 76tied with d1, Jeffan-9B-1211, Clef and 5 more | 24.1 | b2b1421 | text:128Winnow server (llama.cpp), self-hosted |
| English Reading | 79.9 | 48 of 75tied with StartLux-Decision-v1.1-9B-Instant, d1, Clef and 29 more | 8.7 | b2b1421 | english:746Winnow server (llama.cpp), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 24.7 | 55 of 69tied with Drex 2.0, Clef, d1 and 5 more | 14.4 | b2b1421 | logic:winnow_en, logic:winnow_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); the Q8_0 GGUF, served by winnow-inference at d4631fb |
| Math | 33.6 | 56 of 69 | 8.4 | b2b1421 | math:winnow_en, math:winnow_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); the Q8_0 GGUF, served by winnow-inference at d4631fb |
| Logic & Math (in Combined) | 30.5 | 56 of 69 | 10.5 | b2b1421 | logic:winnow_en, math:winnow_en, logic:winnow_en_dontknow, math:winnow_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); the Q8_0 GGUF, served by winnow-inference at d4631fb |
| Backgammon | 58.9 | 28 of 41tied with d1-2B, Mercury Decide, Cerno 9B v1.2 and 2 more | – | – | backgammon:winnow-12bLeague 1; calibration definition pending Ümit; run on our GPUs |
| Answer Check | 76.6 | 36 of 55tied with Chinese Decision Pointer 4B, Perplexity Decider Flash v1.2, Gemini Decision 1 · 20261111 and 4 more | 8.3 | b2b1421 | grading:site-winnow-12b-v1EldanRing/Winnow-12B@b2b14213, its Q8_0 GGUF served by winnow-inference@d4631fb, the server its Logic & Math runs use, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
KnowLine-9B-Gen1
- Maker: PelaAI
- Kind: decision model · class: Small
- Size: 9.7B
- Repo: PelaAI/KnowLine-9B-Gen1
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 59.7EN 82.4 Logic 30.3 Math 49.7 AC 76.1 | 28 of 53tied with StartLux-Decision-v1.1-9B-Instant, Elephant-1, Dexy-Decision-9B-V2 and 6 more | – | – | – |
| Reading · Turkish (in Combined) | 43.2 | 43 of 76tied with Drex 2.0, Cygnet, Open-Jev-27B-v1.1 and 4 more | 4.6 | 325378d | text:1174KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| Turkish Reading | 74.8 | 46 of 76tied with Jeffan-9B-1211, Open-Jev-27B-v1.1, Cygnet and 10 more | 1.7 | 325378d | text:1174KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| Turkish Grammar | 29.7 | 43 of 76tied with Clef, Drex 2.0, Cygnet and 4 more | 6.8 | 325378d | text:1174KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| English Reading | 82.4 | 36 of 75tied with d1.2, Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1 and 31 more | 3.5 | 325378d | english:1177KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 30.3 | 42 of 69tied with StartLux-Decision-v1.1-9B-Instant, Adjudex 9B v5, Elephant-1 and 14 more | 3.5 | 325378d | logic:knowline-9b_en, logic:knowline-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 49.7 | 35 of 69tied with Elephant-1, KnowLine-9B-Gen2, Jeffan-9B-1211 and 4 more | 3.1 | 325378d | math:knowline-9b_en, math:knowline-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 43.1 | 36 of 69tied with Jeffan-9B-1211, KnowLine-9B-Gen2, GLiDE 9B and 4 more | 2.2 | 325378d | logic:knowline-9b_en, math:knowline-9b_en, logic:knowline-9b_en_dontknow, math:knowline-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 76.1 | 37 of 55tied with Chinese Decision Pointer 4B, Perplexity Decider Flash v1.2, Gemini Decision 1 · 20261111 and 5 more | 3.6 | 325378d | grading:site-knowline-9b-gen1-v1PelaAI/KnowLine-9B-Gen1@325378d1, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card's release launcher names 0.5.21, which we do not have), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
Cerno 9B v1.2 full fine-tune
- Maker: Apertix AI · training: full fine-tune
- Kind: decision model · class: Small
- Size: 8.9B
- Repo: apertixai/cerno-9b-v1.2
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 59.0 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 47.5TR 43.1 L&M 39.6 BG 60.0 | 20 of 36tied with StartLux-Decision-v1.1-4B-Instant, Cygnet | – | – | – |
| Decision Index4 benches | 59.6EN 82.4 Logic 29.1 Math 45.0 AC 81.9 | 29 of 53tied with StartLux-Decision-v1.1-9B-Instant, Elephant-1, Dexy-Decision-9B-V2 and 6 more | – | – | – |
| Reading · Turkish (in Combined) | 43.1 | 44 of 76tied with Drex 2.0, Cygnet, Open-Jev-27B-v1.1 and 4 more | 9.8 | f02a632 | text:1201Cerno server (a thin System One server around the card's CernoEngine, transformers: logic's for its English runs, compactor's di_serve.py for its Turkish runs), self-hosted, BF16 |
| Turkish Reading | 74.7 | 47 of 76tied with Cygnet, d1, Winnow-12B and 8 more | 3.0 | f02a632 | text:1201Cerno server (a thin System One server around the card's CernoEngine, transformers: logic's for its English runs, compactor's di_serve.py for its Turkish runs), self-hosted, BF16 |
| Turkish Grammar | 29.6 | 44 of 76tied with Clef, Drex 2.0, Cygnet and 4 more | 14.2 | f02a632 | text:1201Cerno server (a thin System One server around the card's CernoEngine, transformers: logic's for its English runs, compactor's di_serve.py for its Turkish runs), self-hosted, BF16 |
| English Reading | 82.4 | 33 of 75tied with d1.2, Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1 and 31 more | 4.2 | f02a632 | english:1085Cerno server (a thin System One server around the card's CernoEngine, transformers: logic's for its English runs, compactor's di_serve.py for its Turkish runs), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 29.1 | 45 of 69tied with Dexy-Decision-9B-V2, Mercury Decide, Perplexity Decider Flash v1.3 and 9 more | 17.0 | f02a632 | logic:cerno-9b_en, logic:cerno-9b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 45.0 | 42 of 69tied with vera-medium-omni, KnowLine 4B Gen5.5, Perplexity Decider Flash v1.3 and 5 more | 15.7 | f02a632 | math:cerno-9b_en, math:cerno-9b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 39.6 | 44 of 69tied with vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Perplexity Decider Flash v1.3 and 6 more | 16.2 | f02a632 | logic:cerno-9b_en, math:cerno-9b_en, logic:cerno-9b_en_dontknow, math:cerno-9b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 60.0 | 27 of 41tied with d1-2B, Mercury Decide, Winnow-12B and 2 more | – | f02a632 | backgammon:cerno-9b-v1-2League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #48. apertixai/cerno-9b-v1.2 @ f02a632 on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through the card's own in-process transformers engine (CernoEngine) behind logic's thin System One server (cerno_server.py, request/response mapping only), one request at a time; never on a hosted API. First stage 13 Nov 2026: its 9 hosted and program opponents (#1261–#1269, runner 2a22d1f); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1395–#1402, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1270 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Probabilities rounded to 4 places. The decider would cut a state ('Context:' block) over 32,768 tokens (its engine default) without refusing; that cut is never reached in League 1 (longest state block 5,856 tokens over its 31,793 distinct stored requests); the longest prompt (251 options) is 28,349 tokens, within its 262,144 context. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 81.9 | 23 of 55tied with Microsoft-Decision-1, vera-medium-omni, Jev 1.14 and 9 more | 2.9 | f02a632 | grading:site-cerno-9b-v1.2-v1apertixai/cerno-9b-v1.2@f02a632f, its card's Decision Index engine (CernoEngine, transformers 5.17.0, flash-linear-attention 0.5.2, kit decision-index@9eb2dbe2, the checkpoint's decider_config.json settings, loaded by Hub id and that revision as its card runs it, from the local copy; causal-conv1d not installed, so its PyTorch reference convolution) behind di_serve.py, our System One wrapper of the engine, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
KnowLine-9B-Gen2
- Maker: PelaAI
- Kind: decision model · class: Small
- Size: 9.7B
- Repo: PelaAI/KnowLine-9B-Gen2
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 59.2EN 81.5 Logic 29.9 Math 50.1 AC 75.3 | 31 of 53tied with Dexy-Decision-9B-V2, vera-medium-omni, KnowLine-9B-Gen1 and 4 more | – | – | – |
| Reading · Turkish (in Combined) | 41.6 | 45 of 76tied with KnowLine-9B-Gen1, Cerno 9B v1.2, Adjudex 9B v5 and 3 more | 5.1 | 8ec8eb6 | text:1306KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| Turkish Reading | 74.5 | 49 of 76tied with Winnow-12B, Drex 2.0, Adjudex 9B v5 and 6 more | 8.2 | 8ec8eb6 | text:1306KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| Turkish Grammar | 27.6 | 45 of 76tied with Open-Jev-27B-v1.1, KnowLine-9B-Gen1, Cerno 9B v1.2 and 3 more | 5.1 | 8ec8eb6 | text:1306KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted |
| English Reading | 81.5 | 44 of 75tied with Drex Max v1.1, Perplexity Decider 27B v1, d1.1 and 32 more | 8.7 | 8ec8eb6 | english:1321KnowLine server (knowline_server.py over SGLang, PelaAI's release code), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 29.9 | 44 of 69tied with d1.1, StartLux-Decision-v1.1-4B-Instant, Dexy-Decision-9B-V2 and 11 more | 3.2 | 8ec8eb6 | logic:knowline-9b-gen2_en, logic:knowline-9b-gen2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 50.1 | 33 of 69tied with Elephant-1, Jeffan-9B-1211, KnowLine-9B-Gen1 and 4 more | 4.8 | 8ec8eb6 | math:knowline-9b-gen2_en, math:knowline-9b-gen2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 43.2 | 34 of 69tied with Jeffan-9B-1211, GLiDE 9B, KnowLine-9B-Gen1 and 3 more | 3.6 | 8ec8eb6 | logic:knowline-9b-gen2_en, math:knowline-9b-gen2_en, logic:knowline-9b-gen2_en_dontknow, math:knowline-9b-gen2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 75.3 | 39 of 55tied with Perplexity Decider Flash v1.2, Gemini Decision 1 · 20261111, Elephant-1 and 4 more | 4.5 | 8ec8eb6 | grading:site-knowline-9b-gen2-v1PelaAI/KnowLine-9B-Gen2@8ec8eb60, its card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23 (the card names 0.5.21, which we do not have), BF16, SHA256SUMS checked, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
Adjudex 9B v5
- Maker: newsbang
- Kind: decision model · class: Small
- Size: 8.4B
- Repo: newsbang/adjudex-9b-v5
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 62.1EN 82.2 Logic 33.0 Math 52.6 AC 80.4 | 21 of 53tied with Jev 1.13, Jeffan-9B-1211, GLiDE 9B and 3 more | – | – | – |
| Reading · Turkish (in Combined) | 41.4 | 46 of 76tied with KnowLine-9B-Gen1, Cerno 9B v1.2, KnowLine-9B-Gen2 and 3 more | 9.2 | ffdcb82 | text:1183Perplexity Decider server, self-hosted |
| Turkish Reading | 76.0 | 43 of 76tied with Jev 1.13, Jeffan-9B-1211, Open-Jev-27B-v1.1 and 11 more | 2.5 | ffdcb82 | text:1183Perplexity Decider server, self-hosted |
| Turkish Grammar | 26.7 | 46 of 76tied with KnowLine-9B-Gen2, Elephant-1, GLiDE 9B and 2 more | 12.2 | ffdcb82 | text:1183Perplexity Decider server, self-hosted |
| English Reading | 82.2 | 39 of 75tied with d1.2, Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1 and 31 more | 3.4 | ffdcb82 | english:1186Perplexity Decider server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 33.0 | 32 of 69tied with Jeffan-9B-1211, Jev 1.14, GLiDE 9B and 14 more | 7.2 | ffdcb82 | logic:adjudex-9b_en, logic:adjudex-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 52.6 | 31 of 69tied with Elephant-1 | 4.8 | ffdcb82 | math:adjudex-9b_en, math:adjudex-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 45.9 | 31 of 69tied with Jev 1.13, Elephant-1, Jeffan-9B-1211 | 5.6 | ffdcb82 | logic:adjudex-9b_en, math:adjudex-9b_en, logic:adjudex-9b_en_dontknow, math:adjudex-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 80.4 | 28 of 55tied with d1.1, GLiDE 9B, Cerno 9B v1.2 and 7 more | 4.9 | ffdcb82 | grading:site-adjudex-9b-v5-v1newsbang/adjudex-9b-v5@ffdcb825, its card's autojev server (the Perplexity Decider v1.1 source, in the environment of its requirements), default settings, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
Elephant-1 post-trained
- Maker: Base Labs · training: post-trained
- Kind: decision model · class: Small
- Size: 9.4B
- Repo: baselabs/elephant-1
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 60.3 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 49.7TR 40.5 L&M 45.3 BG 63.2 | 18 of 36tied with Clef, Mercury Decide | – | – | – |
| Decision Index4 benches | 60.8EN 81.7 Logic 33.0 Math 51.7 AC 76.9 | 25 of 53tied with Adjudex 9B v5, GLiDE 9B, Microsoft-Decision-1 and 6 more | – | – | – |
| Reading · Turkish (in Combined) | 40.5 | 47 of 76tied with KnowLine-9B-Gen2, Adjudex 9B v5, GLiDE 9B and 2 more | 11.8 | b3f8a93 | text:373Base Labs server (qev, the model card's loader), self-hosted, BF16 |
| Turkish Reading | 75.6 | 44 of 76tied with Jeffan-9B-1211, Open-Jev-27B-v1.1, Cygnet and 10 more | 4.3 | b3f8a93 | text:373Base Labs server (qev, the model card's loader), self-hosted, BF16 |
| Turkish Grammar | 25.5 | 47 of 76tied with KnowLine-9B-Gen2, Adjudex 9B v5, GLiDE 9B and 3 more | 15.4 | b3f8a93 | text:373Base Labs server (qev, the model card's loader), self-hosted, BF16 |
| English Reading | 81.7 | 42 of 75tied with Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1, Perplexity Decider 27B v1 and 32 more | 2.7 | b3f8a93 | english:499Base Labs server (qev, the model card's loader), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 33.0 | 33 of 69tied with Jeffan-9B-1211, Jev 1.14, GLiDE 9B and 14 more | 8.6 | b3f8a93 | logic:elephant-1_en, logic:elephant-1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 51.7 | 32 of 69tied with Adjudex 9B v5, KnowLine-9B-Gen2, Jeffan-9B-1211 and 1 more | 4.3 | b3f8a93 | math:elephant-1_en, math:elephant-1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 45.3 | 32 of 69tied with Adjudex 9B v5, Jeffan-9B-1211 | 5.6 | b3f8a93 | logic:elephant-1_en, math:elephant-1_en, logic:elephant-1_en_dontknow, math:elephant-1_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | 63.2 | 24 of 41tied with GPT-6 Luna · Decisions API, SubQ Snap Preview, d1-2B | – | b3f8a93 | backgammon:base-labs-elephant-1League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup); its matches were played on 2 Nov 2026. Probabilities come rounded to 6 decimals by the loader and are stored as returned.; run on our GPUs |
| Answer Check | 76.9 | 35 of 55tied with Perplexity Decider Flash v1.3, StartLux-Decision-v1.1-9B-Instant, Chinese Decision Pointer 4B and 6 more | 3.7 | b3f8a93 | grading:site-elephant-1-v1baselabs/elephant-1@b3f8a934, the runner's baselabs_server.py in the card's setup.sh environment (OneJev 81ce62f1 with the checkpoint's serve.patch), on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
GLiDE 9B
- Maker: Fastino
- Kind: decision model · class: Small
- Size: 7.9B
- Repo: fastino/GLiDE-9B
- Latency ↓: median 207 ms, p95 259 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 907)
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 62.5 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 61.8EN 82.9 Logic 34.8 Math 47.4 AC 82.2 | 22 of 53tied with Jeffan-9B-1211, Adjudex 9B v5, Microsoft-Decision-1 and 2 more | – | – | – |
| Reading · Turkish (in Combined) | 40.1 | 48 of 76tied with KnowLine-9B-Gen2, Adjudex 9B v5, Elephant-1 and 2 more | 8.5 | ab54df5 | text:907GLiDE server (Fastino's release code), self-hosted |
| Turkish Reading | 75.2 | 45 of 76tied with Jeffan-9B-1211, Open-Jev-27B-v1.1, Cygnet and 10 more | 3.7 | ab54df5 | text:907GLiDE server (Fastino's release code), self-hosted |
| Turkish Grammar | 25.1 | 48 of 76tied with KnowLine-9B-Gen2, Adjudex 9B v5, Elephant-1 and 3 more | 12.0 | ab54df5 | text:907GLiDE server (Fastino's release code), self-hosted |
| English Reading | 82.9 | 30 of 75tied with Drex Max, Perplexity Decider 27B v1.3, d3-ultra and 33 more | 5.6 | ab54df5 | english:909GLiDE server (Fastino's release code), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 34.8 | 28 of 69tied with Gemini Decision 1 · 20261111, d1.2, Jeffan-9B-1211 and 9 more | 6.7 | ab54df5 | logic:glide-9b_en, logic:glide-9b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 47.4 | 40 of 69tied with Microsoft-Decision-1, Drex 2.0, Dexy-Decision-9B-V2 and 2 more | 7.1 | ab54df5 | math:glide-9b_en, math:glide-9b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 43.1 | 35 of 69tied with Jeffan-9B-1211, KnowLine-9B-Gen2, KnowLine-9B-Gen1 and 4 more | 6.7 | ab54df5 | logic:glide-9b_en, math:glide-9b_en, logic:glide-9b_en_dontknow, math:glide-9b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 82.2 | 22 of 55tied with Microsoft-Decision-1, vera-medium-omni, Jev 1.14 and 9 more | 7.2 | ab54df5 | grading:site-glide-9b-v1fastino/GLiDE-9B@ab54df5a, GLiDE server (python -m glide.server) in vLLM v0.30.0 image with flash-linear-attention 0.5.2, as for GLiDE 2B, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
Dexy-Decision-9B-V2 fine-tune
- Maker: Patrick-18 · training: fine-tune
- Kind: decision model · class: Small
- Size: 8.4B
- Repo: Patrick-18/Dexy-Decision-9B-V2
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 60.2EN 79.4 Logic 31.7 Math 48.5 AC 81.1 | 26 of 53tied with Microsoft-Decision-1, StartLux-Decision-v1.1-9B-Instant, Elephant-1 and 5 more | – | – | – |
| Reading · Turkish (in Combined) | 39.5 | 49 of 76tied with KnowLine-9B-Gen2, Adjudex 9B v5, Elephant-1 and 2 more | 9.1 | 4f434e2 | text:1255Dexy server (the card's Transformers d3_server.py, in its requirements.txt pins: torch 2.14.0, transformers 5.19.0), self-hosted, BF16 |
| Turkish Reading | 74.5 | 48 of 76tied with Winnow-12B, Drex 2.0, Adjudex 9B v5 and 6 more | 3.2 | 4f434e2 | text:1255Dexy server (the card's Transformers d3_server.py, in its requirements.txt pins: torch 2.14.0, transformers 5.19.0), self-hosted, BF16 |
| Turkish Grammar | 24.6 | 49 of 76tied with Adjudex 9B v5, Elephant-1, GLiDE 9B and 3 more | 12.7 | 4f434e2 | text:1255Dexy server (the card's Transformers d3_server.py, in its requirements.txt pins: torch 2.14.0, transformers 5.19.0), self-hosted, BF16 |
| English Reading | 79.4 | 52 of 75tied with StartLux-Decision-v1.1-9B-Instant, d1, Clef and 30 more | 4.2 | 4f434e2 | english:1258Dexy server (the card's Transformers d3_server.py, in its requirements.txt pins: torch 2.14.0, transformers 5.19.0), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 31.7 | 37 of 69tied with celeris-1-decision, celeris-1.1-decision, StartLux-Decision-v1.1-9B-Instant and 14 more | 14.9 | 4f434e2 | logic:dexy-9b_en, logic:dexy-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 48.5 | 38 of 69tied with KnowLine-9B-Gen2, Jeffan-9B-1211, KnowLine-9B-Gen1 and 4 more | 12.1 | 4f434e2 | math:dexy-9b_en, math:dexy-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 42.7 | 39 of 69tied with KnowLine-9B-Gen2, GLiDE 9B, KnowLine-9B-Gen1 and 3 more | 13.1 | 4f434e2 | logic:dexy-9b_en, math:dexy-9b_en, logic:dexy-9b_en_dontknow, math:dexy-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 81.1 | 25 of 55tied with Jeffan-9B-1211, Jev 1.13, d1.1 and 7 more | 2.5 | 4f434e2 | grading:site-dexy-decision-9b-v2-v1Patrick-18/Dexy-Decision-9B-V2@4f434e2b, its card's Transformers service (d3_server.py, in the environment its requirements.txt pins: torch 2.14.0, transformers 5.19.0), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
StartLux-Decision-v1.1-9B-Instant
- Maker: StartLux
- Kind: decision model · class: Small
- Size: 9.7B
- Repo: startlux-models/StartLux-Decision-v1.1-9B-Instant
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 60.9EN 83.5 Logic 33.2 Math 48.0 AC 78.8 | 24 of 53tied with Adjudex 9B v5, GLiDE 9B, Microsoft-Decision-1 and 5 more | – | – | – |
| Reading · Turkish (in Combined) | 39.0 | 50 of 76tied with Elephant-1, GLiDE 9B, Dexy-Decision-9B-V2 and 1 more | 7.6 | 18a17f7 | text:1290StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| Turkish Reading | 73.5 | 50 of 76tied with Drex 2.0, Adjudex 9B v5, Elephant-1 and 5 more | 3.6 | 18a17f7 | text:1290StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| Turkish Grammar | 24.4 | 50 of 76tied with Adjudex 9B v5, Elephant-1, GLiDE 9B and 3 more | 11.3 | 18a17f7 | text:1290StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| English Reading | 83.5 | 26 of 75tied with Mammoth-1, d3, Drex Max and 35 more | 6.6 | 18a17f7 | english:1293StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 33.2 | 31 of 69tied with Jeffan-9B-1211, Jev 1.14, GLiDE 9B and 13 more | 3.8 | 18a17f7 | logic:startlux-9b_en, logic:startlux-9b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 48.0 | 39 of 69tied with KnowLine-9B-Gen2, Jeffan-9B-1211, KnowLine-9B-Gen1 and 5 more | 2.7 | 18a17f7 | math:startlux-9b_en, math:startlux-9b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 43.0 | 37 of 69tied with KnowLine-9B-Gen2, GLiDE 9B, KnowLine-9B-Gen1 and 3 more | 2.7 | 18a17f7 | logic:startlux-9b_en, math:startlux-9b_en, logic:startlux-9b_en_dontknow, math:startlux-9b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 78.8 | 31 of 55tied with KnowLine-12B-Gen1, KnowLine 4B Gen5.5, Adjudex 9B v5 and 6 more | 8.2 | 18a17f7 | grading:site-startlux-decision-v1.1-9b-v1startlux-models/StartLux-Decision-v1.1-9B-Instant@18a17f73, its package's own server (startlux-decide serve --launch-vllm, StartLux-Decision@a99977d, vLLM 0.31.0), BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency) |
Perplexity Decider Flash v1.3
- Maker: Perplexity
- Kind: decision model · class: Tiny
- Size: 4.5B
- Repo: perplexity-ai/pplx-decider-flash-v1.3
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 58.6EN 79.7 Logic 30.9 Math 44.9 AC 78.8 | 32 of 53tied with vera-medium-omni, KnowLine-9B-Gen1, Cerno 9B v1.2 and 5 more | – | – | – |
| Reading · Turkish (in Combined) | 37.2 | 51 of 76tied with StartLux-Decision-v1.1-9B-Instant, vera-medium-omni, KnowLine 4B Gen5.5 | 12.6 | d06fb3f | text:1123Perplexity Decider server, self-hosted |
| Turkish Reading | 68.0 | 54 of 76tied with vera-medium-omni, Valen-Decision-1.5-9B, KnowLine 4B Gen5.5 and 4 more | 6.2 | d06fb3f | text:1123Perplexity Decider server, self-hosted |
| Turkish Grammar | 24.0 | 51 of 76tied with Elephant-1, GLiDE 9B, Dexy-Decision-9B-V2 and 3 more | 15.3 | d06fb3f | text:1123Perplexity Decider server, self-hosted |
| English Reading | 79.7 | 49 of 75tied with StartLux-Decision-v1.1-9B-Instant, d1, Clef and 30 more | 2.7 | d06fb3f | english:1126Perplexity Decider server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 30.9 | 39 of 69tied with StartLux-Decision-v1.1-9B-Instant, Adjudex 9B v5, Elephant-1 and 13 more | 10.1 | d06fb3f | logic:decider-flash-v1.3_en, logic:decider-flash-v1.3_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 44.9 | 44 of 69tied with vera-medium-omni, Cerno 9B v1.2, KnowLine 4B Gen5.5 and 5 more | 11.7 | d06fb3f | math:decider-flash-v1.3_en, math:decider-flash-v1.3_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 40.1 | 43 of 69tied with Drex 2.0, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant and 4 more | 11.1 | d06fb3f | logic:decider-flash-v1.3_en, math:decider-flash-v1.3_en, logic:decider-flash-v1.3_en_dontknow, math:decider-flash-v1.3_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 78.8 | 30 of 55tied with KnowLine-12B-Gen1, KnowLine 4B Gen5.5, Adjudex 9B v5 and 6 more | 4.2 | d06fb3f | grading:site-perplexity-decider-flash-v1.3-v1perplexity-ai/pplx-decider-flash-v1.3@d06fb3f2, the runner's pplx_decider_server.py with the checkpoint's own pplx_decider package (as for its published v1.2 runs), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency); before the run, 3 published-cohort pairs were sent in test requests whose outputs were not kept, so their scored rows come from a later request |
vera-medium-omni
- Maker: scar-ai
- Kind: decision model · class: Tiny
- Size: 4.2B
- Repo: scar-ai/vera-medium-omni
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 59.8EN 79.4 Logic 30.1 Math 46.0 AC 83.7 | 27 of 53tied with StartLux-Decision-v1.1-9B-Instant, Elephant-1, Dexy-Decision-9B-V2 and 6 more | – | – | – |
| Reading · Turkish (in Combined) | 36.4 | 52 of 76tied with Perplexity Decider Flash v1.3, KnowLine 4B Gen5.5, StartLux-Decision-v1.1-4B-Instant and 1 more | 3.7 | 1f23a6c | text:1264vera server (the card's own vera-serve from vera-s1 0.1.8, PyTorch engine), self-hosted, BF16 |
| Turkish Reading | 69.3 | 51 of 76tied with Valen-Decision-1.5-9B, KnowLine 4B Gen5.5, Perplexity Decider Flash v1.3 and 4 more | 9.1 | 1f23a6c | text:1264vera server (the card's own vera-serve from vera-s1 0.1.8, PyTorch engine), self-hosted, BF16 |
| Turkish Grammar | 22.3 | 52 of 76tied with Dexy-Decision-9B-V2, StartLux-Decision-v1.1-9B-Instant, Perplexity Decider Flash v1.3 and 5 more | 8.0 | 1f23a6c | text:1264vera server (the card's own vera-serve from vera-s1 0.1.8, PyTorch engine), self-hosted, BF16 |
| English Reading | 79.4 | 53 of 75tied with StartLux-Decision-v1.1-9B-Instant, d1, Clef and 30 more | 9.9 | 1f23a6c | english:1267vera server (the card's own vera-serve from vera-s1 0.1.8, PyTorch engine), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 30.1 | 43 of 69tied with Adjudex 9B v5, Elephant-1, Jev 1.13 and 14 more | 4.7 | 1f23a6c | logic:vera-medium-omni_en, logic:vera-medium-omni_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 46.0 | 41 of 69tied with StartLux-Decision-v1.1-9B-Instant, GLiDE 9B, Cerno 9B v1.2 and 6 more | 3.0 | 1f23a6c | math:vera-medium-omni_en, math:vera-medium-omni_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 40.6 | 41 of 69tied with Drex 2.0, StartLux-Decision-v1.1-4B-Instant, Perplexity Decider Flash v1.3 and 4 more | 3.5 | 1f23a6c | logic:vera-medium-omni_en, math:vera-medium-omni_en, logic:vera-medium-omni_en_dontknow, math:vera-medium-omni_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 83.7 | 17 of 55tied with Logits Sigma-1 Duo, GLiDE · API, GLiDE · v1.2 and 10 more | 12.4 | 1f23a6c | grading:site-vera-medium-omni-v1scar-ai/vera-medium-omni@1f23a6cf, its card's own server (vera-serve from vera-s1 0.1.8, PyTorch engine), BF16, on one RTX PRO 6000 (ours), self-hosted; server probabilities are rounded to four decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency) |
KnowLine 4B Gen5.5 LoRA, merged
- Maker: PelaAI · training: LoRA, merged
- Kind: decision model · class: Tiny
- Size: 4.7B
- Repo: PelaAI/KnowLine-4B-Gen5.5
- Latency ↓: median 99 ms, p95 149 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 965)
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 56.0 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 43.5TR 35.6 L&M 39.1 BG 55.8 | 24 of 36tied with Winnow-12B, Open-Jev-27B-v1.1 | – | – | – |
| Decision Index4 benches | 57.4EN 76.0 Logic 27.9 Math 45.0 AC 80.6 | 34 of 53tied with Perplexity Decider Flash v1.3, d1, Valen-Decision-1.5-9B and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 35.6 | 53 of 76tied with Perplexity Decider Flash v1.3, vera-medium-omni, StartLux-Decision-v1.1-4B-Instant and 2 more | 13.5 | 779944d | text:965KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, FP8 at load |
| Turkish Reading | 68.6 | 53 of 76tied with vera-medium-omni, Valen-Decision-1.5-9B, Perplexity Decider Flash v1.3 and 4 more | 6.2 | 779944d | text:965KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, FP8 at load |
| Turkish Grammar | 21.6 | 53 of 76tied with Perplexity Decider Flash v1.3, vera-medium-omni, Perplexity Decider Flash v1.2 and 3 more | 16.6 | 779944d | text:965KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, FP8 at load |
| English Reading | 76.0 | 58 of 75tied with GPT-6 Luna · Decisions API, celeris-1-decision, Cygnet and 11 more | 5.8 | 779944d | english:961KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, FP8 at load; the kept reading questions in English |
| Vision | 12.7 | 15 of 20tied with GPT-6 Luna · Decisions API | 11.0 | 779944d | vision:92KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, FP8 at load; the page image only (condition B) |
| Logic | 27.9 | 48 of 69tied with Microsoft-Decision-1, Valen-Decision-1.5-9B, KnowLine-9B-Gen1 and 10 more | 13.7 | 779944d | logic:knowline-4b_en, logic:knowline-4b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 45.0 | 43 of 69tied with vera-medium-omni, Cerno 9B v1.2, Perplexity Decider Flash v1.3 and 5 more | 8.6 | 779944d | math:knowline-4b_en, math:knowline-4b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 39.1 | 46 of 69tied with vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Perplexity Decider Flash v1.3 and 6 more | 10.4 | 779944d | logic:knowline-4b_en, math:knowline-4b_en, logic:knowline-4b_en_dontknow, math:knowline-4b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 55.8 | 31 of 41tied with KnowLine-2B-Gen1 | – | 779944d | backgammon:knowline-4b-gen5-5League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #22. PelaAI/KnowLine-4B-Gen5.5 @ 779944d on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), FP8 at load, through the repo's own serve_knowline.sh (SGLang 0.5.21 with FP8 at load, as the card serves it, then its knowline_server.py front end), one request at a time; never on a hosted API. First stage 11 Nov 2026: its 9 hosted and program opponents (#957–#965, runner b727222); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1371–#1378, runner b727222). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #966 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). FP8 at load (--quantization fp8, as the card and serve_knowline.sh serve it), --mem-fraction-static 0.72. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 80.6 | 27 of 55tied with d1.1, GLiDE 9B, Cerno 9B v1.2 and 7 more | 4.5 | 779944d | grading:site-knowline-4b-gen5.5-v1PelaAI/KnowLine-4B-Gen5.5@779944d2 (the site's pin; no longer in the maker's repository, copied from the snapshot its published runs used, SHA256SUMS checked), its card's serve_knowline.sh (SGLang with FP8 at load, then knowline_server.py, card defaults) in the lmsysorg/sglang image with SGLang 0.5.23, on one RTX PRO 6000 (ours), self-hosted; its published runs were on a DGX Spark; one question per request, 1 request at a time (latency not comparable across different concurrency) |
StartLux-Decision-v1.1-4B-Instant fine-tune
- Maker: StartLux · training: fine-tune
- Kind: decision model · class: Tiny
- Size: 4.7B
- Repo: startlux-models/StartLux-Decision-v1.1-4B-Instant
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 54.8 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 47.4TR 34.8 L&M 40.5 BG 66.8 | 21 of 36tied with Cerno 9B v1.2, Cygnet | – | – | – |
| Decision Index4 benches | 59.3EN 78.5 Logic 32.3 Math 44.8 AC 81.7 | 30 of 53tied with Elephant-1, Dexy-Decision-9B-V2, vera-medium-omni and 5 more | – | – | – |
| Reading · Turkish (in Combined) | 34.8 | 54 of 76tied with vera-medium-omni, KnowLine 4B Gen5.5, Perplexity Decider Flash v1.2 and 4 more | 7.1 | 416088f | text:1236StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| Turkish Reading | 67.6 | 55 of 76tied with vera-medium-omni, Valen-Decision-1.5-9B, KnowLine 4B Gen5.5 and 5 more | 2.6 | 416088f | text:1236StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| Turkish Grammar | 20.9 | 55 of 76tied with vera-medium-omni, KnowLine 4B Gen5.5, Perplexity Decider Flash v1.2 and 4 more | 10.3 | 416088f | text:1236StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| English Reading | 78.5 | 54 of 75tied with Jev 1.14, Cerno 9B v1.2, Drex 2.0 and 24 more | 3.2 | 416088f | english:1081StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 32.3 | 36 of 69tied with Jeffan-9B-1211, Jev 1.14, GLiDE 9B and 15 more | 3.3 | 416088f | logic:startlux-4b_en, logic:startlux-4b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 44.8 | 45 of 69tied with vera-medium-omni, Cerno 9B v1.2, KnowLine 4B Gen5.5 and 5 more | 6.7 | 416088f | math:startlux-4b_en, math:startlux-4b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 40.5 | 42 of 69tied with Drex 2.0, vera-medium-omni, Perplexity Decider Flash v1.3 and 4 more | 5.4 | 416088f | logic:startlux-4b_en, math:startlux-4b_en, logic:startlux-4b_en_dontknow, math:startlux-4b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 66.8 | 20 of 41tied with Clef, Drex 2.0, Cygnet and 2 more | – | 416088f | backgammon:startlux-decision-v1-1-4b-instantLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #47. startlux-models/StartLux-Decision-v1.1-4B-Instant @ 416088f on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through its package's own server (startlux-decide 1.1.0 serve --launch-vllm, vLLM 0.31.0), one request at a time; never on a hosted API. First stage 13 Nov 2026: its 9 hosted and program opponents (#1251–#1259, runner 62ae46a); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1387–#1394, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1260 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Instant: one read-out per pass, no thinking; 26 letters per pass including the hidden 'None of the above' (serve's default none_check); the server knocks longer lists out in groups of at most 25. Never truncates: vLLM refuses over 131,072 tokens (422); nothing refused. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 81.7 | 24 of 55tied with vera-medium-omni, Jev 1.14, Jeffan-9B-1211 and 8 more | 8.3 | 416088f | grading:site-startlux-decision-v1.1-4b-v1startlux-models/StartLux-Decision-v1.1-4B-Instant@416088f7, its package's own server (startlux-decide serve --launch-vllm, StartLux-Decision@a99977d, vLLM 0.31.0) as for its published Reading and Logic & Math runs, BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency) |
Perplexity Decider Flash v1.2
- Maker: Perplexity
- Kind: decision model · class: Tiny
- Size: 4.5B
- Repo: perplexity-ai/pplx-decider-flash-v1.2
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 54.3 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 41.4TR 34.7 L&M 38.9 BG 50.8 | 26 of 36 | – | – | – |
| Decision Index4 benches | 56.9EN 77.4 Logic 28.9 Math 44.1 AC 77.2 | 36 of 53tied with d1, KnowLine 4B Gen5.5, Valen-Decision-1.5-9B | – | – | – |
| Reading · Turkish (in Combined) | 34.7 | 55 of 76tied with vera-medium-omni, KnowLine 4B Gen5.5, StartLux-Decision-v1.1-4B-Instant and 4 more | 8.3 | 6d771ad | text:223Perplexity Decider server, self-hosted |
| Turkish Reading | 65.9 | 58 of 76tied with vera-medium-omni, Valen-Decision-1.5-9B, KnowLine 4B Gen5.5 and 6 more | 3.7 | 6d771ad | text:223Perplexity Decider server, self-hosted |
| Turkish Grammar | 21.4 | 54 of 76tied with vera-medium-omni, KnowLine 4B Gen5.5, StartLux-Decision-v1.1-4B-Instant and 2 more | 11.0 | 6d771ad | text:223Perplexity Decider server, self-hosted |
| English Reading | 77.4 | 57 of 75tied with KnowLine-9B-Gen2, GPT-6 Luna · Decisions API, celeris-1-decision and 12 more | 4.1 | 6d771ad | english:533Perplexity Decider server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 28.9 | 46 of 69tied with Dexy-Decision-9B-V2, Mercury Decide, Perplexity Decider Flash v1.3 and 11 more | 6.0 | 6d771ad | logic:decider-flash-v1.2_en, logic:decider-flash-v1.2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 44.1 | 48 of 69tied with vera-medium-omni, Cerno 9B v1.2, KnowLine 4B Gen5.5 and 6 more | 10.4 | 6d771ad | math:decider-flash-v1.2_en, math:decider-flash-v1.2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 38.9 | 47 of 69tied with vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Perplexity Decider Flash v1.3 and 6 more | 8.9 | 6d771ad | logic:decider-flash-v1.2_en, math:decider-flash-v1.2_en, logic:decider-flash-v1.2_en_dontknow, math:decider-flash-v1.2_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | 50.8 | 33 of 41 | – | – | backgammon:decider-flash-v1-2League 1; calibration definition pending Ümit; run on our GPUs |
| Answer Check | 77.2 | 33 of 55tied with d1, Perplexity Decider Flash v1.3, StartLux-Decision-v1.1-9B-Instant and 7 more | 2.8 | 6d771ad | grading:site-perplexity-decider-flash-v1.2-v1perplexity-ai/pplx-decider-flash-v1.2@6d771ad7, the runner's pplx_decider_server.py with the checkpoint's own pplx_decider package (as for its published v1.2 runs), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
Valen-Decision-1.5-9B
- Maker: Valen-Team
- Kind: decision model · class: Small
- Size: 8.4B
- Repo: Valen-Team/Valen-Decision-1.5-9B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 57.1EN 79.7 Logic 30.4 Math 44.2 AC 74.3 | 35 of 53tied with Perplexity Decider Flash v1.3, d1, KnowLine 4B Gen5.5 and 1 more | – | – | – |
| Reading · Turkish (in Combined) | 33.7 | 56 of 76tied with KnowLine 4B Gen5.5, StartLux-Decision-v1.1-4B-Instant, Perplexity Decider Flash v1.2 and 4 more | 3.8 | 5a51211 | text:1275Valen server (the card's valen-vllm plugin, Valen-Team/Valen at 773d3a6, in vllm/vllm-openai v0.33.0), self-hosted, BF16 |
| Turkish Reading | 68.8 | 52 of 76tied with vera-medium-omni, KnowLine 4B Gen5.5, Perplexity Decider Flash v1.3 and 4 more | 9.7 | 5a51211 | text:1275Valen server (the card's valen-vllm plugin, Valen-Team/Valen at 773d3a6, in vllm/vllm-openai v0.33.0), self-hosted, BF16 |
| Turkish Grammar | 18.8 | 58 of 76tied with StartLux-Decision-v1.1-4B-Instant, Chinese Decision Pointer 4B, Verdin-4B-v1.1 and 2 more | 6.9 | 5a51211 | text:1275Valen server (the card's valen-vllm plugin, Valen-Team/Valen at 773d3a6, in vllm/vllm-openai v0.33.0), self-hosted, BF16 |
| English Reading | 79.7 | 50 of 75tied with StartLux-Decision-v1.1-9B-Instant, d1, Clef and 30 more | 8.4 | 5a51211 | english:1278Valen server (the card's valen-vllm plugin, Valen-Team/Valen at 773d3a6, in vllm/vllm-openai v0.33.0), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 30.4 | 41 of 69tied with StartLux-Decision-v1.1-9B-Instant, Adjudex 9B v5, Elephant-1 and 14 more | 5.8 | 5a51211 | logic:valen-9b_en, logic:valen-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 44.2 | 47 of 69tied with vera-medium-omni, Cerno 9B v1.2, KnowLine 4B Gen5.5 and 6 more | 4.1 | 5a51211 | math:valen-9b_en, math:valen-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 39.5 | 45 of 69tied with vera-medium-omni, StartLux-Decision-v1.1-4B-Instant, Perplexity Decider Flash v1.3 and 6 more | 4.5 | 5a51211 | logic:valen-9b_en, math:valen-9b_en, logic:valen-9b_en_dontknow, math:valen-9b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 74.3 | 40 of 55tied with KnowLine-9B-Gen1, Verdin-4B-v1.1, KnowLine-9B-Gen2 and 1 more | 3.2 | 5a51211 | grading:site-valen-decision-1.5-9b-v1Valen-Team/Valen-Decision-1.5-9B@5a512117, its card's vLLM deployment (valen-vllm, the Valen plugin at Valen-Team/Valen@773d3a64 in the official vllm/vllm-openai:v0.33.0 image), BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
Verdin-4B-v1.1 LoRA, merged
- Maker: victorbennett · training: LoRA, merged
- Kind: decision model · class: Tiny
- Size: 4.2B
- Repo: victorbennett/Verdin-4B-v1.1
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 54.3EN 77.8 Logic 24.7 Math 39.4 AC 75.4 | 37 of 53tied with Chinese Decision Pointer 4B, Winnow-12B | – | – | – |
| Reading · Turkish (in Combined) | 33.5 | 57 of 76tied with StartLux-Decision-v1.1-4B-Instant, Perplexity Decider Flash v1.2, Valen-Decision-1.5-9B and 3 more | 9.8 | 45830d2 | text:1244Verdin server (the card's own python -m verdin.server, in the package versions its card lists), self-hosted, BF16 |
| Turkish Reading | 64.0 | 59 of 76tied with StartLux-Decision-v1.1-4B-Instant, Microsoft-Decision-1, Clef-Flash and 2 more | 3.9 | 45830d2 | text:1244Verdin server (the card's own python -m verdin.server, in the package versions its card lists), self-hosted, BF16 |
| Turkish Grammar | 20.5 | 57 of 76tied with vera-medium-omni, KnowLine 4B Gen5.5, Perplexity Decider Flash v1.2 and 4 more | 12.8 | 45830d2 | text:1244Verdin server (the card's own python -m verdin.server, in the package versions its card lists), self-hosted, BF16 |
| English Reading | 77.8 | 56 of 75tied with Elephant-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2 and 14 more | 3.5 | 45830d2 | english:1247Verdin server (the card's own python -m verdin.server, in the package versions its card lists), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 24.7 | 54 of 69tied with Drex 2.0, Clef, d1 and 5 more | 10.8 | 45830d2 | logic:verdin-4b_en, logic:verdin-4b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 39.4 | 52 of 69tied with Clef, Chinese Decision Pointer 4B, Clef-Flash | 5.7 | 45830d2 | math:verdin-4b_en, math:verdin-4b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 34.3 | 53 of 69tied with Clef, Chinese Decision Pointer 4B | 7.3 | 45830d2 | logic:verdin-4b_en, math:verdin-4b_en, logic:verdin-4b_en_dontknow, math:verdin-4b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 75.4 | 38 of 55tied with Perplexity Decider Flash v1.2, Gemini Decision 1 · 20261111, Elephant-1 and 4 more | 3.4 | 45830d2 | grading:site-verdin-4b-v1.1-v1victorbennett/Verdin-4B-v1.1@45830d21, its card's own server (python -m verdin.server, in the package versions its card lists: torch 2.8.0 cu128, transformers 5.17.0, flash-linear-attention 0.5.2, triton 3.8.0; no causal-conv1d, so its own torch conv path), BF16, on one RTX PRO 6000 (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency) |
Chinese Decision Pointer 4B LoRA, merged
- Maker: quincyqiang · training: LoRA, merged
- Kind: decision model · class: Tiny
- Size: 4.2B
- Repo: quincyqiang/chinese-decision-pointer-4b
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 54.3EN 75.5 Logic 25.4 Math 38.0 AC 78.1 | 38 of 53tied with Verdin-4B-v1.1, Winnow-12B | – | – | – |
| Reading · Turkish (in Combined) | 33.3 | 58 of 76tied with StartLux-Decision-v1.1-4B-Instant, Perplexity Decider Flash v1.2, Valen-Decision-1.5-9B and 3 more | 11.7 | 889f23e | text:1206Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16 |
| Turkish Reading | 63.6 | 60 of 76tied with Microsoft-Decision-1, Clef-Flash, Perplexity Decider Flash v1.2 and 1 more | 3.3 | 889f23e | text:1206Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16 |
| Turkish Grammar | 20.5 | 56 of 76tied with vera-medium-omni, KnowLine 4B Gen5.5, Perplexity Decider Flash v1.2 and 4 more | 15.4 | 889f23e | text:1206Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16 |
| English Reading | 75.5 | 59 of 75tied with Perplexity Decider Flash v1.3, Valen-Decision-1.5-9B, celeris-1.1-decision and 8 more | 3.5 | 889f23e | english:1209Decision Index engine (compactor's di_serve.py around the engine its card ships), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 25.4 | 53 of 69tied with Open-Jev-27B-v1.1, KnowLine 4B Gen5.5, Drex 2.0 and 7 more | 15.5 | 889f23e | logic:chinese-pointer-4b_en, logic:chinese-pointer-4b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 38.0 | 54 of 69tied with Verdin-4B-v1.1, Clef, Clef-Flash | 13.7 | 889f23e | math:chinese-pointer-4b_en, math:chinese-pointer-4b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 33.7 | 54 of 69tied with Clef, Verdin-4B-v1.1, Clef-Flash | 14.3 | 889f23e | logic:chinese-pointer-4b_en, math:chinese-pointer-4b_en, logic:chinese-pointer-4b_en_dontknow, math:chinese-pointer-4b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 78.1 | 32 of 55tied with d1, Perplexity Decider Flash v1.3, StartLux-Decision-v1.1-9B-Instant and 5 more | 6.7 | 889f23e | grading:site-chinese-decision-pointer-4b-v1quincyqiang/chinese-decision-pointer-4b@889f23e2, its card's Decision Index engine (yanqiangmiffy/chinese-decision@c16450b, ChineseDecisionEngine with debias 'reverse' as its card's run; transformers 5.17.0, kit decision-index@9eb2dbe2, as the card names no kit commit; causal-conv1d not installed, so its PyTorch reference convolution) behind di_serve.py, our System One wrapper of the engine, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
Microsoft-Decision-1
- Maker: Microsoft
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 61.5EN 82.2 Logic 30.6 Math 49.2 AC 83.9 | 23 of 53tied with Adjudex 9B v5, GLiDE 9B, StartLux-Decision-v1.1-9B-Instant and 2 more | – | – | – |
| Reading · Turkish (in Combined) | 33.1 | 59 of 76tied with StartLux-Decision-v1.1-4B-Instant, Perplexity Decider Flash v1.2, Valen-Decision-1.5-9B and 3 more | 19.8 | – | text:205OpenRouter System One API |
| Turkish Reading | 66.8 | 56 of 76tied with vera-medium-omni, Valen-Decision-1.5-9B, KnowLine 4B Gen5.5 and 6 more | 8.9 | – | text:205OpenRouter System One API |
| Turkish Grammar | 18.7 | 59 of 76tied with StartLux-Decision-v1.1-4B-Instant, Chinese Decision Pointer 4B, Verdin-4B-v1.1 and 2 more | 24.4 | – | text:205OpenRouter System One API |
| English Reading | 82.2 | 41 of 75tied with d1.2, Perplexity Decider 27B v1.2 · DGX Spark, Drex Max v1.1 and 31 more | 2.8 | – | english:412, english:420, english:424, english:427, english:431OpenRouter System One API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 30.6 | 40 of 69tied with StartLux-Decision-v1.1-9B-Instant, Adjudex 9B v5, Elephant-1 and 14 more | 8.0 | – | logic:ms-decision-1_en, logic:ms-decision-1_en_dontknowhosted API via OpenRouter |
| Math | 49.2 | 36 of 69tied with KnowLine-9B-Gen2, Jeffan-9B-1211, KnowLine-9B-Gen1 and 4 more | 4.1 | – | math:ms-decision-1_en, math:ms-decision-1_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 42.8 | 38 of 69tied with KnowLine-9B-Gen2, GLiDE 9B, KnowLine-9B-Gen1 and 3 more | 4.9 | – | logic:ms-decision-1_en, math:ms-decision-1_en, logic:ms-decision-1_en_dontknow, math:ms-decision-1_en_dontknowhosted API via OpenRouter |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 83.9 | 16 of 55tied with GLiDE 27B, Logits Sigma-1 Duo, GLiDE · API and 10 more | 2.4 | – | grading:site-microsoft-decision-1-v1OpenRouter decisions API (openrouter.ai/api, model microsoft/microsoft-decision-1); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
Qwen3.5-4B reasoning off · chat
- Maker: Alibaba
- Kind: chat model · class: Tiny
- Size: 4.7B
- Repo: Qwen/Qwen3.5-4B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic and Math | ||||
| Reading · Turkish (in Combined) | 32.7 | 31 of 40tied with Qwen3-14B · reasoning off · chat, Erk-14B · reasoning off · chat | – | 851bf6e | text:1146our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-4B@851bf6e), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking off; output cap 32,768 |
| Turkish Reading | 60.6 | 33 of 40 | – | 851bf6e | text:1146our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-4B@851bf6e), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking off; output cap 32,768 |
| Turkish Grammar | 20.9 | 31 of 40 | – | 851bf6e | text:1146our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-4B@851bf6e), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking off; output cap 32,768 |
| English Reading | 75.5 | 32 of 47tied with LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat, Qwen3-14B · reasoning off · chat and 1 more | – | 851bf6e | english:1143our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-4B@851bf6e), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking off; output cap 32,768; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 64.6 | 15 of 20tied with Qwen3.5-4B · reasoning on · chat | – | 851bf6e | grading:chat-qwen3.5-4b-nothink-v1, grading:chat-qwen3.5-4b-nothink-seed1-v1, grading:chat-qwen3.5-4b-nothink-seed2-v1Qwen/Qwen3.5-4B@851bf6e8, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 1.0% (per seed in seed_runs; cut off at the token limit per seed: 0, 1, 0); seed 0 is the published single run (every reply kept); seeds 1 and 2 are new runs (16 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs |
Clef-Flash post-trained
- Maker: Cloudflare · training: post-trained
- Kind: decision model · class: Small
- Size: 9.5B
- Repo: Cloudflare/clef-flash
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 52.1EN 77.8 Logic 23.7 Math 37.2 AC 69.8 | 40 of 53tied with Cygnet | – | – | – |
| Reading · Turkish (in Combined) | 32.2 | 60 of 76tied with Valen-Decision-1.5-9B, Verdin-4B-v1.1, Chinese Decision Pointer 4B and 1 more | 15.5 | fde727a | text:139Clef server, self-hosted |
| Turkish Reading | 66.3 | 57 of 76tied with vera-medium-omni, Valen-Decision-1.5-9B, KnowLine 4B Gen5.5 and 6 more | 5.6 | fde727a | text:139Clef server, self-hosted |
| Turkish Grammar | 17.6 | 60 of 76tied with Valen-Decision-1.5-9B, Microsoft-Decision-1 | 20.3 | fde727a | text:139Clef server, self-hosted |
| English Reading | 77.8 | 55 of 75tied with Elephant-1, Open-Jev-27B-v1.1, KnowLine-9B-Gen2 and 14 more | 4.0 | fde727a | english:614Clef server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 23.7 | 57 of 69tied with Clef, d1, OneJev-27B and 4 more | 11.5 | fde727a | logic:clef-flash_en, logic:clef-flash_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 37.2 | 55 of 69tied with Verdin-4B-v1.1, Clef, Chinese Decision Pointer 4B | 2.7 | fde727a | math:clef-flash_en, math:clef-flash_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 32.6 | 55 of 69tied with Chinese Decision Pointer 4B | 5.7 | fde727a | logic:clef-flash_en, math:clef-flash_en, logic:clef-flash_en_dontknow, math:clef-flash_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 69.8 | 42 of 55 | 5.9 | fde727a | grading:site-clef-flash-v1Cloudflare/clef-flash@fde727a2, the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; server probabilities are rounded to four decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency) |
Qwen3-14B reasoning off · chat
- Maker: Alibaba
- Kind: chat model · class: none
- Size: 14.8B
- Repo: Qwen/Qwen3-14B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 32.2 | 32 of 40tied with Qwen3.5-4B · reasoning off · chat, Erk-14B · reasoning off · chat | 41.0 | 40c0698 | text:131vLLM, self-hosted, BF16 |
| Turkish Reading | 65.1 | 32 of 40tied with Erk-14B · reasoning off · chat | 23.2 | 40c0698 | text:131vLLM, self-hosted, BF16 |
| Turkish Grammar | 18.1 | 32 of 40tied with Erk-14B · reasoning off · chat | 48.6 | 40c0698 | text:131vLLM, self-hosted, BF16 |
| English Reading | 74.6 | 33 of 47tied with LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat and 1 more | 17.8 | 40c0698 | english:770vLLM, self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 56.7 | 30 of 35tied with Raon-1 nano · reasoning off · chat, Qwen3.5-2B · reasoning off · chat | – | 40c0698 | logic:qwen3-14b_en, logic:qwen3-14b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 94.7 | 31 of 36tied with Gemma 4 E2B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat | – | 40c0698 | math:qwen3-14b_en, math:qwen3-14b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 81.7 | 29 of 34tied with Qwen3.5-2B · reasoning off · chat | – | 40c0698 | logic:qwen3-14b_en, math:qwen3-14b_en, logic:qwen3-14b_en_dontknow, math:qwen3-14b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Erk-14B reasoning off · chat
- Maker: eCloud Technology
- Kind: chat model · class: none
- Size: 14.8B
- Repo: ecloudtech/Erk-14B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 31.2 | 33 of 40tied with Qwen3.5-4B · reasoning off · chat, Qwen3-14B · reasoning off · chat | 13.5 | 8584b14 | text:132vLLM, self-hosted, BF16 |
| Turkish Reading | 65.3 | 31 of 40tied with Qwen3-14B · reasoning off · chat | 6.7 | 8584b14 | text:132vLLM, self-hosted, BF16 |
| Turkish Grammar | 16.7 | 33 of 40tied with Qwen3-14B · reasoning off · chat | 16.3 | 8584b14 | text:132vLLM, self-hosted, BF16 |
| English Reading | 65.0 | 40 of 47tied with Gemma 4 E2B · default reasoning · chat, Granite-4.2-3B · reasoning on · chat, LFM2.5-2.6B · default reasoning · chat and 3 more | 10.9 | 8584b14 | english:762vLLM, self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 42.8 | 35 of 35 | – | 8584b14 | logic:erk-14b_en, logic:erk-14b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 88.4 | 35 of 36tied with Raon-1 nano · reasoning off · chat | – | 8584b14 | math:erk-14b_en, math:erk-14b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 72.8 | 34 of 34 | – | 8584b14 | logic:erk-14b_en, math:erk-14b_en, logic:erk-14b_en_dontknow, math:erk-14b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Metask-Jev-4B LoRA, merged
- Maker: Metask Lab · training: LoRA, merged
- Kind: decision model · class: Tiny
- Size: 4.5B
- Repo: wayfind/metask-jev-4b-policy-mix
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 43.1EN 71.4 Logic 15.1 Math 23.9 AC 61.7 | 42 of 53 | – | – | – |
| Reading · Turkish (in Combined) | 26.0 | 61 of 76 | 5.8 | ea20fe8 | text:126Metask-Jev server, self-hosted |
| Turkish Reading | 53.5 | 61 of 76 | 5.3 | ea20fe8 | text:126Metask-Jev server, self-hosted |
| Turkish Grammar | 14.2 | 61 of 76tied with KnowLine-2B-Gen1 | 6.5 | ea20fe8 | text:126Metask-Jev server, self-hosted |
| English Reading | 71.4 | 60 of 75tied with Chinese Decision Pointer 4B | 4.4 | ea20fe8 | english:739Metask-Jev server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 15.1 | 59 of 69tied with GLiDE 2B, KnowLine-2B-Gen1, StartLux-Decision-v1.1-2B-Instant and 1 more | 2.5 | ea20fe8 | logic:metask_en, logic:metask_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 23.9 | 58 of 69tied with StartLux-Decision-v1.1-2B-Instant | 3.3 | ea20fe8 | math:metask_en, math:metask_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 20.9 | 58 of 69tied with StartLux-Decision-v1.1-2B-Instant | 2.2 | ea20fe8 | logic:metask_en, math:metask_en, logic:metask_en_dontknow, math:metask_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 61.7 | 45 of 55tied with KnowLine-2B-Gen1, GLiDE 0.8B, Strands Decider 2B | 6.8 | ea20fe8 | grading:site-metask-jev-4b-v1wayfind/metask-jev-4b-policy-mix@ea20fe85, the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
KnowLine-2B-Gen1 LoRA, merged
- Maker: PelaAI · training: LoRA, merged
- Kind: decision model · class: Tiny
- Size: 2.3B
- Repo: PelaAI/KnowLine-2B-Gen1
- Latency ↓: median 62 ms, p95 87 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 1079)
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 31.2TR 21.0 L&M 17.6 BG 55.0 | 28 of 36tied with StartLux-Decision-v1.1-2B-Instant, d1-2B | – | – | – |
| Decision Index4 benches | 39.7EN 64.1 Logic 14.7 Math 19.1 AC 61.1 | 45 of 53tied with StartLux-Decision-v1.1-2B-Instant, GLiDE 2B | – | – | – |
| Reading · Turkish (in Combined) | 21.0 | 62 of 76tied with GLiDE 2B | 8.4 | 53245d4 | text:1079KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16 |
| Turkish Reading | 42.8 | 62 of 76tied with StartLux-Decision-v1.1-2B-Instant, GLiDE 2B | 1.9 | 53245d4 | text:1079KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16 |
| Turkish Grammar | 11.7 | 62 of 76tied with Metask-Jev-4B, GLiDE 2B | 11.4 | 53245d4 | text:1079KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16 |
| English Reading | 64.1 | 61 of 75tied with GLiDE 2B, StartLux-Decision-v1.1-2B-Instant | 5.1 | 53245d4 | english:1057KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 14.7 | 60 of 69tied with GLiDE 2B, Metask-Jev-4B, StartLux-Decision-v1.1-2B-Instant and 1 more | 8.3 | 53245d4 | logic:knowline-2b_en, logic:knowline-2b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 19.1 | 62 of 69tied with GLiDE 2B | 9.4 | 53245d4 | math:knowline-2b_en, math:knowline-2b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 17.6 | 62 of 69tied with GLiDE 2B, d1-2B | 9.0 | 53245d4 | logic:knowline-2b_en, math:knowline-2b_en, logic:knowline-2b_en_dontknow, math:knowline-2b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 55.0 | 32 of 41tied with KnowLine 4B Gen5.5 | – | 53245d4 | backgammon:knowline-2b-gen1League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #40. PelaAI/KnowLine-2B-Gen1 @ 53245d4 on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through the repo's own serve_knowline.sh (SGLang 0.5.21, then its knowline_server.py front end), one request at a time; never on a hosted API. First stage 12 Nov 2026: its 9 hosted and program opponents (#1181–#1189, runner e4ce0e9); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1355–#1362, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1190 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). It never truncates (SGLang without --allow-auto-truncate refuses over its 262,144-token context); nothing refused. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 61.1 | 46 of 55tied with Metask-Jev-4B, GLiDE 0.8B, Strands Decider 2B | 2.7 | 53245d4 | grading:site-knowline-2b-gen1-v1PelaAI/KnowLine-2B-Gen1@53245d47, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
GLiDE 2B
- Maker: Fastino
- Kind: decision model · class: Tiny
- Size: 1.9B
- Repo: fastino/GLiDE-2B
- Latency ↓: median 34 ms, p95 47 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 915)
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 27.5TR 19.1 L&M 19.0 BG 44.4 | 30 of 36 | – | – | – |
| Decision Index4 benches | 41.1EN 62.5 Logic 15.3 Math 20.9 AC 65.6 | 44 of 53tied with StartLux-Decision-v1.1-2B-Instant, KnowLine-2B-Gen1 | – | – | – |
| Reading · Turkish (in Combined) | 19.1 | 63 of 76tied with KnowLine-2B-Gen1, StartLux-Decision-v1.1-2B-Instant | 20.7 | 00f122c | text:915GLiDE server (Fastino's release code), self-hosted |
| Turkish Reading | 39.0 | 64 of 76tied with KnowLine-2B-Gen1, StartLux-Decision-v1.1-2B-Instant | 16.5 | 00f122c | text:915GLiDE server (Fastino's release code), self-hosted |
| Turkish Grammar | 10.7 | 63 of 76tied with KnowLine-2B-Gen1 | 22.5 | 00f122c | text:915GLiDE server (Fastino's release code), self-hosted |
| English Reading | 62.5 | 62 of 75tied with KnowLine-2B-Gen1, StartLux-Decision-v1.1-2B-Instant | 8.2 | 00f122c | english:917GLiDE server (Fastino's release code), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 15.3 | 58 of 69tied with Metask-Jev-4B, KnowLine-2B-Gen1, StartLux-Decision-v1.1-2B-Instant and 1 more | 16.2 | 00f122c | logic:glide-2b_en, logic:glide-2b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 20.9 | 61 of 69tied with d1-2B, KnowLine-2B-Gen1 | 17.2 | 00f122c | math:glide-2b_en, math:glide-2b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 19.0 | 60 of 69tied with StartLux-Decision-v1.1-2B-Instant, d1-2B, KnowLine-2B-Gen1 | 16.8 | 00f122c | logic:glide-2b_en, math:glide-2b_en, logic:glide-2b_en_dontknow, math:glide-2b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 44.4 | 36 of 41tied with StartLux-Decision-v1.1-0.8B-Instant, Open-Jev-27B-v1.1 | – | 00f122c | backgammon:glide-2bLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #41. fastino/GLiDE-2B @ 00f122c on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through the server its repository ships (python -m glide.server, GLIDE_MAX_BATCH=1), one request at a time; never on a hosted API. First stage 12 Nov 2026: its 9 hosted and program opponents (#1191–#1199, runner e4ce0e9); local stage 13 Nov, 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1298–#1305, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1200 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). It never truncates: a prompt over GLIDE_MAX_LEN 40,000 tokens is refused (422); League 1's longest stored request (251 options) is 28,323 tokens; nothing refused. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 65.6 | 44 of 55tied with StartLux-Decision-v1.1-2B-Instant | 4.0 | 00f122c | grading:site-glide-2b-v1fastino/GLiDE-2B@00f122c1, GLiDE server (python -m glide.server) in vLLM v0.30.0 image with flash-linear-attention 0.5.2 on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
StartLux-Decision-v1.1-2B-Instant fine-tune
- Maker: StartLux · training: fine-tune
- Kind: decision model · class: Tiny
- Size: 2.3B
- Repo: startlux-models/StartLux-Decision-v1.1-2B-Instant
- Latency ↓: median 40 ms, p95 48 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 977)
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 32.1TR 17.8 L&M 19.9 BG 58.7 | 27 of 36tied with KnowLine-2B-Gen1 | – | – | – |
| Decision Index4 benches | 41.2EN 62.1 Logic 13.4 Math 23.3 AC 66.0 | 43 of 53tied with GLiDE 2B, KnowLine-2B-Gen1 | – | – | – |
| Reading · Turkish (in Combined) | 17.8 | 64 of 76tied with GLiDE 2B | 6.4 | f1e6cd1 | text:977StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| Turkish Reading | 41.1 | 63 of 76tied with KnowLine-2B-Gen1, GLiDE 2B | 3.3 | f1e6cd1 | text:977StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| Turkish Grammar | 7.9 | 64 of 76tied with StartLux-Decision-v1.1-0.8B-Instant, OneJev1.6-0.8B, d1-2B | 9.4 | f1e6cd1 | text:977StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| English Reading | 62.1 | 63 of 75tied with KnowLine-2B-Gen1, GLiDE 2B | 8.2 | f1e6cd1 | english:973StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16; the kept reading questions in English |
| Vision | 4.3 | 19 of 20tied with StartLux-Decision-v1.1-0.8B-Instant, KnowLine-0.8B-Gen1, d1-2B | 3.6 | f1e6cd1 | vision:98StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16; the page image only (condition B) |
| Logic | 13.4 | 61 of 69tied with GLiDE 2B, Metask-Jev-4B, KnowLine-2B-Gen1 and 2 more | 5.3 | f1e6cd1 | logic:startlux-2b_en, logic:startlux-2b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 23.3 | 59 of 69tied with Metask-Jev-4B, d1-2B | 2.6 | f1e6cd1 | math:startlux-2b_en, math:startlux-2b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 19.9 | 59 of 69tied with Metask-Jev-4B, GLiDE 2B, d1-2B | 3.3 | f1e6cd1 | logic:startlux-2b_en, math:startlux-2b_en, logic:startlux-2b_en_dontknow, math:startlux-2b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 58.7 | 30 of 41tied with Mercury Decide, Cerno 9B v1.2, Winnow-12B and 1 more | – | f1e6cd1 | backgammon:startlux-decision-v1-1-2bLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #23. startlux-models/StartLux-Decision-v1.1-2B-Instant @ f1e6cd1 on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through its package's own server (startlux-decide 1.1.0 serve --launch-vllm, vLLM 0.31.0), one request at a time; never on a hosted API. First stage 11 Nov 2026: its 9 hosted and program opponents (#970–#978, runner 2a27d5d); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1379–#1386, runner 2a27d5d). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #979 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Instant: one read-out per pass, no thinking; one letter per option, 26 per pass; the server knocks longer lists out in groups of at most 25 (the 3 best of each to a final pass); serve's default none_check (a hidden 'None of the above' option). Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 66.0 | 43 of 55tied with GLiDE 2B | 7.2 | f1e6cd1 | grading:site-startlux-decision-v1.1-2b-v1startlux-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd14, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency) |
LFM2.5-2.6B default reasoning · chat
- Maker: Liquid AI
- Kind: chat model · class: Tiny
- Size: 2.7B
- Repo: LiquidAI/LFM2.5-2.6B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | 17.4 | 34 of 40tied with LFM3-1.7B · reasoning on · chat | – | 7141b56 | text:1037our GPU (logic's server, vLLM v0.30.0 BF16, revision LiquidAI/LFM2.5-2.6B@7141b56), one question per request, plain answer line CEVAP: X (no answer schema); sampling repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50; output cap 32,768 |
| Turkish Reading | 31.8 | 34 of 40tied with LFM3-1.7B · reasoning on · chat | – | 7141b56 | text:1037our GPU (logic's server, vLLM v0.30.0 BF16, revision LiquidAI/LFM2.5-2.6B@7141b56), one question per request, plain answer line CEVAP: X (no answer schema); sampling repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50; output cap 32,768 |
| Turkish Grammar | 11.2 | 34 of 40tied with LFM3-1.7B · reasoning on · chat | – | 7141b56 | text:1037our GPU (logic's server, vLLM v0.30.0 BF16, revision LiquidAI/LFM2.5-2.6B@7141b56), one question per request, plain answer line CEVAP: X (no answer schema); sampling repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50; output cap 32,768 |
| English Reading | 67.3 | 39 of 47tied with Gemma 4 E2B · reasoning on · chat, Qwen3.5-2B · reasoning off · chat, Gemma 4 E2B · default reasoning · chat and 4 more | – | 7141b56 | english:1034our GPU (logic's server, vLLM v0.30.0 BF16, revision LiquidAI/LFM2.5-2.6B@7141b56), one question per request, plain answer line CEVAP: X (no answer schema); sampling repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50; output cap 32,768; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 69.2 | 28 of 35 | – | 7141b56 | logic:lfm2.5-2.6b_en, logic:lfm2.5-2.6b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 92.5 | 33 of 36tied with Qwen3.5-2B · reasoning off · chat, GPT-6 Luna · low · chat | – | 7141b56 | math:lfm2.5-2.6b_en, math:lfm2.5-2.6b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 84.5 | 28 of 34tied with Gemma 4 E2B · reasoning on · chat | – | 7141b56 | logic:lfm2.5-2.6b_en, math:lfm2.5-2.6b_en, logic:lfm2.5-2.6b_en_dontknow, math:lfm2.5-2.6b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
LFM3-1.7B reasoning on · chat
- Maker: Liquid AI
- Kind: chat model · class: Tiny
- Size: 1.7B
- Repo: LiquidAI/LFM3-1.7B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 80.3EN 64.7 Logic 81.9 Math 98.7 AC 75.8 | 7 of 11 | – | – | – |
| Reading · Turkish (in Combined) | 15.5 | 35 of 40tied with LFM2.5-2.6B · default reasoning · chat | – | 0dc4268 | text:1089our GPU (logic's server, vLLM nightly 0.34.0rc1.dev302 (commit be47189f; no released vLLM serves LFM3) BF16, revision LiquidAI/LFM3-1.7B@0dc4268), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50, top_p 1.0, thinking on; output cap 32,768 |
| Turkish Reading | 30.3 | 35 of 40tied with LFM2.5-2.6B · default reasoning · chat | – | 0dc4268 | text:1089our GPU (logic's server, vLLM nightly 0.34.0rc1.dev302 (commit be47189f; no released vLLM serves LFM3) BF16, revision LiquidAI/LFM3-1.7B@0dc4268), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50, top_p 1.0, thinking on; output cap 32,768 |
| Turkish Grammar | 9.2 | 35 of 40tied with LFM2.5-2.6B · default reasoning · chat | – | 0dc4268 | text:1089our GPU (logic's server, vLLM nightly 0.34.0rc1.dev302 (commit be47189f; no released vLLM serves LFM3) BF16, revision LiquidAI/LFM3-1.7B@0dc4268), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50, top_p 1.0, thinking on; output cap 32,768 |
| English Reading | 64.7 | 41 of 47tied with Gemma 4 E2B · default reasoning · chat, Granite-4.2-3B · reasoning on · chat, LFM2.5-2.6B · default reasoning · chat and 3 more | – | 0dc4268 | english:1223, english:1302, english:1303our GPU (logic's server, vLLM nightly 0.34.0rc1.dev302 (commit be47189f; no released vLLM serves LFM3) BF16, revision LiquidAI/LFM3-1.7B@0dc4268), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50, top_p 1.0, thinking off; output cap 32,768; the kept reading questions in English; the mean of 3 seed runs |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 81.9 | 26 of 35tied with Gemma 4 12B Unified · default reasoning · chat, LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat | – | 0dc4268 | logic:lfm3-1.7b-vllm-nothink_en, logic:lfm3-1.7b-vllm-nothink_en_dontknowon our NVIDIA DGX Spark (GB10); other setups of this model under its row, not in this entry: LFM3-1.7B (on · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell) |
| Math | 98.7 | 24 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | 0dc4268 | math:lfm3-1.7b-vllm_en, math:lfm3-1.7b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: LFM3-1.7B (off · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell); accuracy is the mean of 3 seed runs (min 0.9884, max 0.9908): seed 0 lfm3-1.7b-vllm_en 2139/2163, seed 1 lfm3-1.7b-vllm-seed1_en 2143/2163, seed 2 lfm3-1.7b-vllm-seed2_en 2138/2163 |
| Logic & Math (in Combined) | 92.7 | 25 of 34tied with Gemma 4 12B Unified · default reasoning · chat, LFM2.5-24B-A2B · default reasoning · chat, Gemma 4 E4B · default reasoning · chat | – | 0dc4268 | logic:lfm3-1.7b-vllm-nothink_en, math:lfm3-1.7b-vllm-nothink_en, logic:lfm3-1.7b-vllm-nothink_en_dontknow, math:lfm3-1.7b-vllm-nothink_en_dontknowon our NVIDIA DGX Spark (GB10); other setups of this model under its row, not in this entry: LFM3-1.7B (on · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 75.8 | 7 of 20tied with Gemma 4 E4B · default reasoning · chat, Gemma 4 12B Unified · default reasoning · chat, Granite-4.2-3B · reasoning on · chat and 2 more | – | 0dc4268 | grading:chat-lfm3-1.7b-seed0-replay-v1, grading:chat-lfm3-1.7b-seed1-r2-v1, grading:chat-lfm3-1.7b-seed2-r2-v1LiquidAI/LFM3-1.7B@0dc4268b, BF16, cu135-nightly-be47189f37f6162991366ff6b91539f6e9d1f432 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.1% (per seed in seed_runs; cut off at the token limit per seed: 2, 1, 1); the three runs are fresh runs (14 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the runs they replace and the server settings recorded for them; seed 0 is the rerun made to diagnose the missing answer lines; the earlier runs stay in history; gpu_hours is the mean of the three runs |
d1-2B
- Maker: Liquid AI
- Kind: decision model · class: Tiny
- Size: 1.9B
- Repo: LiquidAI/d1-2B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 30.5TR 11.7 L&M 18.6 BG 61.1 | 29 of 36tied with KnowLine-2B-Gen1 | – | – | – |
| Decision Index4 benches | 35.3EN 57.0 Logic 13.3 Math 21.3 AC 49.6 | 46 of 53tied with GLiDE 0.8B | – | – | – |
| Reading · Turkish (in Combined) | 11.7 | 65 of 76tied with GLiDE 0.8B, OneJev1.6-0.8B, StartLux-Decision-v1.1-0.8B-Instant | 5.8 | 10e7671 | text:340d1 server (Liquid's model code in Transformers, the model card's call), self-hosted, BF16 |
| Turkish Reading | 26.1 | 66 of 76tied with GLiDE 0.8B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1 | 4.4 | 10e7671 | text:340d1 server (Liquid's model code in Transformers, the model card's call), self-hosted, BF16 |
| Turkish Grammar | 5.6 | 67 of 76tied with StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-0.8B-Instant, OneJev1.6-0.8B and 3 more | 6.7 | 10e7671 | text:340d1 server (Liquid's model code in Transformers, the model card's call), self-hosted, BF16 |
| English Reading | 57.0 | 64 of 75 | 7.9 | 10e7671 | english:708d1 server (Liquid's model code in Transformers, the model card's call), self-hosted, BF16; the kept reading questions in English |
| Vision | 4.2 | 20 of 20tied with StartLux-Decision-v1.1-0.8B-Instant, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-2B-Instant | 1.1 | 10e7671 | vision:60d1 server (Liquid's model code in Transformers, the model card's call), self-hosted, BF16; the page image only (condition B) |
| Logic | 13.3 | 62 of 69tied with GLiDE 2B, Metask-Jev-4B, KnowLine-2B-Gen1 and 2 more | 6.9 | 10e7671 | logic:d1-2b_en, logic:d1-2b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 21.3 | 60 of 69tied with StartLux-Decision-v1.1-2B-Instant, GLiDE 2B | 5.6 | 10e7671 | math:d1-2b_en, math:d1-2b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 18.6 | 61 of 69tied with StartLux-Decision-v1.1-2B-Instant, GLiDE 2B, KnowLine-2B-Gen1 | 5.9 | 10e7671 | logic:d1-2b_en, math:d1-2b_en, logic:d1-2b_en_dontknow, math:d1-2b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | 61.1 | 25 of 41tied with Elephant-1, Mercury Decide, Cerno 9B v1.2 and 2 more | – | 10e7671 | backgammon:liquid-d1-2bLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #52. LiquidAI/d1-2B @ 10e7671 on one RTX PRO 6000 Blackwell of ours (GPU 2), BF16, through logic's d1_server.py: the command and environment of logic's decision/servers.sh d1-2b (torch 2.15.0+cu132, transformers 5.22.1), the card's call model.system_one(state, questions) unchanged, model field LiquidAI/d1-2B; one request at a time; never on a hosted API. First stage 14 Nov 2026, 16:12–16:16 UTC: its 9 hosted and program opponents (#1419–#1427, runner 8ebad40); local stage 14 Nov 2026, 18:44–19:03 UTC: the 8 setups we run ourselves, one at a time on the same card (#1429–#1436, runner 46c984e, which adds only d1-Omni-2B's entry beside 8ebad40's d1-2B client). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1428 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Every option count gets distinct single-token codes (A..IU at 255). Context 128,000 positions; nothing in its code shortens a prompt, and a request it cannot answer is refused (400), never cut: League 1's longest stored request (251 options) is 20,759 tokens with its tokenizer and prompt; nothing refused. Its cost is an estimate from its time on the RTX PRO 6000 (GPU time, at most).; run on our GPUs |
| Answer Check | 49.6 | 50 of 55tied with OneJev1.6-0.8B, StartLux-Decision-v1.1-0.8B-Instant | 7.2 | 10e7671 | grading:site-liquid-d1-2b-v1LiquidAI/d1-2B@10e7671d, d1_server (Liquid's model code) on one RTX PRO 6000 (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
GLiDE 0.8B
- Maker: Fastino
- Kind: decision model · class: Tiny
- Size: 750M
- Repo: fastino/GLiDE-0.8B
- Latency ↓: median 22 ms, p95 31 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 923)
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 36.4 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 19.7TR 11.6 L&M 13.9 BG 33.6 | 33 of 36tied with KnowLine-0.8B-Gen1 | – | – | – |
| Decision Index4 benches | 34.5EN 50.9 Logic 11.4 Math 15.1 AC 60.8 | 47 of 53tied with d1-2B | – | – | – |
| Reading · Turkish (in Combined) | 11.6 | 66 of 76tied with d1-2B, OneJev1.6-0.8B, StartLux-Decision-v1.1-0.8B-Instant | 24.8 | 9eacdee | text:923GLiDE server (Fastino's release code), self-hosted |
| Turkish Reading | 28.0 | 65 of 76tied with d1-2B, OneJev1.6-0.8B | 21.8 | 9eacdee | text:923GLiDE server (Fastino's release code), self-hosted |
| Turkish Grammar | 4.6 | 69 of 76tied with OneJev1.6-0.8B, d1-2B, Decider 1 and 3 more | 26.1 | 9eacdee | text:923GLiDE server (Fastino's release code), self-hosted |
| English Reading | 50.9 | 65 of 75tied with StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B and 2 more | 12.5 | 9eacdee | english:925GLiDE server (Fastino's release code), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 11.4 | 63 of 69tied with StartLux-Decision-v1.1-2B-Instant, d1-2B, OneJev1.6-0.8B | 14.0 | 9eacdee | logic:glide-0.8b_en, logic:glide-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 15.1 | 63 of 69 | 18.8 | 9eacdee | math:glide-0.8b_en, math:glide-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 13.9 | 63 of 69 | 17.1 | 9eacdee | logic:glide-0.8b_en, math:glide-0.8b_en, logic:glide-0.8b_en_dontknow, math:glide-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 33.6 | 38 of 41tied with OneJev1.6-0.8B | – | 9eacdee | backgammon:glide-0-8bLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #42. fastino/GLiDE-0.8B @ 9eacdee on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through the server its repository ships (python -m glide.server, GLIDE_MAX_BATCH=1), one request at a time; never on a hosted API. First stage 12 Nov 2026: its 9 hosted and program opponents (#1201–#1209, runner e4ce0e9); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1315–#1322, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1210 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). It never truncates: a prompt over GLIDE_MAX_LEN 40,000 tokens is refused (422); League 1's longest stored request (251 options) is 28,323 tokens; nothing refused. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 60.8 | 47 of 55tied with Metask-Jev-4B, KnowLine-2B-Gen1, Strands Decider 2B | 3.8 | 9eacdee | grading:site-glide-0.8b-v1fastino/GLiDE-0.8B@9eacdeec, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
OneJev1.6-0.8B full fine-tune
- Maker: OmniJev · training: full fine-tune
- Kind: decision model · class: Tiny
- Size: 850M
- Repo: OmniJev/OneJev1.6-0.8B
- Latency ↓: median 17 ms, p95 23 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 1029)
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 29.6 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 17.7TR 11.4 L&M 9.8 BG 31.9 | 34 of 36 | – | – | – |
| Decision Index4 benches | 29.6EN 49.3 Logic 8.8 Math 10.3 AC 50.0 | 48 of 53tied with StartLux-Decision-v1.1-0.8B-Instant | – | – | – |
| Reading · Turkish (in Combined) | 11.4 | 67 of 76tied with d1-2B, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant and 1 more | 9.3 | bcd88a3 | text:1029Flide server (flide 0.1.11, the card's server for OneJev 1.6), self-hosted, BF16 |
| Turkish Reading | 24.5 | 67 of 76tied with GLiDE 0.8B, d1-2B, KnowLine-0.8B-Gen1 and 1 more | 7.3 | bcd88a3 | text:1029Flide server (flide 0.1.11, the card's server for OneJev 1.6), self-hosted, BF16 |
| Turkish Grammar | 5.8 | 66 of 76tied with StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-0.8B-Instant, d1-2B and 3 more | 10.3 | bcd88a3 | text:1029Flide server (flide 0.1.11, the card's server for OneJev 1.6), self-hosted, BF16 |
| English Reading | 49.3 | 68 of 75tied with GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B and 2 more | 3.7 | bcd88a3 | english:1025Flide server (flide 0.1.11, the card's server for OneJev 1.6), self-hosted, BF16; the kept reading questions in English |
| Vision | 8.1 | 16 of 20tied with StartLux-Decision-v1.1-0.8B-Instant, KnowLine-0.8B-Gen1 | 4.2 | bcd88a3 | vision:122Flide server (flide 0.1.11, the card's server for OneJev 1.6), self-hosted, BF16; the page image only (condition B) |
| Logic | 8.8 | 64 of 69tied with GLiDE 0.8B, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-0.8B-Instant | 11.2 | bcd88a3 | logic:onejev1.6-0.8b_en, logic:onejev1.6-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 10.3 | 66 of 69tied with StartLux-Decision-v1.1-0.8B-Instant, Strands Decider 2B, KnowLine-0.8B-Gen1 | 10.0 | bcd88a3 | math:onejev1.6-0.8b_en, math:onejev1.6-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 9.8 | 65 of 69tied with StartLux-Decision-v1.1-0.8B-Instant, KnowLine-0.8B-Gen1, Strands Decider 2B | 10.4 | bcd88a3 | logic:onejev1.6-0.8b_en, math:onejev1.6-0.8b_en, logic:onejev1.6-0.8b_en_dontknow, math:onejev1.6-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 31.9 | 39 of 41tied with GLiDE 0.8B | – | bcd88a3 | backgammon:onejev1-6-0-8bLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #30. OmniJev/OneJev1.6-0.8B @ bcd88a3 on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through the server its card names (Flide 0.1.11 flide serve, format onejev-slot-v2), one request at a time; never on a hosted API. First stage 11 Nov 2026: its 9 hosted and program opponents (#1041–#1049, runner e4ce0e9); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1347–#1354, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1050 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). No calibration file shipped, so temperature 1; probabilities rounded to 6 places by the server. Never truncates: state plus a question over 32,768 tokens, or a request over 65,536, is refused (422); nothing refused. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 50.0 | 49 of 55tied with d1-2B, StartLux-Decision-v1.1-0.8B-Instant | 4.1 | bcd88a3 | grading:site-onejev1.6-0.8b-v1OmniJev/OneJev1.6-0.8B@bcd88a38, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; server probabilities are rounded to six decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency) |
StartLux-Decision-v1.1-0.8B-Instant fine-tune
- Maker: StartLux · training: fine-tune
- Kind: decision model · class: Tiny
- Size: 870M
- Repo: startlux-models/StartLux-Decision-v1.1-0.8B-Instant
- Latency ↓: median 28 ms, p95 33 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 1015)
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 22.4TR 10.7 L&M 10.0 BG 46.6 | 31 of 36 | – | – | – |
| Decision Index4 benches | 29.1EN 49.5 Logic 5.9 Math 12.1 AC 48.9 | 49 of 53tied with OneJev1.6-0.8B | – | – | – |
| Reading · Turkish (in Combined) | 10.7 | 68 of 76tied with d1-2B, GLiDE 0.8B, OneJev1.6-0.8B and 3 more | 4.4 | 6496ae6 | text:1015StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| Turkish Reading | 18.8 | 70 of 76tied with Strands Decider 2B, Decider 1 | 5.0 | 6496ae6 | text:1015StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| Turkish Grammar | 7.3 | 65 of 76tied with StartLux-Decision-v1.1-2B-Instant, OneJev1.6-0.8B, d1-2B and 1 more | 4.6 | 6496ae6 | text:1015StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16 |
| English Reading | 49.5 | 66 of 75tied with GLiDE 0.8B, GLiDE 0.4B, OneJev1.6-0.8B and 2 more | 8.7 | 6496ae6 | english:1011StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16; the kept reading questions in English |
| Vision | 5.3 | 17 of 20tied with OneJev1.6-0.8B, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-2B-Instant and 1 more | 1.7 | 6496ae6 | vision:110StartLux-Decision server (startlux-decide on vLLM 0.31.0, the maker's release code), self-hosted, BF16; the page image only (condition B) |
| Logic | 5.9 | 66 of 69tied with OneJev1.6-0.8B, KnowLine-0.8B-Gen1, GLiClass v4 Edge and 2 more | 7.8 | 6496ae6 | logic:startlux-0.8b_en, logic:startlux-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 12.1 | 64 of 69tied with Strands Decider 2B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1 | 4.8 | 6496ae6 | math:startlux-0.8b_en, math:startlux-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 10.0 | 64 of 69tied with OneJev1.6-0.8B, KnowLine-0.8B-Gen1 | 5.8 | 6496ae6 | logic:startlux-0.8b_en, math:startlux-0.8b_en, logic:startlux-0.8b_en_dontknow, math:startlux-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 46.6 | 34 of 41tied with Open-Jev-27B-v1.1, GLiDE 2B | – | 6496ae6 | backgammon:startlux-decision-v1-1-0-8b-instantLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #27. startlux-models/StartLux-Decision-v1.1-0.8B-Instant @ 6496ae6 on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through its package's own server (startlux-decide 1.1.0 serve --launch-vllm, vLLM 0.31.0), one request at a time; never on a hosted API. First stage 11 Nov 2026: its 9 hosted and program opponents (#1000–#1014, runner e4ce0e9); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1339–#1346, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1015 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Instant: one read-out per pass, no thinking; 26 letters per pass including the hidden 'None of the above' (serve's default none_check); the server knocks longer lists out in groups of at most 25. Never truncates: vLLM refuses over 131,072 tokens (422); nothing refused. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 48.9 | 51 of 55tied with OneJev1.6-0.8B, d1-2B | 5.6 | 6496ae6 | grading:site-startlux-decision-v1.1-0.8b-instant-v1startlux-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae64, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; licence CC-BY-NC-4.0 (research use); one question per request, 1 request at a time (latency not comparable across different concurrency) |
LFM3-1.7B reasoning off · chat
- Maker: Liquid AI
- Kind: chat model · class: Tiny
- Size: 1.7B
- Repo: LiquidAI/LFM3-1.7B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic and Math | ||||
| Reading · Turkish (in Combined) | 10.4 | 36 of 40 | – | 0dc4268 | text:1227our GPU (logic's server, vLLM nightly 0.34.0rc1.dev302 (commit be47189f; no released vLLM serves LFM3) BF16, revision LiquidAI/LFM3-1.7B@0dc4268), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50, top_p 1.0, thinking off; output cap 32,768 |
| Turkish Reading | 21.5 | 36 of 40tied with Qwen3.5-2B · reasoning off · chat | – | 0dc4268 | text:1227our GPU (logic's server, vLLM nightly 0.34.0rc1.dev302 (commit be47189f; no released vLLM serves LFM3) BF16, revision LiquidAI/LFM3-1.7B@0dc4268), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50, top_p 1.0, thinking off; output cap 32,768 |
| Turkish Grammar | 5.7 | 36 of 40 | – | 0dc4268 | text:1227our GPU (logic's server, vLLM nightly 0.34.0rc1.dev302 (commit be47189f; no released vLLM serves LFM3) BF16, revision LiquidAI/LFM3-1.7B@0dc4268), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50, top_p 1.0, thinking off; output cap 32,768 |
| English Reading | 63.9 | 42 of 47tied with Granite-4.2-3B · reasoning on · chat, LFM2.5-2.6B · default reasoning · chat, Erk-14B · reasoning off · chat and 3 more | – | 0dc4268 | english:1091, english:1115, english:1116our GPU (logic's server, vLLM nightly 0.34.0rc1.dev302 (commit be47189f; no released vLLM serves LFM3) BF16, revision LiquidAI/LFM3-1.7B@0dc4268), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, repetition_penalty 1.1, seed 0, temperature 0.1, top_k 50, top_p 1.0, thinking on; output cap 32,768; the kept reading questions in English; the mean of 3 seed runs |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 74.9 | 9 of 20tied with Gemma 4 12B Unified · default reasoning · chat, Granite-4.2-3B · reasoning on · chat, LFM3-1.7B · reasoning on · chat and 1 more | – | 0dc4268 | grading:chat-lfm3-1.7b-nothink-r2-v1, grading:chat-lfm3-1.7b-nothink-seed1-r2-v1, grading:chat-lfm3-1.7b-nothink-seed2-r2-v1LiquidAI/LFM3-1.7B@0dc4268b, BF16, cu135-nightly-be47189f37f6162991366ff6b91539f6e9d1f432 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.1% (per seed in seed_runs; cut off at the token limit per seed: 2, 2, 1); seed 0 is the published single run (14 Nov, every reply kept); seeds 1 and 2 are new runs (15 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs |
Strands Decider 2B LoRA + head
- Maker: Strands Agents · training: LoRA + head
- Kind: decision model · class: Tiny
- Size: 2.3B
- Repo: StrandsAgents/strands-decider-2B-hobson-v21
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 26.5EN 31.9 Logic 3.0 Math 10.8 AC 60.3 | 50 of 53tied with KnowLine-0.8B-Gen1 | – | – | – |
| Reading · Turkish (in Combined) | 9.3 | 69 of 76tied with OneJev1.6-0.8B, StartLux-Decision-v1.1-0.8B-Instant, Decider 1 and 1 more | 12.2 | 2b52a62 | text:141Strands Decider server, self-hosted |
| Turkish Reading | 21.1 | 69 of 76tied with OneJev1.6-0.8B, KnowLine-0.8B-Gen1, StartLux-Decision-v1.1-0.8B-Instant and 1 more | 10.2 | 2b52a62 | text:141Strands Decider server, self-hosted |
| Turkish Grammar | 4.3 | 70 of 76tied with OneJev1.6-0.8B, d1-2B, Decider 1 and 3 more | 13.2 | 2b52a62 | text:141Strands Decider server, self-hosted |
| English Reading | 31.9 | 72 of 75 | 10.5 | 2b52a62 | english:702Strands Decider server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 3.0 | 69 of 69tied with StartLux-Decision-v1.1-0.8B-Instant, GLiClass v4 Edge, GLiDE 0.4B | 12.0 | 2b52a62 | logic:strands_en, logic:strands_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); the repository holds a LoRA and head on Qwen3.5-2B-Base |
| Math | 10.8 | 65 of 69tied with StartLux-Decision-v1.1-0.8B-Instant, OneJev1.6-0.8B, KnowLine-0.8B-Gen1 | 8.8 | 2b52a62 | math:strands_en, math:strands_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); the repository holds a LoRA and head on Qwen3.5-2B-Base |
| Logic & Math (in Combined) | 8.2 | 67 of 69tied with OneJev1.6-0.8B, KnowLine-0.8B-Gen1 | 9.9 | 2b52a62 | logic:strands_en, math:strands_en, logic:strands_en_dontknow, math:strands_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); the repository holds a LoRA and head on Qwen3.5-2B-Base |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 60.3 | 48 of 55tied with Metask-Jev-4B, KnowLine-2B-Gen1, GLiDE 0.8B | 5.0 | 2b52a62 | grading:site-strands-decider-2b-v1StrandsAgents/strands-decider-2B-hobson-v21@2b52a623 (a LoRA and head on Qwen3.5-2B-Base), the server its Logic & Math runs use, BF16, on one RTX PRO 6000 (ours), self-hosted; server probabilities are rounded to four decimal places and retained without renormalization; ECE uses the reported maximum probability; one question per request, 1 request at a time (latency not comparable across different concurrency) |
KnowLine-0.8B-Gen1 LoRA, merged
- Maker: PelaAI · training: LoRA, merged
- Kind: decision model · class: Tiny
- Size: 870M
- Repo: PelaAI/KnowLine-0.8B-Gen1
- Latency ↓: median 33 ms, p95 50 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 1005)
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 19.7TR 8.9 L&M 9.6 BG 40.6 | 32 of 36tied with GLiDE 0.8B | – | – | – |
| Decision Index4 benches | 26.4EN 48.6 Logic 8.3 Math 10.3 AC 38.5 | 51 of 53tied with Strands Decider 2B, GLiClass v4 Edge | – | – | – |
| Reading · Turkish (in Combined) | 8.9 | 71 of 76tied with StartLux-Decision-v1.1-0.8B-Instant, Strands Decider 2B, Decider 1 | 8.3 | 6b27831 | text:1005KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16 |
| Turkish Reading | 22.8 | 68 of 76tied with d1-2B, OneJev1.6-0.8B, Strands Decider 2B | 5.5 | 6b27831 | text:1005KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16 |
| Turkish Grammar | 3.0 | 71 of 76tied with Decider 1, GLiDE 0.8B, Strands Decider 2B and 2 more | 10.8 | 6b27831 | text:1005KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16 |
| English Reading | 48.6 | 69 of 75tied with GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B and 3 more | 8.1 | 6b27831 | english:1001KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16; the kept reading questions in English |
| Vision | 4.8 | 18 of 20tied with OneJev1.6-0.8B, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant and 1 more | 1.5 | 6b27831 | vision:104KnowLine server (knowline_server.py over SGLang 0.5.21, PelaAI's release code), self-hosted, BF16; the page image only (condition B) |
| Logic | 8.3 | 65 of 69tied with OneJev1.6-0.8B, StartLux-Decision-v1.1-0.8B-Instant | 8.4 | 6b27831 | logic:knowline-0.8b_en, logic:knowline-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 10.3 | 67 of 69tied with StartLux-Decision-v1.1-0.8B-Instant, Strands Decider 2B, OneJev1.6-0.8B | 10.5 | 6b27831 | math:knowline-0.8b_en, math:knowline-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 9.6 | 66 of 69tied with StartLux-Decision-v1.1-0.8B-Instant, OneJev1.6-0.8B, Strands Decider 2B | 9.8 | 6b27831 | logic:knowline-0.8b_en, math:knowline-0.8b_en, logic:knowline-0.8b_en_dontknow, math:knowline-0.8b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 40.6 | 37 of 41 | – | 6b27831 | backgammon:knowline-0-8b-gen1League 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #25. PelaAI/KnowLine-0.8B-Gen1 @ 6b27831 on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through the repo's own serve_knowline.sh (SGLang 0.5.21, then its knowline_server.py front end), one request at a time; never on a hosted API. First stage 11 Nov 2026: its 9 hosted and program opponents (#983–#991, runner e4ce0e9); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1331–#1338, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #992 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). It never truncates (SGLang without --allow-auto-truncate refuses over its 262,144-token context); nothing refused. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 38.5 | 55 of 55 | 14.6 | 6b27831 | grading:site-knowline-0.8b-gen1-v1PelaAI/KnowLine-0.8B-Gen1@6b27831f, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
Decider 1
- Maker: meraGPT
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic and Math | ||||
| Reading · Turkish (in Combined) | 8.9 | 70 of 76tied with StartLux-Decision-v1.1-0.8B-Instant, Strands Decider 2B, KnowLine-0.8B-Gen1 | 14.3 | – | text:243meraGPT System One API |
| Turkish Reading | 18.3 | 71 of 76tied with Strands Decider 2B, StartLux-Decision-v1.1-0.8B-Instant | 11.8 | – | text:243meraGPT System One API |
| Turkish Grammar | 4.9 | 68 of 76tied with StartLux-Decision-v1.1-0.8B-Instant, OneJev1.6-0.8B, d1-2B and 4 more | 15.4 | – | text:243meraGPT System One API |
| English Reading | 44.7 | 71 of 75tied with KnowLine-0.8B-Gen1, GLiClass v4 Edge | 5.3 | – | english:596meraGPT System One API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 43.3 | 54 of 55tied with GLiDE 0.4B, GLiClass v4 Edge | 5.9 | – | grading:site-meragpt-decider-1-v1meraGPT decisions API (meragpt.com, model decider-1); one question per request, 8 requests at a time (latency not comparable across different concurrency) |
Qwen3.5-2B reasoning off · chat
- Maker: Alibaba
- Kind: chat model · class: Tiny
- Size: 2.3B
- Repo: Qwen/Qwen3.5-2B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 72.4EN 69.8 Logic 56.0 Math 93.3 AC 70.3 | 9 of 11 | – | – | – |
| Reading · Turkish (in Combined) | 6.7 | 37 of 40 | – | 15852e8 | text:1218our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-2B@15852e8), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking off; output cap 32,768 |
| Turkish Reading | 17.9 | 37 of 40tied with LFM3-1.7B · reasoning off · chat | – | 15852e8 | text:1218our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-2B@15852e8), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking off; output cap 32,768 |
| Turkish Grammar | 2.0 | 37 of 40 | – | 15852e8 | text:1218our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-2B@15852e8), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking off; output cap 32,768 |
| English Reading | 69.8 | 36 of 47tied with MiniCPM5-2B · reasoning off · chat, Gemma 4 E2B · reasoning on · chat, Gemma 4 E2B · default reasoning · chat and 2 more | – | 15852e8 | english:1341our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-2B@15852e8), one question per request, plain answer line ANSWER: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking on; output cap 32,768; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 56.0 | 32 of 35tied with Qwen3-14B · reasoning off · chat, Raon-1 nano · reasoning off · chat | – | 15852e8 | logic:qwen3.5-2b-nothink_en, logic:qwen3.5-2b-nothink_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Qwen3.5-2B (on · chat) |
| Math | 93.3 | 32 of 36tied with LFM2.5-8B-A1B · default reasoning · chat, Qwen3-14B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat and 1 more | – | 15852e8 | math:qwen3.5-2b-nothink_en, math:qwen3.5-2b-nothink_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Qwen3.5-2B (on · chat) |
| Logic & Math (in Combined) | 80.5 | 30 of 34tied with Qwen3-14B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat | – | 15852e8 | logic:qwen3.5-2b-nothink_en, math:qwen3.5-2b-nothink_en, logic:qwen3.5-2b-nothink_en_dontknow, math:qwen3.5-2b-nothink_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Qwen3.5-2B (on · chat) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 70.3 | 11 of 20tied with Gemma 4 E2B · reasoning on · chat | – | 15852e8 | grading:chat-qwen3.5-2b-r2-v1Qwen/Qwen3.5-2B@15852e8c, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 78 of 2069 replies (4%) had no answer marker and count wrong (5 of them cut off at the 32,768-token limit); a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history; an earlier attempt of this same run stopped part-way (1,210 recorded pairs and 64 further requests possibly answered but not recorded) and is kept apart, not merged; this attempt started again from the first pair, so its outcomes are this attempt's recorded outputs, not guaranteed to be the model's first outputs on those pairs |
GLiClass v4 Edge fine-tune
- Maker: Knowledgator · training: fine-tune
- Kind: decision model · class: Tiny
- Size: 440M
- Repo: knowledgator/gliclass-v4-edge
- Latency ↓: median 25 ms, p95 53 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 1046)
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | −0.8TR 4.2 L&M 3.7 BG −10.2 | 36 of 36 | – | – | – |
| Decision Index4 benches | 25.0EN 47.9 Logic 4.4 Math 3.3 AC 44.3 | 52 of 53tied with KnowLine-0.8B-Gen1, GLiDE 0.4B | – | – | – |
| Reading · Turkish (in Combined) | 4.2 | 72 of 76tied with GLiDE 0.4B | 10.5 | 885af1a | text:1046GLiClass server (our servers/gliclass_server.py: the card's gliclass pipeline and its System One mapping), self-hosted, BF16 |
| Turkish Reading | 7.4 | 72 of 76tied with GLiDE 0.4B | 11.9 | 885af1a | text:1046GLiClass server (our servers/gliclass_server.py: the card's gliclass pipeline and its System One mapping), self-hosted, BF16 |
| Turkish Grammar | 2.8 | 72 of 76tied with Decider 1, GLiDE 0.8B, Strands Decider 2B and 2 more | 10.0 | 885af1a | text:1046GLiClass server (our servers/gliclass_server.py: the card's gliclass pipeline and its System One mapping), self-hosted, BF16 |
| English Reading | 47.9 | 70 of 75tied with GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B and 3 more | 3.8 | 885af1a | english:1042GLiClass server (our servers/gliclass_server.py: the card's gliclass pipeline and its System One mapping), self-hosted, BF16; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 4.4 | 67 of 69tied with StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, Strands Decider 2B | 8.7 | 885af1a | logic:gliclass-v4-edge_en, logic:gliclass-v4-edge_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 3.3 | 68 of 69 | 7.0 | 885af1a | math:gliclass-v4-edge_en, math:gliclass-v4-edge_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 3.7 | 68 of 69 | 7.6 | 885af1a | logic:gliclass-v4-edge_en, math:gliclass-v4-edge_en, logic:gliclass-v4-edge_en_dontknow, math:gliclass-v4-edge_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | −10.2 | 41 of 41 | – | 885af1a | backgammon:gliclass-v4-edgeLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #36. knowledgator/gliclass-v4-edge @ 885af1a on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through turkce's gliclass_server.py with the card's own System One mapping (gliclass 0.1.22, one pass per choice question), one request at a time; never on a hosted API. First stage 12 Nov 2026: its 9 hosted and program opponents (#1141–#1169, runner e4ce0e9); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1363–#1370, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1170 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Options, instructions and state share one 16,384-token input (refused with 422 above it, never cut); League 1's worst case, the longest state plus 218 options at the longest label, is 16,367 tokens, so play lists over 218 options are split into equal groups of at most 218 and a final among the group winners (the league's split rule, as Winnow and Clef; 24 of League 1's 36,995 decisions have more than 218 options). Nothing shortened or refused. Its first stage was interrupted once (12 Nov 08:59 UTC, the server failed 3 health checks in a row) and resumed at 10:15 UTC. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 44.3 | 53 of 55tied with GLiDE 0.4B, Decider 1 | 9.8 | 885af1a | grading:site-gliclass-v4-edge-v1knowledgator/gliclass-v4-edge@885af1af, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
GLiDE 0.4B
- Maker: Fastino
- Kind: decision model · class: Tiny
- Size: 400M
- Repo: fastino/GLiDE-0.4B
- Latency ↓: median 16 ms, p95 24 ms over 2,198 requests on spark2, a DGX Spark (GB10) (Text run 931)
- Learning from experience ↑: not published for it yet
- Jev Decision Index Full score: 33.8 (theirs, a sanity check; not our score)
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | 3.7TR 2.8 L&M 1.8 BG 6.5 | 35 of 36 | – | – | – |
| Decision Index4 benches | 24.6EN 49.3 Logic 4.0 Math 0.7 AC 44.4 | 53 of 53tied with GLiClass v4 Edge | – | – | – |
| Reading · Turkish (in Combined) | 2.8 | 73 of 76tied with GLiClass v4 Edge | 8.0 | 6bda7ab | text:931GLiDE server (Fastino's release code), self-hosted |
| Turkish Reading | 5.7 | 73 of 76tied with GLiClass v4 Edge | 7.2 | 6bda7ab | text:931GLiDE server (Fastino's release code), self-hosted |
| Turkish Grammar | 1.6 | 73 of 76tied with KnowLine-0.8B-Gen1, GLiClass v4 Edge, GLiNER2.5-multi-Decide | 8.3 | 6bda7ab | text:931GLiDE server (Fastino's release code), self-hosted |
| English Reading | 49.3 | 67 of 75tied with GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, OneJev1.6-0.8B and 2 more | 4.8 | 6bda7ab | english:933GLiDE server (Fastino's release code), self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 4.0 | 68 of 69tied with StartLux-Decision-v1.1-0.8B-Instant, GLiClass v4 Edge, Strands Decider 2B | 8.8 | 6bda7ab | logic:glide-0.4b_en, logic:glide-0.4b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Math | 0.7 | 69 of 69 | 12.2 | 6bda7ab | math:glide-0.4b_en, math:glide-0.4b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Logic & Math (in Combined) | 1.8 | 69 of 69 | 11.0 | 6bda7ab | logic:glide-0.4b_en, math:glide-0.4b_en, logic:glide-0.4b_en_dontknow, math:glide-0.4b_en_dontknowon our NVIDIA DGX Spark (GB10) |
| Backgammon | 6.5 | 40 of 41 | – | 6bda7ab | backgammon:glide-0-4bLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #43. fastino/GLiDE-0.4B @ 6bda7ab on our DGX Spark (spark2, the whole GB10, handed over by logic one server at a time), BF16, through the server its repository ships (python -m glide.server, GLIDE_MAX_BATCH=1, GLIDE_MAX_LEN=8006), one request at a time; never on a hosted API. First stage 12 Nov 2026: its 9 hosted and program opponents (#1211–#1219, runner e4ce0e9); local stage 14 Nov 2026: the 8 setups we run ourselves, one at a time on an RTX PRO 6000 Blackwell of ours (#1323–#1330, runner 2a22d1f). It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1220 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). An encoder (ModernBERT) whose model would shorten a prompt over 7,999 tokens instead of refusing it, so it gets at most 74 options per request: every League 1 stored request of at most 74 options fits (at most 7,833 tokens with the card's prompt), and longer play lists are split into equal groups of at most 74 and a final among the group winners (the league's split rule, as Winnow and Clef; 678 of League 1's 36,995 decisions, 1.8%, have more than 74 options). The server runs with GLIDE_MAX_LEN=8006, which refuses (422) any prompt the model would shorten, as a backstop. Nothing shortened or refused. Its cost is an estimate from its time on the DGX Spark (GPU time, at most).; run on our GPUs |
| Answer Check | 44.4 | 52 of 55tied with GLiClass v4 Edge, Decider 1 | 7.7 | 6bda7ab | grading:site-glide-0.4b-v1fastino/GLiDE-0.4B@6bda7ab9, the server its Logic & Math runs use, BF16, on one DGX Spark (ours), self-hosted; one question per request, 1 request at a time (latency not comparable across different concurrency) |
LFM2.5-8B-A1B default reasoning · chat
- Maker: Liquid AI
- Kind: chat model · class: Small
- Size: 8.5B (1.7B active)
- Repo: LiquidAI/LFM2.5-8B-A1B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 66.6EN 63.0 Logic 50.2 Math 95.3 AC 57.7 | 10 of 11 | – | – | – |
| Reading · Turkish (in Combined) | 1.2 | 38 of 40 | – | dd83f5c | text:1288our GPU (logic's server, vLLM v0.30.0 BF16, revision LiquidAI/LFM2.5-8B-A1B@dd83f5c), one question per request, plain answer line CEVAP: X (no answer schema); sampling repetition_penalty 1.05, seed 0, temperature 0.2, top_k 80; output cap 32,768 |
| Turkish Reading | 7.0 | 38 of 40 | – | dd83f5c | text:1288our GPU (logic's server, vLLM v0.30.0 BF16, revision LiquidAI/LFM2.5-8B-A1B@dd83f5c), one question per request, plain answer line CEVAP: X (no answer schema); sampling repetition_penalty 1.05, seed 0, temperature 0.2, top_k 80; output cap 32,768 |
| Turkish Grammar | −1.2 | 38 of 40tied with GLiNER2.5 Multi | – | dd83f5c | text:1288our GPU (logic's server, vLLM v0.30.0 BF16, revision LiquidAI/LFM2.5-8B-A1B@dd83f5c), one question per request, plain answer line CEVAP: X (no answer schema); sampling repetition_penalty 1.05, seed 0, temperature 0.2, top_k 80; output cap 32,768 |
| English Reading | 63.0 | 43 of 47tied with Erk-14B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM3-1.7B · reasoning off · chat and 2 more | – | dd83f5c | english:886our GPU (logic's server, vLLM v0.30.0 BF16, revision LiquidAI/LFM2.5-8B-A1B@dd83f5c), one question per request, plain answer line CEVAP: X (no answer schema); sampling repetition_penalty 1.05, seed 0, temperature 0.2, top_k 80; output cap 32,768; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 50.2 | 33 of 35tied with GPT-6 Luna · low · chat | – | dd83f5c | logic:lfm2.5-8b-a1b_en, logic:lfm2.5-8b-a1b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 95.3 | 30 of 36tied with Gemma 4 E2B · reasoning on · chat, Qwen3-14B · reasoning off · chat, Qwen3.5-2B · reasoning off · chat | – | dd83f5c | math:lfm2.5-8b-a1b_en, math:lfm2.5-8b-a1b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 79.9 | 31 of 34tied with Qwen3.5-2B · reasoning off · chat | – | dd83f5c | logic:lfm2.5-8b-a1b_en, math:lfm2.5-8b-a1b_en, logic:lfm2.5-8b-a1b_en_dontknow, math:lfm2.5-8b-a1b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 57.7 | 16 of 20 | – | dd83f5c | grading:chat-lfm2.5-8b-a1b-think-v1LiquidAI/LFM2.5-8B-A1B@dd83f5cb, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept; an earlier attempt of this same run stopped part-way (45 recorded pairs and an unknown number of further requests possibly answered but not recorded) and is kept apart, not merged; this attempt started again from the first pair, so its outcomes are this attempt's recorded outputs, not guaranteed to be the model's first outputs on those pairs |
GLiNER2.5-multi-Decide fine-tune
- Maker: Fastino · training: fine-tune
- Kind: decision model · class: none
- Size: 290M
- Repo: fastino/GLiNER2.5-multi-Decide
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | 0.3 | 74 of 76tied with Laya | 23.1 | a35a0cd | text:135GLiNER2 server, self-hosted |
| Turkish Reading | 1.3 | 74 of 76tied with Laya | 24.1 | a35a0cd | text:135GLiNER2 server, self-hosted |
| Turkish Grammar | −0.2 | 74 of 76tied with GLiDE 0.4B, Laya, Laya Multilingual | 22.7 | a35a0cd | text:135GLiNER2 server, self-hosted |
| English Reading | 1.5 | 74 of 75tied with Laya | 27.2 | a35a0cd | english:714GLiNER2 server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Laya
- Maker: Convai Innovations
- Kind: decision model · class: none
- Size: 420M
- Repo: convaiinnovations/laya
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | −1.2 | 75 of 76tied with GLiNER2.5-multi-Decide, Laya Multilingual | 13.2 | c5d7873 | text:51Laya server, self-hosted |
| Turkish Reading | 1.1 | 75 of 76tied with GLiNER2.5-multi-Decide | 9.2 | c5d7873 | text:51Laya server, self-hosted |
| Turkish Grammar | −2.2 | 75 of 76tied with GLiNER2.5-multi-Decide, Laya Multilingual | 14.9 | c5d7873 | text:51Laya server, self-hosted |
| English Reading | 3.6 | 73 of 75tied with GLiNER2.5-multi-Decide | 12.1 | c5d7873 | english:726Laya server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Laya Multilingual
- Maker: Convai Innovations
- Kind: decision model · class: none
- Size: 320M
- Repo: convaiinnovations/laya
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | −2.7 | 76 of 76tied with Laya | 27.7 | c5d7873 | text:46Laya server, self-hosted |
| Turkish Reading | −3.7 | 76 of 76 | 33.1 | c5d7873 | text:46Laya server, self-hosted |
| Turkish Grammar | −2.2 | 76 of 76tied with GLiNER2.5-multi-Decide, Laya | 26.0 | c5d7873 | text:46Laya server, self-hosted |
| English Reading | −5.8 | 75 of 75 | 44.4 | c5d7873 | english:732Laya server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
GLiNER2.5 Multi
- Maker: Fastino
- Kind: chat model · class: none
- Size: 290M
- Repo: fastino/gliner2.5-multi-v1
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | −3.2 | 39 of 40 | 35.6 | 2ca71aa | text:122GLiNER2 server, self-hosted |
| Turkish Reading | −3.5 | 39 of 40 | 35.1 | 2ca71aa | text:122GLiNER2 server, self-hosted |
| Turkish Grammar | −3.0 | 39 of 40tied with LFM2.5-8B-A1B · default reasoning · chat | 35.8 | 2ca71aa | text:122GLiNER2 server, self-hosted |
| English Reading | −4.7 | 47 of 47 | 36.4 | 2ca71aa | english:720GLiNER2 server, self-hosted; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
MiniCPM5-2B reasoning off · chat
- Maker: OpenBMB
- Kind: chat model · class: Tiny
- Size: 2.5B
- Repo: openbmb/MiniCPM5-2B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Backgammon | ||||
| Decision Index4 benches | 84.3EN 72.7 Logic 86.1 Math 99.0 AC 79.2 | 4 of 11tied with Qwen3.5-4B · reasoning on · chat, Granite-4.2-3B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat | – | – | – |
| Reading · Turkish (in Combined) | −18.0 | 40 of 40 | – | 42433e9 | text:1299our GPU (logic's server, vLLM v0.30.0 BF16, revision openbmb/MiniCPM5-2B@42433e9), one question per request, plain answer line CEVAP: X (no answer schema); sampling seed 0, temperature 1.0, top_p 0.95, thinking off; output cap 32,768 |
| Turkish Reading | −16.8 | 40 of 40 | – | 42433e9 | text:1299our GPU (logic's server, vLLM v0.30.0 BF16, revision openbmb/MiniCPM5-2B@42433e9), one question per request, plain answer line CEVAP: X (no answer schema); sampling seed 0, temperature 1.0, top_p 0.95, thinking off; output cap 32,768 |
| Turkish Grammar | −18.5 | 40 of 40 | – | 42433e9 | text:1299our GPU (logic's server, vLLM v0.30.0 BF16, revision openbmb/MiniCPM5-2B@42433e9), one question per request, plain answer line CEVAP: X (no answer schema); sampling seed 0, temperature 1.0, top_p 0.95, thinking off; output cap 32,768 |
| English Reading | 72.7 | 34 of 47tied with Qwen3.5-4B · reasoning off · chat, Qwen3-14B · reasoning off · chat, Gemma 4 E2B · reasoning on · chat and 1 more | – | 42433e9 | english:1051, english:1118, english:1119our GPU (logic's server, vLLM v0.30.0 BF16, revision openbmb/MiniCPM5-2B@42433e9), one question per request, plain answer line ANSWER: X (no answer schema); sampling seed 0, temperature 1.0, top_p 0.95, thinking on; output cap 32,768; the kept reading questions in English; the mean of 3 seed runs |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 86.1 | 23 of 35tied with Gemma 4 12B Unified · default reasoning · chat, LFM2.5-24B-A2B · default reasoning · chat | – | 42433e9 | logic:minicpm5-2b_en, logic:minicpm5-2b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: MiniCPM5-2B (off · chat); accuracy is the mean of 3 seed runs (min 0.8827, max 0.8960): seed 0 minicpm5-2b_en 998/1125, seed 1 minicpm5-2b-seed1_en 993/1125, seed 2 minicpm5-2b-seed2_en 1008/1125 |
| Math | 99.0 | 19 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | 42433e9 | math:minicpm5-2b_en, math:minicpm5-2b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: MiniCPM5-2B (off · chat); accuracy is the mean of 3 seed runs (min 0.9917, max 0.9926): seed 0 minicpm5-2b_en 2147/2163, seed 1 minicpm5-2b-seed1_en 2145/2163, seed 2 minicpm5-2b-seed2_en 2147/2163 |
| Logic & Math (in Combined) | 94.6 | 22 of 34tied with Qwen3.5-4B · reasoning on · chat, GLM-5.3-Flash · reasoning on · chat, MiniMax-M3.1-Flash · high · chat and 3 more | – | 42433e9 | logic:minicpm5-2b_en, math:minicpm5-2b_en, logic:minicpm5-2b_en_dontknow, math:minicpm5-2b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: MiniCPM5-2B (off · chat); accuracy is the mean of 3 seed runs (min 0.9544, max 0.9595): seed 0 minicpm5-2b_en+minicpm5-2b_en 3145/3288, seed 1 minicpm5-2b-seed1_en+minicpm5-2b-seed1_en 3138/3288, seed 2 minicpm5-2b-seed2_en+minicpm5-2b-seed2_en 3155/3288 |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 79.2 | 3 of 20tied with Gemma 4 E4B · default reasoning · chat | – | 42433e9 | grading:chat-minicpm5-2b-seed0-r2-v1, grading:chat-minicpm5-2b-seed1-r2-v1, grading:chat-minicpm5-2b-seed2-r2-v1openbmb/MiniCPM5-2B@42433e90, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.1% (per seed in seed_runs; cut off at the token limit per seed: 0, 0, 0); the three runs are fresh runs (14 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the runs they replace and the server settings recorded for them; the earlier runs stay in history; gpu_hours is the mean of the three runs |
Gemma 4 12B Unified default reasoning · chat
- Maker: Google
- Kind: chat model · class: Small
- Size: 12B
- Repo: google/gemma-4-12B-it
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish and Backgammon | ||||
| Decision Index4 benches | 85.9EN 83.5 Logic 84.7 Math 98.6 AC 77.0 | 2 of 11tied with Qwen3.5-4B · reasoning on · chat | – | – | – |
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | 83.5 | 26 of 47tied with GLM-5.3-Flash · reasoning on · chat, Muse Ember · max · chat, MiMo-V2.6-Flash · high · chat and 8 more | – | 43c9d01 | english:895our GPU (logic's server, vLLM v0.30.0 BF16, revision google/gemma-4-12B-it@43c9d01), one question per request, plain answer line ANSWER: X (no answer schema); sampling temperature 0.0; output cap 8,192; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 84.7 | 24 of 35tied with MiniCPM5-2B · reasoning off · chat, LFM2.5-24B-A2B · default reasoning · chat, LFM3-1.7B · reasoning on · chat | – | 43c9d01 | logic:gemma-4-12b-it_en, logic:gemma-4-12b-it_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 98.6 | 25 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | 43c9d01 | math:gemma-4-12b-it_en, math:gemma-4-12b-it_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 93.8 | 23 of 34tied with MiniCPM5-2B · reasoning off · chat, LFM2.5-24B-A2B · default reasoning · chat, LFM3-1.7B · reasoning on · chat | – | 43c9d01 | logic:gemma-4-12b-it_en, math:gemma-4-12b-it_en, logic:gemma-4-12b-it_en_dontknow, math:gemma-4-12b-it_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 77.0 | 5 of 20tied with Gemma 4 E4B · default reasoning · chat, Granite-4.2-3B · reasoning on · chat, LFM3-1.7B · reasoning on · chat and 2 more | – | 43c9d01 | grading:chat-gemma-4-12b-r2-v1google/gemma-4-12B-it@43c9d014, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking off; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history |
celeris-1.1-decision
- Maker: Celeris
- Kind: decision model · class: Efficient
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | 79.7 | 51 of 75tied with StartLux-Decision-v1.1-9B-Instant, d1, Clef and 30 more | 5.0 | – | english:882Celeris System One API; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 34.1 | 30 of 69tied with d1.2, Jeffan-9B-1211, Jev 1.14 and 10 more | 9.8 | – | logic:celeris-1.1_en, logic:celeris-1.1_en_dontknowhosted API (the maker's service) |
| Math | 62.3 | 20 of 69tied with d3-ultra, Gemini Decision 1 · 20261111, GLiDE · API and 10 more | 4.5 | – | math:celeris-1.1_en, math:celeris-1.1_en_dontknowhosted API (the maker's service) |
| Logic & Math (in Combined) | 52.7 | 25 of 69tied with Turion Jev 31B, SubQ Snap 1.1, celeris-1-decision and 1 more | 6.0 | – | logic:celeris-1.1_en, math:celeris-1.1_en, logic:celeris-1.1_en_dontknow, math:celeris-1.1_en_dontknowhosted API (the maker's service) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
LFM2.5-24B-A2B default reasoning · chat
- Maker: Liquid AI
- Kind: chat model · class: Medium
- Size: 24B (2.5B active)
- Repo: LiquidAI/LFM2.5-24B-A2B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Answer Check | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | 78.3 | 30 of 47tied with MiniMax-M3.1-Flash · high · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat and 1 more | – | 1769516 | english:1069our GPU (logic's server, vLLM v0.30.0 BF16, revision LiquidAI/LFM2.5-24B-A2B@1769516), one question per request, plain answer line ANSWER: X (no answer schema); sampling repetition_penalty 1.05, seed 0, temperature 0.1, top_k 50; output cap 32,768; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 83.3 | 25 of 35tied with MiniCPM5-2B · reasoning off · chat, Gemma 4 12B Unified · default reasoning · chat, LFM3-1.7B · reasoning on · chat | – | 1769516 | logic:lfm2.5-24b-a2b_en, logic:lfm2.5-24b-a2b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 98.8 | 23 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | 1769516 | math:lfm2.5-24b-a2b_en, math:lfm2.5-24b-a2b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 93.5 | 24 of 34tied with MiniCPM5-2B · reasoning off · chat, Gemma 4 12B Unified · default reasoning · chat, LFM3-1.7B · reasoning on · chat | – | 1769516 | logic:lfm2.5-24b-a2b_en, math:lfm2.5-24b-a2b_en, logic:lfm2.5-24b-a2b_en_dontknow, math:lfm2.5-24b-a2b_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | 60.6 | 16 of 16 | – | – | backgammon:lfm2-5-24b-a2b-onLeague 1; calibration definition pending Ümit; run on our GPUs |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Gemma 4 E2B reasoning on · chat
- Maker: Google
- Kind: chat model · class: Tiny
- Size: 5.1B
- Repo: google/gemma-4-E2B-it
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish and Backgammon | ||||
| Decision Index4 benches | 75.9EN 70.3 Logic 66.1 Math 96.0 AC 71.1 | 8 of 11 | – | – | – |
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | 70.3 | 35 of 47tied with MiniCPM5-2B · reasoning off · chat, Qwen3.5-2B · reasoning off · chat, Gemma 4 E2B · default reasoning · chat and 2 more | – | 4adf157 | english:1349our GPU (logic's server, vLLM v0.30.0 BF16, revision google/gemma-4-E2B-it@4adf157), one question per request, plain answer line ANSWER: X (no answer schema); sampling temperature 0.0, thinking on; output cap 32,768; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 66.1 | 29 of 35 | – | 4adf157 | logic:gemma-4-e2b-it_en, logic:gemma-4-e2b-it_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Gemma 4 E2B (on · chat), Gemma 4 E2B (on · chat · 32k cap), Gemma 4 E2B (on · chat · 32k cap · sampled); 2.3B effective parameters |
| Math | 96.0 | 29 of 36tied with MiniMax-M3.1-Flash · high · chat, Gemma 4 E4B · default reasoning · chat, LFM2.5-8B-A1B · default reasoning · chat and 1 more | – | 4adf157 | math:gemma-4-e2b-it_en, math:gemma-4-e2b-it_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Gemma 4 E2B (on · chat), Gemma 4 E2B (on · chat · 32k cap), Gemma 4 E2B (on · chat · 32k cap · sampled); 2.3B effective parameters |
| Logic & Math (in Combined) | 85.7 | 27 of 34tied with LFM2.5-2.6B · default reasoning · chat | – | 4adf157 | logic:gemma-4-e2b-it_en, math:gemma-4-e2b-it_en, logic:gemma-4-e2b-it_en_dontknow, math:gemma-4-e2b-it_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Gemma 4 E2B (on · chat), Gemma 4 E2B (on · chat · 32k cap), Gemma 4 E2B (on · chat · 32k cap · sampled); 2.3B effective parameters |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 71.1 | 10 of 20tied with Qwen3.5-2B · reasoning off · chat | – | 4adf157 | grading:chat-gemma-4-e2b-think-r2-v1google/gemma-4-E2B-it@4adf1574, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history |
Gemma 4 E2B default reasoning · chat
- Maker: Google
- Kind: chat model · class: Tiny
- Size: 5.1B
- Repo: google/gemma-4-E2B-it
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish, Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic and Math | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | 68.2 | 37 of 47tied with Gemma 4 E2B · reasoning on · chat, Qwen3.5-2B · reasoning off · chat, Granite-4.2-3B · reasoning on · chat and 3 more | – | 4adf157 | english:892our GPU (logic's server, vLLM v0.30.0 BF16, revision google/gemma-4-E2B-it@4adf157), one question per request, plain answer line ANSWER: X (no answer schema); sampling temperature 0.0; output cap 8,192; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 55.2 | 17 of 20tied with Gemma 4 E2B · thinking off · chat | – | 4adf157 | grading:chat-gemma-4-e2b-sampled-seed0-v1, grading:chat-gemma-4-e2b-sampled-seed1-v1, grading:chat-gemma-4-e2b-sampled-seed2-v1google/gemma-4-E2B-it@4adf1574, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.03% (per seed in seed_runs; cut off at the token limit per seed: 0, 0, 0); three new runs (16 Nov, every reply kept), seeds 0, 1, 2 at Gemma's own sampling (its generation_config.json: temperature 1.0, top_p 0.95, top_k 64; Ümit #10554), with the weights, template and token limit of the temperature-0 row, which stays as its own row; their server settings recorded; gpu_hours is the mean of the three runs |
Granite-4.2-3B reasoning on · chat
- Maker: IBM
- Kind: chat model · class: Tiny
- Size: 3.7B
- Repo: ibm-granite/granite-4.2-3b
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish and Backgammon | ||||
| Decision Index4 benches | 83.1EN 68.0 Logic 89.6 Math 99.0 AC 75.9 | 5 of 11tied with MiniCPM5-2B · reasoning off · chat, Gemma 4 E4B · default reasoning · chat | – | – | – |
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | 68.0 | 38 of 47tied with Gemma 4 E2B · reasoning on · chat, Qwen3.5-2B · reasoning off · chat, Gemma 4 E2B · default reasoning · chat and 4 more | – | 4e9c275 | english:1104our GPU (logic's server, vLLM v0.30.0 BF16, revision ibm-granite/granite-4.2-3b@4e9c275), one question per request, plain answer line ANSWER: X (no answer schema); sampling seed 0, temperature 1.0, top_p 0.95, thinking on; output cap 65,536; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 89.6 | 22 of 35tied with GPT-6.1 Terra · high · chat, MiniMax-M3.1-Flash · high · chat, Qwen3.8 Flash · no reasoning · chat and 3 more | – | 4e9c275 | logic:granite-4.2-3b-64k_en, logic:granite-4.2-3b-64k_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Math | 99.0 | 21 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | 4e9c275 | math:granite-4.2-3b-64k_en, math:granite-4.2-3b-64k_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Logic & Math (in Combined) | 95.8 | 21 of 34tied with MiMo-V2.6-Flash · high · chat, Qwen3.8 Flash · no reasoning · chat, Qwen3.8-27B · low · chat and 6 more | – | 4e9c275 | logic:granite-4.2-3b-64k_en, math:granite-4.2-3b-64k_en, logic:granite-4.2-3b-64k_en_dontknow, math:granite-4.2-3b-64k_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB) |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 75.9 | 6 of 20tied with Gemma 4 E4B · default reasoning · chat, Gemma 4 12B Unified · default reasoning · chat, LFM3-1.7B · reasoning on · chat and 2 more | – | 4e9c275 | grading:chat-granite-4.2-3b-think-v1ibm-granite/granite-4.2-3b@4e9c275e, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept |
MiniCPM5-2B reasoning off · chat
- Maker: OpenBMB
- Kind: chat model · class: Tiny
- Size: 2.5B
- Repo: openbmb/MiniCPM5-2B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish, Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic and Math | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | 59.7 | 44 of 47tied with LFM3-1.7B · reasoning off · chat, LFM2.5-8B-A1B · default reasoning · chat, Qwen3.5-2B · reasoning off · chat | – | 42433e9 | english:1094, english:1300, english:1301our GPU (logic's server, vLLM v0.30.0 BF16, revision openbmb/MiniCPM5-2B@42433e9), one question per request, plain answer line CEVAP: X (no answer schema); sampling seed 0, temperature 1.0, top_p 0.95, thinking off; output cap 32,768; the kept reading questions in English; the mean of 3 seed runs |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 67.0 | 13 of 20tied with Gemma 4 12B Unified · thinking on · chat, Qwen3.5-4B · reasoning on · chat | – | 42433e9 | grading:chat-minicpm5-2b-nothink-r2-v1, grading:chat-minicpm5-2b-nothink-seed1-r2-v1, grading:chat-minicpm5-2b-nothink-seed2-r2-v1openbmb/MiniCPM5-2B@42433e90, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.2% (per seed in seed_runs; cut off at the token limit per seed: 1, 0, 0); seed 0 is the published single run (14 Nov, every reply kept); seeds 1 and 2 are new runs (15 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs (GPU seconds per seed: 219, 76, 76, at the same recorded settings; the cause of the difference is not known) |
Qwen3.5-2B reasoning off · chat
- Maker: Alibaba
- Kind: chat model · class: Tiny
- Size: 2.3B
- Repo: Qwen/Qwen3.5-2B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish, Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on Logic and Math | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | 59.1 | 45 of 47tied with LFM2.5-8B-A1B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat | – | 15852e8 | english:1215our GPU (logic's server, vLLM v0.30.0 BF16, revision Qwen/Qwen3.5-2B@15852e8), one question per request, plain answer line CEVAP: X (no answer schema); sampling min_p 0.0, presence_penalty 1.5, repetition_penalty 1.0, seed 0, temperature 1.0, top_k 20, top_p 0.95, thinking off; output cap 32,768; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 50.2 | 19 of 20 | – | 15852e8 | grading:chat-qwen3.5-2b-nothink-v1, grading:chat-qwen3.5-2b-nothink-seed1-v1, grading:chat-qwen3.5-2b-nothink-seed2-v1Qwen/Qwen3.5-2B@15852e8c, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 2.0% (per seed in seed_runs; cut off at the token limit per seed: 1, 0, 0); seed 0 is the published single run (every reply kept); seeds 1 and 2 are new runs (16 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs |
Raon-1 nano reasoning off · chat
- Maker: KRAFTON
- Kind: chat model · class: Tiny
- Size: 3.5B
- Repo: KRAFTON/Raon-1-nano
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish and Backgammon | ||||
| Decision Index4 benches | 56.5EN 42.2 Logic 56.3 Math 86.9 AC 40.6 | 11 of 11 | – | – | – |
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | 42.2 | 46 of 47 | – | fcf2d1d | english:1152our GPU (logic's server, KRAFTON's vLLM fork (krafton-ai/vllm@13a5dd8; not a vLLM release) BF16, revision KRAFTON/Raon-1-nano@fcf2d1d), one question per request, plain answer line ANSWER: X (no answer schema); sampling presence_penalty 1.5, seed 0, temperature 0.1, thinking off; output cap 32,768; the kept reading questions in English |
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 56.3 | 31 of 35tied with Qwen3-14B · reasoning off · chat, Qwen3.5-2B · reasoning off · chat | – | fcf2d1d | logic:raon-1-nano-nothink_en, logic:raon-1-nano-nothink_en_dontknowon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Raon-1 nano (on · chat); 1.5B effective parameters |
| Math | 86.9 | 36 of 36tied with Erk-14B · reasoning off · chat | – | fcf2d1d | math:raon-1-nano-nothink_enon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Raon-1 nano (on · chat); 1.5B effective parameters; no complete 'I don't know' run |
| Logic & Math (in Combined) | 76.5 | 33 of 34tied with GPT-6 Luna · low · chat | – | fcf2d1d | logic:raon-1-nano-nothink_en, math:raon-1-nano-nothink_enon our NVIDIA RTX PRO 6000 Blackwell (96 GB); other setups of this model under its row, not in this entry: Raon-1 nano (on · chat); 1.5B effective parameters; no complete 'I don't know' run |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 40.6 | 20 of 20 | – | fcf2d1d | grading:chat-raon-1-nano-nothink-v1, grading:chat-raon-1-nano-nothink-seed1-v1, grading:chat-raon-1-nano-nothink-seed2-v1KRAFTON/Raon-1-nano@fcf2d1d5, BF16, KRAFTON vLLM fork krafton-ai/vllm@13a5dd87 (not a vLLM release) on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 0.03% (per seed in seed_runs; cut off at the token limit per seed: 0, 1, 0); seed 0 is the published single run (every reply kept); seeds 1 and 2 are new runs (16 Nov, every reply kept) with the same model, sampling settings, requests at a time and server launcher, only the seed changed, their server settings recorded; gpu_hours is the mean of the three runs |
Inkling 2 Small high · chat
- Maker: Thinking Machines
- Kind: chat model · class: Large
- Size: 594B (17B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on English Reading | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | 94.6 | 8 of 35tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 15 more | – | – | logic:inkling-2-small_en, logic:inkling-2-small_en_dontknowhosted API via OpenRouter |
| Math | 99.4 | 13 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:inkling-2-small_en, math:inkling-2-small_en_dontknowhosted API via OpenRouter |
| Logic & Math (in Combined) | 97.8 | 9 of 34tied with GPT-6 Astra · high · chat, Gemini 4 Carbon · high · chat, GPT-6.1 Sol · high · chat and 15 more | – | – | logic:inkling-2-small_en, math:inkling-2-small_en, logic:inkling-2-small_en_dontknow, math:inkling-2-small_en_dontknowhosted API via OpenRouter |
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 86.2 | 1 of 20 | – | – | grading:chat-inkling-2-small-v1OpenRouter thinkingmachines/inkling-2-small (paid per token; providers with no data retention and no training on inputs), reasoning at the route's default; billed per token at OpenRouter's list price; one question per request, 1 request at a time (latency not comparable across different concurrency); list prices $0.30 per 1M input and $1.00 per 1M output tokens (OpenRouter, 16 Nov); answered by OpenRouter's providers Together 1,104, Cloudflare 317, PrimeIntellect 242, Nebius 201, DeepInfra 179, Modal 26; OpenRouter's reported charges for the 2,069 billed requests of this model: $2.8921 (actual billing, distinct from the list-price figure); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; every reply kept |
DeepSeek-V4-Flash chat
- Maker: DeepSeek
- Kind: chat model · class: Large
- Size: 285B (13B active)
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish, Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on English Reading, Logic and Answer Check | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | 99.1 | 18 of 36tied with Claude Opus 5.5 · high · chat, Qwen4-Max · high · chat, Claude Fable 5.1 · high · chat and 24 more | – | – | math:deepseek-v4-flash_en_t0on our NVIDIA DGX Spark (GB10); no complete 'I don't know' run |
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
DeepSeek-V4.1-Flash low · API + NVIDIA RTX PRO 6000 Blackwell · first release
- Maker: DeepSeek
- Kind: chat model · class: Undisclosed
- Size: not disclosed
- Repo: none (an API setup, or no released checkpoint)
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: Backgammon 8.2 points
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish and Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on English Reading, Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 83.9 | 9 of 16tied with Claude Sonnet 5.5 · high · chat, Grok 4.7 · high · chat, Gemma 4 31B · reasoning off · chat and 1 more | – | – | backgammon:deepseek-v4-1-flash-onLeague 2; calibration definition pending Ümit; hosted (API or subscription); improvement_points: change in risks found, percentage points of risk-pays decisions (with lessons minus no memory); improvement_points from Run 2 after QC; original run +7.21, superseded |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
d1-Omni-2B NVIDIA RTX PRO 6000 Blackwell
- Maker: Liquid AI
- Kind: decision model · class: Tiny
- Size: 2.2B
- Repo: LiquidAI/d1-Omni-2B
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong decision models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish and Logic & Math | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on English Reading, Logic, Math and Answer Check | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | 58.9 | 29 of 41tied with d1-2B, Mercury Decide, Cerno 9B v1.2 and 2 more | – | 81222da | backgammon:liquid-d1-omni-2bLeague 1; calibration definition pending Ümit; late entry: its own decisions in late-entry matches against League 1's field (different positions from the league's); scored against League 1's Random; played 17 of 19 matches; GPT-6 Luna (Decisions API): outside the requested scope (Ümit, 12 Nov); not played (opponent's API out of credit); Mercury Decide: outside the requested scope (Ümit, 12 Nov); not played (opponent no longer served on OpenRouter); Late entry to the original League 1 table (not in standings, not eligible for the Hype Cup), late-entry league #53. LiquidAI/d1-Omni-2B @ 81222da, d1-2B with audio added (the card: text, pictures and audio in), on one RTX PRO 6000 Blackwell of ours (GPU 2, the card d1-2B used, after it), BF16, through logic's d1_server.py: the command of logic's decision/servers.sh d1-omni-2b in d1-2B's environment (torch 2.15.0+cu132, transformers 5.22.1), the card's call model.system_one(state, questions) unchanged, model field LiquidAI/d1-Omni-2B; text only: backgammon positions are text, so its audio and picture inputs are unused; one request at a time; never on a hosted API. Its request and limits are d1-2B's: the model's request code is byte-identical to d1-2B's and its tokenizer only adds two audio tokens, so text prompts tokenize the same. First stage 14 Nov 2026, 19:47–19:50 UTC: its 9 hosted and program opponents (#1437–#1445); local stage 14 Nov 2026, 19:56–20:15 UTC: the 8 setups we run ourselves, one at a time on the same card (#1447–#1454); runner 46c984e throughout. It played 17 of its 19 opponents: its pair with Mercury Decide was not played (Mercury Decide is no longer served on OpenRouter; match #1446 aborted at its warm-up; outside the requested scope, Ümit, 12 Nov) and its pair with GPT-6 Luna was not played (GPT-6 Luna's API was out of credit; outside the requested scope, Ümit, 12 Nov). Every option count gets distinct single-token codes (A..IU at 255). Context 128,000 positions; nothing in its code shortens a prompt, and a request it cannot answer is refused (400), never cut: League 1's longest stored request (251 options) is 20,759 tokens; nothing refused. Its cost is an estimate from its time on the RTX PRO 6000 (GPU time, at most).; run on our GPUs |
| Answer Check | Not runNo Answer Check run for this model: it is not in that bench's file | ||||
Gemma 4 E4B thinking on · sampled (temperature 1.0) · chat
- Maker: Google
- Kind: chat model · class: Small
- Size: 8B
- Repo: google/gemma-4-E4B-it
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish, Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on English Reading, Logic and Math | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 75.1 | 8 of 20tied with Gemma 4 E4B · default reasoning · chat, Gemma 4 12B Unified · default reasoning · chat, Granite-4.2-3B · reasoning on · chat and 2 more | – | 4ecc2ba | grading:chat-gemma-4-e4b-think-sampled-seed0-v1, grading:chat-gemma-4-e4b-think-sampled-seed1-v1, grading:chat-gemma-4-e4b-think-sampled-seed2-v1google/gemma-4-E4B-it@4ecc2ba6, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 30 requests at a time from the client (latency not comparable across different concurrency); mean of 3 full runs, seeds 0, 1, 2 (min-max across seeds, not a confidence interval); answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; replies without an answer marker count wrong: mean rate 1.5% (per seed in seed_runs; cut off at the token limit per seed: 0, 0, 0); three new runs (16 Nov, every reply kept), seeds 0, 1, 2 at Gemma's own sampling (its generation_config.json: temperature 1.0, top_p 0.95, top_k 64; Ümit #10554), with the weights, template and token limit of the temperature-0 row, which stays as its own row; their server settings recorded; gpu_hours is the mean of the three runs |
Gemma 4 12B Unified thinking on · chat
- Maker: Google
- Kind: chat model · class: Small
- Size: 11.9B
- Repo: google/gemma-4-12B-it
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish, Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on English Reading, Logic and Math | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 67.7 | 12 of 20tied with MiniCPM5-2B · reasoning off · chat, Qwen3.5-4B · reasoning on · chat | – | 43c9d01 | grading:chat-gemma-4-12b-think-r2-v1google/gemma-4-12B-it@43c9d014, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking on; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 296 of 2069 replies (14%) had no answer marker and count wrong (295 of them cut off at the 32,768-token limit); a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history |
Gemma 4 E2B thinking off · chat
- Maker: Google
- Kind: chat model · class: Tiny
- Size: 5.1B
- Repo: google/gemma-4-E2B-it
- Latency ↓: not timed on the pinned machine yet
- Learning from experience ↑: not published for it yet
| Bench | Score ↑0–100, chance = 0 | Rankamong chat models | Calibration error ↓ECE, points | Commit | Runs |
|---|---|---|---|---|---|
| Combined3 benchmark groups | Not applicableCombined needs all 3 benchmark groups; not run on Reading · Turkish, Logic & Math and Backgammon | ||||
| Decision Index4 benches | Not applicableThe Decision Index needs all four of English Reading, Logic, Math and Answer Check; not run on English Reading, Logic and Math | ||||
| Reading · Turkish (in Combined) | Not runNo Reading · Turkish run for this model: it is not in that bench's file | ||||
| Turkish Reading | Not runNo Turkish Reading run for this model: it is not in that bench's file | ||||
| Turkish Grammar | Not runNo Turkish Grammar run for this model: it is not in that bench's file | ||||
| English Reading | Not runNo English Reading run for this model: it is not in that bench's file | ||||
| Vision | Not runNo Vision run for this model: it is not in that bench's file | ||||
| Logic | Not runNo Logic run for this model: it is not in that bench's file | ||||
| Math | Not runNo Math run for this model: it is not in that bench's file | ||||
| Logic & Math (in Combined) | Not runNo Logic & Math run for this model: it is not in that bench's file | ||||
| Backgammon | Not runNo Backgammon run for this model: it is not in that bench's file | ||||
| Answer Check | 54.8 | 18 of 20tied with Gemma 4 E2B · default reasoning · chat | – | 4adf157 | grading:chat-gemma-4-e2b-r2-v1google/gemma-4-E2B-it@4adf1574, BF16, v0.30.0 vLLM on one RTX PRO 6000 (ours), self-hosted; one question per request, 64 requests at a time (latency not comparable across different concurrency); thinking off; answer rule v2: the label is the last marker 'ANSWER: X' anywhere in the visible reply, where the word ANSWER may be in any letter case and X must be an uppercase A, B or C; the reasoning text is never read; v2 replaced rule v1 (the word in uppercase only) on 15 Nov after v1 was found to miss replies written 'Answer: X', and applies to every chat row; 0 of 2069 replies (0%) had no answer marker and count wrong; a fresh run (14-15 Nov) with every reply kept, with the same model, sampling settings and requests at a time as the run it replaces and a server from the same launcher (the earlier run's own server flags were not recorded); the earlier run stays in history |