Highlights
Highlights
Gemma 4 E4B, a small chat model, beats every decision model: 83.2% on Logic and 98.1% on Math, against at most 60.7% and 81.6%
Gemma 4 E4B, 8B dense · 4.5B effective, is ahead of each decision model on both tests by the paired test; the decision models' highest scores are those of Karpo Decision 31B (31B dense). Chat models reach 96.7% on Logic and 99.8% on Math, and the smaller Gemma 4 E2B, 5.1B dense · 2.3B effective, is also ahead of every decision model on both.
GLiDE 27B and REAX 1.3: tied on Logic accuracy, non-hallucination 44.3% against 5.3%
On Logic they score 58.4% and 59.3%: 114 questions only GLiDE 27B got right and 124 only REAX 1.3, which the paired test does not tell apart (p = 0.56). Given the option “I don't know”, GLiDE 27B chose it for 44.3% of its answers that were not right and REAX 1.3 for 5.3%; on Math REAX 1.3 is ahead, 78.0% to 72.0%.
The open GLiDE models lose accuracy as they get smaller: on Logic 58.4% for GLiDE 27B, 23.2% for GLiDE 0.4B
Logic accuracy by size: 27B 58.4%, 9B 47.8%, 2B 32.3%, 0.8B 29.2%, 0.4B 23.2% (the paired test does not separate 2B and 0.8B or 0.8B and 0.4B). GLiDE 0.4B is just above chance (20% with five options; its 95% interval 20.8%–25.8%), and given the option “I don't know” it chose it on 785 of 1,125 questions (69.8%).
Text and picture against text alone on the 358 figure questions: C − A +5.1 points
C − A is the mean over 7 image models of each model's accuracy given the text and a picture of the question cut from the Turkish book page (C) minus its accuracy given the text alone (A); the paired test (Holm-corrected) separates C from A for 3 of them. The picture without the text (B): B − A +0.7 points, mean over 7 image models.
On Logic, chat models score 93.6% at 17,920 ms per question, decision models 46.0% at 123 ms (medians)
Medians over the Logic leaderboard's 36 chat models and 69 decision models, each at its own median latency: the chat models' median is 145 times the decision models'. The median cost per 1,000 questions is $1.820 for chat models and $0.026 for decision models (over the 29 and 69 with a recorded cost).
Each card is computed from the data at this build *
A card is left out when its finding no longer holds. “Ahead”, “beats” and “tied” follow the leaderboard's paired test (exact McNemar, Holm-corrected; p < 0.05 separates two setups).