Highlights
KnowLine-31B-Gen1.5 gets the most English reading questions right of the decision models
KnowLine-31B-Gen1.5 answered 512 of the 547 questions right (93.6%).
Against each of the 74 other decision models in the paired test (Holm-adjusted): 11 are too close to call (Copernicus-31B, REAX 1.3, KnowLine-31B-Gen2, Karpo Decision 31B, Gemini Decision 1 20261111, Logits Sigma-1 Duo, GLiDE 27B, d1-ultra, Perplexity Decider 27B v1.2 API · with web, GLiDE v1.2, d1.2-ultra); the other 63 got fewer right.
In English, the best chat models get more questions right than any decision model
Right answers of the 547 questions: the best chat model, Claude Fable 5.2 max · chat, 536 (98.0%); the best decision model, KnowLine-31B-Gen1.5, 512 (93.6%). 20 of the 66 chat model setups got more right than KnowLine-31B-Gen1.5.
No test on the site compares a chat model with a decision model, so these gaps are counted, not tested.
With “I don't know” offered in English, d1.2-ultra holds back a wrong answer most often
With an “I don't know” option in English, it chose it for 109 of its 122 answers that were not right (89.3%), and got 425 of 547 right (77.7%), against 510 for KnowLine-31B-Gen1.5, the most accurate decision model with that run.
Next of the 70 decision models with that run: celeris-1.1-decision, 74.8%, with 416 right (76.1%).
In English, a few decision models choose “I don't know” over a wrong answer far more often than any chat model; the typical one does not
The best chat setup, Claude Fable 5.2 max · chat, reaches 28.6%; 12 of the 70 decision models with an “I don't know” option in English are above it, led by d1.2-ultra at 89.3%.
The median non-hallucination rate is 7.3% for the decision models and 7.5% for the 65 chat model setups.
A 12B open model on one GPU gets 86.8% of the English reading questions
Gemma 4 12B Unified default reasoning · chat answered 475 of 547 right, the best of the 7 small open chat models run in English only.
66 of all 142 English setups got more right; the best got 536 (98.0%).
From a page picture alone, models score 13.6 points lower than from the text
Averaged over 20 image models. Vision's questions are original Turkish page images; the Vision tab has every result. The average covers the decision models only; the Vision tab shows the three chat models apart.