About the Reading bench
Our reading comprehension and grammar test: 2,198 questions answered by 77 decision models, with chat models for comparison. A decision model scores each answer option and picks one. Grammar questions test whether a model knows a rule and can apply it. Every answer is public.
The questions come from a published book; their text stays unpublished, as the publisher owns it. Every setup's answer to every question is in answers.json and the files it lists beside it (the Data page names each). The Method page has the full study: the questions, the answer key, the runs and the tests.
- 2,198 questions
- 20 topics
- 77 decision models
- 137 setups
Results 94aef0c1, updated 16 November 2026; answer key version 2.
How the scores work
- What is scored. 2,198 questions written in Turkish, not translated: 656 reading and 1,542 grammar. Each answer is right or wrong against the answer key; an invalid answer counts as wrong.
- How models answer. A decision model scores each answer option and picks one, with a confidence for it. A chat model gets the question and its options in a prompt and replies with the letter of its answer.
- What a setup is. One model with one setting: its reasoning level, how many questions go in one request (1 or 16), where it ran (a provider's API, a subscription or our GPUs) and whether it searched the web. 137 setups: 77 decision-model and 60 chat-model setups.
- How differences were tested. Paired tests on the same questions, with Holm's correction for the number of comparisons. Setups that share a letter in the ranking were not told apart by the paired test.
- What cost means. An estimate in dollars per 1,000 questions, not a bill: list prices and recorded token use for API setups; for our GPUs, a list price for the same model or weights, or the GPU time.
The rest is on the Method page.
Notes behind the scores
Ranks show observed scores; small gaps need not be meaningful.
Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. No list price: on our own GPUs, with no estimate. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. * Measured on a sample (200 questions).
The leaderboard rules
Row rule: each model's highest reasoning setting (none, off or the chat template's default < minimal < low < medium < high < xhigh < max; on above off for a model with only on and off); a decision model has no setting. Tie-break: several setups at that setting give the row to the earliest run (the lowest run id), never to the higher score. A model with one setup uses it. Every value of a row and of its details box comes from that one chosen setup; its other setups are under the row's expander and under All setups. A setup with web search is its own row; different version names are separate rows; a dated checkpoint joins another model's row only through the map below, never by a similar name.
The dated-checkpoint map (page_config.json, leaderboard.dated_checkpoints), the only model IDs listed under another: deepseek-v4.1-flash-1017 is listed under deepseek-v4.1-flash, by this map. The row shows the original; the checkpoint's setups sit under its other settings.
Ranks show observed scores; small gaps need not be meaningful. The type keys (All, Chat models, Decision models) and the Class keys (Tiny, Small, Medium and Large for an open model by its total parameters; Efficient, Workhorse and Flagship for a closed model, or an open one of undisclosed size, by its list price per 1M input tokens) choose the rows in view; the setups only this site ran are in no class and show under Class: All. On the Turkish part the Tier column holds each row's tier among the rows in view: ranked by accuracy, a row joins the tier of the tier's top row when the all-pairs test behind the full ranking's letters found no difference between them, else it starts the next tier. On Turkish reading, Turkish grammar, English reading and Both, which no all-pairs test covers, the Rank column holds the row's place, 1, 2, 3 without repeats; rows with the same score keep the table's order, which claims no difference. Each row names its model by its official name, with a small tag beside it for how the setup ran (setting · route · snapshot, as “high · chat” or “API · with web”; a decision model with one setup has none), then its class label, starting Open or Closed; its details box gives the size in full and the model page its source.
Each bar runs from 0 to 100%; its whisker is the Wilson 95% interval of the accuracy, the interval each setup's page gives. The paired tests behind the full ranking's letters are on Method.
Not measured or "Not run" in a column: the setup has no such value; the details box leaves such a line out.
Notes on each page
Leaderboard
The best chat setup gets 9.8 in 10 questions right; the best decision model 8.2 in 10
Gemini 4 Carbon · high · chat answered 98.5% of the 2,198 questions right; KnowLine-31B-Gen1.5, the best decision model, 81.8%.
The decision model that chooses “I don't know” most does so on 95 in 100 of its answers that were not right; the chat model that does so most, on 76
GLiDE 0.4B chose “I don't know” on 95.3% of the questions it did not answer right; the top chat model, LFM3-1.7B · reasoning off · chat, on 76.2%. Read it with the share answered.
The median decision setup costs $0.025$24.63 per 1,000 questionsper 1M questions, the median chat setup $1.277$1,277
Costs are list prices or GPU-time estimates, as each setup is labelled; medians over the 60 decision and 60 chat setups with a cost estimate.
At 1 question per request, the median decision setup answers in 0.15 s, the median chat setup in 6.5 s
Median time per request, from our machine, over the 77 decision and 20 chat setups that sent 1 question per request.
122 of 137 setups score higher on reading than on grammar
Among decision models, the best on reading got 93.4% of the 656 reading questions right; the best on grammar, 77.1% of the 1,542 grammar questions.
d1.1 has the smallest calibration gap of the 77 decision models, as shipped: 2.0 points
The gap is how far a model's confidence is from how often it is right. Among decision models, the largest is Laya Multilingual's, 27.7 points.
English reading
The Text tab's 547 kept reading questions in English, answered by 142 setups (76 decision models and 66 chat model setups) with their Text tab settings, 9 of them English only (no Text tab run, with their own settings). The paired comparison with Turkish is on English and Turkish.
The Text tab's 547 kept reading questions in English, answered by 66 chat model setups with their Text tab settings, 8 of them English only (no Text tab run, with their own settings). The paired comparison with Turkish is on English and Turkish.
The Text tab's 547 kept reading questions in English, answered by 76 decision models with their Text tab settings, 1 of them English only (no Text tab run, with their own settings). The paired comparison with Turkish is on English and Turkish.
Limit: Claude Opus 5.5 translated the questions and flagged those whose answer depends on Turkish wording. No person checked the translations or the flags, so an error could move a setup's English accuracy (Our method).
Beside the ranking: Web grounding, Perplexity Decider 27B v1.2 with web search off against on. PWS-Decisions, the decision models on Parallel's benchmark of dated web facts (its Answering habits).
Turkish vs English
547 of the Text tab's 656 reading questions, translated into English, asked of 133 setups (75 decision models and 58 chat model setups), each compared with its own Turkish run.
54 of the 133 setups compared scored higher on the English translation than on the Turkish original; 76 scored lower; three the same; 21 of the 133 differences are significant after Holm's correction over all 133 English setups: Gemini 4 Carbon · high · chat (higher in Turkish), Microsoft-Decision-1 (higher in English), Qwen3.5-4B · reasoning off · chat (higher in English), Metask-Jev-4B (higher in English), LFM2.5-2.6B · default reasoning · chat (higher in English), LFM3-1.7B · reasoning off · chat (higher in English), KnowLine-2B-Gen1 (higher in English), LFM3-1.7B · reasoning on · chat (higher in English), LFM2.5-8B-A1B · default reasoning · chat (higher in English), GLiDE 2B (higher in English), StartLux-Decision-v1.1-2B-Instant (higher in English), MiniCPM5-2B · reasoning off · chat (higher in English), Qwen3.5-2B · reasoning off · chat (higher in English), d1-2B (higher in English), GLiDE 0.8B (higher in English), StartLux-Decision-v1.1-0.8B-Instant (higher in English), GLiDE 0.4B (higher in English), OneJev1.6-0.8B (higher in English), KnowLine-0.8B-Gen1 (higher in English), GLiClass v4 Edge (higher in English) and Decider 1 (higher in English) (Paired test).
547 of the Text tab's 656 reading questions, translated into English, asked of 58 chat model setups, each compared with its own Turkish run.
Seventeen of the 58 chat model setups compared scored higher on the English translation than on the Turkish original; 39 scored lower; two the same; eight of the 58 differences are significant after Holm's correction over all 133 English setups: Gemini 4 Carbon · high · chat (higher in Turkish), Qwen3.5-4B · reasoning off · chat (higher in English), LFM2.5-2.6B · default reasoning · chat (higher in English), LFM3-1.7B · reasoning off · chat (higher in English), LFM3-1.7B · reasoning on · chat (higher in English), LFM2.5-8B-A1B · default reasoning · chat (higher in English), MiniCPM5-2B · reasoning off · chat (higher in English) and Qwen3.5-2B · reasoning off · chat (higher in English) (Paired test).
547 of the Text tab's 656 reading questions, translated into English, asked of 75 decision models, each compared with its own Turkish run.
37 of the 75 decision models compared scored higher on the English translation than on the Turkish original; 37 scored lower; one the same; 13 of the 75 differences are significant after Holm's correction over all 133 English setups: Microsoft-Decision-1 (higher in English), Metask-Jev-4B (higher in English), KnowLine-2B-Gen1 (higher in English), GLiDE 2B (higher in English), StartLux-Decision-v1.1-2B-Instant (higher in English), d1-2B (higher in English), GLiDE 0.8B (higher in English), StartLux-Decision-v1.1-0.8B-Instant (higher in English), GLiDE 0.4B (higher in English), OneJev1.6-0.8B (higher in English), KnowLine-0.8B-Gen1 (higher in English), GLiClass v4 Edge (higher in English) and Decider 1 (higher in English) (Paired test).
Limit: Claude Opus 5.5 translated the questions and flagged those whose answer depends on Turkish wording. No person checked the translations or the flags, so an error could move a setup's English accuracy (Our method).
Beside the ranking: Web grounding, Perplexity Decider 27B v1.2 with web search off against on. PWS-Decisions, the decision models on Parallel's benchmark of dated web facts (its Answering habits).
Hallucination
A wrong answer counts as a hallucination: the model could have chosen “I don't know” instead. This tab is separate from the leaderboard; it ranks nothing on the Text tab and changes none of its numbers.
With “I don't know”, no difference in wrong answers between the option and confidence is significant. With “I don't know” in place of “I don't know”, d1-ultra’s own confidence left significantly fewer wrong answers than “I don't know” did; for the other 8 models that used it, the difference could be chance (What we found).
Web grounding
Perplexity Decider 27B v1.2 API and Perplexity Decider 27B v1.2 API · with web, Perplexity's Decisions API without and with a web search per question; runs per mode: 3 runs on the full bank, 2 runs on the English questions and 3 runs on PWS-Decisions, the ranking keeping each setup's first run; run on 5 November 2026 to 6 November 2026.
PWS-Decisions
PWS-Decisions is Parallel Web Systems' benchmark of 100 public English multiple-choice questions whose answers depend on dated facts on the public web. We asked 38 decision model setups and 28 chat model setups that an API still serves, 3 runs each, one question per request. With web search: GPT-6.1 Sol · high · chat · CLI · with web, Gemini 4 Argon · low · chat · with web, GPT-6 Astra · high · chat · CLI · with web, Claude Fable 5.1 · high · chat · CLI · with web · 2 requests handed to Claude Opus 4.8, counted wrong, GPT-6.1 Terra · high · chat · CLI · with web, Gemini 3.8 Flash · high · chat · with web, Grok 4.7 · high · chat · with web, Claude Opus 5.5 · high · chat · CLI · with web, Claude Sonnet 5.5 · high · chat · CLI · with web, Perplexity Decider 27B v1.2 · API · with web and GPT-6 Luna · low · chat · CLI · with web; every other setup answered without search. This page is not part of the Turkish ranking.
What this measures. Parallel Web Systems made PWS-Decisions. This is an end-to-end comparison of API setups with and without web search on its 100 public questions, which Parallel chose “for their difficulty and alignment with customer outcomes” (its methodology), around its customers' needs for current facts from the web. It is not a general ranking of models: without search, most setups score near random guessing, because the answers depend on dated web facts they were not given.
53 setups have a mean accuracy above the 44.7% chance line (Parallel's random-guess baseline, not a run of ours); every accuracy here is the mean of our 3 runs.
- Dataset
- PWS-Decisions 0.1.0-rc1, Hugging Face
parallel-web/pws-decisions-benchmarkat64da0d8a4ec0 - Scorer
- Parallel's own (
python -m pws_decisions score), unchanged,0f403cf99856 - Runs
- 3 per setup, 198 in all; 0 errors, 0 missing answers
- Run dates
- 5 November 2026 to 6 November 2026
- Published
- Totals only: no question, answer or per-question result
Page images
The 763 questions of the Text tab's 2,198 that point at marked words on the printed page: 229 at underlined words and 543 at numbered words (9 at both). Each model answered them in up to four ways. This tab is ranked on its own; its numbers do not enter the Text tab's ranking.
A · text only
Kedimiz pencerenin önünde <u>sessizce</u> uyuyordu. Bu cümlede altı çizili sözcüğün türü nedir? A) İsim B) Sıfat C) Zarf D) Fiil E) Zamir
The question as text, as in the Text tab, with the bank's markup.
B · image only
Kedimiz pencerenin önünde sessizce uyuyordu.
Bu cümlede altı çizili sözcüğün türü nedir?
A) İsim B) Sıfat C) Zarf D) Fiil E) Zamir
A picture of the question cut from the book's page, alone.
C · text plus image
Kedimiz pencerenin önünde <u>sessizce</u> uyuyordu. Bu cümlede altı çizili sözcüğün türü nedir? A) İsim B) Sıfat C) Zarf D) Fiil E) Zamir
Kedimiz pencerenin önünde sessizce uyuyordu.
Bu cümlede altı çizili sözcüğün türü nedir?
A) İsim B) Sıfat C) Zarf D) Fiil E) Zamir
A's text, then the picture.
D · OCR route
Kedimiz pencerenin önünde sessizce uyuyordu. Bu cümlede altı çizili sözcüğün türü nedir? A) İsim B) Sıfat C) Zarf D) Fiil E) Zamir
One OCR model's text of the picture, for each of 4 OCR models; the most accurate is the best OCR route. Here the underline is lost; the OCR models kept 0% to 90% of underlined words.
Every question
The 2,198 scored questions, in the order of the book. Their text stays unpublished: the publisher owns it; the page and printed number find each one in the book. Each ID opens the question's page, with every setup's pick.
Data
Everything the pages show comes from these files. They hold aggregate numbers and every answer, but not the questions: the publisher owns them.
Summary
No decision model led on every measure:
- Accuracy: Copernicus-31B and KnowLine-31B-Gen1.5 tied at 81.8%, KnowLine-31B-Gen2 at 81.2% and Turion Jev 31B at 80.1%, tied with them; REAX 1.3, Karpo Decision 31B, Gemini Decision 1 and d1-ultra, tied at 77.1% to 77.9%; Logits Sigma-1 Duo and Sieve 31B v3 at 75.0% and 74.2%; GLiDE 27B at 71.1%; then twelve setups at 66.9% to 68.7%.
- Calibration gap, as shipped: d1.1 had the smallest among decision models.
- Better than a flat guess, after refit: Sieve 31B v3 at 34.9%, then Karpo Decision 31B (33.8%) and hosted Perplexity Decider 27B v1.2 (32.1%); the gaps are untested.
- Estimated saving: d1-ultra, the biggest in 7 of 16 LLM setups; Gemini Decision 1 in 5; REAX 1.3 in 2; GLiDE 27B in 2.
- Reading vs grammar: decision models got up to 93.4% of reading questions right, but at most 77.1% of grammar.
More
Method: how the questions, the answer key, the runs and the tests were made. Data: every file behind the site. All benches (draft): every Zeki Works bench for the decision models on one page.