Chat model

Claude Sonnet 5.5

high reasoning · 1 question per request

89.2% of the 2,198 questions right (87.8% to 90.4%, 95% interval). Back to the leaderboard.

We also ran this setup at 16 questions per request: Claude Sonnet 5.5 · high reasoning · 16 questions per request.

Which decision model can go first in front of it? See the Routing tab.

How we called it

Kind: Chat model
Access: Claude Code CLI, Claude subscription
Paid as: Subscription
Model ID: claude-sonnet-5-5
Confidence from: Written estimate
Run: 193, 13 October 2026

Settings

Reasoning: high
Request format: 1 question per request
Requests at once: 1
Temperature: not recorded
Harness seed: 0
Harness commit: 54043fd92cd5
Prompt version: 254597b8be16

Effort levels (Anthropic's effort documentation, read 28 October 2026): low, medium, high (default), xhigh, max. Top level: max. This run: high.

Results

Questions	Accuracy	95% interval	Right
All questions	89.2%	87.8% to 90.4%	1,960 of 2,198
Reading	94.2%	92.1% to 95.8%	618 of 656
Grammar	87.0%	85.3% to 88.6%	1,342 of 1,542

Comparison group: k. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	94.7%	124 of 131
Sentence meanings	Reading	96.1%	148 of 154
How a writer explains and develops ideas	Reading	82.9%	58 of 70
Topic and main point of a paragraph	Reading	95.9%	94 of 98
How a paragraph fits together	Reading	95.3%	101 of 106
Details that support a paragraph’s main point	Reading	95.9%	93 of 97
Word types, such as nouns and verbs	Grammar	93.6%	131 of 140
Groups of words built around a noun	Grammar	93.8%	61 of 65
Verbs and their forms	Grammar	94.5%	86 of 91
Verb forms used like nouns or modifiers	Grammar	93.1%	67 of 72
Word endings and what they do	Grammar	94.9%	75 of 79
How words are built	Grammar	88.4%	61 of 69
The roles of words and phrases in a sentence	Grammar	89.7%	130 of 145
How verbs show who acts and who receives the action	Grammar	87.8%	65 of 74
Types of sentences	Grammar	94.1%	80 of 85
Questions combining several grammar topics	Grammar	83.6%	184 of 220
Sound patterns and changes	Grammar	74.0%	57 of 77
Spelling	Grammar	76.7%	102 of 133
Punctuation	Grammar	76.1%	89 of 117
Problems in sentence wording	Grammar	88.0%	154 of 175

Confidence

Answers kept at 99% or more: 0, of which 0 wrong; 2,198 skipped, the model less than 99% sure
Calibration gap: 9.8 points
AUROC, all questions: 0.84 0.81 to 0.86
AUROC, reading: 0.89 0.85 to 0.93
AUROC, grammar: 0.82 0.80 to 0.85
Brier score: 0.090
Accuracy of its surest answers: most confident fifth: 99.4% right; most confident half: 98.3% right; most confident four fifths: 94.8% right
Typical score of a right answer: 85.0%
Typical score of a wrong answer: 60.0%

Repeats

We did not repeat this setup.

Latency

Latency, median request: 4.64 s (this run)
Latency, 90th percentile: 8.38 s
Requests at once: 1
Throughput: 11.2 questions per minute of request time
Request time: 196.0 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: $7.75
Whole run: $17.03
Basis: Subscription run priced as if we had used the API
Price: $2 input, $0.10 cached input, $2.50 5-minute cache writes, $4 1-hour cache writes, $10 output per 1M tokens (source, 2026-10-08)
Input tokens per question: 1,495
Output tokens per question: 390