Chat model

Claude Sonnet 5.5

low reasoning · 16 questions per request

61.4% of the 2,198 questions right (59.3% to 63.4%, 95% interval). Back to the leaderboard.

How we called it

Kind: Chat model
Access: Claude Code CLI, Claude subscription
Paid as: Subscription
Model ID: claude-sonnet-5-5
Confidence from: Written estimate
Run: 54, 4 October 2026

Settings

Reasoning: low
Request format: 16 questions per request
Requests at once: 1
Temperature: not recorded
Harness seed: 0
Harness commit: not recorded
Prompt version: not recorded

Effort levels (Anthropic's effort documentation, read 28 October 2026): low, medium, high (default), xhigh, max. Top level: max. This run: low.

Results

Questions	Accuracy	95% interval	Right
All questions	61.4%	59.3% to 63.4%	1,349 of 2,198
Reading	82.9%	79.9% to 85.6%	544 of 656
Grammar	52.2%	49.7% to 54.7%	805 of 1,542

Comparison group: tuvwx. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	81.7%	107 of 131
Sentence meanings	Reading	86.4%	133 of 154
How a writer explains and develops ideas	Reading	71.4%	50 of 70
Topic and main point of a paragraph	Reading	92.9%	91 of 98
How a paragraph fits together	Reading	71.7%	76 of 106
Details that support a paragraph’s main point	Reading	89.7%	87 of 97
Word types, such as nouns and verbs	Grammar	53.6%	75 of 140
Groups of words built around a noun	Grammar	47.7%	31 of 65
Verbs and their forms	Grammar	67.0%	61 of 91
Verb forms used like nouns or modifiers	Grammar	61.1%	44 of 72
Word endings and what they do	Grammar	53.2%	42 of 79
How words are built	Grammar	40.6%	28 of 69
The roles of words and phrases in a sentence	Grammar	44.1%	64 of 145
How verbs show who acts and who receives the action	Grammar	71.6%	53 of 74
Types of sentences	Grammar	48.2%	41 of 85
Questions combining several grammar topics	Grammar	30.9%	68 of 220
Sound patterns and changes	Grammar	35.1%	27 of 77
Spelling	Grammar	69.9%	93 of 133
Punctuation	Grammar	53.0%	62 of 117
Problems in sentence wording	Grammar	66.3%	116 of 175

Confidence

Answers kept at 99% or more: 0, of which 0 wrong; 2,198 skipped, the model less than 99% sure
Calibration gap: 7.9 points
AUROC, all questions: 0.79 0.77 to 0.80
AUROC, reading: 0.82 0.78 to 0.86
AUROC, grammar: 0.72 0.70 to 0.75
Brier score: 0.190
Accuracy of its surest answers: most confident fifth: 94.3% right; most confident half: 82.3% right; most confident four fifths: 69.0% right
Typical score of a right answer: 60.0%
Typical score of a wrong answer: 40.0%

Repeats

Run 5 times on a sample of 200 questions.

Answers changed in 5 runs: 72 of 200
Same wrong answer in all 5: 20 of 200
Wrong in at least one run: 92 of 200
Accuracy in each run: 70.0%, 65.5%, 68.0%, 68.0%, 68.5%

Latency

Latency, median request: 3.87 s, measured on a sample (200 questions) over 5 runs
Latency, 90th percentile: 5.85 s
Requests at once: 1
Throughput: 230.3 questions per minute of request time
Request time: 9.6 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: $1.90
Whole run: $4.19
Basis: Subscription run priced as if we had used the API
Price: $2 input, $0.10 cached input, $2.50 5-minute cache writes, $4 1-hour cache writes, $10 output per 1M tokens (source, 2026-10-08)
Input tokens per question: 550
Output tokens per question: 22