Chat model

Claude Fable 5.1

high reasoning · 16 questions per request

96.1% of the 2,198 questions right (95.2% to 96.9%, 95% interval). Back to the leaderboard.

How we called it

Kind: Chat model
Access: Claude Code CLI, Claude subscription
Paid as: Subscription
Model ID: claude-fable-5-1
Confidence from: Written estimate
Run: 310, 28 October 2026

Settings

Reasoning: high
Request format: 16 questions per request
Requests at once: 1
Temperature: not recorded
Harness seed: 0
Harness commit: 0fe372a4a068
Prompt version: 254597b8be16

Results

Questions	Accuracy	95% interval	Right
All questions	96.1%	95.2% to 96.9%	2,113 of 2,198
Reading	95.6%	93.7% to 96.9%	627 of 656
Grammar	96.4%	95.3% to 97.2%	1,486 of 1,542

Comparison group: de. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	95.4%	125 of 131
Sentence meanings	Reading	97.4%	150 of 154
How a writer explains and develops ideas	Reading	91.4%	64 of 70
Topic and main point of a paragraph	Reading	98.0%	96 of 98
How a paragraph fits together	Reading	91.5%	97 of 106
Details that support a paragraph’s main point	Reading	97.9%	95 of 97
Word types, such as nouns and verbs	Grammar	96.4%	135 of 140
Groups of words built around a noun	Grammar	96.9%	63 of 65
Verbs and their forms	Grammar	100.0%	91 of 91
Verb forms used like nouns or modifiers	Grammar	98.6%	71 of 72
Word endings and what they do	Grammar	98.7%	78 of 79
How words are built	Grammar	95.7%	66 of 69
The roles of words and phrases in a sentence	Grammar	95.9%	139 of 145
How verbs show who acts and who receives the action	Grammar	100.0%	74 of 74
Types of sentences	Grammar	98.8%	84 of 85
Questions combining several grammar topics	Grammar	94.1%	207 of 220
Sound patterns and changes	Grammar	94.8%	73 of 77
Spelling	Grammar	96.2%	128 of 133
Punctuation	Grammar	92.3%	108 of 117
Problems in sentence wording	Grammar	96.6%	169 of 175

Confidence

Answers kept at 99% or more: 0, of which 0 wrong; 2,198 skipped, the model less than 99% sure
Calibration gap: 12.0 points
AUROC, all questions: 0.86 0.82 to 0.89
AUROC, reading: 0.93 0.88 to 0.97
AUROC, grammar: 0.82 0.76 to 0.87
Brier score: 0.049
Accuracy of its surest answers: most confident fifth: 99.9% right; most confident half: 99.5% right; most confident four fifths: 98.6% right
Typical score of a right answer: 90.0%
Typical score of a wrong answer: 60.0%

Repeats

We did not repeat this setup.

Latency

Latency, median request: 30.3 s (this run)
Latency, 90th percentile: 64.4 s
Requests at once: 1
Throughput: 27.1 questions per minute of request time
Request time: 81.1 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: $17.04
Whole run: $37.46
Basis: Subscription run priced as if we had used the API
Price: $10 input, $0.25 cached input, $12.50 5-minute cache writes, $20 1-hour cache writes, $50 output per 1M tokens (source, 2026-10-28)
Input tokens per question: 564
Output tokens per question: 174