Chat model

Grok 4.7

high reasoning · 16 questions per request

92.7% of the 2,198 questions right (91.6% to 93.7%, 95% interval). Back to the leaderboard.

How we called it

Kind: Chat model
Access: xAI Responses API, SuperGrok subscription
Paid as: Subscription
Model ID: grok-4.7
Note: xAI did not answer 31 attempts within the time limit, and the runner sent each such request again. The run stopped twice and was resumed from the last answered question: at 1,696 answers, after one request timed out six times (the limit then went from 15 to 30 minutes), and at 2,144, when a retry sent an expired sign-in token. The usage and cost of the timed-out attempts are unknown.
Confidence from: Written estimate
Run: 311, 28 October 2026

Settings

Reasoning: high
Request format: 16 questions per request
Requests at once: 4
Temperature: not recorded
Harness seed: 0
Harness commit: 0fe372a4a068
Prompt version: 254597b8be16

Results

Questions	Accuracy	95% interval	Right
All questions	92.7%	91.6% to 93.7%	2,038 of 2,198
Reading	95.0%	93.0% to 96.4%	623 of 656
Grammar	91.8%	90.3% to 93.0%	1,415 of 1,542

Comparison group: ghij. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	95.4%	125 of 131
Sentence meanings	Reading	98.7%	152 of 154
How a writer explains and develops ideas	Reading	85.7%	60 of 70
Topic and main point of a paragraph	Reading	95.9%	94 of 98
How a paragraph fits together	Reading	92.5%	98 of 106
Details that support a paragraph’s main point	Reading	96.9%	94 of 97
Word types, such as nouns and verbs	Grammar	93.6%	131 of 140
Groups of words built around a noun	Grammar	96.9%	63 of 65
Verbs and their forms	Grammar	95.6%	87 of 91
Verb forms used like nouns or modifiers	Grammar	100.0%	72 of 72
Word endings and what they do	Grammar	93.7%	74 of 79
How words are built	Grammar	88.4%	61 of 69
The roles of words and phrases in a sentence	Grammar	97.2%	141 of 145
How verbs show who acts and who receives the action	Grammar	97.3%	72 of 74
Types of sentences	Grammar	95.3%	81 of 85
Questions combining several grammar topics	Grammar	85.0%	187 of 220
Sound patterns and changes	Grammar	89.6%	69 of 77
Spelling	Grammar	86.5%	115 of 133
Punctuation	Grammar	85.5%	100 of 117
Problems in sentence wording	Grammar	92.6%	162 of 175

Confidence

Answers kept at 99% or more: 0, of which 0 wrong; 2,198 skipped, the model less than 99% sure
Calibration gap: 11.9 points
AUROC, all questions: 0.85 0.82 to 0.88
AUROC, reading: 0.92 0.88 to 0.95
AUROC, grammar: 0.82 0.79 to 0.86
Brier score: 0.074
Accuracy of its surest answers: most confident fifth: 100.0% right; most confident half: 99.0% right; most confident four fifths: 97.2% right
Typical score of a right answer: 86.0%
Typical score of a wrong answer: 62.0%

Repeats

We did not repeat this setup.

Latency

Latency, median request: 401.9 s (this run)
Latency, 90th percentile: 1228.6 s
Requests at once: 4
Throughput: 6.6 questions per minute of request time
Request time: 333.2 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: $14.29
Whole run: $31.41
Basis: Subscription run priced as if we had used the API
Price: $2 input, $0.50 cached input, $6 output per 1M tokens (source, 2026-10-28)
Input tokens per question: 368
Output tokens per question: 2,284