Chat model

Gemma 4 31B

reasoning off · 1 question per request

66.8% of the 2,198 questions right (64.8% to 68.7%, 95% interval). Back to the leaderboard.

We also ran this setup at 16 questions per request: Gemma 4 31B · reasoning off · 16 questions per request.

How we called it

Kind: Chat model
Access: vLLM, self-hosted, FP8
Paid as: Our own hardware
Model ID: gemma-4-31B-it
Parameters: 31.3B
Hardware: 1 × RTX PRO 6000 Blackwell (96 GB); FP8 weights and KV cache
Confidence from: Answer letter probability
Run: 189, 13 October 2026

Settings

Reasoning: off
Request format: 1 question per request
Requests at once: 1
Temperature: not recorded
Harness seed: 0
Harness commit: 385f57507b2a
Prompt version: 254597b8be16

Results

Questions	Accuracy	95% interval	Right
All questions	66.8%	64.8% to 68.7%	1,468 of 2,198
Reading	89.0%	86.4% to 91.2%	584 of 656
Grammar	57.3%	54.8% to 59.8%	884 of 1,542

Comparison group: pqrs. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	86.3%	113 of 131
Sentence meanings	Reading	91.6%	141 of 154
How a writer explains and develops ideas	Reading	87.1%	61 of 70
Topic and main point of a paragraph	Reading	95.9%	94 of 98
How a paragraph fits together	Reading	79.2%	84 of 106
Details that support a paragraph’s main point	Reading	93.8%	91 of 97
Word types, such as nouns and verbs	Grammar	65.0%	91 of 140
Groups of words built around a noun	Grammar	52.3%	34 of 65
Verbs and their forms	Grammar	65.9%	60 of 91
Verb forms used like nouns or modifiers	Grammar	62.5%	45 of 72
Word endings and what they do	Grammar	60.8%	48 of 79
How words are built	Grammar	47.8%	33 of 69
The roles of words and phrases in a sentence	Grammar	48.3%	70 of 145
How verbs show who acts and who receives the action	Grammar	64.9%	48 of 74
Types of sentences	Grammar	50.6%	43 of 85
Questions combining several grammar topics	Grammar	40.0%	88 of 220
Sound patterns and changes	Grammar	41.6%	32 of 77
Spelling	Grammar	69.9%	93 of 133
Punctuation	Grammar	63.2%	74 of 117
Problems in sentence wording	Grammar	71.4%	125 of 175

Confidence

Answers kept at 99% or more: 1,826, of which 487 wrong (share right: 71.3% to 75.3%, 95% interval); 372 skipped, the model less than 99% sure
Calibration gap: 31.2 points
AUROC, all questions: 0.82 0.80 to 0.84
AUROC, reading: 0.88 0.84 to 0.92
AUROC, grammar: 0.75 0.73 to 0.78
Brier score: 0.309
Accuracy of its surest answers: most confident fifth: 97.3% right; most confident half: 89.4% right; most confident four fifths: 74.7% right
Typical score of a right answer: 100.0%
Typical score of a wrong answer: 99.8%

Repeats

We did not repeat this setup.

Latency

Latency, median request: 0.298 s (this run)
Latency, 90th percentile: 0.306 s
Requests at once: 1
Throughput: 195.5 questions per minute of request time
Request time: 11.2 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: Not measured
Basis: No dollar estimate; see hardware and time
Input tokens per question: 318
Output tokens per question: 11