Chat model

Gemma 4 31B

reasoning off · 16 questions per request

73.1% of the 2,198 questions right (71.2% to 74.9%, 95% interval). Back to the leaderboard.

We also ran this setup at 1 question per request: Gemma 4 31B · reasoning off · 1 question per request.

How we called it

Kind: Chat model
Access: vLLM, self-hosted, FP8
Paid as: Our own hardware
Model ID: gemma-4-31B-it
Parameters: 31.3B
Hardware: 1 × RTX PRO 6000 Blackwell (96 GB); FP8 weights and KV cache
Confidence from: Answer letter probability
Run: 119, 5 October 2026

Settings

Reasoning: off
Request format: 16 questions per request
Requests at once: 1
Temperature: not recorded
Harness seed: 0
Harness commit: not recorded
Prompt version: not recorded

Results

Questions	Accuracy	95% interval	Right
All questions	73.1%	71.2% to 74.9%	1,607 of 2,198
Reading	90.9%	88.4% to 92.8%	596 of 656
Grammar	65.6%	63.2% to 67.9%	1,011 of 1,542

Comparison group: o. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	86.3%	113 of 131
Sentence meanings	Reading	93.5%	144 of 154
How a writer explains and develops ideas	Reading	88.6%	62 of 70
Topic and main point of a paragraph	Reading	96.9%	95 of 98
How a paragraph fits together	Reading	84.0%	89 of 106
Details that support a paragraph’s main point	Reading	95.9%	93 of 97
Word types, such as nouns and verbs	Grammar	67.1%	94 of 140
Groups of words built around a noun	Grammar	64.6%	42 of 65
Verbs and their forms	Grammar	75.8%	69 of 91
Verb forms used like nouns or modifiers	Grammar	79.2%	57 of 72
Word endings and what they do	Grammar	68.4%	54 of 79
How words are built	Grammar	55.1%	38 of 69
The roles of words and phrases in a sentence	Grammar	64.1%	93 of 145
How verbs show who acts and who receives the action	Grammar	77.0%	57 of 74
Types of sentences	Grammar	68.2%	58 of 85
Questions combining several grammar topics	Grammar	49.5%	109 of 220
Sound patterns and changes	Grammar	49.4%	38 of 77
Spelling	Grammar	73.7%	98 of 133
Punctuation	Grammar	70.1%	82 of 117
Problems in sentence wording	Grammar	69.7%	122 of 175

Confidence

Answers kept at 99% or more: 1,782, of which 332 wrong (share right: 79.5% to 83.1%, 95% interval); 416 skipped, the model less than 99% sure
Calibration gap: 24.2 points
AUROC, all questions: 0.83 0.82 to 0.85
AUROC, reading: 0.88 0.85 to 0.92
AUROC, grammar: 0.79 0.77 to 0.81
Brier score: 0.242
Accuracy of its surest answers: most confident fifth: 98.6% right; most confident half: 94.3% right; most confident four fifths: 81.8% right
Typical score of a right answer: 100.0%
Typical score of a wrong answer: 99.5%

Repeats

Run 5 times on a sample of 200 questions.

Answers changed in 5 runs: 0 of 200
Same wrong answer in all 5: 59 of 200
Wrong in at least one run: 59 of 200
Accuracy in each run: 70.5%, 70.5%, 70.5%, 70.5%, 70.5%

Latency

Latency, median request: 3.97 s (this run)
Latency, 90th percentile: 4.26 s
Latency at 1 question per request: 0.295 s median, on 200 questions
Requests at once: 1
Throughput: 245.3 questions per minute of request time
Request time: 9.0 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: Not measured
Basis: No dollar estimate; see hardware and time
Input tokens per question: 223
Output tokens per question: 9

Going first in front of an LLM

Replay of recorded answers, cutoff chosen on half the questions, averaged over the splits. Savings are estimates from API prices; the Method page explains how.

LLM behind it	Share taken	Accuracy change	Saving
Gemini 3.8 Flashhigh reasoning · 16 questions per request	14.8%	−0.09 points	15%†
GPT-6 Astrahigh reasoning · 16 questions per request	14.9%	−0.09 points	15%†
GPT-6.1 Solhigh reasoning · 16 questions per request	14.9%	−0.09 points	15%†
GPT-6 Astralow reasoning · 16 questions per request	14.9%	−0.09 points	15%†
GPT-6 Astralow reasoning · 1 question per request	14.6%	−0.09 points	15%†
Gemini 3.8 Flashlow reasoning · 1 question per request	15.2%	−0.10 points	15%†
GPT-6.1 Sollow reasoning · 1 question per request	16.9%	−0.13 points	17%†
GPT-6.1 Sollow reasoning · 16 questions per request	14.9%	−0.09 points	15%†
Claude Opus 5.5high reasoning · 16 questions per request	15.1%	−0.10 points	15%†
GPT-6.1 Terrahigh reasoning · 16 questions per request	17.6%	−0.12 points	18%†
Claude Opus 5.5low reasoning · 1 question per request	16.5%	−0.15 points	17%†
Gemini 3.8 Flashlow reasoning · 16 questions per request	16.9%	−0.09 points	17%†
GPT-6.1 Terralow reasoning · 16 questions per request	15.4%	−0.11 points	15%†
Claude Opus 5.5low reasoning · 16 questions per request	15.0%	−0.10 points	15%†
Claude Sonnet 5.5high reasoning · 1 question per request	26.2%	−0.40 points	26%†
Claude Sonnet 5.5high reasoning · 16 questions per request	60.9%	−0.12 points	61%†

† Leaves out the first model’s hardware cost, which we have not estimated.