Chat model

Gemini 3.8 Flash

low reasoning · 16 questions per request

93.3% of the 2,198 questions right (92.1% to 94.2%, 95% interval). Back to the leaderboard.

We also ran this setup at 1 question per request: Gemini 3.8 Flash · low reasoning · 1 question per request.

Which decision model can go first in front of it? See the Routing tab.

How we called it

Kind: Chat model
Access: Gemini API
Paid as: Provider service
Model ID: gemini-3.8-flash
Confidence from: Written estimate
Run: 123, 5 October 2026

Settings

Reasoning: low
Request format: 16 questions per request
Requests at once: 1
Temperature: not recorded
Harness seed: 0
Harness commit: not recorded
Prompt version: not recorded

Thinking levels (Google's thinking documentation, read 28 October 2026): low, medium (default), high. Top level: high. This run: low.

Results

Questions	Accuracy	95% interval	Right
All questions	93.3%	92.1% to 94.2%	2,050 of 2,198
Reading	95.1%	93.2% to 96.5%	624 of 656
Grammar	92.5%	91.1% to 93.7%	1,426 of 1,542

Comparison group: fghi. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	96.2%	126 of 131
Sentence meanings	Reading	96.8%	149 of 154
How a writer explains and develops ideas	Reading	88.6%	62 of 70
Topic and main point of a paragraph	Reading	98.0%	96 of 98
How a paragraph fits together	Reading	90.6%	96 of 106
Details that support a paragraph’s main point	Reading	97.9%	95 of 97
Word types, such as nouns and verbs	Grammar	95.0%	133 of 140
Groups of words built around a noun	Grammar	95.4%	62 of 65
Verbs and their forms	Grammar	100.0%	91 of 91
Verb forms used like nouns or modifiers	Grammar	94.4%	68 of 72
Word endings and what they do	Grammar	91.1%	72 of 79
How words are built	Grammar	92.8%	64 of 69
The roles of words and phrases in a sentence	Grammar	89.7%	130 of 145
How verbs show who acts and who receives the action	Grammar	98.6%	73 of 74
Types of sentences	Grammar	92.9%	79 of 85
Questions combining several grammar topics	Grammar	86.4%	190 of 220
Sound patterns and changes	Grammar	85.7%	66 of 77
Spelling	Grammar	96.2%	128 of 133
Punctuation	Grammar	88.9%	104 of 117
Problems in sentence wording	Grammar	94.9%	166 of 175

Confidence

Answers kept at 99% or more: 976, of which 15 wrong (share right: 97.5% to 99.1%, 95% interval); 1,222 skipped, the model less than 99% sure
Calibration gap: 3.5 points
AUROC, all questions: 0.77 0.72 to 0.80
AUROC, reading: 0.80 0.71 to 0.89
AUROC, grammar: 0.76 0.72 to 0.80
Brier score: 0.060
Accuracy of its surest answers: most confident fifth: 98.4% right; most confident half: 98.1% right; most confident four fifths: 96.0% right
Typical score of a right answer: 98.0%
Typical score of a wrong answer: 95.0%

Repeats

We did not repeat this setup.

Latency

Latency, median request: 2.19 s (this run)
Latency, 90th percentile: 2.69 s
Requests at once: 1
Throughput: 399.9 questions per minute of request time
Request time: 5.5 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: $0.27
Whole run: $0.59
Basis: Estimated from published prices and recorded token usage
Price: $0.75 input, $0.075 cached input, $3.75 output per 1M tokens (source, 2026-10-08)
Input tokens per question: 220
Output tokens per question: 28