Chat model

DeepSeek-V4.1-Flash, first release

reasoning on · 16 questions per request

92.0% of the 2,198 questions right (90.8% to 93.1%, 95% interval). Back to the leaderboard.

How we called it

Kind: Chat model
Access: SGLang, self-hosted, published mixed precision; first-release weights (revision fb2764a), not the 1017 update
Paid as: Our own hardware
Model ID: deepseek-v4.1-flash
Parameters: 552B (8 to 16B active per token)
Hardware: 4 × RTX PRO 6000 Blackwell (96 GB); FP4 experts, FP8 and BF16 layers
Confidence from: Answer letter probability
Run: 121, 5 October 2026

Settings

Reasoning: on
Request format: 16 questions per request
Requests at once: 8
Temperature: not recorded
Harness seed: 0
Harness commit: not recorded
Prompt version: not recorded

Results

Questions	Accuracy	95% interval	Right
All questions	92.0%	90.8% to 93.1%	2,023 of 2,198
Reading	93.4%	91.3% to 95.1%	613 of 656
Grammar	91.4%	89.9% to 92.7%	1,410 of 1,542

Comparison group: hij. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	95.4%	125 of 131
Sentence meanings	Reading	97.4%	150 of 154
How a writer explains and develops ideas	Reading	84.3%	59 of 70
Topic and main point of a paragraph	Reading	93.9%	92 of 98
How a paragraph fits together	Reading	88.7%	94 of 106
Details that support a paragraph’s main point	Reading	95.9%	93 of 97
Word types, such as nouns and verbs	Grammar	92.1%	129 of 140
Groups of words built around a noun	Grammar	96.9%	63 of 65
Verbs and their forms	Grammar	97.8%	89 of 91
Verb forms used like nouns or modifiers	Grammar	97.2%	70 of 72
Word endings and what they do	Grammar	94.9%	75 of 79
How words are built	Grammar	88.4%	61 of 69
The roles of words and phrases in a sentence	Grammar	93.8%	136 of 145
How verbs show who acts and who receives the action	Grammar	94.6%	70 of 74
Types of sentences	Grammar	95.3%	81 of 85
Questions combining several grammar topics	Grammar	86.4%	190 of 220
Sound patterns and changes	Grammar	85.7%	66 of 77
Spelling	Grammar	94.0%	125 of 133
Punctuation	Grammar	83.8%	98 of 117
Problems in sentence wording	Grammar	89.7%	157 of 175

Confidence

Answers kept at 99% or more: 2,198, of which 175 wrong (share right: 90.8% to 93.1%, 95% interval); 0 skipped, the model less than 99% sure
Calibration gap: 8.0 points
AUROC, all questions: 0.56 0.52 to 0.61
AUROC, reading: 0.60 0.51 to 0.68
AUROC, grammar: 0.56 0.51 to 0.61
Brier score: 0.080
Accuracy of its surest answers: most confident fifth: 93.2% right; most confident half: 93.3% right; most confident four fifths: 93.1% right
Typical score of a right answer: 100.0%
Typical score of a wrong answer: 100.0%

Repeats

Run 5 times on a sample of 200 questions.

Answers changed in 5 runs: 15 of 200
Same wrong answer in all 5: 5 of 200
Wrong in at least one run: 20 of 200
Accuracy in each run: 95.0%, 97.0%, 94.0%, 93.5%, 95.0%

Latency

Latency, median request: 365.1 s (this run)
Latency, 90th percentile: 686.9 s
Requests at once: 8
Throughput: 19.3 questions per minute of request time
Request time: 114.1 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: Not measured
Basis: No dollar estimate; see hardware and time
Input tokens per question: 305
Output tokens per question: 1,006