Chat model

Kimi K3.1 Pro

low reasoning · 16 questions per request

82.8% of the 2,198 questions right (81.1% to 84.3%, 95% interval). Back to the leaderboard.

How we called it

Kind: Chat model
Access: OpenRouter chat API, Moonshot AI's endpoint (mxfp4), fallbacks off
Paid as: Provider service
Model ID: moonshotai/kimi-k3.1-pro
Confidence from: Written estimate
Run: 320, 29 October 2026

Settings

Reasoning: low
Request format: 16 questions per request
Requests at once: 4
Temperature: not recorded
Harness seed: 0
Harness commit: 6cda04f8a3a7
Prompt version: 254597b8be16

Results

Questions	Accuracy	95% interval	Right
All questions	82.8%	81.1% to 84.3%	1,819 of 2,198
Reading	93.8%	91.6% to 95.4%	615 of 656
Grammar	78.1%	75.9% to 80.1%	1,204 of 1,542

Comparison group: l. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	91.6%	120 of 131
Sentence meanings	Reading	98.7%	152 of 154
How a writer explains and develops ideas	Reading	81.4%	57 of 70
Topic and main point of a paragraph	Reading	98.0%	96 of 98
How a paragraph fits together	Reading	90.6%	96 of 106
Details that support a paragraph’s main point	Reading	96.9%	94 of 97
Word types, such as nouns and verbs	Grammar	82.9%	116 of 140
Groups of words built around a noun	Grammar	84.6%	55 of 65
Verbs and their forms	Grammar	86.8%	79 of 91
Verb forms used like nouns or modifiers	Grammar	93.1%	67 of 72
Word endings and what they do	Grammar	81.0%	64 of 79
How words are built	Grammar	73.9%	51 of 69
The roles of words and phrases in a sentence	Grammar	80.0%	116 of 145
How verbs show who acts and who receives the action	Grammar	83.8%	62 of 74
Types of sentences	Grammar	81.2%	69 of 85
Questions combining several grammar topics	Grammar	67.3%	148 of 220
Sound patterns and changes	Grammar	51.9%	40 of 77
Spelling	Grammar	77.4%	103 of 133
Punctuation	Grammar	71.8%	84 of 117
Problems in sentence wording	Grammar	85.7%	150 of 175

Confidence

Answers kept at 99% or more: 1,436, of which 114 wrong (share right: 90.5% to 93.3%, 95% interval); 762 skipped, the model less than 99% sure
Calibration gap: 12.6 points
AUROC, all questions: 0.75 0.72 to 0.78
AUROC, reading: 0.82 0.75 to 0.89
AUROC, grammar: 0.73 0.70 to 0.76
Brier score: 0.141
Accuracy of its surest answers: most confident fifth: 92.7% right; most confident half: 92.2% right; most confident four fifths: 89.4% right
Typical score of a right answer: 99.0%
Typical score of a wrong answer: 97.0%

Repeats

We did not repeat this setup.

Latency

Latency, median request: 44.9 s (this run)
Latency, 90th percentile: 144.5 s
Requests at once: 4
Throughput: 60.0 questions per minute of request time
Request time: 36.6 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: $2.05
Whole run: $4.50
Basis: Estimated from published prices and recorded token usage
Price: $2 input, $0.20 cached input, $10 output per 1M tokens (source, 2026-10-28)
Input tokens per question: 359
Output tokens per question: 134