Chat model

Erk-14B

reasoning off · 16 questions per request

45.0% of the 2,198 questions right (42.9% to 47.1%, 95% interval). Back to the leaderboard.

How we called it

Kind: Chat model
Access: vLLM, self-hosted, BF16
Paid as: Our own hardware
Model ID: erk-14b
Parameters: 14.8B
Hardware: 1 × RTX PRO 6000 Blackwell (96 GB); BF16
Note: Qwen3-14B with continued Turkish training (eCloud)
Confidence from: Answer letter probability
Run: 132, 6 October 2026

Settings

Reasoning: off
Request format: 16 questions per request
Requests at once: 1
Temperature: not recorded
Harness seed: 0
Harness commit: not recorded
Prompt version: not recorded

Results

Questions	Accuracy	95% interval	Right
All questions	45.0%	42.9% to 47.1%	989 of 2,198
Reading	72.3%	68.7% to 75.5%	474 of 656
Grammar	33.4%	31.1% to 35.8%	515 of 1,542

Comparison group: EF. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	64.1%	84 of 131
Sentence meanings	Reading	74.0%	114 of 154
How a writer explains and develops ideas	Reading	65.7%	46 of 70
Topic and main point of a paragraph	Reading	90.8%	89 of 98
How a paragraph fits together	Reading	60.4%	64 of 106
Details that support a paragraph’s main point	Reading	79.4%	77 of 97
Word types, such as nouns and verbs	Grammar	32.1%	45 of 140
Groups of words built around a noun	Grammar	30.8%	20 of 65
Verbs and their forms	Grammar	29.7%	27 of 91
Verb forms used like nouns or modifiers	Grammar	38.9%	28 of 72
Word endings and what they do	Grammar	38.0%	30 of 79
How words are built	Grammar	23.2%	16 of 69
The roles of words and phrases in a sentence	Grammar	26.2%	38 of 145
How verbs show who acts and who receives the action	Grammar	28.4%	21 of 74
Types of sentences	Grammar	29.4%	25 of 85
Questions combining several grammar topics	Grammar	27.3%	60 of 220
Sound patterns and changes	Grammar	26.0%	20 of 77
Spelling	Grammar	36.1%	48 of 133
Punctuation	Grammar	34.2%	40 of 117
Problems in sentence wording	Grammar	55.4%	97 of 175

Confidence

Answers kept at 99% or more: 101, of which 5 wrong (share right: 88.9% to 97.9%, 95% interval); 2,097 skipped, the model less than 99% sure
Calibration gap: 13.5 points
AUROC, all questions: 0.75 0.73 to 0.78
AUROC, reading: 0.83 0.80 to 0.86
AUROC, grammar: 0.64 0.61 to 0.67
Brier score: 0.214
Accuracy of its surest answers: most confident fifth: 84.3% right; most confident half: 63.4% right; most confident four fifths: 50.1% right
Typical score of a right answer: 75.1%
Typical score of a wrong answer: 43.2%

Repeats

We did not repeat this setup.

Latency

Latency, median request: 2.39 s (this run)
Latency, 90th percentile: 2.66 s
Requests at once: 1
Throughput: 392.7 questions per minute of request time
Request time: 5.6 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: Not measured
Basis: No dollar estimate; see hardware and time
Input tokens per question: 253
Output tokens per question: 7