Chat model

Qwen3.8-27B

low reasoning · 16 questions per request

73.9% of the 2,198 questions right (72.1% to 75.7%, 95% interval). Back to the leaderboard.

How we called it

Kind: Chat model
Access: vLLM, self-hosted
Paid as: Our own hardware
Model ID: qwen38-27b-bf16
Parameters: 27.8B
Hardware: 1 × RTX PRO 6000 Blackwell (96 GB); BF16
Note: Open-Jev's base model Reasoning on at the chat template's low effort. Five of the 138 replies never left the thinking part, so their 80 questions have no answer and count as wrong. The GPU was shared with another program during the run, so its request times are longer than on a free GPU.
Confidence from: Answer letter probability
Run: 318, 28 October 2026

Settings

Reasoning: low
Request format: 16 questions per request
Requests at once: 4
Temperature: not recorded
Harness seed: 0
Harness commit: 24bccc767a3a
Prompt version: 254597b8be16

Results

Questions	Accuracy	95% interval	Right
All questions	73.9%	72.1% to 75.7%	1,625 of 2,198
Reading	89.3%	86.7% to 91.5%	586 of 656
Grammar	67.4%	65.0% to 69.7%	1,039 of 1,542

Comparison group: no. Compare with another setup.

80 questions got no valid answer; they count as wrong.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	86.3%	113 of 131
Sentence meanings	Reading	96.1%	148 of 154
How a writer explains and develops ideas	Reading	78.6%	55 of 70
Topic and main point of a paragraph	Reading	95.9%	94 of 98
How a paragraph fits together	Reading	81.1%	86 of 106
Details that support a paragraph’s main point	Reading	92.8%	90 of 97
Word types, such as nouns and verbs	Grammar	77.9%	109 of 140
Groups of words built around a noun	Grammar	80.0%	52 of 65
Verbs and their forms	Grammar	80.2%	73 of 91
Verb forms used like nouns or modifiers	Grammar	84.7%	61 of 72
Word endings and what they do	Grammar	79.7%	63 of 79
How words are built	Grammar	37.7%	26 of 69
The roles of words and phrases in a sentence	Grammar	79.3%	115 of 145
How verbs show who acts and who receives the action	Grammar	79.7%	59 of 74
Types of sentences	Grammar	77.6%	66 of 85
Questions combining several grammar topics	Grammar	43.6%	96 of 220
Sound patterns and changes	Grammar	26.0%	20 of 77
Spelling	Grammar	65.4%	87 of 133
Punctuation	Grammar	65.8%	77 of 117
Problems in sentence wording	Grammar	77.1%	135 of 175

Confidence

Answers kept at 99% or more: 2,113, of which 490 wrong (share right: 75.0% to 78.6%, 95% interval); 85 skipped: 5 with the model less than 99% sure and 80 without a valid answer
Calibration gap: 23.3 points
AUROC, all questions: 0.63 0.61 to 0.66
AUROC, reading: 0.76 0.70 to 0.81
AUROC, grammar: 0.60 0.57 to 0.63
Brier score: 0.233
Accuracy of its surest answers: most confident fifth: 88.1% right; most confident half: 83.9% right; most confident four fifths: 79.6% right
Typical score of a right answer: 100.0%
Typical score of a wrong answer: 100.0%

Repeats

We did not repeat this setup.

Latency

Latency, median request: 648.4 s (this run)
Latency, 90th percentile: 1337.2 s
Requests at once: 4
Throughput: 5.5 questions per minute of request time
Request time: 400.6 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: $2.93
Whole run: $6.44
Basis: Estimated at Alibaba's list price for the same model; our hardware cost is not measured.
Price: $0.50 input, $0.10 cached input, $3 output per 1M tokens (source, 2026-10-18)
Input tokens per question: 238
Output tokens per question: 937