Chat model

Qwen3.8-27B

yes/no scoring · 1 question per request

51.9% of the 2,198 questions right (49.8% to 54.0%, 95% interval). Back to the leaderboard.

How we called it

Kind: Chat model
Access: vLLM, self-hosted, BF16
Paid as: Our own hardware
Model ID: qwen38-27b-bf16
Variant: yes/no scoring
Parameters: 27.8B
Hardware: 1 × RTX PRO 6000 Blackwell (96 GB); BF16
Note: Open-Jev's base model
Confidence from: Option score
Run: 50, 4 October 2026

Settings

Reasoning: off
Request format: 1 question per request
Requests at once: 1
Temperature: not recorded
Harness seed: 0
Harness commit: not recorded
Prompt version: not recorded

Results

Questions	Accuracy	95% interval	Right
All questions	51.9%	49.8% to 54.0%	1,141 of 2,198
Reading	80.5%	77.3% to 83.3%	528 of 656
Grammar	39.8%	37.3% to 42.2%	613 of 1,542

Comparison group: CD. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	69.5%	91 of 131
Sentence meanings	Reading	87.7%	135 of 154
How a writer explains and develops ideas	Reading	72.9%	51 of 70
Topic and main point of a paragraph	Reading	90.8%	89 of 98
How a paragraph fits together	Reading	74.5%	79 of 106
Details that support a paragraph’s main point	Reading	85.6%	83 of 97
Word types, such as nouns and verbs	Grammar	42.1%	59 of 140
Groups of words built around a noun	Grammar	29.2%	19 of 65
Verbs and their forms	Grammar	45.1%	41 of 91
Verb forms used like nouns or modifiers	Grammar	37.5%	27 of 72
Word endings and what they do	Grammar	36.7%	29 of 79
How words are built	Grammar	30.4%	21 of 69
The roles of words and phrases in a sentence	Grammar	37.9%	55 of 145
How verbs show who acts and who receives the action	Grammar	44.6%	33 of 74
Types of sentences	Grammar	31.8%	27 of 85
Questions combining several grammar topics	Grammar	30.0%	66 of 220
Sound patterns and changes	Grammar	32.5%	25 of 77
Spelling	Grammar	46.6%	62 of 133
Punctuation	Grammar	40.2%	47 of 117
Problems in sentence wording	Grammar	58.3%	102 of 175

Confidence

Answers kept at 99% or more: 289, of which 2 wrong (share right: 97.5% to 99.8%, 95% interval); 1,909 skipped, the model less than 99% sure
Calibration gap: 8.9 points
AUROC, all questions: 0.80 0.78 to 0.82
AUROC, reading: 0.91 0.89 to 0.93
AUROC, grammar: 0.69 0.66 to 0.72
Brier score: 0.184
Accuracy of its surest answers: most confident fifth: 95.7% right; most confident half: 74.9% right; most confident four fifths: 58.5% right
Typical score of a right answer: 84.2%
Typical score of a wrong answer: 40.8%

Repeats

Run 5 times on all questions.

Answers changed in 5 runs: 160 of 2,198
Same wrong answer in all 5: 949 of 2,198
Wrong in at least one run: 1,109 of 2,198
Accuracy in each run: 51.9%, 51.7%, 51.6%, 51.4%, 51.6%

Latency

Latency, median request: 0.280 s (its repeated runs)
Latency, 90th percentile: 0.410 s
Requests at once: 1
Throughput: 193.2 questions per minute of request time
Request time: 11.4 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: $0.70
Whole run: $1.55
Basis: Estimated at Alibaba's list price for the same model; our hardware cost is not measured.
Price: $0.50 input, $0.10 cached input, $3 output per 1M tokens (source, 2026-10-18)
Input tokens per question: 1,377
Output tokens per question: 5