Chat model

Qwen3.8 Flash no reasoning · chat

reasoning none · 16 questions per request

Accuracy, all 2,198 questions
63.1%61.0% to 65.1%, 95% interval; 1,386 right
Answers kept at 99% confidence
55 of 2,1989 wrong; share right 71.7% to 91.1%, 95% interval
Cost, $ per 1,000 questionsper 1M questions
$0.047$47.13Basis and price under Cost
Latency
10.0 smedian request, 16 questions per request

Back to the leaderboard.

How we called it

Kind
Chat model
Access
OpenRouter chat API, Alibaba's endpoint, fallbacks off, reasoning off
Paid as
Provider service
Model ID
qwen/qwen3.8-flash
Parameters
not disclosed by makerpages checked: www.alibabacloud.com/help/en/model-studio/model-pricing, openrouter.ai/qwen/qwen3.8-flash; read 11 November 2026
Note
Alibaba's closed chat model (API id qwen3.8-flash), added on the reading bench as on backgammon and Logic & Math
Confidence from
Written estimate
Run
985, 11 November 2026

Settings

Reasoning
none
Request format
16 questions per request
Requests at once
4
Temperature
not recorded
Run record: harness seed, harness commit and prompt version
Harness seed
0
Harness commit
3b4daaeaddf3
Prompt version
254597b8be16

Results

QuestionsAccuracy95% intervalRight
All questions63.1%61.0% to 65.1%1,386 of 2,198
Reading87.2%84.4% to 89.5%572 of 656
Grammar52.8%50.3% to 55.3%814 of 1,542

Comparison group: IJKLMNOP. Compare with another setup.

Accuracy by topic
TopicPartAccuracyRight
Word and phrase meaningsReading85.5%112 of 131
Sentence meaningsReading90.3%139 of 154
How a writer explains and develops ideasReading82.9%58 of 70
Topic and main point of a paragraphReading93.9%92 of 98
How a paragraph fits togetherReading74.5%79 of 106
Details that support a paragraph’s main pointReading94.8%92 of 97
Word types, such as nouns and verbsGrammar52.1%73 of 140
Groups of words built around a nounGrammar44.6%29 of 65
Verbs and their formsGrammar67.0%61 of 91
Verb forms used like nouns or modifiersGrammar61.1%44 of 72
Word endings and what they doGrammar48.1%38 of 79
How words are builtGrammar34.8%24 of 69
The roles of words and phrases in a sentenceGrammar49.0%71 of 145
How verbs show who acts and who receives the actionGrammar67.6%50 of 74
Types of sentencesGrammar64.7%55 of 85
Questions combining several grammar topicsGrammar35.0%77 of 220
Sound patterns and changesGrammar32.5%25 of 77
SpellingGrammar57.1%76 of 133
PunctuationGrammar58.1%68 of 117
Problems in sentence wordingGrammar70.3%123 of 175

English

The Text tab's 547 kept reading questions translated into English, asked with this setup's Text tab settings. Every setup: English reading; against Turkish: English and Turkish.

Right answers
476 of 547, first English run (986)
Accuracy
87.0%
Non-hallucination with “I don't know”
5.6%: 4 “I don't know” of 71 not right (run 988)
Answered with “I don't know”
99.3%: 543 of 547
Error rate with “I don't know”
12.3%: 67 of 543 answers
Cost, $ per 1,000 questionsper 1M questions, first English run
0.05049.69at list prices
Latency, median request, first English run
10.1 s; 4 requests at once
English run
986
Machine, English runs
hosted API
Date
11 November 2026
Request format
16 questions per request
Reasoning
none
Prompt version
5c3a4849752e
Translations, SHA-256
bf0d47cd385a93050c65906977e199ef7e45d5848dbbdef2bc8d124f4d5abf68

Confidence

Answers kept at 99% or more
55, of which 9 wrong (share right: 71.7% to 91.1%, 95% interval); 2,143 skipped, the model less than 99% sure
Calibration gap
28.7 points
AUROC, all questions
0.66 0.64 to 0.68
AUROC, reading
0.68 0.62 to 0.74
AUROC, grammar
0.64 0.62 to 0.67
Brier score
0.305
Accuracy of its surest answers
most confident fifth: 79.4% right; most confident half: 75.2% right; most confident four fifths: 68.3% right
Typical score of a right answer
95.0%
Typical score of a wrong answer
90.0%

Repeats

We did not repeat this setup.

Latency

Latency, median request
10.0 s (this run)
Latency, 90th percentile
12.1 s
Requests at once
4
Throughput
384.6 questions per minute of request time
Request time
5.7 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

Cost per 1,000 questionsper 1M questions
$0.047$47.13
Whole run
$0.10
Basis
Estimated from published prices and recorded token usage
Price
$0.15 input, $0.016 cached input, $0.47 output per 1M tokens (source, 2026-11-11)
Input tokens per question
237
Output tokens per question
25