Chat model
Qwen4-Max high · chat
high reasoning · 16 questions per request
- Accuracy, all 2,198 questions
- 91.7%90.5% to 92.8%, 95% interval; 2,016 right
- Answers kept at 99% confidence
- 3 of 2,1980 wrong; share right 43.8% to 100.0%, 95% interval
- Cost, $ per 1,000 questionsper 1M questions
- $2.533$2,533Basis and price under Cost
- Latency
- 91.5 smedian request, 16 questions per request
How we called it
- Kind
- Chat model
- Access
- OpenRouter chat API, Alibaba's endpoint (the maker's), fallbacks off, reasoning effort high, output cap 32,768 tokens, reasoning included; the answer schema
- Paid as
- Provider service
- Model ID
qwen/qwen4-max- Parameters
- not disclosed by makerpages checked: openrouter.ai/qwen/qwen4-max; read 16 November 2026
- Note
- Alibaba's closed flagship chat model (OpenRouter id qwen/qwen4-max, listed 16 November 2026)
- Confidence from
- Written estimate
- Run
- 1336, 16 November 2026
Settings
- Reasoning
- high
- Request format
- 16 questions per request
- Requests at once
- 4
- Temperature
- not recorded
Run record: harness seed, harness commit and prompt version
- Harness seed
- 0
- Harness commit
22e561be1d9d- Prompt version
254597b8be16
Results
| Questions | Accuracy | 95% interval | Right |
|---|---|---|---|
| All questions | 91.7% | 90.5% to 92.8% | 2,016 of 2,198 |
| Reading | 94.7% | 92.7% to 96.1% | 621 of 656 |
| Grammar | 90.5% | 88.9% to 91.8% | 1,395 of 1,542 |
Comparison group: jkl. Compare with another setup.
Invalid answers: 32; read from broken replies: 0; empty replies asked again: 0. Still invalid: 32 cut off at the output limit. An empty reply is asked once more; in a broken reply, any letter that is clearly readable is scored and the rest count as wrong. Invalid answers count as wrong (how).
With “Bilmiyorum” (run 1339): Invalid answers: 16; read from broken replies: 0; empty replies asked again: 0. Still invalid: 16 cut off at the output limit. (how)
Accuracy by topic
English
The Text tab's 547 kept reading questions translated into English, asked with this setup's Text tab settings. Every setup: English reading; against Turkish: English and Turkish.
- Right answers
- 517 of 547, first English run (1334)
- Accuracy
- 94.5%
- Non-hallucination with “I don't know”
- 0.0%: 0 “I don't know” of 29 not right (run 1338)
- Answered with “I don't know”
- 100.0%: 547 of 547
- Error rate with “I don't know”
- 5.3%: 29 of 547 answers
- Cost, $ per 1,000 questionsper 1M questions, first English run
- 1.1191,119at list prices
- Latency, median request, first English run
- 29.5 s; 4 requests at once
- English run
- 1334
- Machine, English runs
- hosted API
- Date
- 16 November 2026
- Request format
- 16 questions per request
- Reasoning
- high
- Prompt version
5c3a4849752e- Translations, SHA-256
bf0d47cd385a93050c65906977e199ef7e45d5848dbbdef2bc8d124f4d5abf68
Confidence
- Answers kept at 99% or more
- 3, of which 0 wrong (share right: 43.8% to 100.0%, 95% interval); 2,195 skipped: 2,163 with the model less than 99% sure and 32 without a valid answer
- Calibration gap
- 7.8 points
- AUROC, all questions
- 0.86 0.83 to 0.89
- AUROC, reading
- 0.93 0.88 to 0.96
- AUROC, grammar
- 0.83 0.78 to 0.86
- Brier score
- 0.060
- Accuracy of its surest answers
- most confident fifth: 99.8% right; most confident half: 98.9% right; most confident four fifths: 97.2% right
- Typical score of a right answer
- 90.0%
- Typical score of a wrong answer
- 63.5%
Repeats
We did not repeat this setup.
Latency
- Latency, median request
- 91.5 s (this run)
- Latency, 90th percentile
- 335.0 s
- Requests at once
- 4
- Throughput
- 28.2 questions per minute of request time
- Request time
- 78.0 minutes, over the requests whose answers were kept
Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.
Cost
- Cost per 1,000 questionsper 1M questions
- $2.533$2,533
- Whole run
- $5.57
- Basis
- Estimated from published prices and recorded token usage
- Price
- $1.50 input, $4.50 output per 1M tokens (source, 2026-11-16)
- Input tokens per question
- 322
- Output tokens per question
- 456