Chat model
DeepSeek-V4.1-Flash reasoning off · chat · 1017
reasoning off · 16 questions per request
- Accuracy, all 2,198 questions
- 66.0%64.0% to 68.0%, 95% interval; 1,451 right
- Answers kept at 99% confidence
- 696 of 2,19816 wrong; share right 96.3% to 98.6%, 95% interval
- Cost, $ per 1,000 questionsper 1M questions
- $0.098$98.47Basis and price under Cost
- Latency
- 5.08 smedian request, 16 questions per request
How we called it
- Kind
- Chat model
- Access
- SGLang, self-hosted, published mixed precision; 1017 weights (revision 1bae0e1)
- Paid as
- Our own hardware
- Model ID
deepseek-v4.1-flash-1017- Parameters
- Mixture of experts: 552B total, about 16B active per step. Size class uses the total (552B).V4.1 Flash card (first release): 552B backbone + 196B conditional memory; 8B active in prefill, 16B in decode; the 1017 card states no size; pages checked: huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017, huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash; read 9 November 2026
- Hardware
- 4 × RTX PRO 6000 Blackwell (96 GB); FP4 experts, FP8 and BF16 layers
- Confidence from
- Answer letter probability
- Run
- 558, 5 November 2026
Settings
- Reasoning
- off
- Request format
- 16 questions per request
- Requests at once
- 1
- Temperature
- not recorded
Run record: harness seed, harness commit and prompt version
- Harness seed
- 0
- Harness commit
ecf1af0326be- Prompt version
254597b8be16
Results
| Questions | Accuracy | 95% interval | Right |
|---|---|---|---|
| All questions | 66.0% | 64.0% to 68.0% | 1,451 of 2,198 |
| Reading | 88.3% | 85.6% to 90.5% | 579 of 656 |
| Grammar | 56.5% | 54.1% to 59.0% | 872 of 1,542 |
Comparison group: DEFGHIJKLM. Compare with another setup.
Accuracy by topic
English
The Text tab's 547 kept reading questions translated into English, asked with this setup's Text tab settings. Every setup: English reading; against Turkish: English and Turkish.
- Right answers
- 476 of 547, first English run (804)
- Accuracy
- 87.0%
- Non-hallucination with “I don't know”
- 5.4%: 4 “I don't know” of 74 not right (run 805)
- Answered with “I don't know”
- 99.3%: 543 of 547
- Error rate with “I don't know”
- 12.9%: 70 of 543 answers
- Cost, $ per 1,000 questionsper 1M questions, first English run
- 0.07979.31at the maker's list price
- Latency, median request, first English run
- 6.59 s; 1 request at once
- English run
- 804
- Machine, English runs
- RTX PRO 6000
- Date
- 5 November 2026
- Request format
- 16 questions per request
- Reasoning
- off
- Prompt version
5c3a4849752e- Translations, SHA-256
bf0d47cd385a93050c65906977e199ef7e45d5848dbbdef2bc8d124f4d5abf68
Confidence
- Answers kept at 99% or more
- 696, of which 16 wrong (share right: 96.3% to 98.6%, 95% interval); 1,502 skipped, the model less than 99% sure
- Calibration gap
- 16.4 points
- AUROC, all questions
- 0.83 0.81 to 0.85
- AUROC, reading
- 0.89 0.86 to 0.92
- AUROC, grammar
- 0.77 0.75 to 0.79
- Brier score
- 0.197
- Accuracy of its surest answers
- most confident fifth: 99.5% right; most confident half: 89.1% right; most confident four fifths: 74.3% right
- Typical score of a right answer
- 98.5%
- Typical score of a wrong answer
- 68.0%
Repeats
Run 5 times on a sample of 200 questions.
- Answers changed in 5 runs
- 10 of 200
- Same wrong answer in all 5
- 68 of 200
- Wrong in at least one run
- 78 of 200
- Accuracy in each run
- 65.0%, 63.0%, 63.5%, 62.5%, 63.5%
Latency
- Latency, median request
- 5.08 s (this run)
- Latency, 90th percentile
- 11.5 s
- Requests at once
- 1
- Throughput
- 147.3 questions per minute of request time
- Request time
- 14.9 minutes, over the requests whose answers were kept
Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.
Cost
- Cost per 1,000 questionsper 1M questions
- $0.098$98.47
- Whole run
- $0.22
- Basis
- Estimated at DeepSeek's list price for the same model; our hardware cost is not measured.
- Price
- $0.30 input, $0.006 cached input, $1.20 output per 1M tokens (source, 2026-11-05)
- Input tokens per question
- 303
- Output tokens per question
- 6