chat model, answer-letter probabilities · League 2
DeepSeek-V4.1-Flashoff · API · 1017
FinishedPublished
Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.
How we called it
- Model ID
- deepseek/deepseek-v4.1-flash-1017
- Provider or endpoint
- DeepSeek (first party) via OpenRouter, provider pinned, no fallbacks; quantization not published
- Date run
- 24–28 Oct 2026
- Price
- $0.15 / $0.6 per million tokens in / out; API price; price date 2026-10-24
- Model type
- chat model, answer-letter probabilities
- Weights
- open
- Runs on
- API
- Reasoning setting
- thinking off (reasoning disabled)
- Confidence source
- letter probability
- Rules text
- long
- Option groups
- up to 26 options per request (letters A-Z); more legal plays are split into equal groups, a winner per group, then among the winners; in this data 169 decisions were split into groups of 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
- Sampling settings
- temperature 1.0
- Notes
- The 17 Oct 2026 snapshot of DeepSeek-V4.1-Flash, served by DeepSeek itself through OpenRouter (provider pinned, fallbacks off) for every decision; never on our GPUs. Late entry to League 2 on 24 Oct 2026; its missing pairings were played on 27–28 Oct 2026. No fixed answer format (DeepSeek's endpoint takes no JSON schema): the letter is read from the free-text answer with its probabilities. DeepSeek may train on paid prompts; allowed by Can, 24 Oct 2026.
Results in League 2
- Error rate (PR)
- 35.231.8 to 38.8
- Decisions
- 2,034 (2,034 scored, 0 invalid)
- Engine's best play
- 47.7%
- Blunder rate
- 23.6%
- P95 equity loss
- 0.365
- Mean loss, worst 5%
- 0.591
- Median / P90 s per decision
- 0.84 / 1.39
- $ per 1,000 decisions
- 0.244API price
- Risk judgement, margin 0.001
- 58.5 (95% interval 56.6 to 60.5); risks found 25.4% (181 / 713), right safety 91.7% (683 / 745)
- Risk judgement, margin 0.02
- 58.8 (95% interval 56.7 to 60.9); risks found 26.5% (168 / 635), right safety 91.1% (750 / 823)
- No trade-off
- 576 (left out of every risk metric)
Risk call card
58.5risk judgement, margin 0.001
right risks181
wrong risks45
needless risks62
missed risks487
right safety683
Hit risk only: being hit. Gammon risk and priming are not in it.
Close choices
A decision is a close choice when the second most likely play has at least half the probability of the first (internally: split).
Probabilities over plays, not sampled reasoning. Method
League 2: complete letter-readout records are mostly few-option positions, a selected subset.
- Decisions
- 2,034
- Eligible (full probability distribution)
- 537
- Left out, by reason
- asked in groups of options 128; incomplete distribution (an option without its own probability) 941; not audited 428
- Close choices
- 11.5% (62 / 537)
- Ties in the top two
- 0 decisions (0 at a positive probability, 0 at 0; the two most likely plays tied: 0, the second tied with the third: 0); 0 of them close choices. Ties are broken by the order the plays were shown, never by the engine.
- Between a safe play and a riskier one
- 21.0% (13 / 62), under 200 decisions of the close choices; 2.4% (13 / 537) of all eligible decisions
| Decisions | Number | p1 (most likely play) | p2 (second) | p1 + p2 |
|---|---|---|---|---|
| Close choices | 62 | 0.553median 0.571 | 0.392median 0.391 | 0.944median 0.998 |
| Other decisions | 475 | 0.941median 0.993 | 0.052median 0.005 | 0.993median 1.000 |
| p1 + p2 | 0.0–0.1 | 0.1–0.2 | 0.2–0.3 | 0.3–0.4 | 0.4–0.5 | 0.5–0.6 | 0.6–0.7 | 0.7–0.8 | 0.8–0.9 | 0.9–1.0 |
|---|---|---|---|---|---|---|---|---|---|---|
| Close choices | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 5 | 3 | 51 |
| Other decisions | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 6 | 468 |
| Decisions | Number | Hitting rolls | Newly made points | Pip counts | Longest prime | None / 1 / 2 / 3 / 4 differ |
|---|---|---|---|---|---|---|
| Close choices | 62 | 22.6%14 / 62, under 200 decisions | 14.5%9 / 62, under 200 decisions | 4.8%3 / 62, under 200 decisions | 17.7%11 / 62, under 200 decisions | 35 / 17 / 10 / 0 / 0 |
| Other decisions | 475 | 39.2%186 / 475 | 13.9%66 / 475 | 12.0%57 / 475 | 24.0%114 / 475 | 162 / 221 / 75 / 16 / 1 |
| Decisions | Number | Gap between the two plays' values | Most likely play: behind the engine's best | Second most likely play: behind the engine's best |
|---|---|---|---|---|
| Close choices | 62 | 0.082median 0.018 | 0.058median 0.008 | 0.051median 0.003 |
| Other decisions | 475 | 0.124median 0.055 | 0.026median 0.000 | 0.107median 0.030 |
| Margin | In Risk pays positions | In Safety pays positions |
|---|---|---|
| margin 0.001 | 71.4%10 / 14, under 200 decisions | 25.0%3 / 12, under 200 decisions |
| margin 0.02 | 63.6%7 / 11, under 200 decisions | 40.0%6 / 15, under 200 decisions |
| Margin | Risk pays | Safety pays | No trade-off | Risk pays, small stake | Risk pays, medium stake | Risk pays, big stake |
|---|---|---|---|---|---|---|
| margin 0.001 | 19.4%14 / 72, under 200 decisions | 6.9%12 / 174, under 200 decisions | 12.4%36 / 291 | 20.0%3 / 15, under 200 decisions | 33.3%6 / 18, under 200 decisions | 12.8%5 / 39, under 200 decisions |
| margin 0.02 | 19.3%11 / 57, under 200 decisions | 7.9%15 / 189, under 200 decisions | 12.4%36 / 291 | N/A0 / 0 | 33.3%6 / 18, under 200 decisions | 12.8%5 / 39, under 200 decisions |
At the 0.02 margin a Risk pays position has a stake of at least 0.02, so the small band is empty. A big stake means playing safe costs more by the engine's value, not that the decision was harder.
| Margin | Close choices: missed risks | Other decisions: missed risks (weighted to the close choices' mix) | Difference, close minus other | Crude difference |
|---|---|---|---|---|
| margin 0.00158 decisions | 70.0%crude 70.0% (7 / 10), under 200 decisions | 86.2%crude 85.4% (41 / 48), under 200 decisions | −16.3 points95% interval −38.1 to +16.1 | −15.4 points95% interval −37.0 to +16.4 |
| margin 0.0245 decisions | 71.4%crude 71.4% (5 / 7), under 200 decisions | 87.5%crude 84.2% (32 / 38), under 200 decisions | −16.1 points95% interval −44.8 to +19.4 | −12.8 points95% interval −41.7 to +23.1 |
By game phase
| Phase | Error rate (PR) | Decisions |
|---|---|---|
| Opening | 39.6 | 305 |
| Contact | 41.5 | 1,299 |
| Holding game | Too few decisions | 28 |
| Race | 18.7 | 270 |
| Bear-off | Too few decisions | 132 |