chat model, answer-letter probabilities · League 2

DeepSeek-V4.1-Flashoff · API · 1017

FinishedPublished

Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.

How we called it

Model ID
deepseek/deepseek-v4.1-flash-1017
Provider or endpoint
DeepSeek (first party) via OpenRouter, provider pinned, no fallbacks; quantization not published
Date run
24–28 Oct 2026
Price
$0.15 / $0.6 per million tokens in / out; API price; price date 2026-10-24
Model type
chat model, answer-letter probabilities
Weights
open
Runs on
API
Reasoning setting
thinking off (reasoning disabled)
Confidence source
letter probability
Rules text
long
Option groups
up to 26 options per request (letters A-Z); more legal plays are split into equal groups, a winner per group, then among the winners; in this data 169 decisions were split into groups of 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
Sampling settings
temperature 1.0
Notes
The 17 Oct 2026 snapshot of DeepSeek-V4.1-Flash, served by DeepSeek itself through OpenRouter (provider pinned, fallbacks off) for every decision; never on our GPUs. Late entry to League 2 on 24 Oct 2026; its missing pairings were played on 27–28 Oct 2026. No fixed answer format (DeepSeek's endpoint takes no JSON schema): the letter is read from the free-text answer with its probabilities. DeepSeek may train on paid prompts; allowed by Can, 24 Oct 2026.

Results in League 2

Error rate (PR)
35.231.8 to 38.8
Decisions
2,034 (2,034 scored, 0 invalid)
Engine's best play
47.7%
Blunder rate
23.6%
P95 equity loss
0.365
Mean loss, worst 5%
0.591
Median / P90 s per decision
0.84 / 1.39
$ per 1,000 decisions
0.244API price
Risk judgement, margin 0.001
58.5 (95% interval 56.6 to 60.5); risks found 25.4% (181 / 713), right safety 91.7% (683 / 745)
Risk judgement, margin 0.02
58.8 (95% interval 56.7 to 60.9); risks found 26.5% (168 / 635), right safety 91.1% (750 / 823)
No trade-off
576 (left out of every risk metric)

JSON

Risk call card

58.5risk judgement, margin 0.001

right risks181
wrong risks45
needless risks62
missed risks487
right safety683

Hit risk only: being hit. Gammon risk and priming are not in it.

Close choices

A decision is a close choice when the second most likely play has at least half the probability of the first (internally: split).

Probabilities over plays, not sampled reasoning. Method

League 2: complete letter-readout records are mostly few-option positions, a selected subset.

Decisions
2,034
Eligible (full probability distribution)
537
Left out, by reason
asked in groups of options 128; incomplete distribution (an option without its own probability) 941; not audited 428
Close choices
11.5% (62 / 537)
Ties in the top two
0 decisions (0 at a positive probability, 0 at 0; the two most likely plays tied: 0, the second tied with the third: 0); 0 of them close choices. Ties are broken by the order the plays were shown, never by the engine.
Between a safe play and a riskier one
21.0% (13 / 62), under 200 decisions of the close choices; 2.4% (13 / 537) of all eligible decisions
Probabilities of the two most likely plays, mean with median
DecisionsNumberp1 (most likely play)p2 (second)p1 + p2
Close choices620.553median 0.5710.392median 0.3910.944median 0.998
Other decisions4750.941median 0.9930.052median 0.0050.993median 1.000
How much probability the two plays hold together: decisions per bin
p1 + p20.0–0.10.1–0.20.2–0.30.3–0.40.4–0.50.5–0.60.6–0.70.7–0.80.8–0.90.9–1.0
Close choices00000125351
Other decisions000000016468
What differs between the two most likely plays (several can differ; none is picked as the cause)
DecisionsNumberHitting rollsNewly made pointsPip countsLongest primeNone / 1 / 2 / 3 / 4 differ
Close choices6222.6%14 / 62, under 200 decisions14.5%9 / 62, under 200 decisions4.8%3 / 62, under 200 decisions17.7%11 / 62, under 200 decisions35 / 17 / 10 / 0 / 0
Other decisions47539.2%186 / 47513.9%66 / 47512.0%57 / 47524.0%114 / 475162 / 221 / 75 / 16 / 1
Engine values (GNUbg 2-ply, equity), mean with median
DecisionsNumberGap between the two plays' valuesMost likely play: behind the engine's bestSecond most likely play: behind the engine's best
Close choices620.082median 0.0180.058median 0.0080.051median 0.003
Other decisions4750.124median 0.0550.026median 0.0000.107median 0.030
Close choices between a safe play and a riskier one, by position class
MarginIn Risk pays positionsIn Safety pays positions
margin 0.00171.4%10 / 14, under 200 decisions25.0%3 / 12, under 200 decisions
margin 0.0263.6%7 / 11, under 200 decisions40.0%6 / 15, under 200 decisions
Close choices among eligible decisions, by position class and stake band
MarginRisk paysSafety paysNo trade-offRisk pays, small stakeRisk pays, medium stakeRisk pays, big stake
margin 0.00119.4%14 / 72, under 200 decisions6.9%12 / 174, under 200 decisions12.4%36 / 29120.0%3 / 15, under 200 decisions33.3%6 / 18, under 200 decisions12.8%5 / 39, under 200 decisions
margin 0.0219.3%11 / 57, under 200 decisions7.9%15 / 189, under 200 decisions12.4%36 / 291N/A0 / 033.3%6 / 18, under 200 decisions12.8%5 / 39, under 200 decisions

At the 0.02 margin a Risk pays position has a stake of at least 0.02, so the small band is empty. A big stake means playing safe costs more by the engine's value, not that the decision was harder.

Missed risks where the risk is worth taking, close choices vs other decisions
MarginClose choices: missed risksOther decisions: missed risks (weighted to the close choices' mix)Difference, close minus otherCrude difference
margin 0.00158 decisions70.0%crude 70.0% (7 / 10), under 200 decisions86.2%crude 85.4% (41 / 48), under 200 decisions−16.3 points95% interval −38.1 to +16.1−15.4 points95% interval −37.0 to +16.4
margin 0.0245 decisions71.4%crude 71.4% (5 / 7), under 200 decisions87.5%crude 84.2% (32 / 38), under 200 decisions−16.1 points95% interval −44.8 to +19.4−12.8 points95% interval −41.7 to +23.1

By game phase

PhaseError rate (PR)Decisions
Opening39.6305
Contact41.51,299
Holding gameToo few decisions28
Race18.7270
Bear-offToo few decisions132