chat model, answer-letter probabilities · League 2
Qwen3.8 Flashoff
FinishedPublished
Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.
How we called it
- Model ID
- qwen/qwen3.8-flash
- Provider or endpoint
- Alibaba via OpenRouter, quantization not published
- Date run
- 2026-10-20 to 2026-10-26
- Price
- $0.15 / $0.47 per million tokens in / out; API price; price date not recorded
- Model type
- chat model, answer-letter probabilities
- Weights
- Missing
- Runs on
- API
- Reasoning setting
- reasoning off
- Confidence source
- letter probability
- Rules text
- long
- Option groups
- up to 26 options per request (letters A-Z); more legal plays are split into equal groups, a winner per group, then among the winners; in this data 171 decisions were split into groups of 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
- Sampling settings
- temperature 1.0
- Notes
- OpenRouter, pinned to Alibaba (its only provider); replaces Qwen3.8 Flash Next, which is not on OpenRouter; quantization not published
Missing B7 fields: weights, price.date
Results in League 2
- Error rate (PR)
- 33.630.8 to 36.7
- Decisions
- 2,027 (2,027 scored, 0 invalid)
- Engine's best play
- 44.6%
- Blunder rate
- 25.3%
- P95 equity loss
- 0.308
- Mean loss, worst 5%
- 0.533
- Median / P90 s per decision
- 1.70 / 7.47
- $ per 1,000 decisions
- 0.284API price
- Risk judgement, margin 0.001
- 59.5 (95% interval 57.2 to 61.7); risks found 34.4% (243 / 706), right safety 84.6% (613 / 725)
- Risk judgement, margin 0.02
- 59.3 (95% interval 57.1 to 61.5); risks found 35.5% (222 / 625), right safety 83.0% (669 / 806)
- No trade-off
- 596 (left out of every risk metric)
Risk call card
59.5risk judgement, margin 0.001
Hit risk only: being hit. Gammon risk and priming are not in it.
Close choices
A decision is a close choice when the second most likely play has at least half the probability of the first (internally: split).
Probabilities over plays, not sampled reasoning. Method
League 2: complete letter-readout records are mostly few-option positions, a selected subset.
- Decisions
- 2,027
- Eligible (full probability distribution)
- 615
- Left out, by reason
- asked in groups of options 145; incomplete distribution (an option without its own probability) 1,024; not audited 243
- Close choices
- 40.3% (248 / 615)
- Ties in the top two
- 46 decisions (46 at a positive probability, 0 at 0; the two most likely plays tied: 25, the second tied with the third: 21); 30 of them close choices. Ties are broken by the order the plays were shown, never by the engine.
- Between a safe play and a riskier one
- 29.0% (72 / 248) of the close choices; 11.7% (72 / 615) of all eligible decisions
| Decisions | Number | p1 (most likely play) | p2 (second) | p1 + p2 |
|---|---|---|---|---|
| Close choices | 248 | 0.496median 0.500 | 0.354median 0.349 | 0.850median 0.855 |
| Other decisions | 367 | 0.720median 0.731 | 0.192median 0.194 | 0.911median 0.919 |
| p1 + p2 | 0.0–0.1 | 0.1–0.2 | 0.2–0.3 | 0.3–0.4 | 0.4–0.5 | 0.5–0.6 | 0.6–0.7 | 0.7–0.8 | 0.8–0.9 | 0.9–1.0 |
|---|---|---|---|---|---|---|---|---|---|---|
| Close choices | 0 | 0 | 0 | 0 | 1 | 4 | 26 | 59 | 60 | 98 |
| Other decisions | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 37 | 110 | 216 |
| Decisions | Number | Hitting rolls | Newly made points | Pip counts | Longest prime | None / 1 / 2 / 3 / 4 differ |
|---|---|---|---|---|---|---|
| Close choices | 248 | 29.0%72 / 248 | 10.5%26 / 248 | 6.0%15 / 248 | 16.9%42 / 248 | 136 / 76 / 29 / 7 / 0 |
| Other decisions | 367 | 39.2%144 / 367 | 13.1%48 / 367 | 11.7%43 / 367 | 27.8%102 / 367 | 121 / 163 / 76 / 6 / 1 |
| Decisions | Number | Gap between the two plays' values | Most likely play: behind the engine's best | Second most likely play: behind the engine's best |
|---|---|---|---|---|
| Close choices | 248 | 0.073median 0.021 | 0.044median 0.001 | 0.058median 0.003 |
| Other decisions | 367 | 0.124median 0.048 | 0.026median 0.000 | 0.109median 0.030 |
| Margin | In Risk pays positions | In Safety pays positions |
|---|---|---|
| margin 0.001 | 75.0%27 / 36, under 200 decisions | 75.0%45 / 60, under 200 decisions |
| margin 0.02 | 79.3%23 / 29, under 200 decisions | 73.1%49 / 67, under 200 decisions |
| Margin | Risk pays | Safety pays | No trade-off | Risk pays, small stake | Risk pays, medium stake | Risk pays, big stake |
|---|---|---|---|---|---|---|
| margin 0.001 | 43.4%36 / 83, under 200 decisions | 31.7%60 / 189, under 200 decisions | 44.3%152 / 343 | 53.8%7 / 13, under 200 decisions | 37.1%13 / 35, under 200 decisions | 45.7%16 / 35, under 200 decisions |
| margin 0.02 | 41.4%29 / 70, under 200 decisions | 33.2%67 / 202 | 44.3%152 / 343 | N/A0 / 0 | 37.1%13 / 35, under 200 decisions | 45.7%16 / 35, under 200 decisions |
At the 0.02 margin a Risk pays position has a stake of at least 0.02, so the small band is empty. A big stake means playing safe costs more by the engine's value, not that the decision was harder.
| Margin | Close choices: missed risks | Other decisions: missed risks (weighted to the close choices' mix) | Difference, close minus other | Crude difference |
|---|---|---|---|---|
| margin 0.00165 decisions | 74.1%crude 74.1% (20 / 27), under 200 decisions | 88.1%crude 89.5% (34 / 38), under 200 decisions | −14.1 points95% interval −37.9 to +9.4 | −15.4 points95% interval −33.9 to +5.6 |
| margin 0.0255 decisions | 69.6%crude 69.6% (16 / 23), under 200 decisions | 86.1%crude 87.5% (28 / 32), under 200 decisions | −16.5 points95% interval −42.9 to +10.7 | −17.9 points95% interval −38.1 to +4.9 |
By game phase
| Phase | Error rate (PR) | Decisions |
|---|---|---|
| Opening | 36.8 | 298 |
| Contact | 37.1 | 1,337 |
| Holding game | Too few decisions | 27 |
| Race | 20.6 | 249 |
| Bear-off | Too few decisions | 116 |