chat model, answer-letter probabilities · League 2
Gemma 4 31Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark
FinishedPublished
Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.
How we called it
- Model ID
- gemma-4-31B-it
- Provider or endpoint
- NVIDIA RTX PRO 6000 Blackwell (96 GB), fp8 (vLLM); NVIDIA DGX Spark (GB10) 19 Oct 18:36-22:26 UTC and from 20 Oct 21:10 UTC (back on the NVIDIA RTX PRO 6000 Blackwell (96 GB) 21 Oct 01:25-05:00)
- Date run
- 2026-10-19 to 2026-10-26
- Price
- no price per token; estimate: GPU time, at most; source content/own_cost.json (wall clock on our machines × hourly rate); price date not recorded
- Model type
- chat model, answer-letter probabilities
- Weights
- open
- Runs on
- our GPUs
- Reasoning setting
- thinking off
- Confidence source
- letter probability
- Rules text
- long
- Option groups
- up to 26 options per request (letters A-Z); more legal plays are split into equal groups, a winner per group, then among the winners; in this data 183 decisions were split into groups of 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
- Sampling settings
- temperature 1.0
- Notes
- moved between our GPUs and a DGX Spark during the league with the same image, checkpoint and arguments
Results in League 2
- Error rate (PR)
- 22.519.9 to 25.3
- Decisions
- 1,847 (1,847 scored, 0 invalid)
- Engine's best play
- 54.4%
- Blunder rate
- 18.2%
- P95 equity loss
- 0.222
- Mean loss, worst 5%
- 0.376
- Median / P90 s per decision
- 0.31 / 4.03
- $ per 1,000 decisions
- data-v="8.215">8.215estimate · RTX PRO 6000 Blackwell
- Risk judgement, margin 0.001
- 62.6 (95% interval 60.4 to 65.0); risks found 33.2% (194 / 584), right safety 92.1% (732 / 795)
- Risk judgement, margin 0.02
- 63.2 (95% interval 61.0 to 65.6); risks found 35.5% (182 / 513), right safety 90.9% (787 / 866)
- No trade-off
- 468 (left out of every risk metric)
Risk call card
62.6risk judgement, margin 0.001
right risks194
wrong risks49
needless risks63
missed risks341
right safety732
Hit risk only: being hit. Gammon risk and priming are not in it.
Close choices
A decision is a close choice when the second most likely play has at least half the probability of the first (internally: split).
Probabilities over plays, not sampled reasoning. Method
League 2: complete letter-readout records are mostly few-option positions, a selected subset.
- Decisions
- 1,847
- Eligible (full probability distribution)
- 475
- Left out, by reason
- asked in groups of options 148; incomplete distribution (an option without its own probability) 993; not audited 231
- Close choices
- 2.9% (14 / 475)
- Ties in the top two
- 1 decisions (1 at a positive probability, 0 at 0; the two most likely plays tied: 0, the second tied with the third: 1); 0 of them close choices. Ties are broken by the order the plays were shown, never by the engine.
- Between a safe play and a riskier one
- 42.9% (6 / 14), under 200 decisions of the close choices; 1.3% (6 / 475) of all eligible decisions
| Decisions | Number | p1 (most likely play) | p2 (second) | p1 + p2 |
|---|---|---|---|---|
| Close choices | 14 | 0.564median 0.561 | 0.408median 0.415 | 0.971median 0.998 |
| Other decisions | 461 | 0.977median 1.000 | 0.021median 0.000 | 0.998median 1.000 |
| p1 + p2 | 0.0–0.1 | 0.1–0.2 | 0.2–0.3 | 0.3–0.4 | 0.4–0.5 | 0.5–0.6 | 0.6–0.7 | 0.7–0.8 | 0.8–0.9 | 0.9–1.0 |
|---|---|---|---|---|---|---|---|---|---|---|
| Close choices | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 12 |
| Other decisions | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 459 |
| Decisions | Number | Hitting rolls | Newly made points | Pip counts | Longest prime | None / 1 / 2 / 3 / 4 differ |
|---|---|---|---|---|---|---|
| Close choices | 14 | 42.9%6 / 14, under 200 decisions | 14.3%2 / 14, under 200 decisions | 0.0%0 / 14, under 200 decisions | 21.4%3 / 14, under 200 decisions | 6 / 5 / 3 / 0 / 0 |
| Other decisions | 461 | 35.1%162 / 461 | 14.1%65 / 461 | 10.4%48 / 461 | 25.2%116 / 461 | 185 / 178 / 81 / 17 / 0 |
| Decisions | Number | Gap between the two plays' values | Most likely play: behind the engine's best | Second most likely play: behind the engine's best |
|---|---|---|---|---|
| Close choices | 14 | 0.079median 0.040 | 0.028median 0.000 | 0.060median 0.014 |
| Other decisions | 461 | 0.100median 0.036 | 0.024median 0.000 | 0.095median 0.029 |
| Margin | In Risk pays positions | In Safety pays positions |
|---|---|---|
| margin 0.001 | 50.0%2 / 4, under 200 decisions | 100.0%4 / 4, under 200 decisions |
| margin 0.02 | 66.7%2 / 3, under 200 decisions | 80.0%4 / 5, under 200 decisions |
| Margin | Risk pays | Safety pays | No trade-off | Risk pays, small stake | Risk pays, medium stake | Risk pays, big stake |
|---|---|---|---|---|---|---|
| margin 0.001 | 4.9%4 / 81, under 200 decisions | 2.8%4 / 144, under 200 decisions | 2.4%6 / 250 | 8.3%1 / 12, under 200 decisions | 6.9%2 / 29, under 200 decisions | 2.5%1 / 40, under 200 decisions |
| margin 0.02 | 4.3%3 / 69, under 200 decisions | 3.2%5 / 156, under 200 decisions | 2.4%6 / 250 | N/A0 / 0 | 6.9%2 / 29, under 200 decisions | 2.5%1 / 40, under 200 decisions |
At the 0.02 margin a Risk pays position has a stake of at least 0.02, so the small band is empty. A big stake means playing safe costs more by the engine's value, not that the decision was harder.
| Margin | Close choices: missed risks | Other decisions: missed risks (weighted to the close choices' mix) | Difference, close minus other | Crude difference |
|---|---|---|---|---|
| margin 0.00157 decisions | 0.0%crude 0.0% (0 / 2), under 200 decisions | 71.2%crude 63.6% (35 / 55), under 200 decisions | −71.2 points95% interval −100.0 to +0.0 | −63.6 points95% interval −74.6 to −50.0 |
| margin 0.0249 decisions | 0.0%crude 0.0% (0 / 2), under 200 decisions | 66.7%crude 57.4% (27 / 47), under 200 decisions | −66.7 points95% interval −100.0 to +0.0 | −57.4 points95% interval −69.2 to −43.5 |
By game phase
| Phase | Error rate (PR) | Decisions |
|---|---|---|
| Opening | 18.8 | 273 |
| Contact | 27.3 | 1,121 |
| Holding game | Too few decisions | 87 |
| Race | 15.9 | 224 |
| Bear-off | Too few decisions | 142 |