chat model, answer-letter probabilities · League 2

Gemma 4 31Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark

FinishedPublished

Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.

How we called it

Model ID
gemma-4-31B-it
Provider or endpoint
NVIDIA RTX PRO 6000 Blackwell (96 GB), fp8 (vLLM); NVIDIA DGX Spark (GB10) 19 Oct 18:36-22:26 UTC and from 20 Oct 21:10 UTC (back on the NVIDIA RTX PRO 6000 Blackwell (96 GB) 21 Oct 01:25-05:00)
Date run
2026-10-19 to 2026-10-26
Price
no price per token; estimate: GPU time, at most; source content/own_cost.json (wall clock on our machines × hourly rate); price date not recorded
Model type
chat model, answer-letter probabilities
Weights
open
Runs on
our GPUs
Reasoning setting
thinking off
Confidence source
letter probability
Rules text
long
Option groups
up to 26 options per request (letters A-Z); more legal plays are split into equal groups, a winner per group, then among the winners; in this data 183 decisions were split into groups of 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
Sampling settings
temperature 1.0
Notes
moved between our GPUs and a DGX Spark during the league with the same image, checkpoint and arguments

Results in League 2

Error rate (PR)
22.519.9 to 25.3
Decisions
1,847 (1,847 scored, 0 invalid)
Engine's best play
54.4%
Blunder rate
18.2%
P95 equity loss
0.222
Mean loss, worst 5%
0.376
Median / P90 s per decision
0.31 / 4.03
$ per 1,000 decisions
data-v="8.215">8.215estimate · RTX PRO 6000 Blackwell
Risk judgement, margin 0.001
62.6 (95% interval 60.4 to 65.0); risks found 33.2% (194 / 584), right safety 92.1% (732 / 795)
Risk judgement, margin 0.02
63.2 (95% interval 61.0 to 65.6); risks found 35.5% (182 / 513), right safety 90.9% (787 / 866)
No trade-off
468 (left out of every risk metric)

JSON

Risk call card

62.6risk judgement, margin 0.001

right risks194
wrong risks49
needless risks63
missed risks341
right safety732

Hit risk only: being hit. Gammon risk and priming are not in it.

Close choices

A decision is a close choice when the second most likely play has at least half the probability of the first (internally: split).

Probabilities over plays, not sampled reasoning. Method

League 2: complete letter-readout records are mostly few-option positions, a selected subset.

Decisions
1,847
Eligible (full probability distribution)
475
Left out, by reason
asked in groups of options 148; incomplete distribution (an option without its own probability) 993; not audited 231
Close choices
2.9% (14 / 475)
Ties in the top two
1 decisions (1 at a positive probability, 0 at 0; the two most likely plays tied: 0, the second tied with the third: 1); 0 of them close choices. Ties are broken by the order the plays were shown, never by the engine.
Between a safe play and a riskier one
42.9% (6 / 14), under 200 decisions of the close choices; 1.3% (6 / 475) of all eligible decisions
Probabilities of the two most likely plays, mean with median
DecisionsNumberp1 (most likely play)p2 (second)p1 + p2
Close choices140.564median 0.5610.408median 0.4150.971median 0.998
Other decisions4610.977median 1.0000.021median 0.0000.998median 1.000
How much probability the two plays hold together: decisions per bin
p1 + p20.0–0.10.1–0.20.2–0.30.3–0.40.4–0.50.5–0.60.6–0.70.7–0.80.8–0.90.9–1.0
Close choices00000000212
Other decisions000000002459
What differs between the two most likely plays (several can differ; none is picked as the cause)
DecisionsNumberHitting rollsNewly made pointsPip countsLongest primeNone / 1 / 2 / 3 / 4 differ
Close choices1442.9%6 / 14, under 200 decisions14.3%2 / 14, under 200 decisions0.0%0 / 14, under 200 decisions21.4%3 / 14, under 200 decisions6 / 5 / 3 / 0 / 0
Other decisions46135.1%162 / 46114.1%65 / 46110.4%48 / 46125.2%116 / 461185 / 178 / 81 / 17 / 0
Engine values (GNUbg 2-ply, equity), mean with median
DecisionsNumberGap between the two plays' valuesMost likely play: behind the engine's bestSecond most likely play: behind the engine's best
Close choices140.079median 0.0400.028median 0.0000.060median 0.014
Other decisions4610.100median 0.0360.024median 0.0000.095median 0.029
Close choices between a safe play and a riskier one, by position class
MarginIn Risk pays positionsIn Safety pays positions
margin 0.00150.0%2 / 4, under 200 decisions100.0%4 / 4, under 200 decisions
margin 0.0266.7%2 / 3, under 200 decisions80.0%4 / 5, under 200 decisions
Close choices among eligible decisions, by position class and stake band
MarginRisk paysSafety paysNo trade-offRisk pays, small stakeRisk pays, medium stakeRisk pays, big stake
margin 0.0014.9%4 / 81, under 200 decisions2.8%4 / 144, under 200 decisions2.4%6 / 2508.3%1 / 12, under 200 decisions6.9%2 / 29, under 200 decisions2.5%1 / 40, under 200 decisions
margin 0.024.3%3 / 69, under 200 decisions3.2%5 / 156, under 200 decisions2.4%6 / 250N/A0 / 06.9%2 / 29, under 200 decisions2.5%1 / 40, under 200 decisions

At the 0.02 margin a Risk pays position has a stake of at least 0.02, so the small band is empty. A big stake means playing safe costs more by the engine's value, not that the decision was harder.

Missed risks where the risk is worth taking, close choices vs other decisions
MarginClose choices: missed risksOther decisions: missed risks (weighted to the close choices' mix)Difference, close minus otherCrude difference
margin 0.00157 decisions0.0%crude 0.0% (0 / 2), under 200 decisions71.2%crude 63.6% (35 / 55), under 200 decisions−71.2 points95% interval −100.0 to +0.0−63.6 points95% interval −74.6 to −50.0
margin 0.0249 decisions0.0%crude 0.0% (0 / 2), under 200 decisions66.7%crude 57.4% (27 / 47), under 200 decisions−66.7 points95% interval −100.0 to +0.0−57.4 points95% interval −69.2 to −43.5

By game phase

PhaseError rate (PR)Decisions
Opening18.8273
Contact27.31,121
Holding gameToo few decisions87
Race15.9224
Bear-offToo few decisions142