Method

How the games are played, what the models are shown, how every decision is scored, and the rules for intervals and verdicts. The thresholds below were fixed before any number on this site was computed. The page as it was before this site went live: archive, 23 October 2026.

Setup

The question is which model, at which reasoning setting, makes better decisions in which positions. Two kinds of data answer it: league games and shared positions. A setup is one model with one set of settings: the same model at a lower and a higher reasoning effort is two setups. A setup is named by what its requests sent, checked in the request logs: (low), (medium), (high) = the effort level sent; (off) = reasoning turned off, no reasoning tokens; (on) = thinking switched on where the model offers no effort levels (its chat template's default).

Leagues

Setups play each other in leagues: each league is a single round robin. League tables compare players inside one league only, and leagues are never merged into one overall score. See Leagues.

Match rules

How a decision is asked

Shared positions

Every setup answers the same fixed positions with the same prompt. These positions, not the league tables, compare setups across leagues, and the main ranking uses them only. Each position is a board, the dice and the side to move.

SetPositionsHow drawnBy phase
v1 set1,000Recorded positions from the earlier version's leagues, the same positions the earlier report used. Their plays are reshuffled with the option-order seed.from 396 recorded gamesopening 143 · contact 641 · holding 20 · race 136 · bear-off 60
New set1,000GNU Backgammon self-play at expert level, the same number per phase.from 747 recorded games; from 747 self-play games; 15 first-roll positions; 5 duplicates skippedopening 200 · contact 200 · holding 200 · race 200 · bear-off 200
Nackgammon500GNU Backgammon self-play from the Nackgammon starting position (an unfamiliar start, same rules). Its prompt drops the opening table. Superseded 2026-10-27 by the Nackgammon paired test (below).from 301 recorded games; from 301 self-play games; 300 from turns 1 to 6, 200 from later turns; 14 first-roll positions; first roll 3-2 left out (27 legal plays); 14 duplicates skippedopening 300 · contact 145 · holding 8 · race 28 · bear-off 19
Opening control200The new set's opening positions, asked again with the opening table removed from the prompt.15 first-roll positions–
League 1 extension4,000More expert self-play positions with the same phase mix, for League 1 setups only.from 2,938 recorded games; from 2,938 self-play games; 99 duplicates skippedopening 800 · contact 800 · holding 800 · race 800 · bear-off 800

Counts from the position build log. Sampling rules: every position has 2 to 25 legal plays, so no model needs its plays split into groups; each self-play game gives at most one position per phase; within a game and phase the turn is drawn at random from a fixed seed; duplicates are skipped. If a phase cannot be filled, all distinct positions are used and the shortfall is logged.

Game phases

Each position gets the first phase that matches:

  1. Opening: turns 1 to 6, each side's first three moves.
  2. Bear-off: any of GNU Backgammon's bear-off classes.
  3. Race: GNU Backgammon's race class.
  4. Holding: a contact position after turn 6 where one side has escaped all its back checkers and the other keeps one high anchor (2 or more checkers on the opponent's bar, 5- or 4-point, and no other checkers in the opponent's home board or on its bar point). This follows The Gammon Press's definition of a holding game.
  5. Contact: every other position.

Game-plan tags

For analysis only, each contact position after the opening gets the first tag that matches: blitz (a checker on the bar against a home board with 3 or more made points), back game (2 or more anchors in the opponent's home board), holding (the rule above), low anchor (the only anchor is on the opponent's 3-, 2- or ace-point), priming (4 or more made points in a row), or other.

The prompt

The current version is bg-v2. Every player gets it, and it did not change after the first match. It is the format of the earlier version's league #8. Every decision sends these parts in this order:

  1. Rules and strategy: the rules of backgammon, strategy principles (blots and points, distribution, exposure, blocking, hitting, anchors), the best opening plays from computer rollouts, and the match rules. Decision models get a short version; chat models get a longer one with a glossary, game plans and a worked example. Each setup's page says which it got.
  2. Score and where the match stands.
  3. Position from the mover's side: checkers on every point, bar, borne off, pip counts.
  4. Race: the pip lead and whether the two sides are still in contact.
  5. Dice, then the question: which move makes a point in front of the opponent's checkers, reduces your single pieces, or gets you closer to home.
  6. The legal plays in the shuffled order, each followed by what the position looks like after it, computed by the program: hits, points made, blots left and how many of the opponent's 36 rolls hit them, points held, longest prime, pips. Chat models see each play under a letter.

There is no move history: in the earlier version, adding the last 10 moves made no measurable difference. The rules text is published on Every decision; the text built for each position is not, so the positions stay usable as a test.

History

The prompt was built one step at a time in the earlier version's leagues; the next section shows each step and what it changed. bg-v2 keeps that format and changes how the models are asked: the plays are shuffled, an invalid answer is not retried, and every setup gets the same rules and question. The earlier version re-asked up to three times and then played the first legal play, which its order favoured.

Why it looks like this

Earlier version Plays were listed in the runner's order, which tends to put the best play early.

The prompt was changed one step at a time, and a league was run after each step. Leagues stopped part-way are kept, with their finished matches only.

LeagueWhat changedGamesHit the 300-turn limitMoves that hitAverage gameBest opening playError rate
#1Structured state (JSON) with a short rules text and how every earlier game ended3882%62%277 turns28%138
#3Rules and strategy text, then score, time, position and dice; nothing about earlier games17081%60%274 turns77%132
#6+ the question before the choices; Decider v1.1 self-hosted12471%58%255 turns83%120
#7+ what each play leads to, and the race line1350%9.7%50 turns68%38.5
#8As #7 without d1 (7 models)1200%9.5%51 turns69%39.4
#11Repeat, new dice1120%9.3%50 turns58%40.1
#12Repeat, new dice1120%9.7%51 turns69%36.2
#13+ the last 10 moves of the game1060%10.9%52 turns72%39.1

Grey rows did not finish. "Best opening play" is the share of first moves of a game that match the computer-rollout table in the prompt. "Error rate" is GNU Backgammon's, over all model decisions of the league (see Error rate); lower is better. Leagues #2, #4, #5, #9 and #10 failed at start or were stopped within minutes; league #3 is left out of the standings below because its format differs.

Rules and strategy as text, not a data structure

The first leagues sent the position as structured data with a short rules summary. Models opened badly: 28% of first moves matched the known best play. With the rules, strategy principles and the opening table as prose, that rose to 77%. Opening play improved, but the middle game did not: games still ran to the turn limit, because both sides kept hitting and re-entering.

A question that names what a good move does

The question before the choices states three goals in plain words: build a point in front of the opponent's checkers, reduce your blots, bring checkers home. It deliberately leaves hitting out, because hitting on almost every move was the problem. The turn-limit share fell from 81% to 71% (this league also switched to the self-hosted Decider v1.1); hitting barely changed.

Compute the consequences in code, not in the model

Backgammon decisions turn on numbers a language model cannot work out reliably from a list of notations: which checkers are left alone, how many of the opponent's 36 rolls hit them, which points are made. Earlier, each option was just its notation, like bar/19 13/11*, and the model had to imagine the resulting position. Now the program computes those numbers for every option and the model compares them. This is the change that ended the endless games: hits fell from about 60% to under 10% of moves and the average game from about 270 turns to about 50.

Opening play got worse at the same time, from about 80% to between 58% and 69% best plays. The metrics after each option compete with the opening table earlier in the prompt, and on the first move several plays look similar on these numbers.

Recent moves, for longer plans

Each decision is a separate request, so a model cannot see how the position came about. League #13 adds the last 10 moves of the game, both sides. Ten turns add about 200 tokens. More would eat into Open-Jev's 4,096-token limit, which it applies to the whole question repeated for every option.

It made no measurable difference. League #13's error rate was 39.1 (± 2.2), against 38.6 (± 1.2) for #8, #11 and #12 together, and the finishing order barely changed. Ten moves may be too few, or these models may not use them.

The prompt

Earlier version Plays were listed in the runner's order, which tends to put the best play early.

Every decision sends the same parts in the same order:

  1. Rules and strategy (about 6,400 characters): the rules of backgammon, strategy principles (blots and points, distribution, exposure, blocking, hitting, anchors), the best opening plays from computer rollouts, and the match rules above. A longer version with a glossary, game plans and a worked example goes to chat models; decision models get this short one because of Open-Jev's 4,096-token limit.
  2. Score and time: match score, round and game, turn number, both clocks.
  3. Position from the mover's side: checkers on every point, bar, borne off, pip counts.
  4. Race: the pip lead and whether the two sides are still in contact.
  5. The last 10 moves of this game, both sides, in the mover's numbering (league #13 on).
  6. Dice, then the question: Which of the following moves makes a point in front of the opponent's checkers, or reduces your single pieces, or gets you closer to your home?
  7. The legal plays, each followed by what the position looks like after it, computed by the program.

The shape of a request (the rules part is left out; angle brackets stand for the values of the position at hand, and no real position or answer is shown, so that the positions stay usable as a test):

Score: you <n>, opponent <n>.
Match: round <r> of <R>, game <g> of <G> in this round.
Turn <t> of at most <T> in this game.
Clock: <free delay> for this decision, then your bank: <time> left in this game (opponent <time>).

Position (your numbering; you move 24 -> 1):
  your checkers:     <point:count …>
  opponent checkers: <point:count …>
  on the bar: you <n>, opponent <n>
  borne off:  you <n>, opponent <n>
  pip count:  you <n>, opponent <n>
Race: <the pip lead>; <whether the checkers are still in contact>.
Last 10 moves of this game (oldest first, your numbering):
  turn <t>, <you / opponent> rolled <dice>: <play>
  …
Dice: <d1>-<d2>

Which of the following moves makes a point in front of the opponent's checkers, or reduces your single pieces, or gets you closer to your home?

Options (<n>):
<play>: hits <n>; makes <points made>; leaves <n> blots, the opponent hits with <n> of 36 rolls; you hold <n> points (<n> in your home board), longest prime <n>; pips: you <n>, opponent <n>
…  (one line per legal play)

Answer: <one of the plays> (with the model's probability where it gives one)

The numbers after each play are exact. The opponent's hitting rolls are counted by generating its legal plays for each of the 36 rolls and checking whether any of them hits.

Error rate

Wins and losses depend heavily on the dice. A backgammon engine can instead judge every decision directly. GNU Backgammon 1.08 evaluates every legal play at 2-ply, in cubeless money equity (expected points per game, match score ignored). The loss of a decision is the equity of the best play minus the equity of the play made. A play that ends the game scores its points exactly.

Error rate
500 × mean 2-ply equity loss per decision. Lower is better. This is the checker-play performance rating (PR) that backgammon players use.
Decision
A roll with 2 or more legal plays. Forced moves count in no rate.
Invalid answers
Left out of the error rate and shown separately, with their count and share.
Engine's best play
Share of decisions where the play made lost less than 0.001 equity.
Blunder rate
Share of decisions where the play made lost more than 0.08 equity.
Tail error
The 0.95 quantile of the loss per decision, and the mean loss of the worst 0.05 of decisions.
Near-tie
The two best plays are within 0.02 equity. Close plays are not rolled out; the gap groups on the Phases page show how much of a setup's error comes from them.

Human scale (approximate)

Error rates are shown next to eXtreme Gammon rating scale's bands for human players:

BandError rate
World championunder 2.5
World class2.5 to 5
Expert5 to 7.5
Advanced7.5 to 12.5
Intermediate12.5 to 17.5
Casual17.5 to 22.5
Beginner22.5 to 30
Distracted30 or more

The comparison is approximate. Human ratings also count doubling-cube decisions; these games have no cube, so the error rate covers checker plays only. The scale also comes from eXtreme Gammon's own evaluation, not GNU Backgammon at 2-ply. Read a band as a rough guide, not as a measured human level.

Was it the right risk? Definitions

The core question: when a model takes a risk, is it the right risk? And when it plays safe, was safe right? The thresholds below (0.001, 0.02, the stake bands 0.02 and 0.08) were fixed on 2026-10-23, before any of these numbers was computed.

Hitting rolls
For each legal play, how many of the opponent's 36 rolls hit, read from the numbers the model was shown next to that play.
Minimum risk
The fewest hitting rolls any legal play leaves in that position.
Best safe play
Among the plays that leave the minimum risk, the one the engine rates best.
Stake
The equity difference between the engine's best play and the best safe play: how much the risk call is worth in that position.
Position class
No trade-off: every legal play leaves the same number of hitting rolls; counted and left out of every risk metric. Safety pays: stake below 0.001. Risk pays: stake 0.001 or more.
Took a risk
The chosen play leaves more hitting rolls than the minimum. Otherwise the model played safe.
Risk call
Right risk: took a risk, and the chosen play is at least 0.001 better than the best safe play. Wrong risk: took a risk in a Risk pays position, but not one at least 0.001 better than the best safe play. Needless risk: took a risk in a Safety pays position. Missed risk: played safe in a Risk pays position. Right safety: played safe in a Safety pays position. Every decision with a trade-off gets exactly one risk call.

Metrics per setup (the Risk tab, in this order)

  1. Risk judgement: the average of risks found and right safety, × 100. It does not depend on how many positions of each class a setup met. With both classes present, the Safest program scores exactly 50, the "always safe" line. If a setup has no Risk pays or no Safety pays decisions, it shows "N/A: no <class> decisions"; an empty class is never read as 0 or 100. 95% interval: bootstrap by game.
  2. Risks found: right risks ÷ Risk pays decisions.
  3. Right safety: right safety ÷ Safety pays decisions.
  4. Good risks: right risks ÷ all risks taken.
  5. Missed risks: missed risks ÷ Risk pays decisions (too cautious).
  6. Needless risks: needless risks ÷ Safety pays decisions (too bold).
  7. Wrong risks: wrong risks ÷ Risk pays decisions.
  8. Error rate by risk-call category: each part is 500 × the equity lost in that category ÷ all scored decisions of the setup, the same denominator for all six parts (needless risks, missed risks, wrong risks, right risks, right safety, No trade-off), so they add up exactly to the error rate. A right risk or right safety can still lose equity: the chosen play need not be the engine's best. It says where losses happened, not what caused them.
  9. Decisions with a trade-off, and the counts per class.

Every share shows its count and denominator. Every metric is computed per league; league comparisons are not paired. Setups met different positions, and risk judgement is not adjusted for how hard they were; paired comparisons wait for the shared positions.

Breakdowns

Ability and caution

Results published: ability, caution, aim and the cost-derived SDT reference are on the home page's Risk tab (table and chart).

In plain words: ability is how well a setup tells Risk pays positions from Safety pays positions; caution is how much evidence it needs before it takes a risk. A setup can be able and cautious, able and bold, and so on; the two are read separately. Both use the risk classes and risk calls above, at each margin, for every setup and reference row with decisions.

Reference rows

Shown, never ranked, never champions. Random plays a random legal play; GNUbg weak is the x86-64 build: its noise is shifted, so it overplays gammons (see The anchor).

Safest is not the same thing in every league. In League 1, Safest is the program that actually played the matches: among the plays that leave the fewest hitting rolls it takes the one with the fewest blots, then the largest pip lead, then the play shown first; it never asks the engine. It played a fewest-hitting-rolls play in 2,058 of its 2,058 scored decisions and the engine's best safe play in 1,395 of them. Its right-safety part of the error rate, 8.45 PR, is the equity lost on its 749 right-safety decisions × 500 ÷ all 2,058 of its scored decisions: a share of its error rate, not the average loss of a right-safety decision, which is 23.22 PR. In League 2 and League 3, Safest (computed) is a computed reference: the engine's best safe play on every scored decision of that league. Its numbers are not directly comparable with League 1's played program.

Random also played League 1 only; in the other leagues it is computed as the expectation over all legal plays, so its counts can have decimals. A cell with no decisions shows its count, 0, and no average.

Charts

Risk map: across, the Risk pays decisions without the right risk (missed + wrong risks) as a share of Risk pays decisions; up, needless risks as a share of Safety pays decisions. Risk judgement = 100 − (across + up) / 2, so the corner (0, 0) is risk judgement 100; the faint diagonals mark 50, 60, 70, 80, 90. A point's colour is its share of wrong risks among the Risk pays decisions without the right risk. Error rate by risk-call category: the six parts as a stacked bar. Stake chart: risks found, wrong and missed risks per stake band, and big needless risks. Ability and caution: ability across, caution up, programs and computed reference rows as squares, a short bar at the cost-derived SDT reference where it is valid; under it the limit line: a simple model of the risk call (equal-variance signal detection); setups met different positions. Risk call card: one image per league winner. A downloaded chart PNG carries the site's title and subtitle, the league, the margin, the number of decisions, the results version and the site address, and lists the active filters; it keeps the theme in view. Every table on the site has its own Download PNG: the table as shown (tab, filters, sort, theme) with all its columns and rows, the headings above it and the notes under it, then the same small type, with the rows shown and the decisions behind them.

Robustness: the 0.02 margin

At 2-ply the engine can misjudge close plays by more than 0.001. Every risk metric is therefore computed a second time with 0.02 in every place where 0.001 appears above. The 0.02 results are on the Risk tab under the main table, and here:

League 1

Swipe for more columns →

Margin 0.001 (large) and 0.02 (small)
Model and settingRisk judgementRisks foundRight safetyMissed risksNeedless risksWrong risks
Jev 1.13decision model · API59.50.02: 58.427.0%0.02: 26.8%92.1%0.02: 90.0%65.0%0.02: 64.1%7.9%0.02: 10.0%8.1%0.02: 9.0%
DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release59.30.02: 59.029.1%0.02: 30.1%89.5%0.02: 87.9%63.7%0.02: 62.6%10.5%0.02: 12.1%7.1%0.02: 7.3%
GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs58.80.02: 57.933.0%0.02: 33.1%84.6%0.02: 82.7%58.9%0.02: 58.1%15.4%0.02: 17.3%8.1%0.02: 8.8%
Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs57.90.02: 56.633.4%0.02: 33.1%82.5%0.02: 80.2%56.1%0.02: 55.7%17.5%0.02: 19.8%10.5%0.02: 11.2%
celeris-1-decisiondecision model · API57.80.02: 57.323.9%0.02: 24.1%91.7%0.02: 90.5%70.7%0.02: 69.1%8.3%0.02: 9.5%5.4%0.02: 6.8%
Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark57.60.02: 56.723.5%0.02: 22.8%91.7%0.02: 90.6%70.9%0.02: 70.1%8.3%0.02: 9.4%5.6%0.02: 7.1%
CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs57.10.02: 57.327.3%0.02: 28.2%86.8%0.02: 86.4%64.9%0.02: 63.2%13.2%0.02: 13.6%7.7%0.02: 8.5%
Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs56.90.02: 56.733.1%0.02: 33.7%80.7%0.02: 79.7%57.3%0.02: 55.9%19.3%0.02: 20.3%9.5%0.02: 10.4%
SubQ Snap Previewdecision model · API56.50.02: 55.937.9%0.02: 38.4%75.1%0.02: 73.4%45.8%0.02: 44.1%24.9%0.02: 26.6%16.2%0.02: 17.5%
GPT-6 LunaDecisions APIdecision model · API55.70.02: 55.221.6%0.02: 21.7%89.7%0.02: 88.7%71.5%0.02: 70.6%10.3%0.02: 11.3%6.9%0.02: 7.7%
Winnow-12BNVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs55.20.02: 54.533.7%0.02: 33.6%76.6%0.02: 75.3%53.2%0.02: 51.8%23.4%0.02: 24.7%13.1%0.02: 14.6%
Mercury Decidedecision model · API53.40.02: 53.118.7%0.02: 18.4%88.1%0.02: 87.8%74.9%0.02: 73.9%11.9%0.02: 12.2%6.4%0.02: 7.7%
LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark52.40.02: 52.431.5%0.02: 32.2%73.4%0.02: 72.6%47.3%0.02: 44.9%26.6%0.02: 27.4%21.2%0.02: 22.9%
Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs49.10.02: 47.731.3%0.02: 30.6%66.9%0.02: 64.8%42.7%0.02: 42.3%33.1%0.02: 35.2%26.0%0.02: 27.1%
Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs48.50.02: 48.037.3%0.02: 37.7%59.8%0.02: 58.4%34.7%0.02: 33.4%40.2%0.02: 41.6%28.0%0.02: 28.9%
LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell45.70.02: 44.928.8%0.02: 28.4%62.7%0.02: 61.5%48.5%0.02: 48.4%37.3%0.02: 38.5%22.7%0.02: 23.2%
GNUbg weakprogram · CPU67.10.02: 64.366.6%0.02: 66.1%67.6%0.02: 62.6%10.7%0.02: 9.0%32.4%0.02: 37.4%22.7%0.02: 24.9%
Randomprogram · CPU24.90.02: 23.724.8%0.02: 22.7%25.0%0.02: 24.7%19.2%0.02: 19.1%75.0%0.02: 75.3%56.0%0.02: 58.2%
Safestprogram · CPU50.00.02: 50.00.0%0.02: 0.0%100.0%0.02: 100.0%100.0%0.02: 100.0%0.0%0.02: 0.0%0.0%0.02: 0.0%

League 2

Swipe for more columns →

Margin 0.001 (large) and 0.02 (small)
Model and settingRisk judgementRisks foundRight safetyMissed risksNeedless risksWrong risks
DeepSeek-V4.1-Flashlow · API + NVIDIA RTX PRO 6000 Blackwell · first releasechat model, answer-letter probabilities · API69.00.02: 70.245.0%0.02: 49.2%93.0%0.02: 91.2%49.0%0.02: 44.4%7.0%0.02: 8.8%6.0%0.02: 6.4%
DeepSeek-V4.1-Flashlow · API · 1017chat model, answer-letter probabilities · API68.50.02: 68.942.5%0.02: 44.8%94.5%0.02: 93.0%52.1%0.02: 47.9%5.5%0.02: 7.0%5.4%0.02: 7.3%
Claude Sonnet 5.5highchat model, one letter · subscription68.20.02: 69.041.8%0.02: 44.0%94.6%0.02: 94.0%55.5%0.02: 52.1%5.4%0.02: 6.0%2.7%0.02: 3.9%
Claude Fable 5.1low67.80.02: 68.541.0%0.02: 43.7%94.6%0.02: 93.2%53.8%0.02: 49.4%5.4%0.02: 6.8%5.2%0.02: 6.9%
Grok 4.7low67.70.02: 67.042.3%0.02: 42.8%93.0%0.02: 91.1%49.9%0.02: 47.2%7.0%0.02: 8.9%7.7%0.02: 9.9%
Gemini 3.8 Flashhighchat model, one letter · subscription66.90.02: 67.636.1%0.02: 39.2%97.6%0.02: 96.1%61.6%0.02: 56.9%2.4%0.02: 3.9%2.3%0.02: 3.9%
Gemini 3.8 Flashlowchat model, one letter · subscription65.10.02: 66.833.8%0.02: 37.6%96.5%0.02: 96.0%63.2%0.02: 58.3%3.5%0.02: 4.0%3.0%0.02: 4.1%
Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs64.50.02: 65.631.9%0.02: 34.7%97.1%0.02: 96.5%64.9%0.02: 61.3%2.9%0.02: 3.5%3.3%0.02: 3.9%
d1-ultra64.00.02: 63.042.0%0.02: 41.8%86.0%0.02: 84.2%49.6%0.02: 47.4%14.0%0.02: 15.8%8.4%0.02: 10.7%
Claude Sonnet 5.5lowchat model, one letter · subscription63.50.02: 64.230.9%0.02: 33.1%96.2%0.02: 95.2%65.5%0.02: 62.5%3.8%0.02: 4.8%3.6%0.02: 4.4%
Gemma 4 31Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs62.60.02: 63.233.2%0.02: 35.5%92.1%0.02: 90.9%58.4%0.02: 55.8%7.9%0.02: 9.1%8.4%0.02: 8.8%
Qwen3.8-27Blow · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs62.00.02: 61.233.8%0.02: 34.4%90.2%0.02: 87.9%58.3%0.02: 57.3%9.8%0.02: 12.1%7.8%0.02: 8.3%
GLM-5.3lowchat model, answer-letter probabilities · API61.70.02: 61.630.7%0.02: 31.8%92.7%0.02: 91.4%63.2%0.02: 60.6%7.3%0.02: 8.6%6.1%0.02: 7.6%
GLiDEAPI60.40.02: 60.536.9%0.02: 38.4%84.0%0.02: 82.5%51.0%0.02: 48.9%16.0%0.02: 17.5%12.1%0.02: 12.6%
Qwen3.8 Flashoffchat model, answer-letter probabilities · API59.50.02: 59.334.4%0.02: 35.5%84.6%0.02: 83.0%52.7%0.02: 50.6%15.4%0.02: 17.0%12.9%0.02: 13.9%
Jev 1.1459.30.02: 59.229.1%0.02: 29.9%89.4%0.02: 88.5%63.4%0.02: 61.4%10.6%0.02: 11.5%7.4%0.02: 8.8%
DeepSeek-V4.1-Flashoff · API · 1017chat model, answer-letter probabilities · API58.50.02: 58.825.4%0.02: 26.5%91.7%0.02: 91.1%68.3%0.02: 66.1%8.3%0.02: 8.9%6.3%0.02: 7.4%
ClefNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark57.50.02: 57.639.5%0.02: 40.6%75.6%0.02: 74.6%45.2%0.02: 42.8%24.4%0.02: 25.4%15.3%0.02: 16.6%
GNUbg weakprogram · CPU62.30.02: 61.762.7%0.02: 64.8%61.9%0.02: 58.5%14.4%0.02: 10.8%38.1%0.02: 41.5%22.9%0.02: 24.4%
Random (computed)reference27.00.02: 25.925.1%0.02: 23.7%28.8%0.02: 28.1%16.7%0.02: 16.1%71.2%0.02: 71.9%58.2%0.02: 60.2%
Safest (computed)reference50.00.02: 50.00.0%0.02: 0.0%100.0%0.02: 100.0%100.0%0.02: 100.0%0.0%0.02: 0.0%0.0%0.02: 0.0%

League 3

Swipe for more columns →

Margin 0.001 (large) and 0.02 (small)
Model and settingRisk judgementRisks foundRight safetyMissed risksNeedless risksWrong risks
GPT-6 Astralowchat model, one letter · subscription75.80.02: 74.957.2%0.02: 58.1%94.4%0.02: 91.8%39.2%0.02: 37.8%5.6%0.02: 8.2%3.5%0.02: 4.2%
GPT-6 Astramediumchat model, one letter · subscription74.70.02: 74.954.9%0.02: 56.5%94.4%0.02: 93.2%42.4%0.02: 37.9%5.6%0.02: 6.8%2.7%0.02: 5.5%
Gemini 4 Argonhighchat model, one letter · subscription74.10.02: 75.950.8%0.02: 56.4%97.4%0.02: 95.3%45.6%0.02: 40.4%2.6%0.02: 4.7%3.6%0.02: 3.1%
GPT-6.1 Sollow72.70.02: 74.251.7%0.02: 56.2%93.8%0.02: 92.2%44.1%0.02: 38.6%6.2%0.02: 7.8%4.2%0.02: 5.2%
Claude Fable 5.1highchat model, one letter · subscription72.30.02: 72.150.7%0.02: 52.7%94.0%0.02: 91.6%42.2%0.02: 38.6%6.0%0.02: 8.4%7.1%0.02: 8.6%
GPT-6.1 Solhigh72.20.02: 72.450.6%0.02: 52.4%93.7%0.02: 92.5%44.4%0.02: 38.4%6.3%0.02: 7.5%5.0%0.02: 9.2%
Claude Opus 5.5highchat model, one letter · subscription70.00.02: 70.843.4%0.02: 46.3%96.6%0.02: 95.3%51.4%0.02: 47.5%3.4%0.02: 4.7%5.2%0.02: 6.2%
Gemini 4 Argonlowchat model, one letter · subscription69.20.02: 68.747.8%0.02: 49.6%90.7%0.02: 87.8%46.0%0.02: 44.1%9.3%0.02: 12.2%6.2%0.02: 6.3%
Grok 4.7highchat model, one letter · subscription68.50.02: 67.944.0%0.02: 44.6%93.0%0.02: 91.1%50.5%0.02: 47.8%7.0%0.02: 8.9%5.6%0.02: 7.6%
Claude Opus 5.5lowchat model, one letter · subscription66.90.02: 67.140.1%0.02: 41.9%93.8%0.02: 92.4%55.8%0.02: 51.9%6.2%0.02: 7.6%4.1%0.02: 6.2%
Kimi K3.1 Prooffchat model, one letter · API65.90.02: 66.742.4%0.02: 44.7%89.5%0.02: 88.7%49.0%0.02: 45.0%10.5%0.02: 11.3%8.6%0.02: 10.3%
GNUbg weakprogram · CPU62.10.02: 60.058.8%0.02: 58.1%65.5%0.02: 61.8%16.7%0.02: 15.3%34.5%0.02: 38.2%24.5%0.02: 26.6%
Random (computed)reference27.60.02: 26.425.1%0.02: 23.4%30.1%0.02: 29.4%17.5%0.02: 16.9%69.9%0.02: 70.6%57.4%0.02: 59.7%
Safest (computed)reference50.00.02: 50.00.0%0.02: 0.0%100.0%0.02: 100.0%100.0%0.02: 100.0%0.0%0.02: 0.0%0.0%0.02: 0.0%

Plays without a 2-ply evaluation

All 92,888 of the 92,888 league decisions have a 2-ply evaluation of every legal play, so no play needed a CPU evaluation.

Limit: this measures one kind of risk, being hit. Gammon risk and priming are not in it.

Statistics and verdicts

Verdicts

For reasoning (low against high, off against on) and any comparison of A against B on the same positions:

VerdictRule
High helpsThe paired 95% interval of the error difference (high minus low, or B minus A) lies entirely below zero.
High hurtsThe interval lies entirely above zero.
No clear benefitThe interval includes zero. Never read as "the same", "level" or "gains nothing".
Depends on the positionPhase-level effects point in different directions, each with an interval that excludes zero.

Every phase-level verdict is exploratory. "Worse moves" and "more time or cost with no shown gain" are kept as two separate findings and never merged into "high is bad".

Fixed thresholds

These values were fixed on 2026-10-21, before any number on this site was computed, and are not changed after seeing results. Equity values are in points per game.

ThresholdValueMeaning
Error-rate scale500Error rate = this number × mean equity lost per decision.
Engine's best play0.001The chosen play lost less than this much equity.
Blunder0.08The chosen play lost more than this much equity.
Badly wrong0.08The chosen play lost more than this much equity (used in "confident and badly wrong").
Very confident0.9The setup gave its chosen play at least this probability.
Time budget30Seconds of request time per decision; slower decisions count as over the budget.
Minimum sample200Fewer decisions than this in a cell: "Too few decisions" instead of a value.
Near-tie0.02The two best plays are closer than this in equity.
Gap groups0.02, 0.08Limits of the groups by equity gap between the two best plays.
Option groups2–5, 6–10, 11–15, 16–25, 26 or moreGroups by number of legal plays.
Tail error0.95Quantile of the loss per decision reported as tail error.
Worst decisions0.05Share of a setup's decisions, the ones with the largest loss, whose mean loss is reported.
Bootstrap resamples2000Resamples for every interval.
Bootstrap seed20261021Fixed random seed for every interval.
Interval level0.95Coverage of every interval.
Risk thresholds fixed on2026-10-23Date the risk thresholds were fixed, before any risk number was computed.
Risk margin0.001B6.1: stake below it is Safety pays, at or above it Risk pays; a taken risk is right when the chosen play is at least this much better than the best safe play.
Robustness margin0.02B6.6: every risk metric computed again with this margin in place of the risk margin.
Stake bands0.02, 0.08Risk pays decisions by stake: small below the first value, medium up to the second, big above it.
Confidence cutoffs0.9, 0.99Confidence tab: decisions kept at or above each probability.
Held-out splits300Random halves of the games for the cutoff chosen on one half and tested on the other.
Held-out cutoff ruleOn one half of the games (a game's decisions together), the lowest probability cutoff whose kept decisions (probability at or above it) hold no blunder; scored on the other half: share kept and blunder rate among kept, averaged over cutoff_split_runs seeded splits.How that cutoff is chosen.
Surest share0.25Share of a setup's decisions with the highest confidence, for the error rate in the surest quarter.
Needless-risk bands0.02, 0.08Needless risks by the equity the chosen play lost: small, medium, big.
Risk map diagonals50, 60, 70, 80, 90Risk judgement values drawn as faint diagonals on the risk map.
Hero board stake0.05, 0.15The home page board shows a Risk pays decision with a stake in this range: the one closest to the median stake of those decisions in the latest finished league.

Status labels

Every section and every metric shows one status. Where the data does not exist, the status is shown, never a placeholder number.

StatusMeaning
PublishedComputed from finished leagues or complete shared-position runs.
ProvisionalFrom a league that is still running. The numbers will change. They appear only in that league's own tables and enter no cross-league ranking, chart or finding until the league finishes.
Earlier versionFrom an earlier version of the benchmark. Never ranked or charted together with current numbers.
Analysis pendingThe data exists, but the analysis has not been run on it yet.
Not measuredThis was not measured.
N/ADoes not apply to this setup, for example confidence metrics for a setup that returns only a choice.

League status

LeagueMeaning
Running (done / total)Matches are still being played; the numbers are provisional.
FinishedEvery match has been played.
StoppedThe league was ended before all its matches were played; its finished matches are kept.

An average below the minimum sample shows "Too few decisions".

Versions and play order

Numbers from different benchmark versions or prompt formats are never put in one ranking or chart. Every number carries its version.

Earlier version Plays were listed in the runner's order, which tends to put the best play early. Re-asked with the same plays in a random order on 1,000 positions, Gemini 4 Argon's error rate changed by +1.9 (95% interval −0.2 to 4.2), so a help from the order is possible but not proven. The current version shuffles the plays for every setup.

The earlier version also scored differently: every legal play at 0-ply, then the five best and the chosen play again at 2-ply. The current version evaluates every legal play at 2-ply.

Changelog. 28 Oct 2026: setups renamed to the reasoning effort they sent. 5 setups changed from (off) to (low). No score changed. 29 Oct 2026: the Hype Cup and the Nackgammon knockout on Leagues, and every match on Matches. Match results only: no score changed. 6 Nov 2026: seven late entries to the original League 1 table on Leagues (Late entries): SubQ Snap 1.1, Elephant-1 and Mammoth-1 (played 2 Nov), Drex Max v1.1, Drex 2.0, Drex MaxNVIDIA RTX PRO 6000 Blackwell and Drex 2.0NVIDIA RTX PRO 6000 Blackwell (played 6 Nov; their pairs with GPT-6 LunaDecisions API not played, its API out of credit), each with a page of its own; the late-entry table gains the engine's best play, median seconds and $ per 1,000 decisions, with each price's label, source and date read. Late entries count in no standings table and no other setup's numbers: no score changed. 7 Nov 2026: REAX 1.3, a late entry to the original League 1 table on Leagues (Late entries), played on our GPU on 7 Nov (its pair with GPT-6 LunaDecisions API not played, its API out of credit), with a page of its own; no list price is published for it, so it has no cost figure. No score changed. 11 Nov 2026: KnowLine-31B-Gen1.5, a late entry to the original League 1 table on Leagues (Late entries), played on our GPU (NVIDIA DGX Spark) on 10–11 Nov (its pairs with GPT-6 LunaDecisions API, its API out of credit, and Mercury Decide, no longer served on OpenRouter, not played; its first 4 matches, played at once, voided and replayed one at a time), with a page of its own; no list price is published for it, so it has no cost figure. No score changed. 11 Nov 2026: Muse Embermax, a late entry to the original League 1 table on Leagues (Late entries), played through OpenRouter (pinned to Meta, reasoning effort max) on 10–11 Nov (its pairs with GPT-6 Luna (Decisions API), its API out of credit, and Mercury Decide, no longer served on OpenRouter, not played), with a page of its own; its cost is its billed tokens at Meta's list price. No score changed. 11 Nov 2026: Gemini Decision 1, a late entry to the original League 1 table on Leagues (Late entries), played through OpenRouter (pinned to Google AI Studio) on 11 Nov (its pairs with GPT-6 LunaDecisions API, its API out of credit, and Mercury Decide, no longer served on OpenRouter, not played; runner dab6da5 without a probability-map check, then 38024d6 with map-check v2, whose sum rule (within 1e-6) applies to this entry only), with a page of its own; its cost is its billed cost per decision. No score changed. 12 Nov 2026: the late entries' and repeats' pairs with GPT-6 Luna (Decisions API) and Mercury Decide are outside the requested scope (Ümit, 12 Nov): no longer work to close. They stay unplayed and are labelled so beside the reason they were not played, never a score or a loss; a late entry whose only unplayed pairs are these is no longer marked Provisional, since none of its pairs is left to play. No number changed. 12 Nov 2026: Limits (Runner commits) states how many original league matches were started from a runner checkout with uncommitted changes (harness_dirty): provenance only. No number changed. 14 Nov 2026: fourteen small decision models, late entries to the original League 1 table on Leagues (Late entries): GLiDE 2B, GLiDE 0.8B, GLiDE 0.4B, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, KnowLine 4B Gen5.5, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, OneJev1.6-0.8B, GLiClass v4 Edge, Cerno 9B v1.2, Turion Jev 31B and Sieve 31B v3, each served on our DGX Spark, played 11–14 Nov in two stages (its hosted and program opponents, then the 8 setups we run ourselves; its pairs with GPT-6 LunaDecisions API and Mercury Decide not played, outside the requested scope), each with a page of its own; GLiDE 0.4B gets at most 74 options per request and GLiClass v4 Edge at most 218, longer play lists split into groups by the league's rule; their cost is an estimate from their time on the DGX Spark (GPU time, at most). No score changed. 14 Nov 2026: d1-2BNVIDIA RTX PRO 6000 Blackwell, a late entry to the original League 1 table on Leagues (Late entries), played on one RTX PRO 6000 Blackwell of ours on 14 Nov (its pairs with GPT-6 LunaDecisions API and Mercury Decide not played, outside the requested scope), with a page of its own; its cost is an estimate from its time on that card (GPU time, at most). No score changed. 14 Nov 2026: d1-Omni-2BNVIDIA RTX PRO 6000 Blackwell, d1-2B with audio added, a late entry to the original League 1 table on Leagues (Late entries), played text only on the same RTX PRO 6000 Blackwell after d1-2B on 14 Nov (its pairs with GPT-6 LunaDecisions API and Mercury Decide not played, outside the requested scope), with a page of its own; its cost is an estimate from its time on that card (GPU time, at most). No score changed.

Confidence

A cutoff chosen on half of the games

On one half of the games (a game's decisions together), the lowest probability cutoff whose kept decisions (probability at or above it) hold no blunder; scored on the other half: share kept and blunder rate among kept, averaged over cutoff_split_runs seeded splits. 300 splits, by game.

League 1

Swipe for more columns →

Means over the splits of the held-out half
Model and settingShare kept on the other halfBlunders among keptSplits with no cutoffDecisions with probabilities
Jev 1.13decision model · API2.1%3.3%0 of 3001,911
GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs4.6%1.7%0 of 3002,002
DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release4.3%1.5%0 of 3001,589
celeris-1-decisiondecision model · API1.4%8.3%147 of 3001,922
Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs3.6%1.9%0 of 3001,822
CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs0.8%7.3%36 of 3001,893
Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs4.1%1.9%0 of 3001,826
GPT-6 LunaDecisions APIdecision model · API0.7%11.1%46 of 3001,962
SubQ Snap Previewdecision model · API1.5%3.8%0 of 3002,199
Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark0.9%6.4%0 of 3001,825
LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkN/AN/A300 of 3001,964
Mercury Decidedecision model · API1.0%7.1%0 of 3001,853
Winnow-12BNVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs1.7%4.0%0 of 3001,923
Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs2.8%3.1%0 of 3001,782
LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell1.6%4.3%0 of 3001,770
Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs2.3%2.0%0 of 3002,002

League 2

Swipe for more columns →

Means over the splits of the held-out half
Model and settingShare kept on the other halfBlunders among keptSplits with no cutoffDecisions with probabilities
DeepSeek-V4.1-Flashlow · API · 1017chat model, answer-letter probabilities · APIN/AN/A300 of 3001,965
DeepSeek-V4.1-Flashlow · API + NVIDIA RTX PRO 6000 Blackwell · first releasechat model, answer-letter probabilities · API1.3%14.4%151 of 3001,904
Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs9.4%11.0%299 of 3001,870
Gemma 4 31Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs1.6%12.9%150 of 3001,664
Qwen3.8-27Blow · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs0.5%74.2%200 of 3001,891
d1-ultra2.2%2.8%0 of 3002,070
GLM-5.3lowchat model, answer-letter probabilities · API0.6%8.1%0 of 3001,785
Jev 1.142.6%1.8%0 of 3001,910
GLiDEAPI4.0%1.2%0 of 3002,157
Qwen3.8 Flashoffchat model, answer-letter probabilities · API1.5%4.2%0 of 3001,856
DeepSeek-V4.1-Flashoff · API · 1017chat model, answer-letter probabilities · API2.5%3.2%0 of 3001,865
ClefNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark2.8%2.6%0 of 3001,956

Re-read, refit, bar and distributions (offline, from recorded decisions)

Results published: the re-read, calibration and distributions on the Confidence tab, the minimum bar on the Cost tab.

Four analyses of the decisions already recorded; no model was asked again. Every rule below (thresholds, estimators, exclusions, degenerate cases, splits and the hypothesis) was fixed and committed on 2026-10-24, before any of these numbers was computed. Splits, folds and resamples use one map of blocks per league for every setup: a block is the two games of a round, the second with seats swapped and the same dice (checked on the data: same rolls, and no two matches share a dice seed).

Which probabilities are complete

A decision enters these analyses only if its option probabilities are verifiably complete. Each one is checked against the raw response: the response must reproduce the stored probabilities, and every legal play must have its own probability in it: every play for a decision model, every option letter among the returned top log-probabilities for a letter readout. A letter missing there was filled with 0 and the rest renormalised by the runner; such a decision is sparse, whatever the length or sum of its vector. Decisions split into groups, choice-only answers, invalid answers, decisions without a raw response and decisions whose stored vector is not reproduced are counted separately and left out (Confidence tab, per setup).

Re-read risk call (re-read from the model's probabilities, not played)

Hypothesis, recorded before computing: gains will be largest for setups with high ability whose caution is well above the cost-derived SDT reference. It is a hypothesis, not a claim that the reference is optimal. Predictor, from the training half only: d′ × (c − c*); undefined where c* is N/A or d′ is below 0.2; a setup needs it in 150 splits. Test: Spearman rank correlation with the held-out change in risk judgement across setups, leagues pooled; supported only if it is positive and its 95% interval is above 0. Result: ρ = −0.15 (95% interval −0.46 to +0.03, 200 resamples) over 24 setups: not supported.

Calibration as shipped and after a refit

Two events, never mixed: the chosen play (its probability against "the chosen play is the engine's best") and the risk class (P(risk) against "the position is Risk pays", decisions with a trade-off). The refit is Platt scaling (a logistic curve on the log-odds), cross-fitted over 5 folds of blocks: no decision is refitted by a curve that saw its own games. A fold whose other folds hold one outcome only, or whose fit does not converge, is left out of both before and after. Brier score and a reliability diagram in 10 fixed bins, before and after, on the same decisions. The refit is not used to choose anything else.

Minimum bar, then cheapest

Bar: risk judgement at least 60 and error rate at most 35 PR (equity lost, not a share of mistakes). In each split, the model setups that clear the bar on the training half qualify, and the cheapest of them with a price on every training decision is chosen; it is then checked on the other half. Shown: how often each setup qualified, was chosen and cleared the bar on the held-out half; splits where none qualified, or only unpriced setups did. An unpriced setup is never called cheapest; programs are left out (Cost tab).

Confidence on right and wrong decisions

The chosen play's probability (never P(risk)), complete decisions only, in 10 fixed bins: decisions that were the engine's best play against the rest, and right risks against wrong and needless risks (0.001 margin).

Close choices

Stage A: results published (Confidence tab, model pages, Every decision filter). Stage B: five teams, results published. Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark stopped, not reported. (Leagues page, Stage B: five teams).

A decision is a close choice when the second most likely play has at least half the probability of the first (internally: split). Probabilities over plays, not sampled reasoning. The rules below were fixed and committed on 2026-10-24 before any close-choice number was computed. Inspired by "Can LLMs Anticipate Their Cruxes?". They describe the probabilities a setup gave; they say nothing about how it reasoned or why it chose.

Stage A: from recorded decisions

Stage B: five teams: does a setup know where it hesitates?

Stage B: five teams, results published. Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark stopped, not reported. The 5 subscription teams ran on 2026-10-25, 10:23–11:02 UTC; their results are on the Leagues page. Scope. Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark was started but not completed: two reboots and a GPU fault interrupted it, and it was stopped after 311 of 500 play calls on 2026-10-25. Its partial data is not reported. (Ümit's decision, 2026-10-25, 16:05 UTC.) The positions and the scoring rules were fixed on 2026-10-24, before any stage B call; the run conditions below on 2026-10-25.

Nackgammon paired test

Results on the Nackgammon page. These results hold for these rosters only; each group is the setups in the run from that original league. The text below is §10 of the Method specification of 24 Oct 2026 as committed, with its dated notes and amendments (one local database path replaced by its description); §10.14 (groups by original league) says in (e) what had been opened before it was written, corrected in (f); (g) adds the Cup-roster setups after the results had been read, and (h) the exploratory Gemini 4 Argon setup and the output-limit rule. The spec keeps the setups' names from before the relabel of 28 Oct 2026; the results show the current names: Claude Fable 5.1 (on) is Claude Fable 5.1high, Grok 4.7 (on) is Grok 4.7high, GPT-6 Astra (on) is GPT-6 Astramedium, Gemini 4 Argon (off) and (on) are Gemini 4 Argonlow and (high).

10. Nackgammon paired test (amendment dated 2026-10-27, Can; written and committed before the position sets were built and before any model call)

Brief: mailbox/Nackgammon and Terra - brief 2026-10-27.md (with Can's correction of the same day to the Bonferroni rule). Code: tools/v3/nack_paired.py (sets, scores, Random) and tools/v3/selfplay.py (games, unchanged). Freeze: freeze/nack-paired.json. Files: positions/nack-paired/. This section is never changed after the first model call; a bug found later is fixed in a dated addendum below it, the original text kept.

10.0 What it replaces (superseded text kept)

This test replaces the Nackgammon part of the shared positions (A12's 500 Nackgammon positions). The 200-position opening control stays in Method but is not part of this run. The old text, superseded 2026-10-27 by Can (this section), kept for the record:

10.1 Question and hypothesis

Nackgammon has the rules of backgammon; only the start differs: each side starts with 2 checkers on its 24-point, 2 on its 23-point, 4 on its midpoint (13), 3 on its 8-point and 4 on its 6-point. The test measures how much a new starting setup changes a setup's (model, settings and prompt) decision quality and risk calls: how well a whole setup transfers. A larger drop is consistent with relying on familiar positions but is not proof, because Nackgammon positions also differ in structure. "Performance kept" is a valid result.

Hypothesis (a): error rates rise from standard to Nackgammon positions, most in the opening and less later in the game, and the rise differs between setups. It is examined through each setup's drop and label (10.8), the per-phase drops (descriptive, 10.9) and the primary comparisons (10.10).

Error rate (PR) = 500 × mean GNUbg 1.08 2-ply cubeless equity loss of the chosen play against the best legal play. Drop = PR on the Nackgammon positions − PR on their paired standard positions, over the same pairs. Positive = worse on Nackgammon.

10.2 Games, candidates and the one-position-per-game rule

10.3 Phase definitions (tools/v3/classify.py, unchanged; first rule that matches)

  1. opening: turns 1 to 6 (turn 1 = the first move of the game, both sides' turns counted).
  2. bear-off: any of GNUbg's bear-off classes.
  3. race: GNUbg's race class.
  4. holding: a contact position (GNUbg contact or crashed class) where one side has escaped all its back checkers (none on the bar or on its points 14 to 24) and the other keeps one high anchor: 2 or more checkers on its 18-, 20- or 21-point and no other checkers on its points 18 to 24 (nor on the bar).
  5. contact: everything else, including GNUbg's crashed class.

10.4 Game-plan tags (classify.tag, unchanged)

For contact and holding positions only; opening, race and bear-off positions carry no tag and are reported under their phase. First rule that matches, checked for both sides: 1. blitz: a checker on the bar against a home board with 3 or more made points; 2. back game: 2 or more anchors (2+ checkers) on the side's points 18 to 24; 3. holding: the holding rule of 10.3; 4. priming: 4 or more made points in a row; 5. low-anchor: the side's only anchor in the opponent's home board (its points 19 to 24) is on its 22-, 23- or 24-point; 6. other. Note fixed now: a Nackgammon side that still holds its starting 23- and 24-points meets the back-game rule; the tags are reported as the rule gives them. Tags are not a matching variable; the tag mix of both sets is reported.

10.5 Position classes (existing rule, analysis.play_call, margin 0.001)

Hitting rolls of a play = describe.hitting_rolls (= Board::hitting_rolls, the number the model is shown). A position is No trade-off when every legal play leaves the same number of hitting rolls. Otherwise its stake = the 2-ply loss of the best safe play (fewest hitting rolls) against the best play: Risk pays when the stake ≥ 0.001, else Safety pays. Risk calls of an answer: right risk, wrong risk, needless risk, missed risk, right safety (as on the site).

10.6 Engine scoring

GNUbg 1.08.003, engine.PLY2 (2-ply, cubeless money equity, deterministic, prune on), every legal play of every position, through the same code as tools/v3/score.py evaluate. Gap = 2-ply equity of the best play − the second best. Stored with the sets (scores-2ply.jsonl). Random's error on a position = the mean loss over all its legal plays.

10.7 Matching, pairs per phase and class, tolerances

Pairs per phase and class (every pair has the same phase and class on both sides):

PhaseMain: Risk paysMain: Safety paysMain: No trade-offMain totalReserve: Risk paysReserve: Safety paysReserve: No trade-offReserve total
opening30300604545090
contact2424126036361890
holding12126301818945
race002525003838
bear-off002525003737
total6666682009999102300

Tolerances (a standard partner must meet all, against the Nackgammon position):

Procedure (deterministic; h(x) = sha256 of "nack-paired-2026-10-27:" + x):

  1. Nackgammon candidates are taken in order of h(candidate key) (key = source game ID + turn). A candidate is taken when its (phase, class) cell still has quota, its game and its state are unused, and some unused standard candidate (game and state unused) meets every tolerance. Its partner is the one with the smallest distance d = |Δn| / tol_n + |Δg| / tol_g; ties by h(partner key). Both games and both states are then used.
  2. Main set first, to its quotas; then the reserve with the same procedure on what remains.
  3. Shortfall (only if the pool cannot fill a cell): the missing pairs go to the other trade-off class of the same phase, then to No trade-off of the same phase, then to the same class in contact; the procedure runs again for the moved quota. Every move is logged in the freeze and reported. The tolerances are never loosened.
  4. Order: pair IDs np-0001…np-0200 (main) and np-0201…np-0500 (reserve) in order of h("order:" + Nackgammon key) within each set. Position IDs npn-NNNN (Nackgammon) and nps-NNNN (standard). Every setup answers the main 400 positions; an extended setup answers the reserve 600 positions, all of them, in pair-ID order, the same for every extended setup.
  5. Option order: order.option_order with seeds.json's option-order seed, as every v3 set.

Balance reported for both sets (main, reserve): per phase and overall, mean, median and standardised mean difference of n, gap, turn, pip count of the mover, the pip lead, Random's error and the stake; the class and GNUbg-class mix; the tag mix.

10.8 Setups, order, invalid answers, stopping rule, labels

Setups (each runs once, even when it is both in League 1 and in the Cup). Programs are yardsticks and belong to no group: GNUbg weak (GNUbg 1.08 Beginner, its stored League settings, answered through the harness) and Random (no calls: its error is the exact mean loss over all legal plays from the 2-ply values). Safest is not in the run.

Every setup runs with its stored settings (players.settings). Order: GNUbg weak and the decision models first, then the Cup teams not in League 1 (Gemini 3.8 Flashlow, Claude Sonnet 5.5low, GPT-6.1 Sollow, GPT-6 Astra (on), GPT-6.1 Solhigh), then Terra if it runs, and Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark last (the slowest; it is one of the 8).

Prompts. v3 is the leagues' prompt bg-v2 (leagues.prompt_version of Leagues 1–3). Both sets use v3 without the standard opening table, and the two prompts differ only in the line naming the starting setup: prompts/bg-v3-std-notable.md (standard side, sha256 1a145a09…, prompt version bg-v3-std-notable) and prompts/bg-v3-nack.md (Nackgammon side, sha256 13c744f0…, bg-v3-nack). Against bg-v2 the standard side drops the "Opening rolls" section and puts the setup sentence on its own line, naming the standard starting position. The existing nack-v2 is not used here: it states the standard layout. The runner test versions_differ_only_where_designed checks that the two rendered states differ in exactly that line. Each row carries its prompt; --prompt-version is never passed. Answers go to a separate SQLite file, not v3.db.

Invalid or missing answers (after the usual retries; also a timed-out or never-stored answer) are never dropped. Primary rule: such an answer scores the position's Random error (mean loss over all legal plays); in the risk analysis it counts with Random's weights (each legal play 1/n). Sensitivity checks, reported next to the primary result: (i) score it with the worst legal play's loss; (ii) leave its pair out for that setup. The count per setup and set is reported.

Intervals: paired bootstrap over pairs, 10,000 resamples, percentile intervals; the resample indices are shared by all setups (numpy default_rng, seed = the first 8 bytes of h("bootstrap:<look>") as an integer), so group means and differences come from the same resamples. Bootstrap intervals are approximate.

Per-setup stopping rule (b), two looks. First look after the 200 main pairs: the setup's drop with a 97.5% interval [lo, hi]. The setup is extended with the 300 reserve pairs only when lo ≤ 0 ≤ hi and (lo ≤ −5 or hi ≥ +5) PR. Examples: [−2, 6] is extended; [−2, 3] and [3, 9] are not. Final look: an extended setup's drop on all 500 pairs, 97.5% interval; a setup not extended keeps its first-look interval as final. Splitting the error over the two looks targets about 95% coverage overall; this is approximate. Every setup's final 97.5% interval is reported. GNUbg weak follows the same rule. Random needs no calls: it is reported on the 200 and on all 500 pairs.

Outcome label (c), from the final 97.5% interval: kept when −5 ≤ lo and hi ≤ +5; worse when lo > 0 and not kept; better when hi < 0 and not kept; unclear otherwise. At the first look "unclear" is exactly what is extended.

Results per setup: standard PR, Nackgammon PR and the drop, always shown together, never a ranking by drop alone; by phase and by game-plan tag; cost per decision (from the run's logs) alongside. Extensions refine each setup's own estimate and are reported per setup, separate from the primary comparisons. Chart: each setup's standard PR (x) against its Nackgammon PR (y), with GNUbg weak and Random marked and the diagonal drawn.

10.9 Descriptive by phase (part of hypothesis (a), not a test)

Per setup and per group: the drop by phase on the main 200 pairs, with 95% bootstrap intervals, and opening against the other four phases pooled. Labelled descriptive.

10.10 Primary comparisons (main 200 pairs only, which every setup answered)

Each is computed directly as A's drop − B's drop (group drop = mean of its setups' drops, each setup weighted equally), with a paired bootstrap interval over pairs (10.8). Bonferroni (Can's correction, 2026-10-27): the 5% error is divided by M; each interval's confidence level is 1 − 0.05/M. Both groups' (or setups') standard and Nackgammon PR are shown next to every difference. Scope, printed wherever group results appear: these results hold for these rosters only; League 1 is the full league; the other two groups are the Cup's selected top setups.

The comparisons (M = 5):

  1. decision models − workhorses;
  2. decision models − frontier;
  3. workhorses − frontier;
  4. effort pair LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark − LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell (both League 1 decision models);
  5. effort pair GPT-6.1 Solhigh − GPT-6.1 Sollow (both Cup teams).

No other model is in the run at two settings: Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, GPT-6 Astra (on), DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release, Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, Gemini 3.8 Flashlow and Claude Sonnet 5.5low run at one setting only. M = 5 with Terra and without Terra: GPT-6.1 Terra runs at one setting (low) and is a different model from GPT-6.1 Sol, so it creates or completes no effort pair; it only joins the workhorse group, which changes comparisons 1 and 3, not M. Level: 1 − 0.05/5 = 99% for every primary interval, whether or not Terra's go is given. M is final now, before the first model call.

10.11 Secondary analyses (fixed now, labelled secondary, no extra model calls)

  1. Kept advantage over random play: K = (Random's Nackgammon PR − setup's Nackgammon PR) ÷ (Random's standard PR − setup's standard PR), main 200 pairs (and all 500 for extended setups), 95% bootstrap interval. 1 = keeps its whole advantage over random play; 0 = falls to random. Flagged "unstable" when the setup's standard advantage over random is below 20 PR or its 95% interval reaches 0.
  2. Ranking: Spearman correlation across the model setups (yardsticks left out) between standard and Nackgammon PR on the main 200 pairs, 95% bootstrap interval over pairs; the five largest rank changes by name (ties by name).
  3. Risk: every position classed by 10.5 and every answer's risk call. Per setup and set: ability d′ and caution c (§2 rules: log-linear correction, missing-class N/A) and their change from standard to Nackgammon, with 95% paired bootstrap intervals over pairs. The number of positions per class and set is reported; a class with fewer than 50 positions in an analysis is called thin and is not read into.

Everything else (for example GPT-6.1 Terra against GPT-6.1 Sollow) is exploratory and labelled so.

10.12 Engine check (no model calls; engine time only)

10.13 Release

A release bundle is prepared (positions with source game IDs, pairs, matching rules, prompts, settings, raw answers, scores and analysis code) and not published before Can's go.

10 addendum (2026-10-27, 18:05 UTC, Can's answers; written and committed before any model call of this test)

10.14 Groups by original league (amendment dated 2026-10-27, Can; replaces the group definitions in 10.8 and 10.10)

Why: the Hype Cup is now built from the original-league tables (mailbox/Original-league tables - what Can found 2026-10-27.md), not from the final ladder tables, so the groups follow the same principle.

Clear risk (preliminary)

Preliminary / partial. The clear-risk scan (decisions where GNU Backgammon finds the risky or the safe play clearly better), its checks, a rollout check of a sample and a partial quiz of five models (two or four plays shown, not league play) are on the Clear risk page, each with its definitions and settings.

Cost

Cost per 1,000 decisions carries one of three labels. They are never mixed in one figure or ranking.

API price estimate
The provider's list price, on a stated date, × the tokens its API reported (where the API records no cost per answer, × our recorded token counts). The bill itself was not checked.
API-equivalent (subscription)
A setup played through a subscription, which is not billed per token, priced at the API list price × the tokens reported. Not money paid.
GPU time, at most
A setup run on our own machines (NVIDIA RTX PRO 6000 Blackwell (96 GB) at $2.49 per GPU-hour; NVIDIA DGX Spark (GB10) at its purchase price over 3 years plus electricity at the chip's measured draw): the wall clock from its first to its last answer, leaving out every gap over 10 minutes between consecutive answers, × the machine's hourly rate × the GPUs its server used, ÷ its decisions. Its own simultaneous matches count once. Other setups' servers on the same machine at the same time are not separated, so each setup is charged the whole device: an upper bound. A setup that also answered through a hosted API adds those answers' recorded cost. The cost cell shows the estimate first, then a small line naming it and the machine it used most ('estimate · DGX Spark', 'estimate · RTX PRO 6000 Blackwell'; '+ API' where some answers came from a hosted API); how it was counted is in its tooltip (hover, keyboard focus or tap) and its details box. Cost sorting and the cost charts use the estimate.

Every view shows how many of its decisions have a cost (cost coverage).

Time

Limits

What the input adds Earlier version

Computing each play's consequences in code cut the error rate by about two thirds: from 120 in league #6 to 38.5 in league #7, the step that added what each play leads to (hits, blots left, how many of the opponent's 36 rolls hit them, points made). This is the strongest finding so far.

Swipe for more columns →

LeagueWhat changedGamesHit the 300-turn limitMoves that hitAverage gameBest opening playError rate
#1Structured state (JSON) with a short rules text and how every earlier game ended3882%62%277 turns28%138
#3Rules and strategy text, then score, time, position and dice; nothing about earlier games17081%60%274 turns77%132
#6+ the question before the choices; Decider v1.1 self-hosted12471%58%255 turns83%120
#7+ what each play leads to, and the race line1350%9.7%50 turns68%38.5
#8As #7 without d1 (7 models)1200%9.5%51 turns69%39.4
#11Repeat, new dice1120%9.3%50 turns58%40.1
#12Repeat, new dice1120%9.7%51 turns69%36.2
#13+ the last 10 moves of the game1060%10.9%52 turns72%39.1

Each league added one change to the prompt. Error rate: GNU Backgammon's, over all model decisions of the league (earlier scoring), lower is better; grey rows did not finish. The full story: Why it looks like this.

Learning from experience Running (preliminary)

Running (preliminary): see Did they improve? (every model) and Learning across games (the pilot, one model).

Routing and teams Analysis pending

Which decisions a cheap model can handle before handing over to a strong one, and whether a team picks better plays than its best member, wait for the shared-position answers. A team "oracle", the best answer any member gave, is an upper bound and never a working system; the earlier version's team results are on Leagues.

Tail risk

The 95th percentile of the equity lost per decision and the mean loss of the worst 5% of decisions, per league.

League 1

Swipe for more columns →

League decisions; intervals resample whole games
Model and settingError rate (PR), 95% intervalP95 equity lossMean loss, worst 5%Blunder rateConfident and badly wrongDecisions scoredInvalid answers
GNUbg weakprogram · CPU21.418.8 to 24.30.2170.31219.0%N/A1,7440
Jev 1.13decision model · API34.431.0 to 38.00.3520.53425.1%0.4%1,9110
GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs35.031.6 to 39.00.3400.54124.6%0.0%2,0030
DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release35.631.7 to 39.70.3380.55625.2%3.8%1,7360
celeris-1-decisiondecision model · API35.632.0 to 39.40.3390.53125.8%2.7%1,9220
Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs36.933.9 to 40.10.3480.51027.9%0.1%2,0280
CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs38.635.4 to 41.90.3570.57027.5%0.7%2,0160
Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs38.835.3 to 42.50.3890.56926.8%0.5%2,0520
GPT-6 LunaDecisions APIdecision model · API43.339.7 to 46.90.4100.63128.9%0.5%1,9620
SubQ Snap Previewdecision model · API43.939.3 to 48.80.3990.63431.8%0.1%2,2000
Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark45.040.7 to 49.90.4220.68030.7%4.0%2,0050
LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark47.744.1 to 51.20.4310.63132.4%30.3%2,1930
Mercury Decidedecision model · API47.943.3 to 52.90.4810.72830.3%12.9%1,8530
Winnow-12BNVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs49.745.9 to 53.60.4560.68832.3%2.0%2,0150
Safestprogram · CPU51.647.4 to 56.20.4450.67535.1%N/A2,0580
Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs59.654.9 to 64.50.5310.76537.5%0.0%2,0060
LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell61.556.3 to 66.80.5210.74440.8%0.4%1,9520
Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs65.659.7 to 71.90.5960.84638.6%0.0%2,0020
Randomprogram · CPU121.1113.3 to 128.50.8551.08061.5%N/A1,3370

League 2

Swipe for more columns →

League decisions; intervals resample whole games
Model and settingError rate (PR), 95% intervalP95 equity lossMean loss, worst 5%Blunder rateConfident and badly wrongDecisions scoredInvalid answers
Gemini 3.8 Flashhighchat model, one letter · subscription14.912.9 to 17.00.1670.30511.0%N/A2,1680
Gemini 3.8 Flashlowchat model, one letter · subscription16.314.5 to 18.20.1800.27913.5%N/A2,0530
Claude Sonnet 5.5highchat model, one letter · subscription16.714.9 to 18.70.1870.31713.3%N/A2,1640
DeepSeek-V4.1-Flashlow · API · 1017chat model, answer-letter probabilities · API17.115.1 to 19.20.1990.33013.4%13.0%2,22212
Claude Fable 5.1low17.915.9 to 20.10.1850.31315.1%N/A2,0080
DeepSeek-V4.1-Flashlow · API + NVIDIA RTX PRO 6000 Blackwell · first releasechat model, answer-letter probabilities · API19.016.6 to 21.30.1970.37213.9%13.0%2,18838
Grok 4.7low19.116.9 to 21.40.2090.35415.1%N/A2,1520
Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs19.617.5 to 21.70.2030.34815.6%14.2%2,1220
Claude Sonnet 5.5lowchat model, one letter · subscription20.117.8 to 22.40.2150.37915.7%N/A2,1500
Gemma 4 31Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs22.519.9 to 25.30.2220.37618.2%14.4%1,8470
GNUbg weakprogram · CPU27.024.1 to 29.80.2630.38422.1%N/A2,1360
Qwen3.8-27Blow · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs29.026.5 to 31.80.2900.46322.4%20.3%2,1080
d1-ultra30.327.3 to 33.40.2890.48722.4%0.0%2,0710
GLM-5.3lowchat model, answer-letter probabilities · API30.927.8 to 34.00.3100.51221.7%14.4%1,9890
Jev 1.1431.728.3 to 35.70.2860.46524.7%0.1%1,9120
GLiDEAPI33.230.2 to 36.40.3230.51524.1%0.0%2,1590
Qwen3.8 Flashoffchat model, answer-letter probabilities · API33.630.8 to 36.70.3080.53325.3%0.1%2,0270
DeepSeek-V4.1-Flashoff · API · 1017chat model, answer-letter probabilities · API35.231.8 to 38.80.3650.59123.6%8.5%2,0340
ClefNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark36.533.5 to 39.60.3340.53127.1%0.0%1,9780

League 3

Swipe for more columns →

League decisions; intervals resample whole games
Model and settingError rate (PR), 95% intervalP95 equity lossMean loss, worst 5%Blunder rateConfident and badly wrongDecisions scoredInvalid answers
GPT-6.1 Solhigh9.98.6 to 11.30.1030.1877.8%N/A1,3890
GPT-6 Astramediumchat model, one letter · subscription11.19.5 to 12.90.1250.2228.6%N/A1,3620
GPT-6 Astralowchat model, one letter · subscription11.89.5 to 14.20.1290.2469.4%N/A1,2510
GPT-6.1 Sollow12.110.5 to 13.70.1370.2549.4%N/A1,3440
Gemini 4 Argonhighchat model, one letter · subscription12.610.5 to 14.80.1450.2678.9%N/A1,33236
Claude Fable 5.1highchat model, one letter · subscription14.011.5 to 16.30.1600.26611.7%N/A1,2860
Claude Opus 5.5highchat model, one letter · subscription14.412.1 to 17.00.1750.28711.4%N/A1,4770
Claude Opus 5.5lowchat model, one letter · subscription15.413.1 to 18.00.1580.27913.7%N/A1,4110
Gemini 4 Argonlowchat model, one letter · subscription16.513.7 to 19.50.1810.33112.4%N/A1,4840
Grok 4.7highchat model, one letter · subscription18.815.9 to 21.80.2150.36714.0%N/A1,3370
Kimi K3.1 Prooffchat model, one letter · API28.024.5 to 31.80.2670.44821.7%N/A1,1670
GNUbg weakprogram · CPU28.125.0 to 31.40.2540.36423.3%N/A1,4790

Runner changes during the leagues

The anchor

The anchor, GNU Backgammon 1.08 at its Beginner level (0-ply, deterministic noise 0.06), ran on one x86-64 build only, and on x86-64 it is not the Beginner that GNU Backgammon intends. Its deterministic-noise code sums the hash bytes as char, which is signed on x86-64, but subtracts the mean of unsigned bytes, so every noise draw is shifted by about −7 standard deviations. The anchor understates its own winning chances and plays for gammons.

So v3's anchor is "GNUbg 1.08 Beginner, x86-64 build", and its results are reproducible on x86-64 only: a replay there gives the recorded plays. Later releases of the benchmark use a corrected build.

Method experiments

Two experiments with GNU Backgammon players only, run to choose how the leagues work. They are method findings, not results about any model.

Sharing dice across matches

Method finding, not a model result

Each match has its own dice. We tested whether giving round r of every match the same roll schedule would make a league table less a matter of luck. The test used 8 GNU Backgammon players of known strength and no model, and replayed a round robin 500 times under each scheme. The players' order was checked in a separate long run of 11,200 matches: in all 28 pairs the stronger player scored more league points per match.

So the leagues keep separate dice per match. The test players are not language models, whose play varies more from match to match.

Checks of the reference player

Method finding, not a model result

The anchor is GNU Backgammon 1.08 at its Beginner level (0-ply, deterministic noise). We checked whether it is the same player on every machine.

So the v3 anchor is "GNU Backgammon 1.08 Beginner, x86-64 build". It is a fixed reference, not GNU Backgammon's Beginner as published elsewhere. See also The anchor.