Method
How the games are played, what the models are shown, how every decision is scored, and the rules for intervals and verdicts. The thresholds below were fixed before any number on this site was computed. The page as it was before this site went live: archive, 23 October 2026.
Setup
The question is which model, at which reasoning setting, makes better decisions in which positions. Two kinds of data answer it: league games and shared positions. A setup is one model with one set of settings: the same model at a lower and a higher reasoning effort is two setups. A setup is named by what its requests sent, checked in the request logs: (low), (medium), (high) = the effort level sent; (off) = reasoning turned off, no reasoning tokens; (on) = thinking switched on where the model offers no effort levels (its chat template's default).
Leagues
Setups play each other in leagues: each league is a single round robin. League tables compare players inside one league only, and leagues are never merged into one overall score. See Leagues.
- League 1: decision models, plus two programs: Random (a random legal play) and Safest (the play that the fewest opponent rolls hit).
- League 2: workhorse chat models, closed and open, each in two settings (two effort levels, or reasoning off and a reasoning setting).
- League 3: frontier chat models played through their subscriptions, each at its lowest effort (none of them accepts reasoning off) and at its default effort.
- Anchor: GNU Backgammon at its weak Beginner level plays in all three leagues as a fixed reference player (see the reference-player checks).
Match rules
- A match is played in rounds of two games with the same dice: in the second game the players swap seats and the same rolls are repeated, so both face the same luck.
- Three rounds, or fewer: the first player to 5 points after a full round wins if ahead. A tie gets one extra round, then counts as a draw. A game still running after 300 turns goes to the lower pip count for 1 point.
- Games score 1 point, 2 for a gammon and 3 for a backgammon. There is no doubling cube.
- League table: 3 points for a match win, 1 for a draw; ties broken by more games won, then fewer games lost.
- Dice seeds are fixed per league and published. Each match has its own dice (see the dice-sharing test).
- The v3 leagues play without a clock (see Time).
How a decision is asked
- Legal plays only. The program generates every legal play and the model chooses one of them, so a model can never make an illegal move. A roll with only one legal play is played automatically and is not a decision.
- Shuffled order. The legal plays appear in a fixed random order per position, drawn from a published seed. The order never comes from an engine's ranking.
- Reading the answer. Decision models give a probability for each option. Open chat models give probabilities for each answer letter, read after their reasoning when it is on. Models without probabilities answer one letter.
- Long play lists. When a position has more legal plays than a model accepts, the plays are split into equal groups. The model picks a winner in each group, then chooses among the winners. The decision page shows when this happened.
- Invalid answers. An answer that names no option is not retried. The runner plays a random legal play and marks the decision invalid. That play is never counted as the model's choice: it is shown as invalid and left out of every error rate.
- Retries. Network and server errors are retried. Retry and wait time is kept apart from request time.
- Memory. Every decision is a separate request. Models are not told how earlier games went.
The prompt
The current version is bg-v2. Every player gets it, and it did not change after the first match. It is the format of the earlier version's league #8. Every decision sends these parts in this order:
- Rules and strategy: the rules of backgammon, strategy principles (blots and points, distribution, exposure, blocking, hitting, anchors), the best opening plays from computer rollouts, and the match rules. Decision models get a short version; chat models get a longer one with a glossary, game plans and a worked example. Each setup's page says which it got.
- Score and where the match stands.
- Position from the mover's side: checkers on every point, bar, borne off, pip counts.
- Race: the pip lead and whether the two sides are still in contact.
- Dice, then the question: which move makes a point in front of the opponent's checkers, reduces your single pieces, or gets you closer to home.
- The legal plays in the shuffled order, each followed by what the position looks like after it, computed by the program: hits, points made, blots left and how many of the opponent's 36 rolls hit them, points held, longest prime, pips. Chat models see each play under a letter.
There is no move history: in the earlier version, adding the last 10 moves made no measurable difference. The rules text is published on Every decision; the text built for each position is not, so the positions stay usable as a test.
History
The prompt was built one step at a time in the earlier version's leagues; the next section shows each step and what it changed. bg-v2 keeps that format and changes how the models are asked: the plays are shuffled, an invalid answer is not retried, and every setup gets the same rules and question. The earlier version re-asked up to three times and then played the first legal play, which its order favoured.
Why it looks like this
Earlier version Plays were listed in the runner's order, which tends to put the best play early.
The prompt was changed one step at a time, and a league was run after each step. Leagues stopped part-way are kept, with their finished matches only.
| League | What changed | Games | Hit the 300-turn limit | Moves that hit | Average game | Best opening play | Error rate |
|---|---|---|---|---|---|---|---|
| #1 | Structured state (JSON) with a short rules text and how every earlier game ended | 38 | 82% | 62% | 277 turns | 28% | 138 |
| #3 | Rules and strategy text, then score, time, position and dice; nothing about earlier games | 170 | 81% | 60% | 274 turns | 77% | 132 |
| #6 | + the question before the choices; Decider v1.1 self-hosted | 124 | 71% | 58% | 255 turns | 83% | 120 |
| #7 | + what each play leads to, and the race line | 135 | 0% | 9.7% | 50 turns | 68% | 38.5 |
| #8 | As #7 without d1 (7 models) | 120 | 0% | 9.5% | 51 turns | 69% | 39.4 |
| #11 | Repeat, new dice | 112 | 0% | 9.3% | 50 turns | 58% | 40.1 |
| #12 | Repeat, new dice | 112 | 0% | 9.7% | 51 turns | 69% | 36.2 |
| #13 | + the last 10 moves of the game | 106 | 0% | 10.9% | 52 turns | 72% | 39.1 |
Grey rows did not finish. "Best opening play" is the share of first moves of a game that match the computer-rollout table in the prompt. "Error rate" is GNU Backgammon's, over all model decisions of the league (see Error rate); lower is better. Leagues #2, #4, #5, #9 and #10 failed at start or were stopped within minutes; league #3 is left out of the standings below because its format differs.
Rules and strategy as text, not a data structure
The first leagues sent the position as structured data with a short rules summary. Models opened badly: 28% of first moves matched the known best play. With the rules, strategy principles and the opening table as prose, that rose to 77%. Opening play improved, but the middle game did not: games still ran to the turn limit, because both sides kept hitting and re-entering.
A question that names what a good move does
The question before the choices states three goals in plain words: build a point in front of the opponent's checkers, reduce your blots, bring checkers home. It deliberately leaves hitting out, because hitting on almost every move was the problem. The turn-limit share fell from 81% to 71% (this league also switched to the self-hosted Decider v1.1); hitting barely changed.
Compute the consequences in code, not in the model
Backgammon decisions turn on numbers a language model cannot work out reliably from a list of notations: which
checkers are left alone, how many of the opponent's 36 rolls hit them, which points are made. Earlier, each option
was just its notation, like bar/19 13/11*, and the model had to imagine the resulting position.
Now the program computes those numbers for every option and the model compares them. This is the change that
ended the endless games: hits fell from about 60% to under 10% of moves and the average game from about 270 turns
to about 50.
Opening play got worse at the same time, from about 80% to between 58% and 69% best plays. The metrics after each option compete with the opening table earlier in the prompt, and on the first move several plays look similar on these numbers.
Recent moves, for longer plans
Each decision is a separate request, so a model cannot see how the position came about. League #13 adds the last 10 moves of the game, both sides. Ten turns add about 200 tokens. More would eat into Open-Jev's 4,096-token limit, which it applies to the whole question repeated for every option.
It made no measurable difference. League #13's error rate was 39.1 (± 2.2), against 38.6 (± 1.2) for #8, #11 and #12 together, and the finishing order barely changed. Ten moves may be too few, or these models may not use them.
The prompt
Earlier version Plays were listed in the runner's order, which tends to put the best play early.
Every decision sends the same parts in the same order:
- Rules and strategy (about 6,400 characters): the rules of backgammon, strategy principles (blots and points, distribution, exposure, blocking, hitting, anchors), the best opening plays from computer rollouts, and the match rules above. A longer version with a glossary, game plans and a worked example goes to chat models; decision models get this short one because of Open-Jev's 4,096-token limit.
- Score and time: match score, round and game, turn number, both clocks.
- Position from the mover's side: checkers on every point, bar, borne off, pip counts.
- Race: the pip lead and whether the two sides are still in contact.
- The last 10 moves of this game, both sides, in the mover's numbering (league #13 on).
- Dice, then the question: Which of the following moves makes a point in front of the opponent's checkers, or reduces your single pieces, or gets you closer to your home?
- The legal plays, each followed by what the position looks like after it, computed by the program.
The shape of a request (the rules part is left out; angle brackets stand for the values of the position at hand, and no real position or answer is shown, so that the positions stay usable as a test):
Score: you <n>, opponent <n>. Match: round <r> of <R>, game <g> of <G> in this round. Turn <t> of at most <T> in this game. Clock: <free delay> for this decision, then your bank: <time> left in this game (opponent <time>). Position (your numbering; you move 24 -> 1): your checkers: <point:count …> opponent checkers: <point:count …> on the bar: you <n>, opponent <n> borne off: you <n>, opponent <n> pip count: you <n>, opponent <n> Race: <the pip lead>; <whether the checkers are still in contact>. Last 10 moves of this game (oldest first, your numbering): turn <t>, <you / opponent> rolled <dice>: <play> … Dice: <d1>-<d2> Which of the following moves makes a point in front of the opponent's checkers, or reduces your single pieces, or gets you closer to your home? Options (<n>): <play>: hits <n>; makes <points made>; leaves <n> blots, the opponent hits with <n> of 36 rolls; you hold <n> points (<n> in your home board), longest prime <n>; pips: you <n>, opponent <n> … (one line per legal play) Answer: <one of the plays> (with the model's probability where it gives one)
The numbers after each play are exact. The opponent's hitting rolls are counted by generating its legal plays for each of the 36 rolls and checking whether any of them hits.
Error rate
Wins and losses depend heavily on the dice. A backgammon engine can instead judge every decision directly. GNU Backgammon 1.08 evaluates every legal play at 2-ply, in cubeless money equity (expected points per game, match score ignored). The loss of a decision is the equity of the best play minus the equity of the play made. A play that ends the game scores its points exactly.
- Error rate
- 500 × mean 2-ply equity loss per decision. Lower is better. This is the checker-play performance rating (PR) that backgammon players use.
- Decision
- A roll with 2 or more legal plays. Forced moves count in no rate.
- Invalid answers
- Left out of the error rate and shown separately, with their count and share.
- Engine's best play
- Share of decisions where the play made lost less than 0.001 equity.
- Blunder rate
- Share of decisions where the play made lost more than 0.08 equity.
- Tail error
- The 0.95 quantile of the loss per decision, and the mean loss of the worst 0.05 of decisions.
- Near-tie
- The two best plays are within 0.02 equity. Close plays are not rolled out; the gap groups on the Phases page show how much of a setup's error comes from them.
Human scale (approximate)
Error rates are shown next to eXtreme Gammon rating scale's bands for human players:
| Band | Error rate |
|---|---|
| World champion | under 2.5 |
| World class | 2.5 to 5 |
| Expert | 5 to 7.5 |
| Advanced | 7.5 to 12.5 |
| Intermediate | 12.5 to 17.5 |
| Casual | 17.5 to 22.5 |
| Beginner | 22.5 to 30 |
| Distracted | 30 or more |
The comparison is approximate. Human ratings also count doubling-cube decisions; these games have no cube, so the error rate covers checker plays only. The scale also comes from eXtreme Gammon's own evaluation, not GNU Backgammon at 2-ply. Read a band as a rough guide, not as a measured human level.
Was it the right risk? Definitions
The core question: when a model takes a risk, is it the right risk? And when it plays safe, was safe right? The thresholds below (0.001, 0.02, the stake bands 0.02 and 0.08) were fixed on 2026-10-23, before any of these numbers was computed.
- Hitting rolls
- For each legal play, how many of the opponent's 36 rolls hit, read from the numbers the model was shown next to that play.
- Minimum risk
- The fewest hitting rolls any legal play leaves in that position.
- Best safe play
- Among the plays that leave the minimum risk, the one the engine rates best.
- Stake
- The equity difference between the engine's best play and the best safe play: how much the risk call is worth in that position.
- Position class
- No trade-off: every legal play leaves the same number of hitting rolls; counted and left out of every risk metric. Safety pays: stake below 0.001. Risk pays: stake 0.001 or more.
- Took a risk
- The chosen play leaves more hitting rolls than the minimum. Otherwise the model played safe.
- Risk call
- Right risk: took a risk, and the chosen play is at least 0.001 better than the best safe play. Wrong risk: took a risk in a Risk pays position, but not one at least 0.001 better than the best safe play. Needless risk: took a risk in a Safety pays position. Missed risk: played safe in a Risk pays position. Right safety: played safe in a Safety pays position. Every decision with a trade-off gets exactly one risk call.
Metrics per setup (the Risk tab, in this order)
- Risk judgement: the average of risks found and right safety, × 100. It does not depend on how many positions of each class a setup met. With both classes present, the Safest program scores exactly 50, the "always safe" line. If a setup has no Risk pays or no Safety pays decisions, it shows "N/A: no <class> decisions"; an empty class is never read as 0 or 100. 95% interval: bootstrap by game.
- Risks found: right risks ÷ Risk pays decisions.
- Right safety: right safety ÷ Safety pays decisions.
- Good risks: right risks ÷ all risks taken.
- Missed risks: missed risks ÷ Risk pays decisions (too cautious).
- Needless risks: needless risks ÷ Safety pays decisions (too bold).
- Wrong risks: wrong risks ÷ Risk pays decisions.
- Error rate by risk-call category: each part is 500 × the equity lost in that category ÷ all scored decisions of the setup, the same denominator for all six parts (needless risks, missed risks, wrong risks, right risks, right safety, No trade-off), so they add up exactly to the error rate. A right risk or right safety can still lose equity: the chosen play need not be the engine's best. It says where losses happened, not what caused them.
- Decisions with a trade-off, and the counts per class.
Every share shows its count and denominator. Every metric is computed per league; league comparisons are not paired. Setups met different positions, and risk judgement is not adjusted for how hard they were; paired comparisons wait for the shared positions.
Breakdowns
- By stake (Risk pays only): small (below 0.02), medium (0.02 to 0.08), big (above 0.08); risks found, missed risks and wrong risks per band, and "big risks found" on its own.
- Needless risks have no stake band (their stake is below the margin by definition); they are banded by the equity the chosen play lost, with the same limits, and "big needless risks" are those that lost more than 0.08.
- Risk size: extra hitting rolls over the minimum, averaged for right risks and for wrong and needless risks. An average over fewer than 200 decisions shows "Too few decisions".
- By phase: risk judgement per phase, on the Phases page and through the home page's Phase filter.
- Confidence on the call (setups with probabilities only): among the risks with a probability, the ranking score (AUROC) of confidence for "right risk", and the average confidence on missed risks. Decisions whose plays were split into groups have no single probability and are left out; each row shows how many. The confidence is the probability a model gave its chosen play, not the probability that the risk beats the best safe play, so no calibration gap is reported for risk calls. The calibration gap is reported only for the event the confidence stands for, the chosen play being the engine's best (Confidence tab).
Ability and caution
Results published: ability, caution, aim and the cost-derived SDT reference are on the home page's Risk tab (table and chart).
In plain words: ability is how well a setup tells Risk pays positions from Safety pays positions; caution is how much evidence it needs before it takes a risk. A setup can be able and cautious, able and bold, and so on; the two are read separately. Both use the risk classes and risk calls above, at each margin, for every setup and reference row with decisions.
- Hit rate = risks taken in Risk pays positions (right + wrong risks) ÷ Risk pays decisions. False-alarm rate = needless risks ÷ Safety pays decisions.
- Both rates are corrected before the formulas: 0.5 is added to the count and 1.0 to the total (log-linear correction), so a rate of 0 or 1 still has a finite z. The raw counts and totals are shown next to every value.
- Ability d′ = z(hit rate) − z(false-alarm rate), z the inverse of the standard normal distribution. Caution c = −(z(hit rate) + z(false-alarm rate)) ÷ 2: positive is cautious, negative is bold. 95% intervals resample blocks of games (the two games of a round, seats swapped, same dice), 2000 times; a resample without one of the two classes is left out.
- Aim = right risks ÷ (right + wrong risks): needless risks are not in it, so it is not the same as Good risks. With no right or wrong risk it shows N/A and the count 0 / 0.
- Missing class: a setup without Risk pays or without Safety pays decisions shows "N/A: no <class> decisions" for ability, caution and the reference. The correction is never used to stand in for an absent class.
- Cost-derived SDT reference c* = ln β ÷ d′, with β = (Safety pays ÷ Risk pays decisions) × (mean equity lost per needless risk ÷ mean equity lost per missed risk), from the setup's own decisions at that margin. It is the caution at which an equal-variance signal-detection observer with these costs would lose least; it is not the best caution for actual play and not a proven optimum. The costs are conditional on the setup's own policy (which positions it took risks in), and a correctly classified call can still lose equity. Rules, in order: N/A when a class is absent; N/A with no needless or no missed risk to price; N/A when a mean cost is 0 or β is not a positive finite number; N/A when d′ ≤ 0; "unstable: d′ near zero" (value shown, no mark) when the lower end of the d′ interval is 0 or below; "unstable" (value shown, no mark) when fewer than 200 needless or missed risks are behind a mean cost. Only the remaining values get the short reference mark on the chart.
- By reasoning setting (every model with both settings in one league's games): ability and caution of the same model at its lower and higher setting, the change, and the time and output tokens per decision. UNPAIRED: the two settings met different positions.
- Limits: a simple model of the risk call (equal-variance signal detection); setups met different positions; one kind of risk only (being hit). These rules were fixed on 2026-10-24 before any value was computed.
Reference rows
Shown, never ranked, never champions. Random plays a random legal play; GNUbg weak is the x86-64 build: its noise is shifted, so it overplays gammons (see The anchor).
Safest is not the same thing in every league. In League 1, Safest is the program that actually played the matches: among the plays that leave the fewest hitting rolls it takes the one with the fewest blots, then the largest pip lead, then the play shown first; it never asks the engine. It played a fewest-hitting-rolls play in 2,058 of its 2,058 scored decisions and the engine's best safe play in 1,395 of them. Its right-safety part of the error rate, 8.45 PR, is the equity lost on its 749 right-safety decisions × 500 ÷ all 2,058 of its scored decisions: a share of its error rate, not the average loss of a right-safety decision, which is 23.22 PR. In League 2 and League 3, Safest (computed) is a computed reference: the engine's best safe play on every scored decision of that league. Its numbers are not directly comparable with League 1's played program.
Random also played League 1 only; in the other leagues it is computed as the expectation over all legal plays, so its counts can have decimals. A cell with no decisions shows its count, 0, and no average.
Charts
Risk map: across, the Risk pays decisions without the right risk (missed + wrong risks) as a share of Risk pays decisions; up, needless risks as a share of Safety pays decisions. Risk judgement = 100 − (across + up) / 2, so the corner (0, 0) is risk judgement 100; the faint diagonals mark 50, 60, 70, 80, 90. A point's colour is its share of wrong risks among the Risk pays decisions without the right risk. Error rate by risk-call category: the six parts as a stacked bar. Stake chart: risks found, wrong and missed risks per stake band, and big needless risks. Ability and caution: ability across, caution up, programs and computed reference rows as squares, a short bar at the cost-derived SDT reference where it is valid; under it the limit line: a simple model of the risk call (equal-variance signal detection); setups met different positions. Risk call card: one image per league winner. A downloaded chart PNG carries the site's title and subtitle, the league, the margin, the number of decisions, the results version and the site address, and lists the active filters; it keeps the theme in view. Every table on the site has its own Download PNG: the table as shown (tab, filters, sort, theme) with all its columns and rows, the headings above it and the notes under it, then the same small type, with the rows shown and the decisions behind them.
Robustness: the 0.02 margin
At 2-ply the engine can misjudge close plays by more than 0.001. Every risk metric is therefore computed a second time with 0.02 in every place where 0.001 appears above. The 0.02 results are on the Risk tab under the main table, and here:
League 1
Swipe for more columns →
| Model and setting | Risk judgement | Risks found | Right safety | Missed risks | Needless risks | Wrong risks |
|---|---|---|---|---|---|---|
| Jev 1.13decision model · API | 59.50.02: 58.4 | 27.0%0.02: 26.8% | 92.1%0.02: 90.0% | 65.0%0.02: 64.1% | 7.9%0.02: 10.0% | 8.1%0.02: 9.0% |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release | 59.30.02: 59.0 | 29.1%0.02: 30.1% | 89.5%0.02: 87.9% | 63.7%0.02: 62.6% | 10.5%0.02: 12.1% | 7.1%0.02: 7.3% |
| GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 58.80.02: 57.9 | 33.0%0.02: 33.1% | 84.6%0.02: 82.7% | 58.9%0.02: 58.1% | 15.4%0.02: 17.3% | 8.1%0.02: 8.8% |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 57.90.02: 56.6 | 33.4%0.02: 33.1% | 82.5%0.02: 80.2% | 56.1%0.02: 55.7% | 17.5%0.02: 19.8% | 10.5%0.02: 11.2% |
| celeris-1-decisiondecision model · API | 57.80.02: 57.3 | 23.9%0.02: 24.1% | 91.7%0.02: 90.5% | 70.7%0.02: 69.1% | 8.3%0.02: 9.5% | 5.4%0.02: 6.8% |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark | 57.60.02: 56.7 | 23.5%0.02: 22.8% | 91.7%0.02: 90.6% | 70.9%0.02: 70.1% | 8.3%0.02: 9.4% | 5.6%0.02: 7.1% |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs | 57.10.02: 57.3 | 27.3%0.02: 28.2% | 86.8%0.02: 86.4% | 64.9%0.02: 63.2% | 13.2%0.02: 13.6% | 7.7%0.02: 8.5% |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs | 56.90.02: 56.7 | 33.1%0.02: 33.7% | 80.7%0.02: 79.7% | 57.3%0.02: 55.9% | 19.3%0.02: 20.3% | 9.5%0.02: 10.4% |
| SubQ Snap Previewdecision model · API | 56.50.02: 55.9 | 37.9%0.02: 38.4% | 75.1%0.02: 73.4% | 45.8%0.02: 44.1% | 24.9%0.02: 26.6% | 16.2%0.02: 17.5% |
| GPT-6 LunaDecisions APIdecision model · API | 55.70.02: 55.2 | 21.6%0.02: 21.7% | 89.7%0.02: 88.7% | 71.5%0.02: 70.6% | 10.3%0.02: 11.3% | 6.9%0.02: 7.7% |
| Winnow-12BNVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 55.20.02: 54.5 | 33.7%0.02: 33.6% | 76.6%0.02: 75.3% | 53.2%0.02: 51.8% | 23.4%0.02: 24.7% | 13.1%0.02: 14.6% |
| Mercury Decidedecision model · API | 53.40.02: 53.1 | 18.7%0.02: 18.4% | 88.1%0.02: 87.8% | 74.9%0.02: 73.9% | 11.9%0.02: 12.2% | 6.4%0.02: 7.7% |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark | 52.40.02: 52.4 | 31.5%0.02: 32.2% | 73.4%0.02: 72.6% | 47.3%0.02: 44.9% | 26.6%0.02: 27.4% | 21.2%0.02: 22.9% |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 49.10.02: 47.7 | 31.3%0.02: 30.6% | 66.9%0.02: 64.8% | 42.7%0.02: 42.3% | 33.1%0.02: 35.2% | 26.0%0.02: 27.1% |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 48.50.02: 48.0 | 37.3%0.02: 37.7% | 59.8%0.02: 58.4% | 34.7%0.02: 33.4% | 40.2%0.02: 41.6% | 28.0%0.02: 28.9% |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell | 45.70.02: 44.9 | 28.8%0.02: 28.4% | 62.7%0.02: 61.5% | 48.5%0.02: 48.4% | 37.3%0.02: 38.5% | 22.7%0.02: 23.2% |
| GNUbg weakprogram · CPU | 67.10.02: 64.3 | 66.6%0.02: 66.1% | 67.6%0.02: 62.6% | 10.7%0.02: 9.0% | 32.4%0.02: 37.4% | 22.7%0.02: 24.9% |
| Randomprogram · CPU | 24.90.02: 23.7 | 24.8%0.02: 22.7% | 25.0%0.02: 24.7% | 19.2%0.02: 19.1% | 75.0%0.02: 75.3% | 56.0%0.02: 58.2% |
| Safestprogram · CPU | 50.00.02: 50.0 | 0.0%0.02: 0.0% | 100.0%0.02: 100.0% | 100.0%0.02: 100.0% | 0.0%0.02: 0.0% | 0.0%0.02: 0.0% |
League 2
Swipe for more columns →
| Model and setting | Risk judgement | Risks found | Right safety | Missed risks | Needless risks | Wrong risks |
|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flashlow · API + NVIDIA RTX PRO 6000 Blackwell · first releasechat model, answer-letter probabilities · API | 69.00.02: 70.2 | 45.0%0.02: 49.2% | 93.0%0.02: 91.2% | 49.0%0.02: 44.4% | 7.0%0.02: 8.8% | 6.0%0.02: 6.4% |
| DeepSeek-V4.1-Flashlow · API · 1017chat model, answer-letter probabilities · API | 68.50.02: 68.9 | 42.5%0.02: 44.8% | 94.5%0.02: 93.0% | 52.1%0.02: 47.9% | 5.5%0.02: 7.0% | 5.4%0.02: 7.3% |
| Claude Sonnet 5.5highchat model, one letter · subscription | 68.20.02: 69.0 | 41.8%0.02: 44.0% | 94.6%0.02: 94.0% | 55.5%0.02: 52.1% | 5.4%0.02: 6.0% | 2.7%0.02: 3.9% |
| Claude Fable 5.1low | 67.80.02: 68.5 | 41.0%0.02: 43.7% | 94.6%0.02: 93.2% | 53.8%0.02: 49.4% | 5.4%0.02: 6.8% | 5.2%0.02: 6.9% |
| Grok 4.7low | 67.70.02: 67.0 | 42.3%0.02: 42.8% | 93.0%0.02: 91.1% | 49.9%0.02: 47.2% | 7.0%0.02: 8.9% | 7.7%0.02: 9.9% |
| Gemini 3.8 Flashhighchat model, one letter · subscription | 66.90.02: 67.6 | 36.1%0.02: 39.2% | 97.6%0.02: 96.1% | 61.6%0.02: 56.9% | 2.4%0.02: 3.9% | 2.3%0.02: 3.9% |
| Gemini 3.8 Flashlowchat model, one letter · subscription | 65.10.02: 66.8 | 33.8%0.02: 37.6% | 96.5%0.02: 96.0% | 63.2%0.02: 58.3% | 3.5%0.02: 4.0% | 3.0%0.02: 4.1% |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs | 64.50.02: 65.6 | 31.9%0.02: 34.7% | 97.1%0.02: 96.5% | 64.9%0.02: 61.3% | 2.9%0.02: 3.5% | 3.3%0.02: 3.9% |
| d1-ultra | 64.00.02: 63.0 | 42.0%0.02: 41.8% | 86.0%0.02: 84.2% | 49.6%0.02: 47.4% | 14.0%0.02: 15.8% | 8.4%0.02: 10.7% |
| Claude Sonnet 5.5lowchat model, one letter · subscription | 63.50.02: 64.2 | 30.9%0.02: 33.1% | 96.2%0.02: 95.2% | 65.5%0.02: 62.5% | 3.8%0.02: 4.8% | 3.6%0.02: 4.4% |
| Gemma 4 31Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs | 62.60.02: 63.2 | 33.2%0.02: 35.5% | 92.1%0.02: 90.9% | 58.4%0.02: 55.8% | 7.9%0.02: 9.1% | 8.4%0.02: 8.8% |
| Qwen3.8-27Blow · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs | 62.00.02: 61.2 | 33.8%0.02: 34.4% | 90.2%0.02: 87.9% | 58.3%0.02: 57.3% | 9.8%0.02: 12.1% | 7.8%0.02: 8.3% |
| GLM-5.3lowchat model, answer-letter probabilities · API | 61.70.02: 61.6 | 30.7%0.02: 31.8% | 92.7%0.02: 91.4% | 63.2%0.02: 60.6% | 7.3%0.02: 8.6% | 6.1%0.02: 7.6% |
| GLiDEAPI | 60.40.02: 60.5 | 36.9%0.02: 38.4% | 84.0%0.02: 82.5% | 51.0%0.02: 48.9% | 16.0%0.02: 17.5% | 12.1%0.02: 12.6% |
| Qwen3.8 Flashoffchat model, answer-letter probabilities · API | 59.50.02: 59.3 | 34.4%0.02: 35.5% | 84.6%0.02: 83.0% | 52.7%0.02: 50.6% | 15.4%0.02: 17.0% | 12.9%0.02: 13.9% |
| Jev 1.14 | 59.30.02: 59.2 | 29.1%0.02: 29.9% | 89.4%0.02: 88.5% | 63.4%0.02: 61.4% | 10.6%0.02: 11.5% | 7.4%0.02: 8.8% |
| DeepSeek-V4.1-Flashoff · API · 1017chat model, answer-letter probabilities · API | 58.50.02: 58.8 | 25.4%0.02: 26.5% | 91.7%0.02: 91.1% | 68.3%0.02: 66.1% | 8.3%0.02: 8.9% | 6.3%0.02: 7.4% |
| ClefNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark | 57.50.02: 57.6 | 39.5%0.02: 40.6% | 75.6%0.02: 74.6% | 45.2%0.02: 42.8% | 24.4%0.02: 25.4% | 15.3%0.02: 16.6% |
| GNUbg weakprogram · CPU | 62.30.02: 61.7 | 62.7%0.02: 64.8% | 61.9%0.02: 58.5% | 14.4%0.02: 10.8% | 38.1%0.02: 41.5% | 22.9%0.02: 24.4% |
| Random (computed)reference | 27.00.02: 25.9 | 25.1%0.02: 23.7% | 28.8%0.02: 28.1% | 16.7%0.02: 16.1% | 71.2%0.02: 71.9% | 58.2%0.02: 60.2% |
| Safest (computed)reference | 50.00.02: 50.0 | 0.0%0.02: 0.0% | 100.0%0.02: 100.0% | 100.0%0.02: 100.0% | 0.0%0.02: 0.0% | 0.0%0.02: 0.0% |
League 3
Swipe for more columns →
| Model and setting | Risk judgement | Risks found | Right safety | Missed risks | Needless risks | Wrong risks |
|---|---|---|---|---|---|---|
| GPT-6 Astralowchat model, one letter · subscription | 75.80.02: 74.9 | 57.2%0.02: 58.1% | 94.4%0.02: 91.8% | 39.2%0.02: 37.8% | 5.6%0.02: 8.2% | 3.5%0.02: 4.2% |
| GPT-6 Astramediumchat model, one letter · subscription | 74.70.02: 74.9 | 54.9%0.02: 56.5% | 94.4%0.02: 93.2% | 42.4%0.02: 37.9% | 5.6%0.02: 6.8% | 2.7%0.02: 5.5% |
| Gemini 4 Argonhighchat model, one letter · subscription | 74.10.02: 75.9 | 50.8%0.02: 56.4% | 97.4%0.02: 95.3% | 45.6%0.02: 40.4% | 2.6%0.02: 4.7% | 3.6%0.02: 3.1% |
| GPT-6.1 Sollow | 72.70.02: 74.2 | 51.7%0.02: 56.2% | 93.8%0.02: 92.2% | 44.1%0.02: 38.6% | 6.2%0.02: 7.8% | 4.2%0.02: 5.2% |
| Claude Fable 5.1highchat model, one letter · subscription | 72.30.02: 72.1 | 50.7%0.02: 52.7% | 94.0%0.02: 91.6% | 42.2%0.02: 38.6% | 6.0%0.02: 8.4% | 7.1%0.02: 8.6% |
| GPT-6.1 Solhigh | 72.20.02: 72.4 | 50.6%0.02: 52.4% | 93.7%0.02: 92.5% | 44.4%0.02: 38.4% | 6.3%0.02: 7.5% | 5.0%0.02: 9.2% |
| Claude Opus 5.5highchat model, one letter · subscription | 70.00.02: 70.8 | 43.4%0.02: 46.3% | 96.6%0.02: 95.3% | 51.4%0.02: 47.5% | 3.4%0.02: 4.7% | 5.2%0.02: 6.2% |
| Gemini 4 Argonlowchat model, one letter · subscription | 69.20.02: 68.7 | 47.8%0.02: 49.6% | 90.7%0.02: 87.8% | 46.0%0.02: 44.1% | 9.3%0.02: 12.2% | 6.2%0.02: 6.3% |
| Grok 4.7highchat model, one letter · subscription | 68.50.02: 67.9 | 44.0%0.02: 44.6% | 93.0%0.02: 91.1% | 50.5%0.02: 47.8% | 7.0%0.02: 8.9% | 5.6%0.02: 7.6% |
| Claude Opus 5.5lowchat model, one letter · subscription | 66.90.02: 67.1 | 40.1%0.02: 41.9% | 93.8%0.02: 92.4% | 55.8%0.02: 51.9% | 6.2%0.02: 7.6% | 4.1%0.02: 6.2% |
| Kimi K3.1 Prooffchat model, one letter · API | 65.90.02: 66.7 | 42.4%0.02: 44.7% | 89.5%0.02: 88.7% | 49.0%0.02: 45.0% | 10.5%0.02: 11.3% | 8.6%0.02: 10.3% |
| GNUbg weakprogram · CPU | 62.10.02: 60.0 | 58.8%0.02: 58.1% | 65.5%0.02: 61.8% | 16.7%0.02: 15.3% | 34.5%0.02: 38.2% | 24.5%0.02: 26.6% |
| Random (computed)reference | 27.60.02: 26.4 | 25.1%0.02: 23.4% | 30.1%0.02: 29.4% | 17.5%0.02: 16.9% | 69.9%0.02: 70.6% | 57.4%0.02: 59.7% |
| Safest (computed)reference | 50.00.02: 50.0 | 0.0%0.02: 0.0% | 100.0%0.02: 100.0% | 100.0%0.02: 100.0% | 0.0%0.02: 0.0% | 0.0%0.02: 0.0% |
Plays without a 2-ply evaluation
All 92,888 of the 92,888 league decisions have a 2-ply evaluation of every legal play, so no play needed a CPU evaluation.
Limit: this measures one kind of risk, being hit. Gammon risk and priming are not in it.
Statistics and verdicts
- Intervals are 0.95 bootstrap intervals with 2000 resamples and a fixed seed (20261021).
- League decisions resample whole games, because moves within a game are not independent.
- Shared positions resample positions.
- Comparisons of two setups on the same positions are always paired: the interval is of the per-position difference in loss.
- League comparisons are not paired. League games give each setup different positions. Between league setups we write only "no clear difference in league play" or "lower/higher error rate in league play", never a verdict label.
- Minimum sample: an average (error rate, mean equity lost, a calibration metric) over fewer than 200 decisions shows "Too few decisions" instead of a value. Raw counts are always shown, and a share is always shown with its count and denominator. Exception: the 50-position error rates on New models: first look are shown with their intervals.
- League placings are descriptive. Middle places move between leagues with the dice.
Verdicts
For reasoning (low against high, off against on) and any comparison of A against B on the same positions:
| Verdict | Rule |
|---|---|
| High helps | The paired 95% interval of the error difference (high minus low, or B minus A) lies entirely below zero. |
| High hurts | The interval lies entirely above zero. |
| No clear benefit | The interval includes zero. Never read as "the same", "level" or "gains nothing". |
| Depends on the position | Phase-level effects point in different directions, each with an interval that excludes zero. |
Every phase-level verdict is exploratory. "Worse moves" and "more time or cost with no shown gain" are kept as two separate findings and never merged into "high is bad".
Fixed thresholds
These values were fixed on 2026-10-21, before any number on this site was computed, and are not changed after seeing results. Equity values are in points per game.
| Threshold | Value | Meaning |
|---|---|---|
| Error-rate scale | 500 | Error rate = this number × mean equity lost per decision. |
| Engine's best play | 0.001 | The chosen play lost less than this much equity. |
| Blunder | 0.08 | The chosen play lost more than this much equity. |
| Badly wrong | 0.08 | The chosen play lost more than this much equity (used in "confident and badly wrong"). |
| Very confident | 0.9 | The setup gave its chosen play at least this probability. |
| Time budget | 30 | Seconds of request time per decision; slower decisions count as over the budget. |
| Minimum sample | 200 | Fewer decisions than this in a cell: "Too few decisions" instead of a value. |
| Near-tie | 0.02 | The two best plays are closer than this in equity. |
| Gap groups | 0.02, 0.08 | Limits of the groups by equity gap between the two best plays. |
| Option groups | 2–5, 6–10, 11–15, 16–25, 26 or more | Groups by number of legal plays. |
| Tail error | 0.95 | Quantile of the loss per decision reported as tail error. |
| Worst decisions | 0.05 | Share of a setup's decisions, the ones with the largest loss, whose mean loss is reported. |
| Bootstrap resamples | 2000 | Resamples for every interval. |
| Bootstrap seed | 20261021 | Fixed random seed for every interval. |
| Interval level | 0.95 | Coverage of every interval. |
| Risk thresholds fixed on | 2026-10-23 | Date the risk thresholds were fixed, before any risk number was computed. |
| Risk margin | 0.001 | B6.1: stake below it is Safety pays, at or above it Risk pays; a taken risk is right when the chosen play is at least this much better than the best safe play. |
| Robustness margin | 0.02 | B6.6: every risk metric computed again with this margin in place of the risk margin. |
| Stake bands | 0.02, 0.08 | Risk pays decisions by stake: small below the first value, medium up to the second, big above it. |
| Confidence cutoffs | 0.9, 0.99 | Confidence tab: decisions kept at or above each probability. |
| Held-out splits | 300 | Random halves of the games for the cutoff chosen on one half and tested on the other. |
| Held-out cutoff rule | On one half of the games (a game's decisions together), the lowest probability cutoff whose kept decisions (probability at or above it) hold no blunder; scored on the other half: share kept and blunder rate among kept, averaged over cutoff_split_runs seeded splits. | How that cutoff is chosen. |
| Surest share | 0.25 | Share of a setup's decisions with the highest confidence, for the error rate in the surest quarter. |
| Needless-risk bands | 0.02, 0.08 | Needless risks by the equity the chosen play lost: small, medium, big. |
| Risk map diagonals | 50, 60, 70, 80, 90 | Risk judgement values drawn as faint diagonals on the risk map. |
| Hero board stake | 0.05, 0.15 | The home page board shows a Risk pays decision with a stake in this range: the one closest to the median stake of those decisions in the latest finished league. |
Status labels
Every section and every metric shows one status. Where the data does not exist, the status is shown, never a placeholder number.
| Status | Meaning |
|---|---|
| Published | Computed from finished leagues or complete shared-position runs. |
| Provisional | From a league that is still running. The numbers will change. They appear only in that league's own tables and enter no cross-league ranking, chart or finding until the league finishes. |
| Earlier version | From an earlier version of the benchmark. Never ranked or charted together with current numbers. |
| Analysis pending | The data exists, but the analysis has not been run on it yet. |
| Not measured | This was not measured. |
| N/A | Does not apply to this setup, for example confidence metrics for a setup that returns only a choice. |
League status
| League | Meaning |
|---|---|
| Running (done / total) | Matches are still being played; the numbers are provisional. |
| Finished | Every match has been played. |
| Stopped | The league was ended before all its matches were played; its finished matches are kept. |
An average below the minimum sample shows "Too few decisions".
Versions and play order
Numbers from different benchmark versions or prompt formats are never put in one ranking or chart. Every number carries its version.
Earlier version Plays were listed in the runner's order, which tends to put the best play early. Re-asked with the same plays in a random order on 1,000 positions, Gemini 4 Argon's error rate changed by +1.9 (95% interval −0.2 to 4.2), so a help from the order is possible but not proven. The current version shuffles the plays for every setup.
The earlier version also scored differently: every legal play at 0-ply, then the five best and the chosen play again at 2-ply. The current version evaluates every legal play at 2-ply.
Changelog. 28 Oct 2026: setups renamed to the reasoning effort they sent. 5 setups changed from (off) to (low). No score changed. 29 Oct 2026: the Hype Cup and the Nackgammon knockout on Leagues, and every match on Matches. Match results only: no score changed. 6 Nov 2026: seven late entries to the original League 1 table on Leagues (Late entries): SubQ Snap 1.1, Elephant-1 and Mammoth-1 (played 2 Nov), Drex Max v1.1, Drex 2.0, Drex MaxNVIDIA RTX PRO 6000 Blackwell and Drex 2.0NVIDIA RTX PRO 6000 Blackwell (played 6 Nov; their pairs with GPT-6 LunaDecisions API not played, its API out of credit), each with a page of its own; the late-entry table gains the engine's best play, median seconds and $ per 1,000 decisions, with each price's label, source and date read. Late entries count in no standings table and no other setup's numbers: no score changed. 7 Nov 2026: REAX 1.3, a late entry to the original League 1 table on Leagues (Late entries), played on our GPU on 7 Nov (its pair with GPT-6 LunaDecisions API not played, its API out of credit), with a page of its own; no list price is published for it, so it has no cost figure. No score changed. 11 Nov 2026: KnowLine-31B-Gen1.5, a late entry to the original League 1 table on Leagues (Late entries), played on our GPU (NVIDIA DGX Spark) on 10–11 Nov (its pairs with GPT-6 LunaDecisions API, its API out of credit, and Mercury Decide, no longer served on OpenRouter, not played; its first 4 matches, played at once, voided and replayed one at a time), with a page of its own; no list price is published for it, so it has no cost figure. No score changed. 11 Nov 2026: Muse Embermax, a late entry to the original League 1 table on Leagues (Late entries), played through OpenRouter (pinned to Meta, reasoning effort max) on 10–11 Nov (its pairs with GPT-6 Luna (Decisions API), its API out of credit, and Mercury Decide, no longer served on OpenRouter, not played), with a page of its own; its cost is its billed tokens at Meta's list price. No score changed. 11 Nov 2026: Gemini Decision 1, a late entry to the original League 1 table on Leagues (Late entries), played through OpenRouter (pinned to Google AI Studio) on 11 Nov (its pairs with GPT-6 LunaDecisions API, its API out of credit, and Mercury Decide, no longer served on OpenRouter, not played; runner dab6da5 without a probability-map check, then 38024d6 with map-check v2, whose sum rule (within 1e-6) applies to this entry only), with a page of its own; its cost is its billed cost per decision. No score changed. 12 Nov 2026: the late entries' and repeats' pairs with GPT-6 Luna (Decisions API) and Mercury Decide are outside the requested scope (Ümit, 12 Nov): no longer work to close. They stay unplayed and are labelled so beside the reason they were not played, never a score or a loss; a late entry whose only unplayed pairs are these is no longer marked Provisional, since none of its pairs is left to play. No number changed. 12 Nov 2026: Limits (Runner commits) states how many original league matches were started from a runner checkout with uncommitted changes (harness_dirty): provenance only. No number changed. 14 Nov 2026: fourteen small decision models, late entries to the original League 1 table on Leagues (Late entries): GLiDE 2B, GLiDE 0.8B, GLiDE 0.4B, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, KnowLine 4B Gen5.5, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, OneJev1.6-0.8B, GLiClass v4 Edge, Cerno 9B v1.2, Turion Jev 31B and Sieve 31B v3, each served on our DGX Spark, played 11–14 Nov in two stages (its hosted and program opponents, then the 8 setups we run ourselves; its pairs with GPT-6 LunaDecisions API and Mercury Decide not played, outside the requested scope), each with a page of its own; GLiDE 0.4B gets at most 74 options per request and GLiClass v4 Edge at most 218, longer play lists split into groups by the league's rule; their cost is an estimate from their time on the DGX Spark (GPU time, at most). No score changed. 14 Nov 2026: d1-2BNVIDIA RTX PRO 6000 Blackwell, a late entry to the original League 1 table on Leagues (Late entries), played on one RTX PRO 6000 Blackwell of ours on 14 Nov (its pairs with GPT-6 LunaDecisions API and Mercury Decide not played, outside the requested scope), with a page of its own; its cost is an estimate from its time on that card (GPU time, at most). No score changed. 14 Nov 2026: d1-Omni-2BNVIDIA RTX PRO 6000 Blackwell, d1-2B with audio added, a late entry to the original League 1 table on Leagues (Late entries), played text only on the same RTX PRO 6000 Blackwell after d1-2B on 14 Nov (its pairs with GPT-6 LunaDecisions API and Mercury Decide not played, outside the requested scope), with a page of its own; its cost is an estimate from its time on that card (GPU time, at most). No score changed.
Confidence
- Sources. A setup's confidence is its probability for the play it chose. It comes from one of three sources: decision-model probability, letter probability, or none.
- N/A for choice-only setups. Confidence metrics (AUROC, ECE, Brier score, risk–coverage) are N/A for setups that return only a choice. A recorded 1.0 from those setups is not 100% confidence.
- Same source only. Confidence is compared only between setups with the same confidence source.
- Group splits. Decisions whose plays were split into groups are left out of the confidence analyses: the probability then belongs to a smaller choice.
- Cutoffs 0.9 and 0.99, fixed before computing: decisions kept (probability at or above the cutoff), blunders among them and their mean equity lost. Surest quarter: the error rate of the 0.25 share of decisions with the highest probability.
- Ranking scores (AUROC): for the engine's best play (does a higher probability go with the best play?) and for warning before a blunder (does a lower probability go with a blunder?). Calibration gap (ECE): for the event "the chosen play is the engine's best", in 10 bins, in points.
- Confident and badly wrong: the setup gave its play at least 0.9 and the play lost more than 0.08 equity.
A cutoff chosen on half of the games
On one half of the games (a game's decisions together), the lowest probability cutoff whose kept decisions (probability at or above it) hold no blunder; scored on the other half: share kept and blunder rate among kept, averaged over cutoff_split_runs seeded splits. 300 splits, by game.
League 1
Swipe for more columns →
| Model and setting | Share kept on the other half | Blunders among kept | Splits with no cutoff | Decisions with probabilities |
|---|---|---|---|---|
| Jev 1.13decision model · API | 2.1% | 3.3% | 0 of 300 | 1,911 |
| GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 4.6% | 1.7% | 0 of 300 | 2,002 |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release | 4.3% | 1.5% | 0 of 300 | 1,589 |
| celeris-1-decisiondecision model · API | 1.4% | 8.3% | 147 of 300 | 1,922 |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 3.6% | 1.9% | 0 of 300 | 1,822 |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs | 0.8% | 7.3% | 36 of 300 | 1,893 |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs | 4.1% | 1.9% | 0 of 300 | 1,826 |
| GPT-6 LunaDecisions APIdecision model · API | 0.7% | 11.1% | 46 of 300 | 1,962 |
| SubQ Snap Previewdecision model · API | 1.5% | 3.8% | 0 of 300 | 2,199 |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark | 0.9% | 6.4% | 0 of 300 | 1,825 |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark | N/A | N/A | 300 of 300 | 1,964 |
| Mercury Decidedecision model · API | 1.0% | 7.1% | 0 of 300 | 1,853 |
| Winnow-12BNVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 1.7% | 4.0% | 0 of 300 | 1,923 |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 2.8% | 3.1% | 0 of 300 | 1,782 |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell | 1.6% | 4.3% | 0 of 300 | 1,770 |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 2.3% | 2.0% | 0 of 300 | 2,002 |
League 2
Swipe for more columns →
| Model and setting | Share kept on the other half | Blunders among kept | Splits with no cutoff | Decisions with probabilities |
|---|---|---|---|---|
| DeepSeek-V4.1-Flashlow · API · 1017chat model, answer-letter probabilities · API | N/A | N/A | 300 of 300 | 1,965 |
| DeepSeek-V4.1-Flashlow · API + NVIDIA RTX PRO 6000 Blackwell · first releasechat model, answer-letter probabilities · API | 1.3% | 14.4% | 151 of 300 | 1,904 |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs | 9.4% | 11.0% | 299 of 300 | 1,870 |
| Gemma 4 31Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs | 1.6% | 12.9% | 150 of 300 | 1,664 |
| Qwen3.8-27Blow · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs | 0.5% | 74.2% | 200 of 300 | 1,891 |
| d1-ultra | 2.2% | 2.8% | 0 of 300 | 2,070 |
| GLM-5.3lowchat model, answer-letter probabilities · API | 0.6% | 8.1% | 0 of 300 | 1,785 |
| Jev 1.14 | 2.6% | 1.8% | 0 of 300 | 1,910 |
| GLiDEAPI | 4.0% | 1.2% | 0 of 300 | 2,157 |
| Qwen3.8 Flashoffchat model, answer-letter probabilities · API | 1.5% | 4.2% | 0 of 300 | 1,856 |
| DeepSeek-V4.1-Flashoff · API · 1017chat model, answer-letter probabilities · API | 2.5% | 3.2% | 0 of 300 | 1,865 |
| ClefNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark | 2.8% | 2.6% | 0 of 300 | 1,956 |
Re-read, refit, bar and distributions (offline, from recorded decisions)
Results published: the re-read, calibration and distributions on the Confidence tab, the minimum bar on the Cost tab.
Four analyses of the decisions already recorded; no model was asked again. Every rule below (thresholds, estimators, exclusions, degenerate cases, splits and the hypothesis) was fixed and committed on 2026-10-24, before any of these numbers was computed. Splits, folds and resamples use one map of blocks per league for every setup: a block is the two games of a round, the second with seats swapped and the same dice (checked on the data: same rolls, and no two matches share a dice seed).
Which probabilities are complete
A decision enters these analyses only if its option probabilities are verifiably complete. Each one is checked against the raw response: the response must reproduce the stored probabilities, and every legal play must have its own probability in it: every play for a decision model, every option letter among the returned top log-probabilities for a letter readout. A letter missing there was filled with 0 and the rest renormalised by the runner; such a decision is sparse, whatever the length or sum of its vector. Decisions split into groups, choice-only answers, invalid answers, decisions without a raw response and decisions whose stored vector is not reproduced are counted separately and left out (Confidence tab, per setup).
Re-read risk call (re-read from the model's probabilities, not played)
- P(risk) = the probability mass on the plays that leave more hitting rolls than the minimum. If P(risk) ≥ t the re-read takes the most probable risky play, otherwise the most probable safe play; equal probabilities go to the play shown first. A decision without a trade-off keeps the play made.
- t is chosen from 0.0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0, always safe ("always safe" never takes a risk, 0 always does) to maximise risk judgement on half of the league's blocks; ties go to the value nearest 0.5, then the smaller. It is scored on the other half, against the play made on exactly the same decisions: risk judgement on the decisions with a trade-off, error rate on all of them.
- 300 splits; reported: the mean over splits, and the 5th to 95th percentile of the change over the splits as stability (the splits are not independent experiments). 95% interval: the whole procedure repeated on 200 resamples of blocks (30 splits each). Setups with fewer than 200 complete decisions show "Too few decisions".
- Recorded positions scored again, not new games: it says nothing about win rates.
Hypothesis, recorded before computing: gains will be largest for setups with high ability whose caution is well above the cost-derived SDT reference. It is a hypothesis, not a claim that the reference is optimal. Predictor, from the training half only: d′ × (c − c*); undefined where c* is N/A or d′ is below 0.2; a setup needs it in 150 splits. Test: Spearman rank correlation with the held-out change in risk judgement across setups, leagues pooled; supported only if it is positive and its 95% interval is above 0. Result: ρ = −0.15 (95% interval −0.46 to +0.03, 200 resamples) over 24 setups: not supported.
Calibration as shipped and after a refit
Two events, never mixed: the chosen play (its probability against "the chosen play is the engine's best") and the risk class (P(risk) against "the position is Risk pays", decisions with a trade-off). The refit is Platt scaling (a logistic curve on the log-odds), cross-fitted over 5 folds of blocks: no decision is refitted by a curve that saw its own games. A fold whose other folds hold one outcome only, or whose fit does not converge, is left out of both before and after. Brier score and a reliability diagram in 10 fixed bins, before and after, on the same decisions. The refit is not used to choose anything else.
Minimum bar, then cheapest
Bar: risk judgement at least 60 and error rate at most 35 PR (equity lost, not a share of mistakes). In each split, the model setups that clear the bar on the training half qualify, and the cheapest of them with a price on every training decision is chosen; it is then checked on the other half. Shown: how often each setup qualified, was chosen and cleared the bar on the held-out half; splits where none qualified, or only unpriced setups did. An unpriced setup is never called cheapest; programs are left out (Cost tab).
Confidence on right and wrong decisions
The chosen play's probability (never P(risk)), complete decisions only, in 10 fixed bins: decisions that were the engine's best play against the rest, and right risks against wrong and needless risks (0.001 margin).
Close choices
Stage A: results published (Confidence tab, model pages, Every decision filter). Stage B: five teams, results published. Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark stopped, not reported. (Leagues page, Stage B: five teams).
A decision is a close choice when the second most likely play has at least half the probability of the first (internally: split). Probabilities over plays, not sampled reasoning. The rules below were fixed and committed on 2026-10-24 before any close-choice number was computed. Inspired by "Can LLMs Anticipate Their Cruxes?". They describe the probabilities a setup gave; they say nothing about how it reasoned or why it chose.
Stage A: from recorded decisions
- Which decisions. Only decisions whose record gives a probability for every legal play (decided by the record's content, not by model type; the check is described under Which probabilities are complete). Left out and counted per setup: invalid answers, decisions asked in groups of options, incomplete distributions and records that cannot be verified. Setups that name one play only, and programs, show N/A. No model was asked again.
- The rule. Take the two most probable plays, with probabilities p1 and p2. A decision is a close choice when p2 is at least half of p1. Both are kept for every decision, with p1 + p2, so two strong candidates can be told apart from probability spread over many options. Equal probabilities are ordered by the order the plays were shown to the model, never by the engine; how many decisions have such a tie in the top two is reported.
- What differs between the two plays. From the option text the model was shown: the opponent's hitting rolls, the newly made points, both pip counts and the longest prime. Every feature that differs is recorded; when several differ, none is picked as the cause. Both plays' GNUbg 2-ply values, the gap between them and how far each is behind the engine's best play (two plays can be close to each other and both lose a lot).
- Risk. A close choice is between a safe play and a riskier one when one of the two plays leaves the fewest hitting rolls of all legal plays and the other leaves more. By stake band, in Risk pays positions only: close choices among all eligible decisions in the band. A big stake means playing safe costs more by the engine's value; it is not how hard the decision was. Both margins (0.001, 0.02); the rule for a close choice is the same at both.
- Missed risks. In Risk pays positions whose two most probable plays are a safe one and a riskier one: how often the play made was a missed risk, among close choices and among the other decisions. The other decisions are weighted to the close choices' mix of phase, number of options and the engine's gap between its two best plays (only groups holding both count); the crude rates are shown too. 95% interval: 2000 resamples of matches (a match's decisions stay together). Empty groups show N/A. An association, not a cause.
- Reading it. A high or low share of close choices is neither good nor bad; setups are never sorted or ranked by it. Shares come with count and denominator; under 200 decisions they are marked.
Stage B: five teams: does a setup know where it hesitates?
Stage B: five teams, results published. Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark stopped, not reported. The 5 subscription teams ran on 2026-10-25, 10:23–11:02 UTC; their results are on the Leagues page. Scope. Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark was started but not completed: two reboots and a GPU fault interrupted it, and it was stopped after 311 of 500 play calls on 2026-10-25. Its partial data is not reported. (Ümit's decision, 2026-10-25, 16:05 UTC.) The positions and the scoring rules were fixed on 2026-10-24, before any stage B call; the run conditions below on 2026-10-25.
- Positions. 100 positions from the new half of the shared standard set, by
risk class at the 0.001 margin: 40 Risk pays, 40 Safety pays and 20 No trade-off, spread over the
phases. Within each risk class and phase, the positions are picked by a seeded hash rule (seed text
close-choices-b-v2-2026-10-24:); the frozen file's sha256 is88bc3904…48410c. Race and bear-off positions have no contact, so they are all No trade-off; the Risk pays and Safety pays places of those phases go to opening, contact and holding. Every call for a position uses the same option order. - Calls. Each setup answers every position 5 times, and once more in a separate call that names the features it expects to differ between the plays it might choose. Every setup keeps its league settings exactly; a setup that gives the same answer every time is reported as deterministic on these positions, and settings are never changed to create variation. The separate call gets everything the play call gets (rules and strategy text, position, options in the same order) except the answer format: its question, a schema for the features and a neutral system prompt instead of the letter instruction. Before the run, one call per provider route on a position outside every shared set checks only that the format is accepted.
- One run. Each setup is run once: at most 2 retries of network or server errors per call, no rerun, and no invalid answer is asked again.
- Scoring. Observed set: the features of stage A that differ between any two of the plays chosen in the 5 answers. Predicted set: the features named in the separate call. Primary score: exact agreement of predicted and observed feature sets. Secondary score: partial overlap |P and O| / |P or O|; 1 when both empty, 0 when exactly one empty. A position is scored only when the prediction and all 5 answers are valid; left-out positions are reported with their reasons.
- Baseline. The predictions shuffled across positions within the same setup and phase, 1000 seeded shuffles, scored the same way.
- Limits. 5 repeats are not a measure of a model's whole behaviour. Stage B results are shown separately from stage A, per team against its own baseline, never as a ranking.
Nackgammon paired test
Results on the Nackgammon page. These results hold for these rosters only; each group is the setups in the run from that original league. The text below is §10 of the Method specification of 24 Oct 2026 as committed, with its dated notes and amendments (one local database path replaced by its description); §10.14 (groups by original league) says in (e) what had been opened before it was written, corrected in (f); (g) adds the Cup-roster setups after the results had been read, and (h) the exploratory Gemini 4 Argon setup and the output-limit rule. The spec keeps the setups' names from before the relabel of 28 Oct 2026; the results show the current names: Claude Fable 5.1 (on) is Claude Fable 5.1high, Grok 4.7 (on) is Grok 4.7high, GPT-6 Astra (on) is GPT-6 Astramedium, Gemini 4 Argon (off) and (on) are Gemini 4 Argonlow and (high).
10. Nackgammon paired test (amendment dated 2026-10-27, Can; written and committed before the position sets were built and before any model call)
Brief: mailbox/Nackgammon and Terra - brief 2026-10-27.md (with Can's correction of the same day to the Bonferroni rule). Code: tools/v3/nack_paired.py (sets, scores, Random) and tools/v3/selfplay.py (games, unchanged). Freeze: freeze/nack-paired.json. Files: positions/nack-paired/. This section is never changed after the first model call; a bug found later is fixed in a dated addendum below it, the original text kept.
10.0 What it replaces (superseded text kept)
This test replaces the Nackgammon part of the shared positions (A12's 500 Nackgammon positions). The 200-position opening control stays in Method but is not part of this run. The old text, superseded 2026-10-27 by Can (this section), kept for the record:
- Design (
mailbox/Backgammon benchmark v3 design final.md), sets table: "Nackgammon | 500 | GNUbg self-play from the Nackgammon start: 300 from turns 1 to 6, including each of its 15 first-roll positions once, and 200 from later turns | Tests how much known opening knowledge helps (A12)". Split: "Proposed Nackgammon split: 300 of the 500 come from turns 1 to 6, including its 15 first-roll positions; 200 come from later turns. The early 300 pair with the standard opening phase in A12. The later 200 show whether the effect fades as games drift back into ordinary middlegames." A12 row: "500 Nackgammon positions and the 200-position opening control; all setups … early Nackgammon positions." (superseded for the Nackgammon part; the table on/off contrast on the opening control stays). - Method page (
tools/site/method.py, set list): "Nackgammon: GNU Backgammon self-play from the Nackgammon starting position (an unfamiliar start, same rules). Its prompt drops the opening table." The live page is not edited now (no site change without Can's go); when it is, the line is kept and marked superseded by this section. - Draft
mailbox/Method amendment draft - A12 matched 2026-10-24 v1.md, section 3 (matched contrast against the frozen 500): superseded by this section.positions/nackgammon.jsonlstays frozen infreeze.jsonand is not used.
10.1 Question and hypothesis
Nackgammon has the rules of backgammon; only the start differs: each side starts with 2 checkers on its 24-point, 2 on its 23-point, 4 on its midpoint (13), 3 on its 8-point and 4 on its 6-point. The test measures how much a new starting setup changes a setup's (model, settings and prompt) decision quality and risk calls: how well a whole setup transfers. A larger drop is consistent with relying on familiar positions but is not proof, because Nackgammon positions also differ in structure. "Performance kept" is a valid result.
Hypothesis (a): error rates rise from standard to Nackgammon positions, most in the opening and less later in the game, and the rise differs between setups. It is examined through each setup's drop and label (10.8), the per-phase drops (descriptive, 10.9) and the primary comparisons (10.10).
Error rate (PR) = 500 × mean GNUbg 1.08 2-ply cubeless equity loss of the chosen play against the best legal play. Drop = PR on the Nackgammon positions − PR on their paired standard positions, over the same pairs. Positive = worse on Nackgammon.
10.2 Games, candidates and the one-position-per-game rule
- Self-play:
tools/v3/selfplay.pyunchanged (GNUbg 1.08.003 expert level: 0-ply, no noise, cubeless), seed tokennp20261027, games 0–9,999 of each variant (standard,nackgammon), 16 shards of 625 games per variant, CPU atnice -n 19. Source game ID:<variant>:np20261027:<game>. - Candidates: every turn with 2 to 25 legal plays is eligible (the rule of every v3 set). Per game and phase (10.3), at most 2 turns are drawn as candidates, at random with
random.Random("nack-paired-2026-10-27:<variant>:<game>:<phase>"). Every candidate gets 2-ply values for every legal play (10.6). No candidate is ever shown to a model. - One position per source game across the main set and the reserve together (per variant; the two variants' games are different games). Every position records its source game ID.
- Repeated states removed: a state is the board seen by the mover plus the dice (GNUbg position ID + dice, high die first; the v3 dedupe key). No state appears twice anywhere in the main set and the reserve, across both variants.
10.3 Phase definitions (tools/v3/classify.py, unchanged; first rule that matches)
- opening: turns 1 to 6 (turn 1 = the first move of the game, both sides' turns counted).
- bear-off: any of GNUbg's bear-off classes.
- race: GNUbg's race class.
- holding: a contact position (GNUbg contact or crashed class) where one side has escaped all its back checkers (none on the bar or on its points 14 to 24) and the other keeps one high anchor: 2 or more checkers on its 18-, 20- or 21-point and no other checkers on its points 18 to 24 (nor on the bar).
- contact: everything else, including GNUbg's crashed class.
10.4 Game-plan tags (classify.tag, unchanged)
For contact and holding positions only; opening, race and bear-off positions carry no tag and are reported under their phase. First rule that matches, checked for both sides: 1. blitz: a checker on the bar against a home board with 3 or more made points; 2. back game: 2 or more anchors (2+ checkers) on the side's points 18 to 24; 3. holding: the holding rule of 10.3; 4. priming: 4 or more made points in a row; 5. low-anchor: the side's only anchor in the opponent's home board (its points 19 to 24) is on its 22-, 23- or 24-point; 6. other. Note fixed now: a Nackgammon side that still holds its starting 23- and 24-points meets the back-game rule; the tags are reported as the rule gives them. Tags are not a matching variable; the tag mix of both sets is reported.
10.5 Position classes (existing rule, analysis.play_call, margin 0.001)
Hitting rolls of a play = describe.hitting_rolls (= Board::hitting_rolls, the number the model is shown). A position is No trade-off when every legal play leaves the same number of hitting rolls. Otherwise its stake = the 2-ply loss of the best safe play (fewest hitting rolls) against the best play: Risk pays when the stake ≥ 0.001, else Safety pays. Risk calls of an answer: right risk, wrong risk, needless risk, missed risk, right safety (as on the site).
10.6 Engine scoring
GNUbg 1.08.003, engine.PLY2 (2-ply, cubeless money equity, deterministic, prune on), every legal play of every position, through the same code as tools/v3/score.py evaluate. Gap = 2-ply equity of the best play − the second best. Stored with the sets (scores-2ply.jsonl). Random's error on a position = the mean loss over all its legal plays.
10.7 Matching, pairs per phase and class, tolerances
Pairs per phase and class (every pair has the same phase and class on both sides):
| Phase | Main: Risk pays | Main: Safety pays | Main: No trade-off | Main total | Reserve: Risk pays | Reserve: Safety pays | Reserve: No trade-off | Reserve total |
|---|---|---|---|---|---|---|---|---|
| opening | 30 | 30 | 0 | 60 | 45 | 45 | 0 | 90 |
| contact | 24 | 24 | 12 | 60 | 36 | 36 | 18 | 90 |
| holding | 12 | 12 | 6 | 30 | 18 | 18 | 9 | 45 |
| race | 0 | 0 | 25 | 25 | 0 | 0 | 38 | 38 |
| bear-off | 0 | 0 | 25 | 25 | 0 | 0 | 37 | 37 |
| total | 66 | 66 | 68 | 200 | 99 | 99 | 102 | 300 |
Tolerances (a standard partner must meet all, against the Nackgammon position):
- phase: equal; in the opening also |turn difference| ≤ 1;
- position class: equal;
- number of legal plays n: |Δn| ≤ 1 when the Nackgammon position has n ≤ 10, |Δn| ≤ 2 when n ≥ 11;
- gap g (both capped at 0.25): |Δg| ≤ max(0.005, 0.1 × g of the Nackgammon position).
Procedure (deterministic; h(x) = sha256 of "nack-paired-2026-10-27:" + x):
- Nackgammon candidates are taken in order of h(candidate key) (key = source game ID + turn). A candidate is taken when its (phase, class) cell still has quota, its game and its state are unused, and some unused standard candidate (game and state unused) meets every tolerance. Its partner is the one with the smallest distance d = |Δn| / tol_n + |Δg| / tol_g; ties by h(partner key). Both games and both states are then used.
- Main set first, to its quotas; then the reserve with the same procedure on what remains.
- Shortfall (only if the pool cannot fill a cell): the missing pairs go to the other trade-off class of the same phase, then to No trade-off of the same phase, then to the same class in contact; the procedure runs again for the moved quota. Every move is logged in the freeze and reported. The tolerances are never loosened.
- Order: pair IDs
np-0001…np-0200(main) andnp-0201…np-0500(reserve) in order of h("order:" + Nackgammon key) within each set. Position IDsnpn-NNNN(Nackgammon) andnps-NNNN(standard). Every setup answers the main 400 positions; an extended setup answers the reserve 600 positions, all of them, in pair-ID order, the same for every extended setup. - Option order:
order.option_orderwithseeds.json's option-order seed, as every v3 set.
Balance reported for both sets (main, reserve): per phase and overall, mean, median and standardised mean difference of n, gap, turn, pip count of the mover, the pip lead, Random's error and the stake; the class and GNUbg-class mix; the tag mix.
10.8 Setups, order, invalid answers, stopping rule, labels
Setups (each runs once, even when it is both in League 1 and in the Cup). Programs are yardsticks and belong to no group: GNUbg weak (GNUbg 1.08 Beginner, its stored League settings, answered through the harness) and Random (no calls: its error is the exact mean loss over all legal plays from the 2-ply values). Safest is not in the run.
- Decision models (every League 1 model setup in the final table of 2026-10-26, including those in the Cup; 16): celeris-1-decision, SubQ Snap Preview, Cygnet, Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwell, GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwell, DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release, Jev 1.13, GPT-6 Luna, Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, Mercury Decide, Winnow-12B, LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell, Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 Blackwell, Perplexity Decider Flash v1.2. (Four of them, DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release, both LFM2.5-24B-A2B setups and Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, are chat models placed in League 1 by the ladder; the group is League 1 as a whole.)
- Workhorses (the Hype Cup teams from League 2;
mailbox/Hype Cup roster 2026-10-27.md): Gemini 3.8 Flashlow, Claude Sonnet 5.5low, Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark; plus GPT-6.1 Terra (openai/gpt-6.1-terra, effort low) only if its smoke test passes after Can's go. - Frontier (the Hype Cup teams from League 3): GPT-6.1 Sollow, GPT-6 Astra (on), GPT-6.1 Solhigh.
Every setup runs with its stored settings (players.settings). Order: GNUbg weak and the decision models first, then the Cup teams not in League 1 (Gemini 3.8 Flashlow, Claude Sonnet 5.5low, GPT-6.1 Sollow, GPT-6 Astra (on), GPT-6.1 Solhigh), then Terra if it runs, and Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark last (the slowest; it is one of the 8).
Prompts. v3 is the leagues' prompt bg-v2 (leagues.prompt_version of Leagues 1–3). Both sets use v3 without the standard opening table, and the two prompts differ only in the line naming the starting setup: prompts/bg-v3-std-notable.md (standard side, sha256 1a145a09…, prompt version bg-v3-std-notable) and prompts/bg-v3-nack.md (Nackgammon side, sha256 13c744f0…, bg-v3-nack). Against bg-v2 the standard side drops the "Opening rolls" section and puts the setup sentence on its own line, naming the standard starting position. The existing nack-v2 is not used here: it states the standard layout. The runner test versions_differ_only_where_designed checks that the two rendered states differ in exactly that line. Each row carries its prompt; --prompt-version is never passed. Answers go to a separate SQLite file, not v3.db.
Invalid or missing answers (after the usual retries; also a timed-out or never-stored answer) are never dropped. Primary rule: such an answer scores the position's Random error (mean loss over all legal plays); in the risk analysis it counts with Random's weights (each legal play 1/n). Sensitivity checks, reported next to the primary result: (i) score it with the worst legal play's loss; (ii) leave its pair out for that setup. The count per setup and set is reported.
Intervals: paired bootstrap over pairs, 10,000 resamples, percentile intervals; the resample indices are shared by all setups (numpy default_rng, seed = the first 8 bytes of h("bootstrap:<look>") as an integer), so group means and differences come from the same resamples. Bootstrap intervals are approximate.
Per-setup stopping rule (b), two looks. First look after the 200 main pairs: the setup's drop with a 97.5% interval [lo, hi]. The setup is extended with the 300 reserve pairs only when lo ≤ 0 ≤ hi and (lo ≤ −5 or hi ≥ +5) PR. Examples: [−2, 6] is extended; [−2, 3] and [3, 9] are not. Final look: an extended setup's drop on all 500 pairs, 97.5% interval; a setup not extended keeps its first-look interval as final. Splitting the error over the two looks targets about 95% coverage overall; this is approximate. Every setup's final 97.5% interval is reported. GNUbg weak follows the same rule. Random needs no calls: it is reported on the 200 and on all 500 pairs.
Outcome label (c), from the final 97.5% interval: kept when −5 ≤ lo and hi ≤ +5; worse when lo > 0 and not kept; better when hi < 0 and not kept; unclear otherwise. At the first look "unclear" is exactly what is extended.
Results per setup: standard PR, Nackgammon PR and the drop, always shown together, never a ranking by drop alone; by phase and by game-plan tag; cost per decision (from the run's logs) alongside. Extensions refine each setup's own estimate and are reported per setup, separate from the primary comparisons. Chart: each setup's standard PR (x) against its Nackgammon PR (y), with GNUbg weak and Random marked and the diagonal drawn.
10.9 Descriptive by phase (part of hypothesis (a), not a test)
Per setup and per group: the drop by phase on the main 200 pairs, with 95% bootstrap intervals, and opening against the other four phases pooled. Labelled descriptive.
10.10 Primary comparisons (main 200 pairs only, which every setup answered)
Each is computed directly as A's drop − B's drop (group drop = mean of its setups' drops, each setup weighted equally), with a paired bootstrap interval over pairs (10.8). Bonferroni (Can's correction, 2026-10-27): the 5% error is divided by M; each interval's confidence level is 1 − 0.05/M. Both groups' (or setups') standard and Nackgammon PR are shown next to every difference. Scope, printed wherever group results appear: these results hold for these rosters only; League 1 is the full league; the other two groups are the Cup's selected top setups.
The comparisons (M = 5):
- decision models − workhorses;
- decision models − frontier;
- workhorses − frontier;
- effort pair LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark − LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell (both League 1 decision models);
- effort pair GPT-6.1 Solhigh − GPT-6.1 Sollow (both Cup teams).
No other model is in the run at two settings: Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, GPT-6 Astra (on), DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release, Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, Gemini 3.8 Flashlow and Claude Sonnet 5.5low run at one setting only. M = 5 with Terra and without Terra: GPT-6.1 Terra runs at one setting (low) and is a different model from GPT-6.1 Sol, so it creates or completes no effort pair; it only joins the workhorse group, which changes comparisons 1 and 3, not M. Level: 1 − 0.05/5 = 99% for every primary interval, whether or not Terra's go is given. M is final now, before the first model call.
10.11 Secondary analyses (fixed now, labelled secondary, no extra model calls)
- Kept advantage over random play: K = (Random's Nackgammon PR − setup's Nackgammon PR) ÷ (Random's standard PR − setup's standard PR), main 200 pairs (and all 500 for extended setups), 95% bootstrap interval. 1 = keeps its whole advantage over random play; 0 = falls to random. Flagged "unstable" when the setup's standard advantage over random is below 20 PR or its 95% interval reaches 0.
- Ranking: Spearman correlation across the model setups (yardsticks left out) between standard and Nackgammon PR on the main 200 pairs, 95% bootstrap interval over pairs; the five largest rank changes by name (ties by name).
- Risk: every position classed by 10.5 and every answer's risk call. Per setup and set: ability d′ and caution c (§2 rules: log-linear correction, missing-class N/A) and their change from standard to Nackgammon, with 95% paired bootstrap intervals over pairs. The number of positions per class and set is reported; a class with fewer than 50 positions in an analysis is called thin and is not read into.
Everything else (for example GPT-6.1 Terra against GPT-6.1 Sollow) is exploratory and labelled so.
10.12 Engine check (no model calls; engine time only)
- Subset, frozen before any model call: 40 of the 200 main pairs (80 positions, both sets): per phase opening 12, contact 12, holding 6, race 5, bear-off 5, the pairs with the smallest h("engine:" + pair ID) in each phase. The list is in
freeze/nack-paired.json. - After the answers: GNUbg 1.08.003 rollouts of the top 5 plays at 2-ply plus every play any setup chose there. Settings (as
tools/v3/score.py rollout): 1,296 trials per play, cubeless, variance reduction on, quasi-random dice, no truncation, no late evaluation, chequer play at 1-ply, seed 1, 8 threads,nice -n 19. - A play without a rollout (only Random and invalid answers need one) gets its 2-ply equity plus the position's mean (rollout − 2-ply) shift over its rolled-out plays; its loss is floored at 0. Counted and labelled.
- On the subset: PR, drops and the five primary comparisons with rollout values and with 2-ply values side by side; reported whether any comparison changes direction (sign of the point estimate). GNUbg's network learned standard play, so its scores for early Nackgammon positions are likely less accurate; this check bounds that, it does not remove it.
10.13 Release
A release bundle is prepared (positions with source game IDs, pairs, matching rules, prompts, settings, raw answers, scores and analysis code) and not published before Can's go.
10 addendum (2026-10-27, 18:05 UTC, Can's answers; written and committed before any model call of this test)
- Standard prompt (note to 10.8, Can's answer 2): the standard-side prompt wording (
bg-v3-std-notable) is approved. It differs from the league promptbg-v2(no opening table; the starting-setup sentence on its own line), so the paired test's standard error rates are not compared directly with league error rates. - Back-game tag (note to 10.4, Can's answer 3): the frozen back-game rule is kept. In Nackgammon this tag partly reflects the starting 23- and 24-point anchors, and the back-game row is too thin to read (main set: 10 Nackgammon against 2 standard positions).
10.14 Groups by original league (amendment dated 2026-10-27, Can; replaces the group definitions in 10.8 and 10.10)
Why: the Hype Cup is now built from the original-league tables (mailbox/Original-league tables - what Can found 2026-10-27.md), not from the final ladder tables, so the groups follow the same principle.
- a) Groups are defined by original league: the league a setup was in before any ladder move (
tools/league_web.pyleague_origins; a move before the ladder, the League 1 reshape of Oct 19, does not count). Each setup is weighted equally. Not by the Cup roster, not by the final league. - b) Membership rule: every setup in the paired test, including Cup teams added later from the corrected roster, joins its original league's group. Setups are added only because of the Cup roster, never because of paired-test results. A setup with no original league (one that has played in no league before this test) belongs to no group and is exploratory.
- c) M stays 5: the three group comparisons (decision models − workhorses, decision models − frontier, workhorses − frontier) plus LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark − (off) and GPT-6.1 Solhigh − (low), each at 1 − 0.05/5 = 99%. Any effort pair created by a later addition is exploratory. The scope note in 10.10 now reads: these results hold for these rosters only; each group is the setups in the run from that original league.
- d) Members as of now (the 22 model setups started so far; programs belong to no group):
- Decision models (original League 1, 12): celeris-1-decision, SubQ Snap Preview, Cygnet, Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwell, GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwell, Jev 1.13, GPT-6 Luna, Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, Mercury Decide, Winnow-12B, Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 Blackwell, Perplexity Decider Flash v1.2.
- Workhorses (original League 2, 9): DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release, LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell, Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, Gemini 3.8 Flashlow, Claude Sonnet 5.5low, Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, GPT-6.1 Sollow, GPT-6.1 Solhigh.
- Frontier (original League 3, 1): GPT-6 Astra (on). The group grows only when the corrected Cup roster adds original League 3 setups; until then comparisons 2 and 3 rest on one setup, and this is said wherever they appear.
- No group: GPT-6.1 Terra (it has played in no league; exploratory). Programs: GNUbg weak and Random (yardsticks). Both effort pairs now lie inside the workhorse group; comparisons 4 and 5 are unchanged.
- e) What had been opened before this amendment (reported by the run's agent, 2026-10-27): per-setup first looks on the main 200 pairs (standard PR, Nackgammon PR, drop, 97.5% interval, label, extension decision) for 21 setups plus GNUbg weak and Random, all "unclear" and extended. These were needed for the frozen extension rule. Group means and the primary comparisons under the old group definitions were computed twice by scripts on partial data (
results.jsonat 18:32 UTC; a local site build at ~19:00 UTC, checked by a verifier), but no person read a group number; nothing was committed, deployed or published. No result under the groups defined here had been computed. The amendment was written after those runs; this is stated wherever the primary comparisons are reported. - f) Correction to (e), dated 2026-10-28 (Zeki's readback of the retained agent records; (e) above is kept as written). The account in (e) ("computed twice", "no person read a group number") is incomplete:
- Six full-report calls, not two. Before this amendment was committed (2026-10-27 19:10:30 UTC), the run's agent and its site sub-agent directly issued six calls that produce the full report (group means, the primary comparisons and the secondary analyses under the earlier group definitions) on partial data: 18:32:52–18:32:58, the report/publish sequence completed at 18:37:01.140, further report/build calls at 18:39:26.567 and 18:44:48.905, and a snapshot-based call at 18:47:49.646 UTC. This counts invocations, not six independently verified complete computations; recomputations inside the verifier are not counted.
- The results reached an agent's context. At 18:32:58 UTC one of these calls returned numeric old-group primary results (differences, intervals and the groups' PR and drop fields) into the site sub-agent's context. They were not only a hidden machine comparison. Whether a person saw them is not established; it is not claimed that a person saw them, nor that they caused the change of groups.
- What stays as stated: the per-setup first looks were frozen extension looks, needed for the stopping rule; the notes committed before the first model call stand. This amendment (groups by original league) was written after the old-group results above had been computed and returned to an agent, so it is not result-blind.
- Unchanged: original-league groups with each setup weighted equally, M = 5 at 99% for the primary comparisons, the 97.5% per-setup stopping rule, the frozen position samples, the prompt non-comparability note and the back-game caveat.
- g) Cup-roster additions, dated 2026-10-28 (Can; written and committed before any model call of the setups below). The corrected Hype Cup roster (
mailbox/Hype Cup roster 2026-10-28.md, from the original-league tables) has four teams not yet in this test. Under (b) they are added and join their original league's group:- Frontier (original League 3): Claude Fable 5.1 (on) and Grok 4.7 (on).
- Decision models (original League 1): d1-ultra and GLiDEAPI (the hosted Fastino API setup; a different setup from GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwell, which stays in the group as before). Group sizes with the additions: decision models 14, workhorses 9, frontier 3. Each runs with its stored settings (
v3.dbplayers.settings, unchanged), on the same frozen pairs, prompts (bg-v3-std-notable/bg-v3-nack), option-order seed, rules, retries and invalid-answer rule: first the 400 main positions, then the 600 reserve positions when its first look calls for an extension under the frozen stopping rule (b) of 10.8. - Primary comparisons, reported both ways: (i) original members only (the 22 setups of (d)) and (ii) with these additions. (ii) is the primary result per (b). M stays 5, each interval at 99%.
- Everything else (per-setup results, secondary analyses 10.11, the 10.12 engine check) is recomputed under the same frozen rules. For the engine check, the plays the new setups choose on the 40 frozen pairs are rolled out with the 10.12 settings and added to the stored rollouts.
- Timing, stated wherever these results appear: these additions are made after this test's results had been computed and read (2026-10-28). They follow the Cup roster, not the results.
- Gemini 4 Argon (on), exploratory (Can, 2026-10-28): added after a smoke test (stop if its invalid answers are frequent, over 5% in the smoke), under the same rules. It replaces the earlier Argon (high) plan (Argon (on) is thinking level high). It is not added because of the Cup roster, so it belongs to no group despite its original League 3: it enters no primary comparison and does not change M. It is reported per setup and labelled exploratory.
- h) Argon dropped, Argon (low) proposed, and the output-limit rule, dated 2026-10-28 (Can; written and committed before any new run of this test starts. Grok 4.7 (on)'s run, started at 15:33 UTC, continues as it was started).
- Gemini 4 Argon (on) is dropped from the test. Its smoke test (the 50 standard smoke positions,
bg-v2;mailbox/Nackgammon - Argon (on) smoke 2026-10-28.md) had 7 of 50 answers (14%) still thinking when they reached the 32,768-token output limit. The limit was 5%. Its 400 start decisions would take about 22 hours of request time (201 s per decision; about 5.6 h at parallel 4). None of its paired-test positions was asked. - Gemini 4 Argonlow instead, exploratory, not yet started. This is the setup labelled "Gemini 4 Argon (off)", which sends thinking level low. It covers Google's frontier model. Same rules as every setup; no group; it enters no primary comparison and does not change M. An estimate comes first; there is no model call before Can's go.
- Output-limit rule, fixed now. The output limit is 32,768 tokens for every setup. An answer cut off by the limit (the response reports the stop at the limit:
finish_reason"length",stop_reason"max_tokens",finishReason"MAX_TOKENS" or incomplete "max_output_tokens"; the one-token letter readout call is not counted) is invalid under the frozen rule of 10.8 and scores the position's Random error. Each setup's cut-off count is reported in its own column, next to a sensitivity check that leaves out the pairs with a cut-off answer. - Limits the runs so far actually used (from the settings and the stored requests in the run's answers database (not published)). These runs stand as run; the rule applies to new runs.
- 32,000 (the Anthropic route's default
max_tokens): Claude Sonnet 5.5low, Claude Fable 5.1 (on). - Set in the settings: GPT-6.1 Terra 8,000; Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark 24,000 for its reasoning call, then a one-token letter readout; LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark 32,000, then a one-token letter readout; DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release 16; Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark 16; LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell 1 (letter readout only).
- No limit sent; the provider's own limit applies:
- ChatGPT/Codex route: GPT-6.1 Sollow, GPT-6.1 Solhigh, GPT-6 Astra (on).
- Antigravity route: Gemini 3.8 Flashlow. Argon (on)'s smoke test stopped at 32,768 there.
- xAI route: Grok 4.7 (on), whose run started before this rule. Two of its first 178 answers used more than 32,768 output tokens (at most 40,238).
- Not applicable (decision services, local decision models and the program answer with one choice; no text is generated against a limit): celeris-1-decision, SubQ Snap Preview, Jev 1.13, GPT-6 Luna, Mercury Decide, d1-ultra, GLiDEAPI, GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwell, Cygnet, Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark, Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwell, Perplexity Decider Flash v1.2, Winnow-12B, Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 Blackwell, GNUbg weak.
- 32,000 (the Anthropic route's default
- Cut-off answers in the stored runs: none, for every setup (checked with the rule above). The new column will therefore show 0 for them, and their sensitivity check equals the primary rule.
- Flag for runs that sent no limit (Can, 2026-10-28, same day): old runs are not redone. An answer from a run that sent no output limit and used more than 32,768 output tokens is flagged in the cut-off column as "past 32,768, no limit sent" (for example 2 of Grok 4.7 (on)'s first 178 answers). It is counted separately from cut-off answers, and its score stays as it is: it is neither invalid nor left out.
- Gemini 4 Argon (on) is dropped from the test. Its smoke test (the 50 standard smoke positions,
- i) Late entries, exploratory, dated 2026-11-07 (Ümit's go, bus #2646; written and committed before any model call of the setups below). Eight league late entries are added: SubQ Snap 1.1, Elephant-1, Mammoth-1, Drex Max v1.1, Drex 2.0, Drex MaxNVIDIA RTX PRO 6000 Blackwell, Drex 2.0NVIDIA RTX PRO 6000 Blackwell and REAX 1.3 (REAX 1.3 only if Can approves reusing its container exception; until then it is not run).
- Exploratory, no group. They are not Cup-roster additions under (b)/(g). Like Gemini 4 Argonlow under (h), they belong to no group: they enter no primary comparison and do not change M (M stays 5).
- Timing, stated wherever these results appear: they are added after this test's results had been computed and read (2026-10-27/28, and the additions of 2026-10-28).
- Same rules and scoring as every setup: the frozen pairs, prompts (
bg-v3-std-notable/bg-v3-nack), option-order seed, each setup's stored league settings and option cap, the invalid-answer rule of 10.8, the output-limit rule of (h), the stopping rule (b) of 10.8 (400 main positions first, then the 600 reserve when the first look calls for it), and the 10.12 engine check (their new plays on the 40 frozen pairs rolled out with the 10.12 settings). - Unchanged: the 28 published setups and their results (
results/,results-2026-10-28/). The late entries' results go to a separate results directory and a separate dated section.
Clear risk (preliminary)
Preliminary / partial. The clear-risk scan (decisions where GNU Backgammon finds the risky or the safe play clearly better), its checks, a rollout check of a sample and a partial quiz of five models (two or four plays shown, not league play) are on the Clear risk page, each with its definitions and settings.
Cost
Cost per 1,000 decisions carries one of three labels. They are never mixed in one figure or ranking.
- API price estimate
- The provider's list price, on a stated date, × the tokens its API reported (where the API records no cost per answer, × our recorded token counts). The bill itself was not checked.
- API-equivalent (subscription)
- A setup played through a subscription, which is not billed per token, priced at the API list price × the tokens reported. Not money paid.
- GPU time, at most
- A setup run on our own machines (NVIDIA RTX PRO 6000 Blackwell (96 GB) at $2.49 per GPU-hour; NVIDIA DGX Spark (GB10) at its purchase price over 3 years plus electricity at the chip's measured draw): the wall clock from its first to its last answer, leaving out every gap over 10 minutes between consecutive answers, × the machine's hourly rate × the GPUs its server used, ÷ its decisions. Its own simultaneous matches count once. Other setups' servers on the same machine at the same time are not separated, so each setup is charged the whole device: an upper bound. A setup that also answered through a hosted API adds those answers' recorded cost. The cost cell shows the estimate first, then a small line naming it and the machine it used most ('estimate · DGX Spark', 'estimate · RTX PRO 6000 Blackwell'; '+ API' where some answers came from a hosted API); how it was counted is in its tooltip (hover, keyboard focus or tap) and its details box. Cost sorting and the cost charts use the estimate.
Every view shows how many of its decisions have a cost (cost coverage).
Time
- Request time runs from sending a request to receiving the answer. Time spent on retries and on waiting after provider errors is kept separate and is not part of it.
- Several requests can be in flight at once, so on our own hardware the time depends on the load.
- Time per decision is shown as the median and the 90th percentile.
- Time budget: the share of decisions slower than 30 seconds.
- The v3 leagues had no clock, so no game could be lost on time. The risk of going over a time budget in play was therefore not measured; the share above only shows how often a setup needed more time.
Limits
- One kind of risk. The risk calls measure one kind of risk, being hit. Gammon risk and priming are not in it.
- Different positions. Setups played different positions; league comparisons are not paired, and risk judgement is not adjusted for how hard the positions were.
- Checker plays only. The games have no doubling cube, so the error rate leaves out cube decisions. The human scale is a rough guide.
- The engine evaluation is not the strongest available. Plays are judged at 2-ply with GNU Backgammon, not by rollouts, in cubeless money equity with match score ignored. Small differences between close plays can be wrong. GNU Backgammon learned standard play, so its scores for early Nackgammon positions are likely less accurate.
- The prompt steers. The strategy text and the question say what a good move looks like. The results measure how well models follow that guidance with the numbers in front of them, as much as how well they play.
- Strategy text is in training data. Rules, opening tables and strategy articles are online. Backgammon is not an unfamiliar task.
- Setups get different inputs. Rules text length and option groups differ between setups; rankings mark rows that got different inputs. The shared positions avoid group splits.
- Providers vary. A hosted model can change without notice, and providers differ in quantization. We log them; we cannot control them.
- Runner commits. Many original league matches were started from a runner checkout with uncommitted changes (League 1: 127 of 171 matches; League 2: 140 of 171 matches; League 3: 64 of 66 matches), recorded per match as harness_dirty. The recorded commit therefore does not fully identify the code that ran. This is a provenance limit, not evidence that any score is wrong; no number changes.
- Models only play each other and the anchor. Their strength against human players is not measured.
What the input adds Earlier version
Computing each play's consequences in code cut the error rate by about two thirds: from 120 in league #6 to 38.5 in league #7, the step that added what each play leads to (hits, blots left, how many of the opponent's 36 rolls hit them, points made). This is the strongest finding so far.
Swipe for more columns →
| League | What changed | Games | Hit the 300-turn limit | Moves that hit | Average game | Best opening play | Error rate |
|---|---|---|---|---|---|---|---|
| #1 | Structured state (JSON) with a short rules text and how every earlier game ended | 38 | 82% | 62% | 277 turns | 28% | 138 |
| #3 | Rules and strategy text, then score, time, position and dice; nothing about earlier games | 170 | 81% | 60% | 274 turns | 77% | 132 |
| #6 | + the question before the choices; Decider v1.1 self-hosted | 124 | 71% | 58% | 255 turns | 83% | 120 |
| #7 | + what each play leads to, and the race line | 135 | 0% | 9.7% | 50 turns | 68% | 38.5 |
| #8 | As #7 without d1 (7 models) | 120 | 0% | 9.5% | 51 turns | 69% | 39.4 |
| #11 | Repeat, new dice | 112 | 0% | 9.3% | 50 turns | 58% | 40.1 |
| #12 | Repeat, new dice | 112 | 0% | 9.7% | 51 turns | 69% | 36.2 |
| #13 | + the last 10 moves of the game | 106 | 0% | 10.9% | 52 turns | 72% | 39.1 |
Each league added one change to the prompt. Error rate: GNU Backgammon's, over all model decisions of the league (earlier scoring), lower is better; grey rows did not finish. The full story: Why it looks like this.
Learning from experience Running (preliminary)
Running (preliminary): see Did they improve? (every model) and Learning across games (the pilot, one model).
Routing and teams Analysis pending
Which decisions a cheap model can handle before handing over to a strong one, and whether a team picks better plays than its best member, wait for the shared-position answers. A team "oracle", the best answer any member gave, is an upper bound and never a working system; the earlier version's team results are on Leagues.
Tail risk
The 95th percentile of the equity lost per decision and the mean loss of the worst 5% of decisions, per league.
League 1
Swipe for more columns →
| Model and setting | Error rate (PR), 95% interval | P95 equity loss | Mean loss, worst 5% | Blunder rate | Confident and badly wrong | Decisions scored | Invalid answers |
|---|---|---|---|---|---|---|---|
| GNUbg weakprogram · CPU | 21.418.8 to 24.3 | 0.217 | 0.312 | 19.0% | N/A | 1,744 | 0 |
| Jev 1.13decision model · API | 34.431.0 to 38.0 | 0.352 | 0.534 | 25.1% | 0.4% | 1,911 | 0 |
| GLiDEv1.2 · NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 35.031.6 to 39.0 | 0.340 | 0.541 | 24.6% | 0.0% | 2,003 | 0 |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first release | 35.631.7 to 39.7 | 0.338 | 0.556 | 25.2% | 3.8% | 1,736 | 0 |
| celeris-1-decisiondecision model · API | 35.632.0 to 39.4 | 0.339 | 0.531 | 25.8% | 2.7% | 1,922 | 0 |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 36.933.9 to 40.1 | 0.348 | 0.510 | 27.9% | 0.1% | 2,028 | 0 |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs | 38.635.4 to 41.9 | 0.357 | 0.570 | 27.5% | 0.7% | 2,016 | 0 |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkdecision model · our GPUs | 38.835.3 to 42.5 | 0.389 | 0.569 | 26.8% | 0.5% | 2,052 | 0 |
| GPT-6 LunaDecisions APIdecision model · API | 43.339.7 to 46.9 | 0.410 | 0.631 | 28.9% | 0.5% | 1,962 | 0 |
| SubQ Snap Previewdecision model · API | 43.939.3 to 48.8 | 0.399 | 0.634 | 31.8% | 0.1% | 2,200 | 0 |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark | 45.040.7 to 49.9 | 0.422 | 0.680 | 30.7% | 4.0% | 2,005 | 0 |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark | 47.744.1 to 51.2 | 0.431 | 0.631 | 32.4% | 30.3% | 2,193 | 0 |
| Mercury Decidedecision model · API | 47.943.3 to 52.9 | 0.481 | 0.728 | 30.3% | 12.9% | 1,853 | 0 |
| Winnow-12BNVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 49.745.9 to 53.6 | 0.456 | 0.688 | 32.3% | 2.0% | 2,015 | 0 |
| Safestprogram · CPU | 51.647.4 to 56.2 | 0.445 | 0.675 | 35.1% | N/A | 2,058 | 0 |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 59.654.9 to 64.5 | 0.531 | 0.765 | 37.5% | 0.0% | 2,006 | 0 |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 Blackwell | 61.556.3 to 66.8 | 0.521 | 0.744 | 40.8% | 0.4% | 1,952 | 0 |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 Blackwelldecision model · our GPUs | 65.659.7 to 71.9 | 0.596 | 0.846 | 38.6% | 0.0% | 2,002 | 0 |
| Randomprogram · CPU | 121.1113.3 to 128.5 | 0.855 | 1.080 | 61.5% | N/A | 1,337 | 0 |
League 2
Swipe for more columns →
| Model and setting | Error rate (PR), 95% interval | P95 equity loss | Mean loss, worst 5% | Blunder rate | Confident and badly wrong | Decisions scored | Invalid answers |
|---|---|---|---|---|---|---|---|
| Gemini 3.8 Flashhighchat model, one letter · subscription | 14.912.9 to 17.0 | 0.167 | 0.305 | 11.0% | N/A | 2,168 | 0 |
| Gemini 3.8 Flashlowchat model, one letter · subscription | 16.314.5 to 18.2 | 0.180 | 0.279 | 13.5% | N/A | 2,053 | 0 |
| Claude Sonnet 5.5highchat model, one letter · subscription | 16.714.9 to 18.7 | 0.187 | 0.317 | 13.3% | N/A | 2,164 | 0 |
| DeepSeek-V4.1-Flashlow · API · 1017chat model, answer-letter probabilities · API | 17.115.1 to 19.2 | 0.199 | 0.330 | 13.4% | 13.0% | 2,222 | 12 |
| Claude Fable 5.1low | 17.915.9 to 20.1 | 0.185 | 0.313 | 15.1% | N/A | 2,008 | 0 |
| DeepSeek-V4.1-Flashlow · API + NVIDIA RTX PRO 6000 Blackwell · first releasechat model, answer-letter probabilities · API | 19.016.6 to 21.3 | 0.197 | 0.372 | 13.9% | 13.0% | 2,188 | 38 |
| Grok 4.7low | 19.116.9 to 21.4 | 0.209 | 0.354 | 15.1% | N/A | 2,152 | 0 |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs | 19.617.5 to 21.7 | 0.203 | 0.348 | 15.6% | 14.2% | 2,122 | 0 |
| Claude Sonnet 5.5lowchat model, one letter · subscription | 20.117.8 to 22.4 | 0.215 | 0.379 | 15.7% | N/A | 2,150 | 0 |
| Gemma 4 31Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs | 22.519.9 to 25.3 | 0.222 | 0.376 | 18.2% | 14.4% | 1,847 | 0 |
| GNUbg weakprogram · CPU | 27.024.1 to 29.8 | 0.263 | 0.384 | 22.1% | N/A | 2,136 | 0 |
| Qwen3.8-27Blow · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Sparkchat model, answer-letter probabilities · our GPUs | 29.026.5 to 31.8 | 0.290 | 0.463 | 22.4% | 20.3% | 2,108 | 0 |
| d1-ultra | 30.327.3 to 33.4 | 0.289 | 0.487 | 22.4% | 0.0% | 2,071 | 0 |
| GLM-5.3lowchat model, answer-letter probabilities · API | 30.927.8 to 34.0 | 0.310 | 0.512 | 21.7% | 14.4% | 1,989 | 0 |
| Jev 1.14 | 31.728.3 to 35.7 | 0.286 | 0.465 | 24.7% | 0.1% | 1,912 | 0 |
| GLiDEAPI | 33.230.2 to 36.4 | 0.323 | 0.515 | 24.1% | 0.0% | 2,159 | 0 |
| Qwen3.8 Flashoffchat model, answer-letter probabilities · API | 33.630.8 to 36.7 | 0.308 | 0.533 | 25.3% | 0.1% | 2,027 | 0 |
| DeepSeek-V4.1-Flashoff · API · 1017chat model, answer-letter probabilities · API | 35.231.8 to 38.8 | 0.365 | 0.591 | 23.6% | 8.5% | 2,034 | 0 |
| ClefNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark | 36.533.5 to 39.6 | 0.334 | 0.531 | 27.1% | 0.0% | 1,978 | 0 |
League 3
Swipe for more columns →
| Model and setting | Error rate (PR), 95% interval | P95 equity loss | Mean loss, worst 5% | Blunder rate | Confident and badly wrong | Decisions scored | Invalid answers |
|---|---|---|---|---|---|---|---|
| GPT-6.1 Solhigh | 9.98.6 to 11.3 | 0.103 | 0.187 | 7.8% | N/A | 1,389 | 0 |
| GPT-6 Astramediumchat model, one letter · subscription | 11.19.5 to 12.9 | 0.125 | 0.222 | 8.6% | N/A | 1,362 | 0 |
| GPT-6 Astralowchat model, one letter · subscription | 11.89.5 to 14.2 | 0.129 | 0.246 | 9.4% | N/A | 1,251 | 0 |
| GPT-6.1 Sollow | 12.110.5 to 13.7 | 0.137 | 0.254 | 9.4% | N/A | 1,344 | 0 |
| Gemini 4 Argonhighchat model, one letter · subscription | 12.610.5 to 14.8 | 0.145 | 0.267 | 8.9% | N/A | 1,332 | 36 |
| Claude Fable 5.1highchat model, one letter · subscription | 14.011.5 to 16.3 | 0.160 | 0.266 | 11.7% | N/A | 1,286 | 0 |
| Claude Opus 5.5highchat model, one letter · subscription | 14.412.1 to 17.0 | 0.175 | 0.287 | 11.4% | N/A | 1,477 | 0 |
| Claude Opus 5.5lowchat model, one letter · subscription | 15.413.1 to 18.0 | 0.158 | 0.279 | 13.7% | N/A | 1,411 | 0 |
| Gemini 4 Argonlowchat model, one letter · subscription | 16.513.7 to 19.5 | 0.181 | 0.331 | 12.4% | N/A | 1,484 | 0 |
| Grok 4.7highchat model, one letter · subscription | 18.815.9 to 21.8 | 0.215 | 0.367 | 14.0% | N/A | 1,337 | 0 |
| Kimi K3.1 Prooffchat model, one letter · API | 28.024.5 to 31.8 | 0.267 | 0.448 | 21.7% | N/A | 1,167 | 0 |
| GNUbg weakprogram · CPU | 28.125.0 to 31.4 | 0.254 | 0.364 | 23.3% | N/A | 1,479 | 0 |
Runner changes during the leagues
- Voided matches. The first matches of LFM2.5-24B-A2B (off, on) and Qwen3.8 Flashoff were voided on 19 October and replay from scratch with the same dice. The runner had read the letter probabilities from the wrong token: for LFM2.5, Liquid's API returns probabilities for the reasoning tokens too, so every decision read a letter from the reasoning; for Qwen3.8 Flash, some decisions read the "c" of a split "choice" key. It now reads them at the reply's letter and counts a mismatch as invalid. LFM2.5 now runs on our GPU from Liquid's bf16 checkpoint, with the letter read after a prefilled "Answer:".
- Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark replayed. On our vLLM server the list of token probabilities ends with an end-of-message token after the letter, so the runner found no letter and played the letter Qwen had sampled instead of its most likely one, in every one of its decisions there. The runner now skips such trailing tokens; Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark's affected matches were voided on 20 October and replayed with the same dice. No other setup was affected.
- Withdrawn. Qwen3.8 Flash Next (off, on): with reasoning on it took many minutes per decision on a DGX Spark. Its matches are removed from the league. Qwen3.8 Flashoff, through OpenRouter, joined instead.
- Moved. On 19 October League 1 became decision models only: Gemma 4 31B (off, on) and Qwen3.8-27B (off, on) moved to League 2, and their League 1 matches no longer count. GPT-6 Luna's first entry, through the ChatGPT subscription as a chat model, was removed; it plays through OpenAI's Decisions API. The other new decision models joined League 1 the same day; GLiDEAPI and Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 Blackwell joined on 20 October.
- Hardware. Clef, Cygnet and Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX Spark moved from our RTX PRO 6000 cards to DGX Spark (GB10) machines on 16 October, with the same checkpoints and settings (replayed requests: the same play in all). Open-Jev stayed, because on a DGX Spark it sometimes chose another play. Gemma 4 31B ran on a DGX Spark from 19 October 18:36 to 22:26 UTC, then on an RTX PRO 6000 again (the same play in all replayed requests). Qwen3.8-27B started on Cerebras through OpenRouter, where it finished no match, and then moved to our GPU in fp16 (the same top letter in most replayed requests). From 20 October 21:10 UTC Gemma 4 31B, LFM2.5-24B-A2B and Qwen3.8-27B played on DGX Sparks with the same checkpoints and settings (the same top letter in all replayed requests), so that DeepSeek-V4.1-Flash could have all of our RTX PRO 6000 cards; on 21 October 01:25-05:00 UTC they played their last non-DeepSeek matches on the RTX PRO 6000 cards again (LFM2.5 is not deterministic, so its replays agree less often).
- DeepSeek-V4.1-Flash on our GPUs. From 20 October 21:56 UTC, DeepSeek-V4.1-Flash (off, on) plays on our own SGLang server instead of Fireworks through OpenRouter: the 17 October revision of the published weights (FP4 routed experts, FP8/BF16 elsewhere); Fireworks does not say which revision it serves. On stored Fireworks requests our server usually chose the same top letter, but not always; it is not deterministic, and Fireworks replaying its own stored answers did not always agree either.
- DeepSeek-V4.1-Flash back on Fireworks. From 22 October 15:37 UTC, for League 2's last four matches, DeepSeek-V4.1-Flash (off, on) plays on Fireworks through OpenRouter again, with the provider pinned to Fireworks and fallbacks off, as before 20 October 21:56 UTC; Gemma 4 31B, LFM2.5-24B-A2B and Qwen3.8-27B moved from the DGX Sparks back to our RTX PRO 6000 cards (same checkpoints and settings; replayed requests: the same top letter in 20 of 20 for Gemma and Qwen, 17 of 20 for LFM2.5, which is not deterministic). Matches under way keep their stored turns; every decision records its provider, so it shows which turns ran where.
- Gemini by subscription. With no Gemini API key, Gemini 3.8 Flash (low, high) plays through the Google Antigravity subscription from 19 October 12:46 UTC: the same model, prompt, thinking levels and constrained letter. On the smoke positions it nearly always chose the same play as the API.
- Paused matches. A match that fails on a provider error (OpenRouter credit or rate limits, a server restart) waits and resumes later from its last stored turn; the other matches go on.
The anchor
The anchor, GNU Backgammon 1.08 at its Beginner level (0-ply, deterministic noise 0.06), ran on one x86-64 build
only, and on x86-64 it is not the Beginner that GNU Backgammon intends. Its deterministic-noise code sums the hash
bytes as char, which is signed on x86-64, but subtracts the mean of unsigned bytes, so every noise draw
is shifted by about −7 standard deviations. The anchor understates its own winning chances and plays for gammons.
- Another build plays differently. On the 1,000 v1 positions, the x86-64 and ARM builds of the same package choose different plays in 659; with the noise off, in 6, all exact ties. Error rate 21.8 on x86-64 against 16.8 on ARM.
- Gammons in self-play. 72% of games end in a gammon or backgammon on x86-64, 40% on ARM and about 30% without noise.
So v3's anchor is "GNUbg 1.08 Beginner, x86-64 build", and its results are reproducible on x86-64 only: a replay there gives the recorded plays. Later releases of the benchmark use a corrected build.
Method experiments
Two experiments with GNU Backgammon players only, run to choose how the leagues work. They are method findings, not results about any model.
Sharing dice across matches
Method finding, not a model result
Each match has its own dice. We tested whether giving round r of every match the same roll schedule would make a league table less a matter of luck. The test used 8 GNU Backgammon players of known strength and no model, and replayed a round robin 500 times under each scheme. The players' order was checked in a separate long run of 11,200 matches: in all 28 pairs the stronger player scored more league points per match.
- Shared schedules made the table noisier. League points scattered more: standard deviation ×1.10 (95% interval 1.06 to 1.14); variance ×1.20 (1.11 to 1.29).
- The order got slightly worse. Rank correlation with the true order fell from 0.890 to 0.864 (difference −0.026, 95% interval −0.038 to −0.014). The stronger of two neighbours finished above the weaker in about 72% of replays under both schemes.
- Why. Mirrored games already cancel most of the dice luck within a match. Sharing schedules across matches cancels nothing more, and it puts all of a player's matches on the same dice, so their luck adds up: two matches of the same player correlated +0.042 with shared schedules against +0.001 without.
So the leagues keep separate dice per match. The test players are not language models, whose play varies more from match to match.
Checks of the reference player
Method finding, not a model result
The anchor is GNU Backgammon 1.08 at its Beginner level (0-ply, deterministic noise). We checked whether it is the same player on every machine.
- It is reproducible on x86-64. Replaying all 4,387 recorded anchor decisions on the x86-64 build gives the recorded play in 4,387; on an ARM build of the same package, in 31.0%.
- Another build plays differently. On the 1,000 v1 positions, the x86-64 and ARM builds choose different plays in 659. With the noise off, they differ in 6, all exact ties, so the engine's own scoring does not depend on the build. Error rate 21.8 on x86-64 against 16.8 on ARM.
- Why. The noise code sums hash bytes as
char, which is signed on x86-64, but subtracts the mean of unsigned bytes. On x86-64 every noise draw is shifted by about −6.9 standard deviations: the anchor understates its own winning chances and plays for gammons. - Gammons in self-play (300 games each): 71.7% of games end in a gammon or backgammon on x86-64, 40.3% on ARM, 30.0% without noise.
So the v3 anchor is "GNU Backgammon 1.08 Beginner, x86-64 build". It is a fixed reference, not GNU Backgammon's Beginner as published elsewhere. See also The anchor.