About
About the benchmark
Backgammon in one page, how the benchmark works, how the scores work and the leaderboard rules; then the method, the data and how to run it.
Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.
Backgammon in one page
The goal. Two players race their 15 checkers around the board and off it. The first player to bear off all 15 checkers wins.
The board. The board has 24 narrow triangles, called points. Each player moves in the opposite direction. Each player's last six points are their home board.
Moving. A player rolls two dice and moves checkers by those two numbers. One checker can take both numbers, or two checkers can take one each. A roll of doubles (for example 4-4) is played four times. A checker cannot land on a point that holds two or more of the opponent's checkers.
Hitting. A single checker alone on a point is a blot. If the opponent lands on it, the blot is hit: it goes to the middle of the board (the bar). It must come back in through the opponent's home board before any other checker can move. Being hit costs a lot of time in the race.
Bearing off. When all 15 of a player's checkers are in their home board, that player can start taking them off the board.
Gammons. If the loser has borne off no checkers, the winner wins a gammon, worth 2 points. If the loser also still has a checker on the bar or in the winner's home board, it is a backgammon, worth 3 points.
Why this is a test of risk. Almost every turn asks the same question. A safe play leaves no blot. A bolder play leaves a blot but builds a better position. The dice decide whether the risk pays off this time. A strong player takes the risk when it is worth it on average, not only when it feels safe.
Nackgammon. Nackgammon is backgammon with a different starting position. Each player moves one checker from the 13-point and one from the 6-point to their opponent's side, so each side starts with four back checkers instead of two. We use it to test whether a model still plays well when the opening positions are new.
How the benchmark works
- Leagues of full games. Each model setup plays full games against every other setup in its league. There are three leagues, and they are never merged.
- One decision, one request. Every move is sent as its own request. A model sees the match score, but not the moves of earlier turns or earlier games.
- What the model sees. Each request shows the score, the board with pip counts, the race, the dice and every legal play. Next to each play the program lists: checkers hit, points made, blots left and how many of the opponent's 36 rolls hit them, points held, longest prime and pip counts.
- Two kinds of model. Decision models return a probability for every legal play. Chat models choose one play.
- A program judges, not a model. GNU Backgammon 1.08 scores every legal play in every position at 2-ply. No language model judges any move.
- Settings. Each setup is one model at one reasoning setting (off, low, medium, high). We record the time and the tokens of every decision, and its cost where a price is known.
- Reference programs. Three programs are shown as yardsticks and are never ranked: GNUbg weak (a weak setting of the engine, played in every league), Safest and Random. In League 1, Safest picks the play that leaves the fewest hitting rolls, then the fewest blots, then the biggest pip lead. In Leagues 2 and 3, Safest and Random are computed, not played: Safest is the engine's best of the safest plays, Random the average of all legal plays.
- Late entries. A setup added after a league finished plays that league's original field. It is ranked on the risk leaderboard, but it does not change league match-points standings, Cups, or other setups' results.
- Everything is public. Every position, every legal play, the engine's value for each play and every model's choice.
How the scores work
- Risk call. A position is a risk call when its legal plays leave different numbers of hitting rolls. The engine compares its best play with the best safe play: if the best play is at least 0.001 better, risk pays; otherwise safety pays.
- Risks found. The share of risk-pays positions where the model chose a worthwhile risky play: one at least 0.001 better than the best safe play. A risky play that is not that good counts as a wrong risk, not a risk found.
- Safe right. The share of safety-pays positions where the model played safe.
- Risk judgement. The average of risks found and safe right, from 0 to 100. A model that always plays safe scores 50. A model that gets every call right scores 100.
- Error rate. How much a model's moves give away, as judged by the engine: the PR scale that human players use. Lower is better. A move that matches the engine's best play costs nothing.
- Uncertainty. Each score has a 95% interval. Intervals resample whole games, not single moves, because moves inside one game are not independent. Ranks show observed scores; small gaps need not be meaningful.
As the Method page states it
- Who scores. GNU Backgammon 1.08 evaluates every legal play at 2-ply, in cubeless money equity (expected points per game, match score ignored). The loss of a decision is the equity of the best play minus the equity of the play made. Method: Error rate
- Error rate. 500 × mean 2-ply equity loss per decision. Lower is better. This is the checker-play performance rating (PR) that backgammon players use. Method: Error rate
- Risk judgement. The average of risks found and right safety, × 100. It does not depend on how many positions of each class a setup met. With both classes present, the Safest program scores exactly 50, the "always safe" line. Method: Was it the right risk?
- Intervals. Intervals are 0.95 bootstrap intervals with 2000 resamples and a fixed seed (20261021). League decisions resample whole games, because moves within a game are not independent. Method: Statistics and verdicts
- Different positions. Setups played different positions; league comparisons are not paired, and risk judgement is not adjusted for how hard the positions were. Method: Limits
Quoted from the Method page.
Leaderboard rules
- One row per model. The row uses a setting fixed by rule, not the best score: the model's highest reasoning setting. A model with one setup uses that setup.
- Versions. Different version names (Jev 1.13 and 1.14.0) are separate rows. A dated checkpoint of the same model (DeepSeek-V4.1-Flash1017) is the same model, and its row shows the original.
- Late entries count on the leaderboard, with a "late entry" tag, because they play the same league field.
- Order. Rows are sorted by risk judgement. The rank is a sort order, not a verdict. Ranks show observed scores; small gaps need not be meaningful.
- Tap a row to see both halves of its score, the raw counts, its error rate, its record and a link to every decision it made.
How the leaderboard is built
A fixed table: the filters on the Leagues page do not change it. Tap a model (or point at it) for its details and its decisions.
Rank counts 1, 2, 3 down the rows shown, by the measure chosen (Risk judgement, Error rate (PR) or Cost per 1,000 decisions), among the rows of the table's filters (models: All · Chat · Decision; class: All · Tiny · Small · Medium · Large · Efficient · Workhorse · Flagship, combined with the models; a class with no model in view has no key; rows: one per model or all setups); the programs are reference rows in every view and in no class, never ranked. Rank has no tiers: a tier needs a paired exact test (McNemar, Holm) over one question set every model answered, and the leagues played different positions, so no pair of models in different leagues shares one. The tier in a row's details comes from the same 200 positions.
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. The maker's own figure (model card, announcement or model name), never our count; the standard price, never an introductory one, and "(provisional)" where the maker publishes only an introductory or launch price. Never estimated: Undisclosed is a model with neither (never a chat model, each has a per-token list price; a decision model with only a per-request price or none). The label after a model's name: its total (trillions to one decimal from 1,000B, whole from 10B, one decimal under 10B), a mixture of experts as "total MoE · active", or Closed and its price class; a row's details give the maker's figure the class uses and our checkpoint count. Parameters (in a row's details): the logical parameter count of the maker's released checkpoint, linked, with dense or MoE; a mixture of experts as total · active; "not disclosed by maker" where the maker gives none.
Bars run to each score; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful. Leagues played different positions.
Row rule: each model's highest reasoning setting (off < minimal < low < medium < high < xhigh < max), late entries included; a model with one setup uses it; several setups at that setting: the first in league order, then by setup name, never by score.
Each model's row is its highest reasoning setting played, by the effort the requests sent (off < minimal < low < medium < high < xhigh < max; on above off for a model without levels; a decision model's single setup), never its best score; late-entry setups count as the model's setups (a late entry stands in the league order of the league whose field it played). Several setups at that setting: the first in league order, then by setup name (no score). Every value of a row and of its details is that setup's own; its other settings are rows of their own under All setups. Version names are separate models; DeepSeek-V4.1-Flash1017 is under DeepSeek-V4.1-Flash (a dated checkpoint under its original). Every column of every setup: Every number.
Released (in a row's details): the month of the maker's own dated statement that this exact model was released or became available to the public (an announcement before public availability does not count), linked; left out where no such statement is on record (33 of 57 models).
Rules of backgammon, as the prompt states them
The rules as every player reads them in the prompt (bg-v2, the current version: Method: The prompt), quoted unchanged.
Setup
The board has 24 points in a horseshoe-shaped track, 12 on each side. Each player numbers the points from their own side, 1 to 24, and moves from the 24-point toward the 1-point; the two players move in opposite directions. Each player starts with 15 checkers: two on the 24-point, five on the 13-point (the midpoint), three on the 8-point, and five on the 6-point.
Points 1-6 are your home board and points 7-12 your outer board. The 7-point is the bar point.
Moving
- Starting: each player rolls one die. The higher number moves first, playing both dice. Equal numbers are rolled again. After that, players alternate, rolling two dice each turn.
- Using the dice: move one checker by each die. One checker may take both moves, as long as each step lands on a legal point. A roll of 6-3 moves one checker six and another (or the same) checker three.
- Doubles are played four times: 5-5 is four moves of five.
- Using the whole roll: you must use both dice if you can. If only one die can be played, you must play the higher one when possible. If neither can be played, the turn is lost.
Hitting and entering
- A checker may land on an empty point, a point holding your own checkers, or a point holding exactly one opposing checker (a blot). Landing on a blot hits it and sends it to the bar.
- A checker may never land on a point held by two or more opposing checkers, so no point ever holds checkers of both players.
- A player with checkers on the bar must enter them all before moving anything else. Checkers enter in the opponent's home board: a 1 enters on your 24-point (the opponent's 1-point), a 2 on your 23-point, and so on up to a 6 on your 19-point. Entering on a blot hits it.
- If no die matches an open point, you dance and the turn is lost. Against a closed board you cannot enter at all until the opponent opens a point.
Bearing off
Once all 15 of your checkers are in your home board, you may bear them off: a 1 removes a checker from the 1-point, a 2 from the 2-point, and so on. A die higher than your highest occupied point removes a checker from that highest point. For example, with 6-5, no checkers on the 6-point, and two on the 5-point, both checkers come off the 5-point. A checker hit during the bear-off must enter and come home before bearing off continues.
Scoring
The first player to bear off all 15 checkers wins:
| Result | When | Points |
|---|---|---|
| Single game | The loser has borne off at least one checker | 1 |
| Gammon | The loser has borne off no checkers | 2 |
| Backgammon | The loser has borne off no checkers and still has one on the bar or in the winner's home board | 3 |
Method
Every definition, threshold and caveat is on the Method page:
- Setup
- Shared positions
- The prompt
- Why it looks like this
- The prompt
- Error rate
- Human scale (approximate)
- Was it the right risk? Definitions
- Statistics and verdicts
- Fixed thresholds
- Status labels
- Versions and play order
- Confidence
- Re-read, refit, bar and distributions (offline, from recorded decisions)
- Close choices
- Nackgammon paired test
- Clear risk (preliminary)
- Cost
- Time
- Limits
- What the input adds Earlier version
- Learning from experience Running (preliminary)
- Routing and teams Analysis pending
- Tail risk
- Runner changes during the leagues
- The anchor
- Method experiments
Data
results.json New models: first look Every model setup
results.json: every league's results per setup, one league per key, every risk metric at both margins. Per-decision records (positions, plays, the engine's best play) are not published: why.
Every data file, with its size and hash: Data.
How to run it
- Snapshot the league database:
python3 tools/site/snapshot.py v3.db STATE(one read transaction; the prompts each model was sent go toSTATE/prompts.db; the Nackgammon knockout's matches, fromknockout-nack.dbnext to it, toSTATE/knockout.db). - Score every legal play of every decision with GNUbg 1.08 at 2-ply:
tools/site/score.sh STATE/snap.db STATE/plays. - Build and check the site:
python3 tools/site/build.py --state STATE --out OUT. The build fails, and nothing is deployed, if a number does not trace to a data file, counts do not add up, leagues or versions mix, a published finding no longer holds, or a file contains a private address or credential.
The scripts are in the jevgammon repository under tools/site/; the fixed thresholds are in thresholds.json.
Late entries
A late entry joins an original league table after it was played and plays every setup of that table once, in matches of its own. Its games count in no standings table, no league table, no Hype Cup input and no other setup's numbers. The numbers here come from its own late-entry games only, computed as for a setup in a league. Late entries are ranked on the risk leaderboard; they do not change league match-points standings, Cups, or other setups' results.
Error rate and risk judgement: 95% intervals, whole games resampled. Ability d′ and caution c at the 0.001 margin: rates corrected by adding 0.5 to the counts and 1 to the totals; 95% intervals, blocks of two seat-swapped games resampled. Cut off: answers that stopped at an output limit the request sent (finish reason length, stop reason max_tokens, finishReason MAX_TOKENS, incomplete max_output_tokens), in the stored response or error. Effort: the reasoning effort the requests sent. Engine's best play: the share of its scored decisions that chose the engine's best play (equity loss under the best-play threshold). Time: median request seconds per decision (waits between retries not included). Cost per 1,000 decisions: the recorded tokens at the price shown, as labelled there (list or billed price; for a setup on our GPUs the hosted model's list price, not billed).
Every late entry, its numbers and its matches: Leagues: Late entries.
Open questions
What is not settled yet: every analysis in the plan with its status, and the limits on the Method page.
Swipe for more columns →
| Analysis | What | Status | Data file | Waits for the running leagues |
|---|---|---|---|---|
| A1 | Decision quality: error rate, best-play share, blunder rate, intervalsShared positions (main table) | Analysis pending | data/v3/shared.json | no |
| A1 | Decision quality: error rate, best-play share, blunder rate, intervalsLeague decisions, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json, data/v3/league-N/<setup>.json | no |
| A1 | Decision quality: error rate, best-play share, blunder rate, intervalsv1 set | Earlier version | data/v1/v1-set.json | no |
| A2 | Effect of the information given to the model (notation only / consequences / history)Shared positions, 6 setups x 3 prompts | Analysis pending | none yet | no |
| A3 | Return on more reasoning (paired: the same model at a lower and a higher setting)Shared positions | Analysis pending | data/v3/shared.json | no |
| A3 | Return on more reasoning (paired: the same model at a lower and a higher setting)v1 set: DeepSeek-V4.1-Flash off vs thinking | Earlier version | data/v1/v1-set.json (reasoning), data/v1/compare.json | no |
| A4 | Where models fail: phase, strategy tag, number of options, gap between the two best playsShared positions | Analysis pending | data/v3/shared.json | no |
| A4 | Where models fail: phase, strategy tag, number of options, gap between the two best playsLeague decisions, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N/<setup>.json (phases, tags, options, gaps) | no |
| A4 | Where models fail: phase, strategy tag, number of options, gap between the two best playsv1 set | Earlier version | data/v1/v1-set/<setup>.json | no |
| A5 | Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasLeague decisions, per league (Confidence tab: cutoffs 0.90/0.99, surest quarter, AUROC best play and blunder warning, ECE, confident and badly wrong; held-out cutoff in Method) | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (confidence) | no |
| A5 | Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasShared positions (setups with probabilities) | Analysis pending | data/v3/shared.json | no |
| A5 | Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasv1 set (setups with probabilities) | Earlier version | data/v1/v1-set/<setup>.json (quality.auroc, ece, brier) | no |
| A5 | Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasRisk-coverage curves, letter-position bias | Analysis pending | none yet | no |
| A6 | Routing from a cheap model to a strong oneShared positions | Analysis pending | none yet | no |
| A7 | Teams: can a team beat its best member? (oracle as upper bound only)Shared positions / per league | Analysis pending | none yet | no |
| A7 | Teams: can a team beat its best member? (oracle as upper bound only)v1 set, earlier report | Earlier version | content/archive (earlier page section, moved to Leagues) | no |
| A8 | Repeatability: 200 positions asked 5 timesShared positions | Analysis pending | none yet | no |
| A9 | Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timeShared positions | Analysis pending | data/v3/shared.json | no |
| A9 | Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timeLeague decisions, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (quality.time_*, usd_per_1000, cost_coverage) | no |
| A9 | Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timev1 set (time, tokens; cost not measured) | Earlier version | data/v1/v1-set.json | no |
| A10 | League results: standings, W-D-L, games, gammons; safest-play share and hitting rollsStandings per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (standings) | no |
| A10 | League results: standings, W-D-L, games, gammons; safest-play share and hitting rollsRisk calls in play (B6, per league) | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (risk.main, risk.robust) | no |
| A11 | Doubling cube (optional track)- | Not measured | none | no |
| A12 | Known opening knowledge: opening table on/off, NackgammonNackgammon 500 + opening control 200 | Analysis pending | data/v3/shared.json (set sizes only) | no |
| A13 | Does the shared-position score predict league results?Shared positions + finished leagues | Analysis pending | none yet | no |
| High 1 | When does high hurt? Decisions that high fixes and breaks (counts and loss sizes)Shared positions | Analysis pending | none yet | no |
| High 1 | When does high hurt? Decisions that high fixes and breaks (counts and loss sizes)v1 set: DeepSeek-V4.1-Flash off vs thinking | Earlier version | data/v1/compare.json (fixed, broken, fixed_loss, broken_loss) | no |
| High 2 | When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder rateShared positions | Analysis pending | none yet | no |
| High 2 | When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder rateLeague decisions, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (quality.p95_loss, worst_mean, blunder_share) | no |
| High 2 | When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder ratev1 set | Earlier version | data/v1/v1-set.json | no |
| High 3 | When does high hurt? Confident and badly wrongLeague decisions, setups with probabilities, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (quality.confident_badly_wrong) | no |
| High 3 | When does high hurt? Confident and badly wrongv1 set | Earlier version | data/v1/v1-set.json | no |
| High 3 | When does high hurt? Confident and badly wrongShared positions | Analysis pending | none yet | no |
| High 4 | When does high hurt? Going over a time budgetLeague decisions: share of request times over the budget | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (quality.over_budget) | no |
| High 5 | When does high hurt? High only when needed (switching rule on held-out positions)Shared positions | Analysis pending | none yet | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02League decisions, per league, both margins | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (risk.main, risk.robust, references), data/v3/league-N/<setup>.json (risk.*.phases) | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Risk map, error rate by risk-call category, stake chart, risk call cards (league winners) | League 1: Published · League 2: Published · League 3: Published | same; share/risk-card-league-N-*.png | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Every decision marks (position class, stake, risk call, best safe play) | League 1: Published · League 2: Published · League 3: Published | decisions/<league>/<setup>/risk.json.gz and page files | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02v1 set | Earlier version | data/v1/v1-set.json (risk) | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Shared positions | Analysis pending | none yet | no |
| L | Learning from experience- | Not measured | none | no |