About

About the benchmark

Backgammon in one page, how the benchmark works, how the scores work and the leaderboard rules; then the method, the data and how to run it.

Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.

Backgammon in one page

The goal. Two players race their 15 checkers around the board and off it. The first player to bear off all 15 checkers wins.

The board. The board has 24 narrow triangles, called points. Each player moves in the opposite direction. Each player's last six points are their home board.

Moving. A player rolls two dice and moves checkers by those two numbers. One checker can take both numbers, or two checkers can take one each. A roll of doubles (for example 4-4) is played four times. A checker cannot land on a point that holds two or more of the opponent's checkers.

Hitting. A single checker alone on a point is a blot. If the opponent lands on it, the blot is hit: it goes to the middle of the board (the bar). It must come back in through the opponent's home board before any other checker can move. Being hit costs a lot of time in the race.

Bearing off. When all 15 of a player's checkers are in their home board, that player can start taking them off the board.

Gammons. If the loser has borne off no checkers, the winner wins a gammon, worth 2 points. If the loser also still has a checker on the bar or in the winner's home board, it is a backgammon, worth 3 points.

Why this is a test of risk. Almost every turn asks the same question. A safe play leaves no blot. A bolder play leaves a blot but builds a better position. The dice decide whether the risk pays off this time. A strong player takes the risk when it is worth it on average, not only when it feels safe.

Nackgammon. Nackgammon is backgammon with a different starting position. Each player moves one checker from the 13-point and one from the 6-point to their opponent's side, so each side starts with four back checkers instead of two. We use it to test whether a model still plays well when the opening positions are new.

How the benchmark works

How the scores work

As the Method page states it

  • Who scores. GNU Backgammon 1.08 evaluates every legal play at 2-ply, in cubeless money equity (expected points per game, match score ignored). The loss of a decision is the equity of the best play minus the equity of the play made. Method: Error rate
  • Error rate. 500 × mean 2-ply equity loss per decision. Lower is better. This is the checker-play performance rating (PR) that backgammon players use. Method: Error rate
  • Risk judgement. The average of risks found and right safety, × 100. It does not depend on how many positions of each class a setup met. With both classes present, the Safest program scores exactly 50, the "always safe" line. Method: Was it the right risk?
  • Intervals. Intervals are 0.95 bootstrap intervals with 2000 resamples and a fixed seed (20261021). League decisions resample whole games, because moves within a game are not independent. Method: Statistics and verdicts
  • Different positions. Setups played different positions; league comparisons are not paired, and risk judgement is not adjusted for how hard the positions were. Method: Limits

Quoted from the Method page.

Leaderboard rules

How the leaderboard is built

A fixed table: the filters on the Leagues page do not change it. Tap a model (or point at it) for its details and its decisions.

Rank counts 1, 2, 3 down the rows shown, by the measure chosen (Risk judgement, Error rate (PR) or Cost per 1,000 decisions), among the rows of the table's filters (models: All · Chat · Decision; class: All · Tiny · Small · Medium · Large · Efficient · Workhorse · Flagship, combined with the models; a class with no model in view has no key; rows: one per model or all setups); the programs are reference rows in every view and in no class, never ranked. Rank has no tiers: a tier needs a paired exact test (McNemar, Holm) over one question set every model answered, and the leagues played different positions, so no pair of models in different leagues shares one. The tier in a row's details comes from the same 200 positions.

Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. The maker's own figure (model card, announcement or model name), never our count; the standard price, never an introductory one, and "(provisional)" where the maker publishes only an introductory or launch price. Never estimated: Undisclosed is a model with neither (never a chat model, each has a per-token list price; a decision model with only a per-request price or none). The label after a model's name: its total (trillions to one decimal from 1,000B, whole from 10B, one decimal under 10B), a mixture of experts as "total MoE · active", or Closed and its price class; a row's details give the maker's figure the class uses and our checkpoint count. Parameters (in a row's details): the logical parameter count of the maker's released checkpoint, linked, with dense or MoE; a mixture of experts as total · active; "not disclosed by maker" where the maker gives none.

Bars run to each score; the whisker is its 95% interval. Ranks show observed scores; small gaps need not be meaningful. Leagues played different positions.

Row rule: each model's highest reasoning setting (off < minimal < low < medium < high < xhigh < max), late entries included; a model with one setup uses it; several setups at that setting: the first in league order, then by setup name, never by score.

Each model's row is its highest reasoning setting played, by the effort the requests sent (off < minimal < low < medium < high < xhigh < max; on above off for a model without levels; a decision model's single setup), never its best score; late-entry setups count as the model's setups (a late entry stands in the league order of the league whose field it played). Several setups at that setting: the first in league order, then by setup name (no score). Every value of a row and of its details is that setup's own; its other settings are rows of their own under All setups. Version names are separate models; DeepSeek-V4.1-Flash1017 is under DeepSeek-V4.1-Flash (a dated checkpoint under its original). Every column of every setup: Every number.

Released (in a row's details): the month of the maker's own dated statement that this exact model was released or became available to the public (an announcement before public availability does not count), linked; left out where no such statement is on record (33 of 57 models).

Rules of backgammon, as the prompt states them

The rules as every player reads them in the prompt (bg-v2, the current version: Method: The prompt), quoted unchanged.

Setup

The board has 24 points in a horseshoe-shaped track, 12 on each side. Each player numbers the points from their own side, 1 to 24, and moves from the 24-point toward the 1-point; the two players move in opposite directions. Each player starts with 15 checkers: two on the 24-point, five on the 13-point (the midpoint), three on the 8-point, and five on the 6-point.

Points 1-6 are your home board and points 7-12 your outer board. The 7-point is the bar point.

Moving

Hitting and entering

Bearing off

Once all 15 of your checkers are in your home board, you may bear them off: a 1 removes a checker from the 1-point, a 2 from the 2-point, and so on. A die higher than your highest occupied point removes a checker from that highest point. For example, with 6-5, no checkers on the 6-point, and two on the 5-point, both checkers come off the 5-point. A checker hit during the bear-off must enter and come home before bearing off continues.

Scoring

The first player to bear off all 15 checkers wins:

ResultWhenPoints
Single gameThe loser has borne off at least one checker1
GammonThe loser has borne off no checkers2
BackgammonThe loser has borne off no checkers and still has one on the bar or in the winner's home board3

Method

Every definition, threshold and caveat is on the Method page:

Data

results.json: every league's results per setup, one league per key, every risk metric at both margins. Per-decision records (positions, plays, the engine's best play) are not published: why.

Every data file, with its size and hash: Data.

How to run it

  1. Snapshot the league database: python3 tools/site/snapshot.py v3.db STATE (one read transaction; the prompts each model was sent go to STATE/prompts.db; the Nackgammon knockout's matches, from knockout-nack.db next to it, to STATE/knockout.db).
  2. Score every legal play of every decision with GNUbg 1.08 at 2-ply: tools/site/score.sh STATE/snap.db STATE/plays.
  3. Build and check the site: python3 tools/site/build.py --state STATE --out OUT. The build fails, and nothing is deployed, if a number does not trace to a data file, counts do not add up, leagues or versions mix, a published finding no longer holds, or a file contains a private address or credential.

The scripts are in the jevgammon repository under tools/site/; the fixed thresholds are in thresholds.json.

Late entries

A late entry joins an original league table after it was played and plays every setup of that table once, in matches of its own. Its games count in no standings table, no league table, no Hype Cup input and no other setup's numbers. The numbers here come from its own late-entry games only, computed as for a setup in a league. Late entries are ranked on the risk leaderboard; they do not change league match-points standings, Cups, or other setups' results.

Error rate and risk judgement: 95% intervals, whole games resampled. Ability d′ and caution c at the 0.001 margin: rates corrected by adding 0.5 to the counts and 1 to the totals; 95% intervals, blocks of two seat-swapped games resampled. Cut off: answers that stopped at an output limit the request sent (finish reason length, stop reason max_tokens, finishReason MAX_TOKENS, incomplete max_output_tokens), in the stored response or error. Effort: the reasoning effort the requests sent. Engine's best play: the share of its scored decisions that chose the engine's best play (equity loss under the best-play threshold). Time: median request seconds per decision (waits between retries not included). Cost per 1,000 decisions: the recorded tokens at the price shown, as labelled there (list or billed price; for a setup on our GPUs the hosted model's list price, not billed).

Every late entry, its numbers and its matches: Leagues: Late entries.

Open questions

What is not settled yet: every analysis in the plan with its status, and the limits on the Method page.

Swipe for more columns →

Every analysis in the plan, with its status and data file
AnalysisWhatStatusData fileWaits for the running leagues
A1Decision quality: error rate, best-play share, blunder rate, intervalsShared positions (main table)Analysis pendingdata/v3/shared.jsonno
A1Decision quality: error rate, best-play share, blunder rate, intervalsLeague decisions, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json, data/v3/league-N/<setup>.jsonno
A1Decision quality: error rate, best-play share, blunder rate, intervalsv1 setEarlier versiondata/v1/v1-set.jsonno
A2Effect of the information given to the model (notation only / consequences / history)Shared positions, 6 setups x 3 promptsAnalysis pendingnone yetno
A3Return on more reasoning (paired: the same model at a lower and a higher setting)Shared positionsAnalysis pendingdata/v3/shared.jsonno
A3Return on more reasoning (paired: the same model at a lower and a higher setting)v1 set: DeepSeek-V4.1-Flash off vs thinkingEarlier versiondata/v1/v1-set.json (reasoning), data/v1/compare.jsonno
A4Where models fail: phase, strategy tag, number of options, gap between the two best playsShared positionsAnalysis pendingdata/v3/shared.jsonno
A4Where models fail: phase, strategy tag, number of options, gap between the two best playsLeague decisions, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N/<setup>.json (phases, tags, options, gaps)no
A4Where models fail: phase, strategy tag, number of options, gap between the two best playsv1 setEarlier versiondata/v1/v1-set/<setup>.jsonno
A5Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasLeague decisions, per league (Confidence tab: cutoffs 0.90/0.99, surest quarter, AUROC best play and blunder warning, ECE, confident and badly wrong; held-out cutoff in Method)League 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (confidence)no
A5Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasShared positions (setups with probabilities)Analysis pendingdata/v3/shared.jsonno
A5Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasv1 set (setups with probabilities)Earlier versiondata/v1/v1-set/<setup>.json (quality.auroc, ece, brier)no
A5Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasRisk-coverage curves, letter-position biasAnalysis pendingnone yetno
A6Routing from a cheap model to a strong oneShared positionsAnalysis pendingnone yetno
A7Teams: can a team beat its best member? (oracle as upper bound only)Shared positions / per leagueAnalysis pendingnone yetno
A7Teams: can a team beat its best member? (oracle as upper bound only)v1 set, earlier reportEarlier versioncontent/archive (earlier page section, moved to Leagues)no
A8Repeatability: 200 positions asked 5 timesShared positionsAnalysis pendingnone yetno
A9Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timeShared positionsAnalysis pendingdata/v3/shared.jsonno
A9Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timeLeague decisions, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (quality.time_*, usd_per_1000, cost_coverage)no
A9Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timev1 set (time, tokens; cost not measured)Earlier versiondata/v1/v1-set.jsonno
A10League results: standings, W-D-L, games, gammons; safest-play share and hitting rollsStandings per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (standings)no
A10League results: standings, W-D-L, games, gammons; safest-play share and hitting rollsRisk calls in play (B6, per league)League 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (risk.main, risk.robust)no
A11Doubling cube (optional track)-Not measurednoneno
A12Known opening knowledge: opening table on/off, NackgammonNackgammon 500 + opening control 200Analysis pendingdata/v3/shared.json (set sizes only)no
A13Does the shared-position score predict league results?Shared positions + finished leaguesAnalysis pendingnone yetno
High 1When does high hurt? Decisions that high fixes and breaks (counts and loss sizes)Shared positionsAnalysis pendingnone yetno
High 1When does high hurt? Decisions that high fixes and breaks (counts and loss sizes)v1 set: DeepSeek-V4.1-Flash off vs thinkingEarlier versiondata/v1/compare.json (fixed, broken, fixed_loss, broken_loss)no
High 2When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder rateShared positionsAnalysis pendingnone yetno
High 2When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder rateLeague decisions, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (quality.p95_loss, worst_mean, blunder_share)no
High 2When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder ratev1 setEarlier versiondata/v1/v1-set.jsonno
High 3When does high hurt? Confident and badly wrongLeague decisions, setups with probabilities, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (quality.confident_badly_wrong)no
High 3When does high hurt? Confident and badly wrongv1 setEarlier versiondata/v1/v1-set.jsonno
High 3When does high hurt? Confident and badly wrongShared positionsAnalysis pendingnone yetno
High 4When does high hurt? Going over a time budgetLeague decisions: share of request times over the budgetLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (quality.over_budget)no
High 5When does high hurt? High only when needed (switching rule on held-out positions)Shared positionsAnalysis pendingnone yetno
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02League decisions, per league, both marginsLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (risk.main, risk.robust, references), data/v3/league-N/<setup>.json (risk.*.phases)no
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Risk map, error rate by risk-call category, stake chart, risk call cards (league winners)League 1: Published · League 2: Published · League 3: Publishedsame; share/risk-card-league-N-*.pngno
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Every decision marks (position class, stake, risk call, best safe play)League 1: Published · League 2: Published · League 3: Publisheddecisions/<league>/<setup>/risk.json.gz and page filesno
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02v1 setEarlier versiondata/v1/v1-set.json (risk)no
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Shared positionsAnalysis pendingnone yetno
LLearning from experience-Not measurednoneno

More pages