Archived: this is the page as it was before the new site went live (2026-10-23). The current results are on the home page; this content now lives in Leagues and Method.

Decision models play backgammon

Seven decision models play full backgammon leagues against each other. A decision model does not write moves: it is given a description of the position and the complete list of legal plays, and it picks one. We changed what that description contains several times and watched how the games changed.

Updated 23 October 2026 · v1: 12 leagues run, 4 finished with the current prompt · v3: Leagues 1, 2 and 3 finished · every decision scored by GNU Backgammon

Finding

Describing, for every legal play, what the position looks like after it (blots left, how many of the opponent's 36 rolls hit one of them, points made, pips) changed the games more than anything else. GNU Backgammon's error rate fell from about 130 to about 38: each move gave away less than a third as much equity as before. About four in five games used to run to the 300-turn limit; after the change none did.

Among the league players, Gemma 4 31B is best with an error rate of about 25, beginner level on eXtreme Gammon's human scale; every decision model is in the lowest band, distracted (30 or more). Gemma 4 31B won three of the four finished leagues and Jev 1.13.0 one; Open-Jev 27B finished last in all four.

An All-star team that asks all seven and plays the most confident answer does not beat Gemma 4 31B alone: confidence shows how hard a position is, not which model is right.

The strongest players are frontier chat models: GPT-6 Astra (error rate 11.2, advanced on the human scale), then Gemini 4 Argon (16.9) and Claude Opus 5.5 (17.3), both intermediate, ahead of every decision model. They are also the slowest and, at API prices, the most expensive (see Speed and cost).

v3 shuffles the options and runs three leagues (live tables below): decision models (finished: Liquid d1-ultra first, and the weak GNU Backgammon anchor plays better than every decision model), workhorse chat models closed and open (finished: GPT-6.1 Sol (low) first; lowest error rate GPT-6.1 Sol (low), 10.8), and frontier chat models on subscriptions, each with reasoning off and on (finished: GPT-6 Astra (on) first). See v3 leagues.

v3 leagues

Final: data as of 23 October 2026, 02:50 UTC. League 1 has finished; League 2 has finished; League 3 has finished.

v3 keeps the v1 match rules (rounds of two games with the same dice, 3 rounds, 300-turn limit, no cube) and changes how the models are asked:

Every model decision in the matches that count was scored with GNU Backgammon, as in v1 (see Error rate for the method and the human scale): all 81,386 decisions, none skipped.

League 1: decision models

171 of 171 matches finished, 904 games

#SetupPWDLPtsGamesTo playError rate (95%)Engine's best playBlunders
1Liquid d1-ultra1816024867-35030.0 27.2–32.751%23%
2GNUbg weak1816024855-27023.4 20.7–26.450%21%
3Fastino GLiDE1812153757-41037.1 33.7–40.846%27%
4Decider v1.2 27B1812063660-30037.0 33.9–40.247%28%
5Jev 1.14.01812063656-38035.2 31.5–39.446%26%
6Jev 1.13.01812063653-37034.8 31.3–38.551%25%
7Cygnet (Gemma 4 12B)1811163456-44039.1 35.8–42.342%29%
8Clef1810263246-44037.3 33.5–41.145%27%
9Celeris-1 Decision189363058-48032.5 29.2–36.047%24%
10Fastino GLiDE v1.2 (local)1810083055-41035.6 32.0–39.545%26%
11SubQ Snap Preview189182858-44046.6 41.7–52.041%33%
12GPT-6 Luna189092749-51041.9 38.2–45.443%28%
13Decider v1.1 27B189092748-50037.7 34.3–41.146%27%
14Mercury Decide1861111939-55047.1 42.2–52.142%30%
15Winnow-12B1840141242-56048.4 44.9–52.039%32%
16Safest1833121240-70051.1 47.1–55.238%35%
17Open-Jev 27B v1.1182214831-71064.7 59.1–71.036%38%
18Decider Flash v1.2182016628-64058.2 53.2–62.936%38%
19Random18001806-580118.5 110.0–126.916%60%
SetupDecisionsAnswer read asInvalidTime per
decision
$ per 1,000
decisions
Runs on
Liquid d1-ultra2,143option probabilities0.0%0.41 s$0.14Liquid AI decisions API
GNUbg weak1,893–0.0%0.01 s–GNU Backgammon 1.08, Beginner level (0-ply, noise 0.060), x86-64 build (see About the anchor)
Fastino GLiDE2,109option probabilities0.0%0.50 s–Fastino API (responses name the model glide_v2)
Decider v1.2 27B1,849option probabilities0.0%1.24 s$0.09
list-price estimate; runs on our GPUs
our GPU, bf16
Jev 1.14.01,980option probabilities0.0%0.24 s$0.17TypeSafe API
Jev 1.13.01,823option probabilities0.0%0.23 s$0.17TypeSafe API
Cygnet (Gemma 4 12B)1,996option probabilities0.0%1.50 s–our GPUs, bf16; DGX Spark from 16 Oct
Clef1,933option probabilities0.0%4.61 s–our GPUs, bf16; DGX Spark from 16 Oct
Celeris-1 Decision2,157option probabilities0.0%0.19 s$0.15Celeris API (probabilities rounded to 2 decimals)
Fastino GLiDE v1.2 (local)1,931option probabilities0.0%2.03 s–our GPU, bf16, Fastino's released weights and server
SubQ Snap Preview2,129option probabilities0.0%0.25 s$0.19SubQ API
GPT-6 Luna1,957option probabilities0.0%0.21 s$0.34OpenAI Decisions API
Decider v1.1 27B2,011option probabilities0.0%4.26 s$0.09
list-price estimate; runs on our GPUs
our GPUs, bf16; DGX Spark from 16 Oct
Mercury Decide1,822option probabilities0.0%0.52 s$0.07OpenRouter (Inception)
Winnow-12B2,008option probabilities0.0%3.05 s–our GPU, Q8_0 GGUF, author's server
Safest2,119–0.0%0.00 s–program: fewest hitting rolls
Open-Jev 27B v1.12,083option probabilities0.0%2.02 s–our GPU, bf16
Decider Flash v1.21,850option probabilities0.0%0.71 s$0.04
list-price estimate; runs on our GPUs
our GPU, bf16
Random1,317–0.0%0.00 s–program: random legal play

League 2: workhorse chat models, closed and open

136 of 136 matches finished, 726 games

#SetupPWDLPtsGamesTo playError rate (95%)Engine's best playBlunders
1GPT-6.1 Sol (low)1615014561-27010.8 9.3–12.367%9%
2Gemma 4 31B (on)1612043658-34019.2 17.0–21.559%15%
3GPT-6.1 Sol (high)1612043652-30011.0 9.7–12.565%9%
4Gemini 3.8 Flash (low)1611143450-34016.6 14.7–18.659%14%
5Claude Sonnet 5.5 (high)1611053360-36015.2 13.1–17.564%12%
6Gemini 3.8 Flash (high)1611053355-39014.6 12.9–16.463%11%
7DeepSeek V4.1 Flash (on)1611053353-37018.6 16.4–20.962%14%
8Claude Sonnet 5.5 (low)1610153155-33020.2 17.6–22.959%16%
9GNUbg weak168082437-37023.4 20.3–26.751%19%
10Qwen3.8-27B (on)168082436-46031.1 27.9–34.553%23%
11Gemma 4 31B (off)166191940-42022.3 19.4–25.355%18%
12GLM-5.3 (on)1660101833-51029.5 26.6–32.950%22%
13Qwen3.8 Flash (off)1650111536-54034.0 30.7–37.444%25%
14DeepSeek V4.1 Flash (off)1631121038-56033.4 30.7–36.448%24%
15Qwen3.8-27B (off)163013929-59040.7 36.4–45.443%27%
16LFM2.5-24B-A2B (on)162014623-59046.5 42.2–51.139%32%
17LFM2.5-24B-A2B (off)160016010-52063.0 57.0–69.731%42%
SetupDecisionsAnswer read asInvalidTime per
decision
$ per 1,000
decisions
Runs on
GPT-6.1 Sol (low)1,787one letter0.0%6.19 s$9.99ChatGPT subscription
Gemma 4 31B (on)1,917letter probabilities0.0%228.97 s–our GPU, fp8 (vLLM); DGX Spark 19 Oct 18:36-22:26 UTC and from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00)
GPT-6.1 Sol (high)1,760one letter0.0%15.51 s$13.67ChatGPT subscription
Gemini 3.8 Flash (low)1,699one letter0.0%4.41 s$4.05Google Antigravity subscription
Claude Sonnet 5.5 (high)1,979one letter0.0%8.62 s$20.05Claude subscription
Gemini 3.8 Flash (high)2,028one letter0.0%19.46 s$23.05Google Antigravity subscription
DeepSeek V4.1 Flash (on)1,944letter probabilities1.1%98.69 s$12.37Fireworks via OpenRouter (quantization not published) until 20 Oct 21:56 UTC, then our four GPUs (SGLang, published FP4/FP8 weights, 17 Oct revision)
Claude Sonnet 5.5 (low)1,788one letter0.0%3.56 s$13.51Claude subscription
GNUbg weak1,660–0.0%0.01 s–GNU Backgammon 1.08, Beginner level (0-ply, noise 0.060), x86-64 build (see About the anchor)
Qwen3.8-27B (on)1,704letter probabilities0.0%314.13 s–our GPU, fp16 (vLLM); first decisions on Cerebras via OpenRouter; DGX Spark from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00)
Gemma 4 31B (off)1,575letter probabilities0.0%3.55 s–our GPU, fp8 (vLLM); DGX Spark 19 Oct 18:36-22:26 UTC and from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00)
GLM-5.3 (on)1,671letter probabilities0.1%4.51 s$3.42Fireworks via OpenRouter, quantization not published
Qwen3.8 Flash (off)1,806letter probabilities0.0%3.68 s$0.28Alibaba via OpenRouter, quantization not published
DeepSeek V4.1 Flash (off)1,804letter probabilities0.0%2.18 s$0.65Fireworks via OpenRouter (quantization not published) until 20 Oct 21:56 UTC, then our four GPUs (SGLang, published FP4/FP8 weights, 17 Oct revision)
Qwen3.8-27B (off)1,649letter probabilities0.0%1.92 s–our GPU, fp16 (vLLM); first decisions on Cerebras via OpenRouter; DGX Spark from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00)
LFM2.5-24B-A2B (on)1,699letter probabilities0.0%405.65 s–our GPU, bf16 (vLLM); first decisions on Liquid AI's API (voided); DGX Spark from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00)
LFM2.5-24B-A2B (off)1,302letter probabilities0.0%0.27 s–our GPU, bf16 (vLLM); DGX Spark from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00)

League 3: frontier chat models on subscriptions

55 of 55 matches finished, 338 games

Each model plays at two settings, all through the maker's subscription (ChatGPT, Claude, Google Antigravity, SuperGrok), with the same prompt and one answer letter. "On" is the model's default reasoning setting; "off" is its lowest, since none of these models accepts reasoning turned off entirely:

#SetupPWDLPtsGamesTo playError rate (95%)Engine's best playBlunders
1GPT-6 Astra (on)108022433-23012.5 10.5–14.564%10%
2Claude Fable 5.1 (on)107032133-23012.8 10.7–15.165%11%
3Claude Opus 5.5 (on)106041837-27014.3 11.8–17.165%11%
4Gemini 4 Argon (on)105141630-32012.6 10.4–14.965%9%
5Grok 4.7 (on)105141630-32015.1 13.1–17.264%12%
6Gemini 4 Argon (off)105141629-37017.4 14.4–20.660%14%
7GPT-6 Astra (off)104241433-27011.5 9.4–14.066%9%
8Claude Opus 5.5 (off)104061230-30016.2 13.7–19.161%14%
9Grok 4.7 (off)10307926-34019.0 16.5–21.861%15%
10GNUbg weak10226825-37028.0 24.7–31.444%23%
11Claude Fable 5.1 (off)10217732-36017.8 15.4–20.361%14%
SetupDecisionsAnswer read asInvalidTime per
decision
$ per 1,000
decisions
Runs on
GPT-6 Astra (on)1,256one letter0.0%6.71 s$50.83ChatGPT subscription
Claude Fable 5.1 (on)1,203one letter0.0%19.36 s$127.74Claude subscription
Claude Opus 5.5 (on)1,384one letter0.0%9.29 s$36.38Claude subscription
Gemini 4 Argon (on)1,294one letter0.0%43.88 s$41.37Google Antigravity subscription
Grok 4.7 (on)1,243one letter0.0%226.64 s$113.12xAI subscription (SuperGrok)
Gemini 4 Argon (off)1,406one letter0.0%6.13 s$10.22Google Antigravity subscription
GPT-6 Astra (off)1,310one letter0.0%6.25 s$50.44ChatGPT subscription
Claude Opus 5.5 (off)1,261one letter0.0%5.06 s$28.21Claude subscription
Grok 4.7 (off)1,247one letter0.0%44.57 s$31.55xAI subscription (SuperGrok)
GNUbg weak1,443–0.0%0.01 s–GNU Backgammon 1.08, Beginner level (0-ply, noise 0.060), x86-64 build (see About the anchor)
Claude Fable 5.1 (off)1,457one letter0.0%3.66 s$66.50Claude subscription

Error rate: 500 × mean equity lost per decision, with a 95% interval from resampling whole games. "Engine's best play": the chosen play lost less than 0.001; "Blunders": more than 0.08. Invalid answers are scored by the random play made. Only finished matches count in the table; "To play" is the matches left, and a setup's numbers so far can come from a different mix of opponents than another's. Time per decision is the request time without retries, with several requests in flight, so on our own hardware it depends on the load. Cost: the provider's list price × the tokens reported; subscriptions (Claude, ChatGPT, Google Antigravity, SuperGrok) as API-list-price equivalent; the Perplexity Deciders (Decider v1.1 27B, Decider v1.2 27B, Decider Flash v1.2) run on our GPUs, so their figure is a list-price estimate at Perplexity's price, not money paid. "–": no price.

Changes during the leagues

About the anchor

The anchor, GNU Backgammon 1.08 at its Beginner level (0-ply, deterministic noise 0.06), ran on one x86-64 build only, and on x86-64 it is not the Beginner that GNU Backgammon intends. Its deterministic-noise code sums the hash bytes as char, which is signed on x86-64, but subtracts the mean of unsigned bytes, so every noise draw is shifted by about −7 standard deviations. The anchor understates its own winning chances and plays for gammons.

So v3's anchor is "GNUbg 1.08 Beginner, x86-64 build", and its results are reproducible on x86-64 only: a replay there gives the recorded plays. Later releases of the benchmark use a corrected build.

Shared dice tested and rejected

Each match has its own dice. We tested whether giving round r of every match the same roll schedule would make the table less a matter of luck: 8 GNU Backgammon players of known strength, 500 replays of a round robin under each scheme. Shared schedules made league points more scattered (standard deviation ×1.10, 95% interval 1.06 to 1.14) and lowered the rank correlation with the true order from 0.890 to 0.864. Mirrored games already cancel most of the dice luck within a match; sharing schedules across matches cancels nothing more and puts all of a player's matches on the same dice, so their luck adds up. The test players ran on the same x86-64 build as the anchor, so their noise levels are shifted too; that applies to both schemes and does not affect the comparison.

The setup

Each model plays every other model once per league (a single round robin). A match is played in rounds of two games with the same dice: in the second game the players swap seats and the same sequence of rolls is repeated, so both face the same luck. Games score 1 point, 2 for a gammon and 3 for a backgammon; there is no doubling cube.

With seven models, each league has 21 matches and one model sits out each matchday. A match starts as soon as both of its models are free, and no model plays two matches at once.

Models

All seven answer the same kind of request: a text state and one choice question whose options are the legal plays. Two are hosted services; the other five run on our own GPUs.

Jev 1.13.0 · TypeSafe, hosted
TypeSafe's decision model, served through its System One API; the request format the others copy.
GPT-6 Luna · OpenAI Decisions API, hosted
OpenAI's decision endpoint with gpt-6-luna. The only one that sometimes declines to choose; that happened 3 times in early leagues, on positions with 97 to 439 legal plays, and never in the current format.
Decider v1.1 27B · Perplexity, self-hosted
The released pplx-decider-v1.1-27b checkpoint, run with its own server code. It accepts up to 8,192 tokens, so long play lists are sent in groups of 25 (see below). It replaced Perplexity's hosted Decider 27B from league #6 on.
Clef · Cloudflare, self-hosted
A 27B decision model that scores all options in one pass. Groups of up to 110 options.
Open-Jev 27B v1.1 · open reimplementation of Jev, self-hosted
A LoRA adapter and decision head on a 27B base model. It scores every option as its own sequence that repeats the whole question, with a 4,096-token limit per sequence, which is what limits the prompt's length for every model.
Cygnet · blockbrain-ai recipe, self-hosted
Google's gemma-4-12B-it, unchanged, behind a server that labels the options A, B, C… and reads the model's probability for each letter. Groups of up to 26 options.
Gemma 4 31B · our configuration of the Cygnet recipe
The same letter-reading server in front of gemma-4-31B-it, served in FP8. Not a trained decision model: a general chat model whose answer-letter probabilities are read directly. Its calibration setting was fitted for Cygnet's 12B model; that changes its reported confidence, not which play it picks.

When a position has more legal plays than a model's group size, the plays are split into equal groups, the model picks a winner in each, then chooses among the winners. Liquid's d1 played in the early leagues and was dropped after its free tier rate-limited a league to a stop.

The prompt

Every decision sends the same parts in the same order:

  1. Rules and strategy (about 6,400 characters): the rules of backgammon, strategy principles (blots and points, distribution, exposure, blocking, hitting, anchors), the best opening plays from computer rollouts, and the match rules above. A longer version with a glossary, game plans and a worked example goes to chat models; decision models get this short one because of Open-Jev's 4,096-token limit.
  2. Score and time: match score, round and game, turn number, both clocks.
  3. Position from the mover's side: checkers on every point, bar, borne off, pip counts.
  4. Race: the pip lead and whether the two sides are still in contact.
  5. The last 10 moves of this game, both sides, in the mover's numbering (league #13 on).
  6. Dice, then the question: Which of the following moves makes a point in front of the opponent's checkers, or reduces your single pieces, or gets you closer to your home?
  7. The legal plays, each followed by what the position looks like after it, computed by the program.

The shape of a request (the rules part is left out; angle brackets stand for the values of the position at hand, and no real position or answer is shown, so that the positions stay usable as a test):

Score: you <n>, opponent <n>.
Match: round <r> of <R>, game <g> of <G> in this round.
Turn <t> of at most <T> in this game.
Clock: <free delay> for this decision, then your bank: <time> left in this game (opponent <time>).

Position (your numbering; you move 24 -> 1):
  your checkers:     <point:count …>
  opponent checkers: <point:count …>
  on the bar: you <n>, opponent <n>
  borne off:  you <n>, opponent <n>
  pip count:  you <n>, opponent <n>
Race: <the pip lead>; <whether the checkers are still in contact>.
Last 10 moves of this game (oldest first, your numbering):
  turn <t>, <you / opponent> rolled <dice>: <play>
  …
Dice: <d1>-<d2>

Which of the following moves makes a point in front of the opponent's checkers, or reduces your single pieces, or gets you closer to your home?

Options (<n>):
<play>: hits <n>; makes <points made>; leaves <n> blots, the opponent hits with <n> of 36 rolls; you hold <n> points (<n> in your home board), longest prime <n>; pips: you <n>, opponent <n>
…  (one line per legal play)

Answer: <one of the plays> (with the model's probability where it gives one)

The numbers after each play are exact. The opponent's hitting rolls are counted by generating its legal plays for each of the 36 rolls and checking whether any of them hits.

Why it looks like this

The prompt was changed one step at a time, and a league was run after each step. Leagues stopped part-way are kept, with their finished matches only.

LeagueWhat changedGamesHit the 300-turn limitMoves that hitAverage gameBest opening playError rate
#1Structured state (JSON) with a short rules text and how every earlier game ended3882%62%277 turns28%138
#3Rules and strategy text, then score, time, position and dice; nothing about earlier games17081%60%274 turns77%132
#6+ the question before the choices; Decider v1.1 self-hosted12471%58%255 turns83%120
#7+ what each play leads to, and the race line1350%9.7%50 turns68%38.5
#8As #7 without d1 (7 models)1200%9.5%51 turns69%39.4
#11Repeat, new dice1120%9.3%50 turns58%40.1
#12Repeat, new dice1120%9.7%51 turns69%36.2
#13+ the last 10 moves of the game1060%10.9%52 turns72%39.1

Grey rows did not finish. "Best opening play" is the share of first moves of a game that match the computer-rollout table in the prompt. "Error rate" is GNU Backgammon's, over all model decisions of the league (see Error rate); lower is better. Leagues #2, #4, #5, #9 and #10 failed at start or were stopped within minutes; league #3 is left out of the standings below because its format differs.

Rules and strategy as text, not a data structure

The first leagues sent the position as structured data with a short rules summary. Models opened badly: 28% of first moves matched the known best play. With the rules, strategy principles and the opening table as prose, that rose to 77%. Opening play improved, but the middle game did not: games still ran to the turn limit, because both sides kept hitting and re-entering.

A question that names what a good move does

The question before the choices states three goals in plain words: build a point in front of the opponent's checkers, reduce your blots, bring checkers home. It deliberately leaves hitting out, because hitting on almost every move was the problem. The turn-limit share fell from 81% to 71% (this league also switched to the self-hosted Decider v1.1); hitting barely changed.

Compute the consequences in code, not in the model

Backgammon decisions turn on numbers a language model cannot work out reliably from a list of notations: which checkers are left alone, how many of the opponent's 36 rolls hit them, which points are made. Earlier, each option was just its notation, like bar/19 13/11*, and the model had to imagine the resulting position. Now the program computes those numbers for every option and the model compares them. This is the change that ended the endless games: hits fell from about 60% to under 10% of moves and the average game from about 270 turns to about 50.

Opening play got worse at the same time, from about 80% to between 58% and 69% best plays. The metrics after each option compete with the opening table earlier in the prompt, and on the first move several plays look similar on these numbers.

Recent moves, for longer plans

Each decision is a separate request, so a model cannot see how the position came about. League #13 adds the last 10 moves of the game, both sides. Ten turns add about 200 tokens. More would eat into Open-Jev's 4,096-token limit, which it applies to the whole question repeated for every option.

It made no measurable difference. League #13's error rate was 39.1 (± 2.2), against 38.6 (± 1.2) for #8, #11 and #12 together, and the finishing order barely changed. Ten moves may be too few, or these models may not use them.

Error rate

Wins and losses depend heavily on the dice. A backgammon engine can instead judge every decision directly: for each move it finds the best play, and measures how much winning chance, in equity (expected points per game), the chosen play gave away. Averaged over all decisions and multiplied by 500, this is the error rate (performance rating, PR) that backgammon players use, here for checker plays only; forced moves are not counted. eXtreme Gammon's human scale: under 2.5 world champion, 2.5 to 5 world class, 5 to 7.5 expert, 7.5 to 12.5 advanced, 12.5 to 17.5 intermediate, 17.5 to 22.5 casual, 22.5 to 30 beginner, 30 or more distracted. Human ratings also count doubling cube decisions, which these games do not have, so the comparison is approximate.

We scored all 104,045 decisions the models made in every league with GNU Backgammon 1.08. Every legal play was evaluated at 0-ply, then the five best and the chosen play again at 2-ply; the loss is the 2-ply equity of the best play minus that of the chosen one. As a check, on the opening roll the plays that the rollout table calls best lose 0.0002 equity on average, other plays 0.063.

By model, leagues #8, #11 and #12

ModelDecisionsError rateEngine's best playBlunders#8#11#12#13
Gemma 4 31B1,89425.4 ± 2.454%19.5%22.726.327.228.8
Decider v1.1 27B1,94232.7 ± 2.949%24.5%33.033.131.934.2
Jev 1.13.02,06635.7 ± 2.946%26.7%34.236.236.732.0
Clef2,10035.8 ± 3.047%25.6%35.740.530.642.8
GPT-6 Luna2,04439.4 ± 3.546%25.6%40.135.941.737.5
Cygnet2,07843.2 ± 3.442%30.2%46.945.636.740.6
Open-Jev 27B1,75859.6 ± 4.636%36.5%65.863.149.558.1

Error rate: mean equity lost per decision × 500, with a 95% interval that treats decisions as independent (moves within a game are not, so the true interval is somewhat wider). "Engine's best play": share of decisions where the chosen play lost less than 0.001. "Blunders": decisions losing more than 0.08 equity. The last four columns are the error rate in each league; #13 adds the move history.

Results

The standings below use the three finished leagues with computed move consequences and no move history: #8, #11 and #12 (7 models, same prompt, different dice). League #13, with move history, finished 1. Gemma 4 31B, 2. Jev 1.13.0, 3. Decider v1.1 27B, 4. Cygnet, 5. GPT-6 Luna, 6. Clef, 7. Open-Jev 27B.

Final places

Model1st2nd3rd4th5th6th7th
Gemma 4 31B2100000
Jev 1.13.01001100
Decider v1.1 27B0210000
Clef0011010
Cygnet0011010
GPT-6 Luna0000300
Open-Jev 27B0000003

In league #12, Jev 1.13.0 and GPT-6 Luna finished level on every tiebreak and share 5th place.

Across the three leagues

ModelMatches W-D-LGames won-lostPoints for-againstGammons and backgammons wonPicked the safest playShots left on averageTime per decision
Gemma 4 31B14-1-367-2791-362384%5.10.30 s
Decider v1.1 27B12-0-656-4079-562077%6.11.14 s
Jev 1.13.011-0-753-4971-611682%5.10.26 s
Clef9-1-853-4581-642168%7.01.13 s
Cygnet8-0-1052-5465-721182%4.60.22 s
GPT-6 Luna8-0-1048-5462-681187%3.80.20 s
Open-Jev 27B0-0-1815-7518-110265%8.51.69 s

"Safest play": the option whose position the opponent hits with the fewest of its 36 rolls, read from the numbers the model was shown. "Shots left": opponent rolls out of 36 that hit after the chosen play. Both are measured on decisions sent as a single request (about 1,700 to 2,100 per model); the safest available play left 2.7 to 4.2 shots on average, a typical option about 13.

What the numbers say

Can a team beat the best model?

Every model reports a probability for each option. If those probabilities say when a model is right, a team could ask all seven and play the answer of whichever is most sure. To test this, we sent the same 1,000 recorded positions (2 to 25 legal plays each, drawn from leagues #8, #11, #12 and #13) to all seven models with exactly the request the original player received, and scored every legal play with GNU Backgammon at 2-ply: 7,000 answers and 9,439 evaluated plays. Any team and any rule can then be scored offline on the same positions.

Confidence says how hard a position is, not who is right

Within each model, confidence does carry information: across its league decisions, a model is right more often when it is more confident (AUROC 0.66 to 0.74, where 0.5 is chance). But the models use very different scales:

ModelEngine's best playAverage stated probabilityWhen it is most sureit is right
Gemma 4 31B57%0.85 overconfident0.8 – 1.065%
Decider v1.1 27B51%0.580.8 – 1.084%
Jev 1.13.047%0.550.8 – 1.081%
Clef46%0.28 underconfident0.6 – 0.889%
GPT-6 Luna46%0.560.8 – 1.065%
Cygnet44%0.67 overconfident0.8 – 1.063%
Open-Jev 27B36%0.21 underconfident0.6 – 0.892%

From 17,579 league decisions sent as a single request; "right" means the engine's best play (loss below 0.001 equity). The stated probability is the one the model gave its chosen play; the models' own "confidence" fields are defined differently from one another, so they are not used.

Clef and Open-Jev look like the most reliable models when they are sure, but they are only sure on easy positions, where everyone else is right as well. The test that matters is what happens when models disagree on the same position:

When Gemma 4 31B and this model disagreeGemma has the better playThe more confident model has the better play
Decider v1.1 27B62%62%
Clef64%64%
Jev 1.13.069%67%
GPT-6 Luna73%70%
Cygnet77%75%
Open-Jev 27B76%76%

Following the more confident model is never better than simply following Gemma 4 31B.

Teams

Team and ruleError rateEngine's best playDifference from Gemma alone (95%)
Gemma 4 31B alone23.554.4%–
All 7: highest stated probability24.053.8%+0.4 (−0.2 to +1.2)
All 7: lowest expected loss, calibrated26.052.9%+2.5 (+0.7 to +4.3)
All 7: most likely correct, calibrated27.751.9%+4.2 (+2.2 to +6.4)
All 7: probabilities added up27.752.0%+4.2 (+2.1 to +6.6)
Best of 120 formations: Clef + Decider v1.1 + Gemma, probabilities added up23.454.8%−0.2 (−0.7 to +0.2)
Oracle: the best answer any of the 7 gave12.770.1%−10.8

Error rate on the 1,000 shared positions (lower is better). Calibrated rules map each model's stated probability to its observed accuracy or loss with isotonic regression, fitted on one half of the positions and scored on the other (two-fold cross-fitting). Intervals are bootstrap intervals over positions, paired with Gemma 4 31B on the same positions. The best formation was chosen on the same positions it is scored on, which flatters it.

Speed and cost

Measured on the same 1,000 recorded positions as the team test, with exactly the request each original player received. Besides the seven league players, the table includes general language models we tried the same way: DeepSeek V4.1 Flash, Qwen3.8 Flash Next and GLM-4.7-Flash are read through their probabilities for each answer letter, like Gemma 4 31B. GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon and Gemini 3.8 Flash return no such probabilities: each answers one letter, constrained to the option letters, at low reasoning effort (Gemini's lowest available level). Gemini 4 Argon reported no reasoning tokens at that level, so it answered directly. DeepSeek V4.1 Flash also ran a second time with reasoning on (lowest effort), answering one constrained letter after reasoning. GPT-6 Astra and Claude Opus 5.5 ran through ChatGPT and Claude subscriptions.

ModelRuns onError rateTime per decision
median / 90th pct
In flightEstimated API cost
per 1,000 decisions
GPT-6 AstraChatGPT subscription11.24.60 s / 10.95 s4$36.45 API-equivalent
Gemini 4 ArgonGoogle Gemini API16.94.09 s / 28.22 s4$6.97
Claude Opus 5.5Claude subscription17.33.43 s / 6.66 s4$21.65 API-equivalent
DeepSeek V4.1 Flash, reasoning onour GPUs: 418.1106 s / 288 s8own hardware
Gemini 3.8 FlashGoogle Gemini API18.32.87 s / 7.96 s4$3.25
Gemma 4 31Bour GPUs: 123.50.59 s / 0.87 s4own hardware
DeepSeek V4.1 Flashour GPUs: 427.50.57 s / 0.98 s4own hardware
Qwen3.8 Flash Nextour GPUs: 431.10.46 s / 0.71 s4own hardware
Decider v1.1 27Bour GPUs: shares 231.80.82 s / 1.35 s1own hardware
Jev 1.13.0TypeSafe API33.60.25 s / 0.29 s4$0.16
Clefour GPUs: 134.10.94 s / 1.28 s1own hardware
GPT-6 LunaOpenAI Decisions API35.00.22 s / 0.27 s4$0.32
Cygnetour GPUs: 138.90.78 s / 1.12 s4own hardware
GLM-4.7-Flashour GPUs: 150.60.20 s / 0.27 s4own hardware
Open-Jev 27Bour GPUs: 159.91.38 s / 1.91 s1own hardware

Error rate on the 1,000 shared positions (lower is better). Time is wall-clock per request, from sending it to receiving the answer, with the stated number of requests in flight. Our GPUs are NVIDIA RTX PRO 6000 Blackwell cards with 96 GB each, in one machine. The seven league players were asked together, so the self-hosted ones shared the machine; the other models were each asked alone. A run of GLM-5.3-Flash is left out: it played close to random, most likely because of how we had to run it on this machine.

The plays are listed in the order the runner generated them, which tends to put the best play early. Asked again with the same plays in a random order, Gemini 4 Argon's error rate rose from 16.9 to 18.9 (difference 1.9, 95% interval −0.3 to 4.1): probably a small help from the order, not yet a proven one. The next version of the benchmark shuffles the plays for every model.

Cost basis: the provider's list price × the tokens its API reported; the bill itself was not checked. Jev 1.13.0 ($0.042 per 1M input tokens, output free) and GPT-6 Luna on OpenAI's Decisions API ($0.10 per 1M input tokens, output free): prices of 8 October 2026, tokens averaged over their league decisions with 2 to 25 legal plays (3,740 and 3,238 input tokens per decision). Gemini 3.8 Flash ($0.75 per 1M input tokens, $3.75 per 1M output tokens, reasoning billed as output; introductory price of 8 October 2026, which doubles on 1 January 2027): tokens measured on 100 of the positions, 3,373 input and 192 output per decision on average. Gemini 4 Argon ($2 per 1M input, $10 per 1M output; price until 30 November 2026, doubling on 1 December): tokens reported for all 1,000 positions, 3,462 input and 5 output per decision. GPT-6 Astra ($10 per 1M input, $50 per 1M output) and Claude Opus 5.5 ($4 input, $20 output): API list prices of 8 October 2026 × the tokens reported for all 1,000 positions (3,213 input and 86 output, and 4,424 input and 198 output per decision). Both ran on subscriptions, which are not billed per token, so these are API-equivalent estimates. Self-hosted models have no per-request price; a 7-model league of about 690 decisions per player took about 30 minutes on the four GPUs. DeepSeek with reasoning used 6,657 output tokens per decision on average.

Limits