v3 leagues
Final: data as of 23 October 2026, 02:50 UTC. League 1 has finished; League 2 has finished; League 3 has finished.
v3 keeps the v1 match rules (rounds of two games with the same dice, 3 rounds, 300-turn limit, no cube) and changes how the models are asked:
- Shuffled options. The legal plays appear in a fixed random order per position, drawn from a published seed. v1 listed them in move-generation order, which tends to put the best play early.
- One prompt for every player (bg-v2): the league #8 format, with rules and strategy text, the position, dice and every legal play with its computed consequences, each under a letter. No move history.
- How the answer is read. Decision models give a probability for each option; open chat models give probabilities for each answer letter, read after their reasoning when it is on; models without probabilities answer one letter. An answer that names no option is not retried: a random legal play is made and counted as invalid.
- No clock. Some setups needed more than 30 seconds per decision in the smoke test, so the leagues play without one.
- Three leagues. League 1 has decision models only, plus two programs (Random, Safest: the play that the fewest opponent rolls hit). League 2 has workhorse chat models, closed and open, each in two settings (reasoning low and high, or off and on). League 3 has frontier chat models played through their subscriptions, each with reasoning off and on (defined below). GNU Backgammon at its weak Beginner level plays in all three as a fixed anchor.
Every model decision in the matches that count was scored with GNU Backgammon, as in v1 (see Error rate for the method and the human scale): all 81,386 decisions, none skipped.
League 1: decision models
171 of 171 matches finished, 904 games
| # | Setup | P | W | D | L | Pts | Games | To play | Error rate (95%) | Engine's best play | Blunders |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Liquid d1-ultra | 18 | 16 | 0 | 2 | 48 | 67-35 | 0 | 30.0 27.2–32.7 | 51% | 23% |
| 2 | GNUbg weak | 18 | 16 | 0 | 2 | 48 | 55-27 | 0 | 23.4 20.7–26.4 | 50% | 21% |
| 3 | Fastino GLiDE | 18 | 12 | 1 | 5 | 37 | 57-41 | 0 | 37.1 33.7–40.8 | 46% | 27% |
| 4 | Decider v1.2 27B | 18 | 12 | 0 | 6 | 36 | 60-30 | 0 | 37.0 33.9–40.2 | 47% | 28% |
| 5 | Jev 1.14.0 | 18 | 12 | 0 | 6 | 36 | 56-38 | 0 | 35.2 31.5–39.4 | 46% | 26% |
| 6 | Jev 1.13.0 | 18 | 12 | 0 | 6 | 36 | 53-37 | 0 | 34.8 31.3–38.5 | 51% | 25% |
| 7 | Cygnet (Gemma 4 12B) | 18 | 11 | 1 | 6 | 34 | 56-44 | 0 | 39.1 35.8–42.3 | 42% | 29% |
| 8 | Clef | 18 | 10 | 2 | 6 | 32 | 46-44 | 0 | 37.3 33.5–41.1 | 45% | 27% |
| 9 | Celeris-1 Decision | 18 | 9 | 3 | 6 | 30 | 58-48 | 0 | 32.5 29.2–36.0 | 47% | 24% |
| 10 | Fastino GLiDE v1.2 (local) | 18 | 10 | 0 | 8 | 30 | 55-41 | 0 | 35.6 32.0–39.5 | 45% | 26% |
| 11 | SubQ Snap Preview | 18 | 9 | 1 | 8 | 28 | 58-44 | 0 | 46.6 41.7–52.0 | 41% | 33% |
| 12 | GPT-6 Luna | 18 | 9 | 0 | 9 | 27 | 49-51 | 0 | 41.9 38.2–45.4 | 43% | 28% |
| 13 | Decider v1.1 27B | 18 | 9 | 0 | 9 | 27 | 48-50 | 0 | 37.7 34.3–41.1 | 46% | 27% |
| 14 | Mercury Decide | 18 | 6 | 1 | 11 | 19 | 39-55 | 0 | 47.1 42.2–52.1 | 42% | 30% |
| 15 | Winnow-12B | 18 | 4 | 0 | 14 | 12 | 42-56 | 0 | 48.4 44.9–52.0 | 39% | 32% |
| 16 | Safest | 18 | 3 | 3 | 12 | 12 | 40-70 | 0 | 51.1 47.1–55.2 | 38% | 35% |
| 17 | Open-Jev 27B v1.1 | 18 | 2 | 2 | 14 | 8 | 31-71 | 0 | 64.7 59.1–71.0 | 36% | 38% |
| 18 | Decider Flash v1.2 | 18 | 2 | 0 | 16 | 6 | 28-64 | 0 | 58.2 53.2–62.9 | 36% | 38% |
| 19 | Random | 18 | 0 | 0 | 18 | 0 | 6-58 | 0 | 118.5 110.0–126.9 | 16% | 60% |
| Setup | Decisions | Answer read as | Invalid | Time per decision | $ per 1,000 decisions | Runs on |
|---|---|---|---|---|---|---|
| Liquid d1-ultra | 2,143 | option probabilities | 0.0% | 0.41 s | $0.14 | Liquid AI decisions API |
| GNUbg weak | 1,893 | – | 0.0% | 0.01 s | – | GNU Backgammon 1.08, Beginner level (0-ply, noise 0.060), x86-64 build (see About the anchor) |
| Fastino GLiDE | 2,109 | option probabilities | 0.0% | 0.50 s | – | Fastino API (responses name the model glide_v2) |
| Decider v1.2 27B | 1,849 | option probabilities | 0.0% | 1.24 s | $0.09 list-price estimate; runs on our GPUs | our GPU, bf16 |
| Jev 1.14.0 | 1,980 | option probabilities | 0.0% | 0.24 s | $0.17 | TypeSafe API |
| Jev 1.13.0 | 1,823 | option probabilities | 0.0% | 0.23 s | $0.17 | TypeSafe API |
| Cygnet (Gemma 4 12B) | 1,996 | option probabilities | 0.0% | 1.50 s | – | our GPUs, bf16; DGX Spark from 16 Oct |
| Clef | 1,933 | option probabilities | 0.0% | 4.61 s | – | our GPUs, bf16; DGX Spark from 16 Oct |
| Celeris-1 Decision | 2,157 | option probabilities | 0.0% | 0.19 s | $0.15 | Celeris API (probabilities rounded to 2 decimals) |
| Fastino GLiDE v1.2 (local) | 1,931 | option probabilities | 0.0% | 2.03 s | – | our GPU, bf16, Fastino's released weights and server |
| SubQ Snap Preview | 2,129 | option probabilities | 0.0% | 0.25 s | $0.19 | SubQ API |
| GPT-6 Luna | 1,957 | option probabilities | 0.0% | 0.21 s | $0.34 | OpenAI Decisions API |
| Decider v1.1 27B | 2,011 | option probabilities | 0.0% | 4.26 s | $0.09 list-price estimate; runs on our GPUs | our GPUs, bf16; DGX Spark from 16 Oct |
| Mercury Decide | 1,822 | option probabilities | 0.0% | 0.52 s | $0.07 | OpenRouter (Inception) |
| Winnow-12B | 2,008 | option probabilities | 0.0% | 3.05 s | – | our GPU, Q8_0 GGUF, author's server |
| Safest | 2,119 | – | 0.0% | 0.00 s | – | program: fewest hitting rolls |
| Open-Jev 27B v1.1 | 2,083 | option probabilities | 0.0% | 2.02 s | – | our GPU, bf16 |
| Decider Flash v1.2 | 1,850 | option probabilities | 0.0% | 0.71 s | $0.04 list-price estimate; runs on our GPUs | our GPU, bf16 |
| Random | 1,317 | – | 0.0% | 0.00 s | – | program: random legal play |
- Final ranking: 1. Liquid d1-ultra (48 points), 2. GNUbg weak (48 points), 3. Fastino GLiDE (37 points), 4. Decider v1.2 27B (36 points), 5. Jev 1.14.0 (36 points); Random last, with 0 matches won.
- Liquid d1-ultra has the lowest error rate of the decision models, 30.0. Celeris-1 Decision (32.5) is within its interval; every other decision model is at 34.8 or higher.
- The weak GNU Backgammon anchor plays better than every decision model by error rate (23.4); its match wins come with many gammons (see About the anchor).
- 2 decision models play worse than the Safest program by error rate: Decider Flash v1.2 (58.2) and Open-Jev 27B v1.1 (64.7), against 51.1. Mercury Decide (47.1) and Winnow-12B (48.4) within Safest's interval. Random scores 118.5.
- The 6 v1 decision models against their v1 error rates: 4 higher, 2 lower or equal: Jev 1.13.0 34.8 (v1: 35.7), Clef 37.3 (v1: 35.8), Decider v1.1 27B 37.7 (v1: 32.7), Cygnet (Gemma 4 12B) 39.1 (v1: 43.2), GPT-6 Luna 41.9 (v1: 39.4), Open-Jev 27B v1.1 64.7 (v1: 59.6). The games, opponents and option order differ, so this is not a paired comparison.
- New versions: Jev 1.14.0 35.2 (36 points) against Jev 1.13.0 34.8 (36 points): no clear difference in league play; Decider v1.2 27B 37.0 (36 points) against Decider v1.1 27B 37.7 (27 points): no clear difference in league play.
- The hosted decision services (8) take 0.2 s to 0.5 s per decision; the 7 with a price cost $0.07 to $0.34 per 1,000 decisions.
League 2: workhorse chat models, closed and open
136 of 136 matches finished, 726 games
| # | Setup | P | W | D | L | Pts | Games | To play | Error rate (95%) | Engine's best play | Blunders |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-6.1 Sol (low) | 16 | 15 | 0 | 1 | 45 | 61-27 | 0 | 10.8 9.3–12.3 | 67% | 9% |
| 2 | Gemma 4 31B (on) | 16 | 12 | 0 | 4 | 36 | 58-34 | 0 | 19.2 17.0–21.5 | 59% | 15% |
| 3 | GPT-6.1 Sol (high) | 16 | 12 | 0 | 4 | 36 | 52-30 | 0 | 11.0 9.7–12.5 | 65% | 9% |
| 4 | Gemini 3.8 Flash (low) | 16 | 11 | 1 | 4 | 34 | 50-34 | 0 | 16.6 14.7–18.6 | 59% | 14% |
| 5 | Claude Sonnet 5.5 (high) | 16 | 11 | 0 | 5 | 33 | 60-36 | 0 | 15.2 13.1–17.5 | 64% | 12% |
| 6 | Gemini 3.8 Flash (high) | 16 | 11 | 0 | 5 | 33 | 55-39 | 0 | 14.6 12.9–16.4 | 63% | 11% |
| 7 | DeepSeek V4.1 Flash (on) | 16 | 11 | 0 | 5 | 33 | 53-37 | 0 | 18.6 16.4–20.9 | 62% | 14% |
| 8 | Claude Sonnet 5.5 (low) | 16 | 10 | 1 | 5 | 31 | 55-33 | 0 | 20.2 17.6–22.9 | 59% | 16% |
| 9 | GNUbg weak | 16 | 8 | 0 | 8 | 24 | 37-37 | 0 | 23.4 20.3–26.7 | 51% | 19% |
| 10 | Qwen3.8-27B (on) | 16 | 8 | 0 | 8 | 24 | 36-46 | 0 | 31.1 27.9–34.5 | 53% | 23% |
| 11 | Gemma 4 31B (off) | 16 | 6 | 1 | 9 | 19 | 40-42 | 0 | 22.3 19.4–25.3 | 55% | 18% |
| 12 | GLM-5.3 (on) | 16 | 6 | 0 | 10 | 18 | 33-51 | 0 | 29.5 26.6–32.9 | 50% | 22% |
| 13 | Qwen3.8 Flash (off) | 16 | 5 | 0 | 11 | 15 | 36-54 | 0 | 34.0 30.7–37.4 | 44% | 25% |
| 14 | DeepSeek V4.1 Flash (off) | 16 | 3 | 1 | 12 | 10 | 38-56 | 0 | 33.4 30.7–36.4 | 48% | 24% |
| 15 | Qwen3.8-27B (off) | 16 | 3 | 0 | 13 | 9 | 29-59 | 0 | 40.7 36.4–45.4 | 43% | 27% |
| 16 | LFM2.5-24B-A2B (on) | 16 | 2 | 0 | 14 | 6 | 23-59 | 0 | 46.5 42.2–51.1 | 39% | 32% |
| 17 | LFM2.5-24B-A2B (off) | 16 | 0 | 0 | 16 | 0 | 10-52 | 0 | 63.0 57.0–69.7 | 31% | 42% |
| Setup | Decisions | Answer read as | Invalid | Time per decision | $ per 1,000 decisions | Runs on |
|---|---|---|---|---|---|---|
| GPT-6.1 Sol (low) | 1,787 | one letter | 0.0% | 6.19 s | $9.99 | ChatGPT subscription |
| Gemma 4 31B (on) | 1,917 | letter probabilities | 0.0% | 228.97 s | – | our GPU, fp8 (vLLM); DGX Spark 19 Oct 18:36-22:26 UTC and from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00) |
| GPT-6.1 Sol (high) | 1,760 | one letter | 0.0% | 15.51 s | $13.67 | ChatGPT subscription |
| Gemini 3.8 Flash (low) | 1,699 | one letter | 0.0% | 4.41 s | $4.05 | Google Antigravity subscription |
| Claude Sonnet 5.5 (high) | 1,979 | one letter | 0.0% | 8.62 s | $20.05 | Claude subscription |
| Gemini 3.8 Flash (high) | 2,028 | one letter | 0.0% | 19.46 s | $23.05 | Google Antigravity subscription |
| DeepSeek V4.1 Flash (on) | 1,944 | letter probabilities | 1.1% | 98.69 s | $12.37 | Fireworks via OpenRouter (quantization not published) until 20 Oct 21:56 UTC, then our four GPUs (SGLang, published FP4/FP8 weights, 17 Oct revision) |
| Claude Sonnet 5.5 (low) | 1,788 | one letter | 0.0% | 3.56 s | $13.51 | Claude subscription |
| GNUbg weak | 1,660 | – | 0.0% | 0.01 s | – | GNU Backgammon 1.08, Beginner level (0-ply, noise 0.060), x86-64 build (see About the anchor) |
| Qwen3.8-27B (on) | 1,704 | letter probabilities | 0.0% | 314.13 s | – | our GPU, fp16 (vLLM); first decisions on Cerebras via OpenRouter; DGX Spark from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00) |
| Gemma 4 31B (off) | 1,575 | letter probabilities | 0.0% | 3.55 s | – | our GPU, fp8 (vLLM); DGX Spark 19 Oct 18:36-22:26 UTC and from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00) |
| GLM-5.3 (on) | 1,671 | letter probabilities | 0.1% | 4.51 s | $3.42 | Fireworks via OpenRouter, quantization not published |
| Qwen3.8 Flash (off) | 1,806 | letter probabilities | 0.0% | 3.68 s | $0.28 | Alibaba via OpenRouter, quantization not published |
| DeepSeek V4.1 Flash (off) | 1,804 | letter probabilities | 0.0% | 2.18 s | $0.65 | Fireworks via OpenRouter (quantization not published) until 20 Oct 21:56 UTC, then our four GPUs (SGLang, published FP4/FP8 weights, 17 Oct revision) |
| Qwen3.8-27B (off) | 1,649 | letter probabilities | 0.0% | 1.92 s | – | our GPU, fp16 (vLLM); first decisions on Cerebras via OpenRouter; DGX Spark from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00) |
| LFM2.5-24B-A2B (on) | 1,699 | letter probabilities | 0.0% | 405.65 s | – | our GPU, bf16 (vLLM); first decisions on Liquid AI's API (voided); DGX Spark from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00) |
| LFM2.5-24B-A2B (off) | 1,302 | letter probabilities | 0.0% | 0.27 s | – | our GPU, bf16 (vLLM); DGX Spark from 20 Oct 21:10 UTC (back on our GPU 21 Oct 01:25-05:00) |
- GPT-6.1 Sol (low) won with 15 wins in 16 matches. The lowest error rate is GPT-6.1 Sol (low)'s, 10.8.
- Reasoning, lower and higher setting (error rate; a difference counts only when the 95% intervals do not overlap): GPT-6.1 Sol 10.8 (low) and 11.0 (high): no clear difference in league play; 6.2 s and 15.5 s per decision; Gemma 4 31B 22.3 (off) and 19.2 (on): no clear difference in league play; 3.5 s and 229.0 s per decision; Gemini 3.8 Flash 16.6 (low) and 14.6 (high): no clear difference in league play; 4.4 s and 19.5 s per decision; Claude Sonnet 5.5 20.2 (low) and 15.2 (high): lower, and the intervals do not overlap; 3.6 s and 8.6 s per decision; DeepSeek V4.1 Flash 33.4 (off) and 18.6 (on): lower, and the intervals do not overlap; 2.2 s and 98.7 s per decision; Qwen3.8-27B 40.7 (off) and 31.1 (on): lower, and the intervals do not overlap; 1.9 s and 314.1 s per decision; LFM2.5-24B-A2B 63.0 (off) and 46.5 (on): lower, and the intervals do not overlap; 0.3 s and 405.6 s per decision. League games are not the same positions, so these are not paired comparisons.
- The best open setup is 7.8 points behind the best closed one: DeepSeek V4.1 Flash (on) at 18.6, against GPT-6.1 Sol (low) at 10.8.
- The weak GNU Backgammon anchor (23.4): 6 of 6 closed setups have a lower error rate; of the 10 open setups, DeepSeek V4.1 Flash (on) (18.6), Gemma 4 31B (on) (19.2) and Gemma 4 31B (off) (22.3) do.
- Invalid answers: 22 from DeepSeek V4.1 Flash (on) (1.1% of its decisions), 1 from GLM-5.3 (on) (0.1% of its decisions); none from any other setup.
League 3: frontier chat models on subscriptions
55 of 55 matches finished, 338 games
Each model plays at two settings, all through the maker's subscription (ChatGPT, Claude, Google Antigravity, SuperGrok), with the same prompt and one answer letter. "On" is the model's default reasoning setting; "off" is its lowest, since none of these models accepts reasoning turned off entirely:
- GPT-6 Astra: off = reasoning effort low (none and minimal are rejected); on = medium, the default.
- Claude Opus 5.5, Claude Fable 5.1: off = effort low (thinking cannot be disabled, only adaptive thinking is offered); on = effort high, the default.
- Gemini 4 Argon: off = thinking level low, which spends no reasoning tokens; on = high, the default.
- Grok 4.7: off = reasoning effort low (none is rejected), so it still reasons; on = high, the default.
| # | Setup | P | W | D | L | Pts | Games | To play | Error rate (95%) | Engine's best play | Blunders |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra (on) | 10 | 8 | 0 | 2 | 24 | 33-23 | 0 | 12.5 10.5–14.5 | 64% | 10% |
| 2 | Claude Fable 5.1 (on) | 10 | 7 | 0 | 3 | 21 | 33-23 | 0 | 12.8 10.7–15.1 | 65% | 11% |
| 3 | Claude Opus 5.5 (on) | 10 | 6 | 0 | 4 | 18 | 37-27 | 0 | 14.3 11.8–17.1 | 65% | 11% |
| 4 | Gemini 4 Argon (on) | 10 | 5 | 1 | 4 | 16 | 30-32 | 0 | 12.6 10.4–14.9 | 65% | 9% |
| 5 | Grok 4.7 (on) | 10 | 5 | 1 | 4 | 16 | 30-32 | 0 | 15.1 13.1–17.2 | 64% | 12% |
| 6 | Gemini 4 Argon (off) | 10 | 5 | 1 | 4 | 16 | 29-37 | 0 | 17.4 14.4–20.6 | 60% | 14% |
| 7 | GPT-6 Astra (off) | 10 | 4 | 2 | 4 | 14 | 33-27 | 0 | 11.5 9.4–14.0 | 66% | 9% |
| 8 | Claude Opus 5.5 (off) | 10 | 4 | 0 | 6 | 12 | 30-30 | 0 | 16.2 13.7–19.1 | 61% | 14% |
| 9 | Grok 4.7 (off) | 10 | 3 | 0 | 7 | 9 | 26-34 | 0 | 19.0 16.5–21.8 | 61% | 15% |
| 10 | GNUbg weak | 10 | 2 | 2 | 6 | 8 | 25-37 | 0 | 28.0 24.7–31.4 | 44% | 23% |
| 11 | Claude Fable 5.1 (off) | 10 | 2 | 1 | 7 | 7 | 32-36 | 0 | 17.8 15.4–20.3 | 61% | 14% |
| Setup | Decisions | Answer read as | Invalid | Time per decision | $ per 1,000 decisions | Runs on |
|---|---|---|---|---|---|---|
| GPT-6 Astra (on) | 1,256 | one letter | 0.0% | 6.71 s | $50.83 | ChatGPT subscription |
| Claude Fable 5.1 (on) | 1,203 | one letter | 0.0% | 19.36 s | $127.74 | Claude subscription |
| Claude Opus 5.5 (on) | 1,384 | one letter | 0.0% | 9.29 s | $36.38 | Claude subscription |
| Gemini 4 Argon (on) | 1,294 | one letter | 0.0% | 43.88 s | $41.37 | Google Antigravity subscription |
| Grok 4.7 (on) | 1,243 | one letter | 0.0% | 226.64 s | $113.12 | xAI subscription (SuperGrok) |
| Gemini 4 Argon (off) | 1,406 | one letter | 0.0% | 6.13 s | $10.22 | Google Antigravity subscription |
| GPT-6 Astra (off) | 1,310 | one letter | 0.0% | 6.25 s | $50.44 | ChatGPT subscription |
| Claude Opus 5.5 (off) | 1,261 | one letter | 0.0% | 5.06 s | $28.21 | Claude subscription |
| Grok 4.7 (off) | 1,247 | one letter | 0.0% | 44.57 s | $31.55 | xAI subscription (SuperGrok) |
| GNUbg weak | 1,443 | – | 0.0% | 0.01 s | – | GNU Backgammon 1.08, Beginner level (0-ply, noise 0.060), x86-64 build (see About the anchor) |
| Claude Fable 5.1 (off) | 1,457 | one letter | 0.0% | 3.66 s | $66.50 | Claude subscription |
- GPT-6 Astra (on) won with 8 wins in 10 matches. Lowest error rates: GPT-6 Astra (off) 11.5, GPT-6 Astra (on) 12.5, Gemini 4 Argon (on) 12.6, Claude Fable 5.1 (on) 12.8, with overlapping intervals.
- Reasoning, lower and higher setting (error rate; a difference counts only when the 95% intervals do not overlap): GPT-6 Astra 11.5 (off) and 12.5 (on): no clear difference in league play; 6.2 s and 6.7 s per decision; Claude Fable 5.1 17.8 (off) and 12.8 (on): lower, and the intervals do not overlap; 3.7 s and 19.4 s per decision; Claude Opus 5.5 16.2 (off) and 14.3 (on): no clear difference in league play; 5.1 s and 9.3 s per decision; Gemini 4 Argon 17.4 (off) and 12.6 (on): no clear difference in league play; 6.1 s and 43.9 s per decision; Grok 4.7 19.0 (off) and 15.1 (on): no clear difference in league play; 44.6 s and 226.6 s per decision. League games are not the same positions, so these are not paired comparisons.
- 10 of the 10 setups have a lower error rate than the weak GNU Backgammon anchor (28.0).
- No invalid answers.
- Time per decision, reasoning off: 3.7 s to 44.6 s; slowest Grok 4.7 (off).
- Time per decision, reasoning on: 6.7 s to 226.6 s; slowest Grok 4.7 (on).
- Cost at API list prices: $10 to $128 per 1,000 decisions.
Error rate: 500 × mean equity lost per decision, with a 95% interval from resampling whole games. "Engine's best play": the chosen play lost less than 0.001; "Blunders": more than 0.08. Invalid answers are scored by the random play made. Only finished matches count in the table; "To play" is the matches left, and a setup's numbers so far can come from a different mix of opponents than another's. Time per decision is the request time without retries, with several requests in flight, so on our own hardware it depends on the load. Cost: the provider's list price × the tokens reported; subscriptions (Claude, ChatGPT, Google Antigravity, SuperGrok) as API-list-price equivalent; the Perplexity Deciders (Decider v1.1 27B, Decider v1.2 27B, Decider Flash v1.2) run on our GPUs, so their figure is a list-price estimate at Perplexity's price, not money paid. "–": no price.
Changes during the leagues
- Voided matches. The first matches of LFM2.5-24B-A2B (off, on) and Qwen3.8 Flash (off) were voided on 19 October and replay from scratch with the same dice. The runner had read the letter probabilities from the wrong token: for LFM2.5, Liquid's API returns probabilities for the reasoning tokens too, so every decision read a letter from the reasoning; for Qwen3.8 Flash, some decisions read the "c" of a split "choice" key. It now reads them at the reply's letter and counts a mismatch as invalid. LFM2.5 now runs on our GPU from Liquid's bf16 checkpoint, with the letter read after a prefilled "Answer:".
- Qwen3.8-27B (off) replayed. On our vLLM server the list of token probabilities ends with an end-of-message token after the letter, so the runner found no letter and played the letter Qwen had sampled instead of its most likely one, in every one of its decisions there. The runner now skips such trailing tokens; Qwen3.8-27B (off)'s affected matches were voided on 20 October and replayed with the same dice. No other setup was affected.
- Withdrawn. Qwen3.8 Flash Next (off, on): with reasoning on it took many minutes per decision on a DGX Spark. Its matches are removed from the league. Qwen3.8 Flash (off), through OpenRouter, joined instead.
- Moved. On 19 October League 1 became decision models only: Gemma 4 31B (off, on) and Qwen3.8-27B (off, on) moved to League 2, and their League 1 matches no longer count. GPT-6 Luna's first entry, through the ChatGPT subscription as a chat model, was removed; it plays through OpenAI's Decisions API. The other new decision models joined League 1 the same day; Fastino GLiDE and Decider v1.2 27B joined on 20 October.
- Hardware. Clef, Cygnet and Decider v1.1 27B moved from our RTX PRO 6000 cards to DGX Spark (GB10) machines on 16 October, with the same checkpoints and settings (replayed requests: the same play in all). Open-Jev stayed, because on a DGX Spark it sometimes chose another play. Gemma 4 31B ran on a DGX Spark from 19 October 18:36 to 22:26 UTC, then on an RTX PRO 6000 again (the same play in all replayed requests). Qwen3.8-27B started on Cerebras through OpenRouter, where it finished no match, and then moved to our GPU in fp16 (the same top letter in most replayed requests). From 20 October 21:10 UTC Gemma 4 31B, LFM2.5-24B-A2B and Qwen3.8-27B played on DGX Sparks with the same checkpoints and settings (the same top letter in all replayed requests), so that DeepSeek V4.1 Flash could have all of our RTX PRO 6000 cards; on 21 October 01:25-05:00 UTC they played their last non-DeepSeek matches on the RTX PRO 6000 cards again (LFM2.5 is not deterministic, so its replays agree less often).
- DeepSeek V4.1 Flash on our GPUs. From 20 October 21:56 UTC, DeepSeek V4.1 Flash (off, on) plays on our own SGLang server instead of Fireworks through OpenRouter: the 17 October revision of the published weights (FP4 routed experts, FP8/BF16 elsewhere); Fireworks does not say which revision it serves. On stored Fireworks requests our server usually chose the same top letter, but not always; it is not deterministic, and Fireworks replaying its own stored answers did not always agree either.
- DeepSeek V4.1 Flash back on Fireworks. From 22 October 15:37 UTC, for League 2's last four matches, DeepSeek V4.1 Flash (off, on) plays on Fireworks through OpenRouter again, with the provider pinned to Fireworks and fallbacks off, as before 20 October 21:56 UTC; Gemma 4 31B, LFM2.5-24B-A2B and Qwen3.8-27B moved from the DGX Sparks back to our RTX PRO 6000 cards (same checkpoints and settings; replayed requests: the same top letter in 20 of 20 for Gemma and Qwen, 17 of 20 for LFM2.5, which is not deterministic). Matches under way keep their stored turns; every decision records its provider, so it shows which turns ran where.
- Gemini by subscription. With no Gemini API key, Gemini 3.8 Flash (low, high) plays through the Google Antigravity subscription from 19 October 12:46 UTC: the same model, prompt, thinking levels and constrained letter. On the smoke positions it nearly always chose the same play as the API.
- Paused matches. A match that fails on a provider error (OpenRouter credit or rate limits, a server restart) waits and resumes later from its last stored turn; the other matches go on.
About the anchor
The anchor, GNU Backgammon 1.08 at its Beginner level (0-ply, deterministic noise 0.06), ran on one x86-64 build
only, and on x86-64 it is not the Beginner that GNU Backgammon intends. Its deterministic-noise code sums the hash
bytes as char, which is signed on x86-64, but subtracts the mean of unsigned bytes, so every noise draw
is shifted by about −7 standard deviations. The anchor understates its own winning chances and plays for gammons.
- Another build plays differently. On the 1,000 v1 positions, the x86-64 and ARM builds of the same package choose different plays in 659; with the noise off, in 6, all exact ties. Error rate 21.8 on x86-64 against 16.8 on ARM.
- Gammons in self-play. 72% of games end in a gammon or backgammon on x86-64, 40% on ARM and about 30% without noise.
- Gammons in the leagues. Games with the anchor end in a gammon or backgammon in 54% (League 1), 46% (League 2), 26% (League 3); games between two models in 26% (League 1), 27% (League 2), 18% (League 3); games with Random or Safest in 34%.
- Match wins. 26 of the 26 matches the anchor won included a gammon or backgammon by it. Scored at 1 point per game, 10 of those wins are certain (7 in League 1, 2 in League 2, 1 in League 3); the rest would need games that were never played.
So v3's anchor is "GNUbg 1.08 Beginner, x86-64 build", and its results are reproducible on x86-64 only: a replay there gives the recorded plays. Later releases of the benchmark use a corrected build.
Shared dice tested and rejected
Each match has its own dice. We tested whether giving round r of every match the same roll schedule would make the table less a matter of luck: 8 GNU Backgammon players of known strength, 500 replays of a round robin under each scheme. Shared schedules made league points more scattered (standard deviation ×1.10, 95% interval 1.06 to 1.14) and lowered the rank correlation with the true order from 0.890 to 0.864. Mirrored games already cancel most of the dice luck within a match; sharing schedules across matches cancels nothing more and puts all of a player's matches on the same dice, so their luck adds up. The test players ran on the same x86-64 build as the anchor, so their noise levels are shifted too; that applies to both schemes and does not affect the comparison.
The setup
Each model plays every other model once per league (a single round robin). A match is played in rounds of two games with the same dice: in the second game the players swap seats and the same sequence of rolls is repeated, so both face the same luck. Games score 1 point, 2 for a gammon and 3 for a backgammon; there is no doubling cube.
- Match length: 3 rounds, or fewer: the first player to 5 points after a full round (both games) wins if ahead. A tie gets one extra round, then counts as a draw.
- League table: 3 points for a match win, 1 for a draw; ties broken by more games won, then fewer games lost.
- Turn limit: a game still running after 300 turns (both sides counted) goes to the lower pip count for 1 point.
- Clock: 30 seconds free per decision, then a 5-minute bank per game. Running out loses the game.
- Legality: the program generates every legal play and the model chooses one of them, so a model can never make an illegal move. A roll with only one legal play is played automatically.
- Memory: every decision is a separate request. Models are not told how earlier games went.
With seven models, each league has 21 matches and one model sits out each matchday. A match starts as soon as both of its models are free, and no model plays two matches at once.
Models
All seven answer the same kind of request: a text state and one choice question whose options are the legal plays. Two are hosted services; the other five run on our own GPUs.
- Jev 1.13.0 · TypeSafe, hosted
- TypeSafe's decision model, served through its System One API; the request format the others copy.
- GPT-6 Luna · OpenAI Decisions API, hosted
- OpenAI's decision endpoint with
gpt-6-luna. The only one that sometimes declines to choose; that happened 3 times in early leagues, on positions with 97 to 439 legal plays, and never in the current format. - Decider v1.1 27B · Perplexity, self-hosted
- The released
pplx-decider-v1.1-27bcheckpoint, run with its own server code. It accepts up to 8,192 tokens, so long play lists are sent in groups of 25 (see below). It replaced Perplexity's hosted Decider 27B from league #6 on. - Clef · Cloudflare, self-hosted
- A 27B decision model that scores all options in one pass. Groups of up to 110 options.
- Open-Jev 27B v1.1 · open reimplementation of Jev, self-hosted
- A LoRA adapter and decision head on a 27B base model. It scores every option as its own sequence that repeats the whole question, with a 4,096-token limit per sequence, which is what limits the prompt's length for every model.
- Cygnet · blockbrain-ai recipe, self-hosted
- Google's
gemma-4-12B-it, unchanged, behind a server that labels the options A, B, C… and reads the model's probability for each letter. Groups of up to 26 options. - Gemma 4 31B · our configuration of the Cygnet recipe
- The same letter-reading server in front of
gemma-4-31B-it, served in FP8. Not a trained decision model: a general chat model whose answer-letter probabilities are read directly. Its calibration setting was fitted for Cygnet's 12B model; that changes its reported confidence, not which play it picks.
When a position has more legal plays than a model's group size, the plays are split into equal groups, the model picks a winner in each, then chooses among the winners. Liquid's d1 played in the early leagues and was dropped after its free tier rate-limited a league to a stop.
The prompt
Every decision sends the same parts in the same order:
- Rules and strategy (about 6,400 characters): the rules of backgammon, strategy principles (blots and points, distribution, exposure, blocking, hitting, anchors), the best opening plays from computer rollouts, and the match rules above. A longer version with a glossary, game plans and a worked example goes to chat models; decision models get this short one because of Open-Jev's 4,096-token limit.
- Score and time: match score, round and game, turn number, both clocks.
- Position from the mover's side: checkers on every point, bar, borne off, pip counts.
- Race: the pip lead and whether the two sides are still in contact.
- The last 10 moves of this game, both sides, in the mover's numbering (league #13 on).
- Dice, then the question: Which of the following moves makes a point in front of the opponent's checkers, or reduces your single pieces, or gets you closer to your home?
- The legal plays, each followed by what the position looks like after it, computed by the program.
The shape of a request (the rules part is left out; angle brackets stand for the values of the position at hand, and no real position or answer is shown, so that the positions stay usable as a test):
Score: you <n>, opponent <n>. Match: round <r> of <R>, game <g> of <G> in this round. Turn <t> of at most <T> in this game. Clock: <free delay> for this decision, then your bank: <time> left in this game (opponent <time>). Position (your numbering; you move 24 -> 1): your checkers: <point:count …> opponent checkers: <point:count …> on the bar: you <n>, opponent <n> borne off: you <n>, opponent <n> pip count: you <n>, opponent <n> Race: <the pip lead>; <whether the checkers are still in contact>. Last 10 moves of this game (oldest first, your numbering): turn <t>, <you / opponent> rolled <dice>: <play> … Dice: <d1>-<d2> Which of the following moves makes a point in front of the opponent's checkers, or reduces your single pieces, or gets you closer to your home? Options (<n>): <play>: hits <n>; makes <points made>; leaves <n> blots, the opponent hits with <n> of 36 rolls; you hold <n> points (<n> in your home board), longest prime <n>; pips: you <n>, opponent <n> … (one line per legal play) Answer: <one of the plays> (with the model's probability where it gives one)
The numbers after each play are exact. The opponent's hitting rolls are counted by generating its legal plays for each of the 36 rolls and checking whether any of them hits.
Why it looks like this
The prompt was changed one step at a time, and a league was run after each step. Leagues stopped part-way are kept, with their finished matches only.
| League | What changed | Games | Hit the 300-turn limit | Moves that hit | Average game | Best opening play | Error rate |
|---|---|---|---|---|---|---|---|
| #1 | Structured state (JSON) with a short rules text and how every earlier game ended | 38 | 82% | 62% | 277 turns | 28% | 138 |
| #3 | Rules and strategy text, then score, time, position and dice; nothing about earlier games | 170 | 81% | 60% | 274 turns | 77% | 132 |
| #6 | + the question before the choices; Decider v1.1 self-hosted | 124 | 71% | 58% | 255 turns | 83% | 120 |
| #7 | + what each play leads to, and the race line | 135 | 0% | 9.7% | 50 turns | 68% | 38.5 |
| #8 | As #7 without d1 (7 models) | 120 | 0% | 9.5% | 51 turns | 69% | 39.4 |
| #11 | Repeat, new dice | 112 | 0% | 9.3% | 50 turns | 58% | 40.1 |
| #12 | Repeat, new dice | 112 | 0% | 9.7% | 51 turns | 69% | 36.2 |
| #13 | + the last 10 moves of the game | 106 | 0% | 10.9% | 52 turns | 72% | 39.1 |
Grey rows did not finish. "Best opening play" is the share of first moves of a game that match the computer-rollout table in the prompt. "Error rate" is GNU Backgammon's, over all model decisions of the league (see Error rate); lower is better. Leagues #2, #4, #5, #9 and #10 failed at start or were stopped within minutes; league #3 is left out of the standings below because its format differs.
Rules and strategy as text, not a data structure
The first leagues sent the position as structured data with a short rules summary. Models opened badly: 28% of first moves matched the known best play. With the rules, strategy principles and the opening table as prose, that rose to 77%. Opening play improved, but the middle game did not: games still ran to the turn limit, because both sides kept hitting and re-entering.
A question that names what a good move does
The question before the choices states three goals in plain words: build a point in front of the opponent's checkers, reduce your blots, bring checkers home. It deliberately leaves hitting out, because hitting on almost every move was the problem. The turn-limit share fell from 81% to 71% (this league also switched to the self-hosted Decider v1.1); hitting barely changed.
Compute the consequences in code, not in the model
Backgammon decisions turn on numbers a language model cannot work out reliably from a list of notations: which
checkers are left alone, how many of the opponent's 36 rolls hit them, which points are made. Earlier, each option
was just its notation, like bar/19 13/11*, and the model had to imagine the resulting position.
Now the program computes those numbers for every option and the model compares them. This is the change that
ended the endless games: hits fell from about 60% to under 10% of moves and the average game from about 270 turns
to about 50.
Opening play got worse at the same time, from about 80% to between 58% and 69% best plays. The metrics after each option compete with the opening table earlier in the prompt, and on the first move several plays look similar on these numbers.
Recent moves, for longer plans
Each decision is a separate request, so a model cannot see how the position came about. League #13 adds the last 10 moves of the game, both sides. Ten turns add about 200 tokens. More would eat into Open-Jev's 4,096-token limit, which it applies to the whole question repeated for every option.
It made no measurable difference. League #13's error rate was 39.1 (± 2.2), against 38.6 (± 1.2) for #8, #11 and #12 together, and the finishing order barely changed. Ten moves may be too few, or these models may not use them.
Error rate
Wins and losses depend heavily on the dice. A backgammon engine can instead judge every decision directly: for each move it finds the best play, and measures how much winning chance, in equity (expected points per game), the chosen play gave away. Averaged over all decisions and multiplied by 500, this is the error rate (performance rating, PR) that backgammon players use, here for checker plays only; forced moves are not counted. eXtreme Gammon's human scale: under 2.5 world champion, 2.5 to 5 world class, 5 to 7.5 expert, 7.5 to 12.5 advanced, 12.5 to 17.5 intermediate, 17.5 to 22.5 casual, 22.5 to 30 beginner, 30 or more distracted. Human ratings also count doubling cube decisions, which these games do not have, so the comparison is approximate.
We scored all 104,045 decisions the models made in every league with GNU Backgammon 1.08. Every legal play was evaluated at 0-ply, then the five best and the chosen play again at 2-ply; the loss is the 2-ply equity of the best play minus that of the chosen one. As a check, on the opening roll the plays that the rollout table calls best lose 0.0002 equity on average, other plays 0.063.
By model, leagues #8, #11 and #12
| Model | Decisions | Error rate | Engine's best play | Blunders | #8 | #11 | #12 | #13 |
|---|---|---|---|---|---|---|---|---|
| Gemma 4 31B | 1,894 | 25.4 ± 2.4 | 54% | 19.5% | 22.7 | 26.3 | 27.2 | 28.8 |
| Decider v1.1 27B | 1,942 | 32.7 ± 2.9 | 49% | 24.5% | 33.0 | 33.1 | 31.9 | 34.2 |
| Jev 1.13.0 | 2,066 | 35.7 ± 2.9 | 46% | 26.7% | 34.2 | 36.2 | 36.7 | 32.0 |
| Clef | 2,100 | 35.8 ± 3.0 | 47% | 25.6% | 35.7 | 40.5 | 30.6 | 42.8 |
| GPT-6 Luna | 2,044 | 39.4 ± 3.5 | 46% | 25.6% | 40.1 | 35.9 | 41.7 | 37.5 |
| Cygnet | 2,078 | 43.2 ± 3.4 | 42% | 30.2% | 46.9 | 45.6 | 36.7 | 40.6 |
| Open-Jev 27B | 1,758 | 59.6 ± 4.6 | 36% | 36.5% | 65.8 | 63.1 | 49.5 | 58.1 |
Error rate: mean equity lost per decision × 500, with a 95% interval that treats decisions as independent (moves within a game are not, so the true interval is somewhat wider). "Engine's best play": share of decisions where the chosen play lost less than 0.001. "Blunders": decisions losing more than 0.08 equity. The last four columns are the error rate in each league; #13 adds the move history.
- The engine ranks the models almost as the league tables do. Gemma 4 31B is clearly best and Open-Jev 27B clearly worst. Decider v1.1, Jev, Clef and GPT-6 Luna are within about one interval of each other, which is why their league places move around; Cygnet is a little behind them.
- Every model blunders often: from one move in five (Gemma 4 31B) to one in three (Open-Jev 27B). Even Gemma 4 31B chooses the engine's best play only about half the time.
- The prompt mattered more than the model. Before move consequences were computed, the whole field averaged 120 to 138; after, 36 to 40. The worst model now is better than the average model was then.
Results
The standings below use the three finished leagues with computed move consequences and no move history: #8, #11 and #12 (7 models, same prompt, different dice). League #13, with move history, finished 1. Gemma 4 31B, 2. Jev 1.13.0, 3. Decider v1.1 27B, 4. Cygnet, 5. GPT-6 Luna, 6. Clef, 7. Open-Jev 27B.
Final places
| Model | 1st | 2nd | 3rd | 4th | 5th | 6th | 7th |
|---|---|---|---|---|---|---|---|
| Gemma 4 31B | 2 | 1 | 0 | 0 | 0 | 0 | 0 |
| Jev 1.13.0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 |
| Decider v1.1 27B | 0 | 2 | 1 | 0 | 0 | 0 | 0 |
| Clef | 0 | 0 | 1 | 1 | 0 | 1 | 0 |
| Cygnet | 0 | 0 | 1 | 1 | 0 | 1 | 0 |
| GPT-6 Luna | 0 | 0 | 0 | 0 | 3 | 0 | 0 |
| Open-Jev 27B | 0 | 0 | 0 | 0 | 0 | 0 | 3 |
In league #12, Jev 1.13.0 and GPT-6 Luna finished level on every tiebreak and share 5th place.
Across the three leagues
| Model | Matches W-D-L | Games won-lost | Points for-against | Gammons and backgammons won | Picked the safest play | Shots left on average | Time per decision |
|---|---|---|---|---|---|---|---|
| Gemma 4 31B | 14-1-3 | 67-27 | 91-36 | 23 | 84% | 5.1 | 0.30 s |
| Decider v1.1 27B | 12-0-6 | 56-40 | 79-56 | 20 | 77% | 6.1 | 1.14 s |
| Jev 1.13.0 | 11-0-7 | 53-49 | 71-61 | 16 | 82% | 5.1 | 0.26 s |
| Clef | 9-1-8 | 53-45 | 81-64 | 21 | 68% | 7.0 | 1.13 s |
| Cygnet | 8-0-10 | 52-54 | 65-72 | 11 | 82% | 4.6 | 0.22 s |
| GPT-6 Luna | 8-0-10 | 48-54 | 62-68 | 11 | 87% | 3.8 | 0.20 s |
| Open-Jev 27B | 0-0-18 | 15-75 | 18-110 | 2 | 65% | 8.5 | 1.69 s |
"Safest play": the option whose position the opponent hits with the fewest of its 36 rolls, read from the numbers the model was shown. "Shots left": opponent rolls out of 36 that hit after the chosen play. Both are measured on decisions sent as a single request (about 1,700 to 2,100 per model); the safest available play left 2.7 to 4.2 shots on average, a typical option about 13.
What the numbers say
- Gemma 4 31B is the strongest player here, and it is not a trained decision model: a general chat model whose answer-letter probabilities are read. It lost 3 of 18 matches and won the most gammons.
- Safe is not the same as good. GPT-6 Luna picks the safest play most often and leaves the fewest shots, yet finished 5th in all three leagues. Winning also takes making points and building a prime, which costs some risk; Gemma and Clef take more risk and win more gammons.
- Open-Jev 27B collapsed in this format. It finished 2nd in league #3, where options were just notations, and last in every league since, with the most shots left and the highest error rate. We have not found out why; the numbers it is shown are the same as everyone else's.
- Speed does not decide it. The three fastest models are spread across the table, and no model came close to its clock.
Can a team beat the best model?
Every model reports a probability for each option. If those probabilities say when a model is right, a team could ask all seven and play the answer of whichever is most sure. To test this, we sent the same 1,000 recorded positions (2 to 25 legal plays each, drawn from leagues #8, #11, #12 and #13) to all seven models with exactly the request the original player received, and scored every legal play with GNU Backgammon at 2-ply: 7,000 answers and 9,439 evaluated plays. Any team and any rule can then be scored offline on the same positions.
Confidence says how hard a position is, not who is right
Within each model, confidence does carry information: across its league decisions, a model is right more often when it is more confident (AUROC 0.66 to 0.74, where 0.5 is chance). But the models use very different scales:
| Model | Engine's best play | Average stated probability | When it is most sure | it is right |
|---|---|---|---|---|
| Gemma 4 31B | 57% | 0.85 overconfident | 0.8 – 1.0 | 65% |
| Decider v1.1 27B | 51% | 0.58 | 0.8 – 1.0 | 84% |
| Jev 1.13.0 | 47% | 0.55 | 0.8 – 1.0 | 81% |
| Clef | 46% | 0.28 underconfident | 0.6 – 0.8 | 89% |
| GPT-6 Luna | 46% | 0.56 | 0.8 – 1.0 | 65% |
| Cygnet | 44% | 0.67 overconfident | 0.8 – 1.0 | 63% |
| Open-Jev 27B | 36% | 0.21 underconfident | 0.6 – 0.8 | 92% |
From 17,579 league decisions sent as a single request; "right" means the engine's best play (loss below 0.001 equity). The stated probability is the one the model gave its chosen play; the models' own "confidence" fields are defined differently from one another, so they are not used.
Clef and Open-Jev look like the most reliable models when they are sure, but they are only sure on easy positions, where everyone else is right as well. The test that matters is what happens when models disagree on the same position:
| When Gemma 4 31B and this model disagree | Gemma has the better play | The more confident model has the better play |
|---|---|---|
| Decider v1.1 27B | 62% | 62% |
| Clef | 64% | 64% |
| Jev 1.13.0 | 69% | 67% |
| GPT-6 Luna | 73% | 70% |
| Cygnet | 77% | 75% |
| Open-Jev 27B | 76% | 76% |
Following the more confident model is never better than simply following Gemma 4 31B.
Teams
| Team and rule | Error rate | Engine's best play | Difference from Gemma alone (95%) |
|---|---|---|---|
| Gemma 4 31B alone | 23.5 | 54.4% | – |
| All 7: highest stated probability | 24.0 | 53.8% | +0.4 (−0.2 to +1.2) |
| All 7: lowest expected loss, calibrated | 26.0 | 52.9% | +2.5 (+0.7 to +4.3) |
| All 7: most likely correct, calibrated | 27.7 | 51.9% | +4.2 (+2.2 to +6.4) |
| All 7: probabilities added up | 27.7 | 52.0% | +4.2 (+2.1 to +6.6) |
| Best of 120 formations: Clef + Decider v1.1 + Gemma, probabilities added up | 23.4 | 54.8% | −0.2 (−0.7 to +0.2) |
| Oracle: the best answer any of the 7 gave | 12.7 | 70.1% | −10.8 |
Error rate on the 1,000 shared positions (lower is better). Calibrated rules map each model's stated probability to its observed accuracy or loss with isotonic regression, fitted on one half of the positions and scored on the other (two-fold cross-fitting). Intervals are bootstrap intervals over positions, paired with Gemma 4 31B on the same positions. The best formation was chosen on the same positions it is scored on, which flatters it.
- No team beats the best single model. Raw confidence ties with Gemma 4 31B only because its inflated probabilities win almost every vote; correcting the scales makes the team worse.
- The room for a team is large. Gemma 4 31B misses the engine's best play on 456 of the 1,000 positions, and on a third of those another model found it. An oracle would halve the error rate. What is missing is a way to tell which answer is the right one.
- The data is kept for further formations. Other selectors (agreement between models, the number of legal plays, the stage of the game) can be tested on the same 1,000 positions without new requests.
Speed and cost
Measured on the same 1,000 recorded positions as the team test, with exactly the request each original player received. Besides the seven league players, the table includes general language models we tried the same way: DeepSeek V4.1 Flash, Qwen3.8 Flash Next and GLM-4.7-Flash are read through their probabilities for each answer letter, like Gemma 4 31B. GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon and Gemini 3.8 Flash return no such probabilities: each answers one letter, constrained to the option letters, at low reasoning effort (Gemini's lowest available level). Gemini 4 Argon reported no reasoning tokens at that level, so it answered directly. DeepSeek V4.1 Flash also ran a second time with reasoning on (lowest effort), answering one constrained letter after reasoning. GPT-6 Astra and Claude Opus 5.5 ran through ChatGPT and Claude subscriptions.
| Model | Runs on | Error rate | Time per decision median / 90th pct | In flight | Estimated API cost per 1,000 decisions |
|---|---|---|---|---|---|
| GPT-6 Astra | ChatGPT subscription | 11.2 | 4.60 s / 10.95 s | 4 | $36.45 API-equivalent |
| Gemini 4 Argon | Google Gemini API | 16.9 | 4.09 s / 28.22 s | 4 | $6.97 |
| Claude Opus 5.5 | Claude subscription | 17.3 | 3.43 s / 6.66 s | 4 | $21.65 API-equivalent |
| DeepSeek V4.1 Flash, reasoning on | our GPUs: 4 | 18.1 | 106 s / 288 s | 8 | own hardware |
| Gemini 3.8 Flash | Google Gemini API | 18.3 | 2.87 s / 7.96 s | 4 | $3.25 |
| Gemma 4 31B | our GPUs: 1 | 23.5 | 0.59 s / 0.87 s | 4 | own hardware |
| DeepSeek V4.1 Flash | our GPUs: 4 | 27.5 | 0.57 s / 0.98 s | 4 | own hardware |
| Qwen3.8 Flash Next | our GPUs: 4 | 31.1 | 0.46 s / 0.71 s | 4 | own hardware |
| Decider v1.1 27B | our GPUs: shares 2 | 31.8 | 0.82 s / 1.35 s | 1 | own hardware |
| Jev 1.13.0 | TypeSafe API | 33.6 | 0.25 s / 0.29 s | 4 | $0.16 |
| Clef | our GPUs: 1 | 34.1 | 0.94 s / 1.28 s | 1 | own hardware |
| GPT-6 Luna | OpenAI Decisions API | 35.0 | 0.22 s / 0.27 s | 4 | $0.32 |
| Cygnet | our GPUs: 1 | 38.9 | 0.78 s / 1.12 s | 4 | own hardware |
| GLM-4.7-Flash | our GPUs: 1 | 50.6 | 0.20 s / 0.27 s | 4 | own hardware |
| Open-Jev 27B | our GPUs: 1 | 59.9 | 1.38 s / 1.91 s | 1 | own hardware |
Error rate on the 1,000 shared positions (lower is better). Time is wall-clock per request, from sending it to receiving the answer, with the stated number of requests in flight. Our GPUs are NVIDIA RTX PRO 6000 Blackwell cards with 96 GB each, in one machine. The seven league players were asked together, so the self-hosted ones shared the machine; the other models were each asked alone. A run of GLM-5.3-Flash is left out: it played close to random, most likely because of how we had to run it on this machine.
The plays are listed in the order the runner generated them, which tends to put the best play early. Asked again with the same plays in a random order, Gemini 4 Argon's error rate rose from 16.9 to 18.9 (difference 1.9, 95% interval −0.3 to 4.1): probably a small help from the order, not yet a proven one. The next version of the benchmark shuffles the plays for every model.
Cost basis: the provider's list price × the tokens its API reported; the bill itself was not checked. Jev 1.13.0 ($0.042 per 1M input tokens, output free) and GPT-6 Luna on OpenAI's Decisions API ($0.10 per 1M input tokens, output free): prices of 8 October 2026, tokens averaged over their league decisions with 2 to 25 legal plays (3,740 and 3,238 input tokens per decision). Gemini 3.8 Flash ($0.75 per 1M input tokens, $3.75 per 1M output tokens, reasoning billed as output; introductory price of 8 October 2026, which doubles on 1 January 2027): tokens measured on 100 of the positions, 3,373 input and 192 output per decision on average. Gemini 4 Argon ($2 per 1M input, $10 per 1M output; price until 30 November 2026, doubling on 1 December): tokens reported for all 1,000 positions, 3,462 input and 5 output per decision. GPT-6 Astra ($10 per 1M input, $50 per 1M output) and Claude Opus 5.5 ($4 input, $20 output): API list prices of 8 October 2026 × the tokens reported for all 1,000 positions (3,213 input and 86 output, and 4,424 input and 198 output per decision). Both ran on subscriptions, which are not billed per token, so these are API-equivalent estimates. Self-hosted models have no per-request price; a 7-model league of about 690 decisions per player took about 30 minutes on the four GPUs. DeepSeek with reasoning used 6,657 output tokens per decision on average.
- The cheapest strong player is one we run ourselves. Gemma 4 31B costs only GPU time, needs one card, and answers in about 0.6 s.
- The best players cost most and are slowest. GPT-6 Astra makes the fewest errors by far, but at API prices it costs about a hundred times as much per decision as GPT-6 Luna, and one decision in ten takes 11 seconds or more. Between Gemini 4 Argon and Claude Opus 5.5 there is no clear difference (16.9 against 17.3; difference 0.3, 95% interval −1.9 to 2.5, which includes zero), with Argon about a third of Opus's price; Gemini 3.8 Flash scores 18.3 at half Argon's price. Even asking Astra, Opus and Gemini 3.8 Flash together and taking the majority answer is worse than Astra alone (14.4).
- Reasoning helps an open model a lot, at a steep time cost. With reasoning on, DeepSeek V4.1 Flash's error rate falls from 27.5 to 18.1 (95% interval of the gain 6.1 to 12.7), against Opus's 17.3, but a decision takes about 100 seconds instead of half a second.
- The hosted decision services are fastest and cheapest per decision, at about 0.2 s and well under a dollar per 1,000 decisions, but they play worse than Gemma 4 31B.
Limits
- Small sample. Three finished leagues of 21 matches in the main format, about 340 games. Placings in the middle of the table move from league to league; the error rate is the steadier measure.
- The engine evaluation is not the strongest available. Moves are judged at 2-ply with GNU Backgammon, not by full rollouts or eXtreme Gammon, and in cubeless money equity: the games have no doubling cube, and match score is ignored. Small differences between close plays can be wrong, but the losses that count as blunders and the gaps between models are much larger than that.
- Models only play each other. Their absolute strength against human or engine players is unknown. On the engine's scale none of them reaches expert level; the best, GPT-6 Astra, plays like an advanced human.
- Configuration matters. Five models run on servers set up by us, and Gemma 4 31B uses a recipe built for a different model. Hosted services can change without notice.
- The prompt steers. The question and the strategy text say what a good move looks like, so the results measure how well models follow that guidance with the numbers in front of them, as much as how well they play.