Data
Data, versions and how to recompute
Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.
Downloads
results.json New models: first look Every model setup
results.json: every league's results per setup, one league per key, every risk metric at both margins. Per-decision records (positions, plays, the engine's best play) are not published: why.
Swipe for more columns →
Data files are split per version, league and model, so each loads on a phone. Every number on the site is in one of them. The shared rules texts of the prompts are under decisions/contexts/ (2 files).
Analysis status
Swipe for more columns →
| Analysis | What | Status | Data file | Waits for the running leagues |
|---|---|---|---|---|
| A1 | Decision quality: error rate, best-play share, blunder rate, intervalsShared positions (main table) | Analysis pending | data/v3/shared.json | no |
| A1 | Decision quality: error rate, best-play share, blunder rate, intervalsLeague decisions, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json, data/v3/league-N/<setup>.json | no |
| A1 | Decision quality: error rate, best-play share, blunder rate, intervalsv1 set | Earlier version | data/v1/v1-set.json | no |
| A2 | Effect of the information given to the model (notation only / consequences / history)Shared positions, 6 setups x 3 prompts | Analysis pending | none yet | no |
| A3 | Return on more reasoning (paired: the same model at a lower and a higher setting)Shared positions | Analysis pending | data/v3/shared.json | no |
| A3 | Return on more reasoning (paired: the same model at a lower and a higher setting)v1 set: DeepSeek-V4.1-Flash off vs thinking | Earlier version | data/v1/v1-set.json (reasoning), data/v1/compare.json | no |
| A4 | Where models fail: phase, strategy tag, number of options, gap between the two best playsShared positions | Analysis pending | data/v3/shared.json | no |
| A4 | Where models fail: phase, strategy tag, number of options, gap between the two best playsLeague decisions, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N/<setup>.json (phases, tags, options, gaps) | no |
| A4 | Where models fail: phase, strategy tag, number of options, gap between the two best playsv1 set | Earlier version | data/v1/v1-set/<setup>.json | no |
| A5 | Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasLeague decisions, per league (Confidence tab: cutoffs 0.90/0.99, surest quarter, AUROC best play and blunder warning, ECE, confident and badly wrong; held-out cutoff in Method) | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (confidence) | no |
| A5 | Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasShared positions (setups with probabilities) | Analysis pending | data/v3/shared.json | no |
| A5 | Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasv1 set (setups with probabilities) | Earlier version | data/v1/v1-set/<setup>.json (quality.auroc, ece, brier) | no |
| A5 | Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasRisk-coverage curves, letter-position bias | Analysis pending | none yet | no |
| A6 | Routing from a cheap model to a strong oneShared positions | Analysis pending | none yet | no |
| A7 | Teams: can a team beat its best member? (oracle as upper bound only)Shared positions / per league | Analysis pending | none yet | no |
| A7 | Teams: can a team beat its best member? (oracle as upper bound only)v1 set, earlier report | Earlier version | content/archive (earlier page section, moved to Leagues) | no |
| A8 | Repeatability: 200 positions asked 5 timesShared positions | Analysis pending | none yet | no |
| A9 | Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timeShared positions | Analysis pending | data/v3/shared.json | no |
| A9 | Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timeLeague decisions, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (quality.time_*, usd_per_1000, cost_coverage) | no |
| A9 | Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timev1 set (time, tokens; cost not measured) | Earlier version | data/v1/v1-set.json | no |
| A10 | League results: standings, W-D-L, games, gammons; safest-play share and hitting rollsStandings per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (standings) | no |
| A10 | League results: standings, W-D-L, games, gammons; safest-play share and hitting rollsRisk calls in play (B6, per league) | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (risk.main, risk.robust) | no |
| A11 | Doubling cube (optional track)- | Not measured | none | no |
| A12 | Known opening knowledge: opening table on/off, NackgammonNackgammon 500 + opening control 200 | Analysis pending | data/v3/shared.json (set sizes only) | no |
| A13 | Does the shared-position score predict league results?Shared positions + finished leagues | Analysis pending | none yet | no |
| High 1 | When does high hurt? Decisions that high fixes and breaks (counts and loss sizes)Shared positions | Analysis pending | none yet | no |
| High 1 | When does high hurt? Decisions that high fixes and breaks (counts and loss sizes)v1 set: DeepSeek-V4.1-Flash off vs thinking | Earlier version | data/v1/compare.json (fixed, broken, fixed_loss, broken_loss) | no |
| High 2 | When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder rateShared positions | Analysis pending | none yet | no |
| High 2 | When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder rateLeague decisions, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (quality.p95_loss, worst_mean, blunder_share) | no |
| High 2 | When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder ratev1 set | Earlier version | data/v1/v1-set.json | no |
| High 3 | When does high hurt? Confident and badly wrongLeague decisions, setups with probabilities, per league | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (quality.confident_badly_wrong) | no |
| High 3 | When does high hurt? Confident and badly wrongv1 set | Earlier version | data/v1/v1-set.json | no |
| High 3 | When does high hurt? Confident and badly wrongShared positions | Analysis pending | none yet | no |
| High 4 | When does high hurt? Going over a time budgetLeague decisions: share of request times over the budget | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (quality.over_budget) | no |
| High 5 | When does high hurt? High only when needed (switching rule on held-out positions)Shared positions | Analysis pending | none yet | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02League decisions, per league, both margins | League 1: Published · League 2: Published · League 3: Published | data/v3/league-N.json (risk.main, risk.robust, references), data/v3/league-N/<setup>.json (risk.*.phases) | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Risk map, error rate by risk-call category, stake chart, risk call cards (league winners) | League 1: Published · League 2: Published · League 3: Published | same; share/risk-card-league-N-*.png | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Every decision marks (position class, stake, risk call, best safe play) | League 1: Published · League 2: Published · League 3: Published | decisions/<league>/<setup>/risk.json.gz and page files | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02v1 set | Earlier version | data/v1/v1-set.json (risk) | no |
| B6 | Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Shared positions | Analysis pending | none yet | no |
| L | Learning from experience- | Not measured | none | no |
Versions
- v3
- Prompt bg-v2, plays in a seeded random order, invalid answers not retried. Leagues 1 to 3 and the shared positions.
- v1
- Earlier version Earlier prompt; plays listed in the runner's order. The v1 set of 1,000 positions answered by the v1 setups. Never in a v3 table or chart.
- Smoke tests
- 50 positions outside the shared sets, v3 prompt; only on New models: first look.
How to recompute
- Snapshot the league database:
python3 tools/site/snapshot.py v3.db STATE(one read transaction; the prompts each model was sent go toSTATE/prompts.db; the Nackgammon knockout's matches, fromknockout-nack.dbnext to it, toSTATE/knockout.db). - Score every legal play of every decision with GNUbg 1.08 at 2-ply:
tools/site/score.sh STATE/snap.db STATE/plays. - Build and check the site:
python3 tools/site/build.py --state STATE --out OUT. The build fails, and nothing is deployed, if a number does not trace to a data file, counts do not add up, leagues or versions mix, a published finding no longer holds, or a file contains a private address or credential.
The scripts are in the jevgammon repository under tools/site/; the fixed thresholds are in thresholds.json.