Data

Data, versions and how to recompute

Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.

Downloads

results.json: every league's results per setup, one league per key, every risk metric at both margins. Per-decision records (positions, plays, the engine's best play) are not published: why.

Swipe for more columns →

Data files of this build
FileBytessha256
data/analysis-status.json10,926a187f525132d
data/human-scale.json70939008356b91e
data/results.json910,77931dba13e0cf5
data/setups.json164,8851fffeb94db48
data/smoke/part-a.json27,764241ff998d532
data/thresholds.json1,765b5400c053075
data/v1/compare.json107,78050162877a513
data/v1/disagreements.json271,0750a2ee8fc5a5a
data/v1/v1-set/claude-opus-5-5.json49,792e14f7ed18422
data/v1/v1-set/clef.json53,374119859fe5cd3
data/v1/v1-set/cygnet.json53,40101eeb79a3c42
data/v1/v1-set/decider-v1-1-27b.json53,4679ba63fb1a39d
data/v1/v1-set/deepseek-v4-1-flash-thinking.json49,990244d2c5a65f8
data/v1/v1-set/deepseek-v4-1-flash.json52,765982557693ddb
data/v1/v1-set/gemini-3-8-flash.json49,14398eae6cce2dd
data/v1/v1-set/gemini-4-argon-shuffled.json49,992cec52522ed8e
data/v1/v1-set/gemini-4-argon.json50,33486babd76eeb0
data/v1/v1-set/gemma-4-31b.json53,1769df7b904dfda
data/v1/v1-set/glm-4-7-flash.json53,535bc05ad1d8c7b
data/v1/v1-set/glm-5-3-flash-nvfp4.json53,5730e9bd71509f6
data/v1/v1-set/gpt-6-astra.json49,829f815192b7cdc
data/v1/v1-set/gpt-6-luna.json53,0287d224869e1b9
data/v1/v1-set/jev-1-13-0.json53,328820a39a1053b
data/v1/v1-set/open-jev-27b.json53,234e652c7c6e8f5
data/v1/v1-set/qwen3-8-flash-next.json53,321f1fd2a2d9de1
data/v1/v1-set.json172,357ba16d3d3e326
data/v3/clear-risk.json68,837c8d424f9b48e
data/v3/close-choices-b.json4,38407e5a208b079
data/v3/hype-cup.json7,56960257c61259f
data/v3/improve-run2/deepseek-v4.1-flash-on.json4,7352645f6300152
data/v3/improve-run2/gemini-4-argon-on.json4,34399fb2d088e4f
data/v3/improve.json653,8360ec66872635e
data/v3/late-entries/base-labs-elephant-1.json65,407e42e80870f33
data/v3/late-entries/base-labs-mammoth-1.json65,006c9abe20b8fc7
data/v3/late-entries/cerno-9b-v1-2.json65,70817c1858c32c3
data/v3/late-entries/drex-2-0-local.json64,703fb248ee9c820
data/v3/late-entries/drex-2-0.json60,7088d89c5f2a0f5
data/v3/late-entries/drex-max-1-1.json60,929dd9e5e3cfa2c
data/v3/late-entries/drex-max-v1.json65,33395bf49c446f1
data/v3/late-entries/gemini-decision-1.json60,240cc8016243afc
data/v3/late-entries/gliclass-v4-edge.json66,319244a17c20d9e
data/v3/late-entries/glide-0-4b.json65,5679c8ff2850113
data/v3/late-entries/glide-0-8b.json65,242e5e8441d0a44
data/v3/late-entries/glide-2b.json65,559f48e299b9de5
data/v3/late-entries/gpt-6-astra-high.json55,2165421347e1150
data/v3/late-entries/knowline-0-8b-gen1.json65,6969882e6619671
data/v3/late-entries/knowline-2b-gen1.json65,705e1ca4f94aa48
data/v3/late-entries/knowline-31b-gen1-5.json59,79487609031e975
data/v3/late-entries/knowline-4b-gen5-5.json65,4280359ebfd5657
data/v3/late-entries/liquid-d1-2b.json64,66045f14a311ac2
data/v3/late-entries/liquid-d1-omni-2b.json64,1523465a677c188
data/v3/late-entries/muse-ember-max.json56,9522bc45c168988
data/v3/late-entries/onejev1-6-0-8b.json65,5511f2c4a7acee3
data/v3/late-entries/reax-1-3.json64,95544b2d7b71212
data/v3/late-entries/sieve-31b-v3.json65,82642e518e10d03
data/v3/late-entries/startlux-decision-v1-1-0-8b-instant.json65,972475768947ac3
data/v3/late-entries/startlux-decision-v1-1-2b.json65,137f4590e37f5d8
data/v3/late-entries/startlux-decision-v1-1-4b-instant.json65,764c6cdb4690315
data/v3/late-entries/subq-snap-1-1.json60,242efa349c67370
data/v3/late-entries/turion-jev-31b.json65,608ba8b983f2cdf
data/v3/late-entries.json478,183b154625e1785
data/v3/league-1/celeris-1-decision.json62,592f7b079e58ff5
data/v3/league-1/cygnet-gemma-4-12b.json78,286d1139ce56b61
data/v3/league-1/decider-flash-v1-2.json77,2704d49ae7a0f66
data/v3/league-1/decider-v1-1-27b.json85,674c7b3b41772be
data/v3/league-1/decider-v1-2-27b.json76,333e132d10b5c22
data/v3/league-1/deepseek-v4-1-flash-off.json70,209f466f857f6dd
data/v3/league-1/fastino-glide-v1-2-local.json78,9217f1d62f4fbf4
data/v3/league-1/gnubg-weak.json50,7699441e0b91332
data/v3/league-1/gpt-6-luna.json62,6664491e97068aa
data/v3/league-1/jev-1-13-0.json62,650a07fc86bdfa5
data/v3/league-1/lfm2-5-24b-a2b-off.json75,667551cd4d0aa37
data/v3/league-1/lfm2-5-24b-a2b-on.json75,232700e1c638474
data/v3/league-1/mercury-decide.json62,3445a21be00d5be
data/v3/league-1/open-jev-27b-v1-1.json71,82844ddd0ea00a7
data/v3/league-1/qwen3-8-27b-off.json71,57000300be25612
data/v3/league-1/random.json50,4685786d6e2076c
data/v3/league-1/safest.json47,178a45a8401b456
data/v3/league-1/subq-snap-preview.json62,796759cb2a9e67b
data/v3/league-1/winnow-12b.json77,7442266370d7348
data/v3/league-1.json540,09555571aa38c3f
data/v3/league-2/claude-fable-5-1-off.json52,412bd869b125c6f
data/v3/league-2/claude-sonnet-5-5-high.json52,0355e88982669aa
data/v3/league-2/claude-sonnet-5-5-low.json52,249c79cc75ca654
data/v3/league-2/clef.json71,743bd4ceeb97067
data/v3/league-2/deepseek-v4-1-flash-1017-off.json63,0348883861182e0
data/v3/league-2/deepseek-v4-1-flash-1017-on.json61,745d0c653aaadea
data/v3/league-2/deepseek-v4-1-flash-on.json83,31698b305f41571
data/v3/league-2/fastino-glide.json64,74841e83004dcec
data/v3/league-2/gemini-3-8-flash-high.json52,248635a3001b16e
data/v3/league-2/gemini-3-8-flash-low.json52,5205dabd3f33e39
data/v3/league-2/gemma-4-31b-off.json73,941ddf0ceecea4c
data/v3/league-2/gemma-4-31b-on.json97,6077c00025378cb
data/v3/league-2/glm-5-3-on.json62,900b915ac644310
data/v3/league-2/gnubg-weak.json50,646c908023bc54f
data/v3/league-2/grok-4-7-off.json52,8030ad052b4a627
data/v3/league-2/jev-1-14-0.json62,24797974977a213
data/v3/league-2/liquid-d1-ultra.json62,9702c051e355a30
data/v3/league-2/qwen3-8-27b-on.json96,417b6f6319435d7
data/v3/league-2/qwen3-8-flash-off.json63,12940039a0d525a
data/v3/league-2/random-computed.json48,9646fc7fcd13808
data/v3/league-2/safest-computed.json41,56039cf2afa3343
data/v3/league-2.json473,6249df783574729
data/v3/league-3/claude-fable-5-1-on.json52,288993db45f9fdb
data/v3/league-3/claude-opus-5-5-off.json51,989d60d236580ae
data/v3/league-3/claude-opus-5-5-on.json52,0313d587ec5ff8d
data/v3/league-3/gemini-4-argon-off.json52,37021a813c47487
data/v3/league-3/gemini-4-argon-on.json52,44521a6e15e0328
data/v3/league-3/gnubg-weak.json50,893af7dc9240ddf
data/v3/league-3/gpt-6-1-sol-high.json52,244f5dd85d471b4
data/v3/league-3/gpt-6-1-sol-low.json52,4415167ddfc2c3e
data/v3/league-3/gpt-6-astra-off.json52,015781a72a1d5c1
data/v3/league-3/gpt-6-astra-on.json51,7178a6defe6e804
data/v3/league-3/grok-4-7-on.json52,25806617ceac3cd
data/v3/league-3/kimi-k3-1-pro-off.json52,047ff9ed68b3947
data/v3/league-3/random-computed.json49,0092cfa43644537
data/v3/league-3/safest-computed.json41,4376c0202943914
data/v3/league-3.json157,019bcb1ec28f683
data/v3/learning-pilot.json130,9012f49703ea059
data/v3/matches.json454,312965d9562b4f6
data/v3/nack-paired-late.json45,264cb39215c961b
data/v3/nack-paired.json221,13034083d72773c
data/v3/offline-hypothesis.json3,5834d6a2effe786
data/v3/repeat-checks.json37,8874760617f6d54
data/v3/shared.json2,369890f3ace3acc
data/v3/summary.json2300fc844f8e7fd

Data files are split per version, league and model, so each loads on a phone. Every number on the site is in one of them. The shared rules texts of the prompts are under decisions/contexts/ (2 files).

Analysis status

Swipe for more columns →

Every analysis in the plan, with its status and data file
AnalysisWhatStatusData fileWaits for the running leagues
A1Decision quality: error rate, best-play share, blunder rate, intervalsShared positions (main table)Analysis pendingdata/v3/shared.jsonno
A1Decision quality: error rate, best-play share, blunder rate, intervalsLeague decisions, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json, data/v3/league-N/<setup>.jsonno
A1Decision quality: error rate, best-play share, blunder rate, intervalsv1 setEarlier versiondata/v1/v1-set.jsonno
A2Effect of the information given to the model (notation only / consequences / history)Shared positions, 6 setups x 3 promptsAnalysis pendingnone yetno
A3Return on more reasoning (paired: the same model at a lower and a higher setting)Shared positionsAnalysis pendingdata/v3/shared.jsonno
A3Return on more reasoning (paired: the same model at a lower and a higher setting)v1 set: DeepSeek-V4.1-Flash off vs thinkingEarlier versiondata/v1/v1-set.json (reasoning), data/v1/compare.jsonno
A4Where models fail: phase, strategy tag, number of options, gap between the two best playsShared positionsAnalysis pendingdata/v3/shared.jsonno
A4Where models fail: phase, strategy tag, number of options, gap between the two best playsLeague decisions, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N/<setup>.json (phases, tags, options, gaps)no
A4Where models fail: phase, strategy tag, number of options, gap between the two best playsv1 setEarlier versiondata/v1/v1-set/<setup>.jsonno
A5Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasLeague decisions, per league (Confidence tab: cutoffs 0.90/0.99, surest quarter, AUROC best play and blunder warning, ECE, confident and badly wrong; held-out cutoff in Method)League 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (confidence)no
A5Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasShared positions (setups with probabilities)Analysis pendingdata/v3/shared.jsonno
A5Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasv1 set (setups with probabilities)Earlier versiondata/v1/v1-set/<setup>.json (quality.auroc, ece, brier)no
A5Reliability of confidence: AUROC, ECE, Brier, risk-coverage, letter-position biasRisk-coverage curves, letter-position biasAnalysis pendingnone yetno
A6Routing from a cheap model to a strong oneShared positionsAnalysis pendingnone yetno
A7Teams: can a team beat its best member? (oracle as upper bound only)Shared positions / per leagueAnalysis pendingnone yetno
A7Teams: can a team beat its best member? (oracle as upper bound only)v1 set, earlier reportEarlier versioncontent/archive (earlier page section, moved to Leagues)no
A8Repeatability: 200 positions asked 5 timesShared positionsAnalysis pendingnone yetno
A9Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timeShared positionsAnalysis pendingdata/v3/shared.jsonno
A9Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timeLeague decisions, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (quality.time_*, usd_per_1000, cost_coverage)no
A9Time and cost: $ per 1,000 decisions, median and P90 time, request vs retry timev1 set (time, tokens; cost not measured)Earlier versiondata/v1/v1-set.jsonno
A10League results: standings, W-D-L, games, gammons; safest-play share and hitting rollsStandings per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (standings)no
A10League results: standings, W-D-L, games, gammons; safest-play share and hitting rollsRisk calls in play (B6, per league)League 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (risk.main, risk.robust)no
A11Doubling cube (optional track)-Not measurednoneno
A12Known opening knowledge: opening table on/off, NackgammonNackgammon 500 + opening control 200Analysis pendingdata/v3/shared.json (set sizes only)no
A13Does the shared-position score predict league results?Shared positions + finished leaguesAnalysis pendingnone yetno
High 1When does high hurt? Decisions that high fixes and breaks (counts and loss sizes)Shared positionsAnalysis pendingnone yetno
High 1When does high hurt? Decisions that high fixes and breaks (counts and loss sizes)v1 set: DeepSeek-V4.1-Flash off vs thinkingEarlier versiondata/v1/compare.json (fixed, broken, fixed_loss, broken_loss)no
High 2When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder rateShared positionsAnalysis pendingnone yetno
High 2When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder rateLeague decisions, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (quality.p95_loss, worst_mean, blunder_share)no
High 2When does high hurt? Risk in the worst decisions: P95 loss, mean of the worst 5%, blunder ratev1 setEarlier versiondata/v1/v1-set.jsonno
High 3When does high hurt? Confident and badly wrongLeague decisions, setups with probabilities, per leagueLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (quality.confident_badly_wrong)no
High 3When does high hurt? Confident and badly wrongv1 setEarlier versiondata/v1/v1-set.jsonno
High 3When does high hurt? Confident and badly wrongShared positionsAnalysis pendingnone yetno
High 4When does high hurt? Going over a time budgetLeague decisions: share of request times over the budgetLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (quality.over_budget)no
High 5When does high hurt? High only when needed (switching rule on held-out positions)Shared positionsAnalysis pendingnone yetno
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02League decisions, per league, both marginsLeague 1: Published · League 2: Published · League 3: Publisheddata/v3/league-N.json (risk.main, risk.robust, references), data/v3/league-N/<setup>.json (risk.*.phases)no
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Risk map, error rate by risk-call category, stake chart, risk call cards (league winners)League 1: Published · League 2: Published · League 3: Publishedsame; share/risk-card-league-N-*.pngno
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Every decision marks (position class, stake, risk call, best safe play)League 1: Published · League 2: Published · League 3: Publisheddecisions/<league>/<setup>/risk.json.gz and page filesno
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02v1 setEarlier versiondata/v1/v1-set.json (risk)no
B6Was it the right risk? Risk judgement, risks found, right safety, good / missed / needless / wrong risks, error rate by risk-call category, stake bands, risk size, confidence on the call, reference rows, at margins 0.001 and 0.02Shared positionsAnalysis pendingnone yetno
LLearning from experience-Not measurednoneno

Versions

v3
Prompt bg-v2, plays in a seeded random order, invalid answers not retried. Leagues 1 to 3 and the shared positions.
v1
Earlier version Earlier prompt; plays listed in the runner's order. The v1 set of 1,000 positions answered by the v1 setups. Never in a v3 table or chart.
Smoke tests
50 positions outside the shared sets, v3 prompt; only on New models: first look.

How to recompute

  1. Snapshot the league database: python3 tools/site/snapshot.py v3.db STATE (one read transaction; the prompts each model was sent go to STATE/prompts.db; the Nackgammon knockout's matches, from knockout-nack.db next to it, to STATE/knockout.db).
  2. Score every legal play of every decision with GNUbg 1.08 at 2-ply: tools/site/score.sh STATE/snap.db STATE/plays.
  3. Build and check the site: python3 tools/site/build.py --state STATE --out OUT. The build fails, and nothing is deployed, if a number does not trace to a data file, counts do not add up, leagues or versions mix, a published finding no longer holds, or a file contains a private address or credential.

The scripts are in the jevgammon repository under tools/site/; the fixed thresholds are in thresholds.json.