Experiments
Nackgammon
Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.
Nackgammon paired test
Does a model still play well when the starting position changes?
Class: by the maker's total parameters, or by list price when the size is undisclosed *
Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3.
Swipe for more columns →
| Setup (group) | Pairs | Standard PR | Nackgammon PR | Drop (Nackgammon − standard) | 97.5% interval | Label |
|---|---|---|---|---|---|---|
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 500 | 26.33 | 27.93 | +1.60 | [−5.33, +8.30] | unclear |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 500 | 25.73 | 28.80 | +3.07 | [−3.84, +9.81] | unclear |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 500 | 25.30 | 31.47 | +6.17 | [−2.06, +13.95] | unclear |
| Jev 1.13Closed · EfficientLeague 1 group | 500 | 24.67 | 27.44 | +2.77 | [−5.20, +10.44] | unclear |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 500 | 29.56 | 35.44 | +5.87 | [−4.27, +15.07] | unclear |
| celeris-1-decisionClosed · EfficientLeague 1 group | 500 | 26.86 | 30.93 | +4.08 | [−4.58, +12.64] | unclear |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 500 | 31.27 | 32.08 | +0.81 | [−7.98, +9.24] | unclear |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 500 | 28.46 | 28.66 | +0.20 | [−7.64, +7.43] | unclear |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 500 | 31.25 | 32.87 | +1.62 | [−7.07, +10.07] | unclear |
| Mercury DecideClosed · EfficientLeague 1 group | 500 | 31.34 | 34.08 | +2.73 | [−5.94, +10.85] | unclear |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 500 | 31.93 | 34.61 | +2.68 | [−6.26, +11.16] | unclear |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 500 | 31.72 | 29.03 | −2.68 | [−11.03, +5.25] | unclear |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 500 | 40.23 | 48.01 | +7.78 | [−2.16, +18.39] | unclear |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 500 | 47.07 | 43.99 | −3.08 | [−13.33, +6.76] | unclear |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 500 | 11.14 | 9.12 | −2.03 | [−5.57, +1.38] | unclear |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 500 | 12.21 | 10.75 | −1.46 | [−5.01, +2.05] | unclear |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 500 | 16.45 | 15.30 | −1.15 | [−5.89, +3.32] | unclear |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 500 | 16.71 | 15.41 | −1.30 | [−5.48, +2.90] | unclear |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 500 | 17.97 | 16.84 | −1.13 | [−6.19, +4.39] | unclear |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 500 | 23.35 | 29.29 | +5.94 | [−2.48, +13.87] | unclear |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 500 | 26.89 | 31.65 | +4.76 | [−3.56, +12.53] | unclear |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 500 | 37.78 | 42.54 | +4.77 | [−4.14, +13.49] | unclear |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 500 | 46.44 | 50.87 | +4.43 | [−6.53, +15.38] | unclear |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 500 | 9.64 | 10.80 | +1.16 | [−2.17, +4.40] | kept |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 500 | 10.86 | 8.88 | −1.99 | [−5.52, +1.47] | unclear |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 500 | 15.68 | 16.53 | +0.85 | [−3.74, +5.35] | unclear |
| GPT-6.1 Terrano group (exploratory) | 500 | 13.18 | 16.22 | +3.04 | [−0.94, +7.39] | unclear |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 500 | 14.71 | 17.42 | +2.71 | [−2.06, +7.62] | unclear |
| GNUbg weakyardstick | 500 | 25.35 | 20.78 | −4.57 | [−10.32, +1.34] | unclear |
| Randomyardstick | 500 | 85.16 | 81.43 | −3.73 | [−11.43, +3.74] | unclear |
Labels, from the final 97.5% interval (Method §10, outcome label (c)): kept = the interval lies within ±5 PR; dropped = the data file's label “worse”: the interval lies above 0 (lower end > 0) and the setup is not kept; better = the interval lies below 0 and the setup is not kept; unclear = anything else.
Unclear means the test can’t tell, not that the scores are equal. 4 setups were added after the results were read; the primary results are shown with and without them. *
Random play's own drop is −4.6 PR on the main 200 pairs: the Nackgammon positions were slightly easier for random play, so a drop mixes how a model copes with the new start with how hard the positions are; the test cannot separate the two.
One setting per model: a model asked at several settings shows the one with its highest stored risk judgement in league play, on an exact tie the first in league order (chosen from league results, never from this test); All settings shows every setup asked (2 more rows). Decision / Chat is the model type; after each name is its model class (the Class row filters every table and the chart) and under it the paired test's group, the setups from one original league, so a chat model can sit in the League 1 group. Every table further down shows every setup.
Late entries / exploratory
Added after the results were known; exploratory; not in any group comparison (§10.14(i)). Late entry, exploratory (Method §10.14(i), 2026-11-07): added after this test's results had been computed and read; no group, no primary comparison, M unchanged; same rules and scoring as every setup.
Swipe for more columns →
| Setup | Pairs | Standard PR | Nackgammon PR | Change (Nackgammon − standard) | 97.5% interval | Label |
|---|---|---|---|---|---|---|
| SubQ Snap 1.1late entry, exploratory · its answers equal SubQ Snap Preview's answers of 27 October on all 1,000 positions (the same letters and probability vectors) | 500 | 31.25 | 32.87 | +1.62 | [−7.07, +10.07] | unclear |
| Drex Max v1.1late entry, exploratory | 500 | 28.40 | 29.09 | +0.69 | [−6.03, +7.48] | unclear |
| Drex 2.0late entry, exploratory | 500 | 29.39 | 31.35 | +1.95 | [−6.39, +9.95] | unclear |
| Elephant-1late entry, exploratory | 500 | 25.23 | 33.89 | +8.66 | [+0.42, +16.02] | dropped |
| Mammoth-1late entry, exploratory | 500 | 22.97 | 24.81 | +1.84 | [−5.72, +8.72] | unclear |
| Drex Maxlate entry, exploratory | 500 | 26.21 | 27.92 | +1.71 | [−6.67, +9.37] | unclear |
| Drex 2.0locallate entry, exploratory | 500 | 29.19 | 30.79 | +1.60 | [−6.90, +9.75] | unclear |
| Randomyardstick | 500 | 85.16 | 81.43 | −3.73 | [−11.43, +3.74] | unclear |
Labels, from the final 97.5% interval (Method §10, outcome label (c)): kept = the interval lies within ±5 PR; dropped = the data file's label “worse”: the interval lies above 0 (lower end > 0) and the setup is not kept; better = the interval lies below 0 and the setup is not kept; unclear = anything else.
REAX 1.3: not run (pending approval).
Data: nack-paired-late.json
Swipe for more columns →
| Setup (group) | Standard PR, main 200 pairs | Nackgammon PR, main 200 pairs | Drop, main 200 pairs | First look 97.5% interval | Random's drop, main 200 pairs | Extended to 500 pairs | Pairs, final look | Standard PR, final | Nackgammon PR, final | Drop, final | Final 97.5% interval | Random's drop, same pairs as the final look | Label (final interval) | Positions without a valid choice, main 200 pairs (standard / Nackgammon) | Positions without a valid choice, final pairs (standard / Nackgammon) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 22.74 | 24.91 | +2.18 | [−7.28, +11.93] | −4.56 | yes | 500 | 26.33 | 27.93 | +1.60 | [−5.33, +8.30] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 22.93 | 23.38 | +0.45 | [−8.03, +8.77] | −4.56 | yes | 500 | 25.73 | 28.80 | +3.07 | [−3.84, +9.81] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 24.25 | 28.27 | +4.01 | [−11.37, +17.17] | −4.56 | yes | 500 | 25.30 | 31.47 | +6.17 | [−2.06, +13.95] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Jev 1.13Closed · EfficientLeague 1 group | 26.53 | 24.17 | −2.37 | [−17.72, +9.97] | −4.56 | yes | 500 | 24.67 | 27.44 | +2.77 | [−5.20, +10.44] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 28.27 | 34.11 | +5.84 | [−10.57, +20.46] | −4.56 | yes | 500 | 29.56 | 35.44 | +5.87 | [−4.27, +15.07] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| celeris-1-decisionClosed · EfficientLeague 1 group | 29.71 | 28.47 | −1.24 | [−18.27, +14.59] | −4.56 | yes | 500 | 26.86 | 30.93 | +4.08 | [−4.58, +12.64] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 30.42 | 28.64 | −1.78 | [−17.30, +11.94] | −4.56 | yes | 500 | 31.27 | 32.08 | +0.81 | [−7.98, +9.24] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 31.33 | 24.48 | −6.85 | [−22.36, +5.43] | −4.56 | yes | 500 | 28.46 | 28.66 | +0.20 | [−7.64, +7.43] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 31.91 | 29.66 | −2.25 | [−18.09, +11.88] | −4.56 | yes | 500 | 31.25 | 32.87 | +1.62 | [−7.07, +10.07] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Mercury DecideClosed · EfficientLeague 1 group | 32.74 | 28.46 | −4.28 | [−20.11, +8.75] | −4.56 | yes | 500 | 31.34 | 34.08 | +2.73 | [−5.94, +10.85] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 34.33 | 33.43 | −0.89 | [−17.96, +14.97] | −4.56 | yes | 500 | 31.93 | 34.61 | +2.68 | [−6.26, +11.16] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 34.68 | 22.80 | −11.87 | [−27.93, +1.03] | −4.56 | yes | 500 | 31.72 | 29.03 | −2.68 | [−11.03, +5.25] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 39.77 | 42.86 | +3.09 | [−12.09, +20.11] | −4.56 | yes | 500 | 40.23 | 48.01 | +7.78 | [−2.16, +18.39] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 52.08 | 40.96 | −11.12 | [−29.52, +6.79] | −4.56 | yes | 500 | 47.07 | 43.99 | −3.08 | [−13.33, +6.76] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 9.88 | 8.43 | −1.45 | [−5.57, +2.68] | −4.56 | yes | 500 | 11.14 | 9.12 | −2.03 | [−5.57, +1.38] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 11.15 | 9.90 | −1.26 | [−5.70, +3.31] | −4.56 | yes | 500 | 12.21 | 10.75 | −1.46 | [−5.01, +2.05] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 13.96 | 14.50 | +0.54 | [−5.04, +6.02] | −4.56 | yes | 500 | 16.45 | 15.30 | −1.15 | [−5.89, +3.32] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 15.92 | 11.81 | −4.11 | [−9.72, +1.04] | −4.56 | yes | 500 | 16.71 | 15.41 | −1.30 | [−5.48, +2.90] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 18.35 | 13.38 | −4.97 | [−11.81, +1.78] | −4.56 | yes | 500 | 17.97 | 16.84 | −1.13 | [−6.19, +4.39] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 23.91 | 27.90 | +3.98 | [−11.72, +17.96] | −4.56 | yes | 500 | 23.35 | 29.29 | +5.94 | [−2.48, +13.87] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 28.13 | 28.43 | +0.30 | [−14.48, +12.12] | −4.56 | yes | 500 | 26.89 | 31.65 | +4.76 | [−3.56, +12.53] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 38.20 | 37.55 | −0.65 | [−15.17, +14.42] | −4.56 | yes | 500 | 37.78 | 42.54 | +4.77 | [−4.14, +13.49] | −3.73 | unclear | 22 / 16 | 55 / 52 |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 50.75 | 53.98 | +3.23 | [−16.93, +22.07] | −4.56 | yes | 500 | 46.44 | 50.87 | +4.43 | [−6.53, +15.38] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 9.13 | 9.51 | +0.38 | [−4.64, +5.07] | −4.56 | yes | 500 | 9.64 | 10.80 | +1.16 | [−2.17, +4.40] | −3.73 | kept | 0 / 0 | 0 / 0 |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 9.66 | 7.30 | −2.37 | [−6.69, +1.77] | −4.56 | yes | 500 | 10.86 | 8.88 | −1.99 | [−5.52, +1.47] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 15.20 | 14.53 | −0.67 | [−6.58, +5.07] | −4.56 | yes | 500 | 15.68 | 16.53 | +0.85 | [−3.74, +5.35] | −3.73 | unclear | 6 / 9 | 16 / 18 |
| GPT-6.1 Terrano group (exploratory) | 13.52 | 14.40 | +0.89 | [−5.73, +8.59] | −4.56 | yes | 500 | 13.18 | 16.22 | +3.04 | [−0.94, +7.39] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 13.79 | 15.71 | +1.92 | [−4.33, +8.42] | −4.56 | yes | 500 | 14.71 | 17.42 | +2.71 | [−2.06, +7.62] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| GNUbg weakyardstick | 27.69 | 24.13 | −3.56 | [−13.92, +7.29] | −4.56 | yes | 500 | 25.35 | 20.78 | −4.57 | [−10.32, +1.34] | −3.73 | unclear | 0 / 0 | 0 / 0 |
| Randomyardstick | 87.45 | 82.89 | −4.56 | [−18.04, +7.86] | −4.56 | no | 500 | 85.16 | 81.43 | −3.73 | [−11.43, +3.74] | −3.73 | unclear | 0 / 0 | 0 / 0 |
LFM2.5-24B-A2Bon: 107 positions recorded without a valid choice (timeouts or missing choices), 38 in the main 200 pairs and 69 in the reserve; on its final 500 pairs 55 standard / 52 Nackgammon; Grok 4.7high: 34 positions recorded without a valid choice (timeouts or missing choices), 15 in the main 200 pairs and 19 in the reserve; on its final 500 pairs 16 standard / 18 Nackgammon. They are not answers: each is scored by the primary rule (the position's Random error), with both sensitivity checks below. Every other asked setup gave a valid choice on every recorded position.
Method spec 2026-10-24 §10 (groups: §10.14)
Scope. These results hold for these rosters only; each group is the setups in the run from that original league.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Additions. Added under §10.14(g) after this test's results had been computed and read: d1-ultra and GLiDE (League 1 group); Claude Fable 5.1high and Grok 4.7high (League 3 group). They follow the Cup roster, not the results. The primary comparisons are shown both ways: with the additions (the primary result, §10.14(b)) and with the original members only (22 setups, §10.14(d)).
Nackgammon has the rules of backgammon; only the starting position differs. Each setup was asked 200 Nackgammon positions and 200 standard positions matched to them by phase, position class, number of legal plays and the gap between the two best plays. Error rate (PR) = 500 × mean GNUbg 2-ply equity loss. Drop = Nackgammon PR − standard PR on the same pairs; positive = worse on Nackgammon. A setup whose first-look 97.5% interval was wide and contained 0 was asked the 300 reserve pairs too (500 pairs). Every recorded position is scored: an invalid, timed-out or missing answer scores the position's Random error, with the sensitivity checks below. Label from the final 97.5% interval: kept (within ±5 PR), dropped (the data's “worse”), better or unclear. “Unclear” means the interval is too wide to decide; it is not evidence of equivalence.
A larger drop is consistent with relying on familiar positions but is not proof: Nackgammon positions also differ in structure. “Kept” is a valid result.
The paired test's standard-side prompt differs from the league prompt (no opening table; the starting setup on its own line), so the standard error rates here are not compared with league error rates.
Random's own drop: −4.56 PR on the main 200 pairs, −3.73 on all 500 (Random scores slightly better on the Nackgammon positions). It is shown next to every result. Random is computed exactly from GNUbg's 2-ply values (mean loss over all legal plays): it is never asked and is not extended by the stopping rule; it is reported on the main 200 pairs and on all 500.
No group (exploratory), in no group comparison: GPT-6.1 Terra (a setup that had played in no league before this test, §10.14(b)); Gemini 4 Argonlow, added by Can's decision of 2026-10-28 instead of Gemini 4 Argonhigh, not because of the Cup roster (§10.14(h)).
Not a ranking: rows are in fixed order (group, then standard PR) and no table can be sorted.
Standard against Nackgammon error rate
OpenAIGoogleAnthropicxAIDeepSeekLiquid AIAlibabaInceptionFastinoPerplexitySubquadraticCelerisEldanRingTypeSafeZefan CaiBlockbrain Labs GNUbg weak, Random (programs) Above the dashed diagonal: higher error on Nackgammon.
Primary comparisons
Swipe for more columns →
| Comparison (A − B) | A: standard PR | A: Nackgammon PR | A: drop | B: standard PR | B: Nackgammon PR | B: drop | Difference of drops | 99% interval | Random's drop, same pairs |
|---|---|---|---|---|---|---|---|---|---|
| League 1 group − League 2 group | 31.55 | 29.61 | −1.93 | 23.36 | 22.88 | −0.49 | −1.44 | [−8.85, +4.96] | −4.56 |
| League 1 group − League 3 group | 31.55 | 29.61 | −1.93 | 11.33 | 10.45 | −0.89 | −1.04 | [−14.19, +10.26] | −4.56 |
| League 2 group − League 3 group | 23.36 | 22.88 | −0.49 | 11.33 | 10.45 | −0.89 | +0.40 | [−6.68, +6.96] | −4.56 |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | 38.20 | 37.55 | −0.65 | 50.75 | 53.98 | +3.23 | −3.88 | [−26.98, +21.76] | −4.56 |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | 9.88 | 8.43 | −1.45 | 11.15 | 9.90 | −1.26 | −0.19 | [−3.00, +2.63] | −4.56 |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Swipe for more columns →
| Comparison (A − B) | A: standard PR | A: Nackgammon PR | A: drop | B: standard PR | B: Nackgammon PR | B: drop | Difference of drops | 99% interval | Random's drop, same pairs |
|---|---|---|---|---|---|---|---|---|---|
| League 1 group − League 2 group | 31.55 | 29.61 | −1.93 | 24.90 | 23.70 | −1.20 | −0.73 | [−8.24, +5.88] | −4.56 |
| League 1 group − League 3 group | 31.55 | 29.61 | −1.93 | 11.45 | 10.64 | −0.81 | −1.12 | [−14.14, +10.21] | −4.56 |
| League 2 group − League 3 group | 24.90 | 23.70 | −1.20 | 11.45 | 10.64 | −0.81 | −0.39 | [−7.62, +6.32] | −4.56 |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | 52.06 | 44.97 | −7.10 | 50.75 | 53.98 | +3.23 | −10.33 | [−38.92, +19.74] | −4.56 |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | 9.88 | 8.43 | −1.45 | 11.15 | 9.90 | −1.26 | −0.19 | [−3.00, +2.63] | −4.56 |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Swipe for more columns →
| Comparison (A − B) | A: standard PR | A: Nackgammon PR | A: drop | B: standard PR | B: Nackgammon PR | B: drop | Difference of drops | 99% interval | Random's drop, same pairs |
|---|---|---|---|---|---|---|---|---|---|
| League 1 group − League 2 group | 31.55 | 29.61 | −1.93 | 22.52 | 22.36 | −0.16 | −1.77 | [−9.22, +4.92] | −4.56 |
| League 1 group − League 3 group | 31.55 | 29.61 | −1.93 | 11.58 | 10.64 | −0.94 | −0.99 | [−14.18, +10.32] | −4.56 |
| League 2 group − League 3 group | 22.52 | 22.36 | −0.16 | 11.58 | 10.64 | −0.94 | +0.78 | [−6.32, +7.42] | −4.56 |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | 30.59 | 32.91 | +2.32 | 50.75 | 53.98 | +3.23 | −0.91 | [−24.13, +25.20] | −4.56 |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | 9.88 | 8.43 | −1.45 | 11.15 | 9.90 | −1.26 | −0.19 | [−3.00, +2.63] | −4.56 |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Swipe for more columns →
| Comparison (A − B) | A: standard PR | A: Nackgammon PR | A: drop | B: standard PR | B: Nackgammon PR | B: drop | Difference of drops | 99% interval | Random's drop, same pairs |
|---|---|---|---|---|---|---|---|---|---|
| League 1 group − League 2 group | 31.55 | 29.61 | −1.93 | 23.36 | 22.88 | −0.49 | −1.44 | [−8.85, +4.96] | −4.56 |
| League 1 group − League 3 group | 31.55 | 29.61 | −1.93 | 11.33 | 10.45 | −0.89 | −1.04 | [−14.19, +10.26] | −4.56 |
| League 2 group − League 3 group | 23.36 | 22.88 | −0.49 | 11.33 | 10.45 | −0.89 | +0.40 | [−6.68, +6.96] | −4.56 |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | 38.20 | 37.55 | −0.65 | 50.75 | 53.98 | +3.23 | −3.88 | [−26.98, +21.76] | −4.56 |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | 9.88 | 8.43 | −1.45 | 11.15 | 9.90 | −1.26 | −0.19 | [−3.00, +2.63] | −4.56 |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Scope. These results hold for these rosters only; each group is the setups in the run from that original league.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Each difference is A's drop − B's drop (a group's drop = the mean of its setups' drops), main 200 pairs, paired bootstrap. Five comparisons, so each interval is at 99% (Bonferroni, 1 − 0.05/5). Both sides' standard and Nackgammon PR are shown next to every difference. Shown both ways: with the §10.14(g) additions (the primary result) and with the original members only.
Original members only
Swipe for more columns →
| Comparison (A − B) | A: standard PR | A: Nackgammon PR | A: drop | B: standard PR | B: Nackgammon PR | B: drop | Difference of drops | 99% interval | Random's drop, same pairs |
|---|---|---|---|---|---|---|---|---|---|
| League 1 group − League 2 group | 32.88 | 30.25 | −2.63 | 23.36 | 22.88 | −0.49 | −2.14 | [−10.07, +4.88] | −4.56 |
| League 1 group − League 3 group | 32.88 | 30.25 | −2.63 | 9.66 | 7.30 | −2.37 | −0.26 | [−14.64, +12.04] | −4.56 |
| League 2 group − League 3 group | 23.36 | 22.88 | −0.49 | 9.66 | 7.30 | −2.37 | +1.88 | [−5.73, +9.33] | −4.56 |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | 38.20 | 37.55 | −0.65 | 50.75 | 53.98 | +3.23 | −3.88 | [−26.98, +21.76] | −4.56 |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | 9.88 | 8.43 | −1.45 | 11.15 | 9.90 | −1.26 | −0.19 | [−3.00, +2.63] | −4.56 |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
Original members only: the League 3 group has 1 setup (GPT-6 Astramedium); comparisons 2 and 3 rest on it.
Swipe for more columns →
| Comparison (A − B) | A: standard PR | A: Nackgammon PR | A: drop | B: standard PR | B: Nackgammon PR | B: drop | Difference of drops | 99% interval | Random's drop, same pairs |
|---|---|---|---|---|---|---|---|---|---|
| League 1 group − League 2 group | 32.88 | 30.25 | −2.63 | 24.90 | 23.70 | −1.20 | −1.43 | [−9.56, +5.66] | −4.56 |
| League 1 group − League 3 group | 32.88 | 30.25 | −2.63 | 9.66 | 7.30 | −2.37 | −0.26 | [−14.64, +12.04] | −4.56 |
| League 2 group − League 3 group | 24.90 | 23.70 | −1.20 | 9.66 | 7.30 | −2.37 | +1.17 | [−6.49, +8.72] | −4.56 |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | 52.06 | 44.97 | −7.10 | 50.75 | 53.98 | +3.23 | −10.33 | [−38.92, +19.74] | −4.56 |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | 9.88 | 8.43 | −1.45 | 11.15 | 9.90 | −1.26 | −0.19 | [−3.00, +2.63] | −4.56 |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
Original members only: the League 3 group has 1 setup (GPT-6 Astramedium); comparisons 2 and 3 rest on it.
Swipe for more columns →
| Comparison (A − B) | A: standard PR | A: Nackgammon PR | A: drop | B: standard PR | B: Nackgammon PR | B: drop | Difference of drops | 99% interval | Random's drop, same pairs |
|---|---|---|---|---|---|---|---|---|---|
| League 1 group − League 2 group | 32.88 | 30.25 | −2.63 | 22.52 | 22.36 | −0.16 | −2.47 | [−10.57, +4.72] | −4.56 |
| League 1 group − League 3 group | 32.88 | 30.25 | −2.63 | 9.66 | 7.30 | −2.37 | −0.26 | [−14.64, +12.04] | −4.56 |
| League 2 group − League 3 group | 22.52 | 22.36 | −0.16 | 9.66 | 7.30 | −2.37 | +2.21 | [−5.45, +9.55] | −4.56 |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | 30.59 | 32.91 | +2.32 | 50.75 | 53.98 | +3.23 | −0.91 | [−24.13, +25.20] | −4.56 |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | 9.88 | 8.43 | −1.45 | 11.15 | 9.90 | −1.26 | −0.19 | [−3.00, +2.63] | −4.56 |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
Original members only: the League 3 group has 1 setup (GPT-6 Astramedium); comparisons 2 and 3 rest on it.
Swipe for more columns →
| Comparison (A − B) | A: standard PR | A: Nackgammon PR | A: drop | B: standard PR | B: Nackgammon PR | B: drop | Difference of drops | 99% interval | Random's drop, same pairs |
|---|---|---|---|---|---|---|---|---|---|
| League 1 group − League 2 group | 32.88 | 30.25 | −2.63 | 23.36 | 22.88 | −0.49 | −2.14 | [−10.07, +4.88] | −4.56 |
| League 1 group − League 3 group | 32.88 | 30.25 | −2.63 | 9.66 | 7.30 | −2.37 | −0.26 | [−14.64, +12.04] | −4.56 |
| League 2 group − League 3 group | 23.36 | 22.88 | −0.49 | 9.66 | 7.30 | −2.37 | +1.88 | [−5.73, +9.33] | −4.56 |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | 38.20 | 37.55 | −0.65 | 50.75 | 53.98 | +3.23 | −3.88 | [−26.98, +21.76] | −4.56 |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | 9.88 | 8.43 | −1.45 | 11.15 | 9.90 | −1.26 | −0.19 | [−3.00, +2.63] | −4.56 |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
Original members only: the League 3 group has 1 setup (GPT-6 Astramedium); comparisons 2 and 3 rest on it.
Scope. These results hold for these rosters only; each group is the setups in the run from that original league.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Invalid answers, output limit and sensitivity
Swipe for more columns →
| Setup (group) | Invalid answers, standard / Nackgammon (main 200 pairs) | Drop, primary rule | 97.5% interval | Drop, invalid = worst play | 97.5% interval | Pairs, invalid pairs left out | Drop, invalid pairs left out | 97.5% interval | Pairs, cut-off pairs left out | Drop, cut-off pairs left out | 97.5% interval | Cut off at the output limit, main 200 pairs (standard / Nackgammon) | Cut off at the output limit, all recorded positions (standard / Nackgammon) | Past 32,768 output tokens, no limit sent, main 200 pairs (standard / Nackgammon) | Past 32,768 output tokens, no limit sent, all recorded positions (standard / Nackgammon) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 0 / 0 | +2.18 | [−7.28, +11.93] | +2.18 | [−7.28, +11.93] | 200 | +2.18 | [−7.28, +11.93] | 200 | +2.18 | [−7.28, +11.93] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 0 / 0 | +0.45 | [−8.03, +8.77] | +0.45 | [−8.03, +8.77] | 200 | +0.45 | [−8.03, +8.77] | 200 | +0.45 | [−8.03, +8.77] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 0 / 0 | +4.01 | [−11.37, +17.17] | +4.01 | [−11.37, +17.17] | 200 | +4.01 | [−11.37, +17.17] | 200 | +4.01 | [−11.37, +17.17] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Jev 1.13Closed · EfficientLeague 1 group | 0 / 0 | −2.37 | [−17.72, +9.97] | −2.37 | [−17.72, +9.97] | 200 | −2.37 | [−17.72, +9.97] | 200 | −2.37 | [−17.72, +9.97] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 0 / 0 | +5.84 | [−10.57, +20.46] | +5.84 | [−10.57, +20.46] | 200 | +5.84 | [−10.57, +20.46] | 200 | +5.84 | [−10.57, +20.46] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| celeris-1-decisionClosed · EfficientLeague 1 group | 0 / 0 | −1.24 | [−18.27, +14.59] | −1.24 | [−18.27, +14.59] | 200 | −1.24 | [−18.27, +14.59] | 200 | −1.24 | [−18.27, +14.59] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 0 / 0 | −1.78 | [−17.30, +11.94] | −1.78 | [−17.30, +11.94] | 200 | −1.78 | [−17.30, +11.94] | 200 | −1.78 | [−17.30, +11.94] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 0 / 0 | −6.85 | [−22.36, +5.43] | −6.85 | [−22.36, +5.43] | 200 | −6.85 | [−22.36, +5.43] | 200 | −6.85 | [−22.36, +5.43] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 0 / 0 | −2.25 | [−18.09, +11.88] | −2.25 | [−18.09, +11.88] | 200 | −2.25 | [−18.09, +11.88] | 200 | −2.25 | [−18.09, +11.88] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Mercury DecideClosed · EfficientLeague 1 group | 0 / 0 | −4.28 | [−20.11, +8.75] | −4.28 | [−20.11, +8.75] | 200 | −4.28 | [−20.11, +8.75] | 200 | −4.28 | [−20.11, +8.75] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 0 / 0 | −0.89 | [−17.96, +14.97] | −0.89 | [−17.96, +14.97] | 200 | −0.89 | [−17.96, +14.97] | 200 | −0.89 | [−17.96, +14.97] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 0 / 0 | −11.87 | [−27.93, +1.03] | −11.87 | [−27.93, +1.03] | 200 | −11.87 | [−27.93, +1.03] | 200 | −11.87 | [−27.93, +1.03] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 0 / 0 | +3.09 | [−12.09, +20.11] | +3.09 | [−12.09, +20.11] | 200 | +3.09 | [−12.09, +20.11] | 200 | +3.09 | [−12.09, +20.11] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 0 / 0 | −11.12 | [−29.52, +6.79] | −11.12 | [−29.52, +6.79] | 200 | −11.12 | [−29.52, +6.79] | 200 | −11.12 | [−29.52, +6.79] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 0 / 0 | −1.45 | [−5.57, +2.68] | −1.45 | [−5.57, +2.68] | 200 | −1.45 | [−5.57, +2.68] | 200 | −1.45 | [−5.57, +2.68] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 0 / 0 | −1.26 | [−5.70, +3.31] | −1.26 | [−5.70, +3.31] | 200 | −1.26 | [−5.70, +3.31] | 200 | −1.26 | [−5.70, +3.31] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 0 / 0 | +0.54 | [−5.04, +6.02] | +0.54 | [−5.04, +6.02] | 200 | +0.54 | [−5.04, +6.02] | 200 | +0.54 | [−5.04, +6.02] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 0 / 0 | −4.11 | [−9.72, +1.04] | −4.11 | [−9.72, +1.04] | 200 | −4.11 | [−9.72, +1.04] | 200 | −4.11 | [−9.72, +1.04] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 0 / 0 | −4.97 | [−11.81, +1.78] | −4.97 | [−11.81, +1.78] | 200 | −4.97 | [−11.81, +1.78] | 200 | −4.97 | [−11.81, +1.78] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 0 / 0 | +3.98 | [−11.72, +17.96] | +3.98 | [−11.72, +17.96] | 200 | +3.98 | [−11.72, +17.96] | 200 | +3.98 | [−11.72, +17.96] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 0 / 0 | +0.30 | [−14.48, +12.12] | +0.30 | [−14.48, +12.12] | 200 | +0.30 | [−14.48, +12.12] | 200 | +0.30 | [−14.48, +12.12] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 22 / 16 | −0.65 | [−15.17, +14.42] | −7.10 | [−27.35, +13.34] | 166 | +2.32 | [−12.28, +18.07] | 200 | −0.65 | [−15.17, +14.42] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 0 / 0 | +3.23 | [−16.93, +22.07] | +3.23 | [−16.93, +22.07] | 200 | +3.23 | [−16.93, +22.07] | 200 | +3.23 | [−16.93, +22.07] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 0 / 0 | +0.38 | [−4.64, +5.07] | +0.38 | [−4.64, +5.07] | 200 | +0.38 | [−4.64, +5.07] | 200 | +0.38 | [−4.64, +5.07] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 0 / 0 | −2.37 | [−6.69, +1.77] | −2.37 | [−6.69, +1.77] | 200 | −2.37 | [−6.69, +1.77] | 200 | −2.37 | [−6.69, +1.77] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 6 / 9 | −0.67 | [−6.58, +5.07] | −0.46 | [−6.52, +5.59] | 186 | −0.84 | [−7.17, +5.33] | 200 | −0.67 | [−6.58, +5.07] | 0 / 0 | 0 / 0 | 5 / 2 | 7 / 10 |
| GPT-6.1 Terrano group (exploratory) | 0 / 0 | +0.89 | [−5.73, +8.59] | +0.89 | [−5.73, +8.59] | 200 | +0.89 | [−5.73, +8.59] | 200 | +0.89 | [−5.73, +8.59] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 0 / 0 | +1.92 | [−4.33, +8.42] | +1.92 | [−4.33, +8.42] | 200 | +1.92 | [−4.33, +8.42] | 200 | +1.92 | [−4.33, +8.42] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| GNUbg weakyardstick | 0 / 0 | −3.56 | [−13.92, +7.29] | −3.56 | [−13.92, +7.29] | 200 | −3.56 | [−13.92, +7.29] | 200 | −3.56 | [−13.92, +7.29] | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| Randomyardstick | 0 / 0 | −4.56 | [−18.04, +7.86] | −4.56 | [−18.04, +7.86] | 200 | −4.56 | [−18.04, +7.86] | 200 | −4.56 | [−18.04, +7.86] | – | – | – | – |
Output limit (§10.14(h)): an answer cut off at 32,768 output tokens counts as invalid, with its own sensitivity check (cut-off pairs left out). No recorded answer was cut off at the limit. Past 32,768 output tokens from a run that sent no limit (flagged; the answers keep their scores): Grok 4.7high, 17 of its recorded positions (7 standard / 10 Nackgammon), 7 in the main 200 pairs.
Scope. These results hold for these rosters only; each group is the setups in the run from that original league.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
By phase (descriptive)
Swipe for more columns →
| Setup (group) | opening: standard / Nackgammon PR | opening: drop [95%] | contact: standard / Nackgammon PR | contact: drop [95%] | holding: standard / Nackgammon PR | holding: drop [95%] | race: standard / Nackgammon PR | race: drop [95%] | bear-off: standard / Nackgammon PR | bear-off: drop [95%] | Opening drop − other phases' drop [95%] |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 17.73 / 26.99 | +9.25 [−2.30, +20.66] | 39.49 / 29.34 | −10.16 [−28.79, +7.76] | 26.14 / 43.33 | +17.19 [−11.01, +52.43] | 7.15 / 10.53 | +3.38 [−5.64, +14.63] | 6.03 / 1.62 | −4.41 [−9.76, −0.42] | +10.11 [−5.64, +25.57] |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 18.05 / 26.07 | +8.02 [−2.60, +18.11] | 40.73 / 31.12 | −9.62 [−28.23, +8.37] | 25.89 / 31.67 | +5.78 [−15.22, +30.11] | 7.15 / 10.16 | +3.01 [−5.95, +14.23] | 4.14 / 1.62 | −2.53 [−6.40, −0.04] | +10.81 [−3.54, +24.91] |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 18.68 / 28.66 | +9.98 [−1.30, +21.21] | 32.73 / 28.98 | −3.75 [−19.32, +12.24] | 48.27 / 55.33 | +7.06 [−72.46, +68.47] | 7.54 / 14.15 | +6.61 [−4.67, +24.16] | 5.16 / 7.24 | +2.08 [−3.98, +11.87] | +8.52 [−11.10, +29.64] |
| Jev 1.13Closed · EfficientLeague 1 group | 20.21 / 23.69 | +3.49 [−7.33, +14.00] | 31.78 / 28.13 | −3.65 [−21.53, +13.76] | 63.62 / 42.43 | −21.19 [−97.45, +35.95] | 6.34 / 10.79 | +4.46 [−4.55, +15.49] | 4.82 / 7.24 | +2.42 [−3.09, +11.88] | +8.36 [−10.22, +29.77] |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 16.69 / 35.63 | +18.94 [+6.05, +31.44] | 40.34 / 30.85 | −9.49 [−31.25, +11.04] | 60.56 / 64.87 | +4.31 [−74.85, +63.66] | 9.65 / 25.92 | +16.28 [−5.65, +51.13] | 7.01 / 9.57 | +2.57 [−6.03, +13.91] | +18.72 [−2.72, +42.58] |
| celeris-1-decisionClosed · EfficientLeague 1 group | 33.69 / 24.63 | −9.07 [−26.13, +6.22] | 33.67 / 28.66 | −5.01 [−22.42, +12.21] | 51.28 / 59.76 | +8.48 [−74.23, +83.54] | 7.12 / 19.44 | +12.32 [−7.03, +45.44] | 7.35 / 8.75 | +1.40 [−9.03, +13.58] | −11.19 [−36.43, +14.27] |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 19.69 / 37.37 | +17.68 [+4.29, +31.27] | 43.35 / 29.99 | −13.36 [−30.51, +3.94] | 65.89 / 48.33 | −17.56 [−95.10, +44.28] | 6.19 / 8.59 | +2.40 [−5.96, +13.24] | 6.83 / 0.85 | −5.98 [−14.04, −0.23] | +27.80 [+7.23, +50.59] |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 28.33 / 27.23 | −1.10 [−15.92, +13.25] | 38.19 / 31.50 | −6.69 [−24.72, +11.29] | 66.50 / 31.26 | −35.24 [−109.32, +15.97] | 5.97 / 13.21 | +7.24 [−1.28, +18.07] | 5.20 / 4.13 | −1.06 [−5.91, +4.81] | +8.22 [−12.67, +31.39] |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 30.16 / 37.55 | +7.39 [−6.51, +21.06] | 42.13 / 38.64 | −3.49 [−23.46, +18.42] | 60.08 / 42.23 | −17.85 [−94.94, +42.28] | 5.27 / 2.75 | −2.51 [−8.14, +2.98] | 4.44 / 1.02 | −3.41 [−7.89, −0.15] | +13.77 [−7.21, +37.11] |
| Mercury DecideClosed · EfficientLeague 1 group | 26.72 / 30.21 | +3.49 [−10.13, +15.91] | 46.91 / 29.39 | −17.52 [−35.81, −0.62] | 60.18 / 47.77 | −12.41 [−88.53, +41.14] | 7.74 / 25.88 | +18.14 [−4.66, +52.55] | 5.22 / 1.42 | −3.80 [−8.36, −0.66] | +11.10 [−9.50, +33.39] |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 24.97 / 32.92 | +7.95 [−6.25, +21.71] | 45.25 / 33.14 | −12.11 [−31.78, +7.01] | 73.23 / 76.68 | +3.45 [−84.49, +79.88] | 11.29 / 12.97 | +1.68 [−7.12, +10.17] | 6.94 / 3.95 | −2.99 [−9.39, +3.76] | +12.63 [−10.31, +37.00] |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 27.28 / 27.14 | −0.14 [−13.61, +12.94] | 50.09 / 31.86 | −18.23 [−39.05, +1.11] | 66.47 / 26.41 | −40.06 [−113.99, +16.88] | 7.57 / 8.51 | +0.94 [−9.24, +12.63] | 4.39 / 0.63 | −3.76 [−8.54, −0.46] | +16.76 [−4.00, +39.66] |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 32.27 / 33.12 | +0.86 [−15.82, +18.38] | 69.47 / 58.06 | −11.41 [−40.76, +18.75] | 46.43 / 75.18 | +28.75 [−24.00, +99.67] | 10.73 / 24.51 | +13.78 [−1.66, +32.85] | 7.52 / 9.30 | +1.79 [−6.82, +11.56] | −3.19 [−29.19, +21.46] |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 45.60 / 29.70 | −15.90 [−34.91, +3.03] | 88.55 / 60.66 | −27.89 [−63.56, +5.21] | 63.23 / 81.96 | +18.72 [−49.40, +90.12] | 11.32 / 10.76 | −0.56 [−13.57, +12.64] | 7.48 / 1.74 | −5.75 [−13.02, −0.57] | −6.83 [−35.83, +21.69] |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 12.39 / 13.21 | +0.82 [−6.57, +8.28] | 13.67 / 11.24 | −2.43 [−9.76, +5.00] | 13.41 / 7.15 | −6.26 [−19.56, +3.87] | 0.43 / 0.18 | −0.24 [−1.01, +0.38] | 0.00 / 0.00 | +0.00 [+0.00, +0.00] | +3.24 [−5.08, +12.07] |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 12.75 / 16.48 | +3.73 [−4.12, +12.06] | 16.46 / 12.74 | −3.71 [−12.27, +4.62] | 13.70 / 7.37 | −6.33 [−19.68, +3.53] | 1.10 / 0.18 | −0.91 [−2.57, +0.28] | 1.59 / 0.00 | −1.59 [−4.77, +0.00] | +7.12 [−1.90, +16.65] |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 14.50 / 18.95 | +4.45 [−3.80, +13.62] | 24.43 / 24.37 | −0.07 [−12.53, +11.73] | 13.68 / 7.60 | −6.07 [−19.26, +3.64] | 1.85 / 2.86 | +1.01 [−3.88, +7.07] | 0.00 / 0.07 | +0.07 [+0.00, +0.21] | +5.58 [−4.32, +16.34] |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 14.94 / 16.01 | +1.07 [−5.51, +7.26] | 28.74 / 16.38 | −12.35 [−24.99, −0.09] | 14.24 / 13.35 | −0.88 [−14.62, +10.17] | 3.31 / 0.64 | −2.67 [−6.94, +0.62] | 2.15 / 0.07 | −2.08 [−6.01, +0.00] | +7.40 [−1.46, +16.42] |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 19.52 / 19.53 | +0.01 [−9.76, +10.08] | 32.82 / 19.17 | −13.65 [−29.63, +2.24] | 13.15 / 11.62 | −1.54 [−15.22, +9.50] | 3.36 / 0.18 | −3.17 [−6.88, −0.32] | 2.02 / 0.03 | −1.99 [−5.93, +0.06] | +7.11 [−5.19, +19.68] |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 13.39 / 27.97 | +14.58 [+3.87, +25.35] | 38.21 / 27.57 | −10.64 [−31.09, +9.00] | 47.28 / 62.50 | +15.21 [−64.76, +76.60] | 6.31 / 13.97 | +7.66 [−3.66, +24.98] | 4.43 / 0.94 | −3.49 [−8.35, −0.14] | +15.13 [−4.89, +37.40] |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 19.21 / 32.36 | +13.15 [−0.07, +25.87] | 38.40 / 29.03 | −9.37 [−23.98, +4.78] | 61.66 / 49.21 | −12.45 [−88.96, +41.03] | 7.24 / 11.56 | +4.32 [−4.51, +15.35] | 5.56 / 9.54 | +3.97 [−4.31, +15.03] | +18.36 [−1.24, +39.95] |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 33.94 / 37.10 | +3.16 [−13.56, +21.60] | 64.84 / 39.90 | −24.94 [−54.30, +5.53] | 42.82 / 72.20 | +29.38 [−14.29, +77.60] | 11.24 / 19.24 | +8.00 [−7.23, +27.95] | 5.92 / 9.74 | +3.82 [−4.51, +14.94] | +5.44 [−18.48, +29.98] |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 40.54 / 56.04 | +15.51 [−2.57, +33.81] | 65.65 / 65.61 | −0.04 [−28.23, +29.22] | 88.29 / 73.95 | −14.33 [−107.31, +69.03] | 24.63 / 26.07 | +1.45 [−14.37, +21.51] | 20.60 / 25.09 | +4.49 [−6.96, +17.06] | +17.54 [−10.83, +47.11] |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 10.73 / 14.81 | +4.08 [−3.84, +12.23] | 12.57 / 14.10 | +1.53 [−8.40, +11.25] | 14.04 / 5.40 | −8.63 [−21.56, +1.07] | 0.25 / 0.18 | −0.07 [−0.76, +0.55] | 0.00 / 0.00 | +0.00 [+0.00, +0.00] | +5.29 [−4.00, +15.02] |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 8.84 / 12.14 | +3.30 [−3.03, +9.70] | 16.23 / 9.73 | −6.50 [−14.88, +1.78] | 13.57 / 4.68 | −8.88 [−22.08, +1.05] | 0.83 / 0.18 | −0.64 [−2.22, +0.37] | 0.03 / 0.07 | +0.04 [+0.00, +0.13] | +8.10 [+0.51, +16.05] |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 12.73 / 22.83 | +10.09 [+1.10, +19.52] | 28.56 / 17.57 | −10.99 [−21.81, −0.91] | 16.84 / 14.39 | −2.45 [−20.92, +17.77] | 2.30 / 0.93 | −1.37 [−5.37, +1.51] | 0.00 / 1.11 | +1.11 [+0.00, +3.18] | +15.37 [+4.49, +26.73] |
| GPT-6.1 Terrano group (exploratory) | 13.35 / 20.77 | +7.42 [+1.12, +13.95] | 23.74 / 17.13 | −6.61 [−18.78, +5.67] | 14.00 / 19.69 | +5.69 [−18.06, +40.13] | 0.56 / 0.64 | +0.08 [−0.99, +1.35] | 1.76 / 0.00 | −1.76 [−5.12, +0.00] | +9.34 [−1.66, +19.61] |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 15.22 / 18.13 | +2.91 [−7.20, +12.64] | 23.00 / 17.18 | −5.83 [−16.79, +4.71] | 13.58 / 31.67 | +18.09 [+0.69, +42.23] | 2.09 / 2.60 | +0.50 [−4.03, +6.55] | 0.23 / 0.34 | +0.11 [−0.48, +0.81] | +1.42 [−10.49, +13.27] |
| GNUbg weakyardstick | 38.78 / 31.32 | −7.46 [−21.30, +6.50] | 39.90 / 34.06 | −5.85 [−32.05, +20.17] | 22.63 / 26.00 | +3.37 [−12.48, +21.09] | 0.39 / 3.40 | +3.01 [−0.03, +7.82] | 5.15 / 1.53 | −3.61 [−10.52, +0.66] | −5.57 [−23.29, +11.90] |
| Randomyardstick | 95.63 / 84.31 | −11.32 [−23.34, +0.42] | 125.31 / 115.99 | −9.32 [−27.33, +8.75] | 113.57 / 121.18 | +7.61 [−57.23, +63.11] | 19.17 / 22.57 | +3.40 [−7.66, +16.86] | 13.92 / 14.46 | +0.54 [−5.63, +6.89] | −9.66 [−28.68, +9.80] |
Scope. These results hold for these rosters only; each group is the setups in the run from that original league.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
By game-plan tag (descriptive)
Swipe for more columns →
| Setup (group) | (none: bear-off): standard PR · n · Nackgammon PR · n | (none: opening): standard PR · n · Nackgammon PR · n | (none: race): standard PR · n · Nackgammon PR · n | back game: standard PR · n · Nackgammon PR · n | blitz: standard PR · n · Nackgammon PR · n | holding: standard PR · n · Nackgammon PR · n | low-anchor: standard PR · n · Nackgammon PR · n | other: standard PR · n · Nackgammon PR · n | priming: standard PR · n · Nackgammon PR · n |
|---|---|---|---|---|---|---|---|---|---|
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 6.03 · 25 · 1.62 · 25 | 17.73 · 60 · 26.99 · 60 | 7.15 · 25 · 10.53 · 25 | 22.94 · 2 · 44.17 · 10 | 48.46 · 23 · 8.80 · 17 | 26.14 · 30 · 43.33 · 30 | 39.62 · 8 · 31.03 · 7 | 25.00 · 10 · 29.68 · 11 | 37.78 · 17 · 41.69 · 15 |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 4.14 · 25 · 1.62 · 25 | 18.05 · 60 · 26.07 · 60 | 7.15 · 25 · 10.16 · 25 | 22.94 · 2 · 44.17 · 10 | 49.93 · 23 · 12.95 · 17 | 25.89 · 30 · 31.67 · 30 | 39.62 · 8 · 31.03 · 7 | 29.05 · 10 · 29.68 · 11 | 37.78 · 17 · 44.10 · 15 |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 5.16 · 25 · 7.24 · 25 | 18.68 · 60 · 28.66 · 60 | 7.54 · 25 · 14.15 · 25 | 60.79 · 2 · 27.83 · 10 | 27.25 · 23 · 16.11 · 17 | 48.27 · 30 · 55.33 · 30 | 36.13 · 8 · 45.54 · 7 | 44.45 · 10 · 24.72 · 11 | 28.36 · 17 · 39.74 · 15 |
| Jev 1.13Closed · EfficientLeague 1 group | 4.82 · 25 · 7.24 · 25 | 20.21 · 60 · 23.69 · 60 | 6.34 · 25 · 10.79 · 25 | 22.94 · 2 · 35.43 · 10 | 33.90 · 23 · 13.43 · 17 | 63.62 · 30 · 42.43 · 30 | 48.48 · 8 · 37.32 · 7 | 36.46 · 10 · 17.33 · 11 | 19.34 · 17 · 43.56 · 15 |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 7.01 · 25 · 9.57 · 25 | 16.69 · 60 · 35.63 · 60 | 9.65 · 25 · 25.92 · 25 | 22.94 · 2 · 32.07 · 10 | 44.92 · 23 · 14.08 · 17 | 60.56 · 30 · 64.87 · 30 | 36.76 · 8 · 45.54 · 7 | 63.71 · 10 · 30.99 · 11 | 24.12 · 17 · 42.09 · 15 |
| celeris-1-decisionClosed · EfficientLeague 1 group | 7.35 · 25 · 8.75 · 25 | 33.69 · 60 · 24.63 · 60 | 7.12 · 25 · 19.44 · 25 | 22.94 · 2 · 37.21 · 10 | 35.99 · 23 · 30.60 · 17 | 51.28 · 30 · 59.76 · 30 | 31.16 · 8 · 40.57 · 7 | 23.29 · 10 · 14.27 · 11 | 39.10 · 17 · 25.76 · 15 |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 6.83 · 25 · 0.85 · 25 | 19.69 · 60 · 37.37 · 60 | 6.19 · 25 · 8.59 · 25 | 22.94 · 2 · 31.19 · 10 | 56.92 · 23 · 16.55 · 17 | 65.89 · 30 · 48.33 · 30 | 37.56 · 8 · 35.51 · 7 | 25.72 · 10 · 25.86 · 11 | 40.48 · 17 · 44.86 · 15 |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 5.20 · 25 · 4.13 · 25 | 28.33 · 60 · 27.23 · 60 | 5.97 · 25 · 13.21 · 25 | 60.79 · 2 · 27.83 · 10 | 36.62 · 23 · 18.67 · 17 | 66.50 · 30 · 31.26 · 30 | 49.87 · 8 · 40.66 · 7 | 46.30 · 10 · 17.92 · 11 | 27.39 · 17 · 54.16 · 15 |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 4.44 · 25 · 1.02 · 25 | 30.16 · 60 · 37.55 · 60 | 5.27 · 25 · 2.75 · 25 | 17.04 · 2 · 51.48 · 10 | 42.33 · 23 · 34.81 · 17 | 60.08 · 30 · 42.23 · 30 | 48.45 · 8 · 48.65 · 7 | 32.21 · 10 · 16.02 · 11 | 47.68 · 17 · 46.33 · 15 |
| Mercury DecideClosed · EfficientLeague 1 group | 5.22 · 25 · 1.42 · 25 | 26.72 · 60 · 30.21 · 60 | 7.74 · 25 · 25.88 · 25 | 22.94 · 2 · 35.43 · 10 | 58.55 · 23 · 10.87 · 17 | 60.18 · 30 · 47.77 · 30 | 44.31 · 8 · 45.54 · 7 | 45.57 · 10 · 21.24 · 11 | 36.00 · 17 · 44.77 · 15 |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 6.94 · 25 · 3.95 · 25 | 24.97 · 60 · 32.92 · 60 | 11.29 · 25 · 12.97 · 25 | 54.89 · 2 · 27.91 · 10 | 41.91 · 23 · 10.87 · 17 | 73.23 · 30 · 76.68 · 30 | 45.79 · 8 · 41.48 · 7 | 65.55 · 10 · 35.18 · 11 | 36.43 · 17 · 56.47 · 15 |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 4.39 · 25 · 0.63 · 25 | 27.28 · 60 · 27.14 · 60 | 7.57 · 25 · 8.51 · 25 | 17.04 · 2 · 43.32 · 10 | 75.68 · 23 · 14.51 · 17 | 66.47 · 30 · 26.41 · 30 | 47.99 · 8 · 51.90 · 7 | 21.48 · 10 · 20.23 · 11 | 37.18 · 17 · 43.06 · 15 |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 7.52 · 25 · 9.30 · 25 | 32.27 · 60 · 33.12 · 60 | 10.73 · 25 · 24.51 · 25 | 35.25 · 2 · 47.05 · 10 | 85.22 · 23 · 51.71 · 17 | 46.43 · 30 · 75.18 · 30 | 44.13 · 8 · 80.19 · 7 | 56.50 · 10 · 32.79 · 11 | 71.75 · 17 · 80.81 · 15 |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 7.48 · 25 · 1.74 · 25 | 45.60 · 60 · 29.70 · 60 | 11.32 · 25 · 10.76 · 25 | 81.86 · 2 · 52.70 · 10 | 98.15 · 23 · 58.85 · 17 | 63.23 · 30 · 81.96 · 30 | 42.77 · 8 · 70.45 · 7 | 81.20 · 10 · 40.10 · 11 | 102.23 · 17 · 78.53 · 15 |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 0.00 · 25 · 0.00 · 25 | 12.39 · 60 · 13.21 · 60 | 0.43 · 25 · 0.18 · 25 | 11.09 · 2 · 17.22 · 10 | 12.66 · 23 · 10.78 · 17 | 13.41 · 30 · 7.15 · 30 | 13.16 · 8 · 9.99 · 7 | 20.29 · 10 · 12.75 · 11 | 11.68 · 17 · 7.25 · 15 |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 1.59 · 25 · 0.00 · 25 | 12.75 · 60 · 16.48 · 60 | 1.10 · 25 · 0.18 · 25 | 22.94 · 2 · 17.22 · 10 | 11.77 · 23 · 12.98 · 17 | 13.70 · 30 · 7.37 · 30 | 37.19 · 8 · 21.59 · 7 | 20.24 · 10 · 10.68 · 11 | 10.05 · 17 · 6.89 · 15 |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 0.00 · 25 · 0.07 · 25 | 14.50 · 60 · 18.95 · 60 | 1.85 · 25 · 2.86 · 25 | 11.09 · 2 · 39.47 · 10 | 24.68 · 23 · 10.09 · 17 | 13.68 · 30 · 7.60 · 30 | 41.27 · 8 · 36.58 · 7 | 23.58 · 10 · 30.71 · 11 | 18.25 · 17 · 20.13 · 15 |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 2.15 · 25 · 0.07 · 25 | 14.94 · 60 · 16.01 · 60 | 3.31 · 25 · 0.64 · 25 | 22.94 · 2 · 27.83 · 10 | 29.33 · 23 · 6.66 · 17 | 14.24 · 30 · 13.35 · 30 | 36.72 · 8 · 29.48 · 7 | 21.48 · 10 · 15.09 · 11 | 29.13 · 17 · 14.61 · 15 |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 2.02 · 25 · 0.03 · 25 | 19.52 · 60 · 19.53 · 60 | 3.36 · 25 · 0.18 · 25 | 22.94 · 2 · 28.66 · 10 | 28.28 · 23 · 8.77 · 17 | 13.15 · 30 · 11.62 · 30 | 38.14 · 8 · 27.07 · 7 | 40.99 · 10 · 16.96 · 11 | 32.83 · 17 · 22.56 · 15 |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 4.43 · 25 · 0.94 · 25 | 13.39 · 60 · 27.97 · 60 | 6.31 · 25 · 13.97 · 25 | 22.94 · 2 · 27.83 · 10 | 35.63 · 23 · 11.36 · 17 | 47.28 · 30 · 62.50 · 30 | 40.47 · 8 · 45.54 · 7 | 25.00 · 10 · 27.94 · 11 | 50.20 · 17 · 37.11 · 15 |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 5.56 · 25 · 9.54 · 25 | 19.21 · 60 · 32.36 · 60 | 7.24 · 25 · 11.56 · 25 | 22.94 · 2 · 23.84 · 10 | 47.64 · 23 · 11.86 · 17 | 61.66 · 30 · 49.21 · 30 | 44.31 · 8 · 45.54 · 7 | 43.95 · 10 · 24.30 · 11 | 21.69 · 17 · 47.71 · 15 |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 5.92 · 25 · 9.74 · 25 | 33.94 · 60 · 37.10 · 60 | 11.24 · 25 · 19.24 · 25 | 68.33 · 2 · 34.16 · 10 | 58.45 · 23 · 14.94 · 17 | 42.82 · 30 · 72.20 · 30 | 51.18 · 8 · 53.66 · 7 | 67.40 · 10 · 21.02 · 11 | 77.99 · 17 · 79.43 · 15 |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 20.60 · 25 · 25.09 · 25 | 40.54 · 60 · 56.04 · 60 | 24.63 · 25 · 26.07 · 25 | 22.94 · 2 · 45.59 · 10 | 76.34 · 23 · 125.24 · 17 | 88.29 · 30 · 73.95 · 30 | 62.22 · 8 · 45.08 · 7 | 90.40 · 10 · 21.21 · 11 | 43.26 · 17 · 53.51 · 15 |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 0.00 · 25 · 0.00 · 25 | 10.73 · 60 · 14.81 · 60 | 0.25 · 25 · 0.18 · 25 | 17.04 · 2 · 23.32 · 10 | 16.79 · 23 · 6.51 · 17 | 14.04 · 30 · 5.40 · 30 | 16.31 · 8 · 19.97 · 7 | 5.79 · 10 · 18.60 · 11 | 8.55 · 17 · 10.54 · 15 |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 0.03 · 25 · 0.07 · 25 | 8.84 · 60 · 12.14 · 60 | 0.83 · 25 · 0.18 · 25 | 11.09 · 2 · 17.22 · 10 | 10.20 · 23 · 10.78 · 17 | 13.57 · 30 · 4.68 · 30 | 21.16 · 8 · 9.99 · 7 | 23.69 · 10 · 8.27 · 11 | 18.28 · 17 · 4.48 · 15 |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 0.00 · 25 · 1.11 · 25 | 12.73 · 60 · 22.83 · 60 | 2.30 · 25 · 0.93 · 25 | 39.75 · 2 · 18.61 · 10 | 23.28 · 23 · 13.20 · 17 | 16.84 · 30 · 14.39 · 30 | 40.59 · 8 · 45.54 · 7 | 20.82 · 10 · 19.01 · 11 | 33.27 · 17 · 7.71 · 15 |
| GPT-6.1 Terrano group (exploratory) | 1.76 · 25 · 0.00 · 25 | 13.35 · 60 · 20.77 · 60 | 0.56 · 25 · 0.64 · 25 | 39.75 · 2 · 31.19 · 10 | 18.82 · 23 · 6.51 · 17 | 14.00 · 30 · 19.69 · 30 | 28.12 · 8 · 50.56 · 7 | 20.24 · 10 · 12.01 · 11 | 28.52 · 17 · 7.94 · 15 |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 0.23 · 25 · 0.34 · 25 | 15.22 · 60 · 18.13 · 60 | 2.09 · 25 · 2.60 · 25 | 0.00 · 2 · 21.46 · 10 | 22.27 · 23 · 8.73 · 17 | 13.58 · 30 · 31.67 · 30 | 32.68 · 8 · 35.49 · 7 | 15.73 · 10 · 28.54 · 11 | 26.43 · 17 · 7.02 · 15 |
| GNUbg weakyardstick | 5.15 · 25 · 1.53 · 25 | 38.78 · 60 · 31.32 · 60 | 0.39 · 25 · 3.40 · 25 | 17.04 · 2 · 22.11 · 10 | 43.29 · 23 · 56.39 · 17 | 22.63 · 30 · 26.00 · 30 | 32.99 · 8 · 37.50 · 7 | 47.08 · 10 · 17.63 · 11 | 37.05 · 17 · 27.14 · 15 |
| Randomyardstick | 13.92 · 25 · 14.46 · 25 | 95.63 · 60 · 84.31 · 60 | 19.17 · 25 · 22.57 · 25 | 57.49 · 2 · 104.58 · 10 | 125.59 · 23 · 143.55 · 17 | 113.57 · 30 · 121.18 · 30 | 92.05 · 8 · 113.21 · 7 | 135.92 · 10 · 81.52 · 11 | 142.31 · 17 · 118.93 · 15 |
Scope. These results hold for these rosters only; each group is the setups in the run from that original league.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Cost per decision
Swipe for more columns →
| Setup (group) | Decisions, main | Recorded cost, USD per 1,000 decisions, main | API-equivalent USD per 1,000 decisions, main | Positions recorded, all (incl. timed out) | Recorded cost, USD per 1,000, all | API-equivalent USD per 1,000, all |
|---|---|---|---|---|---|---|
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 400 | 0.105 | 0.000 | 1,000 | 0.105 | 0.000 |
| Jev 1.13Closed · EfficientLeague 1 group | 400 | 0.129 | 0.000 | 1,000 | 0.129 | 0.000 |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 400 | 0.267 | 0.000 | 1,000 | 0.267 | 0.000 |
| celeris-1-decisionClosed · EfficientLeague 1 group | 400 | 0.112 | 0.000 | 1,000 | 0.112 | 0.000 |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 400 | 0.056 | 0.000 | 1,000 | 0.056 | 0.000 |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 400 | 0.140 | 0.000 | 1,000 | 0.140 | 0.000 |
| Mercury DecideClosed · EfficientLeague 1 group | 400 | 0.056 | 0.000 | 1,000 | 0.056 | 0.000 |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 400 | 0.056 | 0.000 | 1,000 | 0.056 | 0.000 |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 400 | 0.028 | 0.000 | 1,000 | 0.028 | 0.000 |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 400 | 8.148 | 0.000 | 1,000 | 8.066 | 0.000 |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 400 | 6.022 | 0.000 | 1,000 | 6.016 | 0.000 |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 400 | 8.765 | 0.000 | 1,000 | 8.726 | 0.000 |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 400 | 2.845 | 0.000 | 1,000 | 2.885 | 0.000 |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 400 | 0.293 | 0.000 | 1,000 | 0.267 | 0.000 |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 400 | 72.906 | 0.000 | 1,000 | 73.670 | 0.000 |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 400 | 29.614 | 0.000 | 1,000 | 29.515 | 0.000 |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 400 | 64.087 | 0.000 | 1,000 | 63.440 | 0.000 |
| GPT-6.1 Terrano group (exploratory) | 400 | 3.723 | 0.000 | 1,000 | 3.709 | 0.000 |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 400 | 5.906 | 0.000 | 1,000 | 5.907 | 0.000 |
| GNUbg weakyardstick | 400 | 0.000 | 0.000 | 1,000 | 0.000 | 0.000 |
Recorded cost (the field billed_usd_per_1000) is the cost stored with each answer: the provider's reported cost per response where the route reports one, otherwise the runner's list-price computation (Mercury Decide: list price; its provider-billed amount was not retained and is unknown). No invoice was checked. API-equivalent is a subscription setup priced at list price, not money paid. Setups on our GPUs have no list price.
Scope. These results hold for these rosters only; each group is the setups in the run from that original league.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Secondary analyses
Swipe for more columns →
| Setup (group) | Kept advantage K | 95% interval | Standard advantage over Random (PR) | 95% interval | Unstable |
|---|---|---|---|---|---|
| GNUbg weakyardstick | 0.98 | [0.77, 1.25] | 59.76 | [47.61, 72.84] | no |
| celeris-1-decisionClosed · EfficientLeague 1 group | 0.94 | [0.76, 1.15] | 57.74 | [47.55, 68.65] | no |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 0.96 | [0.76, 1.19] | 55.54 | [45.29, 66.11] | no |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 1.04 | [0.86, 1.27] | 56.12 | [46.07, 66.46] | no |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 1.14 | [0.93, 1.39] | 52.77 | [42.90, 62.93] | no |
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 0.90 | [0.73, 1.09] | 64.71 | [52.87, 77.53] | no |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 0.87 | [0.71, 1.05] | 63.54 | [53.09, 74.34] | no |
| Jev 1.13Closed · EfficientLeague 1 group | 0.96 | [0.80, 1.16] | 60.92 | [50.62, 71.41] | no |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 0.82 | [0.64, 1.03] | 59.18 | [49.11, 69.51] | no |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 0.95 | [0.76, 1.17] | 57.03 | [46.86, 67.46] | no |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 0.92 | [0.68, 1.22] | 49.25 | [38.34, 61.49] | no |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 0.92 | [0.76, 1.10] | 59.32 | [49.48, 69.69] | no |
| Mercury DecideClosed · EfficientLeague 1 group | 0.99 | [0.81, 1.21] | 54.71 | [45.13, 64.95] | no |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 0.93 | [0.74, 1.16] | 53.12 | [43.32, 63.11] | no |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 0.79 | [0.51, 1.19] | 36.70 | [25.62, 47.88] | no |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 1.19 | [0.84, 1.83] | 35.37 | [22.97, 48.64] | no |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 0.84 | [0.58, 1.16] | 47.68 | [36.50, 60.40] | no |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 0.99 | [0.83, 1.18] | 71.53 | [59.26, 84.64] | no |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 0.93 | [0.78, 1.10] | 73.49 | [61.59, 86.24] | no |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 0.96 | [0.81, 1.12] | 76.30 | [64.19, 89.41] | no |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 0.97 | [0.83, 1.13] | 77.79 | [65.54, 90.83] | no |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 0.96 | [0.82, 1.11] | 77.57 | [65.47, 90.77] | no |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 1.01 | [0.84, 1.21] | 69.10 | [57.09, 82.14] | no |
| GPT-6.1 Terrano group (exploratory) | 0.93 | [0.78, 1.10] | 73.93 | [61.79, 86.84] | no |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 0.94 | [0.80, 1.09] | 78.32 | [66.07, 91.51] | no |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 0.95 | [0.79, 1.12] | 72.25 | [60.31, 85.00] | no |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 0.86 | [0.71, 1.05] | 63.20 | [52.20, 74.21] | no |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 0.92 | [0.75, 1.13] | 64.52 | [52.71, 77.38] | no |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 0.91 | [0.77, 1.07] | 73.66 | [61.40, 86.70] | no |
Swipe for more columns →
| Setup (group) | Kept advantage K | 95% interval | Standard advantage over Random (PR) | 95% interval | Unstable |
|---|---|---|---|---|---|
| GNUbg weakyardstick | 1.01 | [0.88, 1.16] | 59.81 | [52.45, 67.38] | no |
| celeris-1-decisionClosed · EfficientLeague 1 group | 0.87 | [0.76, 0.99] | 58.30 | [51.64, 65.13] | no |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 0.90 | [0.78, 1.03] | 53.91 | [47.30, 60.50] | no |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 0.93 | [0.83, 1.05] | 56.70 | [50.32, 63.36] | no |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 0.98 | [0.84, 1.14] | 53.44 | [46.69, 60.36] | no |
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 0.91 | [0.79, 1.05] | 58.83 | [52.10, 65.71] | no |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 0.84 | [0.75, 0.95] | 61.80 | [55.23, 68.46] | no |
| Jev 1.13Closed · EfficientLeague 1 group | 0.89 | [0.80, 1.00] | 60.49 | [54.05, 67.08] | no |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 0.83 | [0.72, 0.96] | 55.60 | [49.00, 62.17] | no |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 0.92 | [0.80, 1.05] | 53.89 | [47.43, 60.46] | no |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 0.82 | [0.68, 0.98] | 47.38 | [40.64, 54.20] | no |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 0.85 | [0.76, 0.96] | 58.27 | [51.96, 64.55] | no |
| Mercury DecideClosed · EfficientLeague 1 group | 0.88 | [0.77, 1.01] | 53.81 | [47.34, 60.28] | no |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 0.88 | [0.77, 1.01] | 53.22 | [46.79, 59.83] | no |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 0.79 | [0.61, 1.00] | 38.72 | [31.77, 45.62] | no |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 0.98 | [0.81, 1.23] | 38.09 | [30.72, 45.47] | no |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 0.74 | [0.59, 0.92] | 44.93 | [37.85, 52.08] | no |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 0.96 | [0.86, 1.08] | 68.45 | [61.28, 75.91] | no |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 0.96 | [0.86, 1.08] | 68.71 | [61.60, 76.16] | no |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 0.97 | [0.87, 1.08] | 72.95 | [65.81, 80.40] | no |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 0.98 | [0.88, 1.08] | 74.29 | [67.10, 81.78] | no |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 0.98 | [0.88, 1.08] | 74.01 | [66.80, 81.46] | no |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 0.96 | [0.85, 1.08] | 67.19 | [60.13, 74.56] | no |
| GPT-6.1 Terrano group (exploratory) | 0.91 | [0.81, 1.01] | 71.98 | [64.84, 79.45] | no |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 0.94 | [0.85, 1.03] | 75.52 | [68.27, 83.14] | no |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 0.93 | [0.83, 1.05] | 69.47 | [62.42, 76.93] | no |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 0.83 | [0.74, 0.94] | 59.85 | [53.18, 66.58] | no |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 0.89 | [0.78, 1.00] | 59.43 | [52.73, 66.34] | no |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 0.91 | [0.81, 1.01] | 70.45 | [63.31, 77.95] | no |
Swipe for more columns →
| Analysis | Model setups | Spearman correlation | 95% interval |
|---|---|---|---|
| Standard against Nackgammon PR, main 200 pairs | 28 | 0.90 | [0.79, 0.92] |
Swipe for more columns →
| Setup | Rank, standard | Rank, Nackgammon | Change |
|---|---|---|---|
| Perplexity Decider 27B v1.2 | 24 | 11 | -13 |
| GPT-6 Luna | 17 | 24 | 7 |
| Cygnet | 20 | 14 | -6 |
| GLiDEv1.2 | 11 | 15 | 4 |
| Gemini 3.8 Flashlow | 9 | 5 | -4 |
Swipe for more columns →
| Setup (group) | Risk pays / Safety pays, standard | Risk pays / Safety pays, Nackgammon | Ability d′, standard | d′, Nackgammon | Change in d′ [95%] | Caution c, standard | c, Nackgammon | Change in c [95%] | Thin (a class under 50 positions) |
|---|---|---|---|---|---|---|---|---|---|
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 66 / 66 | 66 / 66 | 0.82 | 0.37 | −0.45 [−1.15, +0.19] | 0.60 | 0.65 | +0.05 [−0.29, +0.38] | no |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 66 / 66 | 66 / 66 | 0.72 | 0.48 | −0.24 [−0.87, +0.38] | 0.59 | 0.71 | +0.12 [−0.19, +0.45] | no |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 66 / 66 | 66 / 66 | 0.91 | 0.52 | −0.39 [−1.15, +0.27] | 0.49 | 0.52 | +0.03 [−0.32, +0.37] | no |
| Jev 1.13Closed · EfficientLeague 1 group | 66 / 66 | 66 / 66 | 0.59 | 1.10 | +0.51 [−0.17, +1.25] | 0.60 | 0.66 | +0.06 [−0.28, +0.43] | no |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 66 / 66 | 66 / 66 | 0.95 | 0.14 | −0.81 [−1.57, −0.11] | 0.66 | 0.66 | +0.00 [−0.36, +0.38] | no |
| celeris-1-decisionClosed · EfficientLeague 1 group | 66 / 66 | 66 / 66 | 0.43 | 0.85 | +0.42 [−0.32, +1.11] | 0.86 | 0.65 | −0.21 [−0.56, +0.15] | no |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 66 / 66 | 66 / 66 | 0.44 | 0.45 | +0.01 [−0.67, +0.66] | 0.56 | 0.61 | +0.05 [−0.27, +0.38] | no |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 66 / 66 | 66 / 66 | 0.62 | 0.52 | −0.10 [−0.79, +0.60] | 0.70 | 0.69 | −0.01 [−0.34, +0.35] | no |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 66 / 66 | 66 / 66 | 0.56 | 0.29 | −0.27 [−0.95, +0.34] | 0.36 | 0.45 | +0.09 [−0.23, +0.42] | no |
| Mercury DecideClosed · EfficientLeague 1 group | 66 / 66 | 66 / 66 | 0.31 | 0.48 | +0.17 [−0.51, +0.86] | 0.79 | 0.83 | +0.04 [−0.29, +0.37] | no |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 66 / 66 | 66 / 66 | 0.40 | 0.39 | −0.01 [−0.65, +0.60] | 0.35 | 0.23 | −0.12 [−0.43, +0.18] | no |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 66 / 66 | 66 / 66 | 0.30 | 0.88 | +0.58 [−0.10, +1.27] | 0.53 | 0.63 | +0.10 [−0.25, +0.45] | no |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 66 / 66 | 66 / 66 | 0.30 | 0.53 | +0.23 [−0.42, +0.85] | 0.08 | 0.04 | −0.04 [−0.35, +0.25] | no |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 66 / 66 | 66 / 66 | 0.42 | 0.23 | −0.19 [−0.82, +0.43] | 0.06 | 0.23 | +0.17 [−0.14, +0.46] | no |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 66 / 66 | 66 / 66 | 1.18 | 2.05 | +0.87 [+0.16, +1.71] | 0.63 | 0.76 | +0.13 [−0.22, +0.54] | no |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 66 / 66 | 66 / 66 | 1.14 | 1.86 | +0.72 [+0.04, +1.56] | 0.65 | 0.85 | +0.20 [−0.12, +0.63] | no |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 66 / 66 | 66 / 66 | 0.99 | 1.44 | +0.45 [−0.32, +1.27] | 0.80 | 0.91 | +0.11 [−0.25, +0.51] | no |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 66 / 66 | 66 / 66 | 0.79 | 2.17 | +1.38 [+0.73, +1.92] | 0.90 | 1.35 | +0.45 [+0.12, +0.72] | no |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 66 / 66 | 66 / 66 | 0.66 | 1.56 | +0.90 [+0.04, +1.81] | 1.17 | 1.00 | −0.17 [−0.61, +0.30] | no |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 66 / 66 | 66 / 66 | 0.78 | 0.60 | −0.18 [−0.90, +0.55] | 0.50 | 0.65 | +0.15 [−0.20, +0.49] | no |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 66 / 66 | 66 / 66 | 0.31 | 0.37 | +0.06 [−0.64, +0.78] | 0.79 | 0.82 | +0.03 [−0.29, +0.36] | no |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 66 / 66 | 66 / 66 | 0.53 | 0.47 | −0.06 [−0.70, +0.52] | 0.17 | 0.23 | +0.06 [−0.24, +0.35] | no |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 66 / 66 | 66 / 66 | 0.23 | 0.45 | +0.22 [−0.39, +0.86] | 0.19 | −0.00 | −0.19 [−0.49, +0.11] | no |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 66 / 66 | 66 / 66 | 1.33 | 2.01 | +0.68 [+0.00, +1.47] | 0.55 | 0.78 | +0.23 [−0.11, +0.62] | no |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 66 / 66 | 66 / 66 | 1.53 | 2.05 | +0.52 [−0.18, +1.32] | 0.73 | 0.76 | +0.03 [−0.32, +0.43] | no |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 66 / 66 | 66 / 66 | 0.63 | 1.47 | +0.84 [+0.16, +1.76] | 0.47 | 0.89 | +0.42 [+0.09, +0.87] | no |
| GPT-6.1 Terrano group (exploratory) | 66 / 66 | 66 / 66 | 0.91 | 1.44 | +0.53 [−0.29, +1.45] | 0.76 | 1.06 | +0.30 [−0.07, +0.75] | no |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 66 / 66 | 66 / 66 | 1.17 | 1.26 | +0.09 [−0.48, +0.71] | 0.36 | 0.67 | +0.31 [+0.02, +0.61] | no |
| GNUbg weakyardstick | 66 / 66 | 66 / 66 | 0.71 | 0.95 | +0.24 [−0.37, +0.82] | −0.28 | −0.47 | −0.19 [−0.48, +0.09] | no |
| Randomyardstick | 66 / 66 | 66 / 66 | 0.27 | 0.37 | +0.10 [−0.11, +0.30] | −0.77 | −0.82 | −0.05 [−0.15, +0.04] | no |
Swipe for more columns →
| Setup (group) | Risk pays / Safety pays, standard | Risk pays / Safety pays, Nackgammon | Ability d′, standard | d′, Nackgammon | Change in d′ [95%] | Caution c, standard | c, Nackgammon | Change in c [95%] | Thin (a class under 50 positions) |
|---|---|---|---|---|---|---|---|---|---|
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 165 / 165 | 165 / 165 | 0.49 | 0.37 | −0.12 [−0.54, +0.30] | 0.61 | 0.61 | +0.00 [−0.21, +0.21] | no |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 165 / 165 | 165 / 165 | 0.50 | 0.40 | −0.10 [−0.51, +0.33] | 0.68 | 0.68 | +0.00 [−0.19, +0.20] | no |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 165 / 165 | 165 / 165 | 0.92 | 0.43 | −0.49 [−0.92, −0.09] | 0.56 | 0.52 | −0.04 [−0.25, +0.16] | no |
| Jev 1.13Closed · EfficientLeague 1 group | 165 / 165 | 165 / 165 | 0.71 | 0.76 | +0.05 [−0.40, +0.48] | 0.75 | 0.57 | −0.18 [−0.39, +0.04] | no |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 165 / 165 | 165 / 165 | 0.73 | 0.34 | −0.39 [−0.86, +0.02] | 0.79 | 0.58 | −0.21 [−0.43, +0.01] | no |
| celeris-1-decisionClosed · EfficientLeague 1 group | 165 / 165 | 165 / 165 | 0.54 | 0.60 | +0.06 [−0.38, +0.46] | 0.80 | 0.58 | −0.22 [−0.44, −0.01] | no |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 165 / 165 | 165 / 165 | 0.30 | 0.33 | +0.03 [−0.38, +0.42] | 0.56 | 0.63 | +0.07 [−0.13, +0.27] | no |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 165 / 165 | 165 / 165 | 0.72 | 0.43 | −0.29 [−0.70, +0.13] | 0.69 | 0.58 | −0.11 [−0.33, +0.09] | no |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 165 / 165 | 165 / 165 | 0.55 | 0.36 | −0.19 [−0.57, +0.19] | 0.42 | 0.42 | +0.00 [−0.20, +0.20] | no |
| Mercury DecideClosed · EfficientLeague 1 group | 165 / 165 | 165 / 165 | 0.37 | 0.28 | −0.09 [−0.52, +0.33] | 0.84 | 0.83 | −0.01 [−0.23, +0.21] | no |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 165 / 165 | 165 / 165 | 0.46 | 0.31 | −0.15 [−0.53, +0.22] | 0.37 | 0.19 | −0.18 [−0.38, +0.02] | no |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 165 / 165 | 165 / 165 | 0.33 | 0.62 | +0.29 [−0.10, +0.67] | 0.53 | 0.59 | +0.06 [−0.13, +0.27] | no |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 165 / 165 | 165 / 165 | 0.35 | 0.44 | +0.09 [−0.30, +0.48] | 0.05 | −0.05 | −0.10 [−0.29, +0.10] | no |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 165 / 165 | 165 / 165 | 0.49 | 0.21 | −0.28 [−0.66, +0.11] | 0.12 | 0.11 | −0.01 [−0.19, +0.19] | no |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 165 / 165 | 165 / 165 | 1.33 | 1.74 | +0.41 [−0.06, +0.89] | 0.66 | 0.71 | +0.05 [−0.18, +0.29] | no |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 165 / 165 | 165 / 165 | 1.28 | 1.43 | +0.15 [−0.32, +0.61] | 0.68 | 0.72 | +0.04 [−0.18, +0.28] | no |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 165 / 165 | 165 / 165 | 1.08 | 1.22 | +0.14 [−0.31, +0.58] | 0.86 | 0.79 | −0.07 [−0.29, +0.16] | no |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 165 / 165 | 165 / 165 | 0.99 | 1.35 | +0.36 [−0.12, +0.89] | 0.99 | 1.02 | +0.03 [−0.21, +0.30] | no |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 165 / 165 | 165 / 165 | 1.05 | 1.15 | +0.10 [−0.50, +0.65] | 1.24 | 0.95 | −0.29 [−0.57, −0.02] | no |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 165 / 165 | 165 / 165 | 0.83 | 0.61 | −0.22 [−0.67, +0.20] | 0.68 | 0.62 | −0.06 [−0.27, +0.15] | no |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 165 / 165 | 165 / 165 | 0.25 | 0.20 | −0.05 [−0.52, +0.37] | 0.90 | 0.87 | −0.03 [−0.24, +0.19] | no |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 165 / 165 | 165 / 165 | 0.70 | 0.34 | −0.36 [−0.74, +0.00] | 0.20 | 0.19 | −0.01 [−0.19, +0.18] | no |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 165 / 165 | 165 / 165 | 0.20 | 0.60 | +0.40 [+0.01, +0.80] | 0.18 | −0.05 | −0.23 [−0.44, −0.03] | no |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 165 / 165 | 165 / 165 | 1.47 | 1.44 | −0.03 [−0.47, +0.39] | 0.62 | 0.56 | −0.06 [−0.27, +0.15] | no |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 165 / 165 | 165 / 165 | 1.47 | 1.71 | +0.24 [−0.21, +0.72] | 0.74 | 0.72 | −0.02 [−0.25, +0.22] | no |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 165 / 165 | 165 / 165 | 0.88 | 1.05 | +0.17 [−0.26, +0.60] | 0.70 | 0.68 | −0.02 [−0.23, +0.20] | no |
| GPT-6.1 Terrano group (exploratory) | 165 / 165 | 165 / 165 | 1.13 | 0.99 | −0.14 [−0.64, +0.36] | 0.91 | 0.86 | −0.05 [−0.31, +0.19] | no |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 165 / 165 | 165 / 165 | 0.98 | 1.18 | +0.20 [−0.21, +0.60] | 0.48 | 0.57 | +0.09 [−0.10, +0.29] | no |
| GNUbg weakyardstick | 165 / 165 | 165 / 165 | 1.04 | 1.16 | +0.12 [−0.27, +0.50] | −0.31 | −0.42 | −0.11 [−0.30, +0.09] | no |
Scope. These results hold for these rosters only; each group is the setups in the run from that original league.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Secondary, as fixed in §10.11. K = (Random's Nackgammon PR − setup's Nackgammon PR) ÷ (Random's standard PR − setup's standard PR): 1 = keeps its whole advantage over random play, 0 = falls to random. Unstable: standard advantage under 20 PR or its interval reaches 0. Risk: a class with fewer than 50 positions is thin and not read into.
Engine check
Swipe for more columns →
| Setup (group) | 2-ply: standard PR | 2-ply: Nackgammon PR | 2-ply: drop | Rollout: standard PR | Rollout: Nackgammon PR | Rollout: drop |
|---|---|---|---|---|---|---|
| GNUbg weakyardstick | 25.27 | 29.43 | +4.16 | 28.12 | 28.10 | −0.02 |
| celeris-1-decisionClosed · EfficientLeague 1 group | 19.94 | 53.19 | +33.24 | 20.90 | 51.27 | +30.37 |
| SubQ Snap PreviewClosed · EfficientLeague 1 group | 17.11 | 35.79 | +18.67 | 18.47 | 34.63 | +16.16 |
| CygnetNVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 12BLeague 1 group | 17.71 | 22.08 | +4.37 | 19.07 | 19.93 | +0.85 |
| Perplexity Decider 27B v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 15.09 | 20.47 | +5.39 | 16.17 | 19.36 | +3.19 |
| GLiDEv1.2 · NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 13.43 | 20.28 | +6.86 | 14.30 | 19.14 | +4.85 |
| DeepSeek-V4.1-Flashoff · API + NVIDIA RTX PRO 6000 Blackwell · first releaseOpen · 552B MoE · 16B activeLeague 2 group | 10.19 | 22.97 | +12.78 | 11.05 | 20.58 | +9.53 |
| Jev 1.13Closed · EfficientLeague 1 group | 14.66 | 25.27 | +10.61 | 15.82 | 23.79 | +7.97 |
| GPT-6 LunaDecisions APIClosed · EfficientLeague 1 group | 11.27 | 38.90 | +27.63 | 12.17 | 36.34 | +24.17 |
| Perplexity Decider 27B v1.1NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 1 group | 17.21 | 23.64 | +6.43 | 18.36 | 21.31 | +2.95 |
| LFM2.5-24B-A2Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 24B MoE · 2.5B activeLeague 2 group | 24.08 | 46.03 | +21.95 | 25.81 | 44.30 | +18.49 |
| Qwen3.8-27Boff · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 27BLeague 2 group | 9.63 | 23.71 | +14.08 | 10.52 | 22.07 | +11.55 |
| Mercury DecideClosed · EfficientLeague 1 group | 15.28 | 32.27 | +16.99 | 16.11 | 30.64 | +14.54 |
| Winnow-12BNVIDIA RTX PRO 6000 BlackwellOpen · 12BLeague 1 group | 19.40 | 40.94 | +21.54 | 20.82 | 39.56 | +18.74 |
| LFM2.5-24B-A2Boff · NVIDIA RTX PRO 6000 BlackwellOpen · 24B MoE · 2.5B activeLeague 2 group | 36.14 | 72.68 | +36.54 | 37.09 | 71.22 | +34.12 |
| Open-Jev-27B-v1.1NVIDIA RTX PRO 6000 BlackwellOpen · 27BLeague 1 group | 51.03 | 54.35 | +3.31 | 52.10 | 52.82 | +0.72 |
| Perplexity Decider Flash v1.2NVIDIA RTX PRO 6000 BlackwellOpen · 4.0BLeague 1 group | 29.51 | 63.81 | +34.31 | 32.15 | 60.75 | +28.60 |
| Gemini 3.8 FlashlowClosed · WorkhorseLeague 2 group | 15.21 | 10.19 | −5.02 | 16.33 | 8.38 | −7.95 |
| Claude Sonnet 5.5lowClosed · WorkhorseLeague 2 group | 12.96 | 18.62 | +5.66 | 14.17 | 17.06 | +2.89 |
| GPT-6.1 SollowClosed · WorkhorseLeague 2 group | 13.56 | 10.52 | −3.04 | 15.09 | 8.70 | −6.39 |
| GPT-6 AstramediumClosed · FlagshipLeague 3 group | 8.46 | 7.75 | −0.71 | 9.60 | 6.25 | −3.35 |
| GPT-6.1 SolhighClosed · WorkhorseLeague 2 group | 11.73 | 10.46 | −1.28 | 12.53 | 8.87 | −3.66 |
| Gemma 4 31Bon · NVIDIA RTX PRO 6000 Blackwell + NVIDIA DGX SparkOpen · 31BLeague 2 group | 14.93 | 18.07 | +3.13 | 16.05 | 16.73 | +0.69 |
| GPT-6.1 Terrano group (exploratory) | 11.43 | 8.61 | −2.81 | 12.64 | 6.74 | −5.90 |
| Claude Fable 5.1highClosed · FlagshipLeague 3 group · added under §10.14(g), after the results were read | 12.19 | 8.18 | −4.01 | 13.13 | 6.25 | −6.88 |
| Grok 4.7highClosed · WorkhorseLeague 3 group · added under §10.14(g), after the results were read | 11.77 | 16.37 | +4.60 | 12.82 | 14.18 | +1.36 |
| d1-ultraClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 13.92 | 25.38 | +11.46 | 14.94 | 21.96 | +7.03 |
| GLiDEAPIClosed · EfficientLeague 1 group · added under §10.14(g), after the results were read | 13.46 | 22.48 | +9.02 | 14.23 | 21.32 | +7.08 |
| Gemini 4 ArgonlowClosed · Flagshipno group (exploratory) | 8.73 | 11.33 | +2.60 | 9.04 | 9.02 | −0.02 |
| Randomyardstick | 83.84 | 93.55 | +9.72 | 86.10 | 92.16 | +6.06 |
Swipe for more columns →
| Comparison | 2-ply difference | 99% interval | Rollout difference | 99% interval | Direction changed |
|---|---|---|---|---|---|
| League 1 group − League 2 group | +5.57 | [−2.33, +14.99] | +5.35 | [−2.63, +15.07] | no |
| League 1 group − League 3 group | +15.03 | [−1.57, +36.32] | +14.90 | [−1.73, +35.94] | no |
| League 2 group − League 3 group | +9.46 | [−1.84, +23.64] | +9.55 | [−1.79, +23.36] | no |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | −14.59 | [−55.64, +18.77] | −15.63 | [−56.64, +16.95] | no |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | +1.76 | [−2.81, +6.99] | +2.73 | [−1.97, +8.46] | no |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Swipe for more columns →
| Comparison | 2-ply difference | 99% interval | Rollout difference | 99% interval | Direction changed |
|---|---|---|---|---|---|
| League 1 group − League 2 group | +6.36 | [−3.28, +18.64] | +6.17 | [−3.57, +18.72] | no |
| League 1 group − League 3 group | +16.49 | [−1.68, +40.04] | +16.11 | [−2.20, +39.49] | no |
| League 2 group − League 3 group | +10.13 | [−1.63, +24.58] | +9.94 | [−1.65, +24.21] | no |
| LFM2.5-24B-A2Bon − LFM2.5-24B-A2Boff | −14.59 | [−55.64, +18.77] | −15.63 | [−56.64, +16.95] | no |
| GPT-6.1 Solhigh − GPT-6.1 Sollow | +1.76 | [−2.81, +6.99] | +2.73 | [−1.97, +8.46] | no |
Groups by original league (Method §10.14, amendment of 2026-10-27). Method §10 and its notes were committed before any model call, and the per-setup first looks were computed under the frozen extension rule. Separately, before the amendment, six full-report calls were issued on partial data under the earlier group definitions; that all six completed successfully has not been independently established. One returned numeric old-group primary results into the site agent's context before the amendment. Whether a person saw them, and any causation, is not established. Nothing was published before the amendment; the aggregate report and data were pushed to the public repository in commit 3e4f413 on 2026-10-28 (01:03 UTC), before release approval. See §10.14(e) and its dated correction (f) of 2026-10-28.
Original members only: the League 3 group has 1 setup (GPT-6 Astramedium); comparisons 2 and 3 rest on it.
Swipe for more columns →
| Subset | Pairs | Plays rolled out | Unrolled plays used (2-ply + shift) |
|---|---|---|---|
| Engine-check subset | 40 | 442 | 441 |
Settings: GNUbg 1.08.003 rollout: 1,296 trials per play, cubeless, variance reduction on, quasi-random dice, no truncation, no late evaluation, chequer play 1-ply, seed 1, 8 threads, nice 19; candidates: top 5 at 2-ply plus every play any setup chose. The additions' chosen plays that had no rollout yet were rolled out later with the same settings, alone rather than alongside the other candidates of their positions; the earlier rollouts are unchanged.
No primary comparison changes direction between 2-ply and rollout values, either way. GNUbg's network learned standard play, so its values for early Nackgammon positions are likely less accurate; the check bounds that, it does not remove it.
Scope. These results hold for these rosters only; each group is the setups in the run from that original league.
The League 3 group currently has 3 setups (GPT-6 Astramedium, Claude Fable 5.1high, Grok 4.7high); comparisons 2 and 3 rest on them.
Data: nack-paired.json · Method (Method spec 2026-10-24 §10 (groups: §10.14))
Finished. The knockout: Leagues.