Highlights

Highlights

Findings the site already shows, each with its numbers and the page it comes from.

Results f24d1f41, updated 16 November 2026, 21:24 UTC; scored by GNU Backgammon 1.08, 2-ply.

Did they improve?

Preliminary · running

Each chat model wrote its own lessons over its games against GNUbg weak, then took the risk test again. Yes or No is the pre-registered rule's: risks found must rise by more than the model's threshold while right safety drops by no more than the plan allows; an operational threshold, not a statistical test. The other models are still running; their numbers are not results yet.

Risks found before → after, finished models: GLM-5.3low 21.3 → 39.0: Yes · Claude Fable 5.1high 54.0 → 63.4: Yes · Gemini 4 Argonhigh 58.6 → 65.0: No(original run; after QC, Run 2 (E1, S1, S2 replayed): +8.5 pp, now Yes) · Gemma 4 31Bon 34.3 → 42.8: Yes · DeepSeek-V4.1-Flashlow · first release 44.6 → 51.8: Yes(original run; after QC, Run 2 (S1 replayed): +8.2 pp, still Yes) · Gemini 3.8 Flashhigh 38.0 → 46.2: Yes · Claude Sonnet 5.5high 43.0 → 50.0: Yes · Claude Opus 5.5high 47.3 → 54.0: Yes · Qwen3.8-27Blow 32.0 → 51.0: No · Kimi K3.1 Prooff 41.0 → 45.8: No · Qwen3.8 Flashoff 33.3 → 36.0: No · Gemma 4 E4Bon 27.7 → 26.6: No

Did they improve?

Models are too cautious

Most ranked models found a worthwhile risk in a smaller share of the positions where risk paid than the share of positions where safety paid that they played safe. GNUbg weak, a weak setting of the engine, is close to even. Not paired: models met different positions in different leagues.

56 of 57 ranked models · risks found 18.7% (Mercury Decide) to 53.3% (GPT-6 Astrahigh, late entry) · safe right 16.3% (GLiClass v4 EdgeNVIDIA DGX Spark (GB10), late entry) to 97.6% (Gemini 3.8 Flashhigh) · GNUbg weak: 66.6% risks found, 67.6% safe right

The finding on the leaderboard page

Decision models and chat models at the top of the leaderboard

The leaderboard ranks one row per model by risk judgement (higher is better). The best-ranked decision model sits below the top chat model. Models played in different leagues, and a late entry played its league's field after the league finished, so the rows compare only loosely.

GPT-6 Astrahigh (chat model, late entry): rank 1, risk judgement 74.4 · Gemini Decision 120261111 (decision model, late entry): rank 12, risk judgement 64.7

The leaderboard

What thinking adds

The same model at its lowest and its highest reasoning setting, in the same league. Error rate fell for more of them than risk judgement rose. Not paired: each setting played its own games; these are observed differences.

9 models · lower error rate at the highest setting: 9 of 9 · higher risk judgement: 7 of 9 · GPT-6 Astra, low to high (high: late entry): risk judgement 75.8, 74.7, 74.4; error rate 11.8, 11.1, 8.8

The finding on the leaderboard page