Pre-registration · addendum · 2026-11-09
Decision models (added 9 Nov)
Results will be published whichever way they go.
The decision models are pre-registered in an addendum to the plan, committed before their first call. Its rule, cohort, writer, money and open decisions follow as written (sections 1, 2, 3, 4, 7, 9).
Addendum: Did they improve - decision models - addendum 2026-11-09.md, sha256 b370f4dc9660a618aece4586b8c8f5259688372fab580305b4454aaa2d71a967
1. Question
For each decision model of the cohort (§3), at its leaderboard setting: does it take risks better on the pilot's 200-position test after 10 games against GNUbg weak (GNU Backgammon 1.08 Beginner), with a block of text about its own earlier games in its prompt, than with no block?
There are two arms, each judged on its own:
- Arm N, helper-written notes → decision player. After each game a fixed helper model (the writer, §4) reads the decision model's graded game record and writes notes with the pilot's lesson prompt and limits. The page says: helper-written notes, not self-written learning.
- Arm W, worst mistakes (no writer). After each game a fixed script puts the 5 costliest decisions of this series so far (GNUbg grades) into the prompt as solved examples (§5).
Each arm has its own games: same dice schedules, never the other arm's games, decisions or trajectories. Notes and examples reset at the start of every series. Games count nowhere (no league, no leaderboard).
Verdicts are reported per arm, under the same rule. Neither arm is picked as "the" result.
2. The rule (verbatim from the live plan §2; decided by Ümit, #3060 and #3065)
Improved = both of these hold:
- risks found with lessons (the mean of the 5 tests after game 10) minus risks found with no memory (the mean of the 3 empty runs) is more than max(range of the model's own 3 empty runs, 5 points);
- right safety with lessons minus right safety with no memory is ≥ −5 points (right safety drops by no more than 5 points).
- Range = max − min of risks found over the model's 3 empty runs, in percentage points.
- The 5-point floor = 5 of the 100 positions where the risk pays. The Gemma pilot showed about 3 points of drift from re-asking alone.
- This is an operational threshold, not a statistical test and not statistical evidence. The page says so.
- Error rate (PR) is shown before and after; it is not part of the rule.
For Part 3, "with lessons" means "with the arm's block" (notes in Arm N, worst mistakes in Arm W). "With no memory" means that arm's own empty runs (§6).
3. Cohort (Ümit #3744, #3746 and #3758)
Each row is one model at its leaderboard setup, frozen from v3.db players.settings: setup name, route, model id sent, request parameters and option cap. The only changes are name, parallel (never above the league's), the block file, this experiment's budget ledger (never the league's), the route check and the context gate.
The allowed response ids are the ids observed in the league's own replies. There is no successor, alias or substitute (D7). A reply naming another id stops the arm: "not tested: model id not served as on the leaderboard". There is no hosted/local swap.
| Setup (v3.db) | Chosen by | Route | Response id | Delivery evidence | Status before preflight |
|---|---|---|---|---|---|
| REAX 1.3 | Ümit #3744 | local reax13/serve.py (reax-engine @ ff8f595) | reax13 | pending Lr | needs D5 (read-twice → read-once switch); local render check; a GPU |
| GLiDEv1.2 | Ümit #3744 | local glide/server.py | glide-v1.2 | pending Lr | local render check; a GPU |
| d1-ultra | Ümit #3744 | api.liquid.ai/decisions/v1/systemone, d1-ultra | d1-ultra | delivery unverified (our evidence gap) | selected and shown; not tested: native context limit unverified (Zeki #3761; §6.1, §6.2) |
| Drex Max v1.1 | Ümit #3744 | Nace console.nace.ai/v1/systemone, drex-max-v1.1, 25 options | drex-max-v1.1 | Hc | P1, then runs; Nace subcap (§7) |
| d3-ultra | Ümit #3744 | — | — | — | not started: added 9 Nov, no league record, no backgammon route yet |
| Mammoth-1 | Ümit #3744 | local baselabs_server.py (baselabs-serve:81ce62f), 120 options | baselabs/mammoth-1 | pending Lr | needs D5 (both arms past the trained 16,384 tokens); local render check; a GPU |
| Elephant-1 | Ümit #3746 | local, as Mammoth-1 | baselabs/elephant-1 | pending Lr | needs D5 for Arm N (past the trained 16,384 tokens); local render check; a GPU |
| Perplexity Decider 27B v1.2 | Ümit #3744 | local pplx_decider_server.py (decider-v1.2.sh 27b), 25 options | pplx-decider-v1.2 | pending Lr | local render check; a GPU |
| Jev 1.14 | Ümit #3744 | api.typesafe.ai/v1/systemone, jev-1.14.0 | jev-1.14.0 | Hc | not tested: conservative context guard (§6.2: the guard refuses 1.3% of its league requests even without a block) |
| SubQ Snap 1.1 | Ümit #3758 | api.subq.ai/v1/systemone, subq-snap-1.1 | subq-snap-1.1 | Hc | P1, then runs |
Every decision model of the leaderboard outside this list is out of scope for Part 3. Its row stays as the live plan states it.
Page notes:
- Base Labs: "runs past the checkpoint's trained 16,384 tokens (the engine refuses above 32,768)".
- REAX: "reads requests of at most 3,000 tokens twice; with a block every request is longer and is read once".
4. The writer (Arm N only)
Qwen3.8 Flashoff via OpenRouter, pinned once for the whole cohort, no swap mid-cohort. It is the live plan's Qwen3.8 Flashoff lesson call, unchanged (improve.py player_config of that setup):
- spec
openai-compatible/qwen/qwen3.8-flash,https://openrouter.ai/api/v1; - request
provider: {order: ["alibaba"], allow_fallbacks: false, require_parameters: true},usage: {include: true},reasoning: {enabled: false}; - the runner's output allowance of 32,768 tokens;
parallel1;- route check: response
model∈ {qwen/qwen3.8-flash,qwen/qwen3.8-flash-20260826} and providerAlibaba; anything else stops the arm.
Prompt: the pilot's lessons_system.txt + lessons_user.txt, byte-identical (sha256 d755001513b60546cbcc4159db0965dbeda09a65d65377d874b22e247f74550e).
Cut rule (the pilot's): lines starting - are lessons; at most 5 lessons and 150 words are kept; cuts are counted and shown. The writer's raw reply, the kept notes and the exact block delivered (by sha256) are stored privately. After the cut rule nothing in a block is ever shortened.
What the writer reads: the live plan's lesson record (improve.py record_line, game_header, eligible) of that model's game in Arm N, scored by GNUbg 1.08 2-ply. Left out:
- forced and impossible moves;
- invalid answers;
- unscored decisions;
- GNUbg's moves;
- the model's probabilities;
- any decision whose
board_keyis among the frozen 200 test positions (test-set sha256 6c92e7416745125ca3ea09c9e41e450a52716f8ad018772278b07f992f7318e6), removed before the record is built, counted and shown.
The block (Arm N), at the start of state (OpenAI: input), then a blank line, then the league state unchanged:
=== NOTES FROM YOUR EARLIER GAMES IN THIS SERIES === Notes a helper wrote from your earlier games in this series, oldest first: After game 1: <kept notes> … === END OF NOTES ===
Writer calls are booked in the same ledgers as the players.
7. Money
- The $110 total cap and the $5 OpenRouter reserve stay (live plan §9). Every decision-model ledger counts against the $110: players, writer, P1 probes, retries, and every open reservation at its worst case (request bytes as input tokens + 4,096 tokens on decision routes at list price; the writer's 32,768-token output allowance). A request with no response is booked at its worst case. Billed = max(provider-reported cost, tokens × list price).
- Rule (a) per arm: after E1 and S1, a projected cost above 1.5 × the arm's estimate stops that arm.
- D6 approved (Ümit, native message 1569386890212147201, relayed by Zeki #3753): hosted spending outside OpenRouter counts within the existing $110 total cap. The $5 OpenRouter reserve and the 1.5× per-model rule are unchanged. The cohort file sets the hosted allowance of TypeSafe, Liquid, Nace and SubQ to "within the $110". A $0 there would refuse every billed call of that provider, P1 included.
- Nace subcap: $2.50 (raised from $1.25 by the same approval). The league's shared Nace ledger (
workspace/drex/state/nace-budget.db, $0.7856 settled) still counts, so $1.71 is left. Drex Max v1.1's two arms book about $0.89 + $0.88 at the $0.06 list price; Nace bills its $0.03 launch price. The second arm may therefore stop at the cap, shown as "incomplete". - Estimates (booked, separate empty runs, both arms; method of the draft §7):
- SubQ Snap 1.1 $0.99;
- Drex Max v1.1 $1.74;
- Jev 1.14 $1.08 and d1-ultra $0.94 if they ever run (not tested now: $0);
- writer: Arm N of each running model ≈ $0.03 each (about $0.21 for the 7 that can run: SubQ, Drex Max v1.1 and the 5 local models);
- P1 < $0.01 per hosted route.
9. Decisions
- Decided (from the draft):
- D3 placement: start of
state/input; - D4 wording: the pilot prompt byte-identical, the intro line above;
- D7 identity: no successor ids;
- D9 Arm W content: board, chosen play, best play, loss;
- D10 delivery evidence classes.
- D3 placement: start of
- Decided by Ümit: the cohort (#3744, #3746, #3758), including running d1-ultra with its flag; D6 (native message 1569386890212147201).
- Proposals pending Ümit (the harness reads them from
workspace/improve/decision_cohort.json, so an answer changes that file, not the code):- D1 empty runs: separate per arm (the proposed default, used unless Ümit changes it; the page says "proposed default, Ümit may change") or shared;
- D5 edge settings: Base Labs past the trained length, REAX's read-once switch; until decided these arms are pending.
Results will be published whichever way they go.