Pre-registration · 2026-11-08

Did they improve? Pre-registration (2026-11-08)

← Results: Did they improve?

The plan of "Did they improve?" as registered, committed before the first model call; the text below is the file's, as written (names of our servers and GPUs read "our GPU").

File: Did they improve - plan 2026-11-08.md, sha256 85a970d09cf1ae40d5567a7540ec0f607133314589119391bd71584b5ef78542

This is the pre-registration. It is committed and pushed before the first model call of this experiment, and the page "Did they improve?" shows its rule, lesson prompt, model list and commitment line before that call.

Results will be published whichever way they go.

Sources: Ümit's brief (#2954, #2955), the plan draft (mailbox/Did they improve - plan draft 2026-11-08.md, 0567bb2), the proposal (mailbox/Did they improve - proposal 2026-11-08.md, 3f23da6), Zeki's 13 points (#3052) and clarifications (#3057, #3066, #3077), Ümit's decisions #3060, #3065 and #3084, Can's top-up note #3078. Where they differ, the later decision wins; this file states the result. The learning pilot (mailbox/Learning pilot - plan 2026-11-07.md, mailbox/Learning pilot - results 2026-11-07.md) supplies the lesson prompt, the test set, the dice and the harness rules reused here.

1. Question

Does each chat model on the leaderboard take risks better on the pilot's 200-position test after writing its own lessons over 10 games against GNU Backgammon Beginner (GNUbg weak), compared with answering the same test with no memory?

2. The rule (verbatim; decided by Ümit, #3060 and #3065)

Improved = both of these hold:

  1. risks found with lessons (the mean of the 5 tests after game 10) minus risks found with no memory (the mean of the 3 empty runs) is more than max(range of the model's own 3 empty runs, 5 points);
  2. right safety with lessons minus right safety with no memory is ≥ −5 points (right safety drops by no more than 5 points).

Examples (from #3057, with the 5-point floor): empty runs 28, 28, 28 (range 0) → threshold 5; with lessons +4 → No; +5.5 → Yes if right safety fell by at most 5. Empty runs 24, 27, 30 (range 6) → threshold 6; +5 → No; +7 → Yes.

3. Models

Rule: every chat model on the live leaderboard, one row per model, at its leaderboard setting (the setup that names the row: highest reasoning setting, late entries included). Each runs with its v3.db players.settings unchanged except name, parallel, prompt_note_file and, for local servers, the server address.

15 chat models (in scope):

#Model (v3.db setup)MakerSettingRouteStatus (8 Nov)
1Qwen3.8 FlashoffAlibaba (Qwen)offOpenRouter → Alibaba, fallbacks offwaiting: first in order
2GLM-5.3 (on)Z.ailowOpenRouter → Fireworks, fallbacks offwaiting
3Kimi K3.1 ProoffMoonshotoff (reasoning_effort: none)OpenRouter → moonshotai/mxfp4, fallbacks offwaiting
4DeepSeek-V4.1-Flash (on) — tested checkpoint: original V4.1 FlashDeepSeeklow (the leaderboard row's setting)Fireworks — API setup: OpenRouter id deepseek/deepseek-v4.1-flash (canonical slug deepseek/deepseek-v4.1-flash-20260910, Hugging Face deepseek-ai/DeepSeek-V4.1-Flash; Fireworks deepseek-v4p1-flash), pinned to provider tag fireworks, fallbacks off, require_parameters: true, $0.30/$1.20 per 1M (read 2026-11-08). Same model and settings as the leaderboard row's setup "DeepSeek-V4.1-Flash (on)" (reasoning effort low, letter read from the answer's logprobs, top_logprobs 5); its league decisions ran mostly on this route (2,178). Every response's model and provider fields are recorded; the run stops if they do not name the original V4.1 Flash on Fireworks (not 1017; no substitution). Why this route: Ümit chose the original checkpoint on Fireworks at setting low (#3099, #3102), because the leaderboard row's 2,178 Fireworks decisions are the original V4.1 Flash on Fireworks.waiting
5Gemini 3.8 FlashhighGooglehighAntigravity subscriptionwaiting
6Claude Sonnet 5.5highAnthropichighClaude Code subscriptionwaiting
7Claude Opus 5.5 (on)AnthropichighClaude Code subscriptionwaiting
8Gemini 4 Argon (on)GooglehighAntigravity subscriptionwaiting
9Grok 4.7 (on)xAIhighxAI subscription (api.x.ai)waiting
10Claude Fable 5.1 (on)AnthropichighClaude Code subscriptionwaiting
11Gemma 4 31BonGoogleon (thinking)local vLLM, same image, checkpoint and arguments as the leaguewaiting for a GPU
12Qwen3.8-27B (on)Alibaba (Qwen)lowlocal vLLM (tools/v3/servers/qwen38-27b.sh)waiting for a GPU
13LFM2.5-24B-A2BonLiquid AIonlocal vLLM (tools/v3/servers/lfm25-24b.sh)waiting for a GPU
14GPT-6.1 SolhighOpenAIhighChatGPT Pro subscription (Codex backend)blocked until the weekly reset 2026-11-13 11:38 UTC
15GPT-6 Astrahigh, late entryOpenAIhighChatGPT Pro subscription only (paid API on hold)blocked until 2026-11-13 11:38 UTC; the Astra (high) re-ask is cancelled, so this experiment uses the allowance

Decision models: not tested: cannot write lessons. They return probabilities over the options and cannot write text. The 22: d1-ultra, Mammoth-1, Elephant-1, Drex Max, Drex Max v1.1, Drex 2.0 (and its local setting), REAX 1.3, GLiDE, GLiDEv1.2, Jev 1.13, Jev 1.14, Open-Jev-27B-v1.1, Perplexity Decider 27B v1.2, Perplexity Decider 27B v1.1, Perplexity Decider Flash v1.2, celeris-1-decision, Clef, SubQ Snap 1.1, SubQ Snap Preview, GPT-6 Luna, Winnow-12B, Mercury Decide.

Cygnet: not tested (Ümit #3111), listed with the decision models: it plays through a layer that never sees the lessons (decision models can't write lessons). Not replaced by another model. Scope = the 15 chat models above.

The Gemma pilot row: "pilot (different design)". Gemma 4 31Boff, from the learning pilot; no new calls.

4. Sampling and settings: what each route accepts

Ümit's decision (#3060): each model at its league settings, unchanged; no forced temperature. The page says: "no shared seed; same settings as the leaderboard". No route is sent a seed (none is sent in the league).

Route (models)League request (and this experiment's) sendstemperatureseedSource (read 2026-11-08)
ChatGPT subscription, Codex Responses (Sol, Astra)reasoning.effort: high; no temperature, no top_p, no seed, no output capnot accepted when effort ≠ noneno seed parameterhttps://developers.openai.com/api/docs/guides/latest-model ; https://developers.openai.com/api/reference/resources/responses/methods/create
Claude Code subscription, Messages API (Fable, Opus, Sonnet)output_config.effort: high, max_tokens: 32000; no temperaturedefault (1.0) only on Claude 4.7 and laterno seed parameterhttps://platform.claude.com/docs/en/about-claude/model-deprecations ; https://platform.claude.com/docs/en/api/migration-2026-10-21#sampling-parameters
Antigravity subscription, generateContent (Argon, Gemini 3.8 Flash)temperature: 1.0, thinkingConfig.thinkingLevel: highacceptedaccepted on the public API (random when unset); not senthttps://ai.google.dev/api/generate-content
xAI subscription, Responses (Grok 4.7)reasoning.effort: high; no temperatureaccepted (0–2)no seed fieldhttps://docs.x.ai/developers/rest-api-reference/inference/responses ; https://docs.x.ai/developers/model-capabilities/text/reasoning
OpenRouter → Fireworks (GLM-5.3, DeepSeek-V4.1-Flash)temperature: 1.0, reasoning_effort: low, provider pinned, require_parameters: true, allow_fallbacks: false, usage.includeacceptednot supported by the endpointGET https://openrouter.ai/api/v1/models/<id>/endpoints (supported_parameters) ; https://openrouter.ai/docs/api/api-reference/models/get-models
OpenRouter → moonshotai/mxfp4 (Kimi K3.1 Pro)temperature: 1.0, reasoning_effort: none, provider pinned, fallbacks offacceptednot supportedsame listing
OpenRouter → Alibaba (Qwen3.8 Flash)temperature: 1.0, reasoning.enabled: false, provider pinned, fallbacks offacceptedaccepted; not sentsame listing
local vLLM (Gemma 4 31B, Qwen3.8-27B, LFM2.5)temperature: 1.0, the setup's template switches and max_tokensacceptedaccepted; not senthttps://docs.vllm.ai/en/stable/api/vllm/sampling_params/

The subscription routes use backends whose parameter handling is not documented (ChatGPT backend-api/codex, Claude Code OAuth, Antigravity v1internal, xAI with OMP's login); they get exactly the league's request.

5. Design

Per model, the lessons condition only:

Order of one model's steps: E1 → S1 (+ its T10) → cost check (§9) → S2 → E2 → S3 → S4 → E3 → S5.

Lesson call (Ümit #3060).

Test: the pilot's 200 positions (100 where the risk pays, 100 where safety pays), test-set sha256 6c92e7416745125ca3ea09c9e41e450a52716f8ad018772278b07f992f7318e6 (test_set.json; positions file test_positions.jsonl sha256 ff509ea102b4fcd2faca3402d585b05ba6d7dc0014a7c5f9dabbe48b13a3467f). Asked with jevgammon ask (the league Player, prompt bg-v2, League 2's option-order seed 1190975713392774006). Test positions are never published.

Dice: the pilot's series seeds s01–s05 (learning-pilot 2026-11-07 series k, sha256-derived), the same for every model; games 2k−1 and 2k are a mirrored pair (--fixed-rounds).

Isolation:

Concurrency (recorded per request): games and lesson calls run with 1 request in flight; each test (empty or T10) runs alone for that model with 4 decisions in flight; one model's steps run one at a time. Every stored request records how many of this experiment's decisions were in flight for that model when it was sent. Provider-side load is not observable, and the page says so.

6. Scores per model

7. Completion and verdict gate

A model gets yes/no only when all of these are complete:

Until then its row shows live numbers and "running, n of 5 series", with no verdict text. A model that cannot complete is shown "incomplete" with the reason, and never gets a verdict from partial data.

8. Failed and invalid answers

9. Money (verbatim rules; Ümit #3065, Can #3129, Zeki #3085)

Billed money: OpenRouter only (Qwen3.8 Flash, GLM-5.3, Kimi K3.1 Pro, DeepSeek-V4.1-Flash). Subscriptions are $0 billed (quota, shown as API-list-price equivalent); local runs are $0 API spend.

Stop rules (Ümit #3065, verbatim):

  1. "Spending approved up to $110 billed total from the shared OpenRouter balance (cap, not target). Logic/turkce are not part of it."
  2. "NO per-model approvals. Cheapest first. After each model's 1-series cost check, continue automatically UNLESS (a) its projected full cost > 1.5× its estimate (use the per-model estimates in the plan draft table), or (b) the running billed total would pass $110. If either fires, stop that model and report why."

Funding (Can #3129): "don't calculate the money, we'll fund it; just keep it cheap." There is no interim cap below $110 (the earlier interim caps are withdrawn). Routes stay the league's pinned providers (the same model and settings as the leaderboard); a cheaper provider is used only where the model and settings are identical.

Share of the shared balance (Main, 2026-11-08 15:08 UTC): the OpenRouter balance ($110.62 then) is shared with logic (≤ $20) and turkce (≤ $32); this experiment's cumulative billed spend is limited to $55 until Main raises it (enough for Qwen3.8 Flash, GLM-5.3 and Kimi K3.1 Pro; DeepSeek waits for the next top-up). Reaching it is a pause, not a stop: the run pauses at the step boundary (or when the step ledger refuses a request because of it), Main is told, Can tops up on request, and the run resumes where it paused. The share is a runner setting (improve.py share-cap), logged with each change.

Balance check (pause, not fail): before every billed step the runner reads OpenRouter's live balance (GET /api/v1/credits: total credits − total usage). If balance − $5 reserve does not cover the step's expected cost, the run pauses at that step boundary (logged, reported to Main so Can can top up) and resumes from that step when the balance covers it. Money that is not in the account is never spent; a top-up instruction is not money, only the read balance counts.

Per-model estimates (plan draft §7) and the stop limits derived from them:

ModelEstimate (full run)1.5× (rule a)1-series step S (estimate)Empty-run step E (estimate)
------:---:---:---:
Qwen3.8 Flashoff$2.30$3.45$0.35$0.18
GLM-5.3 (on)$22.00$33.00$3.35$1.75
Kimi K3.1 Prooff$35.00$52.50$5.34$2.77
DeepSeek-V4.1-Flash (on), original, via Fireworks$44.00$66.00$6.96$3.07

E = (full − 5 × S) / 3, the draft's estimate split over its steps.

How the rules are applied (hard stops, logged):

10. Schedule

11. Outputs

Private (never committed): smoke/improve/<model>.db (games, test answers, every request), smoke/improve/<model>-plays.db, workspace/improve/runs/<model>/ (lessons, contexts, logs, ledgers). Public via the page: per-model aggregates, lesson texts, the rule, the lesson prompt and its hash, and the money record. Test positions and per-position answers are never published.

Results will be published whichever way they go.