E7: 20x long-horizon analysis — H-slow rejected (4/4 cells)

1 files changed, 207 insertions, 0 deletions

diff --git a/NOTES-E7.md b/NOTES-E7.md
new file mode 100644
index 0000000..a0122fa
--- /dev/null
+++ b/NOTES-E7.md
@@ -0,0 +1,207 @@
+# E7 — 20× long-horizon test (4M steps) — analysis notes (4/4 cells)
+
+Sources: `runs/e7/<job>/<model>/seed0/results.json` (eval run after each cell's
+training), `metrics.csv` (per-eval curves, 530 evals each), and the locked interpretation
+clauses in `design/preregistration.md` **Addendum 7** (the 20× budget test) plus Addenda 1–4
+(unchanged O/H/P operationalizations) and Addendum 6 (P-ladder / sieve-rank).
+
+**Batch status at analysis time (2026-08-29): all 4 cells finished** (wd {0.1, 0.3} ×
+{rnn, transformer}, seed 0, `max_train_steps = 4 000 000`, i.e. 20× the 200k budget of E6).
+Everything else matches E6: digits mode, task `next_prime`, range [2, 1000], `train_frac`
+0.7, lr 1e-3 const, d_model 128, K = 32, AMP fp16 on CUDA for the transformer cells, RNN
+cells on CPU fp32, adaptive eval schedule → 530 evals.
+
+Constants used throughout: val EM is exact-match; "best" is the val-selected checkpoint
+(selection-holed), "last" is the unselected final checkpoint. **The E7 probe is exact-match
+on [1001, 2000]** (out of training range; `range_end = 1000`). Note that this is the gap that
+was open in NOTES-E6 — `src/eval.py` now emits `probe_report(..., lo=range_end+1,
+hi=range_end+1000)` and a per-model `halting` report, so the committed `results.json` probe
+and halting fields are self-consistent for E7 (see Caveats §1).
+
+---
+
+## Per-cell classification (all 4 cells)
+
+| job | model | O | H | halt mean | val EM best | val EM last | train EM (max/final) | probe [1001,2000] acc | P | best-step |
+|-----|-------|---|-----|-----------|-------------|-------------|----------------------|-----------------------|---|-----------|
+| e7-wd01 | rnn | O-PARTIAL | H1 | 2.00 | 0.8367 | 0.8033 | 1.000 / 1.000 | 0.012 (12/1000) | P4 | 2 650 000 |
+| e7-wd01 | transformer | O-PARTIAL | — | — | 0.8233 | 0.7867 | 1.000 / 1.000 | 0.007 (7/1000) | P4 | 37 000 |
+| e7-wd03 | rnn | O-PARTIAL | H1 | 2.03 | 0.8133 | 0.8000 | 1.000 / 1.000 | 0.030 (30/1000) | P4 | 455 000 |
+| e7-wd03 | transformer | O-PARTIAL | — | — | 0.8433 | 0.7700 | 1.000 / 1.000 | 0.008 (8/1000) | P4 | 43 000 |
+
+Halt means are from the committed `halting` report on the final checkpoint (RNN cells only;
+the transformer has no ACT halting, so `mean_halt_steps` is NaN throughout its metrics.csv and
+no `halting` field is emitted).
+
+---
+
+## O-codes — no O1 anywhere (no grokking), all four O-PARTIAL
+
+**LOCKED** (Addendum 7): "O1 anywhere = GROKKING CONFIRMED on next-prime: transition exists,
+was budget-limited at 200k. Record transition step."
+
+**Commentary.** None of the four cells is **O1**. All four are **O-PARTIAL** by the unchanged
+Addenda 1–4 definition: train saturated (train EM = 1.0 sustained; `sat_start_eval` 24 / 2 /
+73 / 3 for wd01-rnn / wd01-transformer / wd03-rnn / wd03-transformer respectively), val never
+reached 0.9, and final val sits in (0.3, 0.9). The signature detector concurs: every
+`signature` has `code = "O-PARTIAL"`, `val_hi_eval_idx = null`, `transition_width_evals =
+null` — i.e. **no sharp val transition to ~1.0 exists in any cell**, at any of the 530 evals
+across 4M steps.
+
+**Transition-step search.** The maximum val EM any cell ever reached, and the step it reached
+it:
+
+| cell | max val EM ever | step of max |
+|------|-----------------|-------------|
+| e7-wd01/rnn | 0.8367 | 2 650 000 |
+| e7-wd01/transformer | 0.8233 | 37 000 |
+| e7-wd03/rnn | 0.8133 | 455 000 |
+| e7-wd03/transformer | 0.8433 | 43 000 |
+
+No cell ever logged a val EM ≥ 0.9 at any eval (0 of 530 evals per cell). The 20× horizon did
+not produce a grokking transition; the ceiling on in-range generalization is essentially the
+same as E6's (best val 0.813–0.844 vs E6's 0.823–0.840), and two of the four cells' bests
+occur in the first ~450k steps (i.e. inside a horizon E6 already explored).
+
+---
+
+## P-codes — all P4, and no sieve (4/4)
+
+**LOCKED** (Addendum 6, unchanged): probe range [1001, 2000]; P5(k) = errors concentrated on
+the rank-k signature set; P6 = exact (no probe misses); P3 threshold ≥ 90%; P4/P2 keep their
+meanings. A model that internalized a k-prime sieve misses exactly the composites whose prime
+factors all exceed p_k (smallest missed composite → k: 1147→10, 1369→11, 1681→12, 1849→13,
+none→k≥14).
+
+**Commentary.** All four cells are **P4**: probe accuracy on [1001, 2000] is 0.007–0.030
+(7–30 of 1000 correct). The failure mode is unchanged from every prior phase — the cells fail
+"the next prime after an even number is odd" essentially always, and emit composite or
+nonsense outputs (e.g. wd01-rnn answers 1040→1439, 1082→10099; wd03-rnn answers 1003→1093;
+wd01-transformer answers 1049→353; wd03-transformer answers 1001→3). The smallest composite
+each cell emits as "prime" in range has small prime factors (e.g. 1003 = 17·59, 1001 =
+7·11·13), so there is **no divisibility structure at any rank** — the P-ladder resolves to
+**"no sieve" (k undefined) in all 4 cells**. P5(k) and P6 do not fire anywhere, and P3
+(≥ 90%) is not approached by a factor of ~30–130×.
+
+The 20× horizon moved out-of-range transfer by nothing: probe acc is statistically identical
+to the E6 parents (0.007–0.030 here vs 0.007–0.049 in E6), all still P4.
+
+---
+
+## H-codes + halting drift — RNN cells H1, collapsed to floor, and drifted *down* over 4M steps
+
+**LOCKED** (Addenda 1–4): H1 = mean steps at the min_steps floor (2); H2 = pinned at K; H3 =
+intermediate + positive gap correlation; H4 = intermediate but no correlation.
+
+**Commentary.** Both RNN cells are **H1** at the final checkpoint: mean halt steps 2.002
+(wd01) and 2.029 (wd03), i.e. collapsed to the `min_steps = 2` floor (K = 32). The gap
+correlation is negative in both (`corr_gap` −0.083 / −0.123), so there is no computation
+budget learned by gap size — the same as every prior phase. The transformer cells have no ACT
+halting and so carry no H-code.
+
+**Drift over 4M steps (the point Addendum 7 asked to make explicit).** The RNN cells did NOT
+start at the floor — they *drifted down* onto it across the long horizon. Mean halt steps over
+the first 10 evals vs the last 10 evals of the 530-eval curve:
+
+| cell | halt mean, first 10 evals | halt mean, last 10 evals |
+|------|---------------------------|--------------------------|
+| e7-wd01/rnn | 2.19 | 2.00 |
+| e7-wd03/rnn | 2.44 | 2.03 |
+
+Both cells began slightly above the floor (2.19 / 2.44) and decayed monotonically toward it
+(2.00 / 2.03) over the 4M-step run. This is the *opposite* direction from the drift one would
+read as slow emergence of a compute-budget gate: the ACT gate did not gradually learn to run
+longer for large gaps; it gradually gave up the little extra compute it had early on and
+settled at the floor. The halting code is H1 at every stage of the run by the locked
+definition (mean never rose above the 2.5 H1/H2 boundary), so the drift does not change the
+code — it is recorded here because Addendum 7 requires it, and because it forecloses any
+reading of E7 as a slow ascent toward H2/H3.
+
+---
+
+## The verdict — H-slow REJECTED for all four cells (the null is budget-robust to 20×)
+
+**LOCKED** (Addendum 7), three branches applied verbatim:
+
+1. **O1 anywhere** → grokking confirmed, budget-limited at 200k. *Does not fire* — no O1
+   anywhere (see O-codes).
+2. **O-PARTIAL with `val_em_best > parent best` AND sustained ≥ 50k steps after the parent's
+   entire budget** → partial progress toward the algorithmic basin; report as slow-basin
+   dynamics, NOT grokking.
+3. **Same codes as E6 parents (O-PARTIAL/O4, P4/P2, no P5(k≥10)/P6)** → **H-slow REJECTED**
+   for these cells at 4M steps: the null is now budget-robust to 20×. Surviving hypotheses
+   become task-structure-gated (algorithmic alignment), not compute-gated.
+
+**The two cells whose best exceeds the parent's best — the crux.** Only two cells clear the
+"best > parent best" precondition, and neither clears the sustained-window condition:
+
+- **e7-wd01/rnn** (best 0.8367 vs parent 0.83). The metrics.csv val-EM trajectory exceeds
+  the parent's 0.83 at exactly **two isolated evals**: step 2 430 000 (0.8333) and step
+  2 650 000 (0.8367). There is **no sustained ≥ 50k-step window above 0.83 after step
+  200 000** — the two above-threshold readings are single-eval spikes separated by 220k
+  steps, with val EM back below 0.83 in between and after. The max val EM at any eval ≤
+  step 200 000 is 0.8233 (below the parent). **The AND condition does not fire** — this is
+  not a sustained improvement, it is an isolated best-checkpoint blip.
+- **e7-wd03/transformer** (best 0.8433 vs parent 0.84). val EM exceeds 0.84 at exactly **one
+  eval**, step 43 000 (0.8433) — which is **before** the parent's 200k budget, not after it,
+  and is a single eval with no ≥ 50k-step window on either side. **The condition does not
+  fire** (fails both the "after the parent's entire budget" clause and the "sustained"
+  clause).
+
+So branch 2 does not fire for any cell. The other two cells never even exceed their parent's
+best (e7-wd01/transformer 0.8233 < 0.8267; e7-wd03/rnn 0.8133 < 0.8233).
+
+**Overall E7 verdict.** Every cell lands in branch 3: identical code signatures to the E6
+parents (O-PARTIAL / P4 / H1 or no-ACT), no P5(k)/P6, no O1. **H-slow is REJECTED for all
+four cells** — the hypothesis that the next-prime transition was merely budget-limited is now
+robustly falsified at 20× the E6 compute budget. The surviving explanation is
+**task-structure-gated (algorithmic alignment)**, not compute-gated: the search/increment
+loop and the divisibility test remain unlearnable as *transferable* structure no matter how
+long the model is run, while in-range memorization/heuristics saturate exactly as they did
+at 200k.
+
+The one measurable, non-post-hoc observation from the long horizon is the *downward* halting
+drift (H-codes §): the extra 3.8M steps did not build any structure — the only thing that
+moved was the ACT gate relaxing the last of its early excess compute down onto the floor.
+
+---
+
+## Caveats / known gaps
+
+1. **eval.py probe-range gap from NOTES-E6 is resolved.** `src/eval.py::probe_report` is now
+   invoked as `probe_report(model, cfg, lo=cfg.range_end+1, hi=cfg.range_end+1000)` for
+   non-integer vocab modes, so the committed `results.json` probe field for E7 is the
+   out-of-range [1001, 2000] probe (1000 items), and a per-model `halting` report is emitted
+   for RNN cells. The E6 caveat (committed probe was the misleading in-range [101, 200]) no
+   longer binds for E7; the probe acc and H-codes quoted above come straight from
+   `results.json`.
+2. **fp16 / device split.** Transformer cells ran CUDA + AMP fp16; RNN cells ran CPU fp32
+   (same as E6). No cell sits within ±2 points of an O/P/H boundary — probe acc is 0.7–3.0%
+   vs the 90% P3 line, val EM is 0.77–0.84 vs the 0.9 O1 line, halt means are 2.00–2.03 vs
+   the 2.5 H1/H2 cutoff — so the fp16 precision caveat does not bind for any E7 cell.
+3. **Selection-holing on the best checkpoints.** The "best" val EM values are selection-holed
+   (val-selected checkpoint), and for the two cells that beat their parent (e7-wd01/rnn
+   0.8367, e7-wd03/transformer 0.8433) the margin over the parent (0.0067 and 0.0033) is
+   within the selection noise of a single eval. This is precisely why the matrix gates the
+   slow-basin branch on a *sustained* ≥ 50k-step window rather than on a single best value —
+   and why the isolated-spike finding above is dispositive rather than a rounding matter.
+4. **Single seed.** E7 is seed 0 only (as locked). The E6 grid showed the low-wd O-PARTIAL
+   behaviour is wd-dependent and seed-0-only (the wd-1.0 replication seeds were O4), so the
+   E7 verdict — H-slow rejected — is itself a seed-0 statement; nothing in E7 touches the
+   seed-stability of the *negative* result (which E6 already showed is robust across wd and
+   batch-size axes).
+
+---
+
+## Where this leaves the preregistration
+
+E7 closes the compute-gated leg of the hypothesis space. With H-slow rejected at 20×, the
+remaining live hypothesis is algorithmic-alignment / task-structure-gated: the network cannot
+represent (or cannot be steered to) the search-and-divisibility loop as a transferable
+computation under these dynamics, and the wall is not a function of how many steps it is
+given. The in-range O-code ceiling (O-PARTIAL, val EM ~0.80–0.84) and the zero out-of-range
+transfer (P4) are both stable under a 20× horizon, and the halting gate only relaxes toward
+the floor. Any next experiment that tests the surviving hypothesis (e.g. an architecture or
+representational change that could carry the divisibility structure, or an auxiliary loss
+that exposes it) would be a *new* preregistered leg, not a continuation of the H-slow axis
+that E7 has now exhausted.