path: root/NOTES-E7.md

# E7 — 20× long-horizon test (4M steps) — analysis notes (4/4 cells)

Sources: `runs/e7/<job>/<model>/seed0/results.json` (eval run after each cell's
training), `metrics.csv` (per-eval curves, 530 evals each), and the locked interpretation
clauses in `design/preregistration.md` **Addendum 7** (the 20× budget test) plus Addenda 1–4
(unchanged O/H/P operationalizations) and Addendum 6 (P-ladder / sieve-rank).

**Batch status at analysis time (2026-08-29): all 4 cells finished** (wd {0.1, 0.3} ×
{rnn, transformer}, seed 0, `max_train_steps = 4 000 000`, i.e. 20× the 200k budget of E6).
Everything else matches E6: digits mode, task `next_prime`, range [2, 1000], `train_frac`
0.7, lr 1e-3 const, d_model 128, K = 32, AMP fp16 on CUDA for the transformer cells, RNN
cells on CPU fp32, adaptive eval schedule → 530 evals.

Constants used throughout: val EM is exact-match; "best" is the val-selected checkpoint
(selection-holed), "last" is the unselected final checkpoint. **The E7 probe is exact-match
on [1001, 2000]** (out of training range; `range_end = 1000`). Note that this is the gap that
was open in NOTES-E6 — `src/eval.py` now emits `probe_report(..., lo=range_end+1,
hi=range_end+1000)` and a per-model `halting` report, so the committed `results.json` probe
and halting fields are self-consistent for E7 (see Caveats §1).

---

## Per-cell classification (all 4 cells)

| job | model | O | H | halt mean | val EM best | val EM last | train EM (max/final) | probe [1001,2000] acc | P | best-step |
|-----|-------|---|-----|-----------|-------------|-------------|----------------------|-----------------------|---|-----------|
| e7-wd01 | rnn | O-PARTIAL | H1 | 2.00 | 0.8367 | 0.8033 | 1.000 / 1.000 | 0.012 (12/1000) | P4 | 2 650 000 |
| e7-wd01 | transformer | O-PARTIAL | — | — | 0.8233 | 0.7867 | 1.000 / 1.000 | 0.007 (7/1000) | P4 | 37 000 |
| e7-wd03 | rnn | O-PARTIAL | H1 | 2.03 | 0.8133 | 0.8000 | 1.000 / 1.000 | 0.030 (30/1000) | P4 | 455 000 |
| e7-wd03 | transformer | O-PARTIAL | — | — | 0.8433 | 0.7700 | 1.000 / 1.000 | 0.008 (8/1000) | P4 | 43 000 |

Halt means are from the committed `halting` report on the final checkpoint (RNN cells only;
the transformer has no ACT halting, so `mean_halt_steps` is NaN throughout its metrics.csv and
no `halting` field is emitted).

---

## O-codes — no O1 anywhere (no grokking), all four O-PARTIAL

**LOCKED** (Addendum 7): "O1 anywhere = GROKKING CONFIRMED on next-prime: transition exists,
was budget-limited at 200k. Record transition step."

**Commentary.** None of the four cells is **O1**. All four are **O-PARTIAL** by the unchanged
Addenda 1–4 definition: train saturated (train EM = 1.0 sustained; `sat_start_eval` 24 / 2 /
73 / 3 for wd01-rnn / wd01-transformer / wd03-rnn / wd03-transformer respectively), val never
reached 0.9, and final val sits in (0.3, 0.9). The signature detector concurs: every
`signature` has `code = "O-PARTIAL"`, `val_hi_eval_idx = null`, `transition_width_evals =
null` — i.e. **no sharp val transition to ~1.0 exists in any cell**, at any of the 530 evals
across 4M steps.

**Transition-step search.** The maximum val EM any cell ever reached, and the step it reached
it:

| cell | max val EM ever | step of max |
|------|-----------------|-------------|
| e7-wd01/rnn | 0.8367 | 2 650 000 |
| e7-wd01/transformer | 0.8233 | 37 000 |
| e7-wd03/rnn | 0.8133 | 455 000 |
| e7-wd03/transformer | 0.8433 | 43 000 |

No cell ever logged a val EM ≥ 0.9 at any eval (0 of 530 evals per cell). The 20× horizon did
not produce a grokking transition; the ceiling on in-range generalization is essentially the
same as E6's (best val 0.813–0.844 vs E6's 0.823–0.840), and two of the four cells' bests
occur in the first ~450k steps (i.e. inside a horizon E6 already explored).

---

## P-codes — all P4, and no sieve (4/4)

**LOCKED** (Addendum 6, unchanged): probe range [1001, 2000]; P5(k) = errors concentrated on
the rank-k signature set; P6 = exact (no probe misses); P3 threshold ≥ 90%; P4/P2 keep their
meanings. A model that internalized a k-prime sieve misses exactly the composites whose prime
factors all exceed p_k (smallest missed composite → k: 1147→10, 1369→11, 1681→12, 1849→13,
none→k≥14).

**Commentary.** All four cells are **P4**: probe accuracy on [1001, 2000] is 0.007–0.030
(7–30 of 1000 correct). The failure mode is unchanged from every prior phase — the cells fail
"the next prime after an even number is odd" essentially always, and emit composite or
nonsense outputs (e.g. wd01-rnn answers 1040→1439, 1082→10099; wd03-rnn answers 1003→1093;
wd01-transformer answers 1049→353; wd03-transformer answers 1001→3). The smallest composite
each cell emits as "prime" in range has small prime factors (e.g. 1003 = 17·59, 1001 =
7·11·13), so there is **no divisibility structure at any rank** — the P-ladder resolves to
**"no sieve" (k undefined) in all 4 cells**. P5(k) and P6 do not fire anywhere, and P3
(≥ 90%) is not approached by a factor of ~30–130×.

The 20× horizon moved out-of-range transfer by nothing: probe acc is statistically identical
to the E6 parents (0.007–0.030 here vs 0.007–0.049 in E6), all still P4.

---

## H-codes + halting drift — RNN cells H1, collapsed to floor, and drifted *down* over 4M steps

**LOCKED** (Addenda 1–4): H1 = mean steps at the min_steps floor (2); H2 = pinned at K; H3 =
intermediate + positive gap correlation; H4 = intermediate but no correlation.

**Commentary.** Both RNN cells are **H1** at the final checkpoint: mean halt steps 2.002
(wd01) and 2.029 (wd03), i.e. collapsed to the `min_steps = 2` floor (K = 32). The gap
correlation is negative in both (`corr_gap` −0.083 / −0.123), so there is no computation
budget learned by gap size — the same as every prior phase. The transformer cells have no ACT
halting and so carry no H-code.

**Drift over 4M steps (the point Addendum 7 asked to make explicit).** The RNN cells did NOT
start at the floor — they *drifted down* onto it across the long horizon. Mean halt steps over
the first 10 evals vs the last 10 evals of the 530-eval curve:

| cell | halt mean, first 10 evals | halt mean, last 10 evals |
|------|---------------------------|--------------------------|
| e7-wd01/rnn | 2.19 | 2.00 |
| e7-wd03/rnn | 2.44 | 2.03 |

Both cells began slightly above the floor (2.19 / 2.44) and decayed monotonically toward it
(2.00 / 2.03) over the 4M-step run. This is the *opposite* direction from the drift one would
read as slow emergence of a compute-budget gate: the ACT gate did not gradually learn to run
longer for large gaps; it gradually gave up the little extra compute it had early on and
settled at the floor. The halting code is H1 at every stage of the run by the locked
definition (mean never rose above the 2.5 H1/H2 boundary), so the drift does not change the
code — it is recorded here because Addendum 7 requires it, and because it forecloses any
reading of E7 as a slow ascent toward H2/H3.

---

## The verdict — H-slow REJECTED for all four cells (the null is budget-robust to 20×)

**LOCKED** (Addendum 7), three branches applied verbatim:

1. **O1 anywhere** → grokking confirmed, budget-limited at 200k. *Does not fire* — no O1
   anywhere (see O-codes).
2. **O-PARTIAL with `val_em_best > parent best` AND sustained ≥ 50k steps after the parent's
   entire budget** → partial progress toward the algorithmic basin; report as slow-basin
   dynamics, NOT grokking.
3. **Same codes as E6 parents (O-PARTIAL/O4, P4/P2, no P5(k≥10)/P6)** → **H-slow REJECTED**
   for these cells at 4M steps: the null is now budget-robust to 20×. Surviving hypotheses
   become task-structure-gated (algorithmic alignment), not compute-gated.

**The two cells whose best exceeds the parent's best — the crux.** Only two cells clear the
"best > parent best" precondition, and neither clears the sustained-window condition:

- **e7-wd01/rnn** (best 0.8367 vs parent 0.83). The metrics.csv val-EM trajectory exceeds
  the parent's 0.83 at exactly **two isolated evals**: step 2 430 000 (0.8333) and step
  2 650 000 (0.8367). There is **no sustained ≥ 50k-step window above 0.83 after step
  200 000** — the two above-threshold readings are single-eval spikes separated by 220k
  steps, with val EM back below 0.83 in between and after. The max val EM at any eval ≤
  step 200 000 is 0.8233 (below the parent). **The AND condition does not fire** — this is
  not a sustained improvement, it is an isolated best-checkpoint blip.
- **e7-wd03/transformer** (best 0.8433 vs parent 0.84). val EM exceeds 0.84 at exactly **one
  eval**, step 43 000 (0.8433) — which is **before** the parent's 200k budget, not after it,
  and is a single eval with no ≥ 50k-step window on either side. **The condition does not
  fire** (fails both the "after the parent's entire budget" clause and the "sustained"
  clause).

So branch 2 does not fire for any cell. The other two cells never even exceed their parent's
best (e7-wd01/transformer 0.8233 < 0.8267; e7-wd03/rnn 0.8133 < 0.8233).

**Overall E7 verdict.** Every cell lands in branch 3: identical code signatures to the E6
parents (O-PARTIAL / P4 / H1 or no-ACT), no P5(k)/P6, no O1. **H-slow is REJECTED for all
four cells** — the hypothesis that the next-prime transition was merely budget-limited is now
robustly falsified at 20× the E6 compute budget. The surviving explanation is
**task-structure-gated (algorithmic alignment)**, not compute-gated: the search/increment
loop and the divisibility test remain unlearnable as *transferable* structure no matter how
long the model is run, while in-range memorization/heuristics saturate exactly as they did
at 200k.

The one measurable, non-post-hoc observation from the long horizon is the *downward* halting
drift (H-codes §): the extra 3.8M steps did not build any structure — the only thing that
moved was the ACT gate relaxing the last of its early excess compute down onto the floor.

---

## Caveats / known gaps

1. **eval.py probe-range gap from NOTES-E6 is resolved.** `src/eval.py::probe_report` is now
   invoked as `probe_report(model, cfg, lo=cfg.range_end+1, hi=cfg.range_end+1000)` for
   non-integer vocab modes, so the committed `results.json` probe field for E7 is the
   out-of-range [1001, 2000] probe (1000 items), and a per-model `halting` report is emitted
   for RNN cells. The E6 caveat (committed probe was the misleading in-range [101, 200]) no
   longer binds for E7; the probe acc and H-codes quoted above come straight from
   `results.json`.
2. **fp16 / device split.** Transformer cells ran CUDA + AMP fp16; RNN cells ran CPU fp32
   (same as E6). No cell sits within ±2 points of an O/P/H boundary — probe acc is 0.7–3.0%
   vs the 90% P3 line, val EM is 0.77–0.84 vs the 0.9 O1 line, halt means are 2.00–2.03 vs
   the 2.5 H1/H2 cutoff — so the fp16 precision caveat does not bind for any E7 cell.
3. **Selection-holing on the best checkpoints.** The "best" val EM values are selection-holed
   (val-selected checkpoint), and for the two cells that beat their parent (e7-wd01/rnn
   0.8367, e7-wd03/transformer 0.8433) the margin over the parent (0.0067 and 0.0033) is
   within the selection noise of a single eval. This is precisely why the matrix gates the
   slow-basin branch on a *sustained* ≥ 50k-step window rather than on a single best value —
   and why the isolated-spike finding above is dispositive rather than a rounding matter.
4. **Single seed.** E7 is seed 0 only (as locked). The E6 grid showed the low-wd O-PARTIAL
   behaviour is wd-dependent and seed-0-only (the wd-1.0 replication seeds were O4), so the
   E7 verdict — H-slow rejected — is itself a seed-0 statement; nothing in E7 touches the
   seed-stability of the *negative* result (which E6 already showed is robust across wd and
   batch-size axes).

---

## Where this leaves the preregistration

E7 closes the compute-gated leg of the hypothesis space. With H-slow rejected at 20×, the
remaining live hypothesis is algorithmic-alignment / task-structure-gated: the network cannot
represent (or cannot be steered to) the search-and-divisibility loop as a transferable
computation under these dynamics, and the wall is not a function of how many steps it is
given. The in-range O-code ceiling (O-PARTIAL, val EM ~0.80–0.84) and the zero out-of-range
transfer (P4) are both stable under a 20× horizon, and the halting gate only relaxes toward
the floor. Any next experiment that tests the surviving hypothesis (e.g. an architecture or
representational change that could carry the divisibility structure, or an auxiliary loss
that exposes it) would be a *new* preregistered leg, not a continuation of the H-slow axis
that E7 has now exhausted.