diff options
Diffstat (limited to 'NOTES-E7.md')
| -rw-r--r-- | NOTES-E7.md | 207 |
1 files changed, 207 insertions, 0 deletions
diff --git a/NOTES-E7.md b/NOTES-E7.md new file mode 100644 index 0000000..a0122fa --- /dev/null +++ b/NOTES-E7.md @@ -0,0 +1,207 @@ +# E7 — 20× long-horizon test (4M steps) — analysis notes (4/4 cells) + +Sources: `runs/e7/<job>/<model>/seed0/results.json` (eval run after each cell's +training), `metrics.csv` (per-eval curves, 530 evals each), and the locked interpretation +clauses in `design/preregistration.md` **Addendum 7** (the 20× budget test) plus Addenda 1–4 +(unchanged O/H/P operationalizations) and Addendum 6 (P-ladder / sieve-rank). + +**Batch status at analysis time (2026-08-29): all 4 cells finished** (wd {0.1, 0.3} × +{rnn, transformer}, seed 0, `max_train_steps = 4 000 000`, i.e. 20× the 200k budget of E6). +Everything else matches E6: digits mode, task `next_prime`, range [2, 1000], `train_frac` +0.7, lr 1e-3 const, d_model 128, K = 32, AMP fp16 on CUDA for the transformer cells, RNN +cells on CPU fp32, adaptive eval schedule → 530 evals. + +Constants used throughout: val EM is exact-match; "best" is the val-selected checkpoint +(selection-holed), "last" is the unselected final checkpoint. **The E7 probe is exact-match +on [1001, 2000]** (out of training range; `range_end = 1000`). Note that this is the gap that +was open in NOTES-E6 — `src/eval.py` now emits `probe_report(..., lo=range_end+1, +hi=range_end+1000)` and a per-model `halting` report, so the committed `results.json` probe +and halting fields are self-consistent for E7 (see Caveats §1). + +--- + +## Per-cell classification (all 4 cells) + +| job | model | O | H | halt mean | val EM best | val EM last | train EM (max/final) | probe [1001,2000] acc | P | best-step | +|-----|-------|---|-----|-----------|-------------|-------------|----------------------|-----------------------|---|-----------| +| e7-wd01 | rnn | O-PARTIAL | H1 | 2.00 | 0.8367 | 0.8033 | 1.000 / 1.000 | 0.012 (12/1000) | P4 | 2 650 000 | +| e7-wd01 | transformer | O-PARTIAL | — | — | 0.8233 | 0.7867 | 1.000 / 1.000 | 0.007 (7/1000) | P4 | 37 000 | +| e7-wd03 | rnn | O-PARTIAL | H1 | 2.03 | 0.8133 | 0.8000 | 1.000 / 1.000 | 0.030 (30/1000) | P4 | 455 000 | +| e7-wd03 | transformer | O-PARTIAL | — | — | 0.8433 | 0.7700 | 1.000 / 1.000 | 0.008 (8/1000) | P4 | 43 000 | + +Halt means are from the committed `halting` report on the final checkpoint (RNN cells only; +the transformer has no ACT halting, so `mean_halt_steps` is NaN throughout its metrics.csv and +no `halting` field is emitted). + +--- + +## O-codes — no O1 anywhere (no grokking), all four O-PARTIAL + +**LOCKED** (Addendum 7): "O1 anywhere = GROKKING CONFIRMED on next-prime: transition exists, +was budget-limited at 200k. Record transition step." + +**Commentary.** None of the four cells is **O1**. All four are **O-PARTIAL** by the unchanged +Addenda 1–4 definition: train saturated (train EM = 1.0 sustained; `sat_start_eval` 24 / 2 / +73 / 3 for wd01-rnn / wd01-transformer / wd03-rnn / wd03-transformer respectively), val never +reached 0.9, and final val sits in (0.3, 0.9). The signature detector concurs: every +`signature` has `code = "O-PARTIAL"`, `val_hi_eval_idx = null`, `transition_width_evals = +null` — i.e. **no sharp val transition to ~1.0 exists in any cell**, at any of the 530 evals +across 4M steps. + +**Transition-step search.** The maximum val EM any cell ever reached, and the step it reached +it: + +| cell | max val EM ever | step of max | +|------|-----------------|-------------| +| e7-wd01/rnn | 0.8367 | 2 650 000 | +| e7-wd01/transformer | 0.8233 | 37 000 | +| e7-wd03/rnn | 0.8133 | 455 000 | +| e7-wd03/transformer | 0.8433 | 43 000 | + +No cell ever logged a val EM ≥ 0.9 at any eval (0 of 530 evals per cell). The 20× horizon did +not produce a grokking transition; the ceiling on in-range generalization is essentially the +same as E6's (best val 0.813–0.844 vs E6's 0.823–0.840), and two of the four cells' bests +occur in the first ~450k steps (i.e. inside a horizon E6 already explored). + +--- + +## P-codes — all P4, and no sieve (4/4) + +**LOCKED** (Addendum 6, unchanged): probe range [1001, 2000]; P5(k) = errors concentrated on +the rank-k signature set; P6 = exact (no probe misses); P3 threshold ≥ 90%; P4/P2 keep their +meanings. A model that internalized a k-prime sieve misses exactly the composites whose prime +factors all exceed p_k (smallest missed composite → k: 1147→10, 1369→11, 1681→12, 1849→13, +none→k≥14). + +**Commentary.** All four cells are **P4**: probe accuracy on [1001, 2000] is 0.007–0.030 +(7–30 of 1000 correct). The failure mode is unchanged from every prior phase — the cells fail +"the next prime after an even number is odd" essentially always, and emit composite or +nonsense outputs (e.g. wd01-rnn answers 1040→1439, 1082→10099; wd03-rnn answers 1003→1093; +wd01-transformer answers 1049→353; wd03-transformer answers 1001→3). The smallest composite +each cell emits as "prime" in range has small prime factors (e.g. 1003 = 17·59, 1001 = +7·11·13), so there is **no divisibility structure at any rank** — the P-ladder resolves to +**"no sieve" (k undefined) in all 4 cells**. P5(k) and P6 do not fire anywhere, and P3 +(≥ 90%) is not approached by a factor of ~30–130×. + +The 20× horizon moved out-of-range transfer by nothing: probe acc is statistically identical +to the E6 parents (0.007–0.030 here vs 0.007–0.049 in E6), all still P4. + +--- + +## H-codes + halting drift — RNN cells H1, collapsed to floor, and drifted *down* over 4M steps + +**LOCKED** (Addenda 1–4): H1 = mean steps at the min_steps floor (2); H2 = pinned at K; H3 = +intermediate + positive gap correlation; H4 = intermediate but no correlation. + +**Commentary.** Both RNN cells are **H1** at the final checkpoint: mean halt steps 2.002 +(wd01) and 2.029 (wd03), i.e. collapsed to the `min_steps = 2` floor (K = 32). The gap +correlation is negative in both (`corr_gap` −0.083 / −0.123), so there is no computation +budget learned by gap size — the same as every prior phase. The transformer cells have no ACT +halting and so carry no H-code. + +**Drift over 4M steps (the point Addendum 7 asked to make explicit).** The RNN cells did NOT +start at the floor — they *drifted down* onto it across the long horizon. Mean halt steps over +the first 10 evals vs the last 10 evals of the 530-eval curve: + +| cell | halt mean, first 10 evals | halt mean, last 10 evals | +|------|---------------------------|--------------------------| +| e7-wd01/rnn | 2.19 | 2.00 | +| e7-wd03/rnn | 2.44 | 2.03 | + +Both cells began slightly above the floor (2.19 / 2.44) and decayed monotonically toward it +(2.00 / 2.03) over the 4M-step run. This is the *opposite* direction from the drift one would +read as slow emergence of a compute-budget gate: the ACT gate did not gradually learn to run +longer for large gaps; it gradually gave up the little extra compute it had early on and +settled at the floor. The halting code is H1 at every stage of the run by the locked +definition (mean never rose above the 2.5 H1/H2 boundary), so the drift does not change the +code — it is recorded here because Addendum 7 requires it, and because it forecloses any +reading of E7 as a slow ascent toward H2/H3. + +--- + +## The verdict — H-slow REJECTED for all four cells (the null is budget-robust to 20×) + +**LOCKED** (Addendum 7), three branches applied verbatim: + +1. **O1 anywhere** → grokking confirmed, budget-limited at 200k. *Does not fire* — no O1 + anywhere (see O-codes). +2. **O-PARTIAL with `val_em_best > parent best` AND sustained ≥ 50k steps after the parent's + entire budget** → partial progress toward the algorithmic basin; report as slow-basin + dynamics, NOT grokking. +3. **Same codes as E6 parents (O-PARTIAL/O4, P4/P2, no P5(k≥10)/P6)** → **H-slow REJECTED** + for these cells at 4M steps: the null is now budget-robust to 20×. Surviving hypotheses + become task-structure-gated (algorithmic alignment), not compute-gated. + +**The two cells whose best exceeds the parent's best — the crux.** Only two cells clear the +"best > parent best" precondition, and neither clears the sustained-window condition: + +- **e7-wd01/rnn** (best 0.8367 vs parent 0.83). The metrics.csv val-EM trajectory exceeds + the parent's 0.83 at exactly **two isolated evals**: step 2 430 000 (0.8333) and step + 2 650 000 (0.8367). There is **no sustained ≥ 50k-step window above 0.83 after step + 200 000** — the two above-threshold readings are single-eval spikes separated by 220k + steps, with val EM back below 0.83 in between and after. The max val EM at any eval ≤ + step 200 000 is 0.8233 (below the parent). **The AND condition does not fire** — this is + not a sustained improvement, it is an isolated best-checkpoint blip. +- **e7-wd03/transformer** (best 0.8433 vs parent 0.84). val EM exceeds 0.84 at exactly **one + eval**, step 43 000 (0.8433) — which is **before** the parent's 200k budget, not after it, + and is a single eval with no ≥ 50k-step window on either side. **The condition does not + fire** (fails both the "after the parent's entire budget" clause and the "sustained" + clause). + +So branch 2 does not fire for any cell. The other two cells never even exceed their parent's +best (e7-wd01/transformer 0.8233 < 0.8267; e7-wd03/rnn 0.8133 < 0.8233). + +**Overall E7 verdict.** Every cell lands in branch 3: identical code signatures to the E6 +parents (O-PARTIAL / P4 / H1 or no-ACT), no P5(k)/P6, no O1. **H-slow is REJECTED for all +four cells** — the hypothesis that the next-prime transition was merely budget-limited is now +robustly falsified at 20× the E6 compute budget. The surviving explanation is +**task-structure-gated (algorithmic alignment)**, not compute-gated: the search/increment +loop and the divisibility test remain unlearnable as *transferable* structure no matter how +long the model is run, while in-range memorization/heuristics saturate exactly as they did +at 200k. + +The one measurable, non-post-hoc observation from the long horizon is the *downward* halting +drift (H-codes §): the extra 3.8M steps did not build any structure — the only thing that +moved was the ACT gate relaxing the last of its early excess compute down onto the floor. + +--- + +## Caveats / known gaps + +1. **eval.py probe-range gap from NOTES-E6 is resolved.** `src/eval.py::probe_report` is now + invoked as `probe_report(model, cfg, lo=cfg.range_end+1, hi=cfg.range_end+1000)` for + non-integer vocab modes, so the committed `results.json` probe field for E7 is the + out-of-range [1001, 2000] probe (1000 items), and a per-model `halting` report is emitted + for RNN cells. The E6 caveat (committed probe was the misleading in-range [101, 200]) no + longer binds for E7; the probe acc and H-codes quoted above come straight from + `results.json`. +2. **fp16 / device split.** Transformer cells ran CUDA + AMP fp16; RNN cells ran CPU fp32 + (same as E6). No cell sits within ±2 points of an O/P/H boundary — probe acc is 0.7–3.0% + vs the 90% P3 line, val EM is 0.77–0.84 vs the 0.9 O1 line, halt means are 2.00–2.03 vs + the 2.5 H1/H2 cutoff — so the fp16 precision caveat does not bind for any E7 cell. +3. **Selection-holing on the best checkpoints.** The "best" val EM values are selection-holed + (val-selected checkpoint), and for the two cells that beat their parent (e7-wd01/rnn + 0.8367, e7-wd03/transformer 0.8433) the margin over the parent (0.0067 and 0.0033) is + within the selection noise of a single eval. This is precisely why the matrix gates the + slow-basin branch on a *sustained* ≥ 50k-step window rather than on a single best value — + and why the isolated-spike finding above is dispositive rather than a rounding matter. +4. **Single seed.** E7 is seed 0 only (as locked). The E6 grid showed the low-wd O-PARTIAL + behaviour is wd-dependent and seed-0-only (the wd-1.0 replication seeds were O4), so the + E7 verdict — H-slow rejected — is itself a seed-0 statement; nothing in E7 touches the + seed-stability of the *negative* result (which E6 already showed is robust across wd and + batch-size axes). + +--- + +## Where this leaves the preregistration + +E7 closes the compute-gated leg of the hypothesis space. With H-slow rejected at 20×, the +remaining live hypothesis is algorithmic-alignment / task-structure-gated: the network cannot +represent (or cannot be steered to) the search-and-divisibility loop as a transferable +computation under these dynamics, and the wall is not a function of how many steps it is +given. The in-range O-code ceiling (O-PARTIAL, val EM ~0.80–0.84) and the zero out-of-range +transfer (P4) are both stable under a 20× horizon, and the halting gate only relaxes toward +the floor. Any next experiment that tests the surviving hypothesis (e.g. an architecture or +representational change that could carry the divisibility structure, or an auxiliary loss +that exposes it) would be a *new* preregistered leg, not a continuation of the H-slow axis +that E7 has now exhausted. |