summaryrefslogtreecommitdiff
path: root/prime-grokking/preregistration.md
diff options
context:
space:
mode:
Diffstat (limited to 'prime-grokking/preregistration.md')
-rw-r--r--prime-grokking/preregistration.md53
1 files changed, 53 insertions, 0 deletions
diff --git a/prime-grokking/preregistration.md b/prime-grokking/preregistration.md
index 56291f0..58cbe5a 100644
--- a/prime-grokking/preregistration.md
+++ b/prime-grokking/preregistration.md
@@ -307,3 +307,56 @@ Benchmarks pre-run: transformer 66 steps/s (AMP), rnn 31 steps/s → batch est.
**E7 (next, after E6 analysis):** long-horizon test — 20× budget (4M steps) on the best E6
cells, per user decision; protocol locked in Addendum 7 post-E6.
+
+---
+
+## Addendum 7 — E7: long-horizon test, 4M steps (locked 2026-08-21 09:20 UTC, pre-run)
+
+**Question (Q3a/Q3c):** Power et al. (2201.02177) observed generalization "well past the point
+of overfitting", with transitions at 10^5–10^6 steps and the explicit finding that smaller
+datasets require MORE optimization. Phases 1–3 and E6 all used a 200k-step budget. E7 tests
+the remaining lever: was every prior null simply an under-trained view of a slow transition?
+
+**Hypothesis being tested (pre-registered):** H-slow: a grokking transition exists for this
+task but requires >200k steps to emerge. Supporting texture from our own data: the E3 cosine
+transformer hit best val EM 0.900 then decayed — a "slingshot" pattern (Nanda et al.) that in
+the literature sometimes returns as full grokking much later.
+
+**Protocol:** SAME as E6 (digits, next_prime, train_frac 0.7, lr 1e-3 const, d_model 128,
+K=32, AMP on CUDA cells) with exactly ONE variable changed per cell vs its E6 parent:
+`max_train_steps 200_000 → 4_000_000` (20×).
+- `--eval_schedule adaptive` (recorded tooling change, not a scientific variable; ~400 evals
+ over the run instead of ~1000; cadence 200/1k/5k/10k by step band).
+- Early stop verified pre-run: fires only when val_em ≥ 1.0 for 5 consecutive evals — cannot
+ truncate a plateau. No protocol change needed.
+- Mixed-device assignment (recorded, benchmarked pre-run): transformer cells → ichi T1000
+ (`--device cuda --compile_model True`, 113 steps/s solo); RNN cells → ichi CPU
+ (`--device cpu`, OMP/MKL pinned to 2 threads via run_sweep env, ~69 steps/s). Rationale:
+ measured cross-over — GPU LOSES to CPU for the sequential tied cell (24.8 vs 69 steps/s);
+ compile breaks dynamic-halting graph capture for the RNN (12 steps/s). Device is recorded
+ per cell; fp32-on-CPU vs fp16-on-GPU numerics differ across model families exactly as in
+ E6, so within-model comparisons stay clean.
+- Concurrency 2 (one GPU cell + one CPU cell), tmux session `e7` ON ICHI ITSELF (SSH-drop
+ safe), logs runs/e7/logs/.
+
+**Cells (4):** wd {0.1, 0.3} × {rnn, transformer}, seed 0, range [2,1000] — the two wd levels
+where E6 showed the strongest in-range solutions (O-PARTIAL band). Selection rule locked
+pre-run: if both E6 parents of a cell are O4 (no solution to extend), the cell is still run
+as specified (the long horizon may rescue memorization itself); no substitutions.
+
+**Interpretation matrix (all else unchanged from Addenda 1–6):**
+- O1 anywhere = GROKKING CONFIRMED on next-prime: transition exists, was budget-limited at
+ 200k. Immediately record transition step; compare across models/wd for Q3 sharpness data.
+- O-PARTIAL with val_em_best > parent's best AND sustained ≥50k steps after the parent's
+ entire budget = partial progress toward the algorithmic basin (report as evidence of
+ slow-basin dynamics, NOT grokking).
+- Same codes as E6 parents (O-PARTIAL/O4, P4/P2, no P5(k≥10)/P6) = H-slow REJECTED for these
+ cells at 4M steps: the null is now budget-robust to 20×. Combined with E6, the surviving
+ hypotheses become task-structure-gated (algorithmic alignment), not compute-gated.
+- Probe ladder P5(k)/P6 as locked in Addendum 6 ([1001,2000] range).
+- Halting codes unchanged; note halt_mean drift over 4M steps explicitly in NOTES-E7.
+
+**Compute estimate (benchmarked):** transformer 4M @ ~113 steps/s ≈ 10–12 h; rnn 4M @ ~69
+steps/s ≈ 16–18 h; pairs run concurrently → batch ≈ 36–40 h total.
+
+**E8 ([2,10000], K=64, sieve-rank probe [10001,20000]) remains gated on E7 analysis.**