diff options
Diffstat (limited to 'prime-grokking/preregistration.md')
| -rw-r--r-- | prime-grokking/preregistration.md | 53 |
1 files changed, 53 insertions, 0 deletions
diff --git a/prime-grokking/preregistration.md b/prime-grokking/preregistration.md index 56291f0..58cbe5a 100644 --- a/prime-grokking/preregistration.md +++ b/prime-grokking/preregistration.md @@ -307,3 +307,56 @@ Benchmarks pre-run: transformer 66 steps/s (AMP), rnn 31 steps/s → batch est. **E7 (next, after E6 analysis):** long-horizon test — 20× budget (4M steps) on the best E6 cells, per user decision; protocol locked in Addendum 7 post-E6. + +--- + +## Addendum 7 — E7: long-horizon test, 4M steps (locked 2026-08-21 09:20 UTC, pre-run) + +**Question (Q3a/Q3c):** Power et al. (2201.02177) observed generalization "well past the point +of overfitting", with transitions at 10^5–10^6 steps and the explicit finding that smaller +datasets require MORE optimization. Phases 1–3 and E6 all used a 200k-step budget. E7 tests +the remaining lever: was every prior null simply an under-trained view of a slow transition? + +**Hypothesis being tested (pre-registered):** H-slow: a grokking transition exists for this +task but requires >200k steps to emerge. Supporting texture from our own data: the E3 cosine +transformer hit best val EM 0.900 then decayed — a "slingshot" pattern (Nanda et al.) that in +the literature sometimes returns as full grokking much later. + +**Protocol:** SAME as E6 (digits, next_prime, train_frac 0.7, lr 1e-3 const, d_model 128, +K=32, AMP on CUDA cells) with exactly ONE variable changed per cell vs its E6 parent: +`max_train_steps 200_000 → 4_000_000` (20×). +- `--eval_schedule adaptive` (recorded tooling change, not a scientific variable; ~400 evals + over the run instead of ~1000; cadence 200/1k/5k/10k by step band). +- Early stop verified pre-run: fires only when val_em ≥ 1.0 for 5 consecutive evals — cannot + truncate a plateau. No protocol change needed. +- Mixed-device assignment (recorded, benchmarked pre-run): transformer cells → ichi T1000 + (`--device cuda --compile_model True`, 113 steps/s solo); RNN cells → ichi CPU + (`--device cpu`, OMP/MKL pinned to 2 threads via run_sweep env, ~69 steps/s). Rationale: + measured cross-over — GPU LOSES to CPU for the sequential tied cell (24.8 vs 69 steps/s); + compile breaks dynamic-halting graph capture for the RNN (12 steps/s). Device is recorded + per cell; fp32-on-CPU vs fp16-on-GPU numerics differ across model families exactly as in + E6, so within-model comparisons stay clean. +- Concurrency 2 (one GPU cell + one CPU cell), tmux session `e7` ON ICHI ITSELF (SSH-drop + safe), logs runs/e7/logs/. + +**Cells (4):** wd {0.1, 0.3} × {rnn, transformer}, seed 0, range [2,1000] — the two wd levels +where E6 showed the strongest in-range solutions (O-PARTIAL band). Selection rule locked +pre-run: if both E6 parents of a cell are O4 (no solution to extend), the cell is still run +as specified (the long horizon may rescue memorization itself); no substitutions. + +**Interpretation matrix (all else unchanged from Addenda 1–6):** +- O1 anywhere = GROKKING CONFIRMED on next-prime: transition exists, was budget-limited at + 200k. Immediately record transition step; compare across models/wd for Q3 sharpness data. +- O-PARTIAL with val_em_best > parent's best AND sustained ≥50k steps after the parent's + entire budget = partial progress toward the algorithmic basin (report as evidence of + slow-basin dynamics, NOT grokking). +- Same codes as E6 parents (O-PARTIAL/O4, P4/P2, no P5(k≥10)/P6) = H-slow REJECTED for these + cells at 4M steps: the null is now budget-robust to 20×. Combined with E6, the surviving + hypotheses become task-structure-gated (algorithmic alignment), not compute-gated. +- Probe ladder P5(k)/P6 as locked in Addendum 6 ([1001,2000] range). +- Halting codes unchanged; note halt_mean drift over 4M steps explicitly in NOTES-E7. + +**Compute estimate (benchmarked):** transformer 4M @ ~113 steps/s ≈ 10–12 h; rnn 4M @ ~69 +steps/s ≈ 16–18 h; pairs run concurrently → batch ≈ 36–40 h total. + +**E8 ([2,10000], K=64, sieve-rank probe [10001,20000]) remains gated on E7 analysis.** |
