prime-grokking: E8 analysis + project freeze 2026-10-09 (main.md experiment log 08-23..09-08 + freeze entry; prereg pointer)HEADmaster

2 files changed, 157 insertions, 0 deletions

diff --git a/prime-grokking/main.md b/prime-grokking/main.md
index a677fa5..7efcd3b 100644
--- a/prime-grokking/main.md
+++ b/prime-grokking/main.md
@@ -170,3 +170,148 @@ cross the gap from pattern matching to computation.
   overfitting. It does not assume primes have a simple closed-form pattern;
   the bounded target remains a test of whether a learned trial-division
   procedure can beat memorization.
+
+### 2026-08-23 — E7 interrupted; evaluation pending
+
+- ichi shut down cleanly at 22:09 UTC on 22 Aug while E7 was running. After
+  recovery, `wd=0.1` transformer had reached 4M steps, but its final evaluator
+  could not load the compiled checkpoint because keys retain the `_orig_mod.`
+  prefix; consequently it has no `results.json` and cannot yet be classified.
+- `wd=0.1` RNN stopped at about 3.2M steps, `wd=0.3` RNN at about 1.9M, and
+  `wd=0.3` transformer did not start. The tmux session is gone and training
+  has no resume support. No locked-matrix outcome, transition step, or H-slow
+  verdict should be reported from this interrupted batch.
+
+### 2026-08-27 — E7 RNNs in flight; E8 decision prioritised
+
+- Both transformer cells of E7 are finalised and reject the slow-basin
+  hypothesis: no transition at 20× budget (best val EM 82.3% @37k then
+  3.9M steps of decay for wd0.1; 84.3% early → 77.0% for wd0.3). The two RNN
+  twins are the remaining cells that close E7 formally — healthy on ichi CPU
+  at the 09:30 delivery check (wd0.1 ≈2.01M/4M, wd0.3 ≈1.31M/4M), revised
+  ETA **Fri 08-28 05:00 UTC / Sat 08-30 03:00 UTC**. Delivery cron recreated
+  for the final fan-out: `c80b2dd6a3f0` → Sun 30 Aug 09:30 BST (waits for
+  both cells, does not claim completion early).
+- **Experiment decision (jayrup's call): E8 [2,10,000] is prioritised over an
+  E7.5/20M-step scaling.** Rationale: E8 removes the cheap `[2,1000]`
+  lookup-table regime (only ~699 training examples) so a successful
+  generalisation would demand a genuinely reusable divisibility procedure —
+  a more direct test of the central hypothesis than throwing more compute at
+  the same memorisable mapping. If E8 produces a dramatic outcome (real
+  grokking or strong failure under capacity pressure) it justifies a
+  standalone post; otherwise E6/E7 fold into the eventual full writeup.
+- Batch-size changes (32→128/512, proposed for faster GPU utilisation) are
+  treated as **new experimental conditions**, not faster equivalents of the
+  batch-32 runs — keep a batch-32 control and equal example-exposure where
+  possible. Renting a vast.ai GPU (4090-class) is gated on a short benchmark
+  first (sustained steps/s after compile warmup, numerical stability,
+  checkpoint/restart, host persistence).
+
+### 2026-08-29 — E7 COMPLETE; H-slow rejected 4/4; E9 (scratchpad) designed ahead of E8
+
+- **E7 formal result (all four cells delivered, commit `c93eeaf`):** every cell
+  reproduces its E6 parent's locked codes exactly — O-PARTIAL / P4 / H1 (or
+  no-ACT for transformers). Probe [1001,2000] dead (7/1000–30/1000). **H-slow
+  REJECTED 4/4 → the null is budget-robust to 20×.** The two "best > parent"
+  near-misses are not slow-basin dynamics: wd0.1-rnn's 0.8367 is one of two
+  isolated single-eval spikes (2.43M, 2.65M, val below 0.83 in between), failing
+  the locked *sustained ≥50k past parent budget* clause; wd0.3-tf's 0.8433 sits
+  at step 43k, *before* the 200k parent budget ends. "Same codes" branch fires
+  for all four. Only measurable motion over 3.8M extra steps: **ACT gate
+  drifted DOWN** onto the min-steps floor (halt 2.19→2.00, 2.44→2.03) — the
+  opposite of a gate learning to budget compute.
+- Full-phase picture now closed: wd no-effect (E2), data 10× zero-transfer
+  (E6), budget 20× no-transition (E7), arch/halting/rep/lr all ruled out. No
+  O1 anywhere — no eval ≥0.9 across 530 evals/4M steps per cell. The wall is
+  **algorithmic (task-structure-gated), not compute-gated.**
+- **New direction (jayrup's idea, my 4-arm → other agent's 6-arm): scratch-
+  pad/filler tokens.** Question worth answering: does outputting *any* tokens
+  before the answer change the **grokking dynamics** (val transition appearing
+  in scratch-space where 20M no-scratch steps never found one)? Sharp,
+  falsifiable reference: Pfau et al. 2024 — fillers help only when the buried
+  computation is parallelizable, and divisible-by-{2,3,5,7} qualifies (4
+  independent checks). 6 arms: A baseline / B structured ground-truth traces
+  (= algorithm supervision, near-cheating, the ceiling — label honestly) /
+  C fixed-16 filler / D1 learned-random / D2 frozen / D3 dynamic-noise (non-
+  digit alphabet for all D arms to block "copy a scratch digit" shortcuts).
+- Other agent preregistered **Addendum 8** and implemented 6 arms (+8 tests,
+  jobs file, explainer HTML). **My review caught a real pre-launch crash:**
+  worst-case structured trace n=887→907 = 371 seq, but positional embedding
+  only enlarged 64→256 → Arm B dies mid-run on the hardest example. Raised to
+  1024 (their later fix, covers probe-range traces + causal-mask buffer + eval
+  early-exit). 56/56 tests pass. **Open:** probe-range trace length vs
+  `_global_lengths` cap (out_max=369 structured vs probe n≈1000→1009 much
+  longer — if probe gen cap is still 369, Arm B's headline probe silently
+  truncates). Not yet launched.
+- **Sequencing: E9 (this scratchpad batch) runs BEFORE E8 [2,10,000]** — if
+  scratch changes the dynamics, E8's design should include a scratch arm from
+  birth. Budget 200k steps default (E6 regime), 400k if a curve looks alive.
+  Don't run scratch on the RNN (already recurrent; E7 RNN cells are the control).
+- **vast.ai now operational** (for E8/E9 GPU cells): CLI 1.5.6 + skill
+  `mlops/vastai-gpu-rental` live, API key registered, voidlaptop SSH key added
+  (id 1290554). Spendable credit **$3.079** (the `credit` field, not
+  `balance`). UK 48GB modded 4090 (offer 28938111, $0.818/hr) is the preferred
+  box if a benchmark confirms a real advantage.
+
+### 2026-08-30 — Scratchpad batch LAUNCHED on ichi; probe-decode-cap blocker fixed; vast-vs-ichi decided
+
+- **Open blocker from 08-29 resolved + root cause nailed:** `greedy_decode` capped generation at the in-range layout bound (`out_max`=369), but the probe window [1001,2000] holds true traces up to **633 tokens** (n=1327). The other agent's 1024 positional fix made the *model* able to represent long traces but left the *decoder* capped → Arm B's probe would read ~0% for every n with a >369-token trace even under perfect generalization (the S4/S5 "ruler too short" vs "no transfer" deadlock). Fixed by keying the decode cap to the positional-table bound (1023); commit `3135fea`, regression-verified via stub model.
+- **Launch-clean pre-flight:** 56/56 tests (incl. 8 new scratchpad tests), 6 arms forward + masked loss finite, worst probe trace 640 ≤ 1024, Addendum 8 prereg locked pre-run against the S1–S5 matrix. **NOTE — naming drift:** this thread labels the scratchpad batch **E8** (tmux `e8`, delivery cron → NOTES-E8.md) and [2,10000] as **E9**; the Aug-29 entry called them E9/E8 respectively. The Addendum-8 prereg is the anchor.
+- **Running on ichi (2-way GPU pipeline, tmux `e8`, jobs reordered so the long arm starts first):** Arm A (baseline) done ~35 min; **Arm B (structured scratchpad) = long pole, 100% GPU ~2–4h**; Arm C (fixed-16 filler) took the freed slot; D1/D2/D3 queued (~35 min each). ETA all 6 **Mon 01:00–03:30 UTC**. Delivery cron **`909ceb7cdf81` Mon 09:30 BST (pinned)**: NOTES-E8.md vs locked S1–S5 matrix → Phase 7 → commit/push → Telegram.
+- **Verdict read (pre-written):** Arm B transition where 20M no-scratch never did → the project's biggest result. Only B moves → S1 (decomposition was the missing ingredient). C/D1-3 also move → S2/S3 (token-space recurrence alone unlocks it; Pfau-2024 confirmation). Nothing moves → S5 (barrier deeper than architecture).
+- **vast.vs-ichi, decided: ichi for this batch** (~3h, free; 3090 shaves a 3h batch to ~1h, not worth setup). **Vast for the [2,10000] follow-up** where Arm-B-class runs go multi-hour/cell with 1.5k-token sequences: RTX 3090 @ $0.16/h (rel 0.997) or 4090 @ $0.33/h; credit $3.08 ≈ ~18h of 3090. Analysis subagent died mid-task on an **OpenRouter key total-limit (HTTP 403 on deepseek-v4-pro)** — affects any deepseek-v4-pro cron until topped up; live read-only search substituted.
+
+### 2026-08-31 — Scratchpad batch: 5 of 6 arms done — filler/random arms a clean NEGATIVE; only Arm B left [S1 vs S5]
+
+Results (ichi, tmux `e8`, desktop glm-5.3-flash):
+
+| Arm | Codes | Best val EM | Final | Probe |
+|---|---|---|---|---|
+| A baseline | O-PARTIAL | 80.7% | 76.0% | P4 (0.3%) |
+| C filler ×16 | O4 | 53.3% | 23.7% | P4 |
+| D1 learned random | O4 | 38.7% | 16.0% | P4 |
+| D2 frozen random | O4 | 41.0% | **0.0%** | P4 |
+| D3 dynamic noise | O4 | 48.3% | **0.0%** | P4 |
+| **B structured** | 🔄 31% (~61.6k/200k) | — | — | — |
+
+- **Read:** filler/random-token arms didn't just fail to help — they actively hurt, all collapsing to O4 (can't even hold memorization), two at 0.0% final. Cuts the Pfau-2024 "pause tokens help" folklore at toy scale. **Kills S2/S3** (token-space recurrence / unsupervised filler alone does not unlock the task). Interpretation: extra compute positions are only useful if the model can learn *what to put there*; meaningless content dilutes the training signal. Everything now rides on **Arm B (structured/supervised scratchpad)**: if B transitions where 20M no-scratch steps never did → **S1** (decomposition was the missing ingredient, the project's biggest result); if B also fails → **S5** (barrier deeper than architecture).
+- **Arm B is the long pole and slow:** 61.6k/200k (31%) after ~24h at **0.72 steps/s** (~95 for short arms). Cause: the codex batch-invariance fix uses a *fixed global batch layout*, so every Arm B batch pays worst-case 369-token decoder cost; attention quadratic → ~130× per-step. My 2–4h estimate badly under-weighted this. Remaining ~138k steps ≈ **53h on the T1000, ETA ~Thu 04:00 UTC**.
+- **Delivery cron NEVER fired (09:30 Mon):** auto-removed "scheduler down past the grace window" — the desktop app's cron ticker wasn't alive at fire time. No analysis ran. Lesson: E7's *poll-and-wait late-scheduled* delivery pattern is the robust one; fixed-morning one-shots die to scheduler gaps. Will rewire Arm B's final delivery as poll-and-wait.
+- **Open decision for jayrup (a/b/c):** (a) let Arm B ride on ichi — free but 2+ days of plug-pull risk (watchdog is gone; would reinstall); (b) rent vast 3090 $0.16/h (~$5–8, finishes ~Wed) and finish there; (c) both, cancel whichever finishes second. The 5 finished arms are safe on disk either way.
+
+### 2026-09-01 — Arm B @ 62% (123k/200k), tracking BELOW the control; S1 not yet materialising
+
+- Arm B mid-run status (desktop glm-5.3-flash, from the GPU-vs-RNN thread): train EM **0.95** (nearly memorized), **val EM last 0.480, best 0.533 @106.2k**. For reference its own E7-era baseline (Arm A) finished at **80.7% best** → structured scratchpad is currently **well below** the no-scratch control, despite the same 200k budget.
+- Honest early read (final verdict still gated on the locked S1–S5 matrix): the scratchpad arm is *harder to optimize* — longer sequences, more tokens per batch, same step budget — and hasn't recouped that anywhere yet. No transition signature at 62%. Still ~77k steps for a late move.
+- **Remaining ~77k ≈ 30h @ 0.71 steps/s → ~Thu 04:00 UTC** (consistent with the 08-31 ETA). ichi uptime 5d11h. Same a/b/c decision stands: ride ichi free to Thu, or rent vast 3090 (~$3–5) for a verdict by ~Wed. Watch this arm, not the dead 5 — everything now hinges on whether *supervised* decomposition (S1) or nothing (S5).
+
+### 2026-09-05 — ichi OUTAGE #3 kills Arm B at 62% (123k/200k); vast-3090 finish option staled
+
+Thread: `Run GPU transformer alongside CPU RNN` (desktop glm-5.3-flash), jayrup pinged "done?".
+
+- **ichi dark again — outage #3**, same plug-pull/power-off failure mode ("last seen 2 days ago", died ~Tue evening). **Arm B (structured scratchpad) froze at 123k/200k (62%)** — it had NOT advanced past its 09-01 snapshot before ichi dropped. The **five finished arms remain safe on disk** (and synced locally).
+- The scratchpad verdict is still fully gated on Arm B: supervised decomposition (S1) vs nothing (S5). Filler/random arms already read as a clean negative (08-31 entry).
+- **Options for the remaining ~77k steps:** ~30h on ichi when/if it returns, OR the **vast.ai RTX 3090** (~10–14h, `$0.164/h` → **~$2–3** vs the **$3.08** credit). **No spend — needs jayrup's go-ahead** per the vast money rule (agent asked explicitly; spend did not happen).
+- **Reliability note:** the ichi-watchdog was removed after E7; three power-off-failure outages now, two of them mid-job. Reinstalling the watchdog (or moving long poles off ichi) is worth reconsidering before any future batch. Watch this arm.
+
+### 2026-09-08 — ichi recovers from outage #3; Arm B restarted fresh at 89.5%-loss; 179k-checkpoint probe first-signal leans S5
+
+Thread: `Run GPU transformer alongside CPU RNN` (desktop glm-5.3-flash, 13 msgs).
+
+- **ichi back up** (up 1d12h, relay-routed when first probed). **Arm B (structured scratchpad) frozen at 179k/200k (89.5%)** since outage #3 — tmux + trainers gone, no resume (checkpoints weights-only, no optimizer state). **Restarted fresh 200k run**, partial preserved as `seed0-partial-outage3` (a weights-only resume would contaminate the pre-registered transition hunt). Fresh run at 0.72 steps/s ≈ **~77h → ~Sat Sep 12 if no outage #4** (ich still 1.5d holed up and longest uptime in a while).
+- **"Why so slow" answered honestly — ~100× estimate error admitted:** Arm B's *fixed global batch layout* means every batch pays the worst-case **371-token** decoder cost (attention quadratic) → **0.72 steps/s vs ~95 for the short arms**, ~130× per-step. True cost ≈ **~10 days of T1000 time in one arm**. This hid in E7 because its 4M-step cells were all short-sequence arms. Outage tax on top: ~190h wall spent, only ~50h actually training. (Under-pricing first flagged 08-31; formally owned 09-08.)
+- **Free diagnostic — probe the 179k checkpoint OOD [1001,2000] on CPU** (detached, no GPU contention): **first 16 → 1/16 correct (6.3%, P4 band)**. Early signal: **no out-of-range generalization even WITH the taught structured scratchpad → S1 collapsing toward S5** (barrier deeper than architecture). Verdict held until all 1000 finish (~90 min); watching whether accuracy is *structured* (small-prime-factor composites land better → evidence of a partial learned sieve) or flat-zero (nothing transferable beyond in-range memorization). If probe ≈0% → S1 collapses to S5; if anywhere near the ~80% in-range → first OOD signal of the entire project, changes E9 design.
+- **OOM pitfall:** probe first ran co-tenant with training on the same 2GB (512-wide batches = 1.85GB attention buffer) → OOM on the 8GB card. Retried chunk-16 (`eval_batch_size` 32) on CPU, detached — clean.
+- **Delivery cron auto-removed again** (2nd time, same "scheduler down past grace window" failure) — the scheduled E8 analysis cron never fired. Final Arm B delivery must be **rewired as a poll-and-wait** job (fire when Arm B lands, whatever the clock says), the E7-robust pattern, not a fixed-morning one-shot.
+- **Finish menu (no spend yet, jayrup's call):** (a) ichi fresh run free → ~Sep 12; (b) vast 3090 ~$3–5 (~25h at 2–3× T1000 → ~Wed Sep 10); (c) both, cancel the loser. The 5 finished arms + `seed0-partial-outage3` are safe on disk.
+
+### 2026-10-09 — PROJECT FROZEN: E8 closes on a negative (filler/random dead); Arm B never finished → S1 vs S5 UNDETERMINED; research resources redirected
+
+Thread: cleanup/freeze pass on voidlaptop. `runs/e8/` rsync'd off ichi and preserved locally (50 files, 21 dirs, 28.5 MB); **NOTES-E8.md** written from the per-arm `metrics.csv`/`results.json`; code repo committed locally (not pushed — public), research notes pushed.
+
+- **E8 analysed (5 of 6 arms done). Real per-arm numbers, read from each arm's own `metrics.csv` (the committed `runs/e8/summary.csv` is blank — it was overwritten by the failed fresh-restart orchestration, NOT the batch summary; ignore it):** A baseline O-PARTIAL 80.7% best / 76.0% final, P4 (0.3%); C filler O4 53.3 / 23.7, P4 0.0%; D1 learned-random O4 38.7 / 16.0, P4 0.0%; D2 frozen-random O4 41.0 / **0.0**, P4 0.0%; D3 dynamic-noise O4 48.3 / **0.0**, P4 0.0%. **No arm reached O1** — max val EM anywhere is A's 0.8067, 0/1000 evals ≥0.9.
+- **Filler/random arms are a clean NEGATIVE → kills S2/S3.** C/D1/D2/D3 all collapse to O4 (they can't even hold train memorization: train EM max 0.50–0.71, ending 0.00–0.27) and transfer exactly nothing (P4 = 0/1000). The S3 a/b/c orderings all fail (D1 is the *lowest* of the three, inverting S3a; none is ≫ baseline, killing S3b/S3c). The "token-space recurrence alone (unsupervised pause/filler/random positions) unlocks next-prime" hypothesis is dead — extra positions only help if the model can learn *what to put there*.
+- **Arm B (structured/supervised scratchpad) — the crux arm — never finished. S1 vs S5 is therefore UNDETERMINED, not resolved.** Partial run `seed0-partial-outage3` stopped at **179k/200k (89.5%)** on **ichi outage #3**; best val EM **0.5933 @162.2k** (an earlier "0.5233" report was wrong — that's only the step-178.2k reading), final 0.5067 @179k, train EM 1.0. No resume exists (weights-only ckpts, no optimizer state), and the fresh 200k restart died at **step 200** on `CUDA error: out of memory` inside the compiled eval pass (torch.compile + cudagraph capturing the 512-wide eval batch on the 8 GB T1000; log traceback in `train.py::_eval_pass`→`evaluate_gpu`→inductor `cudagraph_trees.py::_record`). So Arm B has **no `results.json`** and its only OOD read is a 16-item spot probe (1/16, P4 band) — its O/P are "not run", not negative. Everything hinged on whether supervised decomposition (S1) escapes where 20M no-scratch steps didn't; that arm is missing, so **S5 is consistent-with but not proven** and **S1 is undetermined**.
+- **ichi instability is the proximate blocker:** three power-off outages total, #3 mid-run killed Arm B at 179k; the fixed-morning delivery cron died twice more on scheduler gaps. The 5 finished arms were never at risk (done in the first hours), but the long pole was lost to infra, not science.
+- **Decision — project FROZEN, research resources redirected.** Six independent axes now all fail to move the wall: E2 wd no-effect, E6 10× data zero-transfer, E7 20× budget no-transition, plus architecture/halting/rep/lr, and now E8 unsupervised token-space recurrence (clean negative). The verdict stands: the wall is **algorithmic (task-structure-gated), not compute-gated**. The one open question (S1 vs S5) is blocked on infra we're not going to fund: closing it would need a resumable checkpoint format (optimizer state) so an outage can't waste an arm, and a resume/memory-safe eval path off the compile+cudagraph OOM. Recorded as open, not chased further.
+- **Artifacts:** `runs/e8/` local copy verified (key files: `e8-armA-none/transformer/seed0/metrics.csv`; `e8-armB-structured/transformer/seed0-partial-outage3/metrics.csv`; `e8-armB-structured/transformer/seed0/metrics.csv`; C/D1/D2/D3 `metrics.csv`+`results.json`; `logs/`). `NOTES-E8.md` in the code repo. No training run for this pass.
diff --git a/prime-grokking/preregistration.md b/prime-grokking/preregistration.md
index 58cbe5a..2eeeed0 100644
--- a/prime-grokking/preregistration.md
+++ b/prime-grokking/preregistration.md
@@ -360,3 +360,15 @@ as specified (the long horizon may rescue memorization itself); no substitutions
 steps/s ≈ 16–18 h; pairs run concurrently → batch ≈ 36–40 h total.
 
 **E8 ([2,10000], K=64, sieve-rank probe [10001,20000]) remains gated on E7 analysis.**
+
+---
+
+## Recording note (2026-08-30) — scratchpad/filler experiment launched on ichi
+
+The 6-arm scratchpad experiment designed 2026-08-29 (this thread labels it **E9/E8** —
+naming drift vs the Aug-29 entry; see main.md experiment log) was **launched** on ichi
+(tmux `e8`, 2-way GPU pipeline, Arm B the long pole). Addendum 8 (the S1–S5 prereg) was
+locked pre-run in the **experiment repo**; the canonical research copy still carries only
+this pointer — **provenance gap remains open** (sync the full Addendum 8 text here). The
+Aug-30 probe-decode-cap fix (`3135fea`) is the key correctness correction that makes the
+S4/S5 probe verdict trustworthy.