E2 wd sweep results + analysis + report phase 3

1 files changed, 109 insertions, 0 deletions

diff --git a/NOTES-E2.md b/NOTES-E2.md
new file mode 100644
index 0000000..8db68f4
--- /dev/null
+++ b/NOTES-E2.md
@@ -0,0 +1,109 @@
+# NOTES-E2 — Weight-decay sweep (Addendum 4, §E2)
+
+Phase analysis. Data sources: `runs/e2-wd/summary.csv` (8 rows) + per-run `results.json`
+under `runs/e2-wd/<job>/<model>/seed0/`, and the wd=1.0 control at
+`runs-take2/<model>/seed0/results.json`. Interpretation strictly against the locked E2
+matrix in `design/preregistration.md` (Addendum 4).
+
+---
+
+## Locked reading (stated first)
+
+The E2 matrix has four verdict clauses plus a catch-all. Verdicts, in order:
+
+1. **"O1 at ≥ 1 wd value" → grokking exists.** **NOT MET.** There is no O1 cell anywhere in
+   the sweep. Every one of the 10 cells is either `O-PARTIAL` (train saturated, val ended
+   mid-range) or `O4` (train never sustained saturation). No sharp 0.2→0.9 val transition
+   at any weight decay. → **No grokking for this task at any wd tested.**
+
+2. **"No O1 anywhere + val EM rises monotonically with wd"** (memorization attractor weakens
+   with regularization). **PREMISE FALSE — clause does not fire.** Val EM does not rise
+   monotonically with wd. The RNN *declines* monotonically as wd increases; the transformer
+   is flat (a modest rise from wd 0.01 to 0.1/0.3, then a plateau).
+
+3. **"No O1 anywhere + val EM flat or declining with wd" → memorization strongly dominant;
+   heavier wd only hurts."** **MET — this is the operative clause.** No O1 anywhere, and val
+   EM is flat-to-declining across the sweep. Heavier wd clearly *hurts* the RNN
+   (best 73.3%→20.0%, last 56.7%→10.0% over 0.01→3.0) and is *neutral* for the transformer
+   (best 80.0%→86.7% then flat; last 56.7%→~70–73% plateau). → **Memorization is strongly
+   dominant; heavier weight decay only hurts (or does nothing).**
+
+4. **"RNN never ≥ transformer at any wd (same-seed val EM)" → weight-tying + ACT fails to
+   earn its keep.** **MET (one tie to note).** The RNN never *exceeds* the transformer at any
+   wd on either metric. On the val-selected (best) EM it is strictly below at every wd
+   (e.g. 73.3% vs 80.0% at wd 0.01; 20.0% vs 86.7% at wd 3.0). On the unselected (last) EM
+   it ties the transformer *only* at wd 0.01 (56.7% = 56.7%) and is strictly below at every
+   other wd. → **Architecture-negative: weight-tying + ACT does not earn its keep at this
+   scale** (the single wd-0.01 tie on the last-checkpoint metric is the only softening).
+
+**Locked conclusion:** a clean null. No grokking at any weight decay; val EM is
+flat-to-declining with wd (so memorization dominates and heavier regularization doesn't
+reveal an algorithmic basin); and the tied RNN never beats the transformer. Per the matrix
+the natural next reading is that next-prime's memorization attractor is robust to the full
+wd range tested here.
+
+---
+
+## Data — all 10 cells (5 wd × 2 models, seed 0)
+
+| wd | model | O-code | P-code | H-code | val EM best | val EM last | probe acc | halt mean |
+|---|---|---|---|---|---|---|---|---|
+| 0.01 | rnn | O-PARTIAL | P4 | H1 | 73.3% | 56.7% | 1/100 | 2.00 |
+| 0.01 | transformer | O-PARTIAL | P4 | — | 80.0% | 56.7% | 0/100 | — |
+| 0.1 | rnn | O-PARTIAL | P4 | H1 | 70.0% | 56.7% | 0/100 | 2.04 |
+| 0.1 | transformer | O-PARTIAL | P4 | — | 86.7% | 70.0% | 0/100 | — |
+| 0.3 | rnn | O-PARTIAL | P4 | H1 | 56.7% | 33.3% | 0/100 | 2.48 |
+| 0.3 | transformer | O-PARTIAL | P4 | — | 86.7% | 73.3% | 0/100 | — |
+| 1.0 | rnn | O4 | P4 | H4 | 36.7% | 16.7% | 1/100 | 3.12 |
+| 1.0 | transformer | O-PARTIAL | P4 | — | 86.7% | 70.0% | 0/100 | — |
+| 3.0 | rnn | O4 | P4 | H4 | 20.0% | 10.0% | 0/100 | 3.00 |
+| 3.0 | transformer | O-PARTIAL | P4 | — | 86.7% | 70.0% | 0/100 | — |
+
+Notes: wd 1.0 is the pre-existing control (runs-take2, constant-lr AdamW, the E1 seed-0
+row). "best" = val-selected checkpoint (selection-holed); "last" = unselected final
+checkpoint. Probe acc is out-of-range [101, 200]. Halt mean is RNN only (min_steps floor
+= 2, K = 20).
+
+---
+
+## Interpretive commentary (clearly separated from the locked reading — post-hoc, not pre-registered)
+
+- The sweep is remarkably flat for the transformer and monotone for the RNN, but they move
+  in *opposite* directions. The transformer's best val EM sits at 86.7% from wd 0.1 all the
+  way up to 3.0, and its final EM at ~70–73%. The RNN is best at the *lowest* wd (73.3% at
+  0.01) and collapses to 10–20% by wd 1.0–3.0. So "weight decay" is not a single knob here:
+  it barely touches the transformer and progressively destabilizes the tied cell.
+- The RNN flips O-code from `O-PARTIAL` (wd ≤ 0.3) to `O4` (wd ≥ 1.0): light wd lets it
+  saturate train and partially generalize; heavier wd stops it even from holding a memorized
+  solution. This is the opposite of the "heavy wd pushes toward the cheap algorithm" story.
+- Halting never approaches `H3` (structured). It sits at the floor (H1, mean ~2.0–2.5) at
+  low wd and goes noisy (H4, mean ~3.0–3.1, ρ ≈ 0.22→0.007) at high wd. ACT contributes
+  nothing at any wd.
+- Probe transfer is exactly zero everywhere (P4 in all 10 cells; only two RNN cells score a
+  single lucky point, wd 0.01 and wd 1.0). Whatever solves the in-range holdout — up to
+  86.7% for the transformer — never generalizes one step outside [2, 100].
+
+## Caveats
+
+- Single seed (seed 0). E1 showed O/P/H codes are seed-stable across seeds {0,1,2} for the
+  wd 1.0 control, but the sweep itself has no seed replication.
+- "best" val EM is selection-holed (chosen on val); "last" is the honest unselected number.
+  The architecture comparison in clause 4 is the only place the two diverge (the wd-0.01
+  tie exists on "last" only).
+- No O1 cell means the "flat/declining" clause is the operative reading; it is a sweep-level
+  clause and the two models behave differently within it (RNN declining, transformer flat),
+  which the locked matrix does not further subdivide.
+- 69 training examples, in-range n ∈ [2, 100]; nothing here speaks to larger ranges.
+
+## E3 / E4 implications
+
+- **E3 (lr annealing).** E2 already shows the wd axis does not reveal an algorithmic basin
+  (no O1 at any wd), so annealing alone should not be expected to produce the transition;
+  its locked gate (does final val EM beat the constant-lr control and stop decaying?) remains
+  the right test for *why* the late-run decay happens — optimizer dynamics vs solution
+  instability — independent of grokking.
+- **E4 (halting=False ablation).** E2 gives no `H3` anywhere (H1 at low wd, H4 at high wd),
+  consistent with "ACT didn't help", which is exactly the condition that makes E4 worth
+  running. But at wd 1.0/3.0 the RNN is `O4` (cannot even sustain memorization), so the
+  fixed-K vs ACT comparison should be read with that instability in mind; the informative
+  E4 comparison is at a wd where the RNN at least saturates train (≤ 0.3).