# NOTES-E2 — Weight-decay sweep (Addendum 4, §E2) Phase analysis. Data sources: `runs/e2-wd/summary.csv` (8 rows) + per-run `results.json` under `runs/e2-wd///seed0/`, and the wd=1.0 control at `runs-take2//seed0/results.json`. Interpretation strictly against the locked E2 matrix in `design/preregistration.md` (Addendum 4). --- ## Locked reading (stated first) The E2 matrix has four verdict clauses plus a catch-all. Verdicts, in order: 1. **"O1 at ≥ 1 wd value" → grokking exists.** **NOT MET.** There is no O1 cell anywhere in the sweep. Every one of the 10 cells is either `O-PARTIAL` (train saturated, val ended mid-range) or `O4` (train never sustained saturation). No sharp 0.2→0.9 val transition at any weight decay. → **No grokking for this task at any wd tested.** 2. **"No O1 anywhere + val EM rises monotonically with wd"** (memorization attractor weakens with regularization). **PREMISE FALSE — clause does not fire.** Val EM does not rise monotonically with wd. The RNN *declines* monotonically as wd increases; the transformer is flat (a modest rise from wd 0.01 to 0.1/0.3, then a plateau). 3. **"No O1 anywhere + val EM flat or declining with wd" → memorization strongly dominant; heavier wd only hurts."** **MET — this is the operative clause.** No O1 anywhere, and val EM is flat-to-declining across the sweep. Heavier wd clearly *hurts* the RNN (best 73.3%→20.0%, last 56.7%→10.0% over 0.01→3.0) and is *neutral* for the transformer (best 80.0%→86.7% then flat; last 56.7%→~70–73% plateau). → **Memorization is strongly dominant; heavier weight decay only hurts (or does nothing).** 4. **"RNN never ≥ transformer at any wd (same-seed val EM)" → weight-tying + ACT fails to earn its keep.** **MET (one tie to note).** The RNN never *exceeds* the transformer at any wd on either metric. On the val-selected (best) EM it is strictly below at every wd (e.g. 73.3% vs 80.0% at wd 0.01; 20.0% vs 86.7% at wd 3.0). On the unselected (last) EM it ties the transformer *only* at wd 0.01 (56.7% = 56.7%) and is strictly below at every other wd. → **Architecture-negative: weight-tying + ACT does not earn its keep at this scale** (the single wd-0.01 tie on the last-checkpoint metric is the only softening). **Locked conclusion:** a clean null. No grokking at any weight decay; val EM is flat-to-declining with wd (so memorization dominates and heavier regularization doesn't reveal an algorithmic basin); and the tied RNN never beats the transformer. Per the matrix the natural next reading is that next-prime's memorization attractor is robust to the full wd range tested here. --- ## Data — all 10 cells (5 wd × 2 models, seed 0) | wd | model | O-code | P-code | H-code | val EM best | val EM last | probe acc | halt mean | |---|---|---|---|---|---|---|---|---| | 0.01 | rnn | O-PARTIAL | P4 | H1 | 73.3% | 56.7% | 1/100 | 2.00 | | 0.01 | transformer | O-PARTIAL | P4 | — | 80.0% | 56.7% | 0/100 | — | | 0.1 | rnn | O-PARTIAL | P4 | H1 | 70.0% | 56.7% | 0/100 | 2.04 | | 0.1 | transformer | O-PARTIAL | P4 | — | 86.7% | 70.0% | 0/100 | — | | 0.3 | rnn | O-PARTIAL | P4 | H1 | 56.7% | 33.3% | 0/100 | 2.48 | | 0.3 | transformer | O-PARTIAL | P4 | — | 86.7% | 73.3% | 0/100 | — | | 1.0 | rnn | O4 | P4 | H4 | 36.7% | 16.7% | 1/100 | 3.12 | | 1.0 | transformer | O-PARTIAL | P4 | — | 86.7% | 70.0% | 0/100 | — | | 3.0 | rnn | O4 | P4 | H4 | 20.0% | 10.0% | 0/100 | 3.00 | | 3.0 | transformer | O-PARTIAL | P4 | — | 86.7% | 70.0% | 0/100 | — | Notes: wd 1.0 is the pre-existing control (runs-take2, constant-lr AdamW, the E1 seed-0 row). "best" = val-selected checkpoint (selection-holed); "last" = unselected final checkpoint. Probe acc is out-of-range [101, 200]. Halt mean is RNN only (min_steps floor = 2, K = 20). --- ## Interpretive commentary (clearly separated from the locked reading — post-hoc, not pre-registered) - The sweep is remarkably flat for the transformer and monotone for the RNN, but they move in *opposite* directions. The transformer's best val EM sits at 86.7% from wd 0.1 all the way up to 3.0, and its final EM at ~70–73%. The RNN is best at the *lowest* wd (73.3% at 0.01) and collapses to 10–20% by wd 1.0–3.0. So "weight decay" is not a single knob here: it barely touches the transformer and progressively destabilizes the tied cell. - The RNN flips O-code from `O-PARTIAL` (wd ≤ 0.3) to `O4` (wd ≥ 1.0): light wd lets it saturate train and partially generalize; heavier wd stops it even from holding a memorized solution. This is the opposite of the "heavy wd pushes toward the cheap algorithm" story. - Halting never approaches `H3` (structured). It sits at the floor (H1, mean ~2.0–2.5) at low wd and goes noisy (H4, mean ~3.0–3.1, ρ ≈ 0.22→0.007) at high wd. ACT contributes nothing at any wd. - Probe transfer is exactly zero everywhere (P4 in all 10 cells; only two RNN cells score a single lucky point, wd 0.01 and wd 1.0). Whatever solves the in-range holdout — up to 86.7% for the transformer — never generalizes one step outside [2, 100]. ## Caveats - Single seed (seed 0). E1 showed O/P/H codes are seed-stable across seeds {0,1,2} for the wd 1.0 control, but the sweep itself has no seed replication. - "best" val EM is selection-holed (chosen on val); "last" is the honest unselected number. The architecture comparison in clause 4 is the only place the two diverge (the wd-0.01 tie exists on "last" only). - No O1 cell means the "flat/declining" clause is the operative reading; it is a sweep-level clause and the two models behave differently within it (RNN declining, transformer flat), which the locked matrix does not further subdivide. - 69 training examples, in-range n ∈ [2, 100]; nothing here speaks to larger ranges. ## E3 / E4 implications - **E3 (lr annealing).** E2 already shows the wd axis does not reveal an algorithmic basin (no O1 at any wd), so annealing alone should not be expected to produce the transition; its locked gate (does final val EM beat the constant-lr control and stop decaying?) remains the right test for *why* the late-run decay happens — optimizer dynamics vs solution instability — independent of grokking. - **E4 (halting=False ablation).** E2 gives no `H3` anywhere (H1 at low wd, H4 at high wd), consistent with "ACT didn't help", which is exactly the condition that makes E4 worth running. But at wd 1.0/3.0 the RNN is `O4` (cannot even sustain memorization), so the fixed-K vs ACT comparison should be read with that instability in mind; the informative E4 comparison is at a wd where the RNN at least saturates train (≤ 0.3).