1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
|
# NOTES-E2 — Weight-decay sweep (Addendum 4, §E2)
Phase analysis. Data sources: `runs/e2-wd/summary.csv` (8 rows) + per-run `results.json`
under `runs/e2-wd/<job>/<model>/seed0/`, and the wd=1.0 control at
`runs-take2/<model>/seed0/results.json`. Interpretation strictly against the locked E2
matrix in `design/preregistration.md` (Addendum 4).
---
## Locked reading (stated first)
The E2 matrix has four verdict clauses plus a catch-all. Verdicts, in order:
1. **"O1 at ≥ 1 wd value" → grokking exists.** **NOT MET.** There is no O1 cell anywhere in
the sweep. Every one of the 10 cells is either `O-PARTIAL` (train saturated, val ended
mid-range) or `O4` (train never sustained saturation). No sharp 0.2→0.9 val transition
at any weight decay. → **No grokking for this task at any wd tested.**
2. **"No O1 anywhere + val EM rises monotonically with wd"** (memorization attractor weakens
with regularization). **PREMISE FALSE — clause does not fire.** Val EM does not rise
monotonically with wd. The RNN *declines* monotonically as wd increases; the transformer
is flat (a modest rise from wd 0.01 to 0.1/0.3, then a plateau).
3. **"No O1 anywhere + val EM flat or declining with wd" → memorization strongly dominant;
heavier wd only hurts."** **MET — this is the operative clause.** No O1 anywhere, and val
EM is flat-to-declining across the sweep. Heavier wd clearly *hurts* the RNN
(best 73.3%→20.0%, last 56.7%→10.0% over 0.01→3.0) and is *neutral* for the transformer
(best 80.0%→86.7% then flat; last 56.7%→~70–73% plateau). → **Memorization is strongly
dominant; heavier weight decay only hurts (or does nothing).**
4. **"RNN never ≥ transformer at any wd (same-seed val EM)" → weight-tying + ACT fails to
earn its keep.** **MET (one tie to note).** The RNN never *exceeds* the transformer at any
wd on either metric. On the val-selected (best) EM it is strictly below at every wd
(e.g. 73.3% vs 80.0% at wd 0.01; 20.0% vs 86.7% at wd 3.0). On the unselected (last) EM
it ties the transformer *only* at wd 0.01 (56.7% = 56.7%) and is strictly below at every
other wd. → **Architecture-negative: weight-tying + ACT does not earn its keep at this
scale** (the single wd-0.01 tie on the last-checkpoint metric is the only softening).
**Locked conclusion:** a clean null. No grokking at any weight decay; val EM is
flat-to-declining with wd (so memorization dominates and heavier regularization doesn't
reveal an algorithmic basin); and the tied RNN never beats the transformer. Per the matrix
the natural next reading is that next-prime's memorization attractor is robust to the full
wd range tested here.
---
## Data — all 10 cells (5 wd × 2 models, seed 0)
| wd | model | O-code | P-code | H-code | val EM best | val EM last | probe acc | halt mean |
|---|---|---|---|---|---|---|---|---|
| 0.01 | rnn | O-PARTIAL | P4 | H1 | 73.3% | 56.7% | 1/100 | 2.00 |
| 0.01 | transformer | O-PARTIAL | P4 | — | 80.0% | 56.7% | 0/100 | — |
| 0.1 | rnn | O-PARTIAL | P4 | H1 | 70.0% | 56.7% | 0/100 | 2.04 |
| 0.1 | transformer | O-PARTIAL | P4 | — | 86.7% | 70.0% | 0/100 | — |
| 0.3 | rnn | O-PARTIAL | P4 | H1 | 56.7% | 33.3% | 0/100 | 2.48 |
| 0.3 | transformer | O-PARTIAL | P4 | — | 86.7% | 73.3% | 0/100 | — |
| 1.0 | rnn | O4 | P4 | H4 | 36.7% | 16.7% | 1/100 | 3.12 |
| 1.0 | transformer | O-PARTIAL | P4 | — | 86.7% | 70.0% | 0/100 | — |
| 3.0 | rnn | O4 | P4 | H4 | 20.0% | 10.0% | 0/100 | 3.00 |
| 3.0 | transformer | O-PARTIAL | P4 | — | 86.7% | 70.0% | 0/100 | — |
Notes: wd 1.0 is the pre-existing control (runs-take2, constant-lr AdamW, the E1 seed-0
row). "best" = val-selected checkpoint (selection-holed); "last" = unselected final
checkpoint. Probe acc is out-of-range [101, 200]. Halt mean is RNN only (min_steps floor
= 2, K = 20).
---
## Interpretive commentary (clearly separated from the locked reading — post-hoc, not pre-registered)
- The sweep is remarkably flat for the transformer and monotone for the RNN, but they move
in *opposite* directions. The transformer's best val EM sits at 86.7% from wd 0.1 all the
way up to 3.0, and its final EM at ~70–73%. The RNN is best at the *lowest* wd (73.3% at
0.01) and collapses to 10–20% by wd 1.0–3.0. So "weight decay" is not a single knob here:
it barely touches the transformer and progressively destabilizes the tied cell.
- The RNN flips O-code from `O-PARTIAL` (wd ≤ 0.3) to `O4` (wd ≥ 1.0): light wd lets it
saturate train and partially generalize; heavier wd stops it even from holding a memorized
solution. This is the opposite of the "heavy wd pushes toward the cheap algorithm" story.
- Halting never approaches `H3` (structured). It sits at the floor (H1, mean ~2.0–2.5) at
low wd and goes noisy (H4, mean ~3.0–3.1, ρ ≈ 0.22→0.007) at high wd. ACT contributes
nothing at any wd.
- Probe transfer is exactly zero everywhere (P4 in all 10 cells; only two RNN cells score a
single lucky point, wd 0.01 and wd 1.0). Whatever solves the in-range holdout — up to
86.7% for the transformer — never generalizes one step outside [2, 100].
## Caveats
- Single seed (seed 0). E1 showed O/P/H codes are seed-stable across seeds {0,1,2} for the
wd 1.0 control, but the sweep itself has no seed replication.
- "best" val EM is selection-holed (chosen on val); "last" is the honest unselected number.
The architecture comparison in clause 4 is the only place the two diverge (the wd-0.01
tie exists on "last" only).
- No O1 cell means the "flat/declining" clause is the operative reading; it is a sweep-level
clause and the two models behave differently within it (RNN declining, transformer flat),
which the locked matrix does not further subdivide.
- 69 training examples, in-range n ∈ [2, 100]; nothing here speaks to larger ranges.
## E3 / E4 implications
- **E3 (lr annealing).** E2 already shows the wd axis does not reveal an algorithmic basin
(no O1 at any wd), so annealing alone should not be expected to produce the transition;
its locked gate (does final val EM beat the constant-lr control and stop decaying?) remains
the right test for *why* the late-run decay happens — optimizer dynamics vs solution
instability — independent of grokking.
- **E4 (halting=False ablation).** E2 gives no `H3` anywhere (H1 at low wd, H4 at high wd),
consistent with "ACT didn't help", which is exactly the condition that makes E4 worth
running. But at wd 1.0/3.0 the RNN is `O4` (cannot even sustain memorization), so the
fixed-K vs ACT comparison should be read with that instability in mind; the informative
E4 comparison is at a wd where the RNN at least saturates train (≤ 0.3).
|
