prime-grokking: publish part 2 (wd sweep) + part 1 seed-replication update

3 files changed, 234 insertions, 15 deletions

diff --git a/blog/index.qmd b/blog/index.qmd
index 0eca72e..d46e6ec 100644
--- a/blog/index.qmd
+++ b/blog/index.qmd
@@ -5,6 +5,8 @@ title: "Jayrup Nakawala | Blog"
 RANDOM THOUGHTS, EXPERIMENTS, AND SYSTEMS NOTES
 --------------------------------------------------
 
+- Still No Grokking: The Weight-Decay Sweep — 2026-08-16  
+  curl [jayrup.me/blog/prime-grokking-2](https://jayrup.me/blog/prime-grokking-2)
 - Can a Tiny Network Learn What a Prime Is? — 2026-08-14  
   curl [jayrup.me/blog/prime-grokking-1](https://jayrup.me/blog/prime-grokking-1)
 - What the Jacobian Lens Measures — 2026-08-07  
diff --git a/blog/prime-grokking-1.qmd b/blog/prime-grokking-1.qmd
index 23c97b1..97b1fa2 100644
--- a/blog/prime-grokking-1.qmd
+++ b/blog/prime-grokking-1.qmd
@@ -7,6 +7,10 @@ date: "2026-08-14"
 to see whether either would discover the algorithm instead of memorizing the
 answers. Neither did. The way they failed is more interesting than the failure.*
 
+*Updated 2026-08-16: seeds 1–2 replication added (section 8). The
+weight-decay sweep is a separate post:
+[prime-grokking-2](https://jayrup.me/blog/prime-grokking-2).*
+
 ---
 
 **The short version.** I asked whether a minimal neural network can learn the
@@ -245,9 +249,10 @@ Reading strictly from the locked matrix:
    emerge on its own; it collapsed toward the floor. That's its own small
    result: ACT without stronger shaping didn't produce structure here.
 
-**Honesty note.** This is one seed of a 69-example toy experiment. Per the
-pre-registration, seeds 1 and 2 are required before any claim beyond
-"seed-0 result". The val number for the transformer's best checkpoint (86.7%)
+**Honesty note.** This is a 69-example toy experiment. The numbers above
+are the seed-0 run; the seed replication (section 8) confirmed them, and
+the weight-decay sweep is in [part 2](https://jayrup.me/blog/prime-grokking-2).
+The val number for the transformer's best checkpoint (86.7%)
 is also selection-holed — I used validation for early stopping, so the peak
 is optimistic; the unselected final checkpoint's 70% is the cleaner number.
 And one engineering asterisk: the first attempt at these runs was killed by a
@@ -266,12 +271,10 @@ task, it's not the one these optimizer settings settle into.
 
 ## 6. Where this goes next
 
-In rough priority order:
+Status: the seed replication is done (section 8), and the weight-decay
+sweep is published as [part 2](https://jayrup.me/blog/prime-grokking-2).
+Remaining, in rough priority order:
 
-- **Seeds 1, 2** — the mandatory replication before anything is claimed.
-- **Weight-decay sweep {0.01, 0.1, 0.3, 1.0, 3.0}** — the O2 recipe: if
-  memorization wins, turn the regularization up (and down, for a control) and
-  watch whether any setting produces the jump.
 - **Learning-rate annealing** — a targeted experiment against the late-run
   decay specifically.
 - **Smaller training fraction (40% / 50%)** — fewer examples, more pressure
@@ -285,11 +288,49 @@ In rough priority order:
 
 ## 7. Reproduce
 
-The repo is private for now — it will go public once the seed replication
-lands and the writeup is final. 34 tests cover data correctness, model
+The repo is private for now — it will go public once the writeup series is
+final. 34 tests cover data correctness, model
 shapes, halting, layout invariance, and the evaluator's classification logic
 against ground-truth stub models. Each arm trains in ~2h on 2 CPU cores.
 
+## 8. Update: the seed replication
+
+Per the pre-registration, seeds 1 and 2 were required before any claim
+beyond "seed-0 result". I re-ran both models with identical
+hyperparameters, changing only the seed. The locked criterion: a result is
+**replicated** if it shows the same O-code and P-code across ≥ 2 of the 3
+seeds (0, 1, 2), plus the same H-code for the RNN.
+
+The answer is a clean, boring confirmation — boring in exactly the way a
+replication is supposed to be.
+
+```
+  model        seed  O          H   P   val best  val last  probe (101–200)
+  transformer  0     O-PARTIAL  —   P4  86.7%     70.0%     0 / 100
+  transformer  1     O-PARTIAL  —   P4  70.0%     43.3%     0 / 100
+  transformer  2     O-PARTIAL  —   P4  80.0%     73.3%     0 / 100
+  RNN          0     O4         H4  P4  36.7%     16.7%     1 / 100
+  RNN          1     O4         H4  P4  36.7%     20.0%     0 / 100
+  RNN          2     O4         H4  P4  36.7%     23.3%     0 / 100
+```
+
+Both models hit the bar in **all three seeds**, not just the required two.
+The transformer's O-PARTIAL + P4 is seed-stable: partial in-range
+generalization (val best 70–87%) with **zero** out-of-range transfer in
+every run. The RNN's O4 / H4 + P4 is equally stable: the tied cell couldn't
+hold a memorized solution (val best stuck at 36.7% across all three seeds),
+and its halting gate stayed unstructured near the floor (mean 2.8–3.6 steps
+of 20). No disagreement to report — so per the locked matrix these are now
+real, seed-stable findings, and the weight-decay sweep could proceed on
+seed 0 as planned. That sweep is [part 2](https://jayrup.me/blog/prime-grokking-2).
+
+Caveats unchanged: three seeds of a 69-example toy task is replication, not
+a variance estimate; the "best" column is val-selected and therefore
+selection-holed (the "last" column is the unselected checkpoint); the
+late-run val decay seen in seed 0 recurs in both models; and the probe
+transfer is exactly zero everywhere, so there is nothing downstream to
+over-interpret.
+
 ---
 
 RESOURCES
@@ -297,12 +338,12 @@ RESOURCES
 
 - Grokking (machine learning) — Wikipedia overview:
   https://en.wikipedia.org/wiki/Grokking_(machine_learning)
-- Code for this experiment (goes public once the seed replication lands):
+- Code for this experiment (will be made public):
   <repo link — pending>
-- Part 2: the seed sweep (when published):
-  <link — pending>
+- Part 2 — the weight-decay sweep:
+  https://jayrup.me/blog/prime-grokking-2
 
 ---
 
-*This is part 1. The seed sweep is running next; whatever it shows —
-confirmation or refutation — becomes part 2.*
+*This is part 1. Part 2 covers the weight-decay sweep; the range-extension
+experiment and the halting ablation are the natural part 3.*
diff --git a/blog/prime-grokking-2.qmd b/blog/prime-grokking-2.qmd
new file mode 100644
index 0000000..eceb21a
--- /dev/null
+++ b/blog/prime-grokking-2.qmd
@@ -0,0 +1,176 @@
+---
+title: "Still No Grokking: The Weight-Decay Sweep"
+date: "2026-08-16"
+---
+
+*Part 1 found no grokking at weight decay 1.0, and the seed replication
+confirmed it. So I did the obvious thing: swept the weight decay across
+five orders of magnitude and watched. Still no grokking. The way it
+fails is the result.*
+
+---
+
+**The short version.** The grokking recipe says: if memorization wins,
+turn the regularization up until the cheap algorithmic solution
+outcompetes the lookup table. I re-ran both models across weight decay
+{0.01, 0.1, 0.3, 1.0, 3.0} — a 50× range around the original 1.0 — with
+everything else locked. No grokking at any setting. Validation accuracy
+does not rise with weight decay; it's flat (transformer) to declining
+(RNN). The weight-tied RNN never beats the transformer at any setting —
+tying it only once at the lowest decay on the unselected checkpoint.
+And the tied RNN, the architecture built to *want* the algorithmic
+solution, gets *worse* the heavier the regularization gets. The opposite
+of the grokking story.
+
+---
+
+## 1. Recap: where part 1 left off
+
+In [part 1](https://jayrup.me/blog/prime-grokking-1) I trained two toy
+models on the next-prime function: a plain transformer (407,947 params)
+and a weight-tied recurrent cell with adaptive computation time (36,620
+params) — the "for-loop model" that could in principle represent trial
+division cheaply. Both trained on 69 examples (n ∈ [2, 100]) with heavy
+weight decay (1.0), the strong-regularization regime where grokking
+lives.
+
+Result: no grokking. The transformer memorized and partially generalized
+(peaked at 86.7% validation, decayed to 70%); the tied RNN couldn't even
+hold a memorized solution (16.7% at the end). Out-of-range transfer was
+zero. Then the seed replication (three seeds, identical everything else)
+confirmed all of it: same locked codes in every seed.
+
+That leaves the obvious knob unturned. The whole *theory* of why grokking
+happens is that weight decay makes the algorithmic solution cheaper than
+the memorized one. Weight decay 1.0 was one point on that axis. What if
+the basin lives at 0.01? Or 3.0?
+
+## 2. The sweep
+
+Five weight-decay values — {0.01, 0.1, 0.3, 1.0, 3.0} — × both models,
+seed 0, everything else identical to part 1 (AdamW, lr 1e-3, 200k steps).
+The 1.0 cells are the existing seed-0 runs, reused as control. The
+interpretation was locked in the pre-registration before any of it ran:
+if any setting produced the O1 signature (validation jumping 0.2 → 0.9
+within a few evals), that's grokking. If validation stayed flat or fell
+as decay increased, that's the "memorization strongly dominant" clause.
+
+## 3. What happened
+
+```
+  wd    model        O          H   P   val best  val last  probe  halt mean
+  0.01  RNN          O-PARTIAL  H1  P4  73.3%     56.7%     1/100  2.00
+  0.01  transformer  O-PARTIAL  —   P4  80.0%     56.7%     0/100  —
+  0.1   RNN          O-PARTIAL  H1  P4  70.0%     56.7%     0/100  2.04
+  0.1   transformer  O-PARTIAL  —   P4  86.7%     70.0%     0/100  —
+  0.3   RNN          O-PARTIAL  H1  P4  56.7%     33.3%     0/100  2.48
+  0.3   transformer  O-PARTIAL  —   P4  86.7%     73.3%     0/100  —
+  1.0   RNN          O4         H4  P4  36.7%     16.7%     1/100  3.12
+  1.0   transformer  O-PARTIAL  —   P4  86.7%     70.0%     0/100  —
+  3.0   RNN          O4         H4  P4  20.0%     10.0%     0/100  3.00
+  3.0   transformer  O-PARTIAL  —   P4  86.7%     70.0%     0/100  —
+```
+
+## 4. What it means (locked reading)
+
+Reading strictly against the pre-registered E2 matrix:
+
+1. **No O1 cell anywhere.** Ten cells, five weight-decay values, two
+   architectures — no sharp 0.2 → 0.9 validation transition in any of
+   them. The grokking jump does not exist for this task at any
+   regularization strength tested.
+2. **Validation does not rise with weight decay.** The premise of the
+   "grokking at higher wd" clause is false. The transformer is flat
+   (80–87% best across the whole range); the RNN *declines* monotonically
+   as decay increases (73.3% at 0.01 → 20.0% at 3.0). This lands on the
+   locked clause: **memorization is strongly dominant; heavier weight
+   decay only hurts.** For the RNN, literally. For the transformer, it
+   does nothing.
+3. **The tied RNN never beats the transformer at any wd.** It ties once —
+   at wd 0.01, on the unselected final checkpoint (56.7% = 56.7%) — and
+   loses everywhere else. The architecture built to represent the
+   algorithm cheaply is worse than the plain one at every setting. Per
+   the matrix, that's an architecture-negative result: weight-tying plus
+   ACT does not earn its keep at this scale.
+4. **Halting never structured itself.** H1 (collapsed near the floor,
+   mean ~2.0–2.5 steps of 20) at low wd, H4 (noisy, ~3.0) at high wd.
+   Adaptive computation contributes nothing at any regularization
+   strength.
+5. **Probe transfer stays exactly zero everywhere.** P4 in all ten cells.
+   Whatever the transformer is doing to reach 86.7% on held-out inputs
+   never generalizes one step outside [2, 100].
+
+## 5. The part that's actually interesting
+
+The locked reading is the clean null. But there's a post-hoc observation
+worth separating out, because it's the opposite of what the grokking
+theory would predict:
+
+**Weight decay barely touches the transformer and progressively
+destabilizes the tied cell — two different models, two different
+responses to the same knob.** The transformer sits at 86.7% best from
+wd 0.1 all the way to 3.0, like the regularization is a fly on its back.
+The RNN is best at the *lowest* decay (73.3% at 0.01) and collapses as
+decay rises — it flips from O-PARTIAL (can saturate train, partially
+generalize) at wd ≤ 0.3 to O4 (can't even hold a memorized solution) at
+wd ≥ 1.0.
+
+That's the wrong direction for the theory. The whole bet was: heavy
+decay pushes toward the cheap algorithmic solution. Instead, heavy decay
+pushes the tied cell *away* from even the memorized solution, while the
+plain transformer is indifferent. The weight-tied architecture was
+supposed to be the one that benefits from regularization; it's the one
+that's most fragile to it.
+
+And the halting numbers tell the same story in miniature: the gate moves
+from the floor (2.0 steps) at low wd to noisy (3.0–3.1) at high wd —
+the network uses *slightly more* computation when its weights are being
+squeezed, but it never uses it for anything structured.
+
+## 6. Caveats
+
+- **Seed 0 only for the sweep.** The wd=1.0 codes were seed-stable in
+  the E1 replication, but the sweep itself is un-replicated. The
+  qualitative picture — no O1 anywhere, RNN declines, transformer flat —
+  is the claim, not the exact percentages.
+- **"Best" is selection-holed.** The val-best column is chosen on
+  validation (optimistic); the "last" column is the honest unselected
+  checkpoint. The architecture comparison uses the numbers both ways and
+  only the one tie changes.
+- This is still 69 training examples on n ∈ [2, 100]. Nothing here
+  speaks to larger ranges — which is exactly why the range extension is
+  the experiment that actually discriminates.
+
+## 7. Where this goes next
+
+The weight-decay axis is now mapped and it's flat. That sharpens the
+remaining candidates:
+
+- **Learning-rate annealing** — the late-run decay (both models peak
+  mid-training and roll over) is now the most interesting unexplained
+  behaviour. Annealing is the targeted test of whether it's optimizer
+  dynamics or solution instability.
+- **halting=False ablation** — the informative comparison is at a wd
+  where the RNN can at least saturate train (≤ 0.3), since at 1.0–3.0
+  it's O4 and the comparison would be muddied.
+- **The range extension [2, 1000]** — the discriminator. At ~700
+  training examples the lookup table stops fitting comfortably; the
+  algorithm must win for the task to work at all. If grokking exists for
+  this task, that's where it lives.
+
+---
+
+RESOURCES
+---------
+
+- Part 1 — the original experiment and seed replication:
+  https://jayrup.me/blog/prime-grokking-1
+- Grokking (machine learning) — Wikipedia overview:
+  https://en.wikipedia.org/wiki/Grokking_(machine_learning)
+- Code for this experiment (will be made public):
+  <repo link — pending>
+
+---
+
+*Weight decay was the obvious knob. It's now a documented dead end.
+Next: the range where memorization stops being an option.*