prime-grokking: publish part 2 (wd sweep) + part 1 seed-replication update

1 files changed, 56 insertions, 15 deletions

diff --git a/blog/prime-grokking-1.qmd b/blog/prime-grokking-1.qmd
index 23c97b1..97b1fa2 100644
--- a/blog/prime-grokking-1.qmd
+++ b/blog/prime-grokking-1.qmd
@@ -7,6 +7,10 @@ date: "2026-08-14"
 to see whether either would discover the algorithm instead of memorizing the
 answers. Neither did. The way they failed is more interesting than the failure.*
 
+*Updated 2026-08-16: seeds 1–2 replication added (section 8). The
+weight-decay sweep is a separate post:
+[prime-grokking-2](https://jayrup.me/blog/prime-grokking-2).*
+
 ---
 
 **The short version.** I asked whether a minimal neural network can learn the
@@ -245,9 +249,10 @@ Reading strictly from the locked matrix:
    emerge on its own; it collapsed toward the floor. That's its own small
    result: ACT without stronger shaping didn't produce structure here.
 
-**Honesty note.** This is one seed of a 69-example toy experiment. Per the
-pre-registration, seeds 1 and 2 are required before any claim beyond
-"seed-0 result". The val number for the transformer's best checkpoint (86.7%)
+**Honesty note.** This is a 69-example toy experiment. The numbers above
+are the seed-0 run; the seed replication (section 8) confirmed them, and
+the weight-decay sweep is in [part 2](https://jayrup.me/blog/prime-grokking-2).
+The val number for the transformer's best checkpoint (86.7%)
 is also selection-holed — I used validation for early stopping, so the peak
 is optimistic; the unselected final checkpoint's 70% is the cleaner number.
 And one engineering asterisk: the first attempt at these runs was killed by a
@@ -266,12 +271,10 @@ task, it's not the one these optimizer settings settle into.
 
 ## 6. Where this goes next
 
-In rough priority order:
+Status: the seed replication is done (section 8), and the weight-decay
+sweep is published as [part 2](https://jayrup.me/blog/prime-grokking-2).
+Remaining, in rough priority order:
 
-- **Seeds 1, 2** — the mandatory replication before anything is claimed.
-- **Weight-decay sweep {0.01, 0.1, 0.3, 1.0, 3.0}** — the O2 recipe: if
-  memorization wins, turn the regularization up (and down, for a control) and
-  watch whether any setting produces the jump.
 - **Learning-rate annealing** — a targeted experiment against the late-run
   decay specifically.
 - **Smaller training fraction (40% / 50%)** — fewer examples, more pressure
@@ -285,11 +288,49 @@ In rough priority order:
 
 ## 7. Reproduce
 
-The repo is private for now — it will go public once the seed replication
-lands and the writeup is final. 34 tests cover data correctness, model
+The repo is private for now — it will go public once the writeup series is
+final. 34 tests cover data correctness, model
 shapes, halting, layout invariance, and the evaluator's classification logic
 against ground-truth stub models. Each arm trains in ~2h on 2 CPU cores.
 
+## 8. Update: the seed replication
+
+Per the pre-registration, seeds 1 and 2 were required before any claim
+beyond "seed-0 result". I re-ran both models with identical
+hyperparameters, changing only the seed. The locked criterion: a result is
+**replicated** if it shows the same O-code and P-code across ≥ 2 of the 3
+seeds (0, 1, 2), plus the same H-code for the RNN.
+
+The answer is a clean, boring confirmation — boring in exactly the way a
+replication is supposed to be.
+
+```
+  model        seed  O          H   P   val best  val last  probe (101–200)
+  transformer  0     O-PARTIAL  —   P4  86.7%     70.0%     0 / 100
+  transformer  1     O-PARTIAL  —   P4  70.0%     43.3%     0 / 100
+  transformer  2     O-PARTIAL  —   P4  80.0%     73.3%     0 / 100
+  RNN          0     O4         H4  P4  36.7%     16.7%     1 / 100
+  RNN          1     O4         H4  P4  36.7%     20.0%     0 / 100
+  RNN          2     O4         H4  P4  36.7%     23.3%     0 / 100
+```
+
+Both models hit the bar in **all three seeds**, not just the required two.
+The transformer's O-PARTIAL + P4 is seed-stable: partial in-range
+generalization (val best 70–87%) with **zero** out-of-range transfer in
+every run. The RNN's O4 / H4 + P4 is equally stable: the tied cell couldn't
+hold a memorized solution (val best stuck at 36.7% across all three seeds),
+and its halting gate stayed unstructured near the floor (mean 2.8–3.6 steps
+of 20). No disagreement to report — so per the locked matrix these are now
+real, seed-stable findings, and the weight-decay sweep could proceed on
+seed 0 as planned. That sweep is [part 2](https://jayrup.me/blog/prime-grokking-2).
+
+Caveats unchanged: three seeds of a 69-example toy task is replication, not
+a variance estimate; the "best" column is val-selected and therefore
+selection-holed (the "last" column is the unselected checkpoint); the
+late-run val decay seen in seed 0 recurs in both models; and the probe
+transfer is exactly zero everywhere, so there is nothing downstream to
+over-interpret.
+
 ---
 
 RESOURCES
@@ -297,12 +338,12 @@ RESOURCES
 
 - Grokking (machine learning) — Wikipedia overview:
   https://en.wikipedia.org/wiki/Grokking_(machine_learning)
-- Code for this experiment (goes public once the seed replication lands):
+- Code for this experiment (will be made public):
   <repo link — pending>
-- Part 2: the seed sweep (when published):
-  <link — pending>
+- Part 2 — the weight-decay sweep:
+  https://jayrup.me/blog/prime-grokking-2
 
 ---
 
-*This is part 1. The seed sweep is running next; whatever it shows —
-confirmation or refutation — becomes part 2.*
+*This is part 1. Part 2 covers the weight-decay sweep; the range-extension
+experiment and the halting ablation are the natural part 3.*