diff options
Diffstat (limited to 'blog/prime-grokking-1.qmd')
| -rw-r--r-- | blog/prime-grokking-1.qmd | 71 |
1 files changed, 56 insertions, 15 deletions
diff --git a/blog/prime-grokking-1.qmd b/blog/prime-grokking-1.qmd index 23c97b1..97b1fa2 100644 --- a/blog/prime-grokking-1.qmd +++ b/blog/prime-grokking-1.qmd @@ -7,6 +7,10 @@ date: "2026-08-14" to see whether either would discover the algorithm instead of memorizing the answers. Neither did. The way they failed is more interesting than the failure.* +*Updated 2026-08-16: seeds 1–2 replication added (section 8). The +weight-decay sweep is a separate post: +[prime-grokking-2](https://jayrup.me/blog/prime-grokking-2).* + --- **The short version.** I asked whether a minimal neural network can learn the @@ -245,9 +249,10 @@ Reading strictly from the locked matrix: emerge on its own; it collapsed toward the floor. That's its own small result: ACT without stronger shaping didn't produce structure here. -**Honesty note.** This is one seed of a 69-example toy experiment. Per the -pre-registration, seeds 1 and 2 are required before any claim beyond -"seed-0 result". The val number for the transformer's best checkpoint (86.7%) +**Honesty note.** This is a 69-example toy experiment. The numbers above +are the seed-0 run; the seed replication (section 8) confirmed them, and +the weight-decay sweep is in [part 2](https://jayrup.me/blog/prime-grokking-2). +The val number for the transformer's best checkpoint (86.7%) is also selection-holed — I used validation for early stopping, so the peak is optimistic; the unselected final checkpoint's 70% is the cleaner number. And one engineering asterisk: the first attempt at these runs was killed by a @@ -266,12 +271,10 @@ task, it's not the one these optimizer settings settle into. ## 6. Where this goes next -In rough priority order: +Status: the seed replication is done (section 8), and the weight-decay +sweep is published as [part 2](https://jayrup.me/blog/prime-grokking-2). +Remaining, in rough priority order: -- **Seeds 1, 2** — the mandatory replication before anything is claimed. -- **Weight-decay sweep {0.01, 0.1, 0.3, 1.0, 3.0}** — the O2 recipe: if - memorization wins, turn the regularization up (and down, for a control) and - watch whether any setting produces the jump. - **Learning-rate annealing** — a targeted experiment against the late-run decay specifically. - **Smaller training fraction (40% / 50%)** — fewer examples, more pressure @@ -285,11 +288,49 @@ In rough priority order: ## 7. Reproduce -The repo is private for now — it will go public once the seed replication -lands and the writeup is final. 34 tests cover data correctness, model +The repo is private for now — it will go public once the writeup series is +final. 34 tests cover data correctness, model shapes, halting, layout invariance, and the evaluator's classification logic against ground-truth stub models. Each arm trains in ~2h on 2 CPU cores. +## 8. Update: the seed replication + +Per the pre-registration, seeds 1 and 2 were required before any claim +beyond "seed-0 result". I re-ran both models with identical +hyperparameters, changing only the seed. The locked criterion: a result is +**replicated** if it shows the same O-code and P-code across ≥ 2 of the 3 +seeds (0, 1, 2), plus the same H-code for the RNN. + +The answer is a clean, boring confirmation — boring in exactly the way a +replication is supposed to be. + +``` + model seed O H P val best val last probe (101–200) + transformer 0 O-PARTIAL — P4 86.7% 70.0% 0 / 100 + transformer 1 O-PARTIAL — P4 70.0% 43.3% 0 / 100 + transformer 2 O-PARTIAL — P4 80.0% 73.3% 0 / 100 + RNN 0 O4 H4 P4 36.7% 16.7% 1 / 100 + RNN 1 O4 H4 P4 36.7% 20.0% 0 / 100 + RNN 2 O4 H4 P4 36.7% 23.3% 0 / 100 +``` + +Both models hit the bar in **all three seeds**, not just the required two. +The transformer's O-PARTIAL + P4 is seed-stable: partial in-range +generalization (val best 70–87%) with **zero** out-of-range transfer in +every run. The RNN's O4 / H4 + P4 is equally stable: the tied cell couldn't +hold a memorized solution (val best stuck at 36.7% across all three seeds), +and its halting gate stayed unstructured near the floor (mean 2.8–3.6 steps +of 20). No disagreement to report — so per the locked matrix these are now +real, seed-stable findings, and the weight-decay sweep could proceed on +seed 0 as planned. That sweep is [part 2](https://jayrup.me/blog/prime-grokking-2). + +Caveats unchanged: three seeds of a 69-example toy task is replication, not +a variance estimate; the "best" column is val-selected and therefore +selection-holed (the "last" column is the unselected checkpoint); the +late-run val decay seen in seed 0 recurs in both models; and the probe +transfer is exactly zero everywhere, so there is nothing downstream to +over-interpret. + --- RESOURCES @@ -297,12 +338,12 @@ RESOURCES - Grokking (machine learning) — Wikipedia overview: https://en.wikipedia.org/wiki/Grokking_(machine_learning) -- Code for this experiment (goes public once the seed replication lands): +- Code for this experiment (will be made public): <repo link — pending> -- Part 2: the seed sweep (when published): - <link — pending> +- Part 2 — the weight-decay sweep: + https://jayrup.me/blog/prime-grokking-2 --- -*This is part 1. The seed sweep is running next; whatever it shows — -confirmation or refutation — becomes part 2.* +*This is part 1. Part 2 covers the weight-decay sweep; the range-extension +experiment and the halting ablation are the natural part 3.* |