From 9c50f31c66e788ff08eeac83f84e90e9bc1a921e Mon Sep 17 00:00:00 2001 From: Void Agent Date: Fri, 14 Aug 2026 13:14:09 +0100 Subject: rnn: fully-tied cell (read-in+compute+decode), lambda ramp, ACT regression test; design: prior art + review --- design/experiment-spec.md | 17 +++++++++++++++++ design/preregistration.md | 26 ++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) (limited to 'design') diff --git a/design/experiment-spec.md b/design/experiment-spec.md index c814057..77d1606 100644 --- a/design/experiment-spec.md +++ b/design/experiment-spec.md @@ -125,3 +125,20 @@ cross the gap from pattern matching to computation. --- > Copied from the research repo `prime-grokking/main.md` @ 546dc2c (provenance: `~/Projects/research`, remote ssh://meru/~/projects/research.git). + + +--- + +## Prior art (appended 2026-08-14, Gemini 3.6 Flash design review) + +No prior grokking work on next-prime / primality found. Canonical refs: + +- Power et al. 2022 (arXiv:2201.02177) — grokking, modular arithmetic +- Nanda et al. 2023 (arXiv:2301.05217) — grokking progress measures +- Liu et al. 2022 (arXiv:2205.10343) — empirical grokking study +- Varma et al. 2023 (arXiv:2309.02390) — grokking via circuit efficiency +- Graves 2016 (arXiv:1603.08983) — ACT +- Banino et al. 2021 (arXiv:2107.05407) — PonderNet +- Giannou et al. 2023 (arXiv:2301.13196) — looped transformers +- Xu et al. ICLR 2020 (arXiv:1905.13211) — algorithmic alignment +- Xu et al. 2021 (arXiv:2009.11848) — extrapolation / GNNs diff --git a/design/preregistration.md b/design/preregistration.md index d879912..f800461 100644 --- a/design/preregistration.md +++ b/design/preregistration.md @@ -66,3 +66,29 @@ Caveat: single seed — all architecture comparisons are seed-0 anecdotes until 2. No hyperparameter tuning on the val set. The wd sweep is a separate experiment run only after v1 results, at jayrup's call. 3. Probe interpretation locked above; NOTES.md must compare outcomes against this matrix verbatim (cite codes). 4. Training-range caveat recorded: for n ≤ 100 the sieve only needs divisors {2, 3, 5, 7}; "grokking the algorithm" in-range does not imply the general sieve. + +--- + +## Addendum 1 (2026-08-14, pre-launch — setup amendments only) + +Trigger: Gemini 3.6 Flash design review (experiment repo `design/reviews/gemini-design-review.md`). +The interpretation matrix (O/H/P codes) above is NOT amended; only setup details changed. Original lock commit: 00c696d. + +1. **Fully-tied cell (was: un-tied GRU decoder).** All recurrence — input read-in, K compute steps, AND output-digit decoding — now runs through the SAME 2-layer cell. The earlier draft's GRU decoder would have masked whether the tied cell solved the task. RNN param count ≈ 36.6k (was 168.7k). Regression test added (no GRU/LSTM/RNN modules). +2. **Recurrent input read-in (was: masked mean-pool).** Digits are read through the tied cell with sinusoidal positional encoding. The mean-pool blurred place value ("10" and "100" share the token multiset {1,0}). +3. **λ schedule: linear ramp 1000→5000 steps (was: hard switch at step 1000).** Avoids a discontinuous loss jump late in training. +4. **Future wd sweep revised to {0.01, 0.1, 0.3, 1.0, 3.0}** (10.0 dropped: at lr=1e-3 with AdamW, λ=10 decays weights ~1%/step). Seed-0 default wd=1.0 unchanged. +5. **ACT verification note:** aggregation is Graves (2016) standard — w_t = p_t·Π_{s