diff options
Diffstat (limited to 'design')
| -rw-r--r-- | design/experiment-spec.md | 17 | ||||
| -rw-r--r-- | design/preregistration.md | 26 |
2 files changed, 43 insertions, 0 deletions
diff --git a/design/experiment-spec.md b/design/experiment-spec.md index c814057..77d1606 100644 --- a/design/experiment-spec.md +++ b/design/experiment-spec.md @@ -125,3 +125,20 @@ cross the gap from pattern matching to computation. --- > Copied from the research repo `prime-grokking/main.md` @ 546dc2c (provenance: `~/Projects/research`, remote ssh://meru/~/projects/research.git). + + +--- + +## Prior art (appended 2026-08-14, Gemini 3.6 Flash design review) + +No prior grokking work on next-prime / primality found. Canonical refs: + +- Power et al. 2022 (arXiv:2201.02177) — grokking, modular arithmetic +- Nanda et al. 2023 (arXiv:2301.05217) — grokking progress measures +- Liu et al. 2022 (arXiv:2205.10343) — empirical grokking study +- Varma et al. 2023 (arXiv:2309.02390) — grokking via circuit efficiency +- Graves 2016 (arXiv:1603.08983) — ACT +- Banino et al. 2021 (arXiv:2107.05407) — PonderNet +- Giannou et al. 2023 (arXiv:2301.13196) — looped transformers +- Xu et al. ICLR 2020 (arXiv:1905.13211) — algorithmic alignment +- Xu et al. 2021 (arXiv:2009.11848) — extrapolation / GNNs diff --git a/design/preregistration.md b/design/preregistration.md index d879912..f800461 100644 --- a/design/preregistration.md +++ b/design/preregistration.md @@ -66,3 +66,29 @@ Caveat: single seed — all architecture comparisons are seed-0 anecdotes until 2. No hyperparameter tuning on the val set. The wd sweep is a separate experiment run only after v1 results, at jayrup's call. 3. Probe interpretation locked above; NOTES.md must compare outcomes against this matrix verbatim (cite codes). 4. Training-range caveat recorded: for n ≤ 100 the sieve only needs divisors {2, 3, 5, 7}; "grokking the algorithm" in-range does not imply the general sieve. + +--- + +## Addendum 1 (2026-08-14, pre-launch — setup amendments only) + +Trigger: Gemini 3.6 Flash design review (experiment repo `design/reviews/gemini-design-review.md`). +The interpretation matrix (O/H/P codes) above is NOT amended; only setup details changed. Original lock commit: 00c696d. + +1. **Fully-tied cell (was: un-tied GRU decoder).** All recurrence — input read-in, K compute steps, AND output-digit decoding — now runs through the SAME 2-layer cell. The earlier draft's GRU decoder would have masked whether the tied cell solved the task. RNN param count ≈ 36.6k (was 168.7k). Regression test added (no GRU/LSTM/RNN modules). +2. **Recurrent input read-in (was: masked mean-pool).** Digits are read through the tied cell with sinusoidal positional encoding. The mean-pool blurred place value ("10" and "100" share the token multiset {1,0}). +3. **λ schedule: linear ramp 1000→5000 steps (was: hard switch at step 1000).** Avoids a discontinuous loss jump late in training. +4. **Future wd sweep revised to {0.01, 0.1, 0.3, 1.0, 3.0}** (10.0 dropped: at lr=1e-3 with AdamW, λ=10 decays weights ~1%/step). Seed-0 default wd=1.0 unchanged. +5. **ACT verification note:** aggregation is Graves (2016) standard — w_t = p_t·Π_{s<t}(1−p_s); Σw_t + remainder = 1 by construction. A reviewer initially flagged this as non-normalized; verified correct, comment + covered by tests. +6. **Prior art appended** (below). No prior grokking work on next-prime / primality prediction found — the closest literature is grokking on modular arithmetic (different: group structure) and ACT/PonderNet halting work. + +## References + +- Power et al. (2022), "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets", arXiv:2201.02177 +- Nanda et al. (2023), "Progress measures for grokking via mechanistic interpretability", arXiv:2301.05217 +- Liu et al. (2022), "Towards Understanding Grokking: An Empirical Study", arXiv:2205.10343 +- Varma et al. (2023), "Explaining Grokking Through Circuit Efficiency", arXiv:2309.02390 +- Graves (2016), "Adaptive Computation Time for Recurrent Neural Networks", arXiv:1603.08983 +- Banino et al. (2021), "PonderNet: Learning to Ponder", arXiv:2107.05407 +- Giannou et al. (2023), "Looped Transformers as Programmable Computers", arXiv:2301.13196 +- Xu et al. (ICLR 2020), "What Can Neural Networks Reason About?", arXiv:1905.13211 +- Xu et al. (2021), "How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks", arXiv:2009.11848 |