From 38d6553048808f6b53488894fdb4c83211590ad4 Mon Sep 17 00:00:00 2001
From: Void Agent <void@jayrup.hermes>
Date: Sat, 15 Aug 2026 00:00:41 +0100
Subject: speedup: batched eval (46x, sieve-stub verified), run_sweep
 orchestrator (2-way parallel, idempotent, summaries), E1 jobs; 36 tests

---
 design/preregistration.md | 52 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

(limited to 'design')

diff --git a/design/preregistration.md b/design/preregistration.md
index 91e5036..89e4fee 100644
--- a/design/preregistration.md
+++ b/design/preregistration.md
@@ -165,3 +165,55 @@ results were seen.
 3. **P1 thresholds (replace Addendum 2's):** ≥ 3 errors total, ≥ 3 DISTINCT flagged predictions,
    ≥ 80% of error predictions flagged. A pure {2,3,5,7} sieve produces exactly 22 errors, 100%
    flagged (5 distinct values). P2–P4 unchanged.
+
+---
+
+## Addendum 4 (2026-08-14, pre-launch — Phase 2+ sweep plan)
+
+Precedes all sweep phases. Locked before any of these runs. Infrastructure changes recorded:
+`evaluate()` now batches greedy decoding (measured 46× eval speedup: 4.72 s → 0.10 s per
+eval cycle; semantics preserved — batch-invariance is enforced by regression tests, and a
+known-model sieve stub verifies exact-match counting). New runner: `scripts/run_sweep.py`
+(2-way parallel, one thread per child, fresh deterministic dirs, idempotent skip,
+`summary.csv` aggregation). All sweep runs: CPU, same metrics, early stop, eval cadence
+(every 200) and O/H/P operationalizations as Experiment 1.
+
+### E1 — Seed replication (seeds {1, 2} × {rnn, transformer}; 4 runs)
+Identical hyperparameters to Experiment 1.
+- **"Replicated"** = same O-code and P-code across ≥ 2 of 3 seeds (0, 1, 2); RNN additionally
+  same H-code across ≥ 2 of 3.
+- Transformer O-PARTIAL + P4 in ≥ 2 seeds → partial in-range generalization + zero transfer
+  is seed-stable (a real claim).
+- RNN O4/H4 + P4 in ≥ 2 seeds → tied-cell instability + unstructured halting is seed-stable.
+- Any disagreement → report the per-seed pattern and claim nothing beyond it (high seed
+  variance is itself a finding).
+
+### E2 — Weight-decay sweep (wd ∈ {0.01, 0.1, 0.3, 1.0, 3.0} × both models, seed 0; wd=1.0 is the existing control)
+- **O1 at ≥ 1 wd value** → grokking exists for this task; the transition is
+  regularization-sensitive; that (model, wd) cell becomes the study object for later phases.
+- **No O1 anywhere + val EM rises monotonically with wd** → the memorization attractor
+  weakens with regularization, but no algorithmic basin was found at this scale.
+- **No O1 anywhere + val EM flat or declining with wd** → memorization strongly dominant;
+  heavier wd only hurts.
+- **RNN never ≥ transformer at any wd** (same-seed val EM) → weight-tying + ACT fails to
+  earn its keep at this scale (architecture-negative result).
+- Anything else → report per-cell codes + measurements, no further claim.
+
+### E3 — Learning-rate annealing (both models, seed 0; cosine 1e-3 → 1e-4)
+Targets the late-run val-EM decay. Final val EM ABOVE the constant-lr control and no
+late-run decay → the rollover is optimizer dynamics. Identical or below → the decay is
+solution instability (more fundamental).
+
+### E4 — halting=False ablation (RNN, seed 0)
+Fixed-K (K = 20) beats the ACT RNN (same seed, val EM) → ACT actively hurt it (consistent
+with H4). Equal or worse → the halt gate was not the problem.
+
+### E5 — Training-fraction sweep {40%, 50%} (both models, seed 0)
+- **O1 at a smaller fraction** → grokking needs memorization to be only barely possible
+  (the classic grokking recipe).
+- **Monotone val-EM gain as the fraction shrinks, no O1** → data pressure helps but does
+  not produce the transition.
+- **No change** → memorization dominated at every data level tested.
+
+E3–E5 run after the E1/E2 gate, at jayrup's call. Range scaling ([2, 1000]) is its own
+addendum when it becomes the active phase.
-- 
cgit v1.2.3