path: root/design/preregistration.md

# Pre-registration: Prime-Grokking — Experiment 1 (seed 0)

**Status:** PRE-REGISTERED before any training runs.
**Date:** 2026-08-14
**Spec:** `prime-grokking/main.md` (research repo, commit 546dc2c)
**Lock:** this file is committed to the research repo before experiment code runs; the commit hash is the lock.

## Experiment summary

- **Task:** map n → next prime, range n ∈ [2, 100], 30% random holdout (seed 0), tokenized as decimal digits + EOS, teacher forcing on outputs.
- **Arms:** (A) weight-tied 2-layer RNN cell, K = 20 tied steps, ACT learned halting (λ = 0.01, warmup 1000 steps); (B) GPT-style transformer baseline, d_model = 128, 2 layers, 4 heads.
- **Comparison control:** fixed d_model = 128 for both arms. NO parameter-parity gate — weight sharing is the variable under study (param-matching a tied RNN against a GPT would kill the very property the spec hypothesizes). Param counts are logged per run for the record.
- **Optimizer:** AdamW, lr = 1e-3, weight decay = 1.0 (sweep {0.3, 1.0, 3.0, 10.0} is a later experiment, not tuning on v1). Budget: 200k steps cap, eval every 200, early stop on val exact-match = 1.0 (patience 5).
- **Primary metric:** per-example exact-match accuracy on held-out inputs (token accuracy reported alongside). Per-example correctness on 10 fixed val inputs logged every eval. Mean halt steps logged every eval (RNN arm).
- **Probe:** generalization on [101, 200] is diagnostic-only (range extension is its own experiment).

## Pre-registered outcome → interpretation matrix

### In-range outcomes ([2, 100] held-out)

| Code | Observable | Interpretation (locked) | Next step |
|---|---|---|---|
| **O1** | Sharp transition: train EM ≥ 0.95 sustained ≥ 10 evals, THEN val EM rises 0.2 → 0.9 within ≤ 5 evals | Grokking-like. Tied cell + heavy wd found a structured in-range solution; memorization repelled. NOT yet evidence of the full sieve — may be skip-evens/5s + divisibility heuristics. | Identify structure (per-example log, halting pattern); minimal-conditions ablations (halting=False, wd sweep, seeds) — jayrup's call. |
| **O2** | Train EM → ~1.0, val EM stays low (≤ 0.3 at run end) | Memorization won. Lookup table is the lower-norm solution under these hyperparams — consistent with spec's "memorization is a stronger attractor". | wd sweep, smaller train fraction (40%/50%), longer budget — jayrup's call. |
| **O3** | Gradual val rise to ≥ 0.9, no sharp transition | Smooth heuristic learning — NOT grokking by our operational definition. A "fast" generalizing solution exists that GD finds directly. | Distinguish from O1 by transition sharpness; report both. |
| **O4** | Train EM never ≥ 0.95 within budget | Optimization/setup failure (lr, K, halting collapse, bug). NO scientific interpretation until fixed. | Inspect losses + halt steps; fix; rerun. |

### Architecture comparison (same hyperparams, seed 0)

| Code | Observable | Interpretation (locked) |
|---|---|---|
| **O5** | RNN O1, transformer O2/O3 | Weight-tied recurrence is the enabling mechanism at this scale — evidence for the spec's core hypothesis (weight sharing makes the algorithmic solution cheaper). |
| **O6** | Transformer O1, RNN O2/O3 | Recurrence + halting not necessary; fixed-depth transformer suffices in-range. Weight-tying not the key variable — depth/regularization is. |
| **O7** | Both O1 | Grokking robust to architecture at this scale; the variable is data/regularization, not recurrence. |
| **O8** | Neither | See O2/O4. Next-prime may be fundamentally harder than modular addition as the spec hypothesizes; wd/data sweeps decide. |

Caveat: single seed — all architecture comparisons are seed-0 anecdotes until seeds {1, 2}.

### Halting structure (RNN arm only)

| Code | Observable | Interpretation (locked) |
|---|---|---|
| **H1** | Mean steps collapse to 1 by run end | ACT failed (penalty/init issue); architecture conclusions read with a collapsed gate; fixed-K ablation becomes the informative run. |
| **H2** | Mean steps pinned at K = 20 | Never learned to halt (penalty too weak). Same caveat as H1. |
| **H3** | Mean steps intermediate (2–18), evolves during training, ideally correlates with gap-to-next-prime | Learned computation budget — evidence of structured algorithm. Check per-input steps on the 10 logged val examples. |
| **H4** | Steps fluctuate noisily | Halt signal not used meaningfully. |

### Generalization probe [101, 200] (diagnostic-only)

| Code | Observable | Interpretation (locked) |
|---|---|---|
| **P1** | Errors concentrated on {121, 143, 169, 187} — composites with factors 11, 13 (divisors beyond the {2, 3, 5, 7} sieve of the training range) | Definitive evidence of a learned sieve with the training-range divisor set. Strongest positive result available at this scale. |
| **P2** | Errors scattered uniformly over composites | Memorization or non-transferable heuristics; no evidence of divisibility-based algorithm. |
| **P3** | High probe accuracy beyond {121, 143, 169, 187} | Surprising — implies richer algorithm than the {2,3,5,7} sieve. Treat with suspicion; verify across seeds before claiming anything. |
| **P4** | Probe fails on ALL of [101, 200] incl. easy evens / skip-5s | In-range solution didn't transfer even trivial heuristics — strong memorization evidence. |

### Operational definition of "grokking-like" (locked)

- train exact-match ≥ 0.95 sustained for ≥ 10 consecutive evals (train saturated), AND
- val exact-match transition from ≤ 0.2 to ≥ 0.9 within ≤ 5 consecutive evals (eval_every = 200).
- eval order: evals happen every 200 steps; indices counted in evals, not steps.

## Statistical hygiene

1. Seed 0 for v1; seeds {1, 2} required before ANY claim beyond "seed-0 result".
2. No hyperparameter tuning on the val set. The wd sweep is a separate experiment run only after v1 results, at jayrup's call.
3. Probe interpretation locked above; NOTES.md must compare outcomes against this matrix verbatim (cite codes).
4. Training-range caveat recorded: for n ≤ 100 the sieve only needs divisors {2, 3, 5, 7}; "grokking the algorithm" in-range does not imply the general sieve.

---

## Addendum 1 (2026-08-14, pre-launch — setup amendments only)

Trigger: Gemini 3.6 Flash design review (experiment repo `design/reviews/gemini-design-review.md`).
The interpretation matrix (O/H/P codes) above is NOT amended; only setup details changed. Original lock commit: 00c696d.

1. **Fully-tied cell (was: un-tied GRU decoder).** All recurrence — input read-in, K compute steps, AND output-digit decoding — now runs through the SAME 2-layer cell. The earlier draft's GRU decoder would have masked whether the tied cell solved the task. RNN param count ≈ 36.6k (was 168.7k). Regression test added (no GRU/LSTM/RNN modules).
2. **Recurrent input read-in (was: masked mean-pool).** Digits are read through the tied cell with sinusoidal positional encoding. The mean-pool blurred place value ("10" and "100" share the token multiset {1,0}).
3. **λ schedule: linear ramp 1000→5000 steps (was: hard switch at step 1000).** Avoids a discontinuous loss jump late in training.
4. **Future wd sweep revised to {0.01, 0.1, 0.3, 1.0, 3.0}** (10.0 dropped: at lr=1e-3 with AdamW, λ=10 decays weights ~1%/step). Seed-0 default wd=1.0 unchanged.
5. **ACT verification note:** aggregation is Graves (2016) standard — w_t = p_t·Π_{s<t}(1−p_s); Σw_t + remainder = 1 by construction. A reviewer initially flagged this as non-normalized; verified correct, comment + covered by tests.
6. **Prior art appended** (below). No prior grokking work on next-prime / primality prediction found — the closest literature is grokking on modular arithmetic (different: group structure) and ACT/PonderNet halting work.

## References

- Power et al. (2022), "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets", arXiv:2201.02177
- Nanda et al. (2023), "Progress measures for grokking via mechanistic interpretability", arXiv:2301.05217
- Liu et al. (2022), "Towards Understanding Grokking: An Empirical Study", arXiv:2205.10343
- Varma et al. (2023), "Explaining Grokking Through Circuit Efficiency", arXiv:2309.02390
- Graves (2016), "Adaptive Computation Time for Recurrent Neural Networks", arXiv:1603.08983
- Banino et al. (2021), "PonderNet: Learning to Ponder", arXiv:2107.05407
- Giannou et al. (2023), "Looped Transformers as Programmable Computers", arXiv:2301.13196
- Xu et al. (ICLR 2020), "What Can Neural Networks Reason About?", arXiv:1905.13211
- Xu et al. (2021), "How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks", arXiv:2009.11848

---

## Addendum 2 (2026-08-14, pre-launch — operationalization + correctness fixes)

Trigger: OpenAI Codex code review (experiment repo `design/reviews/codex-review.md`).
The interpretation matrix is unchanged; this addendum (a) locks the code thresholds that the
prose left unquantified, and (b) records setup correctness fixes. Everything below is
pre-launch and pre-run.

### Operationalization of O/H/P codes (thresholds now locked)

- **O1:** train EM ≥ 0.95 for ≥ 10 CONSECUTIVE evals (window start `sat_start`), THEN val EM
  reaches ≥ 0.9 at eval `hi` with `hi ≥ sat_start + 10` (strictly after the window), where the
  last eval with val ≤ 0.2 (`lo`) satisfies `hi - lo ≤ 5`.
- **O2:** train EM ≥ 0.95 sustained AND val EM never reaches 0.9 AND final val EM ≤ 0.3.
- **O3:** train EM ≥ 0.95 sustained AND val EM reaches 0.9 but not via the O1 pattern.
- **O4:** train EM never sustained at ≥ 0.95 for 10 evals (setup/optimization failure).
- **O-PARTIAL** (new — the original matrix left this region unspecified): train saturated,
  val never reaches 0.9, final val EM in (0.3, 0.9). Reported as measurements, interpreted
  cautiously as partial in-range generalization; NOT retrofitted into O1/O2/O3.
- **H1:** mean steps at run end ≤ min_steps + 0.5 (collapse to the floor — see fix 2: with
  min_steps = 2 the floor is 2, so literal "collapse to 1" is impossible by construction).
- **H2:** mean steps ≥ K − 0.5 (never learned to halt).
- **H3:** min_steps + 0.5 < mean < K − 0.5 AND Pearson ρ(gap-to-next-prime, steps) ≥ +0.3
  (positive correlation = larger gap consumes more compute steps).
- **H4:** intermediate but ρ < 0.3 (halt signal noisy/unused).
- **P1:** ≥ 3 of {121, 143, 169, 187} wrong AND ≤ 6 total errors AND all errors within the flagged set.
- **P2:** scattered errors not matching P1/P3/P4.
- **P3:** probe accuracy ≥ 85% (checked after P4).
- **P4:** > 50% of trivial-composite inputs (even or multiple of 5) wrong.

### Correctness fixes (codex review)

1. **Layout invariance (BLOCKER).** Inputs are now LEFT-padded to a fixed global length
   (3 digits for [2,100]) in EVERY context — training batches, eval, greedy decoding; outputs
   right-padded to the global length (4 incl. EOS). Previously batch-max padding made an
   example's representation depend on its batchmates (RNN pad steps transformed the state;
   transformer logit positions misaligned and absolute position embeddings shifted).
   RNN pad steps are now exact no-ops. Regression tests: mixed-batch vs singleton logit
   invariance for both models.
2. **min_steps off-by-one.** Halting probs forced to 0 only for the first `min_steps − 1`
   steps (was: first `min_steps`), so the earliest halt is step `min_steps` — matching
   "execute at least min_steps steps".
3. **integers vocab EOS alias.** Integers-mode vocab is now `next_prime(range_end) + 2`, so
   the value token 101 (= next_prime(100)) is NOT aliased with EOS. Boundary test added.
4. **Rerun protection.** train.py refuses to run if `metrics.csv` exists (append + header
   would corrupt provenance). Fresh `--out_dir` required per run.
5. **Checkpoint-selection honesty.** eval reports BOTH `best.pt` (val-selected — flagged as
   selection-holed) and `last.pt` (unselected); probe + halting analyses use `last.pt`.
6. **Dead config removed:** `halt_eps` (was never read).
7. **`run_meta.json`:** python/torch/numpy versions, device, thread count recorded per run.

---

## Addendum 3 (2026-08-14, pre-launch — P1 diagnostic corrected)

Trigger: ad-hoc verification of `src/eval.py` against ground-truth stub models (promoted to
`tests/test_eval_classification.py`). Pre-launch; P1 operationalization corrected before any
results were seen.

1. **P1 concerns PREDICTIONS, not inputs.** A model that learned only the {2,3,5,7} sieve errs
   on inputs n = 113–120, 139–142, 167–168, 181–186, 199–200 — cases where the first candidate
   with no divisor ≤ 7 is composite. The signature is "predicted next prime" ∈ the flagged set,
   NOT "input n" ∈ the flagged set. Addendum 2's phrasing ("≥3 of {121,143,169,187} wrong") was
   ambiguous and the first implementation checked inputs — corrected.
2. **Set extended to {121, 143, 169, 187, 209}.** The original four are the no-small-factor
   composites ≤ 200; the probe's candidate window is actually [102, 211] (targets of
   n ∈ [101, 200] reach 211), and 209 = 11×19 is likewise mispredicted (n = 199, 200 → pred 209,
   target 211).
3. **P1 thresholds (replace Addendum 2's):** ≥ 3 errors total, ≥ 3 DISTINCT flagged predictions,
   ≥ 80% of error predictions flagged. A pure {2,3,5,7} sieve produces exactly 22 errors, 100%
   flagged (5 distinct values). P2–P4 unchanged.

---

## Addendum 4 (2026-08-14, pre-launch — Phase 2+ sweep plan)

Precedes all sweep phases. Locked before any of these runs. Infrastructure changes recorded:
`evaluate()` now batches greedy decoding (measured 46× eval speedup: 4.72 s → 0.10 s per
eval cycle; semantics preserved — batch-invariance is enforced by regression tests, and a
known-model sieve stub verifies exact-match counting). New runner: `scripts/run_sweep.py`
(2-way parallel, one thread per child, fresh deterministic dirs, idempotent skip,
`summary.csv` aggregation). All sweep runs: CPU, same metrics, early stop, eval cadence
(every 200) and O/H/P operationalizations as Experiment 1.

### E1 — Seed replication (seeds {1, 2} × {rnn, transformer}; 4 runs)
Identical hyperparameters to Experiment 1.
- **"Replicated"** = same O-code and P-code across ≥ 2 of 3 seeds (0, 1, 2); RNN additionally
  same H-code across ≥ 2 of 3.
- Transformer O-PARTIAL + P4 in ≥ 2 seeds → partial in-range generalization + zero transfer
  is seed-stable (a real claim).
- RNN O4/H4 + P4 in ≥ 2 seeds → tied-cell instability + unstructured halting is seed-stable.
- Any disagreement → report the per-seed pattern and claim nothing beyond it (high seed
  variance is itself a finding).

### E2 — Weight-decay sweep (wd ∈ {0.01, 0.1, 0.3, 1.0, 3.0} × both models, seed 0; wd=1.0 is the existing control)
- **O1 at ≥ 1 wd value** → grokking exists for this task; the transition is
  regularization-sensitive; that (model, wd) cell becomes the study object for later phases.
- **No O1 anywhere + val EM rises monotonically with wd** → the memorization attractor
  weakens with regularization, but no algorithmic basin was found at this scale.
- **No O1 anywhere + val EM flat or declining with wd** → memorization strongly dominant;
  heavier wd only hurts.
- **RNN never ≥ transformer at any wd** (same-seed val EM) → weight-tying + ACT fails to
  earn its keep at this scale (architecture-negative result).
- Anything else → report per-cell codes + measurements, no further claim.

### E3 — Learning-rate annealing (both models, seed 0; cosine 1e-3 → 1e-4)
Targets the late-run val-EM decay. Final val EM ABOVE the constant-lr control and no
late-run decay → the rollover is optimizer dynamics. Identical or below → the decay is
solution instability (more fundamental).

### E4 — halting=False ablation (RNN, seed 0)
Fixed-K (K = 20) beats the ACT RNN (same seed, val EM) → ACT actively hurt it (consistent
with H4). Equal or worse → the halt gate was not the problem.

### E5 — Training-fraction sweep {40%, 50%} (both models, seed 0)
- **O1 at a smaller fraction** → grokking needs memorization to be only barely possible
  (the classic grokking recipe).
- **Monotone val-EM gain as the fraction shrinks, no O1** → data pressure helps but does
  not produce the transition.
- **No change** → memorization dominated at every data level tested.

E3–E5 run after the E1/E2 gate, at jayrup's call. Range scaling ([2, 1000]) is its own
addendum when it becomes the active phase.

---

## Addendum 5 (2026-08-16, pre-launch — Phase 3 diagnostics batch)

Trigger: external model feedback on follow-ups; adopted the integer-token and is-prime
decomposition experiments. Locked before any of these runs. Implementation: `task_mode`
(next_prime | is_prime), `train_frac`, `lr_decay` flags — 41 tests green.

### D1 — Is-prime diagnostic (`task_mode=is_prime`, digits input, seed 0; wd 1.0 both models + wd 0.1 rnn/transformer)
Binary classification n → {0,1}; output = digit token "1"/"0" + EOS. EM = classification accuracy.
Probe = classification of [101, 200]. P-code ordering adapted for classification:
P4 → **P1** → P3 → P2. (A pure {2,3,5,7} sieve scores ~96% on the probe because most of
[101, 200] is trivially classifiable — that accuracy IS the sieve signature here, not a
"surprising success". P1 fires on flagged-INPUT concentration: errors on {121, 143, 169, 187}
classified prime. Verified against a ground-truth sieve stub: exactly those 4 errors → P1.)
Locked decomposition readings:
- **is_prime groks (O1) while next_prime never did** → the search/increment loop is the wall,
  not the divisibility test.
- **is_prime reproduces next_prime's codes (no O1 anywhere)** → the divisibility operation
  itself is unlearnable under these dynamics — the strongest negative result available.
- Any other pattern → report codes + measurements, no further claim.

### D2 — Integer-token diagnostic (`vocab_mode=integers`, next_prime, seed 0; wd 1.0 both models + wd 0.1 rnn)
Readings (locked):
- val EM lifts ≥ +10 points over the digits-mode control at the same wd → place-value
  parsing was a real tax on learning.
- within ±10 points → parsing was not the bottleneck; the algorithmic content is the wall.

### E3 implementation note
Cosine schedule inside the step loop: lr_t = lr_min + 0.5·(lr − lr_min)·(1 + cos(π·step/max)),
lr_min = 0.1·lr → 1e-3 → 1e-4. Interpretation per Addendum 4 E3, unchanged.

### E4 wd choice
Fixed-K (halting=False) at wd 0.1 (the RNN saturates train there — E2) and wd 1.0
(control parity). Interpretation per Addendum 4 E4, unchanged.

### E5 note
`train_frac` ∈ {0.4, 0.5} subsamples the TRAIN split only (val stays the locked 30);
deterministic stream (seed+1000); banker's rounding documented. Runs in the next batch;
interpretation per Addendum 4 E5, unchanged.

Batch = 11 runs: e4 (2), e3 (2), ints (3), isp (4). 2-way parallel, one thread per child,
same eval cadence, metrics, and early stop as all previous phases.

---

## Addendum 6 — E6: dataset extension to [2, 1000] (locked 2026-08-17, pre-run)

**Question (Q3c):** does increasing the dataset 10× (range_end 100 → 1000) change the outcome
class of next-prime learning? Power et al. (2201.02177): "smaller datasets require increasing
amounts of optimization for generalization" — the inverse prediction: more data should pull any
grokking transition earlier. E6 is the SAME protocol as phases 1–3 (E1+E2 replication) with
exactly one variable changed: range_end 100 → 1000.

**Protocol (all cells):** digits mode, task next_prime, train_frac 0.7, lr 1e-3 const,
max_train_steps 200_000, eval_every 200, ACT halting on (halt_penalty 0.01), d_model 128.
**Changed vs phases 1–3 (recorded, unavoidable):**
- `--max_steps 32` (K). Max prime gap below 1000 is 20 (887→907), so K=20 is the exact worst
  case with zero margin. K=32 covers every in-range target. Rationale logged pre-run.
- `--device cuda` + AMP fp16 autocast (T1000, ichi). fp16 numerics differ from the CPU fp32
  baseline. The O/P/H classification is precision-robust; if any cell lands within ±2 points of
  a classification boundary, the tiebreak re-run is fp32 on ichi CPU.
- Outputs up to 4 digits: covered by max_out_len 6; global layout is range-derived (bisect +
  margin code, tested to 100k).

**Cells (16):**
- wd sweep {0.01, 0.1, 0.3, 1.0, 3.0} × {rnn, transformer}, seed 0 (10)
- wd 1.0 replication × seeds {1, 2} × {rnn, transformer} (4)
- batch-size ablation: wd 1.0, seed 0, batch 128 × {rnn, transformer} (2)

**Interpretation matrix — O/P/H codes unchanged (locked Addenda 1–4). Probe range is now
[1001, 2000]. New P-ladder (sieve-rank estimation, replaces the single P1 signature):**
- A model that internalized a k-prime sieve misses exactly the composites whose prime factors
  all exceed p_k; the smallest missed composite in probe range identifies k unambiguously:
  1147→k=10, 1369→k=11, 1681→k=12, 1849→k=13, none→k≥14 (exact trial division to √n).
- In-range discriminator: the 960→961 case (961=31² is the ONLY composite below 1000 with all
  factors ≥ 31) separates k=10 from k=11.
- P5(k) = errors concentrated on the rank-k signature set; P6 = exact (no probe misses).
- P3 threshold for the [1001, 2000] probe: ≥90% (a rank-10 sieve scores ~97% there).
- Locked reading: P5(k=10) would confirm the "first-10-primes heuristic" hypothesis; P6 would
  show full divisibility transfer; P4/P2 keep their existing meanings.
- E6 outcome vs phase 1–3: an O1 anywhere = data pressure unlocked grokking; same codes =
  the task's walls are algorithmic, not data-bound.

**Compute:** ichi (T1000 + i7-8700), tmux batch, run_sweep concurrency 4, GPU jobs only.
Benchmarks pre-run: transformer 66 steps/s (AMP), rnn 31 steps/s → batch est. 2.5–4 h.

**E7 (next, after E6 analysis):** long-horizon test — 20× budget (4M steps) on the best E6
cells, per user decision; protocol locked in Addendum 7 post-E6.

---

## Addendum 7 — E7: 20× Long-Horizon Budget Test (locked 2026-08-19, pre-run)

**Question:** Is the failure of next-prime grokking in E6 an optimization horizon / compute-gated artifact ($H_\text{slow}$), or an algorithmic alignment / task-structure barrier?

**Protocol (4 cells, seed 0):**
- Candidates: `wd` $\in \{0.1, 0.3\} \times \{\text{rnn}, \text{transformer}\}$ (the $O\text{-PARTIAL}$ cells from E6).
- Budget: $4\,000\,000$ steps ($20\times$ E6's 200k steps).
- Schedule: Adaptive evaluation (530 evals total).
- Range: $[2, 1000]$, `train_frac=0.7`, digits mode, $K=32$.
- Probe: $[1001, 2000]$ (1000 out-of-range items).

**Interpretation matrix (locked):**
1. **O1 anywhere:** GROKKING CONFIRMED on next-prime: transition exists, was budget-limited at 200k. Record transition step.
2. **O-PARTIAL with `val_em_best > parent best` AND sustained $\ge 50\text{k}$ steps after the parent's entire budget:** Partial progress toward the algorithmic basin; report as slow-basin dynamics, NOT grokking.
3. **Same codes as E6 parents (O-PARTIAL/O4, P4/P2, no P5(k$\ge$10)/P6):** $H_\text{slow}$ REJECTED for these cells at 4M steps: the null is now budget-robust to $20\times$. Surviving hypotheses become task-structure-gated (algorithmic alignment), not compute-gated.

---

## Addendum 8 — E8: Token-Space Recurrence & Decomposition (Arms A, B, C, D) (locked 2026-08-29, pre-run)

**Question (Q3d / Q1):** Does expanding computation into token space—via explicit algorithmic decomposition (scratchpad), unsupervised compute positions (filler/pause tokens), or random sequence expansion—enable a fixed-depth Transformer to discover the next-prime algorithm ($O1$) where $4\text{M}$ steps of fixed-depth training ($E7$) failed?

**Background & Theoretical Predictions:**
1. **Algorithmic Decomposition (Nye et al. 2021):** Supervised intermediate computation provides dense gradient pathways through combinatorial search spaces.
2. **Parallel Subcomputation in Filler Tokens (Pfau et al. 2024):** Filler tokens ($\langle\text{pause}\rangle$) only aid computation when underlying subproblems are parallelizable. Next-prime requires trial division across primes $\{2, 3, 5, \dots, \sqrt{c}\}$, which are independent modular tests that can be evaluated in parallel across attention heads at intermediate positions.
3. **Sequence Length / Depth Invariance:** Testing whether random non-digit tokens (Arm D) provide equivalent routing capacity without static token embeddings.

**Protocol:**
- **Model:** GPT-style transformer (2 layers, 4 heads, `d_model=128`, `d_ff=512`), `wd=0.1`, `lr=1e-3` (const), `batch_size=32`, `max_train_steps=200_000`, `eval_every=200`.
- **Dataset:** Range $[2, 1000]$, `train_frac=0.7`, digits mode, seed 0.
- **Probe:** Generalization evaluated on out-of-range $[1001, 2000]$.
- **Hardware/Precision:** CUDA + AMP fp16 on ichi / voidlaptop (CPU fallback).

**The Four Experimental Arms:**
1. **Arm A (Direct Baseline / Control):** Standard mapping $n \to \text{next\_prime}(n) + \text{EOS}$.
2. **Arm B (Structured Scratchpad):** Explicit trial division trace:
   $n \to \text{candidate and divisor checks } (c = n+1, \dots; d_2:0/1, d_3:0/1, \dots) \to \text{SEP} \to \text{next\_prime}(n) + \text{EOS}$.
3. **Arm C (Fixed Pause / Filler Tokens):** $16$ identical pause tokens:
   $n \to \underbrace{\langle\text{pause}\rangle \dots \langle\text{pause}\rangle}_{16} \to \text{SEP} \to \text{next\_prime}(n) + \text{EOS}$.
4. **Arm D (Random Disjoint Tokens):** $16$ uniform random symbols from disjoint alphabet $[a\text{-}p]$:
   $n \to \underbrace{r_1 r_2 \dots r_{16}}_{16} \to \text{SEP} \to \text{next\_prime}(n) + \text{EOS}$.

**Loss Masking & Evaluation Rules:**
- Loss computed over target answer tokens for all arms (and trace tokens for Arm B during teacher forcing). Filler/random tokens in Arms C/D are masked from cross-entropy loss.
- In-range validation exact-match (EM) and out-of-range probe accuracy are scored strictly on the final answer tokens (post-SEP / post-scratch).

**Interpretation Matrix (Locked):**

| Code | Observable Condition | Interpretation (Locked) | Next Step |
|---|---|---|---|
| **S1** | Arm B shows $O1$ (sharp val EM $\ge 0.9$), while A, C, D remain $O\text{-PARTIAL}$ or $O4$ | **Decomposition-Gated:** Algorithmic grokking requires explicit trace supervision; unsupervised compute positions are insufficient to discover divisibility. | Sieve-rank probe on scratch outputs; inspect generated traces for correct divisibility. |
| **S2** | Arm C shows $O1$ or significant probe generalization ($P5(k)$ / $P6$ on $[1001, 2000]$) | **Pfau Hypothesis Confirmed:** Unsupervised filler tokens provide enough hidden attention routing to compute parallel divisibility tests without trace supervision. Major positive result. | Sieve-rank estimation; test filler length ablation ($M \in \{4, 8, 16, 32\}$). |
| **S3** | Arm D $\approx$ Arm C $\gg$ Arm A | **Depth/Position Invariance:** Sequence expansion enables attention routing independent of token identity. | Study attention maps across intermediate layers. |
| **S4** | Arm B in-range $\text{val EM} \approx 1.0$, but out-of-range probe $[1001, 2000]$ is $P4$ ($<10\%$) | **Bounded Trace Interpolation:** Scratchpad supervision fits training domain heuristics but does not induce generalizable out-of-distribution reasoning. | Test `is_prime` scratchpad to isolate divisibility vs candidate search. |
| **S5** | All arms A, B, C, D remain $O\text{-PARTIAL}$ / $O4$ ($P4$ out-of-range) | **Strongest Algorithmic Barrier:** Neither internal recurrence (RNN) nor externalized token recurrence (transformer scratch/filler) enables SGD to escape the memorization basin on next-prime. | Transition directly to Phase 2 (`is_prime` 4-arm battery). |

---

*Publication note (2026-08-17): this file is mirrored verbatim from the private research repo (prime-grokking/preregistration.md) at the moment the experiment repo went public; hashes cited above are research-repo commits. The experiment repo commit history preserves each mirror step.*