path: root/NOTES-PHASE3.md

# Phase 3 diagnostics — analysis notes (E4, E3, D1, D2)

Sources: `runs/phase3/summary.csv`, per-job `results.json`, `runs/phase3/logs/*.log`,
and the locked interpretation clauses in `design/preregistration.md` (Addendum 4 for E3/E4,
Addendum 5 for D1/D2). Structure per experiment: **LOCKED** clause, then **Commentary**
(measured result), **Caveats**, and **Implications for the [2,1000] phase (E6)**.

Constants used throughout: val EM is exact-match; "best" is the val-selected checkpoint
(selection-holed), "last" is the unselected final checkpoint. Probe for next_prime is
exact-match on [101, 200]; probe for is_prime (D1) is classification accuracy on [101, 200].

---

## E4 — halting=False ablation (RNN, seed 0)

**LOCKED** (Addendum 4): "Fixed-K (K = 20) beats the ACT RNN (same seed, val EM) → ACT
actively hurt it (consistent with H4). Equal or worse → the halt gate was not the problem."
Addendum 5 "E4 wd choice": Fixed-K at wd 0.1 (the RNN saturates train there) and wd 1.0
(control parity).

**Commentary.** Fixed-K (halting=False, K=20) vs the ACT RNN, same seed:

| wd | run | val EM best | val EM last | O | P | H | probe (101–200) | halt mean |
|----|-----|-------------|-------------|---|---|---|-----------------|-----------|
| 0.1 | Fixed-K | **0.767** | 0.367 | O-PARTIAL | P4 | H2 | 0 / 100 | 20.0 |
| 0.1 | ACT (E2 ctrl) | 0.700 | **0.567** | O-PARTIAL | P4 | H1 | 0 / 100 | 2.04 |
| 1.0 | Fixed-K | **0.433** | 0.167 | O4 | P4 | H2 | 0 / 100 | 20.0 |
| 1.0 | ACT (ctrl) | 0.367 | 0.167 | O4 | P4 | H4 | 1 / 100 | 3.12 |

On the **best** checkpoint Fixed-K beats ACT at both wd (+6.7 points at 0.1, +6.6 at 1.0),
which per the locked clause reads as "ACT actively hurt it". On the **last** checkpoint Fixed-K
does *not* beat ACT (it is −20.0 points at wd 0.1 and tied at wd 1.0), which reads "equal or
worse → the halt gate was not the problem". The clause says only "val EM" without specifying
best vs last, and the two metrics disagree, so the reading is metric-dependent. Either way the
**late-run rollover is not cured**: Fixed-K still decays 0.767→0.367 (wd 0.1) and 0.433→0.167
(wd 1.0), and the wd 1.0 cell is still O4 (train never sustains saturation) even with no gate.

**Caveats.** H2 (pinned at 20) is by construction for halting=False — it is not a finding.
The ACT comparison at wd 0.1 uses the E2 sweep run (`runs/e2-wd/wd0.1`) as the control, which
is the correct same-seed ACT RNN. Only two wd cells were run (per Addendum 5), both seed 0.

**Implications for E6.** ACT is not load-bearing: it is at most a minor tax on the *best*
checkpoint and was never the cause of the rollover or the O4 instability. The [2,1000] phase
can run the RNN with fixed-K (simpler, no worse on best) without losing anything, but must not
expect halting removal to stabilize training — the instability needs to be attacked via scale
(more data) or schedule, not via the gate.

---

## E3 — learning-rate annealing (both models, seed 0; cosine 1e-3 → 1e-4)

**LOCKED** (Addendum 4): "Targets the late-run val-EM decay. Final val EM ABOVE the
constant-lr control and no late-run decay → the rollover is optimizer dynamics. Identical or
below → the decay is solution instability (more fundamental)."

**Commentary.** Cosine-annealed runs (wd 1.0, control parity) vs the constant-lr control:

| model | run | val EM best | val EM last | O | P | H | probe (101–200) |
|-------|-----|-------------|-------------|---|---|---|-----------------|
| rnn | cosine | 0.367 | 0.167 | O4 | P4 | H4 | 0 / 100 |
| rnn | const-lr | 0.367 | 0.167 | O4 | P4 | H4 | 1 / 100 |
| transformer | cosine | 0.900 | 0.567 | O3 | P4 | — | 0 / 100 |
| transformer | const-lr | 0.867 | 0.700 | O-PARTIAL | P4 | — | 0 / 100 |

RNN: annealing changed nothing — best 0.367 = 0.367, last 0.167 = 0.167, still O4. Final val
EM is **identical** to the control → the decay is solution instability. Transformer: annealing
lifted the *best* checkpoint (0.900 vs 0.867, a new peak, now O3) but the *final* val EM fell
**below** the control (0.567 vs 0.700) — the rollover actually *deepened* (a 33.3-point drop
from peak vs the control's 16.7-point drop). Final val EM below the control → solution
instability. Neither model cleared the "above the control and no decay" bar, and no O1 appeared.

**Caveats.** The E3 clause keys off *final* val EM; the transformer's improved best is the
selection-holed number and does not rescue the reading. The e3-cos rnn halt_mean was 8.74 (H4,
noisy), vs ~3 for the control — the annealed RNN's gate stayed noisy, but H is not part of the
E3 clause. wd for the annealed runs is 1.0 (control parity); the prereg does not restate it in
the E3 clause itself.

**Implications for E6.** The rollover is not a learning-rate artifact — it is a property of the
solution (the memorization attractor re-asserting itself), so annealing alone will not stabilize
the [2,1000] run. E6's actual lever is the bigger range itself (≈700 examples makes the lookup
table stop fitting, which is the mechanism that could put the algorithmic basin within reach).
Keep a modest anneal if convenient, but do not expect it to buy stability.

---

## D1 — is-prime diagnostic (`task_mode=is_prime`, digits, seed 0; wd 1.0 both + wd 0.1 both)

**LOCKED** (Addendum 5): "is_prime groks (O1) while next_prime never did → the search/increment
loop is the wall, not the divisibility test. / is_prime reproduces next_prime's codes (no O1
anywhere) → the divisibility operation itself is unlearnable under these dynamics — the strongest
negative result available. / Any other pattern → report codes + measurements, no further claim."
Probe = classification of [101, 200]; P-ordering adapted P4 → P1 → P3 → P2, and P1 = errors on
{121,143,169,187} classified prime (the {2,3,5,7}-sieve signature, ≈96% probe acc).

**Commentary.** Measured cells:

| wd | model | val EM best | val EM last | O | P | H | probe acc (cls) |
|----|-------|-------------|-------------|---|---|---|-----------------|
| 1.0 | rnn | 0.900 | 0.800 | O3 | P2 | H1 | 0.60 |
| 1.0 | transformer | 0.900 | 0.867 | O3 | P2 | — | 0.80 |
| 0.1 | rnn | 0.867 | 0.800 | O-PARTIAL | P2 | H1 | 0.79 |
| 0.1 | transformer | 0.900 | 0.833 | O3 | P2 | — | 0.75 |

**No O1 anywhere.** The codes are O3 / O-PARTIAL with P2 — which is *not* "reproduces
next_prime's codes" (next_prime was O4 / O-PARTIAL with P4). So strictly this lands in the
**"any other pattern"** branch: report codes + measurements, no further claim. The informative
facts, reported without a locked claim: (i) is_prime is *much* easier in-range than next_prime —
val EM best 0.867–0.900 everywhere vs next_prime's 0.367 (rnn) / 0.867 (transformer) at wd 1.0 —
so the increment/search loop was a genuine in-range tax; (ii) out-of-range classification is
60–80%, all **below the ~96% trivial-sieve floor**, and the errors are P2 (scattered — primes
misclassified as composite, no concentration on {121,143,169,187}) — so there is **no evidence
the divisibility operation was learned**, even though the in-range task got much easier.

**Caveats.** H1 (halting collapsed to floor, mean 2.00–2.26 steps) is benign here: the is_prime
output is a single digit + EOS, so 2 steps is the correct behaviour, not a pathology. The probe
is classification, so the 60–80% is inflated by trivially-classifiable inputs; the sieve floor of
~96% is the right ceiling to compare against, and we are well below it. Because the branch
resolved to "any other pattern", we are not entitled to the "strongest negative result" phrasing
(which required reproducing next_prime's codes).

**Implications for E6.** The search/increment loop, not the divisibility test, carried most of
the in-range difficulty — but divisibility still did not emerge as an algorithm out-of-range.
E6 should therefore keep the full next_prime task (the range extension is precisely the pressure
that makes the search loop worth solving), and treat is_prime only as a lighter diagnostic, not
as a path to a grokking result on its own.

---

## D2 — integer-token diagnostic (`vocab_mode=integers`, next_prime, seed 0; wd 1.0 both + wd 0.1 rnn)

**LOCKED** (Addendum 5): "val EM lifts ≥ +10 points over the digits-mode control at the same wd →
place-value parsing was a real tax on learning. / within ±10 points → parsing was not the
bottleneck; the algorithmic content is the wall."

**Commentary.** First evaluation crashed (out-of-vocab probe inputs: atomic tokens ≥ 104 exceed
the vocab — out-of-range probing is unrepresentable by construction for integers mode). Fixed:
probe is now N/A for integers mode (regression-tested); D2 is scored on in-range val EM, the
only metric its locked clause uses.

| job | model | wd | val EM best | val EM last | O | P | H | digits control (best/last, same wd) |
|-----|-------|----|-------------|-------------|---|---|---|--------------------------------------|
| int | rnn | 1.0 | 0.133 | 0.000 | O2 | N/A | H4 | 0.367 / 0.167 |
| int | transformer | 1.0 | 0.100 | 0.067 | O2 | N/A | — | 0.867 / 0.700 |
| int01 | rnn | 0.1 | 0.067 | 0.000 | O2 | N/A | H1 | 0.700 / 0.567 |

All three cells are **20–77 points BELOW** the digits-mode control. Neither locked branch
fires literally (the "within ±10 points" branch was written expecting a flat outcome; the
measured outcome is a collapse). The measured fact, reported without a locked claim:
**atomic integer tokens destroy the task** — every run is O2 (train memorizes, val ≈ 0),
even the transformer that reaches 86.7% in digits mode.

**Interpretive commentary (post-hoc, separated).** The digit representation was not a tax —
it was the scaffolding. Compositionality is what let any generalization exist at all: in
digits mode the model shares structure across "42", "43", "4", "3" (tens digit, units digit,
position); in integers mode every number is a unique, once-seen token with nothing shared,
so there is nothing to generalize — pure memorization, and even that barely holds under wd.
This inverts the reviewer's hypothesis: removing the parsing overhead removed the substrate
the in-range heuristic was built on.

**Caveats.** Seed 0, three cells only. "Probe N/A" is a representational fact of the encoding,
not a measurement failure. The integer-token task also changed the output distribution shape
(2-token outputs) — the collapse is attributable to the input representation change, but that
change bundles several things (no positional structure, unique tokens, shorter sequences).

**Implications for E6.** Digits mode is not just the default — it is load-bearing. The [2,1000]
phase should use digits (inputs are 1–4 digits there anyway) and treat integer tokens as a
known-broken encoding, not a diagnostic toggle worth re-running.