Phase 3 diagnostics results + analysis + report phase 4

1 files changed, 159 insertions, 0 deletions

diff --git a/NOTES-PHASE3.md b/NOTES-PHASE3.md
new file mode 100644
index 0000000..b7b1631
--- /dev/null
+++ b/NOTES-PHASE3.md
@@ -0,0 +1,159 @@
+# Phase 3 diagnostics — analysis notes (E4, E3, D1, D2)
+
+Sources: `runs/phase3/summary.csv`, per-job `results.json`, `runs/phase3/logs/*.log`,
+and the locked interpretation clauses in `design/preregistration.md` (Addendum 4 for E3/E4,
+Addendum 5 for D1/D2). Structure per experiment: **LOCKED** clause, then **Commentary**
+(measured result), **Caveats**, and **Implications for the [2,1000] phase (E6)**.
+
+Constants used throughout: val EM is exact-match; "best" is the val-selected checkpoint
+(selection-holed), "last" is the unselected final checkpoint. Probe for next_prime is
+exact-match on [101, 200]; probe for is_prime (D1) is classification accuracy on [101, 200].
+
+---
+
+## E4 — halting=False ablation (RNN, seed 0)
+
+**LOCKED** (Addendum 4): "Fixed-K (K = 20) beats the ACT RNN (same seed, val EM) → ACT
+actively hurt it (consistent with H4). Equal or worse → the halt gate was not the problem."
+Addendum 5 "E4 wd choice": Fixed-K at wd 0.1 (the RNN saturates train there) and wd 1.0
+(control parity).
+
+**Commentary.** Fixed-K (halting=False, K=20) vs the ACT RNN, same seed:
+
+| wd | run | val EM best | val EM last | O | P | H | probe (101–200) | halt mean |
+|----|-----|-------------|-------------|---|---|---|-----------------|-----------|
+| 0.1 | Fixed-K | **0.767** | 0.367 | O-PARTIAL | P4 | H2 | 0 / 100 | 20.0 |
+| 0.1 | ACT (E2 ctrl) | 0.700 | **0.567** | O-PARTIAL | P4 | H1 | 0 / 100 | 2.04 |
+| 1.0 | Fixed-K | **0.433** | 0.167 | O4 | P4 | H2 | 0 / 100 | 20.0 |
+| 1.0 | ACT (ctrl) | 0.367 | 0.167 | O4 | P4 | H4 | 1 / 100 | 3.12 |
+
+On the **best** checkpoint Fixed-K beats ACT at both wd (+6.7 points at 0.1, +6.6 at 1.0),
+which per the locked clause reads as "ACT actively hurt it". On the **last** checkpoint Fixed-K
+does *not* beat ACT (it is −20.0 points at wd 0.1 and tied at wd 1.0), which reads "equal or
+worse → the halt gate was not the problem". The clause says only "val EM" without specifying
+best vs last, and the two metrics disagree, so the reading is metric-dependent. Either way the
+**late-run rollover is not cured**: Fixed-K still decays 0.767→0.367 (wd 0.1) and 0.433→0.167
+(wd 1.0), and the wd 1.0 cell is still O4 (train never sustains saturation) even with no gate.
+
+**Caveats.** H2 (pinned at 20) is by construction for halting=False — it is not a finding.
+The ACT comparison at wd 0.1 uses the E2 sweep run (`runs/e2-wd/wd0.1`) as the control, which
+is the correct same-seed ACT RNN. Only two wd cells were run (per Addendum 5), both seed 0.
+
+**Implications for E6.** ACT is not load-bearing: it is at most a minor tax on the *best*
+checkpoint and was never the cause of the rollover or the O4 instability. The [2,1000] phase
+can run the RNN with fixed-K (simpler, no worse on best) without losing anything, but must not
+expect halting removal to stabilize training — the instability needs to be attacked via scale
+(more data) or schedule, not via the gate.
+
+---
+
+## E3 — learning-rate annealing (both models, seed 0; cosine 1e-3 → 1e-4)
+
+**LOCKED** (Addendum 4): "Targets the late-run val-EM decay. Final val EM ABOVE the
+constant-lr control and no late-run decay → the rollover is optimizer dynamics. Identical or
+below → the decay is solution instability (more fundamental)."
+
+**Commentary.** Cosine-annealed runs (wd 1.0, control parity) vs the constant-lr control:
+
+| model | run | val EM best | val EM last | O | P | H | probe (101–200) |
+|-------|-----|-------------|-------------|---|---|---|-----------------|
+| rnn | cosine | 0.367 | 0.167 | O4 | P4 | H4 | 0 / 100 |
+| rnn | const-lr | 0.367 | 0.167 | O4 | P4 | H4 | 1 / 100 |
+| transformer | cosine | 0.900 | 0.567 | O3 | P4 | — | 0 / 100 |
+| transformer | const-lr | 0.867 | 0.700 | O-PARTIAL | P4 | — | 0 / 100 |
+
+RNN: annealing changed nothing — best 0.367 = 0.367, last 0.167 = 0.167, still O4. Final val
+EM is **identical** to the control → the decay is solution instability. Transformer: annealing
+lifted the *best* checkpoint (0.900 vs 0.867, a new peak, now O3) but the *final* val EM fell
+**below** the control (0.567 vs 0.700) — the rollover actually *deepened* (a 33.3-point drop
+from peak vs the control's 16.7-point drop). Final val EM below the control → solution
+instability. Neither model cleared the "above the control and no decay" bar, and no O1 appeared.
+
+**Caveats.** The E3 clause keys off *final* val EM; the transformer's improved best is the
+selection-holed number and does not rescue the reading. The e3-cos rnn halt_mean was 8.74 (H4,
+noisy), vs ~3 for the control — the annealed RNN's gate stayed noisy, but H is not part of the
+E3 clause. wd for the annealed runs is 1.0 (control parity); the prereg does not restate it in
+the E3 clause itself.
+
+**Implications for E6.** The rollover is not a learning-rate artifact — it is a property of the
+solution (the memorization attractor re-asserting itself), so annealing alone will not stabilize
+the [2,1000] run. E6's actual lever is the bigger range itself (≈700 examples makes the lookup
+table stop fitting, which is the mechanism that could put the algorithmic basin within reach).
+Keep a modest anneal if convenient, but do not expect it to buy stability.
+
+---
+
+## D1 — is-prime diagnostic (`task_mode=is_prime`, digits, seed 0; wd 1.0 both + wd 0.1 both)
+
+**LOCKED** (Addendum 5): "is_prime groks (O1) while next_prime never did → the search/increment
+loop is the wall, not the divisibility test. / is_prime reproduces next_prime's codes (no O1
+anywhere) → the divisibility operation itself is unlearnable under these dynamics — the strongest
+negative result available. / Any other pattern → report codes + measurements, no further claim."
+Probe = classification of [101, 200]; P-ordering adapted P4 → P1 → P3 → P2, and P1 = errors on
+{121,143,169,187} classified prime (the {2,3,5,7}-sieve signature, ≈96% probe acc).
+
+**Commentary.** Measured cells:
+
+| wd | model | val EM best | val EM last | O | P | H | probe acc (cls) |
+|----|-------|-------------|-------------|---|---|---|-----------------|
+| 1.0 | rnn | 0.900 | 0.800 | O3 | P2 | H1 | 0.60 |
+| 1.0 | transformer | 0.900 | 0.867 | O3 | P2 | — | 0.80 |
+| 0.1 | rnn | 0.867 | 0.800 | O-PARTIAL | P2 | H1 | 0.79 |
+| 0.1 | transformer | 0.900 | 0.833 | O3 | P2 | — | 0.75 |
+
+**No O1 anywhere.** The codes are O3 / O-PARTIAL with P2 — which is *not* "reproduces
+next_prime's codes" (next_prime was O4 / O-PARTIAL with P4). So strictly this lands in the
+**"any other pattern"** branch: report codes + measurements, no further claim. The informative
+facts, reported without a locked claim: (i) is_prime is *much* easier in-range than next_prime —
+val EM best 0.867–0.900 everywhere vs next_prime's 0.367 (rnn) / 0.867 (transformer) at wd 1.0 —
+so the increment/search loop was a genuine in-range tax; (ii) out-of-range classification is
+60–80%, all **below the ~96% trivial-sieve floor**, and the errors are P2 (scattered — primes
+misclassified as composite, no concentration on {121,143,169,187}) — so there is **no evidence
+the divisibility operation was learned**, even though the in-range task got much easier.
+
+**Caveats.** H1 (halting collapsed to floor, mean 2.00–2.26 steps) is benign here: the is_prime
+output is a single digit + EOS, so 2 steps is the correct behaviour, not a pathology. The probe
+is classification, so the 60–80% is inflated by trivially-classifiable inputs; the sieve floor of
+~96% is the right ceiling to compare against, and we are well below it. Because the branch
+resolved to "any other pattern", we are not entitled to the "strongest negative result" phrasing
+(which required reproducing next_prime's codes).
+
+**Implications for E6.** The search/increment loop, not the divisibility test, carried most of
+the in-range difficulty — but divisibility still did not emerge as an algorithm out-of-range.
+E6 should therefore keep the full next_prime task (the range extension is precisely the pressure
+that makes the search loop worth solving), and treat is_prime only as a lighter diagnostic, not
+as a path to a grokking result on its own.
+
+---
+
+## D2 — integer-token diagnostic (`vocab_mode=integers`, next_prime, seed 0; wd 1.0 both + wd 0.1 rnn)
+
+**LOCKED** (Addendum 5): "val EM lifts ≥ +10 points over the digits-mode control at the same wd →
+place-value parsing was a real tax on learning. / within ±10 points → parsing was not the
+bottleneck; the algorithmic content is the wall."
+
+**Commentary.** **UNRESOLVED — evaluation crashed on all three integer-token runs**, so the
+locked ±10-point val-EM comparison cannot be scored. The three runs (`int` rnn, `int`
+transformer, `int01` rnn) all completed 200k training steps with `train_em 1.0`, then died in
+the evaluator with `IndexError: index out of range in self` inside `torch.nn.functional.embedding`
+(`rnn._encode` / `transformer.forward` → `self.embed`), reached from `probe_report` → `greedy_decode`.
+The integer-vocab token IDs exceed the embedding's vocab size of 103 (eos=102, pad=103). No
+`results.json` was written, so there is no clean best/last EM and no probe.
+
+Training-time val EM (from the logs) is the only signal, and it is **far below** the digits
+control rather than above it: best_val_em was 0.133 (`int` rnn), 0.100 (`int` transformer),
+0.067 (`int01` rnn), with val EM oscillating ~0.000–0.067 throughout. The digits-mode controls at
+the same wd scored 0.367 (rnn 1.0), 0.867 (transformer 1.0), 0.700 (rnn 0.1). So the integer
+tokens are ~23–80 points *below* the digits control — the opposite of the "parsing tax" lift.
+If this provisional training-time reading holds after the fix, D2 points at **parsing was not the
+bottleneck** (arguably integer tokens even hurt, e.g. by collapsing the halting gate to the floor,
+halt mean 2.00 in `int01`), but this is a bug report, not a scientific negative.
+
+**Caveats.** The crash is a real evaluator bug (integer vocab not wired through the probe/eval
+embedding), not a model failure; the training-time val_em numbers above are not the locked metric.
+Until the vocab bug is fixed and the runs re-evaluated, D2 contributes no clean measurement.
+
+**Implications for E6.** Fix the integer-vocab eval bug before any [2,1000] run that might want
+`vocab_mode=integers`; note that at range [2,1000] the inputs are 1–4 digits, so digits mode (with
+a widened embedding/position budget) is the natural default anyway. Do not carry the parsing-tax
+hypothesis into E6 as if confirmed — the provisional signal points the other way.