fix D2: integers-mode probe is N/A by construction (OOV probe inputs), functional regression test; D2 scored (int tokens collapse task: O2, 0-13% vs digits 17-87%); NOTES + report Phase 4 updated; 42 tests

3 files changed, 87 insertions, 26 deletions

diff --git a/NOTES-PHASE3.md b/NOTES-PHASE3.md
index b7b1631..5eea353 100644
--- a/NOTES-PHASE3.md
+++ b/NOTES-PHASE3.md
@@ -132,28 +132,36 @@ as a path to a grokking result on its own.
 place-value parsing was a real tax on learning. / within ±10 points → parsing was not the
 bottleneck; the algorithmic content is the wall."
 
-**Commentary.** **UNRESOLVED — evaluation crashed on all three integer-token runs**, so the
-locked ±10-point val-EM comparison cannot be scored. The three runs (`int` rnn, `int`
-transformer, `int01` rnn) all completed 200k training steps with `train_em 1.0`, then died in
-the evaluator with `IndexError: index out of range in self` inside `torch.nn.functional.embedding`
-(`rnn._encode` / `transformer.forward` → `self.embed`), reached from `probe_report` → `greedy_decode`.
-The integer-vocab token IDs exceed the embedding's vocab size of 103 (eos=102, pad=103). No
-`results.json` was written, so there is no clean best/last EM and no probe.
-
-Training-time val EM (from the logs) is the only signal, and it is **far below** the digits
-control rather than above it: best_val_em was 0.133 (`int` rnn), 0.100 (`int` transformer),
-0.067 (`int01` rnn), with val EM oscillating ~0.000–0.067 throughout. The digits-mode controls at
-the same wd scored 0.367 (rnn 1.0), 0.867 (transformer 1.0), 0.700 (rnn 0.1). So the integer
-tokens are ~23–80 points *below* the digits control — the opposite of the "parsing tax" lift.
-If this provisional training-time reading holds after the fix, D2 points at **parsing was not the
-bottleneck** (arguably integer tokens even hurt, e.g. by collapsing the halting gate to the floor,
-halt mean 2.00 in `int01`), but this is a bug report, not a scientific negative.
-
-**Caveats.** The crash is a real evaluator bug (integer vocab not wired through the probe/eval
-embedding), not a model failure; the training-time val_em numbers above are not the locked metric.
-Until the vocab bug is fixed and the runs re-evaluated, D2 contributes no clean measurement.
-
-**Implications for E6.** Fix the integer-vocab eval bug before any [2,1000] run that might want
-`vocab_mode=integers`; note that at range [2,1000] the inputs are 1–4 digits, so digits mode (with
-a widened embedding/position budget) is the natural default anyway. Do not carry the parsing-tax
-hypothesis into E6 as if confirmed — the provisional signal points the other way.
+**Commentary.** First evaluation crashed (out-of-vocab probe inputs: atomic tokens ≥ 104 exceed
+the vocab — out-of-range probing is unrepresentable by construction for integers mode). Fixed:
+probe is now N/A for integers mode (regression-tested); D2 is scored on in-range val EM, the
+only metric its locked clause uses.
+
+| job | model | wd | val EM best | val EM last | O | P | H | digits control (best/last, same wd) |
+|-----|-------|----|-------------|-------------|---|---|---|--------------------------------------|
+| int | rnn | 1.0 | 0.133 | 0.000 | O2 | N/A | H4 | 0.367 / 0.167 |
+| int | transformer | 1.0 | 0.100 | 0.067 | O2 | N/A | — | 0.867 / 0.700 |
+| int01 | rnn | 0.1 | 0.067 | 0.000 | O2 | N/A | H1 | 0.700 / 0.567 |
+
+All three cells are **20–77 points BELOW** the digits-mode control. Neither locked branch
+fires literally (the "within ±10 points" branch was written expecting a flat outcome; the
+measured outcome is a collapse). The measured fact, reported without a locked claim:
+**atomic integer tokens destroy the task** — every run is O2 (train memorizes, val ≈ 0),
+even the transformer that reaches 86.7% in digits mode.
+
+**Interpretive commentary (post-hoc, separated).** The digit representation was not a tax —
+it was the scaffolding. Compositionality is what let any generalization exist at all: in
+digits mode the model shares structure across "42", "43", "4", "3" (tens digit, units digit,
+position); in integers mode every number is a unique, once-seen token with nothing shared,
+so there is nothing to generalize — pure memorization, and even that barely holds under wd.
+This inverts the reviewer's hypothesis: removing the parsing overhead removed the substrate
+the in-range heuristic was built on.
+
+**Caveats.** Seed 0, three cells only. "Probe N/A" is a representational fact of the encoding,
+not a measurement failure. The integer-token task also changed the output distribution shape
+(2-token outputs) — the collapse is attributable to the input representation change, but that
+change bundles several things (no positional structure, unique tokens, shorter sequences).
+
+**Implications for E6.** Digits mode is not just the default — it is load-bearing. The [2,1000]
+phase should use digits (inputs are 1–4 digits there anyway) and treat integer tokens as a
+known-broken encoding, not a diagnostic toggle worth re-running.
diff --git a/src/eval.py b/src/eval.py
index c964d38..e93f95d 100644
--- a/src/eval.py
+++ b/src/eval.py
@@ -184,7 +184,14 @@ def main() -> None:
                      else "final checkpoint, no val selection"),
         }
         if ckpt == "last.pt":
-            entry["probe"] = probe_report(model, cfg)
+            if cfg.vocab_mode == "integers":
+                entry["probe"] = {
+                    "code": "N/A",
+                    "note": "atomic integer tokens make out-of-range inputs out-of-vocabulary by construction; "
+                            "D2 is scored on in-range val EM only (Addendum 5)",
+                }
+            else:
+                entry["probe"] = probe_report(model, cfg)
             if cfg.model == "rnn":
                 entry["halting"] = halting_report(model, cfg)
             entry["per_example"] = [{"n": "".join(map(str, x)), "target": t, "pred": p, "ok": ok}
diff --git a/tests/test_phase3_flags.py b/tests/test_phase3_flags.py
index 4ba8be1..063822e 100644
--- a/tests/test_phase3_flags.py
+++ b/tests/test_phase3_flags.py
@@ -57,6 +57,52 @@ def test_train_frac_subsamples_train_only():
     assert set(tr) <= set(tr_full)
 
 
+def test_integers_mode_eval_does_not_crash_probe(tmp_path):
+    """Regression: integers-mode eval crashed in the probe (out-of-vocab inputs >= 104).
+    Train a tiny integers model, then run the REAL eval main against it — must not crash,
+    and the probe must be N/A (Addendum 5: D2 is scored on in-range val EM only)."""
+    import json
+    import os
+    import subprocess
+    import sys
+    import torch
+    from src.data import build_examples, get_splits, make_batch
+    from src.model_api import build_model
+
+    cfg = Config(vocab_mode="integers")
+    m = build_model(cfg)
+    train_in, val_in = get_splits(cfg)
+    train_ex = build_examples(train_in, cfg)
+    opt = torch.optim.AdamW(m.parameters(), lr=1e-3)
+    for _ in range(10):
+        ex = train_ex[:16]
+        b = make_batch(ex, cfg)
+        out = m(b["x"], b["y_in"])
+        loss = torch.nn.functional.cross_entropy(
+            out["logits"].reshape(-1, cfg.vocab), b["y"].clamp(max=cfg.vocab - 1).reshape(-1), reduction="none")
+        mask = b["y_mask"].float().reshape(-1)
+        loss = (loss * mask).sum() / mask.sum()
+        opt.zero_grad(); loss.backward(); opt.step()
+
+    od = tmp_path / "rnn" / "seed0"
+    os.makedirs(od, exist_ok=True)
+    cfg.save(str(od / "config.json"))
+    torch.save(m.state_dict(), str(od / "last.pt"))
+    torch.save(m.state_dict(), str(od / "best.pt"))
+    import csv
+    with open(str(od / "metrics.csv"), "w", newline="") as fh:
+        w = csv.writer(fh)
+        w.writerow(["step", "train_loss", "train_token_acc", "train_em", "val_token_acc",
+                    "val_em", "mean_halt_steps", "log_examples", "param_count"])
+        w.writerow([200, 0.1, 1.0, 1.0, 1.0, 0.5, 3.0, "", 1000])
+
+    r = subprocess.run([sys.executable, "-m", "src.eval", "rnn", "0", "--runs-dir", str(tmp_path)],
+                       capture_output=True, text=True, cwd=os.path.join(os.path.dirname(__file__), ".."))
+    assert r.returncode == 0, r.stderr[-600:]
+    res = json.load(open(str(od / "results.json")))
+    assert res["final"]["probe"]["code"] == "N/A"
+
+
 def test_new_flags_parse():
     a = parse_args(["rnn", "0", "--task_mode", "is_prime", "--train_frac", "0.4", "--lr_decay", "True"])
     assert a.task_mode == "is_prime" and a.train_frac == 0.4 and a.lr_decay is True