diff options
| -rw-r--r-- | NOTES-E1.md | 88 |
1 files changed, 88 insertions, 0 deletions
diff --git a/NOTES-E1.md b/NOTES-E1.md new file mode 100644 index 0000000..dc8c1fa --- /dev/null +++ b/NOTES-E1.md @@ -0,0 +1,88 @@ +# Phase 2 — E1: Seed replication (notes) + +**Status:** locked-criteria analysis only — no code changes, no training runs. All numbers below +are read from `runs/seeds/summary.csv` and each run's `results.json` +(`runs/seeds/rep/<model>/seed<N>/results.json`). Seed 0 numbers are Experiment 1, already +reported in `report/index.html`. + +## Locked criteria (Addendum 4, E1) + +> **E1 — Seed replication (seeds {1, 2} × {rnn, transformer}; 4 runs).** Identical hyperparameters +> to Experiment 1. +> - **"Replicated"** = same O-code and P-code across ≥ 2 of 3 seeds (0, 1, 2); RNN additionally +> same H-code across ≥ 2 of 3. +> - Transformer O-PARTIAL + P4 in ≥ 2 seeds → partial in-range generalization + zero transfer is +> seed-stable (a real claim). +> - RNN O4/H4 + P4 in ≥ 2 seeds → tied-cell instability + unstructured halting is seed-stable. +> - Any disagreement → report the per-seed pattern and claim nothing beyond it. + +## Results + +### Transformer + +| seed | O code | P code | H code | val EM last | val EM best | probe acc | halt mean | +|---|---|---|---|---|---|---|---| +| 0 | O-PARTIAL | P4 | — | 0.700 | 0.867 | 0.00 | — | +| 1 | O-PARTIAL | P4 | — | 0.433 | 0.700 | 0.00 | — | +| 2 | O-PARTIAL | P4 | — | 0.733 | 0.800 | 0.00 | — | + +### RNN + +| seed | O code | P code | H code | val EM last | val EM best | probe acc | halt mean | +|---|---|---|---|---|---|---|---| +| 0 | O4 | P4 | H4 | 0.167 | 0.367 | 0.01 | ~3.1 | +| 1 | O4 | P4 | H4 | 0.200 | 0.367 | 0.00 | 3.584 | +| 2 | O4 | P4 | H4 | 0.233 | 0.367 | 0.00 | 2.839 | + +## Verdict + +**Transformer: replicated (3/3 seeds).** O-PARTIAL + P4 across all three seeds → the locked claim +"partial in-range generalization + zero transfer is seed-stable" is confirmed at 3/3 — strictly +stronger than the ≥ 2-of-3 requirement. + +**RNN: replicated (3/3 seeds).** O4 + H4 + P4 across all three seeds → "tied-cell instability + +unstructured halting is seed-stable" is confirmed at 3/3. + +There is **no disagreement anywhere**: the per-seed code pattern is identical for both models, so +nothing falls back to the "report the pattern and claim nothing" branch. Both seed-0 readings are +now real, seed-stable findings. + +Interpretive commentary (kept separate from the locked findings above): + +- The transformer's **best** EM varies (0.70–0.87) but its **last** EM (0.43–0.73) reproduces the + seed-0 late-run decay; the peak is val-selection-holed and should be read as optimistic. +- The RNN's **val EM best is pinned at 0.367 in all three seeds** — a strikingly consistent + memorization ceiling that the tied cell reaches but never holds (final EM 0.17–0.23). +- **Halt mean sits in a tight 2.8–3.6 band** across all three seeds, all near the floor of 20 + steps — H4 (unstructured) is stable, not a seed-0 fluke. + +## Caveats + +- **Only 3 seeds.** "Seed-stable" here means "consistent across 3 draws," not a proof of low + variance in general. Three is the pre-registered minimum, not a population estimate. +- **Tiny toy task** — 69 training examples, range [2, 100]; nothing here generalizes to larger + networks or ranges. +- **val EM best is val-selected** (early-stopped on validation), so it is selection-holed; the + **last** column (unselected checkpoint) is the cleaner estimate. +- **Late-run decay recurs** in both models — now a replicated phenomenon in its own right, and the + direct target of E3 (annealing). +- **Probe transfer is exactly zero** (0–1/100) in every run. P4 is stable, but the probe is a + coarse instrument: it detects "fails even trivial cases," not graded generalization. + +## E2 implications + +Given these seed-stable results, the weight-decay sweep (wd ∈ {0.01, 0.1, 0.3, 1.0, 3.0}, both +models, seed 0) should test: + +1. **Whether any wd value produces O1.** O-PARTIAL is stable at wd = 1.0; the open question is + whether weaker regularization (easier memorization) or stronger (more pressure toward the + cheaper algorithm) ever flips the transformer into the sharp val jump. Per the E2 matrix, + O1 at ≥ 1 wd value would mean grokking exists for this task. +2. **Whether the RNN ever reaches or beats the transformer** at any wd — the E2 + "RNN never ≥ transformer" clause. With the RNN pinned at 0.367 best-EM across all three seeds, + a wd value that lifts it meaningfully would be the first sign weight-tying + ACT earns its keep. +3. **Whether the late-run decay is wd-sensitive.** The decay is now replicated, so E2 (wd) and E3 + (annealing) are the correct, direct follow-ups rather than exploratory fishing. + +E2 is locked to seed 0 and does not itself require re-replicating seeds 1/2; the seed-stability +claim established by E1 is what licenses proceeding to the sweep. |