1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
|
# Phase 2 — E1: Seed replication (notes)
**Status:** locked-criteria analysis only — no code changes, no training runs. All numbers below
are read from `runs/seeds/summary.csv` and each run's `results.json`
(`runs/seeds/rep/<model>/seed<N>/results.json`). Seed 0 numbers are Experiment 1, already
reported in `report/index.html`.
## Locked criteria (Addendum 4, E1)
> **E1 — Seed replication (seeds {1, 2} × {rnn, transformer}; 4 runs).** Identical hyperparameters
> to Experiment 1.
> - **"Replicated"** = same O-code and P-code across ≥ 2 of 3 seeds (0, 1, 2); RNN additionally
> same H-code across ≥ 2 of 3.
> - Transformer O-PARTIAL + P4 in ≥ 2 seeds → partial in-range generalization + zero transfer is
> seed-stable (a real claim).
> - RNN O4/H4 + P4 in ≥ 2 seeds → tied-cell instability + unstructured halting is seed-stable.
> - Any disagreement → report the per-seed pattern and claim nothing beyond it.
## Results
### Transformer
| seed | O code | P code | H code | val EM last | val EM best | probe acc | halt mean |
|---|---|---|---|---|---|---|---|
| 0 | O-PARTIAL | P4 | — | 0.700 | 0.867 | 0.00 | — |
| 1 | O-PARTIAL | P4 | — | 0.433 | 0.700 | 0.00 | — |
| 2 | O-PARTIAL | P4 | — | 0.733 | 0.800 | 0.00 | — |
### RNN
| seed | O code | P code | H code | val EM last | val EM best | probe acc | halt mean |
|---|---|---|---|---|---|---|---|
| 0 | O4 | P4 | H4 | 0.167 | 0.367 | 0.01 | ~3.1 |
| 1 | O4 | P4 | H4 | 0.200 | 0.367 | 0.00 | 3.584 |
| 2 | O4 | P4 | H4 | 0.233 | 0.367 | 0.00 | 2.839 |
## Verdict
**Transformer: replicated (3/3 seeds).** O-PARTIAL + P4 across all three seeds → the locked claim
"partial in-range generalization + zero transfer is seed-stable" is confirmed at 3/3 — strictly
stronger than the ≥ 2-of-3 requirement.
**RNN: replicated (3/3 seeds).** O4 + H4 + P4 across all three seeds → "tied-cell instability +
unstructured halting is seed-stable" is confirmed at 3/3.
There is **no disagreement anywhere**: the per-seed code pattern is identical for both models, so
nothing falls back to the "report the pattern and claim nothing" branch. Both seed-0 readings are
now real, seed-stable findings.
Interpretive commentary (kept separate from the locked findings above):
- The transformer's **best** EM varies (0.70–0.87) but its **last** EM (0.43–0.73) reproduces the
seed-0 late-run decay; the peak is val-selection-holed and should be read as optimistic.
- The RNN's **val EM best is pinned at 0.367 in all three seeds** — a strikingly consistent
memorization ceiling that the tied cell reaches but never holds (final EM 0.17–0.23).
- **Halt mean sits in a tight 2.8–3.6 band** across all three seeds, all near the floor of 20
steps — H4 (unstructured) is stable, not a seed-0 fluke.
## Caveats
- **Only 3 seeds.** "Seed-stable" here means "consistent across 3 draws," not a proof of low
variance in general. Three is the pre-registered minimum, not a population estimate.
- **Tiny toy task** — 69 training examples, range [2, 100]; nothing here generalizes to larger
networks or ranges.
- **val EM best is val-selected** (early-stopped on validation), so it is selection-holed; the
**last** column (unselected checkpoint) is the cleaner estimate.
- **Late-run decay recurs** in both models — now a replicated phenomenon in its own right, and the
direct target of E3 (annealing).
- **Probe transfer is exactly zero** (0–1/100) in every run. P4 is stable, but the probe is a
coarse instrument: it detects "fails even trivial cases," not graded generalization.
## E2 implications
Given these seed-stable results, the weight-decay sweep (wd ∈ {0.01, 0.1, 0.3, 1.0, 3.0}, both
models, seed 0) should test:
1. **Whether any wd value produces O1.** O-PARTIAL is stable at wd = 1.0; the open question is
whether weaker regularization (easier memorization) or stronger (more pressure toward the
cheaper algorithm) ever flips the transformer into the sharp val jump. Per the E2 matrix,
O1 at ≥ 1 wd value would mean grokking exists for this task.
2. **Whether the RNN ever reaches or beats the transformer** at any wd — the E2
"RNN never ≥ transformer" clause. With the RNN pinned at 0.367 best-EM across all three seeds,
a wd value that lifts it meaningfully would be the first sign weight-tying + ACT earns its keep.
3. **Whether the late-run decay is wd-sensitive.** The decay is now replicated, so E2 (wd) and E3
(annealing) are the correct, direct follow-ups rather than exploratory fishing.
E2 is locked to seed 0 and does not itself require re-replicating seeds 1/2; the seed-stability
claim established by E1 is what licenses proceeding to the sweep.
|
