path: root/NOTES-E6.md

# E6 — dataset extension to [2, 1000] — analysis notes (16/16 cells)

Sources: `runs/e6/<job>/<model>/seed<seed>/results.json` (eval run on ichi after each
cell's training), `metrics.csv` (per-eval curves), and the locked interpretation clauses in
`design/preregistration.md` **Addendum 6** (P-ladder / sieve-rank estimation) plus Addenda 1–4
for the unchanged O/H/P operationalizations.

**Batch status at analysis time (2026-08-18, safety-net cron): all 16 cells finished.**
The ichi sweep completed the fourth batch (wd 3.0 seed 0, batch-128 seed 0) and the
wd-1.0 seeds {1,2}; results were rsync'd back and this note classifies the full 16-cell grid.
(The 23:30 checker had only 8 of 16 finished; this run replaces that partial analysis.)

Constants used throughout: val EM is exact-match; "best" is the val-selected checkpoint
(selection-holed), "last" is the unselected final checkpoint. **The E6 probe is exact-match on
[1001, 2000]** (out of training range; range_end = 1000). The old [101, 200] probe is *in-range*
for E6 and is reported only as an in-range generalization reference, NOT as the out-of-range
probe. Probe was computed on voidlaptop (CPU) by loading `last.pt` with `map_location='cpu'`
(the checkpoints were trained with CUDA + AMP on ichi; eval.py's own `_load` would try
`device=cuda` and cannot run on this host).

---

## Per-cell classification (all 16 cells)

| job | model | O | H | halt mean | val EM best | val EM last | train EM (max/final) | probe [1001,2000] acc | P | pred(960) | probe [101,200] (in-range) |
|-----|-------|---|-----|-----------|-------------|-------------|----------------------|-----------------------|---|-----------|---------------------------|
| e6-wd001 | rnn | O-PARTIAL | H1 | 2.00 | 0.783 | 0.770 | 1.000 / 1.000 | 0.040 (40/1000) | P4 | 967 | 0.94 |
| e6-wd001 | transformer | O-PARTIAL | — | — | 0.840 | 0.803 | 1.000 / 1.000 | 0.007 (7/1000) | P4 | 967 | 0.91 |
| e6-wd01 | rnn | O-PARTIAL | H1 | 2.00 | 0.830 | 0.807 | 1.000 / 1.000 | 0.049 (49/1000) | P4 | 967 | 0.90 |
| e6-wd01 | transformer | O-PARTIAL | — | — | 0.827 | 0.807 | 1.000 / 1.000 | 0.007 (7/1000) | P4 | 967 | 0.92 |
| e6-wd03 | rnn | O-PARTIAL | H1 | 2.03 | 0.823 | 0.787 | 1.000 / 1.000 | 0.039 (39/1000) | P4 | 967 | 0.95 |
| e6-wd03 | transformer | O-PARTIAL | — | — | 0.840 | 0.787 | 1.000 / 1.000 | 0.008 (8/1000) | P4 | 967 | 0.92 |
| e6-wd10 | rnn | O4 | H4 | 2.96 | 0.480 | 0.303 | 0.605 / 0.452 | 0.004 (4/1000) | P4 | 967 | 0.44 |
| e6-wd10 | transformer | O4 | — | — | 0.797 | 0.677 | 0.987 / 0.890 | 0.004 (4/1000) | P4 | 967 | 0.73 |
| e6-wd10 | rnn s1 | O4 | H1 | 2.38 | 0.483 | 0.270 | 0.631 / 0.351 | 0.000 (0/1000) | P4 | 967 | 0.25 |
| e6-wd10 | transformer s1 | O4 | — | — | 0.770 | 0.683 | 0.986 / 0.941 | 0.000 (0/1000) | P4 | 967 | 0.85 |
| e6-wd10 | rnn s2 | O4 | H4 | 2.98 | 0.473 | 0.370 | 0.638 / 0.504 | 0.001 (1/1000) | P4 | 967 | 0.35 |
| e6-wd10 | transformer s2 | O4 | — | — | 0.827 | 0.737 | 0.993 / 0.924 | 0.005 (5/1000) | P4 | 967 | 0.85 |
| e6-wd30 | rnn | O4 | H4 | 2.86 | 0.080 | 0.017 | 0.084 / 0.026 | 0.000 (0/1000) | P4 | 611 | 0.01 |
| e6-wd30 | transformer | O4 | — | — | 0.590 | 0.260 | 0.800 / 0.361 | 0.000 (0/1000) | P4 | 967 | 0.41 |
| e6-b128 | rnn | O4 | H4 | 7.70 | 0.547 | 0.453 | 0.848 / 0.741 | 0.000 (0/1000) | P4 | 967 | 0.64 |
| e6-b128 | transformer | O-PARTIAL | — | — | 0.830 | 0.453 | 1.000 / 0.554 | 0.001 (1/1000) | P4 | 967 | 0.43 |

---

## O-codes — no O1 anywhere (no grokking)

**LOCKED** (Addenda 1–4, unchanged): O1 = sharp transition after sustained train saturation;
O-PARTIAL = train saturated, val never reaches 0.9, final val in (0.3, 0.9); O4 = train never
sustained ≥ 0.95 for 10 evals (setup/optimization failure, no scientific reading).

**Commentary.** Seven cells are **O-PARTIAL**: the six low-wd cells (0.01/0.1/0.3 × both models)
plus the batch-128 transformer. The low-wd six saturate train (train EM = 1.0, saturating from
eval 2–63) and end with val EM 0.77–0.81. Nine cells are **O4**: the entire wd-1.0 block
(all three seeds × both models), the wd-3.0 pair, and the batch-128 RNN. The wd-1.0 RNN's train
EM never exceeds 0.64 (it cannot even memorize the range under wd 1.0); the wd-1.0 transformer
peaks at train EM 0.986–0.993 but never sustains ≥ 0.95 for 10 evals; wd 3.0 collapses outright
(train EM max 0.08–0.80). **No cell shows the O1 grokking signature**, across the full 16-cell
grid — the same headline as phases 1–3, now with the replication seeds and the two ablations
in hand.

**Caveat on the two O-PARTIAL boundary cells.**
- **wd-1.0 transformer** (best val 0.797, final 0.677) reaches a 0.797 *best* checkpoint yet is
  O4 by the locked definition (train never sustained saturation). Its 0.677 final val is NOT
  O-PARTIAL; the O4 branch fires first by construction.
- **batch-128 transformer** IS O-PARTIAL (train saturated for 841 of 1000 evals), but it is a
  *destabilized* O-PARTIAL: at the final eval (step 200 000) its loss spiked 0.011 → 0.45 and
  train EM collapsed 1.0 → 0.554 while val fell 0.82 → 0.453. It trained to saturation, then
  the constant-lr 1e-3 schedule at batch 128 diverged at the very end. This is the same
  late-run rollover phases 1–3 saw, but as a hard spike rather than a gentle decay — the larger
  batch (4× the default 32) makes each optimizer step correspondingly more aggressive.

---

## P-codes — all P4, and the sieve-rank ladder yields "no sieve" (16/16)

**LOCKED** (Addendum 6): probe range [1001, 2000]; P5(k) = errors concentrated on the rank-k
signature set; P6 = exact (no probe misses); P3 threshold ≥ 90% on [1001, 2000]; P4/P2 keep
their existing meanings. A model that internalized a k-prime sieve misses exactly the composites
whose prime factors all exceed p_k; the smallest missed composite identifies k: 1147→k=10,
1369→k=11, 1681→k=12, 1849→k=13, none→k≥14.

**Commentary.** Every one of the 16 cells is **P4**: probe accuracy on [1001, 2000] is 0.0–4.9%
(0–49 of 1000), and 95.3–100% of the 600 trivial inputs (even or multiple of 5) are wrong —
i.e. the models fail "the next prime after an even number is odd" essentially always.
Out-of-range transfer is, as in phases 1–3, zero across the entire grid (no wd value, no model,
no seed, no batch size escapes P4).

**Sieve-rank finding.** No cell internalized a k-prime sieve. The diagnostic is the *smallest
composite the model ever outputs as "prime"* in the probe range; a genuine rank-k sieve's
smallest such composite is one of {1147, 1369, 1681, 1849} (or none, for k ≥ 14). The smallest
in-range composite predictions observed are 1003 = 17·59, 1001 = 7·11·13, 1010 = 2·5·101,
1079 = 13·83, 1099 = 7·157 — all composites with *small* prime factors — and, out of range,
tiny values like 9, 33, 49, 100, 110. A model that outputs 3·557 or 7·157 or 2·5·101 as a prime
has no divisibility structure at any rank. (A handful of the most collapsed cells — wd-1.0 s1
pair, wd-3.0 pair — emit no composite in range at all, but that is an artefact of their garbage
decodes, e.g. wd-3.0 RNN answers 960→611 = 13·47 and emits 2211 out of range; it is not a k ≥ 14
sieve, it is acc = 0.0.) So the P-ladder resolves to **"no sieve" (k undefined) in all 16
cells** — P5(k) and P6 do not fire anywhere, and P3 (≥ 90%) is far from firing.

**The 960→961 discriminator is uninformative here.** 15 of 16 cells output **967** for n = 960
(the correct answer, which is the "k ≥ 11" behaviour); the exception is wd-3.0 RNN, which
outputs 611. But 960 sits in the *training* split (699 train / 300 val, seed 0), so 967 is
memorization of the 960→967 pair, not sieve transfer — a true k=10 sieve would output 961 (it
cannot see 31|961). These models get 960 right *because they memorized it* while simultaneously
failing 95–100% of out-of-range inputs. The discriminator is only meaningful once a cell is
already P5/P6; no cell is.

**In-range reference.** The old [101, 200] probe — which is *in-range* for E6 — scores
0.90–0.95 at low wd (falling to 0.44–0.73 at wd 1.0, 0.01–0.41 at wd 3.0). This is the same
pattern phases 1–3 saw *on their out-of-range probe*: the model generalizes within its training
distribution but transfers nothing beyond it. It is not a positive result, just a reminder that
the correct E6 probe is [1001, 2000].

---

## H-codes — halting still collapsed / noisy, and the batch-128 cell is a wrinkle

**LOCKED** (Addenda 1–4): H1 = mean steps at the min_steps floor (2); H2 = pinned at K; H3 =
intermediate + positive gap correlation; H4 = intermediate but no correlation.

**Commentary.** The three low-wd RNN cells are **H1** (mean 2.00–2.03, collapsed to the
min_steps floor). The wd-1.0 RNN is seed-unstable: seed 0 **H4** (mean 2.96), seed 1 **H1**
(mean 2.38), seed 2 **H4** (mean 2.98) — the ACT gate has no stable structure at wd 1.0. The
wd-3.0 RNN is **H4** (mean 2.86, but the model is so collapsed the reading is moot). The
batch-128 RNN is the one wrinkle: **H4 with mean 7.70 steps** — the larger batch *prevented*
the usual collapse-to-floor (every other RNN bottoms out at 2–3), but there is still no positive
gap correlation (ρ = −0.05), so the gate learned to run ~7 steps unconditionally rather than to
budget compute by gap size. Same as every prior phase: the ACT gate never learned a computation
budget. K = 32 here (vs 20 in phases 1–3) makes no difference.

---

## E6 verdict vs phases 1–3

**LOCKED** (Addendum 6): "an O1 anywhere = data pressure unlocked grokking; same codes = the
task's walls are algorithmic, not data-bound."

**Commentary.** No O1 anywhere, and out-of-range transfer is P4 in all 16 cells — so the
"data pressure unlocked grokking" branch does **not** fire. The outcome *class* is unchanged:
no grokking, zero out-of-range transfer, no divisibility algorithm. The measurable shifts are
**in-range only**:

1. The tied RNN, which was O4 everywhere at [2, 100] (val EM best ≤ 36.7%), is now O-PARTIAL at
   wd 0.01/0.1/0.3 (val EM best 0.78–0.83) — at 699 training examples it finally holds an
   in-range memorized/heuristic solution it could not hold at 69. This is seed-0 only (the wd
   sweep has no replication), and it is wd-dependent (still O4 at wd 1.0). The wd-1.0
   replication seeds {1,2} confirm the *wd-1.0* behaviour is seed-stable (O4 in all three seeds,
   both models) but say nothing about the low-wd O-PARTIAL result.
2. The batch-size ablation is a null: batch 128 does not rescue transfer (still P4, acc 0.0–0.1%)
   and, in the transformer, trades the usual gentle late-run rollover for a hard final-step
   divergence (train 1.0 → 0.55, val 0.83 → 0.45).

Reading the locked clause strictly, the headline is the algorithmic-bound branch: **more data
eased in-range fitting but did not move the task's walls — the search/increment loop and
divisibility test remain unlearnable as transferable structure under these dynamics.** The
clause is binary ("O1 anywhere" vs "same codes"); the measured pattern is "no O1, and the
in-range RNN O-code improved one notch (seed 0, low wd only)". We report the O-code shift as a
measurement rather than forcing it into either binary branch.

---

## Caveats / known gaps

1. **eval.py still hardcodes the [101, 200] probe.** `src/eval.py::probe_report` still defaults
   to `lo=101, hi=200` and only emits P1–P4, so the committed `results.json` P-field and
   `summary.csv` `probe` column carry the *in-range* [101, 200] code (e.g. P3 at 0.94), which is
   misleading for E6. The [1001, 2000] numbers above are from a standalone CPU probe. `eval.py`
   should be taught the E6 probe range + P5(k)/P6 ladder before E7 so its committed results are
   self-consistent — this gap is now the only open tooling item.
2. **`--post-only` cannot run on this host for E6.** The checkpoints are `device=cuda`; eval's
   `_load` honours `cfg.device='cuda'` and cannot run on the CPU-only voidlaptop. Re-scoring on
   voidlaptop loads `last.pt` with `map_location='cpu'` directly (as done here). The
   `runs/e6/summary.csv` was produced by the sweep on ichi and rsync'd back.
3. **fp16 numerics.** E6 ran with AMP fp16 on CUDA. No cell sits within ±2 points of an O/P/H
   boundary (probe acc 0–4.9% vs the 90% P3 line; val EM far from the 0.9 O1 line; halt means
   far from the 2.5 / 31.5 H1/H2 cutoffs), so the precision caveat in Addendum 6 does not bind
   for any cell.
4. **960 is in the training split**, so the 960→967 result is memorization, not sieve evidence
   (see P-codes section). The wd-3.0 RNN's 960→611 is a collapsed model emitting a composite,
   not a k=10 sieve (which would emit 961).

---

## E7 next step (per Addendum 6)

Long-horizon test: **20× budget (4M steps)** on the best E6 cells, per jayrup's decision. The
candidate cells are the low-wd O-PARTIAL cells that hold the strongest in-range solution —
e6-wd01 / e6-wd03 (rnn and transformer, val EM best 0.83–0.84, final 0.79–0.81) — with the
wd-1.0 control for parity. Protocol locked in Addendum 7 post-E6. The full 16-cell E6 read gives
no reason to expect the 20× horizon to unlock transfer (the wall is algorithmic, not a data or
budget shortfall at the margins tested — batch size and wd and data volume all failed to move
it), but the long horizon is exactly the test that separates "slow algorithmic basin" from
"no basin at all".