path: root/NOTES-E6.md

# E6 — dataset extension to [2, 1000] — analysis notes

Sources: `runs/e6/<job>/<model>/seed<seed>/results.json` (eval run on ichi after each
cell's training), `metrics.csv` (per-eval curves), and the locked interpretation clauses in
`design/preregistration.md` **Addendum 6** (P-ladder / sieve-rank estimation) plus Addenda 1–4
for the unchanged O/H/P operationalizations.

**Batch status at analysis time (2026-08-17 ~22:45 UTC): 8 of 16 cells finished, 4 running,
4 queued.** The sweep on ichi (`run_sweep`, concurrency 4) is still in its third batch
(wd 1.0 seeds {1,2}); the fourth batch (wd 3.0 seed 0, batch-128 seed 0) has not started.
This note classifies the 8 finished cells; the remaining 8 are reported as progress only and
will be completed by the 08:30 safety-net cron.

Constants used throughout: val EM is exact-match; "best" is the val-selected checkpoint
(selection-holed), "last" is the unselected final checkpoint. **The E6 probe is exact-match on
[1001, 2000]** (out of training range; range_end = 1000). The old [101, 200] probe is *in-range*
for E6 and is reported only as an in-range generalization reference, NOT as the out-of-range
probe. Probe was computed on voidlaptop (CPU) by loading `last.pt` with `map_location='cpu'`
(the checkpoints were trained with CUDA + AMP on ichi; eval.py's own `_load` would try
`device=cuda` and cannot run on this host).

---

## Per-cell classification (8 finished cells)

| job | model | O | H | halt mean | val EM best | val EM last | train EM (max/final) | probe [1001,2000] acc | P | pred(960) | probe [101,200] (in-range) |
|-----|-------|---|-----|-----------|-------------|-------------|----------------------|-----------------------|---|-----------|---------------------------|
| e6-wd001 | rnn | O-PARTIAL | H1 | 2.00 | 0.783 | 0.770 | 1.000 / 1.000 | 0.040 (40/1000) | P4 | 967 | 0.94 |
| e6-wd001 | transformer | O-PARTIAL | — | — | 0.840 | 0.803 | 1.000 / 1.000 | 0.007 (7/1000) | P4 | 967 | 0.91 |
| e6-wd01 | rnn | O-PARTIAL | H1 | 2.00 | 0.830 | 0.807 | 1.000 / 1.000 | 0.049 (49/1000) | P4 | 967 | 0.90 |
| e6-wd01 | transformer | O-PARTIAL | — | — | 0.827 | 0.807 | 1.000 / 1.000 | 0.007 (7/1000) | P4 | 967 | 0.92 |
| e6-wd03 | rnn | O-PARTIAL | H1 | 2.03 | 0.823 | 0.787 | 1.000 / 1.000 | 0.039 (39/1000) | P4 | 967 | 0.95 |
| e6-wd03 | transformer | O-PARTIAL | — | — | 0.840 | 0.787 | 1.000 / 1.000 | 0.008 (8/1000) | P4 | 967 | 0.92 |
| e6-wd10 | rnn | O4 | H4 | 2.96 | 0.480 | 0.303 | 0.605 / 0.452 | 0.004 (4/1000) | P4 | 967 | 0.44 |
| e6-wd10 | transformer | O4 | — | — | 0.797 | 0.677 | 0.987 / 0.890 | 0.004 (4/1000) | P4 | 967 | 0.73 |

Progress (running / queued cells, at ~22:45 UTC):

| cell | status | step | train EM | val EM |
|------|--------|------|----------|--------|
| e6-wd10 rnn s1 | running | 198000/200000 | 0.323 | 0.267 |
| e6-wd10 rnn s2 | running | 187800 | 0.489 | 0.353 |
| e6-wd10 transformer s1 | running | 180600 | 0.883 | 0.627 |
| e6-wd10 transformer s2 | running | 177800 | 0.887 | 0.700 |
| e6-wd30 rnn s0 | queued | — | — | — |
| e6-wd30 transformer s0 | queued | — | — | — |
| e6-b128 rnn s0 | queued | — | — | — |
| e6-b128 transformer s0 | queued | — | — | — |

---

## O-codes — no O1 anywhere (no grokking)

**LOCKED** (Addenda 1–4, unchanged): O1 = sharp transition after sustained train saturation;
O-PARTIAL = train saturated, val never reaches 0.9, final val in (0.3, 0.9); O4 = train never
sustained ≥ 0.95 for 10 evals (setup/optimization failure, no scientific reading).

**Commentary.** The four low-wd cells (0.01, 0.1, 0.3 × both models) saturate train
(train EM = 1.0, saturating from eval 2–63) and end with val EM 0.77–0.81 → **O-PARTIAL**. The
two wd-1.0 cells are **O4**: the RNN's train EM never exceeds 0.605 (it cannot even memorize
the range under wd 1.0), and the transformer peaks at 0.987 but never sustains ≥ 0.95 for 10
consecutive evals. **No cell shows the O1 grokking signature.** This is the same headline as
phases 1–3: increasing the dataset 10× did not produce the delayed generalization jump.

**Caveat on the transformer at wd 1.0.** It reaches a 0.797 val EM *best* checkpoint (in-range)
yet is O4 by the locked definition (train never sustained saturation). Do not read its 0.677
final val as O-PARTIAL; the O4 branch fires first by construction.

---

## P-codes — all P4, and the sieve-rank ladder yields "no sieve"

**LOCKED** (Addendum 6): probe range [1001, 2000]; P5(k) = errors concentrated on the rank-k
signature set; P6 = exact (no probe misses); P3 threshold ≥ 90% on [1001, 2000]; P4/P2 keep
their existing meanings. A model that internalized a k-prime sieve misses exactly the composites
whose prime factors all exceed p_k; the smallest missed composite identifies k: 1147→k=10,
1369→k=11, 1681→k=12, 1849→k=13, none→k≥14.

**Commentary.** Every finished cell is **P4**: probe accuracy on [1001, 2000] is 0.4–4.9%
(4–49 of 1000), and 574–597 of the 600 trivial inputs (even or multiple of 5) are wrong —
i.e. the models fail "the next prime after an even number is odd" 95–99.5% of the time.
Out-of-range transfer is, as in phases 1–3, essentially zero.

**Sieve-rank finding.** No cell internalized a k-prime sieve. The diagnostic is the *smallest
composite the model ever outputs as "prime"* in the probe range; a genuine rank-k sieve's
smallest such composite is one of {1147, 1369, 1681, 1849} (or none, for k ≥ 14). Every cell
instead emits small composites with small prime factors — e.g. wd-10 transformer predicts 9, 49,
100, 110; wd-0.3 transformer predicts 49; wd-0.01 rnn predicts 100; the smallest *in-range*
composite predictions are 1003 = 17·59, 1079 = 13·83, 1099 = 7·157, 1671 = 3·557. A model that
outputs 3·557 or 7·157 as a prime has no divisibility structure at any rank. So the P-ladder
resolves to **"no sieve" (k undefined) in all 8 cells** — P5(k) and P6 do not fire anywhere, and
P3 (≥ 90%) is far from firing.

**The 960→961 discriminator is uninformative here.** All 8 cells output **967** for n = 960
(the correct answer, which is the "k ≥ 11" behaviour). But 960 sits in the *training* split
(699 train / 300 val, seed 0), so this is memorization of the 960→967 pair, not sieve transfer.
A true k=10 sieve would output 961 (it cannot see 31|961); these models get 960 right *because
they memorized it*, while simultaneously failing 95–99.5% of out-of-range inputs. The
discriminator is only meaningful once a cell is already P5/P6; no cell is.

**In-range reference.** The old [101, 200] probe — which is *in-range* for E6 — scores 0.90–0.95
at low wd (falling to 0.44–0.73 at wd 1.0). This is the same pattern phases 1–3 saw *on their
out-of-range probe*: the model generalizes within its training distribution but transfers
nothing beyond it. It is not a positive result, just a reminder that the correct E6 probe is
[1001, 2000].

---

## H-codes — halting still collapsed / noisy

**LOCKED** (Addenda 1–4): H1 = mean steps at the min_steps floor (2); H2 = pinned at K; H3 =
intermediate + positive gap correlation; H4 = intermediate but no correlation.

**Commentary.** The three low-wd RNN cells are **H1** (mean 2.00–2.03, collapsed to the
min_steps floor), and the wd-1.0 RNN is **H4** (mean 2.96, noisy). Same as every prior phase:
the ACT gate never learned a computation budget. K = 32 here (vs 20 in phases 1–3) makes no
difference — the gate still bottoms out at the floor or hovers unstructured.

---

## E6 verdict vs phases 1–3

**LOCKED** (Addendum 6): "an O1 anywhere = data pressure unlocked grokking; same codes = the
task's walls are algorithmic, not data-bound."

**Commentary.** No O1 anywhere, and out-of-range transfer is P4 in all 8 cells — so the
"data pressure unlocked grokking" branch does **not** fire. The outcome *class* is unchanged:
no grokking, zero out-of-range transfer, no divisibility algorithm. The one measurable shift is
**in-range only**: the tied RNN, which was O4 everywhere at [2, 100] (val EM best ≤ 36.7%), is
now O-PARTIAL at wd 0.01/0.1/0.3 (val EM best 0.78–0.83, final 0.77–0.81) — i.e. at 699
training examples the RNN finally holds an in-range memorized/heuristic solution it could not
hold at 69. That is a real, reportable measurement, but it is not grokking and it does not
transfer. Reading the locked clause strictly, the headline is the algorithmic-bound branch:
**more data eased in-range fitting but did not move the task's walls — the search/increment
loop and divisibility test remain unlearnable as transferable structure under these dynamics.**

**Caveats.** The clause is binary ("O1 anywhere" vs "same codes"); the measured pattern is
"no O1, and the in-range RNN O-code improved one notch". We report the O-code shift as a
measurement rather than forcing it into either binary branch. Seed 0 only for the wd sweep;
the wd-1.0 seeds {1,2} (running) will tell whether the in-range RNN improvement is seed-stable.

---

## Caveats / known gaps

1. **eval.py still hardcodes the [101, 200] probe.** Addendum 6 moves the E6 probe to
   [1001, 2000] and adds the P5(k)/P6 ladder, but `src/eval.py::probe_report` still defaults to
   `lo=101, hi=200` and only emits P1–P4. The [1001, 2000] numbers above were computed by a
   standalone probe (load `last.pt` on CPU, greedy-decode over [1001, 2000]). `eval.py` should
   be taught the E6 probe range + ladder before the remaining 8 cells are classified, otherwise
   their committed `results.json` will again carry the in-range [101, 200] code under the P
   field (harmless for O/H, misleading for P).
2. **`--post-only` cannot run on this host for E6.** The checkpoints are `device=cuda`; eval's
   `_load` honours `cfg.device='cuda'` and cannot run on the CPU-only voidlaptop. Re-scoring must
   happen on ichi (which the sweep already does automatically after each cell trains — the 8
   finished cells were eval'd there). The `runs/e6/summary.csv` will therefore not exist until
   the sweep on ichi completes and is rsync'd back.
3. **fp16 numerics.** E6 ran with AMP fp16 on CUDA (a deliberate, recorded change vs the CPU
   fp32 baseline). All 8 cells sit far from any O/P/H boundary (e.g. probe acc 0.4–4.9% vs the
   90% P3 line; val EM 0.77–0.81 vs the 0.9 O1 line), so the precision caveat in Addendum 6 does
   not bind for any finished cell.
4. **960 is in the training split**, so the 960→967 result is memorization, not sieve evidence
   (see P-codes section).

---

## E7 next step (per Addendum 6)

Long-horizon test: **20× budget (4M steps)** on the best E6 cells, per jayrup's decision. The
candidate cells are the low-wd O-PARTIAL cells that hold the strongest in-range solution —
e6-wd01 / e6-wd03 (rnn and transformer, val EM best 0.83–0.84, final 0.79–0.81) — with the
wd-1.0 control for parity. Protocol locked in Addendum 7 post-E6. The E6 read so far gives no
reason to expect the 20× horizon to unlock transfer (the wall is algorithmic, not a data or
budget shortfall at the margins tested), but the long horizon is exactly the test that separates
"slow algorithmic basin" from "no basin at all".