diff options
Diffstat (limited to 'NOTES-E6.md')
| -rw-r--r-- | NOTES-E6.md | 205 |
1 files changed, 112 insertions, 93 deletions
diff --git a/NOTES-E6.md b/NOTES-E6.md index 06bc685..0adc3a4 100644 --- a/NOTES-E6.md +++ b/NOTES-E6.md @@ -1,15 +1,14 @@ -# E6 — dataset extension to [2, 1000] — analysis notes +# E6 — dataset extension to [2, 1000] — analysis notes (16/16 cells) Sources: `runs/e6/<job>/<model>/seed<seed>/results.json` (eval run on ichi after each cell's training), `metrics.csv` (per-eval curves), and the locked interpretation clauses in `design/preregistration.md` **Addendum 6** (P-ladder / sieve-rank estimation) plus Addenda 1–4 for the unchanged O/H/P operationalizations. -**Batch status at analysis time (2026-08-17 ~22:45 UTC): 8 of 16 cells finished, 4 running, -4 queued.** The sweep on ichi (`run_sweep`, concurrency 4) is still in its third batch -(wd 1.0 seeds {1,2}); the fourth batch (wd 3.0 seed 0, batch-128 seed 0) has not started. -This note classifies the 8 finished cells; the remaining 8 are reported as progress only and -will be completed by the 08:30 safety-net cron. +**Batch status at analysis time (2026-08-18, safety-net cron): all 16 cells finished.** +The ichi sweep completed the fourth batch (wd 3.0 seed 0, batch-128 seed 0) and the +wd-1.0 seeds {1,2}; results were rsync'd back and this note classifies the full 16-cell grid. +(The 23:30 checker had only 8 of 16 finished; this run replaces that partial analysis.) Constants used throughout: val EM is exact-match; "best" is the val-selected checkpoint (selection-holed), "last" is the unselected final checkpoint. **The E6 probe is exact-match on @@ -21,7 +20,7 @@ probe. Probe was computed on voidlaptop (CPU) by loading `last.pt` with `map_loc --- -## Per-cell classification (8 finished cells) +## Per-cell classification (all 16 cells) | job | model | O | H | halt mean | val EM best | val EM last | train EM (max/final) | probe [1001,2000] acc | P | pred(960) | probe [101,200] (in-range) | |-----|-------|---|-----|-----------|-------------|-------------|----------------------|-----------------------|---|-----------|---------------------------| @@ -33,19 +32,14 @@ probe. Probe was computed on voidlaptop (CPU) by loading `last.pt` with `map_loc | e6-wd03 | transformer | O-PARTIAL | — | — | 0.840 | 0.787 | 1.000 / 1.000 | 0.008 (8/1000) | P4 | 967 | 0.92 | | e6-wd10 | rnn | O4 | H4 | 2.96 | 0.480 | 0.303 | 0.605 / 0.452 | 0.004 (4/1000) | P4 | 967 | 0.44 | | e6-wd10 | transformer | O4 | — | — | 0.797 | 0.677 | 0.987 / 0.890 | 0.004 (4/1000) | P4 | 967 | 0.73 | - -Progress (running / queued cells, at ~22:45 UTC): - -| cell | status | step | train EM | val EM | -|------|--------|------|----------|--------| -| e6-wd10 rnn s1 | running | 198000/200000 | 0.323 | 0.267 | -| e6-wd10 rnn s2 | running | 187800 | 0.489 | 0.353 | -| e6-wd10 transformer s1 | running | 180600 | 0.883 | 0.627 | -| e6-wd10 transformer s2 | running | 177800 | 0.887 | 0.700 | -| e6-wd30 rnn s0 | queued | — | — | — | -| e6-wd30 transformer s0 | queued | — | — | — | -| e6-b128 rnn s0 | queued | — | — | — | -| e6-b128 transformer s0 | queued | — | — | — | +| e6-wd10 | rnn s1 | O4 | H1 | 2.38 | 0.483 | 0.270 | 0.631 / 0.351 | 0.000 (0/1000) | P4 | 967 | 0.25 | +| e6-wd10 | transformer s1 | O4 | — | — | 0.770 | 0.683 | 0.986 / 0.941 | 0.000 (0/1000) | P4 | 967 | 0.85 | +| e6-wd10 | rnn s2 | O4 | H4 | 2.98 | 0.473 | 0.370 | 0.638 / 0.504 | 0.001 (1/1000) | P4 | 967 | 0.35 | +| e6-wd10 | transformer s2 | O4 | — | — | 0.827 | 0.737 | 0.993 / 0.924 | 0.005 (5/1000) | P4 | 967 | 0.85 | +| e6-wd30 | rnn | O4 | H4 | 2.86 | 0.080 | 0.017 | 0.084 / 0.026 | 0.000 (0/1000) | P4 | 611 | 0.01 | +| e6-wd30 | transformer | O4 | — | — | 0.590 | 0.260 | 0.800 / 0.361 | 0.000 (0/1000) | P4 | 967 | 0.41 | +| e6-b128 | rnn | O4 | H4 | 7.70 | 0.547 | 0.453 | 0.848 / 0.741 | 0.000 (0/1000) | P4 | 967 | 0.64 | +| e6-b128 | transformer | O-PARTIAL | — | — | 0.830 | 0.453 | 1.000 / 0.554 | 0.001 (1/1000) | P4 | 967 | 0.43 | --- @@ -55,20 +49,30 @@ Progress (running / queued cells, at ~22:45 UTC): O-PARTIAL = train saturated, val never reaches 0.9, final val in (0.3, 0.9); O4 = train never sustained ≥ 0.95 for 10 evals (setup/optimization failure, no scientific reading). -**Commentary.** The four low-wd cells (0.01, 0.1, 0.3 × both models) saturate train -(train EM = 1.0, saturating from eval 2–63) and end with val EM 0.77–0.81 → **O-PARTIAL**. The -two wd-1.0 cells are **O4**: the RNN's train EM never exceeds 0.605 (it cannot even memorize -the range under wd 1.0), and the transformer peaks at 0.987 but never sustains ≥ 0.95 for 10 -consecutive evals. **No cell shows the O1 grokking signature.** This is the same headline as -phases 1–3: increasing the dataset 10× did not produce the delayed generalization jump. - -**Caveat on the transformer at wd 1.0.** It reaches a 0.797 val EM *best* checkpoint (in-range) -yet is O4 by the locked definition (train never sustained saturation). Do not read its 0.677 -final val as O-PARTIAL; the O4 branch fires first by construction. +**Commentary.** Seven cells are **O-PARTIAL**: the six low-wd cells (0.01/0.1/0.3 × both models) +plus the batch-128 transformer. The low-wd six saturate train (train EM = 1.0, saturating from +eval 2–63) and end with val EM 0.77–0.81. Nine cells are **O4**: the entire wd-1.0 block +(all three seeds × both models), the wd-3.0 pair, and the batch-128 RNN. The wd-1.0 RNN's train +EM never exceeds 0.64 (it cannot even memorize the range under wd 1.0); the wd-1.0 transformer +peaks at train EM 0.986–0.993 but never sustains ≥ 0.95 for 10 evals; wd 3.0 collapses outright +(train EM max 0.08–0.80). **No cell shows the O1 grokking signature**, across the full 16-cell +grid — the same headline as phases 1–3, now with the replication seeds and the two ablations +in hand. + +**Caveat on the two O-PARTIAL boundary cells.** +- **wd-1.0 transformer** (best val 0.797, final 0.677) reaches a 0.797 *best* checkpoint yet is + O4 by the locked definition (train never sustained saturation). Its 0.677 final val is NOT + O-PARTIAL; the O4 branch fires first by construction. +- **batch-128 transformer** IS O-PARTIAL (train saturated for 841 of 1000 evals), but it is a + *destabilized* O-PARTIAL: at the final eval (step 200 000) its loss spiked 0.011 → 0.45 and + train EM collapsed 1.0 → 0.554 while val fell 0.82 → 0.453. It trained to saturation, then + the constant-lr 1e-3 schedule at batch 128 diverged at the very end. This is the same + late-run rollover phases 1–3 saw, but as a hard spike rather than a gentle decay — the larger + batch (4× the default 32) makes each optimizer step correspondingly more aggressive. --- -## P-codes — all P4, and the sieve-rank ladder yields "no sieve" +## P-codes — all P4, and the sieve-rank ladder yields "no sieve" (16/16) **LOCKED** (Addendum 6): probe range [1001, 2000]; P5(k) = errors concentrated on the rank-k signature set; P6 = exact (no probe misses); P3 threshold ≥ 90% on [1001, 2000]; P4/P2 keep @@ -76,45 +80,54 @@ their existing meanings. A model that internalized a k-prime sieve misses exactl whose prime factors all exceed p_k; the smallest missed composite identifies k: 1147→k=10, 1369→k=11, 1681→k=12, 1849→k=13, none→k≥14. -**Commentary.** Every finished cell is **P4**: probe accuracy on [1001, 2000] is 0.4–4.9% -(4–49 of 1000), and 574–597 of the 600 trivial inputs (even or multiple of 5) are wrong — -i.e. the models fail "the next prime after an even number is odd" 95–99.5% of the time. -Out-of-range transfer is, as in phases 1–3, essentially zero. +**Commentary.** Every one of the 16 cells is **P4**: probe accuracy on [1001, 2000] is 0.0–4.9% +(0–49 of 1000), and 95.3–100% of the 600 trivial inputs (even or multiple of 5) are wrong — +i.e. the models fail "the next prime after an even number is odd" essentially always. +Out-of-range transfer is, as in phases 1–3, zero across the entire grid (no wd value, no model, +no seed, no batch size escapes P4). **Sieve-rank finding.** No cell internalized a k-prime sieve. The diagnostic is the *smallest composite the model ever outputs as "prime"* in the probe range; a genuine rank-k sieve's -smallest such composite is one of {1147, 1369, 1681, 1849} (or none, for k ≥ 14). Every cell -instead emits small composites with small prime factors — e.g. wd-10 transformer predicts 9, 49, -100, 110; wd-0.3 transformer predicts 49; wd-0.01 rnn predicts 100; the smallest *in-range* -composite predictions are 1003 = 17·59, 1079 = 13·83, 1099 = 7·157, 1671 = 3·557. A model that -outputs 3·557 or 7·157 as a prime has no divisibility structure at any rank. So the P-ladder -resolves to **"no sieve" (k undefined) in all 8 cells** — P5(k) and P6 do not fire anywhere, and -P3 (≥ 90%) is far from firing. - -**The 960→961 discriminator is uninformative here.** All 8 cells output **967** for n = 960 -(the correct answer, which is the "k ≥ 11" behaviour). But 960 sits in the *training* split -(699 train / 300 val, seed 0), so this is memorization of the 960→967 pair, not sieve transfer. -A true k=10 sieve would output 961 (it cannot see 31|961); these models get 960 right *because -they memorized it*, while simultaneously failing 95–99.5% of out-of-range inputs. The -discriminator is only meaningful once a cell is already P5/P6; no cell is. - -**In-range reference.** The old [101, 200] probe — which is *in-range* for E6 — scores 0.90–0.95 -at low wd (falling to 0.44–0.73 at wd 1.0). This is the same pattern phases 1–3 saw *on their -out-of-range probe*: the model generalizes within its training distribution but transfers -nothing beyond it. It is not a positive result, just a reminder that the correct E6 probe is -[1001, 2000]. +smallest such composite is one of {1147, 1369, 1681, 1849} (or none, for k ≥ 14). The smallest +in-range composite predictions observed are 1003 = 17·59, 1001 = 7·11·13, 1010 = 2·5·101, +1079 = 13·83, 1099 = 7·157 — all composites with *small* prime factors — and, out of range, +tiny values like 9, 33, 49, 100, 110. A model that outputs 3·557 or 7·157 or 2·5·101 as a prime +has no divisibility structure at any rank. (A handful of the most collapsed cells — wd-1.0 s1 +pair, wd-3.0 pair — emit no composite in range at all, but that is an artefact of their garbage +decodes, e.g. wd-3.0 RNN answers 960→611 = 13·47 and emits 2211 out of range; it is not a k ≥ 14 +sieve, it is acc = 0.0.) So the P-ladder resolves to **"no sieve" (k undefined) in all 16 +cells** — P5(k) and P6 do not fire anywhere, and P3 (≥ 90%) is far from firing. + +**The 960→961 discriminator is uninformative here.** 15 of 16 cells output **967** for n = 960 +(the correct answer, which is the "k ≥ 11" behaviour); the exception is wd-3.0 RNN, which +outputs 611. But 960 sits in the *training* split (699 train / 300 val, seed 0), so 967 is +memorization of the 960→967 pair, not sieve transfer — a true k=10 sieve would output 961 (it +cannot see 31|961). These models get 960 right *because they memorized it* while simultaneously +failing 95–100% of out-of-range inputs. The discriminator is only meaningful once a cell is +already P5/P6; no cell is. + +**In-range reference.** The old [101, 200] probe — which is *in-range* for E6 — scores +0.90–0.95 at low wd (falling to 0.44–0.73 at wd 1.0, 0.01–0.41 at wd 3.0). This is the same +pattern phases 1–3 saw *on their out-of-range probe*: the model generalizes within its training +distribution but transfers nothing beyond it. It is not a positive result, just a reminder that +the correct E6 probe is [1001, 2000]. --- -## H-codes — halting still collapsed / noisy +## H-codes — halting still collapsed / noisy, and the batch-128 cell is a wrinkle **LOCKED** (Addenda 1–4): H1 = mean steps at the min_steps floor (2); H2 = pinned at K; H3 = intermediate + positive gap correlation; H4 = intermediate but no correlation. **Commentary.** The three low-wd RNN cells are **H1** (mean 2.00–2.03, collapsed to the -min_steps floor), and the wd-1.0 RNN is **H4** (mean 2.96, noisy). Same as every prior phase: -the ACT gate never learned a computation budget. K = 32 here (vs 20 in phases 1–3) makes no -difference — the gate still bottoms out at the floor or hovers unstructured. +min_steps floor). The wd-1.0 RNN is seed-unstable: seed 0 **H4** (mean 2.96), seed 1 **H1** +(mean 2.38), seed 2 **H4** (mean 2.98) — the ACT gate has no stable structure at wd 1.0. The +wd-3.0 RNN is **H4** (mean 2.86, but the model is so collapsed the reading is moot). The +batch-128 RNN is the one wrinkle: **H4 with mean 7.70 steps** — the larger batch *prevented* +the usual collapse-to-floor (every other RNN bottoms out at 2–3), but there is still no positive +gap correlation (ρ = −0.05), so the gate learned to run ~7 steps unconditionally rather than to +budget compute by gap size. Same as every prior phase: the ACT gate never learned a computation +budget. K = 32 here (vs 20 in phases 1–3) makes no difference. --- @@ -123,44 +136,49 @@ difference — the gate still bottoms out at the floor or hovers unstructured. **LOCKED** (Addendum 6): "an O1 anywhere = data pressure unlocked grokking; same codes = the task's walls are algorithmic, not data-bound." -**Commentary.** No O1 anywhere, and out-of-range transfer is P4 in all 8 cells — so the +**Commentary.** No O1 anywhere, and out-of-range transfer is P4 in all 16 cells — so the "data pressure unlocked grokking" branch does **not** fire. The outcome *class* is unchanged: -no grokking, zero out-of-range transfer, no divisibility algorithm. The one measurable shift is -**in-range only**: the tied RNN, which was O4 everywhere at [2, 100] (val EM best ≤ 36.7%), is -now O-PARTIAL at wd 0.01/0.1/0.3 (val EM best 0.78–0.83, final 0.77–0.81) — i.e. at 699 -training examples the RNN finally holds an in-range memorized/heuristic solution it could not -hold at 69. That is a real, reportable measurement, but it is not grokking and it does not -transfer. Reading the locked clause strictly, the headline is the algorithmic-bound branch: -**more data eased in-range fitting but did not move the task's walls — the search/increment -loop and divisibility test remain unlearnable as transferable structure under these dynamics.** - -**Caveats.** The clause is binary ("O1 anywhere" vs "same codes"); the measured pattern is -"no O1, and the in-range RNN O-code improved one notch". We report the O-code shift as a -measurement rather than forcing it into either binary branch. Seed 0 only for the wd sweep; -the wd-1.0 seeds {1,2} (running) will tell whether the in-range RNN improvement is seed-stable. +no grokking, zero out-of-range transfer, no divisibility algorithm. The measurable shifts are +**in-range only**: + +1. The tied RNN, which was O4 everywhere at [2, 100] (val EM best ≤ 36.7%), is now O-PARTIAL at + wd 0.01/0.1/0.3 (val EM best 0.78–0.83) — at 699 training examples it finally holds an + in-range memorized/heuristic solution it could not hold at 69. This is seed-0 only (the wd + sweep has no replication), and it is wd-dependent (still O4 at wd 1.0). The wd-1.0 + replication seeds {1,2} confirm the *wd-1.0* behaviour is seed-stable (O4 in all three seeds, + both models) but say nothing about the low-wd O-PARTIAL result. +2. The batch-size ablation is a null: batch 128 does not rescue transfer (still P4, acc 0.0–0.1%) + and, in the transformer, trades the usual gentle late-run rollover for a hard final-step + divergence (train 1.0 → 0.55, val 0.83 → 0.45). + +Reading the locked clause strictly, the headline is the algorithmic-bound branch: **more data +eased in-range fitting but did not move the task's walls — the search/increment loop and +divisibility test remain unlearnable as transferable structure under these dynamics.** The +clause is binary ("O1 anywhere" vs "same codes"); the measured pattern is "no O1, and the +in-range RNN O-code improved one notch (seed 0, low wd only)". We report the O-code shift as a +measurement rather than forcing it into either binary branch. --- ## Caveats / known gaps -1. **eval.py still hardcodes the [101, 200] probe.** Addendum 6 moves the E6 probe to - [1001, 2000] and adds the P5(k)/P6 ladder, but `src/eval.py::probe_report` still defaults to - `lo=101, hi=200` and only emits P1–P4. The [1001, 2000] numbers above were computed by a - standalone probe (load `last.pt` on CPU, greedy-decode over [1001, 2000]). `eval.py` should - be taught the E6 probe range + ladder before the remaining 8 cells are classified, otherwise - their committed `results.json` will again carry the in-range [101, 200] code under the P - field (harmless for O/H, misleading for P). +1. **eval.py still hardcodes the [101, 200] probe.** `src/eval.py::probe_report` still defaults + to `lo=101, hi=200` and only emits P1–P4, so the committed `results.json` P-field and + `summary.csv` `probe` column carry the *in-range* [101, 200] code (e.g. P3 at 0.94), which is + misleading for E6. The [1001, 2000] numbers above are from a standalone CPU probe. `eval.py` + should be taught the E6 probe range + P5(k)/P6 ladder before E7 so its committed results are + self-consistent — this gap is now the only open tooling item. 2. **`--post-only` cannot run on this host for E6.** The checkpoints are `device=cuda`; eval's - `_load` honours `cfg.device='cuda'` and cannot run on the CPU-only voidlaptop. Re-scoring must - happen on ichi (which the sweep already does automatically after each cell trains — the 8 - finished cells were eval'd there). The `runs/e6/summary.csv` will therefore not exist until - the sweep on ichi completes and is rsync'd back. -3. **fp16 numerics.** E6 ran with AMP fp16 on CUDA (a deliberate, recorded change vs the CPU - fp32 baseline). All 8 cells sit far from any O/P/H boundary (e.g. probe acc 0.4–4.9% vs the - 90% P3 line; val EM 0.77–0.81 vs the 0.9 O1 line), so the precision caveat in Addendum 6 does - not bind for any finished cell. + `_load` honours `cfg.device='cuda'` and cannot run on the CPU-only voidlaptop. Re-scoring on + voidlaptop loads `last.pt` with `map_location='cpu'` directly (as done here). The + `runs/e6/summary.csv` was produced by the sweep on ichi and rsync'd back. +3. **fp16 numerics.** E6 ran with AMP fp16 on CUDA. No cell sits within ±2 points of an O/P/H + boundary (probe acc 0–4.9% vs the 90% P3 line; val EM far from the 0.9 O1 line; halt means + far from the 2.5 / 31.5 H1/H2 cutoffs), so the precision caveat in Addendum 6 does not bind + for any cell. 4. **960 is in the training split**, so the 960→967 result is memorization, not sieve evidence - (see P-codes section). + (see P-codes section). The wd-3.0 RNN's 960→611 is a collapsed model emitting a composite, + not a k=10 sieve (which would emit 961). --- @@ -169,7 +187,8 @@ the wd-1.0 seeds {1,2} (running) will tell whether the in-range RNN improvement Long-horizon test: **20× budget (4M steps)** on the best E6 cells, per jayrup's decision. The candidate cells are the low-wd O-PARTIAL cells that hold the strongest in-range solution — e6-wd01 / e6-wd03 (rnn and transformer, val EM best 0.83–0.84, final 0.79–0.81) — with the -wd-1.0 control for parity. Protocol locked in Addendum 7 post-E6. The E6 read so far gives no -reason to expect the 20× horizon to unlock transfer (the wall is algorithmic, not a data or -budget shortfall at the margins tested), but the long horizon is exactly the test that separates -"slow algorithmic basin" from "no basin at all". +wd-1.0 control for parity. Protocol locked in Addendum 7 post-E6. The full 16-cell E6 read gives +no reason to expect the 20× horizon to unlock transfer (the wall is algorithmic, not a data or +budget shortfall at the margins tested — batch size and wd and data volume all failed to move +it), but the long horizon is exactly the test that separates "slow algorithmic basin" from +"no basin at all". |