diff options
Diffstat (limited to 'design')
| -rw-r--r-- | design/preregistration.md | 44 |
1 files changed, 44 insertions, 0 deletions
diff --git a/design/preregistration.md b/design/preregistration.md index 89e4fee..7efa8ad 100644 --- a/design/preregistration.md +++ b/design/preregistration.md @@ -217,3 +217,47 @@ with H4). Equal or worse → the halt gate was not the problem. E3–E5 run after the E1/E2 gate, at jayrup's call. Range scaling ([2, 1000]) is its own addendum when it becomes the active phase. + +--- + +## Addendum 5 (2026-08-16, pre-launch — Phase 3 diagnostics batch) + +Trigger: external model feedback on follow-ups; adopted the integer-token and is-prime +decomposition experiments. Locked before any of these runs. Implementation: `task_mode` +(next_prime | is_prime), `train_frac`, `lr_decay` flags — 41 tests green. + +### D1 — Is-prime diagnostic (`task_mode=is_prime`, digits input, seed 0; wd 1.0 both models + wd 0.1 rnn/transformer) +Binary classification n → {0,1}; output = digit token "1"/"0" + EOS. EM = classification accuracy. +Probe = classification of [101, 200]. P-code ordering adapted for classification: +P4 → **P1** → P3 → P2. (A pure {2,3,5,7} sieve scores ~96% on the probe because most of +[101, 200] is trivially classifiable — that accuracy IS the sieve signature here, not a +"surprising success". P1 fires on flagged-INPUT concentration: errors on {121, 143, 169, 187} +classified prime. Verified against a ground-truth sieve stub: exactly those 4 errors → P1.) +Locked decomposition readings: +- **is_prime groks (O1) while next_prime never did** → the search/increment loop is the wall, + not the divisibility test. +- **is_prime reproduces next_prime's codes (no O1 anywhere)** → the divisibility operation + itself is unlearnable under these dynamics — the strongest negative result available. +- Any other pattern → report codes + measurements, no further claim. + +### D2 — Integer-token diagnostic (`vocab_mode=integers`, next_prime, seed 0; wd 1.0 both models + wd 0.1 rnn) +Readings (locked): +- val EM lifts ≥ +10 points over the digits-mode control at the same wd → place-value + parsing was a real tax on learning. +- within ±10 points → parsing was not the bottleneck; the algorithmic content is the wall. + +### E3 implementation note +Cosine schedule inside the step loop: lr_t = lr_min + 0.5·(lr − lr_min)·(1 + cos(π·step/max)), +lr_min = 0.1·lr → 1e-3 → 1e-4. Interpretation per Addendum 4 E3, unchanged. + +### E4 wd choice +Fixed-K (halting=False) at wd 0.1 (the RNN saturates train there — E2) and wd 1.0 +(control parity). Interpretation per Addendum 4 E4, unchanged. + +### E5 note +`train_frac` ∈ {0.4, 0.5} subsamples the TRAIN split only (val stays the locked 30); +deterministic stream (seed+1000); banker's rounding documented. Runs in the next batch; +interpretation per Addendum 4 E5, unchanged. + +Batch = 11 runs: e4 (2), e3 (2), ints (3), isp (4). 2-way parallel, one thread per child, +same eval cadence, metrics, and early stop as all previous phases. |
