diff options
| author | Void Agent <void@jayrup.hermes> | 2026-08-02 15:23:54 +0100 |
|---|---|---|
| committer | Void Agent <void@jayrup.hermes> | 2026-08-02 15:23:54 +0100 |
| commit | 5461d3beca4f7bc8aa2feb7d1320056b18e69656 (patch) | |
| tree | 88cdb23f30a778ec7c24897456cf302edd3fe7de | |
| parent | 781a07b0c52c62657a8030eec8bbab76d26fa90f (diff) | |
Loss-reweighting results: q/control 0.944 (CI crosses 1), q/ctrl_random 0.985 — small non-significant causal demotion; freq corr invariant (r~-0.66) across all 9 models; blog section 8 + results.md section 3 filled honestly
| -rw-r--r-- | docs/blog-jlens-frequency.md | 55 | ||||
| -rw-r--r-- | results.md | 30 |
2 files changed, 72 insertions, 13 deletions
diff --git a/docs/blog-jlens-frequency.md b/docs/blog-jlens-frequency.md index 7e01de0..ceb97d5 100644 --- a/docs/blog-jlens-frequency.md +++ b/docs/blog-jlens-frequency.md @@ -249,16 +249,51 @@ running a clean-boundary control (noise token inserted after random word boundaries — still unpredictable, no word-slicing) to rule it out; the numbers above should be read with that caveat until the control lands. -## 8. The causal test (in progress) - -We are currently running the last experiment: train three models per seed, -identical in every way, except one model gives the letter 'q' twice the -learning pressure (2x loss weight on 'q' targets — increasing its effective -frequency without corrupting the text), a control model with normal loss, and a -second control that upweights the same number of random *other* letters. If -doubling 'q's effective frequency causally shrinks its J-lens norm below both -controls, the frequency story is causal, not just correlational. Results land -within hours; this post will be updated. +## 8. The causal test: what actually happened + +The last experiment was the one designed to make the frequency story causal. +Train three models per seed from the *identical* starting weights and the +*identical* minibatch order — the only difference is the loss: one model gives +the letter 'q' twice the learning pressure (2x CE weight on 'q' targets, which +raises its effective frequency without corrupting the text), one is a plain +control, and one upweights the same number of random *other* letters (to check +that "any reweighting" isn't the thing doing the work). Three seeds, three +models each. If doubling 'q's effective frequency causally shrinks its J-lens +norm below both controls, the frequency story is causal, not just +correlational. The faithful lens norm of 'q' (layers 2-4, mean per seed): + +``` + seed | q(2x) control ctrl_random | q/control q/ctrl_random + 0 | 0.0163 0.0174 0.0152 | 0.934 1.069 + 1 | 0.0150 0.0171 0.0161 | 0.881 0.932 + 2 | 0.0161 0.0158 0.0169 | 1.019 0.952 +``` + +Cross-seed: q/control mean = 0.944 (bootstrap 95% CI [0.881, 1.019]), +q/ctrl_random mean = 0.985 (CI [0.932, 1.069]). + +What this shows, honestly: +1. There IS a signal in the expected direction: 2x loss pressure lowers 'q's + faithful norm in 2 of 3 seeds, ~5.6% on average below the plain control. +2. It is small and noisy. The CI crosses 1.0, one seed goes the other way, + and against the random-upweight control the effect essentially vanishes + (0.985). With this power we cannot claim a robust causal demotion from + loss reweighting. +3. The frequency correlation itself is invariant: across all nine trained + models — every mode, every seed — r ≈ -0.63 to -0.69. Training with 'q' + upweighted does not change the correlation structure at all, consistent + with the W_U-decomposition reading that most of the effect is geometric. +4. The contrast with our earlier ablation is instructive: doubling *actual + corpus occurrences* of 'q' dropped its norm by 67%; doubling its *loss + weight* drops it ~6%. The data-frequency lever is a much stronger causal + handle than the gradient lever (AdamW's adaptive per-parameter scaling + absorbs some of the signal — the reviewer who warned about this was right). + +Net: the frequency confound is strongly correlational and geometrically +stable; the causal lever we could afford to test is weak. This is the honest +state of the causal evidence. (Absolute 'q' norms differ across experiments — +base model 0.011 vs these 0.015-0.018 — so only within-experiment +comparisons are meaningful.) ## 9. What we are NOT saying @@ -124,9 +124,33 @@ read with that caveat until it lands. Three models per seed, identical init + minibatch order: 'q' targets weighted x2 in the loss, plain control, and a same-total-loss control upweighting random non-'q' targets. Question: does raising effective frequency causally reduce -'q's faithful J-lens norm? PENDING — run completes within hours of this file -being written; the summary table is printed by -`python3 src/loss_reweight.py --step summary --layers 2,3,4`. +'q's faithful J-lens norm? Faithful lens norm of 'q' (mean over layers 2-4): + +``` + seed | q(2x) control ctrl_random | q/control q/ctrl_random + 0 | 0.0163 0.0174 0.0152 | 0.934 1.069 + 1 | 0.0150 0.0171 0.0161 | 0.881 0.932 + 2 | 0.0161 0.0158 0.0169 | 1.019 0.952 +``` + +Cross-seed: q/control mean = 0.944 (bootstrap 95% CI [0.881, 1.019]), +q/ctrl_random mean = 0.985 (CI [0.932, 1.069]). + +Reading: +1. Expected-direction signal, small: 2x loss pressure lowers 'q' norm in 2/3 + seeds (~5.6% mean vs control), but CI crosses 1.0 and the effect vanishes + vs the random-upweight control (0.985). Not a robust causal demotion at + this power. +2. Frequency correlation is invariant across all 9 models: mean r = -0.66 + (min -0.685, max -0.626) in every mode/seed/layer — training with 'q' + upweighted does not change the correlation structure (geometric, per the + W_U decomposition). +3. Contrast: doubling ACTUAL corpus occurrences of 'q' (original ablation) + dropped its norm 67%; doubling its LOSS weight drops it ~6%. Data-frequency + is the strong causal lever; gradient reweighting under AdamW is weak + (adaptive per-param scaling absorbs part of the signal). +4. Absolute norms differ across experiments (base L3 q = 0.0109 vs these + 0.015-0.018) — only within-experiment comparisons are meaningful. ## 4. Historical / do-not-copy |
