diff options
| -rw-r--r-- | docs/blog-jlens-frequency.md | 29 | ||||
| -rw-r--r-- | results.md | 23 |
2 files changed, 37 insertions, 15 deletions
diff --git a/docs/blog-jlens-frequency.md b/docs/blog-jlens-frequency.md index 3d68759..23529de 100644 --- a/docs/blog-jlens-frequency.md +++ b/docs/blog-jlens-frequency.md @@ -267,18 +267,31 @@ The predictable token scores **~1.4-1.5x higher** than the noise token at identical frequency, in every layer of every seed. Middle-layer ratio across the three seeds: 1.47 ± 0.09, bootstrap 95% CI [1.37, 1.53] — entirely above 1. So the lens is not a pure frequency meter: at equal frequency, the two -tokens differ in norm. Whether that difference is specifically *conditional -predictability* (what "verbalizable" should mean) depends on a control that is -still running — see the caveat below. +tokens differ in norm. One caveat, found by a reviewer: the noise token '#' was inserted at random character positions, which slices through the middle of a word 58% of the time (th#e, ki#ng — letter on both sides), while '@' always sits at a clean word -boundary after "the ". Predictability is therefore not perfectly isolated from -n-gram corruption. We are running a clean-boundary control (noise token -inserted after random word boundaries — 0% word-slicing, still unpredictable) -to rule it out; the numbers above should be read with that caveat until the -control lands. +boundary after "the ". That confounds predictability with n-gram corruption — +so we ran the control that isolates them: '#' inserted at random *word +boundaries* (0% word-slicing, still unpredictable), same 0.0998% frequency, +three fresh seeds. + +The control is done, and it is the honest kind of result — partly confirming, +partly correcting: + +``` + placement of '#' middle-layer ratio @/# bootstrap 95% CI + random (58% slicing) 1.47 ± 0.09 [1.37, 1.53] + clean boundary (0%) 1.31 ± 0.08 [1.26, 1.40] +``` + +The corruption confound was real: it inflated the estimate by about 12%. But +it was not the whole story. At identical frequency, with clean boundaries and +nothing sliced, the predictable token still scores ~1.3x higher than the +unpredictable one, and the CI stays entirely above 1 in every seed. The +conditional-predictability signal — the thing "verbalizable" should mean — +survives the control, modestly smaller than our first estimate. ## 8. The causal test: what actually happened @@ -126,13 +126,22 @@ Middle-layer ratio across seeds: 1.47 +/- 0.09 (SD), bootstrap 95% CI [1.37, 1.53]. The frequency anti-correlation holds, but the lens also carries genuine conditional-predictability signal. -CAVEAT (from adversarial review): '#' was inserted at uniform random character -positions, which slices through the middle of a word 58% of the time (letter -on both sides: th#e, ki#ng); '@' always sits at a clean word boundary after -"the ". Predictability is therefore not perfectly isolated from n-gram -corruption. A clean-boundary control (noise token after random word -boundaries, 0% word-slicing) is running; the numbers above should be read with -that caveat until it lands. +CAVEAT + CONTROL (resolved): '#' was originally inserted at uniform random +character positions, which slices through the middle of a word 58% of the time +(letter on both sides: th#e, ki#ng); '@' always sits at a clean word boundary +after "the ". A clean-boundary control (noise token after random word +boundaries, 0% word-slicing, same frequency) was run with three fresh seeds: + +``` + placement of '#' middle-layer ratio @/# bootstrap 95% CI + random (58% slicing) 1.471 +/- 0.090 [1.368, 1.529] + clean boundary (0%) 1.311 +/- 0.079 [1.258, 1.402] +``` + +Reading: the corruption confound was real (inflated the ratio by ~12%) but not +the whole story — the structure signal survives at clean boundaries, CI +entirely above 1 in every seed. Original run: outputs/synth_pair/seed{s}/; +control run: outputs/synth_pair_clean/seed{s}/. ## 3. Loss-reweighting causal test (`src/loss_reweight.py`) |