diff options
Diffstat (limited to 'results.md')
| -rw-r--r-- | results.md | 49 |
1 files changed, 39 insertions, 10 deletions
@@ -10,18 +10,39 @@ identity, and the check returns cosine similarity **1.0000**. Pearson r between token frequency and J-lens norm, per layer. n = 65 tokens (full char vocab). The correlation survives the faithful implementation at -every layer. +every layer. p-values: all layers p ~ 10^-9 to 10^-13. Spearman rank +correlation is even stronger: faithful -0.69 to -0.85; proxy -0.63 to -0.81 +(not a leverage artifact of extreme common tokens). ``` - Layer proxy r faithful r - L0 -0.661 -0.643 - L1 -0.673 -0.668 - L2 -0.653 -0.672 - L3 -0.648 -0.685 - L4 -0.562 -0.637 - L5 -0.665 -0.606 + Layer proxy r faithful r Spearman(faith) partial r(faith, freq | ||W_U||) + L0 -0.661 -0.643 -0.844 -0.307 + L1 -0.673 -0.668 -0.851 -0.319 + L2 -0.653 -0.672 -0.825 -0.338 + L3 -0.648 -0.685 -0.769 -0.364 + L4 -0.562 -0.637 -0.716 -0.240 + L5 -0.665 -0.606 -0.808 +0.056 ``` +### W_U decomposition (where does the correlation come from?) + +The faithful lens vector for token k is `W_U[k] * J_l`. The unembedding row +norms themselves anti-correlate with frequency: + +``` + r(||W_U[k]||, freq) = -0.606 + r(||W_U[k]||, log10 freq) = -0.693 + Spearman(||W_U[k]||, freq) = -0.808 +``` + +`r(faithful_norm, ||W_U[k]||)` per layer: +0.70 to +1.00 (L5 = +1.00 exactly, +since J = identity at the last layer). After regressing out the W_U row-norm +component, a layer-dependent anti-correlation survives in layers 0-4 (partial +r ≈ -0.24 to -0.36) and vanishes at the last layer (+0.06). Interpretation: +the frequency signal lives partly in the static unembedding geometry (which is +baked into Anthropic's lens by definition) and partly in layer-dependent +dynamics; the mechanism of the latter is not yet pinned down. + Top tokens by faithful norm are consistently rare characters (`?`, `z`, `q`, `$`); bottom are common ones (space, `e`, `t`, `i`). @@ -40,8 +61,16 @@ over layers 0-5: ``` Reading: at equal frequency, the structured token scores ~1.4-1.5x higher. -The frequency anti-correlation holds, but the lens also carries genuine -conditional-predictability signal. +Middle-layer ratio across seeds: 1.47 +/- 0.09 (SD), bootstrap 95% CI +[1.37, 1.53]. The frequency anti-correlation holds, but the lens also carries +genuine conditional-predictability signal. + +CAVEAT (from adversarial review): '#' was inserted at uniform random character +positions, which slices inside words ~95% of the time (th#e, ki#ng); '@' +always sits at a clean word boundary after "the ". Predictability is therefore +not perfectly isolated from n-gram corruption. A clean-boundary control (noise +token after random word boundaries) is planned; the numbers above should be +read with that caveat until it lands. ## 3. Loss-reweighting causal test (`src/loss_reweight.py`) |