diff options
| -rw-r--r-- | docs/blog-jlens-frequency.md | 53 |
1 files changed, 41 insertions, 12 deletions
diff --git a/docs/blog-jlens-frequency.md b/docs/blog-jlens-frequency.md index ef3395f..3d68759 100644 --- a/docs/blog-jlens-frequency.md +++ b/docs/blog-jlens-frequency.md @@ -8,6 +8,24 @@ stronger result.* --- +**The short version.** We reimplemented Anthropic's Jacobian lens faithfully +(verified against their released code) and found that the J-space ranking is +strongly confounded by token frequency: rare tokens score high, common tokens +score low (r ≈ -0.6 to -0.7 at every layer, p ~ 10^-9 or less). Anthropic never +controls for frequency — not in the paper, not in the released code. Digging +into *why* gave us the most interesting result: the frequency signal splits +into two separable parts. Half lives in the static geometry of the model's +word-scoring matrix — baked into the lens by definition, so any user inherits +it. A smaller, layer-dependent part lives in what the layers themselves do, and +vanishes at the final layer. A frequency-matched synthetic pair shows the lens +also carries genuine structure signal (with a caveat we're resolving), and a +causal test found the demotion effect is small under loss reweighting. We are +**not** claiming the J-space doesn't exist. We're claiming that any +"privileged subspace" interpretation needs a frequency control first. The full +story — numbers, mistakes, and all — is below. + +--- + ## 1. The machine that guesses words A language model is, at its heart, a machine that guesses the next word. Show it @@ -165,18 +183,29 @@ statistics are not subtle: Pearson r ≈ -0.61 to -0.69 (p ~ 10^-9 to 10^-13), and Spearman rank correlation is even stronger (-0.69 to -0.85), so the result is not an artifact of a few extreme common tokens. -Where does the correlation come from? This is the question we had to answer -before trusting the result, and the answer is partly boring and partly -interesting. The faithful lens vector for token k is W_U[k]·J_ℓ — the row of -the unembedding matrix times the layer Jacobian. The row norms ||W_U[k]|| -themselves anti-correlate with frequency (r = -0.61, Spearman -0.81). So part -of the effect lives in the static geometry of the final unembedding — which is -exactly the quantity Anthropic's lens reads out by definition. But not all of -it: if we regress out the W_U component, a layer-dependent anti-correlation -survives in layers 0-4 (partial r ≈ -0.24 to -0.36) and vanishes at the last -layer (+0.06). The lens carries a frequency signal both from the ruler it -reads with and from what the layers themselves do. The mechanism of that -layer-dependent part is something we are still investigating. +### The most interesting thing we found: where the correlation comes from + +The faithful lens vector for token k is W_U[k]·J_ℓ — the row of the +unembedding matrix times the layer Jacobian. The correlation can come from +either factor, and the two behave very differently. This decomposition is, we +think, the actual novel mechanistic contribution of this project: **the +frequency confound is not one thing.** + +1. **Static geometry — the ruler.** The unembedding row norms ||W_U[k]|| + themselves anti-correlate with frequency (r = -0.61, Spearman -0.81). + Rare tokens get bigger rows in the word-scoring matrix. Since the lens + reads through W_U by definition, any user of the lens — including + Anthropic's capacity analysis — inherits this bias automatically. A + frequency control would have to live in the geometry, not in the prompts. +2. **Layer dynamics — the layers.** Regress out the W_U component and an + anti-correlation still survives in layers 0-4 (partial r ≈ -0.24 to + -0.36) — something about what the layers themselves do keeps boosting + rare tokens — and it vanishes at the last layer (+0.06). The mechanism + of that layer-dependent part is still under investigation. + +So the lens carries a frequency signal from both the ruler it reads with and +from what the layers do — and the two are separable. That is the finding we +would most want someone to test at scale. A fact-check before we go further. We were about to claim "Anthropic does not control for frequency anywhere," and that is the kind of claim that should be |