diff options
Diffstat (limited to 'docs/blog-jlens-frequency.md')
| -rw-r--r-- | docs/blog-jlens-frequency.md | 79 |
1 files changed, 46 insertions, 33 deletions
diff --git a/docs/blog-jlens-frequency.md b/docs/blog-jlens-frequency.md index 23529de..dec1edf 100644 --- a/docs/blog-jlens-frequency.md +++ b/docs/blog-jlens-frequency.md @@ -109,9 +109,9 @@ J-lens norms, common characters had small ones (r ≈ -0.65). We were excited. We were also wrong. Before publishing anything, we did something slightly unusual: we asked three -large independent AI models to try to tear the work apart — Gemini 3.1 Pro, -Claude Opus 4.6, and GPT-5.6. We gave them our code and our results and asked -them to find the flaws. All three, independently, found the same one: +independent AI reviewers to try to tear the work apart. We gave them our code +and our results and asked them to find the flaws. All three, independently, +found the same one: **Our implementation was not computing Anthropic's Jacobian lens.** @@ -153,12 +153,13 @@ with their exact estimator choices: the frequency correlation is essentially identical at every layer (max delta 0.008, see results.md section 1b), so the result is robust to those choices.) -(One technical note: we capture the residual stream *before* the model's final +One technical note before moving on, because it matters for the +interpretation: we capture the residual stream *before* the model's final layer norm. That matches the paper's definition — the Jacobian stops at the final residual stream and the J-lens vectors are the rows of W_U·J_ℓ, with normalization applied only when *reading* the lens, i.e. softmax(W_U·norm(J_ℓ·h_ℓ)). Under that definition the last-layer identity -check is exact by construction.) +check is exact by construction. ## 6. What we found: frequency is everywhere @@ -197,6 +198,13 @@ frequency confound is not one thing.** reads through W_U by definition, any user of the lens — including Anthropic's capacity analysis — inherits this bias automatically. A frequency control would have to live in the geometry, not in the prompts. + + Where does that geometry come from? Not from initialization: fresh models + show no frequency correlation in their row norms (r ≈ +0.00 to -0.19 + across three seeds). After training it is -0.61. The model *learns* to + push rare tokens' rows outward as it learns the corpus, and nothing in the + objective or the readout ever corrects it. The lens inherits learned + geometry, not a fixed one. 2. **Layer dynamics — the layers.** Regress out the W_U component and an anti-correlation still survives in layers 0-4 (partial r ≈ -0.24 to -0.36) — something about what the layers themselves do keeps boosting @@ -210,17 +218,16 @@ would most want someone to test at scale. A fact-check before we go further. We were about to claim "Anthropic does not control for frequency anywhere," and that is the kind of claim that should be checked, not asserted. We checked it four ways: our own scan of the paper's -text, two independent adversarial reviewers (Gemini 3.6 Flash and GPT-5.6 -Luna) who read the full paper including the appendix, and — after a reader -pointed us to it — Anthropic's own released companion code -(`github.com/anthropics/jacobian-lens`, Apache-2.0). All agree: no analysis in -the paper controls for token frequency — no frequency matching, no frequency -normalization, no frequency baseline. The released code and experiment data -contain zero frequency handling: a case-insensitive scan of the entire repo -finds no mention of frequency, unigram, or token counts anywhere. The one -related detail is an appendix note about a separate baseline method (the -"template lens"), where they filter "high-frequency noise tokens" and -explicitly call that "not a principled approach." To be precise: that note +text, two independent adversarial reviewers who read the full paper including +the appendix, and — after a reader pointed us to it — Anthropic's own released +companion code (`github.com/anthropics/jacobian-lens`, Apache-2.0). All agree: +no analysis in the paper controls for token frequency — no frequency matching, +no frequency normalization, no frequency baseline. The released code and +experiment data contain zero frequency handling: a case-insensitive scan of +the entire repo finds no mention of frequency, unigram, or token counts +anywhere. The one related detail is an appendix note about a separate baseline +method (the "template lens"), where they filter "high-frequency noise tokens" +and explicitly call that "not a principled approach." To be precise: that note concerns the template lens, not the main J-lens — it is not evidence that they observed this confound in the J-lens itself. What we can say, auditably, is: the paper's analyses include no frequency control, its released code has none @@ -228,19 +235,19 @@ either, and its one acknowledgment of high-frequency-token trouble was in a separate method they chose not to use. Any "privileged subspace" interpretation needs a frequency control first. -We are also not alone in noticing the raw lens is distorted by token -statistics. An independent research-engineer analysis of the same paper -(willkn, "Anthropic's J-Lens: A Research Engineer's Analysis", GreaterWrong, -24 Jul 2026), working on GPT-2-medium (355M — thirty times our model), found -that the raw fitted Jacobian "misweights structural tokens (grammar, -punctuation) over semantic content" — its dominant spectral channels carry -~10x the gain of the residual pathway. Structural tokens are the high-frequency -tokens. They also found the Jacobian essentially full-rank (562-858 dimensions -for 90% of spectral variance), matching our toy-scale rank result. Their fix is -a single-parameter shrinkage regularizer (J + λI) that restores next-token -faithfulness — and Anthropic's released fitting code applies no such -regularization. Whether shrinkage also removes the frequency correlation is an -experiment we have not run yet; it is a natural next step. +**Related work.** We are not alone in noticing the raw lens is distorted by +token statistics. An independent research-engineer analysis of the same paper +(willkn, GreaterWrong, 24 Jul 2026), working on GPT-2-medium (355M — thirty +times our model), found that the raw fitted Jacobian "misweights structural +tokens (grammar, punctuation) over semantic content" — dominant spectral +channels carry ~10x the residual pathway's gain, and structural tokens are the +high-frequency tokens. They also found the Jacobian essentially full-rank +(562-858 dimensions for 90% of spectral variance), matching our toy-scale rank +result. Their fix — a single-parameter shrinkage regularizer (J + λI) that +restores next-token faithfulness — is worth testing against our frequency +correlation; Anthropic's released fitting code applies no such regularization. +Whether shrinkage removes the correlation is an experiment we have not run +yet; it is a natural next step. ## 7. But not *only* frequency @@ -262,6 +269,8 @@ training runs showed: 1 0.0224 - 0.0237 0.0148 - 0.0151 1.50 - 1.60 2 0.0215 - 0.0233 0.0154 - 0.0163 1.35 - 1.51 ``` +*(Norm ranges are across layers 0-5 within each seed; ratios are per-layer +@/#. The middle-layer summary follows.)* The predictable token scores **~1.4-1.5x higher** than the noise token at identical frequency, in every layer of every seed. Middle-layer ratio across @@ -319,10 +328,14 @@ q/ctrl_random mean = 0.985 (CI [0.932, 1.069]). What this shows, honestly: 1. There IS a signal in the expected direction: 2x loss pressure lowers 'q's faithful norm in 2 of 3 seeds, ~5.6% on average below the plain control. -2. It is small and noisy. The CI crosses 1.0, one seed goes the other way, - and against the random-upweight control the effect essentially vanishes - (0.985). With this power we cannot claim a robust causal demotion from - loss reweighting. +2. It is small — and the sharper truth is that this design was never powered + to see it. Observed sd 0.07 on the ratio against a mean deficit of 0.056: + a two-sided 80%-power test needs ~13 seeds per arm, and we ran 3. The + experiment could answer "is the effect large?" (no — CI crosses 1.0, one + seed goes the other way, and the effect vanishes against the + random-upweight control) but not "is there any effect at all?". The right + version is ~13 seeds per arm, or a manipulation that moves frequency more + than 2x. 3. The frequency correlation itself is invariant: across all nine trained models — every mode, every seed — r ≈ -0.63 to -0.69. Training with 'q' upweighted does not change the correlation structure at all, consistent |