diff options
Diffstat (limited to 'docs/blog-jlens-frequency.md')
| -rw-r--r-- | docs/blog-jlens-frequency.md | 20 |
1 files changed, 18 insertions, 2 deletions
diff --git a/docs/blog-jlens-frequency.md b/docs/blog-jlens-frequency.md index d0f7ffa..7879535 100644 --- a/docs/blog-jlens-frequency.md +++ b/docs/blog-jlens-frequency.md @@ -136,8 +136,19 @@ faithful lens, correlated with token frequency like this: The correlation survived the faithful implementation — slightly *stronger*, if anything. The rare characters ('?', 'z', 'q', '$') sit at the top of the J-space ranking; the common ones (space, 'e', 't', 'i') sit at the bottom. On -the paper's own quantity, the J-lens ranking is frequency-confounded. Anthropic -does not control for this anywhere in their analysis. +the paper's own quantity, the J-lens ranking is frequency-confounded. + +A fact-check before we go further. We were about to claim "Anthropic does not +control for frequency anywhere," and that is the kind of claim that should be +checked, not asserted. We checked it three ways: our own scan of the paper's +text, and two independent adversarial reviewers (Gemini 3.6 Flash and +GPT-5.6 Luna) who read the full paper including the appendix. All three agree: +the word "frequency" appears nowhere in the paper, no analysis matches tokens +on frequency, and no baseline controls for it. The closest thing is an +appendix note about a separate baseline method (the "template lens") where +they filter "high-frequency noise tokens" — and they explicitly call that +"not a principled approach," then never apply it to the main J-lens. They saw +the effect. They didn't fix it. We think the burden of proof is on them. ## 7. But not *only* frequency @@ -186,6 +197,11 @@ Let us be very careful here, because it would be easy to overclaim. Anthropic's actual capacity claim (which is about *occupancy* — how often J-lens directions are used per position — not about the rank of the word vectors). +- We are **not** saying they have no controls at all. Their occupancy + analysis compares against random-direction baselines, and their probes + subtract mean concept directions. Those are real experimental controls — + but none of them is a token-frequency control, which is the specific thing + our claim is about. - We are **not** saying the lens is useless. The synthetic-pair result shows it carries real structure signal. - We are **not** saying "it's just linear algebra." Our toy models don't show |