at-scale W_U row-norm check (GPT-2): geometric confound generalizes, r(log10)≈-0.45/-0.49; blog draft: drop 'Actually', log-freq lead, probe transparency, norm-vs-readout concession

1 files changed, 53 insertions, 14 deletions

diff --git a/docs/blog-jlens-frequency.md b/docs/blog-jlens-frequency.md
index 0d51c8d..f5a459e 100644
--- a/docs/blog-jlens-frequency.md
+++ b/docs/blog-jlens-frequency.md
@@ -1,4 +1,4 @@
-# What the Jacobian Lens Actually Measures
+# What the Jacobian Lens Measures
 ### A small replication of Anthropic's J-lens, the token-frequency confound we found, and the bug we almost published
 
 *This is a story about trying to look inside a language model. We found something
@@ -11,13 +11,17 @@ stronger result.*
 **The short version.** We reimplemented Anthropic's Jacobian lens faithfully
 (verified against their released code) and found that the J-space ranking is
 strongly confounded by token frequency: rare tokens score high, common tokens
-score low (r ≈ -0.6 to -0.7 at every layer, p ~ 10^-9 or less). Anthropic never
+score low (r ≈ -0.6 to -0.7 at every layer, p ~ 10^-9 or less; against
+log-frequency — the natural scale for Zipfian data — the unembedding geometry
+alone hits r = -0.69). Anthropic never
 controls for frequency — not in the paper, not in the released code. Digging
 into *why* gave us the most interesting result: the frequency signal splits
 into two separable parts. Half lives in the static geometry of the model's
 word-scoring matrix — baked into the lens by definition, so any user inherits
 it. A smaller, layer-dependent part lives in what the layers themselves do, and
-vanishes at the final layer. A frequency-matched synthetic pair shows the lens
+vanishes at the final layer. The static half survives at real scale: GPT-2's
+unembedding rows anti-correlate with token log-frequency too (r ≈ -0.45/-0.49,
+V = 50,257, n = 46,887). A frequency-matched synthetic pair shows the lens
 also carries genuine structure signal (confirmed by a clean-boundary control at
 ~1.3x), and a causal test found the demotion effect is small under loss
 reweighting. We are **not** claiming the J-space doesn't exist. We're claiming
@@ -193,7 +197,9 @@ think, the actual novel mechanistic contribution of this project: **the
 frequency confound is not one thing.**
 
 1. **Static geometry — the ruler.** The unembedding row norms ||W_U[k]||
-   themselves anti-correlate with frequency (r = -0.61, Spearman -0.81).
+   themselves anti-correlate with frequency — most strongly against
+   log-frequency, the natural scale for Zipfian data: r(||W_U[k]||, log10 f) =
+   -0.69 (raw frequency -0.61, Spearman -0.81).
    Rare tokens get bigger rows in the word-scoring matrix. Since the lens
    reads through W_U by definition, any user of the lens — including
    Anthropic's capacity analysis — inherits this bias automatically. A
@@ -365,31 +371,64 @@ Let us be very careful here, because it would be easy to overclaim.
   subtract mean concept directions. Those are real experimental controls —
   but none of them is a token-frequency control, which is the specific thing
   our claim is about.
-- We are **not** saying the lens is useless. The synthetic-pair result shows it
+- We are **not** claiming the lens is useless. The synthetic-pair result shows it
   carries real structure signal.
+- We are **not** claiming that ranking by lens-vector *norm* is the same as
+  ranking by *lens output on real activations*. Our numbers rank tokens by the
+  norm of their faithful J-lens vector — a summary of the readout geometry —
+  not by how strongly, or how often, those directions actually fire in running
+  text. Anthropic's capacity claim is about the latter (occupancy). The norm
+  still matters: it is the size of a token's lever on the readout, and the W_U
+  geometry that drives it enters any readout-based analysis by definition. But
+  the gap between "geometry is frequency-confounded" and "the capacity claim
+  is frequency-confounded" is real, and it is the specific gap an at-scale
+  occupancy test has to close.
 - We are **not** saying "it's just linear algebra." Our toy models don't show
   the compression Anthropic sees in large models; that's a limitation of toy
   models, not evidence against large ones.
 
 What we **are** saying is narrower and, we think, more durable: on the paper's
-own measurement, J-lens *rankings* are strongly confounded by token frequency
-at every scale we can test, and frequency is a variable any J-lens analysis
-should control for. Whether the confound survives at Anthropic's scale is an
-empirical question — one we are taking to bigger models next.
+own measurement, J-lens *norm-rankings* are strongly confounded by token
+frequency at every scale we can test, and frequency is a variable any J-lens
+analysis should control for. Whether the confound survives at Anthropic's scale
+is an empirical question — one we are taking to bigger models next, and one we
+already have a first, partial answer to for the geometric half (Section 10).
 
 ## 10. What's next
 
+We did try bigger once already, and we owe you the number, because a reader
+who opens the repo will find it either way: an early probe on GPT-2 small
+(`src/gpt2_jlens.py`) returned an average correlation of only r ≈ -0.18 across
+layers. We do not count it as evidence, for three concrete reasons: it sampled
+96 token positions out of a 50,257-token vocabulary; it averaged over only 100
+sampled tokens per batch; and it measured a subtly different quantity
+(norm-per-batch rather than norm-of-the-mean). It was a directional probe, and
+it pointed weak. It is logged in `results.md`, flagged do-not-cite — but a
+post that promises "bigger models next" should not pretend the attempt never
+happened.
+
 Toy scale answers the methodological question. Scale answers the real one. We
 want to run the faithful lens on a real language model (V = 50K, d = 768 — the
 regime where Anthropic's claims live) with proper statistical power, and to run
 the occupancy test their capacity claim is actually about. That's the next
 post.
 
-The learned-geometry finding makes one piece of that cheap and decisive: if
-the W_U row-norm anti-correlation is a general property of softmax-output
-models trained on Zipfian data, it should appear in GPT-2's unembedding matrix
-directly — no Jacobian computation at all. One line of code at scale, and it
-either generalizes or it doesn't.
+The learned-geometry finding makes one piece of that cheaper than the probe
+was: if the W_U row-norm anti-correlation is a general property of
+softmax-output models trained on Zipfian data, it should appear in GPT-2's
+unembedding matrix directly — no Jacobian computation at all. So we ran it:
+GPT-2's unembedding row norms correlate with token log-frequency at V = 50,257
+(r ≈ -0.45 on gpt2-small, -0.49 on gpt2-medium, n = 46,887 tokens seen in
+wikitext-103; Spearman -0.46 to -0.50). The decile picture is monotone in both
+models — the rarest tenth of tokens have the biggest rows — and the top-norm
+tokens are rare ones (SPONSORED, soDeliveryDate, Reviewer). The geometric half
+of the confound is not a 65-character vocabulary artifact: it is present in
+two real, independently trained language models at Anthropic's vocabulary
+scale. One honest caveat: it is weaker than at toy scale (-0.45/-0.49 vs
+-0.69), so the char-scale result was not a size artifact, but neither is the
+at-scale effect as strong as the toy suggested. The layer-dynamics half —
+what the layers themselves add beyond the ruler — still needs the faithful
+lens at scale, and that remains the next post.
 
 ## 11. How to reproduce everything