Address Gemini+Opus repo reviews: W_U decomposition, Spearman+p-values, synthetic-pair CIs + corruption caveat, accurate fact-check, softened burden-of-proof, LayerNorm note, reviews caveat

1 files changed, 49 insertions, 16 deletions

diff --git a/docs/blog-jlens-frequency.md b/docs/blog-jlens-frequency.md
index 7879535..d45b183 100644
--- a/docs/blog-jlens-frequency.md
+++ b/docs/blog-jlens-frequency.md
@@ -118,6 +118,13 @@ the Jacobian from a layer to itself is the identity matrix, so the faithful
 J-lens vectors *must* equal the model's word-scoring rows. Our check returned
 cosine similarity 1.0000 — exactly. The ruler is correct.
 
+(One technical note: we capture the residual stream *before* the model's final
+layer norm. That matches the paper's definition — the Jacobian stops at the
+final residual stream and the J-lens vectors are the rows of W_U·J_ℓ, with
+normalization applied only when *reading* the lens, i.e.
+softmax(W_U·norm(J_ℓ·h_ℓ)). Under that definition the last-layer identity
+check is exact by construction.)
+
 ## 6. What we found: frequency is everywhere
 
 On the real trained model, all six layers, both the old (buggy) proxy and the
@@ -136,19 +143,36 @@ faithful lens, correlated with token frequency like this:
 The correlation survived the faithful implementation — slightly *stronger*, if
 anything. The rare characters ('?', 'z', 'q', '$') sit at the top of the
 J-space ranking; the common ones (space, 'e', 't', 'i') sit at the bottom. On
-the paper's own quantity, the J-lens ranking is frequency-confounded.
+the paper's own quantity, the J-lens ranking is frequency-confounded. The
+statistics are not subtle: Pearson r ≈ -0.61 to -0.69 (p ~ 10^-9 to 10^-13),
+and Spearman rank correlation is even stronger (-0.69 to -0.85), so the
+result is not an artifact of a few extreme common tokens.
+
+Where does the correlation come from? This is the question we had to answer
+before trusting the result, and the answer is partly boring and partly
+interesting. The faithful lens vector for token k is W_U[k]·J_ℓ — the row of
+the unembedding matrix times the layer Jacobian. The row norms ||W_U[k]||
+themselves anti-correlate with frequency (r = -0.61, Spearman -0.81). So part
+of the effect lives in the static geometry of the final unembedding — which is
+exactly the quantity Anthropic's lens reads out by definition. But not all of
+it: if we regress out the W_U component, a layer-dependent anti-correlation
+survives in layers 0-4 (partial r ≈ -0.24 to -0.36) and vanishes at the last
+layer (+0.06). The lens carries a frequency signal both from the ruler it
+reads with and from what the layers themselves do. The mechanism of that
+layer-dependent part is something we are still investigating.
 
 A fact-check before we go further. We were about to claim "Anthropic does not
 control for frequency anywhere," and that is the kind of claim that should be
 checked, not asserted. We checked it three ways: our own scan of the paper's
 text, and two independent adversarial reviewers (Gemini 3.6 Flash and
 GPT-5.6 Luna) who read the full paper including the appendix. All three agree:
-the word "frequency" appears nowhere in the paper, no analysis matches tokens
-on frequency, and no baseline controls for it. The closest thing is an
-appendix note about a separate baseline method (the "template lens") where
-they filter "high-frequency noise tokens" — and they explicitly call that
-"not a principled approach," then never apply it to the main J-lens. They saw
-the effect. They didn't fix it. We think the burden of proof is on them.
+no analysis in the paper controls for token frequency — no frequency matching,
+no frequency normalization, no frequency baseline. The only place the word
+shows up is an appendix note about a separate baseline method (the "template
+lens"), where they filter "high-frequency noise tokens" — and they explicitly
+call that "not a principled approach," then never apply it to the main
+J-lens. They saw the effect. They didn't fix it. Any "privileged subspace"
+interpretation needs a frequency control first.
 
 ## 7. But not *only* frequency
 
@@ -172,11 +196,20 @@ training runs showed:
 ```
 
 The predictable token scores **~1.4-1.5x higher** than the noise token at
-identical frequency, in every layer of every seed. So the lens is not a pure
-frequency meter. It genuinely responds to conditional predictability — which,
-honestly, is what "verbalizable" should mean. The J-lens measures *both*:
-a frequency prior that is never subtracted out, and a real structure signal on
-top of it.
+identical frequency, in every layer of every seed. Middle-layer ratio across
+the three seeds: 1.47 ± 0.09, bootstrap 95% CI [1.37, 1.53] — entirely above
+1. So the lens is not a pure frequency meter. It genuinely responds to
+conditional predictability — which, honestly, is what "verbalizable" should
+mean. The J-lens measures *both*: a frequency prior that is never subtracted
+out, and a real structure signal on top of it.
+
+One caveat, found by a reviewer: the noise token '#' was inserted at random
+character positions, which slices *inside* words ~95% of the time (th#e,
+ki#ng), while '@' always sits at a clean word boundary after "the ". That
+means predictability is not perfectly isolated from n-gram corruption. We are
+running a clean-boundary control (noise token inserted after random word
+boundaries — still unpredictable, no word-slicing) to rule it out; the numbers
+above should be read with that caveat until the control lands.
 
 ## 8. The causal test (in progress)
 
@@ -209,10 +242,10 @@ Let us be very careful here, because it would be easy to overclaim.
   models, not evidence against large ones.
 
 What we **are** saying is narrower and, we think, more durable: on the paper's
-own measurement, J-lens *rankings* are strongly confounded by token frequency,
-and any claim about a privileged subspace must control for frequency first.
-Anthropic's paper does not. The burden of proof is on them — and it's a fair
-one.
+own measurement, J-lens *rankings* are strongly confounded by token frequency
+at every scale we can test, and frequency is a variable any J-lens analysis
+should control for. Whether the confound survives at Anthropic's scale is an
+empirical question — one we are taking to bigger models next.
 
 ## 10. What's next