From 6eb47fe35e01dedd42e788d9aa8f199c64962304 Mon Sep 17 00:00:00 2001
From: Void Agent <void@jayrup.hermes>
Date: Fri, 7 Aug 2026 15:13:22 +0100
Subject: blog: add jacobian-lens-frequency post (figures, captions,
 terminal.css sizing)

---
 blog/jacobian-lens-frequency.qmd | 466 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 466 insertions(+)
 create mode 100644 blog/jacobian-lens-frequency.qmd

(limited to 'blog/jacobian-lens-frequency.qmd')

diff --git a/blog/jacobian-lens-frequency.qmd b/blog/jacobian-lens-frequency.qmd
new file mode 100644
index 0000000..6a81809
--- /dev/null
+++ b/blog/jacobian-lens-frequency.qmd
@@ -0,0 +1,466 @@
+---
+title: "What the Jacobian Lens Measures"
+date: "2026-08-07"
+---
+
+*A small replication of Anthropic's J-lens, the token-frequency confound I found, and the bug I almost published.*
+
+*This is a story about trying to look inside a language model. I found something
+Anthropic didn't mention in their paper — and then I found that I'd made a
+mistake, fixed it, and the thing was still there. That second part is the
+stronger result.*
+
+---
+
+**The short version.** I rebuilt Anthropic's Jacobian lens and checked it
+against their released code. In my model, its token ranking has a large and
+simple bias: rare tokens get big scores; common tokens get small ones
+(r ≈ -0.6 to -0.7 at every layer). Anthropic's paper and released code do not
+control for token frequency.
+
+That is not the whole story. I can split the effect in two. Most of it is in
+the model's built-in word-scoring table: training gives rare tokens bigger
+rows there, and the lens necessarily reads through those rows. A smaller part
+comes from the layers themselves. The first effect also appears in GPT-2 at a
+50,257-token vocabulary (r ≈ -0.45/-0.49 against log-frequency).
+
+Nor is the lens *only* measuring frequency. When I gave two invented tokens
+exactly the same frequency, the one the model could predict in context still
+scored about 1.3x higher. So this is not a refutation of J-space. It is a
+more modest claim: before treating a J-lens ranking as evidence for a
+privileged concept workspace, control for frequency first.
+
+---
+
+## 1. The machine that guesses words
+
+A language model is, at its heart, a machine that guesses the next word. Show it
+"the cat sat on the" and it produces a list of probabilities for what comes next:
+"mat" high, "chair" high, "banana" low. Everything it "knows" is wrapped up in
+that guessing.
+
+The interesting question is: *where* does the guessing happen? A modern model
+has dozens of layers, each transforming the sentence a little. Somewhere in
+those layers, the model is deciding that "cat" is an animal, that "sat" is past
+tense, that a location is coming. I would like to watch that happen. The
+problem is that the inside of a transformer is a soup of high-dimensional
+vectors, and no one has a map.
+
+For a long time, people used the "logit lens": at each layer, take the
+representation, and ask "if the model had to guess *right now*, what would it
+guess?" The trouble is that representations change coordinate systems as they
+travel through the layers, so early layers give you nonsense. It's like trying
+to read a letter that's been translated into a language you don't know — at the
+start of the chain, the translation is too rough.
+
+## 2. Anthropic's idea: the Jacobian lens
+
+In 2026, Anthropic published a paper — "Verbalizable Representations Form a
+Global Workspace in Language Models" — introducing a smarter version: the
+*Jacobian lens*. Instead of asking "what would the model guess right now?", it
+asks a sharper question: *"if I nudge this representation a tiny bit, how much
+does the final guess move?"*
+
+That's what a Jacobian is: a table of "how much does each output move when each
+input moves." The lens computes, for every layer, the average nudge-effect of
+that layer's representation on every word in the vocabulary, averaged over a
+thousand different contexts. Words whose representations are strongly "poised"
+to be spoken — ready to be said, should the occasion arise — get big numbers.
+Anthropic calls this collection of word-vectors the **J-space**, and they claim
+it's a kind of "global workspace": a small, privileged subset of the model's
+internal state that can be reported on, modulated, and used for reasoning. They
+even note the resemblance to theories of consciousness, carefully, the way you
+would mention a bear while making clear you are not feeding it.
+
+The headline claim that caught my eye: **the J-space has limited capacity —
+only 10 to 50 concepts are "active" at once.** A tiny privileged workspace
+inside a big model. That's a strong claim. Strong claims deserve strong tests.
+
+## 3. The itch
+
+The moment I read the paper, something felt off. Here's the thing about token
+frequencies: in any language, a handful of words ("the", "of", "and") appear
+all the time, and thousands of words appear almost never. In the model's
+vocabulary of 50,257 tokens, the rarest are nearly invisible.
+
+My first thought was about gradients. A gradient through the model's final
+probability calculation has a built-in quirk: the less likely a word is, the
+larger one of its raw terms can be. That would make rare words look important
+before the model had said anything interesting about them.
+
+That intuition applies directly to the simpler measurement I tried first,
+which differentiated through the final softmax. It does *not* by itself
+explain Anthropic's faithful lens. As I later found, the faithful version has
+a different source of bias: part of it is sitting in the geometry of the
+word-scoring matrix. But it gave me the itch worth checking.
+
+In other words: **a "privileged workspace" might just be a frequency effect
+wearing a fancy hat.**
+
+## 4. My first attempt — and the bug three reviewers found
+
+I set out to test this on a small model I could train myself: a
+10.65-million-parameter character-level transformer (Karpathy's nanoGPT),
+trained on Shakespeare. Small enough to run on a 4GB GPU in a few hours. Big
+enough to have real layers.
+
+My first implementation looked reasonable. I hooked into each layer, computed
+the gradient of log-probability for every character, averaged over contexts,
+and — sure enough — found a strong correlation: rare characters had big
+J-lens norms, common characters had small ones (r ≈ -0.65). I was excited.
+I was also wrong.
+
+Before publishing anything, I did something slightly unusual: I asked three
+independent AI reviewers to try to tear the work apart. I gave them my code
+and results and asked them to find the flaws. All three, independently,
+found the same one:
+
+**My implementation was not computing Anthropic's Jacobian lens.**
+
+Anthropic's lens computes the average Jacobian from a layer to the *final
+representation* — the residual stream — and *then* reads it out through the
+model's word-scoring matrix. My code instead differentiated through the
+softmax directly. That folds a frequency-dependent calibration factor — the
+(1 - p) term — into the thing being averaged. My beautiful correlation might
+have been an artifact of my own measurement.
+
+This is the part of the story I like best, because it's the part that's easy
+to skip: I had built a measurement that *looked* like the paper's and wasn't.
+The reviewers caught it, I fixed it, and the honest result got stronger.
+
+## 5. The right way
+
+I rebuilt the lens to match the paper's definition exactly. In plain English,
+I ask: if I nudge this layer a little, what average change reaches the final
+residual stream? Only after averaging those changes do I use the model's own
+word-scoring table to turn them into token directions. Formally:
+
+> For each layer ℓ, compute the average Jacobian from that layer to the final
+> residual stream, over all source positions, all future positions, and many
+> prompts. The J-lens vector for a word is that matrix read through the
+> model's own unembedding rows.
+
+I verified it the way you verify a ruler. At the last layer, the map from the
+layer to itself must do nothing at all. The faithful J-lens vectors must
+therefore be exactly the model's own word-scoring rows. My check returned
+cosine similarity 1.0000. The ruler is correct.
+
+(I also confirmed my quantity against Anthropic's released reference
+implementation, `github.com/anthropics/jacobian-lens`: their lens is
+`lens_l(h) = unembed(J_l @ h)` with `J_l = E[∂h_final/∂h_l]` — the same
+residual-to-final Jacobian I compute, and my W_U-probed shortcut is
+mathematically equivalent (verified by the identity check above). Their
+estimator has two differences of detail: it excludes the first 16 positions
+(attention sinks) and the last position from the average, and it averages over
+source positions rather than (source, future) pairs. I re-ran my analysis
+with their exact estimator choices: the frequency correlation is essentially
+identical at every layer (max delta 0.008, see results.md section 1b), so the
+result is robust to those choices.)
+
+One technical note before moving on, because it matters for the
+interpretation: I capture the residual stream *before* the model's final
+layer norm. That matches the paper's definition — the Jacobian stops at the
+final residual stream and the J-lens vectors are the rows of W_U·J_ℓ, with
+normalization applied only when *reading* the lens, i.e.
+softmax(W_U·norm(J_ℓ·h_ℓ)). Under that definition the last-layer identity
+check is exact by construction.
+
+## 6. What I found: frequency is everywhere
+
+On the real trained model, all six layers, both the old (buggy) proxy and the
+faithful lens, correlated with token frequency like this:
+
+```
+  Layer   proxy r   faithful r
+  L0      -0.661    -0.643
+  L1      -0.673    -0.668
+  L2      -0.653    -0.672
+  L3      -0.648    -0.685
+  L4      -0.562    -0.637
+  L5      -0.665    -0.606
+```
+
+Figure 1 plots the same numbers: per layer, the faithful lens (blue), the
+buggy proxy (purple), and the Spearman rank correlation (cyan). The Spearman
+bars are the most negative — the effect is not carried by a few extreme
+tokens.
+
+![Figure 1: Token Frequency Anti-Correlation Across Layers](assets/fig1_layer_correlation.png)
+
+
+The correlation survived the faithful implementation — slightly *stronger*, if
+anything. The rare characters ('?', 'z', 'q', '$') sit at the top of the
+J-space ranking; the common ones (space, 'e', 't', 'i') sit at the bottom. On
+the paper's own quantity, the J-lens ranking is frequency-confounded. The
+statistics are not subtle: Pearson r ≈ -0.61 to -0.69 (p ~ 10^-9 to 10^-13),
+and Spearman rank correlation is even stronger (-0.69 to -0.85), so the
+result is not an artifact of a few extreme common tokens.
+
+### Where the correlation comes from
+
+The faithful lens vector for token k is W_U[k]·J_ℓ: one row of the model's
+word-scoring table, passed through the layer map. Think of W_U as the ruler I
+use to read the model. A token-frequency effect could be in the ruler, in the
+layers, or in both. It is in both.
+
+1. **Static geometry — the ruler.** The unembedding row norms ||W_U[k]||
+   themselves anti-correlate with frequency — most strongly against
+   log-frequency, the natural scale for Zipfian data: r(||W_U[k]||, log10 f) =
+   -0.69 (raw frequency -0.61, Spearman -0.81).
+   Rare tokens get bigger rows in the word-scoring matrix. Since the lens
+   reads through W_U by definition, the raw norm ranking inherits that bias
+   automatically. Looking at different prompts cannot remove a bias already
+   built into the ruler.
+
+   Where does that geometry come from? Not from initialization: fresh models
+   show no frequency correlation in their row norms (r ≈ +0.00 to -0.19
+   across three seeds). After training it is -0.61. The model *learns* to
+   push rare tokens' rows outward as it learns the corpus, and nothing in the
+   objective or the readout ever corrects it. The lens inherits learned
+   geometry, not a fixed one.
+2. **Layer dynamics — the layers.** Regress out the W_U component and an
+   anti-correlation still survives in layers 0-4 (partial r ≈ -0.24 to
+   -0.36) — something about what the layers themselves do keeps boosting
+   rare tokens — and it vanishes at the last layer (+0.06). The mechanism
+   of that layer-dependent part is still under investigation.
+
+The left panel of Figure 2 plots that residual directly; the right panel
+shows the ruler's bias appearing during training.
+
+![Figure 2: The frequency confound splits into two separable parts. Left: what remains after the word-scoring-table effect is regressed out — the layer dynamics, strongest in layers 0-4 and gone at the last layer. Right: the table's frequency bias is learned during training — absent at initialization (r ~ 0), strong after (r ≈ -0.61).](assets/fig2_wu_decomposition.png)
+
+
+So the lens carries a frequency signal from both the ruler it reads with and
+from what the layers do — and the two are separable. That is the finding I
+would most want someone to test at scale.
+
+A fact-check before I go further. "Anthropic does not control for frequency"
+is a claim worth checking, not simply making. I searched the paper and its
+appendix, asked two independent reviewers to do the same, and checked
+Anthropic's released companion code (`github.com/anthropics/jacobian-lens`,
+Apache-2.0). I found no frequency matching, normalization, or baseline in
+the J-lens analyses, and no frequency, unigram, or token-count handling in the
+released code or experiment data. Their appendix does mention filtering
+high-frequency noise tokens for a *different* method, the template lens, and
+calls that move unprincipled. It is not a control for the main J-lens.
+
+## 7. But not *only* frequency
+
+Now the twist. Correlation is not causation, so I ran a cleaner test. I made
+a new corpus with two brand-new characters, both at *exactly* the same
+frequency (0.1%):
+
+- `@` — appears only after the trigger "the ". The model can predict it in
+  context. It is *poised to be said*.
+- `#` — appears at random positions. Nothing predicts it.
+
+Same frequency. Different structure. If the J-lens were purely a frequency
+meter, the two tokens would get identical norms. Here is what three separate
+training runs showed:
+
+```
+  seed   @ norm (predictable)   # norm (noise)   ratio
+  0      0.0232–0.0246          0.0152–0.0154     1.51–1.60
+  1      0.0224–0.0237          0.0148–0.0151     1.50–1.60
+  2      0.0215–0.0233          0.0154–0.0163     1.35–1.51
+```
+*(Norm ranges are across layers 0-5 within each seed; ratios are per-layer
+@/#. The middle-layer summary follows.)*
+
+The predictable token scores **~1.4-1.5x higher** than the noise token at
+identical frequency, in every layer of every seed. Middle-layer ratio across
+the three seeds: 1.47 ± 0.09, bootstrap 95% CI [1.37, 1.53] — entirely above
+1. So the lens is not a pure frequency meter: at equal frequency, the two
+tokens differ in norm.
+
+One caveat, found by a reviewer: the noise token '#' was inserted at random
+character positions, which slices through the middle of a word 58% of the time
+(th#e, ki#ng — letter on both sides), while '@' always sits at a clean word
+boundary after "the ". That confounds predictability with n-gram corruption —
+so I ran the control that isolates them: '#' inserted at random *word
+boundaries* (0% word-slicing, still unpredictable), same 0.0998% frequency,
+three fresh seeds.
+
+The control is done, and it is the honest kind of result — partly confirming,
+partly correcting:
+
+```
+  placement of '#'      middle-layer ratio @/#   bootstrap 95% CI
+  random (58% slicing)  1.47 ± 0.09              [1.368, 1.529]
+  clean boundary (0%)    1.31 ± 0.08              [1.258, 1.402]
+```
+
+Figure 3 shows both conditions against the ratio=1 baseline (dashed red line,
+the no-signal case): the clean-boundary bar still clears it, CI included.
+
+![Figure 3: Synthetic Frequency-Matched Pair Control (@ vs # at 0.1% Frequency). Error bars are bootstrap 95% CIs; the dashed line marks ratio 1 (no predictability signal).](assets/fig3_synthetic_pair.png)
+
+
+The corruption confound was real: it inflated the estimate by about 12%. But
+it was not the whole story. At identical frequency, with clean boundaries and
+nothing sliced, the predictable token still scores ~1.3x higher than the
+unpredictable one, and the CI stays entirely above 1 in every seed. The
+conditional-predictability signal — the thing "verbalizable" should mean —
+survives the control, modestly smaller than my first estimate.
+
+## 8. The causal test: what actually happened
+
+The last experiment was the one designed to make the frequency story causal.
+Train three models per seed from the *identical* starting weights and the
+*identical* minibatch order — the only difference is the loss: one model gives
+the letter 'q' twice the learning pressure (2x CE weight on 'q' targets, which
+raises its effective frequency without corrupting the text), one is a plain
+control, and one upweights the same number of random *other* letters (to check
+that "any reweighting" isn't the thing doing the work). Three seeds, three
+models each. If doubling 'q's effective frequency causally shrinks its J-lens
+norm below both controls, the frequency story is causal, not just
+correlational. The faithful lens norm of 'q' (layers 2-4, mean per seed):
+
+```
+  seed | q(2x)  control  ctrl_random | q/control  q/ctrl_random
+  0    | 0.0163  0.0174   0.0152    |   0.934       1.069
+  1    | 0.0150  0.0171   0.0161    |   0.881       0.932
+  2    | 0.0161  0.0158   0.0169    |   1.019       0.952
+```
+
+Figure 4 shows the three arms per seed — q upweighted (red), plain control
+(blue), random-upweight (green) — plus the cross-seed mean with the annotation
+q/control = 0.944, CI [0.881, 1.019]. The CI straddling 1 is the picture of
+"too small to settle it."
+
+![Figure 4: Loss-Reweighting Causal Test Across 3 Seeds. Per-seed bars: q(2x), control, random-upweight; annotation = cross-seed ratio + 95% CI.](assets/fig4_loss_reweighting.png)
+
+
+Cross-seed: q/control mean = 0.944 (bootstrap 95% CI [0.881, 1.019]),
+q/ctrl_random mean = 0.985 (CI [0.932, 1.069]).
+
+The answer is: perhaps, but this experiment is too small to settle it. In two
+of three runs, giving `q` twice the loss weight lowered its norm; on average it
+was 5.6% below the ordinary control. But the confidence interval crosses 1,
+one run went the other way, and the result disappears against the
+random-upweight control. With the variation I saw, a properly powered version
+needs about 13 seeds per arm, not three.
+
+There is a useful lesson in the weak result. Doubling actual occurrences of
+`q` in the training text had previously dropped its norm by 67%. Doubling the
+loss weight moved it only about 6%. Changing what the model sees is a much
+stronger lever than changing the size of its gradient after the fact; AdamW
+appears to absorb part of the latter change.
+
+Meanwhile the overall frequency correlation barely moved: it stayed around
+r ≈ -0.63 to -0.69 across all nine models. That is what I would expect if
+most of the pattern is in the learned word-scoring geometry, rather than a
+fragile effect of one token's loss weight.
+
+Net: the frequency confound is strongly correlational and geometrically
+stable; the causal lever I could afford to test is weak. This is the honest
+state of the causal evidence. (Absolute 'q' norms differ across experiments —
+base model 0.011 vs these 0.015-0.018 — so only within-experiment
+comparisons are meaningful.)
+
+## 9. What I am NOT saying
+
+Let me be very careful here, because it would be easy to overclaim.
+
+- I am **not** saying the J-space doesn't exist. I haven't tested
+  Anthropic's actual capacity claim (which is about *occupancy* — how often
+  J-lens directions are used per position — not about the rank of the word
+  vectors).
+- I am **not** saying they have no controls at all. Their occupancy
+  analysis compares against random-direction baselines, and their probes
+  subtract mean concept directions. Those are real experimental controls —
+  but none of them is a token-frequency control, which is the specific thing
+  my claim is about.
+- I am **not** claiming the lens is useless. The synthetic-pair result shows it
+  carries real structure signal.
+- I am **not** claiming that ranking by lens-vector *norm* is the same as
+  ranking by *lens output on real activations*. My numbers rank tokens by the
+  norm of their faithful J-lens vector — a summary of the readout geometry —
+  not by how strongly, or how often, those directions actually fire in running
+  text. Anthropic's capacity claim is about the latter (occupancy). The norm
+  still matters: it is the size of a token's lever on the readout, and the W_U
+  geometry that drives it enters any readout-based analysis by definition. But
+  the gap between "geometry is frequency-confounded" and "the capacity claim
+  is frequency-confounded" is real, and it is the specific gap an at-scale
+  occupancy test has to close.
+- I am **not** saying "it's just linear algebra." My toy models don't show
+  the compression Anthropic sees in large models; that's a limitation of toy
+  models, not evidence against large ones.
+
+What I **am** saying is narrower and, I think, more durable: on the paper's
+own measurement, J-lens *norm-rankings* are strongly confounded by token
+frequency at every scale I can test, and frequency is a variable any J-lens
+analysis should control for. Whether the confound survives at Anthropic's scale
+is an empirical question — one I am taking to bigger models next, and one I
+already have a first, partial answer to for the geometric half (Section 10).
+
+## 10. What's next
+
+I did try bigger once already, and I owe you the number, because a reader
+who opens the repo will find it either way: an early probe on GPT-2 small
+(`src/gpt2_jlens.py`) returned an average correlation of only r ≈ -0.18 across
+layers. I do not count it as evidence, for three concrete reasons: it sampled
+96 token positions out of a 50,257-token vocabulary; it averaged over only 100
+sampled tokens per batch; and it measured a subtly different quantity
+(norm-per-batch rather than norm-of-the-mean). It was a directional probe, and
+it pointed weak. It is logged in `results.md`, flagged do-not-cite — but a
+post that promises "bigger models next" should not pretend the attempt never
+happened.
+
+Toy scale answers the methodological question. Scale answers the real one. I
+want to run the faithful lens on a real language model (V = 50K, d = 768 — the
+regime where Anthropic's claims live) with proper statistical power, and to run
+the occupancy test their capacity claim is actually about. That's the next
+post.
+
+The learned-geometry finding makes one piece of that cheaper than the probe
+was: if the W_U row-norm anti-correlation is a general property of
+softmax-output models trained on Zipfian data, it should appear in GPT-2's
+unembedding matrix directly — no Jacobian computation at all. So I ran it:
+GPT-2's unembedding row norms correlate with token log-frequency at V = 50,257
+(r ≈ -0.45 on gpt2-small, -0.49 on gpt2-medium, n = 46,887 tokens seen in
+wikitext-103; Spearman -0.46 to -0.50). The decile picture is monotone in both
+models — the rarest tenth of tokens have the biggest rows — and the top-norm
+tokens are rare ones (SPONSORED, soDeliveryDate, Reviewer). The geometric half
+of the confound is not a 65-character vocabulary artifact: it is present in
+two real, independently trained language models at Anthropic's vocabulary
+scale. One honest caveat: it is weaker than at toy scale (-0.45/-0.49 vs
+-0.69), so the char-scale result was not a size artifact, but neither is the
+at-scale effect as strong as the toy suggested. The layer-dynamics half —
+what the layers themselves add beyond the ruler — still needs the faithful
+lens at scale, and that remains the next post.
+
+## 11. How to reproduce everything
+
+All code, data-prep scripts, experiment scripts, tests, and this analysis live
+in the repository: <https://git.jayrup.me/c/jspace-nanogpt.git/>. Summary of
+results in `results.md`. Reproduction steps in the README. The only
+requirements are a Linux machine with Docker, a CUDA GPU (any modern card; I
+used a 4GB Quadro K2200), and the `pytorch/pytorch:2.4.1-cuda11.8` image.
+
+Run the test suite:
+```
+sh scripts/test.sh
+```
+
+Rebuild the main experiment from scratch:
+```
+# 1. train the character-level model on Shakespeare (10.65M params)
+# 2. compute the faithful J-lens + old proxy, all layers:
+python3 src/jlens_v3.py --checkpoint out-shakespeare-char/ckpt.pt \
+    --data_dir data/shakespeare_char --layers 0,1,2,3,4,5
+# 3. synthetic frequency-matched pair:
+python3 src/synthetic_pair.py --step prep
+python3 src/synthetic_pair.py --step train --seed 0
+python3 src/synthetic_pair.py --step jlens --seed 0
+python3 src/synthetic_pair.py --step summary
+# 4. loss-reweighting causal test:
+python3 src/loss_reweight.py --step train --mode q --seed 0
+python3 src/loss_reweight.py --step summary
+```
+
+---
+
+*Written in the spirit of the rule I keep trying to follow: the first
+principle is that you must not fool yourself — and you are the easiest person
+to fool.*
-- 
cgit v1.2.3

