1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
|
# Results
All numbers below are from the faithful J-lens (`src/jlens_v3.py`), which
computes exactly the paper's quantity: rows of `W_U * J_l` where
`J_l = E[ d h_final / d h_l ]` (average residual-to-residual Jacobian, read out
through the unembedding). Verification: at the last layer, J must be the
identity, and the check returns cosine similarity **1.0000**.
## 1. Both-ways comparison: old proxy vs faithful lens (trained 10.65M char model)
Pearson r between token frequency and J-lens norm, per layer. n = 65 tokens
(full char vocab). The correlation survives the faithful implementation at
every layer. p-values: all layers p ~ 10^-9 to 10^-13. Spearman rank
correlation is even stronger: faithful -0.69 to -0.85; proxy -0.63 to -0.81
(not a leverage artifact of extreme common tokens).
```
Layer proxy r faithful r Spearman(faith) partial r(faith, freq | ||W_U||)
L0 -0.661 -0.643 -0.844 -0.307
L1 -0.673 -0.668 -0.851 -0.319
L2 -0.653 -0.672 -0.825 -0.338
L3 -0.648 -0.685 -0.769 -0.364
L4 -0.562 -0.637 -0.716 -0.240
L5 -0.665 -0.606 -0.808 +0.056
```
### W_U decomposition (where does the correlation come from?)
The faithful lens vector for token k is `W_U[k] * J_l`. The unembedding row
norms themselves anti-correlate with frequency:
```
r(||W_U[k]||, freq) = -0.606
r(||W_U[k]||, log10 freq) = -0.693
Spearman(||W_U[k]||, freq) = -0.808
```
`r(faithful_norm, ||W_U[k]||)` per layer: +0.70 to +1.00 (L5 = +1.00 exactly,
since J = identity at the last layer). After regressing out the W_U row-norm
component, a layer-dependent anti-correlation survives in layers 0-4 (partial
r ≈ -0.24 to -0.36) and vanishes at the last layer (+0.06). Interpretation:
the frequency signal lives partly in the static unembedding geometry (which is
baked into Anthropic's lens by definition) and partly in layer-dependent
dynamics; the mechanism of the latter is not yet pinned down.
W_U geometry is LEARNED, not present at init: fresh models (3 seeds, same
config) show r(||W_U[k]||, freq) = +0.002, -0.189, -0.166 (trained: -0.606),
and row-norm spread grows ~4x during training ([0.36, 0.44] -> [0.86, 1.52]).
The model learns to push rare-token unembedding rows outward and nothing
corrects it.
Top tokens by faithful norm are consistently rare characters (`?`, `z`, `q`,
`$`); bottom are common ones (space, `e`, `t`, `i`).
## 1b. Validation against Anthropic's official reference implementation
Anthropic released companion code (`github.com/anthropics/jacobian-lens`,
Apache-2.0). Their lens is `lens_l(h) = unembed(J_l @ h)` with
`J_l = E[∂h_final/∂h_l]` — the same quantity we compute; our W_U-probed
shortcut (rows of `W_U * J_l`) is mathematically equivalent, verified by the
last-layer identity check (cos-sim 1.0000).
Their estimator (`jlens/fitting.py`) differs from our default in two details:
1. `valid_position_mask`: excludes the first 16 positions (attention sinks)
and the last position from the average.
2. Averaging: mean over source positions (each position once) instead of our
mean over (source, future) pairs.
We added `--skip_first N --source_mean` to `src/jlens_v3.py` to mirror their
estimator exactly and re-ran the same model (outputs/jlens_v3_anthropic/).
Result: the frequency correlation is essentially IDENTICAL under their
estimator at every layer (max delta 0.008):
```
Layer default r anthropic r delta
L0 -0.643 -0.651 -0.008
L1 -0.668 -0.669 -0.000
L2 -0.672 -0.673 -0.001
L3 -0.685 -0.687 -0.002
L4 -0.637 -0.635 +0.002
L5 -0.606 -0.606 +0.000
```
The correlation is robust to the estimator's averaging choices (position mask
and normalization). Last-layer identity check passed again (cos-sim 1.0000).
Frequency scan: case-insensitive grep of their entire repo (README, code,
experiment + evaluation data) finds ZERO occurrences of frequency/unigram/
token-count terms. Their README confirms the fit corpus is "a generic web-text
corpus" — no frequency balancing.
## 1c. Independent corroboration (willkn, GreaterWrong, GPT-2-medium)
An independent engineering analysis of the paper (willkn, GreaterWrong, 24 Jul
2026; notebooks: github.com/willkn/jlens_re; full text: docs/willkn-jlens-
greaterwrong-analysis.md), on GPT-2-medium (355M, d=1024, L=24, V=50,257),
found:
- the raw fitted Jacobian "misweights structural tokens (grammar, punctuation)
over semantic content" — dominant spectral channels carry ~10x the gain of
the residual pathway. Structural tokens ARE the high-frequency tokens; this
is the same distortion we measure as frequency anti-correlation, at 30x our
model scale and via a different metric (next-token faithfulness of the
readout).
- the Jacobian is essentially full-rank: 562-858 dimensions capture 90% of
spectral variance, "leaving little room for compression" — corroborates our
dimensional-starvation / full-rank result at scale.
- a single-parameter shrinkage regularizer (J + lambda*I) monotonically
recovers next-token faithfulness (beats logit lens at layer 12: 0.294 vs
0.275). Anthropic's released fitting code applies no such regularization.
- measured noise law rel_F(J_n) ~ c_l/sqrt(n), c_l growing ~4x from layer 20
to layer 4; per-backward cost ~5ms + 9ms/layer on L4 at dim_batch=8.
Untested (natural next experiment): does shrinkage also remove our frequency
correlation?
## 2. Frequency-matched synthetic pair (`src/synthetic_pair.py`)
Two new characters at identical 0.1% unigram frequency in Shakespeare:
`@` appears only after the trigger "the " (predictable in context);
`#` appears at uniform random positions. Faithful J-lens norm per seed, range
over layers 0-5:
```
seed @ norm (predictable) # norm (noise) ratio freq corr r
0 0.0232 - 0.0246 0.0152 - 0.0154 1.51-1.60 -0.59..-0.68
1 0.0224 - 0.0237 0.0148 - 0.0151 1.50-1.60 -0.59..-0.66
2 0.0215 - 0.0233 0.0154 - 0.0163 1.35-1.51 -0.62..-0.66
```
Reading: at equal frequency, the structured token scores ~1.4-1.5x higher.
Middle-layer ratio across seeds: 1.47 +/- 0.09 (SD), bootstrap 95% CI
[1.37, 1.53]. The frequency anti-correlation holds, but the lens also carries
genuine conditional-predictability signal.
CAVEAT + CONTROL (resolved): '#' was originally inserted at uniform random
character positions, which slices through the middle of a word 58% of the time
(letter on both sides: th#e, ki#ng); '@' always sits at a clean word boundary
after "the ". A clean-boundary control (noise token after random word
boundaries, 0% word-slicing, same frequency) was run with three fresh seeds:
```
placement of '#' middle-layer ratio @/# bootstrap 95% CI
random (58% slicing) 1.471 +/- 0.090 [1.368, 1.529]
clean boundary (0%) 1.311 +/- 0.079 [1.258, 1.402]
```
Reading: the corruption confound was real (inflated the ratio by ~12%) but not
the whole story — the structure signal survives at clean boundaries, CI
entirely above 1 in every seed. Original run: outputs/synth_pair/seed{s}/;
control run: outputs/synth_pair_clean/seed{s}/.
## 3. Loss-reweighting causal test (`src/loss_reweight.py`)
Three models per seed, identical init + minibatch order: 'q' targets weighted
x2 in the loss, plain control, and a same-total-loss control upweighting random
non-'q' targets. Question: does raising effective frequency causally reduce
'q's faithful J-lens norm? Faithful lens norm of 'q' (mean over layers 2-4):
```
seed | q(2x) control ctrl_random | q/control q/ctrl_random
0 | 0.0163 0.0174 0.0152 | 0.934 1.069
1 | 0.0150 0.0171 0.0161 | 0.881 0.932
2 | 0.0161 0.0158 0.0169 | 1.019 0.952
```
Cross-seed: q/control mean = 0.944 (bootstrap 95% CI [0.881, 1.019]),
q/ctrl_random mean = 0.985 (CI [0.932, 1.069]).
Reading:
1. Expected-direction signal, small: 2x loss pressure lowers 'q' norm in 2/3
seeds (~5.6% mean vs control), but CI crosses 1.0 and the effect vanishes
vs the random-upweight control (0.985). Not a robust causal demotion at
this power.
2. Frequency correlation is invariant across all 9 models: mean r = -0.66
(min -0.685, max -0.626) in every mode/seed/layer — training with 'q'
upweighted does not change the correlation structure (geometric, per the
W_U decomposition).
3. Contrast: doubling ACTUAL corpus occurrences of 'q' (original ablation)
dropped its norm 67%; doubling its LOSS weight drops it ~6%. Data-frequency
is the strong causal lever; gradient reweighting under AdamW is weak
(adaptive per-param scaling absorbs part of the signal).
4. Absolute norms differ across experiments (base L3 q = 0.0109 vs these
0.015-0.018) — only within-experiment comparisons are meaningful.
## 4. Historical / do-not-copy
- Original proxy finding (r = -0.65, `jlens_v2`): superseded by the faithful
implementation; kept only for the both-ways comparison.
- GPT-2 correlation: an early directional probe on GPT-2 small
(`gpt2_jlens.py`) gave avg r = -0.18 across layers — UNDER-POWERED
(96 token positions, n=100 sampled tokens) and a different quantity
(norm-per-batch vs norm-of-mean). Kept here only as a reproducibility
warning: **do not cite the -0.18 as evidence**; the at-scale test with the
faithful lens is future work.
|
