1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
|
# Results
All numbers below are from the faithful J-lens (`src/jlens_v3.py`), which
computes exactly the paper's quantity: rows of `W_U * J_l` where
`J_l = E[ d h_final / d h_l ]` (average residual-to-residual Jacobian, read out
through the unembedding). Verification: at the last layer, J must be the
identity, and the check returns cosine similarity **1.0000**.
## 1. Both-ways comparison: old proxy vs faithful lens (trained 10.65M char model)
Pearson r between token frequency and J-lens norm, per layer. n = 65 tokens
(full char vocab). The correlation survives the faithful implementation at
every layer. p-values: all layers p ~ 10^-9 to 10^-13. Spearman rank
correlation is even stronger: faithful -0.69 to -0.85; proxy -0.63 to -0.81
(not a leverage artifact of extreme common tokens).
```
Layer proxy r faithful r Spearman(faith) partial r(faith, freq | ||W_U||)
L0 -0.661 -0.643 -0.844 -0.307
L1 -0.673 -0.668 -0.851 -0.319
L2 -0.653 -0.672 -0.825 -0.338
L3 -0.648 -0.685 -0.769 -0.364
L4 -0.562 -0.637 -0.716 -0.240
L5 -0.665 -0.606 -0.808 +0.056
```
### W_U decomposition (where does the correlation come from?)
The faithful lens vector for token k is `W_U[k] * J_l`. The unembedding row
norms themselves anti-correlate with frequency:
```
r(||W_U[k]||, freq) = -0.606
r(||W_U[k]||, log10 freq) = -0.693
Spearman(||W_U[k]||, freq) = -0.808
```
`r(faithful_norm, ||W_U[k]||)` per layer: +0.70 to +1.00 (L5 = +1.00 exactly,
since J = identity at the last layer). After regressing out the W_U row-norm
component, a layer-dependent anti-correlation survives in layers 0-4 (partial
r ≈ -0.24 to -0.36) and vanishes at the last layer (+0.06). Interpretation:
the frequency signal lives partly in the static unembedding geometry (which is
baked into Anthropic's lens by definition) and partly in layer-dependent
dynamics; the mechanism of the latter is not yet pinned down.
Top tokens by faithful norm are consistently rare characters (`?`, `z`, `q`,
`$`); bottom are common ones (space, `e`, `t`, `i`).
## 1b. Validation against Anthropic's official reference implementation
Anthropic released companion code (`github.com/anthropics/jacobian-lens`,
Apache-2.0). Their lens is `lens_l(h) = unembed(J_l @ h)` with
`J_l = E[∂h_final/∂h_l]` — the same quantity we compute; our W_U-probed
shortcut (rows of `W_U * J_l`) is mathematically equivalent, verified by the
last-layer identity check (cos-sim 1.0000).
Their estimator (`jlens/fitting.py`) differs from our default in two details:
1. `valid_position_mask`: excludes the first 16 positions (attention sinks)
and the last position from the average.
2. Averaging: mean over source positions (each position once) instead of our
mean over (source, future) pairs.
We added `--skip_first N --source_mean` to `src/jlens_v3.py` to mirror their
estimator exactly. The robustness re-run (same model, their estimator choices)
is queued behind the loss-reweighting GPU job; results will be appended here.
Frequency scan: case-insensitive grep of their entire repo (README, code,
experiment + evaluation data) finds ZERO occurrences of frequency/unigram/
token-count terms. Their README confirms the fit corpus is "a generic web-text
corpus" — no frequency balancing.
## 1c. Independent corroboration (willkn, GreaterWrong, GPT-2-medium)
An independent engineering analysis of the paper (willkn, GreaterWrong, 24 Jul
2026; notebooks: github.com/willkn/jlens_re; full text: docs/willkn-jlens-
greaterwrong-analysis.md), on GPT-2-medium (355M, d=1024, L=24, V=50,257),
found:
- the raw fitted Jacobian "misweights structural tokens (grammar, punctuation)
over semantic content" — dominant spectral channels carry ~10x the gain of
the residual pathway. Structural tokens ARE the high-frequency tokens; this
is the same distortion we measure as frequency anti-correlation, at 30x our
model scale and via a different metric (next-token faithfulness of the
readout).
- the Jacobian is essentially full-rank: 562-858 dimensions capture 90% of
spectral variance, "leaving little room for compression" — corroborates our
dimensional-starvation / full-rank result at scale.
- a single-parameter shrinkage regularizer (J + lambda*I) monotonically
recovers next-token faithfulness (beats logit lens at layer 12: 0.294 vs
0.275). Anthropic's released fitting code applies no such regularization.
- measured noise law rel_F(J_n) ~ c_l/sqrt(n), c_l growing ~4x from layer 20
to layer 4; per-backward cost ~5ms + 9ms/layer on L4 at dim_batch=8.
Untested (natural next experiment): does shrinkage also remove our frequency
correlation?
## 2. Frequency-matched synthetic pair (`src/synthetic_pair.py`)
Two new characters at identical 0.1% unigram frequency in Shakespeare:
`@` appears only after the trigger "the " (predictable in context);
`#` appears at uniform random positions. Faithful J-lens norm per seed, range
over layers 0-5:
```
seed @ norm (predictable) # norm (noise) ratio freq corr r
0 0.0232 - 0.0246 0.0152 - 0.0154 1.51-1.60 -0.59..-0.68
1 0.0224 - 0.0237 0.0148 - 0.0151 1.50-1.60 -0.59..-0.66
2 0.0215 - 0.0233 0.0154 - 0.0163 1.35-1.51 -0.62..-0.66
```
Reading: at equal frequency, the structured token scores ~1.4-1.5x higher.
Middle-layer ratio across seeds: 1.47 +/- 0.09 (SD), bootstrap 95% CI
[1.37, 1.53]. The frequency anti-correlation holds, but the lens also carries
genuine conditional-predictability signal.
CAVEAT (from adversarial review): '#' was inserted at uniform random character
positions, which slices inside words ~95% of the time (th#e, ki#ng); '@'
always sits at a clean word boundary after "the ". Predictability is therefore
not perfectly isolated from n-gram corruption. A clean-boundary control (noise
token after random word boundaries) is planned; the numbers above should be
read with that caveat until it lands.
## 3. Loss-reweighting causal test (`src/loss_reweight.py`)
Three models per seed, identical init + minibatch order: 'q' targets weighted
x2 in the loss, plain control, and a same-total-loss control upweighting random
non-'q' targets. Question: does raising effective frequency causally reduce
'q's faithful J-lens norm? PENDING — run completes within hours of this file
being written; the summary table is printed by
`python3 src/loss_reweight.py --step summary --layers 2,3,4`.
## 4. Historical / do-not-copy
- Original proxy finding (r = -0.65, `jlens_v2`): superseded by the faithful
implementation; kept only for the both-ways comparison.
- GPT-2 correlation (r = -0.18, `gpt2_jlens.py`): UNDER-POWERED (96 token
positions, n=100 sampled tokens) and computed a different quantity
(norm-per-batch vs norm-of-mean). Directionally consistent but not
publishable evidence on its own.
|
