# Results All numbers below are from the faithful J-lens (`src/jlens_v3.py`), which computes exactly the paper's quantity: rows of `W_U * J_l` where `J_l = E[ d h_final / d h_l ]` (average residual-to-residual Jacobian, read out through the unembedding). Verification: at the last layer, J must be the identity, and the check returns cosine similarity **1.0000**. ## 1. Both-ways comparison: old proxy vs faithful lens (trained 10.65M char model) Pearson r between token frequency and J-lens norm, per layer. n = 65 tokens (full char vocab). The correlation survives the faithful implementation at every layer. p-values: all layers p ~ 10^-9 to 10^-13. Spearman rank correlation is even stronger: faithful -0.69 to -0.85; proxy -0.63 to -0.81 (not a leverage artifact of extreme common tokens). ``` Layer proxy r faithful r Spearman(faith) partial r(faith, freq | ||W_U||) L0 -0.661 -0.643 -0.844 -0.307 L1 -0.673 -0.668 -0.851 -0.319 L2 -0.653 -0.672 -0.825 -0.338 L3 -0.648 -0.685 -0.769 -0.364 L4 -0.562 -0.637 -0.716 -0.240 L5 -0.665 -0.606 -0.808 +0.056 ``` ### W_U decomposition (where does the correlation come from?) The faithful lens vector for token k is `W_U[k] * J_l`. The unembedding row norms themselves anti-correlate with frequency: ``` r(||W_U[k]||, freq) = -0.606 r(||W_U[k]||, log10 freq) = -0.693 Spearman(||W_U[k]||, freq) = -0.808 ``` `r(faithful_norm, ||W_U[k]||)` per layer: +0.70 to +1.00 (L5 = +1.00 exactly, since J = identity at the last layer). After regressing out the W_U row-norm component, a layer-dependent anti-correlation survives in layers 0-4 (partial r ≈ -0.24 to -0.36) and vanishes at the last layer (+0.06). Interpretation: the frequency signal lives partly in the static unembedding geometry (which is baked into Anthropic's lens by definition) and partly in layer-dependent dynamics; the mechanism of the latter is not yet pinned down. Top tokens by faithful norm are consistently rare characters (`?`, `z`, `q`, `$`); bottom are common ones (space, `e`, `t`, `i`). ## 1b. Validation against Anthropic's official reference implementation Anthropic released companion code (`github.com/anthropics/jacobian-lens`, Apache-2.0). Their lens is `lens_l(h) = unembed(J_l @ h)` with `J_l = E[∂h_final/∂h_l]` — the same quantity we compute; our W_U-probed shortcut (rows of `W_U * J_l`) is mathematically equivalent, verified by the last-layer identity check (cos-sim 1.0000). Their estimator (`jlens/fitting.py`) differs from our default in two details: 1. `valid_position_mask`: excludes the first 16 positions (attention sinks) and the last position from the average. 2. Averaging: mean over source positions (each position once) instead of our mean over (source, future) pairs. We added `--skip_first N --source_mean` to `src/jlens_v3.py` to mirror their estimator exactly. The robustness re-run (same model, their estimator choices) is queued behind the loss-reweighting GPU job; results will be appended here. Frequency scan: case-insensitive grep of their entire repo (README, code, experiment + evaluation data) finds ZERO occurrences of frequency/unigram/ token-count terms. Their README confirms the fit corpus is "a generic web-text corpus" — no frequency balancing. ## 1c. Independent corroboration (willkn, GreaterWrong, GPT-2-medium) An independent engineering analysis of the paper (willkn, GreaterWrong, 24 Jul 2026; notebooks: github.com/willkn/jlens_re; full text: docs/willkn-jlens- greaterwrong-analysis.md), on GPT-2-medium (355M, d=1024, L=24, V=50,257), found: - the raw fitted Jacobian "misweights structural tokens (grammar, punctuation) over semantic content" — dominant spectral channels carry ~10x the gain of the residual pathway. Structural tokens ARE the high-frequency tokens; this is the same distortion we measure as frequency anti-correlation, at 30x our model scale and via a different metric (next-token faithfulness of the readout). - the Jacobian is essentially full-rank: 562-858 dimensions capture 90% of spectral variance, "leaving little room for compression" — corroborates our dimensional-starvation / full-rank result at scale. - a single-parameter shrinkage regularizer (J + lambda*I) monotonically recovers next-token faithfulness (beats logit lens at layer 12: 0.294 vs 0.275). Anthropic's released fitting code applies no such regularization. - measured noise law rel_F(J_n) ~ c_l/sqrt(n), c_l growing ~4x from layer 20 to layer 4; per-backward cost ~5ms + 9ms/layer on L4 at dim_batch=8. Untested (natural next experiment): does shrinkage also remove our frequency correlation? ## 2. Frequency-matched synthetic pair (`src/synthetic_pair.py`) Two new characters at identical 0.1% unigram frequency in Shakespeare: `@` appears only after the trigger "the " (predictable in context); `#` appears at uniform random positions. Faithful J-lens norm per seed, range over layers 0-5: ``` seed @ norm (predictable) # norm (noise) ratio freq corr r 0 0.0232 - 0.0246 0.0152 - 0.0154 1.51-1.60 -0.59..-0.68 1 0.0224 - 0.0237 0.0148 - 0.0151 1.50-1.60 -0.59..-0.66 2 0.0215 - 0.0233 0.0154 - 0.0163 1.35-1.51 -0.62..-0.66 ``` Reading: at equal frequency, the structured token scores ~1.4-1.5x higher. Middle-layer ratio across seeds: 1.47 +/- 0.09 (SD), bootstrap 95% CI [1.37, 1.53]. The frequency anti-correlation holds, but the lens also carries genuine conditional-predictability signal. CAVEAT (from adversarial review): '#' was inserted at uniform random character positions, which slices inside words ~95% of the time (th#e, ki#ng); '@' always sits at a clean word boundary after "the ". Predictability is therefore not perfectly isolated from n-gram corruption. A clean-boundary control (noise token after random word boundaries) is planned; the numbers above should be read with that caveat until it lands. ## 3. Loss-reweighting causal test (`src/loss_reweight.py`) Three models per seed, identical init + minibatch order: 'q' targets weighted x2 in the loss, plain control, and a same-total-loss control upweighting random non-'q' targets. Question: does raising effective frequency causally reduce 'q's faithful J-lens norm? PENDING — run completes within hours of this file being written; the summary table is printed by `python3 src/loss_reweight.py --step summary --layers 2,3,4`. ## 4. Historical / do-not-copy - Original proxy finding (r = -0.65, `jlens_v2`): superseded by the faithful implementation; kept only for the both-ways comparison. - GPT-2 correlation (r = -0.18, `gpt2_jlens.py`): UNDER-POWERED (96 token positions, n=100 sampled tokens) and computed a different quantity (norm-per-batch vs norm-of-mean). Directionally consistent but not publishable evidence on its own.