From 34d5e8c4bbad4abb236dcaeb32050bf8c2bdbcf0 Mon Sep 17 00:00:00 2001 From: CaptainJack2491 Date: Fri, 7 Aug 2026 14:32:21 +0100 Subject: Blog: clarify J-lens frequency findings --- results.md | 24 ------------------------ 1 file changed, 24 deletions(-) (limited to 'results.md') diff --git a/results.md b/results.md index 68422db..d396731 100644 --- a/results.md +++ b/results.md @@ -89,30 +89,6 @@ experiment + evaluation data) finds ZERO occurrences of frequency/unigram/ token-count terms. Their README confirms the fit corpus is "a generic web-text corpus" — no frequency balancing. -## 1c. Independent corroboration (willkn, GreaterWrong, GPT-2-medium) - -An independent engineering analysis of the paper (willkn, GreaterWrong, 24 Jul -2026; notebooks: github.com/willkn/jlens_re; full text: docs/willkn-jlens- -greaterwrong-analysis.md), on GPT-2-medium (355M, d=1024, L=24, V=50,257), -found: -- the raw fitted Jacobian "misweights structural tokens (grammar, punctuation) - over semantic content" — dominant spectral channels carry ~10x the gain of - the residual pathway. Structural tokens ARE the high-frequency tokens; this - is the same distortion we measure as frequency anti-correlation, at 30x our - model scale and via a different metric (next-token faithfulness of the - readout). -- the Jacobian is essentially full-rank: 562-858 dimensions capture 90% of - spectral variance, "leaving little room for compression" — corroborates our - dimensional-starvation / full-rank result at scale. -- a single-parameter shrinkage regularizer (J + lambda*I) monotonically - recovers next-token faithfulness (beats logit lens at layer 12: 0.294 vs - 0.275). Anthropic's released fitting code applies no such regularization. -- measured noise law rel_F(J_n) ~ c_l/sqrt(n), c_l growing ~4x from layer 20 - to layer 4; per-backward cost ~5ms + 9ms/layer on L4 at dim_batch=8. - -Untested (natural next experiment): does shrinkage also remove our frequency -correlation? - ## 2. Frequency-matched synthetic pair (`src/synthetic_pair.py`) Two new characters at identical 0.1% unigram frequency in Shakespeare: -- cgit v1.2.3