From 781a07b0c52c62657a8030eec8bbab76d26fa90f Mon Sep 17 00:00:00 2001 From: Void Agent Date: Sun, 2 Aug 2026 14:54:17 +0100 Subject: Add willkn GreaterWrong corroboration: independent GPT-2-medium finding that raw J-lens misweights structural (high-frequency) tokens + full-rank Jacobian; shrinkage J+lambda I as untested next experiment --- results.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) (limited to 'results.md') diff --git a/results.md b/results.md index 574ee06..48c1c48 100644 --- a/results.md +++ b/results.md @@ -69,6 +69,30 @@ experiment + evaluation data) finds ZERO occurrences of frequency/unigram/ token-count terms. Their README confirms the fit corpus is "a generic web-text corpus" — no frequency balancing. +## 1c. Independent corroboration (willkn, GreaterWrong, GPT-2-medium) + +An independent engineering analysis of the paper (willkn, GreaterWrong, 24 Jul +2026; notebooks: github.com/willkn/jlens_re; full text: docs/willkn-jlens- +greaterwrong-analysis.md), on GPT-2-medium (355M, d=1024, L=24, V=50,257), +found: +- the raw fitted Jacobian "misweights structural tokens (grammar, punctuation) + over semantic content" — dominant spectral channels carry ~10x the gain of + the residual pathway. Structural tokens ARE the high-frequency tokens; this + is the same distortion we measure as frequency anti-correlation, at 30x our + model scale and via a different metric (next-token faithfulness of the + readout). +- the Jacobian is essentially full-rank: 562-858 dimensions capture 90% of + spectral variance, "leaving little room for compression" — corroborates our + dimensional-starvation / full-rank result at scale. +- a single-parameter shrinkage regularizer (J + lambda*I) monotonically + recovers next-token faithfulness (beats logit lens at layer 12: 0.294 vs + 0.275). Anthropic's released fitting code applies no such regularization. +- measured noise law rel_F(J_n) ~ c_l/sqrt(n), c_l growing ~4x from layer 20 + to layer 4; per-backward cost ~5ms + 9ms/layer on L4 at dim_batch=8. + +Untested (natural next experiment): does shrinkage also remove our frequency +correlation? + ## 2. Frequency-matched synthetic pair (`src/synthetic_pair.py`) Two new characters at identical 0.1% unigram frequency in Shakespeare: -- cgit v1.2.3