diff options
Diffstat (limited to 'results.md')
| -rw-r--r-- | results.md | 24 |
1 files changed, 24 insertions, 0 deletions
@@ -69,6 +69,30 @@ experiment + evaluation data) finds ZERO occurrences of frequency/unigram/ token-count terms. Their README confirms the fit corpus is "a generic web-text corpus" — no frequency balancing. +## 1c. Independent corroboration (willkn, GreaterWrong, GPT-2-medium) + +An independent engineering analysis of the paper (willkn, GreaterWrong, 24 Jul +2026; notebooks: github.com/willkn/jlens_re; full text: docs/willkn-jlens- +greaterwrong-analysis.md), on GPT-2-medium (355M, d=1024, L=24, V=50,257), +found: +- the raw fitted Jacobian "misweights structural tokens (grammar, punctuation) + over semantic content" — dominant spectral channels carry ~10x the gain of + the residual pathway. Structural tokens ARE the high-frequency tokens; this + is the same distortion we measure as frequency anti-correlation, at 30x our + model scale and via a different metric (next-token faithfulness of the + readout). +- the Jacobian is essentially full-rank: 562-858 dimensions capture 90% of + spectral variance, "leaving little room for compression" — corroborates our + dimensional-starvation / full-rank result at scale. +- a single-parameter shrinkage regularizer (J + lambda*I) monotonically + recovers next-token faithfulness (beats logit lens at layer 12: 0.294 vs + 0.275). Anthropic's released fitting code applies no such regularization. +- measured noise law rel_F(J_n) ~ c_l/sqrt(n), c_l growing ~4x from layer 20 + to layer 4; per-backward cost ~5ms + 9ms/layer on L4 at dim_batch=8. + +Untested (natural next experiment): does shrinkage also remove our frequency +correlation? + ## 2. Frequency-matched synthetic pair (`src/synthetic_pair.py`) Two new characters at identical 0.1% unigram frequency in Shakespeare: |