From 65e03aff79fc2b1b78a500153d14c5532c4ef78f Mon Sep 17 00:00:00 2001 From: Void Agent Date: Wed, 5 Aug 2026 17:10:54 +0100 Subject: at-scale W_U row-norm check (GPT-2): geometric confound generalizes, r(log10)โ‰ˆ-0.45/-0.49; blog draft: drop 'Actually', log-freq lead, probe transparency, norm-vs-readout concession MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) (limited to 'README.md') diff --git a/README.md b/README.md index ae9ba86..de56776 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,11 @@ Workspace in Language Models"* (2026, 4. A causal loss-reweighting test (2x loss weight on 'q' targets vs two controls) tests whether effective frequency causally demotes a token's J-lens norm. See `results.md` for the latest numbers. +5. The geometric half generalizes: GPT-2's unembedding row norms (V = 50,257, + wte == tied lm_head) anti-correlate with token log-frequency + (r โ‰ˆ -0.45/-0.49, gpt2/gpt2-medium; monotone across frequency deciles) โ€” + the learned W_U geometry is not a 65-char vocabulary artifact + (`src/wu_row_norm_check.py`, CPU-only). See `docs/blog-jlens-frequency.md` for the write-up and `results.md` for the numbers. The three independent adversarial reviews that shaped the project @@ -44,6 +49,7 @@ src/jlens_v3.py FAITHFUL J-lens: rows of W_U * J_l (canonical) src/synthetic_pair.py frequency-matched synthetic pair experiment src/loss_reweight.py causal loss-reweighting experiment src/gpt2_jlens.py GPT-2 scale test (under-powered; see results.md) +src/wu_row_norm_check.py at-scale W_U row-norm check (CPU-only; results.md ยง5) tests/ unit tests (see scripts/test.sh) scripts/test.sh canonical test command docs/blog-jlens-frequency.md write-up (Feynman-style) -- cgit v1.2.3