From 65e03aff79fc2b1b78a500153d14c5532c4ef78f Mon Sep 17 00:00:00 2001 From: Void Agent Date: Wed, 5 Aug 2026 17:10:54 +0100 Subject: at-scale W_U row-norm check (GPT-2): geometric confound generalizes, r(log10)≈-0.45/-0.49; blog draft: drop 'Actually', log-freq lead, probe transparency, norm-vs-readout concession MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- results.md | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) (limited to 'results.md') diff --git a/results.md b/results.md index 7acbc56..68422db 100644 --- a/results.md +++ b/results.md @@ -182,6 +182,40 @@ Reading: 4. Absolute norms differ across experiments (base L3 q = 0.0109 vs these 0.015-0.018) — only within-experiment comparisons are meaningful. +## 5. At-scale W_U row-norm check (GPT-2, V=50,257) — RESOLVED + +The geometric half of the confound generalizes. Correlating GPT-2 unembedding +row norms (wte == tied lm_head, so this is exactly W_U's rows) against token +unigram log-frequencies from wikitext-103-raw (117.9M tokens; 46,887/50,257 +vocab covered): + +``` + model r(raw) r(log10, all V) r(log10, seen) Spearman(seen) + gpt2 (d=768) -0.092 -0.451 -0.468 -0.462 + gpt2-medium -0.104 -0.493 -0.515 -0.498 +``` + +n = 46,887 seen tokens; 95% CIs [-0.475, -0.461] (gpt2) and +[-0.522, -0.508] (gpt2-medium). Log10-freq decile -> mean row norm is +monotone in both models (gpt2: 4.02 -> 3.42; medium: 3.93 -> 3.16): rare +tokens get bigger unembedding rows. Top-norm tokens are rare or zero-count +(SPONSORED, soDeliveryDate, Reviewer, advertisement). + +Reading: the learned W_U geometry finding from the 65-char model carries to +real LMs at V = 50,257 — present in two independently trained models, monotone +across the whole vocabulary — but it is WEAKER than at toy scale +(r(log10) -0.45/-0.49 vs -0.69; Spearman -0.46/-0.50 vs -0.81). So the +char-scale result was not a vocabulary-size artifact, and neither is the +at-scale effect as strong as the toy suggested. The layer-dynamics component +(partial r after regressing out W_U) at scale remains untested — that needs +the faithful J-lens on GPT-2 (future work). + +Reproduce, CPU-only, no training, no GPU: +``` +.venv/bin/python src/wu_row_norm_check.py --models gpt2,gpt2-medium +``` +Full log: `outputs/wu_row_norm_check.log`. + ## 4. Historical / do-not-copy - Original proxy finding (r = -0.65, `jlens_v2`): superseded by the faithful @@ -190,5 +224,6 @@ Reading: (`gpt2_jlens.py`) gave avg r = -0.18 across layers — UNDER-POWERED (96 token positions, n=100 sampled tokens) and a different quantity (norm-per-batch vs norm-of-mean). Kept here only as a reproducibility - warning: **do not cite the -0.18 as evidence**; the at-scale test with the - faithful lens is future work. + warning: **do not cite the -0.18 as evidence**; the at-scale W_U geometry + check (section 5) is the powered version of the geometric claim, and the + at-scale test with the faithful lens is future work. -- cgit v1.2.3