at-scale W_U row-norm check (GPT-2): geometric confound generalizes, r(log10)≈-0.45/-0.49; blog draft: drop 'Actually', log-freq lead, probe transparency, norm-vs-readout concession

1 files changed, 37 insertions, 2 deletions

diff --git a/results.md b/results.md
index 7acbc56..68422db 100644
--- a/results.md
+++ b/results.md
@@ -182,6 +182,40 @@ Reading:
 4. Absolute norms differ across experiments (base L3 q = 0.0109 vs these
    0.015-0.018) — only within-experiment comparisons are meaningful.
 
+## 5. At-scale W_U row-norm check (GPT-2, V=50,257) — RESOLVED
+
+The geometric half of the confound generalizes. Correlating GPT-2 unembedding
+row norms (wte == tied lm_head, so this is exactly W_U's rows) against token
+unigram log-frequencies from wikitext-103-raw (117.9M tokens; 46,887/50,257
+vocab covered):
+
+```
+  model           r(raw)   r(log10, all V)   r(log10, seen)   Spearman(seen)
+  gpt2 (d=768)    -0.092   -0.451            -0.468           -0.462
+  gpt2-medium     -0.104   -0.493            -0.515           -0.498
+```
+
+n = 46,887 seen tokens; 95% CIs [-0.475, -0.461] (gpt2) and
+[-0.522, -0.508] (gpt2-medium). Log10-freq decile -> mean row norm is
+monotone in both models (gpt2: 4.02 -> 3.42; medium: 3.93 -> 3.16): rare
+tokens get bigger unembedding rows. Top-norm tokens are rare or zero-count
+(SPONSORED, soDeliveryDate, Reviewer, advertisement).
+
+Reading: the learned W_U geometry finding from the 65-char model carries to
+real LMs at V = 50,257 — present in two independently trained models, monotone
+across the whole vocabulary — but it is WEAKER than at toy scale
+(r(log10) -0.45/-0.49 vs -0.69; Spearman -0.46/-0.50 vs -0.81). So the
+char-scale result was not a vocabulary-size artifact, and neither is the
+at-scale effect as strong as the toy suggested. The layer-dynamics component
+(partial r after regressing out W_U) at scale remains untested — that needs
+the faithful J-lens on GPT-2 (future work).
+
+Reproduce, CPU-only, no training, no GPU:
+```
+.venv/bin/python src/wu_row_norm_check.py --models gpt2,gpt2-medium
+```
+Full log: `outputs/wu_row_norm_check.log`.
+
 ## 4. Historical / do-not-copy
 
 - Original proxy finding (r = -0.65, `jlens_v2`): superseded by the faithful
@@ -190,5 +224,6 @@ Reading:
   (`gpt2_jlens.py`) gave avg r = -0.18 across layers — UNDER-POWERED
   (96 token positions, n=100 sampled tokens) and a different quantity
   (norm-per-batch vs norm-of-mean). Kept here only as a reproducibility
-  warning: **do not cite the -0.18 as evidence**; the at-scale test with the
-  faithful lens is future work.
+  warning: **do not cite the -0.18 as evidence**; the at-scale W_U geometry
+  check (section 5) is the powered version of the geometric claim, and the
+  at-scale test with the faithful lens is future work.