diff options
Diffstat (limited to 'results.md')
| -rw-r--r-- | results.md | 39 |
1 files changed, 37 insertions, 2 deletions
@@ -182,6 +182,40 @@ Reading: 4. Absolute norms differ across experiments (base L3 q = 0.0109 vs these 0.015-0.018) — only within-experiment comparisons are meaningful. +## 5. At-scale W_U row-norm check (GPT-2, V=50,257) — RESOLVED + +The geometric half of the confound generalizes. Correlating GPT-2 unembedding +row norms (wte == tied lm_head, so this is exactly W_U's rows) against token +unigram log-frequencies from wikitext-103-raw (117.9M tokens; 46,887/50,257 +vocab covered): + +``` + model r(raw) r(log10, all V) r(log10, seen) Spearman(seen) + gpt2 (d=768) -0.092 -0.451 -0.468 -0.462 + gpt2-medium -0.104 -0.493 -0.515 -0.498 +``` + +n = 46,887 seen tokens; 95% CIs [-0.475, -0.461] (gpt2) and +[-0.522, -0.508] (gpt2-medium). Log10-freq decile -> mean row norm is +monotone in both models (gpt2: 4.02 -> 3.42; medium: 3.93 -> 3.16): rare +tokens get bigger unembedding rows. Top-norm tokens are rare or zero-count +(SPONSORED, soDeliveryDate, Reviewer, advertisement). + +Reading: the learned W_U geometry finding from the 65-char model carries to +real LMs at V = 50,257 — present in two independently trained models, monotone +across the whole vocabulary — but it is WEAKER than at toy scale +(r(log10) -0.45/-0.49 vs -0.69; Spearman -0.46/-0.50 vs -0.81). So the +char-scale result was not a vocabulary-size artifact, and neither is the +at-scale effect as strong as the toy suggested. The layer-dynamics component +(partial r after regressing out W_U) at scale remains untested — that needs +the faithful J-lens on GPT-2 (future work). + +Reproduce, CPU-only, no training, no GPU: +``` +.venv/bin/python src/wu_row_norm_check.py --models gpt2,gpt2-medium +``` +Full log: `outputs/wu_row_norm_check.log`. + ## 4. Historical / do-not-copy - Original proxy finding (r = -0.65, `jlens_v2`): superseded by the faithful @@ -190,5 +224,6 @@ Reading: (`gpt2_jlens.py`) gave avg r = -0.18 across layers — UNDER-POWERED (96 token positions, n=100 sampled tokens) and a different quantity (norm-per-batch vs norm-of-mean). Kept here only as a reproducibility - warning: **do not cite the -0.18 as evidence**; the at-scale test with the - faithful lens is future work. + warning: **do not cite the -0.18 as evidence**; the at-scale W_U geometry + check (section 5) is the powered version of the geometric claim, and the + at-scale test with the faithful lens is future work. |