diff options
| author | Void Agent <void@jayrup.hermes> | 2026-08-02 16:46:43 +0100 |
|---|---|---|
| committer | Void Agent <void@jayrup.hermes> | 2026-08-02 16:46:43 +0100 |
| commit | 7ca6d8bcfd0db4feb89456100ccf766d02d5f258 (patch) | |
| tree | 839907d1d00b98e87e465f48ad5cd5e45b7820a8 | |
| parent | b1c232ebc00eaa115a1009d9a9f477721777ffe8 (diff) | |
Polish per review: results.md GPT-2 -0.18 demoted to do-not-cite footnote; blog section 10 adds W_U-at-init scale prediction (one-line GPT-2 unembedding check)
| -rw-r--r-- | docs/blog-jlens-frequency.md | 6 | ||||
| -rw-r--r-- | results.md | 10 |
2 files changed, 12 insertions, 4 deletions
diff --git a/docs/blog-jlens-frequency.md b/docs/blog-jlens-frequency.md index dec1edf..8ab2e46 100644 --- a/docs/blog-jlens-frequency.md +++ b/docs/blog-jlens-frequency.md @@ -385,6 +385,12 @@ regime where Anthropic's claims live) with proper statistical power, and to run the occupancy test their capacity claim is actually about. That's the next post. +The learned-geometry finding makes one piece of that cheap and decisive: if +the W_U row-norm anti-correlation is a general property of softmax-output +models trained on Zipfian data, it should appear in GPT-2's unembedding matrix +directly — no Jacobian computation at all. One line of code at scale, and it +either generalizes or it doesn't. + ## 11. How to reproduce everything All code, data-prep scripts, experiment scripts, tests, and this analysis live @@ -186,7 +186,9 @@ Reading: - Original proxy finding (r = -0.65, `jlens_v2`): superseded by the faithful implementation; kept only for the both-ways comparison. -- GPT-2 correlation (r = -0.18, `gpt2_jlens.py`): UNDER-POWERED (96 token - positions, n=100 sampled tokens) and computed a different quantity - (norm-per-batch vs norm-of-mean). Directionally consistent but not - publishable evidence on its own. +- GPT-2 correlation: an early directional probe on GPT-2 small + (`gpt2_jlens.py`) gave avg r = -0.18 across layers — UNDER-POWERED + (96 token positions, n=100 sampled tokens) and a different quantity + (norm-per-batch vs norm-of-mean). Kept here only as a reproducibility + warning: **do not cite the -0.18 as evidence**; the at-scale test with the + faithful lens is future work. |
