summaryrefslogtreecommitdiff
path: root/results.md
diff options
context:
space:
mode:
Diffstat (limited to 'results.md')
-rw-r--r--results.md6
1 files changed, 6 insertions, 0 deletions
diff --git a/results.md b/results.md
index c9e0c23..692ce03 100644
--- a/results.md
+++ b/results.md
@@ -43,6 +43,12 @@ the frequency signal lives partly in the static unembedding geometry (which is
baked into Anthropic's lens by definition) and partly in layer-dependent
dynamics; the mechanism of the latter is not yet pinned down.
+W_U geometry is LEARNED, not present at init: fresh models (3 seeds, same
+config) show r(||W_U[k]||, freq) = +0.002, -0.189, -0.166 (trained: -0.606),
+and row-norm spread grows ~4x during training ([0.36, 0.44] -> [0.86, 1.52]).
+The model learns to push rare-token unembedding rows outward and nothing
+corrects it.
+
Top tokens by faithful norm are consistently rare characters (`?`, `z`, `q`,
`$`); bottom are common ones (space, `e`, `t`, `i`).