diff options
Diffstat (limited to 'results.md')
| -rw-r--r-- | results.md | 6 |
1 files changed, 6 insertions, 0 deletions
@@ -43,6 +43,12 @@ the frequency signal lives partly in the static unembedding geometry (which is baked into Anthropic's lens by definition) and partly in layer-dependent dynamics; the mechanism of the latter is not yet pinned down. +W_U geometry is LEARNED, not present at init: fresh models (3 seeds, same +config) show r(||W_U[k]||, freq) = +0.002, -0.189, -0.166 (trained: -0.606), +and row-norm spread grows ~4x during training ([0.36, 0.44] -> [0.86, 1.52]). +The model learns to push rare-token unembedding rows outward and nothing +corrects it. + Top tokens by faithful norm are consistently rare characters (`?`, `z`, `q`, `$`); bottom are common ones (space, `e`, `t`, `i`). |
