diff options
| author | Void Agent <void@jayrup.hermes> | 2026-08-02 16:43:21 +0100 |
|---|---|---|
| committer | Void Agent <void@jayrup.hermes> | 2026-08-02 16:43:21 +0100 |
| commit | b1c232ebc00eaa115a1009d9a9f477721777ffe8 (patch) | |
| tree | bc01fe0b9480f56efcb099c45eaf8d451f30a98e /results.md | |
| parent | c201b8331c87294911eb3d5d07627cae8429cb5d (diff) | |
Blog revision per reviewer feedback: de-emphasize reviewer name-drops, promote LN technical note, willkn as Related-work, table footnote, section 8 power reframe (~13 seeds/arm needed); add W_U-at-init finding (learned geometry, r~0 at init vs -0.61 trained)
Diffstat (limited to 'results.md')
| -rw-r--r-- | results.md | 6 |
1 files changed, 6 insertions, 0 deletions
@@ -43,6 +43,12 @@ the frequency signal lives partly in the static unembedding geometry (which is baked into Anthropic's lens by definition) and partly in layer-dependent dynamics; the mechanism of the latter is not yet pinned down. +W_U geometry is LEARNED, not present at init: fresh models (3 seeds, same +config) show r(||W_U[k]||, freq) = +0.002, -0.189, -0.166 (trained: -0.606), +and row-norm spread grows ~4x during training ([0.36, 0.44] -> [0.86, 1.52]). +The model learns to push rare-token unembedding rows outward and nothing +corrects it. + Top tokens by faithful norm are consistently rare characters (`?`, `z`, `q`, `$`); bottom are common ones (space, `e`, `t`, `i`). |
