diff options
| -rw-r--r-- | docs/blog-jlens-frequency.md | 5 | ||||
| -rw-r--r-- | results.md | 18 |
2 files changed, 19 insertions, 4 deletions
diff --git a/docs/blog-jlens-frequency.md b/docs/blog-jlens-frequency.md index 4b2b429..34ce5fa 100644 --- a/docs/blog-jlens-frequency.md +++ b/docs/blog-jlens-frequency.md @@ -131,8 +131,9 @@ mathematically equivalent (verified by the identity check above). Their estimator has two differences of detail: it excludes the first 16 positions (attention sinks) and the last position from the average, and it averages over source positions rather than (source, future) pairs. We re-ran our analysis -with their exact estimator choices to confirm the correlation is robust to -those choices — see Section 6.) +with their exact estimator choices: the frequency correlation is essentially +identical at every layer (max delta 0.008, see results.md section 1b), so the +result is robust to those choices.) (One technical note: we capture the residual stream *before* the model's final layer norm. That matches the paper's definition — the Jacobian stops at the @@ -61,8 +61,22 @@ Their estimator (`jlens/fitting.py`) differs from our default in two details: mean over (source, future) pairs. We added `--skip_first N --source_mean` to `src/jlens_v3.py` to mirror their -estimator exactly. The robustness re-run (same model, their estimator choices) -is queued behind the loss-reweighting GPU job; results will be appended here. +estimator exactly and re-ran the same model (outputs/jlens_v3_anthropic/). +Result: the frequency correlation is essentially IDENTICAL under their +estimator at every layer (max delta 0.008): + +``` + Layer default r anthropic r delta + L0 -0.643 -0.651 -0.008 + L1 -0.668 -0.669 -0.000 + L2 -0.672 -0.673 -0.001 + L3 -0.685 -0.687 -0.002 + L4 -0.637 -0.635 +0.002 + L5 -0.606 -0.606 +0.000 +``` + +The correlation is robust to the estimator's averaging choices (position mask +and normalization). Last-layer identity check passed again (cos-sim 1.0000). Frequency scan: case-insensitive grep of their entire repo (README, code, experiment + evaluation data) finds ZERO occurrences of frequency/unigram/ |