diff options
Diffstat (limited to 'results.md')
| -rw-r--r-- | results.md | 23 |
1 files changed, 23 insertions, 0 deletions
@@ -46,6 +46,29 @@ dynamics; the mechanism of the latter is not yet pinned down. Top tokens by faithful norm are consistently rare characters (`?`, `z`, `q`, `$`); bottom are common ones (space, `e`, `t`, `i`). +## 1b. Validation against Anthropic's official reference implementation + +Anthropic released companion code (`github.com/anthropics/jacobian-lens`, +Apache-2.0). Their lens is `lens_l(h) = unembed(J_l @ h)` with +`J_l = E[∂h_final/∂h_l]` — the same quantity we compute; our W_U-probed +shortcut (rows of `W_U * J_l`) is mathematically equivalent, verified by the +last-layer identity check (cos-sim 1.0000). + +Their estimator (`jlens/fitting.py`) differs from our default in two details: +1. `valid_position_mask`: excludes the first 16 positions (attention sinks) + and the last position from the average. +2. Averaging: mean over source positions (each position once) instead of our + mean over (source, future) pairs. + +We added `--skip_first N --source_mean` to `src/jlens_v3.py` to mirror their +estimator exactly. The robustness re-run (same model, their estimator choices) +is queued behind the loss-reweighting GPU job; results will be appended here. + +Frequency scan: case-insensitive grep of their entire repo (README, code, +experiment + evaluation data) finds ZERO occurrences of frequency/unigram/ +token-count terms. Their README confirms the fit corpus is "a generic web-text +corpus" — no frequency balancing. + ## 2. Frequency-matched synthetic pair (`src/synthetic_pair.py`) Two new characters at identical 0.1% unigram frequency in Shakespeare: |