From 1b9b346dc88a3c043af2fa6ef1a1c7ca04a7311b Mon Sep 17 00:00:00 2001 From: Void Agent Date: Sun, 2 Aug 2026 14:45:51 +0100 Subject: Verify against Anthropic's official jacobian-lens repo: estimator match, zero frequency in their code+data, add --skip_first/--source_mean mirror mode --- results.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) (limited to 'results.md') diff --git a/results.md b/results.md index 4c1b604..574ee06 100644 --- a/results.md +++ b/results.md @@ -46,6 +46,29 @@ dynamics; the mechanism of the latter is not yet pinned down. Top tokens by faithful norm are consistently rare characters (`?`, `z`, `q`, `$`); bottom are common ones (space, `e`, `t`, `i`). +## 1b. Validation against Anthropic's official reference implementation + +Anthropic released companion code (`github.com/anthropics/jacobian-lens`, +Apache-2.0). Their lens is `lens_l(h) = unembed(J_l @ h)` with +`J_l = E[∂h_final/∂h_l]` — the same quantity we compute; our W_U-probed +shortcut (rows of `W_U * J_l`) is mathematically equivalent, verified by the +last-layer identity check (cos-sim 1.0000). + +Their estimator (`jlens/fitting.py`) differs from our default in two details: +1. `valid_position_mask`: excludes the first 16 positions (attention sinks) + and the last position from the average. +2. Averaging: mean over source positions (each position once) instead of our + mean over (source, future) pairs. + +We added `--skip_first N --source_mean` to `src/jlens_v3.py` to mirror their +estimator exactly. The robustness re-run (same model, their estimator choices) +is queued behind the loss-reweighting GPU job; results will be appended here. + +Frequency scan: case-insensitive grep of their entire repo (README, code, +experiment + evaluation data) finds ZERO occurrences of frequency/unigram/ +token-count terms. Their README confirms the fit corpus is "a generic web-text +corpus" — no frequency balancing. + ## 2. Frequency-matched synthetic pair (`src/synthetic_pair.py`) Two new characters at identical 0.1% unigram frequency in Shakespeare: -- cgit v1.2.3