1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
|
# Review: GPT-5.6-Terra (codex, 2026-07-31)
Adversarial review of the J-space replication project. Full brief: /tmp/jspace_brief.md.
Codex read the repo and the paper (43,704 tokens used, no files changed).
## Q1. Strongest alternative explanation
The central comparison is not currently about Anthropic's J-lens. jlens_v2.py estimates
E[ grad_{h_l,t} log p(k | x, t') ]
summed over future output positions, whereas the paper first estimates the token-
independent matrix E[ ∂h_final,t' / ∂h_l,t ], then applies final normalization and the
unembedding row for token k. That distinction is fatal for the frequency claim.
The gradient of log probability contains a softmax/calibration term:
∇ log p_k = ∇ z_k − Σ_j p_j ∇ z_j
Thus its norm measures the local sensitivity of the *log probability* of token k,
including output-head geometry, prediction confidence, final LayerNorm's state-dependent
Jacobian, and cancellation across contexts. It is not a token's "amount in J-space."
A token can have a small mean gradient because its effects vary in direction by context
and cancel — not because it is less verbalizable or displaced by a capacity limit.
The `q` intervention is especially confounded. It does not merely double a sufficient
statistic while holding the data-generating problem fixed: it inserts `q` at random
character positions, changing sequence length, every downstream absolute position, local
n-grams, and the conditional distribution of both `q` and its neighbors. In Shakespeare,
`q` is unusually structured; random inserted instances largely destroy that structure.
The 67% drop could therefore reflect a different learned conditional-prediction circuit or
gradient alignment, plus an unmatched training run — not frequency per se.
The rank result is even less probative. A matrix of 65 token-indexed vectors in d
dimensions necessarily has rank at most min(65,d). Observing near-maximal numerical rank
under a 1%-of-top-singular-value cutoff shows that these particular gradient vectors are
reasonably nondegenerate; it does not show that the model's workspace capacity equals that
bound. Anthropic's capacity claim is about sparse nonnegative decomposition of
*activations*, occupancy above random-direction controls, and explained variance at
individual positions — not the global linear rank of the token-vector dictionary. The paper
explicitly notes that the token vectors may span all of residual space; J-space is defined
by sparse use, not a low-rank span.
A further comparability problem: nanoGPT computes norm of the mean gradient, while
gpt2_jlens.py averages norms of per-batch full tensors. Those answer different questions,
so the two correlations cannot jointly support one mechanism.
## Q2. Best single experiment
Run a matched, multi-seed **loss-reweighting plus faithful-lens** experiment.
Train paired models from identical initializations and identical minibatch order on the
unchanged Shakespeare sequences. In one member of each pair, multiply cross-entropy terms
whose target is `q` by 2; in the other, use ordinary loss. This changes the effective
target frequency/importance without injecting malformed `q` contexts or shifting all later
positions. Use at least 5-10 paired seeds. Also include a same-total-loss control that
upweights randomly chosen non-`q` target positions.
For each model, compute both:
1. the present score, ||E ∇ log p(q)||; and
2. a faithful token vector from the averaged final-residual Jacobian, followed by the
unembedding as in the paper.
Report q probability, conditional entropy, unembedding-row norm, vector cosine similarity,
and bootstrap confidence intervals. Cheap because it reuses the small-model setup and
directly separates "measurement artifact" from learned representation.
## Q3. Faithfulness of jlens_v2.py
No: it is directionally related to a Jacobian method, but it is not faithful enough to
validate numerical comparisons with the paper.
Good news: the forward hook captures the output of the selected nanoGPT block (the
post-attention/post-MLP residual stream), a reasonable source capture point. Summing
gradients over source positions and all output positions also includes the causal
future-position dependence the paper intends — masked-impossible pairs have zero gradient.
The methodological error is the target. The paper defines one d_model x d_model average
Jacobian from intermediate residual stream to FINAL residual stream, then reads it through
the model's normal output operations. Our code differentiates log_softmax(logits) directly.
That folds the final LayerNorm, unembedding, and token-dependent softmax subtraction into
the object being averaged. Averaging after these nonlinear/token-dependent operations is
not equivalent to averaging the residual Jacobian and then reading it out.
Two lesser issues: dividing by B*T rather than the number of valid source-future pairs
changes scale (though not within-run rankings at fixed sequence length); 10-20 random
batches is a noisier approximation than the paper's corpus-scale averaging (precision, not
core invalidation).
## Q4. Defensible blog framing
"We found that a simple gradient-of-log-probability proxy on a 10.6M character transformer
is strongly associated with token frequency, and that its token-indexed gradient dictionary
has the expected rank ceiling min(V,d_model). A random-insertion intervention is consistent
with frequency or conditional-prediction structure affecting this proxy, but it does not
isolate frequency, and our current estimator differs materially from Anthropic's
residual-Jacobian J-lens. These results motivate a controlled replication using the
faithful lens and activation-level sparse-occupancy tests."
Do NOT claim to have falsified Anthropic's workspace evidence, shown that its capacity
result is "just linear algebra," demonstrated a consciousness-relevant conclusion is wrong,
or shifted any burden of proof. Anthropic's headline rests on functional interventions,
sparse occupancy, variance controls, and broadcast/generalization tests in addition to
rank; our current experiments test none of those.
|
