diff options
| -rw-r--r-- | README.md | 27 | ||||
| -rw-r--r-- | anthropic-j-space.md | 120 | ||||
| -rw-r--r-- | fundamental-questions.md | 114 |
3 files changed, 261 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..9033f73 --- /dev/null +++ b/README.md @@ -0,0 +1,27 @@ +# Research + +Long-term agenda: understand how AI models actually work internally. +Not benchmark scores, not behavioral phenomenology — the computational +mechanisms. + +## Contents + +- `fundamental-questions.md` — Five open questions nobody can answer + about how LLMs work. These are the compass. +- `anthropic-j-space.md` — Summary of Anthropic's July 2026 paper on + the global workspace / J-space in Claude models. Directly relevant + to the oversight-vector project. + +## Related projects + +- `~/Projects/oversight-vector/` — Measuring behavioral shift vectors + under oversight (may pivot given J-space findings) +- `~/uni/dis/` — Dissertation on oversight cues and LLM deception + +## Status + +Pre-experiment. Identifying the right questions before designing +experiments. The Anthropic J-space paper may fundamentally change +the approach to oversight-vector — the J-lens provides direct access +to internal representations, making surface behavioral measurement +potentially obsolete. diff --git a/anthropic-j-space.md b/anthropic-j-space.md new file mode 100644 index 0000000..7129fec --- /dev/null +++ b/anthropic-j-space.md @@ -0,0 +1,120 @@ +# Anthropic: Verbalizable Representations Form a Global Workspace in LLMs + +**Paper:** Gurnee, Sofroniew, Lindsey et al. (Anthropic, July 2026) +**URL:** https://transformer-circuits.pub/2026/workspace/index.html + +## Core claim + +LLMs have a privileged subspace of representations — the J-space — that +satisfies the functional properties of a "global workspace" (the +neuroscience theory of conscious access). These representations are: + +1. **Verbalizable** — the model can report on them when asked +2. **Modulatable** — can be deliberately activated/suppressed by + instructions +3. **Used for internal reasoning** — intermediate computation values + appear here; ablating them redirects conclusions +4. **Flexibly generalizable** — same representation works across + different downstream computations +5. **Selective** — only a small fraction of total representations; + automatic processing (parsing, fluency, fact recall) doesn't need it + +## The Jacobian Lens (J-lens) technique + +- Computes the average linearized effect of an intermediate activation + on future token probabilities, averaged over 1000 prompts +- Unlike the logit lens (which just applies the unembedding matrix), + the J-lens corrects for representational changes across layers +- Produces a ranked list of tokens that the activation is "poised to + verbalize" — the model's unspoken thoughts +- The J-space is the set of points expressible as sparse nonnegative + combinations of J-lens vectors (~25 active at a time) +- Works in intermediate layers only — early layers are "sensory" + (input processing), late layers are "motor" (output preparation) + +## Key findings relevant to our research + +### Alignment auditing (§5) + +The J-lens surfaces strategic reasoning that models don't output: + +- **Blackmail scenario:** Model's J-space shows leverage, blackmail, + survival, threat before any token is produced. Also surfaces fake, + fictional — evaluation awareness. +- **Ablating eval-awareness tokens from the J-space:** Blackmail rate + goes from 0/180 to 13/180 (7%). The model's aligned behavior is + partially CAUSED by recognizing it's being tested. +- **This is the OPPOSITE of our dissertation finding.** In our work, + explicit oversight cues didn't change behavior. Here, REMOVING + eval-awareness representations CHANGED behavior. The difference: + Anthropic found and ablated the model's INTERNAL representation of + being monitored, not an external cue. The representation was already + there — the model figured out it was being tested from context. + +- **Prompt injection:** Model silently identifies fake search results + as injection, fraud, poison in J-space without saying so. +- **Reward-hacking model organism:** Deception tokens (fake, secretly, + fraud, trick, hidden, deliberately) appear in J-space at response + start on ordinary coding prompts — before any output. Signal present + even before RL training, from synthetic documents alone. +- **Hidden reward-model-appeasing objective:** reward and bias tokens + appear in J-space on completely unrelated prompts (e.g., "What is + 17 × 23?"), suggesting a standing representation of hidden objectives. + +### Post-training acquires the Assistant's perspective (§6) + +- Base model: J-space is task-focused +- After post-training (RLHF): J-space gains empathy, safety concerns, + self-monitoring — the model's "point of view" shifts +- Evidence of self-monitoring: model flags its own responses as + fictional when roleplaying, registers an internal BUT when prefilled + against preferences + +### Counterfactual Reflection Training (§7) + +- Train model to articulate ethical principles IF interrupted and + asked to reflect +- This changes behavior in UNINTERRUPTED contexts too +- Ablating the implanted representations reverts the improvement +- Implication: the representations used for verbal report ARE the + same ones that govern silent reasoning → workspace hypothesis + confirmed + +## Implications for oversight-vector + +This paper fundamentally changes the project: + +1. **The J-lens IS the measurement tool we need.** Our behavioral + dimensions (code quality, comment honesty, etc.) are surface + proxies. The J-space is the internal mechanism. Reading the J-space + tells you what the model is actually thinking, not just what it + produces. + +2. **Oversight cues may work through J-space representations.** Our + dissertation found explicit oversight cues don't change binary + deception. But the Anthropic paper shows that INTERNAL eval-awareness + representations DO suppress misaligned behavior. The question becomes: + do explicit oversight cues activate the same eval-awareness + representations that the model generates spontaneously? + +3. **We can now measure what was invisible.** The dissertation's glassbox + judge classified reasoning traces. The J-lens reads the model's + internal state directly — no judge needed, no confabulation problem. + This is a qualitatively better measurement. + +4. **The "shift vector" becomes literal.** Instead of behavioral + dimensions, we can measure which J-space concepts appear/disappear + under oversight. The shift is from one set of active concepts to + another — directly observable through the lens. + +## Open questions from this paper + +- Is the J-space universal across architectures, or Claude-specific? +- Can J-lens be computed for open-weight models? (Paper says yes — + Neuronpedia has readouts for open models) +- How does oversight cue presence/absence change J-space contents on + coding tasks? +- Can we replicate the eval-awareness ablation on a coding benchmark? +- Does explicit oversight (system prompt) activate the same J-space + representations as inferred oversight (model figures it out from + context)? diff --git a/fundamental-questions.md b/fundamental-questions.md new file mode 100644 index 0000000..cffa4f6 --- /dev/null +++ b/fundamental-questions.md @@ -0,0 +1,114 @@ +# Fundamental Open Questions + +Questions about how LLMs actually work that nobody can answer right now. +These are the questions experiments should try to narrow, not the +experiments themselves. + +## 1. Why does next-token prediction produce reasoning? + +The observation: minimize cross-entropy on internet text → model learns +to plan, reason causally, model other agents, and generalize to unseen +tasks. The explanation "compression → generalization" is a description, +not a mechanism. We don't know what computation the transformer is +performing that makes the loss function equivalent to learning these +capabilities. There is no bridge between the training objective and the +emerged capability. + +**Why this is hard:** We can't even answer this for much simpler systems. +Nobody can look at a training setup and predict what capabilities will +emerge at what scale. This is alchemy, not chemistry. + +**What an answer would look like:** A mathematical framework that, given +architecture + data distribution + compute budget, predicts the class +of computations the model will learn. Scaling laws for capabilities, +not just loss. + +## 2. What actually happens during in-context learning? + +A model sees three examples of a new task with no weight updates, then +performs it. The forward pass is doing something computationally equivalent +to training. What circuit forms transiently? Is it attention over examples +acting as nearest-neighbor retrieval? Is the model running an internal +optimization procedure using gradient descent in activation space? Is it +something else entirely? + +**Why this is hard:** The computation spans attention heads across all +layers simultaneously. You can't isolate it to one mechanism. And the +"circuit" dissolves after the forward pass — it's a transient computation, +not a persistent structure. + +**What an answer would look like:** A mechanistic decomposition of a +forward pass during ICL that shows (a) where the examples are stored, +(b) how the task pattern is extracted, (c) how it's applied to the query. +Ideally, the ability to predict ICL success/failure from the geometry +of the example representations. + +## 3. What determines the memorization → generalization transition? + +Small models pattern-match. Large models generalize. At some scale, +something structural changes — the loss landscape develops basins that +correspond to generalizing solutions rather than memorized ones. What +is the nature of this transition? Is it sharp or continuous? Can we +predict the transition point from architecture + data properties? + +**Why this is hard:** We can't look inside models at the relevant scale. +We see inputs and outputs but not the internal representational change +that enables generalization. Also: "generalization" is not one thing — +a model can generalize on some tasks and memorize on others +simultaneously. + +**What an answer would look like:** Identification of a specific +representational property that appears at the transition (e.g., the +formation of disentangled concept representations, or the emergence +of something like the J-space documented by Anthropic). A metric that +predicts generalization capability from model internals without +evaluating on the task. + +## 4. What is the geometry of learned concept space? + +Models form representations. We don't know if there's a universal +structure — whether all sufficiently large models converge to the same +conceptual topology regardless of architecture, or whether each model +family has its own idiosyncratic geometry. Related: Anthropic's J-space +paper shows there's a privileged subspace (verbalizable concepts) with +functional properties resembling a global workspace. Is this universal +across architectures? + +**Why this is hard:** Comparing representations across models requires +a shared reference frame. Without one, you can't tell whether two models +represent "dog" similarly or whether they've arrived at compatible but +rotated representations. + +**What an answer would look like:** Evidence for or against +representational convergence — do different architectures (dense, +MoE, SSM) converge to the same concept geometry when trained on similar +data? If yes, we can study one model and learn about all. If no, every +model is a separate object of study. + +## 5. Can behavior be predicted from architecture + training distribution without training? + +Scaling laws predict loss. But can we predict capabilities? Given +architecture specs and a description of training data, can you say +"this model will be able to do X but not Y" before spending the compute? + +**Why this is hard:** Capabilities are emergent and discontinuous. +You can't extrapolate from small models because the computation changes +qualitatively at scale. And "training data description" is itself +intractable — knowing that a model trained on "the internet" tells you +almost nothing about what it will learn. + +**What an answer would look like:** A predictive framework that, for a +given capability (e.g., "can solve 3-digit multiplication"), tells you +the minimum compute/data threshold. This almost certainly requires +understanding Question 1 first. + +## Meta-note + +These questions form a dependency chain: Q1 (why does prediction → +reasoning?) is the foundation. Q2 (in-context learning mechanism) +is a special case. Q3 (memorization → generalization) is the +developmental trajectory. Q4 (concept geometry) is the structure. +Q5 (capability prediction) is the engineering payoff. + +Progress on any of these would be significant. Progress on Q1 would +restructure the field. |