path: root/q1-prediction-to-reasoning/main.md

# Q1: Why does next-token prediction produce reasoning?

Parent: `../fundamental-questions.md`

## The bounded version (solved)

For small models on constrained tasks, we CAN answer this. The field has
fully reverse-engineered several cases:

- **Modular addition** (Nanda et al., Power et al.): 1-2 layer transformers
  learn Fourier-like circuits. Grokking — the sudden generalization after
  memorization — corresponds to the formation of sparse, structured weight
  patterns. Every neuron and attention pattern is understood.
- **Induction heads** (Olsson et al.): The specific attention pattern that
  enables in-context learning has been identified, its emergence during
  training is predictable, and we know why it works.
- **Othello-GPT** (Li et al.): A small transformer trained on Othello
  move sequences develops an internal board representation — the world
  model is localized to specific directions in activation space.

## The open problem

We can't answer whether these mechanisms *scale* to large models, or
whether something structurally different happens. The toy examples may
be computational analogs of studying single-celled organisms to
understand human cognition — informative at the margin, but the target
system operates on fundamentally different organizational principles.

## Why can't we reverse-engineer models at scale?

### 1. Combinatorial explosion of circuits

A 70B model has ~80 layers, each with hundreds of attention heads and
thousands of MLP neurons. A single "reasoning" operation doesn't route
through one identifiable circuit — it decomposes across dozens of
layers and hundreds of components, and the specific decomposition
varies by input context. Even if you could isolate one circuit, it
would be one of millions. Understanding "reasoning" would require
understanding the interaction of all of them.

### 2. Superposition

Features dramatically outnumber dimensions. A single neuron in a large
model encodes many unrelated concepts simultaneously, in overlapping,
non-orthogonal directions. Even sparse autoencoders — the best tool we
have — recover features that are themselves polysemantic at scale.
Identifying "the direction for planning" is impossible because there
isn't one — planning is represented as a distributed pattern across
many partially-entangled feature directions.

### 3. Dynamic routing

Small model circuits are relatively static. In large models, the same
parameters participate in different "circuits" depending on context.
The model dynamically routes computation based on input content —
attention patterns route information to different MLP neurons depending
on what's being reasoned about. You can't draw a static circuit diagram
because the diagram changes with every input.

### 4. Our tools assume linearity

Activation patching, logit lens, probing, and even the J-lens all
assume (or approximate) that representations behave linearly and
interventions compose cleanly. At scale, nonlinear interactions between
features become significant and first-order approximations break down.
An activation patch that "should" swap concept A for concept B
according to the linear approximation produces unpredictable effects
because the nonlinear interactions with other features can't be
controlled.

### 5. We lack the right abstraction level

Reverse-engineering a CPU by probing individual transistors is
technically possible but practically useless. The right abstraction
is the instruction set architecture, registers, and memory hierarchy.
For large language models, we don't know what the equivalent
abstraction is. Are "circuits" the right unit? "Features"? Something
else? The J-space paper proposes one candidate — the global workspace —
but it's Claude-specific, single-token-only, and only captures
verbalizable content, not the substrate computation.

### 6. The training process is a black box

We observe weights after training. We don't observe the developmental
trajectory — the sequence of intermediate computations the model
learned and discarded, the phase transitions it passed through, the
circuits that formed and were later subsumed. Understanding why a
model generalizes requires understanding not just what it is, but
how it got there. Training dynamics at scale are largely unstudied
because the compute cost of instrumentation is prohibitive.

## What would "solved" look like?

Not reverse-engineering every circuit. That's the wrong goal. Solved
would mean:

1. **An abstraction that compresses the computation.** Like the ISA for
   CPUs — a description at the right level of granularity that explains
   what computation is happening without specifying every transistor.

2. **Predictive power.** Given architecture + training data description,
   predict what capabilities emerge at what scale, without training.

3. **Causal manipulability.** Intervene on the system at the abstraction
   level to add, remove, or modify specific capabilities without
   affecting others.

The J-space is a candidate for #1. Scaling laws are crude attempts at
#2. Nobody has #3.

## Sub-questions

These decompose Q1 into tractable pieces:

- Does the mechanism that produces generalization in toy models
  (structured weight patterns, Fourier circuits) exist in large models,
  or is it replaced by something else?
- Is there a phase transition in representational structure that
  corresponds to the onset of generalization, and is it detectable
  in model internals *before* behavioral evaluation?
- Does in-context learning use the same mechanism at all scales
  (induction heads), or does the mechanism change?
- What is the right abstraction level for describing computation in
  large models? Are Anthropic's "features" it? Is the J-space a
  component of it?
- Can we predict, from the training data distribution alone, which
  concepts will be represented in the model, before training?

## Attack strategies

Concrete approaches that might yield partial answers to Q1 or its
sub-questions.

### Strategy 1: Distill down, reverse-engineer up

**Idea:** Distill a large model's behavior on a target capability into
the smallest model that can still perform it. Reverse-engineer the small
model's circuit. Probe the large model for that circuit via activation
patching or representational similarity.

**What it can tell you:** Whether the large model uses the same mechanism
or something different. If the circuit exists in the large model, the
mechanism scales. If it doesn't, something qualitatively different
happens at scale.

**What it can't tell you:** Whether the small model's circuit is the
*only* circuit the large model uses. The large model may use the
identified circuit plus others. It also doesn't tell you how the
circuit emerged — only that it's present.

**HPMOR critique:** Distillation implants structure. A better variant
is: train the small model from scratch on the same data distribution,
check if the same circuit emerges spontaneously, *then* probe the large
model. This distinguishes "circuit is a natural consequence of loss +
data" from "circuit was put there by the distillation process."

### Strategy 2: Capability phase transitions via developmental snapshots

**Idea:** Train multiple models of varying scale on identical data,
save checkpoints throughout training. Track when specific capabilities
emerge and what changes in representational structure at those moments.
Look for a phase transition — a qualitative change in how the model
organizes information that coincides with capability emergence.

**What it can tell you:** Whether the generalization mechanism is
continuous (same computation, smaller) or discrete (new computation
appears at threshold). If discrete, what the critical scale is and
what structural change enables it.

**What it can't tell you:** Whether the transition point is data-
dependent, architecture-dependent, or universal.

### Strategy 3: Probe for universal representational geometry

**Idea:** Compare concept representations across models of different
architectures trained on similar data. Use representational similarity
metrics to test whether they converge to the same geometry. Extend
Anthropic's J-space technique to open-weight models and compare.

**What it can tell you:** Whether large models converge to a shared
concept structure (implying the geometry is a stable attractor of
next-token prediction) or diverge (implying it's architecture-
contingent).

**What it can't tell you:** Why they converge or diverge — you need
dynamics for that.

### Strategy 4: Mechanistic distillation (embed, don't train)

**Idea:** For a specific narrow capability, directly test whether a
known circuit from a small model can be embedded in a large model
*and whether the large model uses it*. Take a circuit from a small
model, identify its functional signature (activation patterns,
representational geometry), and search for that signature in the
large model at the corresponding layers.

**What it can tell you:** Whether the mechanism scales. Stronger
than Strategy 1 because it doesn't rely on distillation.

**What it can't tell you:** What else the large model is doing
alongside the identified circuit. Also requires the circuit to be
fully reverse-engineered in the small model first, which is its
own research project.

### Strategy 5: Simplify the data, not the model

**Idea:** Instead of training small models on complex data, train
large models on simple data where the "right answer" is known.
Example: train a 1B model on arithmetic only. The model is large
enough to exhibit generalization, but the task is simple enough
that the correct algorithm is known. Reverse-engineer the model's
learned algorithm and compare to the ground truth.

**What it can tell you:** Whether scale enables discovery of the
known-correct algorithm, and at what point. If yes, you've shown
that next-token prediction + sufficient scale converges to the
right computation.

**What it can't tell you:** Whether this generalizes to complex
domains where there is no single "correct algorithm."

### Strategy 6: Necessary condition ablation

**Idea:** For a specific capability, ablate candidate mechanisms
(specific attention patterns, feature directions, J-space
representations) and test whether the capability breaks. Work
backward from "what must be present" to "what is present."

**What it can tell you:** Causal necessity — which internal
components are required for a capability.

**What it can't tell you:** Sufficiency — whether those components
alone produce the capability.

**HPMOR note:** This is the weakest strategy epistemically. Finding
that X is necessary for Y doesn't tell you why X enables Y or what
computation X performs. It's a starting point, not an endpoint.