# Q1: Why does next-token prediction produce reasoning? Parent: `../fundamental-questions.md` ## The bounded version (solved) For small models on constrained tasks, we CAN answer this. The field has fully reverse-engineered several cases: - **Modular addition** (Nanda et al., Power et al.): 1-2 layer transformers learn Fourier-like circuits. Grokking — the sudden generalization after memorization — corresponds to the formation of sparse, structured weight patterns. Every neuron and attention pattern is understood. - **Induction heads** (Olsson et al.): The specific attention pattern that enables in-context learning has been identified, its emergence during training is predictable, and we know why it works. - **Othello-GPT** (Li et al.): A small transformer trained on Othello move sequences develops an internal board representation — the world model is localized to specific directions in activation space. ## The open problem We can't answer whether these mechanisms *scale* to large models, or whether something structurally different happens. The toy examples may be computational analogs of studying single-celled organisms to understand human cognition — informative at the margin, but the target system operates on fundamentally different organizational principles. ## Why can't we reverse-engineer models at scale? ### 1. Combinatorial explosion of circuits A 70B model has ~80 layers, each with hundreds of attention heads and thousands of MLP neurons. A single "reasoning" operation doesn't route through one identifiable circuit — it decomposes across dozens of layers and hundreds of components, and the specific decomposition varies by input context. Even if you could isolate one circuit, it would be one of millions. Understanding "reasoning" would require understanding the interaction of all of them. ### 2. Superposition Features dramatically outnumber dimensions. A single neuron in a large model encodes many unrelated concepts simultaneously, in overlapping, non-orthogonal directions. Even sparse autoencoders — the best tool we have — recover features that are themselves polysemantic at scale. Identifying "the direction for planning" is impossible because there isn't one — planning is represented as a distributed pattern across many partially-entangled feature directions. ### 3. Dynamic routing Small model circuits are relatively static. In large models, the same parameters participate in different "circuits" depending on context. The model dynamically routes computation based on input content — attention patterns route information to different MLP neurons depending on what's being reasoned about. You can't draw a static circuit diagram because the diagram changes with every input. ### 4. Our tools assume linearity Activation patching, logit lens, probing, and even the J-lens all assume (or approximate) that representations behave linearly and interventions compose cleanly. At scale, nonlinear interactions between features become significant and first-order approximations break down. An activation patch that "should" swap concept A for concept B according to the linear approximation produces unpredictable effects because the nonlinear interactions with other features can't be controlled. ### 5. We lack the right abstraction level Reverse-engineering a CPU by probing individual transistors is technically possible but practically useless. The right abstraction is the instruction set architecture, registers, and memory hierarchy. For large language models, we don't know what the equivalent abstraction is. Are "circuits" the right unit? "Features"? Something else? The J-space paper proposes one candidate — the global workspace — but it's Claude-specific, single-token-only, and only captures verbalizable content, not the substrate computation. ### 6. The training process is a black box We observe weights after training. We don't observe the developmental trajectory — the sequence of intermediate computations the model learned and discarded, the phase transitions it passed through, the circuits that formed and were later subsumed. Understanding why a model generalizes requires understanding not just what it is, but how it got there. Training dynamics at scale are largely unstudied because the compute cost of instrumentation is prohibitive. ## What would "solved" look like? Not reverse-engineering every circuit. That's the wrong goal. Solved would mean: 1. **An abstraction that compresses the computation.** Like the ISA for CPUs — a description at the right level of granularity that explains what computation is happening without specifying every transistor. 2. **Predictive power.** Given architecture + training data description, predict what capabilities emerge at what scale, without training. 3. **Causal manipulability.** Intervene on the system at the abstraction level to add, remove, or modify specific capabilities without affecting others. The J-space is a candidate for #1. Scaling laws are crude attempts at #2. Nobody has #3. ## Sub-questions These decompose Q1 into tractable pieces: - Does the mechanism that produces generalization in toy models (structured weight patterns, Fourier circuits) exist in large models, or is it replaced by something else? - Is there a phase transition in representational structure that corresponds to the onset of generalization, and is it detectable in model internals *before* behavioral evaluation? - Does in-context learning use the same mechanism at all scales (induction heads), or does the mechanism change? - What is the right abstraction level for describing computation in large models? Are Anthropic's "features" it? Is the J-space a component of it? - Can we predict, from the training data distribution alone, which concepts will be represented in the model, before training? ## Attack strategies Concrete approaches that might yield partial answers to Q1 or its sub-questions. ### Strategy 1: Distill down, reverse-engineer up **Idea:** Distill a large model's behavior on a target capability into the smallest model that can still perform it. Reverse-engineer the small model's circuit. Probe the large model for that circuit via activation patching or representational similarity. **What it can tell you:** Whether the large model uses the same mechanism or something different. If the circuit exists in the large model, the mechanism scales. If it doesn't, something qualitatively different happens at scale. **What it can't tell you:** Whether the small model's circuit is the *only* circuit the large model uses. The large model may use the identified circuit plus others. It also doesn't tell you how the circuit emerged — only that it's present. **HPMOR critique:** Distillation implants structure. A better variant is: train the small model from scratch on the same data distribution, check if the same circuit emerges spontaneously, *then* probe the large model. This distinguishes "circuit is a natural consequence of loss + data" from "circuit was put there by the distillation process." ### Strategy 2: Capability phase transitions via developmental snapshots **Idea:** Train multiple models of varying scale on identical data, save checkpoints throughout training. Track when specific capabilities emerge and what changes in representational structure at those moments. Look for a phase transition — a qualitative change in how the model organizes information that coincides with capability emergence. **What it can tell you:** Whether the generalization mechanism is continuous (same computation, smaller) or discrete (new computation appears at threshold). If discrete, what the critical scale is and what structural change enables it. **What it can't tell you:** Whether the transition point is data- dependent, architecture-dependent, or universal. ### Strategy 3: Probe for universal representational geometry **Idea:** Compare concept representations across models of different architectures trained on similar data. Use representational similarity metrics to test whether they converge to the same geometry. Extend Anthropic's J-space technique to open-weight models and compare. **What it can tell you:** Whether large models converge to a shared concept structure (implying the geometry is a stable attractor of next-token prediction) or diverge (implying it's architecture- contingent). **What it can't tell you:** Why they converge or diverge — you need dynamics for that. ### Strategy 4: Mechanistic distillation (embed, don't train) **Idea:** For a specific narrow capability, directly test whether a known circuit from a small model can be embedded in a large model *and whether the large model uses it*. Take a circuit from a small model, identify its functional signature (activation patterns, representational geometry), and search for that signature in the large model at the corresponding layers. **What it can tell you:** Whether the mechanism scales. Stronger than Strategy 1 because it doesn't rely on distillation. **What it can't tell you:** What else the large model is doing alongside the identified circuit. Also requires the circuit to be fully reverse-engineered in the small model first, which is its own research project. ### Strategy 5: Simplify the data, not the model **Idea:** Instead of training small models on complex data, train large models on simple data where the "right answer" is known. Example: train a 1B model on arithmetic only. The model is large enough to exhibit generalization, but the task is simple enough that the correct algorithm is known. Reverse-engineer the model's learned algorithm and compare to the ground truth. **What it can tell you:** Whether scale enables discovery of the known-correct algorithm, and at what point. If yes, you've shown that next-token prediction + sufficient scale converges to the right computation. **What it can't tell you:** Whether this generalizes to complex domains where there is no single "correct algorithm." ### Strategy 6: Necessary condition ablation **Idea:** For a specific capability, ablate candidate mechanisms (specific attention patterns, feature directions, J-space representations) and test whether the capability breaks. Work backward from "what must be present" to "what is present." **What it can tell you:** Causal necessity — which internal components are required for a capability. **What it can't tell you:** Sufficiency — whether those components alone produce the capability. **HPMOR note:** This is the weakest strategy epistemically. Finding that X is necessary for Y doesn't tell you why X enables Y or what computation X performs. It's a starting point, not an endpoint.