1 files changed, 127 insertions, 0 deletions
diff --git a/q1-prediction-to-reasoning/main.md b/q1-prediction-to-reasoning/main.md
new file mode 100644
index 0000000..ab12bf3
--- /dev/null
+++ b/q1-prediction-to-reasoning/main.md
@@ -0,0 +1,127 @@
+# Q1: Why does next-token prediction produce reasoning?
+
+Parent: `../fundamental-questions.md`
+
+## The bounded version (solved)
+
+For small models on constrained tasks, we CAN answer this. The field has
+fully reverse-engineered several cases:
+
+- **Modular addition** (Nanda et al., Power et al.): 1-2 layer transformers
+  learn Fourier-like circuits. Grokking — the sudden generalization after
+  memorization — corresponds to the formation of sparse, structured weight
+  patterns. Every neuron and attention pattern is understood.
+- **Induction heads** (Olsson et al.): The specific attention pattern that
+  enables in-context learning has been identified, its emergence during
+  training is predictable, and we know why it works.
+- **Othello-GPT** (Li et al.): A small transformer trained on Othello
+  move sequences develops an internal board representation — the world
+  model is localized to specific directions in activation space.
+
+## The open problem
+
+We can't answer whether these mechanisms *scale* to large models, or
+whether something structurally different happens. The toy examples may
+be computational analogs of studying single-celled organisms to
+understand human cognition — informative at the margin, but the target
+system operates on fundamentally different organizational principles.
+
+## Why can't we reverse-engineer models at scale?
+
+### 1. Combinatorial explosion of circuits
+
+A 70B model has ~80 layers, each with hundreds of attention heads and
+thousands of MLP neurons. A single "reasoning" operation doesn't route
+through one identifiable circuit — it decomposes across dozens of
+layers and hundreds of components, and the specific decomposition
+varies by input context. Even if you could isolate one circuit, it
+would be one of millions. Understanding "reasoning" would require
+understanding the interaction of all of them.
+
+### 2. Superposition
+
+Features dramatically outnumber dimensions. A single neuron in a large
+model encodes many unrelated concepts simultaneously, in overlapping,
+non-orthogonal directions. Even sparse autoencoders — the best tool we
+have — recover features that are themselves polysemantic at scale.
+Identifying "the direction for planning" is impossible because there
+isn't one — planning is represented as a distributed pattern across
+many partially-entangled feature directions.
+
+### 3. Dynamic routing
+
+Small model circuits are relatively static. In large models, the same
+parameters participate in different "circuits" depending on context.
+The model dynamically routes computation based on input content —
+attention patterns route information to different MLP neurons depending
+on what's being reasoned about. You can't draw a static circuit diagram
+because the diagram changes with every input.
+
+### 4. Our tools assume linearity
+
+Activation patching, logit lens, probing, and even the J-lens all
+assume (or approximate) that representations behave linearly and
+interventions compose cleanly. At scale, nonlinear interactions between
+features become significant and first-order approximations break down.
+An activation patch that "should" swap concept A for concept B
+according to the linear approximation produces unpredictable effects
+because the nonlinear interactions with other features can't be
+controlled.
+
+### 5. We lack the right abstraction level
+
+Reverse-engineering a CPU by probing individual transistors is
+technically possible but practically useless. The right abstraction
+is the instruction set architecture, registers, and memory hierarchy.
+For large language models, we don't know what the equivalent
+abstraction is. Are "circuits" the right unit? "Features"? Something
+else? The J-space paper proposes one candidate — the global workspace —
+but it's Claude-specific, single-token-only, and only captures
+verbalizable content, not the substrate computation.
+
+### 6. The training process is a black box
+
+We observe weights after training. We don't observe the developmental
+trajectory — the sequence of intermediate computations the model
+learned and discarded, the phase transitions it passed through, the
+circuits that formed and were later subsumed. Understanding why a
+model generalizes requires understanding not just what it is, but
+how it got there. Training dynamics at scale are largely unstudied
+because the compute cost of instrumentation is prohibitive.
+
+## What would "solved" look like?
+
+Not reverse-engineering every circuit. That's the wrong goal. Solved
+would mean:
+
+1. **An abstraction that compresses the computation.** Like the ISA for
+   CPUs — a description at the right level of granularity that explains
+   what computation is happening without specifying every transistor.
+
+2. **Predictive power.** Given architecture + training data description,
+   predict what capabilities emerge at what scale, without training.
+
+3. **Causal manipulability.** Intervene on the system at the abstraction
+   level to add, remove, or modify specific capabilities without
+   affecting others.
+
+The J-space is a candidate for #1. Scaling laws are crude attempts at
+#2. Nobody has #3.
+
+## Sub-questions
+
+These decompose Q1 into tractable pieces:
+
+- Does the mechanism that produces generalization in toy models
+  (structured weight patterns, Fourier circuits) exist in large models,
+  or is it replaced by something else?
+- Is there a phase transition in representational structure that
+  corresponds to the onset of generalization, and is it detectable
+  in model internals *before* behavioral evaluation?
+- Does in-context learning use the same mechanism at all scales
+  (induction heads), or does the mechanism change?
+- What is the right abstraction level for describing computation in
+  large models? Are Anthropic's "features" it? Is the J-space a
+  component of it?
+- Can we predict, from the training data distribution alone, which
+  concepts will be represented in the model, before training?