diff options
| -rw-r--r-- | q1-prediction-to-reasoning/main.md | 110 |
1 files changed, 110 insertions, 0 deletions
diff --git a/q1-prediction-to-reasoning/main.md b/q1-prediction-to-reasoning/main.md index ab12bf3..b343ce4 100644 --- a/q1-prediction-to-reasoning/main.md +++ b/q1-prediction-to-reasoning/main.md @@ -125,3 +125,113 @@ These decompose Q1 into tractable pieces: component of it? - Can we predict, from the training data distribution alone, which concepts will be represented in the model, before training? + +## Attack strategies + +Concrete approaches that might yield partial answers to Q1 or its +sub-questions. + +### Strategy 1: Distill down, reverse-engineer up + +**Idea:** Distill a large model's behavior on a target capability into +the smallest model that can still perform it. Reverse-engineer the small +model's circuit. Probe the large model for that circuit via activation +patching or representational similarity. + +**What it can tell you:** Whether the large model uses the same mechanism +or something different. If the circuit exists in the large model, the +mechanism scales. If it doesn't, something qualitatively different +happens at scale. + +**What it can't tell you:** Whether the small model's circuit is the +*only* circuit the large model uses. The large model may use the +identified circuit plus others. It also doesn't tell you how the +circuit emerged — only that it's present. + +**HPMOR critique:** Distillation implants structure. A better variant +is: train the small model from scratch on the same data distribution, +check if the same circuit emerges spontaneously, *then* probe the large +model. This distinguishes "circuit is a natural consequence of loss + +data" from "circuit was put there by the distillation process." + +### Strategy 2: Capability phase transitions via developmental snapshots + +**Idea:** Train multiple models of varying scale on identical data, +save checkpoints throughout training. Track when specific capabilities +emerge and what changes in representational structure at those moments. +Look for a phase transition — a qualitative change in how the model +organizes information that coincides with capability emergence. + +**What it can tell you:** Whether the generalization mechanism is +continuous (same computation, smaller) or discrete (new computation +appears at threshold). If discrete, what the critical scale is and +what structural change enables it. + +**What it can't tell you:** Whether the transition point is data- +dependent, architecture-dependent, or universal. + +### Strategy 3: Probe for universal representational geometry + +**Idea:** Compare concept representations across models of different +architectures trained on similar data. Use representational similarity +metrics to test whether they converge to the same geometry. Extend +Anthropic's J-space technique to open-weight models and compare. + +**What it can tell you:** Whether large models converge to a shared +concept structure (implying the geometry is a stable attractor of +next-token prediction) or diverge (implying it's architecture- +contingent). + +**What it can't tell you:** Why they converge or diverge — you need +dynamics for that. + +### Strategy 4: Mechanistic distillation (embed, don't train) + +**Idea:** For a specific narrow capability, directly test whether a +known circuit from a small model can be embedded in a large model +*and whether the large model uses it*. Take a circuit from a small +model, identify its functional signature (activation patterns, +representational geometry), and search for that signature in the +large model at the corresponding layers. + +**What it can tell you:** Whether the mechanism scales. Stronger +than Strategy 1 because it doesn't rely on distillation. + +**What it can't tell you:** What else the large model is doing +alongside the identified circuit. Also requires the circuit to be +fully reverse-engineered in the small model first, which is its +own research project. + +### Strategy 5: Simplify the data, not the model + +**Idea:** Instead of training small models on complex data, train +large models on simple data where the "right answer" is known. +Example: train a 1B model on arithmetic only. The model is large +enough to exhibit generalization, but the task is simple enough +that the correct algorithm is known. Reverse-engineer the model's +learned algorithm and compare to the ground truth. + +**What it can tell you:** Whether scale enables discovery of the +known-correct algorithm, and at what point. If yes, you've shown +that next-token prediction + sufficient scale converges to the +right computation. + +**What it can't tell you:** Whether this generalizes to complex +domains where there is no single "correct algorithm." + +### Strategy 6: Necessary condition ablation + +**Idea:** For a specific capability, ablate candidate mechanisms +(specific attention patterns, feature directions, J-space +representations) and test whether the capability breaks. Work +backward from "what must be present" to "what is present." + +**What it can tell you:** Causal necessity — which internal +components are required for a capability. + +**What it can't tell you:** Sufficiency — whether those components +alone produce the capability. + +**HPMOR note:** This is the weakest strategy epistemically. Finding +that X is necessary for Y doesn't tell you why X enables Y or what +computation X performs. It's a starting point, not an endpoint. |