# Fundamental Open Questions Questions about how LLMs actually work that nobody can answer right now. These are the questions experiments should try to narrow, not the experiments themselves. ## 1. Why does next-token prediction produce reasoning? The observation: minimize cross-entropy on internet text → model learns to plan, reason causally, model other agents, and generalize to unseen tasks. The explanation "compression → generalization" is a description, not a mechanism. We don't know what computation the transformer is performing that makes the loss function equivalent to learning these capabilities. There is no bridge between the training objective and the emerged capability. **Why this is hard:** We can't even answer this for much simpler systems. Nobody can look at a training setup and predict what capabilities will emerge at what scale. This is alchemy, not chemistry. **What an answer would look like:** A mathematical framework that, given architecture + data distribution + compute budget, predicts the class of computations the model will learn. Scaling laws for capabilities, not just loss. ## 2. What actually happens during in-context learning? A model sees three examples of a new task with no weight updates, then performs it. The forward pass is doing something computationally equivalent to training. What circuit forms transiently? Is it attention over examples acting as nearest-neighbor retrieval? Is the model running an internal optimization procedure using gradient descent in activation space? Is it something else entirely? **Why this is hard:** The computation spans attention heads across all layers simultaneously. You can't isolate it to one mechanism. And the "circuit" dissolves after the forward pass — it's a transient computation, not a persistent structure. **What an answer would look like:** A mechanistic decomposition of a forward pass during ICL that shows (a) where the examples are stored, (b) how the task pattern is extracted, (c) how it's applied to the query. Ideally, the ability to predict ICL success/failure from the geometry of the example representations. ## 3. What determines the memorization → generalization transition? Small models pattern-match. Large models generalize. At some scale, something structural changes — the loss landscape develops basins that correspond to generalizing solutions rather than memorized ones. What is the nature of this transition? Is it sharp or continuous? Can we predict the transition point from architecture + data properties? **Why this is hard:** We can't look inside models at the relevant scale. We see inputs and outputs but not the internal representational change that enables generalization. Also: "generalization" is not one thing — a model can generalize on some tasks and memorize on others simultaneously. **What an answer would look like:** Identification of a specific representational property that appears at the transition (e.g., the formation of disentangled concept representations, or the emergence of something like the J-space documented by Anthropic). A metric that predicts generalization capability from model internals without evaluating on the task. ## 4. What is the geometry of learned concept space? Models form representations. We don't know if there's a universal structure — whether all sufficiently large models converge to the same conceptual topology regardless of architecture, or whether each model family has its own idiosyncratic geometry. Related: Anthropic's J-space paper shows there's a privileged subspace (verbalizable concepts) with functional properties resembling a global workspace. Is this universal across architectures? **Why this is hard:** Comparing representations across models requires a shared reference frame. Without one, you can't tell whether two models represent "dog" similarly or whether they've arrived at compatible but rotated representations. **What an answer would look like:** Evidence for or against representational convergence — do different architectures (dense, MoE, SSM) converge to the same concept geometry when trained on similar data? If yes, we can study one model and learn about all. If no, every model is a separate object of study. ## 5. Can behavior be predicted from architecture + training distribution without training? Scaling laws predict loss. But can we predict capabilities? Given architecture specs and a description of training data, can you say "this model will be able to do X but not Y" before spending the compute? **Why this is hard:** Capabilities are emergent and discontinuous. You can't extrapolate from small models because the computation changes qualitatively at scale. And "training data description" is itself intractable — knowing that a model trained on "the internet" tells you almost nothing about what it will learn. **What an answer would look like:** A predictive framework that, for a given capability (e.g., "can solve 3-digit multiplication"), tells you the minimum compute/data threshold. This almost certainly requires understanding Question 1 first. ## Meta-note These questions form a dependency chain: Q1 (why does prediction → reasoning?) is the foundation. Q2 (in-context learning mechanism) is a special case. Q3 (memorization → generalization) is the developmental trajectory. Q4 (concept geometry) is the structure. Q5 (capability prediction) is the engineering payoff. Progress on any of these would be significant. Progress on Q1 would restructure the field.