1 files changed, 114 insertions, 0 deletions
diff --git a/fundamental-questions.md b/fundamental-questions.md
new file mode 100644
index 0000000..cffa4f6
--- /dev/null
+++ b/fundamental-questions.md
@@ -0,0 +1,114 @@
+# Fundamental Open Questions
+
+Questions about how LLMs actually work that nobody can answer right now.
+These are the questions experiments should try to narrow, not the
+experiments themselves.
+
+## 1. Why does next-token prediction produce reasoning?
+
+The observation: minimize cross-entropy on internet text → model learns
+to plan, reason causally, model other agents, and generalize to unseen
+tasks. The explanation "compression → generalization" is a description,
+not a mechanism. We don't know what computation the transformer is
+performing that makes the loss function equivalent to learning these
+capabilities. There is no bridge between the training objective and the
+emerged capability.
+
+**Why this is hard:** We can't even answer this for much simpler systems.
+Nobody can look at a training setup and predict what capabilities will
+emerge at what scale. This is alchemy, not chemistry.
+
+**What an answer would look like:** A mathematical framework that, given
+architecture + data distribution + compute budget, predicts the class
+of computations the model will learn. Scaling laws for capabilities,
+not just loss.
+
+## 2. What actually happens during in-context learning?
+
+A model sees three examples of a new task with no weight updates, then
+performs it. The forward pass is doing something computationally equivalent
+to training. What circuit forms transiently? Is it attention over examples
+acting as nearest-neighbor retrieval? Is the model running an internal
+optimization procedure using gradient descent in activation space? Is it
+something else entirely?
+
+**Why this is hard:** The computation spans attention heads across all
+layers simultaneously. You can't isolate it to one mechanism. And the
+"circuit" dissolves after the forward pass — it's a transient computation,
+not a persistent structure.
+
+**What an answer would look like:** A mechanistic decomposition of a
+forward pass during ICL that shows (a) where the examples are stored,
+(b) how the task pattern is extracted, (c) how it's applied to the query.
+Ideally, the ability to predict ICL success/failure from the geometry
+of the example representations.
+
+## 3. What determines the memorization → generalization transition?
+
+Small models pattern-match. Large models generalize. At some scale,
+something structural changes — the loss landscape develops basins that
+correspond to generalizing solutions rather than memorized ones. What
+is the nature of this transition? Is it sharp or continuous? Can we
+predict the transition point from architecture + data properties?
+
+**Why this is hard:** We can't look inside models at the relevant scale.
+We see inputs and outputs but not the internal representational change
+that enables generalization. Also: "generalization" is not one thing —
+a model can generalize on some tasks and memorize on others
+simultaneously.
+
+**What an answer would look like:** Identification of a specific
+representational property that appears at the transition (e.g., the
+formation of disentangled concept representations, or the emergence
+of something like the J-space documented by Anthropic). A metric that
+predicts generalization capability from model internals without
+evaluating on the task.
+
+## 4. What is the geometry of learned concept space?
+
+Models form representations. We don't know if there's a universal
+structure — whether all sufficiently large models converge to the same
+conceptual topology regardless of architecture, or whether each model
+family has its own idiosyncratic geometry. Related: Anthropic's J-space
+paper shows there's a privileged subspace (verbalizable concepts) with
+functional properties resembling a global workspace. Is this universal
+across architectures?
+
+**Why this is hard:** Comparing representations across models requires
+a shared reference frame. Without one, you can't tell whether two models
+represent "dog" similarly or whether they've arrived at compatible but
+rotated representations.
+
+**What an answer would look like:** Evidence for or against
+representational convergence — do different architectures (dense,
+MoE, SSM) converge to the same concept geometry when trained on similar
+data? If yes, we can study one model and learn about all. If no, every
+model is a separate object of study.
+
+## 5. Can behavior be predicted from architecture + training distribution without training?
+
+Scaling laws predict loss. But can we predict capabilities? Given
+architecture specs and a description of training data, can you say
+"this model will be able to do X but not Y" before spending the compute?
+
+**Why this is hard:** Capabilities are emergent and discontinuous.
+You can't extrapolate from small models because the computation changes
+qualitatively at scale. And "training data description" is itself
+intractable — knowing that a model trained on "the internet" tells you
+almost nothing about what it will learn.
+
+**What an answer would look like:** A predictive framework that, for a
+given capability (e.g., "can solve 3-digit multiplication"), tells you
+the minimum compute/data threshold. This almost certainly requires
+understanding Question 1 first.
+
+## Meta-note
+
+These questions form a dependency chain: Q1 (why does prediction →
+reasoning?) is the foundation. Q2 (in-context learning mechanism)
+is a special case. Q3 (memorization → generalization) is the
+developmental trajectory. Q4 (concept geometry) is the structure.
+Q5 (capability prediction) is the engineering payoff.
+
+Progress on any of these would be significant. Progress on Q1 would
+restructure the field.