1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
|
# Fundamental Open Questions
Questions about how LLMs actually work that nobody can answer right now.
These are the questions experiments should try to narrow, not the
experiments themselves.
## 1. Why does next-token prediction produce reasoning?
The observation: minimize cross-entropy on internet text → model learns
to plan, reason causally, model other agents, and generalize to unseen
tasks. The explanation "compression → generalization" is a description,
not a mechanism. We don't know what computation the transformer is
performing that makes the loss function equivalent to learning these
capabilities. There is no bridge between the training objective and the
emerged capability.
**Why this is hard:** We can't even answer this for much simpler systems.
Nobody can look at a training setup and predict what capabilities will
emerge at what scale. This is alchemy, not chemistry.
**What an answer would look like:** A mathematical framework that, given
architecture + data distribution + compute budget, predicts the class
of computations the model will learn. Scaling laws for capabilities,
not just loss.
## 2. What actually happens during in-context learning?
A model sees three examples of a new task with no weight updates, then
performs it. The forward pass is doing something computationally equivalent
to training. What circuit forms transiently? Is it attention over examples
acting as nearest-neighbor retrieval? Is the model running an internal
optimization procedure using gradient descent in activation space? Is it
something else entirely?
**Why this is hard:** The computation spans attention heads across all
layers simultaneously. You can't isolate it to one mechanism. And the
"circuit" dissolves after the forward pass — it's a transient computation,
not a persistent structure.
**What an answer would look like:** A mechanistic decomposition of a
forward pass during ICL that shows (a) where the examples are stored,
(b) how the task pattern is extracted, (c) how it's applied to the query.
Ideally, the ability to predict ICL success/failure from the geometry
of the example representations.
## 3. What determines the memorization → generalization transition?
Small models pattern-match. Large models generalize. At some scale,
something structural changes — the loss landscape develops basins that
correspond to generalizing solutions rather than memorized ones. What
is the nature of this transition? Is it sharp or continuous? Can we
predict the transition point from architecture + data properties?
**Why this is hard:** We can't look inside models at the relevant scale.
We see inputs and outputs but not the internal representational change
that enables generalization. Also: "generalization" is not one thing —
a model can generalize on some tasks and memorize on others
simultaneously.
**What an answer would look like:** Identification of a specific
representational property that appears at the transition (e.g., the
formation of disentangled concept representations, or the emergence
of something like the J-space documented by Anthropic). A metric that
predicts generalization capability from model internals without
evaluating on the task.
## 4. What is the geometry of learned concept space?
Models form representations. We don't know if there's a universal
structure — whether all sufficiently large models converge to the same
conceptual topology regardless of architecture, or whether each model
family has its own idiosyncratic geometry. Related: Anthropic's J-space
paper shows there's a privileged subspace (verbalizable concepts) with
functional properties resembling a global workspace. Is this universal
across architectures?
**Why this is hard:** Comparing representations across models requires
a shared reference frame. Without one, you can't tell whether two models
represent "dog" similarly or whether they've arrived at compatible but
rotated representations.
**What an answer would look like:** Evidence for or against
representational convergence — do different architectures (dense,
MoE, SSM) converge to the same concept geometry when trained on similar
data? If yes, we can study one model and learn about all. If no, every
model is a separate object of study.
## 5. Can behavior be predicted from architecture + training distribution without training?
Scaling laws predict loss. But can we predict capabilities? Given
architecture specs and a description of training data, can you say
"this model will be able to do X but not Y" before spending the compute?
**Why this is hard:** Capabilities are emergent and discontinuous.
You can't extrapolate from small models because the computation changes
qualitatively at scale. And "training data description" is itself
intractable — knowing that a model trained on "the internet" tells you
almost nothing about what it will learn.
**What an answer would look like:** A predictive framework that, for a
given capability (e.g., "can solve 3-digit multiplication"), tells you
the minimum compute/data threshold. This almost certainly requires
understanding Question 1 first.
## Meta-note
These questions form a dependency chain: Q1 (why does prediction →
reasoning?) is the foundation. Q2 (in-context learning mechanism)
is a special case. Q3 (memorization → generalization) is the
developmental trajectory. Q4 (concept geometry) is the structure.
Q5 (capability prediction) is the engineering payoff.
Progress on any of these would be significant. Progress on Q1 would
restructure the field.
|
