A visual breakdown of why standard neural networks fail on prime numbers, and how token-space computation tests the boundaries of algorithmic emergence.
When researchers train neural networks on modular arithmetic (like (a + b) mod 97), the network suddenly experiences grokking: after long overfitting, it abruptly discovers the generalized circle algorithm (Fourier transform) because it is simpler and lower-norm than memorization.
Next-Prime is completely different. Finding the next prime after n requires a two-level nested algorithm:
A standard feedforward Transformer has a fixed number of layers. It cannot run an unbounded while-loop. In experiments E1–E7, we proved that even training for 4,000,000 steps never produces grokking on next-prime: the model just memorizes a lookup table for numbers in range, and scores 0%–3% out-of-range.
If the model doesn't have enough internal depth to perform multiple serial operations, we can give it intermediate tokens to think before answering.
Recent theoretical work by Pfau et al. (2024, "Let's Think Dot by Dot") made a surprising prediction: filler tokens (e.g. dots or pauses) only help if the hidden subcomputations are parallelizable. Checking divisors {2, 3, 5, 7} are independent parallel operations! Therefore, next-prime is the ideal testbed for this hypothesis.
To scientifically isolate why intermediate tokens help (or don't), we test four distinct arms:
Input goes directly to output. No intermediate steps.
Tests: Can pure fixed-depth attention solve next-prime? (We know from E7 this fails out-of-range).
Ground-truth trial division steps are spelled out in tokens.
Tests: If the algorithm is explicitly supervised, does the model grok and generalize to unseen ranges?
Fixed 16 identical pause tokens (no intermediate labels).
Tests: Does extra compute depth alone allow the transformer to execute parallel divisor checks silently?
Fixed 16 random letters from a disjoint alphabet ([a-p]).
Tests: Does any sequence expansion help, or is static pause token embedding special?
| Observed Pattern | Scientific Interpretation | Significance |
|---|---|---|
| Arm B >> Arm A, C, D | Algorithm Decomposition Required: Extra compute alone does nothing; explicit intermediate supervision is required to guide SGD. | Confirms Nye et al. Scratchpad mechanism. |
| Arm C >> Arm A | Pfau et al. Confirmed: Unsupervised filler tokens provide enough hidden attention routing to compute parallel divisibility tests. | Major theoretical finding! First clean toy demonstration on non-group tasks. |
| Arm D ≈ Arm C > Arm A | Pure Depth Invariance: Expanding sequence length improves representational capacity regardless of token semantics. | Points to transformer attention capacity dynamics. |
| All Arms Fail ($P4$) Out-of-Range | Absolute Algorithmic Wall: Neither internal recurrence nor token-space scratchpads allow next-token models to generalize prime discovery. | Falsifies scratchpad extrapolation for unbounded search. |
We will train a 2-layer GPT Transformer across all four arms in the $[2, 1000]$ regime for 200k steps with weight decay $0.1$. We evaluate exact-match accuracy both in-range ($[2, 1000]$) and on the unseen probe range ($[1001, 2000]$) to test true algorithmic generalization.