diff options
Diffstat (limited to 'blog/prime-grokking-1.qmd')
| -rw-r--r-- | blog/prime-grokking-1.qmd | 14 |
1 files changed, 13 insertions, 1 deletions
diff --git a/blog/prime-grokking-1.qmd b/blog/prime-grokking-1.qmd index fa14304..23c97b1 100644 --- a/blog/prime-grokking-1.qmd +++ b/blog/prime-grokking-1.qmd @@ -31,7 +31,7 @@ Underneath almost everything in my research agenda sits one question: **what makes a neural network stop memorizing and start generalizing?** For a language model, "generalization" is impossible to define cleanly. So people study the cleanest possible version: tiny networks, toy tasks, and a signature -phenomenon called *grokking*. +phenomenon called *grokking* (see RESOURCES). Grokking, from Power et al. (2022), looks like this: train a small network on modular addition (`a + b mod p`). The training loss collapses to zero almost @@ -292,5 +292,17 @@ against ground-truth stub models. Each arm trains in ~2h on 2 CPU cores. --- +RESOURCES +--------- + +- Grokking (machine learning) — Wikipedia overview: + https://en.wikipedia.org/wiki/Grokking_(machine_learning) +- Code for this experiment (goes public once the seed replication lands): + <repo link — pending> +- Part 2: the seed sweep (when published): + <link — pending> + +--- + *This is part 1. The seed sweep is running next; whatever it shows — confirmation or refutation — becomes part 2.* |