diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 21 |
1 files changed, 18 insertions, 3 deletions
@@ -70,10 +70,17 @@ sh scripts/test.sh Skips gracefully where torch is unavailable. -### 2. Train the base model +### 2. Prepare data + train the base model -Train nanoGPT on `data/shakespeare_char` (10.65M params, 6 layers, d=384, -block 128) and keep the checkpoint at `out-shakespeare-char/ckpt.pt`: +Prepare the character-level Shakespeare dataset (downloads Shakespeare and +builds `data/shakespeare_char/{train,val}.bin` + `meta.pkl`): + +```sh +python3 data/shakespeare_char/prepare.py +``` + +Then train nanoGPT (10.65M params, 6 layers, d=384, block 128) and keep the +checkpoint at `out-shakespeare-char/ckpt.pt`: ```sh python3 train.py config/train_shakespeare_char.py @@ -108,6 +115,14 @@ python3 src/loss_reweight.py --step train --mode ctrl_random --seed 0 python3 src/loss_reweight.py --step summary --layers 2,3,4 ``` +## Data & artifacts + +Raw data files (`.bin`/`.pkl`), checkpoints, and experiment outputs are NOT +committed (gitignored — they are regenerable and large). Committed instead: +`results.md` (the numbers) and `src/stats_decomp.py` (the statistics that +reproduce them from the saved artifacts). To regenerate everything, follow the +reproduction steps above; the full pipeline takes a few hours on a 4GB GPU. + ## References - [Verbalizable Representations Form a Global Workspace in Language Models](https://transformer-circuits.pub/2026/workspace/index.html) — Anthropic, 2026 |