From 31e1656f1fd4409c14108e63eb3e430f657746ac Mon Sep 17 00:00:00 2001 From: Void Agent Date: Sun, 2 Aug 2026 14:24:51 +0100 Subject: Address Luna repo review: dynamic repo root (no hardcoded cwd), J-lens correctness test (last-layer identity), auditable fact-check, mechanism narrative fix, softened conditional-predictability claim, README data-prep + artifacts note --- README.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 155adde..ae9ba86 100644 --- a/README.md +++ b/README.md @@ -70,10 +70,17 @@ sh scripts/test.sh Skips gracefully where torch is unavailable. -### 2. Train the base model +### 2. Prepare data + train the base model -Train nanoGPT on `data/shakespeare_char` (10.65M params, 6 layers, d=384, -block 128) and keep the checkpoint at `out-shakespeare-char/ckpt.pt`: +Prepare the character-level Shakespeare dataset (downloads Shakespeare and +builds `data/shakespeare_char/{train,val}.bin` + `meta.pkl`): + +```sh +python3 data/shakespeare_char/prepare.py +``` + +Then train nanoGPT (10.65M params, 6 layers, d=384, block 128) and keep the +checkpoint at `out-shakespeare-char/ckpt.pt`: ```sh python3 train.py config/train_shakespeare_char.py @@ -108,6 +115,14 @@ python3 src/loss_reweight.py --step train --mode ctrl_random --seed 0 python3 src/loss_reweight.py --step summary --layers 2,3,4 ``` +## Data & artifacts + +Raw data files (`.bin`/`.pkl`), checkpoints, and experiment outputs are NOT +committed (gitignored — they are regenerable and large). Committed instead: +`results.md` (the numbers) and `src/stats_decomp.py` (the statistics that +reproduce them from the saved artifacts). To regenerate everything, follow the +reproduction steps above; the full pipeline takes a few hours on a 4GB GPU. + ## References - [Verbalizable Representations Form a Global Workspace in Language Models](https://transformer-circuits.pub/2026/workspace/index.html) — Anthropic, 2026 -- cgit v1.2.3