diff options
| author | Void Agent <void@jayrup.hermes> | 2026-07-29 17:42:47 +0100 |
|---|---|---|
| committer | Void Agent <void@jayrup.hermes> | 2026-07-29 17:42:47 +0100 |
| commit | 66f99ee30087a5f28ad852e581a0334c7f556091 (patch) | |
| tree | b83da26e340a926e6eb46f1b304b66f693a15e81 /src/config/eval_gpt2_xl.py | |
Initial project setup: J-lens implementation for nanoGPT
- Core J-lens computation (batched per-layer, per-token gradient method)
- Project README with background and experiment plan
- Sync script for meru Docker container deployment
- Upstream nanoGPT model code copied to src/
Architecture: Computes d(log_p(token))/d(residual_stream) averaged
over corpus contexts, replicating Anthropic's Jacobian Lens technique.
Diffstat (limited to 'src/config/eval_gpt2_xl.py')
| -rw-r--r-- | src/config/eval_gpt2_xl.py | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/src/config/eval_gpt2_xl.py b/src/config/eval_gpt2_xl.py new file mode 100644 index 0000000..1bae34f --- /dev/null +++ b/src/config/eval_gpt2_xl.py @@ -0,0 +1,8 @@ +# evaluate the base gpt2 +# n_layer=48, n_head=25, n_embd=1600 +# 1558M parameters +batch_size = 8 +eval_iters = 500 # use more iterations to get good estimate +eval_only = True +wandb_log = False +init_from = 'gpt2-xl' |