eval_gpt2.py « config - jspace-nanogpt.git - J-space on nanoGPT: Jacobian Lens replication & critique. J-lens norms anti-correlate with token frequency (r ~ -0.61 to -0.69); frequency-matched synthetic pair + causal loss-reweighting tests.

summary refs log tree commit diff

path: root/config/eval_gpt2.py

blob: 53978cb9e2408c9172d4b482afbd48784c3f8b27 (plain)

# evaluate the base gpt2
# n_layer=12, n_head=12, n_embd=768
# 124M parameters
batch_size = 8
eval_iters = 500 # use more iterations to get good estimate
eval_only = True
wandb_log = False
init_from = 'gpt2'