# J-space on nanoGPT

Replicating Anthropic's Jacobian Lens ("J-space") technique on a small character-level transformer (nanoGPT).

## Hypothesis

If J-space (the subspace of representations readily available for verbal report) is an **architectural/structural property of transformers** rather than an emergent feature of advanced models, it should appear at all scales — including 10M-parameter char-level models.

## Background

Anthropic's 2026 paper "Verbalizable Representations Form a Global Workspace in Language Models" introduces the Jacobian Lens (J-lens), which computes the average linearized effect of activations on future token probabilities, averaged over many contexts. This reveals a privileged "J-space" of representations that the model can report on, modulate, and use for reasoning.

Full paper: https://transformer-circuits.pub/2026/workspace/index.html

## Experiments

1. **J-space visualization** — Compute J-lens vectors for all vocabulary tokens at each layer. Visualize which characters/concepts enter "verbalizable space" and when.
2. **Ablation test** — Remove J-space components vs random directions vs full activations. Measure prediction quality impact.
3. **Training dynamics** — Save checkpoints during training, compute J-space at each, track when it crystallizes.
4. **Capacity measurement** — Count active J-lens tokens per position.

## Setup

Runs on meru's Quadro K2200 (4GB VRAM) via Docker with GPU passthrough.

## References

- [Verbalizable Representations Form a Global Workspace in Language Models](https://transformer-circuits.pub/2026/workspace/index.html) — Anthropic, 2026
- [nanoGPT](https://github.com/karpathy/nanoGPT) — Andrej Karpathy