diff options
| author | Void Agent <void@jayrup.hermes> | 2026-07-29 17:47:48 +0100 |
|---|---|---|
| committer | Void Agent <void@jayrup.hermes> | 2026-07-29 17:47:48 +0100 |
| commit | db4ec5bb3839bc5cc50d82e427848595d14b3070 (patch) | |
| tree | d2d9fb11e11392077a79e6a10c38d44d1d9f5a38 /data/openwebtext/readme.md | |
| parent | 66f99ee30087a5f28ad852e581a0334c7f556091 (diff) | |
Restructure: nanoGPT at root, custom code in src/
- Move model.py, train.py, configurator.py to root for nanoGPT compatibility
- data/ and config/ directories at root with Shakespeare dataset prep scripts
- src/jlens.py updated to import model from project root
- Cleaned up stale src/config/ and duplicate src/ files
- Fixed .gitignore: exclude out-shakespeare-char/ instead of raw data dirs
Diffstat (limited to 'data/openwebtext/readme.md')
| -rw-r--r-- | data/openwebtext/readme.md | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/data/openwebtext/readme.md b/data/openwebtext/readme.md new file mode 100644 index 0000000..95eb1bf --- /dev/null +++ b/data/openwebtext/readme.md @@ -0,0 +1,15 @@ + +## openwebtext dataset + +after running `prepare.py` (preprocess) we get: + +- train.bin is ~17GB, val.bin ~8.5MB +- train has ~9B tokens (9,035,582,198) +- val has ~4M tokens (4,434,897) + +this came from 8,013,769 documents in total. + +references: + +- OpenAI's WebText dataset is discussed in [GPT-2 paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) +- [OpenWebText](https://skylion007.github.io/OpenWebTextCorpus/) dataset |