From db4ec5bb3839bc5cc50d82e427848595d14b3070 Mon Sep 17 00:00:00 2001 From: Void Agent Date: Wed, 29 Jul 2026 17:47:48 +0100 Subject: Restructure: nanoGPT at root, custom code in src/ - Move model.py, train.py, configurator.py to root for nanoGPT compatibility - data/ and config/ directories at root with Shakespeare dataset prep scripts - src/jlens.py updated to import model from project root - Cleaned up stale src/config/ and duplicate src/ files - Fixed .gitignore: exclude out-shakespeare-char/ instead of raw data dirs --- data/openwebtext/readme.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 data/openwebtext/readme.md (limited to 'data/openwebtext/readme.md') diff --git a/data/openwebtext/readme.md b/data/openwebtext/readme.md new file mode 100644 index 0000000..95eb1bf --- /dev/null +++ b/data/openwebtext/readme.md @@ -0,0 +1,15 @@ + +## openwebtext dataset + +after running `prepare.py` (preprocess) we get: + +- train.bin is ~17GB, val.bin ~8.5MB +- train has ~9B tokens (9,035,582,198) +- val has ~4M tokens (4,434,897) + +this came from 8,013,769 documents in total. + +references: + +- OpenAI's WebText dataset is discussed in [GPT-2 paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) +- [OpenWebText](https://skylion007.github.io/OpenWebTextCorpus/) dataset -- cgit v1.2.3