diff options
Diffstat (limited to 'data/openwebtext/readme.md')
| -rw-r--r-- | data/openwebtext/readme.md | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/data/openwebtext/readme.md b/data/openwebtext/readme.md new file mode 100644 index 0000000..95eb1bf --- /dev/null +++ b/data/openwebtext/readme.md @@ -0,0 +1,15 @@ + +## openwebtext dataset + +after running `prepare.py` (preprocess) we get: + +- train.bin is ~17GB, val.bin ~8.5MB +- train has ~9B tokens (9,035,582,198) +- val has ~4M tokens (4,434,897) + +this came from 8,013,769 documents in total. + +references: + +- OpenAI's WebText dataset is discussed in [GPT-2 paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) +- [OpenWebText](https://skylion007.github.io/OpenWebTextCorpus/) dataset |