Very slow data loading on large dataset

[agemagician]

I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about 1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")

Is there something that I am missing ?

[lhoestq]

When you load a text file for the first time with nlp, the file is converted into Apache Arrow format. Arrow allows to use memory-mapping, which means that you can load an arbitrary large dataset.

Note that as soon as the conversion has been done once, the next time you'll load the dataset it will be much faster.

However for a 1TB dataset, the conversion can indeed take time. You could try to load parts of it in parallel, and then use nlp.concatenate_datasets to get your full dataset.

[thomwolf]

Humm, we can give a look at these large scale datasets indeed.

Do you mind sharing a few stats on your dataset so I can try to test on a similar one?

In particular some orders of magnitudes for the number of files, number of lines per files, line lengths.

[agemagician]

@lhoestq Yes, I understand that the first time requires more time. The concatenate_datasets seems to be a workaround, but I believe a multi-processing method should be integrated into load_dataset to make it easier and more efficient for users.

@thomwolf Sure, here are the statistics:
Number of lines: 4.2 Billion
Number of files: 6K
Number of tokens: 800 Billion
The number of lines is distributed equally across these 6k files.
The line length varies between 100 tokens to 40k tokens.

[thomwolf]

@agemagician you can give a try at a multithreaded version if you want (currently on the #548).

To test it, you just need to copy the new text processing script which is here somewhere on your drive and give it's local path instead of text to load_dataset. E.g. in your example:

train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('./datasets/text.py',   # path to where you've dowloaded the multi-threaded text loading script
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")

[agemagician]

I have already generated the dataset, but now I tried to reload it and it is still very slow.

I also have installed your commit and it is slow, even after the dataset was already generated.
pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257

It uses only a single thread.

Did I miss something ?

[lhoestq]

As mentioned in #548 , each time you call load_dataset with data_files=, they are hashed to get the cache directory name. Hashing can be too slow with 1TB of data. I feel like we should have a faster way of getting a hash that identifies the input data files

[agemagician]

I believe this is really a very important feature, otherwise, we will still have the issue of too slow loading problems even if the data cache generation is fast.

[thomwolf]

Hmm ok then maybe it's the hashing step indeed.

Let's see if we can improve this as well.

(you will very likely have to regenerate your dataset if we change this part of the lib though since I expect modifications on this part of the lib to results in new hashes)

[thomwolf]

Also, @agemagician you have to follow the step I indicate in my previous message here to use the new text loading script.

Just doing pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257 like you did won't use the new script (they are not inside the library but hosted on our hub).

[agemagician]

No problem, I will regenerate it. This will make us see if we solved both issues and now both the data generation step, as well as the hashing step, is fast.

[agemagician]

Any news for the hashing ?

[lhoestq]

I'm working on it today :)