Dataset loading

[Zhouxx0101]

I hope to select some datasets for pre-training. Could you please tell me if the datasets on the autogluon/chronos_datasets can be read locally by using the load_from_disk function? Or is there any other way to load the data sets stored on the disk? If I need to use multiple datasets, do I have to convert their formats separately? Thanks!

If the format needs to be converted to arrow, how should it be done for large datasets, such as weatherbench_hourly?

[shchur]

The question is probably best directed to https://github.com/huggingface/datasets.

There are multiple options:

1. Convert datasets to arrow by calling load_dataset("autogluon/chronos_dataset", ...). Then the datasets can be loaded with load_from_disk using memmap. This allows for efficient iteration from disk without loading all the data into memory. Converting the data might take some time though.
2. Load the parquet files from HF hub and load them with load_dataset("parquet", data_files="path/to/data/*.parquet"). This will avoid converting the data to arrow, but will require loading all datasets into RAM (which can lead to OOM errors).

There are also other advanced options with streaming that are described here https://huggingface.co/datasets/autogluon/chronos_datasets#dealing-with-large-datasets.

You can also only load certain columns from the dataset (e.g. just the target), which will reduce the time/memory footprint.

Each option has some pros and cons, you need to weigh them based on your constraints and requirements. I recommend discussing this with ChatGPT or an LLM of your choice - it will provide more detailed and high-quality suggestions.

[Zhouxx0101]

Thank you very much!