Skip to content

docs: add anchor links to feature sections in README for easy referencing #743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 3, 2025
Merged

docs: add anchor links to feature sections in README for easy referencing #743

Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
b3a435e
Testing permalink
VijayVignesh1 Oct 30, 2025
8fae472
Testing permalink
VijayVignesh1 Oct 30, 2025
6460fff
Testing permalink
VijayVignesh1 Oct 30, 2025
26eab1b
Testing permalink
VijayVignesh1 Oct 30, 2025
1cc0a15
Testing permalink
VijayVignesh1 Oct 30, 2025
aaeaaf6
Testing permalink
VijayVignesh1 Oct 30, 2025
69c35b5
Adding anchor links
VijayVignesh1 Oct 31, 2025
82490bc
Merge branch 'main' into docs/add_anchor_links
VijayVignesh1 Oct 31, 2025
660021d
Cleaning the <a> tag
VijayVignesh1 Nov 1, 2025
a17cb01
adding two word id for <a>
VijayVignesh1 Nov 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 33 additions & 33 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ ld.map(
## Features for optimizing and streaming datasets for model training

<details>
<summary>✅ Stream raw datasets from cloud storage (beta)</summary>
<summary> <a name="stream-raw-datasets-from-cloud-storage-beta"></a> ✅ Stream raw datasets from cloud storage (beta) <a href="#stream-raw-datasets-from-cloud-storage-beta">🔗</a> </summary>
&nbsp;

Effortlessly stream raw files (images, text, etc.) directly from S3, GCS, and Azure cloud storage without any optimization or conversion. Ideal for workflows requiring instant access to original data in its native format.
Expand Down Expand Up @@ -317,7 +317,7 @@ dataset = StreamingRawDataset("s3://bucket/files/", recompute_index=True)
</details>

<details>
<summary> ✅ Stream large cloud datasets</summary>
<summary> <a name="stream-large-cloud-datasets"></a> ✅ Stream large cloud datasets <a href="#stream-large-cloud-datasets">🔗</a> </summary>
&nbsp;

Use data stored on the cloud without needing to download it all to your computer, saving time and space.
Expand Down Expand Up @@ -367,7 +367,7 @@ dataset = StreamingDataset('s3://my-bucket/my-data', cache_dir="/path/to/cache")
</details>

<details>
<summary> ✅ Stream Hugging Face 🤗 datasets</summary>
<summary> <a name="stream-hugging-face-datasets"></a> ✅ Stream Hugging Face 🤗 datasets <a href="#stream-hugging-face-datasets">🔗</a> </summary>

&nbsp;

Expand Down Expand Up @@ -480,7 +480,7 @@ Below is the benchmark for the `Imagenet dataset (155 GB)`, demonstrating that *
</details>

<details>
<summary> ✅ Streams on multi-GPU, multi-node</summary>
<summary> <a name="streams-on-multi-gpu-multi-node"></a> ✅ Streams on multi-GPU, multi-node <a href="#streams-on-multi-gpu-multi-node">🔗</a> </summary>

&nbsp;

Expand Down Expand Up @@ -512,7 +512,7 @@ for batch in val_dataloader:
</details>

<details>
<summary> ✅ Stream from multiple cloud providers</summary>
<summary> <a name="stream-from-multiple-cloud-providers"></a> ✅ Stream from multiple cloud providers <a href="#stream-from-multiple-cloud-providers">🔗</a> </summary>

&nbsp;

Expand Down Expand Up @@ -570,7 +570,7 @@ dataset = ld.StreamingDataset("azure://my-bucket/my-data", storage_options=azure
</details>

<details>
<summary> ✅ Pause, resume data streaming</summary>
<summary> <a name="pause-resume-data-streaming"></a> ✅ Pause, resume data streaming <a href="#pause-resume-data-streaming">🔗</a> </summary>
&nbsp;

Stream data during long training, if interrupted, pick up right where you left off without any issues.
Expand Down Expand Up @@ -604,7 +604,7 @@ for batch_idx, batch in enumerate(dataloader):


<details>
<summary> ✅ Use shared queue for Optimizing</summary>
<summary> <a name="use-shared-queue-for-optimizing"></a> ✅ Use shared queue for Optimizing <a href="#use-shared-queue-for-optimizing">🔗</a> </summary>
&nbsp;

If you are using multiple workers to optimize your dataset, you can use a shared queue to speed up the process.
Expand Down Expand Up @@ -661,7 +661,7 @@ if __name__ == "__main__":


<details>
<summary> ✅ Use a <code>Queue</code> as input for optimizing data</summary>
<summary> <a name="use-queue-as-input-for-optimizing-data"></a> ✅ Use a <code>Queue</code> as input for optimizing data <a href="#use-queue-as-input-for-optimizing-data">🔗</a> </summary>
&nbsp;

Sometimes you don’t have a static list of inputs to optimize — instead, you have a stream of data coming in over time. In such cases, you can use a multiprocessing.Queue to feed data into the optimize() function.
Expand Down Expand Up @@ -718,7 +718,7 @@ if __name__ == "__main__":


<details>
<summary> ✅ LLM Pre-training </summary>
<summary> <a name="llm-pre-training"></a> ✅ LLM Pre-training <a href="#llm-pre-training">🔗</a> </summary>
&nbsp;

LitData is highly optimized for LLM pre-training. First, we need to tokenize the entire dataset and then we can consume it.
Expand Down Expand Up @@ -781,7 +781,7 @@ for batch in tqdm(train_dataloader):
</details>

<details>
<summary> ✅ Filter illegal data </summary>
<summary> <a name="filter-illegal-data"></a> ✅ Filter illegal data <a href="#filter-illegal-data">🔗</a> </summary>
&nbsp;

Sometimes, you have bad data that you don't want to include in the optimized dataset. With LitData, yield only the good data sample to include.
Expand Down Expand Up @@ -843,7 +843,7 @@ if __name__ == "__main__":
</details>

<details>
<summary> ✅ Combine datasets</summary>
<summary> <a name="combine-datasets"></a> ✅ Combine datasets <a href="#combine-datasets">🔗</a> </summary>
&nbsp;

Mix and match different sets of data to experiment and create better models.
Expand Down Expand Up @@ -915,7 +915,7 @@ combined_dataset = CombinedStreamingDataset(
</details>

<details>
<summary> ✅ Parallel streaming</summary>
<summary> <a name="parallel-streaming"></a> ✅ Parallel streaming <a href="#parallel-streaming">🔗</a> </summary>
&nbsp;

While `CombinedDataset` allows to fetch a sample from one of the datasets it wraps at each iteration, `ParallelStreamingDataset` can be used to fetch a sample from all the wrapped datasets at each iteration:
Expand Down Expand Up @@ -965,7 +965,7 @@ parallel_dataset = ParallelStreamingDataset([dset_1, dset_2], transform=transfor
</details>

<details>
<summary> ✅ Cycle datasets</summary>
<summary> <a name="cycle-datasets"></a> ✅ Cycle datasets <a href="#cycle-datasets">🔗</a> </summary>
&nbsp;

`ParallelStreamingDataset` can also be used to cycle a `StreamingDataset`. This allows to dissociate the epoch length from the number of samples in the dataset.
Expand All @@ -992,7 +992,7 @@ You can even set `length` to `float("inf")` for an infinite dataset!
</details>

<details>
<summary> ✅ Merge datasets</summary>
<summary> <a name="merge-datasets"></a> ✅ Merge datasets <a href="#merge-datasets">🔗</a> </summary>
&nbsp;

Merge multiple optimized datasets into one.
Expand Down Expand Up @@ -1027,7 +1027,7 @@ if __name__ == "__main__":
</details>

<details>
<summary> ✅ Transform datasets while Streaming</summary>
<summary> <a name="transform-datasets-while-streaming"></a> ✅ Transform datasets while Streaming <a href="#transform-datasets-while-streaming">🔗</a> </summary>
&nbsp;

Transform datasets on-the-fly while streaming them, allowing for efficient data processing without the need to store intermediate results.
Expand Down Expand Up @@ -1083,7 +1083,7 @@ dataset = StreamingDatasetWithTransform(data_dir, cache_dir=str(cache_dir), shuf
</details>

<details>
<summary> ✅ Split datasets for train, val, test</summary>
<summary> <a name="split-datasets-for-train-val-test"></a> ✅ Split datasets for train, val, test <a href="#split-datasets-for-train-val-test">🔗</a> </summary>

&nbsp;

Expand Down Expand Up @@ -1112,7 +1112,7 @@ print(test_dataset)
</details>

<details>
<summary> ✅ Load a subset of the remote dataset</summary>
<summary> <a name="load-subset-of-remote-dataset"></a> ✅ Load a subset of the remote dataset <a href="#load-subset-of-remote-dataset">🔗</a> </summary>

&nbsp;
Work on a smaller, manageable portion of your data to save time and resources.
Expand All @@ -1130,7 +1130,7 @@ print(len(dataset)) # display the length of your data
</details>

<details>
<summary> ✅ Upsample from your source datasets </summary>
<summary> <a name="upsample-from-source-datasets"></a> ✅ Upsample from your source datasets <a href="#upsample-from-source-datasets">🔗</a> </summary>

&nbsp;
Use to control the size of one iteration of a StreamingDataset using repeats. Contains `floor(N)` possibly shuffled copies of the source data, then a subsampling of the remainder.
Expand All @@ -1148,7 +1148,7 @@ print(len(dataset)) # display the length of your data
</details>

<details>
<summary> ✅ Easily modify optimized cloud datasets</summary>
<summary> <a name="easily-modify-optimized-cloud-datasets"></a> ✅ Easily modify optimized cloud datasets <a href="#easily-modify-optimized-cloud-datasets">🔗</a> </summary>
&nbsp;

Add new data to an existing dataset or start fresh if needed, providing flexibility in data management.
Expand Down Expand Up @@ -1189,7 +1189,7 @@ The `overwrite` mode will delete the existing data and start from fresh.
</details>

<details>
<summary> ✅ Stream parquet datasets</summary>
<summary> <a name="stream-parquet-datasets"></a> ✅ Stream parquet datasets <a href="#stream-parquet-datasets">🔗</a> </summary>
&nbsp;

Stream Parquet datasets directly with LitData—no need to convert them into LitData’s optimized binary format! If your dataset is already in Parquet format, you can efficiently index and stream it using `StreamingDataset` and `StreamingDataLoader`.
Expand Down Expand Up @@ -1248,7 +1248,7 @@ for sample in dataloader:
</details>

<details>
<summary> ✅ Use compression</summary>
<summary> <a name="use-compression"></a> ✅ Use compression <a href="#use-compression">🔗</a> </summary>
&nbsp;

Reduce your data footprint by using advanced compression algorithms.
Expand Down Expand Up @@ -1281,7 +1281,7 @@ Using [zstd](https://github.com/facebook/zstd), you can achieve high compression
</details>

<details>
<summary> ✅ Access samples without full data download</summary>
<summary> <a name="access-samples-without-full-data-download"></a> ✅ Access samples without full data download <a href="#access-samples-without-full-data-download">🔗</a> </summary>
&nbsp;

Look at specific parts of a large dataset without downloading the whole thing or loading it on a local machine.
Expand All @@ -1299,7 +1299,7 @@ print(dataset[42]) # show the 42th element of the dataset
</details>

<details>
<summary> ✅ Use any data transforms</summary>
<summary> <a name="use-any-data-transforms"></a> ✅ Use any data transforms <a href="#use-any-data-transforms">🔗</a> </summary>
&nbsp;

Customize how your data is processed to better fit your needs.
Expand Down Expand Up @@ -1327,7 +1327,7 @@ for batch in dataloader:
</details>

<details>
<summary> ✅ Profile data loading speed</summary>
<summary> <a name="profile-data-loading-speed"></a> ✅ Profile data loading speed <a href="#profile-data-loading-speed">🔗</a> </summary>
&nbsp;

Measure and optimize how fast your data is being loaded, improving efficiency.
Expand All @@ -1345,7 +1345,7 @@ This generates a Chrome trace called `result.json`. Then, visualize this trace b
</details>

<details>
<summary> ✅ Reduce memory use for large files</summary>
<summary> <a name="reduce-memory-use-for-large-files"></a> ✅ Reduce memory use for large files <a href="#reduce-memory-use-for-large-files">🔗</a> </summary>
&nbsp;

Handle large data files efficiently without using too much of your computer's memory.
Expand Down Expand Up @@ -1383,7 +1383,7 @@ outputs = optimize(
</details>

<details>
<summary> ✅ Limit local cache space</summary>
<summary> <a name="limit-local-cache-space"></a> ✅ Limit local cache space <a href="#limit-local-cache-space">🔗</a> </summary>
&nbsp;

Limit the amount of disk space used by temporary files, preventing storage issues.
Expand All @@ -1399,7 +1399,7 @@ dataset = StreamingDataset(..., max_cache_size="10GB")
</details>

<details>
<summary> ✅ Change cache directory path</summary>
<summary> <a name="change-cache-directory-path"></a> ✅ Change cache directory path <a href="#change-cache-directory-path">🔗</a> </summary>
&nbsp;

Specify the directory where cached files should be stored, ensuring efficient data retrieval and management. This is particularly useful for organizing your data storage and improving access times.
Expand All @@ -1417,7 +1417,7 @@ dataset = StreamingDataset(input_dir=Dir(path=cache_dir, url=data_dir))
</details>

<details>
<summary> ✅ Optimize loading on networked drives</summary>
<summary> <a name="optimize-loading-on-networked-drives"></a> ✅ Optimize loading on networked drives <a href="#optimize-loading-on-networked-drives">🔗</a> </summary>
&nbsp;

Optimize data handling for computers on a local network to improve performance for on-site setups.
Expand All @@ -1433,7 +1433,7 @@ dataset = StreamingDataset(input_dir="local:/data/shared-drive/some-data")
</details>

<details>
<summary> ✅ Optimize dataset in distributed environment</summary>
<summary> <a name="optimize-dataset-in-distributed-environment"></a> ✅ Optimize dataset in distributed environment <a href="#optimize-dataset-in-distributed-environment">🔗</a> </summary>
&nbsp;

Lightning can distribute large workloads across hundreds of machines in parallel. This can reduce the time to complete a data processing task from weeks to minutes by scaling to enough machines.
Expand Down Expand Up @@ -1475,7 +1475,7 @@ print(dataset[:])
</details>

<details>
<summary> ✅ Encrypt, decrypt data at chunk/sample level</summary>
<summary> <a name="encrypt-decrypt-data-at-chunk-sample-level"></a> ✅ Encrypt, decrypt data at chunk/sample level <a href="#encrypt-decrypt-data-at-chunk-sample-level">🔗</a> </summary>
&nbsp;

Secure data by applying encryption to individual samples or chunks, ensuring sensitive information is protected during storage.
Expand Down Expand Up @@ -1544,7 +1544,7 @@ This allows the data to remain secure while maintaining flexibility in the encry
</details>

<details>
<summary> ✅ Debug & Profile LitData with logs & Litracer</summary>
<summary> <a name="debug-profile-litdata-with-logs-litracer"></a> ✅ Debug & Profile LitData with logs & Litracer <a href="#debug-profile-litdata-with-logs-litracer">🔗</a> </summary>

&nbsp;

Expand Down Expand Up @@ -1612,7 +1612,7 @@ if __name__ == "__main__":
</details>

<details>
<summary> ✅ Lightning AI Data Connections - Direct download and upload </summary>
<summary> <a name="lightning-ai-data-connections-direct-download-upload"></a> ✅ Lightning AI Data Connections - Direct download and upload <a href="#lightning-ai-data-connections-direct-download-upload">🔗</a> </summary>

&nbsp;

Expand Down Expand Up @@ -1666,7 +1666,7 @@ References to any of the following directories will work similarly:
## Features for transforming datasets

<details>
<summary> ✅ Parallelize data transformations (map)</summary>
<summary> <a name="parallelize-data-transformations-map"></a> ✅ Parallelize data transformations (map) <a href="#parallelize-data-transformations-map">🔗</a> </summary>
&nbsp;

Apply the same change to different parts of the dataset at once to save time and effort.
Expand Down
Loading