Add Common Crawl Preprocessing Tutorial W/ NeMo Curator

[asolergi-nv]

What does this PR do ?

Adds a tutorial preparing the Common Crawl dataset using the new MegatronTokenizerWriter stage from NeMo Curator (NVIDIA-NeMo/Curator#1259). This new stage tokenizes and produces the file prefixes (.bin and .idx files) required for training with Megatron-LM & Megatron-Bridge.

Proper tests and documentation have been added in the Curator repo. We are still waiting for a final review to merge the PR; let’s wait to do that before merging this PR.

To install the development branch run

pip install "nemo-curator[text_cpu] @ git+https://github.com/asolergi-nv/Curator.git@megatron_tokenizer"

Changelog

• Add "Common Crawl Data Preprocessing Tutorial W/ NeMo Curator" tutorial

GitHub Actions CI

See the CI sectionin the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

• NeMo Curator PR: Megatron tokenization pipeline Curator#1259