Fix Mistral tokenizer if no eos_token is added

[devxinvestor]

Context

What is the purpose of this PR? Is it to

• add a new feature
• fix a bug
• update tests and/or documentation
• other (please add here)

Please link to any issues this PR addresses.

Changelog

What are the changes made in this PR?

• Added an add_bos and add_eos attribute to ModelTokenizer to reference in tokenize_messages_no_special_tokens. If not special condition is met, than add_bos and add_eos are set to false, and the tokenized_messages and mask are treated accordingly.

Test plan

Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.

• run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
• add unit tests for any new functionality
• update docstrings for any new or updated methods or classes
• run unit tests via pytest tests
• run recipe tests via pytest tests -m integration_test
• manually run any new or modified recipes with sufficient proof of correctness
• include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

UX

If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example

• I did not change any public API
• I have added an example to docs or docstrings

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1693

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• PyTorch Testing Nodes Undergoing ROCm 6.2.1 Upgrades

❌ 1 New Failure

As of commit 0fe2c82 with merge base 3fddc56 ():

NEW FAILURE - The following job has failed:

• Lint / lint (3.10) (gh)
  ##[error]Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[RdoubleA]

Hey @devxinvestor, thanks for opening this PR. I believe the change should occur in the Mistral tokenizer itself rather than in tokenize_messages_no_special_tokens. In that function, BOS and EOS are controlled by passing in the BOS/EOS id from the caller. If they are not passed in, then it is treated as add_eos=False. In Mistral, it seems like tokenize_messages just adds the EOS by default without any control from the user. Perhaps we can just add a flag to tokenize_messages here:
https://github.com/pytorch/torchtune/blob/57dc6f32398967ff59dd585cbb1c4a4b2d9b6ebe/torchtune/models/mistral/_tokenizer.py#L125

and just pass in the EOS id only if this flag is true?
https://github.com/pytorch/torchtune/blob/57dc6f32398967ff59dd585cbb1c4a4b2d9b6ebe/torchtune/models/mistral/_tokenizer.py#L170

[krammnic]

Probably we can close this as it was fixed in #1806

[ebsmothers]

Yeah agreed @krammnic I'll close this one. Thanks for your PR @devxinvestor!