UnslothTrainer applies ChatML template although passed train dataset is pre-tokenized and contains 'input_ids' field

[crto]

Version: unsloth 2025.2.15, unsloth_zoo 2025.2.7, transformers 4.49.0, trl: 0.15.1

Performing continued pretraining using unsloth/Meta-Llama-3.1-8B-Instruct.
Pre-tokenized dataset that contains 'input_ids' field is passed to UnslothTrainer.
UnslothTrainer contructor starts converting passed train dataset to ChatML:

"Converting train dataset to ChatML (num_proc=8): ..."

Used debugger to pinpoint the issue. It seems that the problem is in
unsloth_compiled_cache/UnslothSFTTrainer.py lines 663-670:

        # Convert the dataset to ChatML if needed
        if isinstance(dataset, Dataset):  # `IterableDataset.map` does not support `desc`
            map_kwargs["desc"] = f"Converting {dataset_name} dataset to ChatML"
        dataset = dataset.map(
            maybe_convert_to_chatml,
            remove_columns="conversations" if "conversations" in dataset.column_names else None,
            **map_kwargs,
        )

Conversion is performed although is_processed variable in line 626 was set to true:

    is_processed = "input_ids" in column_names

I
This did not occur unsloth 2025.1.8, unsloth_zoo 2025.1.5, transformers 4.48.2, trl 0.14.0

Trl sft_trainer.py looks ok (line 404):

if not is_processed:

This looks to be missing in UnslothSFTTrainer.py?

[danielhanchen]

For now it should be fine to ignore since I checked the input_ids and they look fine - it's most likely the tokenization description causing issues.

But I did notice TRL now does 4 tokenizations? I'll move them into 1 and clean the bad description up!

[crto]

Trl sf_trainer.py code in function _prepare_dataset does not perform any dataset processing
(ChatML template conversion/application, dataset tokenization) if passed dataset is pre-tokenized,
UnslothSFTTrainer performs ChatML template conversion/application and dataset tokenization
also if pre-tokenized train dataset is passed. This of course causes training to fail.

So all three above steps in UnslothSFTTrainer should be moved under condition 'if not is_processed:'
the same way this is the case in sft_trainer.py.

Hope my logic is sound ;-)

[crto]

The issue seems to be resolved in version: Unsloth 2025.3.17, Transformers: 4.50.0.dev0. vLLM: 0.8.1.