unsloth=2025.2.15 training result is werd when torch=2.6

[GonChen]

Reminder, the training result is werd when torch=2.6 unsloth=2025.2.15
using same script same model.
Resolve method for me is using torch=2.5 forcely.

conda create --name unsloth_env2     python=3.11     pytorch-cuda=12.1     pytorch=2.5 cudatoolkit xformers -c pytorch -c nvidia -c xformers     -y
pip3 install unsloth torch==2.5.1

This problem troubles me for two days.

[danielhanchen]

Does the training losses look wrong with 2.6?

[zwap-in]

We’re experiencing the same issue with torch 2.5.1, 2.5.0, and 2.6.0. Everything was working fine until two days ago. Any suggestions?

[Oleffa]

I just revisited a colab project from 20.1.2025 (pip install unsloth, no specific version, no code changed since last time using).
During inference, with Gemma 2 27b, 4bit, it would not generate responses anymore for a prompt where with more than 5k tokens despite max_seq_size set to 8192. After some messing around I found that for input sequences longer than 4096 no answer was generated no matter the max_seq_size parameter.
So for now I am truncating input sequences to 4095 with max_new tokens = 2000 and it works as before. This is weird because a few weeks ago I didnt have to worry about max_seq_size.

I am not very knowledgeable with how unsloth works in detail and I only tested this for Gemma2 27b and a fine-tuned version of the same model, but I have the feeling something about how unsloth manages sequence lengths changed, perhaps RoPE? I have this assumption because in all unsloth examples it usually says this:

max_seq_length = 2048 # Supports RoPE Scaling interally, so choose any!

Perhaps you could try limiting the maximum sequence length during training to something small or try inference on your model with long prompts to see if you have the same issue.

Edit: I tried the same things with torch 2.5.1 and 2.6.0, same behavior. I tried running the code with unsloth==2025.1.6 (the version at the date of when things still worked but I get this error: #1829 (comment))

[danielhanchen]

I'm unsure if I solved the issue, but try the latest version! For Colab / Kaggle, please restart and run all. For local machines, please do:

pip install --force-reinstall --upgrade --no-cache-dir --no-deps unsloth unsloth_zoo