save_pretrained_merged doesn't merge the model

[neoneye]

Problem

My goal, I want to save the merged model as a GGUF file, but I'm getting various errors.

The deeper problem seems to be that merging lora+base model isn't saving a merged file.

I think I have successfully done the merging of lora+base model, around 7..14 days ago. Maybe it's something that have broken recently.

Details

My notebook google colab is based on unsloth/llama-3-8b-bnb-4bit and trained using unsloth colab notebook.

My model neoneye/base64-decode-v2-attempt12 contains the adapter_model.safetensors file. It does not contain the full merged model.

I can continue train on my model, and it loads the adapter + base model. So the loading of the LoRA and model is working. The training is working. The push_to_hub is working.

However merging the LoRA with the base model isn't working.

if True: model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit",)

This is the output from save_pretrained_merged. There are no errors.

Unsloth: You have 1 CPUs. Using `safe_serialization` is 10x slower.
We shall switch to Pytorch saving, which will take 3 minutes and not 30 minutes.
To force `safe_serialization`, set it to `None` instead.
Unsloth: Kaggle/Colab has limited disk space. We need to delete the downloaded
model which will save 4-16GB of disk space, allowing you to save on Kaggle/Colab.
Unsloth: Will remove a cached repo with size 5.7G

Unsloth: Merging 4bit and LoRA weights to 16bit...
Unsloth: Will use up to 6.17 out of 12.67 RAM for saving.

 41%|████      | 13/32 [00:01<00:01, 13.39it/s]We will save to Disk and not RAM now.
100%|██████████| 32/32 [00:42<00:00,  1.33s/it]

Unsloth: Saving tokenizer... Done.
Unsloth: Saving model... This might take 5 minutes for Llama-7b...

/usr/local/lib/python3.10/dist-packages/transformers/integrations/peft.py:399: FutureWarning: The `active_adapter` method is deprecated and will be removed in a future version.
  warnings.warn(

config.json: 100%
 1.20k/1.20k [00:00<00:00, 68.7kB/s]

Unsloth: Saving model/adapter_model.bin...
Done.

The biggest file is the lora file, 167mb. It seems like there is no merged file. I guess it should be generating a file around the same size as the base model or bigger, between 5..10 gb. But there is no such file. And no error about no file being generated.

/content/model# ls -la
total 172948
drwxr-xr-x 2 root root      4096 Jun  9 15:20 .
drwxr-xr-x 1 root root      4096 Jun  9 15:24 ..
-rw-r--r-- 1 root root       732 Jun  9 15:23 adapter_config.json
-rw-r--r-- 1 root root 167934026 Jun  9 15:23 adapter_model.bin
-rw-r--r-- 1 root root       172 Jun  9 15:23 generation_config.json
-rw-r--r-- 1 root root       464 Jun  9 15:23 special_tokens_map.json
-rw-r--r-- 1 root root     50614 Jun  9 15:23 tokenizer_config.json
-rw-r--r-- 1 root root   9085698 Jun  9 15:23 tokenizer.json

I'm on Google Colab with plenty of disk space.

Connected to
Python 3 Google Compute Engine backend (GPU)
RAM: 2.91 GB/12.67 GB
Disk: 29.36 GB/201.23 GB

Solution ideas

Am I correct that save_pretrained_merged should output a big merged file?

Inside save_pretrained_merged, check if the output file was generated, if there is no file then print an error.

[danielhanchen]

Weird hmm let me try it in colab

[neph1]

Sounds like my issue: #609

[ai-nikolai]

@danielhanchen is that working now? (What is the preferred usage?)

[Salary-only-17k]

is that working now? (What is the preferred usage?)

[ai-nikolai]

@neph1 Could you provide a bit more clarity (if you know) - how the adapter merging / non-merging is working? (I.e. what happens during the merging and what is the recommended use).

[BaiqingL]

+1 to this issue, using Qwen3 4B model with the provided notebooks https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb#scrollTo=QyEjW-WuYQIm

[BaiqingL]

Adding on, I did solve the issue, I forgot to load the peft model after loading the base model, an error here would be nice!

[rolandtannous]

@BaiqingL @Salary-only-17k @ai-nikolai @ivsanro1 @neoneye @neph1 @yangyushi

A fix has now been merged into pypi for both vision and language models and extensively tested.
reference:
unslothai/unsloth-zoo#135
unslothai/unsloth-zoo#158
#2673
Merging and saving should work now.

Best Practice is save to 16bits (which is the default operation if you do not specify anything), and load in 4 bits if you have to.
For production systems, you might wanna consider post training quantization methods with calibration. These take longer but can also be more reliable.
Saving and merging to 4bits might degrade performance of the model.

To access this recent fix, and if you're running this in a local python environment, please make sure to upgrade your python environment using

pip install --upgrade unsloth_zoo
pip install --upgrade unsloth

If you're in colab, the notebooks install cell will also be updated shortly but should be something like

%%capture
import os
if "COLAB_" not in "".join(os.environ.keys()):
    !pip install unsloth
else:
    # Do this only in Colab notebooks! Otherwise use pip install unsloth
    !pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl triton transformers cut_cross_entropy unsloth_zoo
    !pip install sentencepiece protobuf "datasets>=3.4.1" huggingface_hub hf_transfer
    !pip install --no-deps unsloth

as we're no longer pinning transformers or trl.

Also note that whenever you need to merge while saving or while pushing to hub, make sure to use

save_pretrained_merged()

and

push_to_hub_merged()

If you are not merging at all, use

save_pretrained

and

push_to_hub

instead.

I'll close this for now. Feel free to comment if you face any problems.
Happy finetuning.