Model Merge reduces performance

[mosama1994]

I continuous pretrained a Qwen2.5 0.5B model. When I load the adapter and merge it and then perform inference, the output looks good. But, when I merge with adapter and save model in 16bit or 4bit using the unsloth saving strategy, and then load the saved model for inference the output is not good, the performance degrades. What is the reason for that?

[BugReporterZ]

I've observed this too while doing continuous pretraining tests of various different models around 7-8B size. The merged models obtained with:

model.save_pretrained_merged(modelname, tokenizer, save_method = "merged_16bit",)

have very poor performance and also tokenizer issues with Llama.cpp when attempting to convert them to GGUF format. You can replace the tokenizer files with those of the base model to make it work, but I suspect that's where the problems originate.

On the other hand, adapter checkpoints separately merged using different methods appear to work as expected.

[danielhanchen]

I'm currently working on making merging a much better experience - unfortunately most issues result from chat template issues, but there does seem to be some issues with merging with 4bit models - I'll update you guys!

[BugReporterZ]

As far as I am aware of, adapter merging should very preferably be done with the original (16-bit) weights and not the 4-bit quantized models, so if that's what's occurring in that step, that might be the reason for the performance loss.

[mosama1994]

Merging with bf16 precision also causes the issue. When you merge with bf16 precision model and save the issue is still there. I used the merged_16bit and also tried the peft library to merge and save. If you merge and then do inference, that is fine. But, if you save merged model and do inference from saved model it is not good.

Also, a different question @danielhanchen what tokenizer patching are we doing in unsloth? Would it be possible for you to list down in a few points what is happening there?

[danielhanchen]

@TechieUser2517 @mosama1994 I'm working on refactoring 16bit merging and tracking it here: #1558 - the goal is to allow you to select if you want to merge to the original 16bit weights or upcast the weights

@mosama1994 Tokenizer patches are generally checking the EOS / BOS / PAD tokens and seeing if they correct. If not, we fix them up internally

[mosama1994]

@danielhanchen But even when i load the base model with no quantization and merge, that also does not work properly. When i load model directly from a lora adapter (load_in_4bit is not True, no quantization), and then do merge_and_unload, it works fine in notebook. However, when i save it using save_pretrained, the saved model when used for inference again is no good at all. Any idea?

from unsloth import FastLanguageModel
import torch
max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
dtype = torch.bfloat16 # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = False # Use 4bit quantization to reduce memory usage. Can be False.

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "mosama/Qwen2.5-0.5B-Kaggle-Float16-Pretrained-arb-eng-urd",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
    token = "my hf token", # use one if using gated models like meta-llama/Llama-2-7b-hf
)

model = model.merge_and_unload().to("cuda:0")

model.save_pretrained("my local path")

# Loading the model now:
from unsloth import FastLanguageModel
import torch
max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
dtype = torch.bfloat16 # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = False # Use 4bit quantization to reduce memory usage. Can be False.

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "my local path",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
    token = "my hf token", # use one if using gated models like meta-llama/Llama-2-7b-hf

FastLanguageModel.for_inference(model)

from transformers import TextStreamer

streamer = TextStreamer(tokenizer)
)

model_inputs = tokenizer("مضمون:", return_tensors="pt").to(model.device)
print(model_inputs)
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=200,
    repetition_penalty=1.2,
    temperature=0.5,
    do_sample=True,
    top_p=0.9,
    top_k=20,
    streamer=streamer
)
print("Generated Response!")
response = tokenizer.decode(generated_ids[0], skip_special_tokens=False)
print(response)

[mosama1994]

@danielhanchen Any update on this? Can you take a look at this? I know you created two separate issues for this, but there has been no update on this performance degradation issue.

[rolandtannous]

@BugReporterZ @mosama1994 @shaper @fenglui @HuggingAha @Osakana7777777

A fix has now been merged into pypi for both vision and language models and extensively tested.
reference:
unslothai/unsloth-zoo#135
unslothai/unsloth-zoo#158
#2673
Merging and saving should work now.

Best Practice is save to 16bits (which is the default operation if you do not specify anything), and load in 4 bits if you have to.
For production systems, you might wanna consider post training quantization methods with calibration. These take longer but can also be more reliable.
Saving and merging to 4bits might degrade performance of the model.

To access this recent fix, and if you're running this in a local python environment, please make sure to upgrade your python environment using

pip install --upgrade unsloth_zoo
pip install --upgrade unsloth

If you're in colab, the notebooks install cell will also be updated shortly but should be something like

%%capture
import os
if "COLAB_" not in "".join(os.environ.keys()):
    !pip install unsloth
else:
    # Do this only in Colab notebooks! Otherwise use pip install unsloth
    !pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl triton transformers cut_cross_entropy unsloth_zoo
    !pip install sentencepiece protobuf "datasets>=3.4.1" huggingface_hub hf_transfer
    !pip install --no-deps unsloth

as we're no longer pinning transformers or trl.

Also note that whenever you need to merge while saving or while pushing to hub, make sure to use

save_pretrained_merged()

and

push_to_hub_merged()

If you are not merging at all, use

save_pretrained

and

push_to_hub

instead.

I'll close this for now. Feel free to comment if you face any problems.
Happy finetuning.