anyone has experienced different results after merging the adapters

[acsankar]

After fine-tuning Llama-3 8B 4-bit the results are okay in colab (getting the LLM output can see the results are from custom dataset). This is before merging, Inferencing directly in the below step of Llama-3 unsloth template

But when push the model using the below step by using 16bit to hub, the results are widely varying. I have tried to do 4bit as well

I don't think it's to do with temperature or anything. I feel I am doing something wrong, or the merge is not happening properly as the results are completely different. No information from fine-tuning dataset is reflected. Anyone has experienced this and any potential solutions? How to validate if adapters are merged? Thanks in advance

[danielhanchen]

@acsankar Did you use a chat template with the merged model? Alpaca style?

[vjagannath786]

Experiencing same issue, nor the adapter nor the merged model are behaving weirdly after saving to hub or locally. Performance of model degrades.

[rolandtannous]

@acsankar @vjagannath786

A fix has now been merged into pypi for both vision and language models and extensively tested.
reference:
unslothai/unsloth-zoo#135
unslothai/unsloth-zoo#158
#2673
Merging and saving should work now.

Best Practice is save to 16bits (which is the default operation if you do not specify anything), and load in 4 bits if you have to.
For production systems, you might wanna consider post training quantization methods with calibration. These take longer but can also be more reliable.
Saving and merging to 4bits might degrade performance of the model.

To access this recent fix, and if you're running this in a local python environment, please make sure to upgrade your python environment using

pip install --upgrade unsloth_zoo
pip install --upgrade unsloth

If you're in colab, the notebooks install cell will also be updated shortly but should be something like

%%capture
import os
if "COLAB_" not in "".join(os.environ.keys()):
    !pip install unsloth
else:
    # Do this only in Colab notebooks! Otherwise use pip install unsloth
    !pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl triton transformers cut_cross_entropy unsloth_zoo
    !pip install sentencepiece protobuf "datasets>=3.4.1" huggingface_hub hf_transfer
    !pip install --no-deps unsloth

as we're no longer pinning transformers or trl.

Also note that whenever you need to merge while saving or while pushing to hub, make sure to use

save_pretrained_merged()

and

push_to_hub_merged()

If you are not merging at all, use

save_pretrained

and

push_to_hub

instead.

I'll close this for now. Feel free to comment if you face any problems.
Happy finetuning.