The merged model has no inference process

[xingxindrst]

I used the method “model.save_pretrained_merged(lora_path, tokenizer, save_method = "merged_16bit")” to save the base model and adapter, but when I tested，the model can only answer in the input data format I gave, and I can't get the inference process.What‘s the reason for this?
Also, I tested the llama3 8b model downloaded by unsloth on my specific task, and his answer was not as good as the original llama3 8b.Will its performance be different from the original llama3?

[danielhanchen]

Apologies on the delay - after finetuning, the model will align itself to answer in the way you trained the model - to retain a model's previous capabilities, try concating your dataset with some general datasets.

The model we uploaded should be identical to HF - are you referring to the 4bit quantized one?

[xingxindrst]

Apologies on the delay - after finetuning, the model will align itself to answer in the way you trained the model - to retain a model's previous capabilities, try concating your dataset with some general datasets.

The model we uploaded should be identical to HF - are you referring to the 4bit quantized one?

I think what I am training is an adapter. After merging with the original model, I can theoretically obtain the inference process.

[xingxindrst]

抱歉延迟了 - 微调后，模型将按照您训练模型的方式进行调整以回答问题 - 为了保留模型以前的功能，请尝试将您的数据集与一些通用数据集连接起来。

我们上传的模型应该与 HF 相同 - 您指的是 4 位量化模型吗？
I tried llama3 8b and it can get the inference process, but llama3.1 8b can't.

The “###Response“ part is the fine-tuning data I input, and the ”###Reason“ is the output of the basic model.

[xingxindrst]

It seems that the reasoning process can be obtained using the unsloth2024.7 version, but not the 2024.8 version.

[rolandtannous]

A fix has now been merged into pypi for both vision and language models and extensively tested.
reference:
unslothai/unsloth-zoo#135
unslothai/unsloth-zoo#158
#2673
Merging and saving should work now.

Best Practice is save to 16bits (which is the default operation if you do not specify anything), and load in 4 bits if you have to.
For production systems, you might wanna consider post training quantization methods with calibration. These take longer but can also be more reliable.
Saving and merging to 4bits might degrade performance of the model.

To access this recent fix, and if you're running this in a local python environment, please make sure to upgrade your python environment using

pip install --upgrade unsloth_zoo
pip install --upgrade unsloth

If you're in colab, the notebooks install cell will also be updated shortly but should be something like

%%capture
import os
if "COLAB_" not in "".join(os.environ.keys()):
    !pip install unsloth
else:
    # Do this only in Colab notebooks! Otherwise use pip install unsloth
    !pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl triton transformers cut_cross_entropy unsloth_zoo
    !pip install sentencepiece protobuf "datasets>=3.4.1" huggingface_hub hf_transfer
    !pip install --no-deps unsloth

as we're no longer pinning transformers or trl.

Also note that whenever you need to merge while saving or while pushing to hub, make sure to use

save_pretrained_merged()

and

push_to_hub_merged()

If you are not merging at all, use

save_pretrained

and

push_to_hub

instead.

I'll close this for now. Feel free to comment if you face any problems.
Happy finetuning.