[Bug] Unsloth cannot convert fine-tuned model based on unsloth/phi-4 to GGUF because embedded llama.cpp does not support the architecture LlamaModel

[xyehya]

Describe the bug
Unsloth cannot convert fine-tuned model based on unsloth/phi-4 to GGUF because embedded llama.cpp does not support the architecture LlamaModel which was embedded in phi-4 by unsloth/phi-4 as a bug fix.

1. Environment Setup:
   • OS: [e.g., Ubuntu 2.04]
   • Python Version: [e.g., 3.10]
   • Frameworks/Libraries: unsloth
   • colab / script - was this run in colab or as a script: trying both and same result llama.cpp error (no support for LLamaModel.

2/3. Model Details:

• Model ID: unsloth/phi-4
• Model Configuration: [e.g., lora params, quantization, etc.]

4. Training Configuration:

   • Trainer Args: Not Applicable

5. Reproduction Steps:

   • Minimal script to reproduce error:
     model.save_pretrained_gguf(
     "phi-4-finetune",
     quantization_type = "Q8_0",
     )

6. Expected Behavior:

   • Convert to GGUF

7. Actual Behavior:

   • llama.cpp used for conversion in the script fails to convert the phi-4 fine-tune as the morphed architecture is not supported by llama.cpp

[github-actions]

Is this issue still important to you? Apologies in advance we might have missed this issue as well. For faster response times, please post on our Reddit server - https://www.reddit.com/r/unsloth or our Discord - https://discord.com/invite/unsloth

[rolandtannous]

@xyehya

We just pushed a major fix for llamacpp integration.

1- Please make sure to update your local install of unsloth

pip install --force-reinstall --no-deps unsloth-zoo unsloth

If you're using a notebook on kaggle or colab, it should pull the latest version automatically.

2- You can now use save_pretrained_gguf to quantize and save to GGUF format:

model.save_pretrained_gguf(
            save_directory,
            tokenizer=tokenizer,
            quantization_method=quantization_method,
        )

For vision/multimodal models, use tokenizer=processor

3- You can pass it one or several quant methods (as a list):

quantization_method = "q8_0"

or

quantization_method = ["q8_0", "q4_k_m"]

We support all the quants mentioned in this page except the iq variants.

4- If the model is supported by Ollama, it will now also automatically create a Modelfile for you that you can use to create your model in ollama using:

ollama create model_name -f ./Modelfile

5- Use push_to_hub_gguf if you want to convert to gguf and push to Huggingface Hub instead.

6- Should work for all models supported by unsloth and ollama.

7- It now fully supports linux debian and RPM variants. We are still testing windows (if someone has a windows machine and is on our discord channel ping me to help debug)

Just tested phi-4 and it worked nicely with llama.cpp and ollama.

Let me know how it goes for you.

Thanks!