[Bug] Save to GGUF is taking forever

[luke2023]

1. Did you update? pip install --upgrade unsloth unsloth_zoo yes
2. Colab or Kaggle or local / cloud Local WSL
3. Number GPUs used, use nvidia-smi 1, single 4090
4. Which notebook? Please link! Gemma3-vision+ save GGUF
5. Which Unsloth version, TRL version, transformers version, PyTorch version? Latest version
6. Which trainer? SFTTrainer, GRPOTrainer etc SFTTrainer

model save_pretrained_gguf("-eye-v1"
',quantization_method = "q4_k_m")

Using Gemma3:4b(vision) notebook,ran for 480+ mins but bo other output , also tried this on another WSL computer

[rolandtannous]

Hello,

We are currently dealing with some issues related to save_pretrained_gguf that we are working on resolving. Meanwhile there is some code that uses llama.cpp that you might be able to incorporate : https://colab.research.google.com/drive/1EevDuEa_CYZt59eMGl7acB6DhlApj61O?usp=sharing

Please take note that if you are trying to save a vision model, like Gemma3-4b, you might have to also adjust the code i linked above to align with the docs at llama.cpp as shown here:
https://github.com/ggml-org/llama.cpp/tree/master/docs/multimodal

Hopefully we'll be resolving the gguf saving issues soon.
Thanks

[rolandtannous]

@luke2023

We just pushed a major fix for llamacpp integration.

1- Please make sure to update your local install of unsloth

pip install --force-reinstall --no-deps unsloth-zoo unsloth

If you're using a notebook on kaggle or colab, it should pull the latest version automatically.

2- You can now use save_pretrained_gguf to quantize and save to GGUF format:

model.save_pretrained_gguf(
            save_directory,
            tokenizer=tokenizer,
            quantization_method=quantization_method,
        )

For vision/multimodal models, use tokenizer=processor

3- You can pass it one or several quant methods (as a list):

quantization_method = "q8_0"

or

quantization_method = ["q8_0", "q4_k_m"]

We support all the quants mentioned in this page except the iq variants.

4- If the model is supported by Ollama, it will now also automatically create a Modelfile for you that you can use to create your model in ollama using:

ollama create model_name -f ./Modelfile

5- Use push_to_hub_gguf if you want to convert to gguf and push to Huggingface Hub instead.

6- Should work for all models supported by unsloth and ollama.

7- It now fully supports linux debian and RPM variants. We are still testing windows (if someone has a windows machine and is on our discord channel ping me to help debug)

Have fun

[luke2023]

@rolandtannous
Thank you for the response, I should test this out as soon as possible (expected before Christmas)

[rolandtannous]

Will be closing this for now as the issue has been resolved. Please make sure to follow the instructions given in my last response.

Meanwhile , please feel free to respond if you're still having any issues.

Thank you