Gemma 3n 4-bit finetune Q4_K_M GGUF support

[aphilas]

Q4_K_M is not supported:

RuntimeError: Unsloth: `q4_k_m` quantization type is not supported.
The following quantization types are supported: `['f32', 'f16', 'bf16', 'q8_0']`

Originally posted by @aphilas in #2924

[shimmyshimmer]

Hi there this is a feature request for finetuning GGUF models. At the moment you can finetune safetensor files with Unsloth. See our collection of safetensor models here: https://docs.unsloth.ai/get-started/all-our-models

We will work on supporting GGUF models in the future

[aphilas]

Hi there this is a feature request for finetuning GGUF models.

This is a feature request for saving a finetuned Gemma 3n model to GGUF with Q4_K_M quantization.

[rolandtannous]

I am pushing a PR in the next two days that will allow you to use any quantization scheme allowed by llama.cpp. I'll ping you once it's done.

[rolandtannous]

@aphilas @qingy1337

We just pushed a major fix for llamacpp integration.

1- Please make sure to update your local install of unsloth

pip install --force-reinstall --no-deps unsloth-zoo unsloth

If you're using a notebook on kaggle or colab, it should pull the latest version automatically.

2- You can now use save_pretrained_gguf to quantize and save to GGUF format:

model.save_pretrained_gguf(
            save_directory,
            tokenizer=tokenizer,
            quantization_method=quantization_method,
        )

For vision/multimodal models, use tokenizer=processor

3- You can pass it one or several quant methods (as a list):

quantization_method = "q8_0"

or

quantization_method = ["q8_0", "q4_k_m"]

We support all the quants mentioned in this page except the iq variants.

4- If the model is supported by Ollama, it will now also automatically create a Modelfile for you that you can use to create your model in ollama using:

ollama create model_name -f ./Modelfile

5- Use push_to_hub_gguf if you want to convert to gguf and push to Huggingface Hub instead.

6- Should work for all models supported by unsloth and ollama.

7- It now fully supports linux debian and RPM variants. We are still testing windows (if someone has a windows machine and is on our discord channel ping me to help debug)

Thanks!

[aphilas]

tried to run the notebook but running into oom (might be #3373). i don't have enough time to investigate atm.

[rolandtannous]

@aphilas Yes this is a different issue altogether.
Please feel free to follow up on that issue on #3373

Will be closing this for now as the particular issue mentioned here has been resolved. Please make sure to follow the instructions given in my last response.

Meanwhile , please feel free to respond if you're still having any issues.

Thank you