[Bug] Issue while saving to gguf

[B-Ismail]

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[23], line 1
----> 1 model.save_pretrained_gguf("mistral25", tokenizer, quantization_method = "q4_k_m")

File ~/finetuning/venv/lib/python3.12/site-packages/unsloth/save.py:1865, in unsloth_save_pretrained_gguf(self, save_directory, tokenizer, quantization_method, first_conversion, push_to_hub, token, private, is_main_process, state_dict, save_function, max_shard_size, safe_serialization, variant, save_peft_format, tags, temporary_location, maximum_memory_usage)
   1862 is_sentencepiece_model = check_if_sentencepiece_model(self)
   1864 # Save to GGUF
-> 1865 all_file_locations, want_full_precision = save_to_gguf(
   1866     model_type, model_dtype, is_sentencepiece_model,
   1867     new_save_directory, quantization_method, first_conversion, makefile,
   1868 )
   1870 # Save Ollama modelfile
   1871 modelfile = create_ollama_modelfile(tokenizer, all_file_locations[0])

File ~/finetuning/venv/lib/python3.12/site-packages/unsloth/save.py:1093, in save_to_gguf(model_type, model_dtype, is_sentencepiece, model_directory, quantization_method, first_conversion, _run_installer)
   1091     quantize_location = "llama.cpp/build/bin/quantize"
   1092 else:
-> 1093     raise RuntimeError(
   1094         "Unsloth: The file 'llama.cpp/llama-quantize' or `llama.cpp/quantize` does not exist.\n"\
   1095         "We've also double checked the building directory under 'llama.cpp/build/bin/'.\n"\
   1096         "But we expect this file to exist! Check if the file exists under llama.cpp and investigate the building process of llama.cpp (make/cmake)!"
   1097     )
   1098 pass
   1100 # See https://github.com/unslothai/unsloth/pull/730
   1101 # Filenames changed again!

RuntimeError: Unsloth: The file 'llama.cpp/llama-quantize' or `llama.cpp/quantize` does not exist.
We've also double checked the building directory under 'llama.cpp/build/bin/'.
But we expect this file to exist! Check if the file exists under llama.cpp and investigate the building process of llama.cpp (make/cmake)!

can you help me identify the issue here .

i'm using :

pip show unsloth
Name: unsloth
Version: 2025.8.4
Summary: 2-5X faster LLM finetuning
Home-page:
Author: Unsloth AI team
Author-email: info@unsloth.ai
License:
Location: /home/user7/finetuning/venv/lib/python3.12/site-packages
Requires: accelerate, bitsandbytes, datasets, diffusers, hf_transfer, huggingface_hub, numpy, packaging, peft, protobuf, psutil, sentencepiece, torch, torchvision, tqdm, transformers, triton, trl, tyro, unsloth_zoo, wheel, xformers
Required-by:

[Hochikong]

You can download a prebuilt release of llama.cpp, then copy all the files in the bin folder to the llama.cpp directory inside your Unsloth working directory.

Example:

I trained my model in the following path: /home/jack/LLM/UnslothTesting, and a folder named llama.cpp was created there. Simply copy all the executable files from the prebuilt llama.cpp package to /home/jack/LLM/UnslothTesting/llama.cpp. This should solve your problem.

[rolandtannous]

Hello,

We are currently dealing with some issues related to save_pretrained_gguf that we are working on resolving. Meanwhile there is some code that uses llama.cpp that you might be able to incorporate : https://colab.research.google.com/drive/1EevDuEa_CYZt59eMGl7acB6DhlApj61O?usp=sharing

Please take note that if you are trying to save a vision model you might have to also adjust the code i linked above to align with the docs at llama.cpp as shown here:
https://github.com/ggml-org/llama.cpp/tree/master/docs/multimodal

Hopefully we'll be resolving the gguf saving issues soon.
Thanks

[rolandtannous]

@B-Ismail @Hochikong @Isam-tfares

We just pushed a major fix for llamacpp integration.

1- Please make sure to update your local install of unsloth

pip install --force-reinstall --no-deps unsloth-zoo unsloth

If you're using a notebook on kaggle or colab, it should pull the latest version automatically.

2- You can now use save_pretrained_gguf to quantize and save to GGUF format:

model.save_pretrained_gguf(
            save_directory,
            tokenizer=tokenizer,
            quantization_method=quantization_method,
        )

For vision/multimodal models, use tokenizer=processor

3- You can pass it one or several quant methods (as a list):

quantization_method = "q8_0"

or

quantization_method = ["q8_0", "q4_k_m"]

We support all the quants mentioned in this page except the iq variants.

4- If the model is supported by Ollama, it will now also automatically create a Modelfile for you that you can use to create your model in ollama using:

ollama create model_name -f ./Modelfile

5- Use push_to_hub_gguf if you want to convert to gguf and push to Huggingface Hub instead.

6- Should work for all models supported by unsloth and ollama.

7- It now fully supports linux debian and RPM variants. We are still testing windows (if someone has a windows machine and is on our discord channel ping me to help debug)

8 - If you are on a local machine and can run containers, I highly recommend that you use our official docker containerized environment. No more dependency conflicts. It's safe (non root user) and comes with a prebuilt pretested environment and the most up to date notebooks for you to run. You can find the Docker container image here: https://hub.docker.com/r/unsloth/unsloth
The how to guide for the docker container can be found here: https://docs.unsloth.ai/get-started/install-and-update/docker

Thanks!

[rolandtannous]

Will be closing this for now as the issue has been resolved. Please make sure to follow the instructions given in my last response.

Meanwhile , please feel free to respond if you're still having any issues.

Thank you