[Bug] The model.save_pretrained_gguf() function wont run in .venv

[OliDo99]

The model.save_pretrained_gguf() function wont run in .venv environment the only thing that solved the issue was changing enviroments to conda
Saving a model in .venv using said function will throw this error -> Unsloth: The file ('llama.cpp/llama-quantize' or 'llama.cpp/llama-quantize.exe' if you are on Windows WSL) or 'llama.cpp/quantize' does not exist.
But we expect this file to exist! Maybe the llama.cpp developers changed the name or check extension of the llama-quantize file.

But the same code with same setup will run perfectly in conda
Is that a feature or a bug?

1. Dataset Details:

   • Dataset: amarmahdi1112/human_chat_dataset_processed
   • Formating Function:
     def format_instruction(sample):
     return {
     "text": tokenizer.apply_chat_template(
     [
     {"role": "user", "content": sample["instruction"]},
     {"role": "assistant", "content": sample["output"]},
     ],
     tokenize=False,
     add_generation_prompt=False
     )
     }

2. Model Details:
   Model didn't affect the result

3. Training Config:
   trainer = SFTTrainer(
   model = model,
   tokenizer = tokenizer,
   train_dataset = dataset,
   dataset_text_field = "text",
   dataset_num_proc = 2,
   max_seq_length = 2048,
   packing = False,
   args = TrainingArguments(
   per_device_train_batch_size = 2,
   gradient_accumulation_steps = 4,
   warmup_steps = 5,
   max_steps = 60,
   learning_rate = 2e-4,
   fp16 = not is_bfloat16_supported(),
   bf16 = is_bfloat16_supported(),
   logging_steps = 1,
   optim = "adamw_8bit",
   weight_decay = 0.01,
   lr_scheduler_type = "linear",
   seed = 3407,
   output_dir = "outputs",
   report_to = "none",
   ),
   )

4. Reproduction Steps:
   setup a .venv environment
   set a model you wish to fine tune, a tokenizer and the dataset
   set parameters for your

5. Expected Behavior:
   It should run regardless of the environment, right?

6. Actual Behavior:
   No it does not

[rolandtannous]

@OliDo99

We just pushed a major fix for llamacpp integration.

1- Please make sure to update your local install of unsloth

pip install --force-reinstall --no-deps unsloth-zoo unsloth

If you're using a notebook on kaggle or colab, it should pull the latest version automatically.

2- You can now use save_pretrained_gguf to quantize and save to GGUF format:

model.save_pretrained_gguf(
            save_directory,
            tokenizer=tokenizer,
            quantization_method=quantization_method,
        )

For vision/multimodal models, use tokenizer=processor

3- You can pass it one or several quant methods (as a list):

quantization_method = "q8_0"

or

quantization_method = ["q8_0", "q4_k_m"]

We support all the quants mentioned in this page except the iq variants.

4- If the model is supported by Ollama, it will now also automatically create a Modelfile for you that you can use to create your model in ollama using:

ollama create model_name -f ./Modelfile

5- Use push_to_hub_gguf if you want to convert to gguf and push to Huggingface Hub instead.

6- Should work for all models supported by unsloth and ollama.

7- It now fully supports linux debian and RPM variants. We are still testing windows (if someone has a windows machine and is on our discord channel ping me to help debug)

If you can try in .venv and let me know.
Alternatively use conda or its non license restricted equivalent mamba (that's what i use).
Our docker container image is also worth checking out as it comes with a prebuilt env + all the notebooks https://docs.unsloth.ai/get-started/install-and-update/docker

Thanks!

[rolandtannous]

Will be closing this for now as the issue has been resolved. Please make sure to follow the instructions given in my last response.

Meanwhile , please feel free to respond if you're still having any issues.

Thank you