feat: IQ quants support

[mr-september]

Problem
GGUF models quantized with IQ quants fail to load.

Success Criteria
Load and play as usual

Additional context
IQ quants: ggml-org/llama.cpp#4773

Example model with both traditional Q and new IQ quants: https://huggingface.co/bartowski/Starling_Monarch_Westlake_Garten-7B-v0.1-GGUF

[Van-QA]

hi @mr-september,

• As I tested, the imported gguf model (Starling_Monarch_Westlake_Garten-7B-v0.1-Q2_K) is working. Can you tell us more details about the issue that you are facing?

• If the issue is about broken UI due to long model name, we will resolve it soon 🙏
• In case you need it, here is the guideline on how to import GGUF model: https://jan.ai/docs/models#import-or-symlink-local-models.

Many thanks

[mr-september]

Thanks for the quick reply. I think the models you downloaded are using traditional quants, could you please check with an IQ quant model? For example:

Also, is there some logs or some other outputs I could share which may help with troubleshooting?

[Van-QA]

thank @mr-september,
We were able to reproduce the issue using Starling_Monarch_Westlake_Garten-7B-v0.1-IQ4_XS.gguf. Dev team will investigate the issue soon.