Improve Mistral integration

[juliendenize]

Context

To improve the integration of Mistral we propose to include the following features in llama.cpp:

1. a dedicated script for Mistral models that rely on the official implementation:convert_mistral_to_gguf.py. It uses our tokenizer, parameters and safetensor files.
2. add the mistral architecture as the supported model.
3. add the possibility to pass token ids to the llama.cpp server /completions route. This feature is important to allow using llama.cpp models that do not support chat templates.

For the 3. feature we use the following branch of mistral-common: improve_llama_cpp_integration with the following PR mistralai/mistral-common#113. This branch adds a FastAPI app to mistral-common to ease the encoding and decoding of requests.

Code snippet

(optional) Convert a model

HF_TOKEN=... python convert_mistral_to_gguf.py \
https://huggingface.co/mistralai/Devstral-Small-2505 --ctx-train 131072 --outtype bf16

Launch Mistral-Common server

First install mistral-common from the FastAPI branch:

pip install git+https://github.com/mistralai/mistral-common.git@improve_llama_cpp_integration[server]

Then launch the server:

HF_TOKEN=... mistral_common mistralai/Devstral-Small-2505 --port 6000

Launch llama.cpp server

We assume that devstralQ4_K_M.gguf weights have been downloaded following this snippet: https://huggingface.co/mistralai/Devstral-Small-2505_gguf#local-inference-gguf

Launch the llama.cpp server:

./build/bin/llama-server  -c 0 --top-p 0.95 --temp 0.15 \
-m mistralai/Devstral-Small-2505_gguf/devstralQ4_K_M.gguf

Perform completion inference

import requests

mistral_common_url = "http://127.0.0.1:6000"
llama_cpp_url = "http://127.0.0.1:8080"

def tokenize(messages, url):
    response = requests.post(f"{url}/tokenize/messages", json=messages)
    return response.json()

def detokenize(tokens, url):
    response = requests.post(f"{url}/detokenize", json={"tokens": tokens})
    return response.json()

def detokenize_message(tokens, url):
    response = requests.post(f"{url}/detokenize", json={"tokens": tokens, "as_message": True})
    return response.json()

def generate(tokens, url):
    response = requests.post(f"{url}/completions", json={
        "prompt": tokens,
        "stream": False,
        "return_tokens": True
    })
    return response.json()

messages = [
    {"role": "system", "content": "You are Devstral a cool coding agent that can help users with their coding needs."},
    {"role": "user", "content": "Who are you and what can you do?"}
]

tokens = tokenize(messages, mistral_common_url)
print(tokens)

generated = generate(tokens, llama_cpp_url)["tokens"]
print(generated)

detokenized = detokenize(generated, mistral_common_url)
print(detokenized)

detokenized_message = detokenize_message(generated, mistral_common_url)
print(detokenized_message)

You can also tokenize ChatCompletionRequest or OpenAI requests using this:

def tokenize_request(chat_request, url):
    if not isinstance(chat_request, dict):
        chat_request = chat_request.model_dump()
    response = requests.post(f"{url}/tokenize/request", json=chat_request)
    return response.json()

request = {
    "messages": messages,
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location"],
                },
            },
        }
    ],
}

tokenize_request(request, mistral_common_url)