Support Voxtral (MistralAI) via LiteLLM Gateway

[flefevre]

Hello,

First of all, thank you for this project!

I would like to know if Murmure could support using a Voxtral LLM from MistralAI served through a LiteLLM gateway.

In my setup, LiteLLM acts as a unified gateway for multiple models and providers. It exposes an OpenAI-compatible API endpoint while routing requests to different backends (including Mistral models). This allows switching models or providers without changing client integrations.

The idea would be to configure Murmure to send requests to a LiteLLM endpoint, which would then forward them to a Voxtral model from MistralAI, more specialised for french language.

Example architecture:

Murmure
↓
LiteLLM Gateway (OpenAI-compatible API)
↓
Voxtral (MistralAI)

Questions

• Does Murmure currently support using an OpenAI-compatible endpoint that is not the official OpenAI API?
• If not, would you be open to supporting custom OpenAI-compatible endpoints such as LiteLLM?
• Are there any architectural constraints that would prevent using Voxtral through such a gateway?

If helpful, I would be happy to test this setup.

Thanks again for the project!

[Kieirra]

Hello,

and thank you for the kind words and for taking the time to create this discussion.

Does Murmure currently support using an OpenAI-compatible endpoint that is not the official OpenAI API?
If not, would you be open to supporting custom OpenAI-compatible endpoints such as LiteLLM?

Version 1.8.0 includes a way to connect LLM models in the LLM Connect feature via an OpenAI-compatible API (official or not) using "Remote Server". I am not familiar with LiteLLM, but normally it should work; that's the purpose of this new feature.

Unfortunately, this is not yet available in version 1.7.0.

Are there any architectural constraints that would prevent using Voxtral through such a gateway?

There are no architectural issues. The reason I didn’t implement it right away was more due to privacy concerns, as if users send their data to the cloud, they lose the privacy benefit of Murmure. However, many users have internal servers or enterprise servers hosting solutions like VLLM, LMStudio, etc. So I couldn’t ignore this use case. I ultimately decided to add the feature, and everyone can now use it, though it does pose some risks if data is sent to the cloud.

If helpful, I would be happy to test this setup.

You can clone the repository and run the command pnpm tauri build to generate a binary for your platform. Otherwise, I can generate a binary for you to test if you tell me the platform you are on, but remember to delete it once version 1.8.0 is released.

[thiswillbeyourgithub]

I suspect there might be a misunderstanding here. The user asked if it could use a remote model for transcription. But you mentioned using online services for LLM string modification.

Unless I'm mistaken:

• Voxtral is an "audio to text" model. Two versions : one in a small model to download, and one beefy API-only model.
• Murmure 1.8.0, AFAIU, allows using a remote API to call LLM, giving them as inputs the output of the local transcription model. So the API only allows "text to text" models.

[Kieirra]

Yes, there is an ambiguity because Voxtral seems to be able to handle both text and audio, So I don't really know if the poster wants to "replace parakeet" or just use it in post-processing mode. I assumed he wanted to use it in text mode but I might be wrong.

But yes, in Murmure, users can't replace parakeet simply.

Murmure 1.8.0, AFAIU, allows using a remote API to call LLM, giving them as inputs the output of the local transcription model. So the API only allows "text to text" models.

Correct.