Optional GPU / NPU execution on Linux

[BorisLord]

Right now inference runs on CPU via ORT. On Linux, we could optionally use the GPU or NPU for faster inference and better battery life.

I'd like to propose contributing this for Linux. I'm not in a position to work on macOS (CoreML) or Windows (DirectML) — those would need to be picked up separately by someone with the matching setup.

The goal is to keep a single .deb that transparently handles all cases: the app probes the available execution providers at startup and picks the best one, silently falling back to CPU if no GPU is usable. No extra packages, no variants to maintain.

Linux targets to probe for:

• Intel iGPU / NPU — OpenVINO. Bundle the runtime inside the .deb (~100 MB added) so Intel users benefit out of the box.
• NVIDIA — CUDA EP, enabled only if the CUDA libraries (libcublas / libcudnn) are already present on the machine. No .deb dependency on the CUDA toolkit, so non-NVIDIA users aren't forced to install several GB of runtime.
• AMD — ROCm is currently too brittle to support.
• WebGPU is still experimental in ort but would eventually replace both OpenVINO and CUDA as a universal Linux path through Vulkan.

CPU remains the default fallback for anything that doesn't find a usable GPU path.

Open questions:

• OK with +~100 MB on the .deb to bundle the OpenVINO runtime?
• Preferred starting point: OpenVINO first (broader hardware reach on Intel laptops) or CUDA opt-in first (simpler to integrate)?

[Kieirra]

Hi Boris, thanks for the write-up and for #310.

The CPU default is deliberate. Even with a load-on-demand scheme, the typical Murmure flow runs Parakeet and the local LLM together (transcribe, then hand off to LLM Connect), so both end up in VRAM at the same time. On an 8 to 12 GB consumer card, that gigabyte for Parakeet competes with headroom the LLM needs, and LLM Connect is a central feature here (Smart Speech Mic and Voice Mode both build on it). CPU inference with the int8 model is already fast enough in daily use, and I am not fully sure a GPU path would bring a clear battery win on short dictation either.

Beyond that, I try to keep the code simple and minimal wherever I can, so maintenance stays manageable in the long run. Probing providers, bundling OpenVINO, handling silent fallbacks all add real surface area, and you noted yourself that ROCm is too brittle. Adding +100 MB to the .deb right after #310 also feels like a small step back on that effort.

For now I would rather not take this on, but I am not closing the door for the future. Does that make sense?

[BorisLord]

Yeah, makes sense — VRAM co-existence convinced me. Parakeet + Ollama on an 8–12 GB card might push each other out.

Small thing on #310: the win was mostly binary (-22 MB); the .deb barely moved (-5 MB, Parakeet dominates). OpenVINO here would be a feature/weight trade, not an undo.
Not the moment to take this one