Audio input support (Phi 4 multimodal) (#1448)

* Deps

* Add conformer

* Nemo loading

* Position embeds

* Load t5 attn bias

* Attn and feed forward

* Add conv module and glu pointwise

* Implement relative attn bias

* Add the forward methods

* Add encoder embedding

* Fix oproj

* Some loading

* Conformer loads!

* Fully loading speech stack

* Merger

* Dont need that

* First pass at audio processing

* Read samples

* Optional

* Small loading fix

* Runs but not correct yet

* Improved audio processing?

* Works with this

* Fix t5 attn bias

* It works!

* Comment

* Use some other crates

* Clippy

* Allow bf16 on metal

* Add prefix_audio

* Remove unused

* Typo

* User specified

* Add audio url parsing

* AudioProjectionMode -> InputMode

* Audio prefix caching

* Fix bug in audio prefix caching

* Support both at the same time!

* Tweak logging

* Support stereo

* Add mistralrs-audio

* Support batching

* Add server and rust api example

* Add python api

* Fix add_multimodal_message

* Fix unfold for conformer

* Streaming example

* Add web chat support

* Add modalities registry

Author: EricLBuehler

.typos.toml

@@ -10,7 +10,9 @@ extend-ignore-identifiers-re = [
     "thr",
     "nd",
     "uneeded",
-    "tese"
+    "tese",
+    "seperable",
+    "Seperable",
 ]
 
 [files]

Cargo.lock

@@ -10,6 +10,7 @@ members = [
     "mistralrs-quant",
     "mistralrs-paged-attn",
     "mistralrs-web-chat",
+    "mistralrs-audio",
 ]
 resolver = "2"
 
@@ -140,6 +141,7 @@ ahash = "0.8.12"
 num-traits = "0.2.19"
 libc = "0.2.172"
 bm25 = "2.2.1"
+symphonia = { version = "0.5.4", default-features = false, features = ["mp3", "flac", "vorbis", "wav", "isomp4", "ogg", "pcm"] }
 lazy_static = "1.5"
 paste = "1.0.15"
 byteorder = "1.5.0"
@@ -159,12 +161,18 @@ include_dir = "0.7.4"
 http = "1.3.1"
 hyper = "1.6.0"
 bindgen_cuda = { git = "https://github.com/guoqingbao/bindgen_cuda.git", version = "0.1.6" }
+rubato = "0.16.2"
+rustfft = "6.3.0"
+hound = "3.5.1"
+apodize = "1.0.0"
+
 mistralrs-core = { path = "mistralrs-core" }
 mistralrs-paged-attn = { path = "mistralrs-paged-attn" }
 mistralrs-quant = { path = "mistralrs-quant" }
 mistralrs-vision = { path = "mistralrs-vision" }
 mistralrs-server-core = { path = "mistralrs-server-core" }
 mistralrs = { path = "mistralrs" }
+mistralrs-audio = { path = "mistralrs-audio" }
 
 [profile.release-with-debug]
 inherits = "release"

Cargo.toml

@@ -11,7 +11,7 @@ Blazingly fast LLM inference.
 | <a href="https://ericlbuehler.github.io/mistral.rs/mistralrs/"><b>Rust Documentation</b></a> | <a href="https://github.com/EricLBuehler/mistral.rs/blob/master/mistralrs-pyo3/API.md"><b>Python Documentation</b></a> | <a href="https://discord.gg/SZrecqK8qw"><b>Discord</b></a> | <a href="https://matrix.to/#/#mistral.rs:matrix.org"><b>Matrix</b></a> |
 </p>
 
-Mistral.rs is a cross-platform, highly multimodal inference engine featuring support for **text**, **vision**, **image generation**, and **speech generation** models!
+Mistral.rs is a cross-platform, highly-multimodal inference engine that brings you local **text→text**, **text + vision→text**, **text + speech + vision→text**, **text→speech**, and **text→image** workflows — all in one blazing-fast package!
 
 Please submit requests for new models [here](https://github.com/EricLBuehler/mistral.rs/issues/156).