Add automatic vision loader type

[EricLBuehler]

Summary by CodeRabbit

• New Features

  • Vision model initialization now automatically selects the appropriate loader type based on the model configuration, removing the need to specify the loader type manually.
  • Added support for optionally specifying the architecture when building vision models.

• Refactor

  • Simplified vision model builder APIs, making the loader type parameter optional and introducing a method to set it if needed.
  • Updated trait and method signatures to accept configuration parameters, enabling more flexible model handling.
  • Renamed and reorganized loader types for clarity and improved modularity.

• Style

  • Example usages and documentation updated to remove explicit architecture or loader type specifications, reflecting the streamlined initialization process.

[coderabbitai]

Walkthrough

This update refactors the handling of vision model loader types throughout the codebase. The vision loader type is now optional in model selection and builder patterns, enabling automatic detection from model configuration when not explicitly provided. Public API signatures, trait implementations, and builder methods have been adjusted to support this flexibility, and example usages have been simplified accordingly.

Changes

File(s)	Change Summary
mistralrs-core/src/model_selected.rs mistralrs-core/src/toml_selector.rs	Changed the arch field in VisionPlain variant from required to optional (Option<VisionLoaderType>) in both runtime and TOML model selection enums.
mistralrs-core/src/pipeline/loaders/normal_loaders.rs mistralrs-core/src/pipeline/loaders/mod.rs mistralrs-core/src/pipeline/mod.rs mistralrs-core/src/pipeline/normal.rs	Renamed AutoLoader to AutoNormalLoader and updated all references and exports. Adjusted imports and usage in the normal pipeline module.
mistralrs-core/src/pipeline/loaders/vision_loaders.rs mistralrs-core/src/pipeline/vision.rs	Introduced AutoVisionLoader for automatic loader selection based on config. Updated VisionModelLoader trait signatures to accept config parameters. Added loader selection logic and delegation. Adjusted builder to take Option<VisionLoaderType>.
mistralrs-pyo3/src/which.rs	Made the arch parameter in Plain and VisionPlain variants of the Which enum optional (Option<...>) with default None.
mistralrs-pyo3/src/lib.rs	Updated parse_which to conditionally convert arch using .map(Into::into) for VisionPlain.
mistralrs/src/vision_model.rs	Made loader_type in VisionModelBuilder optional. Changed constructor to omit loader type and added with_loader_type method. Updated UqffVisionModelBuilder accordingly.
mistralrs-core/src/pipeline/vision.rs	Updated loader builder to accept Option<VisionLoaderType>. Adjusted method calls to pass config parameter.
mistralrs/examples/gemma3/main.rs mistralrs/examples/idefics2/main.rs mistralrs/examples/idefics3/main.rs mistralrs/examples/llama4/main.rs mistralrs/examples/llama_vision/main.rs mistralrs/examples/llama_vision_multiturn/main.rs mistralrs/examples/llava/main.rs mistralrs/examples/llava_next/main.rs mistralrs/examples/minicpmo_2_6/main.rs mistralrs/examples/mistral3/main.rs mistralrs/examples/phi3v/main.rs mistralrs/examples/phi4mm/main.rs mistralrs/examples/qwen2_5vl/main.rs mistralrs/examples/qwen2vl/main.rs mistralrs/examples/smolvlm/main.rs mistralrs/examples/uqff_vision/main.rs mistralrs/examples/vision_auto_device_map/main.rs	Removed explicit VisionLoaderType arguments from VisionModelBuilder and similar builders in all vision model examples, simplifying initialization.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant VisionModelBuilder
    participant AutoVisionLoader
    participant ConcreteLoader

    User->>VisionModelBuilder: new(model_id)
    User->>VisionModelBuilder: with_loader_type(loader_type)? (optional)
    User->>VisionModelBuilder: build()
    VisionModelBuilder->>AutoVisionLoader: build (if loader_type is None)
    AutoVisionLoader->>AutoVisionLoader: parse config for architecture
    AutoVisionLoader->>ConcreteLoader: instantiate loader based on config
    AutoVisionLoader->>ConcreteLoader: delegate load/configure methods
    ConcreteLoader-->>AutoVisionLoader: result
    AutoVisionLoader-->>VisionModelBuilder: loader instance
    VisionModelBuilder-->>User: ready-to-use model

Loading

Poem

In the warren of code where the vision bunnies dwell,
Loaders now choose themselves—oh, isn’t that swell!
No more must you fuss, or specify each kind,
The right loader hops out, with config in mind.
With options and traits all neatly aligned,
This patch brings new freedom—of the automatic kind!
🐇✨

Tip

⚡️ Faster reviews with caching

• CodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 16th. To opt out, configure Review - Disable Cache at either the organization or repository level. If you prefer to disable all data retention across your organization, simply turn off the Data Retention setting under your Organization Settings.

Enjoy the performance boost—your workflow just got faster.

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                3           62           53            0            9
 Dockerfile              1           41           22           10            9
 JSON                   12          107          106            0            1
 Makefile                1            6            5            0            1
 Python                 82         3665         3128          137          400
 Shell                   1           63           26           18           19
 Plain Text              3         3723            0         2413         1310
 TOML                   19          557          512            6           39
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       3            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               54         4898            0         3733         1165
 |- BASH                 8          105          102            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                21          729          616            0          113
 |- TOML                 2           75           63            0           12
 (Total)                           5940          902         3733         1305
-------------------------------------------------------------------------------
 Rust                  365       122573       109641         2418        10514
 |- Markdown           165         1843           25         1669          149
 (Total)                         124416       109666         4087        10663
===============================================================================
 Total                 546       135716       113512         8737        13467
===============================================================================
  

[coderabbitai]

Actionable comments posted: 1

🔭 Outside diff range comments (1)

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (1)

83-106: 🛠️ Refactor suggestion

Trait signature now forces JSON string to be passed around – consider accepting parsed config instead

Adding config: &str to every VisionModelLoader helper ( is_gptx, supports_paged_attention, supports_prefix_cacher, prefixer ) keeps the public API uniform, but it also means:

1. Every implementation ends up calling serde_json::from_str(config)? each time the helper is invoked.
2. Call-sites that already have a parsed config are forced to re-serialise or keep the raw bytes around.

For frequently-called helpers (e.g. inside an inner decoding loop) this will allocate and parse JSON many times and can become a non-trivial hot-spot.

-pub trait VisionModelLoader { 
-    fn is_gptx(&self, config: &str) -> bool;
-    … 
-}
+pub trait VisionModelLoader<Cfg: DeserializeOwned> {
+    fn is_gptx(&self, cfg: &Cfg) -> bool;
+    …
+}

The generic parameter keeps the public shape but lets each caller re-use an already-parsed config, saving allocations and CPU cycles.
If changing the trait is out of scope, at minimum document the performance cost and ensure the helpers are never called from tight loops.

🧹 Nitpick comments (7)

mistralrs-core/src/pipeline/loaders/normal_loaders.rs (4)

255-257: Scope and Derive Config Struct
AutoNormalLoaderConfig is correctly private. For improved debugging when deserializing, consider deriving Debug (and possibly Clone) on this struct.

---

259-288: Improve Error Context in get_loader
The length check is correct, but the error message could include the unexpected payload for easier troubleshooting.

Example diff:

-        if auto_cfg.architectures.len() != 1 {
-            anyhow::bail!("Expected to have one name for `architectures` config field.")
-        }
+        match auto_cfg.architectures.as_slice() {
+            [name] => { /* proceed */ }
+            other => anyhow::bail!(
+                "Expected exactly one architecture, found {}: {:?}", other.len(), other
+            ),
+        }

---

332-345: Delegate IsqModelLoader Implementation
Delegation through Self::get_loader(config)?… is accurate. This pattern is repeated below; consider extracting a small helper or macro to reduce boilerplate.

---

347-385: Delegate DeviceMappedModelLoader Implementation
Forwarding all trait methods to the selected loader is correct. For maintainability, you might factor out the repeated Self::get_loader(config)?… blocks into a private method or macro.

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (3)

167-189: from_causal_lm_name mapping brittle & missing several real-world names

1. Many HF classes use the pattern *ForCausalLM and *ForConditionalGeneration.
   Some here end with neither ("MiniCPMO").
2. Matching the full string means every new upstream class name requires a code change.

Consider normalising the incoming string and matching by suffix:

match name.trim_end_matches("ForCausalLM")
          .trim_end_matches("ForConditionalGeneration") {
    "Phi3V"   => Ok(Self::Phi3V),
    "MiniCPMO" => Ok(Self::MiniCpmO),
    …
    _ => bail!(…),
}

That reduces maintenance friction and avoids user-visible “unsupported model class” errors for trivially renamed upstream classes.

---

242-253: Log message allocation

once_log_info(format!(…)) allocates a String every time. If the macro accepts &'static str + args, prefer once_log_info!("Automatic loader type determined to be {}", tp); to avoid an extra heap allocation.

---

273-332: Delegation boilerplate – consider a macro

The impl VisionModelLoader for AutoVisionLoader repeats the same pattern for every method: Self::get_loader(config)?.....
A small macro or a helper with_loader!(config, |ldr| ldr.method(...)) would cut ~70 lines and remove the risk of forgetting to forward a new method in the future.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9a228b8 and 18731f8.

📒 Files selected for processing (28)

• mistralrs-core/src/model_selected.rs (1 hunks)
• mistralrs-core/src/pipeline/loaders/mod.rs (1 hunks)
• mistralrs-core/src/pipeline/loaders/normal_loaders.rs (4 hunks)
• mistralrs-core/src/pipeline/loaders/vision_loaders.rs (31 hunks)
• mistralrs-core/src/pipeline/mod.rs (1 hunks)
• mistralrs-core/src/pipeline/normal.rs (2 hunks)
• mistralrs-core/src/pipeline/vision.rs (5 hunks)
• mistralrs-core/src/toml_selector.rs (1 hunks)
• mistralrs-pyo3/src/lib.rs (1 hunks)
• mistralrs-pyo3/src/which.rs (2 hunks)
• mistralrs/examples/gemma3/main.rs (1 hunks)
• mistralrs/examples/idefics2/main.rs (1 hunks)
• mistralrs/examples/idefics3/main.rs (1 hunks)
• mistralrs/examples/llama4/main.rs (1 hunks)
• mistralrs/examples/llama_vision/main.rs (1 hunks)
• mistralrs/examples/llama_vision_multiturn/main.rs (1 hunks)
• mistralrs/examples/llava/main.rs (1 hunks)
• mistralrs/examples/llava_next/main.rs (1 hunks)
• mistralrs/examples/minicpmo_2_6/main.rs (1 hunks)
• mistralrs/examples/mistral3/main.rs (1 hunks)
• mistralrs/examples/phi3v/main.rs (1 hunks)
• mistralrs/examples/phi4mm/main.rs (1 hunks)
• mistralrs/examples/qwen2_5vl/main.rs (2 hunks)
• mistralrs/examples/qwen2vl/main.rs (1 hunks)
• mistralrs/examples/smolvlm/main.rs (1 hunks)
• mistralrs/examples/uqff_vision/main.rs (1 hunks)
• mistralrs/examples/vision_auto_device_map/main.rs (2 hunks)
• mistralrs/src/vision_model.rs (5 hunks)

⏰ Context from checks skipped due to timeout of 90000ms (8)

• GitHub Check: Docs
• GitHub Check: Clippy
• GitHub Check: Check (macOS-latest, stable)
• GitHub Check: Test Suite (ubuntu-latest, stable)
• GitHub Check: Test Suite (macOS-latest, stable)
• GitHub Check: Test Suite (windows-latest, stable)
• GitHub Check: Check (ubuntu-latest, stable)
• GitHub Check: Check (windows-latest, stable)

🔇 Additional comments (49)

mistralrs/examples/smolvlm/main.rs (2)

2-2: Clean import update

Removed the unnecessary VisionLoaderType import since it's no longer needed with the new automatic vision loader type detection.

---

8-8: API simplification looks good

The VisionModelBuilder::new() now only requires the model ID without explicitly specifying the loader type. This simplification aligns with the PR objective of adding automatic vision loader type detection.

mistralrs/examples/uqff_vision/main.rs (2)

2-2: Import cleanup

Removed the unused VisionLoaderType import, which is consistent with the API changes.

---

6-9: Builder API simplification

The UqffVisionModelBuilder::new() now just takes the model ID and files without requiring the explicit vision loader type. This is consistent with the PR objective of making loader types optional and automatically detected.

mistralrs/examples/minicpmo_2_6/main.rs (2)

2-2: Import simplification

Removed the now unused VisionLoaderType import, which is no longer needed with automatic loader type detection.

---

6-6: Simpler builder API

The VisionModelBuilder::new() call has been simplified to only require the model ID. This is consistent with the PR objective of automatic vision loader type detection.

mistralrs/examples/llava/main.rs (2)

2-2: Cleaned up imports

Removed the unnecessary VisionLoaderType import, aligning with the new automatic vision loader type functionality.

---

6-6: Simplified builder API

The VisionModelBuilder::new() now only requires the model ID parameter, eliminating the need for explicit vision loader type specification. This change is consistent with the PR objective of automatic loader type detection.

mistralrs/examples/gemma3/main.rs (1)

2-2: Great API simplification!

Removing the explicit VisionLoaderType import and parameter makes the API more user-friendly. The code is now cleaner and requires less domain knowledge to use, while maintaining the same functionality through automatic loader type detection.

Also applies to: 6-6

mistralrs/examples/vision_auto_device_map/main.rs (1)

3-4: Code simplification improves example clarity

Removing the explicit VisionLoaderType import and parameter from the builder pattern creates a more streamlined API. This change is particularly beneficial in this example focusing on auto device mapping, as it reduces cognitive load for users.

Also applies to: 16-16

mistralrs/examples/qwen2vl/main.rs (1)

2-2: Simplified model initialization

The removal of VisionLoaderType from imports and constructor arguments makes the example more concise and easier to understand. This change effectively demonstrates how the automatic loader type detection works with the Qwen2VL model.

Also applies to: 8-8

mistralrs/examples/llama_vision/main.rs (1)

2-2: Consistent API improvement

This change follows the same pattern as other examples, removing the need to explicitly specify the vision loader type. The consistency across all example files demonstrates a thorough approach to this API improvement. Now users can focus on their application logic rather than selecting the correct loader type.

Also applies to: 9-9

mistralrs/examples/llava_next/main.rs (1)

2-2: API simplification looks good

The removal of VisionLoaderType from imports and simplification of the VisionModelBuilder::new() call improves the API by reducing unnecessary parameters. The automatic vision loader type detection works correctly with the LLaVA model.

Also applies to: 6-6

mistralrs/examples/llama_vision_multiturn/main.rs (1)

2-2: Simplified API improves developer experience

The removal of VisionLoaderType from imports and simplification of the VisionModelBuilder::new() call makes the multi-turn vision example clearer and more concise. The automatic loader detection correctly works with the Llama 3.2 Vision model.

Also applies to: 8-8

mistralrs/examples/mistral3/main.rs (1)

2-2: API simplification consistent with Mistral3 model

The removal of explicit VisionLoaderType::Mistral3 argument and corresponding import simplifies the example while maintaining the same functionality. The automatic loader type detection works correctly with the Mistral model based on its identifier.

Also applies to: 6-6

mistralrs/examples/idefics3/main.rs (1)

2-2: Consistent API improvement for Idefics3 model

The removal of the explicit VisionLoaderType::Idefics3 parameter and its import simplifies the example while preserving functionality. The automatic loader type detection works correctly with the Idefics3 model based on its identifier.

Also applies to: 8-8

mistralrs/examples/llama4/main.rs (2)

2-2: Simplified import by removing unnecessary VisionLoaderType

The import statement has been streamlined by removing the VisionLoaderType import, which is no longer needed due to the automatic detection capability added in this PR.

---

6-10: API usage simplified with automatic vision loader detection

The model builder initialization has been simplified by removing the explicit VisionLoaderType::Llama4 parameter. The code now leverages the new automatic vision loader type detection feature, making it more concise while maintaining the same functionality.

mistralrs/examples/idefics2/main.rs (2)

2-2: Simplified import by removing unnecessary VisionLoaderType

The import statement has been cleaned up by removing the unused VisionLoaderType import, which is now handled automatically by the library.

---

6-10: API usage simplified with automatic vision loader detection

The model builder initialization has been simplified by removing the explicit VisionLoaderType::Idefics2 parameter. This change leverages the new automatic loader type detection feature, making the code more concise while maintaining the same functionality.

mistralrs/examples/phi3v/main.rs (2)

2-2: Simplified import by removing unnecessary VisionLoaderType

The import statement has been streamlined by removing the unused VisionLoaderType import, consistent with the changes in other example files.

---

6-10: API usage simplified with automatic vision loader detection

The model builder initialization no longer requires an explicit VisionLoaderType::Phi3V parameter, reducing boilerplate while maintaining functionality through automatic detection.

mistralrs/examples/phi4mm/main.rs (2)

2-2: Simplified import by removing unnecessary VisionLoaderType

The import statement has been cleaned up by removing the VisionLoaderType import, which is now handled automatically, maintaining consistency with the other example files.

---

6-10: API usage simplified with automatic vision loader detection

The model builder initialization has been simplified by removing the explicit VisionLoaderType::Phi4MM parameter. This change leverages the new automatic loader type detection capability, making the API more user-friendly while preserving functionality.

mistralrs-core/src/toml_selector.rs (1)

397-397: Appropriate change to make vision loader type optional

Changing the arch field in the VisionPlain variant to Option<VisionLoaderType> aligns with the pattern used in other model variants where architecture types are optional. This enables automatic detection of the vision loader type from model configuration.

mistralrs/examples/qwen2_5vl/main.rs (2)

3-3: Import cleaned up after vision loader type made optional

Removed the unnecessary VisionLoaderType import since it's no longer needed with the automatic loader type detection.

---

30-30: Simplified API usage with automatic loader detection

The model builder initialization is now more concise, requiring only the model ID and leveraging the automatic vision loader type detection capability. This simplification makes the API more user-friendly.

mistralrs-core/src/model_selected.rs (1)

475-475: Consistent change to make vision arch parameter optional

Making the arch field optional in ModelSelected::VisionPlain mirrors the change in TomlModelSelected::VisionPlain, ensuring consistency throughout the codebase for optional architecture specification. This change enables more flexible vision model initialization patterns.

mistralrs-core/src/pipeline/mod.rs (1)

35-35: Updated exports with specialized loader types

The public re-exports now include AutoNormalLoader and AutoVisionLoader instead of the previous generic AutoLoader. This change supports the new architecture where loader types are more specialized and vision loader types can be automatically detected.

mistralrs-pyo3/src/lib.rs (1)

411-411: Updated build parameter to handle optional architecture

The code now correctly uses .map(Into::into) instead of direct conversion, preserving the Option type. This matches the changes to make vision loader type optional, allowing automatic detection from model configuration.

mistralrs-core/src/pipeline/loaders/mod.rs (2)

20-20: Renamed AutoLoader to AutoNormalLoader

Good naming change that clearly distinguishes between normal and vision loaders, providing better API clarity.

---

28-30: Added AutoVisionLoader to re-exports

Properly adds the new AutoVisionLoader to the exports, enabling the automatic vision loader selection functionality. This is consistent with the architecture changes in the rest of the PR.

mistralrs-pyo3/src/which.rs (2)

400-401: Made arch parameter optional for Plain variant

Constructor now takes an optional architecture parameter with a default of None, enabling automatic architecture selection.

---

414-415: Made arch parameter optional for VisionPlain variant

The VisionPlain variant now accepts an optional VisionArchitecture, allowing the code to automatically select the appropriate loader based on model configuration when not specified.

mistralrs-core/src/pipeline/normal.rs (1)

224-224: Updated to use AutoNormalLoader

Consistent rename from AutoLoader to AutoNormalLoader, maintaining compatibility with the changes in the loaders module.

mistralrs-core/src/pipeline/vision.rs (5)

4-8: Good addition of AutoVisionLoader in imports

The addition of AutoVisionLoader to the imports aligns with the new automatic vision loader type detection feature.

---

156-172: Well-implemented optional loader type support

The build method now accepts an Option<VisionLoaderType> instead of requiring an explicit type, with a proper fallback to AutoVisionLoader when None is provided. This enables automatic loader selection based on model configuration.

This change improves the API by making the loader type specification optional, which simplifies client code when the specific loader type doesn't need to be explicitly specified.

---

254-256: Good adaptation of supports_paged_attention method call

The method call now correctly passes the configuration string to supports_paged_attention, allowing for configuration-dependent behavior.

---

693-694: Correctly updated supports_prefix_cacher method call

The no_prefix_cache field is now correctly determined by passing the configuration to the supports_prefix_cacher method.

---

702-703: Updated prefixer method call with config parameter

The prefixer method now correctly receives the configuration string, ensuring the proper vision prompt prefixer is selected based on the model configuration.

mistralrs/src/vision_model.rs (4)

32-32: Good change to make loader_type optional

The loader_type field is now correctly defined as Option<VisionLoaderType>, enabling support for automatic vision loader selection.

---

50-77: Well-implemented constructor simplification

The new method has been updated to remove the now-optional loader_type parameter, initializing it to None by default to leverage automatic loader detection.

This change simplifies the API and makes it more user-friendly, as users don't need to specify a loader type when creating a new vision model.

---

121-126: Great addition of with_loader_type builder method

The new builder method provides a clean way to explicitly set the loader type when needed, while keeping it optional by default. The method is well-documented with clear comments explaining its purpose.

---

312-316: Correctly updated UqffVisionModelBuilder constructor

The UqffVisionModelBuilder::new method has been updated to remove the loader_type parameter, aligning with the changes in VisionModelBuilder::new.

mistralrs-core/src/pipeline/loaders/normal_loaders.rs (2)

252-253: Rename Clear and Aligned with Vision Loader
The new AutoNormalLoader struct name clearly distinguishes normal model loading logic and aligns with the introduced AutoVisionLoader, improving API consistency.

---

291-330:

✅ Verification successful

Delegate NormalModelLoader to Concrete Loaders
The impl NormalModelLoader for AutoNormalLoader correctly forwards calls to the dynamically chosen loader. Please verify that any future loader signature changes propagate here.

You can run this to confirm consistency:

---

🏁 Script executed:

rg "get_loader" -A2 src/pipeline/loaders/normal_loaders.rs

Length of output: 137

---

I wasn’t able to locate the file at the given path. Let’s find where normal_loaders.rs actually lives and confirm the get_loader calls:

---

🏁 Script executed:

#!/bin/bash
# Locate the file
file=$(fd normal_loaders.rs)
echo "File found at: $file"

# Show lines around get_loader usage
rg "get_loader" -A2 "$file"

Length of output: 2144

---

Confirm Delegation of All NormalModelLoader Methods
I’ve verified that AutoNormalLoader forwards every current NormalModelLoader method to the underlying loader via Self::get_loader(config)?. When adding new methods to the trait, be sure to propagate them here as well.

• File: mistralrs-core/src/pipeline/loaders/normal_loaders.rs
• Delegated methods include:

• load, load_xlora
• get_config_repr
• supports_paged_attention, is_gptx
• immediate_isq_predicates(_moqe)
• isq_layer_regexes(_moqe)
• non_mapped_size_in_bytes
• num_layers, layer_sizes_in_bytes
• mapped_max_act_size_elems, non_mapped_max_act_size_elems
• model_config

Re-run this to confirm coverage remains in sync:

rg "get_loader" -A2 mistralrs-core/src/pipeline/loaders/normal_loaders.rs

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (3)

213-231: Nice Display implementation

Straight-forward and exhaustive – thanks for adding it; it will make logs and error messages much clearer.

---

245-247: Hard-error on multi-architecture configs may reject valid HF checkpoints

HF occasionally ships configs with multiple entries in architectures. Silently selecting the first entry is usually what users expect.

-if auto_cfg.architectures.len() != 1 {
-    anyhow::bail!("Expected exactly one architecture in config");
-}
+if auto_cfg.architectures.is_empty() {
+    anyhow::bail!("`architectures` field empty in config");
+}
+let name = &auto_cfg.architectures[0];

This keeps the strictness for empty configs while being forgiving for multi-arch cases.

---

255-269: Possible typo: VLlama4Loader vs Llama4 variant

VisionLoaderType::Llama4 => Box::new(VLlama4Loader) builds the VLlama4Loader (note the extra V).
If this was meant to be consistent with the other loaders (VLlamaLoader, VLlama4Loader), fine; otherwise confirm the struct name is correct – a misspelling will cause a compile error only if both variants exist.

[coderabbitai]

Actionable comments posted: 2

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18731f8 and eb55d1e.

📒 Files selected for processing (20)

• README.md (7 hunks)
• docs/CHAT_TOK.md (2 hunks)
• docs/DEVICE_MAPPING.md (1 hunks)
• docs/GEMMA3.md (2 hunks)
• docs/IDEFICS2.md (2 hunks)
• docs/IDEFICS3.md (4 hunks)
• docs/ISQ.md (1 hunks)
• docs/LLAMA4.md (2 hunks)
• docs/LLaVA.md (2 hunks)
• docs/MINICPMO_2_6.md (4 hunks)
• docs/MISTRAL3.md (3 hunks)
• docs/PAGED_ATTENTION.md (1 hunks)
• docs/PHI3V.md (2 hunks)
• docs/PHI4MM.md (2 hunks)
• docs/QUANTS.md (1 hunks)
• docs/QWEN2VL.md (4 hunks)
• docs/TOOL_CALLING.md (1 hunks)
• docs/TOPOLOGY.md (2 hunks)
• docs/VLLAMA.md (4 hunks)
• docs/WEB_SEARCH.md (1 hunks)

✅ Files skipped from review due to trivial changes (12)

• docs/DEVICE_MAPPING.md
• docs/WEB_SEARCH.md
• docs/PAGED_ATTENTION.md
• docs/TOOL_CALLING.md
• docs/LLAMA4.md
• docs/QUANTS.md
• docs/TOPOLOGY.md
• docs/IDEFICS2.md
• docs/CHAT_TOK.md
• docs/MISTRAL3.md
• docs/VLLAMA.md
• docs/ISQ.md

🧰 Additional context used

🪛 markdownlint-cli2 (0.17.2)

README.md

67-67: Fenced code blocks should have a language specified
null

(MD040, fenced-code-language)

---

96-96: Fenced code blocks should have a language specified
null

(MD040, fenced-code-language)

---

102-102: Fenced code blocks should have a language specified
null

(MD040, fenced-code-language)

⏰ Context from checks skipped due to timeout of 90000ms (8)

• GitHub Check: Test Suite (ubuntu-latest, stable)
• GitHub Check: Test Suite (windows-latest, stable)
• GitHub Check: Test Suite (macOS-latest, stable)
• GitHub Check: Check (macOS-latest, stable)
• GitHub Check: Clippy
• GitHub Check: Check (windows-latest, stable)
• GitHub Check: Check (ubuntu-latest, stable)
• GitHub Check: Docs

🔇 Additional comments (32)

docs/PHI3V.md (2)

54-54: CLI Simplification Approved
The example correctly omits the explicit -a phi3v flag and relies on the new automatic loader detection via the vision-plain subcommand.

---

105-111: Rust Example Updated Correctly
Removing the VisionLoaderType import and calling VisionModelBuilder::new with only the model identifier aligns with the updated API.

docs/PHI4MM.md (2)

52-52: CLI Example Simplified
The cargo run command no longer requires the -a phi4mm flag and correctly uses auto-detection.

---

102-109: Rust Example Streamlined
The import removal of VisionLoaderType and the simplified VisionModelBuilder::new("microsoft/Phi-4-multimodal-instruct") usage reflect the optional loader type refactor.

docs/GEMMA3.md (2)

50-50: CLI Invocation Updated
Removing the -a gemma3 argument from the cargo run example makes the usage consistent with automatic loader selection.

---

106-112: Rust Builder Call Simplification
Dropping the explicit VisionLoaderType::Gemma3 import and passing only the model ID simplifies the builder API as intended.

docs/LLaVA.md (3)

58-58: First CLI Example Simplified
The removal of the explicit loader flag from the first cargo run line cleanly demonstrates the auto-detection behavior.

---

60-60: Alternate CLI Example Simplified
The second example correctly omits the explicit loader type, relying on the vision-plain subcommand.

---

110-110: Rust Import Cleaned Up
Removing the VisionLoaderType import matches the new builder signature without altering functionality.

README.md (10)

49-49: Example CLI for Llama 4 Updated
The -a flag has been removed in favor of the new auto-loader mechanism.

---

62-62: Gemma 3 Quick Run Simplified
Dropping the -a gemma flag is consistent with the automatic detection update.

---

68-68: Mistral 3 Example CLI Streamlined
The example correctly uses vision-plain without an explicit loader argument.

---

91-91: Llama 3.2 Vision Command Simplified
The removal of the -a flag aligns with the refactored API.

---

97-97: Phi 4 Multimodal Quick Run
Omitting the -a phi4mm flag is intentional and correct under the new design.

---

103-103: Phi 4 Mini Example Updated
No longer specifying the loader accelerates setup and leverages the default.

---

360-360: Token Source CLI Example
The order of flags is clear and the -a flag omission is expected here as well.

---

372-372: Local Model Path Example
Using plain without -a properly relies on the auto-selected loader for local models.

---

510-510: Interactive Plain Model Run
The simplified -i plain -m invocation matches the updated subcommand behavior.

---

516-516: Interactive Vision Model Run
This example cleanly demonstrates vision-plain without an explicit loader type.

docs/MINICPMO_2_6.md (5)

18-23: Approve ToC enhancements
The newly nested Table of Contents is well-structured and aligns with the layout used in other model docs, improving discoverability of each section.

---

35-35: Removed explicit loader flag in interactive CLI
Dropping the -a minicpmo argument correctly defers to auto-detection, while retaining the --isq Q4K quantization flag.

---

97-97: Removed explicit loader flag in server CLI
The HTTP server example now omits -a minicpmo in favor of automatic loader resolution; the --isq Q4K option remains in place.

---

146-146: Cleaned up Rust imports
The VisionLoaderType import has been removed, leaving only the necessary types for auto loader mode.

---

153-153: Updated Rust builder signature
VisionModelBuilder::new now takes only MODEL_ID, relying on the new auto-loader. Ensure no downstream code still passes the old enum argument.

docs/QWEN2VL.md (3)

17-23: Approve ToC enhancements
The added nested Table of Contents organizes the documentation consistently with other vision model pages.

---

144-144: Cleaned up Rust imports
Removed VisionLoaderType from the import list, reflecting the shift to auto loader resolution.

---

151-151: Updated Rust builder signature
VisionModelBuilder::new now only requires the MODEL_ID string, leveraging auto detection of the loader type.

docs/IDEFICS3.md (5)

17-24: Approve ToC enhancements
The enhanced nested Table of Contents provides clear navigation for all new sections, aligning with other model docs.

---

40-40: Removed explicit loader flag in interactive CLI
The -a idefics3 argument has been dropped to enable automatic loader detection; the --isq Q4K flag remains.

---

112-112: Removed explicit loader flag in server CLI
Omitting the -a idefics3 argument aligns the HTTP server example with the auto-loader design while retaining quantization settings.

---

160-160: Cleaned up Rust imports
Eliminated the VisionLoaderType import, matching the updated builder API.

---

166-166: Updated Rust builder signature
Transitioned VisionModelBuilder::new to accept only MODEL_ID, deferring loader selection to the new auto logic.