Unified/automatic flash attention enabler

[EricLBuehler]

Now we are enabling flash attention via the feature flags ( --features flash-attn and --features flash-attn-v3). Documentation stays the same.

@Slowki does this cause any major conflicts with code on your end?

Summary by CodeRabbit

• Refactor
  • Removed the flash attention configuration option from all model, loader, and builder interfaces.
  • Simplified model and vision loader APIs by eliminating the use_flash_attn flag and related logic.
  • Updated configuration structures to no longer include flash attention toggles.
  • Streamlined model initialization and configuration loading for text, vision, and diffusion models.
  • Adjusted logging and removed flash attention-related messages in application entry points.

[coderabbitai]

Walkthrough

This change removes the use_flash_attn boolean flag and all related configuration, propagation, and logic from the codebase. The field is deleted from model, loader, and pipeline configs, as well as from macro, builder, and loader interfaces. All method signatures, struct definitions, and trait implementations are updated to eliminate this parameter.

Changes

File(s)	Change Summary
mistralrs-bench/src/main.rs, mistralrs-server/src/main.rs, mistralrs-pyo3/src/lib.rs	Removed all references to use_flash_attn in main functions and loader configurations; deleted related logging and imports.
mistralrs-core/src/attention.rs	Removed use_flash_attn from SdpaParams; usage now centralized via crate::using_flash_attn().
mistralrs-core/src/model_loader.rs, mistralrs-core/src/pipeline/diffusion.rs, mistralrs-core/src/pipeline/normal.rs, mistralrs-core/src/pipeline/vision.rs, mistralrs-core/src/toml_selector.rs	Removed use_flash_attn from loader, builder, and config structs; updated builder and loader method signatures and instantiations accordingly.
mistralrs-core/src/pipeline/loaders/normal_loaders.rs, mistralrs-core/src/pipeline/loaders/vision_loaders.rs, mistralrs-core/src/pipeline/loaders/diffusion_loaders.rs	Removed use_flash_attn from all loader trait method signatures and implementations; simplified config deserialization.
mistralrs-core/src/pipeline/macros.rs	Removed $use_flash_attn parameter from all model loader macros and macro invocations.
mistralrs-core/src/pipeline/mod.rs	Removed re-export of DiffusionSpecificConfig.
mistralrs-core/src/lib.rs	Reordered re-exported pipeline items.
mistralrs-core/src/models/*, mistralrs-core/src/vision_models/*, mistralrs-core/src/xlora_models/*	Removed use_flash_attn field from all model config structs and related serde default functions; eliminated all propagation into attention parameters and struct initializations.
mistralrs/src/text_model.rs, mistralrs/src/vision_model.rs, mistralrs/src/diffusion_model.rs, mistralrs/src/anymoe.rs, mistralrs/src/lora_model.rs, mistralrs/src/speculative.rs, mistralrs/src/xlora_model.rs	Removed use_flash_attn from all builder structs, constructors, and build methods; updated config construction accordingly.
mistralrs-core/src/models/quantized_*.rs, mistralrs-core/src/vision_models/*/text.rs, mistralrs-core/src/vision_models/*/vision.rs, mistralrs-core/src/vision_models/siglip.rs, mistralrs-core/src/xlora_models/quantized_*.rs	Removed explicit setting of use_flash_attn in SdpaParams struct initializations.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant LoaderBuilder
    participant ModelConfig
    participant Attention

    User->>LoaderBuilder: Create builder (no use_flash_attn)
    LoaderBuilder->>ModelConfig: Build config (no use_flash_attn)
    ModelConfig->>Attention: Construct attention (no use_flash_attn)
    Attention->>Attention: Use global flash attention logic if needed

Loading

Poem

A bunny hopped and cleared the way,
No more toggles for flash attn today!
Models, configs, and loaders, all in sync,
Simpler code—faster than you think.
With every field that's swept aside,
The codebase gleams with bunny pride!
🐇✨

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 99a035c and a5af951.

📒 Files selected for processing (3)

• mistralrs-core/src/pipeline/diffusion.rs (3 hunks)
• mistralrs-core/src/pipeline/normal.rs (2 hunks)
• mistralrs-core/src/pipeline/vision.rs (2 hunks)

🚧 Files skipped from review as they are similar to previous changes (3)

• mistralrs-core/src/pipeline/vision.rs
• mistralrs-core/src/pipeline/normal.rs
• mistralrs-core/src/pipeline/diffusion.rs

⏰ Context from checks skipped due to timeout of 90000ms (8)

• GitHub Check: Check (macOS-latest, stable)
• GitHub Check: Clippy
• GitHub Check: Docs
• GitHub Check: Check (ubuntu-latest, stable)
• GitHub Check: Check (windows-latest, stable)
• GitHub Check: Test Suite (macOS-latest, stable)
• GitHub Check: Test Suite (windows-latest, stable)
• GitHub Check: Test Suite (ubuntu-latest, stable)

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                3           62           53            0            9
 Dockerfile              1           41           22           10            9
 JSON                   11          104          103            0            1
 Makefile                1            6            5            0            1
 Python                 82         3665         3128          137          400
 Shell                   1           63           26           18           19
 Plain Text              3         3723            0         2413         1310
 TOML                   19          554          509            6           39
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       3            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               54         4898            0         3733         1165
 |- BASH                 8          105          102            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                21          729          616            0          113
 |- TOML                 2           75           63            0           12
 (Total)                           5940          902         3733         1305
-------------------------------------------------------------------------------
 Rust                  365       122557       109710         2394        10453
 |- Markdown           165         1829           25         1660          144
 (Total)                         124386       109735         4054        10597
===============================================================================
 Total                 545       135694       113575         8713        13406
===============================================================================
  

[coderabbitai]

Actionable comments posted: 0

🔭 Outside diff range comments (2)

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (1)

82-91: 💡 Verification agent

🧩 Analysis chain

Verify flash attention feature flag setup for backwards compatibility.

The VisionModelLoader trait has removed the use_flash_attn parameter, which is part of the shift to using feature flags for flash attention control. Make sure that the feature flag mechanism is properly set up elsewhere in the codebase to maintain backward compatibility.

---

🏁 Script executed:

#!/bin/bash
# Check for feature flag declarations and usage of flash attention features

# Look for feature flag definitions in Cargo.toml files
echo "Checking for flash-attn feature flags in Cargo.toml files:"
fd -t f "Cargo.toml" | xargs grep -l "flash-attn"

# Look for runtime feature detection for flash attention
echo -e "\nChecking for flash attention runtime detection:"
rg "using_flash_attn|crate::using_flash_attn"

# Look for any remaining use_flash_attn references that might need updating
echo -e "\nChecking for any remaining use_flash_attn references:"
rg "use_flash_attn" --max-count=20

Length of output: 2808

---

Ensure examples and docs are updated for feature-flag-based flash attention

We’ve moved to controlling flash attention purely via the flash-attn Cargo feature (with the using_flash_attn() runtime check) and removed the use_flash_attn parameter from VisionModelLoader::load(). Any code that still passes use_flash_attn: … will no longer compile.

Please update the following:

• Enable the flash-attn feature in your Cargo.toml (already declared).
• Remove or replace all use_flash_attn: false/true in example code under
  • mistralrs/examples/lower_level/**/*
• Update references in docs/UQFF.md (and any other markdown) to point users to the new feature flag.
• (Optional) Add a short migration note in CHANGELOG or RELEASE_NOTES explaining how to opt into flash attention.

mistralrs-core/src/pipeline/loaders/normal_loaders.rs (1)

210-215: ⚠️ Potential issue

Incorrect enum value returned for "qwen3" – will mis-route loaders at runtime

FromStr::from_str maps the string "qwen3" to Self::DeepSeekV3 instead of the expected Self::Qwen3.
This breaks automatic loader resolution (both CLI & API), silently falling back to the wrong model family and very likely panicking later when the DeepSeekV3 loader cannot deserialize a Qwen3 config.

-            "qwen3" => Ok(Self::DeepSeekV3),
+            "qwen3" => Ok(Self::Qwen3),

Please add a unit-test covering every branch of this match to avoid future regressions (e.g. table-driven test over the Display output).

🧹 Nitpick comments (3)

mistralrs-core/src/pipeline/loaders/normal_loaders.rs (3)

291-301: Repeated serde_json::from_str incurs avoidable cost & code duplication

Each call to AutoLoader::load delegates to Self::get_loader(config)? which re-parses config once again inside the concrete loader.
For long-running services this extra parse is negligible, but in tight loops (e.g. batch benchmarking) it shows up in profiles and bloats binary size.

A tiny refactor keeps behaviour identical while eliminating the extra parse:

-        Self::get_loader(config)?.load(config, vb, normal_loading_metadata, attention_mechanism)
+        let loader = Self::get_loader(config)?;
+        loader.load(config, vb, normal_loading_metadata, attention_mechanism)

Now config is parsed exactly once in the concrete loader.
Consider applying the same pattern to load_xlora, get_config_repr, etc.

---

480-485: Parsing the same config over and over – extract & reuse

Inside several *_size_in_bytes helpers you perform:

let cfg: crate::models::<model>::Config = serde_json::from_str(config)?;

This happens multiple times per loader (sometimes four or five).
Because these helpers are often called back-to-back during device-mapping, the config is re-parsed repeatedly.

If you keep the current trait shape, you can at least move the parsed struct into
NormalLoadingMetadata or cache it inside a small OnceCell in each loader struct.

The change is not urgent but will:

• Shave milliseconds off model start-up.
• Reduce temporary allocations.
• Shorten the source file considerably.

Let me know if you’d like a PR snippet demonstrating a lightweight cache.

Also applies to: 667-673, 858-864

---

321-327: supports_paged_attention defaulting to true may hide future omissions

For new architectures added later, forgetting to override this method will silently report support, potentially causing hard-to-trace runtime errors.

Consider changing the default to anyhow::bail!("not implemented") or Ok(false) and explicitly enabling it per loader.
This follows the “fail-fast” principle and surfaces integration gaps during development rather than in production.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7a794da and 99a035c.

📒 Files selected for processing (69)

• mistralrs-bench/src/main.rs (1 hunks)
• mistralrs-core/src/attention.rs (2 hunks)
• mistralrs-core/src/lib.rs (1 hunks)
• mistralrs-core/src/model_loader.rs (2 hunks)
• mistralrs-core/src/models/deepseek2.rs (0 hunks)
• mistralrs-core/src/models/deepseek3.rs (0 hunks)
• mistralrs-core/src/models/gemma.rs (0 hunks)
• mistralrs-core/src/models/gemma2.rs (1 hunks)
• mistralrs-core/src/models/llama.rs (0 hunks)
• mistralrs-core/src/models/mistral.rs (0 hunks)
• mistralrs-core/src/models/mixtral.rs (0 hunks)
• mistralrs-core/src/models/phi2.rs (0 hunks)
• mistralrs-core/src/models/phi3.rs (0 hunks)
• mistralrs-core/src/models/phi3_5_moe.rs (0 hunks)
• mistralrs-core/src/models/quantized_llama.rs (0 hunks)
• mistralrs-core/src/models/quantized_phi2.rs (0 hunks)
• mistralrs-core/src/models/quantized_phi3.rs (0 hunks)
• mistralrs-core/src/models/quantized_qwen2.rs (0 hunks)
• mistralrs-core/src/models/quantized_starcoder2.rs (0 hunks)
• mistralrs-core/src/models/qwen2.rs (0 hunks)
• mistralrs-core/src/models/qwen3.rs (0 hunks)
• mistralrs-core/src/models/qwen3_moe.rs (0 hunks)
• mistralrs-core/src/models/starcoder2.rs (0 hunks)
• mistralrs-core/src/pipeline/diffusion.rs (1 hunks)
• mistralrs-core/src/pipeline/loaders/diffusion_loaders.rs (0 hunks)
• mistralrs-core/src/pipeline/loaders/normal_loaders.rs (81 hunks)
• mistralrs-core/src/pipeline/loaders/vision_loaders.rs (24 hunks)
• mistralrs-core/src/pipeline/macros.rs (0 hunks)
• mistralrs-core/src/pipeline/mod.rs (1 hunks)
• mistralrs-core/src/pipeline/normal.rs (1 hunks)
• mistralrs-core/src/pipeline/vision.rs (1 hunks)
• mistralrs-core/src/toml_selector.rs (0 hunks)
• mistralrs-core/src/vision_models/gemma3/config.rs (0 hunks)
• mistralrs-core/src/vision_models/gemma3/text.rs (0 hunks)
• mistralrs-core/src/vision_models/idefics2/mod.rs (0 hunks)
• mistralrs-core/src/vision_models/llama4/config.rs (0 hunks)
• mistralrs-core/src/vision_models/llama4/text.rs (0 hunks)
• mistralrs-core/src/vision_models/llama4/vision.rs (0 hunks)
• mistralrs-core/src/vision_models/llava/config.rs (0 hunks)
• mistralrs-core/src/vision_models/llava/llava_llm/llama.rs (0 hunks)
• mistralrs-core/src/vision_models/llava/llava_llm/mistral.rs (0 hunks)
• mistralrs-core/src/vision_models/mllama/config.rs (0 hunks)
• mistralrs-core/src/vision_models/mllama/text.rs (0 hunks)
• mistralrs-core/src/vision_models/mllama/vision.rs (0 hunks)
• mistralrs-core/src/vision_models/phi3/mod.rs (0 hunks)
• mistralrs-core/src/vision_models/phi4/config.rs (1 hunks)
• mistralrs-core/src/vision_models/phi4/mod.rs (0 hunks)
• mistralrs-core/src/vision_models/qwen2_5_vl/text.rs (0 hunks)
• mistralrs-core/src/vision_models/qwen2vl/text.rs (0 hunks)
• mistralrs-core/src/vision_models/siglip.rs (0 hunks)
• mistralrs-core/src/xlora_models/gemma.rs (0 hunks)
• mistralrs-core/src/xlora_models/gemma2.rs (0 hunks)
• mistralrs-core/src/xlora_models/llama.rs (0 hunks)
• mistralrs-core/src/xlora_models/mistral.rs (0 hunks)
• mistralrs-core/src/xlora_models/mixtral.rs (0 hunks)
• mistralrs-core/src/xlora_models/phi2.rs (0 hunks)
• mistralrs-core/src/xlora_models/phi3.rs (0 hunks)
• mistralrs-core/src/xlora_models/quantized_llama.rs (0 hunks)
• mistralrs-core/src/xlora_models/quantized_phi3.rs (0 hunks)
• mistralrs-core/src/xlora_models/starcoder2.rs (0 hunks)
• mistralrs-pyo3/src/lib.rs (2 hunks)
• mistralrs-server/src/main.rs (1 hunks)
• mistralrs/src/anymoe.rs (0 hunks)
• mistralrs/src/diffusion_model.rs (1 hunks)
• mistralrs/src/lora_model.rs (0 hunks)
• mistralrs/src/speculative.rs (0 hunks)
• mistralrs/src/text_model.rs (0 hunks)
• mistralrs/src/vision_model.rs (0 hunks)
• mistralrs/src/xlora_model.rs (0 hunks)

💤 Files with no reviewable changes (54)

• mistralrs-core/src/models/quantized_starcoder2.rs
• mistralrs/src/speculative.rs
• mistralrs-core/src/vision_models/qwen2vl/text.rs
• mistralrs-core/src/xlora_models/phi3.rs
• mistralrs-core/src/vision_models/mllama/text.rs
• mistralrs-core/src/vision_models/phi4/mod.rs
• mistralrs/src/lora_model.rs
• mistralrs-core/src/xlora_models/quantized_phi3.rs
• mistralrs-core/src/xlora_models/mixtral.rs
• mistralrs-core/src/vision_models/llava/llava_llm/llama.rs
• mistralrs-core/src/models/mixtral.rs
• mistralrs-core/src/models/quantized_phi3.rs
• mistralrs-core/src/models/phi2.rs
• mistralrs/src/anymoe.rs
• mistralrs-core/src/xlora_models/phi2.rs
• mistralrs-core/src/xlora_models/quantized_llama.rs
• mistralrs-core/src/models/quantized_qwen2.rs
• mistralrs-core/src/vision_models/mllama/vision.rs
• mistralrs-core/src/vision_models/llama4/text.rs
• mistralrs-core/src/xlora_models/llama.rs
• mistralrs-core/src/models/quantized_phi2.rs
• mistralrs-core/src/xlora_models/gemma2.rs
• mistralrs-core/src/xlora_models/gemma.rs
• mistralrs-core/src/models/qwen3_moe.rs
• mistralrs-core/src/vision_models/llama4/vision.rs
• mistralrs-core/src/models/starcoder2.rs
• mistralrs-core/src/models/deepseek2.rs
• mistralrs-core/src/models/gemma.rs
• mistralrs/src/xlora_model.rs
• mistralrs-core/src/vision_models/llama4/config.rs
• mistralrs-core/src/models/qwen2.rs
• mistralrs-core/src/models/llama.rs
• mistralrs-core/src/xlora_models/mistral.rs
• mistralrs-core/src/vision_models/gemma3/config.rs
• mistralrs/src/vision_model.rs
• mistralrs-core/src/xlora_models/starcoder2.rs
• mistralrs-core/src/vision_models/mllama/config.rs
• mistralrs-core/src/vision_models/qwen2_5_vl/text.rs
• mistralrs-core/src/models/mistral.rs
• mistralrs/src/text_model.rs
• mistralrs-core/src/models/deepseek3.rs
• mistralrs-core/src/vision_models/siglip.rs
• mistralrs-core/src/models/qwen3.rs
• mistralrs-core/src/models/phi3_5_moe.rs
• mistralrs-core/src/vision_models/gemma3/text.rs
• mistralrs-core/src/models/phi3.rs
• mistralrs-core/src/toml_selector.rs
• mistralrs-core/src/vision_models/phi3/mod.rs
• mistralrs-core/src/models/quantized_llama.rs
• mistralrs-core/src/vision_models/llava/config.rs
• mistralrs-core/src/vision_models/idefics2/mod.rs
• mistralrs-core/src/pipeline/macros.rs
• mistralrs-core/src/pipeline/loaders/diffusion_loaders.rs
• mistralrs-core/src/vision_models/llava/llava_llm/mistral.rs

🧰 Additional context used

🧬 Code Graph Analysis (3)

mistralrs-core/src/pipeline/normal.rs (4)

mistralrs/src/model.rs (1)

• inner (285-287)

mistralrs-core/src/pipeline/loaders/normal_loaders.rs (17)

• get_config_repr (107-107)
• get_config_repr (321-323)
• get_config_repr (424-427)
• get_config_repr (611-614)
• get_config_repr (802-805)
• get_config_repr (986-990)
• get_config_repr (1179-1183)
• get_config_repr (1362-1366)
• get_config_repr (1533-1537)
• get_config_repr (1722-1726)
• get_config_repr (1914-1918)
• get_config_repr (2100-2104)
• get_config_repr (2293-2296)
• get_config_repr (2611-2614)
• get_config_repr (2930-2934)
• get_config_repr (3108-3112)
• config (69-69)

mistralrs-core/src/lib.rs (1)

• config (619-621)

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (1)

• config (77-77)

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (16)

mistralrs-core/src/pipeline/loaders/normal_loaders.rs (18)

• get_config_repr (107-107)
• get_config_repr (321-323)
• get_config_repr (424-427)
• get_config_repr (611-614)
• get_config_repr (802-805)
• get_config_repr (986-990)
• get_config_repr (1179-1183)
• get_config_repr (1362-1366)
• get_config_repr (1533-1537)
• get_config_repr (1722-1726)
• get_config_repr (1914-1918)
• get_config_repr (2100-2104)
• get_config_repr (2293-2296)
• get_config_repr (2611-2614)
• get_config_repr (2930-2934)
• get_config_repr (3108-3112)
• config (69-69)
• from_str (199-217)

mistralrs-core/src/models/llama.rs (2)

• config (659-661)
• new (338-355)

mistralrs-core/src/models/qwen2.rs (1)

• config (651-653)

mistralrs-core/src/vision_models/phi3/mod.rs (1)

• config (1224-1226)

mistralrs-core/src/models/gemma2.rs (1)

• config (696-698)

mistralrs-core/src/models/mixtral.rs (1)

• config (768-770)

mistralrs-core/src/models/phi3.rs (1)

• config (707-709)

mistralrs-core/src/models/phi3_5_moe.rs (1)

• config (882-884)

mistralrs-core/src/models/starcoder2.rs (1)

• config (691-693)

mistralrs-core/src/vision_models/idefics2/mod.rs (1)

• config (1296-1298)

mistralrs-core/src/vision_models/gemma3/text.rs (2)

• config (722-724)
• cfg (508-518)

mistralrs-core/src/vision_models/idefics3/mod.rs (1)

• config (333-335)

mistralrs-core/src/vision_models/mllama/mod.rs (2)

• config (184-186)
• new (83-113)

mistralrs-core/src/vision_models/llava/llava_inputs_processor.rs (1)

• serde_json (50-50)

mistralrs-core/src/vision_models/llava/llava_next_inputs_processor.rs (1)

• serde_json (53-53)

mistralrs-core/src/vision_models/mllama/text.rs (3)

• new (33-65)
• new (99-151)
• new (231-268)

mistralrs-core/src/pipeline/loaders/normal_loaders.rs (15)

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (29)

• get_config_repr (91-91)
• get_config_repr (279-282)
• get_config_repr (539-542)
• get_config_repr (864-867)
• get_config_repr (1109-1112)
• get_config_repr (1346-1349)
• get_config_repr (1726-1729)
• get_config_repr (2003-2006)
• get_config_repr (2282-2285)
• get_config_repr (2555-2558)
• get_config_repr (2871-2874)
• get_config_repr (3146-3149)
• get_config_repr (3460-3463)
• get_config_repr (3761-3764)
• config (77-77)
• from_str (164-181)
• model_config (481-496)
• model_config (806-822)
• model_config (1051-1067)
• model_config (1288-1304)
• model_config (1659-1675)
• model_config (1945-1960)
• model_config (2221-2237)
• model_config (2493-2509)
• model_config (2805-2820)
• model_config (3089-3104)
• model_config (3398-3418)
• model_config (3700-3716)
• model_config (4051-4067)

mistralrs-core/src/models/deepseek2.rs (1)

• config (1167-1169)

mistralrs-core/src/models/llama.rs (1)

• config (659-661)

mistralrs-core/src/models/mistral.rs (1)

• config (678-680)

mistralrs-core/src/models/deepseek3.rs (1)

• config (1220-1222)

mistralrs-core/src/models/qwen2.rs (4)

• config (651-653)
• new (61-132)
• new (251-292)
• new (337-460)

mistralrs-core/src/models/gemma.rs (1)

• config (637-639)

mistralrs-core/src/models/mixtral.rs (1)

• config (768-770)

mistralrs-core/src/models/phi2.rs (1)

• config (703-705)

mistralrs-core/src/models/phi3.rs (1)

• config (707-709)

mistralrs-core/src/models/phi3_5_moe.rs (1)

• config (882-884)

mistralrs-core/src/models/starcoder2.rs (1)

• config (691-693)

mistralrs-core/src/models/qwen3.rs (1)

• cfg (505-513)

mistralrs-core/src/models/qwen3_moe.rs (1)

• cfg (706-714)

mistralrs-core/src/pipeline/loaders/mod.rs (3)

• from_str (170-191)
• new (102-124)
• model_config (471-471)

⏰ Context from checks skipped due to timeout of 90000ms (8)

• GitHub Check: Clippy
• GitHub Check: Docs
• GitHub Check: Test Suite (windows-latest, stable)
• GitHub Check: Test Suite (macOS-latest, stable)
• GitHub Check: Test Suite (ubuntu-latest, stable)
• GitHub Check: Check (macOS-latest, stable)
• GitHub Check: Check (windows-latest, stable)
• GitHub Check: Check (ubuntu-latest, stable)

🔇 Additional comments (42)

mistralrs-bench/src/main.rs (2)

14-14: Import clean-up update.

The import has been simplified from use tracing::{info, warn}; to only import info since the warn import is no longer needed after removing flash attention configuration logging.

---

362-365: Simplified loader builder API.

The LoaderBuilder no longer requires the with_use_flash_attn call, aligning with the codebase's shift toward enabling flash attention through feature flags instead of runtime configuration.

mistralrs-core/src/pipeline/mod.rs (1)

26-26: Removed DiffusionSpecificConfig re-export.

The export list has been updated to only include DiffusionLoader and DiffusionLoaderBuilder, removing DiffusionSpecificConfig which contained the now-removed use_flash_attn flag. This aligns with the broader refactoring that eliminates flash attention configuration from model configs.

mistralrs-core/src/vision_models/phi4/config.rs (1)

6-6: Simplified imports after removing flash attention configuration.

The import list has been cleaned up by removing serde-related imports that were only needed for the now-removed use_flash_attn field, streamlining the dependencies.

mistralrs-server/src/main.rs (2)

40-40: Import clean-up update.

The import has been simplified from use tracing::{info, warn}; to only import info since the warn import is no longer needed after removing flash attention configuration logging.

---

292-297: Simplified loader builder API.

The LoaderBuilder no longer requires the with_use_flash_attn call, aligning with the codebase's shift toward enabling flash attention through feature flags instead of runtime configuration.

mistralrs-core/src/pipeline/vision.rs (1)

398-398: Removed flash attention configuration from model config logging.

The info logging no longer includes flash attention configuration information since the use_flash_attn field has been removed from the config representation. This streamlines the logging output to focus on the remaining configuration parameters.

mistralrs-core/src/attention.rs (2)

285-286: Documentation improvement correctly reflects new flash attention handling.

The updated documentation accurately reflects that flash attention now uses either V2 or V3 kernels on CUDA devices, aligning with the feature flag approach rather than per-parameter configuration.

---

301-301: Centralized flash attention detection properly implemented.

The code now uses crate::using_flash_attn() to determine flash attention availability instead of checking a per-instance parameter. This centralizes the flash attention decision logic, making it consistent across the codebase and ensuring feature flag control.

mistralrs-core/src/lib.rs (1)

89-92: Re-export list updated to align with removed config types.

The change in re-export order and removal of DiffusionSpecificConfig aligns with the broader refactoring that removes the use_flash_attn flag from pipeline and model loading infrastructure.

mistralrs-core/src/models/gemma2.rs (1)

27-28: Config struct's derive macro updated appropriately.

Changed from serde::Serialize to serde::Deserialize, which better reflects how this configuration is used in the codebase. Note that this means the config can now be deserialized but not automatically serialized.

mistralrs-core/src/pipeline/normal.rs (1)

463-463: Simplified logging reflects config structure changes.

The logging statement has been updated to reflect the removal of the use_flash_attn field from configuration structures. The call to get_config_repr now expects a configuration without the flash attention flag.

mistralrs-pyo3/src/lib.rs (2)

26-27: Import list cleaned up after removing flash attention boolean flag

The import list has been adjusted to align with the removal of the use_flash_attn boolean flag from the codebase.

---

416-417: Simplified DiffusionLoaderBuilder initialization

The DiffusionLoaderBuilder::new call has been simplified to only pass the model ID, removing the DiffusionSpecificConfig that previously contained the use_flash_attn boolean flag. This aligns with the PR objective of enabling flash attention through feature flags instead of a boolean parameter.

mistralrs-core/src/model_loader.rs (2)

14-15: Import list updated after removing flash attention flag

The import list has been adjusted to reflect the removal of types related to the use_flash_attn boolean flag.

---

542-543: Simplified DiffusionLoaderBuilder initialization

The DiffusionLoaderBuilder::new call has been simplified to only take the model ID parameter, removing the DiffusionSpecificConfig that previously contained the use_flash_attn boolean flag. This is consistent with the PR objective of migrating from a boolean flag to feature flags for flash attention.

mistralrs/src/diffusion_model.rs (1)

80-81: Simplified DiffusionLoaderBuilder initialization

The DiffusionLoaderBuilder::new call has been simplified to only pass the model ID, removing the DiffusionSpecificConfig that previously contained the use_flash_attn boolean flag. This is part of the PR objective to enable flash attention through feature flags instead of a boolean parameter.

I also notice that the DiffusionModelBuilder struct itself has been simplified, with the use_flash_attn field being removed, which is consistent with the broader refactoring effort.

mistralrs-core/src/pipeline/diffusion.rs (1)

53-58: Simplified DiffusionLoaderBuilder constructor

The DiffusionLoaderBuilder::new method has been simplified to only accept an optional model ID parameter, removing the previously required config parameter that contained the use_flash_attn boolean flag. This is consistent with the PR objective of enabling flash attention through feature flags rather than a boolean parameter.

I also notice that the DiffusionLoaderBuilder struct itself has been simplified, no longer containing a config field, which aligns with the broader refactoring effort to remove the use_flash_attn flag from the codebase.

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (24)

91-91: Simplified trait method signature by removing flash attention parameter.

The get_config_repr method signature has been modified to remove the use_flash_attn: bool parameter, which is part of the broader refactoring to control flash attention through feature flags instead of a boolean parameter.

---

267-270: Config parameter handling improved for Phi3VLoader load method.

The load method now deserializes the config directly without modifying a use_flash_attn field, which aligns with using feature flags for flash attention control instead of a method parameter.

---

279-282: Simplified Phi3VLoader get_config_repr implementation.

The implementation now directly deserializes and returns the config without handling a use_flash_attn parameter.

---

527-530: Config parameter handling improved for Idefics2Loader load method.

The load method now directly deserializes the config without modifying a use_flash_attn field.

---

539-542: Simplified Idefics2Loader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

852-855: Config parameter handling improved for LLaVANextLoader load method.

The load method now directly deserializes the config without modifying a use_flash_attn field.

---

864-867: Simplified LLaVANextLoader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

1097-1100: Config parameter handling improved for LLaVALoader load method.

The load method now directly deserializes the config without modifying a use_flash_attn field.

---

1109-1112: Simplified LLaVALoader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

1334-1337: Config parameter handling improved for VLlamaLoader load method.

The load method now directly deserializes the config without modifying a use_flash_attn field.

---

1346-1349: Simplified VLlamaLoader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

1726-1729: Simplified Qwen2VLLoader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

1991-1994: Config parameter handling improved for Idefics3Loader load method.

The load method now directly deserializes the config without modifying a use_flash_attn field.

---

2003-2006: Simplified Idefics3Loader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

2270-2273: Config parameter handling improved for MiniCpmOLoader load method.

The load method now directly deserializes the config without modifying a use_flash_attn field.

---

2282-2285: Simplified MiniCpmOLoader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

2543-2546: Config parameter handling improved for Phi4MMLoader load method.

The load method now directly deserializes the config without modifying a use_flash_attn field.

---

2555-2558: Simplified Phi4MMLoader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

2871-2874: Simplified Qwen2_5VLLoader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

3146-3149: Simplified Gemma3Loader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

3448-3451: Config parameter handling improved for Mistral3Loader load method.

The load method now directly deserializes the config without modifying a use_flash_attn field.

---

3460-3463: Simplified Mistral3Loader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

---

3749-3752: Config parameter handling improved for VLlama4Loader load method.

The load method now directly deserializes the config without modifying a use_flash_attn field.

---

3761-3764: Simplified VLlama4Loader get_config_repr implementation.

Implementation now directly deserializes and returns the config without manipulating a use_flash_attn parameter.

[Slowki]

Nothing major, I already have to do some rebasing anyway so it's no big deal!

[EricLBuehler]

Sounds great, wanted to make sure!