Skip to content

FIX #245 - Set required DirectML session options (memory_pattern + sequential execution)#246

Merged
Anush008 merged 2 commits intoAnush008:mainfrom
markgandolfo:pass-additional-flags-to-ort-for-directml
Apr 10, 2026
Merged

FIX #245 - Set required DirectML session options (memory_pattern + sequential execution)#246
Anush008 merged 2 commits intoAnush008:mainfrom
markgandolfo:pass-additional-flags-to-ort-for-directml

Conversation

@markgandolfo
Copy link
Copy Markdown
Contributor

proposed fix for #245

DirectML requires two ONNX Runtime session options that fastembed wasn't setting:

  • enable_mem_pattern must be false
  • execution_mode must be sequential (not parallel)

Without these, DirectML can OOM on operations that would otherwise fit in VRAM. This became visible after candle 0.10 changed GPU memory allocation patterns.

on the directml page

If creating the onnxruntime InferenceSession object directly, you must set the appropriate fields on the onnxruntime::SessionOptions struct. Specifically, execution_mode must be set to ExecutionMode::ORT_SEQUENTIAL, and enable_mem_pattern must be false.

Changes:

  • Added directml = ["ort/directml"] feature flag in Cargo.toml
  • Patched all 4 session builder sites (try_new and try_new_from_user_defined in both text_embedding/impl.rs and image_embedding/impl.rs) to detect DirectML at runtime and apply the required settings
  • Uses #[cfg(feature = "directml")] guards so it compiles on all platforms
  • No change for CPU, CUDA, or any other execution provider.

Copy link
Copy Markdown
Owner

@Anush008 Anush008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Thanks for taking the time to contribute @markgandolfo.

@Anush008 Anush008 merged commit 138602f into Anush008:main Apr 10, 2026
1 check passed
github-actions bot pushed a commit that referenced this pull request Apr 10, 2026
## [5.13.2](v5.13.1...v5.13.2) (2026-04-10)

### 🐛 Bug Fixes

* Set required DirectML session options (memory_pattern + sequential execution) ([#246](#246)) ([138602f](138602f))
@github-actions
Copy link
Copy Markdown

🎉 This PR is included in version 5.13.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DirectML: Missing required session options (memory_pattern, execution_mode) causes OOM on Windows

2 participants