Skip to content

feat: Inference engines abstraction for object detection model family with Transformers and ONNXRuntime#2959

Merged
cau-git merged 16 commits intomainfrom
cau/inference-engine-object-detection
Feb 10, 2026
Merged

feat: Inference engines abstraction for object detection model family with Transformers and ONNXRuntime#2959
cau-git merged 16 commits intomainfrom
cau/inference-engine-object-detection

Conversation

@cau-git
Copy link
Member

@cau-git cau-git commented Feb 6, 2026

This PR establishes an implementation of generic inference engines for the object-detection model family.
It defines the required option types, base classes, factories, onnx inference engine, and a stage for layout models (in ONNX format) to demonstrate the principle (WIP).

TODO:

  • Transformers inference engine
  • Working layout stage with both ONNX and transformers inference engine.
  • All inference engines generically understand how to apply model settings that can be read from HF configuration, such as this for e.g. class label mapping.
  • Cleanup of arguments
  • Support for remote API model "inference engine"
  • Example unit

What this PR does not include

  • The standard layout model implementation still remains the default
  • No CLI-level exposure for new runtime choices

For reference: https://github.com/docling-project/docling/blob/main/docs/usage/model_catalog.md

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@mergify
Copy link

mergify bot commented Feb 6, 2026

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@github-actions
Copy link
Contributor

github-actions bot commented Feb 6, 2026

DCO Check Passed

Thanks @cau-git, all your commits are properly signed off. 🎉

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
… engines

- Add abstract get_label_mapping() method to BaseObjectDetectionEngine
- Implement label loading from config.json in OnnxRuntimeObjectDetectionEngine
- Refactor LayoutObjectDetectionModel to use engine-provided labels instead of hardcoded mapping
- Centralizes label mapping logic in the inference engine layer

This eliminates hardcoded label dictionaries and makes label mappings configurable through model configs.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Implement TransformersObjectDetectionEngine as a PyTorch-based alternative
to ONNX Runtime. Works as drop-in replacement for both layout and table
detection models with support for CPU, CUDA, and MPS devices.

- Add TransformersObjectDetectionEngine with AutoModelForObjectDetection
- Update TransformersObjectDetectionEngineOptions (score_threshold, torch_dtype)
- Update factory to instantiate Transformers engine
- Switch OBJECT_DETECTION_LAYOUT_HERON preset to use Transformers by default
- Add logging configuration to layout_object_detection_example.py

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git cau-git marked this pull request as ready for review February 10, 2026 09:15
@dosubot
Copy link

dosubot bot commented Feb 10, 2026

Related Documentation

Checked 13 published document(s) in 1 knowledge base(s). No updates required.

How did I do? Any feedback?  Join Discord

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
@cau-git cau-git changed the title feat: inference engine for object detection feat: Inference engines abstraction for object detection model family Feb 10, 2026
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Copy link
Member

@dolfim-ibm dolfim-ibm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great 🚀

@dolfim-ibm dolfim-ibm changed the title feat: Inference engines abstraction for object detection model family feat: Inference engines abstraction for object detection model family with Transformers and ONNXRuntime Feb 10, 2026
@cau-git cau-git merged commit 14e474c into main Feb 10, 2026
26 of 27 checks passed
@cau-git cau-git deleted the cau/inference-engine-object-detection branch February 10, 2026 15:15
cau-git added a commit that referenced this pull request Feb 11, 2026
…mily with HF Transformers and ONNX runtime

Implements runtime abstraction for image classification models with support for both ONNX Runtime and HuggingFace Transformers engines. Users can switch between engines without model retraining, similar to the object detection abstraction (#2959).

Key components:
- BaseImageClassificationEngine with factory pattern
- OnnxRuntimeImageClassificationEngine and TransformersImageClassificationEngine implementations
- Shared HfVisionModelMixin for common HF model utilities
- Engine-specific configuration options
- Test suite and example demonstrating runtime engine switching

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
cau-git added a commit that referenced this pull request Feb 18, 2026
…ication) and KServe v2 API support (#2979)

* feat: Inference engines abstraction for image classification model family with HF Transformers and ONNX runtime

Implements runtime abstraction for image classification models with support for both ONNX Runtime and HuggingFace Transformers engines. Users can switch between engines without model retraining, similar to the object detection abstraction (#2959).

Key components:
- BaseImageClassificationEngine with factory pattern
- OnnxRuntimeImageClassificationEngine and TransformersImageClassificationEngine implementations
- Shared HfVisionModelMixin for common HF model utilities
- Engine-specific configuration options
- Test suite and example demonstrating runtime engine switching

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add missing files and re-export for backward compat

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Don't run with OCR in the example.

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Remove excess onnxruntime related options for inuts and outputs

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: centralize torch compile defaults with DOCLING_INFERENCE_COMPILE_TORCH_MODELS

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* feat: Add Kserve2 API engine for image classifier and object detection models (#2999)

* fix: add failed pages to DoclingDocument for page break consistency (#2939)

* fix: add failed pages to DoclingDocument for page break consistency

When some PDF pages fail to parse, they were not added to
DoclingDocument.pages, causing page break markers to be incorrect
during export. This adds failed/skipped pages with their size info
(if available) to maintain correct page numbering and structure.

- Add _add_failed_pages_to_document() method in StandardPdfPipeline
- Add test cases for failed page handling
- Add test cases for normal page handling (regression test)
- Add test PDF files

Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com>

* fix: ensure resource cleanup and simplify type hints

- Wrap page_backend usage in try-finally to guarantee unload (prevents resource leaks).
- Simplify redundant 'float | None | None' type hint.

Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com>

* fix: add groundtruth for normal_4pages.pdf and exclude failing PDFs from e2e test

Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com>

* fix: ensure correct status assertion for failed pages in tests

Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com>

---------

Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com>

* fix: Use timezone-aware datetime (#2947)

* Use timezone-aware datetime for profiling timestamps

Updated timestamp recording to use timezone-aware datetime.

Signed-off-by: Nikhil Singh <124866156+Ritinikhil@users.noreply.github.com>

* run formatter

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Nikhil Singh <124866156+Ritinikhil@users.noreply.github.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>

* fix(asciidoc): handle commas in image alt text (#2983)

* Fix: Handle commas in AsciiDoc image alt text

  - Modified _parse_picture() to gracefully handle alt text containing commas
  - Commas in alt text are now preserved instead of causing ValueError
  - Added test case with realistic auto-generated alt text
  - split('=', 1) prevents issues when values contain '=' characters

* DCO Remediation Commit for n0rdp0l <n90.w135@gmail.com>

I, n0rdp0l <n90.w135@gmail.com>, hereby add my Signed-off-by to this commit: ee75249

Signed-off-by: n0rdp0l <n90.w135@gmail.com>

* style: fix ruff formatting in test_backend_asciidoc.py

Signed-off-by: n0rdp0l <n90.w135@gmail.com>

---------

Signed-off-by: n0rdp0l <n90.w135@gmail.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>

* chore: bump version to 2.73.1 [skip ci]

* First attempt at establishing API Kserve2 facet

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* refactor: improve KServe v2 engine implementation after code review

- Add comprehensive error handling to KserveV2HttpClient
  - Catch and wrap Timeout, ConnectionError, HTTPError with context
  - Validate response formats with clear error messages

- Refactor URL building to eliminate duplication
  - Extract _build_model_url() helper method
  - Single source of truth for infer_url and model_metadata_url

- Make URL required parameter (remove default localhost:8000)
  - Update ApiKserveV2*EngineOptions to require explicit URL
  - Add preset validation with helpful error messages

- Rename constants for clarity: TRITON_* → KSERVE_V2_*
  - Add comment explaining KServe v2 uses Triton type system

- Improve error messages with actual values
  - Show counts, shapes, and supported types in validation errors

- Document official KServe Python SDK alternative
  - Note async-only requirement and alpha status

- Update tests for required URL parameter

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Cleanup in kserve http helper and options

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Further cleanup

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fix for remote-services on tablemodel

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* fix: improved deserialization of engine_options (#3008)

* add registry of discriminated subclasses

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* fix detection of engine_type value

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* Add options serialization improvements

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com>
Signed-off-by: Nikhil Singh <124866156+Ritinikhil@users.noreply.github.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: n0rdp0l <n90.w135@gmail.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Co-authored-by: jhchoi1182 <jhchoi1182@gmail.com>
Co-authored-by: Nikhil Singh <124866156+Ritinikhil@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Felix Wente <63914035+n0rdp0l@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>

* Fixes from review

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* DCO Remediation Commit for Christoph Auer <cau@zurich.ibm.com>

I, Christoph Auer <cau@zurich.ibm.com>, hereby add my Signed-off-by to this commit: 4cdb01e

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* DCO Remediation Commit for Christoph Auer <60343111+cau-git@users.noreply.github.com>

I, Christoph Auer <60343111+cau-git@users.noreply.github.com>, hereby add my Signed-off-by to this commit: e293ba3

Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Add fallback for API variants

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Recreate uv.lock

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

---------

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: jhchoi1182 <jhchoi1182@gmail.com>
Signed-off-by: Nikhil Singh <124866156+Ritinikhil@users.noreply.github.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: n0rdp0l <n90.w135@gmail.com>
Signed-off-by: Christoph Auer <60343111+cau-git@users.noreply.github.com>
Co-authored-by: jhchoi1182 <jhchoi1182@gmail.com>
Co-authored-by: Nikhil Singh <124866156+Ritinikhil@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Felix Wente <63914035+n0rdp0l@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants