Skip to content

Conversation

@evkotov
Copy link
Contributor

@evkotov evkotov commented Nov 3, 2025

Details:

Problem:
OpenVINO rejects ONNX models with GatherND when batch dimensions are broadcastable but not equal (e.g., data=[1,3,4,4], indices=[2,2,2], batch_dims=1). However, ONNXRuntime accepts and successfully executes these models.

Root Cause:

  • Shape inference used strict DimType::merge() requiring exact equality of batch dimensions
  • Reference implementation and CPU plugin did not support broadcasting on batch dimensions

Solution:
Implemented full broadcasting support for GatherND batch dimensions to match ONNX specification and ONNXRuntime behavior:

  1. Shape Inference (gather_nd_shape_inference.hpp):

    • Replaced DimType::merge() with DimType::broadcast_merge() to allow dimension 1 to broadcast to any size N
    • Updated validation message: "must be the same" → "must be broadcastable"
  2. Reference Implementation (gather_nd.hpp):

    • Compute broadcast batch shape as max(data_dim, indices_dim) for each batch dimension
    • Calculate broadcast-aware strides (stride=0 for dimension 1)
    • Multi-dimensional batch index conversion with broadcasting logic
  3. CPU Plugin (gather_nd.cpp, gather_nd.h):

    • Extended GatherNDAttributes with idxDims field
    • Added broadcast support fields: batchShape, srcBatchStrides, idxBatchStrides
    • Updated both gatherBlocks() and gatherElementwise() methods with broadcast-aware iteration
    • Lambda helper getBatchOffsets() for clean broadcast offset calculation
  4. Documentation (gather-nd-5.rst, gather-nd-8.rst):

    • Updated batch_dims description: "must be equal" → "must be broadcastable"
  5. Tests:

    • Type prop tests: 6 new broadcasting tests (3 for v5, 3 for v8), updated fail tests
    • Shape inference tests: Added broadcasting validation test
    • Functional tests: 5 new reference tests with actual data validation (2 for v5, 3 for v8)

Tickets:

  • 169059

@evkotov evkotov self-assigned this Nov 3, 2025
@evkotov evkotov requested review from a team as code owners November 3, 2025 14:28
@evkotov evkotov added the category: ONNX FE OpenVINO ONNX FrontEnd label Nov 3, 2025
@github-actions github-actions bot added category: Core OpenVINO Core (aka ngraph) category: CPU OpenVINO CPU plugin and removed category: ONNX FE OpenVINO ONNX FrontEnd labels Nov 3, 2025
Copy link
Contributor

@t-jankowski t-jankowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test in src/core/tests/type_prop/gather_nd.cpp.

@evkotov
Copy link
Contributor Author

evkotov commented Nov 5, 2025

Please add a test in src/core/tests/type_prop/gather_nd.cpp.

I added

@evkotov evkotov requested a review from t-jankowski November 5, 2025 16:56
NODE_VALIDATION_CHECK(
op,
DimType::broadcast_merge(output_dims[dim_idx], data_pshape[dim_idx], indices_pshape[dim_idx]),
"Batch dimensions of data and indices must be broadcastable.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GatherND op specification mentions that batch needs to be equal, so it would also need to be updated.

Do we know whether reference/plugin implementations would handle such broadcast? I think it would make sense to add few tests to make sure result of such broadcasted GatherND would produce expected outputs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed refernece implementation, docs and CPU plugin

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually such change is request of new version of operator.
It has to be check if three is no compatibility with previous version of OV.
If this is for ONNX then mayb ONNX FE can process node on front end to provide compatible graph for OV

@evkotov evkotov requested review from a team as code owners November 6, 2025 14:00
@evkotov evkotov requested review from tsavina and removed request for a team November 6, 2025 14:00
@github-actions github-actions bot added category: docs OpenVINO documentation category: TEMPLATE OpenVINO Template plugin labels Nov 6, 2025
@maxnick
Copy link
Contributor

maxnick commented Nov 6, 2025

@nshchego, could you please review the CPU part?

@maxnick maxnick requested a review from nshchego November 6, 2025 14:09
@evkotov evkotov requested a review from praasz November 6, 2025 14:13
NODE_VALIDATION_CHECK(
op,
DimType::broadcast_merge(output_dims[dim_idx], data_pshape[dim_idx], indices_pshape[dim_idx]),
"Batch dimensions of data and indices must be broadcastable.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually such change is request of new version of operator.
It has to be check if three is no compatibility with previous version of OV.
If this is for ONNX then mayb ONNX FE can process node on front end to provide compatible graph for OV

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: Core OpenVINO Core (aka ngraph) category: CPU OpenVINO CPU plugin category: docs OpenVINO documentation category: TEMPLATE OpenVINO Template plugin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants