Skip to content

Fix QLinearConv/PRelu edge cases and add NMS argmax option#871

Merged
PINTO0309 merged 1 commit intomainfrom
fix-qconv
Feb 12, 2026
Merged

Fix QLinearConv/PRelu edge cases and add NMS argmax option#871
PINTO0309 merged 1 commit intomainfrom
fix-qconv

Conversation

@PINTO0309
Copy link
Owner

@PINTO0309 PINTO0309 commented Feb 12, 2026

Summary

This PR improves robustness in quantized conversion paths and adds a new NonMaxSuppression option for class-score shrinking.

It addresses three concrete conversion failures seen during real model conversion, and adds a user-facing CLI/API switch for NMS behavior.

Background / Motivation

While converting quantized and post-process-heavy models, the following failure patterns were observed:

  1. QLinearConv aborted when output shape metadata was None in the auto_pad == 'NOTSET' path.
  2. QLinearConv depthwise weight reshape failed with mismatched element counts (invalid reshape target).
  3. PRelu failed broadcasting when slope was channel-first style (e.g. [C,1,1]) but runtime tensor layout was channel-last.

In addition, for some NMS post-processing models (e.g. DAMO-YOLO style layouts), users requested a mode to shrink scores class dimension by argmax before NMS.

Changes

1) QLinearConv robustness fixes

File: onnx2tf/ops/QLinearConv.py

  • Guarded SAME-padding shape comparison against missing metadata:
    • Before: accessed output_tensor_shape[2:] directly
    • After: checks both input/output shapes are not None before comparison
  • Fixed depthwise filter reshape target:
    • Before: [..., input_weights_shape[2], input_weights_shape[3] // group]
    • After: [..., -1, input_weights_shape[3] // group]
    • This aligns behavior with Conv.py and avoids invalid reshape size errors.

2) PRelu slope layout alignment

File: onnx2tf/ops/PRelu.py

  • Reworked slope handling to align slope layout with runtime input tensor layout.
  • Added reshape logic for patterns like:
    • input: [N,H,W,C]
    • slope: [C,1,1]
    • transformed slope: [1,1,C]
  • Implemented in a layout-agnostic way based on runtime tensor shape inspection.

3) New NMS option: --output_nms_with_argmax (-onwa)

Files:

  • onnx2tf/onnx2tf.py
  • onnx2tf/ops/NonMaxSuppression.py
  • README.md

Added new CLI/API option to shrink class dimension of NMS scores:

  • Input scores: [B, C, N]
  • With option: argmax/reduce_max over class axis -> [B, 1, N]

Implementation details in NonMaxSuppression.py:

  • Compute per-box class ids with argmax(scores, axis=1)
  • Use reduced scores for NMS (reduce_max(..., keepdims=True))
  • Reconstruct output [batch_index, class_index, box_index] by gathering class ids for selected boxes
  • If scores rank is not 3, warn and fallback to existing behavior
image image

4) Documentation and version update

Files:

  • README.md

  • onnx2tf/__init__.py

  • pyproject.toml

  • Documented the new NMS argmax option in README (CLI and Python API sections).

  • Updated version from 2.0.8 to 2.0.9.

Validation

  • python -m py_compile onnx2tf/onnx2tf.py onnx2tf/ops/QLinearConv.py onnx2tf/ops/PRelu.py onnx2tf/ops/NonMaxSuppression.py onnx2tf/__init__.py
  • Sanity-checked NMS output tuple construction for argmax mode and verified output format [batch, class, box].

Compatibility / Risk

  • Default behavior is unchanged unless --output_nms_with_argmax is explicitly enabled.
  • QLinearConv and PRelu changes are defensive and targeted at previously failing shape/layout edge cases.

Issue

@PINTO0309 PINTO0309 merged commit 02236ea into main Feb 12, 2026
3 checks passed
@PINTO0309 PINTO0309 deleted the fix-qconv branch February 12, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant