TRT export

[bpfliegel]

Hello,

I swear I tried everything to match cuda+cudnn+onnx+trt+pytorch versions in order to build both onnx and trt plugins and be able to convert a maskrcnn model to trt, but no luck so far.

Docs are a bit contradictory in this respect, e.g. these pages:

https://mmdetection.readthedocs.io/en/latest/tutorials/pytorch2onnx.html
All models above are tested with Pytorch==1.6.0 and onnxruntime==1.5.1, except for CornerNet.
https://mmdetection.readthedocs.io/en/latest/tutorials/onnx2tensorrt.html
All models above are tested with Pytorch==1.6.0, onnx==1.7.0 and TensorRT-7.2.1.6.Ubuntu-16.04.x86_64-gnu.cuda-10.2.cudnn8.0 (this is likely onnxruntime 1.5.x).
https://mmcv.readthedocs.io/en/latest/deployment/onnxruntime_op.html
Please be noted that only onnxruntime>=1.8.1 of CPU version on Linux platform is tested by now.

For instance: onnxruntime 1.8.1 installed from pip assumes cuda 11.0, but for PyTorch 1.6.0 there is no cuda 11.0 build (PyTorch team introduced it only up from 1.7.0).

So far, I generally ended up with the infamous 'UNSUPPORTED_NODE: Assertion failed: axis >= 0 && axis < nbDims' error with pretty much anything I tried - like these guys: #5639 and #5108.

Please kindly let me know what combination works for sure with the latest 2.16.0 version. I have a guess, which I will try now :) I also promise I will make a docker image for the community that other folks don't pull their hair out.

CUDA version: ???
CuDNN version: ???
PyTorch version: ???
ONNX version: ???
ONNX runtime version: ???
TRT version: ???

Thanks a lot, Balint

[RangiLyu]

The onnxruntime version is determined by your mmcv version. With the latest mmcv, you should install onnxruntime>=1.8.1. But if you are using old version of pytorch and cuda, I'm afraid that you need to use old version of mmcv which support lower version of onnxruntime.

[RangiLyu]

@RunningLeon Please take a look at this issue.

[neverrop]

Got the same issue for converting point_rend model to trt. 😢

[fbagci]

same issue, any suggestions?

[RunningLeon]

@fbagci @neverrop Hi, can you post your error messages and outputs from python mmdet/utils/collect_env.py
BTW, you could use the new deployment toolkit mmdeploy and you could check the supported model list here: https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/supported_models.md

[fbagci]

@RunningLeon thanks for response, the output for python mmdet/utils/collect_env.py is:

fatal: not a git repository (or any of the parent directories): .git
sys.platform: linux
Python: 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0]
CUDA available: True
GPU 0: GeForce RTX 2080 Ti
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.2, V10.2.89
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.6.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.2-Product Build 20210312 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.5.0 (Git Hash e2ac1fac44c5078ca927cb9b90e1b3066a0b2ed0)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 10.2
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75
  - CuDNN 7.6.5
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF, 

TorchVision: 0.9.1
OpenCV: 4.5.5
MMCV: 1.4.3
MMCV Compiler: GCC 7.5
MMCV CUDA Compiler: 10.2
MMDetection: 2.20.0+

Also:

python mmdetection/tools/deployment/pytorch2onnx.py configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py work_dirs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth --output-file fasterrcnn.onnx
produces fasterrcnn.onnx successfull.

However,
python mmdetection/tools/deployment/onnx2tensorrt.py configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py ./fasterrcnn.onnx --trt-file fasterrcnn.trt
does not work and outputs
mmdetection/tools/deployment/onnx2tensorrt.py:202: UserWarning: Arguments like `--to-rgb`, `--mean`, `--std`, `--dataset` would be parsed directly from config file and are deprecated and will be removed in future releases. removed in future releases.') Traceback (most recent call last): File "mmdetection/tools/deployment/onnx2tensorrt.py", line 254, in <module> verbose=args.verbose) File "mmdetection/tools/deployment/onnx2tensorrt.py", line 45, in onnx2tensorrt max_workspace_size=max_workspace_size) File "/backend/mmcv/mmcv/tensorrt/tensorrt_utils.py", line 63, in onnx2trt raise RuntimeError(f'parse onnx failed:\n{error_msgs}') RuntimeError: parse onnx failed: In node -1 (importTopK): UNSUPPORTED_NODE: Assertion failed: inputs.at(1).is_weights()

[RunningLeon]

@fbagci @neverrop Hi, can you post your error messages and outputs from python mmdet/utils/collect_env.py BTW, you could use the new deployment toolkit mmdeploy and you could check the supported model list here: https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/supported_models.md

@fbagci Hi, you may need to add one env variable with pytorch2onnx.py tool： https://github.com/open-mmlab/mmdetection/blob/master/docs/en/tutorials/pytorch2onnx.md#description-of-all-arguments-1

BTW, please try with mmdeploy, which is more convenient.

[fbagci]

@fbagci Hi, you may need to add one env variable with pytorch2onnx.py tool： https://github.com/open-mmlab/mmdetection/blob/master/docs/en/tutorials/pytorch2onnx.md#description-of-all-arguments-1
image

@neverrop Setting env var does not solve problem. So, i tried to use mmdeploy. Since the local installations are tricky and open to error for library version inconsistencies, I used dockerfile in PR. This did not worked as well. The deploy.py converted torch model to onnx successfully but failed while converting onnx to tensorrt. Since PR is not merged there may be some missing parts. My usage scenario is;

git clone https://github.com/AllentDan/mmdeploy.git

cd mmdeploy

git checkout docker

cp /path/to/models mmdeploy/models # Copy fasterrcnn checkpoint, config and example image from mmdetection to mmdeploy project.

docker build docker/GPU/ -t mmdeploy:gpu

docker run --gpus all -it -v $(pwd)/models:/root/workspace/mmdeploy/models -p 8080:8081 mmdeploy:gpu

cd mmdeploy

python tools/deploy.py configs/mmdet/detection/detection_tensorrt_dynamic-160x160-608x608.py models/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py models/faster_rcnn/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth models/faster_rcnn/test_img.jpg --work-dir models/faster_rcnn/deploy --device cuda:0

output is

  warnings.warn("Exporting aten::index operator of advanced indexing in opset " +
2022-01-17 13:32:13,731 - mmdeploy - INFO - torch2onnx success.
[2022-01-17 13:32:13.819] [mmdeploy] [info] [model.cpp:97] Register 'DirectoryModel'
[2022-01-17 13:32:13.819] [mmdeploy] [error] [cuda_device.cpp:268] error getting device count: initialization error
terminate called after throwing an instance of 'system_error2::status_error<mmdeploy::StatusDomain>'
  what():  unknown (6) @ /root/workspace/mmdeploy/csrc/device/cuda/cuda_device.cpp:269
Aborted (core dumped)

I suspect that cuda version in container is incorrect. Cuda version argument in dockerfile is set to 10.2 by default but in container nvcc -Voutputs

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0

[AllentDan]

Hi, @fbagci. The CUDA arg only gets used for conda installing pytorch and cudatoolkit. Actually, conda already separates the env to a virtual one which only influences the usage of pytorch. I am not sure, but the docker container just works fine in my T4 machine. BTW, for dynamic exporting, 160x160-608x608 also failed in my machine but in visualize part. The config is not suitable for dynamically exporting faster-rcnn, 320x320-1344x1344 is the right one which worked all fine.