MMDeploy on Jetson AGX Xavier

[habjoel]

Hey, sorry for asking so many questions but I'm really trying to get this work...
I am a bit confused about instructions on how to install mmdeploy on the Jetson. (https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_install_mmdeploy_on_jetsons.html)

At the step of installing mmdeploy itself, it states to refer to [https://mmdeploy.readthedocs.io/en/latest/build.html] but these are only the download instructions... so I assume I should rather follow [https://mmdeploy.readthedocs.io/en/latest/build/linux.html] for the installation.

I want to build mmdeploy with onnx and tensorrt to be able to export to those frameworks. Therefore, I'll need do the following steps:

cd ${MMDEPLOY_DIR}
mkdir -p build && cd build
cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS="ort;trt" -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} -DTENSORRT_DIR=${TENSORRT_DIR} -DCUDNN_DIR=${CUDNN_DIR} ..
make -j$(nproc)

However, I am not sure how to set the environment variables TENSORRT_DIR and CUDNN_DIR as the Jetson has tensorrt and cudnn installed natively! how should I set these?

Thanks a lot!

[tehkillerbee]

On Jetson platforms, including AGX Xavier, TENSORRT is installed in /usr/include/aarch64-linux-gnu. We are currently working on a more automated install using a script

First install the necessary prerequisites. On Jetson, you will need the following

• Spdlog (from apt or source)

# remove libspdlog, if it has already been installed from apt
sudo apt-get remove libspdlog-dev -y

echo "Building and installing latest spdlog from source"
git clone https://github.com/gabime/spdlog.git spdlog
cd spdlog
git pull
git checkout tags/v1.8.1
mkdir build -p && cd build
# we must build spdlog with -fPIC enabled
cmake .. -DCMAKE_POSITION_INDEPENDENT_CODE=ON && make -j
sudo make install
sudo ldconfig

• Cmake >3.14 (latest from source

# purge existing
sudo apt-get purge cmake
sudo snap remove cmake
#
wget https://github.com/Kitware/CMake/releases/download/v3.23.1/cmake-3.23.1-linux-${ARCH}.sh
chmod +x cmake-3.23.1-linux-${ARCH}.sh
sudo ./cmake-3.23.1-linux-${ARCH}.sh --prefix=/usr --skip-license

• Build essentials, if not already installed. This also installs a compatible gcc compiler

sudo apt-get install build-essential gcc make git curl -y

• Opencv: Should be installed per default on Jetson

After installing the necessary prerequisites needed for MMDeploy SDK, you might need to setup your python environment.
The exact steps needed depends on whether you use conda or "standard" venv. The example below uses standard venv

# python 3.6 on jetson requires a slightly different get-pip
curl https://bootstrap.pypa.io/pip/3.6/get-pip.py -o get-pip.py

# misc python prerequisites
sudo apt-get install protobuf-compiler libprotoc-dev libssl-dev curl ninja-build -y
sudo apt-get install libopenblas-dev python3-venv python3-dev python3-setuptools -y
sudo -H python3 get-pip.py
sudo -H pip3 install testresources
sudo -H pip3 install --upgrade setuptools wheel

# create venv and switch to it
python3 -m venv venv-mmdeploy --system-site-packages #system site packages to keep trt from system installation
source venv-mmdeploy/bin/activate

# Python dependencies, needed for mmdeploy, mmcv
python3 get-pip.py
pip3 install testresources
pip3 install --upgrade setuptools wheel

# Latest PIL is not compatible with mmcv=1.4.1
pip3 install Pillow==7.0.0

if [[ "$ARCH" == aarch64 ]]
then
# protofbuf on jetson is quite old - must be upgraded
pip3 install --upgrade protobuf
# Install numpy 1.19.4 as newer versions might give "Illegal instruction (core dumped)" on Jetson
# TODO Numpy might be installed per default so we should not remove it. 
pip3 install numpy==1.19.4
else
pip3 install numpy
fi
pip3 install opencv-python
pip3 install matplotlib

# pytorch, torchvision, torchaudio
if [[ "$ARCH" == aarch64 ]]
then
# pytorch
wget https://nvidia.box.com/shared/static/fjtbno0vpo676a25cgvuqc1wty0fkkg6.whl -O torch-1.10.0-cp36-cp36m-linux_aarch64.whl
pip3 install torch-1.10.0-cp36-cp36m-linux_aarch64.whl
# torchvision
sudo apt-get install libjpeg-dev zlib1g-dev libpython3-dev libavcodec-dev libavformat-dev libswscale-dev -y
sudo rm -r torchvision
git clone --branch v0.11.1 https://github.com/pytorch/vision torchvision
cd torchvision
export BUILD_VERSION=0.11.1  # where 0.x.0 is the torchvision version  
python3 setup.py install
cd ../
# torchaudio
#sudo apt-get install -y sox libsox-dev libsox-fmt-all
#sudo rm -r torchaudio
#git clone -b v0.10.0 https://github.com/pytorch/audio torchaudio
#cd torchaudio
#git submodule update --init --recursive
#python3 setup.py install
#cd ../
# mmcv
pip3 uninstall mmcv-full
pip3 install mmcv-full==1.4.1 -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.10.0/index.html
else
pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
# mmcv
pip3 uninstall mmcv-full
pip3 install mmcv-full==1.4.1 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html
fi

• Then set the env. variables needed when building mmdeploy

ARCH=$(uname -i)
echo 'export TENSORRT_DIR="/usr/include/'${ARCH}'-linux-gnu/"' >> ${HOME}/.bashrc
echo 'export CUDNN_DIR="/usr/include/'${ARCH}'-linux-gnu/"' >> ${HOME}/.bashrc
echo 'export LD_LIBRARY_PATH="/usr/lib/'${ARCH}'-linux-gnu/:$LD_LIBRARY_PATH"' >> ${HOME}/.bashrc
#apply bashrc changes
 exec bash
# validate installed version. Should be  >=8.0.1.6
cat /usr/include/${ARCH}-linux-gnu/NvInferVersion.h | grep NV_TENSORRT

• PPLCV, required for MMDeploy+tensorrt

## ppl.cv
git clone https://github.com/openppl-public/ppl.cv.git ppl.cv
cd ppl.cv
git pull
git checkout tags/v0.6.2

# build
mkdir build -p && cd build
cmake -DHPCC_USE_CUDA=ON -DCMAKE_INSTALL_PREFIX=/usr/local/aarch64-linux-gnu .. && make -j && sudo make install
sudo ldconfig

• MMDeploy

## mmdeploy SDK
git clone https://github.com/open-mmlab/mmdeploy.git mmdeploy
cd mmdeploy
git pull
git checkout tags/v0.4.0
# reinit submodules
git submodule update --init --recursive

# python dependencies
source venv-mmdeploy/bin/activate

## h5py (Required by mmdeploy)
## h5py not directly supported by jetson and must be built/installed manually
sudo apt-get install pkg-config libhdf5-10* libhdf5-dev -y
sudo pip3 install Cython
sudo env H5PY_SETUP_REQUIRES=0 pip3 install -U h5py==2.9.0

pip install -e .

# build
mkdir build -p && cd build
cmake .. \
  -DMMDEPLOY_BUILD_SDK=ON \
  -DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} \
  -DCMAKE_CXX_COMPILER=g++ \
  -Dpplcv_DIR=/usr/local/aarch64-linux-gnu/lib/cmake/ppl \
  -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \
  -DMMDEPLOY_TARGET_BACKENDS=trt \
  -DMMDEPLOY_CODEBASES=all \
  -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
  -DTENSORRT_DIR=${TENSORRT_DIR}
  -DCUDNN_DIR=${CUDNN_DIR}
cmake --build . -- -j$(nproc) && sudo cmake --install .
sudo ldconfig

## build mmdeploy examples
cp -r ${WORKING_DIR}/MMDeploy/demo/csrc ${WORKING_DIR}/MMDeploy/build/example
cd ${WORKING_DIR}/MMDeploy/build/example
rm -r build
mkdir build -p && cd build
cmake -DMMDeploy_DIR=/usr/local/aarch64-linux-gnu ..
make all

There are many steps involved and they are not that straightforward.
In PR #399 a new build install script is being developed. The steps listed in this comment are essentially the same as this script. Please feel free to test or give feedback to this script.

[habjoel]

Hi @tehkillerbee,
Thanks a lot for the detailed instructions! I installed it on the L4T ML Docker container from NVidia and the steps worked. (https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-ml). However, I'm a bit confused as to where I can now find the SDK as this procedure with copying the example files over to build/example gives me no /install directioy.
According to "Get Started" it should be somehow like this:

install
├── example
├── include
│   ├── c
│   └── cpp
└── lib

I am going to use MMDeploy to convert a Swin instance segmentation model. I know that it's not officially supported yet but I already converted it to ONNX on my Laptop and it worked well. Only problem is that it runs very slowly on ONNX Runtime on the Jetson (Inference Time ~0.5s) so I am hoping for some real speedup through TensorRT.

General question: Would you advise me to use the TensorRT Python API for Inferencing or should i rather use the SDK from mmdeploy? Also: there is really no documentation of the SDK or am I wrong?

[tehkillerbee]

@habjoel Regarding installation of the SDK, in the above script, it will be installed in the "${INSTALL_PREFIX}" folder. I assume you have set this prefix according to the MMDeploy documentation? If you build MMDeploy SDK according to the documentation, you should probably leave out DCMAKE_INSTALL_PREFIX. This should give you a folder structure that corresponds to the Get started.

In my scripts, i've set DCMAKE_INSTALL_PREFIX to /usr/local/aarch64-linux-gnu. This makes it easier to cross compile MMDeploy tools, as the SDK is installed in a standard location.

  mkdir build -p && cd build
  cmake .. \
    -DMMDEPLOY_BUILD_SDK=ON \
    -DCMAKE_CXX_COMPILER=g++ \
    -Dpplcv_DIR=${INSTALL_PREFIX}/lib/cmake/ppl \
    -DMMDEPLOY_TARGET_DEVICES="cuda;cpu" \
    -DMMDEPLOY_TARGET_BACKENDS=trt \
    -DMMDEPLOY_CODEBASES=all \
    -DMMDEPLOY_BUILD_SDK_PYTHON_API=ON \
    -DTENSORRT_DIR=${TENSORRT_DIR}
    -DCUDNN_DIR=${CUDNN_DIR}

I would suggest you try the C API example code as a starting point instead of the Python API. I've gotten both speed and memory improvements in this way, although I am using a Mask RCNN instance segmentation model so your results may vary. It would be interesting to hear your progress, if you get the Swin model working. What Jetson are you using?

I do not think there is a huge amount of documentation describing the SDK but the example code is a good starting point. You can probably use the object_detection.cpp example directly, after you have converted your model to TensorRT engine.

[habjoel]

@tehkillerbee Alright, thanks for the info!

I am using a Jetson AGX Xavier Developer Kit. I'm really trying to get this work but I am getting the following error during conversion and I don't know what to do about it:

RuntimeError: Failed to parse onnx, In node 49 (parseGraph): INVALID_NODE: Invalid Node - Pad_49
[shuffleNode.cpp::symbolicExecute::387] Error Code 4: Internal Error (Reshape_38: IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [-1,2])

2022-05-09 16:44:18,197 - mmdeploy - ERROR - onnx2tensorrt of ./end2end.onnx failed.

Obviously onnx2tensorrt somehow fails to parse the end2end.onnx file.
I guess the problem lies within the generation of the .onnx file, where I keep getting the following warnings:

WARNING: The shape inference of mmdeploy::MMCVMultiLevelRoiAlign type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.

WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.

Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.

I see that TRTBatchedNMS and MMCVMultiLevelRoiAlign are part of the TensorRT Ops provided by MMDeploy (https://mmdeploy.readthedocs.io/en/latest/ops/tensorrt.html). So something concerning this is not working. How can I confirm that these plugins are built properly?

I used the config file instance-seg_tensorrt-fp16_static-800x1344.py but changed the input sizes to 480x640.

[tehkillerbee]

@habjoel The warnings regarding MMCVMultiLevelRoiAlign and TRTBatchedNMS also occur for me when I generate the trt engine. Whether they are serious, I am not sure, since the model output appears correct.

What version of JetPack/TensorRT are you running?

Also, can you include the full modifications you made to instance-seg_tensorrt-fp16_static-800x1344.py? Did you only change the max resolution or did you set min/max/opt to be identical?

[habjoel]

@tehkillerbee Alright, I am running on a Docker container with the following configuration:
JetPack 4.6.1 (L4T R32.7.1)

l4t-ml:r32.7.1-py3
TensorFlow 1.15.5
PyTorch v1.10.0
torchvision v0.11.0
torchaudio v0.10.0
onnx 1.11.0
CuPy 9.2.0
numpy 1.19.5
numba 0.53.1
OpenCV 4.5.0 (with CUDA)
pandas 1.1.5
scipy 1.5.4
scikit-learn 0.24.2
JupyterLab 2.2.9

The TensorRT version is 8.2.1.8.

My config looks as follows:

_base_ = [
    '../_base_/base_instance-seg_static.py',
    '../../_base_/backends/tensorrt-fp16.py'
]

onnx_config = dict(input_shape=(640, 480))
backend_config = dict(
    common_config=dict(max_workspace_size=1 << 30),
    model_inputs=[
        dict(
            input_shapes=dict(
                input=dict(
                    min_shape=[1, 3, 480, 640],
                    opt_shape=[1, 3, 480, 640],
                    max_shape=[1, 3, 480, 640])))
    ])

I don't know if the config is really the problem since I tried it using instance-seg_tensorrt-fp16_dynamic-320x320-1344x1344.py and the same error occured with onnx2tensorrt unable to parse end2end.onnx

[habjoel]

@tehkillerbee
Alright, I think the problem really is that I am trying to convert the Swin model. I tried the converion using standard Mask-RCNN and it worked...
Do you know how I could somehow make this thing work for the Swin?

[habjoel]

@tehkillerbee
I managed to get rid of the error

[shuffleNode.cpp::symbolicExecute::387] Error Code 4: Internal Error (Reshape_38: IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [-1,2])

Which is caused because the constants cannot be folded during conversion to onnx

Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.

To get rid of it, I used the polygraphy tool of TensorRT on the intermediate onnx model (end2end.onnx) according to this: https://github.com/NVIDIA/TensorRT/tree/master/tools/Polygraphy/examples/cli/surgeon/02_folding_constants

I then tried using tools/onnx2tensorrt.py to convert the "fixed" onnx model further to tensorrt. However, it throws yet another error that I cannot solve...

[05/12/2022-13:47:14] [TRT] [E] [layers.cpp::TopKLayer::4233] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/layers.cpp::TopKLayer::4233, condition: k > 0 && k <= MAX_TOPK_K
)
2022-05-12 13:47:14,724 - mmdeploy - ERROR - Failed to parse onnx, In node 1945 (importTopK): UNSUPPORTED_NODE: Assertion failed: layer && "Failed to add TopK layer."

I am also getting the same error if I use the trtexec tool from TensorRT. I found something on nvidia docs concerning this: https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#topk-layer

Maybe it is caused by some unsupported ops that come with the swin transformer... but is there really an op within the swin transformer that is different from the ops other exportable models use (e.g. mask-rcnn)?

[tehkillerbee]

@habjoel My guess would be an unsupported ops - but I am no expert on this so I cannot say for sure, sorry.
I would probably consult this list to see what ops that are supported by onnx-tensorrt:
here

Are there no other relevant warnings or errors listed in the log?

[habjoel]

@tehkillerbee Yeah, I guess so. I am going to wait until MMDeploy officially supports Swin (expected in June) and work with the standard Mask-RCNN for now as it is too time consuming for me to write a custom plugin or similar.

[habjoel]

@tehkillerbee By the way, have you implemented a script that runs your exported Mask RCNN model using the TensorRT API? If yes, would you mind sharing it with me? that would greatly speed up my process!

[tehkillerbee]

@habjoel The tool I use is largely based on the MMDeploy SDK example listed here. You should be able to use this example code directly with Mask RCNN.

[lvhan028]

Closing it since no more discussions for a month. Feel free to reopen it if it is still an issue.