[Inference] Add tuned trt_dynamic_shape mode. #34806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

jiweibo merged 32 commits into PaddlePaddle:develop from jiweibo:tuned_dynamic_shape

Sep 14, 2021

Contributor

jiweibo commented Aug 11, 2021 •

edited

Loading

PR types

New features

PR changes

Others

Describe

trt support tuned_dynamic_shape mode.

1、Use offline data for autotune, get the shape range information at runtime and save it in the shape_range_info.pbtxt .

config.CollectShapeRangeInfo("shape_range_info.pbtxt")

the file looks like

shape_range_info {
  name: "fc_11.tmp_0"
  min_shape: 1
  min_shape: 32
  min_shape: 768
  max_shape: 1
  max_shape: 128
  max_shape: 768
  opt_shape: 1
  opt_shape: 32
  opt_shape: 768
}
shape_range_info {
  name: "fc_11.tmp_1"
  min_shape: 1
  min_shape: 32
  min_shape: 768
  max_shape: 1
  max_shape: 128
  max_shape: 768
  opt_shape: 1
  opt_shape: 32
  opt_shape: 768
}
shape_range_info {
  name: "fc_12.tmp_0"
  min_shape: 1
  min_shape: 32
  min_shape: 768
  max_shape: 1
  max_shape: 128
  max_shape: 768
  opt_shape: 1
  opt_shape: 32
  opt_shape: 768
}
...

2、Comment config.CollectShapeInfo("shape_range_info.pbtxt") and set config.EnableTunedTensorRtDynamicShape("shape_range_info.pbtxt", /*allow_build_at_runtime*/ true);

3、DEMO：https://github.com/PaddlePaddle/Paddle-Inference-Demo/tree/master/c%2B%2B/paddle-trt/tuned_dynamic_shape


          add tuned trt dynamic shape mode.

2af763b

paddle-bot-old bot commented Aug 11, 2021

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

jiweibo added 3 commits

August 11, 2021 09:16


          update io test

96ee635


          add proto file

122f108


          add python api.

99bd981

jiweibo commented

View reviewed changes

paddle/fluid/inference/api/analysis_config.cc Show resolved Hide resolved

jiweibo commented

View reviewed changes

paddle/fluid/inference/api/analysis_config.cc Outdated Show resolved Hide resolved

jiweibo commented

View reviewed changes

paddle/fluid/inference/api/analysis_config.cc Outdated Show resolved Hide resolved

jiweibo commented

View reviewed changes

paddle/fluid/inference/tensorrt/engine.cc Show resolved Hide resolved

jiweibo commented

View reviewed changes

paddle/fluid/inference/utils/shape_info.proto Outdated Show resolved Hide resolved

jiweibo commented

View reviewed changes

paddle/fluid/inference/utils/io_utils.h Outdated Show resolved Hide resolved

jiweibo added 19 commits

August 13, 2021 01:56


          add ut

c3e8ab2


          fix initial error

450cf95


          update ShapeInfo to ShapeRangeInfo

fdf4650


          imporve ci coverage.

59531f8


          fix failed ut

6efc679


          support serialize when rebuild engine

e575abc


          fix engine_op_test

a5bdc92


          Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

abf72d9

… tuned_dynamic_shape


          Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

300804b

… tuned_dynamic_shape


          Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

8fade9b

… tuned_dynamic_shape


          Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

8abb25a

… tuned_dynamic_shape


          add ut.

f4614a9


          Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

d4576f7

… tuned_dynamic_shape


          fix ut error.

0f58860


          Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

27fd96c

… tuned_dynamic_shape


          Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

23b2a85

… tuned_dynamic_shape


          add ut.

309e332


          add trt engine ut.

dcfc7cb


          fix compile error

315aab9


          skip ci error

7850fa2

jiweibo requested review from Shixiaowei02, cryoco and shangzhizhou

September 8, 2021 08:55

shangzhizhou reviewed

View reviewed changes

paddle/fluid/inference/api/analysis_predictor_tester.cc Outdated Show resolved Hide resolved

paddle/fluid/inference/api/analysis_predictor_tester.cc Outdated Show resolved Hide resolved

paddle/fluid/inference/tensorrt/engine.h Outdated Show resolved Hide resolved

paddle/fluid/inference/tests/api/trt_dynamic_shape_test.cc Show resolved Hide resolved

jiweibo added 5 commits

September 9, 2021 08:06


          update tests.

b7254e1


          update test

e615f38


          fix ut error.

c3df13f


          update

00a1833


          Merge branch 'develop' into tuned_dynamic_shape

853f024

cryoco previously approved these changes

View reviewed changes

cryoco left a comment

LGTM

Shixiaowei02 reviewed

View reviewed changes

paddle/fluid/inference/api/paddle_analysis_config.h Outdated Show resolved Hide resolved

CheQiXiao previously approved these changes

View reviewed changes

jiweibo added 2 commits

September 10, 2021 08:23


          update

8085ff1


          Merge branch 'tuned_dynamic_shape' of https://github.com/jiweibo/Paddle…

d816963

… into tuned_dynamic_shape

jiweibo dismissed stale reviews from CheQiXiao and cryoco via

d816963

September 10, 2021 08:24


          fix compile error

4a871e4

Shixiaowei02 approved these changes

View reviewed changes

CheQiXiao approved these changes

View reviewed changes

cryoco approved these changes

View reviewed changes

jiweibo requested a review from shangzhizhou

September 13, 2021 03:02

shangzhizhou approved these changes

View reviewed changes

Member

shangzhizhou left a comment

LGTM

lanxianghit approved these changes

View reviewed changes

Contributor

lanxianghit left a comment

LGTM for API changes, compatibility confirmed.

raindrops2sea approved these changes

View reviewed changes

Member

raindrops2sea left a comment

LGTM

jiweibo merged commit 7c96efe into PaddlePaddle:develop

jiweibo deleted the tuned_dynamic_shape branch

September 14, 2021 02:23

jiweibo mentioned this pull request

c and go api support tuned trt. #35771

Merged

AnnaTrainingG pushed a commit to AnnaTrainingG/Paddle that referenced this pull request


          [Inference] Add tuned trt_dynamic_shape mode. (PaddlePaddle#34806)

ffde341

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

cryoco cryoco approved these changes

raindrops2sea raindrops2sea approved these changes

Shixiaowei02 Shixiaowei02 approved these changes

lanxianghit lanxianghit approved these changes

shangzhizhou shangzhizhou approved these changes

+1 more reviewer

CheQiXiao CheQiXiao approved these changes

Labels

None yet