Skip to content

[INFER][LLM] Add the AutoPredictor for inference#9445

Merged
ZHUI merged 9 commits into
PaddlePaddle:developfrom
zeroRains:auto
Dec 3, 2024
Merged

[INFER][LLM] Add the AutoPredictor for inference#9445
ZHUI merged 9 commits into
PaddlePaddle:developfrom
zeroRains:auto

Conversation

@zeroRains
Copy link
Copy Markdown
Contributor

@zeroRains zeroRains commented Nov 17, 2024

PR types

New features

PR changes

Others

Description

整合现有全部Predictor的选择条件,创建AutoPredicotr,可以处理static graph/dynamic graph,inference_mode/not inference_mode,block_attn/not block_attn的情况。

调用方式

predictor = AutoPredictor.create_predictor(predictor_args, config, model_args, tokenizer, model=model)

AutoPredictor的执行流程图如下:

Predictor流程图

当前LLM最简单的调用流程如下图:

8CD0A83C3720B34F3C11FC75FE048AFB

@paddle-bot
Copy link
Copy Markdown

paddle-bot Bot commented Nov 17, 2024

Thanks for your contribution!

@codecov
Copy link
Copy Markdown

codecov Bot commented Nov 26, 2024

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 53.08%. Comparing base (4b02477) to head (42b3e6a).
Report is 229 commits behind head on develop.

Files with missing lines Patch % Lines
paddlenlp/transformers/auto/modeling.py 0.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #9445      +/-   ##
===========================================
+ Coverage    52.93%   53.08%   +0.14%     
===========================================
  Files          688      699      +11     
  Lines       109379   110476    +1097     
===========================================
+ Hits         57899    58641     +742     
- Misses       51480    51835     +355     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yuanlehome yuanlehome self-assigned this Nov 27, 2024
@yuanlehome yuanlehome self-requested a review November 27, 2024 03:27
@ZHUI ZHUI merged commit 2c1387f into PaddlePaddle:develop Dec 3, 2024
@zeroRains zeroRains deleted the auto branch December 3, 2024 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants