[Doc] V1 user guide #13991

JenZhao · 2025-02-27T20:42:02Z

This PR adds the v1 User Guide as a living document, to be updated and expanded over time.

github-actions · 2025-02-27T20:42:13Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Jennifer Zhao <[email protected]>

docs/source/getting_started/v1_user_guide.md

robertgshaw2-redhat

Looking good! can you finish this off tomorrow? I want to include a link to this page in the logs

Signed-off-by: Jennifer Zhao <[email protected]>

docs/source/getting_started/v1_user_guide.md

Signed-off-by: Jennifer Zhao <[email protected]>

ywang96

Left some comments - PTAL

docs/source/getting_started/v1_user_guide.md

Co-authored-by: Roger Wang <[email protected]>

Signed-off-by: Jennifer Zhao <[email protected]>

Signed-off-by: Roger Wang <[email protected]>

WoosukKwon · 2025-03-04T10:31:00Z

Thanks for the PR! I will take a look tmr (Tue).

Signed-off-by: Jennifer Zhao <[email protected]>

LiuXiaoxuanPKU

Thanks for the guide. It's great with lots of information.

Just some concerns:

On the high level, I feel it's a bit too negative? We talk a lot about features we don't support or need to be optimized. I feel we should highlight some optimized features, such as chunked prefill and prefix caching.
I feel we need to mention the scheduler (scheduling policy) somewhere, currently we batch prefill tokens and decoding tokens in the same batch, we don't prioritize prefill/decode.

docs/source/getting_started/v1_user_guide.md

LiuXiaoxuanPKU · 2025-03-05T23:08:53Z

docs/source/getting_started/v1_user_guide.md

+(e.g., see [PR #13096](https://github.com/vllm-project/vllm/pull/13096)).
+
+- **Spec Decode**: Currently, only ngram-based spec decode is supported in V1. There
+  will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)).


Suggested change

will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)).

will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)). We will prioritize the support for Eagle, MTP compared to draft model based spec decode.

Signed-off-by: Jennifer Zhao <[email protected]>

docs/source/getting_started/v1_user_guide.md

Co-authored-by: Cyrus Leung <[email protected]>

simon-mo

Thank you for working on this!!

docs/source/getting_started/v1_user_guide.md

simon-mo · 2025-03-13T17:36:36Z

Oh we need a small section on hardware support saying NVIDIA is natively supported, AMD and TPU are work in progress.

Signed-off-by: Jennifer Zhao <[email protected]>

simon-mo

This is a great first verison!

Signed-off-by: Jennifer Zhao <[email protected]> Signed-off-by: Roger Wang <[email protected]> Signed-off-by: Jennifer Zhao <[email protected]> Co-authored-by: Jennifer Zhao <[email protected]> Co-authored-by: Jennifer Zhao <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Signed-off-by: Louis Ulmer <[email protected]>

Signed-off-by: Jennifer Zhao <[email protected]> Signed-off-by: Roger Wang <[email protected]> Signed-off-by: Jennifer Zhao <[email protected]> Co-authored-by: Jennifer Zhao <[email protected]> Co-authored-by: Jennifer Zhao <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>

Signed-off-by: Jennifer Zhao <[email protected]> Signed-off-by: Roger Wang <[email protected]> Signed-off-by: Jennifer Zhao <[email protected]> Co-authored-by: Jennifer Zhao <[email protected]> Co-authored-by: Jennifer Zhao <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Signed-off-by: Mu Huai <[email protected]>

mergify bot added the documentation Improvements or additions to documentation label Feb 27, 2025

JenZhao and others added 10 commits February 27, 2025 13:33

add vLLM V1 User Guide Template

904690e

Signed-off-by: Jennifer Zhao <[email protected]>

wip update v1 user guide

c294f75

Signed-off-by: Jennifer Zhao <[email protected]>

update logprob desc

2283371

Signed-off-by: Jennifer Zhao <[email protected]>

update logprobs desc

1486cb8

Signed-off-by: Jennifer Zhao <[email protected]>

update

f9bb563

Signed-off-by: Jennifer Zhao <[email protected]>

fix

5002550

Signed-off-by: Jennifer Zhao <[email protected]>

update

b395576

Signed-off-by: Jennifer Zhao <[email protected]>

update

658957f

Signed-off-by: Jennifer Zhao <[email protected]>

Update v1_user_guide.md

eacf90a

Signed-off-by: Jennifer Zhao <[email protected]>

Merge branch 'vllm-project:main' into v1_guide

bf852c6

robertgshaw2-redhat reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

robertgshaw2-redhat reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

robertgshaw2-redhat reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

robertgshaw2-redhat reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Show resolved Hide resolved

robertgshaw2-redhat requested changes Mar 2, 2025

View reviewed changes

update unsupported

560b267

Signed-off-by: Jennifer Zhao <[email protected]>

shen-shanshan reviewed Mar 2, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

address comments

33d759f

Signed-off-by: Jennifer Zhao <[email protected]>

ywang96 reviewed Mar 2, 2025

View reviewed changes

JenZhao and others added 8 commits March 1, 2025 21:39

Apply suggestions from code review

2c28989

Co-authored-by: Roger Wang <[email protected]>

address comments

4e978c2

Signed-off-by: Jennifer Zhao <[email protected]>

remove merged pr

32887cd

Signed-off-by: Jennifer Zhao <[email protected]>

fix

fcffa3c

Signed-off-by: Jennifer Zhao <[email protected]>

link pr #13361

047f46d

Signed-off-by: Jennifer Zhao <[email protected]>

link pr #13997

d858c6a

Signed-off-by: Jennifer Zhao <[email protected]>

update

54d51cf

Signed-off-by: Roger Wang <[email protected]>

pre-commit

b934056

Signed-off-by: Roger Wang <[email protected]>

JenZhao marked this pull request as ready for review March 2, 2025 20:50

JenZhao added 3 commits March 4, 2025 20:47

update

79549c9

Signed-off-by: Jennifer Zhao <[email protected]>

update

aa7bed4

Signed-off-by: Jennifer Zhao <[email protected]>

update

f80562b

Signed-off-by: Jennifer Zhao <[email protected]>

LiuXiaoxuanPKU reviewed Mar 5, 2025

View reviewed changes

simon-mo added this to the v0.8.0 milestone Mar 12, 2025

JenZhao added 5 commits March 12, 2025 18:41

Merge branch 'vllm-project:main' into v1_guide

781a224

[Doc] Update v1 guide (#4)

6021f71

Signed-off-by: Jennifer Zhao <[email protected]>

update

b4691c7

Signed-off-by: Jennifer Zhao <[email protected]>

remove notes

7c48862

Signed-off-by: Jennifer Zhao <[email protected]>

fix

e4c5e81

Signed-off-by: Jennifer Zhao <[email protected]>

DarkLight1337 reviewed Mar 13, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

DarkLight1337 reviewed Mar 13, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

Apply suggestions from code review

4090ac9

Co-authored-by: Cyrus Leung <[email protected]>

simon-mo reviewed Mar 13, 2025

View reviewed changes

docs/source/getting_started/v1_user_guide.md Show resolved Hide resolved

docs/source/getting_started/v1_user_guide.md Show resolved Hide resolved

docs/source/getting_started/v1_user_guide.md Outdated Show resolved Hide resolved

JenZhao added 6 commits March 13, 2025 18:51

update structured output

b1dad35

Signed-off-by: Jennifer Zhao <[email protected]>

sort table, add benchmark placeholder

c5cc253

Signed-off-by: Jennifer Zhao <[email protected]>

minor

5b1374d

Signed-off-by: Jennifer Zhao <[email protected]>

add hardware support

35b7af7

Signed-off-by: Jennifer Zhao <[email protected]>

address comments

16d9ce5

Signed-off-by: Jennifer Zhao <[email protected]>

update

dedc925

Signed-off-by: Jennifer Zhao <[email protected]>

simon-mo approved these changes Mar 15, 2025

View reviewed changes

simon-mo merged commit aaacf17 into vllm-project:main Mar 15, 2025
17 of 18 checks passed

ckhordiasma mentioned this pull request Apr 17, 2025

[do not merge] pr test for nm changes into 2.20 red-hat-data-services/vllm#107

Closed

	will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)).
	will be follow-up work to support other types of spec decode (e.g., see [PR #13933](https://github.com/vllm-project/vllm/pull/13933)). We will prioritize the support for Eagle, MTP compared to draft model based spec decode.

Uh oh!

[Doc] V1 user guide #13991

[Doc] V1 user guide #13991

Conversation

JenZhao commented Feb 27, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertgshaw2-redhat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ywang96 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WoosukKwon commented Mar 4, 2025

Uh oh!

LiuXiaoxuanPKU left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LiuXiaoxuanPKU Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

simon-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simon-mo commented Mar 13, 2025

Uh oh!

simon-mo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

JenZhao commented Feb 27, 2025 •

edited by github-actions bot

Loading