Skip to content

GPU Model Runner V2 #25266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
WoosukKwon merged 173 commits into from
Nov 21, 2025
Merged

GPU Model Runner V2 #25266

WoosukKwon merged 173 commits into from
Nov 21, 2025
+2,639 −12

Conversation

@WoosukKwon
Copy link
Collaborator

@WoosukKwon WoosukKwon commented Sep 19, 2025 •
edited by github-actions bot
Loading

Key Changes

  • Remove persistent batch
    • No “reordering” & complex bookkeeping
    • Almost all CPU states are Numpy arrays → We can vectorize most of the Python loops in pre-/post-processing
    • Simpler handling for requests resumed from preemption
  • GPU-persistent block tables
    • The CPU does not have the block tables at all. GPU maintains the persistent block tables.
    • In every step, we only send the “diff”s to the GPU, and use a Triton kernel to update the persistent block tables
    • We also use another Triton kernel to create new ephemeral block tables used for this forward pass.
    • More scalable as max_model_len and num_kv_groups increase
  • Triton-native sampler
    • No -1 temperature hack for greedy sampling
    • Efficient support for per-request seeds
    • Efficient support for logprobs by only materializing the top-k logprobs instead of the whole vocab
    • Memory-efficient implementation of prompt logprobs
  • Simple implementation of DP
  • Simple CUDA graphs
  • Efficient support for structured outputs

@WoosukKwon
wip
33a3a26 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
Merge branch 'main' into woosuk/input-prep
699bd79
@WoosukKwon
merge
c472982 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
wip
79e5eb3 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
rename
64c8cce 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
merge
48bca9a 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
wip
a1e3745 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
Merge branch 'main' into woosuk/input-prep
da9cd26
@WoosukKwon
fix
7b4b72e 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
merge
65f9369 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
fix
b1d5273 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
simplify
a851aaa 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
merge
e570b0a 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
Merge branch 'main' into woosuk/input-prep
d6d719f
@WoosukKwon
Merge branch 'main' into woosuk/input-prep
b21393c
@WoosukKwon
minor
efba25e 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
fix
e451045 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
minor
19c0dfc 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
minor
4055781 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
fix
9ee9d0e 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
merge
efcb786 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
minor
e696f78 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
optimize spec
c11d1e6 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
work
22771e5 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
MAX_SPEC_LEN
ba1a58f 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
fix
62d23b3 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
fix
af7b6c5 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
fix
01bf16e 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
top_p top_k
cc340e2 
Signed-off-by: Woosuk Kwon <[email protected]>
@WoosukKwon
merge
4c2a337 
Signed-off-by: Woosuk Kwon <[email protected]>
@mergify mergify bot removed the needs-rebase label Nov 20, 2025
@WoosukKwon WoosukKwon added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 20, 2025
@LucasWilkinson LucasWilkinson mentioned this pull request Nov 21, 2025
Open
@WoosukKwon WoosukKwon merged commit 30b44a1 into main Nov 21, 2025
54 of 58 checks passed
@github-project-automation github-project-automation bot moved this to Done in NVIDIA Nov 21, 2025
@WoosukKwon WoosukKwon deleted the woosuk/model-runner-v2 branch November 21, 2025 16:20
njhill
njhill reviewed Nov 21, 2025
View reviewed changes
Copy link
Member

@njhill njhill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved :)

@DarkLight1337
Copy link
Member

DarkLight1337 commented Nov 21, 2025

Looks like this is failing pre-commit on main

@WoosukKwon
Copy link
Collaborator Author

WoosukKwon commented Nov 21, 2025

yeah let me fix the error

The PR passed the pre-commit test in CI somehow.

WoosukKwon added a commit that referenced this pull request Nov 21, 2025
@WoosukKwon
[Chore] Fix pre-commit error after #25266 (#29190)
1bed891
ywang96 pushed a commit to ywang96/vllm that referenced this pull request Nov 23, 2025
@WoosukKwon @ywang96
GPU Model Runner V2 (vllm-project#25266)
43f7b31 
Signed-off-by: Woosuk Kwon <[email protected]>
ywang96 pushed a commit to ywang96/vllm that referenced this pull request Nov 23, 2025
@WoosukKwon @ywang96
[Chore] Fix pre-commit error after vllm-project#25266 (vllm-project#2…
2cf6dc4 
…9190)
@Selkh
Copy link

Selkh commented Nov 24, 2025

Will V2 support async spec decoding in another way or pick the impl in V1?

lpapavassiliou pushed a commit to lpapavassiliou/vllm that referenced this pull request Nov 24, 2025
@WoosukKwon @lpapavassiliou
GPU Model Runner V2 (vllm-project#25266)
34b3cfe 
Signed-off-by: Woosuk Kwon <[email protected]>
lpapavassiliou pushed a commit to lpapavassiliou/vllm that referenced this pull request Nov 24, 2025
@WoosukKwon @lpapavassiliou
[Chore] Fix pre-commit error after vllm-project#25266 (vllm-project#2…
3084748 
…9190)
RunkaiTao pushed a commit to RunkaiTao/vllm that referenced this pull request Nov 24, 2025
@WoosukKwon @RunkaiTao
GPU Model Runner V2 (vllm-project#25266)
d4bf1d9 
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Runkai Tao <[email protected]>
RunkaiTao pushed a commit to RunkaiTao/vllm that referenced this pull request Nov 24, 2025
@WoosukKwon @RunkaiTao
[Chore] Fix pre-commit error after vllm-project#25266 (vllm-project#2…
0cf7a78 
…9190)

Signed-off-by: Runkai Tao <[email protected]>
bringlein pushed a commit to bringlein/vllm that referenced this pull request Nov 26, 2025
@WoosukKwon @bringlein
GPU Model Runner V2 (vllm-project#25266)
efb8444 
Signed-off-by: Woosuk Kwon <[email protected]>
bringlein pushed a commit to bringlein/vllm that referenced this pull request Nov 26, 2025
@WoosukKwon @bringlein
[Chore] Fix pre-commit error after vllm-project#25266 (vllm-project#2…
5e91584 
…9190)
devpatelio pushed a commit to SumanthRH/vllm that referenced this pull request Nov 29, 2025
@WoosukKwon @devpatelio
GPU Model Runner V2 (vllm-project#25266)
508eb1e 
Signed-off-by: Woosuk Kwon <[email protected]>
devpatelio pushed a commit to SumanthRH/vllm that referenced this pull request Nov 29, 2025
@WoosukKwon @devpatelio
[Chore] Fix pre-commit error after vllm-project#25266 (vllm-project#2…
76e4d62 
…9190)
kitaekatt pushed a commit to kitaekatt/vllm that referenced this pull request Dec 1, 2025
@WoosukKwon @kitaekatt
GPU Model Runner V2 (vllm-project#25266)
a671ce7 
Signed-off-by: Woosuk Kwon <[email protected]>
kitaekatt pushed a commit to kitaekatt/vllm that referenced this pull request Dec 1, 2025
@WoosukKwon @kitaekatt
[Chore] Fix pre-commit error after vllm-project#25266 (vllm-project#2…
fccff94 
…9190)
charlotte12l pushed a commit to charlotte12l/vllm that referenced this pull request Dec 5, 2025
@WoosukKwon @charlotte12l
GPU Model Runner V2 (vllm-project#25266)
0c35a22 
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Xingyu Liu <[email protected]>
charlotte12l pushed a commit to charlotte12l/vllm that referenced this pull request Dec 5, 2025
@WoosukKwon @charlotte12l
[Chore] Fix pre-commit error after vllm-project#25266 (vllm-project#2…
2d795b7 
…9190)

Signed-off-by: Xingyu Liu <[email protected]>
Zhathw pushed a commit to Zhathw/vllm that referenced this pull request Dec 6, 2025
@WoosukKwon @Zhathw
GPU Model Runner V2 (vllm-project#25266)
4b2c7c2 
Signed-off-by: Woosuk Kwon <[email protected]>
Zhathw pushed a commit to Zhathw/vllm that referenced this pull request Dec 6, 2025
@WoosukKwon @Zhathw
[Chore] Fix pre-commit error after vllm-project#25266 (vllm-project#2…
4c33c8e 
…9190)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LucasWilkinson LucasWilkinson LucasWilkinson left review comments

@NickLucche NickLucche NickLucche left review comments

@ProExpertProg ProExpertProg ProExpertProg left review comments

@njhill njhill njhill left review comments

@simon-mo simon-mo simon-mo left review comments

@heheda12345 heheda12345 heheda12345 left review comments

@houseroad houseroad houseroad left review comments

@jikunshang jikunshang jikunshang left review comments

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@comaniac comaniac Awaiting requested review from comaniac

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

@ApostaC ApostaC Awaiting requested review from ApostaC ApostaC is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@pavanimajety pavanimajety Awaiting requested review from pavanimajety pavanimajety is a code owner

+2 more reviewers

@yinghai yinghai yinghai left review comments

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build documentation Improvements or additions to documentation nvidia ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

NVIDIA
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[RFC]: Redesigning Persistent Batch in vLLM