[Bugfix] Fix error with penalties when speculative decoding and structural output are enabled #26586
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…put are enabled Signed-off-by: southfreebird <[email protected]>
southfreebird
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request addresses a critical bug that causes a RuntimeError when speculative decoding and structured output are used together with logit processors. The root cause is that stale speculative token data could persist in InputBatch if the scheduler drops all draft tokens for a request, leading to out-of-bounds errors in subsequent penalty calculations. The fix correctly ensures that InputBatch.spec_token_ids is always updated, even with an empty list of tokens, thus preventing state corruption. The change is logical, well-commented, and effectively resolves the issue. The implementation looks correct.
benchislett
reviewed
Oct 10, 2025
vllm/v1/worker/gpu_model_runner.py
Outdated
| # meet the structural schema. This means that | ||
| # scheduler_output.scheduled_spec_decode_tokens might be empty, | ||
| # even when speculative decoding is enabled. So, we moved this line | ||
| # from the 'if' block above. |
Collaborator
There was a problem hiding this comment.
Please rephrase the comment so that it explains the state of the code and not the change to the code. Comments about moved lines can become less meaningful over time with refactoring
benchislett
reviewed
Oct 10, 2025
Signed-off-by: southfreebird <[email protected]>
benchislett
reviewed
Oct 16, 2025
njhill
approved these changes
Oct 18, 2025
Member
|
@southfreebird could you rebase on latest main? |
…dec-and-structural-output
Pradyun92
pushed a commit
to Pradyun92/vllm
that referenced
this pull request
Oct 20, 2025
2 critical fixes that cannot be implemented as plugins: 1. Qwen3 tool parser fix (line 523): - Fixes missing opening brace in streaming tool calls - One-line fix: removed buggy condition - Upstreamable: Yes 2. Eagle rejection sampler fix (gpu_model_runner.py): - Cherry-picked from PR vllm-project#26586 (pending upstream merge) - Fixes RuntimeError with Eagle + penalties - Moved spec_token_ids assignment outside if block Plus minor fixes: - DeepSeek R1 reasoning parser import - Config __init__.py ordering See: IN_TREE_MODIFICATIONS.md for details Signed-off-by: Pradyun Ramadorai <[email protected]>
Pradyun92
pushed a commit
to Pradyun92/vllm
that referenced
this pull request
Oct 20, 2025
Merged 8 commits from origin/main including: - PR vllm-project#26586: Eagle rejection sampler fix (previously cherry-picked) - LoRA CUDA graph specialization (vllm-project#25914) - Bee-8B VLM model support (vllm-project#27012) - Utilities reorganization (network_utils, async_utils, etc.) - Multiple bug fixes and improvements In-Tree Modifications: - Removed Eagle rejection sampler cherry-pick (now in upstream) - Kept Qwen3 tool parser fix (still needed, line 523) - Only 1 active in-tree modification remaining Plugin Compatibility: - All 10 plugin patches load successfully - No target class changes required - Clean merge with no conflicts Documentation Updates: - Updated IN_TREE_MODIFICATIONS.md (moved Eagle fix to Removed/Obsolete) - Updated CLAUDE.md merge history - Verified clean diff with origin/main (3 files, all documented) Signed-off-by: Pradyun Ramadorai <[email protected]>
lywa1998
pushed a commit
to lywa1998/vllm
that referenced
this pull request
Oct 20, 2025
…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <[email protected]>
adabeyta
pushed a commit
to adabeyta/vllm
that referenced
this pull request
Oct 20, 2025
…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <[email protected]>
albertoperdomo2
pushed a commit
to albertoperdomo2/vllm
that referenced
this pull request
Oct 23, 2025
…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <[email protected]> Signed-off-by: Alberto Perdomo <[email protected]>
0xrushi
pushed a commit
to 0xrushi/vllm
that referenced
this pull request
Oct 26, 2025
…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <[email protected]> Signed-off-by: 0xrushi <[email protected]>
0xrushi
pushed a commit
to 0xrushi/vllm
that referenced
this pull request
Oct 26, 2025
…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <[email protected]> Signed-off-by: 0xrushi <[email protected]>
ilmarkov
pushed a commit
to neuralmagic/vllm
that referenced
this pull request
Nov 7, 2025
…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <[email protected]>
rtourgeman
pushed a commit
to rtourgeman/vllm
that referenced
this pull request
Nov 10, 2025
…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <[email protected]>
Zhathw
pushed a commit
to Zhathw/vllm
that referenced
this pull request
Nov 12, 2025
…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <[email protected]>
devpatelio
pushed a commit
to SumanthRH/vllm
that referenced
this pull request
Nov 29, 2025
…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix an error that appears after #19482 when logit processors (such as penalties) are enabled together with speculative decoding and structural output. The example of the error:
Purpose
Test Plan
Test Result