[PD] Heterogenous TP + #7 #14

NickLucche · 2025-05-15T09:55:41Z

No description provided.

…me in PR vllm-project#17483 (vllm-project#17961) Signed-off-by: tjtanaa <[email protected]>

…7861) Signed-off-by: Isotr0py <[email protected]>

…roject#17964) Signed-off-by: youkaichao <[email protected]>

… engine when xgrammar validation fails (vllm-project#17623) Signed-off-by: Jason Cheng <[email protected]> Co-authored-by: Russell Bryant <[email protected]>

Signed-off-by: reidliu41 <[email protected]> Co-authored-by: reidliu41 <[email protected]>

Signed-off-by: wangli <[email protected]>

…ject#17971) Signed-off-by: Siyuan Liu <[email protected]>

…m-project#17976) Signed-off-by: Brayden Zhong <[email protected]>

Signed-off-by: 许文卿 <[email protected]> Signed-off-by: Xu Wenqing <[email protected]>

Signed-off-by: Aaruni Aggarwal <[email protected]>

Signed-off-by: Jee Jee Li <[email protected]>

Signed-off-by: Max de Bayser <[email protected]> Co-authored-by: igmainc <[email protected]>

Signed-off-by: ApostaC <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]> Signed-off-by: [email protected] <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Brent Salisbury <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: ApostaC <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: mgoin <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Brent Salisbury <[email protected]>

…lm-project#16855) Signed-off-by: jberkhahn <[email protected]>

…ON blobs without CLI (vllm-project#17994) Signed-off-by: Harry Mellor <[email protected]>

…roject#17968)

…project#17245) Signed-off-by: Jade Zheng <[email protected]>

…ect#18020)

…s to counting tokens (vllm-project#18003) Signed-off-by: Chen Zhang <[email protected]>

Signed-off-by: [email protected] <[email protected]>

Signed-off-by: Yang Wang <[email protected]>

Signed-off-by: Russell Bryant <[email protected]>

Signed-off-by: wwl2755 <[email protected]>

Signed-off-by: mgoin <[email protected]>

Signed-off-by: wp-alpha <[email protected]> Co-authored-by: wangpeng66 <[email protected]>

Signed-off-by: Harry Mellor <[email protected]>

Signed-off-by: mgoin <[email protected]> Signed-off-by: Dipika <[email protected]> Co-authored-by: Dipika <[email protected]>

Signed-off-by: Harry Mellor <[email protected]>

…ect#17905) Signed-off-by: Hissu Hyvarinen <[email protected]>

…h sparse attention support (vllm-project#11844)

Signed-off-by: Jee Jee Li <[email protected]>

…roject#13779) Signed-off-by: Arjun Kathuria <[email protected]>

Signed-off-by: mgoin <[email protected]>

…llm-project#17935) Signed-off-by: DarkLight1337 <[email protected]>

…the `StructuredOutputBackend`. (vllm-project#17845) Signed-off-by: chaunceyjiang <[email protected]>

Signed-off-by: calvin chen <[email protected]>

Signed-off-by: Nick Hill <[email protected]>

Signed-off-by: drisspg <[email protected]>

…llm-project#17999) Signed-off-by: Chen Zhang <[email protected]> Co-authored-by: Woosuk Kwon <[email protected]>

… PP (vllm-project#17916) Signed-off-by: Jin Huang <[email protected]> Co-authored-by: Jin Huang <[email protected]>

fix descr indexing change remote worker selection indexing; test ptp2-dtp4 Signed-off-by: nicklucche <[email protected]>

Signed-off-by: nicklucche <[email protected]>

Signed-off-by: Nick Hill <[email protected]> Signed-off-by: nicklucche <[email protected]>

Signed-off-by: nicklucche <[email protected]>

github-actions · 2025-05-15T09:55:51Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

tjtanaa and others added 30 commits May 11, 2025 09:14

[BUG] [ROCm] [MLA] Fix variable name bug due to change in variable na…

7de18d5

…me in PR vllm-project#17483 (vllm-project#17961) Signed-off-by: tjtanaa <[email protected]>

[Model] Broadcast Ovis2 implementation to fit Ovis1.6 (vllm-project#1…

021c16c

…7861) Signed-off-by: Isotr0py <[email protected]>

[misc] add instructions on how to install nvshmem/pplx/deepep (vllm-p…

d45fe33

…roject#17964) Signed-off-by: youkaichao <[email protected]>

[Bugfix] validate grammar and throw 400 error instead of crashing the…

08bf784

… engine when xgrammar validation fails (vllm-project#17623) Signed-off-by: Jason Cheng <[email protected]> Co-authored-by: Russell Bryant <[email protected]>

[bugfix] fix the wrong parser (vllm-project#17958)

ada50aa

Signed-off-by: reidliu41 <[email protected]> Co-authored-by: reidliu41 <[email protected]>

[Bugfix] Fix pydantic.errors.PydanticUserError (vllm-project#17962)

19a3c78

Signed-off-by: wangli <[email protected]>

[Bugfix][TPU] Use np array when updating cache slot_mapping (vllm-pro…

4307830

…ject#17971) Signed-off-by: Siyuan Liu <[email protected]>

[Fix] Benchmark "EngineClient" has no attribute "model_config" (vll…

891b9d3

…m-project#17976) Signed-off-by: Brayden Zhong <[email protected]>

[Feature] Support DeepSeekV3 Function Call (vllm-project#17784)

3a5ea75

Signed-off-by: 许文卿 <[email protected]> Signed-off-by: Xu Wenqing <[email protected]>

Correcting testcases in builkite job for IBM Power (vllm-project#17675)

9fbf2bf

Signed-off-by: Aaruni Aggarwal <[email protected]>

[Misc] Improve modelscope import error (vllm-project#17983)

7ea6cb2

Signed-off-by: Jee Jee Li <[email protected]>

Initialize the delta tool call fields explicitly (vllm-project#17340)

05a4324

Signed-off-by: Max de Bayser <[email protected]> Co-authored-by: igmainc <[email protected]>

[Lora][Frontend]Add default local directory LoRA resolver plugin. (vl…

98ea356

…lm-project#16855) Signed-off-by: jberkhahn <[email protected]>

Construct KVTransferConfig properly from Python instead of using JS…

72a3f6b

…ON blobs without CLI (vllm-project#17994) Signed-off-by: Harry Mellor <[email protected]>

[CI/Build] Fix TPU V1 Test mixed use of & and && across tests (vllm-p…

b9fd0d7

…roject#17968)

[Core] Use platform-agnostic device control for DP engine core (vllm-…

289199f

…project#17245) Signed-off-by: Jade Zheng <[email protected]>

Enabling "Weight Loading Multiple GPU Test - Large Models" (vllm-proj…

e9c730c

…ect#18020)

[v1][KVCacheManager] Change prefix caching metric from counting block…

302f3ac

…s to counting tokens (vllm-project#18003) Signed-off-by: Chen Zhang <[email protected]>

[Chore] Remove unused method (vllm-project#18024)

195adb4

Signed-off-by: [email protected] <[email protected]>

Enable standard language model for torhc nightly (vllm-project#18004)

2b0db9b

Signed-off-by: Yang Wang <[email protected]>

[CI] Make JSON output tests less likely to fail (vllm-project#17859)

ebab1ac

Signed-off-by: Russell Bryant <[email protected]>

[V1][Spec Decode] Eagle unit tests (vllm-project#17350)

dc99053

Signed-off-by: wwl2755 <[email protected]>

Fix FBGEMM integration (vllm-project#18002)

f065de4

Signed-off-by: mgoin <[email protected]>

[Model] Support MiMo-7B inference with MTP (vllm-project#17433)

acee8f4

Signed-off-by: wp-alpha <[email protected]> Co-authored-by: wangpeng66 <[email protected]>

Update some more deprecated type hinting (vllm-project#17998)

9d7ea9d

Signed-off-by: Harry Mellor <[email protected]>

Use NVFP4 Marlin for CompressedTensorsW4A16Fp4 (vllm-project#18000)

307939f

Signed-off-by: mgoin <[email protected]> Signed-off-by: Dipika <[email protected]> Co-authored-by: Dipika <[email protected]>

Remove noisy warnings from SchedulerConfig (vllm-project#17995)

d67085c

Signed-off-by: Harry Mellor <[email protected]>

[ROCm] Skip tests for quantizations incompatible with ROCm (vllm-proj…

f6518b2

…ect#17905) Signed-off-by: Hissu Hyvarinen <[email protected]>

Implements dual-chunk-flash-attn backend for dual chunk attention wit…

60f7624

…h sparse attention support (vllm-project#11844)

jeejeelee and others added 21 commits May 12, 2025 20:36

[Misc] Slight spelling modification (vllm-project#18039)

c06af9a

Signed-off-by: Jee Jee Li <[email protected]>

[ROCm]: Fix build from source failure with gcc14 and ROCm 6.3 (vllm-p…

d8487ef

…roject#13779) Signed-off-by: Arjun Kathuria <[email protected]>

[Bugfix] Fixes for new marlin moe usage (vllm-project#18017)

1df491c

Signed-off-by: mgoin <[email protected]>

[Bugfix] Avoid repeatedly creating dummy data during engine startup (v…

61e0a50

…llm-project#17935) Signed-off-by: DarkLight1337 <[email protected]>

[Feature][V1] Support tool_choice: required when using Xgrammar as …

dc1a821

…the `StructuredOutputBackend`. (vllm-project#17845) Signed-off-by: chaunceyjiang <[email protected]>

cleanup invalid prints (vllm-project#18050)

4854572

Signed-off-by: calvin chen <[email protected]>

[BugFix] Fix 4-GPU RLHF tests (vllm-project#18007)

ee5be83

Signed-off-by: Nick Hill <[email protected]>

Fix Broken macro for cutlass moe (vllm-project#18049)

e57e4d6

Signed-off-by: drisspg <[email protected]>

[v1][KVCacheManager] Avoid full cache hit by controlling max_length (v…

f0d610a

…llm-project#17999) Signed-off-by: Chen Zhang <[email protected]> Co-authored-by: Woosuk Kwon <[email protected]>

[Bugfix][V1] Only get input embeddings w/ multi-modal models if first…

8dd0671

… PP (vllm-project#17916) Signed-off-by: Jin Huang <[email protected]> Co-authored-by: Jin Huang <[email protected]>

split kv_cache along head dim

afccc5f

fix descr indexing change remote worker selection indexing; test ptp2-dtp4 Signed-off-by: nicklucche <[email protected]>

rebase cruft

e96f832

Signed-off-by: nicklucche <[email protected]>

address race condition

f0ba728

Signed-off-by: nicklucche <[email protected]>

integration test

d0c44bb

Signed-off-by: nicklucche <[email protected]>

address review: move src_xfer_side_handle init out of hotpath and docs

bde92a5

Signed-off-by: nicklucche <[email protected]>

address review: clarity

f674fc7

Signed-off-by: nicklucche <[email protected]>

typing

230a8b8

Signed-off-by: nicklucche <[email protected]>

revert proxy server

8274635

Signed-off-by: nicklucche <[email protected]>

fix req_finished sync bug

abc710a

Signed-off-by: nicklucche <[email protected]>

Avoid extra collective when aggregating

425b98e

Signed-off-by: Nick Hill <[email protected]> Signed-off-by: nicklucche <[email protected]>

rebase cruft

8cad890

Signed-off-by: nicklucche <[email protected]>

NickLucche closed this May 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[PD] Heterogenous TP + #7 #14

[PD] Heterogenous TP + #7 #14

Uh oh!

NickLucche commented May 15, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

34 participants

[PD] Heterogenous TP + #7 #14

[PD] Heterogenous TP + #7 #14

Uh oh!

Conversation

NickLucche commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

34 participants

NickLucche commented May 15, 2025 •

edited

Loading