SYNC RHOAI 2.24 to midstream v0.10.0.1 release #175

heyselbi · 2025-08-08T06:27:06Z

Includes:

Upstream's v0.10.0 release
Cherry picks:
- Fix CUDA permute/unpermute for use with DeepGemm Moe vllm-project/vllm#17934 --> resolves MOE models on ROCM
- [Bugfix][ROCm] Fix for warp_size uses on host vllm-project/vllm#21205 --> resolves MOE models on ROCM
- heyselbi@297ea4e --> resolves XET issue not present anymore
Remove VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
Remove hf[xet] requirement
Updates to Dockerfile.ubi and Dockerfile.rocm.ubi
- upgrade tgis_adapter to 0.8.0
- nccl version update for [Bug]: CUDA Illegal Access Error when running 2x Blackwell for tensor or pipeline parallel. vllm-project/vllm#19166
- from @dtrifiro 👍
  - Dockerfile.ubi: we're still using flashinfer 0.2.5 (https://storage.googleapis.com/neuralmagic-public-pypi/dist/flashinfer_python-0.2.5-cp38-abi3-linux_x86_64.whl)
  - Dockerfile.rocm.ubi: ARG FA_BRANCH="3cea2fb" (v2.6.3-21-g3cea2fb6), we've been using the same revision that upstream has, which is 1a7f4dfa (v2.7.4.post1-26-g1a7f4dfa)
  - Dockerfile.rocm still has libsodium
  - We could probably remove the VLLM_USE_V1 env vars since V0 has been removed in v0.10.0
  - removed libsodium from rocm contianer
  - bump flashinfer to 0.2.8 (https://storage.googleapis.com/nm-public-pypi/dist/flashinfer_python-0.2.8-cp39-abi3-linux_x86_64.whl)
  - bump ROCm flash-attention to v2.7.4.post1-26-g1a7f4dfa
  - remove ROCm VLLM_USE_V1 flag

Signed-off-by: Yong Hoon Shin <[email protected]>

…Seek TP (vllm-project#20769) Signed-off-by: Alexander Matveev <[email protected]>

…ites from the cache (vllm-project#20942) Signed-off-by: Richard Zou <[email protected]>

… DeepGEMM (vllm-project#20933) Signed-off-by: mgoin <[email protected]>

…ct#20945) Signed-off-by: mgoin <[email protected]>

Signed-off-by: Xiongfei Wei <[email protected]>

Signed-off-by: Ricardo Decal <[email protected]>

…t#20763) Signed-off-by: reidliu41 <[email protected]>

Signed-off-by: NickLucche <[email protected]>

Signed-off-by: Ruheena Suhani Shaik <[email protected]>

vllm-project#20934) Signed-off-by: Pavani Majety <[email protected]>

…oject#20926) Signed-off-by: Isotr0py <[email protected]>

…ject#20940) Signed-off-by: Boyuan Feng <[email protected]>

…#20534) Signed-off-by: Jennifer He <[email protected]> Signed-off-by: <[email protected]> Signed-off-by: Jen H <[email protected]>

Signed-off-by: Woosuk Kwon <[email protected]>

…0918) Signed-off-by: ilmarkov <[email protected]> Co-authored-by: ilmarkov <[email protected]>

…ject#20961) Signed-off-by: reidliu41 <[email protected]>

Signed-off-by: Ricardo Decal <[email protected]>

…roject#20206) Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

…llm-project#20602) Signed-off-by: Ricardo Decal <[email protected]>

Signed-off-by: Yifei Teng <[email protected]>

…li (vllm-project#20840) Signed-off-by: Thomas Parnell <[email protected]>

Signed-off-by: wangli <[email protected]>

Signed-off-by: Rui Qiao <[email protected]>

Signed-off-by: Harry Mellor <[email protected]>

Signed-off-by: Christian Pinto <[email protected]>

Signed-off-by: Patrick von Platen <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>

…roject#20994) Signed-off-by: DarkLight1337 <[email protected]>

…m-project#20979) Signed-off-by: Harry Mellor <[email protected]>

Signed-off-by: Harry Mellor <[email protected]>

heyselbi · 2025-08-08T06:32:09Z

/ok-to-test

heyselbi · 2025-08-08T06:33:19Z

/build-rocm

heyselbi · 2025-08-08T06:33:24Z

/build-cuda

ckhordiasma · 2025-08-08T13:01:47Z

/build-cuda

ckhordiasma · 2025-08-08T13:14:24Z

/build-cuda

heyselbi · 2025-08-08T16:42:23Z

/build-cuda

heyselbi · 2025-08-09T10:59:05Z

/build-cuda

ckhordiasma · 2025-08-09T13:44:22Z

/build-cuda

dtrifiro · 2025-08-09T16:00:06Z

@ckhordiasma I pushed v0.10.0.0 tags to this repo and to vllm-rocm. Can you make sure that versioning (tags) is (are) correct when the branches are ready? Thanks!

heyselbi · 2025-08-10T05:49:03Z

/build-cuda

heyselbi · 2025-08-11T09:15:56Z

/build-cuda

heyselbi · 2025-08-11T10:01:05Z

/build-cuda

heyselbi · 2025-08-11T10:17:11Z

/build-cuda

heyselbi · 2025-08-11T15:24:34Z

/build-cuda

heyselbi · 2025-08-11T15:33:53Z

/build-cuda

heyselbi · 2025-08-11T19:32:51Z

/build-cuda

heyselbi · 2025-08-15T19:14:32Z

/build-cuda

heyselbi · 2025-08-18T17:51:57Z

Validation is passed on Midstream side. PSAP team is currently running some final benchmarks. As no apparent blockers are expected, merging it for now to make it on time to release tomorrow.

sarckk and others added 30 commits July 14, 2025 23:11

Fall back if flashinfer comm module not found (vllm-project#20936)

61e2082

Signed-off-by: Yong Hoon Shin <[email protected]>

SM100 Cutlass MLA decode with unrestricted num_heads (< 128) for Deep…

8cdc371

…Seek TP (vllm-project#20769) Signed-off-by: Alexander Matveev <[email protected]>

[BugFix] VLLM_DISABLE_COMPILE_CACHE=1 should disable all reads and wr…

ba8c300

…ites from the cache (vllm-project#20942) Signed-off-by: Richard Zou <[email protected]>

[Bugfix] Fix incorrect dispatch for CutlassBlockScaledGroupedGemm and…

bcdfb2a

… DeepGEMM (vllm-project#20933) Signed-off-by: mgoin <[email protected]>

[CI/Build] Split Entrypoints Test into LLM and API Server (vllm-proje…

946aadb

…ct#20945) Signed-off-by: mgoin <[email protected]>

Use w8a8 quantized matmul Pallas kernel (vllm-project#19170)

d4170fa

Signed-off-by: Xiongfei Wei <[email protected]>

[Docs] Add Kuberay to deployment integrations (vllm-project#20592)

054c865

Signed-off-by: Ricardo Decal <[email protected]>

feat: add image zoom to improve image viewing experience (vllm-projec…

37e2eca

…t#20763) Signed-off-by: reidliu41 <[email protected]>

[CI] Fix flaky test_streaming_response test (vllm-project#20913)

80305c1

Signed-off-by: NickLucche <[email protected]>

Enabled BnB NF4 inference on Gaudi (vllm-project#20172)

016b8d1

Signed-off-by: Ruheena Suhani Shaik <[email protected]>

[Bugfix] Switch bailout logic for kv-cache-dtype with SM100 Flashinfer (

9ad0a45

vllm-project#20934) Signed-off-by: Pavani Majety <[email protected]>

[Doc] Clearer mistral3 and pixtral model support description (vllm-pr…

fc01791

…oject#20926) Signed-off-by: Isotr0py <[email protected]>

[cold start] replace VLLM_COMPILE_DEPYF with debug_dump_dir (vllm-pro…

91b3d19

…ject#20940) Signed-off-by: Boyuan Feng <[email protected]>

[Model] Add AutoWeightsLoader support for BERT, RoBERTa (vllm-project…

85bd659

…#20534) Signed-off-by: Jennifer He <[email protected]> Signed-off-by: <[email protected]> Signed-off-by: Jen H <[email protected]>

Implement Async Scheduling (vllm-project#19970)

d4d3094

Signed-off-by: Woosuk Kwon <[email protected]>

[Misc] Refactor AllReduceFusionPass. Remove parameter (vllm-project#2…

37a7d5d

…0918) Signed-off-by: ilmarkov <[email protected]> Co-authored-by: ilmarkov <[email protected]>

[frontend] Add --help=page option for paginated help output (vllm-pro…

68d28e3

…ject#20961) Signed-off-by: reidliu41 <[email protected]>

[Docs] Improve documentation for RLHF example (vllm-project#20598)

235bfd5

Signed-off-by: Ricardo Decal <[email protected]>

[frontend] Refactor CLI Args for a better modular integration (vllm-p…

f148c44

…roject#20206) Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

[Docs] Improve documentation for ray cluster launcher helper script (v…

33d5600

…llm-project#20602) Signed-off-by: Ricardo Decal <[email protected]>

[TPU] Optimize kv cache update kernel (vllm-project#20415)

c586b55

Signed-off-by: Yifei Teng <[email protected]>

[V1] [Hybrid] Refactor mamba state shape calculation; enable V1 via c…

3534c39

…li (vllm-project#20840) Signed-off-by: Thomas Parnell <[email protected]>

[MISC] Add init files for python package (vllm-project#20908)

20149d8

Signed-off-by: wangli <[email protected]>

[doc] Add more details for Ray-based DP (vllm-project#20948)

d912781

Signed-off-by: Rui Qiao <[email protected]>

[Deprecation] Remove TokenizerPoolConfig (vllm-project#20968)

56fe4be

Signed-off-by: Harry Mellor <[email protected]>

[v1][core] Support for attention free models (vllm-project#20811)

4ffd963

Signed-off-by: Christian Pinto <[email protected]>

Voxtral (vllm-project#20970)

e7e3e6d

Signed-off-by: Patrick von Platen <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>

[CI/Build] Fix wrong path in Transformers Nightly Models Test (vllm-p…

c847e34

…roject#20994) Signed-off-by: DarkLight1337 <[email protected]>

[Deprecation] Remove everything scheduled for removal in v0.10.0 (vll…

313ae8c

…m-project#20979) Signed-off-by: Harry Mellor <[email protected]>

Configure Gemini (vllm-project#20971)

5bac613

Signed-off-by: Harry Mellor <[email protected]>

heyselbi added 2 commits August 8, 2025 11:04

Update tgis adapter to 0.8.0

01e9d83

remove VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

d3548b7

heyselbi mentioned this pull request Aug 8, 2025

Sync v0.10.0 rhoai rocm red-hat-data-services/vllm-rocm#52

Merged

update vllm version

7053645

dtrifiro approved these changes Aug 11, 2025

View reviewed changes

[TEMP] Make openai entrypoint default and add template files

9d8b4d6

Correct chown

f777d8d

update template to examples directory

73b8071

install blobfile for Kimi model support

efe58e0

fix the typo

32bb10d

add nccl upgrade to tgis adapter

6acccf0

heyselbi merged commit 9f294e4 into red-hat-data-services:main Aug 18, 2025
4 checks passed

SYNC RHOAI 2.24 to midstream v0.10.0.1 release #175

SYNC RHOAI 2.24 to midstream v0.10.0.1 release #175

Uh oh!

Conversation

heyselbi commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

heyselbi commented Aug 8, 2025

Uh oh!

heyselbi commented Aug 8, 2025

Uh oh!

heyselbi commented Aug 8, 2025

Uh oh!

ckhordiasma commented Aug 8, 2025

Uh oh!

ckhordiasma commented Aug 8, 2025

Uh oh!

heyselbi commented Aug 8, 2025

Uh oh!

heyselbi commented Aug 9, 2025

Uh oh!

ckhordiasma commented Aug 9, 2025

Uh oh!

dtrifiro commented Aug 9, 2025

Uh oh!

heyselbi commented Aug 10, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 11, 2025

Uh oh!

heyselbi commented Aug 15, 2025

Uh oh!

heyselbi commented Aug 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

heyselbi commented Aug 8, 2025 •

edited

Loading