Skip to content

[CI] Refactor test_sequence_parallel.py and add a warmup run for more accurate performance stat#1165

Merged
hsliuustc0106 merged 101 commits intovllm-project:mainfrom
mxuax:Non-Intrusive-SP
Feb 3, 2026
Merged

[CI] Refactor test_sequence_parallel.py and add a warmup run for more accurate performance stat#1165
hsliuustc0106 merged 101 commits intovllm-project:mainfrom
mxuax:Non-Intrusive-SP

Conversation

@mxuax
Copy link
Contributor

@mxuax mxuax commented Feb 3, 2026

Purpose

This PR refactors test_sequence_parallel.py, and adds a warmup run for more accurate performance stat to fix the issue6 mentioned in #1143. The redundant tests are also removed to speedup the test.

Issue 6: Test Validity Anomaly (Ring vs. Ulysses)
Evidence: In 4-GPU Sequence Parallel tests:
Ulysses: ~29.0s execution.
Ring/Hybrid: ~0.3s execution.
The reason is the time consumption during NCCL initialization. By adding a warm-up run, all those can be eliminated. 

Test Plan

pytest tests/e2e/offline_inference/test_sequence_parallel.py -v -s

Test Result

Hardware: H800
Time duration 388.15s (0:06:28)

======================================================================
SUMMARY
======================================================================
Mode            GPUs   Size       Baseline     SP           Speedup    Status
----------------------------------------------------------------------
ulysses-2       2      256x256   95ms         110ms        0.86x      PASS
ring-2          2      256x256   95ms         168ms        0.57x      PASS
hybrid-2x2      4      256x256   N/A          314ms        N/A        PASS
ulysses-4       4      272x272   N/A          9168ms        N/A        PASS
======================================================================

NOTE: The performance downgrade is because we use a very small image size(256*256), and the generation bottleneck is communication-bound instead of computational-bound.


Essential Elements of an Effective PR Description Checklist
  • [ - ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • [ - ] The test plan, such as providing test command.
  • [ - ] The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

mxuax and others added 30 commits January 14, 2026 15:06
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
… to support cp_plan

Signed-off-by: mxuax <mxuax@connect.ust.hk>
…paration in a block

Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Removed context parallelism plan and related comments.

Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
…llustration

Signed-off-by: mxuax <mxuax@connect.ust.hk>
…e lengths

- Add sp_attention_mask, sp_padding_size, sp_original_seq_len to ForwardContext
- Add auto_pad option to SequenceParallelInput
- Implement _shard_with_auto_pad in SequenceParallelSplitHook
- Update SequenceParallelGatherHook to remove padding
- Update QwenImage _sp_plan with auto_pad=True
- Update QwenImageCrossAttention to use sp_attention_mask

Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
… code, add some comment

Signed-off-by: mxuax <mxuax@connect.ust.hk>
…struction

Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
mxuax and others added 15 commits January 30, 2026 12:20
Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
…e performance stat

Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: mxuax <mxuax@connect.ust.hk>
@mxuax mxuax requested a review from hsliuustc0106 as a code owner February 3, 2026 02:52
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e483ab08fb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@mxuax
Copy link
Contributor Author

mxuax commented Feb 3, 2026

This PR is ready. @congw729 @wtomin @SamitHuang @ZJY0516 @hsliuustc0106

@congw729
Copy link
Contributor

congw729 commented Feb 3, 2026

The codes are fine. Please rebase to origin/main branch to clean the commits log.

Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
@mxuax mxuax requested a review from ZJY0516 February 3, 2026 03:43
@tzhouam tzhouam added the ready label to trigger buildkite CI label Feb 3, 2026
@hsliuustc0106
Copy link
Collaborator

image this looks much better for each single function @yenuo26 @congw729

@hsliuustc0106 hsliuustc0106 merged commit 9494d69 into vllm-project:main Feb 3, 2026
7 checks passed
@yenuo26
Copy link
Contributor

yenuo26 commented Feb 3, 2026

image this looks much better for each single function @yenuo26 @congw729

Buildkite automatically collapses logs starting with "Running". If we want to achieve this effect, we can add a fixture function in conftest.py to add "Running" information before each test function runs.

futurenitian pushed a commit to futurenitian/vllm-omni that referenced this pull request Feb 4, 2026
… accurate performance stat (vllm-project#1165)

Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: future fu <3172516720@qq.com>
futurenitian pushed a commit to futurenitian/vllm-omni that referenced this pull request Feb 4, 2026
… accurate performance stat (vllm-project#1165)

Signed-off-by: mxuax <mxuax@connect.ust.hk>
Signed-off-by: XU Mingshi <91017482+mxuax@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: future fu <3172516720@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants