Skip to content

Conversation

@FightingZhen
Copy link
Collaborator

@FightingZhen FightingZhen commented Sep 19, 2025

What does this PR do?

This PR is committed for solving 3 things:

  1. All test cases in e2e_ascend CI pipeline use 8 NPUs by default, which prevents the machine's original performance from being fully utilized. This PR is committed for solving this problem. Thank @zheliuyu for finding this problem :)
  2. Remove qwen3 grpo test case in e2e_ascend.yml because it is similar to qwen2.5 grpo.
  3. Remove ppo test case in e2e_ascend.yml because it is not work since first commit [model, ci] feat: add qwen3-8b ppo script on ASCEND NPU #3502 , @xvxuopop is working for solving this.
  4. Update e2e_ascend CI scan path for covering most of file modification case.
  5. Ignore loading libnuma.so for Ascend NPU.
  6. Fix qwen2_vl flash-attention related functions unavailable error for Ascend NPU.

Checklist Before Starting

  • Search for similar PRs. Paste at least one query link here: ...
  • Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
    • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
    • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
    • {type} is in feat, fix, refactor, chore, test
    • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
    • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

Not related.

API and Usage Example

Not related.

Design & Code Changes

Not related.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

@FightingZhen FightingZhen changed the title Update ngpus_per_node to 16 for all e2e_ascend CI test cases [ci] chore: Update ngpus_per_node to 16 for all e2e_ascend CI test cases Sep 19, 2025
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the number of GPUs/NPUs from 8 to 16 across several test scripts to better utilize hardware resources. The changes are correct, but I've identified an opportunity to improve maintainability and consistency. In several scripts, the number of GPUs is hardcoded. I've suggested using an environment variable with a default value, which makes the scripts more flexible and aligns with the pattern already present in one of the modified files.

@FightingZhen FightingZhen changed the title [ci] chore: Update ngpus_per_node to 16 for all e2e_ascend CI test cases [CI] chore: Update ngpus_per_node to 16 for all e2e_ascend CI test cases Sep 19, 2025
@FightingZhen FightingZhen force-pushed the ascend_ci_update branch 2 times, most recently from c22f154 to 3dcfd68 Compare September 20, 2025 09:01
@FightingZhen FightingZhen changed the title [CI] chore: Update ngpus_per_node to 16 for all e2e_ascend CI test cases [WIP][CI] chore: Update ngpus_per_node to 16 for all e2e_ascend CI test cases Sep 20, 2025
@FightingZhen FightingZhen changed the title [WIP][CI] chore: Update ngpus_per_node to 16 for all e2e_ascend CI test cases [CI] chore: Update ngpus_per_node to 16 for all e2e_ascend CI test cases Sep 22, 2025
@FightingZhen FightingZhen force-pushed the ascend_ci_update branch 3 times, most recently from a9e232e to f9c6ffb Compare September 23, 2025 02:06
@FightingZhen FightingZhen changed the title [CI] chore: Update ngpus_per_node to 16 for all e2e_ascend CI test cases [CI] chore: Update e2e_ascend CI config Sep 23, 2025
@vermouth1992 vermouth1992 merged commit 5547dbe into volcengine:main Sep 23, 2025
81 of 84 checks passed
masoudhashemi pushed a commit to masoudhashemi/verl that referenced this pull request Oct 19, 2025
### What does this PR do?
This PR is committed for solving 3 things:
1. All test cases in `e2e_ascend` CI pipeline use 8 NPUs by default,
which prevents the machine's original performance from being fully
utilized. This PR is committed for solving this problem. Thank @zheliuyu
for finding this problem :)
2. Remove qwen3 grpo test case in `e2e_ascend.yml` because it is similar
to qwen2.5 grpo.
3. Remove ppo test case in `e2e_ascend.yml` because it is not work since
first commit volcengine#3502 , @xvxuopop is working for solving this.
4. Update e2e_ascend CI scan path for covering most of file modification
case.
5. Ignore loading `libnuma.so` for Ascend NPU.
6. Fix qwen2_vl flash-attention related functions unavailable error for
Ascend NPU.

### Checklist Before Starting

- [x] Search for similar PRs. Paste at least one query link here: ...
- [x] Format the PR title as `[{modules}] {type}: {description}` (This
will be checked by the CI)
- `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`,
`trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`,
`ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`,
`env`, `tool`, `ckpt`, `doc`, `data`
- If this PR involves multiple modules, separate them with `,` like
`[megatron, fsdp, doc]`
  - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test`
- If this PR breaks any API (CLI arguments, config, function signature,
etc.), add `[BREAKING]` to the beginning of the title.
  - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching`

### Test

Not related.

### API and Usage Example

Not related.

### Design & Code Changes

Not related.

### Checklist Before Submitting

> [!IMPORTANT]
> Please check all the following items before requesting a review,
otherwise the reviewer might deprioritize this PR for review.

- [ ] Read the [Contribute
Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md).
- [ ] Apply [pre-commit
checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting):
`pre-commit install && pre-commit run --all-files --show-diff-on-failure
--color=always`
- [ ] Add / Update [the
documentation](https://github.com/volcengine/verl/tree/main/docs).
- [ ] Add unit or end-to-end test(s) to [the CI
workflow](https://github.com/volcengine/verl/tree/main/.github/workflows)
to cover all the code. If not feasible, explain why: ...
- [ ] Once your PR is ready for CI, send a message in [the `ci-request`
channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the
`verl` Slack
workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).
(If not accessible, please try [the Feishu group
(飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)
techkang pushed a commit to techkang/verl that referenced this pull request Oct 31, 2025
### What does this PR do?
This PR is committed for solving 3 things:
1. All test cases in `e2e_ascend` CI pipeline use 8 NPUs by default,
which prevents the machine's original performance from being fully
utilized. This PR is committed for solving this problem. Thank @zheliuyu
for finding this problem :)
2. Remove qwen3 grpo test case in `e2e_ascend.yml` because it is similar
to qwen2.5 grpo.
3. Remove ppo test case in `e2e_ascend.yml` because it is not work since
first commit volcengine#3502 , @xvxuopop is working for solving this.
4. Update e2e_ascend CI scan path for covering most of file modification
case.
5. Ignore loading `libnuma.so` for Ascend NPU.
6. Fix qwen2_vl flash-attention related functions unavailable error for
Ascend NPU.

### Checklist Before Starting

- [x] Search for similar PRs. Paste at least one query link here: ...
- [x] Format the PR title as `[{modules}] {type}: {description}` (This
will be checked by the CI)
- `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`,
`trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`,
`ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`,
`env`, `tool`, `ckpt`, `doc`, `data`
- If this PR involves multiple modules, separate them with `,` like
`[megatron, fsdp, doc]`
  - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test`
- If this PR breaks any API (CLI arguments, config, function signature,
etc.), add `[BREAKING]` to the beginning of the title.
  - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching`

### Test

Not related.

### API and Usage Example

Not related.

### Design & Code Changes

Not related.

### Checklist Before Submitting

> [!IMPORTANT]
> Please check all the following items before requesting a review,
otherwise the reviewer might deprioritize this PR for review.

- [ ] Read the [Contribute
Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md).
- [ ] Apply [pre-commit
checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting):
`pre-commit install && pre-commit run --all-files --show-diff-on-failure
--color=always`
- [ ] Add / Update [the
documentation](https://github.com/volcengine/verl/tree/main/docs).
- [ ] Add unit or end-to-end test(s) to [the CI
workflow](https://github.com/volcengine/verl/tree/main/.github/workflows)
to cover all the code. If not feasible, explain why: ...
- [ ] Once your PR is ready for CI, send a message in [the `ci-request`
channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the
`verl` Slack
workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).
(If not accessible, please try [the Feishu group
(飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)
mtian8 pushed a commit to mtian8/verl that referenced this pull request Nov 1, 2025
### What does this PR do?
This PR is committed for solving 3 things:
1. All test cases in `e2e_ascend` CI pipeline use 8 NPUs by default,
which prevents the machine's original performance from being fully
utilized. This PR is committed for solving this problem. Thank @zheliuyu
for finding this problem :)
2. Remove qwen3 grpo test case in `e2e_ascend.yml` because it is similar
to qwen2.5 grpo.
3. Remove ppo test case in `e2e_ascend.yml` because it is not work since
first commit volcengine#3502 , @xvxuopop is working for solving this.
4. Update e2e_ascend CI scan path for covering most of file modification
case.
5. Ignore loading `libnuma.so` for Ascend NPU.
6. Fix qwen2_vl flash-attention related functions unavailable error for
Ascend NPU.

### Checklist Before Starting

- [x] Search for similar PRs. Paste at least one query link here: ...
- [x] Format the PR title as `[{modules}] {type}: {description}` (This
will be checked by the CI)
- `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`,
`trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`,
`ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`,
`env`, `tool`, `ckpt`, `doc`, `data`
- If this PR involves multiple modules, separate them with `,` like
`[megatron, fsdp, doc]`
  - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test`
- If this PR breaks any API (CLI arguments, config, function signature,
etc.), add `[BREAKING]` to the beginning of the title.
  - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching`

### Test

Not related.

### API and Usage Example

Not related.

### Design & Code Changes

Not related.

### Checklist Before Submitting

> [!IMPORTANT]
> Please check all the following items before requesting a review,
otherwise the reviewer might deprioritize this PR for review.

- [ ] Read the [Contribute
Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md).
- [ ] Apply [pre-commit
checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting):
`pre-commit install && pre-commit run --all-files --show-diff-on-failure
--color=always`
- [ ] Add / Update [the
documentation](https://github.com/volcengine/verl/tree/main/docs).
- [ ] Add unit or end-to-end test(s) to [the CI
workflow](https://github.com/volcengine/verl/tree/main/.github/workflows)
to cover all the code. If not feasible, explain why: ...
- [ ] Once your PR is ready for CI, send a message in [the `ci-request`
channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the
`verl` Slack
workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).
(If not accessible, please try [the Feishu group
(飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)
wangboxiong320 pushed a commit to wangboxiong320/verl that referenced this pull request Nov 1, 2025
### What does this PR do?
This PR is committed for solving 3 things:
1. All test cases in `e2e_ascend` CI pipeline use 8 NPUs by default,
which prevents the machine's original performance from being fully
utilized. This PR is committed for solving this problem. Thank @zheliuyu
for finding this problem :)
2. Remove qwen3 grpo test case in `e2e_ascend.yml` because it is similar
to qwen2.5 grpo.
3. Remove ppo test case in `e2e_ascend.yml` because it is not work since
first commit volcengine#3502 , @xvxuopop is working for solving this.
4. Update e2e_ascend CI scan path for covering most of file modification
case.
5. Ignore loading `libnuma.so` for Ascend NPU.
6. Fix qwen2_vl flash-attention related functions unavailable error for
Ascend NPU.

### Checklist Before Starting

- [x] Search for similar PRs. Paste at least one query link here: ...
- [x] Format the PR title as `[{modules}] {type}: {description}` (This
will be checked by the CI)
- `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`,
`trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`,
`ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`,
`env`, `tool`, `ckpt`, `doc`, `data`
- If this PR involves multiple modules, separate them with `,` like
`[megatron, fsdp, doc]`
  - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test`
- If this PR breaks any API (CLI arguments, config, function signature,
etc.), add `[BREAKING]` to the beginning of the title.
  - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching`

### Test

Not related.

### API and Usage Example

Not related.

### Design & Code Changes

Not related.

### Checklist Before Submitting

> [!IMPORTANT]
> Please check all the following items before requesting a review,
otherwise the reviewer might deprioritize this PR for review.

- [ ] Read the [Contribute
Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md).
- [ ] Apply [pre-commit
checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting):
`pre-commit install && pre-commit run --all-files --show-diff-on-failure
--color=always`
- [ ] Add / Update [the
documentation](https://github.com/volcengine/verl/tree/main/docs).
- [ ] Add unit or end-to-end test(s) to [the CI
workflow](https://github.com/volcengine/verl/tree/main/.github/workflows)
to cover all the code. If not feasible, explain why: ...
- [ ] Once your PR is ready for CI, send a message in [the `ci-request`
channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the
`verl` Slack
workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ).
(If not accessible, please try [the Feishu group
(飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants