[model] feat: add qwen3 grpo 8b/32b script on ASCEND NPU #3310

tardis-key · 2025-09-02T10:45:16Z

What does this PR do?

add examples/grpo_trainer/run_qwen3_32b_npu.sh

add examples/grpo_trainer/run_qwen3_8b_npu.sh

Add concise overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review.

Checklist Before Starting

Search for similar PRs. Paste at least one query link here: ...
Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
- {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
- If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
- {type} is in feat, fix, refactor, chore, test
- If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
- Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

Read the Contribute Guide.
Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
Add / Update the documentation.
Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...

already support in #3300

Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

gemini-code-assist

Code Review

This pull request adds a new script to run GRPO training for the Qwen3-32B model on NPUs. The changes include adding the script itself and a placeholder entry in the documentation. My review found a critical issue in the new shell script that would prevent it from running correctly. I've provided a suggestion to fix it.

examples/grpo_trainer/run_qwen3_32b_npu.sh

* [recipe] fix: Remove redundant parameters to resolve errors in the script caused by the latest Verl main branch. * Add the Qwen3-8B-GRPO training script and related data. Signed-off-by: ZLiao <[email protected]> * Change project name DAPO to GRPO Signed-off-by: ZLiao <[email protected]> * Delete redundant arguments Signed-off-by: ZLiao <[email protected]> --------- Signed-off-by: ZLiao <[email protected]> Co-authored-by: X. HU <[email protected]>

tardis-key · 2025-09-10T01:50:47Z

@FightingZhen hi, please review~

FightingZhen · 2025-09-11T13:12:24Z

examples/grpo_trainer/run_qwen3-8b_npu.sh

@@ -0,0 +1,67 @@
+# Tested successfully on the hiyouga/verl:ngc-th2.6.0-cu126-vllm0.8.4-flashinfer0.2.2-cxx11abi0 image.


These comments can be removed before merged

FightingZhen · 2025-09-11T13:14:29Z

examples/grpo_trainer/run_qwen3-8b_npu.sh

+TRAIN_FILE=${TRAIN_FILE:-"${RAY_DATA_HOME}/data/dapo-math-17k.parquet"}
+TEST_FILE=${TEST_FILE:-"${RAY_DATA_HOME}/data/aime-2024.parquet"}
+
+ray job submit --no-wait --runtime-env="${RUNTIME_ENV}" \


can 8b and 32b shell script use same format? one use ray job submit while another use common python command looks strange :(

tardis-key · 2025-09-12T02:50:05Z

The modified scripts are currently undergoing retesting.

tardis-key · 2025-09-12T06:30:35Z

Training methods in scripts have been unified according to the review comments. Testing has been completed, and five steps have been successfully completed.

…3310) ### What does this PR do? add examples/grpo_trainer/run_qwen3_32b_npu.sh <img width="1014" height="1111" alt="image" src="https://github.com/user-attachments/assets/8cd59fc2-5f6a-419e-87ac-bf35a71856fb" /> add examples/grpo_trainer/run_qwen3_8b_npu.sh <img width="844" height="930" alt="image" src="https://github.com/user-attachments/assets/5c23c7a4-8729-4007-8828-027a8cda4779" /> > Add **concise** overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... > already support in volcengine#3300 - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Signed-off-by: ZLiao <[email protected]> Co-authored-by: ZLiao <[email protected]>

StrongerXi · 2025-10-21T23:22:20Z

examples/grpo_trainer/run_qwen3-8b_npu.sh

+    actor_rollout_ref.actor.use_torch_compile=False \
+    actor_rollout_ref.ref.use_torch_compile=False \


@tardis-key hi I'm curious why you turn off compile, and instead use entropy_from_logits_with_chunking below? Does compile have npu specific issues here?

npu torch compile is only supported after 2.7.x, and the matched version in verl is 2.5.1

@StrongerXi

I see thanks!

…3310) ### What does this PR do? add examples/grpo_trainer/run_qwen3_32b_npu.sh <img width="1014" height="1111" alt="image" src="https://github.com/user-attachments/assets/8cd59fc2-5f6a-419e-87ac-bf35a71856fb" /> add examples/grpo_trainer/run_qwen3_8b_npu.sh <img width="844" height="930" alt="image" src="https://github.com/user-attachments/assets/5c23c7a4-8729-4007-8828-027a8cda4779" /> > Add **concise** overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... > already support in volcengine#3300 - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Signed-off-by: ZLiao <[email protected]> Co-authored-by: ZLiao <[email protected]>

tardis-key requested review from eric-haibin-lin and zhaochenyang20 as code owners September 2, 2025 10:45

gemini-code-assist bot reviewed Sep 2, 2025

View reviewed changes

examples/grpo_trainer/run_qwen3_32b_npu.sh Outdated Show resolved Hide resolved

tardis-key changed the title ~~[WIP][model] feat: add qwen3-32b grpo script on npu~~ [WIP][model] feat: add qwen3-32b grpo script on ASCEND NPU Sep 2, 2025

tardis-key marked this pull request as draft September 2, 2025 10:46

tardis-key force-pushed the qwen3_32b branch 2 times, most recently from ca4e74e to f6e214a Compare September 3, 2025 02:41

tardis-key marked this pull request as ready for review September 3, 2025 02:52

tardis-key changed the title ~~[WIP][model] feat: add qwen3-32b grpo script on ASCEND NPU~~ [model] feat: add qwen3-32b grpo script on ASCEND NPU Sep 3, 2025

tardis-key changed the title ~~[model] feat: add qwen3-32b grpo script on ASCEND NPU~~ [model] feat: add qwen3 grpo script on ASCEND NPU Sep 3, 2025

tardis-key marked this pull request as draft September 3, 2025 08:04

tardis-key force-pushed the qwen3_32b branch from f6e214a to 55a54a4 Compare September 8, 2025 06:40

add qwen3-32b grpo script on npu

5a72467

tardis-key force-pushed the qwen3_32b branch from 55a54a4 to 5a72467 Compare September 8, 2025 06:41

tardis-key changed the title ~~[model] feat: add qwen3 grpo script on ASCEND NPU~~ [model] feat: add qwen3 grpo 8b/32b script on ASCEND NPU Sep 8, 2025

tardis-key marked this pull request as ready for review September 8, 2025 08:30

FightingZhen reviewed Sep 11, 2025

View reviewed changes

tardis-key force-pushed the qwen3_32b branch from fb1edf0 to 7f1c759 Compare September 12, 2025 02:07

unify the filename and training menthods of the two scripts

9149b85

tardis-key force-pushed the qwen3_32b branch from 7f1c759 to 9149b85 Compare September 12, 2025 03:06

tardis-key requested a review from FightingZhen September 12, 2025 06:30

FightingZhen approved these changes Sep 13, 2025

View reviewed changes

vermouth1992 approved these changes Sep 15, 2025

View reviewed changes

vermouth1992 merged commit 2061894 into volcengine:main Sep 15, 2025
5 checks passed

StrongerXi reviewed Oct 21, 2025

View reviewed changes

		@@ -0,0 +1,67 @@
		# Tested successfully on the hiyouga/verl:ngc-th2.6.0-cu126-vllm0.8.4-flashinfer0.2.2-cxx11abi0 image.

		actor_rollout_ref.actor.use_torch_compile=False \
		actor_rollout_ref.ref.use_torch_compile=False \

[model] feat: add qwen3 grpo 8b/32b script on ASCEND NPU #3310

[model] feat: add qwen3 grpo 8b/32b script on ASCEND NPU #3310

Conversation

tardis-key commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist Before Starting

Test

API and Usage Example

Design & Code Changes

Checklist Before Submitting

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

tardis-key commented Sep 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tardis-key commented Sep 12, 2025

Uh oh!

tardis-key commented Sep 12, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tardis-key commented Sep 2, 2025 •

edited

Loading