[sglang, recipe] feat: add SGLang as rollout engine for one-step-off-policy

[KAMiPan]

What does this PR do?

This PR extends the one-step-off-policy recipe by adding SGLang as an alternative rollout engine to vLLM, allowing flexible backend selection and improving training efficiency.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: Add async vllm backend support for one-step-off-policy training in disaggregated architecture #3460
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

To validate this solution, we adopted the existing experimental configuration from the recipe one-step-off-policy.

The evaluation demonstrates that the proposed SGLang rollout engine integration achieves effective acceleration in one-step-off-policy asynchronous training, providing users with enhanced rollout engine options for diverse deployment scenarios.

Experimental Results

• Machine Configuration: 2 nodes with 16 H20 GPUs each
  • Generation: 4 GPUs
  • Training: 12 GPUs
• Model: Qwen2.5-Math-7B
• Max Response Length: 8,192 tokens
• Algorithm: DAPO
• Rollout Engine: vLLM, SGLang

training mode	engine	step	gen	wait_prev_gen	generate_sequences	old_log_prob	update_actor	total time	acc/best@32/mean	acc/maj@32/mean
colocate sync	SGLang+FSDP2	452	131	-	125	54	199	12h25m	0.6560	0.4471
one-step-overlap async	SGLang+FSDP2	406	-	12	305	58	245	11h12m (+11%)	0.6303	0.4443

• colocate sync: step ≈ gen + old_log_prob + update_actor
• one-step-overlap async: step ≈ max(wait_prev_gen + generate_sequences, old_log_prob + update_actor)

API and Usage Example

Configuration Example

# Using SGLang engine
python3 -m recipe.one_step_off_policy.main_ppo \
    actor_rollout_ref.rollout.name=sglang \
    # ... other configuration parameters

# Using vLLM engine
python3 -m recipe.one_step_off_policy.main_ppo \
    actor_rollout_ref.rollout.name=vllm \
    # ... other configuration parameters

Script Usage

# Using SGLang engine
bash dapo_7b_math_fsdp2_sglang_4_12.sh
bash dapo_7b_math_fsdp2_sglang_colocate.sh

# Using vLLM engine
bash dapo_7b_math_fsdp2_4_12.sh
bash dapo_7b_math_fsdp2_colocate.sh

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[CLAassistant]

All committers have signed the CLA.

[gemini-code-assist]

Code Review

This pull request adds SGLang as a rollout engine option for the one-step-off-policy recipe, providing an alternative to vLLM. The changes include new configuration files and experiment scripts for SGLang, and modifications to fsdp_workers.py to handle weight synchronization for the new engine. While the implementation is mostly correct, I've identified a significant performance issue in the weight synchronization logic for both SGLang and the existing vLLM implementation. My review includes a comment detailing this issue and a recommendation for refactoring to improve efficiency.

[gemini-code-assist]

This block for updating weights is inside a for loop that iterates over each weight tensor. Calling loop.run_until_complete for sglang and inference_model.load_weights for vllm on each tensor individually is inefficient.

For sglang, run_until_complete has significant overhead from setting up and tearing down the event loop. For vllm, it results in many small load_weights calls.

To improve performance, you should refactor this to batch the weight updates. Collect all (key, tensor) pairs in a list within the loop, and then make a single call to load_weights (for vllm) or run_until_complete (for sglang) with the entire list of tensors after the loop finishes. This will significantly reduce overhead.

[KAMiPan]

@vermouth1992 @wuxibin89 @yushengsu-thu

[wuxibin89]

Please move this import into local scope.

[wuxibin89]

All ShardingManagers have been deprecated in #3285 and will be removed in release v0.7