[roadmap] verl development Q2

[eric-haibin-lin]

Past roadmap for reference: #22

Agentic RL: Environment interaction & tool support [P0]

• Integrate SandBox for code generation tasks (more mature than verl's current code sandbox)
• search / environment interaction via http / grpc
• multi-turn optimizations, better kv cache management and streaming generations (potential inference engine dependency) [sglang] feat: Add SGLang async multi-turn rollout with tool support #1037 [rollout] feat: introduce vLLM AsyncLLM to support multi-turn rollout #1138 will leave kv cache optimization to inference engines
• further multi-turn rollout improvements see Multi-turn rollout & agentic RL Status & Roadmap zhaochenyang20/Awesome-ML-SYS-Tutorial#131

Scaling up RL & system performance [P0]

• Ring Attention
• Ulyssess sequence parallel for VLM models, e.g Qwen2VL
• reference system tuning script for best RL throughput on different types of accelerators
• multi-node rollout (potential inference engine dependency)
• alignment loss fused kernels Feat/memory optimized loss #1212

Usability improvement

• make the current ray trainer easier to extend (without modifying verl source code or forks). Currently users can define their custom reward via command line without modifying verl source code. Ideally in the main training loop we should allow custom dataset as well https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py#L41.
• collect benchmark result on torchtitan vs megatron
• support TorchTitan nd parallelism for better usability

Latest Model & Algorithm Support

See https://verl.readthedocs.io/en/latest/advance/fsdp_extension.html for adding models with FSDP backend
See https://verl.readthedocs.io/en/latest/advance/megatron_extension.html for adding models with Megatron backend.

• gemma3 https://github.com/volcengine/verl/pull/1613/files
• deepseek v3 - since it's large, we should start with SFT for correctness verification, optimize it with fused kernels/recomputation, before moving to RL [Project] deepseek R1 infrastructure #708 Add DeepSeek 671B GRPO example #1771
• qwen3 & qwen3-moe Adding Qwen3 and Qwen3MoE huggingface/transformers#36878
  and any other popular models.
• OLMo2
• Dr. GRPO [Feature Request]Support Dr. GRPO for Unbiased Optimization in RL Training #742

Component Continuous Updates

• verify ulysses sequence parallelism support works with latest version of transformers >= v4.50
• replace FSDP1 with FSDP2 [fsdp] feat: support fsdp2 training and inference in fsdp_workers #1026
• add activation offloading optimization https://github.com/volcengine/verl/pull/1220/files

dataset & benchmark

• gpqa diamond (english)
• LiveCodeBench (code)
• SWE-bench Verified (code)
• CNMO 2024 (math)
• codecontests (Code Generation)
• TACO (Code Generation)
• competition_math (Math)

Please also help provide scripts to reproduce evaluation performance of public released models.

Efficient RL / codesign [P1]

• lora support for RL, and provide convergence report [feat] Add: new feature -- LoRA support for PPO  #1127

Wide Hardware Coverage

Make the experience on non-nvidia GPUs more smooth

• stable Ascend NPUs suppport, with reproducible examples and logs
• stable AMD GPUs suppport, with sglang
• AMD GPU with mcore support

Make verl easier to extend with custom train/infer engine and roles

• [single_controller][decorator] Define a DynamicEnum class to make Dispatch and Execute extensible. #1424
• [RFC] engine interface for training backends (FSDP, FSDP2, torchtitan, Megatron, Mindspore, PAI-Megatron, etc) #1371

other community requests

• retool We need Retool !! #1169
• load balance [Feature Request] Load Balancing for rollout phase #658

[uygnef]

Are there any plans to align the alignment GRPO in Megatron training with FSDP training? We've been struggling for weeks to match the performance(loss/ eval score). We alignment fist step loss and grad (diff less than 0.01), but Megatron's performance degrades after training for several hundred steps.

[eric-haibin-lin]

Are there any plans to align the alignment GRPO in Megatron training with FSDP training? We've been struggling for weeks to match the performance(loss/ eval score). We alignment fist step loss and grad (diff less than 0.01), but Megatron's performance degrades after training for several hundred steps.

@ETOgaosion is your Megatron GRPO training aligned?

[ETOgaosion]

I will provide some examples on wandb.ai , our tests using scripts in examples/grpo_training can align during the whole 200 steps e2e, from the wandb results.

But I do think we still need to develop a calibration tool to dive into it. Because in future, we will have 3 backends: FSDP, veRL implementation of Megatron models, and Megatron GPTModel with TransformerEngine. Alignment does matter.

[ETOgaosion]

Thanks for sharing the recurrent steps https://wandb.ai/uygnef-didi/GRPO_numinamath_test/reports/FSDP-vs-Megatron--VmlldzoxMTk4Mjk0NA?accessToken=tn7ijeeixbeip8xoij0au270xozoh7xjqghmmftp4x96ueoy9wy01a810kh7zlw4 @uygnef . We will check this when development tasks are basically done and before we start to train larger models.

Any key observations to help us identify the root cause underlying this matter would be a significant contribution.

[eric-haibin-lin]

For better usability / extension, comments from the author of code-R1:

From a software engineering perspective, I think it would be more user-friendly for first-time users if we simplify the interface parameters as much as possible! Also, I think improving customization would be very helpful, allowing users to treat verl as a library as much as possible and reduce the need for forking, which would make it easier to accept upstream updates. If there's an interface that allows users to slightly adjust rewards in a prompt group, I think it would enable users to easily customize more algorithm details.
I feel that we can use "prompt: List[response]" as input for the reward function, which would make it easier to adjust the rewards within a prompt group, such as implementing some algorithms for length penalty in Kimi.

[uygnef]

Any key observations to help us identify the root cause underlying this matter would be a significant contribution.
The variances of actor/pg_loss and actor/kl_loss in FSDP and Megatron differ significantly, but average is same.

[sophiayyya]

Hi, what does "reference system tuning script for best RL throughput on different types of accelerators" mean? Does it indicate a series of reference scripts containing parallel size, placement methods, etc for different backend? Thanks!

[BearBiscuit05]

Hi, what does "reference system tuning script for best RL throughput on different types of accelerators" mean? Does it indicate a series of reference scripts containing parallel size, placement methods, etc for different backend? Thanks!

Yes, we will first provide the minimum resource requirements for training some models, the optimal configuration requirements, and the corresponding parallel methods, all presented in the form of a table. Later, we may consider improving a script for automatic configuration search.

[casper-hansen]

Do you plan to release Streaming Rollout System (SRS) to veRL?

[eric-haibin-lin]

Do you plan to release Streaming Rollout System (SRS) to veRL?

We are looking for contribution from the community since the implementation is based on a custom inference engine not suitable for open source.

[uygnef]

Thanks for sharing the recurrent steps https://wandb.ai/uygnef-didi/GRPO_numinamath_test/reports/FSDP-vs-Megatron--VmlldzoxMTk4Mjk0NA?accessToken=tn7ijeeixbeip8xoij0au270xozoh7xjqghmmftp4x96ueoy9wy01a810kh7zlw4 @uygnef . We will check this when development tasks are basically done and before we start to train larger models.

Any key observations to help us identify the root cause underlying this matter would be a significant contribution.

Hi @ETOgaosion ,

We resolved the issue by adjusting the learning rates—FSDP uses 1e-6 and Megatron uses 7e-7. This might not directly address your case, but sharing in case it helps.

Logs for reference: WandB link.