[trainer] refactor: make main_ppo TaskRunner more modular

[eric-haibin-lin]

What does this PR do?

• Added __init__() method to initialize self.role_worker_mapping = {}
• Extracted worker setup logic into dedicated methods:
  • add_actor_rollout_worker() - handles strategy-specific worker imports and setup (lines 130-153)
  • add_critic_worker() - sets up critic worker role mapping (lines 170-176)
  • init_resource_pool_mgr() - creates resource pool specifications (lines 178-187)
  • add_reward_model_worker() - conditionally adds reward model workers (lines 195-203)
  • add_ref_policy_worker() - conditionally adds reference policy workers (lines 205-208)

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

relying on existing unit tests

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[gemini-code-assist]

Code Review

This pull request refactors the TaskRunner class in verl/trainer/main_ppo.py to improve modularity by extracting worker setup logic into separate methods. The changes are well-structured and enhance code organization. However, I've identified a critical bug in the init_resource_pool_mgr method where self.mapping is incorrectly re-assigned, which would discard previously set mappings for reward and reference policy workers, leading to a KeyError during runtime. A fix is provided to address this issue.