[Feature Request]Support Dr. GRPO for Unbiased Optimization in RL Training

[Peng-YM]

Paper

Dr. GRPO Paper

Motivation/Benefits

• Fixes optimization bias while maintaining reasoning performance
• Reduces average incorrect response length by 38% (Fig.5 in paper)
• Backward-compatible with existing GRPO workflows

[Peng-YM]

See Also: trl@huggingface's PR, we could also add an option scale_rewards=False for GRPO algorithm.

[eric-haibin-lin]

PR is welcome!

[sunjin-k]

scale_grpo_adv is the first part of Dr.GRPO.

The second part is already implemented through loss_agg_mode == "seq-mean-token-sum", but in the paper's implementation they multiply it by a constant factor MAX_TOKENS (data.max_response_length in verl), probably to be able to reuse hyperparameters between GRPO to some extent.

Should we should add ... / max_response_length to loss_agg_mode == "seq-mean-token-sum"?
Or add another option for that?

[Peng-YM]

PR #990 has implemented the Dr. GRPO algorithm, many thanks for your awesome work! @sunjin-k @eric-haibin-lin