Feature request: GRPO support

[tikikun]

As you all might have already known by now DeepSeek-R1 with its GRPO training was quite successful, should we consider bringing GRPO into torchtune?

[vgoklani]

Yes please. We need a reference-level implementation that's both clear and concise. Should obviously leverage FSDP2

[felipemello1]

Hi all, thanks for you interest! We have a couple of PRs cooking. Please feel free to take a look and contribute/review.

[akashc1]

I've also started working on GRPO on my end, @felipemello1 can you point me to those PRs? would love to collaborate on this.

I'm especially interested in the RLVR approach used in DeepSeek R1 and Tulu 3

[felipemello1]

Take a look at this one that is being done by an user: #2326

[felipemello1]

@akashc1 @dzheng256 @tikikun @vgoklani and others, please check the new comment with a task list. If you guys feel like contributing, that would be a good starting point: #2326