Enable trl GRPO trainer #2088

schoi-habana · 2025-06-26T21:53:55Z

This enables GRPO trainer.
The baseline is #1898
This reproduce the example from HF https://huggingface.co/learn/cookbook/en/fine_tuning_llm_grpo_trl on gaudi

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

#1360)

Co-authored-by: regisss <[email protected]>

regisss

I left a few comments. Can you also add some tests for this new trainer in https://github.com/huggingface/optimum-habana/blob/main/tests/test_trl.py please?

examples/trl/requirements_grpo.txt

examples/trl/README.md

optimum/habana/trl/trainer/grpo_config.py

optimum/habana/trl/trainer/grpo_trainer.py

github-actions · 2025-07-08T22:54:31Z

The code quality check failed, please run make style.

github-actions · 2025-07-08T22:56:32Z

The code quality check failed, please run make style.

schoi-habana · 2025-07-09T05:02:46Z

I left a few comments. Can you also add some tests for this new trainer in https://github.com/huggingface/optimum-habana/blob/main/tests/test_trl.py please?

@regisss thanks for the review! i will add some tests soon.
the tests need to run with PT_HPU_LAZY_MODE=1. where do you usually add it to tests?

regisss · 2025-07-09T08:56:38Z

I left a few comments. Can you also add some tests for this new trainer in https://github.com/huggingface/optimum-habana/blob/main/tests/test_trl.py please?

@regisss thanks for the review! i will add some tests soon. the tests need to run with PT_HPU_LAZY_MODE=1. where do you usually add it to tests?

Right now it is added at the level of the makefile:

optimum-habana/Makefile

Line 20 in ee0dcd6

export PT_HPU_LAZY_MODE=1

Since CI tests are executed through makefile commands, that's everything we need. However, if you want to run it with pytest, you'll have to pass the env variable explicitly.
Most tests require lazy mode so I think you can just add yours and that should be fine.

github-actions · 2025-07-09T21:41:32Z

The code quality check failed, please run make style.

regisss

LGTM

Signed-off-by: Wang, Yi A <[email protected]> Signed-off-by: Daniel Socek <[email protected]> Signed-off-by: U. Artie Eoff <[email protected]> Signed-off-by: Vivek Kumar <[email protected]> Signed-off-by: Urszula <[email protected]> Co-authored-by: regisss <[email protected]> Co-authored-by: Jimin Ha <[email protected]> Co-authored-by: Shiv Kaul <[email protected]> Co-authored-by: Yeonsil Yoon <[email protected]> Co-authored-by: Harish Subramony <[email protected]> Co-authored-by: Vidya Galli <[email protected]> Co-authored-by: Iman Gohari <[email protected]> Co-authored-by: Bhargav <[email protected]> Co-authored-by: Chetan Kumar Verma <[email protected]> Co-authored-by: Sheng Yang <[email protected]> Co-authored-by: Yaser Afshar <[email protected]> Co-authored-by: Wang, Yi <[email protected]> Co-authored-by: Akihiro Takahashi <[email protected]> Co-authored-by: Edward Mascarenhas <[email protected]> Co-authored-by: Libin Tang <[email protected]> Co-authored-by: Sayantan Sarkar <[email protected]> Co-authored-by: Daniel Socek <[email protected]> Co-authored-by: Silvia Colabrese <[email protected]> Co-authored-by: Mieszko Dziadowiec <[email protected]> Co-authored-by: Dmitry <[email protected]> Co-authored-by: Urszula Golowicz <[email protected]> Co-authored-by: Nikolay Protasov <[email protected]> Co-authored-by: U. Artie Eoff <[email protected]> Co-authored-by: Luca Calabria <[email protected]> Co-authored-by: Harshvardhan Chauhan <[email protected]> Co-authored-by: Shifani Rajabose <[email protected]> Co-authored-by: Dmitry <[email protected]> Co-authored-by: Mounika Mandava <[email protected]> Co-authored-by: ZhengHongming888 <[email protected]> Co-authored-by: Alexey Fadeev <[email protected]> Co-authored-by: Vivek Kumar <[email protected]> Co-authored-by: Vivek Kumar <[email protected]> Co-authored-by: Ilyas Moutawwakil <[email protected]> Co-authored-by: Rafal Bogdanowicz <[email protected]> Co-authored-by: Rafal <[email protected]> Co-authored-by: Jan Kamiński <[email protected]> Co-authored-by: Karol Brejna <[email protected]> Co-authored-by: Piotr Bielak <[email protected]> Co-authored-by: Piotr Bielak <[email protected]> Co-authored-by: karol-brejna-i <[email protected]> Co-authored-by: IlyasMoutawwakil <[email protected]>

Signed-off-by: Wang, Yi A <[email protected]> Signed-off-by: Daniel Socek <[email protected]> Signed-off-by: U. Artie Eoff <[email protected]> Signed-off-by: Vivek Kumar <[email protected]> Signed-off-by: Urszula <[email protected]> Co-authored-by: Sun Choi <[email protected]> Co-authored-by: regisss <[email protected]> Co-authored-by: Jimin Ha <[email protected]> Co-authored-by: Shiv Kaul <[email protected]> Co-authored-by: Yeonsil Yoon <[email protected]> Co-authored-by: Harish Subramony <[email protected]> Co-authored-by: Vidya Galli <[email protected]> Co-authored-by: Iman Gohari <[email protected]> Co-authored-by: Bhargav <[email protected]> Co-authored-by: Chetan Kumar Verma <[email protected]> Co-authored-by: Sheng Yang <[email protected]> Co-authored-by: Yaser Afshar <[email protected]> Co-authored-by: Wang, Yi <[email protected]> Co-authored-by: Akihiro Takahashi <[email protected]> Co-authored-by: Edward Mascarenhas <[email protected]> Co-authored-by: Libin Tang <[email protected]> Co-authored-by: Sayantan Sarkar <[email protected]> Co-authored-by: Daniel Socek <[email protected]> Co-authored-by: Silvia Colabrese <[email protected]> Co-authored-by: Mieszko Dziadowiec <[email protected]> Co-authored-by: Dmitry <[email protected]> Co-authored-by: Urszula Golowicz <[email protected]> Co-authored-by: Nikolay Protasov <[email protected]> Co-authored-by: U. Artie Eoff <[email protected]> Co-authored-by: Luca Calabria <[email protected]> Co-authored-by: Harshvardhan Chauhan <[email protected]> Co-authored-by: Shifani Rajabose <[email protected]> Co-authored-by: Dmitry <[email protected]> Co-authored-by: Mounika Mandava <[email protected]> Co-authored-by: ZhengHongming888 <[email protected]> Co-authored-by: Alexey Fadeev <[email protected]> Co-authored-by: Vivek Kumar <[email protected]> Co-authored-by: Vivek Kumar <[email protected]> Co-authored-by: Ilyas Moutawwakil <[email protected]> Co-authored-by: Rafal Bogdanowicz <[email protected]> Co-authored-by: Rafal <[email protected]> Co-authored-by: Jan Kamiński <[email protected]> Co-authored-by: Karol Brejna <[email protected]> Co-authored-by: Piotr Bielak <[email protected]> Co-authored-by: Piotr Bielak <[email protected]> Co-authored-by: karol-brejna-i <[email protected]> Co-authored-by: IlyasMoutawwakil <[email protected]>

regisss and others added 30 commits September 2, 2024 16:51

Upgrade to commit 74e19e81e2a23809af192532b9b0e7ea202be6f2

5e88ce7

Merge branch 'main' into transformers_future

43bc4eb

Add specific commit in setup.py

8eea643

Upgrade to commit e48e5f1f13e05380e24f4f31f5fee07aa6f959eb

a7be363

Merge branch 'main' into transformers_future

f0b909a

Fix default cache

d99f18f

Merge branch 'main' into transformers_future

39b7a76

Merge branch 'main' into transformers_future

da66ecf

Merge branch 'main' into transformers_future

5547767

Merge branch 'main' into transformers_future

bf89e41

Merge branch 'main' into transformers_future

98b0da5

Upgrade to commit 238b13478df209ab534f2195a397dc64a3930883

47ad03c

Fix

94c23ba

Upgrade to v4.45.0

c19dedd

Merge branch 'main' into transformers_future

c12fd7e

Fix

fc399fa

Add bias to gptj (#1363)

9216159

Switch roberta from sdpa to eager attn (#1361)

679365a

Update bloom attention forward reshape follwing the transformer change (

1abd6ee

#1360)

Workaround for Llava/Llava-next

8043d2c

Fix reshape error in mamba (#1369)

047e7ff

Merge branch 'main' into transformers_future

f89e03b

Merge branch 'main' into transformers_future

2ae546a

Fix contrastive search

1b8a3f7

Fix local variable 'image_features' referenced before assignment (#1383)

2332afb

Use model.generation_config instead of model.config (#1384)

f62ecde

Co-authored-by: regisss <[email protected]>

Make style

a8fb8ac

Merge branch 'main' into transformers_future

c7ff331

Upgrade to Transformers v4.47.1

dd07c16

Fix Transformers version to install

1924c89

schoi-habana force-pushed the schoi/grpo_from_pr1898_gradient_ckpt branch from 882b81b to a65a9d6 Compare June 27, 2025 22:58

schoi-habana marked this pull request as ready for review June 27, 2025 23:05

schoi-habana requested a review from regisss as a code owner June 27, 2025 23:05

regisss reviewed Jul 8, 2025

View reviewed changes

examples/trl/requirements_grpo.txt Outdated Show resolved Hide resolved

examples/trl/README.md Show resolved Hide resolved

optimum/habana/trl/trainer/grpo_config.py Show resolved Hide resolved

optimum/habana/trl/trainer/grpo_trainer.py Outdated Show resolved Hide resolved

schoi-habana force-pushed the schoi/grpo_from_pr1898_gradient_ckpt branch from 4c1597f to df5f215 Compare July 8, 2025 22:53

schoi-habana force-pushed the schoi/grpo_from_pr1898_gradient_ckpt branch from df5f215 to 6b347df Compare July 8, 2025 22:56

incorporating the review

86dcc6a

schoi-habana force-pushed the schoi/grpo_from_pr1898_gradient_ckpt branch from 6b347df to 86dcc6a Compare July 8, 2025 22:58

astachowiczhabana self-assigned this Jul 9, 2025

astachowiczhabana added the synapse1.22 label Jul 9, 2025

add tests for grpo

3c0a6a6

schoi-habana force-pushed the schoi/grpo_from_pr1898_gradient_ckpt branch from 961d0ac to 3c0a6a6 Compare July 9, 2025 20:40

update tests in Makefile

7962be0

schoi-habana force-pushed the schoi/grpo_from_pr1898_gradient_ckpt branch from d6c11d6 to 7962be0 Compare July 9, 2025 21:43

yafshar mentioned this pull request Jul 10, 2025

Upgrade to TRL 0.17.0 version #2130

Draft

3 tasks

karol-brejna-i assigned pbielak and unassigned astachowiczhabana Jul 11, 2025

regisss approved these changes Jul 28, 2025

View reviewed changes

regisss merged commit 84bc77c into main Jul 28, 2025
6 of 9 checks passed

regisss deleted the schoi/grpo_from_pr1898_gradient_ckpt branch July 28, 2025 10:44

yafshar mentioned this pull request Jul 28, 2025

Implementation of GRPO in Gaudi TRL #1898

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable trl GRPO trainer #2088

Enable trl GRPO trainer #2088

Uh oh!

schoi-habana commented Jun 26, 2025 •

edited

Loading

Uh oh!

regisss left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

schoi-habana commented Jul 9, 2025

Uh oh!

regisss commented Jul 9, 2025

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

regisss left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

34 participants

Enable trl GRPO trainer #2088

Enable trl GRPO trainer #2088

Uh oh!

Conversation

schoi-habana commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before submitting

Uh oh!

regisss left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

schoi-habana commented Jul 9, 2025

Uh oh!

regisss commented Jul 9, 2025

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

regisss left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

34 participants

schoi-habana commented Jun 26, 2025 •

edited

Loading