Add fix for evo2 generate/inference by jwilber · Pull Request #14027 · NVIDIA-NeMo/NeMo

jwilber · 2025-06-26T05:36:18Z

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Fixes Evo 2 inference issue where state wasn't correctly accumulating during generate.

Background:
Documented issue example: NVIDIA/bionemo-framework#890

Collection: [Note which collection this PR will affect]
GPT/hyena

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Jared Wilber <[email protected]>

nemo/collections/llm/gpt/model/megatron/hyena/hyena_utils.py

+        # x1, x2, v all of shape torch.Size([1, 4096, 63])
+        u = torch.cat([x2, x1, v], dim=1) # torch.Size([1, 12288, 63])
+        L = u.shape[-1]
+        poles = rearrange(self.filter.p, "d n -> d n 1") # n = 16


farhadrgh

Left some comments, LGTM

nemo/collections/llm/gpt/model/megatron/hyena/hyena_mixer.py

nemo/collections/llm/gpt/model/megatron/hyena/engine.py

Signed-off-by: Jared Wilber <[email protected]>

nemo/collections/llm/gpt/model/megatron/hyena/hyena_mixer.py

nemo/collections/llm/gpt/model/megatron/hyena/hyena_utils.py

Signed-off-by: Jared Wilber <[email protected]>

nemo/collections/llm/gpt/model/megatron/hyena/hyena_utils.py

+        update_filter_state("inner_fir", state=fir_state)
+        return rearrange(y, "b l d -> b d l")  # b l d
+
+    def forward(self, x1, x2, v, _hyena_use_cp=True, inference_context=None):


Signed-off-by: Jared Wilber <[email protected]>

Signed-off-by: jwilber <[email protected]>

Signed-off-by: Jared Wilber <[email protected]>

Signed-off-by: jwilber <[email protected]>

nemo/collections/llm/gpt/model/megatron/hyena/hyena_utils.py

jstjohn

Approved with some cosmetic comments and one code block diff to verify.

Signed-off-by: Jared Wilber <[email protected]>

Signed-off-by: jwilber <[email protected]>

Signed-off-by: Jared Wilber <[email protected]>

Signed-off-by: jwilber <[email protected]>

chtruong814 · 2025-07-11T16:57:10Z

Although the code coverage is below the targeted patch change, we will go ahead and merge this given the need to handle a support request. This has a minimal impact for the coverage of the llm collection overall.

* Add fix for evo2 generate/inference Signed-off-by: Jared Wilber <[email protected]> * Add Farhad's suggested refactor Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * Remove unused and truism code Signed-off-by: Jared Wilber <[email protected]> * add inference_context conditional check to use new ops Signed-off-by: Jared Wilber <[email protected]> * add hyena operator tests Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * remove unused code Signed-off-by: Jared Wilber <[email protected]> * add doc strings for flake8 Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * remove unnecessary assignments Signed-off-by: Jared Wilber <[email protected]> * invoke original forward for non-inference calls Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Fix reset issue Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * add docstring Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Remove test Signed-off-by: Jared Wilber <[email protected]> * Add tests for hyena operator Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * simplify context manager in tests Signed-off-by: Jared Wilber <[email protected]> * remove unused import in test Signed-off-by: Jared Wilber <[email protected]> * Add env vars for test Signed-off-by: Jared Wilber <[email protected]> * mark tests gpu only Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> --------- Signed-off-by: Jared Wilber <[email protected]> Signed-off-by: jwilber <[email protected]> Co-authored-by: Jared Wilber <[email protected]> Co-authored-by: jwilber <[email protected]> Co-authored-by: Charlie Truong <[email protected]> Signed-off-by: Amir Hussein <[email protected]>

* Add fix for evo2 generate/inference Signed-off-by: Jared Wilber <[email protected]> * Add Farhad's suggested refactor Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * Remove unused and truism code Signed-off-by: Jared Wilber <[email protected]> * add inference_context conditional check to use new ops Signed-off-by: Jared Wilber <[email protected]> * add hyena operator tests Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * remove unused code Signed-off-by: Jared Wilber <[email protected]> * add doc strings for flake8 Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * remove unnecessary assignments Signed-off-by: Jared Wilber <[email protected]> * invoke original forward for non-inference calls Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Fix reset issue Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * add docstring Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Remove test Signed-off-by: Jared Wilber <[email protected]> * Add tests for hyena operator Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * simplify context manager in tests Signed-off-by: Jared Wilber <[email protected]> * remove unused import in test Signed-off-by: Jared Wilber <[email protected]> * Add env vars for test Signed-off-by: Jared Wilber <[email protected]> * mark tests gpu only Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> --------- Signed-off-by: Jared Wilber <[email protected]> Signed-off-by: jwilber <[email protected]> Co-authored-by: Jared Wilber <[email protected]> Co-authored-by: jwilber <[email protected]> Co-authored-by: Charlie Truong <[email protected]>

* Add fix for evo2 generate/inference Signed-off-by: Jared Wilber <[email protected]> * Add Farhad's suggested refactor Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * Remove unused and truism code Signed-off-by: Jared Wilber <[email protected]> * add inference_context conditional check to use new ops Signed-off-by: Jared Wilber <[email protected]> * add hyena operator tests Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * remove unused code Signed-off-by: Jared Wilber <[email protected]> * add doc strings for flake8 Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * remove unnecessary assignments Signed-off-by: Jared Wilber <[email protected]> * invoke original forward for non-inference calls Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Fix reset issue Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * add docstring Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Remove test Signed-off-by: Jared Wilber <[email protected]> * Add tests for hyena operator Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * simplify context manager in tests Signed-off-by: Jared Wilber <[email protected]> * remove unused import in test Signed-off-by: Jared Wilber <[email protected]> * Add env vars for test Signed-off-by: Jared Wilber <[email protected]> * mark tests gpu only Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> --------- Signed-off-by: Jared Wilber <[email protected]> Signed-off-by: jwilber <[email protected]> Co-authored-by: Jared Wilber <[email protected]> Co-authored-by: jwilber <[email protected]> Co-authored-by: Charlie Truong <[email protected]> Signed-off-by: Amir Hussein <[email protected]>

* Add fix for evo2 generate/inference Signed-off-by: Jared Wilber <[email protected]> * Add Farhad's suggested refactor Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * Remove unused and truism code Signed-off-by: Jared Wilber <[email protected]> * add inference_context conditional check to use new ops Signed-off-by: Jared Wilber <[email protected]> * add hyena operator tests Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * remove unused code Signed-off-by: Jared Wilber <[email protected]> * add doc strings for flake8 Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * remove unnecessary assignments Signed-off-by: Jared Wilber <[email protected]> * invoke original forward for non-inference calls Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Fix reset issue Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * add docstring Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Remove test Signed-off-by: Jared Wilber <[email protected]> * Add tests for hyena operator Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * simplify context manager in tests Signed-off-by: Jared Wilber <[email protected]> * remove unused import in test Signed-off-by: Jared Wilber <[email protected]> * Add env vars for test Signed-off-by: Jared Wilber <[email protected]> * mark tests gpu only Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> --------- Signed-off-by: Jared Wilber <[email protected]> Signed-off-by: jwilber <[email protected]> Co-authored-by: Jared Wilber <[email protected]> Co-authored-by: jwilber <[email protected]> Co-authored-by: Charlie Truong <[email protected]>

* Add fix for evo2 generate/inference Signed-off-by: Jared Wilber <[email protected]> * Add Farhad's suggested refactor Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * Remove unused and truism code Signed-off-by: Jared Wilber <[email protected]> * add inference_context conditional check to use new ops Signed-off-by: Jared Wilber <[email protected]> * add hyena operator tests Signed-off-by: Jared Wilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * remove unused code Signed-off-by: Jared Wilber <[email protected]> * add doc strings for flake8 Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * remove unnecessary assignments Signed-off-by: Jared Wilber <[email protected]> * invoke original forward for non-inference calls Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Fix reset issue Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * add docstring Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * Remove test Signed-off-by: Jared Wilber <[email protected]> * Add tests for hyena operator Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> * lint Signed-off-by: Jared Wilber <[email protected]> * simplify context manager in tests Signed-off-by: Jared Wilber <[email protected]> * remove unused import in test Signed-off-by: Jared Wilber <[email protected]> * Add env vars for test Signed-off-by: Jared Wilber <[email protected]> * mark tests gpu only Signed-off-by: Jared Wilber <[email protected]> * Apply isort and black reformatting Signed-off-by: jwilber <[email protected]> --------- Signed-off-by: Jared Wilber <[email protected]> Signed-off-by: jwilber <[email protected]> Co-authored-by: Jared Wilber <[email protected]> Co-authored-by: jwilber <[email protected]> Co-authored-by: Charlie Truong <[email protected]> Signed-off-by: Guyue Huang <[email protected]>

Add fix for evo2 generate/inference

6498cd9

Signed-off-by: Jared Wilber <[email protected]>

github-advanced-security bot found potential problems Jun 26, 2025

View reviewed changes

farhadrgh approved these changes Jun 26, 2025

View reviewed changes

nemo/collections/llm/gpt/model/megatron/hyena/hyena_mixer.py Show resolved Hide resolved

nemo/collections/llm/gpt/model/megatron/hyena/engine.py Show resolved Hide resolved

Jared Wilber added 2 commits June 26, 2025 13:19

Add Farhad's suggested refactor

c8f34b2

Signed-off-by: Jared Wilber <[email protected]>

lint

422b056

Signed-off-by: Jared Wilber <[email protected]>

ntajbakhsh assigned JRD971000 Jun 27, 2025

jstjohn reviewed Jun 27, 2025

View reviewed changes

nemo/collections/llm/gpt/model/megatron/hyena/hyena_mixer.py Outdated Show resolved Hide resolved

jstjohn reviewed Jun 27, 2025

View reviewed changes

nemo/collections/llm/gpt/model/megatron/hyena/hyena_mixer.py Outdated Show resolved Hide resolved

jstjohn reviewed Jun 27, 2025

View reviewed changes

nemo/collections/llm/gpt/model/megatron/hyena/hyena_mixer.py Outdated Show resolved Hide resolved

jstjohn reviewed Jun 27, 2025

View reviewed changes

nemo/collections/llm/gpt/model/megatron/hyena/hyena_utils.py Outdated Show resolved Hide resolved

Jared Wilber added 2 commits June 30, 2025 13:49

Remove unused and truism code

0e6de40

Signed-off-by: Jared Wilber <[email protected]>

add inference_context conditional check to use new ops

cd4d96b

Signed-off-by: Jared Wilber <[email protected]>

github-advanced-security bot found potential problems Jul 1, 2025

View reviewed changes

Jared Wilber and others added 9 commits July 1, 2025 13:36

add hyena operator tests

181b115

Signed-off-by: Jared Wilber <[email protected]>

lint

87c28d2

Signed-off-by: Jared Wilber <[email protected]>

remove unused code

ad018c3

Signed-off-by: Jared Wilber <[email protected]>

add doc strings for flake8

a21f429

Signed-off-by: Jared Wilber <[email protected]>

Merge branch 'main' into jwilber/fix-evo2-generate

942c241

Apply isort and black reformatting

8c0a32e

Signed-off-by: jwilber <[email protected]>

remove unnecessary assignments

1d283f8

Signed-off-by: Jared Wilber <[email protected]>

invoke original forward for non-inference calls

9e34b8e

Signed-off-by: Jared Wilber <[email protected]>

Apply isort and black reformatting

be0c14e

Signed-off-by: jwilber <[email protected]>

jstjohn reviewed Jul 3, 2025

View reviewed changes

nemo/collections/llm/gpt/model/megatron/hyena/hyena_utils.py Show resolved Hide resolved

jstjohn reviewed Jul 3, 2025

View reviewed changes

nemo/collections/llm/gpt/model/megatron/hyena/hyena_utils.py Show resolved Hide resolved

jstjohn previously approved these changes Jul 3, 2025

View reviewed changes

Fix reset issue

2d3692b

Signed-off-by: Jared Wilber <[email protected]>

jwilber dismissed jstjohn’s stale review via 2d3692b July 3, 2025 22:57

jwilber and others added 3 commits July 3, 2025 22:58

Apply isort and black reformatting

e8afda4

Signed-off-by: jwilber <[email protected]>

Merge branch 'main' into jwilber/fix-evo2-generate

d3e78f6

add docstring

9c35b31

Signed-off-by: Jared Wilber <[email protected]>

JRD971000 added the Run CICD label Jul 10, 2025

JRD971000 temporarily deployed to test July 10, 2025 12:32 — with GitHub Actions Inactive

Add env vars for test

dab1c53

Signed-off-by: Jared Wilber <[email protected]>

ko3n1g added Run CICD and removed Run CICD labels Jul 10, 2025

ko3n1g temporarily deployed to test July 10, 2025 17:24 — with GitHub Actions Inactive

mark tests gpu only

a3202d8

Signed-off-by: Jared Wilber <[email protected]>

ko3n1g added Run CICD and removed Run CICD labels Jul 10, 2025

Apply isort and black reformatting

df97bf5

Signed-off-by: jwilber <[email protected]>

ko3n1g added Run CICD and removed Run CICD labels Jul 10, 2025

ko3n1g temporarily deployed to test July 10, 2025 20:01 — with GitHub Actions Inactive

Merge branch 'main' into jwilber/fix-evo2-generate

3eed182

ko3n1g added Run CICD and removed Run CICD labels Jul 11, 2025

ko3n1g temporarily deployed to test July 11, 2025 04:37 — with GitHub Actions Inactive

Merge branch 'main' into jwilber/fix-evo2-generate

d7cf0a9

ko3n1g added Run CICD and removed Run CICD labels Jul 11, 2025

ko3n1g temporarily deployed to test July 11, 2025 12:21 — with GitHub Actions Inactive

JRD971000 approved these changes Jul 11, 2025

View reviewed changes

chtruong814 merged commit 15dc5cb into NVIDIA-NeMo:main Jul 11, 2025
403 of 406 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fix for evo2 generate/inference#14027

Add fix for evo2 generate/inference#14027
chtruong814 merged 35 commits intoNVIDIA-NeMo:mainfrom
jwilber:jwilber/fix-evo2-generate

jwilber commented Jun 26, 2025 •

edited

Loading

Uh oh!

Check warning

farhadrgh left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check notice

Uh oh!

Uh oh!

jstjohn left a comment

Uh oh!

chtruong814 commented Jul 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

jwilber commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

Check warning

Uh oh!

farhadrgh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check notice

Uh oh!

Uh oh!

jstjohn left a comment

Choose a reason for hiding this comment

Uh oh!

chtruong814 commented Jul 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jwilber commented Jun 26, 2025 •

edited

Loading