Skip to content

Remove flaky Evo2 dataset performance test #14371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 31, 2025
Merged

Remove flaky Evo2 dataset performance test #14371

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a60336b
Less stringent timing threshold for dataset performance test
jstjohn Jul 31, 2025
3ff55c4
Remove timing test from the dataset test suite
jstjohn Jul 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions tests/collections/llm/gpt/data/megatron/hyena/test_evo2_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1045,11 +1045,11 @@ def benchmark_phylo_tag_masking(num_iterations: int = 1000) -> tuple[float, floa
return old_time, new_time


def test_phylo_tag_masking_speed():
def test_phylo_tag_masking_speed(minimal_threshold: float = 0.6):
num_iterations = 2000
old_time, new_time = benchmark_phylo_tag_masking(num_iterations=num_iterations)
# Assert performance equivalent to within 20% or better on a small example.
assert old_time / num_iterations > (new_time / num_iterations) * 0.8
# Assert performance equivalent to no worse than 60% of the original implementation.
assert old_time / num_iterations > (new_time / num_iterations) * minimal_threshold


if __name__ == "__main__":
Expand Down
Loading