Skip to content

Add functional tests for init, train step, and inference for every supported released model #15433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chtruong814 merged 14 commits into from
Mar 14, 2026
Merged

Add functional tests for init, train step, and inference for every supported released model #15433

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
9bd4767
Add functional tests for all supported NeMo speech models
pzelasko Feb 18, 2026
122af95
Split up the functional tests into separate files
pzelasko Feb 23, 2026
1dd0378
Fix all 15 failing model training step tests (80/80 pass)
pzelasko Feb 24, 2026
558ae87
remove test reading csv file
pzelasko Feb 24, 2026
ff7f815
model registry json
pzelasko Feb 24, 2026
7100512
remove numba workaround
pzelasko Feb 24, 2026
aad889d
Merge branch 'main' into model-support-functional-tests
pzelasko Feb 24, 2026
4958efe
fix lint
pzelasko Feb 25, 2026
408ae1a
Exclude functional tests from Unit_Test_X_Others
pzelasko Feb 25, 2026
9b5cd89
Merge branch 'main' into model-support-functional-tests
pzelasko Mar 2, 2026
67b05f0
Address review, fix specific tests
pzelasko Mar 5, 2026
eace8eb
Merge branch 'main' into model-support-functional-tests
pzelasko Mar 11, 2026
4183cf0
Conversion of the new tests to e2e_nightly + register them in Github …
pzelasko Mar 11, 2026
e0e5621
Revert change
pzelasko Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
241 changes: 241 additions & 0 deletions .github/workflows/cicd-main-speech.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,247 @@ jobs:
script: L2_TTS_InferEvaluate_Magpietts_MoE_ZeroShot
- runner: self-hosted-azure
script: L2_TTS_InferEvaluatelongform_Magpietts_MoE_ZeroShot
# Model support functional tests
- runner: self-hosted-azure
Copy link
Collaborator

@chtruong814 chtruong814 Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pzelasko regarding scheduling this, we actually run nightly tests on main branch already. Looks like one of the CI tests needs to be fixed.
https://github.com/NVIDIA-NeMo/NeMo/actions/runs/22421911989
https://github.com/NVIDIA-NeMo/NeMo/blob/main/.github/workflows/cicd-main.yml#L17

In any case, to separate these out to run nightly only, it may be better to move the tests to a new group called e2e-nightly or whatever you want on it and apply a condition like this:
https://github.com/NVIDIA-NeMo/NeMo/blob/main/.github/workflows/cicd-main.yml#L246

The condition could be something like:

if: ${{ github.event_name == 'schedule' }}
needs: unit-tests

Copy link
Collaborator Author

@pzelasko pzelasko Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chtruong814 added, can you check if it looks OK? Thanks!

script: L2_Model_Support_nvidia__stt_de_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_es_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_it_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_ua_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_pl_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_hr_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_be_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_fr_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_ru_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_nl_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_fa_fastconformer_hybrid_large
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_ka_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_kk_ru_fastconformer_hybrid_large
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_uz_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_ar_fastconformer_hybrid_large_pc_v1_0
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_hy_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_medium_streaming_80ms_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_medium_streaming_80ms
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_pt_fastconformer_hybrid_large_pc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_es_fastconformer_hybrid_large_pc_nc
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_ar_fastconformer_hybrid_large_pcd_v1_0
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_large_streaming_multi
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_ctc_large
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_transducer_large
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_ctc_xlarge
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_transducer_xlarge
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_transducer_xxlarge
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_ctc_xxlarge
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__stt_en_fastconformer_tdt_large
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_stt_en_fastconformer_hybrid_large_streaming_1040ms
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_stt_multilingual_fastconformer_hybrid_large_pc_blend_eu
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_rnnt_1_1b
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_ctc_1_1b
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_rnnt_0_6b
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_ctc_0_6b
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_tdt_1_1b
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_tdt_ctc_1_1b
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_tdt_ctc_0_6b_ja
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_tdt_ctc_110m
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_tdt_0_6b_v2
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_rnnt_110m_da_dk
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_tdt_0_6b_v3
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_ctc_0_6b_Vietnamese
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__canary_1b
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__canary_1b_flash
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__canary_180m_flash
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__canary_1b_v2
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__parakeet_realtime_eou_120m_v1
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__multitalker_parakeet_streaming_0_6b_v1
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__nemotron_speech_streaming_en_0_6b
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__canary_qwen_2_5b
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__diar_sortformer_4spk_v1
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__diar_streaming_sortformer_4spk_v2
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__diar_streaming_sortformer_4spk_v2_1
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_titanet_large
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__speakerverification_en_titanet_large
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__ssl_en_nest_large_v1_0
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__ssl_en_nest_xlarge_v1_0
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_vad_multilingual_marblenet
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_vad_multilingual_frame_marblenet
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__Frame_VAD_Multilingual_MarbleNet_v2_0
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__se_den_sb_16k_small
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__se_der_sb_16k_small
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__sr_ssl_flowmatching_16k_430m
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_mel_codec_44khz_medium
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_mel_codec_22khz_fullband_medium
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__low_frame_rate_speech_codec_22khz
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__audio_codec_22khz
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__audio_codec_44khz
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__mel_codec_22khz
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__mel_codec_44khz
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__nemo_nano_codec_22khz_1_78kbps_12_5fps
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__nemo_nano_codec_22khz_1_89kbps_21_5fps
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__nemo_nano_codec_22khz_0_6kbps_12_5fps
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__tts_en_fastpitch
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__tts_hifigan
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_nvidia__magpie_tts_multilingual_357m
timeout: 15
- runner: self-hosted-azure
script: L2_Model_Support_tts_en_e2e_fastspeech2hifigan
timeout: 15
needs: [unit-tests]
runs-on: ${{ matrix.runner }}
name: ${{ matrix.is-optional && 'PLEASEFIXME_' || '' }}${{ matrix.script }}
Expand Down
22 changes: 7 additions & 15 deletions nemo/core/utils/cuda_python_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
# limitations under the License.

import contextlib
import inspect

import numpy as np
import torch
Expand Down Expand Up @@ -171,21 +170,16 @@ def with_conditional_node(while_loop_kernel, while_loop_args, while_loop_conditi
# Use driver API here because of bug in cuda-python runtime API: https://github.com/NVIDIA/cuda-python/issues/55
# TODO: Change call to this after fix goes in (and we bump minimum cuda-python version to 12.4.0):
# node, = cu_call(cudart.cudaGraphAddNode(graph, dependencies, len(dependencies), driver_params))
# depending on cuda-python version, number of parameters vary
num_cuda_graph_add_node_params = len(inspect.signature(cuda.cuGraphAddNode).parameters)
if num_cuda_graph_add_node_params == 5:
# CUDA 13 (cuda-python >= 13.0.0) adds an edgeData parameter to cuGraphAddNode and
# cudaStreamUpdateCaptureDependencies; CUDA 12 does not accept it.
_cuda13 = Version(cuda_python_version) >= Version("13.0.0")
if _cuda13:
(node,) = cu_call(cuda.cuGraphAddNode(graph, dependencies, None, len(dependencies), driver_params))
elif num_cuda_graph_add_node_params == 4:
(node,) = cu_call(cuda.cuGraphAddNode(graph, dependencies, len(dependencies), driver_params))
else:
raise NeMoCUDAPythonException("Unexpected number of parameters for `cuGraphAddNode`")
(node,) = cu_call(cuda.cuGraphAddNode(graph, dependencies, len(dependencies), driver_params))
body_graph = driver_params.conditional.phGraph_out[0]

# depending on cuda-python version, number of parameters vary
num_cuda_stream_update_capture_dependencies_params = len(
inspect.signature(cudart.cudaStreamUpdateCaptureDependencies).parameters
)
if num_cuda_stream_update_capture_dependencies_params == 5:
if _cuda13:
cu_call(
cudart.cudaStreamUpdateCaptureDependencies(
torch.cuda.current_stream(device=device).cuda_stream,
Expand All @@ -195,7 +189,7 @@ def with_conditional_node(while_loop_kernel, while_loop_args, while_loop_conditi
cudart.cudaStreamUpdateCaptureDependenciesFlags.cudaStreamSetCaptureDependencies,
)
)
elif num_cuda_stream_update_capture_dependencies_params == 4:
else:
cu_call(
cudart.cudaStreamUpdateCaptureDependencies(
torch.cuda.current_stream(device=device).cuda_stream,
Expand All @@ -204,8 +198,6 @@ def with_conditional_node(while_loop_kernel, while_loop_args, while_loop_conditi
cudart.cudaStreamUpdateCaptureDependenciesFlags.cudaStreamSetCaptureDependencies,
)
)
else:
raise NeMoCUDAPythonException("Unexpected number of parameters for `cudaStreamUpdateCaptureDependencies`")
body_stream = torch.cuda.Stream(device)
previous_stream = torch.cuda.current_stream(device=device)
cu_call(
Expand Down
Loading
Loading