Tdt buffered inference fix #13500

hainan-xv · 2025-05-08T16:55:10Z

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Fix issues with TDT buffered inference that results in bad accuracy.

Collection: [Note which collection this PR will affect]

ASR

Changelog

change alignment representation of TDT to include skipped frames as well
add a boundary matching algo in the TDT buffered inference procedure.

Usage

You can potentially add a usage example below

python NeMo/examples/asr/asr_chunked_inference/rnnt/speech_to_text_buffered_infer_rnnt.py \
               ... \
               merge_algo=tdt

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

hainan-xv · 2025-05-08T16:59:16Z

@nithinraok can you take a look by any chance? So far I'm not sure it's 100% correct yet, since it's noticeably worse than offline inference in my testing. I'm not sure how much we're expecting the gap to be though... Could you let me know if it works OK on your end?

nithinraok · 2025-05-08T17:01:19Z

@weiqingw4ng tested it for few samples and it worked great! I will add some numbers here with parakeet-tdt-0.6b-v2 on LS dataset in sometime.

nemo/collections/asr/parts/utils/streaming_utils.py

nithinraok · 2025-05-09T01:40:01Z

Below table shows evaluation of same size, same data trained parakeet models with various decoders. Evaluated with greedy (total buffer size, chunk len in sec)

Decoder	MODE	LS-test clean	LS-test other
tdt	offline	1.69	3.19
tdt	greedy (30,10)	2.07	4.01
tdt	greedy (8,0.16)	3.23	5.19
rnnt	offline	1.82	3.47
rnnt	greedy (30,10)	3.09	5.14
ctc	offline	2.2	3.8
ctc	greedy (30,10)	2.73	4.95
ctc	greedy (8,0.16)	5.39	7.03

nithinraok

Can you add a test case for buffered tdt inference

nithinraok · 2025-05-09T01:52:15Z

examples/asr/asr_chunked_inference/rnnt/speech_to_text_buffered_infer_rnnt.py

    # Change Decoding Config
    with open_dict(cfg.decoding):
-        if cfg.stateful_decoding:
+        if cfg.stateful_decoding or cfg.merge_algo == 'tdt':


Add this info to the doc string above and also add to script usage at the top of this script.

artbataev · 2025-05-09T10:07:29Z

examples/asr/asr_chunked_inference/rnnt/speech_to_text_buffered_infer_rnnt.py


    # Merge algorithm for transducers
-    merge_algo: Optional[str] = 'middle'  # choices=['middle', 'lcs'], choice of algorithm to apply during inference.
+    merge_algo: Optional[str] = (


Suggestion: set merge_algo to None (meaning "auto")
In the code use something like:

if merge_algo is None: merge_algo = "tdt" if model.is_tdt() else "middle"

Thanks! Could you also set default to None in the config (here)?

hainan-xv · 2025-05-09T16:27:16Z

Can you add a test case for buffered tdt inference

I can't seem to find test code for the existing buffered RNNT inference to base the new test on. Not sure how to do it efficiently... What I can think of is to include a small trained TDT model in the repo and include a short audio recording also to run this, but this will take quite some work... Shall we do this later?

Signed-off-by: Francesco Bertolotti <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

…NeMo#13493) Signed-off-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

Signed-off-by: Hainan Xu <[email protected]>

* fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * make fp8 tests non-optional Signed-off-by: Alexandros Koumparoulis <[email protected]> * switch to gemma Signed-off-by: Alexandros Koumparoulis <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

* beep boop: Update changelog Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Add changelog highlights Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Charlie Truong <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Charlie Truong <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

* tests: Disable flaky test Signed-off-by: oliver könig <[email protected]> * remove breakpoint Signed-off-by: oliver könig <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

* Fix 2.3.0 changelog Signed-off-by: Charlie Truong <[email protected]> * Update 2.3.0 changelog Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

Signed-off-by: Pranav Prashant Thombre <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

* add extra hyena tests * Apply isort and black reformatting Signed-off-by: JRD971000 <[email protected]> * fix num gpus * keep sft optional --------- Signed-off-by: JRD971000 <[email protected]> Co-authored-by: JRD971000 <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

Signed-off-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

nithinraok · 2025-05-09T18:28:48Z

I think something messed up. Now its showing 21 file changes

…ed_inference_fix

hainan-xv · 2025-05-10T08:11:22Z

I think something messed up. Now its showing 21 file changes

Not sure what happened back then. I did a pull of the latest main and re-push'ed. It seems it's fixed now.

Signed-off-by: Hainan Xu <[email protected]>

…ed_inference_fix

Signed-off-by: hainan-xv <[email protected]>

nithinraok

LGTM! Thanks @hainan-xv

artbataev

Thanks a lot!

* added use-fast tokenizer argument (NVIDIA-NeMo#12986) Signed-off-by: Francesco Bertolotti <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * ci: Run selective triggering on dockerfiles and dependencies (NVIDIA-NeMo#13493) Signed-off-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * fix buffered inference for tdt Signed-off-by: Hainan Xu <[email protected]> * small fixes Signed-off-by: Hainan Xu <[email protected]> * [automodel] fallback FP8 + LCE -> FP8 + CE (NVIDIA-NeMo#13349) * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * make fp8 tests non-optional Signed-off-by: Alexandros Koumparoulis <[email protected]> * switch to gemma Signed-off-by: Alexandros Koumparoulis <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * Update changelog for `r2.3.0` (NVIDIA-NeMo#13501) * beep boop: Update changelog Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Add changelog highlights Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Charlie Truong <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Charlie Truong <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * Update 2.3.0 changelog (NVIDIA-NeMo#13503) Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * ci: Remove trt-llm breakpoint (NVIDIA-NeMo#13499) * tests: Disable flaky test Signed-off-by: oliver könig <[email protected]> * remove breakpoint Signed-off-by: oliver könig <[email protected]> --------- Signed-off-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * Update 2.3.0 changelog (NVIDIA-NeMo#13504) * Fix 2.3.0 changelog Signed-off-by: Charlie Truong <[email protected]> * Update 2.3.0 changelog Signed-off-by: Charlie Truong <[email protected]> --------- Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * Enabling flash decode for float16 precision only (NVIDIA-NeMo#13471) Signed-off-by: Pranav Prashant Thombre <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * Fix changelog formatting (NVIDIA-NeMo#13505) Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * Updating the long context performance number for B200 (NVIDIA-NeMo#13468) * Add without CP numbers for B200 and merge the captioning texts of both into one. Signed-off-by: Youngeun Kwon <[email protected]> * figure removed Signed-off-by: Youngeun Kwon <[email protected]> --------- Signed-off-by: Youngeun Kwon <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * Autodetect model_type and dtype for deployment using TRT-LLM backend (NVIDIA-NeMo#13209) * Autodetect model_type and dtype for deployment using TRT-LLM backed Signed-off-by: Jan Lasek <[email protected]> * Handling kv_cache_qformat parameter Signed-off-by: Jan Lasek <[email protected]> * Apply isort and black reformatting Signed-off-by: janekl <[email protected]> * Docstring update Signed-off-by: Jan Lasek <[email protected]> --------- Signed-off-by: Jan Lasek <[email protected]> Signed-off-by: janekl <[email protected]> Co-authored-by: janekl <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * remove unused variable Signed-off-by: Hainan Xu <[email protected]> * Apply isort and black reformatting Signed-off-by: hainan-xv <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * add doc string, cleaner way of setting mergo_algo Signed-off-by: Hainan Xu <[email protected]> * Apply isort and black reformatting Signed-off-by: hainan-xv <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * add extra hyena tests (NVIDIA-NeMo#13097) * add extra hyena tests * Apply isort and black reformatting Signed-off-by: JRD971000 <[email protected]> * fix num gpus * keep sft optional --------- Signed-off-by: JRD971000 <[email protected]> Co-authored-by: JRD971000 <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * ci: Add mode files to filter (NVIDIA-NeMo#13517) Signed-off-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]> * change default merge_algo for buffered inference to None Signed-off-by: Hainan Xu <[email protected]> * Apply isort and black reformatting Signed-off-by: hainan-xv <[email protected]> --------- Signed-off-by: Francesco Bertolotti <[email protected]> Signed-off-by: Hainan Xu <[email protected]> Signed-off-by: oliver könig <[email protected]> Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Pranav Prashant Thombre <[email protected]> Signed-off-by: Youngeun Kwon <[email protected]> Signed-off-by: Jan Lasek <[email protected]> Signed-off-by: janekl <[email protected]> Signed-off-by: hainan-xv <[email protected]> Signed-off-by: JRD971000 <[email protected]> Co-authored-by: Francesco Bertolotti <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: oliver könig <[email protected]> Co-authored-by: Hainan Xu <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Charlie Truong <[email protected]> Co-authored-by: pthombre <[email protected]> Co-authored-by: Youngeun Kwon <[email protected]> Co-authored-by: Jan Lasek <[email protected]> Co-authored-by: janekl <[email protected]> Co-authored-by: hainan-xv <[email protected]> Co-authored-by: Ali Taghibakhshi <[email protected]> Co-authored-by: JRD971000 <[email protected]> Signed-off-by: jianbinc <[email protected]>

github-actions bot added the ASR label May 8, 2025

github-advanced-security bot found potential problems May 8, 2025

View reviewed changes

nemo/collections/asr/parts/utils/streaming_utils.py Fixed Show fixed Hide fixed

hainan-xv force-pushed the tdt_buffered_inference_fix branch from 13c3a00 to f8a7ecc Compare May 8, 2025 17:08

nithinraok requested a review from artbataev May 8, 2025 17:13

nithinraok added Run CICD skip-linting labels May 9, 2025

nithinraok requested changes May 9, 2025

View reviewed changes

ko3n1g added Run CICD and removed Run CICD labels May 9, 2025

artbataev reviewed May 9, 2025

View reviewed changes

ko3n1g added Run CICD and removed Run CICD labels May 9, 2025

f14-bertolotti and others added 10 commits May 9, 2025 12:28

added use-fast tokenizer argument (NVIDIA-NeMo#12986)

57fabaa

Signed-off-by: Francesco Bertolotti <[email protected]> Co-authored-by: Alexandros Koumparoulis <[email protected]> Co-authored-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

ci: Run selective triggering on dockerfiles and dependencies (NVIDIA-…

d15485e

…NeMo#13493) Signed-off-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

fix buffered inference for tdt

465ab59

Signed-off-by: Hainan Xu <[email protected]>

small fixes

f9684be

Signed-off-by: Hainan Xu <[email protected]>

Update 2.3.0 changelog (NVIDIA-NeMo#13503)

7cbf683

Signed-off-by: Charlie Truong <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

Enabling flash decode for float16 precision only (NVIDIA-NeMo#13471)

896f4b6

Signed-off-by: Pranav Prashant Thombre <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

JRD971000 and others added 2 commits May 9, 2025 12:29

ci: Add mode files to filter (NVIDIA-NeMo#13517)

62284b6

Signed-off-by: oliver könig <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

hainan-xv force-pushed the tdt_buffered_inference_fix branch from 26c24bf to 62284b6 Compare May 9, 2025 16:30

hainan-xv requested review from chtruong814, ko3n1g, pablo-garay and thomasdhc as code owners May 9, 2025 16:30

github-actions bot added the CI label May 9, 2025

ko3n1g added Run CICD and removed Run CICD labels May 9, 2025

Merge branch 'main' of https://github.com/NVIDIA/NeMo into tdt_buffer…

ebe7743

…ed_inference_fix

github-actions bot removed the CI label May 10, 2025

ko3n1g added Run CICD and removed Run CICD labels May 10, 2025

ko3n1g temporarily deployed to test May 10, 2025 22:57 — with GitHub Actions Inactive

Hainan Xu added 2 commits May 12, 2025 09:25

change default merge_algo for buffered inference to None

47ce9a2

Signed-off-by: Hainan Xu <[email protected]>

Merge branch 'main' of https://github.com/NVIDIA/NeMo into tdt_buffer…

d46ae4c

…ed_inference_fix

ko3n1g added Run CICD and removed Run CICD labels May 12, 2025

Apply isort and black reformatting

a32fd25

Signed-off-by: hainan-xv <[email protected]>

ko3n1g added Run CICD and removed Run CICD labels May 12, 2025

ko3n1g temporarily deployed to test May 12, 2025 15:25 — with GitHub Actions Inactive

nithinraok approved these changes May 12, 2025

View reviewed changes

artbataev approved these changes May 12, 2025

View reviewed changes

nithinraok merged commit 4749e42 into NVIDIA-NeMo:main May 12, 2025
123 checks passed

artbataev mentioned this pull request May 22, 2025

New streaming RNN-T and TDT inference #9106

Merged

8 tasks

Tdt buffered inference fix #13500

Tdt buffered inference fix #13500

Uh oh!

Conversation

hainan-xv commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

hainan-xv commented May 8, 2025

Uh oh!

nithinraok commented May 8, 2025

Uh oh!

Uh oh!

nithinraok commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nithinraok left a comment

Choose a reason for hiding this comment

Uh oh!

nithinraok May 9, 2025

Choose a reason for hiding this comment

Uh oh!

hainan-xv May 9, 2025

Choose a reason for hiding this comment

Uh oh!

artbataev May 9, 2025

Choose a reason for hiding this comment

Uh oh!

hainan-xv May 9, 2025

Choose a reason for hiding this comment

Uh oh!

hainan-xv May 9, 2025

Choose a reason for hiding this comment

Uh oh!

artbataev May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hainan-xv May 12, 2025

Choose a reason for hiding this comment

Uh oh!

hainan-xv commented May 9, 2025

Uh oh!

nithinraok commented May 9, 2025

Uh oh!

hainan-xv commented May 10, 2025

Uh oh!

nithinraok left a comment

Choose a reason for hiding this comment

Uh oh!

artbataev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

hainan-xv commented May 8, 2025 •

edited

Loading

nithinraok commented May 9, 2025 •

edited

Loading

artbataev May 12, 2025 •

edited

Loading