Support for multi-channel audio data

[yunbin]

Describe the bug

NeMo training and decoding scripts do not support multi-channel audio data.

Steps/Code to reproduce bug

It does not support specifying which channel to use for each audio file in each line in train.manifest.json or test.manifest.json file.

I was able to run ./examples/asr/speech_to_text_eval.py with "channel_selector=" to specify the channel for all the audio in a manifest.json file, but I can't find a way to specifying them for each audio file inside the manifest.json file.

Expected behavior

Can NeMo team add this useful feature to work with a diverse set of multi-channel training and testing audio data so data from different channel can be mixed within a manifest.json file?

Environment overview (please complete the following information)

NeMo was installed by pip in a conda environment. It works for single channel audio data.

[anteju]

Thanks @yunbin, we'll take a look at adding this functionality over the next couple weeks. We'll post future updates here.

[yunbin]

@anteju Any update on getting the channel feature implemented in NeMo training scripts?

[anteju]

@yunbin, please check #9007.
The change there enables using channel selector from manifest for nvidia/canary-1b model.

[Jesteinbe]

@pzelasko @anteju - i've been able to use this as long as i use python $NEMO_ROOT/examples/asr/transcribe_speech.py channel_selector=[0,1] but I really want to be able to specify the channel in the manifest file itself so that I can decode a series of utterances regardless of which channel they're on. otherwise i have to run this script once for channel 0 and again for channel 1. am i doing something wrong? is there a correct field i could be using in the manifest?

FYI i'm currently using Nemo v2.4.0 and python 3.12.0 .

[anteju]

@Jesteinbe, could you please check the "Toy Example" attached to #9007?
Direct link to example scripts: https://github.com/NVIDIA/NeMo/files/15069660/pr_example_01.tar.gz

Please check the manifest: it includes a field called reference_channel. You should be able to provide the name of the custom field as string to transcribe_speech, i.e., something like

python transcribe_speech.py <argument> channel_selector=reference_channel

For details, please check the scripts above.
If you cannot run the example due to any bugs, please share the exact error and steps to reproduce it.

[Jesteinbe]

@anteju - Thanks for the info. based on what you describe in the PR I expected that i should be able to specify the channel field in the manifest with channel_selector so I was surprised when I couldn't get it to work. For the record, this is the error I got when i ran:

python ~/git/NeMo/examples/asr/transcribe_speech.py cuda=0 model_path=./stt_enes_conformer_ctc_large_codesw.nemo dataset_manifest=debug.manifest.jsonl output_filename=foo.json channel_selector=reference_channel

where debug.manifest.jsonl looks like this:
{"audio_filepath": "debug.mp3", "offset": 0.0, "duration": 3.02, "reference_channel":1}

The model loads and i get the normal printouts but when it finally is about to decode i get this error:

Error executing job with overrides: ['cuda=0', 'model_path=./stt_enes_conformer_ctc_large_codesw.nemo', 'dataset_manifest=debug.manifest.jsonl', 'output_filename=foo.json', 'channel_selector=reference_channel']
Traceback (most recent call last):
  File "/home/git/NeMo/examples/asr/transcribe_speech.py", line 426, in main
    transcriptions = asr_model.transcribe(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/models/ctc_models.py", line 182, in transcribe
    return super().transcribe(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/parts/mixins/transcription.py", line 270, in transcribe
    for processed_outputs in generator:
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/parts/mixins/transcription.py", line 365, in transcribe_generator
    for test_batch in tqdm(dataloader, desc="Transcribing", disable=not verbose):
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 733, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 789, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
            ~~~~~~~~~~~~^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/data/audio_to_text.py", line 477, in __getitem__
    return self._process_sample(index)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/data/audio_to_text.py", line 486, in _process_sample
    features = self.featurizer.process(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 202, in process
    audio = AudioSegment.from_file(
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/parts/preprocessing/segment.py", line 379, in from_file
    return cls(
           ^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/parts/preprocessing/segment.py", line 187, in __init__
    raise ValueError(
ValueError: ('Input signal is one-dimensional, channel selector (%s) cannot not be used.', 'reference_channel')

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

As you suggested, I also tried running the toy example from the PR but when I hit an error in the first step (bash test_transcribe.sh 0). I got an error about the manifest having the duration field set to "null":

Error executing job with overrides: ['pretrained_name=nvidia/canary-1b', 'dataset_manifest=./mc_manifest.json', 'gt_lang_attr_name=source_lang', 'channel_selector=0']
Traceback (most recent call last):
  File "/home/git/NeMo/examples/asr/transcribe_speech.py", line 364, in main
    filepaths, sorted_manifest_path = prepare_audio_data(cfg)
                                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/parts/utils/transcribe_utils.py", line 336, in prepare_audio_data
    raise ValueError(
ValueError: Requested presort_manifest=True, but line {"audio_filepath": "./audio/mc_00.wav", "reference_channel": 0, "duration": null, "text_ch0": "at n e c the need for international managers will keep rising", "text_ch1": "economists were encouraged by a one point six percent increase in new orders for non defense capital goods an important indicator of future business investment", "taskname": "asr", "source_lang": "en", "target_lang": "en", "pnc": "no", "answer": "na"} in manifest ./mc_manifest.json                             lacks a 'duration' field.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

I installed nemo with pip install nemo-toolkit['all']==2.4.0 and even grabbed a clean version of the repo (using tag v2.4.0) to be sure. Let me know if you have any thoughts.

[anteju]

(1) For the toy example:
Could you add presort_manifest=False in the call to transcribe_speech.py and rerun?
Alternatively, you can add duration field in the manifest (mc_manifest.json).

(2) For your example:
How many channels do you have in your audio file "audio_filepath": "debug.mp3"?

[Jesteinbe]

@anteju - I set presort_manifest=False and now running bash test_transcribe.sh 0 runs without error but bash test_transcribe.sh 1 errors out with this:

Error executing job with overrides: ['pretrained_name=nvidia/canary-1b', 'dataset_manifest=./mc_manifest.json', 'gt_lang_attr_name=source_lang', 'channel_selector=1', 'presort_manifest=False']
Traceback (most recent call last):
  File "/home/jo23828/git/deleteme/NeMo/examples/asr/transcribe_speech.py", line 426, in main
    transcriptions = asr_model.transcribe(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/models/aed_multitask_models.py", line 532, in transcribe
    return super().transcribe(audio=audio, override_config=trcfg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/parts/mixins/transcription.py", line 270, in transcribe
    for processed_outputs in generator:
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/parts/mixins/transcription.py", line 365, in transcribe_generator
    for test_batch in tqdm(dataloader, desc="Transcribing", disable=not verbose):
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 733, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 789, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    data = self.dataset[possibly_batched_index]
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/asr/data/audio_to_text_lhotse_prompted.py", line 86, in __getitem__
    formatted = [apply_prompt_format_fn(cut, self.prompt) for cut in cuts]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/common/data/prompt_fn.py", line 91, in apply_prompt_format_fn
    return fn(example, prompt)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/.conda/envs/nemo_2.4/lib/python3.12/site-packages/nemo/collections/common/prompts/canary.py", line 171, in canary
    prompt.PROMPT_LANGUAGE_SLOT: ifnone(cut.supervisions[0].language, cut.custom.get("target_lang")),
                                        ~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

I get the exact same error when i run bash test_transcribe.sh reference_channel with the only difference coming from a channel_selector="reference_channel" instead of 1 :

< Error executing job with overrides: ['pretrained_name=nvidia/canary-1b', 'dataset_manifest=./mc_manifest.json', 'gt_lang_attr_name=source_lang', 'channel_selector=1', 'presort_manifest=False']
---
> Error executing job with overrides: ['pretrained_name=nvidia/canary-1b', 'dataset_manifest=./mc_manifest.json', 'gt_lang_attr_name=source_lang', 'channel_selector=reference_channel', 'presort_manifest=False']

As for your second question, I have 2 channels in my debug.mp3 audio file. I could get both channels to decode correctly, e.g. if i pass channel_selector=0 or channel_selector=1 to transcribe_speech.py so I know that's not the issue. i just couldn't get it to work by passing it the string of the field to read it from in the manifest.

[Jesteinbe]

@anteju another quick update - When I first posted my issue I was decoding with different models:

https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_enes_conformer_ctc_large_codesw
https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_multilingual_fastconformer_hybrid_large_pc

For both of those, I could specify channel_selector to be 0 or 1 without issue but it errored out if i tried to specify a field in the manifest.

Conversely, I just tried to decode my debug.mp3 with the canary-1b-flash model and see the same behavior as when i ran the toy problem (i.e. channel 0 worked but channel 1 errors out with the message above). It seems like maybe there's at least one bug in how the different architectures handle the channel_selector option.