Possibly index out of range bug

[azziko]

Hello,

I was refering to your project as a reference for my implementation of canary with streamAtt, and largely adopting the code from the repo. In my tests I have noticed that when PunctuationTextHistory is used, it is possible that the first input ends with a punctuation and gets discarded from the history. Subsequently, in the _update_speech... it is possible that cross_attn.shape[0] = 0, implying index out of range at https://github.com/hlt-mt/simulstream/blob/1f8b3337c5222995166555bfb36aa3fe69b94409/simulstream/server/speech_processors/base_streamatt.py#L168C1-L168C58. It just might be a fault in my implementation specifically.

Looking at the steamAtt Seamless implementation(I didn't verify this with tests), I also see that this is a plausible option.

I would appreaciate it if you confirmed or denied, so I can see where I'm making a mistake.

Thank you

[mgaido91]

Thanks for reporting this @azziko. I agree with your analysis, but interestingly we have never faced this issue. May you share your test so that we can run it on our code?

[azziko]

Hello @mgaido91,

In my _generate implementation, using "nvidia/canary-1b-v2", I do:

    def _generate(self, speech: torch.Tensor) -> Tuple[List[str], torch.Tensor]:
        new_hypothesis = self.model.transcribe(
            speech, 
            override_config=self.multitask_transcription_conf
        )
        
        return new_hypothesis[0].tokens, new_hypothesis[0].xatt_scores[self.cross_attn_layer].mean(dim=0)

Note: I realize that the text_history is never injected in the decoder here, but for the first call I believe that's irrelevant since there is no previous history in the first call. The decoder_input_ids are introduced later inside from the model's cfg from what I have tracked.

I'm also looking for ways to combine a bit lower level API to control the decoder's input, but still keeping it simple. If you could point me with a reference to something similar, I would appreciate it too.

My decoding strategy:

strategy: beam
    compute_hypothesis_token_set: true
    preserve_alignments: true
    confidence_cfg:
      preserve_frame_confidence: false
      preserve_token_confidence: false
      preserve_word_confidence: false
      exclude_blank: true
      aggregation: min
      tdt_include_duration: false
      method_cfg:
        name: entropy
        entropy_type: tsallis
        alpha: 0.33
        entropy_norm: exp
        temperature: DEPRECATED
    compute_langs: false
    greedy:
      temperature: null
      max_generation_delta: -1
      preserve_alignments: false
      preserve_token_confidence: false
      confidence_method_cfg:
        name: entropy
        entropy_type: tsallis
        alpha: 0.33
        entropy_norm: exp
        temperature: DEPRECATED
      n_samples: 1
    beam:
      beam_size: 5
      search_type: default
      len_pen: 1.0
      max_generation_delta: -1
      return_best_hypothesis: true
      preserve_alignments: true
      ngram_lm_model: null
      ngram_lm_alpha: 0.0
      boosting_tree:
        model_path: null
        key_phrases_file: null
        key_phrases_list: null
        key_phrase_items_list: null
        context_score: 1.0
        depth_scaling: 1.0
        unk_score: 0.0
        final_eos_score: 1.0
        score_per_phrase: 0.0
        source_lang: en
        use_triton: true
        uniform_weights: false
        use_bpe_dropout: false
        num_of_transcriptions: 5
        bpe_alpha: 0.3
      boosting_tree_alpha: 0.0
    temperature: 1.0
    return_xattn_scores: true

[mgaido91]

I'm also looking for ways to combine a bit lower level API to control the decoder's input, but still keeping it simple. If you could point me with a reference to something similar, I would appreciate it too.

I am not aware either, it was in my TODO list after the cross-attention got merged in NeMo but unfortunately in this period I am busy with other topics so had no time to look at that. I see there is a method to pass a prompt but not sure if that works/is feasible for forcing the decoder_input_ids.

BTW, in case you find one and complete your canary streamatt implementation, if you want to submit a PR to this codebase I'd be more than happy and grateful to review and merge.

Getting back to the original topic of the issue, we will check if we can force the potential issue you mentioned to happen in the next days and keep you posted, thanks.

[sarapapi]

Hi @azziko, what you mentioned here is indeed an incorrect behavior of the code. I've just submitted a PR to correct it. Thanks a lot for spotting it!

[mgaido91]

FIxed in #18 .

[azziko]

Hello @sarapapi @mgaido91 ,

Thank you for the quick fix and replies!

I am not aware either, it was in my TODO list after the cross-attention got merged in NeMo but unfortunately in this period I am busy with other topics so had no time to look at that. I see there is a method to pass a prompt but not sure if that works/is feasible for forcing the decoder_input_ids.

@mgaido91 I opened an issue regarding injecting decoder_input_ids, but when I look at the code now, the prompt you mentioned seems to indeed support the previous context. See: https://github.com/NVIDIA-NeMo/NeMo/blob/0ae9f88b19479db12605d1bd0889ce4a1f4c0f20/nemo/collections/common/prompts/canary2.py#L55

I'm not sure, however, how crucial is the fact that in the input sequence to the decoder it comes before |BOS| and if forcing order is necessary. I will try playing with it and if I succeed I will be happy to help with a PR.