MagpieTTS decoder model on top of NeMo main branch

[paarthneekhara]

No description provided.

[blisc]

Have we tested the non-Lhotse path?

[blisc]

Some more comments from WIP review

[blisc]

Can you add to this as to how this differs from magpietts_inference.py?

[blisc]

Should we subclass Tokenizer instead of instantiation within the class?

[paarthneekhara]

Subclassing Tokenizer may not be the best idea because we will have to reassign some internal methods/params to self for it to work correctly. If we call from_file() from the init function, we'll have to reassign normalizer, pre_tokenizer, post_processor and decoder of the loaded model to self. I am moving the imports to the top. It would look something like this if we were to subclass Tokenizer.

class IPABPETokenizer(Tokenizer):
    """Simple IPA BPE tokenizer subclassing HuggingFace tokenizers.Tokenizer.

    Args:
        tokenizer_path: Path to the tokenizer.json file (or directory containing it).
    """

    def __init__(self, tokenizer_path: str):
        if os.path.isdir(tokenizer_path):
            tokenizer_file = os.path.join(tokenizer_path, "tokenizer.json")
        else:
            tokenizer_file = tokenizer_path

        if not os.path.exists(tokenizer_file):
            raise ValueError(f"Tokenizer file not found: {tokenizer_file}")

        loaded = Tokenizer.from_file(tokenizer_file)
        super().__init__(loaded.model)
        self.normalizer = loaded.normalizer
        self.pre_tokenizer = loaded.pre_tokenizer
        self.post_processor = loaded.post_processor
        self.decoder = loaded.decoder

        self.tokens = self.get_vocab()
        self.pad = self.tokens.get("<pad>", None)

    def encode(self, text: str) -> List[int]:
        """Encode IPA text to token IDs."""
        return super().encode(text).ids

    def decode(self, tokens: List[int]) -> str:
        """Decode token IDs back to IPA text."""
        return super().decode(tokens)

[blisc]

Does this affect existing MagpieTTS checkpoints?

[paarthneekhara]

@shehzeen Can you check this.

[shehzeen]

MagpieTTS should work the same way. Nemotron tokenizer has pad_token_id as None, so we are using the unk_token_id as the pad_token_id in EasyMagpie. In Magpie tokenizers, tokenizer.pad_token_id is not None (if it's ever None, the tokenizer setup would error out in the old code). So the code functionality should stay the same for MagpieTTS since the tokenizers have the pad_token_id.

[blisc]

FYI: @rlangman @rfejgin since we were talking about this. Let's lift this from the decoder PR and move it to main early

[blisc]

I'm not sure when you call this function, but this should be part of the tokenizer class not a util function in the dataset.

[blisc]

However, this only exists in the Lhotse file but not the non-Lhotse file?

[blisc]

This is a really large file. Can we split it up? Some suggestions

• Anything that's common with Encoder-Decoder Magpie, let's move to a separate base class:
  • The code manipulaiton functions
  • The local transformer functions
  • etc
• Let's move the dataclasses to another file, although we can debate this
• Let's move worker_init_fn too since it should be common to both models
• Could consider splitting training and inference into two classes as well

[blisc]

Switch print statments to logging

[blisc]

This function does not depend on self. Consider moving all helper print functions into a separate file and call them within the model instead of defining additional class functions