Fine-tuning and Pretrain for the Japanese Language

[kadirnar]

I'm fine-tuning the Orpheus-TTS model to add Japanese language support. The results are bad because I set the context_length value to 8192. I will retrain it after setting this value to 2048. Please help me find quality datasets if you know Japanese.

Datasets:

• https://huggingface.co/datasets/Sin2pi/JA_audio_JA_text_180k_samples (training continues)
• https://huggingface.co/datasets/kadirnar/StoryTTS

Model:

• https://huggingface.co/kadirnar/Orpheus-StoryTTS-JA

[webbigdata-jp]

Hello.
I am currently pre-training the Japanese version.

I have checked your model.

1. kadirnar/Orpheus-StoryTTS-JA
   -> It doesn't sound Japanese.
2. kadirnar/Orpheus-TTS-Starrail
   -> It's a bit noisy, but it's Japanese.

By the way, I have seen various your Japanese models.
Is there a reason why you are making a model for each dataset?

And how do you plan to proceed in the future?
I am planning to finetune after pre-training is finished, but I am still thinking about what kind of dataset to use for finetune.
I have not been able to find a Japanese speech dataset with tags or emotions.
Would you like to collaborate in some way?

[kadirnar]

Hello. I am currently pre-training the Japanese version.

I have checked your model.

1. kadirnar/Orpheus-StoryTTS-JA
   -> It doesn't sound Japanese.
2. kadirnar/Orpheus-TTS-Starrail
   -> It's a bit noisy, but it's Japanese.

By the way, I have seen various your Japanese models. Is there a reason why you are making a model for each dataset?

And how do you plan to proceed in the future? I am planning to finetune after pre-training is finished, but I am still thinking about what kind of dataset to use for finetune. I have not been able to find a Japanese speech dataset with tags or emotions. Would you like to collaborate in some way?

I stopped fine-tuning. I'm doing pretraining now. I got good results with some datasets. However, I'm preparing a very large dataset to do better. But I'm having a bit of difficulty because I don't know Japanese.

• https://voca.ro/16oqkjkaohMW
• https://voca.ro/1keiAXK7hufm
• https://voca.ro/1kMmp6R7HaSh

[webbigdata-jp]

@kadirnar
The quality of your model is as good as my current pretrained model, so I think it's promising.

• https://vocaroo.com/16hiHyyhDgfZ
• https://vocaroo.com/16h4R9pnAcHY
• https://vocaroo.com/1meXku3i57uR

However, I feel like there's not much point in two people doing the same thing.
I rent a public GPU, so it costs money. If you have a GPU that you can use freely, I would like to stop training and leave it to you.
In that case, I can help you check the dataset, but what kind of help do you need?

[kadirnar]

@webbigdata-jp I'm using 8xH100 for pretraining. It's very costly. And I'm not considering open source release yet since the quality still isn't good enough. I think I need a minimum of 5000 hours of data for pretraining. Therefore, developing a new model will take a long time.

[webbigdata-jp]

@kadirnar
Okay. Good luck.

It's true that it's hard because of the GPU cost and the disk space. It's impressive that you're targeting many languages.

[webbigdata-jp]

Although there were many difficulties, I have completed a high-quality Japanese TTS model, VoiceCore.
Thank you to Orpheus and SNAC for the base models, and to kadirnar for joining in the discussion.

[kadirnar]

@amuvarma13 Would having NSFW content in the dataset negatively affect the model's naturalness? I found a 500-hour dataset, but it contains NSFW data. Should I use this? My goal is to develop a TTS model for everyday conversation. I'm asking about pretrain.

[amuvarma13]

I don't think it would - in fact nsfw is probably more expressive from an acoustic perspective so I could I even see an argument for it leading to higher quality results.