Fine-tuning for the Spanish Language

[kadirnar]

I want to add Spanish language to the Orpheus-TTS model. I need help with finding a quality dataset. I will modify the context_length value and retrain it to improve word pronunciation.

Texts:

• si tú fuiste el primero y más animoso en herir cuando los dioses mataron a fiase mis uñas y mis vencejos no te dejarán carecer de buenas advertencias
• vengo a verte pasar todos los días aporciten cantados siempre lejos tus ojos son dos rubios capitanes tu labio es un brevísimo pañuelo rojo que honde an uno dios de sangre
• entró efectivamente el tártaro con áspero continente al aviso de un paje de antecámaraacompañában le al lado paio gómez desotro mayor y hernán sánchez de pazuelos

Dataset: https://huggingface.co/datasets/freds0/cml_tts_dataset_spanish/
Model: https://huggingface.co/kadirnar/Orpheus-TTS-Cml-Spanish

Output:

• https://voca.ro/1guOGUMYoHG6
• https://voca.ro/1nvza5BmmY99
• https://voca.ro/18qbpv1Rp8X6

[giandiego]

Hi!

Saw you're looking for a quality Spanish dataset for Orpheus. I just prepared one focused on LATAM Spanish that might work well for you:

Dataset: https://huggingface.co/datasets/GianDiego/latam-spanish-speech-orpheus-tts-24khz

Quick highlights:

• LATAM Spanish: Various accents (AR, CL, PE, CO, VE, PR).
• Quality: Based on the high-quality Google/OpenSLR recordings.
• Orpheus-Ready: Already processed to 24kHz, 16-bit mono.
• Variety: Multiple male & female speakers per country.

The variety of speakers and accents might help with the pronunciation improvements you're aiming for.

Hope this helps with your retraining! Let me know how it turns out.

[kadirnar]

Can you make the dataset public?

[giandiego]

Yes, the dataset is fully public and available for anyone to use. You can access it directly at:

https://huggingface.co/datasets/GianDiego/latam-spanish-speech-orpheus-tts-24khz

[gianpaj]

@kadirnar Have you tried the model? canopylabs/3b-es_it-ft-research_release

This is how the first sentence sounds like with javi voice
https://uxjubqdyhv4aowsi.public.blob.vercel-storage.com/audio/javi-e3e14b44.wav

si tú fuiste el primero y más animoso en herir cuando los dioses mataron a fiase mis uñas y mis vencejos no te dejarán carecer de buenas advertencias

out.mp4

_{Generated with SexyVoice.ai (plug :)}

[C00reNUT]

Hello, may I ask you what kind of GPU is needed for finetuning the model? Is 24GB VRAM enough?

[Josebd98]

But what about the Spanish model? I think they have a pretrained model, don’t they? I’ve collected a large amount of audio for training: https://huggingface.co/datasets/sirekist98/spanish_Audios

I just want to know if the results using the pretrained model are good or not.

[ukemamaster]

But what about the Spanish model? I think they have a pretrained model, don’t they? I’ve collected a large amount of audio for training: https://huggingface.co/datasets/sirekist98/spanish_Audios

I just want to know if the results using the pretrained model are good or not.

Hi, any updates on this?

[Josebd98]

Hello! Yes, I fine-tuned the model with the dataset. The results are really good — you can try it here:
https://huggingface.co/spaces/sirekist98/orpheustts_spanish_tuned

You can use the model here: https://huggingface.co/sirekist98/orpheustts_spanish_finetuned

[ukemamaster]

Hi @Josebd98 ,
How is your fine tuned model performance?
I fine tuned one in Spanish, but i have the issue of skipping random words in the streaming inference provided by orpheus. I posted the issue here. Could you please review it and suggest how can i avoid this issue?
Which tokenizer should i use for inference after fine tuning, the base one? Or the fine tuned one?

[Josebd98]

Hi @ukemamaster

I actually use the base tokenizer for inference, even after fine-tuning. From my experience, it doesn’t really matter if you use the base tokenizer or a tokenizer reloaded from your fine-tuned model — since typically the tokenizer doesn’t change during fine-tuning in TTS workflows like Orpheus (only the acoustic or acoustic+language model is updated).

However, if your fine-tuning involved special tokens or changes to the vocab (which is uncommon for standard Orpheus TTS setups), then yes, you’d need to use the updated tokenizer. Otherwise, sticking to the base tokenizer is perfectly fine.