Bug description
If we change the default value from false to true, Training FP or Vocoder will crash for use_weighted_sampler
config/everyvoice-spec-to-wav.yaml: use_weighted_sampler: true
config/everyvoice-text-to-spec.yaml: use_weighted_sampler: true
I tried with various multi-speaker , multi-language, single speaker etc and getting the same message no matter what.
Side note: I wonder if we can have better document about this feature or en example as to why this can be usefull to the end users. The schema says this:
"use_weighted_sampler": {
"default": false,
"description": "Whether to use a sampler which oversamples from the minority language or speaker class for balanced training.",
"title": "Use Weighted Sampler",
"type": "boolean"
How to reproduce the bug
Edit below config :
config/everyvoice-spec-to-wav.yaml: use_weighted_sampler: true
config/everyvoice-text-to-spec.yaml: use_weighted_sampler: true
then run this on a GPU node to train an FP or Vocoder , both will not work when set to true :
everyvoice train text-to-spec --devices 1 --nodes 1 config/everyvoice-text-to-spec.yaml
or
everyvoice train spec-to-wav --devices 1 --nodes 1 config/everyvoice-spec-to-wav.yaml
Error messages and logs
[rank0]: ╭───────────────────── Traceback (most recent call last) ──────────────────────╮
[rank0]: │ /gpfs/fs5/nrc/nrc-fs1/ict/others/u/tes001/TxT2SPEECH/EveryVoice_2025-05-21/e │
[rank0]: │ veryvoice/model/feature_prediction/FastSpeech2_lightning/fs2/cli/train.py:33 │
[rank0]: │ in train │
[rank0]: │ │
[rank0]: │ 30 │ │
[rank0]: │ 31 │ model_kwargs = {"lang2id": lang2id, "speaker2id": speaker2id, "stat │
[rank0]: │ 32 │ │
[rank0]: │ ❱ 33 │ train_base_command( │
[rank0]: │ 34 │ │ model_config=FastSpeech2Config, │
[rank0]: │ 35 │ │ model=FastSpeech2, │
[rank0]: │ 36 │ │ data_module=FastSpeech2DataModule, │
... ( Truncated)
[rank0]: │ 50 │ │ elif isinstance(dataset, torch.utils.data.TensorDataset): │
[rank0]: │ 51 │ │ │ return dataset.tensors[1] │
[rank0]: │ 52 │ │ elif isinstance(dataset, torch.utils.data.Dataset): │
[rank0]: │ ❱ 53 │ │ │ return dataset.get_labels() │
[rank0]: │ 54 │ │ else: │
[rank0]: │ 55 │ │ │ raise NotImplementedError │
[rank0]: │ 56 │
[rank0]: │ │
[rank0]: │ /gpfs/fs5/nrc/nrc-fs1/ict/others/u/tes001/TxT2SPEECH/EveryVoice_2025-05-21/e │
[rank0]: │ veryvoice/model/feature_prediction/FastSpeech2_lightning/fs2/dataset.py:210 │
[rank0]: │ in get_labels │
[rank0]: │ │
[rank0]: │ 207 │ │ return len(self.dataset) │
[rank0]: │ 208 │ │
[rank0]: │ 209 │ def get_labels(self): │
[rank0]: │ ❱ 210 │ │ return [x["label"] for x in self.dataset] │
[rank0]: │ 211 │
[rank0]: │ 212 │
[rank0]: │ 213 class FastSpeech2DataModule(BaseDataModule): │
[rank0]: │ │
[rank0]: │ /gpfs/fs5/nrc/nrc-fs1/ict/others/u/tes001/TxT2SPEECH/EveryVoice_2025-05-21/e │
[rank0]: │ veryvoice/model/feature_prediction/FastSpeech2_lightning/fs2/dataset.py:210 │
[rank0]: │ in <listcomp> │
[rank0]: │ │
[rank0]: │ 207 │ │ return len(self.dataset) │
[rank0]: │ 208 │ │
[rank0]: │ 209 │ def get_labels(self): │
[rank0]: │ ❱ 210 │ │ return [x["label"] for x in self.dataset] │
[rank0]: │ 211 │
[rank0]: │ 212 │
[rank0]: │ 213 class FastSpeech2DataModule(BaseDataModule): │
[rank0]: ╰──────────────────────────────────────────────────────────────────────────────╯
[rank0]: KeyError: 'label'
Loading EveryVoice modules: 100%|██████████| 4/4 [00:04<00:00, 1.06s/it]
srun: error: ib14gpu-001: task 0: Exited with exit code 1
Environment
Current environment
(EveryVoice_2025-05-21) [U24-GPSC5]:$ everyvoice --diagnostic
EveryVoice Diagnostic information
EveryVoice version: 0.3.0
Python version: 3.10.17 | packaged by conda-forge | (main, Apr 10 2025, 22:19:12) [GCC 13.3.0]
System: Linux 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64
Environment type: conda
Modules installed using conda:
packages in environment at /gpfs/fs5/nrc/nrc-fs1/ict/others/u/tes001/miniforge3/envs/EveryVoice_2025-05-21:
Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
alsa-lib 1.2.14 hb9d3cd8_0 conda-forge
aom 3.9.1 hac33072_0 conda-forge
attr 2.5.1 h166bdaf_1 conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
ca-certificates 2025.4.26 hbd8a1cb_0 conda-forge
cairo 1.18.4 h3394656_0 conda-forge
dav1d 1.2.1 hd590300_0 conda-forge
dbus 1.13.6 h5008d03_3 conda-forge
expat 2.7.0 h5888daf_0 conda-forge
ffmpeg 7.1.1 gpl_h127656b_905 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 h77eed37_3 conda-forge
fontconfig 2.15.0 h7e30c49_1 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.13.3 ha770c72_1 conda-forge
fribidi 1.0.10 h36c2ea0_0 conda-forge
gdk-pixbuf 2.42.12 hb9ae30d_0 conda-forge
gettext 0.24.1 h5888daf_0 conda-forge
gettext-tools 0.24.1 h5888daf_0 conda-forge
gmp 6.3.0 hac33072_2 conda-forge
graphite2 1.3.13 h59595ed_1003 conda-forge
harfbuzz 11.2.1 h3beb420_0 conda-forge
icu 75.1 he02047a_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
ld_impl_linux-64 2.43 h712a8e2_4 conda-forge
lerc 4.0.0 h0aef613_1 conda-forge
level-zero 1.22.1 h84d6215_0 conda-forge
libabseil 20250127.1 cxx17_hbbce691_0 conda-forge
libasprintf 0.24.1 h8e693c7_0 conda-forge
libasprintf-devel 0.24.1 h8e693c7_0 conda-forge
libass 0.17.3 h52826cd_2 conda-forge
libcap 2.75 h39aace5_0 conda-forge
libdeflate 1.24 h86f0d12_0 conda-forge
libdrm 2.4.124 hb9d3cd8_0 conda-forge
libegl 1.7.0 ha4b6fd6_2 conda-forge
libexpat 2.7.0 h5888daf_0 conda-forge
libffi 3.4.6 h2dba641_1 conda-forge
libflac 1.4.3 h59595ed_0 conda-forge
libfreetype 2.13.3 ha770c72_1 conda-forge
libfreetype6 2.13.3 h48d6fc4_1 conda-forge
libgcc 15.1.0 h767d61c_2 conda-forge
libgcc-ng 15.1.0 h69a702a_2 conda-forge
libgcrypt-lib 1.11.1 hb9d3cd8_0 conda-forge
libgettextpo 0.24.1 h5888daf_0 conda-forge
libgettextpo-devel 0.24.1 h5888daf_0 conda-forge
libgl 1.7.0 ha4b6fd6_2 conda-forge
libglib 2.84.1 h3618099_1 conda-forge
libglvnd 1.7.0 ha4b6fd6_2 conda-forge
libglx 1.7.0 ha4b6fd6_2 conda-forge
libgomp 15.1.0 h767d61c_2 conda-forge
libgpg-error 1.55 h3f2d84a_0 conda-forge
libhwloc 2.11.2 default_h0d58e46_1001 conda-forge
libiconv 1.18 h4ce23a2_1 conda-forge
libjpeg-turbo 3.1.0 hb9d3cd8_0 conda-forge
liblzma 5.8.1 hb9d3cd8_1 conda-forge
liblzma-devel 5.8.1 hb9d3cd8_1 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libogg 1.3.5 hd0c01bc_1 conda-forge
libopenvino 2025.0.0 hdc3f47d_3 conda-forge
libopenvino-auto-batch-plugin 2025.0.0 h4d9b6c2_3 conda-forge
libopenvino-auto-plugin 2025.0.0 h4d9b6c2_3 conda-forge
libopenvino-hetero-plugin 2025.0.0 h981d57b_3 conda-forge
libopenvino-intel-cpu-plugin 2025.0.0 hdc3f47d_3 conda-forge
libopenvino-intel-gpu-plugin 2025.0.0 hdc3f47d_3 conda-forge
libopenvino-intel-npu-plugin 2025.0.0 hdc3f47d_3 conda-forge
libopenvino-ir-frontend 2025.0.0 h981d57b_3 conda-forge
libopenvino-onnx-frontend 2025.0.0 h0e684df_3 conda-forge
libopenvino-paddle-frontend 2025.0.0 h0e684df_3 conda-forge
libopenvino-pytorch-frontend 2025.0.0 h5888daf_3 conda-forge
libopenvino-tensorflow-frontend 2025.0.0 h684f15b_3 conda-forge
libopenvino-tensorflow-lite-frontend 2025.0.0 h5888daf_3 conda-forge
libopus 1.5.2 hd0c01bc_0 conda-forge
libpciaccess 0.18 hd590300_0 conda-forge
libpng 1.6.47 h943b412_0 conda-forge
libprotobuf 5.29.3 h501fc15_1 conda-forge
librsvg 2.58.4 he92a37e_3 conda-forge
libsndfile 1.2.2 hc60ed4a_1 conda-forge
libsqlite 3.49.2 hee588c1_0 conda-forge
libstdcxx 15.1.0 h8f9b012_2 conda-forge
libstdcxx-ng 15.1.0 h4852527_2 conda-forge
libsystemd0 257.4 h4e0b6ca_1 conda-forge
libtiff 4.7.0 hf01ce69_5 conda-forge
libudev1 257.4 hbe16f8c_1 conda-forge
libunwind 1.6.2 h9c3ff4c_0 conda-forge
liburing 2.9 h84d6215_0 conda-forge
libusb 1.0.28 h73b1eb8_1 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libva 2.22.0 h4f16b4b_2 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libvpx 1.14.1 hac33072_0 conda-forge
libwebp-base 1.5.0 h851e524_0 conda-forge
libxcb 1.17.0 h8a09558_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libxkbcommon 1.9.2 h65c71a3_0 conda-forge
libxml2 2.13.8 h4bc477f_0 conda-forge
libzlib 1.3.1 hb9d3cd8_2 conda-forge
lz4-c 1.10.0 h5888daf_1 conda-forge
mad 0.15.1b h9c3ff4c_1 conda-forge
mpg123 1.32.9 hc50e24c_0 conda-forge
ncurses 6.5 h2d0b736_3 conda-forge
ocl-icd 2.3.3 hb9d3cd8_0 conda-forge
opencl-headers 2024.10.24 h5888daf_0 conda-forge
openh264 2.6.0 hc22cd8d_0 conda-forge
openssl 3.5.0 h7b32b05_1 conda-forge
pango 1.56.3 h9ac818e_1 conda-forge
pcre2 10.45 hc749103_0 conda-forge
pip 25.1.1 pyh8b19718_0 conda-forge
pixman 0.46.0 h29eaf8c_0 conda-forge
pthread-stubs 0.4 hb9d3cd8_1002 conda-forge
pugixml 1.15 h3f63f65_0 conda-forge
pulseaudio-client 17.0 hac146a9_1 conda-forge
python 3.10.17 hd6af730_0_cpython conda-forge
readline 8.2 h8c095d6_2 conda-forge
sdl2 2.32.54 h3f2d84a_0 conda-forge
sdl3 3.2.14 he3e324a_0 conda-forge
setuptools 80.8.0 pyhff2d567_0 conda-forge
snappy 1.2.1 h8bd8927_1 conda-forge
sox 14.4.2 h59a48fd_1020 conda-forge
svt-av1 3.0.2 h5888daf_0 conda-forge
tbb 2022.1.0 h4ce085d_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
wayland 1.23.1 h3e06ad9_1 conda-forge
wayland-protocols 1.43 hd8ed1ab_0 conda-forge
wheel 0.45.1 pyhd8ed1ab_1 conda-forge
x264 1!164.3095 h166bdaf_2 conda-forge
x265 3.5 h924138e_3 conda-forge
xkeyboard-config 2.44 hb9d3cd8_0 conda-forge
xorg-libice 1.1.2 hb9d3cd8_0 conda-forge
xorg-libsm 1.2.6 he73a12e_0 conda-forge
xorg-libx11 1.8.12 h4f16b4b_0 conda-forge
xorg-libxau 1.0.12 hb9d3cd8_0 conda-forge
xorg-libxcursor 1.2.3 hb9d3cd8_0 conda-forge
xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge
xorg-libxext 1.3.6 hb9d3cd8_0 conda-forge
xorg-libxfixes 6.0.1 hb9d3cd8_0 conda-forge
xorg-libxrender 0.9.12 hb9d3cd8_0 conda-forge
xorg-libxscrnsaver 1.2.4 hb9d3cd8_0 conda-forge
zlib 1.3.1 hb9d3cd8_2 conda-forge
zstd 1.5.7 hb8e6e7a_2 conda-forge
torch modules installed using pip:
pytorch-lightning==2.5.1.post0
torch==2.3.1+cu118
torchaudio==2.3.1+cu118
torchinfo==1.8.0
torchmetrics==1.7.1
Other modules installed using pip:
absl-py==2.2.2
aiofiles==24.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.11.18
aiosignal==1.3.2
annotated-types==0.7.0
anyio==4.9.0
anytree==2.13.0
arrow==1.3.0
async-timeout==5.0.1
attrs==25.3.0
audioread==3.0.1
black==24.10.0
certifi==2025.4.26
cffi==1.17.1
cfgv==3.4.0
charset-normalizer==3.4.2
chevron==0.14.0
click==8.1.8
clipdetect==0.1.4
coloredlogs==15.0.1
contourpy==1.3.2
coverage==7.8.1
cycler==0.12.1
decorator==5.2.1
deepdiff==8.5.0
distlib==0.3.9
dnspython==2.7.0
docopt==0.6.2
editdistance==0.8.1
einops==0.5.0
email_validator==2.2.0
et_xmlfile==2.0.0
-e git+https://github.com/roedoejet/EveryVoice.git@9edbcad0061db0db216779c8f54a277af7eabca1#egg=EveryVoice
exceptiongroup==1.3.0
fastapi==0.115.12
ffmpy==0.5.0
filelock==3.18.0
flake8==7.2.0
fonttools==4.58.0
frozenlist==1.6.0
fsspec==2025.5.0
g2p==2.2.1
gitlint-core==0.19.1
gradio==5.30.0
gradio_client==1.10.1
grapheme==0.6.0
groovy==0.1.2
grpcio==1.71.0
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
huggingface-hub==0.31.4
humanfriendly==10.0
identify==2.6.10
idna==3.10
iniconfig==2.1.0
ipatok==0.4.2
isort==6.0.1
Jinja2==3.1.6
joblib==1.5.0
jsonschema==4.23.0
jsonschema-specifications==2025.4.1
kiwisolver==1.4.8
librosa==0.9.2
lightning==2.5.1.post0
lightning-utilities==0.14.3
llvmlite==0.44.0
loguru==0.6.0
lxml==5.4.0
Markdown==3.8
markdown-it-py==3.0.0
MarkupSafe==3.0.2
matplotlib==3.9.4
mccabe==0.7.0
mdurl==0.1.2
merge-args==0.1.5
mpmath==1.3.0
multidict==6.4.4
munkres==1.1.4
mypy==1.15.0
mypy_extensions==1.1.0
networkx==3.4.2
nltk==3.9.1
nodeenv==1.9.1
numba==0.61.2
numpy==1.26.4
nvidia-cublas-cu11==11.11.3.6
nvidia-cuda-cupti-cu11==11.8.87
nvidia-cuda-nvrtc-cu11==11.8.89
nvidia-cuda-runtime-cu11==11.8.89
nvidia-cudnn-cu11==8.7.0.84
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.3.0.86
nvidia-cusolver-cu11==11.4.1.48
nvidia-cusparse-cu11==11.7.5.86
nvidia-nccl-cu11==2.20.5
nvidia-nvtx-cu11==11.8.86
openpyxl==3.1.5
ordered-set==4.1.0
orderly-set==5.4.1
orjson==3.10.18
packaging==24.2
pandas==2.2.3
panphon==0.20.0
pathspec==0.12.1
pep440==0.1.2
pillow==11.2.1
platformdirs==4.3.8
pluggy==1.6.0
pooch==1.8.2
pre_commit==4.2.0
prompt_toolkit==3.0.51
pronunciation-dictionary==0.0.6
propcache==0.3.1
protobuf==4.25.7
pycodestyle==2.13.0
pycountry==22.3.5
pycparser==2.22
pydantic==2.7.4
pydantic_core==2.18.4
pydub==0.25.1
pyflakes==3.3.2
Pygments==2.19.1
pympi-ling==1.70.2
pyparsing==3.2.3
pysdtw==0.0.5
pytest==7.4.4
python-dateutil==2.9.0.post0
python-multipart==0.0.20
python-slugify==5.0.0
pytz==2025.2
pyworld-prebuilt==0.3.4.4
PyYAML==6.0.2
questionary==1.10.0
readalongs==1.2.1
referencing==0.36.2
regex==2024.11.6
requests==2.32.3
resampy==0.4.3
rich==14.0.0
rpds-py==0.25.1
ruff==0.11.10
safehttpx==0.1.6
scikit-learn==1.6.1
scipy==1.15.3
semantic-version==2.10.0
sh==2.2.2
shellingham==1.5.4
simple-term-menu==1.5.2
six==1.17.0
sniffio==1.3.1
soundfile==0.13.1
soundswallower==0.6.5
starlette==0.46.2
sympy==1.14.0
tabulate==0.9.0
tensorboard==2.19.0
tensorboard-data-server==0.7.2
termcolor==2.3.0
text-unidecode==1.3
TextGrid==1.6.1
textgrid-tools==0.0.8
threadpoolctl==3.6.0
tomli==2.2.1
tomlkit==0.13.2
tqdm==4.67.1
triton==2.3.1
typer==0.15.4
types-python-dateutil==2.9.0.20250516
types-PyYAML==6.0.12.20250516
types-requests==2.32.0.20250515
types-setuptools==80.8.0.20250521
types-tabulate==0.9.0.0
types-tqdm==4.67.0.20250516
typing_extensions==4.13.2
tzdata==2025.2
unicodecsv==0.14.1
urllib3==2.4.0
uvicorn==0.34.2
virtualenv==20.31.2
wcwidth==0.2.13
websockets==15.0.1
webvtt-py==0.4.2
Werkzeug==3.1.3
yarl==1.20.0
yaspin==3.1.0
More info
No response
Bug description
If we change the default value from
falsetotrue, Training FP or Vocoder will crash foruse_weighted_samplerconfig/everyvoice-spec-to-wav.yaml: use_weighted_sampler: true
config/everyvoice-text-to-spec.yaml: use_weighted_sampler: true
I tried with various multi-speaker , multi-language, single speaker etc and getting the same message no matter what.
Side note: I wonder if we can have better document about this feature or en example as to why this can be usefull to the end users. The schema says this:
How to reproduce the bug
Edit below config :
then run this on a GPU node to train an FP or Vocoder , both will not work when set to
true:everyvoice train text-to-spec --devices 1 --nodes 1 config/everyvoice-text-to-spec.yamlor
everyvoice train spec-to-wav --devices 1 --nodes 1 config/everyvoice-spec-to-wav.yamlError messages and logs
Environment
Current environment
More info
No response