Dataset Viewer issue for indonesian-nlp/librivox-indonesia

[cahya-wirawan]

Link

https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia

Description

I created a new speech dataset https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia, but the dataset preview doesn't work with following error message:

Server error
Status code:   400
Exception:     TypeError
Message:       unsupported operand type(s) for +: 'NoneType' and 'str'

Please help, I am not sure what the problem here is. Thanks a lot.

Owner

Yes

[albertvillanova]

The error is not related to the dataset viewer. I'm having a look...

[cahya-wirawan]

Thanks @albertvillanova for checking the issue. Actually, I can use the dataset like following:

>>> from datasets import load_dataset
>>> ds=load_dataset("indonesian-nlp/librivox-indonesia")
No config specified, defaulting to: librivox-indonesia/all
Reusing dataset librivox-indonesia (/root/.cache/huggingface/datasets/indonesian-nlp___librivox-indonesia/all/1.0.0/9a934a42bfb53dc103003d191618443b8a786bea2bd7bb0bc2d9454b8494521e)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 500.87it/s]
>>> ds
DatasetDict({
    train: Dataset({
        features: ['path', 'language', 'reader', 'sentence', 'audio'],
        num_rows: 7815
    })
})
>>> ds["train"][0]
{'path': '/root/.cache/huggingface/datasets/downloads/extracted/c8ead52370fa28feb64643ea9d05cd7d820192dc8a1700d665ec45ec7624f5a3/librivox-indonesia/sundanese/universal-declaration-of-human-rights/human_rights_un_sun_brc_0000.mp3', 'language': 'sun', 'reader': '3174', 'sentence': 'pernyataan umum ngeunaan hak hak asasi manusa sakabeh manusa', 'audio': {'path': '/root/.cache/huggingface/datasets/downloads/extracted/c8ead52370fa28feb64643ea9d05cd7d820192dc8a1700d665ec45ec7624f5a3/librivox-indonesia/sundanese/universal-declaration-of-human-rights/human_rights_un_sun_brc_0000.mp3', 'array': array([ 0.        ,  0.        ,  0.        , ..., -0.02419001,
       -0.01957154, -0.01502833], dtype=float32), 'sampling_rate': 44100}}

It would be just nice if I also can see it using dataset viewer.

[albertvillanova]

Yes, the issue arises when streaming (that is used by the viewer): your script does not support streaming and to support it in this case there are some subtleties that we are explaining better in our docs in a work-in progress pull request:

• Docs for creating an audio dataset #4872

Just note that when streaming, local_extracted_archive is None, and this code line generates the error:

filepath = local_extracted_archive + "/librivox-indonesia/audio_transcription.csv"

For a proper implementation, you could have a look at: https://huggingface.co/datasets/common_voice/blob/main/common_voice.py

You can test your script locally by passing streaming=True to load_dataset:

ds = load_dataset("indonesian-nlp/librivox-indonesia", split="train", streaming=True); item = next(iter(ds)); item

[cahya-wirawan]

Great, I will have a look and update the script. Thanks.

[cahya-wirawan]

Hi @albertvillanova , I just add the streaming functionality and it works in the first try :-) Thanks a lot!

[albertvillanova]

Awesome!!! 🤗