Better DuplicateKeysError error to help the user debug the issue

[lhoestq]

As mentioned in #2552 it would be nice to improve the error message when a dataset fails to build because there are duplicate example keys.

The current one is

datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 48
Keys should be unique and deterministic in nature

and we could have something that guides the user to debugging the issue:

DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.
Please fix the dataset script at <path/to/the/dataset/script>

[SupritYoung]

excuse me, my datasets version is 2.2.2, but I also just see the error info like

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature

[lhoestq]

Hi ! for which dataset do you have this error ?

Also note that this issue is just about improving the error message, which is not very friendly x)

[VijayKalmath]

@lhoestq I would like to take a hit at improving the error message. Will open a draft PR and will reach out to you for review

[VijayKalmath]

DuplicateKeysError: both 42th and 1337th examples have the same keys 48.

@lhoestq when you mention 42th and 1337th in the above case , are these values the examples' "id" or are they the examples' index ?

[lhoestq]

Hi ! Thanks @VijayKalmath :)

In the general case, examples don't have an "id" field, so I think it should correspond to the index

[VijayKalmath]

@lhoestq , I have opened a draft PR for this Issue.

I wanted to check with you if there is a way to get <path/to/the/dataset/script> currently or do I need to add extra code to find that.

If I need to find the script , I can assume that the generator function will always be in datasets/{dataset_name}/{dataset_name}.py.

[lhoestq]

Thanks !

I wanted to check with you if there is a way to get <path/to/the/dataset/script> currently or do I need to add extra code to find that.

You don't have access to this info inside the ArrowWriter unfortunately. This info is available in builder.py in the DatasetBuilder code that uses the ArrowWriter though, maybe a try-catch there can do the job