Support multilabel metrics

[albertvillanova]

Currently, multilabel metrics are not supported because predictions and references are defined as Value("int32").

This PR creates a new feature type OptionalSequence which can act as either Value("int32") or Sequence(Value("int32")), depending on the data passed.

Close #2554.

[lhoestq]

Hi ! Thanks for the fix :)

If I understand correctly, OptionalSequence doesn't have an associated arrow type that we know in advance unlike the other feature types, because it depends on the type of the examples.

For example, I tested this and it raises an error:

import datasets as ds
import pyarrow as pa

features = ds.Features({"a": ds.features.OptionalSequence(ds.Value("int32"))})
batch = {"a": [[0]]}

writer = ds.ArrowWriter(features=features, stream=pa.BufferOutputStream())
writer.write_batch(batch)
# ArrowInvalid: Could not convert [0] with type list: tried to convert to int

This error happens because features.type is StructType(struct<a: int32>).

Another way to add support for multilabel would be to have several configurations for these metrics. By default it would set the features without sequences, and for the multi label configuration it would use features with sequences. Let me know what you think

[albertvillanova]

Hi @lhoestq, thanks for your feedback :)

Definitely, your suggested approach is simpler. I am going to refactor all my PR unless we could envision some other use cases where an OptionalSequence might be convenient, but for now I can't think of any...

[lhoestq]

Great thanks :)

[fcakyon]

@albertvillanova @lhoestq I couldnt find the related docs in F1 card: https://huggingface.co/spaces/evaluate-metric/f1

How do I perform multilabel F1 evaluation using evaluate package?

[albertvillanova]

I was going to transfer your question to the evaluate GitHub repository, but I saw you have already done it (and even opened a PR):

• How to calculate multi-label F1 score? evaluate#219
• add multilabel f1 eval usage evaluate#221

Thanks, @fcakyon.

[fcakyon]

Sorry to bomb you on multiple channels 😅 @albertvillanova, I have solved my problems, and opened a PR so that others also don't get confused 👍