[Arrow writer, Trivia_qa] Could not convert TagMe with type str: converting to null type

[patrickvonplaten]

Running the following code

import nlp
ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
ds.map(lambda x: x, load_from_cache_file=False)

triggers a ArrowInvalid: Could not convert TagMe with type str: converting to null type error.

On the other hand if we remove a certain column of trivia_qa which seems responsible for the bug, it works:

import nlp
ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
ds.map(lambda x: x, remove_columns=["entity_pages"], load_from_cache_file=False)

. Seems quite hard to debug what's going on here... @lhoestq @thomwolf - do you have a good first guess what the problem could be?

Note BTW: I think this could be a good test to check that the datasets work correctly: Take a tiny portion of the dataset and check that it can be written correctly.

[patrickvonplaten]

Here the full error trace:

ArrowInvalid                              Traceback (most recent call last)
<ipython-input-1-7aaf3f011358> in <module>
      1 import nlp
      2 ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
----> 3 ds.map(lambda x: x, load_from_cache_file=False)

~/python_bin/nlp/arrow_dataset.py in map(self, function, with_indices, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, arrow_schema, disable_nullable)
    549
    550         if update_data:
--> 551             writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file
    552
    553             # Create new Dataset from buffer or file

~/python_bin/nlp/arrow_writer.py in finalize(self, close_stream)
    182     def finalize(self, close_stream=True):
    183         if self.pa_writer is not None:
--> 184             self.write_on_file()
    185             self.pa_writer.close()
    186         if close_stream:

~/python_bin/nlp/arrow_writer.py in write_on_file(self)
    104         """
    105         if self.current_rows:
--> 106             pa_array = pa.array(self.current_rows, type=self._type)
    107             first_example = pa.array(self.current_rows[0:1], type=self._type)[0]
    108             # Sanity check

~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

~/hugging_face/venv_3.7/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Could not convert TagMe with type str: converting to null type

[patrickvonplaten]

Actually thinking a bit more about it, it's probably a data sample that is not correct in trivia_qa. But I'm a bit surprised though that we managed to write it in .arrow format and now cannot write it anymore after an "identity" mapping.

[lhoestq]

I don't have this error :x

[patrickvonplaten]

Interesting, maybe I have a very old cache of trivia_qa...thanks for checking

[lhoestq]

I'm running it right now on colab to double check

[patrickvonplaten]

Actually, I know what the problem is...I'm quite sure it's a bug. Here we take some test inputs: https://github.com/huggingface/nlp/blob/0e0ef12c14d2175e0b0bd7d8aa814b09e2cd7e1f/src/nlp/arrow_dataset.py#L472

It might be that in the test inputs, a Sequence type value is an emtpy list. So in my case I have ds[0]["entity_pages']["wiki_context"] = []. => this leads to an arrow_schema equal to null for ["entity_pages']["wiki_context"] => see line: https://github.com/huggingface/nlp/blob/0e0ef12c14d2175e0b0bd7d8aa814b09e2cd7e1f/src/nlp/arrow_dataset.py#L501 instead of list of string which it should for other examples.

Guess it's an edge case, but it can happen.

[thomwolf]

Good point, I think the schema should be infered at the writing stage where we have a writer_batch_size number of examples (typically 10k) so it's even less likely to run into this scenario.