python DeltaTable.schema().to_pyarrow() does not support data type `void`

[leo-schick]

Environment

pip datalake==0.14.0

Environment:

• Cloud provider: /
• OS: Linux

---

Bug

What happened:
I wanted to read the schema from a delta table in pyarrow format via

from deltalake import DeltaTable

file_uri = "..."
deltaTable = DeltaTable(file_uri, storage_options=...)
pyarrow_schema = deltaTable.schema().to_pyarrow()

It failed with error:

Traceback (most recent call last):
[...]
  File "my_python_file.py", line 63, in __
    pyarrow_schema = deltaTable.schema().to_pyarrow()
Exception: Schema error: Invalid data type for Arrow: void

What you expected to happen:
pyarrow has a null data type. I would expect that in case of a void type in delta, the pyarrow schema returns the column with null data type instead of throwing an exception.

How to reproduce it:

More details:

[leo-schick]

p.s i tested this with deltalake 0.11.0 and had there the same issue. So I guess this has never been implemented.

[ion-elgreco]

This is some info on void that I can find in the delta protocol definition:

Note: Existing tables may have void data type columns. Behavior is undefined for void data type columns but it is recommended to drop any void data type columns on reads (as is implemented by the Spark connector)

https://github.com/delta-io/delta/blob/master/PROTOCOL.md

[leo-schick]

@ion-elgreco The sentence from the protocol you mentioned seems to describe it for reading operations. Since we do not perform reading operations here, I think there are two options to fix this:

1. return a column with type void/null (as proposed above)
2. skip the column since it has data type void/null

For me, I do not care if the column is there or not. I just don't want to have it break when there exist a column with type void.

I assume that returning a column would be the right choice since somehow the delta table metadata includes the column - so - it should be returend I guess. Maybe an option skip columns of type void could be added for schema reading...

[roeap]

The delta protocol actually has something to say about void columns :)

Note: Existing tables may have void data type columns. Behavior is undefined for void data type columns but it is recommended to drop any void data type columns on reads (as is implemented by the Spark connector).

SO i guess to not throw any errors, we could drop all void fields ...