ldsantos0911 opened a new issue, #2119:
URL: https://github.com/apache/iceberg-python/issues/2119
### Question
```
import pyarrow as pa
schema = pa.schema(
[
pa.field("col1", pa.string(), nullable=True),
]
)
# table is a Table with the above schema
df = pa.Table.from_pylist(
[
{"col1": None}
]
)
table.overwrite(df)
```
In the above example, we encounter an error like this
`UnsupportedPyArrowTypeException: Column 'col1' has an unsupported type: null`,
with underlying cause
```
in _ConvertToIceberg.primitive(self, primitive)
1211 return FixedType(primitive.byte_width)
-> 1213 raise TypeError(f"Unsupported type: {primitive}")
TypeError: Unsupported type: null
```
Is there any reason we wouldn't want to support the case where pyarrow has
marked a Field as `null`? As a workaround/fix, I was thinking that we could
exclude `pa.null()` Fields in `visit_pyarrow(obj: pa.StructType, visitor:
PyArrowSchemaVisitor[T])`. This way, the column would effectively be missing
and any required/nullable enforcement would be performed accordingly. Would
this have any undesired consequences?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
