rahulsmahadev opened a new pull request, #4360:
URL: https://github.com/apache/iceberg-python/pull/4360
# Rationale for this change
`Table.append` fails for a table identity-partitioned on a `uuid` column:
```python
schema = Schema(NestedField(1, "id", LongType()), NestedField(2, "u",
UUIDType()))
tbl = catalog.create_table("db.t", schema=schema,
partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "u")))
tbl.append(pa.table({"id": pa.array([1], pa.int64()), "u":
pa.array([uuid.UUID(int=1).bytes], pa.uuid())}))
# pyarrow.lib.ArrowNotImplementedError: Keys of type extension<arrow.uuid>
```
The identity transform returns the `arrow.uuid` column unchanged, and
PyArrow has no group by or equality kernel for extension types, so the
`group_by` in `_determine_partitions` fails before any file is written.
Bucketing on the uuid column works because the bucket transform returns an int.
This change groups and filters on the storage type of extension-typed
partition columns (`fixed_size_binary(16)` for uuid). PyArrow does not plan
compute kernels for extension types (apache/arrow#39024 was closed as not
planned), so this stays needed. The data columns are unchanged, so the Parquet
files keep the uuid type, and the partition values are the 16 bytes the
manifests store for a uuid.
Not changed here: an equality filter on a uuid identity partition still
plans no files, because partition pruning compares the UUID read from the
manifest with a bytes literal. That also affects tables written by other
engines, and I'll send it as a separate fix.
## Are these changes tested?
Yes. Two new tests in `tests/io/test_pyarrow.py` fail with
`ArrowNotImplementedError: Keys of type extension<arrow.uuid>` without this
change and pass with it, on pyarrow 25 and 18:
- `test_determine_partitions_with_uuid_column`: the partition keys
(including a null partition), the rows of each partition, and that the data
keeps the uuid type.
- `test_append_uuid_to_table_partitioned_on_uuid`: an append through the
`catalog` fixture, the rows read back, and the partition values of the data
files.
## Are there any user-facing changes?
Bug fix only: appending to a table identity-partitioned on a uuid column no
longer raises. No API change.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]