rahulsmahadev opened a new pull request, #4360:
URL: https://github.com/apache/iceberg-python/pull/4360

   # Rationale for this change
   
   `Table.append` fails for a table identity-partitioned on a `uuid` column:
   
   ```python
   schema = Schema(NestedField(1, "id", LongType()), NestedField(2, "u", 
UUIDType()))
   tbl = catalog.create_table("db.t", schema=schema, 
partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "u")))
   tbl.append(pa.table({"id": pa.array([1], pa.int64()), "u": 
pa.array([uuid.UUID(int=1).bytes], pa.uuid())}))
   # pyarrow.lib.ArrowNotImplementedError: Keys of type extension<arrow.uuid>
   ```
   
   The identity transform returns the `arrow.uuid` column unchanged, and 
PyArrow has no group by or equality kernel for extension types, so the 
`group_by` in `_determine_partitions` fails before any file is written. 
Bucketing on the uuid column works because the bucket transform returns an int.
   
   This change groups and filters on the storage type of extension-typed 
partition columns (`fixed_size_binary(16)` for uuid). PyArrow does not plan 
compute kernels for extension types (apache/arrow#39024 was closed as not 
planned), so this stays needed. The data columns are unchanged, so the Parquet 
files keep the uuid type, and the partition values are the 16 bytes the 
manifests store for a uuid.
   
   Not changed here: an equality filter on a uuid identity partition still 
plans no files, because partition pruning compares the UUID read from the 
manifest with a bytes literal. That also affects tables written by other 
engines, and I'll send it as a separate fix.
   
   ## Are these changes tested?
   
   Yes. Two new tests in `tests/io/test_pyarrow.py` fail with 
`ArrowNotImplementedError: Keys of type extension<arrow.uuid>` without this 
change and pass with it, on pyarrow 25 and 18:
   
   - `test_determine_partitions_with_uuid_column`: the partition keys 
(including a null partition), the rows of each partition, and that the data 
keeps the uuid type.
   - `test_append_uuid_to_table_partitioned_on_uuid`: an append through the 
`catalog` fixture, the rows read back, and the partition values of the data 
files.
   
   ## Are there any user-facing changes?
   
   Bug fix only: appending to a table identity-partitioned on a uuid column no 
longer raises. No API change.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to