rahulsmahadev opened a new pull request, #4397:
URL: https://github.com/apache/iceberg-python/pull/4397
# Rationale for this change
PyIceberg already models partition statistics in the table metadata:
`PartitionStatisticsFile`, `TableMetadata.partition_statistics`, and the
`SetPartitionStatisticsUpdate` and `RemovePartitionStatisticsUpdate` table
updates (#2192). Table and Transaction have no builder for these updates;
callers currently need to construct the update objects and commit them through
the catalog API.
Java has this API. `Table.updatePartitionStatistics()` and
`Transaction.updatePartitionStatistics()` return an `UpdatePartitionStatistics`
with `setPartitionStatistics(file)` and `removePartitionStatistics(snapshotId)`
(`api/src/main/java/org/apache/iceberg/UpdatePartitionStatistics.java`,
`core/src/main/java/org/apache/iceberg/SetPartitionStatistics.java`).
This PR adds the same API to PyIceberg:
- `Table.update_partition_statistics()` and
`Transaction.update_partition_statistics()`, next to `update_statistics()`.
- `UpdatePartitionStatistics` in
`pyiceberg/table/update/partition_statistics.py`, modelled on
`UpdateStatistics`. Pending operations are applied on commit, for chained and
`with` block operations alike, with sets applied before removals as in Java.
- `set_partition_statistics(file)` replaces the file already set for the
same snapshot. `None` is a no-op, as in Java.
- `remove_partition_statistics(snapshot_id)` uses the existing
`RemovePartitionStatisticsUpdate`.
```python
table.update_partition_statistics().set_partition_statistics(partition_statistics_file).commit()
with table.update_partition_statistics() as update:
update.set_partition_statistics(partition_statistics_file)
update.remove_partition_statistics(snapshot_id)
```
Related open PRs: #4087 fixes `UpdateStatistics.remove_statistics`, which
replaces the pending operations instead of appending to them. This PR does not
copy that pattern. #4364 removes partition statistics when their snapshots are
removed. This PR changes only the `RemovePartitionStatisticsUpdate` handler in
`pyiceberg/table/update/__init__.py`, a different part of that file from the
snapshot removal handler that #4364 changes, and does not touch the snapshot
expiry code. #4364 only removes entries that exist, so the no-op change does
not affect its path.
## Are these changes tested?
Yes. Five tests in `tests/table/test_statistics.py` use the `catalog`
fixture (in-memory and SQL catalogs) and go through the public API:
- `test_update_partition_statistics`: set, replace the file for the same
snapshot, set in a `with` block, remove, and reload the table from the catalog.
- `test_update_partition_statistics_set_and_remove_in_one_update`:
operations on different snapshots are both applied. On the same snapshot the
removal wins in either call order, with or without an existing file. Repeated
sets keep the last file, and repeated removals are no-ops.
- `test_update_partition_statistics_in_transaction`: pending changes are
visible in `transaction.table_metadata`, but not in `table.metadata` or a
catalog reload, until the transaction commits. An exception rolls them back.
- `test_remove_partition_statistics_without_file_is_noop`: removing a
snapshot without a file, once or repeated, commits and leaves the metadata
unchanged.
- `test_set_partition_statistics_none_is_noop`:
`set_partition_statistics(None)` changes nothing.
The low-level handler is covered by
`test_remove_partition_statistics_update_without_file_is_noop` in
`tests/table/test_init.py`.
All five tests fail on main with `AttributeError`: the transaction test
reports that `Transaction` has no `update_partition_statistics` attribute, and
the other four report the missing attribute on `Table`. All five pass with this
change. The low-level test in `tests/table/test_init.py` fails on main with
`ValueError` and passes with this change. I also checked that a copy of the
`UpdateStatistics` pattern, where removal discards the pending sets, fails the
set-and-remove test.
## Are there any user-facing changes?
Yes. New methods `Table.update_partition_statistics()` and
`Transaction.update_partition_statistics()`, and a new
`UpdatePartitionStatistics` class. The docs are in `mkdocs/docs/api.md`. Two
behaviours follow Java:
- Removing partition statistics that are absent is a no-op, matching Java.
This also changes the existing `RemovePartitionStatisticsUpdate` handler, which
previously raised `ValueError`.
- Within a builder, sets are applied before removals, matching Java.
Removing a snapshot's statistics wins over setting them, regardless of call
order.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]