karubian opened a new pull request, #4324: URL: https://github.com/apache/iceberg-python/pull/4324
Closes https://github.com/apache/iceberg-python/issues/4323 # Rationale for this change `PartitionFieldStats.update` takes bounds with Python's `min` and `max`, which keep whichever of -0.0 and 0.0 came first. The spec says the lower bound must not be +0.0 when -0.0 is a value, and the upper bound must not be -0.0 when +0.0 is a value: https://iceberg.apache.org/spec/#manifest-lists After `min` and `max`, a zero now replaces a zero bound of the other sign when it belongs there, so -0.0 sorts below 0.0. That is the order Iceberg Java's `PartitionSummary` gets from `Double.compareTo`. Nonzero floats and other types take the same path as before. `StatsAggregator` in `pyiceberg/io/pyarrow.py` aggregates data file column bounds with the same `min` and `max`. For files PyIceberg writes this is already correct, because Arrow's Parquet writer stores a zero minimum as -0.0 and a zero maximum as +0.0, already in pyarrow 18.0.0, the oldest version PyIceberg supports: https://github.com/apache/arrow/blob/apache-arrow-18.0.0/cpp/src/parquet/statistics.cc#L338-L367. Files from other writers added with `add_files` may not follow that rule, which I'll raise separately. #4274 is a different signed zero bug, on the write path, and doesn't overlap with this. ## Are these changes tested? Yes. `test_construct_partition_summaries_orders_negative_zero_below_zero` builds the summary for `[0.0, -0.0]` and `[-0.0, 0.0]`, for float and double fields, and fails on main. `make lint` passes. `make test` passes apart from four Kerberos tests in `tests/catalog/test_hive.py`, which need `puresasl`, and that isn't installed here. ## Are there any user-facing changes? Yes. Manifest lists for float and double partition fields that hold both zeros now get the bounds the spec requires. There is no API change. Please add the `changelog` label. AI disclosure: the code, the test and this description were drafted with Claude Code at my direction. I reviewed them and ran lint and the tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
