rahulsmahadev opened a new pull request, #4409:
URL: https://github.com/apache/iceberg-python/pull/4409
# Rationale for this change
Iceberg v3 allows `date` to be promoted to `timestamp` and `timestamp_ns`.
The promotion does not rewrite the
bounds of existing data files, so a bound written before the promotion is
still a 4-byte day count from
1970-01-01. The spec says readers must infer the write-time type from the
length of the bound: in the
"Schema Evolution" section of `format/spec.md`, a 4-byte `timestamp` or
`timestamp_ns` bound is a `date`.
`from_bytes` registers `TimestampType` and `TimestampNanoType` with the
8-byte decoder, so such a bound fails:
```python
from pyiceberg.conversions import from_bytes, to_bytes
from pyiceberg.types import DateType, TimestampType
from_bytes(TimestampType(), to_bytes(DateType(), 18793)) # 2021-06-15
# on main: struct.error: unpack requires a buffer of 8 bytes
# with this change: 1623715200000000
```
On a v3 table with retained date bounds after promotion to `timestamp`,
`tbl.scan(row_filter=GreaterThan("d", "2022-01-01T00:00:00")).plan_files()`
raises this `struct.error` when evaluating those bounds. `tbl.inspect.files()`
raises it after promotion to either `timestamp` or `timestamp_ns`.
The fix decodes a 4-byte bound of `TimestampType` and `TimestampNanoType` as
a signed day count, scaled to microseconds (x 86,400,000,000) or nanoseconds (x
86,400,000,000,000), and returns the exact Python int. The 8-byte path is
unchanged. The fallback decodes 4-byte bounds; unsupported lengths still raise
`struct.error`. The change adds no per-file containers or caches. `TimeType`,
`TimestamptzType` and `TimestamptzNanoType` keep their decoder, because the
spec does not allow `date` to be promoted to them. The fix is in `from_bytes`,
the one decoder that the inclusive and strict metrics evaluators and `inspect`
all use, so no call site changes.
Planning compares such bounds exactly, because Python ints have no int64
limit. Metadata tables are different: converting a bound outside the Arrow
range to a timestamp raises `OverflowError`. For date bounds promoted to
`timestamp_ns`, representable midnight dates run from 1677-09-22 through
2262-04-11 inclusive; metadata inspection raises `OverflowError` for dates
outside that interval. The spec's runtime failure rule (Schema Evolution)
applies to promoted data values. It does not apply to bounds, because the
bounds rules only require a lower bound at or below all values and an upper
bound at or above all values, so a bound can be wider than the data
(`format/spec.md`, the data file fields `lower_bounds` and `upper_bounds`). An
out-of-range bound therefore does not show that a data value overflows.
This PR follows the spec rather than Java. Java's
`Conversions.fromByteBuffer`
(`api/src/main/java/org/apache/iceberg/types/Conversions.java`, lines
171-179) returns the raw day count for a
4-byte `TIMESTAMP` or `TIMESTAMP_NANO` bound without scaling it. Comparing
that count with a microsecond literal
could prune a file that has matching rows.
Alternatives considered: B, raise `ValueError` for bounds that do not fit in
int64. That fails planning or inspection even when every row is representable,
because a bound need not occur in the data. C, treat such bounds as unknown, or
leave them unavailable in metadata tables. That avoids the errors, but it
changes the evaluator and metadata contracts, so it is left for a follow-up.
Actual-value overflow checks belong to date-promotion support.
This extends #3293, which added the same kind of 4-byte fallback for int to
long and float to double bounds, and #4314, which added the inspect test for
those types (it closes #3744). Neither covers date to timestamp.
Out of scope: normal schema promotion and reading date data files as
timestamps remain unsupported (related closed PR #2586). Comparison filters on
`timestamp_ns` remain unsupported for string, integer and datetime literals
(#4078).
# Are these changes tested?
Yes. `test_from_bytes_date_bound_promoted_to_timestamp` in
`tests/test_conversions.py` checks exact micros and nanos for 4-byte day
counts, including a date before 1970, and checks that the 8-byte encoding is
unchanged. `test_from_bytes_date_bound_promoted_to_timestamp_beyond_int64`
checks the exact scaled value for the first day count outside the int64 range,
in both directions. `test_plan_and_inspect_after_date_to_timestamp_promotion`
in `tests/expressions/test_evaluator.py` uses the public API with each catalog
fixture. Two filters plan exactly the expected data file each, and
`inspect.files()` returns the exact bounds. It simulates the promotion with
`update_schema(allow_incompatible_changes=True)`, because PyIceberg cannot
write v3 metadata yet (#1551), and the bound decoder does not depend on the
format version. The new rows in
`test_inclusive_metrics_eval_bounds_after_promotion` and
`test_strict_metrics_eval_bounds_after_promotion` cover both metrics
evaluators, including a
day count before the epoch. The strict row compares against a literal after
the epoch, so a missing scale fails it.
Without this change, the new tests fail with `struct.error: unpack requires
a buffer of 8 bytes`, including the beyond-int64 test. With this change, they
pass.
# Are there any user-facing changes?
Yes. Filtered file planning for `timestamp` and metadata inspection for
`timestamp` and `timestamp_ns` now decode retained 4-byte date bounds. This
does not add date-to-timestamp data reading or nanosecond literal binding.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]