rahulsmahadev opened a new pull request, #4409:
URL: https://github.com/apache/iceberg-python/pull/4409

   # Rationale for this change
   
   Iceberg v3 allows `date` to be promoted to `timestamp` and `timestamp_ns`. 
The promotion does not rewrite the
   bounds of existing data files, so a bound written before the promotion is 
still a 4-byte day count from
   1970-01-01. The spec says readers must infer the write-time type from the 
length of the bound: in the
   "Schema Evolution" section of `format/spec.md`, a 4-byte `timestamp` or 
`timestamp_ns` bound is a `date`.
   
   `from_bytes` registers `TimestampType` and `TimestampNanoType` with the 
8-byte decoder, so such a bound fails:
   
   ```python
   from pyiceberg.conversions import from_bytes, to_bytes
   from pyiceberg.types import DateType, TimestampType
   
   from_bytes(TimestampType(), to_bytes(DateType(), 18793))  # 2021-06-15
   # on main: struct.error: unpack requires a buffer of 8 bytes
   # with this change: 1623715200000000
   ```
   
   On a v3 table with retained date bounds after promotion to `timestamp`, 
`tbl.scan(row_filter=GreaterThan("d", "2022-01-01T00:00:00")).plan_files()` 
raises this `struct.error` when evaluating those bounds. `tbl.inspect.files()` 
raises it after promotion to either `timestamp` or `timestamp_ns`.
   
   The fix decodes a 4-byte bound of `TimestampType` and `TimestampNanoType` as 
a signed day count, scaled to microseconds (x 86,400,000,000) or nanoseconds (x 
86,400,000,000,000), and returns the exact Python int. The 8-byte path is 
unchanged. The fallback decodes 4-byte bounds; unsupported lengths still raise 
`struct.error`. The change adds no per-file containers or caches. `TimeType`, 
`TimestamptzType` and `TimestamptzNanoType` keep their decoder, because the 
spec does not allow `date` to be promoted to them. The fix is in `from_bytes`, 
the one decoder that the inclusive and strict metrics evaluators and `inspect` 
all use, so no call site changes.
   
   Planning compares such bounds exactly, because Python ints have no int64 
limit. Metadata tables are different: converting a bound outside the Arrow 
range to a timestamp raises `OverflowError`. For date bounds promoted to 
`timestamp_ns`, representable midnight dates run from 1677-09-22 through 
2262-04-11 inclusive; metadata inspection raises `OverflowError` for dates 
outside that interval. The spec's runtime failure rule (Schema Evolution) 
applies to promoted data values. It does not apply to bounds, because the 
bounds rules only require a lower bound at or below all values and an upper 
bound at or above all values, so a bound can be wider than the data 
(`format/spec.md`, the data file fields `lower_bounds` and `upper_bounds`). An 
out-of-range bound therefore does not show that a data value overflows.
   
   This PR follows the spec rather than Java. Java's 
`Conversions.fromByteBuffer`
   (`api/src/main/java/org/apache/iceberg/types/Conversions.java`, lines 
171-179) returns the raw day count for a
   4-byte `TIMESTAMP` or `TIMESTAMP_NANO` bound without scaling it. Comparing 
that count with a microsecond literal
   could prune a file that has matching rows.
   
   Alternatives considered: B, raise `ValueError` for bounds that do not fit in 
int64. That fails planning or inspection even when every row is representable, 
because a bound need not occur in the data. C, treat such bounds as unknown, or 
leave them unavailable in metadata tables. That avoids the errors, but it 
changes the evaluator and metadata contracts, so it is left for a follow-up. 
Actual-value overflow checks belong to date-promotion support.
   
   This extends #3293, which added the same kind of 4-byte fallback for int to 
long and float to double bounds, and #4314, which added the inspect test for 
those types (it closes #3744). Neither covers date to timestamp.
   
   Out of scope: normal schema promotion and reading date data files as 
timestamps remain unsupported (related closed PR #2586). Comparison filters on 
`timestamp_ns` remain unsupported for string, integer and datetime literals 
(#4078).
   
   # Are these changes tested?
   
   Yes. `test_from_bytes_date_bound_promoted_to_timestamp` in 
`tests/test_conversions.py` checks exact micros and nanos for 4-byte day 
counts, including a date before 1970, and checks that the 8-byte encoding is 
unchanged. `test_from_bytes_date_bound_promoted_to_timestamp_beyond_int64` 
checks the exact scaled value for the first day count outside the int64 range, 
in both directions. `test_plan_and_inspect_after_date_to_timestamp_promotion` 
in `tests/expressions/test_evaluator.py` uses the public API with each catalog 
fixture. Two filters plan exactly the expected data file each, and 
`inspect.files()` returns the exact bounds. It simulates the promotion with 
`update_schema(allow_incompatible_changes=True)`, because PyIceberg cannot 
write v3 metadata yet (#1551), and the bound decoder does not depend on the 
format version. The new rows in 
`test_inclusive_metrics_eval_bounds_after_promotion` and 
`test_strict_metrics_eval_bounds_after_promotion` cover both metrics 
evaluators, including a
  day count before the epoch. The strict row compares against a literal after 
the epoch, so a missing scale fails it.
   
   Without this change, the new tests fail with `struct.error: unpack requires 
a buffer of 8 bytes`, including the beyond-int64 test. With this change, they 
pass.
   
   # Are there any user-facing changes?
   
   Yes. Filtered file planning for `timestamp` and metadata inspection for 
`timestamp` and `timestamp_ns` now decode retained 4-byte date bounds. This 
does not add date-to-timestamp data reading or nanosecond literal binding. 
Please add the `changelog` label.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to