avison9 opened a new pull request, #4314:
URL: https://github.com/apache/iceberg-python/pull/4314
Closes #3744
# Rationale for this change
#3744 reports that after a spec-allowed promotion (`int` → `long`, `float` →
`double`), every metadata
inspection API raises `struct.error` on tables that still have files written
before the promotion. Bounds
are stored without a type tag, and older files keep the 4-byte encoding
while the current schema type is
8 bytes.
This is already fixed on `main` by #3293 (released in 0.12.0):
`from_bytes(LongType/DoubleType)` now falls
back to 4-byte decoding, and every bounds decode in `inspect.py` goes
through it. The issue was reported
against 0.11.1, which predates that fix.
#3293 only added coverage for the metrics evaluators, though; the `inspect`
path was fixed as a side effect
with no test pinning it. This PR adds that coverage so the crash can't
silently come back, e.g. if
`inspect` moves to a different decode path (#4297).
```python
tbl.append(pa.table({"qty": pa.array([1, 2], pa.int32())}))
with tbl.update_schema() as u:
u.update_column("qty", field_type=LongType())
tbl.inspect.files()
# 0.11.1: struct.error: unpack requires a buffer of 8 bytes
# main: readable_metrics.qty -> lower_bound=1, upper_bound=2
```
## Are these changes tested?
Test-only change. `test_inspect_bounds_after_type_promotion` writes with
`int`/`float`, promotes to
`long`/`double`, and asserts correct decoded bounds from `entries`, `files`,
`data_files`, `all_files`,
`all_data_files`, and the `manifests()` partition summaries.
Verified that it fails with the original `struct.error` when #3293's change
to `conversions.py` is reverted,
and passes on `main`. Full unit suite: 4524 → 4525 passed, no new failures;
`make lint` clean.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
