JohnEarle opened a new pull request, #17834:
URL: https://github.com/apache/iceberg/pull/17834
Closes #17833
ContentFileParser omits `dataSequenceNumber` and `fileSequenceNumber` when
serializing content files, and the REST `ContentFile` schema does not define
them. Any file scan task that crosses through the
server-side scan planning endpoints arrives with null sequence numbers on
its delete files.
Delete files apply only to data files with an equal or smaller data sequence
number, so a merge-on-read scan built from such tasks cannot evaluate delete
applicability.
I hit this running an engine's full connector test suite
against a REST catalog that implements the scan planning endpoints: every
DELETE/UPDATE/MERGE-related test failed with an NPE unboxing
`DeleteFile.dataSequenceNumber()`, while append-only tables were unaffected.
With this fix applied on both sides, the same suite passes: 76 tests,
0 failures (previously 7 merge-on-read errors).
Changes:
- `ContentFileParser` writes `data-sequence-number` and
`file-sequence-number` when present, and restores them on read
- The REST `ContentFile` schema documents both optional fields
(regenerated model included)
- Round-trip coverage in `TestContentFileParser` (the comparator now also
checks sequence numbers for every existing case) and a scan task
round-trip in `TestFileScanTaskParser`
Both directions stay compatible: readers ignore unknown fields, and JSON
written without the fields deserializes exactly as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
