mbutrovich opened a new pull request, #3035:
URL: https://github.com/apache/iceberg-rust/pull/3035
## Which issue does this PR close?
- Closes #2792.
## What changes are included in this PR?
Draft PR, stacked on #2868. This is Task 3 of the deletion-vector read epic:
reading and applying V3 deletion vectors during a scan.
- `CachingDeleteFileLoader` reads a deletion vector's `deletion-vector-v1`
blob directly from its Puffin file by byte range (`content_offset` /
`content_size_in_bytes`), decrypting first via `EncryptedInputFile` when the
entry carries key metadata.
- After decoding, the blob's cardinality is checked against the manifest
entry's `record_count`, matching the same check Iceberg-Java does in
`BitmapPositionDeleteIndex.deserializeBitmap`. Mismatch is a `DataInvalid`
error, not a silent wrong result.
- `DeleteFileIndex` now indexes deletion vectors by `referenced_data_file`
and applies them ahead of position delete files, since a DV supersedes any
position deletes for the same data file per spec.
- A few spec invariants are now enforced as errors instead of silently
mishandled:
- a `PositionDeletes` entry with `content_offset` set but no
`referenced_data_file`
- two deletion vectors referencing the same data file (at most one is
allowed per data file per snapshot)
- a deletion vector whose partition or sequence number disagrees with the
data file it claims to reference
- `FileScanTaskDeleteFile` gained a `record_count` field, populated from the
manifest entry, used for the cardinality check above.
- `DeleteVector::deserialize` (from #2866) is now actually wired in; removed
its `#[allow(dead_code)]`.
No data-file read path changes outside of deletion vectors. Existing
position delete and equality delete handling is unchanged.
## Are these changes tested?
Yes.
- Unit tests in `delete_file_index.rs` for: DV supersedes partition-scoped
and path-scoped position deletes, DV coexists with equality deletes, rejecting
a DV missing `referenced_data_file`, rejecting duplicate DVs for one data file,
rejecting a DV with a mismatched partition or stale sequence number.
- Unit tests in `caching_delete_file_loader.rs` for: reading and applying a
DV, reading an encrypted DV, rejecting a cardinality mismatch, and the
coordinate/cardinality validation helpers directly.
- End-to-end tests in `positional_deletes.rs` reading a real Parquet data
file with a DV applied through `ArrowReader`, including the
cardinality-mismatch failure path through the full read.
## AI Disclosure
Developed with the help of Claude Code, but I understand and support these
changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
