raghav-reglobe opened a new pull request, #2681:
URL: https://github.com/apache/iceberg-rust/pull/2681
## What
Make iceberg-rust **apply V3 deletion vectors when scanning**. Today the scan
applies V2 Parquet positional/equality deletes but ignores V3 deletion
vectors,
so rows marked deleted by a DV are incorrectly returned. This is the **read**
complement to the DV-**write** work in #2678.
End to end:
- `DeleteFileIndex` recognizes a DV (a `PositionDeletes` entry stored as
Puffin
with a `referenced_data_file`) and indexes it by the data file it applies
to,
superseding positional-delete files for that data file (per spec). Mirrors
apache/iceberg-go `buildDVIndex` / `matchDVToData`.
- The caching delete-file loader reads the `deletion-vector-v1` blob located
by
the manifest entry's `content_offset` / `content_size_in_bytes` — the
spec's
"direct access" path — and parses it with
`DeleteVector::from_serialized_bytes`.
- This works whether the DV is wrapped in a full Puffin container or written
as a
standalone blob (e.g. by DuckDB), and falls back to the Puffin footer to
honor a
compression codec if a (non-conforming) writer ever compresses a DV blob.
## Stacked on #2678
This branch **contains #2678's commits (the DV-write side) plus this PR's
read
commits**, both rebased on current `main`. Please review #2678 first; this
PR's
own changes are the commits from `feat(scan): carry file_format +
referenced_data_file`
onward (6 commits touching `delete_file_index.rs`,
`arrow/caching_delete_file_loader.rs`,
`scan/task.rs`, `delete_vector.rs`). It can be reduced to just those once
#2678 lands.
## Validation
- Round-trip unit test (`test_load_deletes_applies_v3_deletion_vector`),
ported from
iceberg-go `TestFilterByDeletionVector` / `TestReadAllDeletionVectors`.
- **Real-data cross-engine check**: iceberg-rust scans tables whose DVs were
written
by **DuckDB** and returns the correct post-delete row counts (900K/1M and
9M/10M),
proving it reads a foreign-written spec DV — not just its own.
- Full `cargo test -p iceberg --lib`: 1375 passed, 0 failed; clippy clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
