manuzhang opened a new pull request, #958:
URL: https://github.com/apache/iceberg-cpp/pull/958
## What
Plan changelog tasks for format version 3 tables that use deletion vectors:
- Add `DeletedRowsScanTask`, a `ChangelogScanTask` with `added_deletes()` and
`existing_deletes()`, for rows deleted by a deletion vector committed
against a
data file that already existed.
- `IncrementalChangelogScan` reads the delete manifests written by the
changelog
snapshots and indexes the deletion vectors each snapshot added and
removed. A
vector committed together with its data file is attached to that file's
`AddedRowsScanTask`; a vector removed together with its data file becomes
the
existing delete of the `DeletedDataFileScanTask`; a vector for an existing
data
file locates the file in the committing snapshot's data manifests and
produces a
`DeletedRowsScanTask` carrying the added vector and the vector it replaced.
- Delete manifests on format version 1 and 2 tables, position delete files
and
equality delete files are rejected with `NotSupported`.
This is planning only, matching the existing changelog tasks: a consumer of
`DeletedRowsScanTask` reads the data file and emits the positions set in the
added
vector that are not set in the existing vector.
## Why
`IncrementalChangelogScan` failed on any snapshot that carried a delete
manifest,
so a table with row-level deletes could not produce a changelog at all.
Format
version 3 makes this tractable: each data file has at most one live deletion
vector and a new vector replaces the previous one, so the rows a snapshot
deleted
are exactly the difference between the vector it added and the vector it
removed,
without reading any earlier delete files.
## Behavior change
- Changelog scans of format version 3 tables with deletion vectors now
succeed and
may return `DeletedRowsScanTask`, which callers that switch on task type
must
handle.
- The error for delete files on format version 1 and 2 tables is now
"Delete files are only supported in changelog scans of format version 3
tables".
- For every changelog snapshot that adds deletion vectors to existing data
files,
planning reads that snapshot's data manifests to find the referenced files.
- Only delete manifests written by snapshots in the range are read. Position
or
equality delete files that a format version 3 table still carries in
manifests
untouched during the range are therefore neither applied nor rejected.
## Testing
New cases in `incremental_changelog_scan_test.cc` cover a deletion vector
added to
an existing file, a vector replacing an earlier one (single snapshot and
across
two snapshots), a vector committed with its data file, a vector removed with
its
data file, partition pruning of vectors by the data filter, vectors outside
the
scan range, and the `NotSupported` errors for format version 2 delete files,
position delete files and equality delete files. `scan_test` passes locally.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
