lawofcycles opened a new issue, #15394: URL: https://github.com/apache/iceberg/issues/15394
### Feature Request / Improvement #10935 is working on adding delete file support to `IncrementalChangelogScan`, which will enable changelog scans for merge-on-read tables. This issue proposes a further optimization for v3 tables using row lineage. ### Motivation With the delete file support from #10935, changelog scans will need to read and merge delete files with data files to identify changed rows. For v3 tables with row lineage, this can be done more efficiently by leveraging row-level metadata instead. ### Proposal For v3 tables with row lineage, the changelog scan could use - `_last_updated_sequence_number` to filter to only rows changed within the scan range, avoiding full data file reads and delete file merging - `_row_id` to reliably match the same logical row across data files for UPDATE detection, without depending on identifier columns Note that row lineage does not track changes made via equality deletes ([spec](https://iceberg.apache.org/spec/#row-lineage)), so the general delete file merging path from #10935 remains necessary as a fallback. ### Scope This is an optimization on top of the delete file support in #10935, targeting v3 tables with row lineage only. ### Query engine Spark / Core ### Willingness to contribute - [ ] I can contribute this improvement/feature independently - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
