DAlperin opened a new pull request, #2219: URL: https://github.com/apache/iceberg-rust/pull/2219
## Which issue does this PR close? Closes #2218 ## What changes are included in this PR? This PR adds support for row-level changes (inserts, updates, and deletes) to Iceberg tables through a new DeltaWriter implementation. This enables CDC (Change Data Capture), upsert operations, and efficient row-level deletions. Key Components: 1. RowDeltaAction Transaction (could be supplanted by https://github.com/apache/iceberg-rust/pull/2203 - New transaction type for applying row-level changes atomically - Supports both data files and delete files in a single transaction - Located in transaction/row_delta.rs 2. Position Delete Writer - Writes position delete files for efficient row-level deletions - Deletes rows by file path and row position - Located in writer/base_writer/position_delete_writer.rs 3. Delta Writer (Combined Writer) - Orchestrates data file, position delete, and equality delete writers - Intelligently routes operations based on row tracking - Memory-bounded row tracking with configurable limits - Falls back to equality deletes for older/evicted rows - Located in writer/combined_writer/delta_writer.rs Input Format: The DeltaWriter expects RecordBatch with an operations column: - Value 1 = Insert/Update (write to data file) - Value -1 = Delete (write to delete file) Memory Management: - Tracks recently written rows for efficient position deletes - Configurable max_seen_rows limit (default: 100,000) - FIFO eviction when limit is reached - Can be disabled (set to 0) to use only equality deletes <!-- Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates. --> ## Are these changes tested? Unit tests for each new writer are added. <!-- Specify what test covers (unit test, integration test, etc.). If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
