comphead opened a new issue, #3702:
URL: https://github.com/apache/iceberg-rust/issues/3702

   ### What's the feature are you trying to implement?
   
   Bin-pack data file compaction, the default strategy of iceberg-java's 
`rewrite_data_files`. It selects data files outside the target size range or 
with too many deletes, packs them into groups, rewrites each group into files 
of the target size, and commits a `replace` snapshot. The goal is parity with 
iceberg-java's `BinPackRewriteFilePlanner` and its rewrite commit.
   
   #### Tasks
   
   - [ ] #3417 `MergingSnapshotProducer` (PR #3564)
   - [ ] #1607 `RewriteFilesAction` (PR #3585), including a starting data 
sequence number like iceberg-java's `RewriteFiles.dataSequenceNumber`
   - [ ] #3419 Validation: fail the commit when a delete committed during the 
rewrite targets a rewritten file
   - [ ] Bin-pack planner (no ticket yet): select files and pack them into 
groups, with iceberg-java's options and defaults
   - [ ] Rewrite runner (no ticket yet): read each group with deletes applied, 
write files at the target size, and commit in one commit or in batches with 
partial progress
   - [ ] v3 tables: carry row lineage into rewritten files, the same gap as 
#3401 (read side in #2879)
   
   #### Related
   
   - #128 Size-based scan task packing. iceberg-java uses one `BinPacking` 
utility for scan planning and for rewrite groups.
   - apache/datafusion-comet#7282 Bin-pack `rewrite_data_files` in Comet, with 
a planner ported from iceberg-java
   
   ### Willingness to contribute
   
   I can contribute to this feature independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to