wombatu-kun commented on issue #16854: URL: https://github.com/apache/iceberg/issues/16854#issuecomment-4737223403
Two answers, to the point: **1. Write side - try `write.distribution-mode=hash`.** `write.target-file-size-bytes` is only a per-task *upper* bound (it splits big tasks, never enlarges or merges small ones), so raising it does nothing for small partitions. You're on `range`, which scatters a small partition's rows across many tasks - likely your "6K rows -> 720 files." `hash` sends each partition value to one task (~1 file per small partition; hot partitions still split to target size via AQE). Worth an A/B vs `range`. That said, a file can't span a partition boundary, so a long tail of tiny partitions will always yield small files at write time - this part can't be fixed write-side. **2. Yes, `rewrite_data_files` is the canonical, scalable fix.** It bin-packs per partition (`min-input-files=5` default, so your 720-file partitions qualify). At your scale use `partial-progress.enabled=true` (incremental commits), `rewrite-job-order=files-desc` (worst first), `where` (partition-by-partition), and `max-file-group-size-bytes` (bound per-job work). So yes: ingest produces what it produces, then compaction sizes it - run it incrementally and it's affordable. Docs PR: apache/iceberg#16855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
