dramaticlly opened a new pull request, #15590: URL: https://github.com/apache/iceberg/pull/15590
Add DataFileAccumulator to automatically flush data files to manifest files when the accumulated count reaches a configurable threshold (default 100K). This spreads manifest writing I/O over time for bulk operations that add large numbers of data files (e.g. 500K+), reducing peak memory and avoid OOM from unbound collection. - Add DataFileAccumulator (analogous to RollingManifestWriter but by count) - Refactor MergingSnapshotProducer to delegate file accumulation - Flushed files are lazily read back from manifests for validation - Flushed manifests survive commit retries and are cleaned on failure ## Performance numbers Reused https://github.com/apache/iceberg/blob/main/core/src/jmh/java/org/apache/iceberg/AppendBenchmark.java to quantify the performance by comparing against different combination of tables and files count, detailed setup in https://github.com/dramaticlly/iceberg/commit/383b5605ac59a62686e7a3f7ebfba6d0dfc03a48 ``` End to end Latency (prepare files, add and commit and avg across 3 runs) ┌───────┬───────────────────────┬─────────┬──────────┬──────┐ │ Files │ Schema │ dynamic │ baseline │ diff │ ├───────┼───────────────────────┼─────────┼──────────┼──────┤ │ 100k │ narrow (5 col) │ 1.651s │ 1.613s │ +2% │ ├───────┼───────────────────────┼─────────┼──────────┼──────┤ │ │ fact_table (30 col) │ 2.352s │ 2.282s │ +3% │ ├───────┼───────────────────────┼─────────┼──────────┼──────┤ │ │ ml_features (150 col) │ 11.835s │ 6.861s │ +73% │ ├───────┼───────────────────────┼─────────┼──────────┼──────┤ │ 500k │ narrow │ 2.757s │ 2.625s │ +5% │ ├───────┼───────────────────────┼─────────┼──────────┼──────┤ │ │ fact_table │ 10.054s │ 10.841s │ -7% │ ├───────┼───────────────────────┼─────────┼──────────┼──────┤ │ │ ml_features │ 49.986s │ OOM │ -- │ ├───────┼───────────────────────┼─────────┼──────────┼──────┤ │ 1M │ narrow │ 5.683s │ 4.791s │ +19% │ ├───────┼───────────────────────┼─────────┼──────────┼──────┤ │ │ fact_table │ 19.274s │ 19.871s │ -3% │ ├───────┼───────────────────────┼─────────┼──────────┼──────┤ │ │ ml_features │ 94.476s │ OOM │ -- │ └───────┴───────────────────────┴─────────┴──────────┴──────┘ Manifests written per snapshot commit (avg across 3 runs) ┌───────┬─────────────┬─────────┬──────────┬──────┐ │ Files │ Schema │ dynamic │ baseline │ diff │ ├───────┼─────────────┼─────────┼──────────┼──────┤ │ 100k │ narrow │ 5.7 │ 5.7 │ same │ ├───────┼─────────────┼─────────┼──────────┼──────┤ │ │ fact_table │ 20 │ 20 │ same │ ├───────┼─────────────┼─────────┼──────────┼──────┤ │ │ ml_features │ 44 │ 40 │ +10% │ ├───────┼─────────────┼─────────┼──────────┼──────┤ │ 500k │ narrow │ 16 │ 16 │ same │ ├───────┼─────────────┼─────────┼──────────┼──────┤ │ │ fact_table │ 100 │ 64 │ +56% │ ├───────┼─────────────┼─────────┼──────────┼──────┤ │ │ ml_features │ 223 │ OOM │ -- │ ├───────┼─────────────┼─────────┼──────────┼──────┤ │ 1M │ narrow │ 34 │ 32 │ +6% │ ├───────┼─────────────┼─────────┼──────────┼──────┤ │ │ fact_table │ 202 │ 112 │ +80% │ ├───────┼─────────────┼─────────┼──────────┼──────┤ │ │ ml_features │ 446 │ OOM │ -- │ └───────┴─────────────┴─────────┴──────────┴──────┘ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
