dramaticlly opened a new pull request, #15590: URL: https://github.com/apache/iceberg/pull/15590
Add DataFileAccumulator to automatically flush data files to manifest files when the accumulated count reaches a configurable threshold (default 10K). This spreads manifest writing I/O over time for bulk operations that add large numbers of data files (e.g. 500K+), reducing peak memory and commit latency. - Add TableProperties.MANIFEST_FLUSH_FILE_COUNT_THRESHOLD - Add DataFileAccumulator (analogous to RollingManifestWriter but by count) - Refactor MergingSnapshotProducer to delegate file accumulation - Flushed files are lazily read back from manifests for validation - Flushed manifests survive commit retries and are cleaned on failure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
