KranzL opened a new issue, #1982: URL: https://github.com/apache/iceberg-go/issues/1982
We run compactions against tables with fairly wide rows and found it hard to keep worker memory under control. Looking at the read+write pipeline, the knobs that matter are either hardcoded or only settable as table properties: - the scanner decodes `read.parquet.batch-size` rows per Arrow batch (default 131072) and there is no per-scan override, so lowering it for a compaction means changing the table property for every reader - each rolling data writer buffers up to 64 record batches in its input channel (`rollingDataWriterQueueCapacity` is a constant) - `write.parquet.row-group-limit` is also table-wide only Peak memory ends up being roughly decode batch size times buffered batches times row width. With the defaults and rows in the KB range that arithmetic gets into gigabytes per worker before the parquet writer even flushes anything. I'd like to add per-scan and per-write overrides for these (a scan option for the decode batch size, a write option for the writer queue capacity and the row group limit) and forward them through `CompactionGroupOption` so `ExecuteCompactionGroup` can run with tighter bounds without touching table properties. Defaults would stay exactly as they are today. I have a working patch with tests and would like to pick this up, happy to put up a PR if this sounds reasonable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
