liurenjie1024 commented on issue #624: URL: https://github.com/apache/iceberg-rust/issues/624#issuecomment-2419144850
Hi, @camuel >Does anyone has any insights on how computation heavy is the compaction workload really? Like on a beefy machine what compaction rate will be possible? Like 1GB/sec? 10GB/sec? A ball part figure? Separately what is possible compaction rate from first principles and what is typical compaction rate per node with today Iceberg impl.? I think it's hard to say the compaction rate. Compaction is a mixed workload, it involves io(read/write files), computation(reclustering, binpacking), so it's difficult to estimate the compaction rate. Things gets more complicated when we need to handle deletion files. As with partition spec change, it's typically lazy in iceberg, e.g. partition spec chanage is a metadata only operation in iceberg, and it's not part of compaction typically. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
