coderfender commented on PR #12824: URL: https://github.com/apache/iceberg/pull/12824#issuecomment-2814148621
@yogevyuval , The goal here is to provide user an option to limit the number of files to be rewritten (either through compaction , data rewrite etc) . In a use case (like mine) where there are 1 billion plus files in a lake house, the user might want to iteratively run compaction to reduce the file count to an acceptable level rather than going all in at the very first time. This option should help improve rewrite spark jobs and the users can tune this param to optimize scale and reliability -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
