vfsfitvnm commented on issue #16932: URL: https://github.com/apache/iceberg/issues/16932#issuecomment-4845367882
I second this, especially: > So today neither mode works well at scale: the Hadoop path parallelizes but makes too many calls, and the cheaper prefix path makes few calls but doesn't parallelize. Also, I see that the current implementation is: 1. Collect all tracked file paths; 2. Collect all existing file paths (I think this is what makes it unsustainable for very large tables at terabyte-scale); 3. Compare them and delete untracked files older than timestamp. Instead, untracked files could be deleted on-the-go/incrementally: 1. Collect all tracked file paths; 2. Collect a subset of existing file paths (e.g. pagination), check if they tracked and take action immediately. Then, collect the next subset. I'm not entirely sure this is feasible given the API the procedure relies. But other than that, would the proposed approach be better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
