gtrettenero commented on PR #16336: URL: https://github.com/apache/iceberg/pull/16336#issuecomment-4451839716
I couldn't find an issue for this, but I was trying to rewrite the manifests on a table with ~2B files and 48k manifests. Although it was able to get through most of rewrite without OOMing (with a 36gb executor memory), when it had to copy the added manifests it was doing so serially in the driver and was taking hours. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
