kinolaev commented on PR #16956: URL: https://github.com/apache/iceberg/pull/16956#issuecomment-6794874568
Hi @laskoviymishka, could you please take a look at this PR when you get a chance? It aims to prevent the loading amplification problem between data files and equality deletes by introducing a shared equality delete structure that can be used by multiple data files within a task. This makes Spark viable even for fast-changing CDC tables. Trino generally performs better here, but because it caches equality deletes per job rather than per task, it risks OOMs when job-level memory is tight. With this change, Spark can finish these jobs reliably. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
