qinghui-xu opened a new issue, #18242: URL: https://github.com/apache/iceberg/issues/18242
### Feature Request / Improvement **Scenario:** Our Flink job keeps upserting to a V2 table, with some spiky loads during the day. Spark compaction actions are running regularly to ensure a decent performance for the table. When Flink job is under spikes, the Spark compactions are failing with OOM. **Root cause:** Compaction is using Spark iceberg reader, which loads equality deletes into memory in order to build a local filter. In hot partitions, there are huge amount of equality deletes that quickly eat up executor memory. **Proposal:** Use a merge / join approach when equality deletes are large, and stay with the local filter approach for moderate size equality deletes. Details in the [design doc](https://docs.google.com/document/d/15L44T7TcCyi8Bc_OTTkX72PywtDf1G5l84LEzYxEgvo/edit?usp=sharing). **Test:** We have implemented a PoC based on 1.10.2, and it shows some promising results. There is a huge improvement in terms of memory efficiency: for the same compaction workload, we use `10 executors X 8GB memory` with the PoC vs. `100 execcutors X 32GB` memory with 1.10.2, with an equivalent execution run time (and the 1.10.2 version fails when there are ~ 100 million eq deletes, while the PoC can handle up to 100 billion eq deletes under some stress tests). ### Query engine Spark ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
