adiranjan25 commented on issue #18597: URL: https://github.com/apache/iceberg/issues/18597#issuecomment-6815723258
Hi @ammarchalifah , I'd like to take ownership of investigating and implementing this improvement. I'm planning to focus on the Spark–Iceberg Storage Partition Join (SPJ) behavior described here, particularly the executor memory pressure caused by grouping data across multiple partition keys and the trade-off between shuffle reduction, parallelism, and small-file proliferation. My proposed approach is to: 1. Reproduce the behavior using a representative Iceberg table partitioned by date and bucketed ID, with a `MERGE INTO` workload joining on ID. 2. Establish baseline measurements for executor memory consumption, shuffle volume, task distribution, and execution time. 3. Investigate whether finer-grained, locality-aware partitioning or a node-local shuffle strategy can reduce executor memory pressure without increasing the physical table's bucket count. 4. Develop a focused implementation with correctness tests and reproducible performance benchmarks. 5. Submit a PR with the implementation details, benchmark results, and documented trade-offs. Before making significant architectural changes, I'll review the existing Spark SPJ and Iceberg scan-planning mechanisms and share the proposed design here for maintainer feedback. **I'm actively taking this issue forward and intend to submit a PR.** If anyone is already working on an overlapping implementation, please let me know so we can coordinate and avoid duplicate effort. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
