kevinjqliu commented on issue #17665: URL: https://github.com/apache/iceberg/issues/17665#issuecomment-5684991122
We got the band together for the first virtual meetup!! ### Apache Iceberg Virtual Meetup: GSoC Projects and Commutative Compaction * **Friday, September 18, 9:00 AM - 10:00 AM PDT** * See the event page at https://luma.com/b4j0ctoq This is also available on the "Iceberg Dev Calendar" We got **TWO** presentations from community members. #### Following the Read Path: From Spark Readers to Object Storage w/ Varun Lakhyani My GSoC project began with a straightforward goal: improve Apache Iceberg's Spark reader performance through parallelism. But tracing the read path across Spark, Iceberg, Parquet, and object storage revealed the original optimization was only part of the story. Understanding those interactions and validating with benchmarks led me to a different optimization first. This session explores that investigation, the design decisions and benchmarking methodology behind it, and the performance improvements that followed, along with lessons on open source collaboration. #### Commutative Compaction w/ Chris Douglas Table formats optimize data layouts in immutable stores by committing compactions. Despite making no logical changes, layout changes conflict with concurrent transactions, forcing costly restarts of both compactions and user transactions. We introduce compaction maps, a compact data structure encoding remapping functions to repair layout conflicts and restore commutativity in data lake settings. We implement compaction maps in Apache Iceberg and demonstrate that repairing even large conflicts through large compactions takes less than a second, including I/O costs in three major US clouds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
