RussellSpitzer commented on PR #11525: URL: https://github.com/apache/iceberg/pull/11525#issuecomment-2473885000
> It's true the broadcast would be compressed by default via `spark.broadcast.compress` and that would minimize the space pretty well. I think the concern is more so when we need to load the map broadcast variable on executor side we'd ultimately need to decompress all the chunks of the map. So the goal for relativization was to minimize how much would take in the in-memory representation of the map after decompression. Let me know if that makes sense.| Won't it all be in memory anyway? Java should do string interning with prefix? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
