singhpk234 commented on PR #8644: URL: https://github.com/apache/iceberg/pull/8644#issuecomment-1738199905
> Also, I'd like to find a way to add a mode that downloads files only on executors and not on the Spark driver. One way I have seen folks hacking this around is via ``` boolean onDriver = org.apache.spark.TaskContext.get() == null; ``` It would be interesting how we pass this info to the AsyncFileIO so that this mode can be set by a per engine based on the where it's accessing the AsyncFIleIO -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
