Akeron-Zhu commented on issue #9268: URL: https://github.com/apache/iceberg/issues/9268#issuecomment-2756506856
Hi, @IgorBerman @akshayakp97 @rdblue , I also encountered this problem in last year, it is because the Spark3 DSV2 only prune column at V2ScanRelationPushDown, but the later RewriteSubquery rule generated new predicates which can be use to prune column, but Spark does not prune column again. I write an optimize rule to solve this problem, and I create a PR for Spark after passed the tests of spark. The url is: [https://github.com/apache/spark/pull/50399](url) Hope this helps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
