tmnd1991 commented on PR #9233: URL: https://github.com/apache/iceberg/pull/9233#issuecomment-1846764267
> How do you determine that the SystemFunctions are not pushed down? > > Spark will push down predicate(which includes predicates containing system functions) through join(except for full outer join), see: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1912 . So I don't think you need to handle joins specifically in ReplaceStaticInvoke. Thanks @advancedxy , now, that explains a lot of what I was observing happening in my project. During a MERGE (which is 2 joins, one LeftSemi + one FullOuter) I was observing that the first join was correctly pruning the partitions, while the secondo one, was not. Adding this patch though helps pruning more partitions, this is because the batch scan on the target table cannot prune partitions because the file names (collected as a result of the first join) are not known when performing physical planning. I think we should limit the replacement to the "full outer" case, what do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
