advancedxy commented on PR #9233: URL: https://github.com/apache/iceberg/pull/9233#issuecomment-1846909628
> Adding this patch though helps pruning more partitions, this is because the batch scan on the target table cannot prune partitions because the file names (collected as a result of the first join) are not known when performing physical planning. I think we should limit the replacement to the "full outer" case, what do you think? Could you elaborate a bit more? the planning tree string/dag of Spark SQL would be helpful. If the join type is full outer, the predicate could not be pushed down, therefore the partition pruning is unlikely to be performed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
