andygrove opened a new issue, #7370: URL: https://github.com/apache/datafusion-comet/issues/7370
### What is the problem the feature request solves? A Comet Native Executor cluster (#7289) runs whole queries without a JVM. A scan with dynamic partition pruning filters its partitions by the result of a subquery (`InSubqueryExec`, `SubqueryBroadcastExec`) that today runs in Spark. The driver doesn't support those yet, so a query whose plan uses DPP can't run on the cluster. ### Describe the potential solution Run the pruning subquery before submitting the query, as the prototype does for scalar subqueries, and prune the scan's file partitions on the driver before it builds each partition's plan. A broadcast-reuse variant could instead run the subquery's build side as a stage of the same query. ### Additional context Scalar subqueries already run first on the driver and are passed to the cluster as literals; see `withSubqueryResults` in `CometNativeExecutorsExec.scala`. TPC-H doesn't use DPP, since its tables aren't partitioned. Part of #7288. ### Willingness to contribute Yes. I can contribute this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
