dwsmith1983 commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5302683820
**Update:** pushed two follow-up commits extending the scan's pruning and object-store behavior. **`perf: fetch Delta deletion vectors and footers concurrently`** DV blob and footer reads were sequential: two serial round-trips per DV'd file before the scan could start, which scales badly on object stores. They now fetch with a bounded fan-out of 8, preserving file order and fail-fast error semantics. Covered by a new end-to-end unit test (inline DVs, on-disk DVs, pass-through files, exact row selections, output ordering). **`feat: push resolved scalar-subquery filters into the native Delta scan`** predicates like `id >= (SELECT max(ts) FROM checkpoint)` previously contributed nothing to the native scan: subquery results don't exist at planning, so the scan decoded the full table and Spark's covering FilterExec did all the filtering. They are now resolved at execution time and appended as pushed filters, so row-group and page-index pruning fire the same as for literal bounds. Three version-specific traps handled: 1. Spark 3.x strips subquery predicates from a scan's `dataFilters` (`FileSourceStrategy`); Spark 4.x keeps them. The contrib harvests them from the covering FilterExec at claim time and dedups, so both behaviors converge. 2. The DV plan shape interposes nodes between the filter and the scan, so the harvest matches the nearest filter above the scan, guarded by references scan output. 3. `MergeScalarSubqueries` fuses multiple scalar subqueries into one struct-returning subquery accessed via `GetStructField`; that subtree is folded to a literal before serialization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
