adriangb commented on issue #22883: URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5808424593
I've been experimenting with a way to handle "optional" filters, meaning filters that are not needed for correctness, such as hash join / TopK dynamic filters and prune-only copies. Design notes, including how Trino, DuckDB, ClickHouse, Velox and Spark handle this: https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt In short: an `Optional(...)` wrapper marks the filter. A consumer may skip it only when it finds it on the root AND chain. Per-stream gates pause optional filters that remove too few rows. `PushedDown` keeps its two states. This is an experiment. The PRs are drafts and stacked: - #25673 `OptionalFilterPhysicalExpr` + `split_optional` + proto - #25681 hash join / TopK / aggregate mark their dynamic filters optional (only EXPLAIN changes) - #25674 `OptionalFilterGate` + `datafusion.execution.optional_filter_mode` (`always` default | `adaptive` | `pruning_only`) - #25682 the Parquet scan uses the mode for optional row filters - #25683 `FilterExec`: skip optional conjuncts, plus a smaller take on adaptive reordering (#22698 stays open for its history) Independent of the stack but related: - #25677 collapse partitioned InList join filters into one `IN` list (from #24235) - #25670 docs: what `PushedDown::No` means (why no `Inexact` state) - #22384 rebased -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
