lxc512157407 commented on PR #24821: URL: https://github.com/apache/datafusion/pull/24821#issuecomment-5649766506
Hi @2010YOUY01, thank you for taking a look! **On architecture**: fair point — baking stats-based simplification into `FilterExecBuilder` does set a precedent of per-operator special cases, and the cleaner long-term home would be a stats-aware expression simplification pass. The complication we hit: at the logical layer (`SimplifyExpressions`) the statistics of a Filter's *input* are not readily available (they originate at the TableScan and flow through the plan), which is why this landed in the physical builder as a first step. Would a `PhysicalOptimizerRule` be an acceptable middle ground for you (it sees the physical plan and input statistics, without special-casing inside the operator itself), or would you rather this not land at all until stats are plumbed into expression simplification? **On measurable benefit**: honest answer — we have selectivity evidence (join-key `IS NOT NULL` filters passing 100% of rows, measured `6.00M/6.00M` on TPC-H SF=10) but not an end-to-end runtime comparison yet. We agree the `is_not_null` kernel itself is fast (null-buffer inversion) and `filter_record_batch` with an all-true mask is zero-copy in Arrow, so the remaining win is only the per-batch fixed overhead (FilterExecStream wrapper, coalescer, metrics) — which may well be noise-level. We will prepare a benchmark before pushing this further; if it shows no measurable improvement, we're happy to close this PR and keep the discussion documented in #24942 instead. Also fixed the last CI nit in d505b21f9 (single-node physical plan renders inline in explain output) — the remaining sqllogictest diffs across jobs were all that one formatting expectation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
