kita-renji opened a new pull request, #25764: URL: https://github.com/apache/datafusion/pull/25764
## Which issue does this PR close? - Closes #25507. ## Rationale for this change `PullUpCorrelatedExpr` pulls a correlated `Filter` up through any `Join`. That is fine for an inner join and for the preserved side of an outer join, but below the side an outer join fills with NULLs the filter decides which rows are unmatched, so pulling it above the join changes the result. With the tables from the issue, `EXISTS (SELECT 1 FROM a LEFT JOIN (SELECT * FROM b WHERE b.y = o.k) AS b ON a.id = b.id WHERE b.y IS NULL)` returns `false` for both rows instead of `true`, and the `IN` form returns `false` instead of `NULL` for `o.k = 5`. The same happens for `WHERE [NOT] EXISTS`, scalar subqueries, the left side of a `RIGHT JOIN`, either side of a `FULL JOIN`, a filter nested under an inner join on the nullable side, and `LATERAL` subqueries (a `t1, LATERAL (... t3 LEFT JOIN (... WHERE t2.t1_id = t1.id) ...)` query returns 3 rows instead of 9). ## What changes are included in this PR? `PullUpCorrelatedExpr::f_down` now has an arm for `LogicalPlan::Join`: if an input the join does not preserve holds outer references, the subquery is marked as not pull-up-able, the same way it already is for a `Union` or `Sort` with outer references. Which sides are preserved comes from `lr_is_preserved` in `push_down_filter.rs`, which filter pushdown uses to decide whether a filter above a join can move into one side. That is the same question asked the other way around. The non-output side of a semi, anti or mark join counts as not preserved. The check stops at `LogicalPlan::Subquery` nodes, since their outer references belong to a nested scope (for example the right side of a nested `LATERAL` that is decorrelated separately). `f_down` already treats `Subquery` as a scope boundary. Such subqueries now stay correlated and fail with a not-implemented error instead of returning wrong results. Correlated filters on a preserved side, on either side of an inner join, or above the outer join are still pulled up as before. This is independent of #25284, which adds an `unsupported()` helper for the same pattern. I kept the diff away from the lines it touches so both merge cleanly (I checked the merge locally); whichever lands second can switch the new arm to `self.unsupported(plan)`. ## Are these changes tested? Yes, new sqllogictest cases in `subquery.slt` and `lateral_join.slt`: - EXPLAIN showing the subquery from the issue stays correlated. - `EXISTS` / `IN` in a projection, `WHERE EXISTS`, `WHERE NOT EXISTS`, a scalar subquery, `RIGHT JOIN`, `FULL JOIN` (both sides), a filter nested under an inner join on the nullable side, and a `LATERAL` subquery now error instead of returning wrong results. All of these fail without the fix. - A correlated filter on the preserved side of `LEFT` and `RIGHT` joins, on the nullable side of an inner join, and above a `LEFT JOIN` is still decorrelated and returns the right rows (checked against DuckDB). The whole sqllogictest suite passes with no changes to existing plans. ## Are there any user-facing changes? Queries with a correlated filter below the nullable side of an outer join inside a subquery now fail with a not-implemented error instead of silently returning wrong results. Supporting them properly needs a different decorrelation strategy. I can open a follow-up issue for that. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
