fordN opened a new pull request, #23585: URL: https://github.com/apache/datafusion/pull/23585
## Which issue does this PR close? - Backport of https://github.com/apache/datafusion/pull/23395 to branch-54 (https://github.com/apache/datafusion/issues/22547) - Closes #23220. ## Rationale for this change The reported bug showed that when volatile (non-deterministic) expressions are referenced multiple times in an outer function the query results can be incorrect. To produce correct results, a volatile expression (e.g. `random()`, `uuid()`) aliased once in a subquery and referenced multiple times must be evaluated once and reused. Since 52.0.0 this evaluation pattern for volatile functions has been broken; the physical projection-pushdown rule merges the outer projection into the file `DataSourceExec`, inlining the aliased volatile expression at each reference site. Instead of being evaluated once, the volatile expression is evaluated N times, so the references diverge: ```sql SELECT s.r AS x, s.r AS y FROM (SELECT random() AS r FROM t) AS s; -- x != y on >= 52.0.0 ``` This was correct in 51.0.0 and regressed in 52.0.0/53.0.0. It reproduces on file scans (Parquet/CSV) but not in-memory tables, and was surfaced downstream in Ibis. Worth noting, #10337 appears to report the same class of bug but a different cause (I'll try to look into that issue soon as well). ## What changes are included in this PR? `FileScanConfig::try_swapping_with_projection` now declines to merge a projection into the file source when the merge would inline a volatile expression that the incoming projection references. The check reuses the existing `is_volatile()` utility from `datafusion_physical_expr_common` — the same volatility gate the physical `ProjectionPushdown` and `FilterPushdown` rules already apply — so file-source projection merging is now consistent with them. Deterministic expressions still merge freely. The guard blocks when a volatile inner expression is referenced at least once (not only more than once), because a single outer expression can itself duplicate the value (e.g.`r + r`). ## Are these changes tested? Yes: - Unit tests in `file_scan_config.rs`: - volatile referenced ≥1× → blocked; - deterministic computed and column-only → allowed; - unreferenced volatile → allowed; - single-expression self-reference (`r + r`) → blocked; - volatile nested in arithmetic → blocked; - empty → allowed. - A regression test in `projection_pushdown.slt` asserting `x = y` for the aliased-`random()` pattern over a Parquet scan. - Full `datafusion-sqllogictest` suite passes. ## Are there any user-facing changes? No API changes! Query results for the affected pattern are corrected (a volatile expression aliased in a subquery is no longer duplicated by projection pushdown into a file scan). Physical plans for such queries now retain a `ProjectionExec` above the scan rather than inlining the volatile expression into the `DataSourceExec` projection. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
