friendlymatthew opened a new issue, #21307: URL: https://github.com/apache/datafusion/issues/21307
### Is your feature request related to a problem or challenge? `ProjectionExec::project_statistics()` only propagates column statistics for plain `Column` references and `Literal` values. Any `ScalarFunctionExpr` like `get_field()`, `variant_get`, or any UDF produces `ColumnStatistics::new_unknown()`: https://github.com/apache/datafusion/blob/bc2b36cf56846e0c697b0f8b98619f346a72a9bf/datafusion/physical-expr/src/projection.rs#L716-L719 This means every column produced by a scalar function has `Absent` min/max/null_count/distinct_count, even when the function is a pure extraction (`get_field`, `variant_get`), a monotonic transformation (`cast`, `abs`), or any UDF where output stats are derivable from input stats ### Describe the solution you'd like Add an optional method to `ScalarUdfImpl`: something like `fn output_statistics(&self, input_statistics: &[ColumnStatistics]) -> Option<ColumnStatistics>` This way, `ProjectionExec::project_statistics` would call this before falling back to unknown ### Describe alternatives you've considered _No response_ ### Additional context This matters for struct/variant queries especially, most output columns come through `get_field` or equivalent UDFs, so the cost model is effectively blind! - **cost based join ordering** can't estimate cardinality for join keys produced by udfs - **aggregation planning** can't estimate group count for group by on udf outputs - **filterexec selectivity** - can't narrow row estimates for filters on udf outputs Here's an MRE: https://github.com/apache/datafusion/compare/main...pydantic:datafusion:stats-propagation-mre?expand=1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
