asolimando commented on issue #21583: URL: https://github.com/apache/datafusion/issues/21583#issuecomment-5992705361
Apache Hive has a [configuration option](https://github.com/apache/hive/blob/925df927248c1be11b84df3e8a0c3e43955e133e/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1906) driving the use of either formula: the default is the current implementation that picks the max NDV value (for Hive it's [`maxNdvForCorrelatedColumns`](https://github.com/apache/hive/blob/925df927248c1be11b84df3e8a0c3e43955e133e/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java#L207)), which works well for correlated columns. When columns are independent, Hive uses the same exponential backoff formula proposed here: [HiveRelMdSelectivity.java#L217-L235](https://github.com/apache/hive/blob/925df927248c1be11b84df3e8a0c3e43955e133e/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java#L217-L235). So I am unsure we can just replace one formula with another. Until we have information about column independence, I think we will need a similar configuration option, as Apache Hive does. The `StatisticsRegistry` framework can already carry such information through statistics extensions (`ExtendedStatistics`) from a custom provider, without changes to DataFusion (see [this test](https://github.com/apache/datafusion/blob/982fca6907fcfc12051fe64b989f3e708459a133/datafusion/physical-plan/src/statistics.rs#L679-L720)), but the built-in join estimate does not read it yet. If in the future we standardize how to pass and consume optional information about correlated columns in DataFusion (for example a distinct count over the join key columns together, the kind of statistic PostgreSQL collects with [`CREATE STATISTICS ... (ndistinct)`](https://www.postgresql.org/docs/current/planner-stats.html#PLANNER-STATS-EXTENDED-N-DISTINCT-COUNTS)), we can propagate this extended statistics by default, and the join cardinality estimate can use it when it is available, and fall back to the user-provided configuration otherwise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
