jayzhan211 commented on code in PR #25576:
URL: https://github.com/apache/datafusion/pull/25576#discussion_r4165717644
##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -963,17 +961,14 @@ fn summarize_distinct_counts(
// Return early if there's no chance to reach the required coverage.
let remaining = num_row_groups - row_group_idx - 1;
if ndv_count + remaining < required_count {
- return Precision::Absent;
+ return Ok(Precision::Absent);
}
}
- match max_distinct_count {
- Some(distinct_count) if num_row_groups == 1 => {
- Precision::Exact(distinct_count as usize)
- }
+ Ok(match max_distinct_count {
Review Comment:
Dropping single-row-group NDV to `Inexact` is required, not just
conservative. arrow-rs's NDV is hash-based, and for dictionary columns it
hashes keys. With the old `Exact` rule, a file written with this option gives
the wrong `COUNT(DISTINCT)` result from stats. Repro with `main`'s rule put
back: `2 6`, expected `6 6`. This turns off the stats fold for all parquet
files (see the `clickbench.slt` change), so please state it in the PR
description / upgrade notes. Please also pin it in `parquet_ndv_write.slt`:
```sql
statement ok
set datafusion.execution.batch_size = 2;
query I
COPY (SELECT arrow_cast(column1, 'Dictionary(Int32, Utf8)') AS d, column2 AS
x
FROM (VALUES ('a', 1), ('b', 2), ('c', 3), ('d', 4), ('e', 5), ('f',
6)))
TO 'test_files/scratch/parquet_ndv_write/dict.parquet'
STORED AS PARQUET;
----
6
statement ok
CREATE EXTERNAL TABLE dict_ndv
STORED AS PARQUET
LOCATION 'test_files/scratch/parquet_ndv_write/dict.parquet';
# NDV written for `d` is wrong (2), so it must not be used as Exact
query II
SELECT count(DISTINCT d), count(DISTINCT x) FROM dict_ndv;
----
6 6
```
##########
datafusion/common/src/config.rs:
##########
@@ -1514,6 +1514,12 @@ config_namespace! {
/// default parquet writer setting
pub bloom_filter_ndv: Option<u64>, default = None
+ /// (writing) Write the number of distinct values (NDV) for each column
Review Comment:
arrow-rs `ArrowColumnWriter::write_internal` hashes dictionary **keys** to
count distinct values. Keys from batches with different dictionaries collide,
so a 6-value dictionary column written in 2-row batches is stored as
`Distinct=Inexact(2)`. Results stay correct now that NDV is `Inexact`, but the
estimate is badly off. Fine to handle in a follow-up: please file an arrow-rs
issue and mention the limitation in this option's doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
