xudong963 commented on code in PR #21907:
URL: https://github.com/apache/datafusion/pull/21907#discussion_r3182433912
##########
datafusion/datasource-parquet/src/row_group_filter.rs:
##########
@@ -272,6 +274,7 @@ impl RowGroupAccessPlanFilter {
parquet_schema,
row_group_metadatas,
arrow_schema,
+ missing_null_counts_as_zero: true,
Review Comment:
I checked the current DataFusion parquet reader options and I don’t think
this is currently exposed as a reader-level DataFusion option (We can open
another PR for it!). The existing behavior was implicitly the
StatisticsConverter default, which treats missing null counts as zero.
So the regular pruning path keeps that existing behavior.
The fully matched path is intentionally stricter because it is proving every
row passes the predicate. For that proof, missing null counts must remain
unknown; otherwise nullable columns with missing null_count can be incorrectly
marked fully matched.
I’ll add comments/named constructors to make this distinction clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
