amogh-jahagirdar commented on code in PR #16692:
URL: https://github.com/apache/iceberg/pull/16692#discussion_r3482556371
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java:
##########
@@ -128,6 +130,27 @@ public Boolean or(Boolean leftResult, Boolean rightResult)
{
return leftResult || rightResult;
}
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> Boolean predicate(BoundPredicate<T> pred) {
+ // A column that is absent from this file but carries an initial-default
reads as the default
+ // for every row, not as null. The per-predicate handlers below assume
an absent column is all
+ // nulls (valueCount == null), which would skip the row group and drop
the backfilled rows
+ // (e.g. for col = <default> or the IsNotNull engines infer). Evaluate
such predicates against
+ // the default value instead. See #16690.
+ if (pred.term() instanceof BoundReference) {
+ int id = ((BoundReference<T>) pred.term()).fieldId();
+ if (!valueCounts.containsKey(id)) {
Review Comment:
I was discussing with @cbb330 we (or at least I) may have just misread this.
This is valueCounts which are required
https://github.com/apache/parquet-format/blame/8a5e04bdecf100e8e981daacfa117e8b5aadacb9/src/main/thrift/parquet.thrift#L903
So @pvary I think we can continue to use value counts as a reliable
indication of a fields presence without having to check the file schema, I have
some other changes in my branch to his PR for building up the fields with
defaults into a map one time among some other test changes which I think is
just slightly better for the case where we expect default values to generally
be rare.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
