amogh-jahagirdar commented on code in PR #16692:
URL: https://github.com/apache/iceberg/pull/16692#discussion_r3482556371
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java:
##########
@@ -128,6 +130,27 @@ public Boolean or(Boolean leftResult, Boolean rightResult)
{
return leftResult || rightResult;
}
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> Boolean predicate(BoundPredicate<T> pred) {
+ // A column that is absent from this file but carries an initial-default
reads as the default
+ // for every row, not as null. The per-predicate handlers below assume
an absent column is all
+ // nulls (valueCount == null), which would skip the row group and drop
the backfilled rows
+ // (e.g. for col = <default> or the IsNotNull engines infer). Evaluate
such predicates against
+ // the default value instead. See #16690.
+ if (pred.term() instanceof BoundReference) {
+ int id = ((BoundReference<T>) pred.term()).fieldId();
+ if (!valueCounts.containsKey(id)) {
Review Comment:
I was discussing with @cbb330 we (or at least I) may have just misread this.
This is valueCounts which are required
https://github.com/apache/parquet-format/blame/8a5e04bdecf100e8e981daacfa117e8b5aadacb9/src/main/thrift/parquet.thrift#L903
So @pvary I think we can continue to use this, I have some other changes in
my branch to his PR for building up the fields with defaults into a map one
time among some other test changes which I think is just slightly better for
the case where we expect default values to generally be rare.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
