aokolnychyi commented on code in PR #10943:
URL: https://github.com/apache/iceberg/pull/10943#discussion_r1840744978
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/BaseRowReader.java:
##########
@@ -48,6 +50,17 @@ abstract class BaseRowReader<T extends ScanTask> extends
BaseReader<InternalRow,
super(table, taskGroup, tableSchema, expectedSchema, caseSensitive);
}
+ BaseRowReader(
+ Table table,
+ ScanTaskGroup<T> taskGroup,
+ Schema tableSchema,
+ Schema expectedSchema,
+ boolean caseSensitive,
+ Integer pushedLimit) {
+ this(table, taskGroup, tableSchema, expectedSchema, caseSensitive);
+ this.pushedLimit = pushedLimit;
Review Comment:
My initial thought is that we can use `ExpressionUtil$selectsPartitions` to
test if the filter selects entire files. If so, we can simply find matching
files so that the sum of `recordCount` exceeds the pushed limit. It may be more
complicated for filters that only select some records within files. Let me
explore this PR more for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
