szehon-ho commented on code in PR #17956:
URL: https://github.com/apache/iceberg/pull/17956#discussion_r4010974568
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java:
##########
@@ -264,4 +293,98 @@ protected String groupingKeyDesc() {
.map(NestedField::name)
.collect(Collectors.joining(", "));
}
+
+ protected List<PartitionPredicate> partitionPredicates() {
+ return partitionPredicates;
+ }
+
+ private boolean matchesPartitionPredicates(
+ T task, Map<Integer, PartitionPredicateEvaluator> evaluatorsBySpecId) {
+ PartitionPredicateEvaluator evaluator =
+ evaluatorsBySpecId.computeIfAbsent(
+ task.spec().specId(), ignored -> new
PartitionPredicateEvaluator(task.spec()));
+ return evaluator.eval(task.partition());
+ }
+
+ private List<PartitionField> activePartitionFields(PartitionSpec spec) {
Review Comment:
Done in `9b2f995f15`. Made `activePartitionFields` and `activePartitionType`
private static methods.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/BaseSparkScanBuilder.java:
##########
@@ -62,19 +63,18 @@
abstract class BaseSparkScanBuilder implements ScanBuilder {
private static final Logger LOG =
LoggerFactory.getLogger(BaseSparkScanBuilder.class);
- private static final Predicate[] NO_PREDICATES = new Predicate[0];
-
private final SparkSession spark;
private final Table table;
private final Schema schema;
private final SparkReadConf readConf;
private final boolean caseSensitive;
private final Set<String> metaFieldNames = Sets.newLinkedHashSet();
private final InMemoryMetricsReporter metricsReporter = new
InMemoryMetricsReporter();
+ private final List<PartitionPredicate> partitionPredicates =
Lists.newArrayList();
+ private final List<Predicate> pushedPredicates = Lists.newArrayList();
private Schema projection;
private List<Expression> filters = Lists.newArrayList();
- private Predicate[] pushedPredicates = NO_PREDICATES;
Review Comment:
Done in `9b2f995f15`. Made `filters` final and moved it alongside the other
final fields.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/BaseSparkScanBuilder.java:
##########
@@ -62,19 +63,18 @@
abstract class BaseSparkScanBuilder implements ScanBuilder {
private static final Logger LOG =
LoggerFactory.getLogger(BaseSparkScanBuilder.class);
- private static final Predicate[] NO_PREDICATES = new Predicate[0];
Review Comment:
Done in `9b2f995f15`. Restored the blank line between the static logger and
instance fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
