szehon-ho commented on code in PR #7582:
URL: https://github.com/apache/iceberg/pull/7582#discussion_r1253852920
##########
core/src/main/java/org/apache/iceberg/PositionDeletesTable.java:
##########
@@ -155,30 +177,75 @@ protected List<String> scanColumns() {
return context().returnColumnStats() ? DELETE_SCAN_WITH_STATS_COLUMNS :
DELETE_SCAN_COLUMNS;
}
+ /**
+ * Sets a filter on the base table to use for this scan.
+ *
+ * <p>This must be set exclusively from {@link #filter(Expression)}
+ *
+ * @param expr expression filter on base table
+ * @return this for method chaining
+ */
+ public BatchScan baseTableFilter(Expression expr) {
+ if (filterSet) {
+ throw new ValidationException("Filter condition is already set");
+ }
+
+ return new PositionDeletesBatchScan(
+ table(),
+ schema(),
+ context(),
+ Expressions.and(baseTableFilter, expr),
+ true /* base filter set */,
+ filterSet);
+ }
+
+ /**
+ * Sets a filter to use for this scan.
+ *
+ * <p>This must be set exclusively from {@link
#baseTableFilter(Expression)}
+ *
+ * @param expr expression filter
+ * @return this for method chaining
+ */
+ @Override
+ public BatchScan filter(Expression expr) {
+ if (baseTableFilterSet) {
+ throw new ValidationException("Base table filter is already set");
+ }
+
+ return new PositionDeletesBatchScan(
+ table(),
+ schema(),
+ context().filterRows(Expressions.and(context().rowFilter(), expr)),
+ baseTableFilter,
+ baseTableFilterSet,
+ true /* filter set */);
+ }
+
@Override
protected CloseableIterable<ScanTask> doPlanFiles() {
String schemaString = SchemaParser.toJson(tableSchema());
// prepare transformed partition specs and caches
- Map<Integer, PartitionSpec> transformedSpecs =
transformSpecs(tableSchema(), table().specs());
+ Map<Integer, PartitionSpec> transformedSpecs =
transformSpecsIfNecessary();
Review Comment:
OK , the latest change supports both filters now. I use another
ManifestEvaluator, gotten via ManifestEvaluator.forPartitionFilter(), which
internally does the Projection
##########
core/src/main/java/org/apache/iceberg/PositionDeletesTable.java:
##########
@@ -155,30 +177,75 @@ protected List<String> scanColumns() {
return context().returnColumnStats() ? DELETE_SCAN_WITH_STATS_COLUMNS :
DELETE_SCAN_COLUMNS;
}
+ /**
+ * Sets a filter on the base table to use for this scan.
+ *
+ * <p>This must be set exclusively from {@link #filter(Expression)}
+ *
+ * @param expr expression filter on base table
+ * @return this for method chaining
+ */
+ public BatchScan baseTableFilter(Expression expr) {
+ if (filterSet) {
+ throw new ValidationException("Filter condition is already set");
+ }
+
+ return new PositionDeletesBatchScan(
+ table(),
+ schema(),
+ context(),
+ Expressions.and(baseTableFilter, expr),
+ true /* base filter set */,
+ filterSet);
+ }
+
+ /**
+ * Sets a filter to use for this scan.
+ *
+ * <p>This must be set exclusively from {@link
#baseTableFilter(Expression)}
+ *
+ * @param expr expression filter
+ * @return this for method chaining
+ */
+ @Override
+ public BatchScan filter(Expression expr) {
+ if (baseTableFilterSet) {
+ throw new ValidationException("Base table filter is already set");
+ }
+
+ return new PositionDeletesBatchScan(
+ table(),
+ schema(),
+ context().filterRows(Expressions.and(context().rowFilter(), expr)),
+ baseTableFilter,
+ baseTableFilterSet,
+ true /* filter set */);
+ }
+
@Override
protected CloseableIterable<ScanTask> doPlanFiles() {
String schemaString = SchemaParser.toJson(tableSchema());
// prepare transformed partition specs and caches
- Map<Integer, PartitionSpec> transformedSpecs =
transformSpecs(tableSchema(), table().specs());
+ Map<Integer, PartitionSpec> transformedSpecs =
transformSpecsIfNecessary();
+
+ LoadingCache<Integer, String> specStringCache =
+ partitionCacheOf(transformedSpecs, PartitionSpecParser::toJson);
+
+ LoadingCache<Integer, ManifestEvaluator> evalCache =
+ partitionCacheOf(
+ transformedSpecs,
+ spec -> ManifestEvaluator.forRowFilter(effectiveFilter(), spec,
isCaseSensitive()));
LoadingCache<Integer, ResidualEvaluator> residualCache =
partitionCacheOf(
transformedSpecs,
spec ->
ResidualEvaluator.of(
spec,
- shouldIgnoreResiduals() ? Expressions.alwaysTrue() :
filter(),
+ shouldIgnoreResiduals() ? Expressions.alwaysTrue() :
effectiveFilter(),
Review Comment:
yea, made this back to filter()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
