aokolnychyi commented on code in PR #6524:
URL: https://github.com/apache/iceberg/pull/6524#discussion_r1062923292
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java:
##########
@@ -106,41 +106,41 @@ public SparkScanBuilder caseSensitive(boolean
isCaseSensitive) {
@Override
public Filter[] pushFilters(Filter[] filters) {
List<Expression> expressions =
Lists.newArrayListWithExpectedSize(filters.length);
- List<Filter> pushed = Lists.newArrayListWithExpectedSize(filters.length);
+ List<Filter> pushableFilters =
Lists.newArrayListWithExpectedSize(filters.length);
+ List<Filter> postScanFilters =
Lists.newArrayListWithExpectedSize(filters.length);
for (Filter filter : filters) {
- Expression expr = null;
try {
- expr = SparkFilters.convert(filter);
Review Comment:
We also have to wrap the code that converts the filter. It is unlikely to
throw an exception but we have to make sure it does not fail the query. There
are 3 calls that can throw an exception now. I did not want to have nested
try-catch because it looked to complicated with added logic.
Github renders it in a way that's really hard to read. It does not seem to
be that bad in IDE.
```
try {
Expression expr = SparkFilters.convert(filter);
if (expr != null) {
// try binding the expression to ensure it can be pushed down
Binder.bind(schema.asStruct(), expr, caseSensitive);
expressions.add(expr);
pushableFilters.add(filter);
}
if (expr == null || requiresSparkFiltering(expr)) {
postScanFilters.add(filter);
} else {
LOG.info("Evaluating completely on Iceberg side: {}", filter);
}
} catch (Exception e) {
LOG.warn("Failed to check if {} can be pushed down: {}", filter,
e.getMessage());
postScanFilters.add(filter);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
