ConeyLiu commented on code in PR #7886:
URL: https://github.com/apache/iceberg/pull/7886#discussion_r1264463187
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -63,4 +63,9 @@ private SparkSQLProperties() {}
// Controls the WAP branch used for write-audit-publish workflow.
// When set, new snapshots will be committed to this branch.
public static final String WAP_BRANCH = "spark.wap.branch";
+
+ // Controls whether to push down Iceberg system function
+ public static final String SYSTEM_FUNC_PUSH_DOWN_ENABLED =
+ "spark.sql.iceberg.system-function-push-down.enabled";
+ public static final boolean SYSTEM_FUNC_PUSH_DOWN_ENABLED_DEFAULT = false;
Review Comment:
Compared with `StaticInvoke`, `ApplyFunctionExpression` may have a little
decrease in performance because it can not leverage codegen.
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkV2Filters.java:
##########
@@ -360,10 +417,70 @@ private static boolean hasNoInFilter(Predicate predicate)
{
}
private static boolean isSupportedInPredicate(Predicate predicate) {
- if (!isRef(childAtIndex(predicate, 0))) {
+ if (!canConvertToTerm(childAtIndex(predicate, 0))) {
return false;
} else {
return
Arrays.stream(predicate.children()).skip(1).allMatch(SparkV2Filters::isLiteral);
}
}
+
+ /** Should be called after {@link #canConvertToTerm} passed */
+ private static <T> UnboundTerm<Object> toTerm(T input) {
+ if (input instanceof NamedReference) {
+ return Expressions.ref(SparkUtil.toColumnName((NamedReference) input));
+ } else {
+ return udfToTerm((UserDefinedScalarFunc) input);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ private static UnboundTerm<Object> udfToTerm(UserDefinedScalarFunc udf) {
+ switch (udf.name().toLowerCase(Locale.ROOT)) {
+ case "years":
+ if (udf.children().length == 1 && isRef(udf.children()[0])) {
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
