npawar commented on a change in pull request #6503:
URL: https://github.com/apache/incubator-pinot/pull/6503#discussion_r567030092
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -278,11 +280,48 @@ public static void validateIngestionConfig(TableConfig
tableConfig, @Nullable Sc
"Arguments of a transform function '" + arguments + "' cannot
contain the destination column '"
+ columnName + "'");
}
+ columnToFunctionEvaluator.put(columnName, expressionEvaluator);
+ columnToTransformExpression.put(columnName, transformFunction);
+ }
+ Map<String, Integer> transformFunctionChainDepth = new HashMap<>();
+ for (String column : columnToFunctionEvaluator.keySet()) {
+ if (transformFunctionChainDepth(column, columnToFunctionEvaluator,
transformFunctionChainDepth) > 2) {
+ Set<String> derivedArguments =
columnToFunctionEvaluator.get(column).getArguments().stream()
+ .filter(a -> transformFunctionChainDepth.get(a) ==
2).collect(Collectors.toSet());
+ throw new IllegalStateException(String.format(
+ "Derived columns: [%s] cannot be used as arguments to the
transform function: %s of derived column: %s.",
+ derivedArguments, columnToTransformExpression.get(column),
column));
+ }
}
}
}
}
+ /**
+ * Returns the depth of chaining in transform functions. Eg:
Review comment:
true.
i'm starting to think, this validation is an overkill. You have already
handled in your PR, that if the column is not found, then just generate
default. By adding this validation, we will be preventing derived columns even
in cases where it could have been handled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
