Jackie-Jiang commented on a change in pull request #6503:
URL: https://github.com/apache/incubator-pinot/pull/6503#discussion_r567067659
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -278,11 +280,48 @@ public static void validateIngestionConfig(TableConfig
tableConfig, @Nullable Sc
"Arguments of a transform function '" + arguments + "' cannot
contain the destination column '"
+ columnName + "'");
}
+ columnToFunctionEvaluator.put(columnName, expressionEvaluator);
+ columnToTransformExpression.put(columnName, transformFunction);
+ }
+ Map<String, Integer> transformFunctionChainDepth = new HashMap<>();
+ for (String column : columnToFunctionEvaluator.keySet()) {
+ if (transformFunctionChainDepth(column, columnToFunctionEvaluator,
transformFunctionChainDepth) > 2) {
+ Set<String> derivedArguments =
columnToFunctionEvaluator.get(column).getArguments().stream()
+ .filter(a -> transformFunctionChainDepth.get(a) ==
2).collect(Collectors.toSet());
+ throw new IllegalStateException(String.format(
+ "Derived columns: [%s] cannot be used as arguments to the
transform function: %s of derived column: %s.",
+ derivedArguments, columnToTransformExpression.get(column),
column));
+ }
}
}
}
}
+ /**
+ * Returns the depth of chaining in transform functions. Eg:
Review comment:
I changed my PR to not generating the column when the derived one cannot
be created so that user has a chance to fix it without re-downloading the
segment. But yes, I think we might not need this validation as segment creation
job can handle this properly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
