npawar commented on a change in pull request #5681:
URL: https://github.com/apache/incubator-pinot/pull/5681#discussion_r453979435
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/util/IngestionUtils.java
##########
@@ -67,12 +70,24 @@ private static void extractFieldsFromSchema(Schema schema,
Set<String> fields) {
* Extracts the fields needed by a RecordExtractor from given {@link
IngestionConfig}
*/
private static void extractFieldsFromIngestionConfig(@Nullable
IngestionConfig ingestionConfig, Set<String> fields) {
- if (ingestionConfig != null && ingestionConfig.getFilterConfig() != null) {
- String filterFunction =
ingestionConfig.getFilterConfig().getFilterFunction();
- if (filterFunction != null) {
- FunctionEvaluator functionEvaluator =
FunctionEvaluatorFactory.getExpressionEvaluator(filterFunction);
- if (functionEvaluator != null) {
- fields.addAll(functionEvaluator.getArguments());
+ if (ingestionConfig != null) {
+ FilterConfig filterConfig = ingestionConfig.getFilterConfig();
+ if (filterConfig != null) {
+ String filterFunction = filterConfig.getFilterFunction();
+ if (filterFunction != null) {
+ FunctionEvaluator functionEvaluator =
FunctionEvaluatorFactory.getExpressionEvaluator(filterFunction);
+ if (functionEvaluator != null) {
+ fields.addAll(functionEvaluator.getArguments());
+ }
+ }
+ }
+ List<TransformConfig> transformConfigs =
ingestionConfig.getTransformConfigs();
+ if (transformConfigs != null) {
+ for (TransformConfig transformConfig : transformConfigs) {
+ FunctionEvaluator expressionEvaluator =
+
FunctionEvaluatorFactory.getExpressionEvaluator(transformConfig.getTransformFunction());
+ fields.addAll(expressionEvaluator.getArguments());
+ fields.add(transformConfig.getColumnName());
Review comment:
if transformation is already done at source, this ensures we avoid doing
it again. This behavior is being carried over from previous behavior when it
used to be in schema.
Added a comment too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
