Jackie-Jiang commented on a change in pull request #6494:
URL: https://github.com/apache/incubator-pinot/pull/6494#discussion_r566522822
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java
##########
@@ -324,8 +345,69 @@ protected void removeColumnV1Indices(String column)
*/
protected void createColumnV1Indices(String column)
throws Exception {
+ TableConfig tableConfig = _indexLoadingConfig.getTableConfig();
+ if (tableConfig != null && tableConfig.getIngestionConfig() != null
+ && tableConfig.getIngestionConfig().getTransformConfigs() != null) {
+ List<TransformConfig> transformConfigs =
tableConfig.getIngestionConfig().getTransformConfigs();
+ for (TransformConfig transformConfig : transformConfigs) {
+ if (transformConfig.getColumnName().equals(column)) {
+ String transformFunction = transformConfig.getTransformFunction();
+ FunctionEvaluator functionEvaluator =
FunctionEvaluatorFactory.getExpressionEvaluator(transformFunction);
+
+ // Check if all arguments exist in the segment
+ // TODO: Support chained derived column
+ boolean skipCreatingDerivedColumn = false;
+ List<String> arguments = functionEvaluator.getArguments();
+ List<ColumnMetadata> argumentsMetadata = new
ArrayList<>(arguments.size());
+ for (String argument : arguments) {
+ ColumnMetadata columnMetadata =
_segmentMetadata.getColumnMetadataFor(argument);
+ if (columnMetadata == null) {
+ LOGGER.warn(
+ "Skip creating derived column: {} because argument: {} does
not exist in the segment, creating default value column instead",
+ column, argument);
+ skipCreatingDerivedColumn = true;
+ break;
+ }
+ argumentsMetadata.add(columnMetadata);
+ }
+ if (skipCreatingDerivedColumn) {
+ break;
+ }
+
+ // TODO: Support raw derived column
+ if (_indexLoadingConfig.getNoDictionaryColumns().contains(column)) {
+ LOGGER.warn("Skip creating raw derived column: {}, creating
default value column instead", column);
Review comment:
I've considered that, but that requires more code change because we
assume the default column is always added, and the metadata will be changed
accordingly.
There is a work-around to re-download the segments if the user puts the
unsupported config, they can update the schema with a different default null
value, which will trigger the update of the default column.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
