gortiz commented on code in PR #10191: URL: https://github.com/apache/pinot/pull/10191#discussion_r1093056041
########## pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/IndexType.java: ########## @@ -0,0 +1,171 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.segment.spi.index; + +import com.fasterxml.jackson.databind.JsonNode; +import java.io.IOException; +import java.util.Map; +import javax.annotation.Nullable; +import org.apache.pinot.segment.spi.ColumnMetadata; +import org.apache.pinot.segment.spi.creator.IndexCreationContext; +import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer; +import org.apache.pinot.segment.spi.store.SegmentDirectory; +import org.apache.pinot.spi.config.table.FieldConfig; +import org.apache.pinot.spi.config.table.TableConfig; +import org.apache.pinot.spi.data.Schema; +import org.apache.pinot.spi.utils.JsonUtils; + + +/** + * TODO: implement mutable indexes. + * @param <C> the class that represents how this object is configured. + * @param <IR> the {@link IndexReader} subclass that should be used to read indexes of this type. + * @param <IC> the {@link IndexCreator} subclass that should be used to create indexes of this type. + */ +public interface IndexType<C, IR extends IndexReader, IC extends IndexCreator> { + + /** + * The unique id that identifies this index type. + * In case there is more than one implementation for a given index, then all should share the same id in order to be + * correctly registered in the {@link IndexService}. + * This is also the value being used as the default toString implementation and the one used as keys when config is + * specified. + * + * <p>Therefore the returned value for each index should be constant across different Pinot versions.</p> + */ + String getId(); + + /** + * Returns an internal name used in some parts of the code (mainly in format v1 and metadata) that is persisted on + * disk. + * + * <p>Therefore the returned value for each index should be constant across different Pinot versions.</p> + */ + String getIndexName(); + + default Class<C> getIndexConfigClass() { + throw new UnsupportedOperationException(); + } + + /** + * The default config when it is not explicitly defined by the user. + * + * Can return null if the index should be disabled by default. + */ + @Nullable + default C getDefaultConfig() { + return null; + } + + /** + * This method is called to transform from a JSON node to a config object. + * + * This is usually used to deserialize {@link FieldConfig#getIndexes() fieldConfigLists.indexes.(indexId)}. + * @throws IOException + */ + @Nullable + default C deserialize(JsonNode node) + throws IOException { + return JsonUtils.jsonNodeToObject(node, getIndexConfigClass()); + } + + /** + * This method can be overridden by indexes that support alternative configuration formats where the configuration is + * spread on different fields in the TableConfig. + * + * Configuration that can be read from the {@link FieldConfig#getIndexes() fieldConfigLists.indexes} shall not be + * included here. + */ + default IndexDeclaration<C> deserializeSpreadConf(TableConfig tableConfig, Schema schema, String column) { Review Comment: I don't think we should, for several reasons: 1. It is quite more repetitive (why do we always need to add this fragment?). We can add an abstract implementation that includes the repetitive code, but then the subclasses will have the same methods we have right now. 2. It means that whenever we want to instantiate a config object we either need to instantiate it directly by calling the constructor or we would need to have a `TableConfig` and a `Schema`. Right now we can also deserialize from a JsonNode or a String (with `JsonUtils.stringToJsonNode(str)`). 3. As shown in https://github.com/apache/pinot/pull/10184, we are going to need to transform all the indexes into a `Map<String, FieldIndexConfigs>`. Therefore we are going to need to add some logic to read all the indexes, which means that we are going to have to use some utility class to read the. Therefore the glue that calls each deserialize method isn't going to add boilerplate in the calling code. This is important: **These methods are not going to be directly called by code that needs to deserialize config**. There are not examples in this PR because it is a partial PR. 4. We still need to have the `Map<String, IndexDeclaration<C>> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig)` to support current tests. This is not included here because it is a partial PR. 5. New indexes are not going to need to implement `deserializeSpreadConf` and they are not going to need to implement `C deserialize(JsonNode node)` if and only if they require to do some fancy thing with Jackson. Therefore most of the time new indexes are not going to need to implement neither of these methods. We need to be careful when analyzing these partial PRs. Some design decisions (like this one) may have been taken in a specific way because there is a requirement in the PR with all the changes and it may be difficult to understand if only the partial PR is taken into consideration. I've added this note explaining that in the general issue https://github.com/apache/pinot/issues/10183#issuecomment-1411839122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
