jamesyfshao commented on a change in pull request #5394: URL: https://github.com/apache/incubator-pinot/pull/5394#discussion_r446714977
########## File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/callback/DataManagerCallback.java ########## @@ -0,0 +1,106 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.core.data.manager.callback; + +import org.apache.pinot.core.data.manager.SegmentDataManager; +import org.apache.pinot.spi.annotations.InterfaceStability; +import org.apache.pinot.spi.data.readers.GenericRow; +import org.apache.pinot.spi.stream.MessageBatch; +import org.apache.pinot.spi.stream.StreamPartitionMsgOffset; + +import java.io.IOException; + +/** + * Component inject to {@link org.apache.pinot.core.data.manager.SegmentDataManager} for handling extra logic for + * upsert-enabled pinot ingestion mode. + */ +@InterfaceStability.Evolving +public interface DataManagerCallback { + + void init() throws IOException; + + /** + * create a {@link IndexSegmentCallback} object to allow SegmentDataManager to create proper IndexSegment that supports + * either append/upsert mode + * + * In append-tables callback, this method will create a DefaultIndexSegmentCallback + * In upsert-tables callback, this method will create a UpsertDataManagerCallbackImpl + * + * The {@link IndexSegmentCallback} will be used in the constructor of IndexSegment + */ + IndexSegmentCallback getIndexSegmentCallback(); Review comment: So for dataManager in pinot, there are mostly 3 layer: 1. TableDataManager: this is the class that handles when a segment should be add/remove from a pinot server and the handling of all segment lock. We add a callback for this class _TableDataManagerCallback_ to ensure that we can add listener for segment addition & removal operation in Pinot 2. DataManager: this is the class that handles the ingestion from input stream. As some of the information related to the input stream is only visible at this layer (aka offset information), we are adding _DataManagerCallback_ that interact with the offset information and sending information to key-coordinator queue here to make it consistent. 3. IndexSegment: this is the class that handles the interaction between incoming data and underlying storage (either in-memory or on-disk). There are some information that is only visible at this layer (actual data storage, docId information, etc). We are adding _IndexSegmentCallback_ that interact with the physical data storage and access the docId information here. We *could* make all 3 of these callback into one callback class and make the all 3 managers I listed above to interact with this one single callback, but I feel make the callbacks function unclear and hard to get to know intention of each method. By separating the corresponding callback class out, we make each them more focus on their own interaction and self-contain. However, it would also means that we need a way to create the appropriate callback objects from the higher level callback object (the same way we create a DataManager from TableDataManager, and IndexSegment from DataManager). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
