gortiz commented on code in PR #15245: URL: https://github.com/apache/pinot/pull/15245#discussion_r2024207226
########## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/blocks/MseBlock.java: ########## @@ -0,0 +1,178 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.pinot.query.runtime.blocks; + +import org.apache.pinot.common.utils.DataSchema; +import org.apache.pinot.core.common.Block; + + +/// Blocks used by [MultiStageOperator][org.apache.pinot.query.runtime.operator.MultiStageOperator] to share +/// information. +/// +/// Blocks always go from upstream (the children of the operator) to downstream (the parent of the operator) and can be +/// classified in the following categories: +/// - [Data] blocks: contain data that can be processed by the operator. +/// - [Eos] blocks: signal the end of a stream. These blocks can be either [successful][SuccessMseBlock] or +/// [error][ErrorMseBlock]. +/// +/// ## The MseBlock API +/// A MseBlock itself is not very useful, as they have almost no methods. +/// Instead, they are used as a common sub-interface for [data][Data] and [end-of-stream][Eos] blocks, +/// which are then subclassed to provide the actual functionality. +/// This pattern follows the principles of Java 17 sealed interfaces and the intention is implement them as such once +/// Pinot source code is migrated to Java 17 or newer, specially in Java 21 where pattern matching can also be used, +/// removing the need for the [Visitor] pattern. +/// +/// Meanwhile, the API force callers to do some castings, but it is a trade-off to have a more robust and maintainable +/// codebase given that we can rely on Java typesystem to verify some important properties at compile time instead of +/// adding runtime checks. +/// +/// The alternative to this pattern would be to have a single class with all methods, adding runtime checks to verify +/// whether it is acceptable to call a method or not. This is the approach that was used in the removed +/// TransferableBlock class, which was used for all possible block type combinations. As a result each method +/// had to include a runtime check to verify if it was legal to call it given some conditions imposed by its attributes. +/// This approach was error-prone and hard to maintain, as it was easy to forget to add a check in a new method or to +/// know which methods could be called at a given point in time. +/// +/// ## MseBlock vs DataBlock +/// MseBlock are conceptually close to [DataBlocks][org.apache.pinot.common.datablock.DataBlock]. +/// MseBlocks are sent from one operator to another while DataBlocks are a way to codify data. It is important to notice +/// that MseBlocks don't store stats, while DataBlocks do. +/// +/// When a MseBlock needs to be sent to another server, it will serialize it into a DataBlock. Then, when a DataBlock +/// is received by another server, it will deserialize it into a MseBlock (plus stats if needed). This is done by Review Comment: It is a design decision. MseBlock not having stats improves the quality of the code. Current TransferableBlocks include stats just because we use them as the mechanism to collect stats. This PR uses a different mechanism that is cleaner: Once the send operator is going to send the EOS or Error block, stats are calculated. You can see that new operators are easier to read because of that and we even fixed a bug in the intersection operator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
