walterddr commented on code in PR #9782:
URL: https://github.com/apache/pinot/pull/9782#discussion_r1023369641
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java:
##########
@@ -0,0 +1,120 @@
+package org.apache.pinot.query.runtime.operator;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.PriorityQueue;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.datablock.DataBlock;
+import org.apache.pinot.common.datablock.DataBlockUtils;
+import org.apache.pinot.common.datablock.MetadataBlock;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.blocks.InstanceResponseBlock;
+import org.apache.pinot.core.operator.blocks.results.BaseResultsBlock;
+import org.apache.pinot.query.runtime.blocks.TransferableBlock;
+
+
+/**
+ * Leaf-stage transfer block operator is used to wrap around the leaf stage
process results. They are passed to the
+ * Pinot server to execute query thus only one {@link DataTable} were
returned. However, to conform with the
+ * intermediate stage operators. an additional {@link MetadataBlock} needs to
be transfer after the data block.
+ *
+ * <p>In order to achieve this:
+ * <ul>
+ * <li>The leaf-stage result is split into data payload block and metadata
payload block.</li>
+ * <li>In case the leaf-stage result contains error or only metadata, we
skip the data payload block.</li>
+ * <li>Leaf-stage result blocks are in the {@link
DataSchema.ColumnDataType#getStoredColumnDataTypes()} format
+ * thus requires canonicalization.</li>
Review Comment:
since we move leaf stage canonicalization out. we
1. no longer need to add test to TransferableBlockUtils for this handling
2. no longer need to test under splitBlock conditions
3. no longer need to handle in-place updates
the drawback is we create one additional list. but we will remove this class
once we conform with the desired schema anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
