navina commented on a change in pull request #8047:
URL: https://github.com/apache/pinot/pull/8047#discussion_r789193506
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/stream/PartitionGroupConsumer.java
##########
@@ -39,4 +58,19 @@
*/
MessageBatch fetchMessages(StreamPartitionMsgOffset startOffset,
StreamPartitionMsgOffset endOffset, int timeoutMs)
throws TimeoutException;
+
+ /**
+ * Checkpoints the consumption state of the stream partition group in the
source
+ *
+ * This is useful in systems that require preserving consumption state on
the source in order to resume or replay
+ * consumption of data.
+ * The offset returned will be used for offset comparisons and persisted to
the ZK segment metadata. Hence, the
Review comment:
catching up? catching up happens only in non-winner replicas, iiuc. This
method won't even be invoked as only winning replicas will checkpoint to
source.
offset returned will be persisted in ZK as the endOffset of this segment and
used as startingOffset of the next segment. It should still be comparable
within the server. This system-specific implementation of
`StreamPartitionMsgOffset` should provide these guarantees.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
