npawar opened a new issue #8052: URL: https://github.com/apache/pinot/issues/8052
PR https://github.com/apache/pinot/pull/7756/files introduced a check for liveTable in the path of adding new partitionGroups. This branch was introduced in `setupNewPartitionGroup` (prior to this PR we had just the else part) ``` if (isLiveTable) { startOffset = getPartitionGroupSmallestOffset(streamConfig, partitionGroupId); } else { startOffset = partitionGroupMetadata.getStartOffset(); } ``` For new table, we go to the `else`, and for existing table detecting a new partitionGroup we go to `if`. Within the `if`, a call is made to `getNewPartitionGroupMetadataList(streamConfig, Collections.emptyList());`. For Kinesis like streams, the response from this call depends on the currentList passed. If a Kinesis like stream received empty list, it will only return the very first parent shards. This is because in Kinesis, the shards have a sequence (we started with 0, split it to 1, 2, so 1, 2 will only be returned if the current state tells it that 0 is done ingesting). As a result of this PR change, no new shards can get detected in Kinesis. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
