sajjad-moradi opened a new pull request #7267:
URL: https://github.com/apache/pinot/pull/7267
## Description
This PR adds a new realtime consumption status checker which is used at
startup time to have a more accurate estimate of the catchup period in which no
query execution happens and consumers try to catch up to the latest messages
available in streams.
To achieve this, every time status check is called,
_haveAllConsumingSegmentsReachedStreamLatestOffset_ method, list of consuming
segments is gathered and then for each segment, we check if segment's latest
ingested offset has reached the latest stream offset. To prevent chasing a
moving target, once the latest stream offset is fetched, it will not be fetched
again and subsequent status check calls compare latest ingested offset with the
already fetched stream offset.
The new status checker is being added in two phases. First phase adds the
new status checker, but it doesn't apply its output. Instead it only logs its
behavior. When the behavior is analysed and approved for different tables with
different consumption rates, we can safely wire new status checker and use its
functionality.
Another approach would be to define a new config and disable it by default.
Since this feature is not urgent, @mcvsubbu suggested not to define yet another
config and go with this two-phase approach.
## Testing Done
- unit tests
- modified _SegmentPartitionLLCRealtimeClusterIntegrationTest_ integration
test locally and verified the expected behavior. I didn't create a new IT
because they're time consuming and different edge cases are already tested in
the unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
