jugomezv commented on PR #10418: URL: https://github.com/apache/pinot/pull/10418#issuecomment-1471336951
> > When all events are filtered and no actual events are processed, set consumption delay to zero as we are caught up processing actual events. > > All events filtered in a given batch doesn't imply that we are caught up. In this scenario, the consumer returned messages , however, it was all filtered for whatever reason (is it tombstones today??). This scenario should be treated differently than when fetch messages doesn't return any messages from the consumer. Have I misunderstood the hidden assumptions in the current realtime segment data manager's implementation ? > > > This does not affect metric reports in active tables that have incoming events, only inactive tables that have long sequences of filtered events. > > What do you mean by "inactive tables"? Are these tables that are not consuming or tables that are consuming from an empty source? Or something else? @navina inactive tables I refer to tables that do not have events that are not filtered. The problem this is trying to tackle is the following sequence of events: EventSequnce:[Good Event-NotFiltered][allfilteredbatch][allfilteredbatch].........[allfilteredbatch] Delay reprted: DelayForGoodEvent=t,t+delta, t+2delta.........................................t+n*delta If we could measure the delay on filtered events we will be OK but sadly those events are filtered, so we cant compute delay on them. The ever increasing time is confusing as it lead us to believe that we are not consuming but in fact we are, we are just filtering messages. Hopefully, you follow the situation else I can set some time to zoom, I am open to any other suggestions that you and @Jackie-Jiang may have but current behavior does not seem right as it does not reflect the delay incurred by filtered events either. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
