sajjad-moradi commented on PR #9244: URL: https://github.com/apache/pinot/pull/9244#issuecomment-1225016104
> > I thought your requirement is to not have many uningested messages in the topic because ingestion interferes with query execution: > > Both are a concern, but the ingestion lag is much more of a problem than the slow execution. For a topic with ~4,000 events/second, it takes us ~1-2 minutes per hour of ingestion lag to catch up when a server is restarted. Our segment seal threshold is 6 hours, so each restart takes ~3-6 minutes to reach the offset at startup. That brief period where we're then serving queries at 3-6 minutes behind is a bigger issue than the latency. Using freshness as the threshold happens to help with both. Even if we start serving queries at 15 seconds behind, that's relatively few remaining events to consume, so I don't expect query latency to be impacted any longer. Makes sense. Even in the case of a huge spike in the latest 15s, consumption rate limiter doesn't allow immediate consumption which in turns prevents latency in query execution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
