jadami10 commented on PR #9244: URL: https://github.com/apache/pinot/pull/9244#issuecomment-1224964658
> I thought your requirement is to not have many uningested messages in the topic because ingestion interferes with query execution: Both are a concern, but the ingestion lag is much more of a problem than the slow execution. For a topic with ~4,000 events/second, it takes us ~1-2 minutes per hour of ingestion lag to catch up when a server is restarted. Our segment seal threshold is 6 hours, so each restart takes ~3-6 minutes to reach the offset at startup. That brief period where we're then serving queries at 3-6 minutes behind is a bigger issue than the latency. Using freshness as the threshold happens to help with both. Even if we start serving queries at 15 seconds behind, that's relatively few remaining events to consume, so I don't expect query latency to be impacted any longer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
