Jackie-Jiang opened a new pull request, #9896: URL: https://github.com/apache/pinot/pull/9896
Currently we seek offset for each fetch call, which adds unnecessary overhead, and also cause very noisy logs from `KafkaConsumer`. Ideally we should only seek offset once when setting up the consumer, but to be more error prune, in this PR we track the last fetched offset, and seek the offset when the next fetch doesn't follow the offset from the previous fetch. Also improve the filtering logic to only process the message when it should be kept. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
