sajjad-moradi opened a new pull request #6291: URL: https://github.com/apache/incubator-pinot/pull/6291
## Description Bursty pattern of incoming Kafka events introduces long time GC pauses on Pinot server. This behavior has been observed in use cases in LinkedIn. It's also reported in this issue: #6138. This PR introduces rate limit on Kafka consumption which can be configured in StreamConfig. The rate limit parameter is defined for the whole topic and the rate limit for each partition will be simply topic rate limit divided by the partition count of that topic. ## Testing Done Leveraged `quick-start-hybrid.sh`: - Set the limit rate parameter to 20 in `airlineStats_realtime_table_config` (there are 10 partitions in the corresponding Kafka topic) - Increased kafka publishing rate to produce more than 20 events per second - Logged throttling timestamps and verified that each partition only consumed 2 events per second ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
