[GitHub] [incubator-pinot] dongxiaoman opened a new issue #6138: Real time server query performance hit when catching up Kafka backlogs

[GitBox]

dongxiaoman opened a new issue #6138:
URL: https://github.com/apache/incubator-pinot/issues/6138


   Right now our upstream Kafka log has a "batching" behavior that every a few 
minutes there is a sudden spike in Kafka traffic (due to upstream Flink job's 
checkpointing behavior). This cause a periodical Kafka backlog for our Pinot 
server. For example, every 30 minutes, we will suddenly have lots of Kafka 
backlog to catch up.
   
   This is fine but the query performance suffers a lot during that backlog 
period. Based on our quick chart analysis it could be I/O bound.
   Below are some of the charts we can share, showing correlation.
   
![doc1](https://user-images.githubusercontent.com/11821736/95919938-51c94380-0d63-11eb-9146-6f8acfd55e2e.png)
   
![doc2](https://user-images.githubusercontent.com/11821736/95919971-6574aa00-0d63-11eb-9f36-17e4cb761d56.png)
   
![doc3](https://user-images.githubusercontent.com/11821736/95919980-69a0c780-0d63-11eb-9a80-2a8e8c9f7221.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org