jasperjiaguo commented on PR #11496: URL: https://github.com/apache/pinot/pull/11496#issuecomment-1720244864
> Great work! Out of curiosity, can you elaborate more on the netty channel deadlock? Or is there some blog I can read about describing the issue? Hey @Jackie-Jiang the situation we had was exactly as Sidd described: >A really bad query that was fetching around 150MB of data from each of the 160 servers (fan out was 160) caused direct memory OOM on broker. Note that this was a soft OOM (broker didn't crash unlike Java heap space OOM) So the broker is able to process the responese from one server technically, but all servers (150+) together sending back these mid sized responses together will create a situation where each channel allocates some amont of direct buffer, but then the direct memory is filled and no server can proceed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
