tibrewalpratik17 commented on issue #13990: URL: https://github.com/apache/pinot/issues/13990#issuecomment-2347049700
> On Pinot side, we should triage why a metadata fetch call can spawn 75k threads. It doesn't look correct to me Yes we were able to find the root cause for this. We spawn one thread per segment in case of server-metadata fetch. Ref: https://github.com/apache/pinot/blob/266073eee0a56ae811c65cb0828cff294212aa48/pinot-controller/src/main/java/org/apache/pinot/controller/util/ServerSegmentMetadataReader.java#L185-L194 Ideally we can do one call per server rather than one call per segment. Plus we also use an unbounded thread pool for submitting these calls per segment causing 75k threads to spawn: https://github.com/apache/pinot/blob/266073eee0a56ae811c65cb0828cff294212aa48/pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java#L254-L255 We can limit the number of threads in this pool to something reasonable. I think what we are interested to know more is why didn't leader-election kick off even when zk-client session was lost. We will raise an issue in Helix and link it here. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
