ankitsultana opened a new issue, #15741: URL: https://github.com/apache/pinot/issues/15741
We were seeing an issue where queries would fail with "Client cancelled" on the receiver side, and "RST_STREAM" error on the sender side. Outside of this, there were no logs to indicate what the issue was. Enabling gRPC Debug logs showed that the issue was `errorCode=8`, which refers to resource exhaustion: https://grpc.github.io/grpc/core/md_doc_statuscodes.html After increasing the buffer limit the queries work fine. ``` {"@timestamp":"2025-05-08T02:47:38.502+00:00","@version":1,"message":"[id: 0x047e448d, L:/10.148.247.202:44029 - R:/10.148.33.163:32918] OUTBOUND RST_STREAM: streamId=15 errorCode=8","logger_name":"io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler","thread_name":"grpc-default-worker-ELG-3-1","level":"DEBUG","level_value":10000,"HOSTNAME":"some-hostname"} ``` As part of this ticket we should improve logging for this scenario. This is kinda unexpected because in the past I have seen "RESOURCE_EXHAUSTED" errors when gRPC message size limits are breached, but maybe BiDir calls are not handled gracefully by gRPC? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
