rhodo opened a new pull request, #15686: URL: https://github.com/apache/pinot/pull/15686
This PR introduce a memory based throttling mechanism for requests towards grpc query server: when grpc query server's direct memory usage is high and exceed certain threshold, it will start rejecting further requests with RESOURCE_EXHAUSTED grpc status code until memory usage back down below that threshold. Testing: We simulated sustained synthetic query traffic targeting a server constrained by network bandwidth and CPU(# of grpc thread). **Without memory throttling**: direct memory usage can reach up to 1GB in the test setup. <img width="836" alt="Screenshot 2025-05-01 at 1 08 20 AM" src="https://github.com/user-attachments/assets/b683bf8a-89cf-45f6-a7f2-262cc999c3e2"; /> **With throttling enabled (threshold = 400MB)**: Memory usage is capped more effectively. <img width="812" alt="Screenshot 2025-05-01 at 12 57 45 AM" src="https://github.com/user-attachments/assets/37b42760-b602-43f8-87d0-85a0f411aa88"; /> meanwhile we also observe rejected requests due to throttling: <img width="774" alt="Screenshot 2025-05-01 at 1 40 31 AM" src="https://github.com/user-attachments/assets/07babc59-545d-4325-94fa-73ac810a4d65"; /> On the client side, the following exception is seen: ``` Error in task: java.lang.RuntimeException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Server under memory pressure (used: 427819008 bytes, threshold: 419430400 bytes) ``` Note: This mechanism provides best-effort control over memory usage. It does not guarantee memory will remain strictly below the threshold at all times. For instance: - Large query responses may cause a brief memory overshoot. - Concurrent request bursts can overshoot the threshold if they arrive in parallel before throttling is triggered. However, such concurrency is naturally bounded by the gRPC thread pool size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
