geniusjoe commented on PR #25522: URL: https://github.com/apache/pulsar/pull/25522#issuecomment-4301505659
Based on the scenarios I’m currently using, when brokers and consumer networks are throttled, message dispatch can slow down, causing back pressure on the broker side and leading to high direct memory usage. If, at this time, the Pulsar ledger handler attempts to read entries from BK, it may encounter transient direct memory OOM due to direct memory exhaustion. However, in the current architecture, even with the `ExitOnOutOfMemoryError` JVM parameter configured, the system does not shut down. Instead, it backs off and retries the read, and the part of the entry bytebuf that was not successfully read is automatically reclaimed by Netty’s internal exception handling, preventing any memory leaks. I personally think the current exception handling logic is quite good, as it helps avoid OOM that could be triggered by back pressure when brokers and consumer networks are throttled. I’m not entirely sure whether Netty 4.2’s `adaptiveByteBuf` is fully compatible with the logic of Netty 4.1’s `pooledByteBuf` in this part. When upgrading to a major version of Netty, if the new `adaptiveByteBuf` does not provide significant improvements in performance or other aspects, it may be worth evaluating whether to adopt the new adaptiveByteBufor continue reusing the old `pooledByteBuf` mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
