[
https://issues.apache.org/jira/browse/HADOOP-18389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18036023#comment-18036023
]
ASF GitHub Bot commented on HADOOP-18389:
-----------------------------------------
github-actions[bot] closed pull request #4691: HADOOP-18389. Limit stacked call
of one connection in client to avoid possible OOM in server
URL: https://github.com/apache/hadoop/pull/4691
> Limit stacked call of one connection in client to avoid possible oom in server
> ------------------------------------------------------------------------------
>
> Key: HADOOP-18389
> URL: https://issues.apache.org/jira/browse/HADOOP-18389
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: ZanderXu
> Assignee: ZanderXu
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-08-04-00-22-28-865.png,
> image-2022-08-04-00-23-18-427.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In our prod environment, we encountered an accident that JN OOM because
> Server#Connection#responseQueue used 97% memory.
> After analyzed the memory of JN and found that there are 2w+ called stacked
> in one Server#Connection#responseQueue, because the network between NN and JN
> jitters with some tcp packet loss.
> !image-2022-08-04-00-22-28-865.png|width=561,height=254!
> !image-2022-08-04-00-23-18-427.png|width=559,height=356!
>
> In this case, I think Client.java should support limit the stacked calls of
> one connection to avoid the possible OOM in Server. When the number of
> stacked calls is more than the limit size, we can just throw one IOException
> to the method caller.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
