[
https://issues.apache.org/jira/browse/HADOOP-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006670#comment-16006670
]
Jason Lowe commented on HADOOP-14412:
-------------------------------------
Looks like this was caused by YARN-4002. A typical stack trace we are seeing
looks like this:
{noformat}
java.util.HashMap.resize(HashMap.java:734)
java.util.HashMap.putVal(HashMap.java:662)
java.util.HashMap.put(HashMap.java:611)
java.util.HashSet.add(HashSet.java:219)
java.util.AbstractCollection.addAll(AbstractCollection.java:344)
org.apache.hadoop.util.HostsFileReader.getHostDetails(HostsFileReader.java:211)
org.apache.hadoop.yarn.server.resourcemanager.NodesListManager.isValidNode(NodesListManager.java:367)
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(ResourceTrackerService.java:532)
org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceTrackerPBServiceImpl.nodeHeartbeat(ResourceTrackerPBServiceImpl.java:71)
org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$2.callBlockingMethod(ResourceTracker.java:97)
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:999)
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:879)
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:808)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2507)
{noformat}
> HostsFileReader#getHostDetails is very expensive on large clusters
> ------------------------------------------------------------------
>
> Key: HADOOP-14412
> URL: https://issues.apache.org/jira/browse/HADOOP-14412
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 2.8.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
>
> After upgrading one of our large clusters to 2.8 we noticed many IPC server
> threads of the resourcemanager spending time in NodesListManager#isValidNode
> which in turn was calling HostsFileReader#getHostDetails. The latter is
> creating complete copies of the include and exclude sets for every node
> heartbeat, and these sets are not small due to the size of the cluster.
> These copies are causing multiple resizes of the underlying HashSets being
> filled and creating lots of garbage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
