[
https://issues.apache.org/jira/browse/HADOOP-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013467#comment-16013467
]
Hudson commented on HADOOP-14412:
---------------------------------
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11741 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11741/])
HADOOP-14412. HostsFileReader#getHostDetails is very expensive on large
(rohithsharmaks: rev d87a63a9019d74a1c338c724e050952843a153e5)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
> HostsFileReader#getHostDetails is very expensive on large clusters
> ------------------------------------------------------------------
>
> Key: HADOOP-14412
> URL: https://issues.apache.org/jira/browse/HADOOP-14412
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 2.8.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Fix For: 2.9.0, 3.0.0-alpha3, 2.8.2
>
> Attachments: HADOOP-14412.001.patch, HADOOP-14412.002.patch,
> HADOOP-14412-branch-2.001.patch, HADOOP-14412-branch-2.002.patch,
> HADOOP-14412-branch-2.002.patch, HADOOP-14412-branch-2.8.002.patch
>
>
> After upgrading one of our large clusters to 2.8 we noticed many IPC server
> threads of the resourcemanager spending time in NodesListManager#isValidNode
> which in turn was calling HostsFileReader#getHostDetails. The latter is
> creating complete copies of the include and exclude sets for every node
> heartbeat, and these sets are not small due to the size of the cluster.
> These copies are causing multiple resizes of the underlying HashSets being
> filled and creating lots of garbage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
