I'm trying to setup hadoop cluster for version 2.8.2 with two slaves.
Whenever I run:
hadoop jar
$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar
teragen -Dmapreduce.job.maps=1000 10t random-data
I get the following error for one of the slave:
17/10/29 00:02:51 INFO mapreduce.Job: Task Id :
attempt_1509256119340_0001_m_000445_1, Status : FAILED
Container launch failed for container_1509256119340_0001_01_000682 :
java.net.ConnectException: Call From linux-01/127.0.0.1 to
localhost:37411 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
Going to that slave's yarn-hadoop-nodemanager-linux-02.log, I get:
2017-10-28 23:48:42,719 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node ID
assigned is : localhost:37411
2017-10-28 23:48:42,725 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting
to ResourceManager at linux-01.local/192.168.1.1:8031
2017-10-28 23:48:42,762 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0
NM container statuses: []
2017-10-28 23:48:42,767 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering
with RM using containers :[]
2017-10-28 23:48:43,181 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager:
Rolling master-key for container-tokens, got key with id -410082181
2017-10-28 23:48:43,187 INFO
org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM:
Rolling master-key for container-tokens, got key with id -1296212863
2017-10-28 23:48:43,188 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with
ResourceManager as localhost:37411 with total resource of <memory:8192,
vCores:8>
2017-10-28 23:48:43,188 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying
ContainerManager to unblock new container-requests
2017-10-28 23:53:46,156 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:53:46,156 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:53:47,156 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:53:48,159 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:53:49,162 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:53:50,164 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:53:51,169 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:53:53,174 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:56:48,588 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:56:50,592 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:56:51,595 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
2017-10-28 23:56:52,598 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
couldn't find app application_1509256119340_0001 while processing
FINISH_CONTAINERS event
In it, you can see "Node ID assigned is : localhost:37411", which
matches the complain from the mapreduce job.
I don't know why the resourcemanager, in my linux-01 won't call slave
linux-02, but instead call th enon-existent localhost:37411... I'm very
confused.
I can clarify and/or provide more info if needed.
Cheers,
Joey Andres
