Hi Wei, Can you please share your yarn-site.xml and the command you used for running wordcount?
Thanks, Konstantinos On Tue, Jun 13, 2017 at 11:44 Jasson Chenwei <[email protected]> wrote: > hi all > > I have set up a distributed scheduler using a new feature in Hadoop-3.0. My > Hadoop version is hadoop-3.0.0-appha3. I have enabled the > opportunistic container and distributed scheduler in > yarn-site.xml following the guide. But wordcount application master failed > to launch as follow: > > *2017-06-13 12:34:11,036 INFO [main] org.eclipse.jetty.server.Server: > Started @5116ms* > *2017-06-13 12:34:11,036 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: > Web app mapreduce started at 45559* > *2017-06-13 12:34:11,039 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE > job_1497050650910_0004* > *2017-06-13 12:34:11,041 INFO [main] > org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class > java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler: > class org.apache.hadoop.ipc.DefaultRpcScheduler* > *2017-06-13 12:34:11,042 INFO [Socket Reader #1 for port 36026] > org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36026* > *2017-06-13 12:34:11,045 INFO [IPC Server Responder] > org.apache.hadoop.ipc.Server: IPC Server Responder: starting* > *2017-06-13 12:34:11,045 INFO [IPC Server listener on 36026] > org.apache.hadoop.ipc.Server: IPC Server listener on 36026: starting* > *2017-06-13 12:34:11,083 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: > nodeBlacklistingEnabled:true* > *2017-06-13 12:34:11,083 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: > maxTaskFailuresPerNode is 3* > *2017-06-13 12:34:11,083 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: > blacklistDisablePercent is 33* > *2017-06-13 12:34:11,090 INFO [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the > mappers will be scheduled using OPPORTUNISTIC containers* > *2017-06-13 12:34:11,132 INFO [main] org.apache.hadoop.yarn.client.RMProxy: > Connecting to ResourceManager at localhost/127.0.0.1:8030 > <http://127.0.0.1:8030>* > *2017-06-13 12:34:11,193 WARN [main] org.apache.hadoop.ipc.Client: > Exception encountered while connecting to the server : > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1497050650910_0004_000002* > *2017-06-13 12:34:11,203 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > registering* > *org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1497050650910_0004_000002* > * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)* > * at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)* > * at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)* > * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)* > * at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)* > * at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)* > * at > > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)* > * at > > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)* > * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)* > * at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)* > * at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)* > * at java.lang.reflect.Method.invoke(Method.java:498)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)* > * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)* > * at > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)* > * at > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)* > * at > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)* > * at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)* > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:966)* > * at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)* > * at > > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)* > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1263)* > * at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)* > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1716)* > * at java.security.AccessController.doPrivileged(Native Method)* > * at javax.security.auth.Subject.doAs(Subject.java:422)* > * at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)* > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1712)* > * at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1643)* > *Caused by: > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1497050650910_0004_000002* > * at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)* > * at org.apache.hadoop.ipc.Client.call(Client.java:1427)* > * at org.apache.hadoop.ipc.Client.call(Client.java:1337)* > * at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)* > * at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:1)* > * at com.sun.proxy.$Proxy85.registerApplicationMaster(Unknown Source)* > * at > > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:107)* > * ... 25 more* > *2017-06-13 12:34:11,207 INFO [main] > org.apache.hadoop.service.AbstractService: Service RMCommunicator failed in > state STARTED; cause: > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1497050650910_0004_000002* > *org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1497050650910_0004_000002* > * at > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)* > * at > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)* > * at > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)* > * at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)* > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:966)* > * at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)* > * at > > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)* > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1263)* > * at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)* > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1716)* > * at java.security.AccessController.doPrivileged(Native Method)* > * at javax.security.auth.Subject.doAs(Subject.java:422)* > * at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)* > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1712)* > * at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1643)* > *Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: > Invalid AMRMToken from appattempt_1497050650910_0004_000002* > * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)* > * at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)* > * at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)* > * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)* > * at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)* > * at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)* > * at > > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)* > * at > > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)* > * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)* > * at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)* > * at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)* > * at java.lang.reflect.Method.invoke(Method.java:498)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)* > * at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)* > * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)* > * at > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)* > * ... 14 more* > > > > Is there any configuration missing that I have a AMRMToken erros? > > I appreciate your help! > > Thanks! > > > Wei Chen > -- Konstantinos
