Do you forgot restart the NodeManager service when updated the yarn-site.xml?
On Sun, Jun 16, 2019 at 5:17 PM Mike IT Expert <[email protected]> wrote: > Dear All, > > I like to run a python mapreduce sample program. I use the below command > and I get an exception as provided below: > > I googled a few solutions they suggest to update yarn-site.xml as below > but I still get the same exception message. > > Please let me know what you think. > > Thank you > > > hadoop-2.8.5/etc/hadoop/yarn-site.xml > ################################ > as below: > > <configuration> > <property> > <name>mapreduceyarn.nodemanager.aux-services</name> > <value>mapreduce_shuffle</value> > </property> > <property> > <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> > <value>org.apache.hadoop.mapred.ShuffleHandler</value> > </property> > </configuration> > > Command and Output > #################### > hadoop@mike-laptop:/tmp/kmeans/line_cust$ ./run.sh > 19/06/16 14:31:59 WARN streaming.StreamJob: -file option is deprecated, > please use generic option -files instead. > packageJobJar: [/tmp/kmeans/line_cust/lineCustReduceJoin.py, > /tmp/kmeans/line_cust/lineCustReduceGroup.py, > /tmp/hadoop-unjar6079850917434748483/] [] > /tmp/streamjob3876527625248696490.jar tmpDir=null > 19/06/16 14:31:59 INFO client.RMProxy: Connecting to ResourceManager at / > 0.0.0.0:8032 > 19/06/16 14:32:00 INFO client.RMProxy: Connecting to ResourceManager at / > 0.0.0.0:8032 > 19/06/16 14:32:01 INFO mapred.FileInputFormat: Total input files to > process : 1 > 19/06/16 14:32:01 INFO mapreduce.JobSubmitter: number of splits:2 > 19/06/16 14:32:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: > job_1560691739419_0001 > 19/06/16 14:32:01 INFO impl.YarnClientImpl: Submitted application > application_1560691739419_0001 > 19/06/16 14:32:02 INFO mapreduce.Job: The url to track the job: > http://localhost:8088/proxy/application_1560691739419_0001/ > 19/06/16 14:32:02 INFO mapreduce.Job: Running job: job_1560691739419_0001 > 19/06/16 14:32:08 INFO mapreduce.Job: Job job_1560691739419_0001 running > in uber mode : false > 19/06/16 14:32:08 INFO mapreduce.Job: map 0% reduce 0% > 19/06/16 14:32:09 INFO mapreduce.Job: Task Id : > attempt_1560691739419_0001_m_000000_0, Status : FAILED > Container launch failed for container_1560691739419_0001_01_000002 : > org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The > auxService:mapreduce_shuffle does not exist > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 19/06/16 14:32:09 INFO mapreduce.Job: Task Id : > attempt_1560691739419_0001_m_000001_0, Status : FAILED > Container launch failed for container_1560691739419_0001_01_000003 : > org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The > auxService:mapreduce_shuffle does not exist > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 19/06/16 14:32:11 INFO mapreduce.Job: Task Id : > attempt_1560691739419_0001_m_000000_1, Status : FAILED > Container launch failed for container_1560691739419_0001_01_000004 : > org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The > auxService:mapreduce_shuffle does not exist > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 19/06/16 14:32:12 INFO mapreduce.Job: Task Id : > attempt_1560691739419_0001_m_000001_1, Status : FAILED > Container launch failed for container_1560691739419_0001_01_000005 : > org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The > auxService:mapreduce_shuffle does not exist > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 19/06/16 14:32:14 INFO mapreduce.Job: Task Id : > attempt_1560691739419_0001_m_000000_2, Status : FAILED > Container launch failed for container_1560691739419_0001_01_000006 : > org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The > auxService:mapreduce_shuffle does not exist > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 19/06/16 14:32:15 INFO mapreduce.Job: Task Id : > attempt_1560691739419_0001_m_000001_2, Status : FAILED > Container launch failed for container_1560691739419_0001_01_000007 : > org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The > auxService:mapreduce_shuffle does not exist > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182) > at > org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162) > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:393) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 19/06/16 14:32:17 INFO mapreduce.Job: map 100% reduce 100% > 19/06/16 14:32:17 INFO mapreduce.Job: Job job_1560691739419_0001 failed > with state FAILED due to: Task failed task_1560691739419_0001_m_000000 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > > 19/06/16 14:32:17 INFO mapreduce.Job: Counters: 14 > Job Counters > Failed map tasks=7 > Killed map tasks=1 > Killed reduce tasks=1 > Launched map tasks=7 > Other local map tasks=5 > Data-local map tasks=2 > Total time spent by all maps in occupied slots (ms)=5 > Total time spent by all reduces in occupied slots (ms)=0 > Total time spent by all map tasks (ms)=5 > Total time spent by all reduce tasks (ms)=0 > Total vcore-milliseconds taken by all map tasks=5 > Total vcore-milliseconds taken by all reduce tasks=0 > Total megabyte-milliseconds taken by all map tasks=5120 > Total megabyte-milliseconds taken by all reduce tasks=0 > 19/06/16 14:32:17 ERROR streaming.StreamJob: Job not successful! > Streaming Command Failed! > > > >
