Hey, I was following the following blog post to copy the dump in case the container goes OOM - https://yhemanth.wordpress.com/2013/03/28/taking-memory- dumps-of-hadoop-tasks/
But for some reason the dump is not getting pushed to hdfs, I get the following logs - -- Log Type: stderr Log Upload Time: Wed Aug 17 12:01:55 +0530 2016 Log Length: 833 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/grid/6/yarn/local/usercache/fk-fdp-cdm/filecache/2682/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/grid/7/yarn/local/filecache/10/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]*./copy_dump.sh: 2: ./copy_dump.sh: Bad substitution* Log Type: stdout Log Upload Time: Wed Aug 17 12:01:55 +0530 2016 Log Length: 272 *java.lang.OutOfMemoryError: Java heap space Dumping heap to ./heapdump.hprof ... Heap dump file created [1906572521 <%5B1906572521> bytes in 7.933 secs] # # java.lang.OutOfMemoryError: Java heap space # -XX:OnOutOfMemoryError="./copy_dump.sh" # Executing /bin/sh -c "./copy_dump.sh"...* -- copy_dump.sh looks like this - #!/bin/sh hadoop fs -copyFromLocal heapdump.hprof /tmp/heapdump_akshay_aggarwal/${PWD//\//_}.hprof And I've added the following params to my job - -files hdfs:///user/fk-fdp-cdm/scripts/copy_dump.sh#copy_dump.sh \ -archives ${metadata_archive} \ -D mapred.create.symlink=yes \ -D mapreduce.reduce.java.opts='-Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./heapdump.hprof -XX:OnOutOfMemoryError=./copy_dump.sh' \ Any pointers to what might be wrong here? Thanks, Akshay Aggarwal
