Yes, that almost worked. Now the application starts on Yarn and after a while an exception is thrown and the application exits with code 10.
Application About Jobs Tools Log Type: stdout Log Upload Time: Sat Jan 23 12:49:41 +0100 2016 Log Length: 21097 UnJar appMaster.jar to tmp/twill.launcher-1453549768670-0 Launch class (org.apache.twill.internal.appmaster.ApplicationMasterMain) with classpath: [file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/classes, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/resources, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-cli-1.2.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/scala-library-2.10.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-classic-1.0.9.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-math3-3.1.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-core-1.0.9.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/xmlenc-0.52.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jsch-0.1.42.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/httpclient-4.1.2.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-configuration-1.6.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/metrics-core-2.2.0.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jetty-6.1.26.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-api-2.7.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-annotations-2.7.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/guice-3.0.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-net-3.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jetty-util-6.1.26.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/kafka_2.10-0.8.0.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-api-0.6.0-incubating.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-api-1.7.10.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/paranamer-2.3.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/protobuf-java-2.5.0.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/apacheds-kerberos-codec-2.0.0-M15.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/avro-1.7.4.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-compress-1.4.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-auth-2.7.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/zookeeper-3.4.6.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-core-1.9.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/curator-client-2.7.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-zookeeper-0.6.0-incubating.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-client-1.9.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/gson-2.2.4.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-common-2.7.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-hdfs-2.7.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/api-asn1-api-1.0.0-M20.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-discovery-core-0.6.0-incubating.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-collections-3.2.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/netty-3.7.0.Final.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-log4j12-1.7.10.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-common-2.7.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-mapper-asl-1.9.13.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/zkclient-0.3.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-jaxrs-1.9.13.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-xc-1.9.13.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jsr305-3.0.0.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/snappy-java-1.0.4.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/log4j-1.2.17.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-codec-1.4.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/asm-all-5.0.2.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/netty-all-4.0.23.Final.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/servlet-api-2.5.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/guava-13.0.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jopt-simple-3.2.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/curator-framework-2.7.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/hadoop-yarn-client-2.7.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-httpclient-3.1.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-discovery-api-0.6.0-incubating.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-lang-2.6.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/httpcore-4.1.2.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-yarn-0.6.0-incubating.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/api-util-1.0.0-M20.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/htrace-core-3.1.0-incubating.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-common-0.6.0-incubating.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-io-2.4.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jersey-server-1.9.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/apacheds-i18n-2.0.0-M15.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/commons-logging-1.1.3.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/twill-core-0.6.0-incubating.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/jackson-core-asl-1.9.13.jar, file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/javax.inject-1.jar] Launching main: public static void org.apache.twill.internal.appmaster.ApplicationMasterMain.main(java.lang.String[]) throws java.lang.Exception [] 12:49:29.586 [main] DEBUG o.a.h.s.a.util.KerberosName - Kerberos krb5 configuration not found, setting default realm to empty 12:49:30.083 [main] DEBUG o.a.h.h.p.d.s.DataTransferSaslUtil - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection 12:49:30.552 [main] INFO o.apache.twill.internal.ServiceMain - Starting service ApplicationMasterService [NEW]. 12:49:30.600 [kafka-publisher] WARN o.a.t.i.k.c.SimpleKafkaPublisher - Broker list is empty. No Kafka producer is created. 12:49:30.704 [TrackerService STARTING] INFO o.a.t.i.appmaster.TrackerService - Tracker service started at http://hdfs-ix03.se-ix.delta.prod:51793 12:49:30.922 [TwillZKPathService STARTING] INFO o.a.t.i.ServiceMain$TwillZKPathService - Creating container ZK path: zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b 12:49:31.102 [kafka-publisher] INFO o.a.t.i.k.c.SimpleKafkaPublisher - Update Kafka producer broker list: hdfs-ix03.se-ix.delta.prod:58668 12:49:31.288 [ApplicationMasterService] INFO o.a.t.internal.AbstractTwillService - Create live node zookeeper01.se-ix.delta.prod/JarApp/instances/be4bbf01-5e72-4147-b2eb-b84e19214b5b 12:49:31.308 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - Start application master with spec: {"name":"JarApp","runnables":{"JarRunnable":{"name":"JarRunnable","runnable":{"classname":"org.apache.twill.ext.BundledJarRunnable","name":"JarRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1},"files":[{"name":"twill-app-1.0.0-SNAPSHOT.jar","uri":"hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/twill-app-1.0.0-SNAPSHOT.jar.e09cf92c-56f7-42a8-84ac-53f2665afa1d.jar","lastModified":1453549766870,"size":7090,"archive":false,"pattern":null}]}},"orders":[{"names":["JarRunnable"],"type":"STARTED"}],"placementPolicies":[],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}} 12:49:31.318 [main] INFO o.apache.twill.internal.ServiceMain - Service ApplicationMasterService [RUNNING] started. 12:49:31.344 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - Request 1 container with capability <memory:512, vCores:1> for runnable JarRunnable 12:49:33.368 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - Got container container_e29_1453498444043_0012_01_000002 12:49:33.369 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - Starting runnable JarRunnable with RunnableProcessLauncher{container=org.apache.twill.internal.yarn.Hadoop21YarnContainerInfo@5e82cebd} 12:49:33.417 [ApplicationMasterService] INFO o.a.t.i.a.RunnableProcessLauncher - Launching in container container_e29_1453498444043_0012_01_000002 at hdfs-ix03.se-ix.delta.prod:45454, [$JAVA_HOME/bin/java -Djava.io.tmpdir=tmp -Dyarn.container=$YARN_CONTAINER_ID -Dtwill.runnable=$TWILL_APP_NAME.$TWILL_RUNNABLE_NAME -cp launcher.jar:$HADOOP_CONF_DIR -Xmx359m org.apache.twill.launcher.TwillLauncher container.jar org.apache.twill.internal.container.TwillContainerMain true 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr] 12:49:33.473 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - Runnable JarRunnable fully provisioned with 1 instances. 12:49:35.302 [zk-client-EventThread] INFO o.a.t.i.TwillContainerLauncher - Container LiveNodeData updated: {"data":{"containerId":"container_e29_1453498444043_0012_01_000002","host":"hdfs-ix03.se-ix.delta.prod"}} 12:49:37.484 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - Container container_e29_1453498444043_0012_01_000002 completed with COMPLETE:Exception from container-launch. Container id: container_e29_1453498444043_0012_01_000002 Exit code: 10 Stack trace: ExitCodeException exitCode=10: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 10 . 12:49:37.488 [ApplicationMasterService] WARN o.a.t.i.appmaster.RunningContainers - Container container_e29_1453498444043_0012_01_000002 exited abnormally with state COMPLETE, exit code 10. 12:49:37.496 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - All containers completed. Shutting down application master. 12:49:37.498 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - Stop application master with spec: {"name":"JarApp","runnables":{"JarRunnable":{"name":"JarRunnable","runnable":{"classname":"org.apache.twill.ext.BundledJarRunnable","name":"JarRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1},"files":[{"name":"twill-app-1.0.0-SNAPSHOT.jar","uri":"hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/twill-app-1.0.0-SNAPSHOT.jar.e09cf92c-56f7-42a8-84ac-53f2665afa1d.jar","lastModified":1453549766870,"size":7090,"archive":false,"pattern":null}]}},"orders":[{"names":["JarRunnable"],"type":"STARTED"}],"placementPolicies":[],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}} 12:49:37.500 [ApplicationMasterService] INFO o.a.t.i.appmaster.RunningContainers - Stopping all instances of JarRunnable 12:49:37.500 [ApplicationMasterService] INFO o.a.t.i.appmaster.RunningContainers - Terminated all instances of JarRunnable 12:49:37.512 [ApplicationMasterService] INFO o.a.t.i.a.ApplicationMasterService - Application directory deleted: hdfs://hdpcluster/user/stoffe/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b 12:49:37.512 [ApplicationMasterService] INFO o.a.t.internal.AbstractTwillService - Remove live node zookeeper01.se-ix.delta.prod/JarApp/instances/be4bbf01-5e72-4147-b2eb-b84e19214b5b 12:49:37.516 [ApplicationMasterService] INFO o.a.t.internal.AbstractTwillService - Service ApplicationMasterService with runId be4bbf01-5e72-4147-b2eb-b84e19214b5b shutdown completed 12:49:37.516 [main] INFO o.apache.twill.internal.ServiceMain - Service ApplicationMasterService [TERMINATED] completed. 12:49:39.676 [kafka-publisher] WARN o.a.t.i.k.c.SimpleKafkaPublisher - Broker list is empty. No Kafka producer is created. 12:49:40.037 [TwillZKPathService STOPPING] INFO o.a.t.i.ServiceMain$TwillZKPathService - Removing container ZK path: zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b 12:49:40.248 [TrackerService STOPPING] INFO o.a.t.i.appmaster.TrackerService - Tracker service stopped at http://hdfs-ix03.se-ix.delta.prod:51793 Main class completed. Launcher completed Cleanup directory tmp/twill.launcher-1453549768670-0 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/tmp/twill.launcher-1453549768670-0/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder] 16/01/23 12:49:29 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 16/01/23 12:49:30 INFO utils.VerifiableProperties: Verifying properties 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property log.dir is overridden to /hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property default.replication.factor is overridden to 1 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property port is overridden to 58668 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property socket.request.max.bytes is overridden to 104857600 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property socket.send.buffer.bytes is overridden to 1048576 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property log.flush.interval.ms is overridden to 1000 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property zookeeper.connect is overridden to zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property broker.id is overridden to 1 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property log.retention.hours is overridden to 24 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property socket.receive.buffer.bytes is overridden to 1048576 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property zookeeper.connection.timeout.ms is overridden to 3000 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property num.partitions is overridden to 1 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property log.flush.interval.messages is overridden to 10000 16/01/23 12:49:30 INFO utils.VerifiableProperties: Property log.segment.bytes is overridden to 536870912 16/01/23 12:49:30 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 16/01/23 12:49:30 INFO server.KafkaServer: [Kafka Server 1], Starting 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] Log directory '/hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs' not found, creating it. 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] Starting log cleaner every 600000 ms 16/01/23 12:49:30 INFO log.LogManager: [Log Manager on Broker 1] Starting log flusher every 3000 ms with the following overrides Map() 16/01/23 12:49:30 INFO network.Acceptor: Awaiting socket connections on 0.0.0.0:58668. 16/01/23 12:49:30 INFO network.SocketServer: [Socket Server on Broker 1], Started 16/01/23 12:49:30 INFO server.KafkaZooKeeper: connecting to ZK: zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka 16/01/23 12:49:30 INFO zkclient.ZkEventThread: Starting ZkClient event thread. 16/01/23 12:49:31 INFO zkclient.ZkClient: zookeeper state changed (SyncConnected) 16/01/23 12:49:31 INFO utils.ZkUtils$: Registered broker 1 at path /brokers/ids/1 with address hdfs-ix03.se-ix.delta.prod:58668. 16/01/23 12:49:31 INFO server.KafkaServer: [Kafka Server 1], Connecting to ZK: zookeeper01.se-ix.delta.prod/JarApp/be4bbf01-5e72-4147-b2eb-b84e19214b5b/kafka 16/01/23 12:49:31 INFO utils.VerifiableProperties: Verifying properties 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property metadata.broker.list is overridden to hdfs-ix03.se-ix.delta.prod:58668 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property request.required.acks is overridden to 1 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property partitioner.class is overridden to org.apache.twill.internal.kafka.client.IntegerPartitioner 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property compression.codec is overridden to snappy 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property key.serializer.class is overridden to org.apache.twill.internal.kafka.client.IntegerEncoder 16/01/23 12:49:31 INFO utils.VerifiableProperties: Property serializer.class is overridden to org.apache.twill.internal.kafka.client.ByteBufferEncoder 16/01/23 12:49:31 INFO utils.Mx4jLoader$: Will not load MX4J, mx4j-tools.jar is not in the classpath 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Controller starting up 16/01/23 12:49:31 INFO server.ZookeeperLeaderElector: 1 successfully elected as leader 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Broker 1 starting become controller state transition 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Controller 1 incremented epoch to 1 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with correlation id 0 for 1 topic(s) Set(log) 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to hdfs-ix03.se-ix.delta.prod:58668 for producing 16/01/23 12:49:31 INFO controller.RequestSendThread: [Controller-1-to-broker-1-send-thread], Starting 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Currently active brokers in the cluster: Set(1) 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Currently shutting brokers in the cluster: Set() 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Current list of topics in the cluster: Set() 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state machine on controller 1]: No state transitions triggered since no partitions are assigned to brokers 1 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state machine on controller 1]: Invoking state change to OnlineReplica for replicas 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state machine on controller 1]: Started replica state machine with initial state -> Map() 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition state machine on Controller 1]: Started partition state machine with initial state -> Map() 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Broker 1 is ready to serve as the new controller with epoch 1 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Partitions being reassigned: Map() 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Partitions already reassigned: List() 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Resuming reassignment of partitions: Map() 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Partitions undergoing preferred replica election: 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { "partitions":{ "0":[ 1 ] }, "version":1 } 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Partitions that completed preferred replica election: 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Resuming preferred replica election for partitions: 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Starting preferred replica leader election for partitions 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition state machine on Controller 1]: Invoking state change to OnlinePartition for partitions 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: Controller startup complete 16/01/23 12:49:31 INFO server.KafkaApis: [KafkaApi-1] Auto creation of topic log with 1 partitions and replication factor 1 is successful! 16/01/23 12:49:31 INFO server.KafkaServer: [Kafka Server 1], Started 16/01/23 12:49:31 INFO server.ZookeeperLeaderElector$LeaderChangeListener: New leader is 1 16/01/23 12:49:31 INFO controller.ControllerEpochListener: [ControllerEpochListener on 1]: Initialized controller epoch to 1 and zk version 0 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from hdfs-ix03.se-ix.delta.prod:58668 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic log -> No partition metadata for topic log due to kafka.common.LeaderNotAvailableException}] for topic [log]: class kafka.common.LeaderNotAvailableException 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with correlation id 1 for 1 topic(s) Set(log) 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to hdfs-ix03.se-ix.delta.prod:58668 for producing 16/01/23 12:49:31 INFO controller.PartitionStateMachine$TopicChangeListener: [TopicChangeListener on Controller 1]: New topics: [Set(log)], deleted topics: [Set()], new partition replica assignment [Map([log,0] -> List(1))] 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { "partitions":{ "0":[ 1 ] }, "version":1 } 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: New topic creation callback for [log,0] 16/01/23 12:49:31 INFO controller.KafkaController: [Controller 1]: New partition creation callback for [log,0] 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from hdfs-ix03.se-ix.delta.prod:58668 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition state machine on Controller 1]: Invoking state change to NewPartition for partitions [log,0] 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic log -> No partition metadata for topic log due to kafka.common.LeaderNotAvailableException}] for topic [log]: class kafka.common.LeaderNotAvailableException 16/01/23 12:49:31 INFO network.Processor: Closing socket connection to /10.3.24.22. 16/01/23 12:49:31 INFO network.Processor: Closing socket connection to /10.3.24.22. 16/01/23 12:49:31 ERROR async.DefaultEventHandler: Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: log 16/01/23 12:49:31 INFO async.DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 3 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state machine on controller 1]: Invoking state change to NewReplica for replicas PartitionAndReplica(log,0,1) 16/01/23 12:49:31 INFO controller.PartitionStateMachine: [Partition state machine on Controller 1]: Invoking state change to OnlinePartition for partitions [log,0] 16/01/23 12:49:31 INFO controller.ReplicaStateMachine: [Replica state machine on controller 1]: Invoking state change to OnlineReplica for replicas PartitionAndReplica(log,0,1) 16/01/23 12:49:31 INFO server.ReplicaManager: [Replica Manager on Broker 1]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:1;ControllerEpoch:1;CorrelationId:6;ClientId:id_1-host_null-port_58668;PartitionState:(log,0) -> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:hdfs-ix03.se-ix.delta.prod,port:58668 16/01/23 12:49:31 INFO server.ReplicaFetcherManager: [ReplicaFetcherManager on broker 1] Removing fetcher for partition [log,0] 16/01/23 12:49:31 INFO log.Log: [Kafka Log on Broker 1], Completed load of log log-0 with log end offset 0 16/01/23 12:49:31 INFO log.LogManager: [Log Manager on Broker 1] Created log for partition [log,0] in /hadoop/yarn/local/usercache/stoffe/appcache/application_1453498444043_0012/container_e29_1453498444043_0012_01_000001/kafka-logs. 16/01/23 12:49:31 WARN server.HighwaterMarkCheckpoint: No highwatermark file is found. Returning 0 as the highwatermark for partition [log,0] 16/01/23 12:49:31 INFO server.ReplicaManager: [Replica Manager on Broker 1]: Handled leader and isr request Name:LeaderAndIsrRequest;Version:0;Controller:1;ControllerEpoch:1;CorrelationId:6;ClientId:id_1-host_null-port_58668;PartitionState:(log,0) -> (LeaderAndIsrInfo:(Leader:1,ISR:1,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:1);Leaders:id:1,host:hdfs-ix03.se-ix.delta.prod,port:58668 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with correlation id 2 for 1 topic(s) Set(log) 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to hdfs-ix03.se-ix.delta.prod:58668 for producing 16/01/23 12:49:31 INFO admin.AdminUtils$: Topic creation { "partitions":{ "0":[ 1 ] }, "version":1 } 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from hdfs-ix03.se-ix.delta.prod:58668 16/01/23 12:49:31 INFO network.Processor: Closing socket connection to /10.3.24.22. 16/01/23 12:49:31 WARN producer.BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic log -> No partition metadata for topic log due to kafka.common.LeaderNotAvailableException}] for topic [log]: class kafka.common.LeaderNotAvailableException 16/01/23 12:49:31 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:hdfs-ix03.se-ix.delta.prod,port:58668 with correlation id 3 for 1 topic(s) Set(log) 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to hdfs-ix03.se-ix.delta.prod:58668 for producing 16/01/23 12:49:31 INFO producer.SyncProducer: Disconnecting from hdfs-ix03.se-ix.delta.prod:58668 16/01/23 12:49:31 INFO network.Processor: Closing socket connection to /10.3.24.22. 16/01/23 12:49:31 INFO producer.SyncProducer: Connected to hdfs-ix03.se-ix.delta.prod:58668 for producing 16/01/23 12:49:33 INFO impl.AMRMClientImpl: Received new token for : hdfs-ix03.se-ix.delta.prod:45454 16/01/23 12:49:33 INFO impl.ContainerManagementProtocolProxy: Opening proxy : hdfs-ix03.se-ix.delta.prod:45454 16/01/23 12:49:35 INFO network.Processor: Closing socket connection to /10.3.24.22. 16/01/23 12:49:35 INFO network.Processor: Closing socket connection to /10.3.24.22. 16/01/23 12:49:39 INFO server.KafkaServer: [Kafka Server 1], Shutting down 16/01/23 12:49:39 INFO server.KafkaZooKeeper: Closing zookeeper client... 16/01/23 12:49:39 INFO zkclient.ZkEventThread: Terminate ZkClient event thread. 16/01/23 12:49:39 INFO network.SocketServer: [Socket Server on Broker 1], Shutting down 16/01/23 12:49:39 INFO network.SocketServer: [Socket Server on Broker 1], Shutdown completed 16/01/23 12:49:39 INFO server.KafkaRequestHandlerPool: [Kafka Request Handler on Broker 1], shutting down 16/01/23 12:49:39 INFO server.KafkaRequestHandlerPool: [Kafka Request Handler on Broker 1], shutted down completely 16/01/23 12:49:39 INFO utils.KafkaScheduler: Shutdown Kafka scheduler 16/01/23 12:49:39 INFO server.ReplicaManager: [Replica Manager on Broker 1]: Shut down 16/01/23 12:49:39 INFO server.ReplicaFetcherManager: [ReplicaFetcherManager on broker 1] shutting down 16/01/23 12:49:39 INFO server.ReplicaFetcherManager: [ReplicaFetcherManager on broker 1] shutdown completed 16/01/23 12:49:39 INFO server.ReplicaManager: [Replica Manager on Broker 1]: Shutted down completely 16/01/23 12:49:40 INFO controller.RequestSendThread: [Controller-1-to-broker-1-send-thread], Shutting down 16/01/23 12:49:40 INFO controller.RequestSendThread: [Controller-1-to-broker-1-send-thread], Stopped 16/01/23 12:49:40 INFO controller.RequestSendThread: [Controller-1-to-broker-1-send-thread], Shutdown completed 16/01/23 12:49:40 INFO controller.KafkaController: [Controller 1]: Controller shutdown complete 16/01/23 12:49:40 INFO server.KafkaServer: [Kafka Server 1], Shut down completed 16/01/23 12:49:40 INFO impl.ContainerManagementProtocolProxy: Opening proxy : hdfs-ix03.se-ix.delta.prod:45454 16/01/23 12:49:40 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered. 16/01/23 12:49:40 INFO producer.SyncProducer: Disconnecting from hdfs-ix03.se-ix.delta.prod:58668 16/01/23 12:49:40 WARN async.DefaultEventHandler: Failed to send producer request with correlation id 35 to broker 1 with data for partitions [log,0] java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:957) at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:204) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) at kafka.utils.Utils$.read(Unknown Source) at kafka.network.BoundedByteBufferReceive.readFrom(Unknown Source) at kafka.network.Receive$class.readCompletely(Unknown Source) at kafka.network.BoundedByteBufferReceive.readCompletely(Unknown Source) at kafka.network.BlockingChannel.receive(Unknown Source) at kafka.producer.SyncProducer.liftedTree1$1(Unknown Source) at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Unknown Source) at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown Source) at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(Unknown Source) at kafka.metrics.KafkaTimer.time(Unknown Source) at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(Unknown Source) at kafka.producer.SyncProducer$$anonfun$send$1.apply(Unknown Source) at kafka.producer.SyncProducer$$anonfun$send$1.apply(Unknown Source) at kafka.metrics.KafkaTimer.time(Unknown Source) at kafka.producer.SyncProducer.send(Unknown Source) at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(Unknown Source) at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(Unknown Source) at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(Unknown Source) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) at scala.collection.mutable.HashMap.foreach(HashMap.scala:98) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(Unknown Source) at kafka.producer.async.DefaultEventHandler.handle(Unknown Source) at kafka.producer.Producer.send(Unknown Source) at kafka.javaapi.producer.Producer.send(Unknown Source) at org.apache.twill.internal.kafka.client.SimpleKafkaPublisher$SimplePreparer.send(SimpleKafkaPublisher.java:122) at org.apache.twill.internal.logging.KafkaAppender.doPublishLogs(KafkaAppender.java:268) at org.apache.twill.internal.logging.KafkaAppender.publishLogs(KafkaAppender.java:228) at org.apache.twill.internal.logging.KafkaAppender.access$700(KafkaAppender.java:66) at org.apache.twill.internal.logging.KafkaAppender$2.run(KafkaAppender.java:280) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 16/01/23 12:49:40 INFO async.DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 3 16/01/23 12:49:40 INFO producer.Producer: Shutting down producer 16/01/23 12:49:40 INFO producer.ProducerPool: Closing all sync producers On Sat, Jan 23, 2016 at 1:22 AM, Terence Yim <[email protected]> wrote: > Hi, > > It's due to a very old version of ASM library that bring it by hadoop/yarn. > Please add exclusion of asm library to all hadoop dependencies. > > <exclusion> > <groupId>asm</groupId> > <artifactId>asm</artifactId> > </exclusion> > > Terence > > > On Fri, Jan 22, 2016 at 2:34 PM, Kristoffer Sjögren <[email protected]> > wrote: > >> Further adding the following dependencies cause another exception. >> >> <dependency> >> <groupId>com.google.guava</groupId> >> <artifactId>guava</artifactId> >> <version>13.0</version> >> </dependency> >> <dependency> >> <groupId>org.apache.hadoop</groupId> >> <artifactId>hadoop-hdfs</artifactId> >> <version>2.7.1</version> >> </dependency> >> >> Exception in thread " STARTING" >> java.lang.IncompatibleClassChangeError: class >> org.apache.twill.internal.utils.Dependencies$DependencyClassVisitor >> has interface org.objectweb.asm.ClassVisitor as super class >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:760) >> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) >> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:362) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:361) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> at >> org.apache.twill.internal.utils.Dependencies.findClassDependencies(Dependencies.java:86) >> at >> org.apache.twill.internal.ApplicationBundler.findDependencies(ApplicationBundler.java:198) >> at >> org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:155) >> at >> org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:126) >> at >> org.apache.twill.yarn.YarnTwillPreparer.createAppMasterJar(YarnTwillPreparer.java:402) >> at >> org.apache.twill.yarn.YarnTwillPreparer.access$200(YarnTwillPreparer.java:108) >> at >> org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:299) >> at >> org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:289) >> at >> org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:97) >> at >> org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:76) >> at >> org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:175) >> at >> com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) >> at java.lang.Thread.run(Thread.java:745) >> >> On Fri, Jan 22, 2016 at 11:32 PM, Kristoffer Sjögren <[email protected]> >> wrote: >> > Add those dependencies fail with the following exception. >> > >> > Exception in thread "main" java.lang.AbstractMethodError: >> > >> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy()Lorg/apache/hadoop/io/retry/FailoverProxyProvider$ProxyInfo; >> > at >> org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73) >> > at >> org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64) >> > at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) >> > at >> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:149) >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:569) >> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:512) >> > at >> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:142) >> > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) >> > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) >> > at >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) >> > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) >> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) >> > at >> org.apache.twill.yarn.YarnTwillRunnerService.createDefaultLocationFactory(YarnTwillRunnerService.java:615) >> > at >> org.apache.twill.yarn.YarnTwillRunnerService.<init>(YarnTwillRunnerService.java:149) >> > at deephacks.BundledJarExample.main(BundledJarExample.java:70) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:497) >> > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) >> > >> > On Fri, Jan 22, 2016 at 10:58 PM, Terence Yim <[email protected]> wrote: >> >> Hi, >> >> >> >> If you run it from IDE, you and simply add a dependency on hadoop with >> >> version 2.7.1. E.g. if you are using Maven, you can add the following to >> >> your pom.xml dependencies section. >> >> >> >> <dependency> >> >> <groupId>org.apache.hadoop</groupId> >> >> <artifactId>hadoop-yarn-api</artifactId> >> >> <version>2.7.1</version> >> >> </dependency> >> >> <dependency> >> >> <groupId>org.apache.hadoop</groupId> >> >> <artifactId>hadoop-yarn-common</artifactId> >> >> <version>2.7.1</version> >> >> </dependency> >> >> <dependency> >> >> <groupId>org.apache.hadoop</groupId> >> >> <artifactId>hadoop-yarn-client</artifactId> >> >> <version>2.7.1</version> >> >> </dependency> >> >> <dependency> >> >> <groupId>org.apache.hadoop</groupId> >> >> <artifactId>hadoop-common</artifactId> >> >> <version>2.7.1</version> >> >> </dependency> >> >> >> >> Terence >> >> >> >> On Fri, Jan 22, 2016 at 12:47 PM, Kristoffer Sjögren <[email protected]> >> >> wrote: >> >> >> >>> I run it from IDE right now, but would like to create a command line >> >>> app eventually. >> >>> >> >>> I should clarify that the exception above is thrown on the YARN node, >> >>> not in the IDE. >> >>> >> >>> On Fri, Jan 22, 2016 at 9:32 PM, Terence Yim <[email protected]> wrote: >> >>> > Hi Kristoffer, >> >>> > >> >>> > The example itself shouldn't need any modification. However, how do >> >>> > you run that class? Do you run it from IDE or from command line using >> >>> > "java" command? >> >>> > >> >>> > Terence >> >>> > >> >>> > On Fri, Jan 22, 2016 at 12:16 PM, Kristoffer Sjögren < >> [email protected]> >> >>> wrote: >> >>> >> Hi Terence, >> >>> >> >> >>> >> I'm quite new to Twill and not sure how to do that exactly. Could >> you >> >>> >> show me how to modify the following example to do the same? >> >>> >> >> >>> >> >> >>> >> https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java >> >>> >> >> >>> >> On Fri, Jan 22, 2016 at 7:24 PM, Terence Yim <[email protected]> >> wrote: >> >>> >>> Hi Kristoffer, >> >>> >>> >> >>> >>> Seems like the exception comes from the YARN class >> "ConverterUtils". I >> >>> >>> believe need to start the application with the version 2.7.1 Hadoop >> >>> >>> Jars. How to do start the twill application? Usually on a cluster >> with >> >>> >>> hadoop installed, you can get all the hadoop jars in the classpath >> by >> >>> >>> running this: >> >>> >>> >> >>> >>> export CP=`hadoop classpath` >> >>> >>> java -cp .:$CP YourApp ... >> >>> >>> >> >>> >>> Assuming your app classes and Twill jars are in the current >> directory. >> >>> >>> >> >>> >>> Terence >> >>> >>> >> >>> >>> On Fri, Jan 22, 2016 at 4:54 AM, Kristoffer Sjögren < >> [email protected]> >> >>> wrote: >> >>> >>>> Here's the full stacktrace. >> >>> >>>> >> >>> >>>> Exception in thread "main" >> java.lang.reflect.InvocationTargetException >> >>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >>> >>>> at >> >>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >>> >>>> at >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>> >>>> at java.lang.reflect.Method.invoke(Method.java:497) >> >>> >>>> at >> org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:89) >> >>> >>>> Caused by: java.lang.RuntimeException: >> >>> >>>> java.lang.reflect.InvocationTargetException >> >>> >>>> at >> com.google.common.base.Throwables.propagate(Throwables.java:160) >> >>> >>>> at >> >>> >> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) >> >>> >>>> at >> >>> >> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:77) >> >>> >>>> ... 5 more >> >>> >>>> Caused by: java.lang.reflect.InvocationTargetException >> >>> >>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> >>> Method) >> >>> >>>> at >> >>> >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> >>> >>>> at >> >>> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> >>> >>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >> >>> >>>> at >> >>> >> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) >> >>> >>>> ... 6 more >> >>> >>>> Caused by: java.lang.IllegalArgumentException: Invalid >> ContainerId: >> >>> >>>> container_e25_1453466340022_0004_01_000001 >> >>> >>>> at >> >>> >> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) >> >>> >>>> at >> >>> >> org.apache.twill.internal.yarn.AbstractYarnAMClient.<init>(AbstractYarnAMClient.java:83) >> >>> >>>> at >> >>> >> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.<init>(Hadoop21YarnAMClient.java:65) >> >>> >>>> at >> >>> >> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.<init>(Hadoop22YarnAMClient.java:34) >> >>> >>>> ... 11 more >> >>> >>>> Caused by: java.lang.NumberFormatException: For input string: >> "e25" >> >>> >>>> at >> >>> >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> >>> >>>> at java.lang.Long.parseLong(Long.java:589) >> >>> >>>> at java.lang.Long.parseLong(Long.java:631) >> >>> >>>> at >> >>> >> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) >> >>> >>>> at >> >>> >> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) >> >>> >>>> ... 14 more >> >>> >>>> >> >>> >>>> On Thu, Jan 21, 2016 at 10:59 PM, Kristoffer Sjögren < >> >>> [email protected]> wrote: >> >>> >>>>> Hi >> >>> >>>>> >> >>> >>>>> I'm trying the basic example [1] on yarn 2.7.1 but get an >> exception >> >>> as >> >>> >>>>> soon as the application starts on the resource manager that >> tells me >> >>> >>>>> the container id cannot be parsed. >> >>> >>>>> >> >>> >>>>> java.lang.IllegalArgumentException: Invalid containerId: >> >>> >>>>> container_e04_1427159778706_0002_01_000001 >> >>> >>>>> >> >>> >>>>> I don't have the exact stacktrace but I recall it failing in >> >>> >>>>> ConverterUtils.toContainerId because it assumes that that the >> first >> >>> >>>>> token is an application attempt to be parsed as an integer. This >> >>> class >> >>> >>>>> resides in hadoop-yarn-common 2.3.0. >> >>> >>>>> >> >>> >>>>> Is there any way to either tweak the container id or make twill >> use >> >>> >>>>> the 2.7.1 jar instead? >> >>> >>>>> >> >>> >>>>> Cheers, >> >>> >>>>> -Kristoffer >> >>> >>>>> >> >>> >>>>> >> >>> >>>>> [1] >> >>> >> https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java >> >>> >>
