Further adding the following dependencies cause another exception. <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>13.0</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.7.1</version> </dependency>
Exception in thread " STARTING" java.lang.IncompatibleClassChangeError: class org.apache.twill.internal.utils.Dependencies$DependencyClassVisitor has interface org.objectweb.asm.ClassVisitor as super class at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:760) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.twill.internal.utils.Dependencies.findClassDependencies(Dependencies.java:86) at org.apache.twill.internal.ApplicationBundler.findDependencies(ApplicationBundler.java:198) at org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:155) at org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:126) at org.apache.twill.yarn.YarnTwillPreparer.createAppMasterJar(YarnTwillPreparer.java:402) at org.apache.twill.yarn.YarnTwillPreparer.access$200(YarnTwillPreparer.java:108) at org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:299) at org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:289) at org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:97) at org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:76) at org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:175) at com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) at java.lang.Thread.run(Thread.java:745) On Fri, Jan 22, 2016 at 11:32 PM, Kristoffer Sjögren <[email protected]> wrote: > Add those dependencies fail with the following exception. > > Exception in thread "main" java.lang.AbstractMethodError: > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy()Lorg/apache/hadoop/io/retry/FailoverProxyProvider$ProxyInfo; > at > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64) > at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) > at > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:149) > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:569) > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:512) > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:142) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) > at > org.apache.twill.yarn.YarnTwillRunnerService.createDefaultLocationFactory(YarnTwillRunnerService.java:615) > at > org.apache.twill.yarn.YarnTwillRunnerService.<init>(YarnTwillRunnerService.java:149) > at deephacks.BundledJarExample.main(BundledJarExample.java:70) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) > > On Fri, Jan 22, 2016 at 10:58 PM, Terence Yim <[email protected]> wrote: >> Hi, >> >> If you run it from IDE, you and simply add a dependency on hadoop with >> version 2.7.1. E.g. if you are using Maven, you can add the following to >> your pom.xml dependencies section. >> >> <dependency> >> <groupId>org.apache.hadoop</groupId> >> <artifactId>hadoop-yarn-api</artifactId> >> <version>2.7.1</version> >> </dependency> >> <dependency> >> <groupId>org.apache.hadoop</groupId> >> <artifactId>hadoop-yarn-common</artifactId> >> <version>2.7.1</version> >> </dependency> >> <dependency> >> <groupId>org.apache.hadoop</groupId> >> <artifactId>hadoop-yarn-client</artifactId> >> <version>2.7.1</version> >> </dependency> >> <dependency> >> <groupId>org.apache.hadoop</groupId> >> <artifactId>hadoop-common</artifactId> >> <version>2.7.1</version> >> </dependency> >> >> Terence >> >> On Fri, Jan 22, 2016 at 12:47 PM, Kristoffer Sjögren <[email protected]> >> wrote: >> >>> I run it from IDE right now, but would like to create a command line >>> app eventually. >>> >>> I should clarify that the exception above is thrown on the YARN node, >>> not in the IDE. >>> >>> On Fri, Jan 22, 2016 at 9:32 PM, Terence Yim <[email protected]> wrote: >>> > Hi Kristoffer, >>> > >>> > The example itself shouldn't need any modification. However, how do >>> > you run that class? Do you run it from IDE or from command line using >>> > "java" command? >>> > >>> > Terence >>> > >>> > On Fri, Jan 22, 2016 at 12:16 PM, Kristoffer Sjögren <[email protected]> >>> wrote: >>> >> Hi Terence, >>> >> >>> >> I'm quite new to Twill and not sure how to do that exactly. Could you >>> >> show me how to modify the following example to do the same? >>> >> >>> >> >>> https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java >>> >> >>> >> On Fri, Jan 22, 2016 at 7:24 PM, Terence Yim <[email protected]> wrote: >>> >>> Hi Kristoffer, >>> >>> >>> >>> Seems like the exception comes from the YARN class "ConverterUtils". I >>> >>> believe need to start the application with the version 2.7.1 Hadoop >>> >>> Jars. How to do start the twill application? Usually on a cluster with >>> >>> hadoop installed, you can get all the hadoop jars in the classpath by >>> >>> running this: >>> >>> >>> >>> export CP=`hadoop classpath` >>> >>> java -cp .:$CP YourApp ... >>> >>> >>> >>> Assuming your app classes and Twill jars are in the current directory. >>> >>> >>> >>> Terence >>> >>> >>> >>> On Fri, Jan 22, 2016 at 4:54 AM, Kristoffer Sjögren <[email protected]> >>> wrote: >>> >>>> Here's the full stacktrace. >>> >>>> >>> >>>> Exception in thread "main" java.lang.reflect.InvocationTargetException >>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> >>>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> >>>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>>> at java.lang.reflect.Method.invoke(Method.java:497) >>> >>>> at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:89) >>> >>>> Caused by: java.lang.RuntimeException: >>> >>>> java.lang.reflect.InvocationTargetException >>> >>>> at com.google.common.base.Throwables.propagate(Throwables.java:160) >>> >>>> at >>> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) >>> >>>> at >>> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:77) >>> >>>> ... 5 more >>> >>>> Caused by: java.lang.reflect.InvocationTargetException >>> >>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> >>>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >>> >>>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> >>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>> >>>> at >>> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) >>> >>>> ... 6 more >>> >>>> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: >>> >>>> container_e25_1453466340022_0004_01_000001 >>> >>>> at >>> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) >>> >>>> at >>> org.apache.twill.internal.yarn.AbstractYarnAMClient.<init>(AbstractYarnAMClient.java:83) >>> >>>> at >>> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.<init>(Hadoop21YarnAMClient.java:65) >>> >>>> at >>> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.<init>(Hadoop22YarnAMClient.java:34) >>> >>>> ... 11 more >>> >>>> Caused by: java.lang.NumberFormatException: For input string: "e25" >>> >>>> at >>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >>> >>>> at java.lang.Long.parseLong(Long.java:589) >>> >>>> at java.lang.Long.parseLong(Long.java:631) >>> >>>> at >>> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) >>> >>>> at >>> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) >>> >>>> ... 14 more >>> >>>> >>> >>>> On Thu, Jan 21, 2016 at 10:59 PM, Kristoffer Sjögren < >>> [email protected]> wrote: >>> >>>>> Hi >>> >>>>> >>> >>>>> I'm trying the basic example [1] on yarn 2.7.1 but get an exception >>> as >>> >>>>> soon as the application starts on the resource manager that tells me >>> >>>>> the container id cannot be parsed. >>> >>>>> >>> >>>>> java.lang.IllegalArgumentException: Invalid containerId: >>> >>>>> container_e04_1427159778706_0002_01_000001 >>> >>>>> >>> >>>>> I don't have the exact stacktrace but I recall it failing in >>> >>>>> ConverterUtils.toContainerId because it assumes that that the first >>> >>>>> token is an application attempt to be parsed as an integer. This >>> class >>> >>>>> resides in hadoop-yarn-common 2.3.0. >>> >>>>> >>> >>>>> Is there any way to either tweak the container id or make twill use >>> >>>>> the 2.7.1 jar instead? >>> >>>>> >>> >>>>> Cheers, >>> >>>>> -Kristoffer >>> >>>>> >>> >>>>> >>> >>>>> [1] >>> https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java >>>
