Hi, If you run it from IDE, you and simply add a dependency on hadoop with version 2.7.1. E.g. if you are using Maven, you can add the following to your pom.xml dependencies section.
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-api</artifactId> <version>2.7.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-common</artifactId> <version>2.7.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-client</artifactId> <version>2.7.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.7.1</version> </dependency> Terence On Fri, Jan 22, 2016 at 12:47 PM, Kristoffer Sjögren <[email protected]> wrote: > I run it from IDE right now, but would like to create a command line > app eventually. > > I should clarify that the exception above is thrown on the YARN node, > not in the IDE. > > On Fri, Jan 22, 2016 at 9:32 PM, Terence Yim <[email protected]> wrote: > > Hi Kristoffer, > > > > The example itself shouldn't need any modification. However, how do > > you run that class? Do you run it from IDE or from command line using > > "java" command? > > > > Terence > > > > On Fri, Jan 22, 2016 at 12:16 PM, Kristoffer Sjögren <[email protected]> > wrote: > >> Hi Terence, > >> > >> I'm quite new to Twill and not sure how to do that exactly. Could you > >> show me how to modify the following example to do the same? > >> > >> > https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java > >> > >> On Fri, Jan 22, 2016 at 7:24 PM, Terence Yim <[email protected]> wrote: > >>> Hi Kristoffer, > >>> > >>> Seems like the exception comes from the YARN class "ConverterUtils". I > >>> believe need to start the application with the version 2.7.1 Hadoop > >>> Jars. How to do start the twill application? Usually on a cluster with > >>> hadoop installed, you can get all the hadoop jars in the classpath by > >>> running this: > >>> > >>> export CP=`hadoop classpath` > >>> java -cp .:$CP YourApp ... > >>> > >>> Assuming your app classes and Twill jars are in the current directory. > >>> > >>> Terence > >>> > >>> On Fri, Jan 22, 2016 at 4:54 AM, Kristoffer Sjögren <[email protected]> > wrote: > >>>> Here's the full stacktrace. > >>>> > >>>> Exception in thread "main" java.lang.reflect.InvocationTargetException > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>>> at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>>> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>> at java.lang.reflect.Method.invoke(Method.java:497) > >>>> at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:89) > >>>> Caused by: java.lang.RuntimeException: > >>>> java.lang.reflect.InvocationTargetException > >>>> at com.google.common.base.Throwables.propagate(Throwables.java:160) > >>>> at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > >>>> at > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:77) > >>>> ... 5 more > >>>> Caused by: java.lang.reflect.InvocationTargetException > >>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > >>>> at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > >>>> at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > >>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > >>>> at > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > >>>> ... 6 more > >>>> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: > >>>> container_e25_1453466340022_0004_01_000001 > >>>> at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > >>>> at > org.apache.twill.internal.yarn.AbstractYarnAMClient.<init>(AbstractYarnAMClient.java:83) > >>>> at > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.<init>(Hadoop21YarnAMClient.java:65) > >>>> at > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.<init>(Hadoop22YarnAMClient.java:34) > >>>> ... 11 more > >>>> Caused by: java.lang.NumberFormatException: For input string: "e25" > >>>> at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > >>>> at java.lang.Long.parseLong(Long.java:589) > >>>> at java.lang.Long.parseLong(Long.java:631) > >>>> at > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > >>>> at > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > >>>> ... 14 more > >>>> > >>>> On Thu, Jan 21, 2016 at 10:59 PM, Kristoffer Sjögren < > [email protected]> wrote: > >>>>> Hi > >>>>> > >>>>> I'm trying the basic example [1] on yarn 2.7.1 but get an exception > as > >>>>> soon as the application starts on the resource manager that tells me > >>>>> the container id cannot be parsed. > >>>>> > >>>>> java.lang.IllegalArgumentException: Invalid containerId: > >>>>> container_e04_1427159778706_0002_01_000001 > >>>>> > >>>>> I don't have the exact stacktrace but I recall it failing in > >>>>> ConverterUtils.toContainerId because it assumes that that the first > >>>>> token is an application attempt to be parsed as an integer. This > class > >>>>> resides in hadoop-yarn-common 2.3.0. > >>>>> > >>>>> Is there any way to either tweak the container id or make twill use > >>>>> the 2.7.1 jar instead? > >>>>> > >>>>> Cheers, > >>>>> -Kristoffer > >>>>> > >>>>> > >>>>> [1] > https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java >
