Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id :
attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException:
org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[email protected]]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [email protected]
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do
any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the
generic option -libjars.
Can you give the stack trace of your problem for which class it is giving
ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[email protected]]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [email protected]
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache
commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math
jar as lib/commons-math3.jar I have placed this jar file in HADOOP_HOME/lib
folder but still I am getting ClassNotFoundException.
I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar
<inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop jar
myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not working.
Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer ***************** This e-mail contains
PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the
addressee(s). If you are not the intended recipient, please notify the sender
by e-mail and delete the original message. Further, you are not to copy,
disclose, or distribute this e-mail or its contents to any other person and any
such actions are unlawful. This e-mail may contain viruses. Infosys has taken
every reasonable precaution to minimize this risk, but is not liable for any
damage you may sustain as a result of any virus in this e-mail. You should
carry out your own virus checks before opening the e-mail or attachment.
Infosys reserves the right to monitor and review the content of all messages
sent to or from this e-mail address. Messages sent to or from this e-mail
address may be stored on the Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
