It seems to me this issue is the direct result of MAPREDUCE-6577 <https://issues.apache.org/jira/browse/MAPREDUCE-6577> Since you’re on a CDH cluster, I would suggest you to move up to CDH5.7.2 or above where this bug is fixed.
Best, Wei-Chiu Chuang > On Oct 4, 2016, at 1:26 PM, Wei-Chiu Chuang <[email protected]> wrote: > > I see. Sorry for the confusion. > > It seems to me the warning message a bit misleading. This message may also be > printed if libhadoop can not be loaded for any reason. > Can you turn on debug log and see if the log contains either "Loaded the > native-hadoop library” or "Failed to load native-hadoop with error”? > > > Wei-Chiu Chuang > >> On Oct 4, 2016, at 1:12 PM, Uthayan Suthakar <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Wei-Chiu, >> >> My Hadoop version is Hadoop 2.6.0-cdh5.7.0. >> >> But when I checked the native, it shows that it is installed: >> >> hadoop checknative >> 16/10/04 21:01:30 INFO bzip2.Bzip2Factory: Successfully loaded & initialized >> native-bzip2 library system-native >> 16/10/04 21:01:30 INFO zlib.ZlibFactory: Successfully loaded & initialized >> native-zlib library >> Native library checking: >> hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0 >> zlib: true /lib64/libz.so.1 >> snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1 >> lz4: true revision:99 >> bzip2: true /lib64/libbz2.so.1 >> openssl: true /usr/lib64/libcrypto.so >> >> Thanks. >> >> Uthay >> >> >> On 4 October 2016 at 21:05, Wei-Chiu Chuang <[email protected] >> <mailto:[email protected]>> wrote: >> Hi Uthayan, >> what’s the version of Hadoop you have? Hadoop 2.7.3 binary does not ship >> with snappy precompiled. If this is the version you have you may have to >> rebuild Hadoop yourself to include it. >> >> Wei-Chiu Chuang >> >>> On Oct 4, 2016, at 12:59 PM, Uthayan Suthakar <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hello guys, >>> >>> I have a job that reads compressed (Snappy) data but when I run the job, it >>> is throwing an error "native snappy library not available: this version of >>> libhadoop was built without snappy support". >>> . >>> I followed this instruction but it did not resolve the issue: >>> https://community.hortonworks.com/questions/18903/this-version-of-libhadoop-was-built-without-snappy.html >>> >>> <https://community.hortonworks.com/questions/18903/this-version-of-libhadoop-was-built-without-snappy.html> >>> >>> The check native command show that snappy is installed. >>> hadoop checknative >>> 16/10/04 21:01:30 INFO bzip2.Bzip2Factory: Successfully loaded & >>> initialized native-bzip2 library system-native >>> 16/10/04 21:01:30 INFO zlib.ZlibFactory: Successfully loaded & initialized >>> native-zlib library >>> Native library checking: >>> hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0 >>> zlib: true /lib64/libz.so.1 >>> snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1 >>> lz4: true revision:99 >>> bzip2: true /lib64/libbz2.so.1 >>> openssl: true /usr/lib64/libcrypto.so >>> >>> I also have a code in the job to check whether native snappy is loaded, >>> which is returning true. >>> >>> Now, I have no idea why I'm getting this error. Also, I had no issue >>> reading Snappy data using MapReduce job on the same cluster, Could anyone >>> tell me what is wrong? >>> >>> >>> >>> Thank you. >>> >>> Stack: >>> >>> >>> java.lang.RuntimeException: native snappy library not available: this >>> version of libhadoop was built without snappy support. >>> at >>> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) >>> at >>> org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:193) >>> at >>> org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:178) >>> at >>> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:111) >>> at >>> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67) >>> at >>> org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:237) >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>> at >>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) >>> at >>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>> at org.apache.spark.scheduler.Task.run(Task.scala:89) >>> at >>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >> >> >
