Hi Paula, I am not sure how to answer your questions but is there a reason why you are using an EC2 instance instead of amazonz EMR (elastic Map reduce) Hadoop cluster. As far as I know you can set that up to work with HDFS setup as well as S3 buckets if you don’t need a long term cluster to stay online.
Regards, Jonathan From: Paula Logan <[email protected]> Sent: 10 September 2021 16:13 To: [email protected] Subject: hadoop-hdfs-native-client Help Hello, I am new to building Hadoop locally, and am having some issues. Please let me know if this information should be sent to a different distro. (1) Can Hadoop 3.3.1 be compiled and run with OpenJDK 11 or is OpenJDK 1.8 needed for compile while 1.8 or 11 can be used to run hadoop? (2) I am compiling and testing Hadoop 3.3.1 on RHEL 8.4 on the command line not via any IDE inside an AWS instance. I have encountered an issue with Native Test Case #35 (all other 39 Native Test Cases succeed). First here is my maven command: mvn -e -X test -Pnative,parallel-tests,shelltest,yarn-ui -Dtest=allNative -Dparallel-tests=true -Drequire.bzip2=true -Drequire.fuse=true -Drequire.isal=true -Disal.prefix=/usr/local -Disal.lib=/usr/local/lib64 -Dbundle.isal=true -Drequire.openssl=true -Dopenssl.prefix=/usr -Dopenssl.include=/usr/include -Dopenssl.lib=/usr/lib64 -Dbundle.openssl=true -Dbundle.openssl.in.bin=true -Drequire.pmdk=true -Dpmdk.lib=/usr/lib64 -Dbundle.pmdk=true -Drequire.snappy=true -Dsnappy.prefix=/usr -Dsnappy.include=/usr/include -Dsnappy.lib=/usr/lib64 -Dbundle.snappy=true -Drequire.valgrind=true -Dhbase.profile=2.0 -Drequire.zstd=true -Dzstd.prefix=/usr -Dzstd.include=/usr/include -Dzstd.lib=/usr/lib64 -Dbundle.zstd=true -Dbundle.zstd.in.bin=true -Drequire.test.libhadoop=true This is what I get for Test Case #35: [exec] 35/40 Test #35: test_libhdfs_threaded_hdfspp_test_shim_static ..............***Failed 31.58 sec [exec] testRecursiveJvmMutex error: [exec] ClassNotFoundException: RuntimeExceptionjava.lang.NoClassDefFoundError: RuntimeException [exec] Caused by: java.lang.ClassNotFoundException: RuntimeException [exec] at java.net.URLClassLoader.findClass(URLClassLoader.java:382) [exec] at java.lang.ClassLoader.loadClass(ClassLoader.java:418) [exec] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) [exec] at java.lang.ClassLoader.loadClass(ClassLoader.java:351) [exec] 2021-09-02 22:31:09,706 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:<init>(529)) - starting cluster: numNameNodes=1, numDataNodes=1 [exec] 2021-09-02 22:31:10,134 INFO namenode.NameNode (NameNode.java:format(1249)) - Formatting using clusterid: testClusterID [exec] 2021-09-02 22:31:10,156 INFO namenode.FSEditLog (FSEditLog.java:newInstance(229)) - Edit logging is async:true [exec] 2021-09-02 22:31:10,182 INFO namenode.FSNamesystem (FSNamesystem.java:<init>(814)) - KeyProvider: null [exec] 2021-09-02 22:31:10,184 INFO namenode.FSNamesystem (FSNamesystemLock.java:<init>(141)) - fsLock is fair: true [exec] 2021-09-02 22:31:10,185 INFO namenode.FSNamesystem (FSNamesystemLock.java:<init>(159)) - Detailed lock hold time metrics enabled: false [exec] 2021-09-02 22:31:10,185 INFO namenode.FSNamesystem (FSNamesystem.java:<init>(847)) - fsOwner = ec2-user (auth:SIMPLE) [exec] 2021-09-02 22:31:10,185 INFO namenode.FSNamesystem (FSNamesystem.java:<init>(848)) - supergroup ... [exec] 2021-09-02 22:31:13,204 INFO ipc.Server (Server.java:logException(3020)) - IPC Server handler 7 on default port 44945, call Call#6 Retry#-1 org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 127.0.0.1:37362: java.io.FileNotFoundException: File does not exist: /tlhData0001/file1 ... [exec] 98% tests passed, 1 tests failed out of 40 [exec] [exec] Total Test time (real) = 270.30 sec [exec] [exec] The following tests FAILED: [exec] 35 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] Errors while running CTest [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main 3.3.1 ........................... SUCCESS [ 0.707 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.743 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.692 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.955 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 0.106 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.101 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.194 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 0.806 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.192 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 0.452 s] [INFO] Apache Hadoop Common ............................... SUCCESS [ 54.493 s] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.123 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 2.087 s] [INFO] Apache Hadoop Registry ............................. SUCCESS [ 2.538 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.055 s] [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 16.283 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 24.263 s] [INFO] Apache Hadoop HDFS Native Client ................... FAILURE [04:49 min] [INFO] Apache Hadoop HttpFS ............................... SKIPPED ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 06:49 min [INFO] Finished at: 2021-09-02T22:34:18Z [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (native_tests) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 8 [ERROR] around Ant part ...<exec failonerror="true" dir="/home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/" executable="ctest">... @ 6:150 in /home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml [ERROR] -> [Help 1] This RuntimeException error also appears for Native Test Case 2 but that test case doesn't fail. Also, see a lot of "File not found" messages. Assume at this point that the NoClassDefFoundError causes the code that creates the files to be skipped and once the NoClassDefFoundError is fixed, these files will be generated. The compile of Hadoop 3.3.1 on RHEL 8.4 succeeded without issues. I have JAVA_HOME and JRE_HOME set in .bashrc to OpenJDK 1.8 and have added these into the $PATH. export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el8_4.x86_64 export JAVA_OPTS="-Xms2048m -Xmx4096m -XX:+UseZGC" export JRE_HOME=${JAVA_HOME}/jre export LIBHDFS_OPTS="-Xms2048m -Xmx4096m" export MAVEN_HOME=/usr/share/maven export MAVEN_OPTS="-Xms256m -Xmx1536m" export PROTOBUF_HOME=/usr/local export PATH=/home/ec2-user/.local/bin:/home/ec2-user/bin:${JAVA_HOME}/bin:${JRE_HOME}/bin:${MAVEN_HOME}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin It appears the $JRE_HOME/lib/rt.jar isn't being included in the maven.test.classpath or in the native module tests. I tried setting CLASSPATH and JAVA_LIBRARY_PATH in .bashrc and tried passing via 'mvn' command, but still no success. I followed the procedures in the BUILDING.txt file for the CentOS 8 as that was the closest to RHEL 8.4. ${HADOOP_SRC_HOME}/hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml <properties> <native_cmake_args></native_cmake_args> <native_ctest_args></native_ctest_args> <native_make_args></native_make_args> Do I need to supply anything for any of these three properties? (3) Here is some more snippets of information from the maven log that I captured. [exec] 98% tests passed, 1 tests failed out of 40 [exec] [exec] Total Test time (real) = 270.30 sec [exec] [exec] The following tests FAILED: [exec] 35 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] Errors while running CTest [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main 3.3.1 ........................... SUCCESS [ 0.707 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.743 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.692 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.955 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 0.106 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.101 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.194 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 0.806 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.192 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 0.452 s] [INFO] Apache Hadoop Common ............................... SUCCESS [ 54.493 s] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.123 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 2.087 s] [INFO] Apache Hadoop Registry ............................. SUCCESS [ 2.538 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.055 s] [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 16.283 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 24.263 s] [INFO] Apache Hadoop HDFS Native Client ................... FAILURE [04:49 min] [INFO] Apache Hadoop HttpFS ............................... SKIPPED ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 06:49 min [INFO] Finished at: 2021-09-02T22:34:18Z [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (native_tests) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 8 [ERROR] around Ant part ...<exec failonerror="true" dir="/home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/" executable="ctest">... @ 6:150 in /home/ec2-user/workspaces/hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml [ERROR] -> [Help 1] (4) What are the following properties? require.test.libhadoop - not sure of the purpose of this property or if I need it or is it just for hadoop project developers bundle.<type> -vs- bundle.<type>.in.bin - What is the difference of bundle vs bundle.in.bin? Please let me know what I might be missing or if a (or some) native files need to be modified to have the rt.jar (RuntimeException class contained within) be included. Wasn't sure if this was an OpenJDK 1.8 vs OpenJDK 11 issue as the JRE binary is located in a different directory in OpenJDK 11. I am not using OpenJDK 11 at all nor is it installed in my RHEL 8.4 AWS instance. I didn't submit a ticket as I assume there is something that I am not doing correctly or forgetting to include/do. Any help you can give me would be very much appreciated. Paula
