ACCUMULO-4426 Lift MR-IT instructions to TESTING.md
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/def8f90e Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/def8f90e Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/def8f90e Branch: refs/heads/1.8 Commit: def8f90eff10d8c598030abfa433ec8907aae735 Parents: e29055f Author: Josh Elser <els...@apache.org> Authored: Fri Aug 26 11:03:23 2016 -0400 Committer: Josh Elser <els...@apache.org> Committed: Fri Aug 26 11:03:23 2016 -0400 ---------------------------------------------------------------------- TESTING.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/accumulo/blob/def8f90e/TESTING.md ---------------------------------------------------------------------- diff --git a/TESTING.md b/TESTING.md index 03ee687..2195108 100644 --- a/TESTING.md +++ b/TESTING.md @@ -133,6 +133,27 @@ but for different instances, the correct version is terminated. `HADOOP_CONF_DIR files to construct the FileSystem object for the cluster can be constructed (e.g. core-site.xml and hdfs-site.xml), which is typically required to interact with HDFS. +## MapReduce job for Integration tests + +[ACCUMULO-3871][6] (re)introduced the ability to parallelize the execution of the Integration Test suite by the use +of MapReduce/YARN. When a YARN cluster is available, this can drastically reduce the amount of time to run all tests. + +To run the tests, you first need a list of the tests. A simple way to get a list, is to scan the accumulo-test jar file for them. + +`jar -tf lib/accumulo-test.jar | grep IT.class | tr / . | sed -e 's/.class$//' >accumulo-integration-tests.txt` + +Then, put the list of files into HDFS: + +`hdfs dfs -mkdir /tmp` +`hdfs dfs -put accumulo-integration-tests.txt /tmp/tests` + +Finally, launch the job, providing the list of tests to run and a location to store the test results. Optionally, a built +native library shared object can be provided to the Mapper's classpath to enable MiniAccumuloCluster to use the native maps +instead of the Java-based implementation. (Note that the below paths are the JAR and shared object are based on an installation. +These files do exist in the build tree, but at different locations) + +`yarn jar lib/accumulo-test.jar org.apache.accumulo.test.mrit.IntegrationtestMapReduce -libjars lib/native/libaccumulo.so /tmp/accumulo-integration-tests.txt /tmp/accumulo-integration-test-results` + # Manual Distributed Testing Apache Accumulo also contains a number of tests which are suitable for running against large clusters for hours to days @@ -144,4 +165,4 @@ at a time, for example the [Continuous Ingest][1] and [Randomwalk test][2] suite [3]: https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html [4]: http://maven.apache.org/surefire/maven-surefire-plugin/ [5]: http://maven.apache.org/surefire/maven-failsafe-plugin/ - +[6]: https://issues.apache.org/jira/browse/ACCUMULO-3871
