ACCUMULO-2379 Document running functional tests under MR as a different user
Functional tests might not run under MapReduce in clusters where a different user from the Accumulo user runs mappers. Needed configuration changes are now documented in the functional test README. Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/ac18b569 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/ac18b569 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/ac18b569 Branch: refs/heads/1.6.0-SNAPSHOT Commit: ac18b569738b6dd8f950ee5d747af5176bbc8d13 Parents: 3898317 Author: Bill Havanki <bhava...@cloudera.com> Authored: Wed Feb 19 11:05:12 2014 -0500 Committer: Bill Havanki <bhava...@cloudera.com> Committed: Wed Feb 19 11:05:12 2014 -0500 ---------------------------------------------------------------------- test/system/auto/README | 94 ++++++++++++++++++++++++++++++-------------- 1 file changed, 65 insertions(+), 29 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/accumulo/blob/ac18b569/test/system/auto/README ---------------------------------------------------------------------- diff --git a/test/system/auto/README b/test/system/auto/README index 4773393..45ed158 100644 --- a/test/system/auto/README +++ b/test/system/auto/README @@ -1,58 +1,94 @@ Apache Accumulo Functional Tests -These scripts run a series of tests against a small local accumulo instance. To run these scripts, -you must have hadoop and zookeeper installed and running. You will need a functioning C compiler to build -a shared library needed for one of the tests. The test suite is known to run on Linux RedHat Enterprise +These scripts run a series of tests against a small local accumulo instance. +To run these scripts, you must have Hadoop and Zookeeper installed and running. +You will need a functioning C compiler to build a shared library needed for +one of the tests. The test suite is known to run on Linux RedHat Enterprise version 5, and Mac OS X 10.5. -The tests are shown as being run from the ACCUMULO_HOME directory, but they should run from any directory. -Make sure to create "logs" and "walogs" directories in ACCUMULO_HOME. Also, ensure that accumulo-env.sh -specifies its ACCUMULO_LOG_DIR in the following way: +How to Run + +The tests are shown as being run from the ACCUMULO_HOME directory, but they +should run from any directory. Make sure to create "logs" and "walogs" +directories in ACCUMULO_HOME. Also, ensure that accumulo-env.sh specifies its +ACCUMULO_LOG_DIR in the following way: + test -z "$ACCUMULO_LOG_DIR" && export ACCUMULO_LOG_DIR=$ACCUMULO_HOME/logs $ ./test/system/auto/run.py -l - -Will list all the test names. You can run the suite like this: - + +will list all the test names. You can run the suite like this: + $ ./test/system/auto/run.py - + You can select tests using a case-insensitive regular expression: - + $ ./test/system/auto/run.py -t simple $ ./test/system/auto/run.py -t SunnyDay - + To run tests repeatedly: $ ./test/system/auto/run.py -r 3 -If you are attempting to debug what is causing a test to fail, you can run the tests in "verbose" mode: +If you are attempting to debug what is causing a test to fail, you can run the +tests in "verbose" mode: $ python test/system/auto/run.py -t SunnyDay -v 10 - -If a test is failing, and you would like to examine logs from the run, you can run the test in "dirty" -mode which will keep the test from cleaning up all the logs at the end of the run: - + +If a test is failing, and you would like to examine logs from the run, you can +run the test in "dirty" mode which will keep the test from cleaning up all the +logs at the end of the run: + $ ./test/system/auto/run.py -t some.failing.test -d - -If the test suite hangs, and you would like to re-run the tests starting with the last test that failed: - + +If the test suite hangs, and you would like to re-run the tests starting with +the last test that failed: + $ ./test/system/auto/run.py -s start.over.test - -If tests tend to time out (on slower hardware, for example), you can scale up the timeout values by a -multiplier. This example triples timeouts: + +If tests tend to time out (on slower hardware, for example), you can scale up +the timeout values by a multiplier. This example triples timeouts: $ ./test/system/auto/run.py -f 3 -Test results are normally printed to the console, but you can send them to XML files compatible with -Jenkins: +Test results are normally printed to the console, but you can send them to XML +files compatible with Jenkins: $ ./test/system/auto/run.py -x -The full test suite can take nearly an hour. If you have a larger hadoop cluster at your disposal, you can -run the tests as a map-reduce job: +Running under MapReduce + +The full test suite can take nearly an hour. If you have a larger Hadoop +cluster at your disposal, you can run the tests as a MapReduce job: $ python test/system/auto/run.py -l > tests $ hadoop fs -put tests /user/hadoop/tests - $ ./bin/accumulo org.apache.accumulo.server.test.functional.RunTests /user/hadoop/tests /user/hadoop/results + $ ./bin/accumulo org.apache.accumulo.server.test.functional.RunTests \ + /user/hadoop/tests /user/hadoop/results + +The example above runs every test. You can trim the tests file to include +only the tests you wish to run. + +In some clusters, the user under which MR jobs run is different from the user +under which Accumulo is installed, and this can cause failures running the +tests. Various configuration and permission changes can be made to help the +tests run, including the following: + +* Opening up directory and file permissions on each cluster node so that the MR + user has the same read/write privileges as the Accumulo user. Adding the MR + user to a shared group is one easy way to accomplish this. Access is required + to the Accumulo installation, log, write-ahead log, and configuration + directories. +* Creating a user directory in HDFS, named after and owned by the MR user, + e.g., /user/mruser. +* Setting the ZOOKEEPER_HOME and HADOOP_CONF_DIR environment variables for the + MR user. These can be set using the mapred.child.env property in + mapred-site.xml, e.g.: + + <property> + <name>mapred.child.env</name> + <value>ZOOKEEPER_HOME=/path/to/zookeeper,HADOOP_CONF_DIR=/path/to/hadoop/conf</value> + </property> - +Each functional test is run by a mapper, and so you can check the mapper logs +to see any error messages tests produce.
