[
https://issues.apache.org/jira/browse/HADOOP-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13446:
-----------------------------------
Attachment: HADOOP-13446-HADOOP-13345.001.patch
I'm attaching patch v001, implementing what I had in mind. Summary:
# Add maven-failsafe-plugin to the build.
# Rename all test classes that require connecting to S3 from Test*.java to
ITest*.java to fit the maven-failsafe-plugin naming convention. These will run
during the Maven integration-test phase instead of the test phase.
# For test classes that are true unit tests wihtout any dependency on S3, leave
them untouched. These will continue to use the maven-surefire-plugin naming
convention and run during the test phase. There are 5 existing test suite
classes like this already, and I expect S3Guard will add more.
The patch file is large, but it's mostly the result of scripting a bunch of
renames. The Java code in the test classes is unchanged, so I don't expect
anyone to do a line-by-line review of the test suites.
Due to Maven quirks, the execution definition for maven-failsafe-plugin when
running in sequential mode must not go into the main build definition.
Otherwise, it would override the parallel definition. The workaround for this
is to put the definition for sequential mode behind its own sequential-tests
profile, triggered only when the parallel-tests profile is off. This requires
use of activation by presence (or lack of presence) of a property, not a
profile name. For this to work properly, we are now going to have to specify
{{-Dparallel-tests}} instead of {{-Pparallel-tests}} if we want a parallel run.
(Usage when running in sequential mode is unchanged.)
Considering use of the property and use of the new Maven lifecycle phases, that
means we support the following usage:
*{{mvn test}}* - Run only the unit tests. No connection to S3 is required.
These tests run even if S3 credentials are not present in the environment.
*{{mvn verify}}* - Run unit tests and integration tests. If S3 credentials are
not present, then the integration tests are skipped automatically.
*{{mvn -Dparallel-tests test}}* - Run only the unit tests, in parallel.
*{{mvn -Dparallel-tests verify}}* - Run unit tests and integration tests, in
parallel. If S3 credentials are not present, then the integration tests are
skipped automatically.
Our policy now should be that every patch must demonstrate a clean {{mvn
verify}} run before committing.
I recommend not running {{mvn -Pparallel-tests verify}} ({{-P}} instead of
{{-D}}). Due to the Maven quirks described above, you won't actually get
parallel execution.
I have tested all of the above scenarios. Running a specific test with
{{-Dtest=}} works too.
> S3Guard: Support running isolated unit tests separate from AWS integration
> tests.
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-13446
> URL: https://issues.apache.org/jira/browse/HADOOP-13446
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-13446-HADOOP-13345.001.patch
>
>
> Currently, the hadoop-aws module only runs Surefire if AWS credentials have
> been configured. This implies that all tests must run integrated with the
> AWS back-end. It also means that no tests run as part of ASF pre-commit.
> This issue proposes for the hadoop-aws module to support running isolated
> unit tests without integrating with AWS. This will benefit S3Guard, because
> we expect the need for isolated mock-based testing to simulate eventual
> consistency behavior. It also benefits hadoop-aws in general by allowing
> pre-commit to do something more valuable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
