[
https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285053#comment-15285053
]
Chris Nauroth commented on HADOOP-13113:
----------------------------------------
bq. I should warn that I'm seeing errors here...
[[email protected]], here are my observations based on additional experiments.
# If I configure a dedicated bucket per file system (S3, S3N and S3A), then I
consistently get a full pass on parallel test runs. This was the setup I was
using when I developed the patch.
# All of my buckets are in AWS region US-west-2 (Oregon), which is very close
to my laptop physically.
# If I configure the same bucket to be used by the test suites for all 3 file
systems, then sometimes I see some flaky behavior. I don't know if this is due
to data problems, such as perhaps S3N tests leaving bits behind that confuse
S3A later, or if the increased activity on a shared bucket simply surfaces
eventual consistency problems with greater likelihood.
# One test run surfaced a bug, and I have submitted a patch for that on
HADOOP-13158.
# {{TestS3NContractRootDir#testListEmptyRootDirectory}} probably needs the same
retry loop that we put in place for the same test in {{TestS3AContractRootDir}}.
I'll continue to look for ways to stabilize this.
> Enable parallel test execution for hadoop-aws.
> ----------------------------------------------
>
> Key: HADOOP-13113
> URL: https://issues.apache.org/jira/browse/HADOOP-13113
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Fix For: 2.8.0
>
> Attachments: HADOOP-13113.001.patch, HADOOP-13113.002.patch,
> HADOOP-13113.003.patch, HADOOP-13113.004.patch
>
>
> The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend
> most of their time blocked on network I/O with the S3 back-end, but they
> don't saturate the bandwidth of the NIC. We can improve overall execution
> time by enabling parallel test execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
