[
https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276944#comment-15276944
]
Chris Nauroth commented on HADOOP-13113:
----------------------------------------
One interesting finding from repeated parallel test runs is that I'm seeing a
greater occurrence of failures in
{{TestS3AContractRootDir#testListEmptyRootDirectory}}, despite the fact that
this is one of the test suites that I force to run in serial.
{code}
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 9.085 sec <<<
FAILURE! - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir)
Time elapsed: 3.041 sec <<< FAILURE!
java.lang.AssertionError: Deleted file: unexpectedly found
s3a://cnauroth-test-aws-s3a/fork-2 as
S3AFileStatus{path=s3a://cnauroth-test-aws-s3a/fork-2; isDirectory=true;
modification_time=0; access_time=0; owner=; group=; permission=rwxrwxrwx;
isSymlink=false}
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:706)
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:566)
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:544)
at
org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:134)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
I do see an intermittent failure of this test even without the parallel test
patch, so it's not something introduced by this patch. I think the test is
subject to failure because of the eventual consistency behavior of deletes
against S3. Reviewing debug logs, in the failure case, I get the "Deleting
fake empty directory" message, even though it's definitely not empty at this
point. If I put a retry loop around the test, sometimes it fails first and
then succeeds on the retry.
It might be interesting to explore if the order of operations in
{{S3AFileSystem#listStatus}} could be changed to help with this. Maybe the
common prefixes could be analyzed first before deciding the key represents a
fake empty directory. I'm not sure if it's completely solvable though, because
any kind of S3 listing modification is subject to eventual consistency effects.
I'm not planning to change anything related to this within the scope of the
current patch though.
> Enable parallel test execution for hadoop-aws.
> ----------------------------------------------
>
> Key: HADOOP-13113
> URL: https://issues.apache.org/jira/browse/HADOOP-13113
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-13113.001.patch, HADOOP-13113.002.patch
>
>
> The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend
> most of their time blocked on network I/O with the S3 back-end, but they
> don't saturate the bandwidth of the NIC. We can improve overall execution
> time by enabling parallel test execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
