[
https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524139#comment-15524139
]
Steve Loughran commented on HADOOP-13560:
-----------------------------------------
Testing: s3 ireland. One transient failure that I'm taking as a sighting of a
eventual consistency mismatch: an object appearing in a HEAD probe after it's
deletion.
{code}
testDeleteEmptyDirRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete)
Time elapsed: 1.234 sec <<< FAILURE!
java.lang.AssertionError: Deleted file: unexpectedly found
s3a://hwdev-steve-ireland/test/testDeleteEmptyDirRecursive as
S3AFileStatus{path=s3a://hwdev-steve-ireland/test/testDeleteEmptyDirRecursive;
isDirectory=true; modification_time=0; access_time=0; owner=; group=;
permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:714)
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:572)
at
org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:550)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertDeleted(AbstractFSContractTestBase.java:349)
at
org.apache.hadoop.fs.contract.AbstractContractDeleteTest.testDeleteEmptyDirRecursive(AbstractContractDeleteTest.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
We may want to think about having the base test slightly more consistency
aware, that is: retrying if the FS has a blobstore flag set in its contract.
For now though, helps collect stats on frequency of this being visible.
> S3ABlockOutputStream to support huge (many GB) file writes
> ----------------------------------------------------------
>
> Key: HADOOP-13560
> URL: https://issues.apache.org/jira/browse/HADOOP-13560
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-13560-branch-2-001.patch,
> HADOOP-13560-branch-2-002.patch, HADOOP-13560-branch-2-003.patch,
> HADOOP-13560-branch-2-004.patch
>
>
> An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights
> that metadata isn't copied on large copies.
> 1. Add a test to do that large copy/rname and verify that the copy really
> works
> 2. Verify that metadata makes it over.
> Verifying large file rename is important on its own, as it is needed for very
> large commit operations for committers using rename
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
