[
https://issues.apache.org/jira/browse/HADOOP-16380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866597#comment-16866597
]
Steve Loughran commented on HADOOP-16380:
-----------------------------------------
Cause (Which you can verify by changing the limit on L2626 in S3AFileSystem
{code}
S3ListRequest request = createListObjectsRequest(key, path, 1);
{code}
If the single entry returned by the LIST is one which has a tombstone marker,
then the child list is considered empty, so the empty directory marker is set.
As a result, in a delete(path, false) operation, the S3AFilesystem will see the
empty dir marker and conclude that (a) the directory is free to be deleted
(even if recursive==false), and implement the delete simply by adding an empty
directory marker at the destination. Which then creates orphan files underneath.
I can verify this because changing the limit to a number > than the number of
outstanding tombstones stops the test {{testRmNonEmptyRootDirNonRecursive}}
failing.
{code}
S3ListRequest request = createListObjectsRequest(key, path, 100);
{code}
h2. Analysis
This is a problem which has been lurking for some time. It's been surfacing
more on the root dir tests because its under root where lots of files get
created and deleted, so it has the most tombstone markers, and the possibility
exists that under the list there's an object dir1/file4 which exists in the
store but where there is a tombstone marker for dir1/ At which point things get
confused.
h3. Store status _as returned by LIST_
{code}
/dir1/file2
/file1
{code}
This could be the actual store status, or it could be the status as listed due
to eventual consistency on the delete call
h3. metastore status
{code}
/dir1 -> isDeleted
/file1 -> is file
{code}
h3. list response
{code}
LIST (prefix=/, delim=/, limit=1) =>
commonprefixes=dir1
truncated=true
{code}
h3. S3AFileSystem.innerGetFileStatus("/", needEmptyDirectory=true)
# does the LIST call
# gets the /dir entry back (its sorted ahead of "file")
# sees that there is a tombstone
# returns an S3AFileStatus with emptyDirectory = true
* For any path other than root, any empty dir marker is maybe a parent is
created, and S3Guard a tombstone is added
* For /, the delete call will simply downgrade to a no-op
h2. Solutions
This isn't something which can be defended against by tombstones. What we
should be doing is looking for when we need an empty dir is immediately
conclude that the dir is not-empty if there is any entry in the metastore which
is not a tombstone. This will let us bypass S3 too
Really we should be combining that getFileStatus with the real listing; there's
no need to duplicate the LIST calls
> ITestS3AContractRootDir failing on trunk: tombstones mislead about directory
> empty status
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-16380
> URL: https://issues.apache.org/jira/browse/HADOOP-16380
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
>
> I'm seeing reproducible failures of {{ITestS3AContractRootDir}} which look
> like consistency problems *even when S3Guard is enabled*.
> Suspicion: root dir listings are still inconsistent, due to the way we don't
> keep root entries in the table
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
