[
https://issues.apache.org/jira/browse/HADOOP-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873538#comment-16873538
]
Steve Loughran commented on HADOOP-16384:
-----------------------------------------
Update: I believe that the updating of deleted parent entries was causing the
failure which arose in HADOOP-15183
Imagine a sequence of
# file /a/b created (== a -> dir;, a/b -> file
# directory /a deleted (== a ->dir tombstone, a/b -> file tombstone)
# file /a is created (a -> file)
# file /a is deleted (== a -> file tombstone)
I believe the presence of the entry which is !dir is causing the problems.
The latest patch downgrades to a noop if the parent is deleted (which avoids a
put of the updated dir tombstone and the surfacing of problems there);
It doesn't downgrade to a nop-op if the parent entry is a file though. I'm
thinking of doing this, because we are pruning cruft here. If you debug a prune
run there is a lot of entries which are obsolete - we should only update the
parent dir if it exists, is a directory and isn't deleted. Next iteration.
Testing: in progress
> ITestS3AContractRootDir failing: inconsistent DDB tables
> --------------------------------------------------------
>
> Key: HADOOP-16384
> URL: https://issues.apache.org/jira/browse/HADOOP-16384
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: hwdev-ireland-new.csv
>
>
> HADOOP-15183 added detection and rejection of prune updates when the store is
> inconsistent (i.e. when it tries to update an entry twice in the same
> operation, the second time with one that is inconsistent with the first)
> Now that we can detect this, we should address it. We are lucky here in that
> my DDB table is currently inconsistent: prune is failing.
> Plan
> # new test to run in the sequential phase, which does a s3guard prune against
> the bucket used in tests
> # use this to identify/debug the issue
> # replicate the problem in the ITestDDBMetastore tests
> # decide what to do in this world. Tell the user to run fsck? skip?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
