[
https://issues.apache.org/jira/browse/HADOOP-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042997#comment-16042997
]
Aaron Fabbri commented on HADOOP-14488:
---------------------------------------
One thing that comes to mind is the bug I noticed in the delete logic
[~mackrorysd] added to {{InconsistentAmazonS3Client}}:
{code}
private ObjectListing restoreListObjects(ListObjectsRequest request,
ObjectListing rawListing) {
List<S3ObjectSummary> outputList = rawListing.getObjectSummaries();
List<String> outputPrefixes = rawListing.getCommonPrefixes();
for (String key : new HashSet<>(delayedDeletes.keySet())) {
Delete delete = delayedDeletes.get(key);
if (isKeyDelayed(delete.time(), key)) {
// TODO this works fine for flat directories but:
// if you have a delayed key /a/b/c/d and you are listing /a/b,
// this incorrectly will add /a/b/c/d to the listing for b
if (key.startsWith(request.getPrefix())) {
{code}
Seems like that could affect the set of files moved during rename. I'll try to
get a patch out today, or at least send you a snippet to retest with.
> s3guard listStatus fails after renaming file into directory
> -----------------------------------------------------------
>
> Key: HADOOP-14488
> URL: https://issues.apache.org/jira/browse/HADOOP-14488
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Steve Loughran
> Priority: Blocker
> Attachments: HADOOP-14488-HADOOP-13345-001.patch, output.txt
>
>
> Running scala integration test with inconsistent s3 client & local DDB enabled
> {code}
> fs.rename("work/task-00/part-00", work)
> fs.listStatus(work)
> {code}
> The list status work fails with a message about the childStatus not being a
> child of the parent.
> Hypothesis: rename isn't updating the child path entry
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
