[
https://issues.apache.org/jira/browse/HADOOP-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HADOOP-13760:
-----------------------------------
Attachment: HADOOP-13760-HADOOP-13345.005.patch
Thanks for the analysis, [~fabbri]. The really helpful thing is remembering
that not consistently listing everything for a rename did exist before. I
changed LeafNodesIterator to only return things it KNOWS are leaf nodes, and
relies on S3 to list everything else. So brand new empty directories may still
get missed in a rename, but files (even brand new ones), older directories, and
deleted stuff gets handled correctly. And it shouldn't perform too badly like
some solutions I had considered. Strictly an improvement. So I'm pretty happy
with this version of the patch - except still having some test issues...
I've been seeing ITestS3AEncryptionSSE.testEncryptionOverRename and
ITestS3AContractRootDir.testRecursiveRootListing fail like I was earlier. They
fail when running all tests (not even in parallel) but not when run
individually. So it seems like an unrelated test issue, but doesn't seem to
happen without my change. Also seeing
ITestS3AEmptyDirectory.testDirectoryBecomesEmpty and a few tests in
ITestS3AFileContextURI fail, but the circumstances are also inconsistent and
they don't seem related to my changes. Will investigate more...
> S3Guard: add delete tracking
> ----------------------------
>
> Key: HADOOP-13760
> URL: https://issues.apache.org/jira/browse/HADOOP-13760
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Aaron Fabbri
> Assignee: Sean Mackrory
> Attachments: HADOOP-13760-HADOOP-13345.001.patch,
> HADOOP-13760-HADOOP-13345.002.patch, HADOOP-13760-HADOOP-13345.003.patch,
> HADOOP-13760-HADOOP-13345.004.patch, HADOOP-13760-HADOOP-13345.005.patch
>
>
> Following the S3AFileSystem integration patch in HADOOP-13651, we need to add
> delete tracking.
> Current behavior on delete is to remove the metadata from the MetadataStore.
> To make deletes consistent, we need to add a {{isDeleted}} flag to
> {{PathMetadata}} and check it when returning results from functions like
> {{getFileStatus()}} and {{listStatus()}}. In HADOOP-13651, I added TODO
> comments in most of the places these new conditions are needed. The work
> does not look too bad.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
