[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727354#comment-15727354
]
Mingliang Liu commented on HADOOP-13449:
----------------------------------------
Thanks for your insightful comments, [~fabbri]. You summary looks very good to
me. Having follow-up JIRAs looks a good step as most of them can be addressed
separately === parallelly.
# For point 1, I'll change this in the new patch. I think by induction it is
good enough to include this optimization which should be minor change.
# For the example of pre-existing items, I was thinking the import phase may
have already loaded the S3 directory tree to the DDB table, e.g. by command
line tool. For better implementing {{isEmpty}}, I think working with
S3AFileStatus is a good idea. Steve also suggested we should query DDB lazily
only when we need the isEmpty information. Obviously we can't enable this
within DDBMetatdataStore itself.
# The integration tests you shared are very helpful. I'll have a look at the
{{ITestS3AFileOperationCost#testFakeDirectoryDeletion}} first. Will get back to
this the day after tomorrow.
# For the invalid creds in URI, the DDBClientFactory itself uses the same
{{createAWSCredentialProviderSet}} as S3ClientFactory does so it should honor
the creds in URI name. But after FS#initialization, S3AFS has stripped the
creds and returns the scheme://host only URI to create a MetadataStore. We can
refuse to support this case as it's very unsafe and deprecated. One approach
though is to pass the {{name}} URI which contains the creds to
{{S3Guard#getMetadataStore()}}.
{code:title=S3AFileSystem#initialize()}
- metadataStore = S3Guard.getMetadataStore(this);
+ metadataStore = S3Guard.getMetadataStore(this, name);
{code}
By the way, do you have suggestions about the failing {{testPutNew}} unit test?
My current idea is to override this test method in DDB and local so they can
have different behaviors; alternatively we can enforce the put API of metadata
store so both DDB and local implement that for newly put item, all its
ancestors are existent.
> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch,
> HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch,
> HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch,
> HADOOP-13449-HADOOP-13345.005.patch, HADOOP-13449-HADOOP-13345.006.patch,
> HADOOP-13449-HADOOP-13345.007.patch, HADOOP-13449-HADOOP-13345.008.patch,
> HADOOP-13449-HADOOP-13345.009.patch, HADOOP-13449-HADOOP-13345.010.patch,
> HADOOP-13449-HADOOP-13345.011.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
