[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593227#comment-15593227
]
Aaron Fabbri commented on HADOOP-13449:
---------------------------------------
I'm curious what you guys are doing about path normalization.. For
LocalMetadataStore, I'm using Path as the hashtable key. We already agreed to
require all incoming paths to be absolute, but this is not quite enough to
ensure consistency. Some callers may hand in an absolute path without scheme
and bucket (a.k.a URI host) qualification, and others may hand in a path to the
same object with both of those things.
I first tried stripping scheme and bucket away (via
{Path#getPathWithoutSchemeAndAuthority()}). This causes a problem, though, if
multiple FileSystem instances share a metadata store singleton. In this case,
we actually need the host portion of the URI to distinguish between the
different buckets that may be sharing a single MetadataStore.
So, now I'm thinking that MetadataStore implementations, and DirListingMetadata
API, need to do something like fs.qualify(path) for incoming paths.
Questions
1. Does this sound right (always using qualified paths as lookup keys)?
2. Is there something less heavyweight we should do besides fs.qualify(path)?
I'd like common case to be fast.
3. Should MetadataStore impls. and DirListingMetadata be responsible for this
path conversion, or should we assert that the caller already did it?
3.b. If the former, are we ok with DirListingMetadata keeping a reference to an
"owning" FileSystem instance via its constructor?
> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.wip.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
