[
https://issues.apache.org/jira/browse/HADOOP-16396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878128#comment-16878128
]
Hudson commented on HADOOP-16396:
---------------------------------
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16858 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16858/])
HADOOP-16396. Allow authoritative mode on a subdirectory. (#1043) (github: rev
34747c373f40a33d36a2e21ecb33fa791ccd939f)
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitOperations.java
* (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java
* (add)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestAuthoritativePath.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java
> Allow authoritative mode on a subdirectory
> ------------------------------------------
>
> Key: HADOOP-16396
> URL: https://issues.apache.org/jira/browse/HADOOP-16396
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Sean Mackrory
> Assignee: Sean Mackrory
> Priority: Major
> Attachments: HADOOP-16396.001.patch, HADOOP-16396.002.patch,
> HADOOP-16396.003.patch
>
>
> Let's allow authoritative mode to be applied only to a subset of a bucket.
> This is coming primarily from a Hive warehousing use-case where Hive-managed
> tables can benefit from query planning, but can't speak for the rest of the
> bucket. This should be limited in scope and is not a general attempt to allow
> configuration on a per-path basis, as configuration is currently done on a
> per-process of a per-bucket basis.
> I propose a new property (we could overload
> fs.s3a.metadatastore.authoritative, but that seems likely to cause confusion
> somewhere). A string would be allowed that would then be qualified in the
> context of the FileSystem, and used to check if it is a prefix for a given
> path. If it is, we act as though authoritative mode is enabled. If not, we
> revert to the existing behavior of fs.s3a.metadatastore.authoritative (which
> in practice will probably be false, the default, if the new property is in
> use).
> Let's be clear about a few things:
> * Currently authoritative mode only short-cuts the process to avoid a
> round-trip to S3 if we know it is safe to do so. This means that even when
> authoritative mode is enabled for a bucket, if the metadata store does not
> have a complete (or "authoritative") current listing cached, authoritative
> mode still has no effect. This will still apply.
> * This will only apply to getFileStatus and listStatus, and internal calls to
> their internal counterparts. No other API is currently using authoritative
> mode to change behavior.
> * This will only apply to getFileStatus and listStatus calls INSIDE the
> configured prefix. If there is a recursvie listing on the parent of the
> configured prefix, no change in behavior will be observed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
