[
https://issues.apache.org/jira/browse/HADOOP-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109478#comment-16109478
]
Steve Loughran commented on HADOOP-13757:
-----------------------------------------
Rajesh was suggesting that we can skip any check in fs.init, instead relying on
the first list/create/open to force an implicit check for the bucket existing.
If you are dynamically created FS instances for queries (Hive LLAP, Spark) then
the extra https HEAD / check becomes an overhead.
but it will fail at a different place, and potentially get reported
differently. Assuming that s3guard will be setting up a link with DDB during FS
init, I'm not going to worry about it until after s3guard is merged in, and
then only if its shown to be a perf problem in s3guard-enabled buckets. Why? If
you care about performance and are working with AWS S3, you'll need s3guard
turned on.
> Remove verifyBuckets overhead in S3AFileSystem::initialize()
> ------------------------------------------------------------
>
> Key: HADOOP-13757
> URL: https://issues.apache.org/jira/browse/HADOOP-13757
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Rajesh Balamohan
> Priority: Minor
>
> {{S3AFileSystem.initialize()}} invokes verifyBuckets, but in case the bucket
> does not exist and gets a 403 error message, it ends up returning {{true}}
> for {{s3.doesBucketExists(bucketName}}. In that aspect, verifyBuckets() is
> an unnecessary call during initialization.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
