[
https://issues.apache.org/jira/browse/HADOOP-18683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913385#comment-17913385
]
Steve Loughran commented on HADOOP-18683:
-----------------------------------------
I see what you mean. there'd be a vendor/version option which would be used as
per bucket settings.
we are getting into the complicated world of configuration management, and as
someone who has done research in that area (
https://dl.acm.org/doi/10.1145/1496909.1496915 ) , i'm reluctant to make it
more complex.
today we have the following levels of config
* constants.java
* core-default.xml
* fs.s3a.*
* fs.s3a.bucket.*
each one can overwrite the other, no cross-references, no removal, and no
sharing of attributes across buckets X Y Z
a full config language would let you add templates
"fs.s3a.store.netapp."...
and then you'd declare a bucket as an extension of that store.
would also let you have stores like "usw-1" and "london" which would give the
london and us-west settings.
...of course, you now see those are action regions within a store class. At
which point you have just entered the world of template based component
definition languages.
I'm not saying these are bad (*hey, i got into hadoop by using one to deploy
hadoop*), but that it becomes complex fast.
you want it I would want that notion of fs.s3a.product for product, then
something for each deployment (usw-1, steve's minio docker image, ozone site).
I think that should be enough.
> S3A Add new store vendor config option
> --------------------------------------
>
> Key: HADOOP-18683
> URL: https://issues.apache.org/jira/browse/HADOOP-18683
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.5
> Reporter: Ahmar Suhail
> Priority: Minor
>
> Add in a new fs.s3a.store.vendor config, where users can specify the storage
> vendor they are using (eg: aws, netapp, minio).
> This will allow us to configure S3A correctly per vendor. For example, if the
> vendor is not AWS, you probably want to use ListObjectsV1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
