[
https://issues.apache.org/jira/browse/HADOOP-19740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18038331#comment-18038331
]
ASF GitHub Bot commented on HADOOP-19740:
-----------------------------------------
steveloughran commented on PR #8058:
URL: https://github.com/apache/hadoop/pull/8058#issuecomment-3532453285
also, I think we may want to add an enum of fallbacks
```
fs.s3a.endpoint.region.fallback = <ec2, sdk, central>
```
this would define the policy for calculating the region of a bucket which
isn't set explicitly, or inferred from the endpoint (vpce, aws, external).
today's behaviour is "central", which doesn't work when the network is
locked down in a VPC in a different region.
if you set ec2 or sdk, then unless the region is declared (or set to those
explicit region names) then the fallback will be whichever of the defaults.
There's one more thing to consider here: should we always do an EC2/IAM
probe before falling back to central?
pro: works great in EC2 with local regions
con: change in behavior when working with remote buckts.
I think the fallback option has better compatibility.
now, given we are doing ec2 region resolution in our own code, we have a
choice of what to do when the resolution fails. so what does
endopoint.region=ec2
endpoint.fallback=ec2
mean?
so maybe we only need two fallbacks
central: today
sdk: do the chain of env vars, sysprops and iam info.
> S3A: add explicit "sdk" and "ec2" regions for region resolution through SDK
> and EC2 IAM
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-19740
> URL: https://issues.apache.org/jira/browse/HADOOP-19740
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Affects Versions: 3.4.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Add explicit regions to hand off to the sdk
> * sdk: "use the sdk chain"
> * ec2: "we are in EC2, use the local region": use the iAM logic inside the
> SDK directly.
> empty string "" also hands off to the SDK; the warning will be removed
> also: if an endpoint is set and it is not parsed as a vpce endpoint, we will
> automatically add the endpoint name "external". This avoids the need to make
> up an external region when working with an endpoint.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
