steveloughran commented on PR #5421: URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1484983979
`Endpoint is not set || Endpoint is set && ends in amazonaws.com || ARN is set` is roughly what storediag does: https://github.com/steveloughran/cloudstore/blob/trunk/src/main/java/org/apache/hadoop/fs/store/diag/S3ADiagnosticsInfo.java#L672 it also looks for amazonaws.cn; not sure about where else it should probe. having a flag "is.aws" would be good as a single switch to move to all 3rd party stuff, but the problem there is that features there may change over time too; fs.s3a.store.vendor would let you have a table of providers (aws, ozone, hitachi, amplidata, minio, netapp) and choose the right settings w.r.t path vs hostname, checksums, .... we'd have the vendor settings from properties too, e.g fs.s3a.vendor.ozone.change.detection.mode = none so they could be overridden in core-site/per-bucket that could actually simplify a lot of our internal "doesn't work with vendor XYZ" where problems 1 and 2 are 1. you forgot to set the endpoint and aws are rejecting you/your bucket 2. use path resolution -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
