[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17771078#comment-17771078
]
ASF GitHub Bot commented on HADOOP-18889:
-----------------------------------------
steveloughran opened a new pull request, #6141:
URL: https://github.com/apache/hadoop/pull/6141
1. handle getBucketLocation() unsupported
2. append / to marker tool searches if the first call fails.
3. downgrade change control errors a bit better. TODO: useful error text/docs
4. handle unknown storage classes.
5. Docs on how to bind to and test third party stores, including google gcs.
Not doing the region logic; this is other test failures.
Exception translation/resilience
* new AWSUnsupportedFeatureException ex for unsupported/unavailable errors
* handle 501 method unimplemented the same
* all ex > 500 mapped to the AWSStatus500Exception
* precondition errors handled a bit better
* GCS throttle exception also recognised.
* New class ErrorHandling to move our error handling logic too, initially
somewhere for new stuff
* ErrorHandling.maybeTranslateNetworkException() scans down an exception
chain and if the inner one is of a known type
- UnknownHostException
- NoRouteToHostException
- ConnectException
Then a new exception of that specific type is created, with the top level ex
its cause.
This is done to retain the whole stack chain.
* reduce the number of retries within the AWS SDK
* and those of our S3ARetryPolicy
The reduction in retries is because its clear when you try to create a
bucket
which doesn't resolve that the time for even an UnknownHostException to
eventually fail over 90s, which then hit the s3a retry code.
- Reducing the sdk retries means these escalate to our code better.
- Cutting back on our own retries makes it a bit more responsive for most
real deployments.
- maybeTranslateNetworkException() and s3a retry policy means that
UnknownHostException is recognised and fails fast.
Lots of changes to the test code:
* more switches to turn off test suites (acls, v1list, ...)
* more resilience to different test failures
* pull up skip() checks into config create phase, rather than waiting for fs
to be instantiated (faster; keeps the fs cache clean)
* tuning for new exceptions
### How was this patch tested?
* AWS S3 london: no regressions observed.
* a third party endpoint: showed troublespots, especially how handling of
UnknownHostException needed to be radically improved.
Gave me the stack traces for the docs too
Docs cover google gcs binding; I've not tested that yet, but the settings
came from
a colleague who has connected the v1 sdk to it.
A full test run against GCS should be something we require here as it
supports
a different subset of the s3 apis (no bulk delete, v1 list only...)
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [X] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> S3A: V2 SDK client does not work with third-party store
> -------------------------------------------------------
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
