steveloughran commented on code in PR #8628: URL: https://github.com/apache/hadoop/pull/8628#discussion_r3650148019
########## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/qualifying_a_new_sdk.md: ########## @@ -0,0 +1,1899 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL +NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in +RFC 2119. + +# Qualifying an AWS SDK + + +> The AWS SDKs and CLI are designed for usage with official AWS services. +> We may introduce and enable new features by default, such as these new default integrity protections, +> prior to them being supported or otherwise handled by third-party service implementations. + +That is a quote from an +[announcement of a somewhat incompatible change](https://github.com/aws/aws-sdk-java-v2/discussions/5802) +which shipped in v2.30.0 of the AWS SDK. + +It highlights the SDK team's point of view: their job is to work on the SDK to support AWS's own services. +Compatibility with third-party services is not their problem, and they do not test against such stores. + +This makes sense from their perspective: if someone implements their own S3 store, then it is +their task to make it compatible with AWS S3, even as that is a moving target with no public +formal API specification. + +The S3A connector is one of the most popular of S3 connectors used to connect +JVM-hosted big-data applications to AWS S3 *and to other S3-compatible stores*. +We do not have the luxury of saying "third-party stores are not our problem", so have to make +sure that our release works with all stores. + +And because of that broad adoption, we need to make sure that it works in +different deployment scenarios, with different configurations even within AWS. + +The task of qualifying an AWS SDK is a lot more than just incrementing a number in a maven POM file: +it is determining whether the SDK is safe to adopt, and, if safe, +identifying and making any changes in our code that are needed to migrate. + + +## Introduction + +The S3A connector is utterly dependent upon the AWS SDK; even a minor change can have serious consequences. +That is: changing a single number in a maven file can bring new features and needed bug fixes. +It can also cause a lot of damage, albeit unintentionally. + +Some example regressions encountered previously include: +* The SDK printing a warning message telling developers off every time a specific object in the SDK is instantiated + This breaks all tests which look for specific output strings and runs a risk of generating support calls asking "why is my application telling me off?" +* A change in the semantics of calling `abort()` on a stream. + This was a valid design decision — however, it was unexpected. + And again the warning message printed every time the stream was closed prematurely flooded application logs. +* Instabilities in the shading of third-party libraries (slf4j, etc.), with consequences such as the inability to enable any form of logging. +* The shaded library continuing to declare dependencies which are redundant due to the shading. Review Comment: pretty much every upgrade was a pain point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
