[
https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748718#comment-17748718
]
ASF GitHub Bot commented on HADOOP-18820:
-----------------------------------------
mukund-thakur commented on code in PR #5872:
URL: https://github.com/apache/hadoop/pull/5872#discussion_r1277902721
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/aws_sdk_upgrade.md:
##########
@@ -49,14 +137,67 @@ has been replaced by
[software.amazon.awssdk.auth.credentials.AwsCredentialsProv
changed.
The change in interface will mean that custom credential providers will need
to be updated to now
-implement `AwsCredentialsProvider` instead of `AWSCredentialProvider`.
+implement `software.amazon.awssdk.auth.credentials.AwsCredentialsProvider`
instead of
+`com.amazonaws.auth.AWSCredentialsProvider`.
+
+#### Original v1 `AWSCredentialsProvider` interface
+
+Note how the interface begins with the capitalized "AWS" acronym.
+The v2 interface starts with "Aws". This is a very subtle change
+for developers to spot.
+Compilers _will_ detect and report the type mismatch.
+
Review Comment:
this is repeated below.
##########
hadoop-project/pom.xml:
##########
@@ -1132,18 +1133,29 @@
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>${aws-java-sdk.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bundle</artifactId>
<version>${aws-java-sdk-v2.version}</version>
<exclusions>
<exclusion>
- <groupId>io.netty</groupId>
+ <groupId>*</groupId>
Review Comment:
We are excluding everything as this is a big shaded jar correct?
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3_select.md:
##########
@@ -663,6 +679,24 @@ to the `get()` call: do it.
## Troubleshooting
+### `NoClassDefFoundError: software/amazon/eventstream/MessageDecoder`
+
+Select operation failing with a missing evenstream class.
+
+```
+java.io.IOException: java.lang.NoClassDefFoundError:
software/amazon/eventstream/MessageDecoder
Review Comment:
the eventstream dependency has been added in the hadoop-aws pom. Do we still
get this error? I have seen this too.
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md:
##########
@@ -70,14 +70,45 @@ These are Hadoop filesystem client classes, found in the
`hadoop-aws` JAR.
An exception reporting this class as missing means that this JAR is not on
the classpath.
-### `ClassNotFoundException: com.amazonaws.services.s3.AmazonS3Client`
-(or other `com.amazonaws` class.)
+### `NoClassDefFoundError: software/amazon/awssdk/crt/s3/S3MetaRequest`
+
+The library `aws-crt.jar` is not on the classpath. Its classes
+are not in the AWS `bundle.jar` file, yet are needed for uploading
+and renaming objects.
+
+Fix: add.
+
+```
+java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError:
software/amazon/awssdk/crt/s3/S3MetaRequest
+at
software.amazon.awssdk.services.s3.internal.crt.S3MetaRequestPauseObservable.<init>(S3MetaRequestPauseObservable.java:33)
+at
software.amazon.awssdk.transfer.s3.internal.DefaultS3TransferManager.uploadFile(DefaultS3TransferManager.java:205)
+at org.apache.hadoop.fs.s3a.S3AFileSystem.putObject(S3AFileSystem.java:3064)
+at org.apache.hadoop.fs.s3a.S3AFileSystem.executePut(S3AFileSystem.java:4054)
+
+```
+### `ClassNotFoundException: software.amazon.awssdk.services.s3.S3Client`
-This means that the `aws-java-sdk-bundle.jar` JAR is not on the classpath:
+(or other `software.amazon` class.)
+
+This means that the AWS V2 SDK `bundle.jar` JAR is not on the classpath:
add it.
Review Comment:
these jars are now added in the hadoop-aws pom by default. Do we still need
this documentation?
> AWS SDK v2: make the v1 bridging support optional
> -------------------------------------------------
>
> Key: HADOOP-18820
> URL: https://issues.apache.org/jira/browse/HADOOP-18820
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> The AWS SDK v2 code includes the v1 sdk core for plugin support of
> * existing credential providers
> * delegation token binding
> I propose we break #2 and rely on those who have implemented to to upgrade.
> apart from all the needless changes the v2 SDK did to the api (why?) this is
> fairly straighforward
> for #1: fix through reflection, retaining a v1 sdk dependency at test time so
> we can verify that the binder works.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
