AwasthiSomesh opened a new issue, #11145:
URL: https://github.com/apache/iceberg/issues/11145
### Apache Iceberg version
1.6.1 (latest release)
### Query engine
Hive
### Please describe the bug 🐞
I am trying to configure AWS S3 configuration with the Hadoop and Hive setup.
But while trying so we are seeing following exception :
hadoop fs -ls s3a://somesh.qa.bucket/ -:
Fatal internal error java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem
not found
**To resolve this I have added hadoop-aws-3.3.6.jar and
aws-java-sdk-bundle-1.12.770.jar in Hadoop classpath.**
**i.e is under : /usr/local/hadoop/share/hadoop/common/lib**
**And S3 related configurations in the core-site.xml file: under
/usr/local/hadoop/etc/hadoop directory.**
<property>
<name>fs.default.name</name>
<value>s3a://infa.qa.bucket</value>
</property>
<property>
<name>fs.s3a.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
<property>
<name>fs.s3a.endpoint</name>
<value>s3.us-west-2.amazonaws.com</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<value>{Access _Key_Value}</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>{Secret_Key_Value}</value>
</property>
<property>
<name>fs.s3a.path.style.access</name>
<value>false</value>
</property>
Now when we try hadoop fs -ls s3a://infa.qa.bucket/
We are observing following exception :
2024-08-22 13:50:11,294 INFO impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
2024-08-22 13:50:11,376 INFO impl.MetricsSystemImpl: Scheduled Metric
snapshot period at 10 second(s).
2024-08-22 13:50:11,376 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system started
2024-08-22 13:50:11,434 WARN util.VersionInfoUtils: The AWS SDK for Java 1.x
entered maintenance mode starting July 31, 2024 and will reach end of support
on December 31, 2025. For more information, see
https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/
You can print where on the file system the AWS SDK for Java 1.x core runtime
is located by setting the AWS_JAVA_V1_PRINT_LOCATION environment variable or
aws.java.v1.printLocation system property to 'true'.
This message can be disabled by setting the
AWS_JAVA_V1_DISABLE_DEPRECATION_ANNOUNCEMENT environment variable or
aws.java.v1.disableDeprecationAnnouncement system property to 'true'.
The AWS SDK for Java 1.x is being used here:
at java.lang.Thread.getStackTrace(Thread.java:1564)
at
com.amazonaws.util.VersionInfoUtils.printDeprecationAnnouncement(VersionInfoUtils.java:81)
at com.amazonaws.util.VersionInfoUtils.<clinit>(VersionInfoUtils.java:59)
at
com.amazonaws.internal.EC2ResourceFetcher.<clinit>(EC2ResourceFetcher.java:44)
at
com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.<init>(InstanceMetadataServiceCredentialsFetcher.java:38)
at
com.amazonaws.auth.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:111)
at
com.amazonaws.auth.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:91)
at
com.amazonaws.auth.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:75)
at
com.amazonaws.auth.InstanceProfileCredentialsProvider.<clinit>(InstanceProfileCredentialsProvider.java:58)
at
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.initializeProvider(EC2ContainerCredentialsProviderWrapper.java:66)
at
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.<init>(EC2ContainerCredentialsProviderWrapper.java:55)
at
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider.<init>(IAMInstanceCredentialsProvider.java:53)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:727)
at org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList(S3AUtils.java:659)
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:585)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:959)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:586)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3712)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3663)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:347)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:264)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:247)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:105)
at org.apache.hadoop.fs.shell.Command.run(Command.java:191)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:97)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
ls: s3a://infa.qa.bucket/:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
2024-08-22 13:50:14,248 INFO impl.MetricsSystemImpl: Stopping
s3a-file-system metrics system...
2024-08-22 13:50:14,248 INFO impl.MetricsSystemImpl: s3a-file-system metrics
system stopped.
2024-08-22 13:50:14,248 INFO impl.MetricsSystemImpl: s3a-file-system metrics
syst
**Could you please help us to resolve this issue as soon as possible**
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [X] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
