[
https://issues.apache.org/jira/browse/HADOOP-19696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18028867#comment-18028867
]
ASF GitHub Bot commented on HADOOP-19696:
-----------------------------------------
steveloughran commented on PR #7980:
URL: https://github.com/apache/hadoop/pull/7980#issuecomment-3387174385
Latest build generates stack traces from gcs and obs filesystem incomplete
CP in service loader. Both need to move to core-default.xml *only* which is
faster anyway.
```
2025-10-09 20:06:44,452 [main] WARN fs.FileSystem
(FileSystem.java:loadFileSystems(3539)) - Cannot load filesystem
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
Provider org.apache.hadoop.fs.gs.GoogleHadoopFileSystem could not be
instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3525)
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3562)
at
org.apache.hadoop.fs.store.diag.StoreDiag.probeForFileSystemClass(StoreDiag.java:671)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:223)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:170)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:97)
at
org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:1255)
at
org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:1264)
at storediag.main(storediag.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:333)
at org.apache.hadoop.util.RunJar.main(RunJar.java:254)
Caused by: java.lang.NoClassDefFoundError: com/google/auth/Credentials
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 18 more
Caused by: java.lang.ClassNotFoundException: com.google.auth.Credentials
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 23 more
2025-10-09 20:06:44,456 [main] WARN fs.FileSystem
(FileSystem.java:loadFileSystems(3539)) - Cannot load filesystem
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
Provider org.apache.hadoop.fs.obs.OBSFileSystem could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3525)
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3562)
at
org.apache.hadoop.fs.store.diag.StoreDiag.probeForFileSystemClass(StoreDiag.java:671)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:223)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:170)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:97)
at
org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:1255)
at
org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:1264)
at storediag.main(storediag.java:25)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:333)
at org.apache.hadoop.util.RunJar.main(RunJar.java:254)
Caused by: java.lang.NoClassDefFoundError:
com/obs/services/exception/ObsException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 18 more
Caused by: java.lang.ClassNotFoundException:
com.obs.services.exception.ObsException
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 23 more
FileSystem for s3a:// is: org.apache.hadoop.fs.s3a.S3AFileSystem
Loaded from:
file:/Users/stevel/Projects/Releases/hadoop-3.5.0-SNAPSHOT/share/hadoop/common/lib/hadoop-aws-3.5.0-SNAPSHOT.jar
via sun.misc.Launcher$AppClassLoader@41906a77
```
> hadoop binary distribution to move cloud connectors to hadoop common/lib
> ------------------------------------------------------------------------
>
> Key: HADOOP-19696
> URL: https://issues.apache.org/jira/browse/HADOOP-19696
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, fs/gcs, fs/huawei, fs/s3
> Affects Versions: 3.4.2
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Place all the cloud connector hadoop-* artifacts and dependencies into
> hadoop/common/lib so that the stores can be directly accessed.
> * filesystem operations against abfs, s3a, gcs, etc don't need any effort
> setting things up.
> * Releases without the aws bundle.jar can be trivially updated by adding any
> version of the sdk libraries to the common/lib dir.
> This adds a lot more stuff into the distribution, so I'm doing the following
> design
> * all hadoop-* modules in common/lib
> * minimal dependencies for hadoop-azure and hadoop-gcs (once we get those
> right!)
> * hadoop-aws: everything except bundle.jar
> * other connectors: only included with explicit profiles.
> ASF releases will support azure out the box, the others once you add the
> dependencies. And anyone can build their own release with everything
> One concern here, we make hadoop-cloud-storage artifact incomplete at pulling
> in things when depended on. We may need a separate module for the distro
> setup.
> Noticed during this that the hadoop-tos component is shaded and includes
> stuff (httpclient5) that we need under control. Filed HADOOP-19708 and
> incorporating here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
