Hi, thanks for the fast reply. The PR is here [1].
It works, if I exclude the client-api and client-api-runtime from being scanned in surefire, which is a hacky workaround for the actual issue. The hadoop-commons jar is a transient dependency of the minicluster, which is used for testing. Debugging the situation shows, that HttpServer2 is in the same package in hadoop-commons as well as in the client-api but with differences in methods / classes used, so depending on the classpath order the wrong class is loaded. Stacktraces are in the first GH Action run.here: [1]. A reproducer would be to check out Storm, go to storm-hdfs and remove the exclusion in [2] and run the tests in that module, which will fail due to a missing jetty server class (as the HTTPServer2 class is loaded from client-api instead of minicluster). Gruß & Thx Richard [1] https://github.com/apache/storm/pull/3637 [2] https://github.com/apache/storm/blob/e44f72767370d10a682446f8f36b75242040f675/external/storm-hdfs/pom.xml#L120 On 2024/04/11 21:29:13 Ayush Saxena wrote: > Hi Richard, > I am not able to decode the issue properly here, It would have been > better if you shared the PR or the failure trace as well. > QQ: Why are you having hadoop-common as an explicit dependency? Those > hadoop-common stuff should be there in hadoop-client-api > I quickly checked once on the 3.4.0 release and I think it does have them. > > ``` > ayushsaxena@ayushsaxena client % jar tf hadoop-client-api-3.4.0.jar | > grep org/apache/hadoop/fs/FileSystem.class > org/apache/hadoop/fs/FileSystem.class > `` > > You didn't mention which shaded classes are being reported as > missing... I think spark uses these client jars, you can use that as > an example, can grab pointers from here: [1] & [2] > > -Ayush > > [1] https://github.com/apache/spark/blob/master/pom.xml#L1361 > [2] https://issues.apache.org/jira/browse/SPARK-33212 > > On Thu, 11 Apr 2024 at 17:09, Richard Zowalla <[email protected]> wrote: > > > > Hi all, > > > > we are using "hadoop-minicluster" in Apache Storm to test our hdfs > > integration. > > > > Recently, we were cleaning up our dependencies and I noticed, that if I > > am adding > > > > <dependency> > > <groupId>org.apache.hadoop</groupId> > > <artifactId>hadoop-client-api</artifactId> > > <version>${hadoop.version}</version> > > </dependency> > > <dependency> > > <groupId>org.apache.hadoop</groupId> > > <artifactId>hadoop-client-runtime</artifactId> > > <version>${hadoop.version}</version> > > </dependency> > > > > and have > > <dependency> > > <groupId>org.apache.hadoop</groupId> > > <artifactId>hadoop-minicluster</artifactId> > > <version>${hadoop.version}</version> > > <scope>test</scope> > > </dependency> > > > > as a test dependency to setup a mini-cluster to test our storm-hdfs > > integration. > > > > This fails weirdly because of missing (shaded) classes as well as a > > class ambiquity with HttpServer2. > > > > It is present as a class inside of the "hadoop-client-api" and within > > "hadoop-common". > > > > Is this setup wrong or should we try something different here? > > > > Gruß > > Richard > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
