I was able to run the Maven build (~900 modules) with the profiler.
Used `Apache Maven 4.0.0-rc-3` and the simplest possible command `mvn
initialize` which demonstrates reproducible degradation, the build is using
6 worker threads.
Maven 3 runs this command in 5 sec, Maven 4 - 6 minutes (no mistake).
YourKit highlighted as the most hot
`org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(RepositorySystemSession,
CollectRequest)` method.
I collected several stacktraces during the build and found this common part
of the stacktrace
```
at
org.eclipse.aether.util.graph.manager.AbstractDependencyManager.deriveChildManager
at
org.eclipse.aether.internal.impl.collect.bf.BfDependencyCollector.doRecurse(BfDependencyCollector.java:361)
at
org.eclipse.aether.internal.impl.collect.bf.BfDependencyCollector.processDependency(BfDependencyCollector.java:320)
at
org.eclipse.aether.internal.impl.collect.bf.BfDependencyCollector.doCollectDependencies(BfDependencyCollector.java:202)
at
org.eclipse.aether.internal.impl.collect.DependencyCollectorDelegate.collectDependencies(DependencyCollectorDelegate.java:222)
at
org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:79)
at
org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:241)
at
org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:157)
at
org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:260)
at
org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectArtifacts(LifecycleDependencyResolver.java:208)
- locked <xxxxxxxxxxxx> (a
org.apache.maven.project.artifact.DefaultProjectArtifactsCache$CacheKey)
at
org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:128)
at
org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:368)
at
org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:307)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:214)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:179)
at
org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:168)
at
org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:165)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:110)
at
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder.lambda$createBuildCallable$1(MultiThreadedBuilder.java:191)
at
org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$$Lambda$1189/0x0000000800526240.call(Unknown
Source)
at java.util.concurrent.FutureTask.run(java.base@17.0.14
/FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.14
/Executors.java:539)
at java.util.concurrent.FutureTask.run(java.base@17.0.14
/FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.14
/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.14
/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.14/Thread.java:840)
```
As you see, the top method in the snippet is `deriveChildManager` and in
the captured stack traces there are different combinations of stacktraces
above it. But the below part is always the same.
In this gist you can find 4 real stacktraces of the same Maven process
captured during the build when it was already slow
https://gist.github.com/seregamorph/bace2d5087ba195279583704bb41a5c4
Things I'd like to focus attention is: it looks like it's degrading
smoothly, maybe the degradation depends on the depth of dependency tree,
maybe on the number of transitive dependencies (correlating values). But in
the very beginning (first 400 modules) it goes quite fast. It does not fail
with OOM, so probably the problem is around some nested iterations.
Let me know what else I can do (including patching maven to have more
sensors) as I'm not sure how to provide a reproducible project model (it is
private).
I tried the gradle performance testing tool
https://gist.github.com/melix/c09f6d8d27b0005302cc3317c9c9be05 , but the
project that it generates does not reproduce the problem (probably because
the dependency tree of the huge multi-module project is almost flat).
On Fri, May 30, 2025 at 1:59 PM Xeno Amess <xenoam...@gmail.com> wrote:
> @Guillaume Nodet
> [MNG-8510] Avoid duplicate Utils classes (#2173)
>
> https://github.com/JetBrains/intellij-community/blob/4054fd36584fb2bbb1ed983d7d36cefca040c2cf/plugins/maven/maven40-server-impl/src/com/intellij/maven/server/m40/Maven40ServerEmbedderImpl.java#L74
>
> yes we can release the version 4.0.0, but after the releasing things like
> this class/file-names would not be changed easily (of course and other
> bigger things), I don't know if people be prepared for a...stablization
>
> Xeno Amess <xenoam...@gmail.com> 于2025年5月29日周四 17:21写道：
>
> > all running with skipTests
> >
> > log4j both success, mvn3 2:00 mvn4 1:50
> > dubbo both success, mvn3 3:08 mvn4 3:19
> >
> > hadoop failed due to network reason(for yarn frontend, well considering
> > about where I'm living it is normal...)
> >
> > for structs 2, 90%+ time is wasted on antrun for docs generation in
> > struts2-assembly. both mvn3 and mvn4 be very very slow there.
> >
> >
> > jetty seems has profile-plugin leak when using mvn4.
> > maven-assembly-plugin:3.7.1 wrongly invoking a bom sub-project, which is
> > only defined in a profile (named `config`) which shall not be on.
> >
> > quarkus cannot build with mvn4, maven-invoker-plugin failed
> >
> > Xeno Amess <xenoam...@gmail.com> 于2025年5月29日周四 14:52写道：
> >
> >> nope. neither can the github master latest.
> >>
> >> Xeno Amess <xenoam...@gmail.com> 于2025年5月29日周四 14:40写道：
> >>
> >>> I used apache-maven-4.0.0-rc-3 btw. I would have a try at the latest
> >>> github code version...
> >>>
> >>> Xeno Amess <xenoam...@gmail.com> 于2025年5月29日周四 14:39写道：
> >>>
> >>>> jetty is a good example.
> >>>> mvn3 can build but mvn4 cannot.
> >>>>
> >>>> maven-assembly-plugin:3.7.1 wrongly invoking a bom sub-project
> >>>>
> >>>> [INFO] ---------------< org.eclipse.jetty.ee10:jetty-ee10-bom
> >>>> >----------------
> >>>> [INFO] Building EE10 :: BOM 12.0.22-SNAPSHOT
> >>>> [90/354]
> >>>> [INFO] from jetty-ee10/jetty-ee10-bom/pom.xml
> >>>> [INFO] --------------------------------[ pom
> >>>> ]---------------------------------
> >>>> [INFO] Going to calculate checksum for project
> >>>> [groupId=org.eclipse.jetty.ee10, artifactId=jetty-ee10-bom,
> >>>> version=12.0.22-SNAPSHOT]
> >>>> [INFO] Project inputs calculated in 0 ms. XXMM checksum
> >>>> [7b7b6120251dfd1d] calculated in 0 ms.
> >>>> [INFO] Attempting to restore project
> >>>> org.eclipse.jetty.ee10:jetty-ee10-bom from build cache
> >>>> [INFO] Local build found by checksum 7b7b6120251dfd1d
> >>>> [INFO] Found cached build, restoring
> >>>> org.eclipse.jetty.ee10:jetty-ee10-bom from cache by checksum
> >>>> 7b7b6120251dfd1d
> >>>> [WARNING] Cached build doesn't contains all requested plugin
> executions
> >>>> (missing:
> >>>> [org.apache.maven.plugins:maven-resources-plugin:3.3.1:copy-resources
> >>>> {execution: copy-resources},
> >>>> org.apache.maven.plugins:maven-assembly-plugin:3.7.1:single
> {execution:
> >>>> config-assembly}]), cannot restore
> >>>> [INFO] Local build was not found by checksum 7b7b6120251dfd1d for
> >>>> org.eclipse.jetty.ee10:jetty-ee10-bom
> >>>> [INFO]
> >>>> [INFO] --- spotless:2.44.4:check (default) @ jetty-ee10-bom ---
> >>>> [INFO] Index file does not exist. Fallback to an empty index
> >>>> [INFO] Sorting file /tmp/pom11239457262347372795.xml
> >>>> [INFO] Pom file is already sorted, exiting
> >>>> [INFO] Spotless.Pom is keeping 1 files clean - 0 needs changes to be
> >>>> clean, 1 were already clean, 0 were skipped because caching
> determined they
> >>>> were already clean
> >>>> [INFO]
> >>>> [INFO] --- checkstyle:3.6.0:check (checkstyle-check) @ jetty-ee10-bom
> >>>> ---
> >>>> [INFO] 开始检查……
> >>>> 检查完成。
> >>>> [INFO] You have 0 Checkstyle violations.
> >>>> [INFO]
> >>>> [INFO] --- enforcer:3.5.0:enforce (ban-javax-servlet-api) @
> >>>> jetty-ee10-bom ---
> >>>> [INFO] Rule 0:
> >>>> org.apache.maven.enforcer.rules.dependency.BannedDependencies passed
> >>>> [INFO]
> >>>> [INFO] --- enforcer:3.5.0:enforce (enforce-java) @ jetty-ee10-bom ---
> >>>> [INFO] Rule 3:
> >>>>
> org.eclipse.jetty.toolchain.enforcer.rules.RequireOsgiCompatibleVersionRule(versionOsgiRule)
> >>>> passed
> >>>> [INFO] Rule 4:
> >>>> org.apache.maven.enforcer.rules.dependency.RequireUpperBoundDeps
> passed
> >>>> [INFO]
> >>>> [INFO] --- build-helper:3.6.0:parse-version (set-osgi-version) @
> >>>> jetty-ee10-bom ---
> >>>> [INFO]
> >>>> [INFO] --- jacoco:0.8.13:prepare-agent (jacoco-initialize) @
> >>>> jetty-ee10-bom ---
> >>>> [INFO] argLine set to
> >>>>
> -javaagent:/home/xenoamess/.m2/repository/org/jacoco/org.jacoco.agent/0.8.13/org.jacoco.agent-0.8.13-runtime.jar=destfile=/home/xenoamess/workspace/jetty.project/jetty-ee10/jetty-ee10-bom/target/jacoco.exec,excludes=**/org/eclipse/jetty/ant/**:*/org/eclipse/jetty/maven/its/**:**/org/eclipse/jetty/embedded/**:**/org/eclipse/jetty/asyncrest/**:**/org/eclipse/jetty/demo/**:**/org/eclipse/jetty/gcloud/**:**/org/eclipse/jetty/infinispan/**:**/org/eclipse/jetty/osgi/**:**/org/eclipse/jetty/spring/**:**/org/eclipse/jetty/http/spi/**:**/org/eclipse/jetty/tests/**:**/org/eclipse/jetty/test/**
> >>>> [INFO]
> >>>> [INFO] --- jacoco:0.8.13:prepare-agent (jacoco-setup-m-invoker-p) @
> >>>> jetty-ee10-bom ---
> >>>> [INFO] invoker.mavenOpts set to
> >>>>
> -javaagent:/home/xenoamess/.m2/repository/org/jacoco/org.jacoco.agent/0.8.13/org.jacoco.agent-0.8.13-runtime.jar=destfile=/home/xenoamess/workspace/jetty.project/jetty-ee10/jetty-ee10-bom/target/jacoco.exec,excludes=**/org/eclipse/jetty/ant/**:*/org/eclipse/jetty/maven/its/**:**/org/eclipse/jetty/embedded/**:**/org/eclipse/jetty/asyncrest/**:**/org/eclipse/jetty/demo/**:**/org/eclipse/jetty/gcloud/**:**/org/eclipse/jetty/infinispan/**:**/org/eclipse/jetty/osgi/**:**/org/eclipse/jetty/spring/**:**/org/eclipse/jetty/http/spi/**:**/org/eclipse/jetty/tests/**:**/org/eclipse/jetty/test/**
> >>>> [INFO]
> >>>> [INFO] --- remote-resources:3.2.0:process (copy-shared-resources) @
> >>>> jetty-ee10-bom ---
> >>>> [INFO] Preparing remote bundle
> >>>> org.eclipse.jetty:build-resources:12.0.22-SNAPSHOT
> >>>> [INFO] Copying 0 resource from 1 bundle.
> >>>> [INFO]
> >>>> [INFO] --- jetty-version:2.7:attach-version-text (attach-version) @
> >>>> jetty-ee10-bom ---
> >>>> [INFO]
> >>>> [INFO] --- resources:3.3.1:copy-resources (copy-resources) @
> >>>> jetty-ee10-bom ---
> >>>> [INFO] skip non existing resourceDirectory
> >>>>
> /home/xenoamess/workspace/jetty.project/jetty-ee10/jetty-ee10-bom/src/main/config
> >>>> [INFO]
> >>>> [INFO] --- flatten:1.7.0:flatten (flatten) @ jetty-ee10-bom ---
> >>>> [INFO] Generating flattened POM of project
> >>>> org.eclipse.jetty.ee10:jetty-ee10-bom:pom:12.0.22-SNAPSHOT...
> >>>> [INFO]
> >>>> [INFO] --- bundle:5.1.9:manifest (default) @ jetty-ee10-bom ---
> >>>> [WARNING] Ignoring project type pom - supportedProjectTypes = [jar,
> >>>> maven-plugin]
> >>>> [INFO]
> >>>> [INFO] --- source:3.3.1:jar-no-fork (attach-sources) @ jetty-ee10-bom
> >>>> ---
> >>>> [INFO]
> >>>> [INFO] --- jacoco:0.8.13:report (jacoco-site) @ jetty-ee10-bom ---
> >>>> [INFO] Skipping JaCoCo execution due to missing execution data file.
> >>>> [INFO]
> >>>> [INFO] --- assembly:3.7.1:single (config-assembly) @ jetty-ee10-bom
> ---
> >>>>
> >>>>
> >>>>
> >>>> [INFO] Mimir session closed (RETRIEVED=0 CACHED=0)
> >>>> [ERROR] Failed to execute goal
> >>>> org.apache.maven.plugins:maven-assembly-plugin:3.7.1:single
> >>>> (config-assembly) on project jetty-ee10-bom: Failed to create
> assembly:
> >>>> Error creating assembly archive config: archive cannot be empty ->
> [Help 1]
> >>>> [ERROR]
> >>>>
> >>>>
> >>>> checked it did not invoke in mvn3 building log...
> >>>>
> >>>> Sergey Chernov <serega.mo...@gmail.com> 于2025年5月29日周四 00:17写道：
> >>>>
> >>>>> Hey, Delany. I will look into this.
> >>>>> Also to avoid confusion let's continue this discussion in the proper
> >>>>> thread
> >>>>> as it's not related to maven 4 release
> >>>>>
> >>>>> On Wed, May 28, 2025 at 4:22 PM Delany <delany.middle...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> > Hi Sergei,
> >>>>> >
> >>>>> > I'm running with this extension going forward. I don't use the
> build
> >>>>> cache
> >>>>> > (yet) so that's not an issue.
> >>>>> > When trying to promote unit tests to a company that's resistant its
> >>>>> helpful
> >>>>> > to be able to say the tests won't slow down the build.
> >>>>> >
> >>>>> > We have one huge module though (project we're calling it now) with
> >>>>> > extensive tests, and its a bottleneck for all that follow.
> >>>>> > Typically we had to skip tests, but now we don't have to. Build
> time
> >>>>> was
> >>>>> > cut in half.
> >>>>> >
> >>>>> > I did notice jacoco coverage was disabled as I bound the
> >>>>> > jacoco-maven-plugin report goal to the prepare-package phase.
> >>>>> > As the documentation says, the phases have been switched around.
> >>>>> > But can you make it work with the
> buildplan-maven-plugin:list-phases
> >>>>> goal
> >>>>> > so I can actually see which phases runs when, or is that up to this
> >>>>> plugin
> >>>>> > to figure out?
> >>>>> >
> >>>>> > Kind regards,
> >>>>> > Delany
> >>>>> >
> >>>>> >
> >>>>> > On Wed, 28 May 2025 at 14:55, Sergey Chernov <
> serega.mo...@gmail.com
> >>>>> >
> >>>>> > wrote:
> >>>>> >
> >>>>> > > Guillaume, that's awesome. It's good that there is no rush with
> it.
> >>>>> > > Once I'll have insights, I'll share them.
> >>>>> > >
> >>>>> > > Xeno, the project I'm checking is private unfortunately. Also
> this
> >>>>> can be
> >>>>> > > quite challenging to find a good Maven multi-module project as a
> >>>>> > reference
> >>>>> > > for performance testing and compatibility validations, as many of
> >>>>> big
> >>>>> > > projects e.g. have flaky tests and other issues.
> >>>>> > > Some of the biggest that I've seen:
> >>>>> > > * https://github.com/quarkusio/quarkus (has lots of
> >>>>> customizations in
> >>>>> > > .mvn/extensions.xml)
> >>>>> > > * https://github.com/apache/dubbo
> >>>>> > > * https://github.com/apache/hadoop
> >>>>> > > * https://github.com/apache/struts
> >>>>> > > * https://github.com/vaadin/flow and
> >>>>> https://github.com/vaadin/framework
> >>>>> > >
> >>>>> > >
> >>>>> > > On Wed, May 28, 2025 at 2:16 PM Guillaume Nodet <
> gno...@apache.org
> >>>>> >
> >>>>> > wrote:
> >>>>> > >
> >>>>> > > > After considering the concerns about API stability, plugin
> >>>>> > > > compatibility, and the performance issues reported by Sergey
> (4x
> >>>>> > > > slower in rc3), I agree with Sylwester, Benjamin, and Elliotte
> >>>>> that we
> >>>>> > > > should opt for an rc4 to address these issues. My proposal is
> to
> >>>>> > > > release a final release candidate (rc4) soon, aiming for a
> >>>>> General
> >>>>> > > > Availability (GA) release in September. Post rc4, I suggest
> >>>>> creating a
> >>>>> > > > 4.0.x branch to start work on 4.1.0. For 4.1.0, we could
> >>>>> introduce API
> >>>>> > > > changes like records and sequenced collections, as well as new
> >>>>> > > > features such as mixins, cascading profiles, server aliases,
> JDK
> >>>>> 21
> >>>>> > > > (still under discussion) and more....
> >>>>> > > >
> >>>>> > > > The recent API changes for JPMS, as Martin noted, only affect
> the
> >>>>> > > > experimental API, so they shouldn’t block the release. On
> >>>>> Matthias’s
> >>>>> > > > suggestion, enabling GitHub Issues across all repositories
> >>>>> before the
> >>>>> > > > GA seems like a good idea to improve user feedback. Let’s focus
> >>>>> on
> >>>>> > > > stabilizing rc4 and addressing performance bottlenecks,
> possibly
> >>>>> > > > exploring Martin’s O(N²) concerns with profiling, as Sergey
> >>>>> offered.
> >>>>> > > >
> >>>>> > > > What do you think about this plan?
> >>>>> > > >
> >>>>> > > > On a side note, I created a few months ago a project to check
> >>>>> Maven 4
> >>>>> > > > compatibility with all Apache projects using Maven. The
> results
> >>>>> for
> >>>>> > > > RC-3 can be found at
> >>>>> > > > https://github.com/gnodet/maven4-testing/issues/2812. Most of
> >>>>> the
> >>>>> > > > errors are caused by the use of incompatible plugins (which
> >>>>> needs to
> >>>>> > > > be updated) or incorrect POMs (as Maven 4 is more strict about
> a
> >>>>> few
> >>>>> > > > things). What would be nice is a tool to integrate in the
> shell
> >>>>> that
> >>>>> > > > would fix those known incompatibilities, so that users would
> >>>>> have an
> >>>>> > > > easier migration path. And we could apply it to this project
> to
> >>>>> make
> >>>>> > > > more green boxes.
> >>>>> > > >
> >>>>> > > > Le mer. 21 mai 2025 à 08:11, Guillaume Nodet <
> gno...@apache.org>
> >>>>> a
> >>>>> > > écrit :
> >>>>> > > > >
> >>>>> > > > > Hey Maven Devs,
> >>>>> > > > >
> >>>>> > > > > We're gearing up to release a new version from the master
> >>>>> branch. I'm
> >>>>> > > > thinking we should go for 4.0.0 instead of rc-4. What do you
> all
> >>>>> think?
> >>>>> > > Any
> >>>>> > > > feedback or ideas on the versioning or release plan? Let’s hear
> >>>>> it!
> >>>>> > > > >
> >>>>> > > > > Cheers,
> >>>>> > > > >
> >>>>> > > > > Guillaume
> >>>>> > > > >
> >>>>> > > > >
> >>>>> > > >
> >>>>> > > >
> >>>>> > > > --
> >>>>> > > > ------------------------
> >>>>> > > > Guillaume Nodet
> >>>>> > > >
> >>>>> > > >
> >>>>> ---------------------------------------------------------------------
> >>>>> > > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
> >>>>> > > > For additional commands, e-mail: dev-h...@maven.apache.org
> >>>>> > > >
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
> >>>>
>
