Hi Sergey, Your maven-turbo-builder sounds very similar to the concurrent builder in Maven 4, so I’d like to share some insights for comparison.
The Maven 4 concurrent builder decomposes projects at the phase level, with phases ordered based on fine-grained constraints. Unlike the traditional builder, these constraints allow more flexibility. For instance, if project A depends on project B at test scope, both can start building concurrently, but A’s test phase will wait until B completes its necessary phases (e.g., resource preparation, compilation, and class post-processing) to ensure the required artifacts are available. Downloads are deferred to avoid pausing the build unnecessarily, which aligns with optimizing build performance. To maintain compatibility with existing Maven behavior, the standard phase order is preserved. However, with the concurrent builder, phases form a dependency tree rather than a linear list, as detailed in the Maven 4 lifecycle documentation [1]. This structure allows phases like package to run independently of test, and intra-project dependencies can rely on a ready phase, which precedes package. This enables earlier scheduling of downstream tasks. Regarding your suggestion about upfront downloads, I agree it could be beneficial to initiate downloads early with a strategy to prioritize artifacts as needed. However, since dependency downloads typically occur only once per project, their impact on overall build time may be less significant compared to phase scheduling optimizations. The maven-turbo-builder’s approach—scheduling downstream builds after the upstream package phase and reordering test before package—is an interesting optimization. It seems to align with the concurrent builder’s philosophy but takes a more aggressive stance by breaking from traditional phase ordering. Your reported 20m to 14m reduction in build time for a large project like Miro’s monolith is impressive and highlights the potential of such optimizations on multi-core systems. I’d love to hear more about your PoC and explore how it could inform Maven’s evolution. Have you considered testing it against Maven 4’s concurrent builder to compare performance and compatibility? I’ll take a look at your code in the maven-turbo-builder subdirectory and try it out on a sample project. Thanks for sharing, and I’m excited to continue this discussion! Cheers, Guillaume Nodet [1] https://maven.apache.org/ref/4.0.0-rc-3/maven-impl-modules/maven-core/lifecycles.html Le mar. 27 mai 2025 à 21:26, Sergey Chernov <serega.mo...@gmail.com> a écrit : > > Thanks for asking, I planned to raise this topic soon in the mailing list > > Here you can find a presentation board > https://miro.com/app/board/uXjVLYUPRas=/ which also has a 5 minute video > with explanation (click Play on the first slide, no auth is required). > > Long story short: standard Maven builder (either single or multi-threaded) > has a pretty strong contract: it schedules the module to be built not > earlier than all upstream dependencies are resolved (if it's a jar library > - it's downloaded from remote to .m2, if it's another module - ALL build > phases are executed). And it's a bottleneck in multi-module projects built > on multi-core CPUs. > > It looks like takari-smart-builder respects this contract, but schedules > modules in different order, also resolving libraries eagerly for > optimization purposes. > > The maven-turbo-builder is a more radical change: > > 1. it schedules downstream dependency build once the upstream finished > package phase (instead of all requested phases), because package creates > jar artifact which is needed by downstream projects > 2. it changes order of test and package phases (test phases are before > package by default), because it allows to schedule downstream > dependencies earlier. In other words, it suggests optimistic strategy there > instead of original pessimistic > > As a result, this solution leverages multi-core CPUs way more efficiently. > E.g., for our huge Miro monolith application the test execution time (with > compiling and running only unit tests) is reduced from 20m to 14m. The > parallelization is so good that I even have to use less cores than T1C, > like T0.6C because of Amdahl's law effects. The takari-smart-builder, for > comparison, gives a very small boost (few percent if all library > dependencies are already downloaded to .m2). > > In the project readme you can find extension adoption for sample > maven-surefire project or validate it on your own. > > The extension is also considered as PoC to change the standard behavior of > Maven, which was quite good when it was created 20 years ago. At this > moment we did not have multi-core machines, but today this can be revisited > to maximize build performance. Also *this is a breaking change* for plugins > that rely on the order or phases, e.g. when "test-jar" classified artifact > is used as a dependency (which should be avoided IMO 😊). There can be > different strategies like opt-in this behavior or introduce a new phase > before test. While these disadvantages may look severe, the advantage is a > huge potential performance boost. > > I'm curious about your opinion about it and it would be great if you can > try it (you can inspect the code, it's very small - see maven-turbo-builder > subdir; other modules are side playground projects which are not directly > related to turbo builder extension). > > > On Tue, May 27, 2025 at 5:26 PM Tamás Cservenák <ta...@cservenak.net> wrote: > > > Hey Sergey, > > > > Can you tell us more about this "pet project" of yours? > > > > How is it different from, for example, the takari-smart-builder? > > > > Thanks > > T > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > > For additional commands, e-mail: dev-h...@maven.apache.org > > > > -- ------------------------ Guillaume Nodet --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
