[
https://issues.apache.org/jira/browse/MRESOLVER-229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454942#comment-17454942
]
wei cai edited comment on MRESOLVER-229 at 12/8/21, 3:59 AM:
-------------------------------------------------------------
My misunderstanding, maven downloads jars in parallel with max 5 threads but is
partial, and MRESOLVER-7 did provide a parallel solution make use of full
threads.
Regarding MNG-7129
I think MNG-7129 is different solution than my ticket, MNG-7129 is using
similar solution like gradle enterprise maven extension, it would cache the
plugins/tests execution result based on the principle that "if inputs does not
change, then outputs should be identical."
It focuses on pure build part and does not cover download part.
This ticket is actually a quick win version of MRESOLVER-7, aims to solve the
download slowness. MRESOLVER-7 is introducing papalism at maven-resolver's
DefaultDependencyCollector, and this ticket is suggesting creating a new layer
on top of DefaultDependencyCollector, it is like when maven starts to build, it
would first call DefaultDependencyWarmup.warmup to warmup dependencies in
parallel, and then call DefaultDependencyCollector.collectDependencies, when
collectDependencies completes, it would call
DefaultDependencyWarmup.cacheUpdate to update cache.
The API call chain:
DefaultDependencyWarmup.warmup ->
DefaultDependencyCollector.collectDependencies ->
DefaultDependencyWarmup.cacheUpdate
The flow of DefaultDependencyWarmup.warmup is like:
* read .mvn/artifacts_cache.xml (the xml stores the GAV list required by the
project like groupdId:artifactId:version;groupdId2:artifactId2:version2)
* if yes, parallel resolve the artifactIds defined in artifacts_cache.xml
The flow of DefaultDependencyWarmup.cacheUpdate is like:
* update the GAV list to .mvn/artifacts_cache.xml
was (Author: wecai):
My misunderstanding, maven downloads jars in parallel with max 5 threads but is
partial, and MRESOLVER-7 did provide a parallel solution make use of full
threads.
Regarding MNG-7129
I think MNG-7129 is different solution than my ticket, MNG-7129 is using
similar solution like gradle enterprise maven extension, it would cache the
plugins/tests execution result based on the principle that "if inputs does not
change, then outputs should be identical."
This ticket is actually a quick win version of MRESOLVER-7, aims to solve the
download slowness. MRESOLVER-7 is introducing papalism at maven-resolver's
DefaultDependencyCollector, and this ticket is suggesting creating a new layer
on top of DefaultDependencyCollector, it is like when maven starts to build, it
would first call DefaultDependencyWarmup.warmup to warmup dependencies in
parallel, and then call DefaultDependencyCollector.collectDependencies, when
collectDependencies completes, it would call
DefaultDependencyWarmup.cacheUpdate to update cache.
The API call chain:
DefaultDependencyWarmup.warmup ->
DefaultDependencyCollector.collectDependencies ->
DefaultDependencyWarmup.cacheUpdate
The flow of DefaultDependencyWarmup.warmup is like:
* read .mvn/artifacts_cache.xml (the xml stores the GAV list required by the
project like groupdId:artifactId:version;groupdId2:artifactId2:version2)
* if yes, parallel resolve the artifactIds defined in artifacts_cache.xml
The flow of DefaultDependencyWarmup.cacheUpdate is like:
* update the GAV list to .mvn/artifacts_cache.xml
> Improve the download speed by downloading poms & jars in parallel as a warm-up
> ------------------------------------------------------------------------------
>
> Key: MRESOLVER-229
> URL: https://issues.apache.org/jira/browse/MRESOLVER-229
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
> Affects Versions: 1.7.1
> Reporter: wei cai
> Priority: Major
>
> We are tuning maven builds in our company, so far, we've made 2 big changes
> that we think they can benefit the opensource community.
> * maven-resolver change by a skip & reconcile approach
> https://issues.apache.org/jira/browse/MRESOLVER-228?filter=allopenissues
> * download poms and jars in parallel ahead as a warm-up
> It is not a new issue that downloading dependencies in maven is slow. Similar
> issues reported in:
> https://issues.apache.org/jira/browse/MRESOLVER-7
> Our solution is a bit different, basically we:
> * Collect artifacts required for the project by listening to the maven
> dependency resolve event and store as a artifacts.json (GAV list & its
> repository ids) somewhere, in our case, we are storing the artifacts.json in
> a backend service.
> * Before maven build kick starts, we called the maven-resolver api
> (DefaultDependencyCollector) to resolve all artifacts in artifacts.json in
> parallel.
> * After warm-up completed, let maven still build the project as normal. This
> means even something missed in artifacts.json, maven could still find the
> missing part and resolve them.
> This solution does not help for the very 1st build as no artifacts.json is
> built yet, however for any future builds, download can happen in parallel and
> thus it would be very quick.
> This solution helps speed up the maven builds in below cases:
> * All CI builds
> * New hires or new machines as there is no local maven repo cache
> * Snapshot updates
> * When a new dependency added, especially for a dependency having lots of
> transitive dependencies
> * Version upgrade such as Spring boot version upgrade
> [~michael-o]
> As a opensource solution, may be we can store the artifacts.json to .mvn in
> the git repository root, in this way, the artifacts list cache goes with the
> code.
> Please share your comments with this solution.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
