[
https://issues.apache.org/jira/browse/MRESOLVER-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wei cai updated MRESOLVER-247:
------------------------------
Summary: Avoid unnecessary dependency resolution by a Skip solution based
on BFS (was: Avoid unnecessary dependency resolution )
> Avoid unnecessary dependency resolution by a Skip solution based on BFS
> -----------------------------------------------------------------------
>
> Key: MRESOLVER-247
> URL: https://issues.apache.org/jira/browse/MRESOLVER-247
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
> Affects Versions: 1.7.3
> Reporter: wei cai
> Priority: Major
> Attachments: exclusion-matters.png, skip-duplicate.png,
> skip-version-conflicts.png
>
>
> This Jira is related with https://issues.apache.org/jira/browse/MRESOLVER-240
> and https://issues.apache.org/jira/browse/MRESOLVER-228
> There was discussions about the DFS or BFS algorithm for maven resolver in
> MRESOLVER-228, Changing to BFS would make MRESOLVER-228 & MRESOLVER-7 much
> easier to implement. Here is the plan for multiple changes requested recently:
> * DFS > BFS - preparation for parallel download
> * Skip - avoid unnecessary version resolution （Covered in this JIRA）
> * Download descriptors in parallel (MRESOLVER-7)
>
> When comes to resolve the huge amount of dependencies of an enterprise level
> project, the maven resolver is very slow to resolve the dependency
> graph/tree. Take one of our app as example, it could take *10minutes+ and 16G
> memory* to print out the result of {*}mvn dependency:tree{*}.
> This is because there are many dependencies declared in the project, and some
> of the dependencies would introduce *600+* transitive dependencies, and
> exclusions are widely used to solve dependency conflicts.
> By checking the
> [code|https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java#L500],
> we know the exclusion is also part of the cache key. This means when the
> exclusions up the tree differs, the cached resolution result for the same GAV
> won't be picked up and need s to be recalculated.
>
> !exclusion-matters.png!
>
> From above figure, we know:
> * In 1st case, D will be resolved only once as there are no exclusions/same
> exclusions up the tree.
> * In 2nd case, the B and C have different exclusions and D needs to be
> recalculated, if D is a heavy dependency which introduce many transitive
> dependencies, all D and its children needs to be recalculated. Recalculating
> all of these nodes introduces 2 issues:
> ** Slow in resolving dependencies.
> ** Lots of DependencyNodes cached (all calculated/recalculated nodes would
> be cached) and will consume huge memory.
> To improve the speed of maven resolver's dependency resolution, I
> implemented a skip approach.
> *CASE 1: Skip duplicate node*
> !skip-duplicate.png!
> From above figure, the 1st R is resolved at depth 3, and the 2nd R is
> resolved again because the depth is at 2 which is lower, the 3rd R at depth 3
> and the 4th R at depth 4 are simply skipped as R is already resolved at depth
> 2. This is because the same node with deeper depth is most likely won't be
> picked up by maven as maven employs a "nearest transitive dependency in the
> tree depth and the first in resolution" strategy.
> The 3rd R and 4th R will have children set as zero and marked as skipped by
> the R at depth 2 in 2nd tree path.
> *CASE 2: Skip version conflict*
> *!skip-version-conflicts.png!*
>
> After we enabled the resolver patch in maven, we are seeing 10% ~70% build
> time reduced for different projects depend on how complex the dependencies
> are, and the result of *mvn dependency:tree* and *mvn dependency:list* remain
> the same.
> We've verified the resolver performance patch leveraging an automation
> solution to certify 2000+ apps of our company by comparing the *mvn
> dependency:tree* and *mvn dependency:list* result with/without the
> performance patch.
> Please help review the PR.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
