[
https://issues.apache.org/jira/browse/MRESOLVER-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577258#comment-17577258
]
ASF GitHub Bot commented on MRESOLVER-7:
----------------------------------------
caiwei-ebay commented on code in PR #178:
URL: https://github.com/apache/maven-resolver/pull/178#discussion_r941047977
##########
maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java:
##########
@@ -365,6 +477,64 @@ else if ( descriptorResult == DataPool.NO_DESCRIPTOR )
return descriptorResult;
}
+ static class ParallelDescriptorResolver
+ {
+ final ExecutorService executorService;
Review Comment:
Uploaded a diagram to explain how the dependency resolution works.
In a Maven build:
- DependencyCollector focus on resolve dependency graph by downloading and
parsing metadata.xml and pom.xmls which is running in sequence as of now.
- MojoExecutor calls ensureDependencyResolved which download jars in
parallel based on the dependency graph calculated by DependencyCollector.
Using the former (fully unitize get <List, List>) in DependencyCollector
seems not possible, I think we need to use a separate thread pool for
dependency resolution in DependencyCollector which includes 2 steps:
- Resolve version range (which resort to VersionRangeResolver ->
MetadataResolver -> RepositoryConnector)
- Resolve descriptor (download or pick local poms and then parse poms, which
resort to ArtifactDescriptorReader -> ArtifactResolver -> RepositoryConnector)
By running the above 2 steps in a separate thread pool can achieve better
parallelism.
Like the MetadataResolver, it is also using a separate ThreadPool to
download metadata.xml one by one from multiple repositories.
May I know any concerns that we should not use another thread pool for
dependency resolution here?
> Download dependency POMs in parallel
> ------------------------------------
>
> Key: MRESOLVER-7
> URL: https://issues.apache.org/jira/browse/MRESOLVER-7
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
> Affects Versions: Aether 1.0.2
> Reporter: Harald Wellmann
> Priority: Major
> Attachments: resolve_deps.png, resolver.log
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> h3. Background
> When building a project with dependencies not yet available in the local
> repository, I noticed that Maven 3.3.9/Aether 1.0.2 first downloads the
> dependency POMs _sequentially_ and then proceeds downloading the dependency
> JARs with up to 5 threads _in parallel_.
> Due to this, when first building a project with a large number of
> dependencies, downloading a large number of small POMs may take a lot longer
> than downloading the much larger JARs, or even longer than building the
> project itself, especially when a repository manager is used which increases
> the download latency.
> h3. Enhancement
> Download POMs of (transitive) dependencies in parallel to significantly speed
> up initial builds of large projects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
