gnodet commented on code in PR #2041:
URL: https://github.com/apache/maven-resolver/pull/2041#discussion_r3738367906
##########
src/site/markdown/expected-checksums.md:
##########
@@ -18,171 +18,104 @@ specific language governing permissions and limitations
under the License.
-->
-Checksums in Resolver were historically used during transport,
-to ensure Artifact integrity. In addition, latest Resolver may
-use checksums in various other ways too, for example to ensure
-Artifact integrity during resolution.
+Resolver uses checksums during transport to verify that the artifact is intact.
+Resolver also uses checksums at other times. For example, it verifies that the
artifact is intact during resolution.
-The bare essence of all checksum uses in Resolver is
-"integrity validation": Resolver calculates by various
-means the "calculated" checksum (for given payload),
-then obtains somehow the "expected" checksum (for same payload)
-and compares the two.
+Checksums in Resolver provide integrity verification. Resolver determines the
*calculated checksum* for an artifact by applying a mathematical algorithm to
its local copy of the artifact. It can read the bytes and calculate the
checksum during the download or from a file already present on the local file
system. Then it retrieves the *expected checksum* for the same artifact. It
compares the two checksums to see if the artifact is intact or corrupt.
-The "calculated" checksum is uninteresting from technical viewpoint,
-as it is calculated by standard means: either during payload
-streaming, or in worst case, from file already present on local
-file system (transport dependant).
-
-Instead, this page covers all the "expected" checksum varieties.
+This page covers the different ways Resolver can retrieve the *expected
checksum* for an artifact that it compares against the locally calculated
checksum.
## Transport Checksum Strategies
-Historically, the "obtain expected checksum" was implemented as simple
-request against Artifact checksum URL (Artifact URL appended by ".sha1"). This
logic
-is still present in current Resolver, but is "decorated" and extended in
multiple
-ways.
-
-Resolver has broadened the "obtain checksum" step for "expected" checksum with
two new strategies,
-so the three expected checksum kinds in transport are: "Provided", "Remote
Included" and
-"Remote External". All these strategies provide the source of "expected"
checksum,
-but it differs **how** Resolver obtains these.
-
-The new **Provided** kind of expected checksums are provided to resolver by
some alternative
-means, possibly ahead of any transport operation. There is an SPI extension
point that users may
-implement, to have own ways to provide checksums to resolver. Alternatively,
one may use Resolver out of the
-box implementation, that simply delegates "provided checksums" to "trusted
checksums" (more about them later).
-
-The new **Remote Included** checksums are in some way included by remote
party, typically
-in their response. Since advent of modern Repository Managers, most of
-them already sends checksums (usually the "standard" SHA-1 and MD5)
-in their response headers. Moreover, Maven Central, and even Google Mirror of
Maven Central
-sends them as well. By extracting these checksums from response, we can get
hashes
-that were provided by remote repository along with its content. This saves one
HTTP round-trip, as we
-got both, content and checksums in one response.
-
-Finally, the **Remote External** checksums are the "classic" checksums we all
know: They are laid down
-next to Artifact files, external in other words on the remote repository,
according
-to remote repository layout. To obtain Remote External checksum, new HTTP
request against remote repository is
-required. The order of requested checksums will follow the order given in
layout configuration,
-asking for checksums in same order as the parameter contains algorithm names.
-
-During single artifact retrieval, these strategies are executed in above
specified order,
-and only if current strategy has "no answer", the next strategy is attempted.
Hence, if
-resolver is able to get "expected" checksum from Provided Checksum Source, the
Remote Included
-and Remote External sources will not be consulted. Important implication:
given that almost
-all MRMs and remote repositories (Maven Central, Google Mirror of Maven
Central) send "standard" (SHA-1, MD5)
-checksums in their response, if any of the standard checksum are enabled,
validation will
-be probably satisfied by "Remote Included" strategy and "Remote External" will
be skipped.
-
-The big win here is that by obtaining hashes using "Remote Included" and not
by "Remote External"
-strategy, we can halve the count of HTTP requests to download an Artifact.
+Resolver has three strategies for retrieving the expected checksum:
*Provided*, *Remote Included*, and *Remote External*. Appending a checksum
extension to the artifact URL is an example of the Remote External strategy.
+The strategies differ in **how** Resolver gets the checksum.
-Related configuration keys:
-* `aether.layout.maven2.checksumAlgorithms` A comma-separated list of checksum
algorithms. Order is important, as
- transport will ask for those in specified order (default is "SHA-1,MD5"),
and first received and matched causes
- integrity validation algorithm to stop.
+**Provided** checksums are supplied to the Resolver through the Resolver's
Java API. Users can implement or install an SPI extension point that loads
checksums. The checksums can load before any transport operation.
+Users can also use the trusted checksum SPI bundled with the Resolver
implementation. The section "Trusted Checksums" below describes this approach.
-Note: Since Maven 3.9.x you can use expression
`${session.rootDirectory}/.mvn/checksums/` to store checksums along with
-sources as `session.rootDirectory` will become an absolute path pointing to
the root directory of your project (where
-usually the `.mvn` directory is).
+**Remote Included** checksums are part of the response that the artifact
itself arrives in. Most modern repositories send checksums in the HTTP response
headers. Maven Central sends the SHA-1 checksum as a hexadecimal string in the
x-checksum-sha1 HTTP header.
+It sends the hexadecimal encoded MD5 checksum in the x-checksum-md5 HTTP
header.
+Other repositories send similar HTTP headers.
+Resolver gets the content and the checksums in one HTTP request.
+**Remote External** checksums are separate resources in the remote repository.
The remote repository stores them next to the artifact files. To get a Remote
External checksum, Resolver sends a new HTTP GET request for the artifact
checksum URL. This is the artifact URL with an algorithm extension such as
".sha1" appended. For example, if the artifact URL is
https://repo1.maven.org/maven2/xom/xom/1.3.9/xom-1.3.9.jar, then the SHA-1
checksum URL is
https://repo1.maven.org/maven2/xom/xom/1.3.9/xom-1.3.9.jar.sha1, and the MD5
checksum URL is https://repo1.maven.org/maven2/xom/xom/1.3.9/xom-1.3.9.jar.md5.
-### Provided Checksums
-There is a Resolver SPI `ProvidedChecksumsSource` that makes possible to feed
Provided Checksums to Resolver ahead
-of actual transport. These checksums are used **during transport only** to
verify transported payload (artifacts)
-integrity. Hence, Provided checksums are NOT usable to verify already cached
artifacts integrity (unless you build
-with empty repository, of course, that forces all of your artifact go through
transport).
+When retrieving an artifact, Resolver looks for a checksum in this order until
it finds one:
-Resolver out of the box provides one SPI implementation: one that simply
delegates to "trusted checksums".
+1. Provided
+2. Remote Included
+3. Remote External
-### Remote Included Checksums
+If Resolver gets the expected checksum from the Provided source, it does not
consult the Remote Included and Remote External sources.
-**Note: Remote Included checksums work only with transport-http, they do NOT
work with transport-wagon!**
-
-By using "Remote Included" checksum feature, we are able to halve the issued
HTTP request
-count, since many repository services along Maven Central emits the reference
checksums in
-the artifact response itself (as HTTP headers). Hence, we are able to get the
-artifact and reference "expected" checksum using only one HTTP round-trip.
+Almost all repository managers and remote repositories send standard checksums
in their responses.
+If any standard checksum algorithm is enabled, the Remote Included strategy
usually finds a checksum. Then Resolver skips the Remote External strategy.
Related configuration keys:
-* `aether.connector.basic.smartChecksums` to enable or disable Remote Included
checksums.
+* `aether.checksums.checksumAlgorithms` A comma-separated list of checksum
algorithms. The order is important. The transport asks for the checksums in the
specified order. The default is "SHA-1,MD5". The first available algorithm will
be used. For example, if you prefer MD5 but are willing to use SHA-1, set
`aether.checksums.checksumAlgorithms` to "MD5,SHA-1".
-The Remote Included checksums support several "strategies" to extract
checksums from HTTP response header.
+In Maven 3.9.x and later, you can use the expression
`${session.rootDirectory}/.mvn/checksums/` to store checksums alongside
sources. `session.rootDirectory` becomes an absolute path. The path points to
the root directory of the project. The `.mvn` directory is usually in the root
directory.
-#### Sonatype Nexus 2
+### Provided Checksums
-Sonatype Nexus 2 uses SHA-1 hash to generate `ETag` header in "shielded" (à la
Plexus Cipher)
-way. Naturally, this means only SHA-1 is available in artifact response header.
+The Resolver SPI `ProvidedChecksumsSource` feeds the Provided Checksums to
Resolver before the actual transport. Resolver uses these checksums during
transport to verify the integrity of the transported payload. Provided
checksums cannot verify the integrity of cached artifacts. If you build with an
empty repository, all artifacts go through transport. Then the Provided
checksums can verify them.
-Emitted by: Sonatype Nexus2 only.
+Resolver provides one SPI implementation with the distribution for loading
trusted checksums.
+### Remote Included Checksums
-#### Non-standard `X-` headers
+**Note:** Remote Included checksums only work with transport-http. They do not
work with transport-wagon.
-Maven Central emits headers `x-checksum-sha1` and `x-checksum-md5` along with
artifact response.
-Google GCS on the other hand uses `x-goog-meta-checksum-sha1` and
`x-goog-meta-checksum-md5`
-headers. AWS S3 uses `x-amz-meta-checksum-sha1` and `x-amz-meta-checksum-md5`
headers.
-Resolver will detect all these and use their value.
+Maven Central and many other repositories include the reference checksums in
the HTTP response headers. Resolver gets the artifact and the expected checksum
with one HTTP request.
-Emitted by: Maven Central, GCS, AWS S3, some CDNs and probably more.
+Resolver checks several unstandardized `X-` headers for checksums:
+
+* `x-checksum-sha1` and `x-checksum-md5`: Maven Central
+* `x-goog-meta-checksum-sha1` and `x-goog-meta-checksum-md5`: the Google
Mirror of Maven Central
+* `x-amz-meta-checksum-sha1` and `x-amz-meta-checksum-md5`: AWS S3
Review Comment:
This line documents `x-amz-meta-checksum-sha1` and `x-amz-meta-checksum-md5`
for AWS S3, but the codebase has no implementation for these headers.
`XChecksumExtractor.java` only handles:
- `x-checksum-sha1` / `x-checksum-md5`
- `x-goog-meta-checksum-sha1` / `x-goog-meta-checksum-md5`
`grep -ri "amz" --include="*.java"` across the entire repository returns
zero results. This bullet should be removed unless code support is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
