This is an automated email from the ASF dual-hosted git repository. quantranhong1999 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git
commit c1b7246294b4eb90b6b03c7f1edbdd804990dc10 Author: Benoit TELLIER <[email protected]> AuthorDate: Sat Sep 5 00:07:52 2026 +0200 JAMES-4224 Make 128 bit entropy the new default --- docs/modules/servers/partials/configure/jvm.adoc | 32 +++++++++++----------- .../sample-configuration/jvm.properties | 11 ++++---- .../org/apache/james/blob/api/BlobIdEncoding.java | 4 +-- .../org/apache/james/blob/api/BlobIdEntropy.java | 16 ++++++----- .../apache/james/blob/api/BlobIdEncodingTest.java | 8 +++++- .../apache/james/blob/api/BlobIdEntropyTest.java | 10 +++++++ .../blob/api/DeduplicationBlobStoreContract.java | 7 +++-- .../org/apache/james/blob/api/PlainBlobIdTest.java | 3 +- .../aws/S3WithMinIOGenerationAwareBlobIdTest.java | 2 +- upgrade-instructions.md | 18 ++++++++++++ 10 files changed, 75 insertions(+), 36 deletions(-) diff --git a/docs/modules/servers/partials/configure/jvm.adoc b/docs/modules/servers/partials/configure/jvm.adoc index eda2806d8f..2ad1a7ab83 100644 --- a/docs/modules/servers/partials/configure/jvm.adoc +++ b/docs/modules/servers/partials/configure/jvm.adoc @@ -101,31 +101,31 @@ message is reindexed, so the space comes back progressively rather than at once. == Change the entropy of the blobId -By default a blobId carries 256 bits of entropy: the full SHA-256 of the content it addresses for -deduplicated blobs, and as many random bits for the randomly generated ones. The property -`james.blobid.entropy` shortens them, in bits. +By default a blobId carries 128 bits of entropy: the leading 128 bits of the SHA-256 of the content it +addresses for deduplicated blobs, and as many random bits for the randomly generated ones. The property +`james.blobid.entropy` changes that, in bits. Ex in `jvm.properties` ---- -james.blobid.entropy=128 +james.blobid.entropy=256 ---- -Optional. Integer, a multiple of 8 within [128, 256]. Defaults to 256. +Optional. Integer, a multiple of 8 within [128, 256]. Defaults to 128. -Shorter ids cost less everywhere an id is stored. The object key itself, but above all the Cassandra -columns referencing it, which hold one id per message rather than one per blob: going from 256 to 128 -bits takes a body blob id from 50 to 28 characters. Truncated ids are also left unpadded, which is where -the last two characters go. +The length of an id is paid everywhere it is stored. The object key itself, but above all the Cassandra +columns referencing it, which hold one id per message rather than one per blob: 256 bits takes a body +blob id from 28 to 50 characters. Ids below the full 256 bits are also left unpadded, which is worth two +more characters. -128 bits is the sensible alternative to the default. The birthday bound puts a collision at -`n^2/2^129`, ie. 1.5e-19 for ten billion blobs, twenty orders of magnitude below the silent error rate of -the storage underneath; and truncating a cryptographic hash to its leading bits is standard practice -(NIST SP 800-107, FIPS 180-4). Values below 128 bits are rejected: a collision in a deduplicated store -means a message silently inheriting the body of another. +128 bits is ample for a content addressed store. The birthday bound puts a collision at `n^2/2^129`, ie. +1.5e-19 for ten billion blobs, twenty orders of magnitude below the silent error rate of the storage +underneath; and truncating a cryptographic hash to its leading bits is standard practice (NIST SP +800-107, FIPS 180-4). Values below 128 bits are rejected: a collision in a deduplicated store means a +message silently inheriting the body of another. `256` spells ids out the way releases up to 3.9.x did. -WARNING: This is an install time setting, not one to flip on a live deployment. Lowering it loses +WARNING: This is an install time setting, not one to flip on a live deployment. Changing it loses nothing, since ids are stored alongside the messages and existing blobs stay readable, but content -already stored under a longer id will not deduplicate against its shorter counterpart until it is +already stored under a differently spelled id will not deduplicate against its counterpart until it is rewritten. == Improve listing support for MinIO diff --git a/server/apps/distributed-app/sample-configuration/jvm.properties b/server/apps/distributed-app/sample-configuration/jvm.properties index 4a6fd73b98..29831fa8f4 100644 --- a/server/apps/distributed-app/sample-configuration/jvm.properties +++ b/server/apps/distributed-app/sample-configuration/jvm.properties @@ -102,12 +102,13 @@ jmx.remote.x.mlet.allow.getMBeansFromURL=false # james.jmap.preview.length=128 # Bits of entropy carried by a blobId: the SHA-256 is truncated to that many leading bits, and randomly -# generated ids draw that many. A multiple of 8 within [128, 256], defaults to 256. -# 128 shortens a body blobId from 50 to 28 chars, in the object key and in every Cassandra column -# referencing it, for a collision probability of 1.5e-19 at ten billion blobs. +# generated ids draw that many. A multiple of 8 within [128, 256], defaults to 128. +# 256 spells ids out the way releases up to 3.9.x did, taking a body blobId from 28 to 50 chars in the +# object key and in every Cassandra column referencing it, for no practical collision benefit: 128 bits +# already puts a collision at 1.5e-19 for ten billion blobs. # Install time setting: changing it on a live deployment stops new writes from deduplicating against -# blobs already stored under a longer id. -# james.blobid.entropy=128 +# blobs already stored under a differently spelled id. +# james.blobid.entropy=256 # Count of octet from which hashing shall be done out of the IO threads in deduplicating blob store # james.deduplicating.blobstore.thread.switch.threshold=32768 diff --git a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobIdEncoding.java b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobIdEncoding.java index ecceb81c41..655c2d365a 100644 --- a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobIdEncoding.java +++ b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobIdEncoding.java @@ -30,7 +30,7 @@ import com.google.common.io.BaseEncoding; * * <p>Truncated ids are left unpadded: they exist to be short, and padding them back up would give away * part of what {@link BlobIdEntropy} saved. Ids at full entropy keep the padding of their encoding, so - * that ids of existing deployments are left untouched.</p> + * that a deployment pinned there keeps spelling ids the way releases up to 3.9.x did.</p> */ public class BlobIdEncoding { public static final String ENCODING_PROPERTY = "james.blob.id.hash.encoding"; @@ -58,7 +58,7 @@ public class BlobIdEncoding { @VisibleForTesting BlobIdEncoding(BaseEncoding encoding) { - if (BlobIdEntropy.entropyBits() == BlobIdEntropy.DEFAULT_ENTROPY_BITS) { + if (BlobIdEntropy.entropyBits() == BlobIdEntropy.MAX_ENTROPY_BITS) { this.encoding = encoding; } else { this.encoding = encoding.omitPadding(); diff --git a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobIdEntropy.java b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobIdEntropy.java index 294ee7200f..da1e3a5dce 100644 --- a/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobIdEntropy.java +++ b/server/blob/blob-api/src/main/java/org/apache/james/blob/api/BlobIdEntropy.java @@ -29,15 +29,17 @@ import com.google.common.base.Preconditions; /** * How many bits of entropy a blob id carries, as set by the {@code james.blobid.entropy} system property. * - * <p>Defaults to {@value #DEFAULT_ENTROPY_BITS} bits, the full SHA-256 output, so that ids of existing - * deployments are left untouched. {@code 128} is the sensible alternative: the birthday bound puts a - * collision at {@code n^2/2^129}, ie. 1.5e-19 for ten billion blobs, and truncating a cryptographic hash - * to its leading bits is standard practice (NIST SP 800-107, FIPS 180-4).</p> + * <p>Defaults to {@value #DEFAULT_ENTROPY_BITS} bits: the birthday bound puts a collision at + * {@code n^2/2^129}, ie. 1.5e-19 for ten billion blobs, and truncating a cryptographic hash to its + * leading bits is standard practice (NIST SP 800-107, FIPS 180-4). {@value #MAX_ENTROPY_BITS}, the full + * SHA-256 output, spells ids out the way releases up to 3.9.x did.</p> */ public class BlobIdEntropy { public static final String ENTROPY_BITS_PROPERTY = "james.blobid.entropy"; - public static final int DEFAULT_ENTROPY_BITS = 256; + public static final int DEFAULT_ENTROPY_BITS = 128; private static final int MIN_ENTROPY_BITS = 128; + /** The full SHA-256 output: the longest an id can usefully get. */ + static final int MAX_ENTROPY_BITS = 256; private static final int BITS_PER_BYTE = 8; private static final SecureRandom SECURE_RANDOM = new SecureRandom(); @@ -57,8 +59,8 @@ public class BlobIdEntropy { int bits = Integer.parseInt(value); Preconditions.checkArgument(bits % BITS_PER_BYTE == 0, "'%s' must be a multiple of %s, got %s", ENTROPY_BITS_PROPERTY, BITS_PER_BYTE, bits); - Preconditions.checkArgument(bits >= MIN_ENTROPY_BITS && bits <= DEFAULT_ENTROPY_BITS, - "'%s' must be within [%s, %s], got %s", ENTROPY_BITS_PROPERTY, MIN_ENTROPY_BITS, DEFAULT_ENTROPY_BITS, bits); + Preconditions.checkArgument(bits >= MIN_ENTROPY_BITS && bits <= MAX_ENTROPY_BITS, + "'%s' must be within [%s, %s], got %s", ENTROPY_BITS_PROPERTY, MIN_ENTROPY_BITS, MAX_ENTROPY_BITS, bits); return bits; } catch (NumberFormatException e) { throw new IllegalArgumentException("Invalid '" + ENTROPY_BITS_PROPERTY + "' value: '" + value + "'. Expected a bit count, eg. 128 or 256", e); diff --git a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BlobIdEncodingTest.java b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BlobIdEncodingTest.java index 6b3f983656..0c948938e3 100644 --- a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BlobIdEncodingTest.java +++ b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BlobIdEncodingTest.java @@ -63,7 +63,13 @@ class BlobIdEncodingTest { @Test void shouldDefaultToBase64Url() { assertThat(BlobIdEncoding.fromSystemProperties().encode(PAYLOAD)) - .isEqualTo(BaseEncoding.base64Url().encode(PAYLOAD)); + .isEqualTo(BaseEncoding.base64Url().omitPadding().encode(PAYLOAD)); + } + + @Test + void shouldOmitPaddingAtTruncatedEntropy() { + assertThat(BlobIdEncoding.fromSystemProperties().encode(PAYLOAD)) + .doesNotEndWith("="); } @Test diff --git a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BlobIdEntropyTest.java b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BlobIdEntropyTest.java index e573c4d64a..abd59ec5e9 100644 --- a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BlobIdEntropyTest.java +++ b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/BlobIdEntropyTest.java @@ -40,6 +40,16 @@ class BlobIdEntropyTest { assertThat(BlobIdEntropy.parse("128")).isEqualTo(128); } + @Test + void parseShouldAcceptTheFullHashLength() { + assertThat(BlobIdEntropy.parse("256")).isEqualTo(BlobIdEntropy.MAX_ENTROPY_BITS); + } + + @Test + void defaultShouldBeTruncated() { + assertThat(BlobIdEntropy.DEFAULT_ENTROPY_BITS).isEqualTo(128); + } + @Test void parseShouldRejectNonNumericValue() { assertThatThrownBy(() -> BlobIdEntropy.parse("many")) diff --git a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/DeduplicationBlobStoreContract.java b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/DeduplicationBlobStoreContract.java index ab20cdf5a5..9da5950986 100644 --- a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/DeduplicationBlobStoreContract.java +++ b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/DeduplicationBlobStoreContract.java @@ -43,6 +43,7 @@ public interface DeduplicationBlobStoreContract { } String SHORT_STRING = "toto"; + String SHORT_STRING_BLOB_ID = "MfemXjFVhqwZi9eYtmKc5A"; BlobStore testee(); @@ -58,7 +59,7 @@ public interface DeduplicationBlobStoreContract { BlobId blobId = Mono.from(store.save(defaultBucketName, SHORT_STRING, storagePolicy)).block(); - assertThat(blobId).isEqualTo(blobIdFactory().parse("MfemXjFVhqwZi9eYtmKc5JA9CJlHbVdBqfMuLlIbamY=")); + assertThat(blobId).isEqualTo(blobIdFactory().parse(SHORT_STRING_BLOB_ID)); } @ParameterizedTest @@ -69,7 +70,7 @@ public interface DeduplicationBlobStoreContract { BlobId blobId = Mono.from(store.save(defaultBucketName, SHORT_BYTEARRAY, storagePolicy)).block(); - assertThat(blobId).isEqualTo(blobIdFactory().parse("MfemXjFVhqwZi9eYtmKc5JA9CJlHbVdBqfMuLlIbamY=")); + assertThat(blobId).isEqualTo(blobIdFactory().parse(SHORT_STRING_BLOB_ID)); } @ParameterizedTest @@ -82,6 +83,6 @@ public interface DeduplicationBlobStoreContract { // This fix is ok because it will only affect deduplication, after this change the same content might be assigned a different blobid // and thus might be duplicated in the store. No data can be lost since no api allows for externally deterministic blob id construction // before this change. - assertThat(blobId).isEqualTo(blobIdFactory().of("MfemXjFVhqwZi9eYtmKc5JA9CJlHbVdBqfMuLlIbamY=")); + assertThat(blobId).isEqualTo(blobIdFactory().of(SHORT_STRING_BLOB_ID)); } } diff --git a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/PlainBlobIdTest.java b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/PlainBlobIdTest.java index d555deefac..0a33b09dce 100644 --- a/server/blob/blob-api/src/test/java/org/apache/james/blob/api/PlainBlobIdTest.java +++ b/server/blob/blob-api/src/test/java/org/apache/james/blob/api/PlainBlobIdTest.java @@ -31,7 +31,8 @@ import com.google.common.io.BaseEncoding; import nl.jqno.equalsverifier.EqualsVerifier; class PlainBlobIdTest { - private static final BaseEncoding ENCODING = BaseEncoding.base64Url(); + // Mirrors how the factory spells ids at the default, truncated, entropy: base64url, unpadded. + private static final BaseEncoding ENCODING = BaseEncoding.base64Url().omitPadding(); private static final PlainBlobId.Factory BLOB_ID_FACTORY = new PlainBlobId.Factory(); diff --git a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3WithMinIOGenerationAwareBlobIdTest.java b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3WithMinIOGenerationAwareBlobIdTest.java index a13a9e772c..1c7f7012ae 100644 --- a/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3WithMinIOGenerationAwareBlobIdTest.java +++ b/server/blob/blob-s3/src/test/java/org/apache/james/blob/objectstorage/aws/S3WithMinIOGenerationAwareBlobIdTest.java @@ -126,7 +126,7 @@ public class S3WithMinIOGenerationAwareBlobIdTest implements BlobStoreContract { String blobIdString = blobId.asString(); // Then: BlobId string and parsed BlobId should match expectations - assertThat(blobIdString).isEqualTo("1/628/M/f/emXjFVhqwZi9eYtmKc5JA9CJlHbVdBqfMuLlIbamY="); + assertThat(blobIdString).isEqualTo("1/628/M/f/emXjFVhqwZi9eYtmKc5A"); assertThat(blobId).isEqualTo(blobIdFactory().parse(blobIdString)); } diff --git a/upgrade-instructions.md b/upgrade-instructions.md index 8ca73e0c8d..2dd12243b5 100644 --- a/upgrade-instructions.md +++ b/upgrade-instructions.md @@ -22,6 +22,24 @@ Change list: - [JAMES-4210 SMTP AuthHook deprecation](#james-4210-smtp-authhook-deprecation) - [JAMES-4210 POP3 USER/PASS requires TLS by default](#james-4210-pop3-userpass-requires-tls-by-default) - [JAMES-4210 ManageSieve SASL adoption](#james-4210-managesieve-sasl-adoption) + - [JAMES-4225 Blob ids default to 128 bits of entropy](#james-4225-blob-ids-default-to-128-bits-of-entropy) + +### JAMES-4225 Blob ids default to 128 bits of entropy + +Date: 05/09/2026 + +Concerned products: all products using a blob store + +Blob ids now carry 128 bits of entropy instead of the full 256 bits of a SHA-256, which shortens a body +blob id from 50 to 28 characters in the object key and in every metadata store column referencing it. + +Existing blobs stay readable. However, content stored under the previous spelling will not deduplicate +against its shorter counterpart until it is rewritten. To keep the previous behaviour, add to +`jvm.properties`: + +``` +james.blobid.entropy=256 +``` ### JAMES-4210 POP3 USER/PASS requires TLS by default --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
