moomindani opened a new pull request, #17179:
URL: https://github.com/apache/iceberg/pull/17179
This PR adds a new S3FileIO property to select the checksum algorithm the
AWS SDK uses for data integrity protection on S3 upload requests:
- `s3.checksum-algorithm`: valid values are the algorithms supported by S3 —
`CRC32`, `CRC32C`, `CRC64NVME`, `SHA1`, `SHA256`, and (since AWS SDK 2.42.x)
`MD5`, `SHA512`, `XXHASH3`, `XXHASH64`, `XXHASH128` — case-insensitive. When
unset, the AWS SDK default algorithm (CRC32) is used, so there is no behavior
change.
### Background
S3FileIO has no way to choose the checksum algorithm today; the SDK always
uses its default (CRC32). Users may want `CRC64NVME` for efficient full-object
integrity checks of multipart uploads, or `SHA256` where a cryptographic hash
is required. See #17178 for details.
### Usage
Set the catalog property `s3.checksum-algorithm` to the desired algorithm
name, e.g. with Spark:
```
spark-sql --conf
spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
--conf
spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \
--conf spark.sql.catalog.my_catalog.s3.checksum-algorithm=CRC64NVME \
...
```
### Naming
The property is named `s3.checksum-algorithm` to sit alongside the existing
`s3.checksum-enabled`, which also concerns upload checksums. An alternative
considered was `s3.write.checksum-algorithm`, following the `s3.write.*`
convention used for other upload-only properties (e.g.
`s3.write.storage-class`), since this property only affects upload requests.
Open to renaming if reviewers prefer that convention. For reference, Hadoop S3A
calls the equivalent option `fs.s3a.create.checksum.algorithm`.
### Implementation notes
- The algorithm is applied per request in `S3OutputStream`
(`PutObjectRequest`, `CreateMultipartUploadRequest`, `UploadPartRequest`).
- Part checksums returned by `UploadPart` are passed on to
`CompleteMultipartUpload` via `CompletedPart`; S3 rejects the completion with
`InvalidPart` otherwise when an algorithm is specified.
- One interaction worth noting: setting a request-level checksum algorithm
causes the SDK to calculate checksums even when the SDK request checksum
calculation is configured to `when_required` (see #17177).
### Testing
- Unit tests in `TestS3FileIOProperties` for parsing (case-insensitivity,
invalid values, unset leaves the SDK default untouched)
- Integration test `TestS3OutputStream#testWriteWithChecksumAlgorithm`
verifying the algorithm is applied to PutObject, CreateMultipartUpload, and
UploadPart requests
- Verified against MinIO (testcontainers) and real AWS S3 with `CRC64NVME`
and `SHA256`, for both single PUT and multipart uploads: request headers carry
the configured algorithm (`x-amz-sdk-checksum-algorithm` / `x-amz-checksum-*`
trailers), uploads and reads succeed, and content roundtrips correctly.
Fixes #17178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
