moomindani opened a new pull request, #17177: URL: https://github.com/apache/iceberg/pull/17177
This PR adds two new S3FileIO properties to control the AWS SDK checksum behavior introduced in AWS SDK 2.30.0: - `s3.request-checksum-calculation`: controls when the SDK calculates a checksum for S3 requests (`when_supported` / `when_required`) - `s3.response-checksum-validation`: controls when the SDK validates checksums on S3 responses (`when_supported` / `when_required`) When a property is not set, the corresponding AWS SDK default (`WHEN_SUPPORTED` since SDK 2.30.0) is left unchanged, so this PR introduces no behavior change for existing users. ### Background Since AWS SDK 2.30.0, the SDK calculates a CRC32 checksum for all requests that support one (`RequestChecksumCalculation.WHEN_SUPPORTED` by default) and sends it as `x-amz-sdk-checksum-algorithm` / `x-amz-trailer: x-amz-checksum-crc32` headers. Some S3-compatible object stores (e.g. Dell ECS, GCS S3-compatible API) reject these headers, which breaks writes through S3FileIO. S3FileIO currently has no way to configure this SDK behavior; `s3.checksum-enabled` only controls the legacy Content-MD5 checksum. Setting `s3.request-checksum-calculation=when_required` stops the SDK from sending these headers. This revives and reworks #15391 by @rcjverhoef (thanks!), preserving the original commit. The rework decouples the new knobs from `s3.checksum-enabled` into standalone properties that follow the SDK defaults when unset. ### Design notes - Defaults intentionally follow the AWS SDK defaults (no behavior change). Flipping the default to `when_required` for better out-of-the-box compatibility with S3-compatible stores would be a behavior change and can be discussed separately on the dev list. - A follow-up PR will add an `s3.checksum-algorithm` property to select the checksum algorithm per request. - The properties are applied in all S3 client factories (default, assume-role, LakeFormation, S3FileIO default factory), for sync, async, and CRT async clients. ### Testing - Unit tests in `TestS3FileIOProperties` for parsing, defaults (unset leaves SDK defaults untouched), and builder application - Verified against MinIO (testcontainers) and real AWS S3 with a header-capturing interceptor: with the default settings PutObject carries `x-amz-sdk-checksum-algorithm: CRC32` and `x-amz-trailer: x-amz-checksum-crc32`; with `when_required` no checksum headers are sent. S3FileIO write/read/delete succeed under both settings. Fixes #14439 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
