moomindani opened a new issue, #17178: URL: https://github.com/apache/iceberg/issues/17178
### Feature Request / Improvement S3FileIO currently has no way to choose which checksum algorithm the AWS SDK uses for data integrity protection on S3 writes. Since AWS SDK 2.30.0, the SDK calculates a CRC32 checksum by default for requests that support one, and the only related knobs are the legacy Content-MD5 (`s3.checksum-enabled`) and, once #17177 lands, the calculation/validation policies (`s3.request-checksum-calculation` / `s3.response-checksum-validation`) — none of which select the algorithm itself. I propose adding a new property: - `s3.checksum-algorithm`: selects the checksum algorithm the SDK uses for S3FileIO writes. When unset, the AWS SDK default (CRC32) is used, so there is no behavior change. S3 supports CRC32, CRC32C, CRC64NVME, SHA-1, and SHA-256, and added MD5, SHA-512, XXHash3, XXHash64, and XXHash128 in April 2026 (available in the AWS SDK since 2.42.x, already covered by the SDK version Iceberg uses today). Use cases include: - CRC64NVME for efficient full-object integrity checks (AWS-recommended for multipart uploads, since it composes across parts) - SHA-256 where compliance requirements mandate a cryptographic hash - Matching the algorithm an organization already standardizes on for object verification The property would be applied per request in `S3OutputStream` (`PutObjectRequest`, `CreateMultipartUploadRequest`, `UploadPartRequest`), following how other write options are applied there. One interaction to document: setting a request-level checksum algorithm causes the SDK to calculate checksums even when `s3.request-checksum-calculation` is `when_required`, so an explicit algorithm effectively opts back in to checksum calculation. Related: #14439, #17177 ### Query engine None ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
