[
https://issues.apache.org/jira/browse/HADOOP-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18015243#comment-18015243
]
Steve Loughran commented on HADOOP-15224:
-----------------------------------------
note that whatever algorithm you set, the sha256 checksum is set, which then
fails when working with third party stores. This must be an SDK issue
{code}
Exiting with status -1: org.apache.hadoop.fs.s3a.AWSBadRequestException: PUT
0-byte object on test/dir-bf299600-87cb-4434-886b-648c3d64cdcd:
software.amazon.awssdk.services.s3.model.S3Exception: The Content-SHA256 you
specified did not match what we received (Service: S3, Status Code: 400,
Request ID: 0c07c87a:196b249a2df:e85c3:b79, Extended Request ID:
64869ab8a95e700d3dc4dd0fa2286a071ca5a5e8b536e4d323483616f3edd1be) (SDK Attempt
Count: 1):XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not
match what we received (Service: S3, Status Code: 400, Request ID:
0c07c87a:196b249a2df:e85c3:b79, Extended Request ID:
64869ab8a95e700d3dc4dd0fa2286a071ca5a5e8b536e4d323483616f3edd1be) (SDK Attempt
Count: 1)
{code}
> S3A: Add option to set checksum on S3 objects
> ---------------------------------------------
>
> Key: HADOOP-15224
> URL: https://issues.apache.org/jira/browse/HADOOP-15224
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Raphael Azzolini
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.2
>
>
> The option fs.s3a.create.checksum.algorithm allows checksums to be set on
> file upload; It supports the following values:
> 'CRC32', 'CRC32C', 'SHA1', and 'SHA256'
> * This can protect against corruption of data even before uploading
> commences, and enables support for buckets with S3 Object Lock activated.
> * Third party stores may not support these checksum algorithms, or support a
> limited subset
> * Cryptographically strong hashes, here SHA1 and SHA256, are alway slower
> than simple CRC checksums.
> For verifying data is not tampered with before data upload, CRC32C should
> suffice
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
