raphaelazzolini commented on PR #7396:
URL: https://github.com/apache/hadoop/pull/7396#issuecomment-3090764761
@steveloughran this works with object lock, I just ran the following test:
* Created a new S3 bucket with object lock enabled
* Executed the command bellow, expecting the first test (without checksum
config) to fail
```
mvn clean verify -Dit.test=TestS3AEncryption,TestChecksumSupport -Dtest=none
```
#### Results
```
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[ERROR] Tests run: 10, Failures: 0, Errors: 10, Skipped: 0, Time elapsed:
3.64 s <<< FAILURE! - in
org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[ERROR]
test_010_CreateHugeFile(org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks)
Time elapsed: 1.387 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: PUT 0-byte object on
job-00/test: software.amazon.awssdk.services.s3.model.InvalidRequestException:
Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests
with Object Lock parameters (Service: S3, Status Code: 400, Request ID:
K7HJNJ5F2YPMDKPT, Extended Request ID:
satzMHpF3kUamAsECmEAg9cYPLAUpAKVhdIX9kMWR7di5KycSDCvlxGKcnmIPD3HGCIg7WSOels=):InvalidRequest:
Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests
with Object Lock parameters (Service: S3, Status Code: 400, Request ID:
K7HJNJ5F2YPMDKPT, Extended Request ID:
satzMHpF3kUamAsECmEAg9cYPLAUpAKVhdIX9kMWR7di5KycSDCvlxGKcnmIPD3HGCIg7WSOels=)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:265)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
(...)
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AChecksum
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.259
s - in org.apache.hadoop.fs.s3a.ITestS3AChecksum
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[ERROR]
ITestS3AHugeFilesArrayBlocks>AbstractSTestS3AHugeFiles.setup:102->S3AScaleTestBase.setup:85->AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:205->AbstractFSContractTestBase.mkdirs:363
» AWSBadRequest
[INFO]
[ERROR] Tests run: 11, Failures: 0, Errors: 10, Skipped: 0
```
---
Then, I added the checksum to the config and ran the same tests again.
```
<property>
<name>fs.s3a.create.checksum.algorithm</name>
<value>CRC32C</value>
</property>
```
#### Results
```
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
9.636 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AChecksum
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.05
s - in org.apache.hadoop.fs.s3a.ITestS3AChecksum
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
