roded created JCLOUDS-1549:
------------------------------
Summary: Multipart upload fails on AWS S3 buckets with Object Lock
and retention
Key: JCLOUDS-1549
URL: https://issues.apache.org/jira/browse/JCLOUDS-1549
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 2.2.0
Reporter: roded
AWS S3 buckets with object lock and a default object retention period
(governance/compliance) require the content MD5 header for PUT operations
([https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html])
When multipart uploading a blob into such a bucket, a response is returned
indicating that the content MD5 header is missing.
The following snippet:
{code:java}
BlobStoreContext blobStoreContext = ContextBuilder.newBuilder("aws-s3")
.endpoint("https://s3.amazonaws.com";)
.credentials("...", "...")
.buildView(BlobStoreContext.class);
BlobStore blobStore = blobStoreContext.getBlobStore();
ByteSource payloadByteSource =
ByteSource.wrap("payloaddata".getBytes(StandardCharsets.UTF_8));
Blob blob = blobStore.blobBuilder("multipart-uploaded-blob")
.userMetadata(Collections.emptyMap())
.payload(payloadByteSource)
.contentLength(payloadByteSource.size())
.contentMD5(payloadByteSource.hash(Hashing.md5()))
.build();
blobStore.putBlob("my-temp-object-locked-bucket", blob,
PutOptions.Builder.multipart());
blobStoreContext.close();
{code}
Results in the exception:
{code:java}
com.google.common.util.concurrent.UncheckedExecutionException:
org.jclouds.aws.AWSResponseException: request PUT
https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB',
requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=',
code='InvalidRequest', message='Content-MD5 HTTP header is required for Put
Part requests with Object Lock parameters',
context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}
at
com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
at
com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511)
at
org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:395)
at
org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
at
org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
at
io.model9.backup.common.objectstorage.MultipartMd5Test.multipartMd5(MultipartMd5Test.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: org.jclouds.aws.AWSResponseException: request PUT
https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB',
requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=',
code='InvalidRequest', message='Content-MD5 HTTP header is required for Put
Part requests with Object Lock parameters',
context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:76)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy44.uploadPart(Unknown Source)
at
org.jclouds.s3.blobstore.S3BlobStore.uploadMultipartPart(S3BlobStore.java:391)
at
org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:415)
at
org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:402)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at java.lang.Thread.getStackTrace(Thread.java:1559)
at
org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:393)
... 25 more
{code}
Here are the requests (URL and headers) initiated by JClouds in the above
snippet:
{code:java}
GET >> https://s3.amazonaws.com/my-temp-object-locked-bucket?location
{Host=[s3.amazonaws.com],
x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=...,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=db96f64af0bc4457ceda679805b6bb70656494b0ae5ae4d6501f1b19847dc31a]}
POST >>
https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploads
{Content-Type=[application/unknown], Content-MD5=[nirLkofY1bN5hCACAirgRA==],
Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com],
x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=...,
SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date,
Signature=fdb6a51c9cfbe116f8da033e079d4a9033a63b1d7a3abe8256546b2363f2dddb]}
PUT >>
https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
{Content-Type=[application/unknown], Content-Length=[11],
Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com],
x-amz-content-sha256=[9e8f1b13c357dd00b99dd1760be67a3fdeb3ed1e3605307e549ab74c483a43fa],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..,
SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date,
Signature=9aec5a51bbffaa95321a054d8d6aad40b7c7d87d6967562592f8aaf51bb5e7aa]}
DELETE >>
https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
{Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com],
x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=...,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=f2e53347d054d7c4a08eb1d62b1ff828d90135d0563d85ee8547626788dfd290]}
{code}
It looks like the blob's MD5 hash is added to the POST to initiate the
multipart upload but is missing from the part's PUT itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
