[
https://issues.apache.org/jira/browse/HADOOP-18889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767666#comment-17767666
]
Steve Loughran commented on HADOOP-18889:
-----------------------------------------
[~ahmarsu] thanks; will look at tomorrow.
one little regression i've hit is that 3rd party stores may return 404 an abort
MPU request if the request has already been aborted...it's no longer idempotent
{code}
[ERROR]
testMultipartUploadAbort(org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader)
Time elapsed: 11.787 s <<< ERROR!
java.io.FileNotFoundException: Aborting multipart upload ID
AQAAAYq4pVYYAJ8Ebo0GR4a3z_Yzdv14njmie8U on
fork-0005/test/testMultipartUploadAbort:
software.amazon.awssdk.services.s3.model.NoSuchUploadException: The specified
multipart upload does not exist. The upload ID might be invalid, or the
multipart upload might have been aborted or completed. (Service: S3, Status
Code: 404, Request ID: 0c07c871:18a4aedd1ab:35b98:1c16, Extended Request ID:
9578ba52d8520581c0645b2eb9ee12ccf205b96fbbe8f335826ea72de53a8fa7):NoSuchUpload
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:248)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:255)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.abortMultipartUpload(WriteOperationHelper.java:395)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.abortMultipartCommit(WriteOperationHelper.java:471)
at
org.apache.hadoop.fs.s3a.impl.S3AMultipartUploader.lambda$abort$3(S3AMultipartUploader.java:243)
at org.apache.hadoop.util.LambdaUtils.eval(LambdaUtils.java:52)
at
org.apache.hadoop.fs.s3a.impl.StoreContext.lambda$submit$0(StoreContext.java:374)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java
{code}
I don't intend to make any changes to production code here. the test in
question actually has a flag to indicate whether aborts are repeatable; hdfs
says no and so the FNFE raised there is accepted.
Options
# add another test switch to change behaviour
# document it's a known quirk and not worry about it. This is not a
production-time issue after all.
looking at option #2, personally.
> S3A: V2 SDK client does not work with third-party store
> -------------------------------------------------------
>
> Key: HADOOP-18889
> URL: https://issues.apache.org/jira/browse/HADOOP-18889
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
>
> testing against an external store without specifying region now blows up
> because the region is queried off eu-west-1.
> What are we do to here? require the region setting *which wasn't needed
> before? what even region do we provide for third party stores?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
