[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15576:
------------------------------------
Status: Patch Available (was: Open)
HADOOP-15576 patch 005; patch 004 + my edits
* reorder imports
* fix some javadoc typos
* more tests, including one of a single part upload
* ITestS3AContractMultipartUploader is a scale test (run with -Dscale),
configurable for part size, and requests a smaller # of parts (not yet
configurable)
* Test assertions show that with S3Guard on, the MPU fails as the data length
recorded in the DDB table is always 0
w.r.t the assertion failure, I think the data included with each s3a part
handle needs to include the length of the block, so the aggregate length of the
upload can be built up for the commit.
> S3A Multipart Uploader to work with S3Guard and encryption
> -----------------------------------------------------------
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2
> Reporter: Steve Loughran
> Assignee: Ewan Higgs
> Priority: Blocker
> Attachments: HADOOP-15576-005.patch, HADOOP-15576.001.patch,
> HADOOP-15576.002.patch, HADOOP-15576.003.patch, HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
