[
https://issues.apache.org/jira/browse/HADOOP-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567845#comment-16567845
]
Steve Loughran commented on HADOOP-15576:
-----------------------------------------
HADOOP-15576 patch 007
* S3A part handles marshall (header, len, etag); unmarshall validates header &
extracts len and etag. Unit tests for this. Uses java DataInputStream, nothing
fancy.
* total length of unmarshalled parts is used
* S3A MPU rejects empty handle list, as does file uploader
* Test to verify that MPUs reject empty handle lists
* S3A moves test to the sequential bit at the end, just because its uploading
so much data (which I want to cut back on more for that reason)
Testing, US-west-1.
The rule "1+ handle must have been uploaded" is new, but it stops the MPU
complete on S3 failing. The other stores did work. I think the design needs a
policy here of allow vs reject, and be consistent. Note also the requirement
that after a complete fails, abort() still cleans. up. Again, something to
specify in HDFS-13713.
Big issue there: what would this mean for a distcp working this way? I'd
propose: 0-byte files get treated as special, or at least there's a requirement
for a 0-byte upload. Which, if supported, is something else to test for.
[~ehiggs]: your turn on this again. Remember to use -Dscale to run the tests
now; think about making that count of parts configurable so you can do a full
scale 1000-part upload
> S3A Multipart Uploader to work with S3Guard and encryption
> -----------------------------------------------------------
>
> Key: HADOOP-15576
> URL: https://issues.apache.org/jira/browse/HADOOP-15576
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2
> Reporter: Steve Loughran
> Assignee: Ewan Higgs
> Priority: Blocker
> Attachments: HADOOP-15576-005.patch, HADOOP-15576-007.patch,
> HADOOP-15576.001.patch, HADOOP-15576.002.patch, HADOOP-15576.003.patch,
> HADOOP-15576.004.patch
>
>
> The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with
> the tests to demonstrate this
> # move from low-level calls of S3A client to calls of WriteOperationHelper;
> adding any new methods are needed there.
> # Tests. the tests of HDFS-13713.
> # test execution, with -DS3Guard, -DAuth
> There isn't an S3A version of {{AbstractSystemMultipartUploaderTest}}, and
> even if there was, it might not show that S3Guard was bypassed, because
> there's no checks that listFiles/listStatus shows the newly committed files.
> Similarly, because MPU requests are initiated in S3AMultipartUploader,
> encryption settings are't picked up. Files being uploaded this way *are not
> being encrypted*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
