[
https://issues.apache.org/jira/browse/HADOOP-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164662#comment-16164662
]
ASF GitHub Bot commented on HADOOP-13600:
-----------------------------------------
Github user steveloughran commented on the issue:
https://github.com/apache/hadoop/pull/157
In HADOOP-13786 I'm wrapping every single s3 client call with retry policy,
then expanding the inconsistent client to generate more faults (initially
throttle, later connection setup/response parsing). I'd really like this work
to actually await that, as without it this code isn't going to be resilient to
large copies where you are much more likely to hit parallel IO. And we need to
make sure there's a good failure policy set up there
> S3a rename() to copy files in a directory in parallel
> -----------------------------------------------------
>
> Key: HADOOP-13600
> URL: https://issues.apache.org/jira/browse/HADOOP-13600
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.3
> Reporter: Steve Loughran
> Assignee: Sahil Takiar
> Attachments: HADOOP-13600.001.patch
>
>
> Currently a directory rename does a one-by-one copy, making the request
> O(files * data). If the copy operations were launched in parallel, the
> duration of the copy may be reducable to the duration of the longest copy.
> For a directory with many files, this will be significant
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
