[
https://issues.apache.org/jira/browse/HADOOP-11183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14326422#comment-14326422
]
Steve Loughran commented on HADOOP-11183:
-----------------------------------------
h3. Exceptions
# we're going to need them translated into IOExceptions
# I'd like typed for the basic FileNotFound exception & permissions, though not
so typed that they lose data
h3. Statistics.
I don't have a good opinion here. For the swift stuff I didn't do enough
integration with Hadoop stats; I did collect counters on the various HTTP
verbs, including average time, which helped track down some throttling of
DELETE.
We can evolve statistics, especially if you highlight "unstable".
h3. Failures
Fail-fast would be my preference, unless there's some attempt to retry on a
transient outage. That might be a good option on some IOEs, (connection &c),
but not others (Authentication failures)
> Memory-based S3AOutputstream
> ----------------------------
>
> Key: HADOOP-11183
> URL: https://issues.apache.org/jira/browse/HADOOP-11183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.6.0
> Reporter: Thomas Demoor
> Assignee: Thomas Demoor
> Attachments: HADOOP-11183-004.patch, HADOOP-11183.001.patch,
> HADOOP-11183.002.patch, HADOOP-11183.003.patch, design-comments.pdf
>
>
> Currently s3a buffers files on disk(s) before uploading. This JIRA
> investigates adding a memory-based upload implementation.
> The motivation is evidently performance: this would be beneficial for users
> with high network bandwidth to S3 (EC2?) or users that run Hadoop directly on
> an S3-compatible object store (FYI: my contributions are made in name of
> Amplidata).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
