[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124573#comment-16124573
]
Steve Loughran commented on HADOOP-14749:
-----------------------------------------
# We haven't any tests of IAM role with r/o bucket and r/w of DDB. I did
accidentally try to use s3guard and landsat, and don't remember it being a
particularly good experience. That's really something we need to test before
making any statement about its use. Hence my caution. I will state "this has
been undertested and may not work"
# I already have throttling in the fault injection in HADOOP-13786.,
[here|https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java].
Shows how we don't handle it at all well today. It's not an optimistic promise
then, more a "forward looking on a future merge of existing code" statement.
The hard part isn't this injection, it's reworking every since direct call of
the AWS SDK (other than via xfer manager) to handle the error. Hence
[S3ALambda|https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ALambda.java],
which I use for all the committer and upload calls in
[WriteOperationHelper|https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java].
We will need to go through every single API GET/HEAD call the same way for
resilience.
> review s3guard docs & code prior to merge
> -----------------------------------------
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch,
> HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch,
> HADOOP-14749-HADOOP-13345-004.patch, HADOOP-14749-HADOOP-13345-005.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
