[
https://issues.apache.org/jira/browse/HADOOP-13664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526069#comment-15526069
]
Steve Loughran commented on HADOOP-13664:
-----------------------------------------
Example log of some transients. I suspect that the two calls were actually
back-to-back, with a 60s timeout in the connect; that would correspond with the
interval of 62 between the events.
{code}
test_050_readHugeFile: Reading /tests3a/scale/hugefile
2016-09-27 13:52:32,540 [JUnit] INFO s3a.S3AFileSystem
(S3AInputStream.java:onReadFailure(334)) - Got exception while trying to read
from stream s3a://hwdev-steve-ireland/tests3a/scale/hugefile trying to recover:
java.net.SocketException: Operation timed out
2016-09-27 13:53:34,868 [JUnit] INFO s3a.S3AFileSystem
(S3AInputStream.java:onReadFailure(334)) - Got exception while trying to read
from stream s3a://hwdev-steve-ireland/tests3a/scale/hugefile trying to recover:
java.net.SocketException: Operation timed out
{code}
A timeout failure like this is almost self recovering from, and given that any
socket failure is likely to be followed immediately by a reconnect, the current
policy may be sufficient. But other problems (e.g transient DNS failure) aren't
so forgiving.
> S3AInputStream to use a retry policy on read failures
> -----------------------------------------------------
>
> Key: HADOOP-13664
> URL: https://issues.apache.org/jira/browse/HADOOP-13664
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Priority: Minor
>
> {{S3AInputStream}} has some retry logic to handle failures on a read: log and
> retry. We should move this over to a (possibly hard coded RetryPolicy with
> some sleep logic, so that longer-than-just-transient read failures can be
> handled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
