[
https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743481#comment-15743481
]
Sean Mackrory commented on HADOOP-13826:
----------------------------------------
[~Thomas Demoor] Thanks. The more I think about it, the more I prefer .003
patch. I played around with my original approach (distinct resource pools for
different types of tasks) today. I restructured it so tasks were segregated
immediately upon being passed from TransferManager, instead of having layers of
shared queues on top of them. Even then, control tasks were able to saturate
their pool and deadlock it. But the unbounded pool you suggested fixed that
problem - but wanting to avoid unbounded pools is my main concern with the .003
patch anyway.
It's also still a bit kludgy trying to separate tasks based on internal Amazon
APIs (which I griped about this morning:
https://github.com/aws/aws-sdk-java/issues/939) and to a lesser extent when
S3AFastOutputStream still causes tasks to be submitted to the executor wrapped
in other types of Callable.
> S3A Deadlock in multipart copy due to thread pool limits.
> ---------------------------------------------------------
>
> Key: HADOOP-13826
> URL: https://issues.apache.org/jira/browse/HADOOP-13826
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.3
> Reporter: Sean Mackrory
> Assignee: Sean Mackrory
> Priority: Critical
> Attachments: HADOOP-13826.001.patch, HADOOP-13826.002.patch,
> HADOOP-13826.003.patch
>
>
> In testing HIVE-15093 we have encountered deadlocks in the s3a connector. The
> TransferManager javadocs
> (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html)
> explain how this is possible:
> {quote}It is not recommended to use a single threaded executor or a thread
> pool with a bounded work queue as control tasks may submit subtasks that
> can't complete until all sub tasks complete. Using an incorrectly configured
> thread pool may cause a deadlock (I.E. the work queue is filled with control
> tasks that can't finish until subtasks complete but subtasks can't execute
> because the queue is filled).{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
