[
https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906832#comment-14906832
]
Steve Loughran commented on HADOOP-11684:
-----------------------------------------
bq. Anyone have example where many threads would be entering S3AFilesystem to
do writes
any multi-threaded process with lots of workers committing the output of their
work to s3. Examples: MR, Tez, Spark work
HADOOP-11446 caught this with HBase FS snapshots.
bq. Since the behavior of these parameters has changed, should we add an entry
to release notes notifying folks of the new behavior?
s3a has still been in stabilisation phase, so I'm not so sure as to how much we
need to detail. the main thing is "do the changes consistute some form of
regression?". If they improve things, then they aren't. After all, currently
when the thread pool fills up, you are in trouble —so you do need to
overallocate
> S3a to use thread pool that blocks clients
> ------------------------------------------
>
> Key: HADOOP-11684
> URL: https://issues.apache.org/jira/browse/HADOOP-11684
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.0
> Reporter: Thomas Demoor
> Assignee: Thomas Demoor
> Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch,
> HADOOP-11684-003.patch
>
>
> Currently, if fs.s3a.max.total.tasks are queued and another (part)upload
> wants to start, a RejectedExecutionException is thrown.
> We should use a threadpool that blocks clients, nicely throtthling them,
> rather than throwing an exception. F.i. something similar to
> https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
