[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249093#comment-16249093
]
Genmao Yu commented on HADOOP-15027:
------------------------------------
[~wujinhu] Thanks for the work. Some comments follow:
1. I have doubt about the necessity of concurrent reading in one input stream.
IMHO, it is indeed helpful to accelerate reading data. However in distributed
scene, we have split job into multiple tasks, i.e. we have read data in
parallel in task level. In Aliyun environment, as far as I know one OSS client
may achieve tens of MB per second.
2. {{move thread pool from InputStream to FileSystem}}: can this lead to
blocking between multiple input stream? I mean if threads are working on slow
stuff for one abnormal file (?), other input streams should be waiting for a
long time.
3. expensive cost to use thread pool.
So my concern is the necessity to use concurrency mechanism in one input stream.
4. some code style warning: 80 line length limit.
5. possible reuse {{CacheItem}} ?
> Improvements for Hadoop read from AliyunOSS
> -------------------------------------------
>
> Key: HADOOP-15027
> URL: https://issues.apache.org/jira/browse/HADOOP-15027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
> Affects Versions: 3.0.0
> Reporter: wujinhu
> Assignee: wujinhu
> Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch
>
>
> Currently, read performance is poor when Hadoop reads from AliyunOSS. It
> needs about 1min to read 1GB from OSS.
> Class AliyunOSSInputStream uses single thread to read data from AliyunOSS,
> so we can refactor this by using multi-thread pre read to improve this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
