[
https://issues.apache.org/jira/browse/HADOOP-18876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769351#comment-17769351
]
ASF GitHub Bot commented on HADOOP-18876:
-----------------------------------------
mukund-thakur commented on PR #6009:
URL: https://github.com/apache/hadoop/pull/6009#issuecomment-1736352041
Yes, byte buffers are faster but large file uploads with more parallelism
can cause OOM. That is the reason disk is default.
What kind of uploads you have performed? Could you please share file size,
block size and number of parallel threads.
CC @steveloughran
> ABFS: Change default from disk to bytebuffer for fs.azure.data.blocks.buffer
> ----------------------------------------------------------------------------
>
> Key: HADOOP-18876
> URL: https://issues.apache.org/jira/browse/HADOOP-18876
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
> Affects Versions: 3.3.6
> Reporter: Anmol Asrani
> Assignee: Anmol Asrani
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.6
>
>
> Change default from disk to bytebuffer for fs.azure.data.blocks.buffer.
> Gathered from multiple workload runs, the presented data underscores a
> noteworthy enhancement in performance. The adoption of ByteBuffer for
> *reading operations* exhibited a remarkable improvement of approximately
> *64.83%* when compared to traditional disk-based reading. Similarly, the
> implementation of ByteBuffer for *write operations* yielded a substantial
> efficiency gain of about {*}60.75%{*}. These findings underscore the
> consistent and substantial advantages of integrating ByteBuffer across a
> range of workload scenarios.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
