[
https://issues.apache.org/jira/browse/HADOOP-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-17937:
------------------------------------
Summary: ITestS3ADeleteFilesOneByOne. testBulkRenameAndDelete OOM: Direct
buffer memory (was: .ITestS3ADeleteFilesOneByOne. testBulkRenameAndDelete OOM:
Direct buffer memory)
> ITestS3ADeleteFilesOneByOne. testBulkRenameAndDelete OOM: Direct buffer memory
> ------------------------------------------------------------------------------
>
> Key: HADOOP-17937
> URL: https://issues.apache.org/jira/browse/HADOOP-17937
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
> Environment: fs.s3a.fast.upload.buffer = "bytebuffer"
> Reporter: Steve Loughran
> Priority: Minor
>
> on a test setup with bytebuffer, the parallel zero-byte file create phase
> OOMed
> fs.s3a.fast.upload.buffer = "bytebuffer" [core-site.xml]
> fs.s3a.fast.upload.active.blocks = "8" [core-site.xml]
> fs.s3a.multipart.size = "32M" [core-site.xml]
> Root cause: bytebuffer is being allocated on block creation, so every empty
> file took up 32MB of off-heap storage only for this to be released unused in
> close()
> If this allocation was postponed until the first write(), then empty files
> wouldn't need any memory allocation. Do the same on-demand creation for byte
> arrays and filesystem would also have benefits.
> this has implications for HADOOP-17195, which has abfs using a fork of the
> buffering code
> changing the code there to be on-demand would be a good incentive for s3a to
> adopt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
