[
https://issues.apache.org/jira/browse/HADOOP-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17419870#comment-17419870
]
Steve Loughran commented on HADOOP-17937:
-----------------------------------------
{code}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 14.193
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[ERROR]
testBulkRenameAndDelete(org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne)
Time elapsed: 13.743 s <<< ERROR!
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at
org.apache.hadoop.util.DirectBufferPool.getBuffer(DirectBufferPool.java:61)
at
org.apache.hadoop.fs.s3a.S3ADataBlocks$ByteBufferBlockFactory.requestBuffer(S3ADataBlocks.java:526)
at
org.apache.hadoop.fs.s3a.S3ADataBlocks$ByteBufferBlockFactory.access$100(S3ADataBlocks.java:507)
at
org.apache.hadoop.fs.s3a.S3ADataBlocks$ByteBufferBlockFactory$ByteBufferBlock.<init>(S3ADataBlocks.java:571)
at
org.apache.hadoop.fs.s3a.S3ADataBlocks$ByteBufferBlockFactory.create(S3ADataBlocks.java:520)
at
org.apache.hadoop.fs.s3a.S3ADataBlocks$ByteBufferBlockFactory.create(S3ADataBlocks.java:507)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:207)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.<init>(S3ABlockOutputStream.java:185)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerCreateFile(S3AFileSystem.java:1640)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$create$6(S3AFileSystem.java:1560)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2250)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2269)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:1559)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1176)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1156)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1045)
at
org.apache.hadoop.fs.contract.ContractTestUtils.createFile(ContractTestUtils.java:636)
at
org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes.lambda$put$7(ITestPartialRenamesDeletes.java:840)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:87)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Code is creating 50 files in parallel, using path.getName() as the string.
Given its actually generating text, adopting an on-demand buffer constructor
isn't going to help. Possibly useful elsewhere tho'
> ITestS3ADeleteFilesOneByOne. testBulkRenameAndDelete OOM: Direct buffer memory
> ------------------------------------------------------------------------------
>
> Key: HADOOP-17937
> URL: https://issues.apache.org/jira/browse/HADOOP-17937
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
> Environment: fs.s3a.fast.upload.buffer = "bytebuffer"
> Reporter: Steve Loughran
> Priority: Minor
>
> on a test setup with bytebuffer, the parallel zero-byte file create phase
> OOMed
> fs.s3a.fast.upload.buffer = "bytebuffer" [core-site.xml]
> fs.s3a.fast.upload.active.blocks = "8" [core-site.xml]
> fs.s3a.multipart.size = "32M" [core-site.xml]
> Root cause: bytebuffer is being allocated on block creation, so every empty
> file took up 32MB of off-heap storage only for this to be released unused in
> close()
> If this allocation was postponed until the first write(), then empty files
> wouldn't need any memory allocation. Do the same on-demand creation for byte
> arrays and filesystem would also have benefits.
> this has implications for HADOOP-17195, which has abfs using a fork of the
> buffering code
> changing the code there to be on-demand would be a good incentive for s3a to
> adopt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
