Seems like a virtual memory limit. On Tue, Jun 14, 2016 at 3:35 PM, Jeremy Hanna (JIRA) <j...@apache.org> wrote:
> > [ > https://issues.apache.org/jira/browse/CASSANDRA-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330782#comment-15330782 > ] > > Jeremy Hanna edited comment on CASSANDRA-12009 at 6/14/16 10:34 PM: > -------------------------------------------------------------------- > > This is a limits problem. Often OOMs are not due to limited memory but > insufficient OS limits for the process. > > Check out the output from running > > {code} > ulimit -a > {code} > > or > > {code} > cat /proc/<cassandra pid>/limits > {code} > > What you'll likely need is to make sure your memlock setting is set to > unlimited. Also recommended is to set nofiles to something like 100000 and > nproc to 32768. > > > > > was (Author: jeromatron): > This is a limits problem. > > Check out the output from running > > {code} > ulimit -a > {code} > > or > > {code} > cat /proc/<cassandra pid>/limits > {code} > > What you'll likely need is to make sure your memlock setting is set to > unlimited. Also recommended is to set nofiles to something like 100000 and > nproc to 32768. > > > > > Nodetool scrub crashes: java.lang.OutOfMemoryError: Map failed > > -------------------------------------------------------------- > > > > Key: CASSANDRA-12009 > > URL: > https://issues.apache.org/jira/browse/CASSANDRA-12009 > > Project: Cassandra > > Issue Type: Bug > > Environment: # java -version > > openjdk version "1.8.0_91" > > OpenJDK Runtime Environment (build 1.8.0_91-b14) > > OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) > > # uname -a > > Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed > Jun 1 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > > # rpm -qa datastax-ddc > > datastax-ddc-3.5.0-1.noarch > > Reporter: Victor Trac > > > > Running nodetool scrub fails after a day or two. Originally, the > instance had 8GB of RAM with 4GB of Xmx/Xms. After seeing the OOM error > several times, I upgraded the box to a 24GB RAM box, giving 20GB to Xms/Xmx > to see if it would finish the scrub operation. Both cases failed with the > same error as below (tried on both the CMS and G1 GC collectors): > > {code} > > INFO [Service Thread] 2016-06-14 15:01:17,968 StatusLogger.java:106 - > system_traces.events 0,0 > > DEBUG [GossipTasks:1] 2016-06-14 15:01:18,019 Gossiper.java:336 - > Convicting /10.240.0.4 with status removed - alive false > > WARN [CompactionExecutor:69] 2016-06-14 15:01:18,151 > OutputHandler.java:57 - Error reading row (stacktrace follows): > > org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed > > at > org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873) > [apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81) > [apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366) > [apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314) > [apache-cassandra-3.5.0.jar:3.5.0] > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_91] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_91] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_91] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > > Caused by: java.io.IOException: Map failed > > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) > ~[na:1.8.0_91] > > at > org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > ... 23 common frames omitted > > Caused by: java.lang.OutOfMemoryError: Map failed > > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91] > > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_91] > > ... 24 common frames omitted > > WARN [CompactionExecutor:69] 2016-06-14 15:01:18,152 > OutputHandler.java:52 - Row starting at position 31397042159 is unreadable; > skipping to next > > WARN [CompactionExecutor:69] 2016-06-14 15:01:18,257 > OutputHandler.java:57 - Error reading row (stacktrace follows): > > org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed > > at > org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873) > [apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81) > [apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366) > [apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314) > [apache-cassandra-3.5.0.jar:3.5.0] > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_91] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_91] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_91] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > > Caused by: java.io.IOException: Map failed > > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) > ~[na:1.8.0_91] > > at > org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > ... 23 common frames omitted > > Caused by: java.lang.OutOfMemoryError: Map failed > > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91] > > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_91] > > ... 24 common frames omitted > > WARN [CompactionExecutor:69] 2016-06-14 15:01:18,258 > OutputHandler.java:52 - Row starting at position 31397303324 is unreadable; > skipping to next > > WARN [CompactionExecutor:69] 2016-06-14 15:01:18,363 > OutputHandler.java:57 - Error reading row (stacktrace follows): > > org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed > > at > org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) > ~[apache-cassandra-3.5.0.jar:3.5Gq > > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873) > [apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81) > [apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366) > [apache-cassandra-3.5.0.jar:3.5.0] > > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314) > [apache-cassandra-3.5.0.jar:3.5.0] > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_91] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_91] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_91] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > > Caused by: java.io.IOException: Map failed > > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) > ~[na:1.8.0_91] > > at > org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.5.0.jar:3.5.0] > > ... 23 common frames omitted > > Caused by: java.lang.OutOfMemoryError: Map failed > > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91] > > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_91] > > ... 24 common frames omitted > > WARN [CompactionExecutor:69] 2016-06-14 15:01:18,363 > OutputHandler.java:52 - Row starting at position 31397313498 is unreadable; > skipping to next > > {code} > > > > -- > This message was sent by Atlassian JIRA > (v6.3.4#6332) >
