Are you sure you are using a 64 bit JVM? Are you sure you really changed your vmem limit to unlimited? That should have resolved the OOME from mmap.
Or: can you run "cat /proc/sys/vm/max_map_count"? This is a limit on the total number of maps in a single process, that Linux imposes. But the default limit is usually high (64K), so it'd be surprising if you are hitting that unless it's lower in your env. The amount of [free] RAM on the machine should have no bearing on whether mmap succeeds or fails; it's the available address space (32 bit is tiny; 64 bit is immense) and then any OS limits imposed. Mike McCandless http://blog.mikemccandless.com On Thu, Sep 22, 2011 at 5:27 AM, Ralf Matulat <ralf.matu...@bundestag.de> wrote: > Good morning! > Recently we slipped into an OOME by optimizing our index. It looks like it's > regarding to the nio class and the memory-handling. > I'll try to describe the environment, the error and what we did to solve the > problem. Nevertheless, none of our approaches was successful. > > The environment: > > - Tested with both, SOLR 3.3 & 3.4 > - SuSE SLES 11 (X64)virtual machine with 16GB RAM > - ulimi: virtual memory 14834560 (14GB) > - Java: java-1_6_0-ibm-1.6.0-124.5 > - Apache Tomcat/6.0.29 > > - Index Size (on filesystem): ~5GB, 1.1 million text documents. > > The error: > First, building the index from scratch with a mysql DIH, with an empty > index-Dir works fine. > Building an index with &command=full-import, when the old segment files > still in place, fails with an OutOfMemoryException. Same as optimizing the > index. > Doing an optimize fails after some time with: > > SEVERE: java.io.IOException: background merge hit exception: > _6p(3.4):Cv1150724 _70(3.4):Cv667 _73(3.4):Cv7 _72(3.4):Cv4 _71(3.4):Cv1 > into _74 [optimize] > at > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2552) > at > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2472) > at > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410) > at > org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85) > at > org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154) > at > org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107) > at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:61) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:735) > Caused by: java.io.IOException: Map failed > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:765) > at > org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264) > at > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216) > at > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:89) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115) > at > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:710) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4378) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3917) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) > Caused by: java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:762) > ... 9 more > > Then we changed mergeScheduler and mergePolicy to > > <mergeScheduler class="org.apache.lucene.index.SerialMergeScheduler" /> > <mergePolicy class="org.apache.lucene.index.LogByteSizeMergePolicy"/> > > which lead to a slightly different error-message: > > SEVERE: java.io.IOException: Map failed > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:765) > at > org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264) > at > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216) > at > org.apache.lucene.index.TermVectorsReader.<init>(TermVectorsReader.java:85) > at > org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:221) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117) > at > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:710) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4248) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3917) > at > org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37) > at > org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2725) > at > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2535) > at > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2472) > at > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410) > at > org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85) > at > org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154) > at > org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107) > at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:61) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:735) > Caused by: java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:762) > ... 33 more > > Both seem to have in common the "sun.nio.ch.FileChannelImpl.map" running out > of memory. > > We looked at "free" while doing the optimize and we saw, that the optimize > ate up all the free RAM. When no RAM was left, the exception was thrown. No > swapping etc. at all. > > > What we did to solve the problem: > > We discovered a link from this list and tried the mentioned solutions: > http://www.mail-archive.com/solr-user@lucene.apache.org/msg54094.html > > No success. > We set ulimit -v unlimited, changed mergePolicy (see above). > Last thing we tried: We decreased the RAM of the JVM. At least that worked. > The optimize finished. > > But on a long run this is no solution. Our index is growing constantly and > RAM, which is sufficient today, maybe is not tomorrow. We wonder how the > memory management works. If you are running another process on that machine, > using a reasonable amount of RAM in a short period of time, an optimize > would fail when the optimize starts at the wrong point in time. > Or what are you doing, if you got an 25GB index? Do you need a +50GB RAM > Server to do an optimize? That does not make any sense and so there must be > something we can do about it - something we missed. > > Maybe someone get some bright ideas? > Thanks a lot and best regards > Ralf >
