Dear Mike,
thanks for your your reply.
Just a couple of minutes we found a solution or - to be honest - where
we went wrong.
Our failure was the use of ulimit. We missed, that ulimit sets the vmem
for each shell seperatly. So we set 'ulimit -v unlimited' on a shell,
thinking that we've done the job correctly.
As we recognized our mistake, we added 'ulimit -v unlimited' to our
init-Skript of the tomcat-instance and now it looks like everything
works as aspected.
Need some further testing with the java versions, but I'm quite optimistic.
Best regards
Ralf
Am 22.09.2011 14:46, schrieb Michael McCandless:
Are you sure you are using a 64 bit JVM?
Are you sure you really changed your vmem limit to unlimited? That
should have resolved the OOME from mmap.
Or: can you run "cat /proc/sys/vm/max_map_count"? This is a limit on
the total number of maps in a single process, that Linux imposes. But
the default limit is usually high (64K), so it'd be surprising if you
are hitting that unless it's lower in your env.
The amount of [free] RAM on the machine should have no bearing on
whether mmap succeeds or fails; it's the available address space (32
bit is tiny; 64 bit is immense) and then any OS limits imposed.
Mike McCandless
http://blog.mikemccandless.com
On Thu, Sep 22, 2011 at 5:27 AM, Ralf Matulat<ralf.matu...@bundestag.de> wrote:
Good morning!
Recently we slipped into an OOME by optimizing our index. It looks like it's
regarding to the nio class and the memory-handling.
I'll try to describe the environment, the error and what we did to solve the
problem. Nevertheless, none of our approaches was successful.
The environment:
- Tested with both, SOLR 3.3& 3.4
- SuSE SLES 11 (X64)virtual machine with 16GB RAM
- ulimi: virtual memory 14834560 (14GB)
- Java: java-1_6_0-ibm-1.6.0-124.5
- Apache Tomcat/6.0.29
- Index Size (on filesystem): ~5GB, 1.1 million text documents.
The error:
First, building the index from scratch with a mysql DIH, with an empty
index-Dir works fine.
Building an index with&command=full-import, when the old segment files
still in place, fails with an OutOfMemoryException. Same as optimizing the
index.
Doing an optimize fails after some time with:
SEVERE: java.io.IOException: background merge hit exception:
_6p(3.4):Cv1150724 _70(3.4):Cv667 _73(3.4):Cv7 _72(3.4):Cv4 _71(3.4):Cv1
into _74 [optimize]
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2552)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2472)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:61)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:735)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:765)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264)
at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:89)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:710)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4378)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3917)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:762)
... 9 more
Then we changed mergeScheduler and mergePolicy to
<mergeScheduler class="org.apache.lucene.index.SerialMergeScheduler" />
<mergePolicy class="org.apache.lucene.index.LogByteSizeMergePolicy"/>
which lead to a slightly different error-message:
SEVERE: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:765)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:264)
at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
at
org.apache.lucene.index.TermVectorsReader.<init>(TermVectorsReader.java:85)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:221)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:710)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4248)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3917)
at
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2725)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2535)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2472)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:107)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:61)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:735)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:762)
... 33 more
Both seem to have in common the "sun.nio.ch.FileChannelImpl.map" running out
of memory.
We looked at "free" while doing the optimize and we saw, that the optimize
ate up all the free RAM. When no RAM was left, the exception was thrown. No
swapping etc. at all.
What we did to solve the problem:
We discovered a link from this list and tried the mentioned solutions:
http://www.mail-archive.com/solr-user@lucene.apache.org/msg54094.html
No success.
We set ulimit -v unlimited, changed mergePolicy (see above).
Last thing we tried: We decreased the RAM of the JVM. At least that worked.
The optimize finished.
But on a long run this is no solution. Our index is growing constantly and
RAM, which is sufficient today, maybe is not tomorrow. We wonder how the
memory management works. If you are running another process on that machine,
using a reasonable amount of RAM in a short period of time, an optimize
would fail when the optimize starts at the wrong point in time.
Or what are you doing, if you got an 25GB index? Do you need a +50GB RAM
Server to do an optimize? That does not make any sense and so there must be
something we can do about it - something we missed.
Maybe someone get some bright ideas?
Thanks a lot and best regards
Ralf
--
Ralf Matulat
Deutscher Bundestag
Platz der Republik 1
11011 Berlin
Referat IT 1 - Anwendungsadministration
ralf.matu...@bundestag.de
Tel.: 030 - 227 34260
