Re: MMapDirectory failed to map a 23G compound index segment

[François Schiettecatte]

My memory of this is a little rusty but isn't mmap also limited by mem + swap 
on the box? What does 'free -g' report?

François

On Sep 7, 2011, at 12:25 PM, Rich Cariens wrote:

> Ahoy ahoy!
> 
> I've run into the dreaded OOM error with MMapDirectory on a 23G cfs compound
> index segment file. The stack trace looks pretty much like every other trace
> I've found when searching for OOM & "map failed"[1]. My configuration
> follows:
> 
> Solr 1.4.1/Lucene 2.9.3 (plus
> SOLR-1969<https://issues.apache.org/jira/browse/SOLR-1969>
> )
> CentOS 4.9 (Final)
> Linux 2.6.9-100.ELsmp x86_64 yada yada yada
> Java SE (build 1.6.0_21-b06)
> Hotspot 64-bit Server VM (build 17.0-b16, mixed mode)
> ulimits:
>    core file size     (blocks, -c)     0
>    data seg size    (kbytes, -d)     unlimited
>    file size     (blocks, -f)     unlimited
>    pending signals    (-i)     1024
>    max locked memory     (kbytes, -l)     32
>    max memory size     (kbytes, -m)     unlimited
>    open files    (-n)     256000
>    pipe size     (512 bytes, -p)     8
>    POSIX message queues     (bytes, -q)     819200
>    stack size    (kbytes, -s)     10240
>    cpu time    (seconds, -t)     unlimited
>    max user processes     (-u)     1064959
>    virtual memory    (kbytes, -v)     unlimited
>    file locks    (-x)     unlimited
> 
> Any suggestions?
> 
> Thanks in advance,
> Rich
> 
> [1]
> ...
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(Unknown
> Source)
> at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(Unknown
> Source)
> at org.apache.lucene.store.MMapDirectory.openInput(Unknown Source)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(Unknown Source)
> 
> at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> at org.apache.lucene.index.DirectoryReader.<init>(Unknown Source)
> at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(Unknown Source)
> at org.apache.lucene.index.DirectoryReader$1.doBody(Unknown Source)
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Unknown
> Source)
> at org.apache.lucene.index.DirectoryReader.open(Unknown Source)
> at org.apache.lucene.index.IndexReader.open(Unknown Source)
> ...
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> ...