My colleague and I thought the same thing - that this is an O/S
configuration issue.
/proc/sys/vm/max_map_count = 65536
I honestly don't know how many segments were in the index. Our merge factor
is 10 and there were around 4.4 million docs indexed. The OOME was raised
when the MMapDirectory was
On Tue, Sep 20, 2011 at 12:32 PM, Michael McCandless
wrote:
>
> Or: is it possible you reopened the reader several times against the
> index (ie, after committing from Solr)? If so, I think 2.9.x never
> unmaps the mapped areas, and so this would "accumulate" against the
> system limit.
In order
Since you hit OOME during mmap, I think this is an OS issue not a JVM
issue. Ie, the JVM isn't running out of memory.
How many segments were in the unoptimized index? It's possible the OS
rejected the mmap because of process limits. Run "cat
/proc/sys/vm/max_map_count" to see how many mmaps are
Thanks. It's definitely repeatable and I may spend some time plumbing this
further. I'll let the list know if I find anything.
The problem went away once I optimized the index down to a single segment
using a simple IndexWriter driver. This was a bit strange since the
resulting index contained sim
I remember now: by memory-mapping one block of address space that big, the
garbage collector has problems working around it. If the OOM is repeatable,
you could try watching the app with jconsole and watch the memory spaces.
Lance
On Thu, Sep 8, 2011 at 8:58 PM, Lance Norskog wrote:
> Do you ne
Do you need to use the compound format?
On Thu, Sep 8, 2011 at 3:57 PM, Rich Cariens wrote:
> I should add some more context:
>
> 1. the problem index included several cfs segment files that were around
> 4.7G, and
> 2. I'm running four SOLR instances on the same box, all of which have
>
I should add some more context:
1. the problem index included several cfs segment files that were around
4.7G, and
2. I'm running four SOLR instances on the same box, all of which have
similiar problem indeces.
A colleague thought perhaps I was bumping up against my 256,000 open files
FWiW I optimized the index down to a single segment and now I have no
trouble opening an MMapDirectory on that index, even though the 23G cfx
segment file remains.
On Thu, Sep 8, 2011 at 4:27 PM, Rich Cariens wrote:
> Thanks for the response. "free -g" reports:
>
> totalused
Thanks for the response. "free -g" reports:
totalusedfreesharedbuffers
cached
Mem: 141 95 46 0
093
-/+ buffers/cache: 2 139
Swap: 3 0 3
2011/9/7 François Schiettecatte
>
My memory of this is a little rusty but isn't mmap also limited by mem + swap
on the box? What does 'free -g' report?
François
On Sep 7, 2011, at 12:25 PM, Rich Cariens wrote:
> Ahoy ahoy!
>
> I've run into the dreaded OOM error with MMapDirectory on a 23G cfs compound
> index segment file. Th
10 matches
Mail list logo
