cene.apache.org
> Sent: Friday, February 29, 2008 11:38:31 AM
> Subject: Re: Optimization taking days/weeks
>
>
> We are a bit concerned regarding the index size. At least no response (so
> far) as indicated that the size is unmanagable. We killed the process -
> will mov
cene.apache.org
> Sent: Friday, February 29, 2008 11:28:32 AM
> Subject: RE: Optimization taking days/weeks
>
>
> We will review the java settings. The current settings are a bit low - but
> the indexed typically does not reach even 50% of the allocated 1024MB Max
> Heap.
>
o: solr-user@lucene.apache.org
> Sent: Friday, February 29, 2008 11:22:04 AM
> Subject: Re: Optimization taking days/weeks
>
>
> Yes indeed - it was spending all of its time in garbage collection. We will
> be moving to Java6.
> Thanks for your suggestion.
>
We are a bit concerned regarding the index size. At least no response (so
far) as indicated that the size is unmanagable. We killed the process -
will move to Java6 - and will use vmstat to monitor the new optimization
process.
At what index size would you begin to worry? Or is it a combinatio
We will review the java settings. The current settings are a bit low - but
the indexed typically does not reach even 50% of the allocated 1024MB Max
Heap.
Yes the index is large - only 3 fields are stored - and I have set the
positionIncrementGap to 50 (down from 100) in an attempt to reduce ind
Yes indeed - it was spending all of its time in garbage collection. We will
be moving to Java6.
Thanks for your suggestion.
Frances
Yonik Seeley wrote:
>
> Have you checked if this is due to running out of heap memory?
> When that happens, the garbage collector can start taking a lot of CPU.
This sounds too familiar...
>java settings used - java -Xmx1024M -Xms1024M
Sounds like your settings are pretty low... if you're using 64bit JVM, you
should be able to set
these much higher, maybe give it like 8gb.
Another thing, you may want to look at reducing the index size... is there a
That's a tiny little index there ;) Circa 100GB?
What do you see if you run vmstat 2 while the optimization is happening?
Non-idle CPU? A pile of IO? Is there a reason for such a small heap on a
machine with 32GB of RAM?
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
--
Have you checked if this is due to running out of heap memory?
When that happens, the garbage collector can start taking a lot of CPU.
If you are using a Java6 JVM, it should have management enabled by
default and you should be able to connect to it via jconsole and
check.
-Yonik
On Thu, Feb 28,
