Hi Constantijn,
yes I use Linux 64bit and thanks for the help.
Bernd
Am 31.05.2011 12:22, schrieb Constantijn Visinescu:
Hi Bernd,
I'm assuming Linux here, if you're running something else these
instructions might differ slightly.
First get a heap dump with:
jmap -heap:format=b,file=/path/to/generate/heapdumpfile.hprof 1234
with 1234 being the PID (process id) of the JVM
After you get a Heap dump you can analyze it with Eclipse MAT (Memory
Analyzer Tool).
Just a heads up if you're doing this in production: the JVM will
freeze completely while generating the heap dump, which will seem like
a giant stop the world GC with a 10GB heap.
Good luck with finding out what's eating your memory!
Constantijn
P.S.
Sorry about altering the subject line, but the spam assassin used by
the mailing list was rejecting my post because it had replication in
the subject line. hope it doesn't mess up the thread.
On Tue, May 31, 2011 at 8:43 AM, Bernd Fehling
<bernd.fehl...@uni-bielefeld.de> wrote:
Some more info,
after one week the servers have the following status:
Master (indexing only)
+ looks good and has heap size of about 6g from 10g OldGen
+ has loaded meanwhile 2 times the index from scratch via DIH
+ has added new documents into existing index via DIH
+ has optimized and replicated
+ no full GC within one week
Slave A (search only) Online
- looks bad and has heap size of 9.5g from 10g OldGen
+ was replicated
- several full GC
Slave B (search only) Backup
+ looks good has heap size of 4 g from 10g OldGen
+ was replicated
+ no full GC within one week
Conclusion:
+ DIH, processing, indexing, replication are fine
- the search is crap and "eats up" OldGen heap which can't be
cleaned up by full GC. May be memory leaks or what ever...
Due to this Solr 3.1 can _NOT_ be recommended as high-availability,
high-search-load search engine because of unclear heap problems
caused by the search. The search is "out of the box", so no
self produced programming errors.
Any tools available for JAVA to analyze this?
(like valgrind or electric fence for C++)
Is it possible to analyze a heap dump produced with jvisualvm?
Which tools?
Bernd
Am 30.05.2011 15:51, schrieb Bernd Fehling:
Dear list,
after switching from FAST to Solr I get the first _real_ data.
This includes search times, memory consumption, perfomance of solr,...
What I recognized so far is that something eats up my OldGen and
I assume it might be replication.
Current Data:
one master - indexing only
two slaves - search only
over 28 million docs
single instance
single core
index size 140g
current heap size 16g
After startup I have about 4g heap in use and about 3.5g of OldGen.
After one week and some replications OldGen is filled close to 100
percent.
If I start an optimize under this condition I get OOM of heap.
So my assumption is that something is eating up my heap.
Any idea how to trace this down?
May be a memory leak somewhere?
Best regards
Bernd
--
*************************************************************
Bernd Fehling Universitätsbibliothek Bielefeld
Dipl.-Inform. (FH) Universitätsstr. 25
Tel. +49 521 106-4060 Fax. +49 521 106-4052
bernd.fehl...@uni-bielefeld.de 33615 Bielefeld
BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
