> IndexSchema is using 62% of the memory
That seems odd. Can you see what objects are taking all the RAM in
the
IndexSchema?
We investigated this and found out that a dictionary was loaded for
each core, taking loads of memory. We the the config to
shareSchema=true. The memory usage decreased a lot and Solr is crashing
less often, but the problem still exists. We sometimes see a "GC
overhead limit exceeded" log entry now.
We made a new memory dump. It's about 4GB. The strange this is: Eclipse
Memory Analyzer talks about "Size: 799,2 MB". It seems the rest is in
the "Unreachable Objects" (2,5 GB). The "Unreachable Objects" are full
of byte[] and BytesRef objects:
https://www.dropbox.com/s/6kysc21rkmr67r7/Screenshot%202014-03-07%2015.38.51.png
I think this is the memory leak?
We'd need to actually see a large chunk of the end of the actual
logfile.
I put the log here (anonymised some shard names):
https://www.dropbox.com/s/0seosviys5wrvzh/catalina.log
Are there any messages in the operating system logs?
No, not at all.
Full details about the computer, operating system,
Dell PowerEdge servers, 16GB RAM, Debian Wheezy
Solr startup options
-server -verbose:gc -Xloggc:/var/log/jvm.log -Xmx4096m
-Dcom.sun.management.jmxremote -Djava.awt.headless=true
-DzkHost=192.168.40.30:2181,192.168.40.33:2181,192.168.40.37:2181/solr
and your index
About 70 cores, 5 servers, 12GB indexes in total (every core has 2
shards, so it's 6 GB per server). The most used schema is:
https://www.dropbox.com/s/6fhlvsh6v1rxyck/schema.xml
Thanks,
Martin
