You were right. I've attached VisualVM to the process and forced a System.gc(): used memory went down to near 1.8gb.
So, i don't understand VisualVM dump reports. It said that all those char[] references have a SolrDispatchFilter instance as CG root. Another example (1M+ references with the exact same size 32792 [default buffer size]): https://dl.dropboxusercontent.com/u/4740964/solr_references.png Can you give any hint on how to read this kind of reference graph? Thanks! On Tue, Aug 20, 2013 at 7:20 PM, Samuel García Martínez < samuelgmarti...@gmail.com> wrote: > Thanks for the quick answer. > > We are experiencing slower indexing speed (x2/x2.5 time consumed) and the > memory consumed during this process oscilates between 10g (max allowed, so > JVM performs full gc's) and 8gb, but never goes under 8gb. > > I'll try your suggestions and see how it goes. > > Thanks! > El 20/08/2013 13:29, "Erick Erickson" <erickerick...@gmail.com> escribió: > > Hmmm, first time I've seen this, but are you seeing a problem >> or is this just a theoretical concern? There are some >> references that are held around until GC, these could be >> perfectly legitimate, unused but un-collected references. >> >> What I'd try before worrying much: >> 1> attach jconsole and press the GC button and see if lots of those >> go away. >> 2> _reduce_ the memory allocated to Solr and see if you can get by. >> In this case, reduce it to, say, 4G. If your memory consumption >> hits 4G and you continue running, you're probably seeing uncollected >> memory. >> >> Best >> Erick >> >> >> On Tue, Aug 20, 2013 at 4:24 AM, Samuel García Martínez < >> samuelgmarti...@gmail.com> wrote: >> >> > Hi all, we are facing a high memory usage in our Solr 3.6 master (not in >> > the slaves) even during "idle" (non indexing) periods. >> > >> > container: Tomcat 6.0.29 >> > maxthreads: 1.5k (i think this setting is wrong, 300 would be enough) >> > solr: solr 3.6.0 >> > setup: multitenant environment with 120+ cores and near 5M docs spread >> over >> > all indexes. >> > uptime: 100+ days >> > qps on this machine: 10 qps >> > used heap: 8gb >> > JVM params: >> > >> > >> -Djava.util.logging.config.file=/opt/opensearch/tomcat/conf/logging.properties >> > -Dmaster=true >> > -Dmaster.url="" >> > -Xms5G >> > -Xmx10G >> > -XX:MaxPermSize=256m >> > -XX:SurvivorRatio=5 >> > -XX:NewRatio=2 >> > -XX:+UseParNewGC >> > -XX:+UseConcMarkSweepGC >> > -Dcom.sun.management.jmxremote >> > -Dcom.sun.management.jmxremote.port=22222 >> > -Dcom.sun.management.jmxremote.authenticate=false >> > -Dcom.sun.management.jmxremote.ssl=false >> > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >> > -Djava.endorsed.dirs=/opt/opensearch/tomcat/endorsed >> > -Dcatalina.base=/opt/opensearch/tomcat >> > -Dcatalina.home=/opt/opensearch/tomcat >> > -Djava.io.tmpdir=/opt/opensearch/tomcat/temp >> > >> > >> > I think we have a high memory usage due to the next report: >> > char[] 3.1M instances 5.3Gb -----> this is what i'm talking about. >> > LinkedHashMap$Entry 2M instances 121Mb >> > String 1.48M 53Mb >> > >> > >> > Checking for "cg roots" at all these instances, i found that almost all >> > these references are contained in ISOLatin1AccentFilter.output -> >> > TokenStreamImpl - Map$Entry -> Map -> CloseableThreadLocal -> >> > TokenizerChain >> > >> > Do we had to add any CMS param to the JVM params? Or is this a memory >> leak >> > due to the ThreadLocal's? >> > >> > I verified that those char[] didn't belong to FieldCache.default, so >> this >> > high memory usage is not due the faceting and high cadinality values. >> > >> > PS: we reduced the number of threads and memory (and char[] instances) >> > decreased significantly. >> > -- >> > Un saludo, >> > Samuel García. >> > >> > -- Un saludo, Samuel García.
