> One thing that could fit the pattern you describe would be Solr caches > filling up and getting you too close to your JVM or memory limit
This [uncommitted] issue would solve that problem by allowing the GC to collect caches that become too large, though in practice, the cache setting would need to be fairly large for an OOM to occur from them: https://issues.apache.org/jira/browse/SOLR-1513 On Thu, Feb 16, 2012 at 7:14 PM, Bryan Loofbourrow <bloofbour...@knowledgemosaic.com> wrote: > A couple of thoughts: > > We wound up doing a bunch of tuning on the Java garbage collection. > However, the pattern we were seeing was periodic very extreme slowdowns, > because we were then using the default garbage collector, which blocks > when it has to do a major collection. This doesn't sound like your > problem, but it's something to be aware of. > > One thing that could fit the pattern you describe would be Solr caches > filling up and getting you too close to your JVM or memory limit. For > example, if you have large documents, and have defined a large document > cache, that might do it. > > I found it useful to point jconsole (free with the JDK) at my JVM, and > watch the pattern of memory usage. If the troughs at the bottom of the GC > cycles keep rising, you know you've got something that is continuing to > grab more memory and not let go of it. Now that our JVM is running > smoothly, we just see a sawtooth pattern, with the troughs approximately > level. When the system is under load, the frequency of the wave rises. Try > it and see what sort of pattern you're getting. > > -- Bryan > >> -----Original Message----- >> From: Matthias Käppler [mailto:matth...@qype.com] >> Sent: Thursday, February 16, 2012 7:23 AM >> To: solr-user@lucene.apache.org >> Subject: Frequent garbage collections after a day of operation >> >> Hey everyone, >> >> we're running into some operational problems with our SOLR production >> setup here and were wondering if anyone else is affected or has even >> solved these problems before. We're running a vanilla SOLR 3.4.0 in >> several Tomcat 6 instances, so nothing out of the ordinary, but after >> a day or so of operation we see increased response times from SOLR, up >> to 3 times increases on average. During this time we see increased CPU >> load due to heavy garbage collection in the JVM, which bogs down the >> the whole system, so throughput decreases, naturally. When restarting >> the slaves, everything goes back to normal, but that's more like a >> brute force solution. >> >> The thing is, we don't know what's causing this and we don't have that >> much experience with Java stacks since we're for most parts a Rails >> company. Are Tomcat 6 or SOLR known to leak memory? Is anyone else >> seeing this, or can you think of a reason for this? Most of our >> queries to SOLR involve the DismaxHandler and the spatial search query >> components. We don't use any custom request handlers so far. >> >> Thanks in advance, >> -Matthias >> >> -- >> Matthias Käppler >> Lead Developer API & Mobile >> >> Qype GmbH >> Großer Burstah 50-52 >> 20457 Hamburg >> Telephone: +49 (0)40 - 219 019 2 - 160 >> Skype: m_kaeppler >> Email: matth...@qype.com >> >> Managing Director: Ian Brotherston >> Amtsgericht Hamburg >> HRB 95913 >> >> This e-mail and its attachments may contain confidential and/or >> privileged information. If you are not the intended recipient (or have >> received this e-mail in error) please notify the sender immediately >> and destroy this e-mail and its attachments. Any unauthorized copying, >> disclosure or distribution of this e-mail and its attachments is >> strictly forbidden. This notice also applies to future messages.
