Another thing you may wish to ponder is this blog entry from Mike McCandless: http://blog.mikemccandless.com/2011/04/just-say-no-to-swapping.html
In it, he discusses the poor interaction between OS swapping, and long-neglected allocations in a JVM. You're on Linux, which has decent control over swapping decisions, so you may find that a tweak is in order, especially if you can discover evidence that the hard drive is being worked hard during GC. If the problem exists, it might be especially pronounced in your large JVM. I have no direct evidence of thrashing during GC (I am not sure how to go about gathering such evidence), but I have seen, on a Windows machine, a Tomcat running Solr refuse to shut down for many minutes, while a Resource Monitor session reports that that same Tomcat process is frantically reading from the page file the whole time. So there is something besides plausibility to the idea. -- Bryan > -----Original Message----- > From: Mou [mailto:mouna...@gmail.com] > Sent: Monday, July 16, 2012 9:09 PM > To: solr-user@lucene.apache.org > Subject: Re: Using Solr 3.4 running on tomcat7 - very slow search > > Thanks Brian. Excellent suggestion. > > I haven't used VisualVM before but I am going to use it to see where CPU > is > going. I saw that CPU is overly used. I haven't seen so much CPU use in > testing. > Although I think GC is not a problem, splitting the jvm per shard would be > a good idea. > > > On Mon, Jul 16, 2012 at 9:44 PM, Bryan Loofbourrow [via Lucene] < > ml-node+s472066n3995446...@n3.nabble.com> wrote: > > > 5 min is ridiculously long for a query that used to take 65ms. That > ought > > to be a great clue. The only two things I've seen that could cause that > > are thrashing, or GC. Hard to see how it could be thrashing, given your > > hardware, so I'd initially suspect GC. > > > > Aim VisualVM at the JVM. It shows how much CPU goes to GC over time, in > a > > nice blue line. And if it's not GC, try out its Sampler tab, and see > where > > the CPU is spending its time. > > > > FWIW, when asked at what point one would want to split JVMs and shard, > on > > the same machine, Grant Ingersoll mentioned 16GB, and precisely for GC > > cost reasons. You're way above that. Maybe multiple JVMs and sharding, > > even on the same machine, would serve you better than a monster 70GB > JVM. > > > > -- Bryan > > > > > -----Original Message----- > > > From: Mou [mailto:[hidden > email]<http://user/SendEmail.jtp?type=node&node=3995446&i=0>] > > > > > Sent: Monday, July 16, 2012 7:43 PM > > > To: [hidden > email]<http://user/SendEmail.jtp?type=node&node=3995446&i=1> > > > Subject: Using Solr 3.4 running on tomcat7 - very slow search > > > > > > Hi, > > > > > > Our index is divided into two shards and each of them has 120M docs , > > > total > > > size 75G in each core. > > > The server is a pretty good one , jvm is given memory of 70G and about > > > same > > > is left for OS (SLES 11) . > > > > > > We use all dynamic fields except th eunique id and are using long > > queries > > > but almost all of them are filter queires, Each query may have 10 -30 > fq > > > parameters. > > > > > > When I tested the index ( same size) but with max heap size 40 G, > > queries > > > > > were blazing fast. I used solrmeter to load test and it was happily > > > serving > > > 12000 queries or more per min with avg 65 ms qtime.We had an excellent > > > filtercache hit ratio. > > > > > > This index is only used for searching and being replicated every 7 sec > > > from > > > the master. > > > > > > But now in production server it is horribly slow and taking 5 > > mins(qtime) > > > > > to > > > return a query ( same query). > > > What could go wrong? > > > > > > Really appreciate your suggestions on debugging this thing.. > > > > > > > > > > > > -- > > > View this message in context: > http://lucene.472066.n3.nabble.com/Using- > > > Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436.html > > > Sent from the Solr - User mailing list archive at Nabble.com. > > > > > > ------------------------------ > > If you reply to this email, your message will be added to the > discussion > > below: > > > > http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7- > very-slow-search-tp3995436p3995446.html > > To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow > search, click > > > here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=uns > ubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg > 1MTA5MTUw> > > . > > > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=mac > ro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespace > s.BasicNamespace-nabble.view.web.template.NabbleNamespace- > nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21na > bble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml- > send_instant_email%21nabble%3Aemail.naml> > > > > > -- > View this message in context: http://lucene.472066.n3.nabble.com/Using- > Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995449.html > Sent from the Solr - User mailing list archive at Nabble.com.
