Hmm.. About <1>: Yep, GC is one of the "more art than science" bits of Java/Solr. Siiiggggh.
About <2>: that's what autowarming is about. Particularly the filterCache and queryResultCache. My guess is that you have the autowarm count on those two caches set to zero. Try setting it to some modest number like 16 or 32. The whole _point_ of those parameters is to smooth out these kinds of spikes. Additionally, the newSearcher event (also in solrconfig.xml) is explicitly intended to allow you to hard-code queries that fill the internal caches as well as the mmap OS memory from disk, people include facets, sorts and the like in that event. It's fired every time a new searcher is opened (i.e. whenever you commit and open a new searcher)... FirstSearcher is for restarts. The difference is that newSearcher presumes Solr has been running for a while and the autowarm counts have something to work from. OTOH, when you start Solr there's no history to autowarm so firstSeracher can be quite a bit more complex than newSearcher. Practically, most people just copy newSearcher into firstSearcher on the assumption that restarting Solr is pretty rare..... about <3> MMap stuff will be controlled by the OS I think. I actually worked with a much more primitive system at one point that would be dog-slow during off-hours. Someone wrote an equivalent of a cron job to tickle the app upon occasion to prevent periodic slowness..... for a nauseating set of details about hard and soft commits, see: https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Best, Erick On Sat, Sep 24, 2016 at 11:35 AM, Rallavagu <rallav...@gmail.com> wrote: > > > On 9/22/16 5:59 AM, Shawn Heisey wrote: >> >> On 9/22/2016 5:46 AM, Muhammad Zahid Iqbal wrote: >>> >>> Did you find any solution to slow searches? As far as I know jetty >>> container default configuration is bit slow for large production >>> environment. >> >> >> This might be true for the default configuration that comes with a >> completely stock jetty downloaded from eclipse.org, but the jetty >> configuration that *Solr* ships with is adequate for just about any Solr >> installation. The Solr configuration may require adjustment as the >> query load increases, but the jetty configuration usually doesn't. >> >> Thanks, >> Shawn >> > > It turned out to be a "sequence of performance testing sessions" in order to > locate slowness. Though I am not completely done with it, here are my > finding so far. We are using NRT configuration (warm up count to 0 for > caches and NRTCachingDirectoryFactory for index directory) > > 1. Essentially, solr searches (particularly with edismax and relevance) > generate lot of "garbage" that makes GC activity to kick in more often. This > becomes even more when facets are included. This has huge impact on QTimes > (I have 12g heap and configured 6g to NewSize). > > 2. After a fresh restart (or core reload) when searches are performed, Solr > would initially "populate" mmap entries and this is adding to total QTimes > (I have made sure that index files are cached at filesystem layer using > vmtouch - https://hoytech.com/vmtouch). When run the same test again with > mmap entries populated from previous tests, it shows improved QTimes > relative to previous test. > > 3. Seems the populated mmap entries are flushed away after certain idle time > (not sure if it is controlled by Solr or underlying OS). This will make > subsequent searches to fetch from "disk" (even though the disk items are > cached by OS). > > So, what I am gonna try next is to tune the field(s) for facets to reduce > the index size if possible. Though I am not sure, if it will have impact but > would attempt to change the "caches" even though they will be invalidated > after a softCommit (every 10 minutes in my case). > > Any other tips/clues/suggestions are welcome. Thanks. >
