BTW, what is NRT? Dennis Gearon
Signature Warning ---------------- EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Fri, 9/17/10, Peter Sturge <peter.stu...@gmail.com> wrote: > From: Peter Sturge <peter.stu...@gmail.com> > Subject: Re: Tuning Solr caches with high commit rates (NRT) > To: solr-user@lucene.apache.org > Date: Friday, September 17, 2010, 2:18 AM > Hi, > > It's great to see such a fantastic response to this thread > - NRT is > alive and well! > > I'm hoping to collate this information and add it to the > wiki when I > get a few free cycles (thanks Erik for the heads up). > > In the meantime, I thought I'd add a few tidbits of > additional > information that might prove useful: > > 1. The first one to note is that the techniques/setup > described in > this thread don't fix the underlying potential for > OutOfMemory errors > - there can always be an index large enough to ask of its > JVM more > memory than is available for cache. > These techniques, however, mitigate the risk, and provide > an efficient > balance between memory use and search performance. > There are some interesting discussions going on for both > Lucene and > Solr regarding the '2 pounds of baloney into a 1 pound bag' > issue of > unbounded caches, with a number of interesting strategies. > One strategy that I like, but haven't found in discussion > lists is > auto-limiting cache size/warming based on available > resources (similar > to the way file system caches use free memory). This would > allow > caches to adjust to their memory environment as indexes > grow. > > 2. A note regarding lockType in solrconfig.xml for dual > Solr > instances: It's best not to use 'none' as a value for > lockType - this > sets the lockType to null, and as the source comments note, > this is a > recipe for disaster, so, use 'simple' instead. > > 3. Chris mentioned setting maxWarmingSearchers to 1 as a > way of > minimizing the number of onDeckSearchers. This is a prudent > move -- > thanks Chris for bringing this up! > > All the best, > Peter > > > > > On Tue, Sep 14, 2010 at 2:00 PM, Peter Karich <peat...@yahoo.de> > wrote: > > Peter Sturge, > > > > this was a nice hint, thanks again! If you are here in > Germany anytime I > > can invite you to a beer or an apfelschorle ! :-) > > I only needed to change the lockType to none in the > solrconfig.xml, > > disable the replication and set the data dir to the > master data dir! > > > > Regards, > > Peter Karich. > > > >> Hi Peter, > >> > >> this scenario would be really great for us - I > didn't know that this is > >> possible and works, so: thanks! > >> At the moment we are doing similar with > replicating to the readonly > >> instance but > >> the replication is somewhat lengthy and > resource-intensive at this > >> datavolume ;-) > >> > >> Regards, > >> Peter. > >> > >> > >>> 1. You can run multiple Solr instances in > separate JVMs, with both > >>> having their solr.xml configured to use the > same index folder. > >>> You need to be careful that one and only one > of these instances will > >>> ever update the index at a time. The best way > to ensure this is to use > >>> one for writing only, > >>> and the other is read-only and never writes to > the index. This > >>> read-only instance is the one to use for > tuning for high search > >>> performance. Even though the RO instance > doesn't write to the index, > >>> it still needs periodic (albeit empty) commits > to kick off > >>> autowarming/cache refresh. > >>> > >>> Depending on your needs, you might not need to > have 2 separate > >>> instances. We need it because the 'write' > instance is also doing a lot > >>> of metadata pre-write operations in the same > jvm as Solr, and so has > >>> its own memory requirements. > >>> > >>> 2. We use sharding all the time, and it works > just fine with this > >>> scenario, as the RO instance is simply another > shard in the pack. > >>> > >>> > >>> On Sun, Sep 12, 2010 at 8:46 PM, Peter Karich > <peat...@yahoo.de> > wrote: > >>> > >>> > >>>> Peter, > >>>> > >>>> thanks a lot for your in-depth > explanations! > >>>> Your findings will be definitely helpful > for my next performance > >>>> improvement tests :-) > >>>> > >>>> Two questions: > >>>> > >>>> 1. How would I do that: > >>>> > >>>> > >>>> > >>>>> or a local read-only instance that > reads the same core as the indexing > >>>>> instance (for the latter, you'll need > something that periodically refreshes - i.e. runs > commit()). > >>>>> > >>>>> > >>>> 2. Did you try sharding with your current > setup (e.g. one big, > >>>> nearly-static index and a tiny write+read > index)? > >>>> > >>>> Regards, > >>>> Peter. > >>>> > >>>> > >>>> > >>>>> Hi, > >>>>> > >>>>> Below are some notes regarding Solr > cache tuning that should prove > >>>>> useful for anyone who uses Solr with > frequent commits (e.g. <5min). > >>>>> > >>>>> Environment: > >>>>> Solr 1.4.1 or branch_3x trunk. > >>>>> Note the 4.x trunk has lots of neat > new features, so the notes here > >>>>> are likely less relevant to the 4.x > environment. > >>>>> > >>>>> Overview: > >>>>> Our Solr environment makes extensive > use of faceting, we perform > >>>>> commits every 30secs, and the indexes > tend be on the large-ish side > >>>>> (>20million docs). > >>>>> Note: For our data, when we commit, we > are always adding new data, > >>>>> never changing existing data. > >>>>> This type of environment can be tricky > to tune, as Solr is more geared > >>>>> toward fast reads than frequent > writes. > >>>>> > >>>>> Symptoms: > >>>>> If anyone has used faceting in > searches where you are also performing > >>>>> frequent commits, you've likely > encountered the dreaded OutOfMemory or > >>>>> GC Overhead Exeeded errors. > >>>>> In high commit rate environments, this > is almost always due to > >>>>> multiple 'onDeck' searchers and > autowarming - i.e. new searchers don't > >>>>> finish autowarming their caches before > the next commit() > >>>>> comes along and invalidates them. > >>>>> Once this starts happening on a > regular basis, it is likely your > >>>>> Solr's JVM will run out of memory > eventually, as the number of > >>>>> searchers (and their cache arrays) > will keep growing until the JVM > >>>>> dies of thirst. > >>>>> To check if your Solr environment is > suffering from this, turn on INFO > >>>>> level logging, and look for: > 'PERFORMANCE WARNING: Overlapping > >>>>> onDeckSearchers=x'. > >>>>> > >>>>> In tests, we've only ever seen this > problem when using faceting, and > >>>>> facet.method=fc. > >>>>> > >>>>> Some solutions to this are: > >>>>> Reduce the commit rate to allow > searchers to fully warm before the > >>>>> next commit > >>>>> Reduce or eliminate the > autowarming in caches > >>>>> Both of the above > >>>>> > >>>>> The trouble is, if you're doing NRT > commits, you likely have a good > >>>>> reason for it, and > reducing/elimintating autowarming will very > >>>>> significantly impact search > performance in high commit rate > >>>>> environments. > >>>>> > >>>>> Solution: > >>>>> Here are some setup steps we've used > that allow lots of faceting (we > >>>>> typically search with at least 20-35 > different facet fields, and date > >>>>> faceting/sorting) on large indexes, > and still keep decent search > >>>>> performance: > >>>>> > >>>>> 1. Firstly, you should consider using > the enum method for facet > >>>>> searches (facet.method=enum) unless > you've got A LOT of memory on your > >>>>> machine. In our tests, this method > uses a lot less memory and > >>>>> autowarms more quickly than fc. (Note, > I've not tried the new > >>>>> segement-based 'fcs' option, as I > can't find support for it in > >>>>> branch_3x - looks nice for 4.x > though) > >>>>> Admittedly, for our data, enum is not > quite as fast for searching as > >>>>> fc, but short of purchsing a > Thaiwanese RAM factory, it's a worthwhile > >>>>> tradeoff. > >>>>> If you do have access to LOTS of > memory, AND you can guarantee that > >>>>> the index won't grow beyond the memory > capacity (i.e. you have some > >>>>> sort of deletion policy in place), fc > can be a lot faster than enum > >>>>> when searching with lots of facets > across many terms. > >>>>> > >>>>> 2. Secondly, we've found that LRUCache > is faster at autowarming than > >>>>> FastLRUCache - in our tests, about 20% > faster. Maybe this is just our > >>>>> environment - your mileage may vary. > >>>>> > >>>>> So, our filterCache section in > solrconfig.xml looks like this: > >>>>> <filterCache > >>>>> class="solr.LRUCache" > >>>>> size="3600" > >>>>> initialSize="1400" > >>>>> autowarmCount="3600"/> > >>>>> > >>>>> For a 28GB index, running in a > quad-core x64 VMWare instance, 30 > >>>>> warmed facet fields, Solr is running > at ~4GB. Stats filterCache size > >>>>> shows usually in the region of ~2400. > >>>>> > >>>>> 3. It's also a good idea to have some > sort of > >>>>> firstSearcher/newSearcher event > listener queries to allow new data to > >>>>> populate the caches. > >>>>> Of course, what you put in these is > dependent on the facets you need/use. > >>>>> We've found a good combination is a > firstSearcher with as many facets > >>>>> in the search as your environment can > handle, then a subset of the > >>>>> most common facets for the > newSearcher. > >>>>> > >>>>> 4. We also set: > >>>>> > <useColdSearcher>true</useColdSearcher> > >>>>> just in case. > >>>>> > >>>>> 5. Another key area for search > performance with high commits is to use > >>>>> 2 Solr instances - one for the high > commit rate indexing, and one for > >>>>> searching. > >>>>> The read-only searching instance can > be a remote replica, or a local > >>>>> read-only instance that reads the same > core as the indexing instance > >>>>> (for the latter, you'll need something > that periodically refreshes - > >>>>> i.e. runs commit()). > >>>>> This way, you can tune the indexing > instance for writing performance > >>>>> and the searching instance as above > for max read performance. > >>>>> > >>>>> Using the setup above, we get > fantastic searching speed for small > >>>>> facet sets (well under 1sec), and > really good searching for large > >>>>> facet sets (a couple of secs depending > on index size, number of > >>>>> facets, unique terms etc. etc.), > >>>>> even when searching against largeish > indexes (>20million docs). > >>>>> We have yet to see any OOM or GC > errors using the techniques above, > >>>>> even in low memory conditions. > >>>>> > >>>>> I hope there are people that find this > useful. I know I've spent a lot > >>>>> of time looking for stuff like this, > so hopefullly, this will save > >>>>> someone some time. > >>>>> > >>>>> > >>>>> Peter > >>>>> > >>>>> > > > > >
