Hi Upayavira, You might have gotten it. Yes we noticed maxdocs was way bigger than numdocs. There were a lot of files ending in '.del' in the index folder also. We started on 1.3 also. I don't currently have any solr config settings for MergePolicy at all. Am I going to want to put something like this into my index defaults section?
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy"> <int name="maxMergeAtOnce">10</int> <int name="segmentsPerTier">10</int> </mergePolicy> Thanks Robi -----Original Message----- From: Upayavira [mailto:u...@odoko.co.uk] Sent: Monday, June 17, 2013 12:29 PM To: solr-user@lucene.apache.org Subject: Re: yet another optimize question The key figures are numdocs vs maxdocs. Maxdocs-numdocs is the number of deleted docs in your index. This is a 3.6 system you say. But has it been upgraded? I've seen folks who've upgraded from 1.4 or 3.0/3.1 over time, keeping the old config. The consequence of this is that they don't get the right config for the TieredMergePolicy, and therefore don't get to use it, seeing the old behaviour which does require periodic optimise. Upayavira On Mon, Jun 17, 2013, at 07:21 PM, Petersen, Robert wrote: > Hi Otis, > > Right I didn't restart the JVMs except on the one slave where I was > experimenting with using G1GC on the 1.7.0_21 JRE. Also some time ago I > made all our caches small enough to keep us from getting OOMs while still > having a good hit rate. Our index has about 50 fields which are mostly > int IDs and there are some dynamic fields also. These dynamic fields > can be used for custom faceting. We have some standard facets we > always facet on and other dynamic facets which are only used if the > query is filtering on a particular category. There are hundreds of > these fields but since they are only for a small subset of the overall > index they are very sparsely populated with regard to the overall > index. With CMS GC we get a sawtooth on the old generation (I guess > every replication and commit causes it's usage to drop down to 10GB or > so) and it seems to be the old generation which is the main space > consumer. With the G1GC, the memory map looked totally different! I > was a little lost looking at memory consumption with that GC. Maybe > I'll try it again now that the index is a bit smaller than it was last > time I tried it. After four days without running an optimize now it > is 21GB. BTW our indexing speed is mostly bound by the DB so reducing the > segments might be ok... > > Here is a quick snapshot of one slaves memory map as reported by > PSI-Probe, but unfortunately I guess I can't send the history graphics > to the solr-user list to show their changes over time: > Name Used Committed Max > Initial Group > Par Survivor Space 20.02 MB 108.13 MB 108.13 MB > 108.13 MB HEAP > CMS Perm Gen 42.29 MB 70.66 MB 82.00 MB 20.75 > MB NON_HEAP > Code Cache 9.73 MB 9.88 MB 48.00 MB 2.44 MB NON_HEAP > CMS Old Gen 20.22 GB 30.94 GB 30.94 GB > 30.94 GB HEAP > Par Eden Space 42.20 MB 865.31 MB 865.31 MB 865.31 > MB HEAP > Total 20.33 GB 31.97 GB 32.02 GB > 31.92 GB TOTAL > > And here's our current cache stats from a random slave: > > name: queryResultCache > class: org.apache.solr.search.LRUCache > version: 1.0 > description: LRU Cache(maxSize=488, initialSize=6, autowarmCount=6, > regenerator=org.apache.solr.search.SolrIndexSearcher$3@461ff4c3) > stats: lookups : 619 > hits : 36 > hitratio : 0.05 > inserts : 592 > evictions : 101 > size : 488 > warmupTime : 2949 > cumulative_lookups : 681225 > cumulative_hits : 73126 > cumulative_hitratio : 0.10 > cumulative_inserts : 602396 > cumulative_evictions : 428868 > > > name: fieldCache > class: org.apache.solr.search.SolrFieldCacheMBean > version: 1.0 > description: Provides introspection of the Lucene FieldCache, this is > **NOT** a cache that is managed by Solr. > stats: entries_count : 359 > > > name: documentCache > class: org.apache.solr.search.LRUCache > version: 1.0 > description: LRU Cache(maxSize=2048, initialSize=512, > autowarmCount=10, regenerator=null) > stats: lookups : 12710 > hits : 7160 > hitratio : 0.56 > inserts : 5636 > evictions : 3588 > size : 2048 > warmupTime : 0 > cumulative_lookups : 10590054 > cumulative_hits : 6166913 > cumulative_hitratio : 0.58 > cumulative_inserts : 4423141 > cumulative_evictions : 3714653 > > > name: fieldValueCache > class: org.apache.solr.search.FastLRUCache > version: 1.0 > description: Concurrent LRU Cache(maxSize=280, initialSize=280, > minSize=252, acceptableSize=266, cleanupThread=false, autowarmCount=6, > regenerator=org.apache.solr.search.SolrIndexSearcher$1@143eb77a) > stats: lookups : 1725 > hits : 1481 > hitratio : 0.85 > inserts : 122 > evictions : 0 > size : 128 > warmupTime : 4426 > cumulative_lookups : 3449712 > cumulative_hits : 3281805 > cumulative_hitratio : 0.95 > cumulative_inserts : 83261 > cumulative_evictions : 3479 > > > name: filterCache > class: org.apache.solr.search.FastLRUCache > version: 1.0 > description: Concurrent LRU Cache(maxSize=248, initialSize=12, > minSize=223, acceptableSize=235, cleanupThread=false, > autowarmCount=10, > regenerator=org.apache.solr.search.SolrIndexSearcher$2@36e831d6) > stats: lookups : 3990 > hits : 3831 > hitratio : 0.96 > inserts : 239 > evictions : 26 > size : 244 > warmupTime : 1 > cumulative_lookups : 5745011 > cumulative_hits : 5496150 > cumulative_hitratio : 0.95 > cumulative_inserts : 351485 > cumulative_evictions : 276308 > > -----Original Message----- > From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] > Sent: Saturday, June 15, 2013 5:52 AM > To: solr-user@lucene.apache.org > Subject: Re: yet another optimize question > > Hi Robi, > > I'm going to guess you are seeing smaller heap also simply because you > restarted the JVM recently (hm, you don't say you restarted, maybe I'm > making this up). If you are indeed indexing continuously then you > shouldn't optimize. Lucene will merge segments itself. Lower > mergeFactor will force it to do it more often (it means slower > indexing, bigger IO hit when segments are merged, more per-segment > data that Lucene/Solr need to read from the segment for faceting and > such, etc.) so maybe you shouldn't mess with that. Do you know what > your caches are like in terms of size, hit %, evictions? We've > recently seen people set those to a few hundred K or even higher, > which can eat a lot of heap. We have had luck with G1 recently, too. > Maybe you can run jstat and see which of the memory pools get filled > up and change/increase appropriate JVM param based on that? How many > fields do you index, facet, or group on? > > Otis > -- > Performance Monitoring - http://sematext.com/spm/index.html > Solr & ElasticSearch Support -- http://sematext.com/ > > > > > > On Fri, Jun 14, 2013 at 8:04 PM, Petersen, Robert > <robert.peter...@mail.rakuten.com> wrote: > > Hi guys, > > > > We're on solr 3.6.1 and I've read the discussions about whether to optimize > > or not to optimize. I decided to try not optimizing our index as was > > recommended. We have a little over 15 million docs in our biggest index > > and a 32gb heap for our jvm. So without the optimizes the index folder > > seemed to grow in size and quantity of files. There seemed to be an upper > > limit but eventually it hit 300 files consuming 26gb of space and that > > seemed to push our slave farm over the edge and we started getting the > > dreaded OOMs. We have continuous indexing activity, so I stopped the > > indexer and manually ran an optimize which made the index become 9 files > > consuming 15gb of space and our slave farm started having acceptable memory > > usage. Our merge factor is 10, we're on java 7. Before optimizing, I > > tried on one slave machine to go with the latest JVM and tried switching > > from the CMS GC to the G1GC but it hit OOM condition even faster. So it > > seems like I have to continue to schedule a regular optimize. Right now it > > has been a couple of days since running the optimize and the index is > > slowly growing bigger, now up to a bit over 19gb. What do you guys think? > > Did I miss something that would make us able to run without doing an > > optimize? > > > > Robert (Robi) Petersen > > Senior Software Engineer > > Search Department > >
