Re: yet another optimize question

2013-06-20 Thread Jack Krupansky
Subject: RE: yet another optimize question Petersen, Robert [robert.peter...@mail.rakuten.com] wrote: We actually have hundreds of facet-able fields, but most are specialized and are only faceted upon if the user has drilled into the particular category to which they are applicable and so they are

RE: yet another optimize question

2013-06-20 Thread Toke Eskildsen
Petersen, Robert [robert.peter...@mail.rakuten.com] wrote: > We actually have hundreds of facet-able fields, but most are specialized > and are only faceted upon if the user has drilled into the particular category > to which they are applicable and so they are only indexed for products > in those

RE: yet another optimize question

2013-06-19 Thread Petersen, Robert
Sent: Wednesday, June 19, 2013 10:50 AM To: solr-user@lucene.apache.org Subject: Re: yet another optimize question I generally run with an 8GB heap for a system that does no faceting. 32GB does seem rather large, but you really should have room for bigger caches. The Akamai cache will reduce your hit

Re: yet another optimize question

2013-06-19 Thread Walter Underwood
wun...@wunderwood.org] > Sent: Tuesday, June 18, 2013 6:57 PM > To: solr-user@lucene.apache.org > Subject: Re: yet another optimize question > > Your query cache is far too small. Most of the default caches are too small. > > We run with 10K entries and get a hit rate around 0.30

RE: yet another optimize question

2013-06-19 Thread Petersen, Robert
[mailto:wun...@wunderwood.org] Sent: Tuesday, June 18, 2013 6:57 PM To: solr-user@lucene.apache.org Subject: Re: yet another optimize question Your query cache is far too small. Most of the default caches are too small. We run with 10K entries and get a hit rate around 0.30 across four servers. This

Re: yet another optimize question

2013-06-19 Thread Andre Bois-Crettez
facet fields eh? Thanks for the tip. Thanks Robi -Original Message- From: Andre Bois-Crettez [mailto:andre.b...@kelkoo.com] Sent: Tuesday, June 18, 2013 3:03 AM To: solr-user@lucene.apache.org Subject: Re: yet another optimize question Recently we had steadily increasing memory usage and

Re: yet another optimize question

2013-06-18 Thread Walter Underwood
umulative_inserts : 83261 >> cumulative_evictions : 3479 >> >> >> name:filterCache >> class: org.apache.solr.search.FastLRUCache >> version: 1.0 >> description: Concurrent LRU Cache(maxSize=248, initialSize=12, >> minSize=223, accept

RE: yet another optimize question

2013-06-18 Thread Petersen, Robert
, June 18, 2013 3:03 AM To: solr-user@lucene.apache.org Subject: Re: yet another optimize question Recently we had steadily increasing memory usage and OOM due to facets on dynamic fields. The default facet.method=fc need to build a large array of maxdocs ints for each field (a fieldCache or

RE: yet another optimize question

2013-06-18 Thread Petersen, Robert
already in effect for me? 10 10 Thanks Robi -Original Message- From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] Sent: Monday, June 17, 2013 6:36 PM To: solr-user@lucene.apache.org Subject: Re: yet another optimize question Yes, in one of the example sol

RE: yet another optimize question

2013-06-18 Thread Petersen, Robert
nSize=223, acceptableSize=235, cleanupThread=false, autowarmCount=10, > regenerator=org.apache.solr.search.SolrIndexSearcher$2@36e831d6) > stats: lookups : 3990 > hits : 3831 > hitratio : 0.96 > inserts : 239 > evictions : 26 > size : 244 > warmupTime : 1 > cumulative_lookups : 5745

Re: yet another optimize question

2013-06-18 Thread Andre Bois-Crettez
Recently we had steadily increasing memory usage and OOM due to facets on dynamic fields. The default facet.method=fc need to build a large array of maxdocs ints for each field (a fieldCache or fieldValueCahe entry), whether it is sparsely populated or not. Once you have reduced your number of ma

Re: yet another optimize question

2013-06-17 Thread Otis Gospodnetic
5, cleanupThread=false, autowarmCount=10, > regenerator=org.apache.solr.search.SolrIndexSearcher$2@36e831d6) > stats: lookups : 3990 > hits : 3831 > hitratio : 0.96 > inserts : 239 > evictions : 26 > size : 244 > warmupTime : 1 > cumulative_lookups : 5745011 > cum

Re: yet another optimize question

2013-06-17 Thread Otis Gospodnetic
my index defaults section? > > >10 >10 > > > Thanks > Robi > > -Original Message- > From: Upayavira [mailto:u...@odoko.co.uk] > Sent: Monday, June 17, 2013 12:29 PM > To: solr-user@lucene.apache.org > Subject: Re: yet another optimize que

RE: yet another optimize question

2013-06-17 Thread Petersen, Robert
to want to put something like this into my index defaults section? 10 10 Thanks Robi -Original Message- From: Upayavira [mailto:u...@odoko.co.uk] Sent: Monday, June 17, 2013 12:29 PM To: solr-user@lucene.apache.org Subject: Re: yet another optimize question The key figures are num

Re: yet another optimize question

2013-06-17 Thread Upayavira
90 > hits : 3831 > hitratio : 0.96 > inserts : 239 > evictions : 26 > size : 244 > warmupTime : 1 > cumulative_lookups : 5745011 > cumulative_hits : 5496150 > cumulative_hitratio : 0.95 > cumulative_inserts : 351485 > cumulative_evictions : 276308 >

RE: yet another optimize question

2013-06-17 Thread Petersen, Robert
umulative_hitratio : 0.95 cumulative_inserts : 351485 cumulative_evictions : 276308 -Original Message- From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] Sent: Saturday, June 15, 2013 5:52 AM To: solr-user@lucene.apache.org Subject: Re: yet another optimize question Hi Robi, I

Re: yet another optimize question

2013-06-15 Thread Otis Gospodnetic
Hi Robi, I'm going to guess you are seeing smaller heap also simply because you restarted the JVM recently (hm, you don't say you restarted, maybe I'm making this up). If you are indeed indexing continuously then you shouldn't optimize. Lucene will merge segments itself. Lower mergeFactor will for