On 12/5/06, Gmail Account <[EMAIL PROTECTED]> wrote:
Sorry.. I put the wrong subject on my message. I also wanted to mention that my cpu jumps to to almost 100% each query.
There's nothing wrong with CPU jumping to 100% each query, that just means you aren't IO bound :-) It's the 3 seconds that are the problem.
> I'm having slow performance with my solr index. I'm not sure what to do. I > need some suggestions on what to try. I have updated all my records in the > last couple of days. I'm not sure how much it degraded because of that, > but it now takes about 3 seconds per search.
> - I did an optimize index through Luke with compound format and noticed > in the solrconfig file that useCompoundFile is set to false.
Don't do this unless you really know what you are doing... Luke is probably using a different version of Lucene than Solr, and it could be dangerous.
> numDocs : 1185814 > maxDoc : 2070472
Your index isn't optmized... you can see that by the difference between maxDoc and numDocs (which represent deleted documents still in the index). This will slow things down in several ways. - Since you only have 2GB of RAM on your PC, the larger index files means less effective OS level caching - if you are using filters, any larger than 3000 will be double the size (maxDoc bits) - lowest level lucene search functions iterate over all documents for a given term, even if they are deleted (they are filtered out by consulting a bitvector of deleted docs) Do an optimize of the index via Solr (use your "commit" command as a template, but substitute "optimize").
My cache statistics don't > look so good either.
Ouch... the filterCache doesn't look good. Are you using faceted browsing? It looks like the number of terms in the field you are faceting by is larger than the number of items the filterCache can hold... which means a 0% cache hit rate w/ a LRU cache. Can you give some examples of what your queries look like? Depending on the size of the corpus and what faceted queries you want to run, you may need more than 2GB of ram for faster queries. If you always keep the index optimized, this will lower the size of some of the filters and allow you to increase the number of cached filters at least.
> - I changed one of the fields in the schema from text_ws to string
OK, as long as you re-indexed.
> - I added a field (type="text" indexed="false" stored="true")
OK -Yonik
> My schema and solrconfig are the same as the example except I have a few > more fields. My pc is winXP and has 2gig of ram. Below are some stats from > the solr admin stat page. > > Thanks! > > > caching : true > numDocs : 1185814 > maxDoc : 2070472 > readerImpl : MultiReader > > name: filterCache > class: org.apache.solr.search.LRUCache > version: 1.0 > description: LRU Cache(maxSize=512, initialSize=512, > autowarmCount=256, > [EMAIL PROTECTED]) > stats: lookups : 658446 > hits : 30 > hitratio : 0.00 > inserts : 658420 > evictions : 657908 > size : 512 > cumulative_lookups : 658446 > cumulative_hits : 30 > cumulative_hitratio : 0.00 > cumulative_inserts : 658420 > cumulative_evictions : 657908 > > > name: queryResultCache > class: org.apache.solr.search.LRUCache > version: 1.0 > description: LRU Cache(maxSize=512, initialSize=512, > autowarmCount=256, > [EMAIL PROTECTED]) > stats: lookups : 88 > hits : 83 > hitratio : 0.94 > inserts : 6 > evictions : 0 > size : 5 > cumulative_lookups : 88 > cumulative_hits : 83 > cumulative_hitratio : 0.94 > cumulative_inserts : 6 > cumulative_evictions : 0 > > > name: documentCache > class: org.apache.solr.search.LRUCache > version: 1.0 > description: LRU Cache(maxSize=512, initialSize=512) > stats: lookups : 780 > hits : 738 > hitratio : 0.94 > inserts : 42 > evictions : 0 > size : 42 > cumulative_lookups : 780 > cumulative_hits : 738 > cumulative_hitratio : 0.94 > cumulative_inserts : 42 > cumulative_evictions : 0
