Re: Doing url search in solr is slow

2012-01-10 Thread yu shen
Hi Erick, I change all my url fields into text (they were string fields before), and added a WordDelimiterFilterFactory, so that url fields can be tokenized into several words. But I still got around 15 seconds response time measured using debugyQuery=on, and most of the time still spend on DebugC

Re: Doing url search in solr is slow

2012-01-09 Thread yu shen
Hi Erick, I only added debugyQuery=on to the url, and did not do any configuration with regard to DebugComponent. Seems like 'string' type should be substituted with 'text' type. I will paste the result here after I did some experiments. Spark 2012/1/9 Erick Erickson > Do you by chance have t

Re: Doing url search in solr is slow

2012-01-09 Thread Erick Erickson
Do you by chance have the debugQuery on by default? Because if you look down in the "timing" section, you can see the times the various components took to do their work, there are two sections "prepare" and "process". The cumulative time is 17.156 seconds. Of which 17.156 seconds is reported to be

Re: Doing url search in solr is slow

2012-01-09 Thread yu shen
Hi Erick, Thanks for you reply. Actually I did the following search: survey_url:http\://www.someurl.com/sch/i.html* referal_url:http\:// www.someurl.com/sch/i.html* page_url:http\://www.someurl.com/sch/i.html* I did not prepend any asterisk to the field value, but only append to them. I analyze

Re: Doing url search in solr is slow

2012-01-09 Thread Erick Erickson
Yu Shen & Arian: We can't help much without more information. In particular, how are the fields in question analyzed? What is the result of looking at the admin/analysis page? What do you get when you attach &debugQuery=on to the query? You might review: http://wiki.apache.org/solr/UsingMailingLi

Re: Doing url search in solr is slow

2012-01-09 Thread François Schiettecatte
About the search 'referal_url:*www.someurl.com*', having a wildcard at the start will cause a dictionary scan for every term you search on unless you use ReversedWildcardFilterFactory. That could be the cause of your slowdown if you are I/O bound, and even if you are CPU bound for that matter.

Re: Doing url search in solr is slow

2012-01-08 Thread Arian
g url search in solr is slow Hi, My solr document has up to 20 fields, containing data from product name, date, url etc. The volume of documents is around 1.5m. My symptom is when doing url search like [ url:*www.someurl.com* referal_url:*www.someurl.com* page_url:*www.someurl.com*] will

Doing url search in solr is slow

2012-01-08 Thread yu shen
Hi, My solr document has up to 20 fields, containing data from product name, date, url etc. The volume of documents is around 1.5m. My symptom is when doing url search like [ url:*www.someurl.com* referal_url:*www.someurl.com* page_url:*www.someurl.com*] will get a extraordinary long response ti