RE: Performance stats for indeces with over 10MM documents

2008-01-03 Thread Alex Benjamen
we currently use a relational system, and it doesn't perform. Also, even though a lot of our queries are structured, we do combine them with text search, so for instance, there could be an additional clause which is a free text search for a favorite TV show -- I had exactly

Re: Performance stats for indeces with over 10MM documents

2008-01-03 Thread Walter Underwood
I had exactly the same thought. That query is not an information retrieval (text search) query. It is data retrieval and would work great on a relational database. wunder On 1/2/08 9:53 PM, "John Stewart" <[EMAIL PROTECTED]> wrote: > Alex, > > Not to be a pain, but the response I had when looki

Re: Performance stats for indeces with over 10MM documents

2008-01-02 Thread Mike Klaas
On 2-Jan-08, at 9:52 PM, Alex Benjamen wrote: Thanks for the input, it's really valueable. Several forum users have suggested using fq to separate the caching of filters, and I can immediately see how this would help. I'm changing the code right now and going to run some benchmarks, hopeful

Re: Performance stats for indeces with over 10MM documents

2008-01-02 Thread John Stewart
Alex, Not to be a pain, but the response I had when looking at the query was, why not do this in a SQL database, which is designed precisely to process this sort of request at speed? I've noticed that people sometimes try to get Solr to act as a generalized information store -- I'm not sure that'

RE: Performance stats for indeces with over 10MM documents

2008-01-02 Thread Alex Benjamen
Mike, Thanks for the input, it's really valueable. Several forum users have suggested using fq to separate the caching of filters, and I can immediately see how this would help. I'm changing the code right now and going to run some benchmarks, hopefully see a big gain just from that > - use

Re: Performance stats for indeces with over 10MM documents

2008-01-02 Thread Mike Klaas
On 2-Jan-08, at 5:47 PM, Alex Benjamen wrote: gender:m AND status:(2 || 8 || 6 || 3) AND age:(26 || 27 || 28 || 29) AND orientation:3 gender:f AND age:(27 || 28 || 29 || 30 || 31 || 32 || 33 || 34 || 35 || 36 ) AND orientation:2 AND photos:y gender:f AND (activity:y) AND age:(28 || 29 || 30

Re: Performance stats for indeces with over 10MM documents

2008-01-02 Thread Erik Hatcher
Are you faceting? Please provide the exact lines logged in Solr's console for the offending queries - that would show us _exactly_ what you're hitting Solr with, not just the q parameter as you seem to have provided. Erik On Jan 2, 2008, at 8:47 PM, Alex Benjamen wrote: JDS: Th

RE: Performance stats for indeces with over 10MM documents

2008-01-02 Thread Alex Benjamen
JDS: > That's too slow. Can you provide more details about your schema, queries etc? Ofcourse - I'm using the standard config which comes with solr, and I've added the following fields :

Re: Performance stats for indeces with over 10MM documents

2008-01-02 Thread John Stewart
Alex, That's too slow. Can you provide more details about your schema, queries etc? jds On Jan 2, 2008 7:28 PM, Alex Benjamen <[EMAIL PROTECTED]> wrote: > Hi, > > I'm very interested in sharing performance stats with those who have indeces > that > contain more than 10MM documents. It seems th

Performance stats for indeces with over 10MM documents

2008-01-02 Thread Alex Benjamen
Hi, I'm very interested in sharing performance stats with those who have indeces that contain more than 10MM documents. It seems that the response times and QPS drops drastically with the number of documents in the index. This overall makes sense, but it would be good to know what kind of QPS o