Re: Fail to huge collection extraction

2012-09-10 Thread neosky
To Alex, Thanks for you advice. I did ask and I can understand the requirement is necessary for them. They won't browser all the result in one page. But they will use the query result to do some additional research. So what they want are something exact match the query. So they need to pull out al

Re: Fail to huge collection extraction

2012-09-09 Thread Erick Erickson
Alexandre: I'll buy you a beer sometime, it's just sooo pleasant when someone else has the same worldview I do http://searchhub.org/dev/2011/11/03/stop-being-so-agreeable/ neosky: Particularly look at the paragraph that has "the XY problem" in it. Best Erick On Sun, Sep 9, 2012 at 8:56 AM,

Re: Fail to huge collection extraction

2012-09-09 Thread Alexandre Rafalovitch
I am sorry, but your customer is extremely unlikely to want the whole result in his browser. It is just a red flag that they are converting their (business) requirements into your (IT) language and that's what they end up with. Go the other way, ask them to pretend that you've done it already and

Re: Fail to huge collection extraction

2012-09-09 Thread neosky
Thanks Alex! Yes, you hit my key points. Actually I have to implement both of the requirements. The first one works very well as the reason you state. Now I have a website client which is 20 records per page. It is fast. However, my customer also wants to use Servlet to download the whole query se

Re: Fail to huge collection extraction

2012-09-08 Thread Alexandre Rafalovitch
I think the point here is the question about your use of the data. If you want to show it to the client, then you are unlikely to need details of more than 1 screenful of records (e.g. 10). When user goes to another screen, you rerun the query and specify values 11-20, etc. SOLR does not have a pr

Re: Fail to huge collection extraction

2012-09-08 Thread neosky
I am sorry that I can't get your point. Would you explain a little more? I am still struggling with this problem. It seems crash by no meaning sometimes. Even I reduce to 5000 records each time, but sometimes it works well with 1 per page. -- View this message in context: http://lucene.4720

Re: Fail to huge collection extraction

2012-08-29 Thread Erick Erickson
I really think you need to think about firing successive page requests at the index and reporting in chunks. Best Erick On Mon, Aug 27, 2012 at 2:56 PM, neosky wrote: > I am using Solr 3.5 and Jetty 8.12 > I need to pull out huge query results at a time(for example, 1 million > documents, probab