Re: Result merging takes too long

2014-04-13 Thread remi tassing
Hi, After looking at SearchHandler.java I believe we might have a bug on debug information shown for distributed queries. >@Erick: So as you can see, there's an extra round trip to each shard to get the full document. Perhaps this is what you're seeing? <4> seems like it might be what you're seei

Re: Result merging takes too long

2014-03-17 Thread Shalin Shekhar Mangar
That's great Jeff! Thanks for sharing your experience. SOLR-5768 will make it even better. https://issues.apache.org/jira/browse/SOLR-5768 On Tue, Mar 18, 2014 at 3:35 AM, Jeff Wartes wrote: > > This is highly anecdotal, but I tried SOLR-1880 with 4.7 for some tests I > was running, and saw almo

Re: Result merging takes too long

2014-03-17 Thread Jeff Wartes
This is highly anecdotal, but I tried SOLR-1880 with 4.7 for some tests I was running, and saw almost a 30% improvement in latency. If you¹re only doing document selection, it¹s definitely worth having. I¹m reasonably certain that the patch would work in 4.6 too, but the test file relies on some

Re: Result merging takes too long

2014-03-16 Thread Chris W
Can you share a sample query ? Ensure you have filterquery, fl fields and query result cache settings well tuned To give you an example: A month ago I had an issue where a few of our queries were taking 3+seconds with 5 shards and as I added more shards the query was taking even longer. I figured

Re: Result merging takes too long

2014-03-16 Thread Shalin Shekhar Mangar
On Thu, Mar 13, 2014 at 1:49 PM, remi tassing wrote: > > I've used the fl=id parameter to avoid retrieving the actual documents > (step <4> in your mail) but the problem still exists. > Any ideas on how to find the merging time(step <3>)? Actually that doesn't skip steps #4 and #5. That optimizat

Re: Result merging takes too long

2014-03-16 Thread remi tassing
>So I have to ask what the end goal is here. In our case, the purpose of sharding was/is to speed up the process. We've noticed that as the index size was growing, response speed kept going down so we decided to split the index across 5 machines. >Are your response times really in need of improvem

Re: Result merging takes too long

2014-03-15 Thread Erick Erickson
I wouldn't expect the merge times to be significant at all, _assuming_ you're not doing something like setting a very high &start= parameter or returning a whole of rows. Now, it may be that you're sharding with too small a document set to really notice a difference. Sharding isn't really about sp

Re: Result merging takes too long

2014-03-13 Thread remi tassing
Hi Erick, I've used the fl=id parameter to avoid retrieving the actual documents (step <4> in your mail) but the problem still exists. Any ideas on how to find the merging time(step <3>)? Remi On Tue, Mar 11, 2014 at 7:29 PM, Erick Erickson wrote: > In SolrCloud there are a couple of round tri

Re: Result merging takes too long

2014-03-11 Thread Erick Erickson
In SolrCloud there are a couple of round trips that _may_ be what you're seeing. First, though, the QTime is the time spent querying, it does NOT include assembling the documents from disk for return etc., so bear that in mind But here's the sequence as I understand it from the receiving node

Result merging takes too long

2014-03-11 Thread remi tassing
Hi, I've just setup a SolrCloud with Tomcat. 5 Shards with one replication each and total 10million docs (evenly distributed). I've noticed the query response time is faster than using one single node but still not as fast as I expected. After turning debugQuery on, I noticed the query time is d