> Thanks Eric for your response. If you find pagination is not the main
> culprit, what other factors do you guys suggest I need to tweak to test
> that?
Well, is basic search slow? What are your response times for plain
un-warmed top-20 searches?
I have restarted Solr and I have tried running a query "Football" on Solr
and here are the results
for start=0, rows=10 it took around 3.391 seconds
for start=1000, rows=10 it took around 21.569 seconds *(btw, after trying
the query the second time, it took around 332 ms, could you explain this
behavior?)*
I am not quite sure what do you mean by un-warmed search, but I do have
autowarmed set to true for filtercache
btw, here is the log for both queries and it looks like that indeed it does
take that long for Solr to query
INFO - 2015-10-09 18:46:17.937; [c:sabr102 s:shard2 r:core_node1
x:sabr102_shard2_replica1] org.apache.solr.core.SolrCore;
[sabr102_shard2_replica1] webapp=/solr path=/select
params={ids=592367114956177408,590296378955407362,585347065619750912,584382847948951552&distrib=false&wt=javabin&version=2&rows=10&df=text&shard.url=http://
[MySolrIP]:8983/solr/sabr102_shard2_replica1/|http://[MySolrIP]:7574/solr/sabr102_shard2_replica2/&NOW=1444416374563&start=0&shards.purpose=64&q=(content_text:Football)&isShard=true&preferLocalShards=false}
status=0 QTime=13
INFO - 2015-10-09 18:46:17.953; [c:sabr102 s:shard1 r:core_node2
x:sabr102_shard1_replica1] org.apache.solr.core.SolrCore;
[sabr102_shard1_replica1] webapp=/solr path=/select
params={start=0&q=(content_text:Football)&rows=10} hits=24408 status=0
QTime=3391
INFO - 2015-10-09 18:46:43.207; [c:sabr102 s:shard2 r:core_node1
x:sabr102_shard2_replica1] org.apache.solr.core.SolrCore;
[sabr102_shard2_replica1] webapp=/solr path=/select
params={distrib=false&wt=javabin&version=2&rows=1010&df=text&fl=id&fl=score&shard.url=http://
[MySolrIP]:8983/solr/sabr102_shard2_replica1/|http://[MySolrIP]:7574/solr/sabr102_shard2_replica2/&NOW=1444416403161&start=0&shards.purpose=4&q=(content_text:Football)&isShard=true&fsv=true&preferLocalShards=false}
hits=12198 status=0 QTime=32
INFO - 2015-10-09 18:47:04.727; [c:sabr102 s:shard1 r:core_node2
x:sabr102_shard1_replica1] org.apache.solr.core.SolrCore;
[sabr102_shard1_replica1] webapp=/solr path=/select
params={start=1000&q=(content_text:Football)&rows=10} hits=24408 status=0
QTime=21569
> As I mentioned, by navigating to 20000 results using start and row I
> am getting time out from Solr.NET and I need a way to fix that.
You still haven't answered my question: Do your users actually need to page
that far?
No, they do not need to navigate to that level but I was checking the edge
cases. Moreover, based on my previous query results, even navigating to the
100th page (1000 results as each page has 10 results, which they can easily
do from the query strings in the URL or jumping bunch of pages at once in
the UI as I am giving access to 10 pages at a time like Google or LinkedIn)
the performance results are not promising.
It shows that the shard-searches themselves is not what is slowing you
down. Are the returned documents very large? Try setting fl=id,score and
see if it brings response times below 1 second.
I have around 50-60 fields per document in schema but not all of them get
populated for each document. The main field that I am searching on is
called content_text but that is usually small.
I have tried running the following query on Solr
http://[MySolrMachine]:8983/solr/sabr102/select?q=(content_text:Football)&start=1000&rows=10&fl=id,score
and it took around 13.567 seconds *(the same goes here after running the
query the second time, it took around 244 ms)*
The log shows that it did take Solr that long
INFO - 2015-10-09 18:54:44.271; [c:sabr102 s:shard1 r:core_node2
x:sabr102_shard1_replica1] org.apache.solr.core.SolrCore;
[sabr102_shard1_replica1] webapp=/solr path=/select
params={fl=id,score&start=1000&q=(content_text:Football)&rows=10}
hits=24408 status=0 QTime=13567
INFO - 2015-10-09 19:02:41.732; [c:sabr102 s:shard2 r:core_node1
x:sabr102_shard2_replica1] org.apache.solr.core.SolrCore;
[sabr102_shard2_replica1] webapp=/solr path=/select
params={distrib=false&wt=javabin&version=2&rows=1010&df=text&fl=id&fl=score&shard.url=http://
[MySolrIP]:8983/solr/sabr102_shard2_replica1/|http://[MySolrIP]:7574/solr/sabr102_shard2_replica2/&NOW=1444417361716&start=0&shards.purpose=4&q=(content_text:Football)&isShard=true&fsv=true&preferLocalShards=false}
hits=12198 status=0 QTime=9
*Why is it the case that for some reasons shard1 is taking way more longer
than shard2?*
I do note that one of your queries has rows=1010, a typo?
No that was not a typo,
Try again with rows=0&start=1000 to see if it's something weird with getting
the stored data, but that's highly doubtful.
I have tried the query "Salman" with rows=0, start=1000 and it took around
13.819 seconds.
I think the only real way to get to the bottom of it will be to slap a
profiler on it and see where the time is being spent.
Can you direct me to a good profiler for Solr?
Regards,
Salman
On Fri, Oct 9, 2015 at 8:02 PM, Erick Erickson <erickerick...@gmail.com>
wrote:
> OK, this makes very little sense. The individual queries are taking < 100ms
> yet the total response is 29 seconds. I do note that one of your
> queries has rows=1010, a typo?
>
> Anyway, not at all sure what's going on here. If these are gigantic files
> you're
> returning, then it could be decompressing time, unlikely but possible.
>
> Try again with rows=0&start=1000 to see if it's something weird with
> getting
> the stored data, but that's highly doubtful.
>
> I think the only real way to get to the bottom of it will be to slap a
> profiler
> on it and see where the time is being spent.
>
> Best,
> Erick
>
> On Fri, Oct 9, 2015 at 9:53 AM, Toke Eskildsen <t...@statsbiblioteket.dk>
> wrote:
> > Salman Ansari <salman.rah...@gmail.com> wrote:
> >> Thanks Eric for your response. If you find pagination is not the main
> >> culprit, what other factors do you guys suggest I need to tweak to test
> >> that?
> >
> > Well, is basic search slow? What are your response times for plain
> un-warmed top-20 searches?
> >
> >> As I mentioned, by navigating to 20000 results using start and row I
> >> am getting time out from Solr.NET and I need a way to fix that.
> >
> > You still haven't answered my question: Do your users actually need to
> page that far?
> >
> >
> > Again: I know there can be 10 million results. Why would your users need
> to page through all of them? Why would they need to page trough just the
> first 1000? What are they trying to achieve?
> >
> > If they used it automatically for full export of the result set, then I
> can understand it, but you talked about next & previous page, which
> indicates that this is a manual process. A manual process that requires
> clicking next 1000 times is a severe indicator that something can be done
> differently.
> >
> > - Toke Eskildsen
>
