Emir, Using tlog replica types addresses my immediate problem.
The secondary issue is that all of our searches show inconsistent results. These are all normal paging use cases. We regularly test our relevancy, and these differences creates confusion in the testers. Moreover, we are migrating from Endeca which has very consistent results. I'm hoping that using the global stats cache will make the other searches more stable. I think we will eventually move to favoring tlog replicas. We have a couple of collections where NRT makes sense, but those collections don't need to return data in relevancy order. I think NRT should be considered a niche use case for a search engine, tlog and pull replicas are a much better fit for a search engine (imho) On Tue, Feb 27, 2018 at 4:01 AM, Emir Arnautović < emir.arnauto...@sematext.com> wrote: > Hi Webster, > Since you are returning all hits, returning the last page is almost as > heavy for Solr as returning all documents. Maybe you should consider just > returning one large page and completely avoid this issue. > I agree with you that this should be handled by Solr. ES solved this issue > with “preference” search parameter where you can set session id as > preference and it will stick to the same shards. I guess you could try > similar thing on your own but that would require you to send list of shards > as parameter for your search and balance it for different sessions. > > HTH, > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > > > On 26 Feb 2018, at 21:03, Webster Homer <webster.ho...@sial.com> wrote: > > > > Erick, > > > > No we didn't look at that. I will add it to the list. We have not seen > > performance issues with solr. We have much slower technologies in our > > stack. This project was to replace a system that was too slow. > > > > Thank you, I will look into it > > > > Webster > > > > On Mon, Feb 26, 2018 at 1:13 PM, Erick Erickson <erickerick...@gmail.com > > > > wrote: > > > >> Did you try enabling distributed IDF (statsCache)? See: > >> https://lucene.apache.org/solr/guide/6_6/distributed-requests.html > >> > >> It's may not totally fix the issue, but it's worth trying. It does > >> come with a performance penalty of course. > >> > >> Best, > >> Erick > >> > >> On Mon, Feb 26, 2018 at 11:00 AM, Webster Homer <webster.ho...@sial.com > > > >> wrote: > >>> Thanks Shawn, I had settled on this as a solution. > >>> > >>> All our use cases for Solr is to return results in order of relevancy > to > >>> the query, so having a deterministic sort would defeat that purpose. > >> Since > >>> we wanted to be able to return all the results for a query, I > originally > >>> looked at using the Streaming API, but that doesn't support returning > >>> results sorted by relevancy > >>> > >>> I disagree with you about NRT replicas though. They may function as > >>> designed, but since they cannot guarantee consistent results their > design > >>> is buggy, at least it is for a search engine. > >>> > >>> > >>> On Mon, Feb 26, 2018 at 12:20 PM, Shawn Heisey <apa...@elyograg.org> > >> wrote: > >>> > >>>> On 2/26/2018 10:26 AM, Webster Homer wrote: > >>>>> We need the results by relevancy so the application sorts the results > >> by > >>>>> score desc, and the unique id ascending as the tie breaker > >>>> > >>>> This is the reason for the discrepancy, and why the different replica > >>>> types don't have the same issue. > >>>> > >>>> Each NRT replica can have different deleted documents than the others, > >>>> just due to the way that NRT replicas work. Deleted documents affect > >>>> relevancy scoring. When one replica has say 5000 deleted documents > and > >>>> another has 200, or has 5000 but they're different docs, a relevancy > >>>> sort can end up different. So when Solr goes to one replica for page > 1 > >>>> and another for page 2 (which is expected due to SolrCloud's internal > >>>> load balancing), you may end up with duplicate documents or documents > >>>> missing. Because deleted documents are not counted or returned, > >>>> numFound will be consistent, as long as the index doesn't change > between > >>>> the queries for pages. > >>>> > >>>> If you were using a deterministic sort rather than relevancy, this > >>>> wouldn't be happening, because deleted documents have no influence on > >>>> that kind of sort. > >>>> > >>>> With TLOG or PULL, the replicas are absolutely identical, so there is > no > >>>> difference, unless the index is changing as you page through the > >> results. > >>>> > >>>> I think changing replica types is the only solution here. NRT > replicas > >>>> are working as they were designed -- there's no bug, even though > >>>> problems like this do sometimes turn up. > >>>> > >>>> Thanks, > >>>> Shawn > >>>> > >>>> > >>> > >>> -- > >>> > >>> > >>> This message and any attachment are confidential and may be privileged > or > >>> otherwise protected from disclosure. If you are not the intended > >> recipient, > >>> you must not copy this message or attachment or disclose the contents > to > >>> any other person. If you have received this transmission in error, > please > >>> notify the sender immediately and delete the message and any attachment > >>> from your system. Merck KGaA, Darmstadt, Germany and any of its > >>> subsidiaries do not accept liability for any omissions or errors in > this > >>> message which may arise as a result of E-Mail-transmission or for > damages > >>> resulting from any unauthorized changes of the content of this message > >> and > >>> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its > >>> subsidiaries do not guarantee that this message is free of viruses and > >> does > >>> not accept liability for any damages caused by any virus transmitted > >>> therewith. > >>> > >>> Click http://www.emdgroup.com/disclaimer to access the German, French, > >>> Spanish and Portuguese versions of this disclaimer. > >> > > > > -- > > > > > > This message and any attachment are confidential and may be privileged or > > otherwise protected from disclosure. If you are not the intended > recipient, > > you must not copy this message or attachment or disclose the contents to > > any other person. If you have received this transmission in error, please > > notify the sender immediately and delete the message and any attachment > > from your system. Merck KGaA, Darmstadt, Germany and any of its > > subsidiaries do not accept liability for any omissions or errors in this > > message which may arise as a result of E-Mail-transmission or for damages > > resulting from any unauthorized changes of the content of this message > and > > any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its > > subsidiaries do not guarantee that this message is free of viruses and > does > > not accept liability for any damages caused by any virus transmitted > > therewith. > > > > Click http://www.emdgroup.com/disclaimer to access the German, French, > > Spanish and Portuguese versions of this disclaimer. > > -- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith. Click http://www.emdgroup.com/disclaimer to access the German, French, Spanish and Portuguese versions of this disclaimer.
