RE: Ghost Documents or Shards out of Sync
Hi, Thank you your replies - much appreciated! I'm afraid it's neither one... Let me add some background. A user triggers an operation which under the hood needs to update a single field. Atomic update fails with a message that one of the mandatory fields is missing (which is strange by itself). When I query Solr for the exact document (fq with the document id) I sometimes get the expected single result and sometimes zero. Those queries are done sometimes couple of days later so auto commits necessarily have been performed. Only after a restart, the query is working and then atomic update succeeds. One more thing that might be important - we're using nested schema, and we recently encountered several issues that make me think that this combination - nested and atomic updates (of parent documents) - is the root cause. Ronen. -Original Message- From: Mike Drob Sent: יום ב 01 פברואר 2021 22:58 To: solr-user@lucene.apache.org Subject: Re: Ghost Documents or Shards out of Sync To expand on what Jason suggested, if the issue is the non-deterministic ordering due to staggered commits per replica, you may have more consistency with TLOG replicas rather than the NRT replicas. In this case, the underlying segment files should be identical and lead to more predictable results. On Mon, Feb 1, 2021 at 2:50 PM Jason Gerlowski wrote: > Hi Ronen, > > The first thing I'd figure out in your situation is whether the > results are actually different each time, or whether the ordering is > what differs (which might push a particular result off the page you're > looking at, giving the appearance that it didn't match). > > In the case of the former, this can happen briefly if queries come in > when some but not all replicas have seen a commit. But usually this > is a transient concern - either waiting for the next autocommit or > triggering an explicit commit resolves the discrepancy in this case. > Since you only see identical results after a restart, this _doesn't_ > sound like what you're seeing. > > In the case of the latter (same results, differently ordered) this is > expected sometimes. Solr sorts on relevance by default with the > internal Lucene document ID being a tiebreaker. Both the relevance > statistics and Lucene's document IDs can differ across SolrCloud > replicas (due to non-deterministic conditions such as the segment > merging and deleted-doc removal that Lucene does under the hood), and > this can produce differently-ordered result sets for users that issue > the same query repeatedly. > > Good luck narrowing things down! > > Jason > > On Mon, Jan 25, 2021 at 3:32 AM Ronen Nussbaum wrote: > > > > Hi All, > > > > I'm using Solr Cloud (version 8.3.0) with shards and replicas > (replication > > factor of 2). > > Recently, I've encountered several times that running the same query > > repeatedly yields different results. Restarting the nodes fixes the > problem > > (until next time). > > I assume that some shards are not synchronized and I have several > questions: > > 1. What can cause this - many atomic updates? issues with commits? > > 2. Can I trigger the "fixing" mechanism that Solr runs at restart by > > an > API > > call or some other method? > > > > Thanks in advance, > > Ronen. > This electronic message may contain proprietary and confidential information of Verint Systems Inc., its affiliates and/or subsidiaries. The information is intended to be for the use of the individual(s) or entity(ies) named above. If you are not the intended recipient (or authorized to receive this e-mail for the intended recipient), you may not use, copy, disclose or distribute to anyone this message or any information contained in this message. If you have received this electronic message in error, please notify us by replying to this e-mail.
Re: Authentication for all but selects
Thanks Craig!
I got the following to work.
{
"authentication":{
"blockUnknown":false,
"class":"solr.BasicAuthPlugin",
"credentials":{"solr":"..."}},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"user-role":{"solr":"admin"},
"permissions":[
{
"name":"all",
"role":"admin",
"index":1},
{
"name":"open_select",
"collection":"*",
"path":"/select",
"role":null,
"index":2}],
"":{"v":0}}}
On 2/5/21, 3:35 PM, "Oakley, Craig (NIH/NLM/NCBI) [C]"
wrote:
What works for us is having something like this at the bottom of
security.json:
{
"name":"open_select",
"path":"/select/*",
"role":null,
"index":9},
{
"name":"catch-all-nocollection",
"collection":null,
"path":"/*",
"role":"allgen",
"index":10},
{
"name":"catch-all-collection",
"path":"/*",
"role":"allgen",
"index":11}],
"":{"v":9}}}
The clause with the name open_select specifically allows selects to run
without any role ("role":null)
The last two clauses say that anything else (with any collection and
without any collection) requires allgen role: and that is a role that I grant
to all users generally
Other permissions can go higher up in security.json (disallowing normal
users from running DELETEREPLICA, and things like that); but these are the
three clauses which I think should allow select without any login (and without
any password), while everything else does require a login and password.
-Original Message-
From: Robert Douglas
Sent: Friday, February 05, 2021 1:19 PM
To: solr-user@lucene.apache.org
Subject: Authentication for all but selects
Hello all,
We are working on some migrations and we want to be incorporating
authentication more uniformly across all our installations of Solr, but we are
getting stuck on allowing Select statements through without authentication
while having authentication on with RBAP for everything else. For some of our
apps the authentication for Selects isn’t an issue but for others, where we
can’t really touch the code, it is.
Is there a way of doing this?
Cheers,
R
Robert Douglas
DevOps Engineer
Cornell University Library
