bq: you think it is caused by the DBQ deleting a document while a document with that same ID
No. I'm saying that DBQ has no idea _if_ that would be the case so can't carry out the operations in parallel because it _might_ be the case. Shawn: IIUC, here's the problem. For deleteById, I can guarantee the sequencing through the same optimistic locking that regular updates use (i.e. the _version_ field). But I'm kind of guessing here. Best, Erick On Tue, Nov 7, 2017 at 8:51 AM, Shawn Heisey <apa...@elyograg.org> wrote: > On 11/5/2017 12:20 PM, Chris Troullis wrote: >> The issue I am seeing is when some >> threads are adding/updating documents while other threads are issuing >> deletes (using deleteByQuery), solr seems to get into a state of extreme >> blocking on the replica > > The deleteByQuery operation cannot coexist very well with other indexing > operations. Let me tell you about something I discovered. I think your > problem is very similar. > > Solr 4.0 and later is supposed to be able to handle indexing operations > at the same time that the index is being optimized (in Lucene, > forceMerge). I have some indexes that take about two hours to optimize, > so having indexing stop while that happens is a less than ideal > situation. Ongoing indexing is similar in many ways to a merge, enough > that it is handled by the same Merge Scheduler that handles an optimize. > > I could indeed add documents to the index without issues at the same > time as an optimize, but when I would try my full indexing cycle while > an optimize was underway, I found that all operations stopped until the > optimize finished. > > Ultimately what was determined (I think it was Yonik that figured it > out) was that *most* indexing operations can happen during the optimize, > *except* for deleteByQuery. The deleteById operation works just fine. > > I do not understand the low-level reasons for this, but apparently it's > not something that can be easily fixed. > > A workaround is to send the query you plan to use with deleteByQuery as > a standard query with a limited fl parameter, to retrieve matching > uniqueKey values from the index, then do a deleteById with that list of > ID values instead. > > Thanks, > Shawn >
