Re: Delete by query in SOLR 6.3

Hi Rakesh, Since Solr has to maintain eventual consistency of all replicas, it has to block updates while DBQ is running. Here is blog post with high level explaination of the issue: http://www.od-bits.com/2018/03/dbq-or-delete-by-query.html

Re: Delete By Query issue followed by Delete By Id Issues

Hi Emir, We are deleting a larger subset of docs with a particular value which we know based on the id and only updating a few of the deleted. Our document is of the form __, we need to delete all that has the same , that are no longer in DB and then update only a few that has been updated in DB.

Re: Delete By Query issue followed by Delete By Id Issues

Hi Sujatha, Did I get it right that you are deleting the same documents that will be updated afterward? If that’s the case, then you can simply skip deleting, and just send updated version of document. Solr (Lucene) does not have delete - it’ll just flag document as deleted. Updating document (a

Re: Delete By Query issue followed by Delete By Id Issues

Thanks,Shawn. Our use case is something like this in a batch load of several 1000's of documents,we do a delete first followed by update.Example delete all 1000 docs and send an update request for 1000. What we see is that there are many missing docs due to DBQ re-ordering of the order of delet

Re: Delete By Query issue followed by Delete By Id Issues

On 6/21/2018 9:59 AM, sujatha sankaran wrote: > Currently from our business perspective we find that we are left with no > options for deleting docs in a batch load as : > > DBQ+ batch does not work well together > DBI+ custom routing (batch load / normal)would not work as well. I would expect

Re: Delete By Query issue followed by Delete By Id Issues

Thanks,Shawn. Currently from our business perspective we find that we are left with no options for deleting docs in a batch load as : DBQ+ batch does not work well together DBI+ custom routing (batch load / normal)would not work as well. We are not sure how we can proceed unless we don't hav

Re: Delete By Query issue followed by Delete By Id Issues

On 6/20/2018 3:46 PM, sujatha sankaran wrote: > Thanks,Shawn. Very useful information. > > Please find below the log details:- Is your collection using the implicit router?  You didn't say.  If it is, then I think you may not be able to use deleteById.  This is indeed a bug, one that has been re

Re: Delete By Query issue followed by Delete By Id Issues

Thanks,Shawn. Very useful information. Please find below the log details:- 2018-06-20 17:19:06.661 ERROR (updateExecutor-2-thread-8226-processing-crm_v2_01_shard3_replica1 x:crm_v2_01_shard3_replica2 r:core_node4 n:masked:8983_solr s:shard3 c:crm_v2_01) [c:crm_v2_01 s:shard3 r:core_node4 x:cr

Re: Delete By Query issue followed by Delete By Id Issues

On 6/15/2018 3:14 PM, sujatha sankaran wrote: We were initially having an issue with DBQ and heavy batch updates which used to result in many missing updates. After reading many mails in mailing list which mentions that DBQ and batch update do not work well together, we switched to DBI. But we

Re: Delete by query, including negative filters

Thanks Erick, The *'s were accidental, if that makes any difference whatsoever. On 09/04/16 15:42, Erick Erickson wrote: Should work, or -merchant_id:(12345 OR 9876*) But do be aware that Solr is not strict boolean logic. The above is close enough for this purpose. Here's an excellent write

Re: Delete by query, including negative filters

Should work, or -merchant_id:(12345 OR 9876*) But do be aware that Solr is not strict boolean logic. The above is close enough for this purpose. Here's an excellent writeup on this subtlety: https://lucidworks.com/blog/2011/12/28/why-not-and-or-and-not/ Best, Erick On Sat, Apr 9, 2016 at 3:51 A

Re: Delete by query using JSON?

Ouch! I certainly did not mean to sound snippy. But perhaps it did, at least to some people. I'll try to be even more PC in the future. I am glad the problem was resolved in the end. Newsletter and resources for Solr beginners and intermediates: http://www.solr-start.com/ On 23 March 2016 a

Re: Delete by query using JSON?

On Tue, Mar 22, 2016 at 10:25:06PM -0400, Jack Krupansky wrote: > See the correct syntax example here: > https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-SendingJSONUpdateCommands > > Your query is fine. Thanks; I thought the query

Re: Delete by query using JSON?

On Tue, Mar 22, 2016 at 04:27:03PM -0700, Walter Underwood wrote: > “Why do you care?” might not be the best way to say it, but it is > essential to understand the difference between selection (filtering) > and ranking. > > As Solr params: > > * q is ranking and filtering > * fq is filtering on

Re: Delete by query using JSON?

See the correct syntax example here: https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-SendingJSONUpdateCommands Your query is fine. -- Jack Krupansky On Tue, Mar 22, 2016 at 3:07 PM, Paul Hoffman wrote: > I've been struggling to

Re: Delete by query using JSON?

“Why do you care?” might not be the best way to say it, but it is essential to understand the difference between selection (filtering) and ranking. As Solr params: * q is ranking and filtering * fq is filtering only * bq is ranking only When deleting documents, ordering does not matter, which i

Re: Delete by query using JSON?

"why do you care? just do this ..." I see this a lot on mailing lists these days, it's usually a learning curve/task/question. I know I fall into these types of questions/tasks regularly. Which usually leads to "don't tell me my approach is wrong, just explain what's going on, and why", or

Re: Delete by query using JSON?

Why do you care? The difference between Q and FQ are the scoring. For delete, you delete all of them regardless of scoring and there is no difference. Just chuck them all into Q. Regards, Alex. Newsletter and resources for Solr beginners and intermediates: http://www.solr-start.com/ On

Re: Delete By query on a multi-value field

: Monday, February 2, 2015 11:44 PM To: solr-user@lucene.apache.org Subject: Re: Delete By query on a multi-value field Hi Jean, Please see the issues https://issues.apache.org/jira/browse/SOLR-3862 https://issues.apache.org/jira/browse/SOLR-5992 Both of them are resolved. The *remove *clause

Re: Delete By query on a multi-value field

Hi Jean, Please see the issues https://issues.apache.org/jira/browse/SOLR-3862 https://issues.apache.org/jira/browse/SOLR-5992 Both of them are resolved. The *remove *clause (atomic update) has been added to 4.9.0 release. Haven't checked it though. Thanks, Lokesh On Tue, Feb 3, 2015 at 7:26 A

Re: Delete by query with soft commit

Hi Jess; Could you check here first: http://search-lucene.com/m/QTPaSxpsW/Commit+Within+and+%252Fupdate%252Fextract+handler&subj=Re+Commit+Within+and+update+extract+handler and then here: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Thanks; Fu

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

Cool, useful info. As soon as I can duplicate the issue I'll work out what we need to do differently for this case. - Mark On Mar 7, 2013, at 10:19 AM, Brett Hoerner wrote: > As an update to this, I did my SolrCloud dance and made it 2xJVMs per > machine (2 machines still, the same ones) and

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

As an update to this, I did my SolrCloud dance and made it 2xJVMs per machine (2 machines still, the same ones) and spread the load around. Each Solr instance now has 16 total shards (master for 8, replica for 8). *drum roll* ... I can repeatedly run my delete script and nothing breaks. :) On Th

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

No, not a poor idea at all, definitely a valid setup. - Mark On Mar 7, 2013, at 9:30 AM, Brett Hoerner wrote: > As a side note, do you think that was a poor idea? I figured it's better to > spread the master "load" around? > > > On Thu, Mar 7, 2013 at 11:29 AM, Mark Miller wrote: > >> >> O

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

As a side note, do you think that was a poor idea? I figured it's better to spread the master "load" around? On Thu, Mar 7, 2013 at 11:29 AM, Mark Miller wrote: > > On Mar 7, 2013, at 9:03 AM, Brett Hoerner wrote: > > > To be clear, neither is really "the replica", I have 32 shards and each >

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

On Mar 7, 2013, at 9:03 AM, Brett Hoerner wrote: > To be clear, neither is really "the replica", I have 32 shards and each > physical server is the leader for 16, and the replica for 16. Ah, interesting. That actually could be part of the issue - some brain cells are firing. I'm away from home

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

Here is the other server when it's locked: https://gist.github.com/3529b7b6415756ead413 To be clear, neither is really "the replica", I have 32 shards and each physical server is the leader for 16, and the replica for 16. Also, related to the max threads hunch: my working cluster has many, many f

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

Any chance you can grab the stack trace of a replica as well? (also when it's locked up of course). - Mark On Mar 6, 2013, at 3:34 PM, Brett Hoerner wrote: > If there's anything I can try, let me know. Interestingly, I think I have > noticed that if I stop my indexer, do my delete, and restart

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

If there's anything I can try, let me know. Interestingly, I think I have noticed that if I stop my indexer, do my delete, and restart the indexer then I'm fine. Which goes along with the update thread contention theory. On Wed, Mar 6, 2013 at 5:03 PM, Mark Miller wrote: > This is what I see: >

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

This is what I see: We currently limit the number of outstanding update requests at one time to avoid a crazy number of threads being used. It looks like a bunch of update requests are stuck in socket reads and are taking up the available threads. It looks like the deletes are hanging out wait

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

It does not look like a deadlock, though it could be a distributed one. Or it could be a livelock, though that's less likely. Here is what we used to recommend in similar situations for large Java systems (BEA Weblogic): 1) Do thread dump of both systems before anything. As simultaneous as you can

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

Thans Brett, good stuff (though not a good problem). We def need to look into this. - Mark On Mar 6, 2013, at 1:53 PM, Brett Hoerner wrote: > Here is a dump after the delete, indexing has been stopped: > https://gist.github.com/bretthoerner/c7ea3bf3dc9e676a3f0e > > An interesting hint that I

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

Here is a dump after the delete, indexing has been stopped: https://gist.github.com/bretthoerner/c7ea3bf3dc9e676a3f0e An interesting hint that I forgot to mention: it doesn't always happen on the first delete. I manually ran the delete cron, and the server continued to work. I waited about 5 minut

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

4.1, I'll induce it again and run jstack. On Wed, Mar 6, 2013 at 1:50 PM, Mark Miller wrote: > Which version of Solr? > > Can you use jconsole, visualvm, or jstack to get some stack traces and see > where things are halting? > > - Mark > > On Mar 6, 2013, at 11:45 AM, Brett Hoerner wrote: > >

Re: Delete By Query suddenly halts indexing on SolrCloud cluster

Which version of Solr? Can you use jconsole, visualvm, or jstack to get some stack traces and see where things are halting? - Mark On Mar 6, 2013, at 11:45 AM, Brett Hoerner wrote: > I have a SolrCloud cluster (2 machines, 2 Solr instances, 32 shards, > replication factor of 2) that I've been

Re: Delete by Query not working properly

Yes, text/xml is deprecated. But it is less typing for curl commands! Why Solr doesn't default to application/xml is a mystery. -- Jack Krupansky -Original Message- From: Walter Underwood Sent: Tuesday, December 18, 2012 10:47 AM To: solr-user@lucene.apache.org Subject: Re: Dele

Re: Delete by Query not working properly

Never use text/xml, always application/xml. wunder On Dec 18, 2012, at 6:03 AM, Jack Krupansky wrote: > Make sure that your curl command includes: > > -H "Content-Type: text/xml" > > For example, > > curl http://localhost:8983/solr/update?commit=true -H "Content-Type: > text/xml" --data-bina

Re: Delete by Query not working properly

ecember 18, 2012 9:03 AM To: solr-user@lucene.apache.org Subject: Re: Delete by Query not working properly Make sure that your curl command includes: -H "Content-Type: text/xml" For example, curl http://localhost:8983/solr/update?commit=true -H "Content-Type: text/xml" --d

Re: Delete by Query not working properly

Make sure that your curl command includes: -H "Content-Type: text/xml" For example, curl http://localhost:8983/solr/update?commit=true -H "Content-Type: text/xml" --data-binary ' id:doc-2012-12-18' -- Jack Krupansky -Original Message- From: Dixline Sent: Tuesday, December 18, 201

Re: Delete by Query not working properly

Surely you are deleting documents that have the term Perl in the skills_s field, but are leaving behind another document that has Perl in the default field (usually 'text'). Before doing the delete, do 'q=Perl skills_s:Perl' and see if you get more than one document. That's what it looks like anyh

Re: delete by query don't work

In order to clear all the indexed data please try to use this code private void Btn_Delete_Click(object sender, EventArgs e) { var solrUrl = this.textBoxSolrUrl.Text; indexer.FixtureSetup(solrUrl); indexer.Delete(); MessageBox.Sho

Re: delete by query don't work

On Mon, 2012-06-18 at 11:45 +0200, ramzesua wrote: > Hi all. I am using solr 4.0 and trying to clear index by query. At first I > use *:* with commit, but index is still not > empty. I tried another queries, but it not help me. Then I tried delete by > `id`. It works fine, but I need clear all inde

Re: delete by query don't work

Well, it would help if you defined what behavior you're seeing. When you say delete-by-query doesn't work, what is the symptom? What does "empty" mean? Because if you're just looking at your index directory and expecting to see files disappear, you'll be disappointed. When you delete documents in

Re: Delete by Query with limited number of rows

Hi Erick, hi Yury, thanks to your input I found a perfect solution for my case. Even though this is not a solr-only solution, I will just briefly describe how it works since it might be of interest to others: I have put up a mysql database holding two tables. The first only has a primarykey with

Re: Delete by Query with limited number of rows

There's nothing built into Solr that lets you do this automatically. About the best you can do is probably a delete by query going back some fixed time interval. So rather than keeping the last N documents, you keep documents that are, say, no more than 1 month old (or whatever you determine your i

Re: Delete by Query with limited number of rows

Hi Yury, thank you very much for your quick reply. Currently I have a timestamp field (solr.DateField) and every time I add a document I use "NOW" for the timestamp field. I only commit documents on the core every four hours. This works fine with the timestamp since I can use "NOW". However, I cou

Re: Delete by Query with limited number of rows

On 11/12/2011 4:08 PM, mikr00 wrote: > Similar to a first in first out list. The problem is: It's easy to check the > limit, but how can I delete the oldest documents to go again below the > limit? Can I do it with a delete by query request? In that case, I would > probably have to limit the number

Re: delete by query

i resolved: http://10.0.0.178:8983/solr/update?stream.body= (field1:value1)AND(field2:value2) Thanx 2011/3/28 Gastone Penzo > Hi, > i want to use delete by query method to delete indexes. > i try for example: > > http://10.0.0.178:8983/solr/update?stream.body= > field1:value > > and it works >

Re: Delete by query or Id very slow

Hi, As Tom suggested removing optimize and passing the ids as list (instead of for loop) will surely increase the speed of deletion. We have a program which fetches complete list of ID from back end (around 10 million) and compares it with the complete list of id's present in SOLR document and d

Re: Delete by query or Id very slow

I'd bet it's the optimize that's taking the time, and not the delete. You don't really need to optimize these days, and you certainly don't need to do it on every delete. And you can give solr a list of ids to delete, which would be more efficient. I don't believe you can tell which ones have fai

Re: Delete by query or Id very slow

Thank you Tom for responding. On an average the docs are around 25-35 KB. The code is as follows, Kindly let me know if you see anything weird, a second pair of eyes always helps :-) public List deleteDocs(List ids) throws SolrCustomException { CommonsHttpSolrServer server = (CommonsHt

Re: Delete by query or Id very slow

That''s a pretty low number of documents for auto complete. It means that when getting to 850,000 documents, you will create 8500 segments, and that's not counting merges. How big are your documents? I just created an 850,000 document (and a 3.5 m doc index) with tiny documents (id and title), and

Re: Delete by query issue

: Here's the problem: the standard Solr parser is a little weird about : negative queries. The way to make this work is to say : *:* AND -field:[* TO *] the default parser actually works ok ... it's a bug specific to deletion... https://issues.apache.org/jira/browse/SOLR-381 -Hoss

Re: Delete by query issue

Thanks Lance. I'll give that a try going forward. On Wed, Aug 25, 2010 at 9:59 PM, Lance Norskog wrote: > Here's the problem: the standard Solr parser is a little weird about > negative queries. The way to make this work is to say >*:* AND -field:[* TO *] > > This means "select everything A

Re: Delete by query issue

Here's the problem: the standard Solr parser is a little weird about negative queries. The way to make this work is to say *:* AND -field:[* TO *] This means "select everything AND only these documents without a value in the field". On Wed, Aug 25, 2010 at 7:55 PM, Max Lynch wrote: > I was t

Re: Delete by query issue

I was trying to filter out all documents that HAVE that field. I was trying to delete any documents where that field had empty values. I just found a way to do it, but I did a range query on a string date in the Lucene DateTools format and it worked, so I'm satisfied. However, I believe it worke

Re: Delete by query issue

Excuse me, what's the hyphen before the field name 'date_added_solr'? Is this some kind of new query format that I didn't know? -date_added_solr:[* TO *]' - Original Message - From: "Max Lynch" To: Sent: Thursday, August 26, 2010 6:12 AM Subject: Delete by query issue > Hi, > I am

Re: Delete by query discrepancy

Cool, thanks - just wanted to make sure I'm not insane. Makes sense that there would be a difference if the index is built fresh in that case. On Tue, Feb 16, 2010 at 11:59, Mark Miller wrote: > Mat Brown wrote: >> Hi all, >> >> Trying to debug a very sneaky bug in a small Solr extension that I >

Re: Delete by query discrepancy

Mat Brown wrote: > Hi all, > > Trying to debug a very sneaky bug in a small Solr extension that I > wrote, and I've come across an odd situation. Here's what my test > suite does: > > deleteByQuery("*:*"); > // add some documents > commit(); > // test the search > > This works fine. The test suite

Re: Delete by query

The problem is that negative queries are ignored by Lucene. Why this is still true I have no idea. On Mon, Jan 25, 2010 at 6:23 AM, Noam G. wrote: > > Hi David, > > Thank you very much - that did the trick :-) > > Noam. > -- > View this message in context: > http://old.nabble.com/Delete-by-query

RE: Delete by query

Hi David, Thank you very much - that did the trick :-) Noam. -- View this message in context: http://old.nabble.com/Delete-by-query-tp27306968p27307336.html Sent from the Solr - User mailing list archive at Nabble.com.

RE: Delete by query

solrServer.deleteByQuery("*:* AND -version_uuid:e04534e2-28db-4420-a5f3-300477872c11"); Should do the trick. -Original Message- From: Noam G. [mailto:noam...@gmail.com] Sent: 25 January 2010 13:58 To: solr-user@lucene.apache.org Subject: Delete by query Hi, I have an index with 3M doc

Re: Delete by query isn't working for a specific query

Thank you both for your nice answers. I will try it out. 2008/10/27 Erik Hatcher <[EMAIL PROTECTED]> > I don't think delete-by-query supports purely negative queries, even though > they are supported for q and fq parameters for searches. > > Try using: > > *:* AND -deptId:[1 TO *] > >Er

Re: Delete by query isn't working for a specific query

I don't think delete-by-query supports purely negative queries, even though they are supported for q and fq parameters for searches. Try using: *:* AND -deptId:[1 TO *] Erik On Oct 27, 2008, at 9:21 AM, Alexander Ramos Jardim wrote: Hey pals, I am trying to delete a couple docu

Re: Delete by query isn't working for a specific query

Alexander Ramos Jardim wrote: Hey pals, I am trying to delete a couple documents that don't have any value on a given integer field. This is the command I am executing: $curl http://:/solr/update -H 'Content-Type:text/xml' -d '-(deptId:[1 TO *])' $curl http://:/solr/update -H 'Content-Type:text

Re: Delete by Query is not deleting

I know what the problem was, the query let use someones characters without escapes, but using delete by query don't. thanks, see you ;-) Athok wrote: > > Hello, I am begining with solr and i have a problem with the delete by > query. If i do a query to solr, it give me the results that i hope

Re: Delete by Query is not deleting

Yes, when the file is index, i send a commit Noble Paul നോബിള്‍ नोब्ळ् wrote: > > are you sure you committed after the 'delete' ? > > On Wed, Sep 10, 2008 at 2:26 PM, Athok <[EMAIL PROTECTED]> wrote: >> >> Hello, I am begining with solr and i have a problem with the delete by >> query. >> If i

Re: Delete by Query is not deleting

are you sure you committed after the 'delete' ? On Wed, Sep 10, 2008 at 2:26 PM, Athok <[EMAIL PROTECTED]> wrote: > > Hello, I am begining with solr and i have a problem with the delete by query. > If i do a query to solr, it give me the results that i hope but when the > query is sent by XML as a

Re: delete by query multiple Ids

On Jun 26, 2007, at 6:46 AM, michael ravits wrote: hello solrs is it possible to query multiple specific ids? something like this: mediaId:6720,6721,6722,8762,9754 sure, but you need to use proper query parser syntax: mediaId:(6720 OR 6721 OR )