Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-28 Thread Shawn Heisey
On 10/27/2011 5:56 AM, Michael Sokolov wrote: From everything you've said, it certainly sounds like a low-level I/O problem in the client, not a server slowdown of any sort. Maybe Perl is using the same connection over and over (keep-alive) and Java is not. I really don't know. One thing I'v

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-27 Thread Shawn Heisey
On 10/27/2011 5:56 AM, Michael Sokolov wrote: From everything you've said, it certainly sounds like a low-level I/O problem in the client, not a server slowdown of any sort. Maybe Perl is using the same connection over and over (keep-alive) and Java is not. I really don't know. One thing I'v

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-27 Thread Shawn Heisey
On 10/27/2011 1:36 AM, Michael Kuhlmann wrote: Why do you first query for these documents? Why don't you just delete them? Solr won't harm if no documents are affected by your delete query, and you'll get the number of affected documents in your response anyway. When deleting, Solrj nearly does

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-27 Thread Michael Sokolov
From everything you've said, it certainly sounds like a low-level I/O problem in the client, not a server slowdown of any sort. Maybe Perl is using the same connection over and over (keep-alive) and Java is not. I really don't know. One thing I've heard is that StreamingUpdateSolrServer (I t

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-27 Thread Michael Kuhlmann
Sorry, I was wrong. Am 27.10.2011 09:36, schrieb Michael Kuhlmann: > and you'll get the number of affected documents in your response anyway. That's not true, you don't get the affected document count. Anyway, it's still true that you don't need to check for documents first, at least not when you

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-27 Thread Michael Kuhlmann
Am 26.10.2011 18:29, schrieb Shawn Heisey: > For inserting, I do use a Collection of SolrInputDocuments. The delete > process grabs values from idx_delete, does a query like the above (the > part that's slow in Java), then if any documents are found, issues a > deleteByQuery with the same string.

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-26 Thread Shawn Heisey
On 10/26/2011 6:16 PM, Michael Sokolov wrote: Have you checked to see when you are committing? Is the pattern the same in both instances? If you are committing after each delete request in Java, but not in Perl, that could slow things down. The commit happens separately, not during the proce

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-26 Thread Michael Sokolov
Have you checked to see when you are committing? Is the pattern the same in both instances? If you are committing after each delete request in Java, but not in Perl, that could slow things down. On 10/25/2011 5:53 PM, Shawn Heisey wrote: On 10/20/2011 11:00 AM, Shawn Heisey wrote: I've got t

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-26 Thread Shawn Heisey
On 10/26/2011 10:29 AM, Shawn Heisey wrote: One possible thing I can do to make the Java code even faster is to set rows to zero before doing the query, since I only need numFound, not the actual results. The Perl code does NOT do this, and yet it's super fast. It turns out I already thought

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-26 Thread Shawn Heisey
On 10/26/2011 1:30 AM, Michael Kuhlmann wrote: Hi, Am 25.10.2011 23:53, schrieb Shawn Heisey: On 10/20/2011 11:00 AM, Shawn Heisey wrote: [...] I've noticed a performance discrepancy when processing every one of my delete records, currently about 25000 of them. I din't understand what a delet

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-26 Thread Michael Kuhlmann
Hi, Am 25.10.2011 23:53, schrieb Shawn Heisey: > On 10/20/2011 11:00 AM, Shawn Heisey wrote: >> [...] I've noticed a performance discrepancy when >> processing every one of my delete records, currently about 25000 of >> them. I din't understand what a delete record is. Do you delete records in So

Re: Query/Delete performance difference between straight HTTP and SolrJ

2011-10-25 Thread Shawn Heisey
On 10/20/2011 11:00 AM, Shawn Heisey wrote: I've got two build systems for my Solr index that I wrote. The first one is in Perl and uses GET/POST requests via HTTP, the second is in Java using SolrJ. I've noticed a performance discrepancy when processing every one of my delete records, curren