Re: lots of inserts very fast, out of heap or file descs

2007-02-24 Thread Brian Whitman
On Feb 24, 2007, at 1:26 PM, gmail wrote: do you have a script/data that makes this happen? all you've got to do is apache-solr-nightly/example/exampledocs ryan$ while [ 0 -lt 1 ]; do ./ post.sh hd.xml; done with the request handler pointing to /update. Use # lsof | grep solr | wc -l to

Re: lots of inserts very fast, out of heap or file descs

2007-02-24 Thread gmail
do you have a script/data that makes this happen? I'm on a windows dev box - it does not get "too many open files" but i'll figure it out. ryan

Re: lots of inserts very fast, out of heap or file descs

2007-02-24 Thread Brian Whitman
On Feb 24, 2007, at 1:16 AM, Chris Hostetter wrote: Based on Brain's email, it sounds like it didn't work in *exactly* the same way, because it caused some filedescriptor leaks (and possibly some memory leaks) Hopefully Ryan will be a rock star and spot the probably immediately -- Thank

Re: lots of inserts very fast, out of heap or file descs

2007-02-24 Thread Ryan McKinley
it sounds like we may have a very bad bug in the XmlUpdateRequestHandler I haven't looked at this yet, but if i understand the description, it would have to be a problem with the SolrDispatchFilter and/or the SolrRequestParsers. the part this *is* exactly the same is the XmlUpdateRequestHandl

Re: lots of inserts very fast, out of heap or file descs

2007-02-23 Thread Chris Hostetter
i'll try to look into it later this weekend. : Date: Fri, 23 Feb 2007 22:33:10 -0500 : From: Brian Whitman <[EMAIL PROTECTED]> : Reply-To: solr-user@lucene.apache.org : To: solr-user@lucene.apache.org : Subject: Re: lots of inserts very fast, out of heap or file descs : : On Feb

Re: lots of inserts very fast, out of heap or file descs

2007-02-23 Thread Brian Whitman
On Feb 23, 2007, at 8:31 PM, Yonik Seeley wrote: -- it does not go down until I restart solr. This would be the cause of my too many files open problem. Turning off autocommit / not commiting after every add keeps this count steady at 100-200. The files are all of type: [...] Bug or feature?

Re: lots of inserts very fast, out of heap or file descs

2007-02-23 Thread Yonik Seeley
On 2/23/07, Brian Whitman <[EMAIL PROTECTED]> wrote: > > Try not committing so often (perhaps until you are done). > Don't use post.sh, or modify it to remove the commit. > OK, I modified it to not commit after and I also realized I had SOLR-126 (autocommit) on, which I disabled. Is there a rule

Re: lots of inserts very fast, out of heap or file descs

2007-02-23 Thread Brian Whitman
Try not committing so often (perhaps until you are done). Don't use post.sh, or modify it to remove the commit. OK, I modified it to not commit after and I also realized I had SOLR-126 (autocommit) on, which I disabled. Is there a rule of thumb on when to commit / optimize? Part of the

Re: lots of inserts very fast, out of heap or file descs

2007-02-23 Thread Mike Klaas
On 2/23/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 2/23/07, Brian Whitman <[EMAIL PROTECTED]> wrote: > I'm trying to add lots of documents at once (hundreds of thousands) > in a loop. I don't need these docs to appear as results until I'm > done, though. > > For a simple test, I call the post

Re: lots of inserts very fast, out of heap or file descs

2007-02-23 Thread Yonik Seeley
On 2/23/07, Brian Whitman <[EMAIL PROTECTED]> wrote: I'm trying to add lots of documents at once (hundreds of thousands) in a loop. I don't need these docs to appear as results until I'm done, though. For a simple test, I call the post.sh script in a loop with the same moderately sized xml file.

lots of inserts very fast, out of heap or file descs

2007-02-23 Thread Brian Whitman
I'm trying to add lots of documents at once (hundreds of thousands) in a loop. I don't need these docs to appear as results until I'm done, though. For a simple test, I call the post.sh script in a loop with the same moderately sized xml file. This adds a 20K doc and then commits. Repeat