Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Thanks Yonik. I'm not a java developer by trade, but the objects mentioned in the heap dump along with the objects mentioned in the jira issue sound eerily familiar to me. Last year, we encountered a memory leak on the Lucene.Net project, stemming from the FieldCache class having references left

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Yonik Seeley
On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: Yonik - is this the JIRA entry you're referring to? http://issues.apache.org/jira/browse/LUCENE-754 Yes. But from the heap dump you provided, that doesn't look like the issue. -Yonik

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Yonik - is this the JIRA entry you're referring to? http://issues.apache.org/jira/browse/LUCENE-754 On 4/2/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: > We are doing incremental updates, and we optimize quite a bit. mergeFactor > presentl

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Major version is 1.0. The bits are from a nightly build from early September 2006. We do have plans to upgrade solr soon. On 4/2/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: > We are doing incremental updates, and we optimize quite a bit. m

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Sorry for the confusion. We do have caching disabled. I was asking the question because I wasn't certain if the configurable cache settings applied throughout, or if the FieldCache in lucene still came in play. The two integer-based facets are single valued per document. The string-based facet

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Yonik Seeley
On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: We are doing incremental updates, and we optimize quite a bit. mergeFactor presently set to 10. maxDoc count = 144156 numDocs count = 144145 What version of Solr are you using? Another potential OOM (multiple threads generating the same Fie

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Thanks for the pointers, Mike. I'm trying to determine the math to resolve some strange numbers we're seeing. Here's the top dozen lines from a jmap analysis on a heap dump: SizeCount Class description - 428246064 1792204 i

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Chris Hostetter
: values = 50. More to the point, is there an algorithm that I can use to : estimate the cache consumption rate for facet queries? I'm confused ... i thought you said in your orriginal mail that you had all the caching disabled? (except for FieldCache which is so low level in Lucene it's always

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
On 4/2/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 4/1/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: > Our scenario: 150MB index, 14 documents, read/write servers in place > using standard replication. Running Tomcat 5.5.17 on Redhat Enterprise > Linux 4. Java configured to start with -

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Yonik Seeley
On 4/1/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: Our scenario: 150MB index, 14 documents, read/write servers in place using standard replication. Running Tomcat 5.5.17 on Redhat Enterprise Linux 4. Java configured to start with -Xmx1024m. We encounter java heap out-of-memory issues on

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Mike Klaas
On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: Hoping I can get a better response with a more directed question: I haven't answered your original question as it seems that general java memory debugging techniques would be the most useful thing here. With facet queries and the fields use

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Hoping I can get a better response with a more directed question: With facet queries and the fields used, what qualifies as a "large" number of values? The wiki uses U.S. states as an example, so the number of unique values = 50. More to the point, is there an algorithm that I can use to estima

Troubleshooting java heap out-of-memory

2007-04-01 Thread Jeff Rodenburg
I've read through the list entries here, the Lucene list, and the wiki docs and am not resolving a major pain point for us. We've been trying to determine what could possibly cause us to hit this in our given environment, and am hoping more eyes on this issue can help. Our scenario: 150MB index