Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Thanks Yonik. I'm not a java developer by trade, but the objects mentioned in the heap dump along with the objects mentioned in the jira issue sound eerily familiar to me. Last year, we encountered a memory leak on the Lucene.Net project, stemming from the FieldCache class having references left

Re: embedding solr

2007-04-02 Thread Erik Hatcher
Yonik and Ryan, Thank you for the quick and helpful responses. I'll have to do some hacking from here on out and see where I get to, but I'm happy to know I'm in good company and that what I'm attempting is on a path already slightly worn. :) Erik On Apr 2, 2007, at 9:55 PM, Ry

Re: embedding solr

2007-04-02 Thread Ryan McKinley
I have embedded solr skipping HTTP transport altogether. It was remarkably easy to link directly to request handlers skipping the dispatch filter and using the DocList and associated data in the SolrQueryResponse directly. Assuming the existing TCP/IP interface is sending strings around, filling

Re: embedding solr

2007-04-02 Thread Yonik Seeley
On 4/2/07, Erik Hatcher <[EMAIL PROTECTED]> wrote: I have a client need to embed Solr behind an already built custom TCP/ IP interface (currently for Lucene, but want to swap in Solr to benefit from its additional goodness of course). Have folks already done this? Experiences? Or perhaps th

embedding solr

2007-04-02 Thread Erik Hatcher
I have a client need to embed Solr behind an already built custom TCP/ IP interface (currently for Lucene, but want to swap in Solr to benefit from its additional goodness of course). Have folks already done this? Experiences? Or perhaps there are some thoughts on why this may or may n

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Yonik Seeley
On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: Yonik - is this the JIRA entry you're referring to? http://issues.apache.org/jira/browse/LUCENE-754 Yes. But from the heap dump you provided, that doesn't look like the issue. -Yonik

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Yonik - is this the JIRA entry you're referring to? http://issues.apache.org/jira/browse/LUCENE-754 On 4/2/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: > We are doing incremental updates, and we optimize quite a bit. mergeFactor > presentl

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Major version is 1.0. The bits are from a nightly build from early September 2006. We do have plans to upgrade solr soon. On 4/2/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: > We are doing incremental updates, and we optimize quite a bit. m

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Sorry for the confusion. We do have caching disabled. I was asking the question because I wasn't certain if the configurable cache settings applied throughout, or if the FieldCache in lucene still came in play. The two integer-based facets are single valued per document. The string-based facet

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Yonik Seeley
On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: We are doing incremental updates, and we optimize quite a bit. mergeFactor presently set to 10. maxDoc count = 144156 numDocs count = 144145 What version of Solr are you using? Another potential OOM (multiple threads generating the same Fie

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Thanks for the pointers, Mike. I'm trying to determine the math to resolve some strange numbers we're seeing. Here's the top dozen lines from a jmap analysis on a heap dump: SizeCount Class description - 428246064 1792204 i

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Chris Hostetter
: values = 50. More to the point, is there an algorithm that I can use to : estimate the cache consumption rate for facet queries? I'm confused ... i thought you said in your orriginal mail that you had all the caching disabled? (except for FieldCache which is so low level in Lucene it's always

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
On 4/2/07, Yonik Seeley <[EMAIL PROTECTED]> wrote: On 4/1/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: > Our scenario: 150MB index, 14 documents, read/write servers in place > using standard replication. Running Tomcat 5.5.17 on Redhat Enterprise > Linux 4. Java configured to start with -

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Yonik Seeley
On 4/1/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: Our scenario: 150MB index, 14 documents, read/write servers in place using standard replication. Running Tomcat 5.5.17 on Redhat Enterprise Linux 4. Java configured to start with -Xmx1024m. We encounter java heap out-of-memory issues on

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Mike Klaas
On 4/2/07, Jeff Rodenburg <[EMAIL PROTECTED]> wrote: Hoping I can get a better response with a more directed question: I haven't answered your original question as it seems that general java memory debugging techniques would be the most useful thing here. With facet queries and the fields use

Re: Troubleshooting java heap out-of-memory

2007-04-02 Thread Jeff Rodenburg
Hoping I can get a better response with a more directed question: With facet queries and the fields used, what qualifies as a "large" number of values? The wiki uses U.S. states as an example, so the number of unique values = 50. More to the point, is there an algorithm that I can use to estima

Re: Snippets of indexed text

2007-04-02 Thread Thierry Collogne
Glad I could help you. On 02/04/07, Pierre-Yves LANDRON <[EMAIL PROTECTED]> wrote: >And this is the part for the highlighted text : > > > > > > > > > Yes it works just fine ! and it's great. :) Thanks Thierry : you were right, i didn't look for the right tag in the response. ( My prob

Find docs close to a date

2007-04-02 Thread nick19701
Let me use this date for example: 2007-03-25T17:22:00Z My docs have a date field and I need to find the two docs with a date which is closest to 2007-03-25T17:22:00Z. I use the following two queries to accomplish the task. date:{* TO 2007-03-25T17:22:00Z};date desc&start=0&rows=1 date:{2007-03-

Re: Snippets of indexed text

2007-04-02 Thread Pierre-Yves LANDRON
And this is the part for the highlighted text : Yes it works just fine ! and it's great. :) Thanks Thierry : you were right, i didn't look for the right tag in the response. ( My problem with facets parameters is still unresolved but i will work on that later) The more i'm usi