On 2-Jun-08, at 2:09 PM, Norskog, Lance wrote:
Solr 1.2 ignores the 'number of documents' attribute. It honors the
"every 30 minutes" attribute.
Only if you specify both, I think. There was a bug in the
implementation.
-Mike
Lance
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik
Seeley
Sent: Sunday, June 01, 2008 6:47 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr indexing configuration help
On Sun, Jun 1, 2008 at 4:43 AM, Gaku Mak <[EMAIL PROTECTED]> wrote:
I have tried Yonik's suggestions with the following:
1) all autowarming are off
2) commented out firstsearch and newsearcher event handlers
3) increased autocommit interval to 600 docs and 30 minutes
(previously 50 docs and 5 minutes)
Glad it looks like your memory issues are solved, but I really
wouldn't
use "docs" at all for an autocommit criteria.... it will just slow
down
your full index builds.
-Yonik
In addition, I updated the java option with the following:
-d64 -server -Xms2048M -Xmx3072M -XX:-HeapDumpOnOutOfMemoryError
-XX:+UseSerialGC
Results:
I'm currently at 100,000 documents now with about 9.0GB index on a
quad machine with 4GB ram. The stress test is to add 20 documents
every 30 seconds now.
It seems like the serial GC works better than the other two
alternatives (-XX:+UseParallelGC or -XX:+UseConcMarkSweepGC) for some
reason. I have not seen any OOM since the changes mentioned above
(yet). If others have better experience with other GC and know how
to
configure it properly, please let me know because using serial GC
just
doesn't sound right on a quad machine.
Additional questions:
Does anyone know how solr/lucene use heap in terms of their
generations (young vs tenured) on the indexing environment? If we
have this answer, we would be able to better configure the
young/tenured ratio in the heap. Any help is appreciated! Thanks!
Now, I'm looking into configuring the slave machines. Well, that's a
separate question.
Yonik Seeley wrote:
Some things to try:
- turn off autowarming on the master
- turn off autocommit, unless you really need it, or change it to be
less agressive: autocommitting every 50 docs is bad if you are
rapidly adding documents.
- set maxWarmingSearchers to 1 to prevent the buildup of searchers
-Yonik
On Fri, May 30, 2008 at 3:39 PM, Gaku Mak <[EMAIL PROTECTED]> wrote:
I started running the test on 2 other machines with similar specs
but more RAM (4G). One of them now has about 60k docs and still
running fine. On the other machine, solr died at about 43k docs. A
short while before solr died, I saw that there were 5 searchers at
the same time. Do any of you know why would solr create 5
searchers,
and if that could cause solr to die? Is there any way to prevent
this? Also is there a way to totally disable the searcher and
whether that is a way to optimize the solr master?
I copied the following from the SOLR Statistics page in case it has
interested info:
name: [EMAIL PROTECTED] main
class: org.apache.solr.search.SolrIndexSearcher
version: 1.0
description: index searcher
stats: caching : true
numDocs : 42754
maxDoc : 42754
readerImpl : MultiSegmentReader
readerDir :
org.apache.lucene.store.FSDirectory@/var/lib/solr/peoplesolr_0002/
so
lr/data/index
indexVersion : 1211702500453
openedAt : Fri May 30 10:04:15 PDT 2008 registeredAt : Fri May 30
10:05:05 PDT 2008
name: [EMAIL PROTECTED] main
class: org.apache.solr.search.SolrIndexSearcher
version: 1.0
description: index searcher
stats: caching : true
numDocs : 42754
maxDoc : 42754
readerImpl : MultiSegmentReader
readerDir :
org.apache.lucene.store.FSDirectory@/var/lib/solr/peoplesolr_0002/
so
lr/data/index
indexVersion : 1211702500453
openedAt : Fri May 30 10:03:24 PDT 2008 registeredAt : Fri May 30
10:03:41 PDT 2008
name: [EMAIL PROTECTED] main
class: org.apache.solr.search.SolrIndexSearcher
version: 1.0
description: index searcher
stats: caching : true
numDocs : 42675
maxDoc : 42675
readerImpl : MultiSegmentReader
readerDir :
org.apache.lucene.store.FSDirectory@/var/lib/solr/peoplesolr_0002/
so
lr/data/index
indexVersion : 1211702500450
openedAt : Fri May 30 10:00:53 PDT 2008 registeredAt : Fri May 30
10:01:05 PDT 2008
name: [EMAIL PROTECTED] main
class: org.apache.solr.search.SolrIndexSearcher
version: 1.0
description: index searcher
stats: caching : true
numDocs : 42697
maxDoc : 42697
readerImpl : MultiSegmentReader
readerDir :
org.apache.lucene.store.FSDirectory@/var/lib/solr/peoplesolr_0002/
so
lr/data/index
indexVersion : 1211702500451
openedAt : Fri May 30 10:02:20 PDT 2008 registeredAt : Fri May 30
10:02:22 PDT 2008
name: [EMAIL PROTECTED] main
class: org.apache.solr.search.SolrIndexSearcher
version: 1.0
description: index searcher
stats: caching : true
numDocs : 42724
maxDoc : 42724
readerImpl : MultiSegmentReader
readerDir :
org.apache.lucene.store.FSDirectory@/var/lib/solr/peoplesolr_0002/
so
lr/data/index
indexVersion : 1211702500452
openedAt : Fri May 30 10:02:55 PDT 2008 registeredAt : Fri May 30
10:02:57 PDT 2008
Thank you all so much for your help. I really appreciate it.
-Gaku
Yonik Seeley wrote:
It's most likely a
1) hardware issue: bad memory
OR
2) incompatible libraries (most likely libc version for the JVM).
If you have another box around, try that.
-Yonik
--
View this message in context:
http://www.nabble.com/Solr-indexing-configuration-help-
tp17524364p17
566612.html Sent from the Solr - User mailing list archive at
Nabble.com.
--
View this message in context:
http://www.nabble.com/Solr-indexing-configuration-help-
tp17524364p1758
3518.html Sent from the Solr - User mailing list archive at
Nabble.com.
