http://lucene.apache.org/java/3_0_3/api/contrib-misc/org/apache/lucene/index/BalancedSegmentMergePolicy.html
Look in solrconfig.xml for where MergePolicy is configured.
On Sun, May 1, 2011 at 6:31 PM, Lance Norskog wrote:
> Yes, indexing generally slows down querying. Most sites do indexing in
>
Yes, indexing generally slows down querying. Most sites do indexing in
one Solr and queries from another. The indexing system does 'merging',
which involves copying data around in files.
With 10g allocated to a 1g load, the JVM is doing a lot more garbage
collection that it would with a 1.5g alloc
Couple of things. One you are not swaping which is a good thing. Second (and I
am not sure what delay you selected for dstat, I would assume the default of 1
second) there is some pretty heavy write activity like this:
26 1 71 2 0 0 |4096B 1424k| 0 0 | 719 415 | 197M 11G|1.00
Thanks for the tool recommendation! This is the dstat output during
commit bombardment / concurrent search requests:
total-cpu-usage -dsk/total- ---paging-- ---system-- swap---
--io/total- ---file-locks--
usr sys idl wai hiq siq| read writ| in out | int csw | used free|
read w
If you are on linux, I would recommend two tools you can use to track what is
going on on the machine, atop ( http://freshmeat.net/projects/atop/ ) and dstat
( http://freshmeat.net/projects/dstat/ ).
atop in particular has been very useful to me in tracking down performance
issues in real time
Committing too frequently is very costly, since this calls fsync on
numerous files under-the-hood, which strains the IO system and can cut
into queries. If you really want to commit frequently, turning on compound
file format could help things, since that's 1 file to fsync instead of N, per
segment
Thanks Craig,
I'm beginning to think that frequent commits are causing the performance
degradation, even though there shouldn't be any *concurrent* commits in
our system. There's just one indexing thread per Solr core,
waitFlush/waitSearcher are set to true.
For now I'd pretty much rule out RAM s
Daniel,
I've been able to post documents to Solr without degrading the performance
of search. But, I did have to make some changes to the solrconfig.xml
(ramBufferSize, mergeFactor, autoCommit, etc).
What I found to be helpful was having a look at what was the causing the OS
to grind. If you
