Re: Experience with large merge factors

2010-10-07 Thread Jan Høydahl / Cominvent
Vote for or implement :) this issue: https://issues.apache.org/jira/browse/SOLR-1565 StreamingUpdateSolrServer should support RequestWriter API -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > As far as I know. It is the simplest, but I found that is uses a 'lot' of

Re: Experience with large merge factors

2010-10-07 Thread Michael McCandless
On Wed, Oct 6, 2010 at 9:57 PM, Burton-West, Tom wrote: > Hi Mike, > >>.Do you use multiple threads for indexing?  Large RAM buffer size is >>>also good, but I think perf peaks out mabye around 512 MB (at least >>>based on past tests)? > > We are using Solr, I'm not sure if Solr uses multiple thre

Re: Experience with large merge factors

2010-10-07 Thread Thijs
ecosystem search :: http://search-lucene.com/ - Original Message From: "Burton-West, Tom" To: "solr-user@lucene.apache.org" Sent: Wed, October 6, 2010 9:57:12 PM Subject: RE: Experience with large merge factors Hi Mike, .Do you use multiple threads for indexing? Large

Re: Experience with large merge factors

2010-10-06 Thread Otis Gospodnetic
ngUpdateSolrServer is the simplest thing to use for this. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message > From: "Burton-West, Tom" > To: "solr-user@lucene.apache.org"

RE: Experience with large merge factors

2010-10-06 Thread Burton-West, Tom
Hi Mike, >.Do you use multiple threads for indexing? Large RAM buffer size is >>also good, but I think perf peaks out mabye around 512 MB (at least >>based on past tests)? We are using Solr, I'm not sure if Solr uses multiple threads for indexing. We have 30 "producers" each sending documents

Re: Experience with large merge factors

2010-10-05 Thread Lance Norskog
You could do periodic small optimizes. The optimize command now includes 'maxSegments' which limits the target number of segments. It is possible to write a Lucene program that collects a bunch of segments and annoints it as an index. This gives you a way to collect segments after you write them w

Re: Experience with large merge factors

2010-10-05 Thread Michael McCandless
4 weeks is a depressingly long time to re-index! Do you use multiple threads for indexing? Large RAM buffer size is also good, but I think perf peaks out mabye around 512 MB (at least based on past tests)? Believe it or not, merging is typically compute bound. It's costly to decode & re-encode

Recall: Experience with large merge factors

2010-10-05 Thread Nguyen, Vincent (CDC/OD/OADS) (CTR)
Nguyen, Vincent (CDC/OD/OADS) (CTR) would like to recall the message, "Experience with large merge factors".

Recall: Experience with large merge factors

2010-10-05 Thread Nguyen, Vincent (CDC/OD/OADS) (CTR)
Nguyen, Vincent (CDC/OD/OADS) (CTR) would like to recall the message, "Experience with large merge factors".

RE: Experience with large merge factors

2010-10-05 Thread Nguyen, Vincent (CDC/OD/OADS) (CTR)
[mailto:tburt...@umich.edu] Sent: Tuesday, October 05, 2010 2:41 PM To: solr-user@lucene.apache.org Subject: Experience with large merge factors Hi all, At some point we will need to re-build an index that totals about 3 terabytes in size (split over 12 shards). At our current indexing speed we estimate

Experience with large merge factors

2010-10-05 Thread Burton-West, Tom
Hi all, At some point we will need to re-build an index that totals about 3 terabytes in size (split over 12 shards). At our current indexing speed we estimate that this will take about 4 weeks. We would like to reduce that time. It appears that our main bottleneck is disk I/O during index m