RE: large scale indexing issues / single threaded bottleneck

2011-11-01 Thread Roman Alekseenkov
We have a rate of 2K small docs/sec which translates into 90 GB/day of index space You should be fine Roman Awasthi, Shishir wrote: > > Roman, > How frequently do you update your index? I have a need to do real time > add/delete to SOLR documents at a rate of approximately 20/min. > The total n

Re: large scale indexing issues / single threaded bottleneck

2011-10-30 Thread Roman Alekseenkov
Guys, thank you for all the replies. I think I have figured out a partial solution for the problem on Friday night. Adding a whole bunch of debug statements to the info stream showed that every document is following "update document" path instead of "add document" path. Meaning that all document I

Re: large scale indexing issues / single threaded bottleneck

2011-10-28 Thread Roman Alekseenkov
I'm wondering if this is relevant: https://issues.apache.org/jira/browse/LUCENE-2680 - Improve how IndexWriter flushes deletes against existing segments Roman On Fri, Oct 28, 2011 at 11:38 AM, Roman Alekseenkov wrote: > Hi everyone, > > I'm looking for some help with Solr i

large scale indexing issues / single threaded bottleneck

2011-10-28 Thread Roman Alekseenkov
Hi everyone, I'm looking for some help with Solr indexing issues on a large scale. We are indexing few terabytes/month on a sizeable Solr cluster (8 masters / serving writes, 16 slaves / serving reads). After certain amount of tuning we got to the point where a single Solr instance can handle ind