Re: Avoid re indexing

2015-08-02 Thread Upayavira
You do not want to add a new shard, first you want your docs evenly spread, secondly, they are spread using hash ranges, to add more capacity, you spread out those hash ranges using shard splitting. "Adding" a new shard doesnt really make any sense here. Unless you go for implicit routing where you

Re: Avoid re indexing

2015-08-01 Thread Nagasharath
Yes, shard splitting will only help in managing large clusters and to improve query performance. In my case as index size is fully grown (no capacity to hold in the existing shards) across the collection adding a new shard will help and for which I have to re index. > On 01-Aug-2015, at 6:34 p

Re: Avoid re indexing

2015-08-01 Thread Upayavira
Erm, that doesn't seem to make sense. Seems like you are talking about *merging* shards. Say you had two shards, 3m docs each: shard1: 3m docs shard2: 3m docs If you split shard1, you would have: shard1_0: 1.5m docs shard1_1: 1.5m docs shard2: 3m docs You could, of course, then split shard2. Y

Re: Avoid re indexing

2015-08-01 Thread Nagasharath
If my current shard is holding 3 million documents will the new subshard after splitting also be able to hold 3 million documents? If that is the case After shard splitting the sub shards should hold 6 million documents if a shard is split in to two. Am I right? > On 01-Aug-2015, at 5:43 pm, Upa

Re: Avoid re indexing

2015-08-01 Thread Upayavira
On Sat, Aug 1, 2015, at 11:29 PM, naga sharathrayapati wrote: > I am using solrj to index documents > > i agree with you regarding the index update but i should not see any > deleted documents as it is a fresh index. Can we actually identify what > are > those deleted documents? If you post doc

Re: Avoid re indexing

2015-08-01 Thread naga sharathrayapati
I am using solrj to index documents i agree with you regarding the index update but i should not see any deleted documents as it is a fresh index. Can we actually identify what are those deleted documents? if there is no option of adding shards to existing collection i do not like the idea of re

Re: Avoid re indexing

2015-08-01 Thread Upayavira
On Sat, Aug 1, 2015, at 10:30 PM, naga sharathrayapati wrote: > I have an exception with one of the document after indexing 6 mil > documents > out of 10 mil, is there any way i can avoid re indexing the 6 mil > documents? How are you indexing your documents? Are you using the DIH? Personally, I