On Thu, Dec 31, 2009 at 10:26 PM, Chris Hostetter <hossman_luc...@fucit.org> wrote: > why do we bother detecthing/removing the duplicates? > > strictly speaking docs with duplicate IDs on multiple shards is a "garbage > in" situation, i can understanding Solr taking a little extra effort to > not fail hard if this situation is encountered, but why update the > numFound at all, or remove the duplicates from the list? ... why not leave > them in as is? (then numFound would never change)
Distrib search keys some things off of the unique id, so when we encountered duplicates in the past it failed hard. IIRC only keeping one doc with the same id was actually the easiest way to not fail hard. -Yonik http://www.lucidimagination.com
