On 9/2/2015 2:24 PM, Renee Sun wrote: > I have a sharded index. When I re-index a document (vs new index, which is > different process), I need to delete the old one first to avoid dup. We all > know that if there is only one core, the newly added document will replace > the old one, but with multiple core indexes, we will have to issue delete > command first to ALL shards since we do NOT know/remember which core the old > document was indexed to ... > > I also wanted to know if there is a better way handling this efficiently. > > Anyways, we are sending delete to all cores of this customer, one of them > hit , others did not. > > But consequently, when I need to decide about commit, I do NOT want blindly > commit to all cores, I want to know which one actually had the old doc so I > only send commit to that core. > > I could alternatively use query first and skip if it did not hit, but delete > if it does, and I can't short circuit since we have dups :-( based on a > historical reason. > > any suggestion how to make this more efficiently?
I have a sharded index too. It is a more complicated sharding mechanism than you would get in a default SolrCloud install (and my servers are NOT running in cloud mode). It's a hot/cold shard system, with one hot shard and six cold shards. Even though the shard that contains any given document is *always* something that can be calculated according to a configuration that changes at most once a day, I send all deletes to every shard like you do. Each batch of documents in the delete list (currently set to a batch size of 500) is sent to each shard. The deleteByQuery method on my Core class (this is a java program) queries the Solr core to see if any documents are found. If they are, then the delete request is sent to Solr. Any successful Solr update operation (add, delete, etc) will set a "commit" flag in the class instance, which is checked by the commit method. When a commit is requested on the Core class, if the flag is true, a commit is sent to Solr. If the commit succeeds, the flag is cleared. Thanks, Shawn
