This solrcloud has had some issues of late. We had a network glitch which caused a shard leader of one of the collections write over 5000 0 length tlogs to its filesystem. Whenever it started up it ran out of file handles which killed the IndexWriter and caused lots of unhappy collections. This may be related to that. No one was alerted to the errors for several days.
These guys have been out of sync for a while.Indeed one of the collections we just did a data load to today and it stayed bad. I can say that we have NEVER done a FORCELEADER, unless some internal solrcloud code does this. the dataload we did had no errors and both replicas are in active state. No replica was offline Oh just went back and looked and saw that the empty replica in the collection we just loaded has now caught up and has data. It took a while but it now matches its leader. Perhaps all we need to do is new data loads to the out of whack collections? On Fri, Oct 6, 2017 at 2:04 PM, Webster Homer <webster.ho...@sial.com> wrote: > We are using Solr 6.2.0 in solrcloud mode > > I have a QA solrcloud that has multiple collections. All collections have > 2 shards each with two replicas. > > I have several replicas where the numDocs in the same shard do not match. > In two collections with three different shards I have one replica with data > and the other has no data. All six replicas appear healthy in the Solr > console. > > So how does that happen where two replicas in the same shard have > different amounts of data? > > How do you diagnose this when the replicas are active and seemingly > healthy? > > How do I get the replicas with no data, get data from their leader? In all > three cases the replica with data is the leader. > > I also see two other collections where the replica's numDocs don't quite > match > In those two cases the leader has a few more docs than the other replica > > How to remedy this situation? > > This solrcloud is a target of CDCR replication, but I'm not sure why that > would matter since I believe cdcr has the shard leaders communicate and the > followers should just get their updates from their leader as they would > from a normal update > > I'm just lucky that this is not a production solrcloud! Still need to know > how to fix it. > > Thanks! > -- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith. Click http://www.emdgroup.com/disclaimer to access the German, French, Spanish and Portuguese versions of this disclaimer.
