This will certainly force-replicate the entire index but I question whether it was necessary.
These are basically snapshots that are different versions and do not necessarily have any correlation to the _current_ index. If you're not having any search problems, i.e. if all the replicas are returning the same number of documents, you shouldn't need to be this drastic. A quick check is to issue a q=*:*&distrib=false against each member of a shard. If the numFound is identical, then your shards are in sync. You really can't tell much of anything from even the segment counts or file names in the index directory on replicas in a shard. Since Solr forwards the _raw_ document to each replica, and since most commits are time-based, they'll often vary in terms of segments. Then the merge policy kicks in and since the segments on one replica may be slightly different from another, different segments may be chosen to merge.... Other than seeing all the directories laying around, what was the problem? Was there an operational issue like differing numbers of docs found for identical queries or anything else strange besides just more directories than you expected? FWIW, Erick On Mon, Aug 17, 2015 at 11:54 AM, Jeff Courtade <courtadej...@gmail.com> wrote: > Hi, > > So it turns out that the index directory has nothign to do with what index > is actually in use > > I found that we had mismatched version numbers on our shards so this is > what we had to do to fix that. > > > In production today we discovered that our shard replicas were on different > version numbers. > this means that the shards had some differences in data between leader and > replica. > > we have two shards > > shard1 ps01 ps03 > shard2 ps02 ps04 > > checking this url showed different version numbers on a given shard. Both > leader and replica is a shard should have the same version number. > > > http://ps0X:8983/solr/#/~cores/collection1 > > shard1 ps01 7752045 ps03 7752095 > > shard2 ps02 7792045 ps04 7790323 > > So to fix this we did the following. > > Stop ingestion/aspire no updates should be being made while you are doing > this. > > > For each shard stop the server with the lowest version number > In this case it is shard1 ps01 shard2 ps04 > > so stop solr on ps01 > ps03 will become leader if it is not already in the cloud console > > http://ps0X:8983/solr/#/~cloud > > then move or remove everything in this directory. It should be empty. > > > /opt/solr/solr-4.7.2/example/solr/collection1/data/ > > restart solr on ps01 > > watch that data directory it should get a few files and an > index.201508XXXXX directory where the index is downloaded form the leader > > du -sh should show that growing. > > in the cloud console ps01 will show as recovering while this is going on > until it is complete. Once it is done it will go green in teh cloud console. > > once it is green check the version number on ps01 and ps03 they should be > the same now. > > Repeat this for shard2 and you are done. > > > -- > Thanks, > > Jeff Courtade > M: 240.507.6116 > > On Mon, Aug 17, 2015 at 10:57 AM, Jeff Courtade <courtadej...@gmail.com> > wrote: > >> Hi, >> >> I have SOLR cloud running on SOLR 4.7.2 >> 2 shards one replica each. >> >> The size of the index directories is odd on ps03 >> >> >> ps01 shard1 leader >> >> 41G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20150815024352580 >> >> >> ps03 shard 2 replica >> >> 59G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20140906125148419 >> 18G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20150129181248396 >> 24G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20150511233039042 >> 24G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20150527121503268 >> 41G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20150806034052366 >> 4.0K >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20150814152030017 >> >> >> ps02 shard 2 leader >> >> 31G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20150527161148429 >> 39G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20150815151640598 >> >> >> ps04 shard 2 replica >> >> 61G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20140820212651780 >> 39G >> /opt/solr/solr-4.7.2/example/solr/collection1/data/index.20150815170546642 >> >> >> what can i do to remedy this? >> >> -- >> Thanks, >> >> Jeff Courtade >> M: 240.507.6116 >>
