bq. Is this the correct approach ? It works, but it might not be ideal. Recent versions of ZooKeeper have an alternate config for this max limit though, and it is preferable to use that.
See maxSessionTimeout in http://zookeeper.apache.org/doc/r3.3.1/zookeeperAdmin.html - Mark On Mon Jan 26 2015 at 4:44:41 PM Vijay Sekhri <sekhrivi...@gmail.com> wrote: > Hi Erick, > > The older message seems to be deleted so I am sending a new one > http://osdir.com/ml/solr-user.lucene.apache.org/2015-01/msg00773.html > > > In solr.xml file I had zk timeout set to* <int > name="zkClientTimeout">${zkClientTimeout:450000}</int>* > One thing that made a it a bit better now is the zk tick time and syncLimit > settings. I set it to a higher value as below. This may not be advisable > though. > > tickTime=30000 > initLimit=30 > syncLimit=20 > > Now we observed that replicas do not go in recovery that often as before. > In the whole cluster at a given time I would have a couple of replicas in > recovery whereas earlier it were multiple replicas from every shard . > On the wiki https://wiki.apache.org/solr/SolrCloud it says the "The > maximum > is 20 times the tickTime." in the FAQ so I decided to increase the tick > time. Is this the correct approach ? > > One question I have is that if auto commit settings has anything to do with > this or not ? Does it induce extra work for the searchers because of which > this would happen? I have tried with following settings > * <autoSoftCommit>* > * <maxDocs>500000</maxDocs>* > * <maxTime>900000</maxTime>* > * </autoSoftCommit>* > > * <autoCommit>* > * <maxDocs>200000</maxDocs>* > * <maxTime>30000</maxTime>* > * <openSearcher>false</openSearcher>* > * </autoCommit>* > > I have increased the heap size to 15GB for each JVM instance . I > monitored during full indexing how the heap usage looks like and it never > goes beyond 8 GB . I don't see any Full GC happening at any point . > > > Our rate is a variable rate . It is not a sustained rate of 6000/second , > however there are intervals where it would reach that much and come down > and grow again and come down. So if I would take an average it would be > 600/second only but that is not real rate at any given time. > Version of solr cloud is *4.10*. All indexers are basically java programs > running on different host using CloudSolrServer api. > As I mentioned it is much better now than before , however not completely > as expected .If possible we would like to have none of them go in recovery > > I captured some logs before and after recovery > > 14:16:52,774 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [IFD][recoveryExecutor-7-thread-1]: delete > "_5r2_1.del" > 14:16:52,774 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [IFD][recoveryExecutor-7-thread-1]: 0 msec > to > checkpoint > 14:16:52,774 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [IFD][recoveryExecutor-7-thread-1]: now > checkpoint "_4qe(4.10.0):C4312879/1nts ; isCommit = false] > 14:16:52,774 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [IFD][recoveryExecutor-7-thread-1]: delete > "_5r4_1.del" > 14:16:52,774 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [IFD][recoveryExecutor-7-thread-1]: 0 msec > to > checkpoint > 14:16:52,775 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [DW][recoveryExecutor-7-thread-1]: > recoveryExecutor-7-thread-1 finishFullFlush success=true > 14:16:52,775 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > findMerges: 34 segments > 14:16:52,777 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_554(4.10.0):C3995865/780418:delGen=23 size=3669.307 MB [skip: too > large] > 14:16:52,777 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_4qe(4.10.0):C4312879/1370113:delGen=57 size=3506.254 MB [skip: too > large] > 14:16:52,777 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5co(4.10.0):C871785/93995:delGen=11 size=853.668 MB > 14:16:52,777 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5kb(4.10.0):C424868/49572:delGen=12 size=518.704 MB > 14:16:52,777 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5hm(4.10.0):C457977/83353:delGen=12 size=470.422 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_56u(4.10.0):C286775/11906:delGen=15 size=312.952 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5f5(4.10.0):C116528/43621:delGen=2 size=95.529 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5m7(4.10.0):C122852/54010:delGen=12 size=84.949 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5p7(4.10.0):C37457/649:delGen=2 size=54.241 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5of(4.10.0):C38545/5677:delGen=1 size=50.672 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qm(4.10.0):C29770/4:delGen=2 size=34.052 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5o8(4.10.0):C27155/7531:delGen=1 size=31.008 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5r2(4.10.0):C32769/4:delGen=2 size=27.410 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5r3(4.10.0):C26057 size=23.893 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5r4(4.10.0):C23934/3:delGen=2 size=22.004 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5nx(4.10.0):C33236/20669:delGen=2 size=19.861 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5r5(4.10.0):C24222/1:delGen=1 size=19.546 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5q0(4.10.0):C13610/2625:delGen=7 size=12.480 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qv(4.10.0):C3973/1:delGen=1 size=3.402 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qo(4.10.0):C3456/2:delGen=2 size=3.147 MB > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qq(4.10.0):C1997 size=1.781 MB [floored] > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5q2(4.10.0):C779 size=1.554 MB [floored] > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qh(4.10.0):C1765/1:delGen=1 size=1.549 MB [floored] > 14:16:52,778 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qc(4.10.0):C1828/1:delGen=1 size=1.401 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qr(4.10.0):C1468 size=1.390 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qp(4.10.0):C1729/1:delGen=1 size=1.351 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qa(4.10.0):C967 size=1.235 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qi(4.10.0):C1241 size=1.146 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qy(4.10.0):C1407 size=1.050 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5q1(4.10.0):C402/1:delGen=1 size=0.954 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qw(4.10.0):C802/1:delGen=1 size=0.821 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qx(4.10.0):C638 size=0.818 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qt(4.10.0):C30 size=0.072 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > seg=_5qu(4.10.0):C27 size=0.063 MB [floored] > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > allowedSegmentCount=31 vs count=34 (eligible count=32) tooBigCount=2 > 14:16:52,779 INFO [org.apache.solr.update.LoggingInfoStream] > (recoveryExecutor-7-thread-1) [TMP][recoveryExecutor-7-thread-1]: > maybe=_5co(4.10.0):C871785/93995:delGen=11 > _5kb(4.10.0):C424868/49572:delGen=12 _5hm(4.10.0):C457977/83353:delGen=12 > _56u(4.10.0):C286775/11906:delGen=15 _5f5(4.10.0):C116528/43621:delGen=2 > _5m7(4.10.0):C122852/54010:delGen=12 _5p7(4.10.0):C37457/649:delGen=2 > _5of(4.10.0):C38545/5677:delGen=1 _5qm(4.10.0):C29770/4:delGen=2 > _5o8(4.10.0):C27155/7531:delGen=1 score=0.7313777596572674 skew=0.341 > nonDelRatio=0.852 tooLarge=false size=2506.203 MB > ..... > > 14:16:58,367 INFO [org.apache.solr.update.LoggingInfoStream] > (searcherExecutor-6-thread-1) [IFD][searcherExecutor-6-thread-1]: delete > "_5p7_1.del" > 14:16:58,367 INFO [org.apache.solr.update.LoggingInfoStream] > (searcherExecutor-6-thread-1) [IFD][searcherExecutor-6-thread-1]: delete > "_5nx_1.del" > 14:16:58,367 INFO [org.apache.solr.update.LoggingInfoStream] > (searcherExecutor-6-thread-1) [IFD][searcherExecutor-6-thread-1]: delete > "_5co_a.del" > 14:16:58,367 INFO [org.apache.solr.update.LoggingInfoStream] > (searcherExecutor-6-thread-1) [IFD][searcherExecutor-6-thread-1]: delete > "_5q0_6.del" > 14:16:58,367 INFO [org.apache.solr.update.LoggingInfoStream] > (searcherExecutor-6-thread-1) [ > > -- > ********************************************* > Vijay Sekhri > ********************************************* >
