For the sessionexpiredexception, the solr is throwing this exception and then the shard goes down.
>From the following discussion, it seems to be that the solr is loosing connection to zookeeper and throws the exception. In the zoo keeper configuration file, zoo.cfg, is it safe to increase the synclimit shown in below snippet. # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/sanfs/mnt/vol01/solr/zookeeperdata/2 # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 Thanks, Satya On Mon, May 8, 2017 at 12:04 PM Satya Marivada <satya.chaita...@gmail.com> wrote: > The 3g memory is doing well, performing a gc at 600-700 MB. > > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC > > Here are my jvm start up > > The start up parameters are: > > java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4 > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 > -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark > -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly > -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 > -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled > -XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps > -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime > -Xloggc:/sanfs/mnt/vol01/solr/solr-6.3.0/server/logs/solr_gc.log > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M > -DzkClientTimeout=15000 ....... > > On Mon, May 8, 2017 at 11:50 AM Walter Underwood <wun...@wunderwood.org> > wrote: > >> Which garbage collector are you using? The default GC will probably give >> long pauses. >> >> You need to use CMS or G1. >> >> wunder >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> >> >> > On May 8, 2017, at 8:48 AM, Erick Erickson <erickerick...@gmail.com> >> wrote: >> > >> > 3G of memory should not lead to long GC pauses unless you're running >> > very close to the edge of available memory. Paradoxically, running >> > with 6G of memory may lead to _fewer_ noticeable pauses since the >> > background threads can do the work, well, in the background. >> > >> > Best, >> > Erick >> > >> > On Mon, May 8, 2017 at 7:29 AM, Satya Marivada >> > <satya.chaita...@gmail.com> wrote: >> >> Hi Piyush and Shawn, >> >> >> >> May I ask what is the solution for it, if it is the long gc pauses? I >> am >> >> skeptical about the same problem in our case too. We have started with >> 3G >> >> of memory for the heap. >> >> Did you have to adjust some of the memory allotted? Very much >> appreciated. >> >> >> >> Thanks, >> >> Satya >> >> >> >> On Sat, May 6, 2017 at 12:36 PM Piyush Kunal <piyush.ku...@myntra.com> >> >> wrote: >> >> >> >>> We already faced this issue and found out the issue to be long GC >> pauses >> >>> itself on either client side or server side. >> >>> Regards, >> >>> Piyush >> >>> >> >>> On Sat, May 6, 2017 at 6:10 PM, Shawn Heisey <apa...@elyograg.org> >> wrote: >> >>> >> >>>> On 5/3/2017 7:32 AM, Satya Marivada wrote: >> >>>>> I see below exceptions in my logs sometimes. What could be causing >> it? >> >>>>> >> >>>>> org.apache.zookeeper.KeeperException$SessionExpiredException: >> >>>> >> >>>> Based on my limited research, this would tend to indicate that the >> >>>> heartbeats ZK uses to detect when sessions have gone inactive are not >> >>>> occurring in a timely fashion. >> >>>> >> >>>> Common causes seem to be: >> >>>> >> >>>> JVM Garbage collections. These can cause the entire JVM to pause >> for an >> >>>> extended period of time, and this time may exceed the configured >> >>> timeouts. >> >>>> >> >>>> Excess client connections to ZK. ZK limits the number of connections >> >>>> from each client address, with the idea of preventing denial of >> service >> >>>> attacks. If a client is misbehaving, it may make more connections >> than >> >>>> it should. You can try increasing the limit in the ZK config, but if >> >>>> this is the reason for the exception, then something's probably >> wrong, >> >>>> and you may be just hiding the real problem. >> >>>> >> >>>> Although we might have bugs causing the second situation, the first >> >>>> situation seems more likely. >> >>>> >> >>>> Thanks, >> >>>> Shawn >> >>>> >> >>>> >> >>> >> >>
