Garbage Collectors Summary: https://apps.sematext.com/spm-reports/s/rgRnwuShgI
Pool Size: https://apps.sematext.com/spm-reports/s/H16ndqichM First Stopping recovery warning: 4:00, OOM error: 6:30. 2014-03-24 16:35 GMT+02:00 Shalin Shekhar Mangar <shalinman...@gmail.com>: > I am guessing that it is all related to memory issues. I guess that as > the used heap increases, full GC cycles increase causing ZK timeouts > which in turn cause more recoveries to be initiated. In the end, > everything blows up with the out of memory errors. Do you log GC > activity on your servers? > > I suggest that you rollback to 4.6.1 for now and upgrade to 4.7.1 when > it releases next week. > > On Mon, Mar 24, 2014 at 7:51 PM, Lukas Mikuckis <lukasmikuc...@gmail.com> > wrote: > > Yes, we upgraded solr from 4.6.1 to 4.7 3 weeks ago (2 weeks before solr > > started crashing). > > When we were upgrading, we just upgraded solr and changed versions in > > collections configs. > > > > When solr crashes we get OOM but only 2h after first Stopping recovery > > warnings. > > > > Maybe you have any ideas when Stopping recovery warnings are thrown? > > Because now we have no idea what could cause this issue. > > > > Mon, 24 Mar 2014 04:03:17 GMT Shalin Shekhar Mangar < > shalinman...@gmail.com > >>: > >> > >> Did you upgrade recently to Solr 4.7? 4.7 has a bad bug which can > >> cause out of memory issues. Can you check your logs for out of memory > >> errors? > >> > >> On Sun, Mar 23, 2014 at 9:07 PM, Lukas Mikuckis < > lukasmikuc...@gmail.com> > > wrote: > >> > Solr version: 4.7 > >> > > >> > Architecture: > >> > 2 solrs (1 shard, leader + replica) > >> > 3 zookeepers > >> > > >> > Servers: > >> > * zookeeper + solr (heap 4gb) - RAM 8gb, 2 cpu cores > >> > * zookeeper + solr (heap 4gb) - RAM 8gb, 2 cpu cores > >> > * zookeeper > >> > > >> > Solr data: > >> > * 21 collections > >> > * Many fields, small docs, docs count per collection from 1k to 500k > >> > > >> > About a week ago solr started crashing. It crashes every day, 3-4 > times > > a > >> > day. Usually at nigh. I can't tell anything what could it be related > to > >> > because at that time we haven't done any configuration changes. Load > >> > haven't changed too. > >> > > >> > > >> > Everything starts with Stopping recovery for .. warnings (every > > warnings is > >> > repeated several times): > >> > > >> > WARN org.apache.solr.cloud.RecoveryStrategy; Stopping recovery for > >> > zkNodeName=core_node1core=****************** > >> > > >> > WARN org.apache.solr.cloud.ElectionContext; cancelElection did not > find > >> > election node to remove > >> > > >> > WARN org.apache.solr.update.PeerSync; no frame of reference to tell > if > >> > we've missed updates > >> > > >> > WARN - 2014-03-23 04:00:26.286; org.apache.solr.update.PeerSync; no > > frame > >> > of reference to tell if we've missed updates > >> > > >> > WARN - 2014-03-23 04:00:30.728; org.apache.solr.handler.SnapPuller; > > File > >> > _f9m_Lucene41_0.doc expected to be 6218278 while it is 7759879 > >> > > >> > WARN - 2014-03-23 04:00:54.126; > >> > org.apache.solr.update.UpdateLog$LogReplayer; Starting log replay > >> > > > > tlog{file=/path/solr/collection1_shard1_replica2/data/tlog/tlog.0000000000000003272 > >> > refcount=2} active=true starting pos=356216606 > >> > > >> > Then again Stopping recovery for .. warnings: > >> > > >> > WARN org.apache.solr.cloud.RecoveryStrategy; Stopping recovery for > >> > zkNodeName=core_node1core=****************** > >> > > >> > ERROR - 2014-03-23 05:19:29.566; org.apache.solr.common.SolrException; > >> > org.apache.solr.common.SolrException: No registered leader was found > > after > >> > waiting for 4000ms , collection: collection1 slice: shard1 > >> > > >> > ERROR - 2014-03-23 05:20:03.961; org.apache.solr.common.SolrException; > >> > org.apache.solr.common.SolrException: I was asked to wait on state > down > > for > >> > IP:PORT_solr but I still do not see the requested state. I see state: > >> > active live:false > >> > > >> > > >> > After this serves mostly didn't recover. > >> > >> > >> > >> -- > >> Regards, > >> Shalin Shekhar Mangar. > >> > >> > > > > -- > Regards, > Shalin Shekhar Mangar. >
