Re: Gossip intermittently marks node as DOWN

2014-03-19 Thread Phil Luckhurst
I think we've found the issue! It seems that the times on those Cassandra servers was being kept in sync by vmware tools using the time of the vmware host machine. We have now turned that off and are using the ntp service to keep the times in sync like we do for our physical servers and we have no

Re: Gossip intermittently marks node as DOWN

2014-03-04 Thread Johnny Miller
That looks healthy - nothing blocked or dropped. On 4 Mar 2014, at 16:12, Phil Luckhurst wrote: > Here's the tpstats output from both nodes. > > > > > > > Johnny Miller wrote >> What is nodetool tpstats telling you? > > > > > > -- > View this message in context: > http://cassandra-

Re: Gossip intermittently marks node as DOWN

2014-03-04 Thread Phil Luckhurst
Here's the tpstats output from both nodes. Johnny Miller wrote > What is nodetool tpstats telling you? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Gossip-intermittently-marks-node-as-DOWN-tp7593189p7593206.html Sent from the cassandr

Re: Gossip intermittently marks node as DOWN

2014-03-04 Thread Johnny Miller
What is nodetool tpstats telling you? On 4 Mar 2014, at 15:10, Phil Luckhurst wrote: > It was created with the default settings so we have 256 per node. > > > Fabrice Facorat wrote >> From what I understand, this can happen when having many nodes and >> vnodes by node. How many vnodes did you

Re: Gossip intermittently marks node as DOWN

2014-03-04 Thread Phil Luckhurst
It was created with the default settings so we have 256 per node. Fabrice Facorat wrote > From what I understand, this can happen when having many nodes and > vnodes by node. How many vnodes did you configure on your nodes ? > > 2014-03-04 11:37 GMT+01:00 Phil Luckhurst < > phil.luckhurst@ > >

Re: Gossip intermittently marks node as DOWN

2014-03-04 Thread Fabrice Facorat
>From what I understand, this can happen when having many nodes and vnodes by node. How many vnodes did you configure on your nodes ? 2014-03-04 11:37 GMT+01:00 Phil Luckhurst : > The VMs are hosted on the same ESXi server and they are just running > Cassandra. We seem to get this happen even if t

RE: Gossip intermittently marks node as DOWN

2014-03-04 Thread Phil Luckhurst
The VMs are hosted on the same ESXi server and they are just running Cassandra. We seem to get this happen even if the nodes appear to be idle; about 2 to 4 times per hour. Phil -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Gossip-interm

RE: Gossip intermittently marks node as DOWN

2014-03-04 Thread Romain HARDOUIN
Set phi_convict_threshold to 12 is a good idea if your network is busy. Are your VMs located in different datacenters? Did you check if the nodes are not overloaded? An unresponsive node can be seen as down even if it's temporary. Romain Phil Luckhurst a écrit sur 03/03/2014 15:16:25 : > De