Re: Dropped messages on random nodes.

2017-01-24 Thread Dikang Gu
> > > We have a 100 nodes ish cluster, I find that there are dropped messages > on > > random nodes in the cluster, which caused error spikes and P99 latency > > spikes as well. > > > > I tried to figure out the cause. I do not see any obvious bottleneck in >

Re: Dropped messages on random nodes.

2017-01-23 Thread Brandon Williams
on-gc pause is also troubling. (I assume non-gc since there wasn't anything logged by the GC inspector.) On Mon, Jan 23, 2017 at 12:36 AM, Dikang Gu wrote: > Hello there, > > We have a 100 nodes ish cluster, I find that there are dropped messages on > random nodes in the cluster, whi

Re: Dropped messages on random nodes.

2017-01-23 Thread Roopa Tangirala
> > Blake > > > On January 22, 2017 at 10:37:22 PM, Dikang Gu (dikan...@gmail.com) wrote: > > Btw, the C* version is 2.2.5, with several backported patches. > > On Sun, Jan 22, 2017 at 10:36 PM, Dikang Gu wrote: > > > Hello there, > > > > We have a

Re: Dropped messages on random nodes.

2017-01-23 Thread Blake Eggleston
. On Sun, Jan 22, 2017 at 10:36 PM, Dikang Gu wrote: > Hello there, > > We have a 100 nodes ish cluster, I find that there are dropped messages on > random nodes in the cluster, which caused error spikes and P99 latency > spikes as well. > > I tried to figure out the ca

Re: Dropped messages on random nodes.

2017-01-22 Thread Dikang Gu
Btw, the C* version is 2.2.5, with several backported patches. On Sun, Jan 22, 2017 at 10:36 PM, Dikang Gu wrote: > Hello there, > > We have a 100 nodes ish cluster, I find that there are dropped messages on > random nodes in the cluster, which caused error spikes and P99 latency

Dropped messages on random nodes.

2017-01-22 Thread Dikang Gu
Hello there, We have a 100 nodes ish cluster, I find that there are dropped messages on random nodes in the cluster, which caused error spikes and P99 latency spikes as well. I tried to figure out the cause. I do not see any obvious bottleneck in the cluster, the C* nodes still have plenty of