Re: Restarting nodes and reported load

2017-06-02 Thread Daniel Steuernol
to Al Tobey's tuning guide frequently if nothing else for the tools he mentions and notes on the java gc. I want to say heap size of 15G sounds a little high but I am starting to talk a bit out of my depth when it comes to java tuning. see datastax's official cassandra 2.1 jvm tuning

Re: Restarting nodes and reported load

2017-06-01 Thread Daniel Steuernol
observe the (increasing) latencies? Something must be writing to keyspaces, something I presume is reading. What is the workload?- when using SSD, there are some /devices optimizations for SSD's. I wonder if those were done (they will cause some IO latency, but not like this)Daemeon C.M. Re

Re: Restarting nodes and reported load

2017-06-01 Thread Daniel Steuernol
er it is replacing a down node or inserting a new node, having a large amount of data on each node will mean that it takes longer for a node to join the cluster if it is streaming the data.Kind regards,AnthonyOn 30 May 2017 at 02:43, Daniel Steuernol <dan...@sendwithus.com> wrote:The cluster i

Re: Restarting nodes and reported load

2017-05-30 Thread Daniel Steuernol
the day are dangerous men, for they may act their dreams with open eyes, to make it possible.” — T.E. Lawrence On Tue, May 30, 2017 at 2:00 PM, tommaso barbugli <tbarbu...@gmail.com> wrote:Am I the only one thinking 3TB is way too much data for a single node on a VM?On Tue, May 30, 2017 at 10:3

Re: Restarting nodes and reported load

2017-05-30 Thread Daniel Steuernol
the dreamers of the day are dangerous men, for they may act their dreams with open eyes, to make it possible.” — T.E. Lawrence On Tue, May 30, 2017 at 1:36 PM, Daniel Steuernol <dan...@sendwithus.com> wrote:I don't believe incremental repair is enabled, I have never enabled it on

Re: Restarting nodes and reported load

2017-05-30 Thread Daniel Steuernol
enabled and snapshots are occupying the space.run nodetool clearsnapshot command.On Tue, May 30, 2017 at 11:12 AM, Daniel Steuernol <dan...@sendwithus.com> wrote:It's 3-4TB per node, and by load rises, I'm talking about load as reported by nodetool status.

Re: Restarting nodes and reported load

2017-05-30 Thread Daniel Steuernol
, May 30, 2017 at 11:12 AM, Daniel Steuernol <dan...@sendwithus.com> wrote:It's 3-4TB per node, and by load rises, I'm talking about load as reported by nodetool status. On May 30 2017, at 10:25 am, daemeon reiydelle <d

Re: Restarting nodes and reported load

2017-05-30 Thread Daniel Steuernol
r in total?On Tue, May 30, 2017 at 6:48 PM, Daniel Steuernol <dan...@sendwithus.com> wrote:I should also mention that I am running cassandra 3.10 on the cluster On May 29 2017, at 9:43 am, Daniel Steuernol <dan...@sendwithus.com> wrote:

Re: Restarting nodes and reported load

2017-05-30 Thread Daniel Steuernol
I should also mention that I am running cassandra 3.10 on the cluster On May 29 2017, at 9:43 am, Daniel Steuernol wrote: The cluster is running with RF=3, right now each node is storing about 3-4 TB of data. I'm

Re: Restarting nodes and reported load

2017-05-29 Thread Daniel Steuernol
uch data do you store per node and what kind of servers do you use (core count, RAM, disk, ...)?Cheers,TommasoOn Mon, May 29, 2017 at 6:22 PM, Daniel Steuernol <dan...@sendwithus.com> wrote:I am running a 6 node cluster, and I have noticed that the reported load on each node rises throughout the we

Restarting nodes and reported load

2017-05-29 Thread Daniel Steuernol
I am running a 6 node cluster, and I have noticed that the reported load on each node rises throughout the week and grows way past the actual disk space used and available on each node. Also eventually latency for operations suffers and the nodes have to be restarted. A couple questions on this, is

Re: Nodes stopping

2017-05-11 Thread Daniel Steuernol
m2017-05-11 22:29 GMT+01:00 Daniel Steuernol <dan...@sendwithus.com>:Thank you, it's an Out of memory crash according to dmesg. I have the heap size set to 15G in the jvm.options for cassandra, and there is 30G on the machine. On May 11 2017,

Re: Nodes stopping

2017-05-11 Thread Daniel Steuernol
wrote: Have a look at dmesg. It have already happened to me regarding type i instances at AWS. On 11-05-2017 22:17, Daniel Steuernol wrote: I had 2 nodes go down today, here is the ERRORs from the system log on both nodes https://gist.githu

Re: Nodes stopping

2017-05-11 Thread Daniel Steuernol
10 pm, Cogumelos Maravilha wrote: Can you grep ERROR system.log On 11-05-2017 21:52, Daniel Steuernol wrote: There is nothing in the system log about it being drained or shutdown, I'm not sure how els

Re: Nodes stopping

2017-05-11 Thread Daniel Steuernol
down, OR pre-empting cassandra process?On Thu, May 11, 2017 at 1:30 PM, Daniel Steuernol <dan...@sendwithus.com> wrote:I have a 6 node cassandra cluster running, and frequently a node will go down with no obvious error in the logs. This is starting to happen quite often, almost daily now. Any s

Nodes stopping

2017-05-11 Thread Daniel Steuernol
I have a 6 node cassandra cluster running, and frequently a node will go down with no obvious error in the logs. This is starting to happen quite often, almost daily now. Any suggestions on how to track down what is causing the node to stop?