>>
>> Another thought for surprise demands on memory. I don’t know if this is
>> in 3.11.0, you’ll have to check the C* bash scripts for launching the
>> service. The number of malloc arenas haven’t always been curtailed, and
>> that could result in an explosion in m
uot;user@cassandra.apache.org"
> *Date: *Wednesday, February 26, 2020 at 9:55 PM
> *To: *"user@cassandra.apache.org"
> *Subject: *Re: Hints replays very slow in one DC
>
>
>
> *Message from External Sender*
>
> Nodes are going down due to Out of Memory
To: "user@cassandra.apache.org"
Subject: Re: Hints replays very slow in one DC
Message from External Sender
Nodes are going down due to Out of Memory and we are using 31GB heap size in
DC1 , however in DC2 (Which serves the traffic) has 16GB heap .
Why we had to increase heap in DC1 is beca
>
> Nodes are going down due to Out of Memory and we are using 31GB heap size
> in DC1 , however in DC2 (Which serves the traffic) has 16GB heap .
> Why we had to increase heap in DC1 is because , DC1 nodes were going down
> due Out of Memory issue but DC2 nodes never went down .
>
It doesn't soun
You may find throttle rate of hinted handoff on node and adjust if needed.
nodetool gethintedhandoffthrottlekb and you may also set by nodetool
sethintedhandoffthrottlekb
I would also check disk stats where hints are stored either by sar or iostat.
Sincerely,
Aakash Pandhi
On Wednesday,
Is dc1 a simple standby DC? Or you run some operations(e.g. compute for
analysis) on the same? Have you found the root cause of the oom? Do you
see any specific Cassandra operation (e.g repair) is causing oom?
One tip: try upgrading to 3.11.6 as lots of bugs has been fixed since 3.11.0
On Wed, Fe
Nodes are going down due to Out of Memory and we are using 31GB heap size
in DC1 , however in DC2 (Which serves the traffic) has 16GB heap .
Why we had to increase heap in DC1 is because , DC1 nodes were going down
due Out of Memory issue but DC2 nodes never went down .
We also noticed below kind
What's the reason for nodes going down? Is it because the cluster is
overloaded? Hints will get handed off periodically when nodes come back to
life but if they happen to go down again or become unresponsive (for
whatever reason), the handoff will be delayed until the next cycle. I think
it's every
DC2 is our main datacenter which serves all the traffic.
This cluster has Materialized views.
On Tue, Feb 25, 2020 at 9:32 PM Erick Ramirez
wrote:
> Krish, with the limited info and assuming things like hint throttle and
> delivery threads all being equal, my guess would be DC1 is your primary
Krish, with the limited info and assuming things like hint throttle and
delivery threads all being equal, my guess would be DC1 is your primary DC
and is busier than DC2. Got any diagnostic data/troubleshooting info you
could share? Otherwise, it's a little difficult to speculate as to what may
be
Hi,
We have 2 datacenters in our cassandra cluster.
Whenever a node goes down in DC1 and hints gets collected in all other
nodes then what we have noticed is that hints replay is very very slow in
DC1 node but if a node goes down in DC2 and comes back, hints replay fast.
We are on 3.11.0
We are u
11 matches
Mail list logo
