Hi Ernie,
Thank you really much for trying to help.
Unfortunately we haven't been able to reproduce this issue without k8s.
I would like to share some additional info about this issue. We know
that this is not the "correct" way to shut down member gracefully, since
it's complete TCP communication towards the other members is forcefully
terminated even before it receives shutdown indication. This issue with
removals from previous mail can be easily avoided by enforcing policy
that the TCP connections towards other members aren't terminated until
member terminates them by itself during graceful shutdown.
Another thing is that the removals occur only when when we shut down
complete k8s node as described in previous mail. When we reproduce the
same situation without shutting down k8s node then the removals aren't
initiated:
"In parallel terminate member TCP communication and initiate gracefully
shut down the member. What happens then is that availability check over
UDP using Heartbeat messages towards other members pass, and removals
aren't initiated towards other members"
Given the above additional test result when using k8s with 1.12 release,
and due to fact that we were unable to reproduce the issue on geode
only, we are not sure that the problem is actually in geode. Please feel
free to contact me by slack or mail for any additional questions or
things you would like to discuss.
BRs,
Jakov
On 19. 10. 2020. 20:26, Ernie Burghardt wrote:
Hi Jakov,
I'm looking into your question(s)... curious if you've run into this in a
non-k8s cluster?
Might help focus the investigation...
Thanks,
EB
On 10/13/20, 7:51 AM, "Jakov Varenina" <jakov.varen...@est.tech> wrote:
Hi all,
sorry for bothering, but we have noticed some differences in behavior
between 1.11 and 1.12 releases and need your help in understanding them.
First I would like to mention that we are running geode in Kubernetes.
We perform shutdown of the worker node that is hosting one member(e.g.
coordinator locator). Shutdown procedure affect member in a following way:
1. TCP shared unordered connections towards other members are terminated
2. Member receives graceful shut-down indication and starts with the
shut-down procedure
Usually connections starting to be terminated first and the shut-down
indication comes short after (e.g. ~10 milliseconds in difference). The
step 1. triggers availability check towards the other members for which
TCP connection has been previously lost. At this point of time
coordinator is unaware of ongoing shut-down and assumes that all other
members are actually having issues due to connection loss. Even after
coordinator receives the graceful shut-down indication this process of
availability check is not stopped. What happens later on is that
availability check fail for all members and coordinator initiates their
removal with RemoveMemberMessage. This message is succesfully received
on the other members forcing them to shut-down.
In geode 1.11 everything is same except the fact that availability check
pass and therefore removals aren't initiated.
In logs it can be seen that for both releases TCP availability check
fail, but HeartbeatMessageRequest/HearbeatMessage check fails only on
1.12 and pass on 1.11. In 1.12 release it can be seen that heartbeat
request and heartbeat messages are sent but does not reach their
destination members. RemoveMemberMessage which are sent later on reach
their destination successfully. Does anybody know what was changed in
1.12 that could lead to such difference in behavior?
Additionally, availability check is not stopped when graceful shutdown
is initiated. Do you think that this could be improved, so that member
stops ongoing availability check when detects gracefull shutdown? Just
to add that shutdown procedure is also delayed due to unsuccessful
attempts to estabilsh TCP connections towards the other members.
BRs,
Jakov
