Hi Jakov, Do you see this issue if you drain the worker node before shutting it down? For reference: https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/. I know this doesn’t directly answer your questions, but it seems like draining the node first would help the members receive the shutdown signal before the TCP connections are terminated. For what it’s worth, we also run Geode on Kubernetes but haven’t encountered this issue. Knowing whether it happens when you drain the node first would help us to be able to reproduce it.
Thanks, Aaron On Oct 20, 2020, at 7:49 AM, Jakov Varenina <jakov.varen...@est.tech<mailto:jakov.varen...@est.tech>> wrote: Hi Ernie, Thank you really much for trying to help. Unfortunately we haven't been able to reproduce this issue without k8s. I would like to share some additional info about this issue. We know that this is not the "correct" way to shut down member gracefully, since it's complete TCP communication towards the other members is forcefully terminated even before it receives shutdown indication. This issue with removals from previous mail can be easily avoided by enforcing policy that the TCP connections towards other members aren't terminated until member terminates them by itself during graceful shutdown. Another thing is that the removals occur only when when we shut down complete k8s node as described in previous mail. When we reproduce the same situation without shutting down k8s node then the removals aren't initiated: "In parallel terminate member TCP communication and initiate gracefully shut down the member. What happens then is that availability check over UDP using Heartbeat messages towards other members pass, and removals aren't initiated towards other members" Given the above additional test result when using k8s with 1.12 release, and due to fact that we were unable to reproduce the issue on geode only, we are not sure that the problem is actually in geode. Please feel free to contact me by slack or mail for any additional questions or things you would like to discuss. BRs, Jakov On 19. 10. 2020. 20:26, Ernie Burghardt wrote: Hi Jakov, I'm looking into your question(s)... curious if you've run into this in a non-k8s cluster? Might help focus the investigation... Thanks, EB On 10/13/20, 7:51 AM, "Jakov Varenina" <jakov.varen...@est.tech<mailto:jakov.varen...@est.tech>> wrote: Hi all, sorry for bothering, but we have noticed some differences in behavior between 1.11 and 1.12 releases and need your help in understanding them. First I would like to mention that we are running geode in Kubernetes. We perform shutdown of the worker node that is hosting one member(e.g. coordinator locator). Shutdown procedure affect member in a following way: 1. TCP shared unordered connections towards other members are terminated 2. Member receives graceful shut-down indication and starts with the shut-down procedure Usually connections starting to be terminated first and the shut-down indication comes short after (e.g. ~10 milliseconds in difference). The step 1. triggers availability check towards the other members for which TCP connection has been previously lost. At this point of time coordinator is unaware of ongoing shut-down and assumes that all other members are actually having issues due to connection loss. Even after coordinator receives the graceful shut-down indication this process of availability check is not stopped. What happens later on is that availability check fail for all members and coordinator initiates their removal with RemoveMemberMessage. This message is succesfully received on the other members forcing them to shut-down. In geode 1.11 everything is same except the fact that availability check pass and therefore removals aren't initiated. In logs it can be seen that for both releases TCP availability check fail, but HeartbeatMessageRequest/HearbeatMessage check fails only on 1.12 and pass on 1.11. In 1.12 release it can be seen that heartbeat request and heartbeat messages are sent but does not reach their destination members. RemoveMemberMessage which are sent later on reach their destination successfully. Does anybody know what was changed in 1.12 that could lead to such difference in behavior? Additionally, availability check is not stopped when graceful shutdown is initiated. Do you think that this could be improved, so that member stops ongoing availability check when detects gracefull shutdown? Just to add that shutdown procedure is also delayed due to unsuccessful attempts to estabilsh TCP connections towards the other members. BRs, Jakov
