[
https://issues.apache.org/jira/browse/GEODE-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruce Schuchardt reopened GEODE-3780:
-------------------------------------
Another variant of this bug was hit in network-down testing.
The problem is happening when the network partition isn't instantaneous, which
is the case with most test frameworks that use iptable manipulation.
The network from the losing side to the surviving side was shut down and
process 13989 initiates a final existence check on one of the surviving-side
processes:
{noformat}
system.log: [info 2018/08/21 21:56:43.082 PDT gemfire1_host1_13989 <Geode
Failure Detection thread 1> tid=0x84] Performing final check for suspect member
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025 reason=Unable to
send messages to this member via JGroups
{noformat}
But the network from the surviving-side -> the losing side was still open and
13989 received messages from it:
{noformat}
system.log: [info 2018/08/21 21:56:48.084 PDT gemfire1_host1_13989 <Geode
Failure Detection thread 1> tid=0x84] Final check failed but detected recent
message traffic for suspect member
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025
system.log: [info 2018/08/21 21:56:48.085 PDT gemfire1_host1_13989 <Geode
Failure Detection thread 1> tid=0x84] Final check passed for suspect member
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025
{noformat}
The health monitor eventually suspected the other surviving-side member and
kicked it out, but it never performed another final check on
gemfire4_host2_12365 and so did not shut down:
{noformat}
system.log: [info 2018/08/21 21:56:48.086 PDT gemfire1_host1_13989 <Geode
Failure Detection thread 2> tid=0x85] Final check failed - requesting removal
of suspect member 10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026
system.log: [info 2018/08/21 21:56:50.536 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026
system.log: [info 2018/08/21 21:56:50.536 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025
system.log: [info 2018/08/21 21:56:53.036 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026
system.log: [info 2018/08/21 21:56:53.036 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025
system.log: [info 2018/08/21 21:56:55.536 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026
system.log: [info 2018/08/21 21:56:58.036 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.108.137(gemfire1_host1_13989:13989)<v14>:1024
system.log: [info 2018/08/21 21:56:58.036 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026
system.log: [info 2018/08/21 21:57:00.536 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.108.137(gemfire1_host1_13989:13989)<v14>:1024
system.log: [info 2018/08/21 21:57:00.536 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026
system.log: [info 2018/08/21 21:57:03.036 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.108.137(gemfire1_host1_13989:13989)<v14>:1024
system.log: [info 2018/08/21 21:57:03.036 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] Failure detection is now watching
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026
system.log: [info 2018/08/21 21:57:05.536 PDT gemfire1_host1_13989 <Geode
Failure Detection Scheduler> tid=0x22] All other members are suspect at this
point
{noformat}
> suspected member is never watched again after passing final check
> -----------------------------------------------------------------
>
> Key: GEODE-3780
> URL: https://issues.apache.org/jira/browse/GEODE-3780
> Project: Geode
> Issue Type: Bug
> Components: membership
> Reporter: Bruce Schuchardt
> Assignee: Bruce Schuchardt
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In a network-down test we saw a node on the losing side of the network
> partition perform final checks on members on the winning side. One of the
> final checks mysteriously succeeded
> [info 2017/09/17 12:24:45.552 PDT
> gemfire1_rs-FullRegression-2017-09-15-21-00-35-client-10_8941 <Geode Failure
> Detection thread 4> tid=0x128] Final check failed but detected recent message
> traffic for suspect member
> 10.32.109.252(gemfire3_rs-FullRegression-2017-09-15-21-00-35-client-16_6135:6135)<v2>:1026
> [info 2017/09/17 12:24:45.552 PDT
> gemfire1_rs-FullRegression-2017-09-15-21-00-35-client-10_8941 <Geode Failure
> Detection thread 4> tid=0x128] Final check passed for suspect member
> 10.32.109.252(gemfire3_rs-FullRegression-2017-09-15-21-00-35-client-16_6135:6135)<v2>:1026
> After this the suspected member was never checked again and the losing side
> failed to shut down.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
