[ 
https://issues.apache.org/jira/browse/GEODE-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruce Schuchardt reopened GEODE-3780:
-------------------------------------

Another variant of this bug was hit in network-down testing.

The problem is happening when the network partition isn't instantaneous, which 
is the case with most test frameworks that use iptable manipulation.

The network from the losing side to the surviving side was shut down and 
process 13989 initiates a final existence check on one of the surviving-side 
processes:

{noformat}
system.log: [info 2018/08/21 21:56:43.082 PDT gemfire1_host1_13989 <Geode 
Failure Detection thread 1> tid=0x84] Performing final check for suspect member 
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025 reason=Unable to 
send messages to this member via JGroups
{noformat}

But the network from the surviving-side -> the losing side was still open and 
13989 received messages from it:

{noformat}
system.log: [info 2018/08/21 21:56:48.084 PDT gemfire1_host1_13989 <Geode 
Failure Detection thread 1> tid=0x84] Final check failed but detected recent 
message traffic for suspect member 
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025

system.log: [info 2018/08/21 21:56:48.085 PDT gemfire1_host1_13989 <Geode 
Failure Detection thread 1> tid=0x84] Final check passed for suspect member 
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025
{noformat}

The health monitor eventually suspected the other surviving-side member and 
kicked it out, but it never performed another final check on 
gemfire4_host2_12365 and so did not shut down:

{noformat}
system.log: [info 2018/08/21 21:56:48.086 PDT gemfire1_host1_13989 <Geode 
Failure Detection thread 2> tid=0x85] Final check failed - requesting removal 
of suspect member 10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026

system.log: [info 2018/08/21 21:56:50.536 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026

system.log: [info 2018/08/21 21:56:50.536 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025

system.log: [info 2018/08/21 21:56:53.036 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026

system.log: [info 2018/08/21 21:56:53.036 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.111.37(gemfire4_host2_12365:12365:locator)<ec><v1>:1025

system.log: [info 2018/08/21 21:56:55.536 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026

system.log: [info 2018/08/21 21:56:58.036 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.108.137(gemfire1_host1_13989:13989)<v14>:1024

system.log: [info 2018/08/21 21:56:58.036 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026

system.log: [info 2018/08/21 21:57:00.536 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.108.137(gemfire1_host1_13989:13989)<v14>:1024

system.log: [info 2018/08/21 21:57:00.536 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026

system.log: [info 2018/08/21 21:57:03.036 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.108.137(gemfire1_host1_13989:13989)<v14>:1024

system.log: [info 2018/08/21 21:57:03.036 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] Failure detection is now watching 
10.32.111.37(gemfire3_host2_12393:12393)<v2>:1026

system.log: [info 2018/08/21 21:57:05.536 PDT gemfire1_host1_13989 <Geode 
Failure Detection Scheduler> tid=0x22] All other members are suspect at this 
point
{noformat}


> suspected member is never watched again after passing final check
> -----------------------------------------------------------------
>
>                 Key: GEODE-3780
>                 URL: https://issues.apache.org/jira/browse/GEODE-3780
>             Project: Geode
>          Issue Type: Bug
>          Components: membership
>            Reporter: Bruce Schuchardt
>            Assignee: Bruce Schuchardt
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.7.0
>
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> In a network-down test we saw a node on the losing side of the network 
> partition perform final checks on members on the winning side.  One of the 
> final checks mysteriously succeeded
> [info 2017/09/17 12:24:45.552 PDT 
> gemfire1_rs-FullRegression-2017-09-15-21-00-35-client-10_8941 <Geode Failure 
> Detection thread 4> tid=0x128] Final check failed but detected recent message 
> traffic for suspect member 
> 10.32.109.252(gemfire3_rs-FullRegression-2017-09-15-21-00-35-client-16_6135:6135)<v2>:1026
> [info 2017/09/17 12:24:45.552 PDT 
> gemfire1_rs-FullRegression-2017-09-15-21-00-35-client-10_8941 <Geode Failure 
> Detection thread 4> tid=0x128] Final check passed for suspect member 
> 10.32.109.252(gemfire3_rs-FullRegression-2017-09-15-21-00-35-client-16_6135:6135)<v2>:1026
> After this the suspected member was never checked again and the losing side 
> failed to shut down.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to