Hi all, Got a really odd one here...
I had a cluster in the lab where it was built and tested. Then we deployed it, and I've been trying to update the stonith config. It _seems_ to take at first, then it reverts back to the old config. This is super annoying, obviously. :) So I can confirm that the APC PDUs work; ==== [root@an-a02n01 ~]# fence_apc_snmp -a 10.201.2.3 -n 3 -o status Status: ON [root@an-a02n01 ~]# fence_apc_snmp -a 10.201.2.4 -n 3 -o status Status: ON ==== Here's the config as-shipped; ==== # pcs stonith config apc_snmp_node1_an-pdu03 Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=5 Operations: monitor interval=60 (apc_snmp_node1_an-pdu03-monitor-interval-60) Target: an-a02n01 Level 1 - ipmilan_node1 Level 2 - apc_snmp_node1_an-pdu03,apc_snmp_node1_an-pdu04 Level 3 - delay_node1 Target: an-a02n02 Level 1 - ipmilan_node2 Level 2 - apc_snmp_node2_an-pdu03,apc_snmp_node2_an-pdu04 Level 3 - delay_node2 ==== So in this example, I am trying to update 'apc_snmp_node1_an-pdu03' to change the IP from 10.201.2.1 -> 10.201.2.3 and to change the port from port 5 -> 3. ==== # pcs stonith update apc_snmp_node1_an-pdu03 ip=10.201.2.3 pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=3 # pcs stonith config apc_snmp_node1_an-pdu03 Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) Attributes: ip=10.201.2.3 pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=3 ==== As you can see, initially it appears to work. However, after a minute, the config reverts; ==== # pcs stonith config apc_snmp_node1_an-pdu03 Resource: apc_snmp_node1_an-pdu03 (class=stonith type=fence_apc_snmp) Attributes: ip=10.201.2.1 pcmk_host_list=an-a02n01 pcmk_off_action=reboot port=5 ==== This happens to all four stonith devices (two per node, two nodes). I've tried doing a 'pcs stonith disable <id>' for all four devices, and did a 'pcs stonith cleanup' to make sure errors were cleared before updating, still no luck. Any idea what I'm doing wrong? The logs from the node I run the update on, followed by the logs on the peer: digimer ==== Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Updating all attributes after cib_refresh_notify event Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_IDLE -> S_POLICY_ENGINE Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_ELECTION -> S_INTEGRATION Jul 23 16:44:43 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n01 because resource parameters have changed Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:37:55 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Clearing failure of apc_snmp_node1_an-pdu03 on an-a02n02 because resource parameters have changed Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:37:42 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Calculated transition 173, saving inputs in /var/lib/pacemaker/pengine/pe-input-193.bz2 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627083488 -> (unset) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627083475 -> (unset) Jul 23 16:44:47 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 173 aborted by deletion of lrm_rsc_op[@id='apc_snmp_node1_an-pdu03_last_failure_0']: Resource operation removal Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 173 (Complete=3, Pending=0, Fired=0, Skipped=1, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-193.bz2): Stopped Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Calculated transition 174, saving inputs in /var/lib/pacemaker/pengine/pe-input-194.bz2 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Initiating monitor operation apc_snmp_node1_an-pdu03_monitor_60000 on an-a02n02 Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 174 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-194.bz2): Complete Jul 23 16:44:48 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_IDLE -> S_POLICY_ENGINE Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n02 ) due to node availability Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Calculated transition 175, saving inputs in /var/lib/pacemaker/pengine/pe-input-195.bz2 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 175 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-195.bz2): Complete Jul 23 16:45:09 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Updating all attributes after cib_refresh_notify event Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_IDLE -> S_POLICY_ENGINE Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_ELECTION -> S_INTEGRATION Jul 23 16:45:11 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Actions: Start apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Calculated transition 176, saving inputs in /var/lib/pacemaker/pengine/pe-input-196.bz2 Jul 23 16:45:16 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 176 aborted by operation apc_snmp_node1_an-pdu03_start_0 'modify' on an-a02n02: Event failed Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 176 action 31 (apc_snmp_node1_an-pdu03_start_0 on an-a02n02): expected 'ok' but got 'error' Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 176 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-196.bz2): Complete Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: (unset) -> INFINITY Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: (unset) -> 1627083929 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 ) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Calculated transition 177, saving inputs in /var/lib/pacemaker/pengine/pe-input-197.bz2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n02 -> an-a02n01 ) Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Calculated transition 178, saving inputs in /var/lib/pacemaker/pengine/pe-input-198.bz2 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 on an-a02n02 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Initiating start operation apc_snmp_node1_an-pdu03_start_0 locally on an-a02n01 Jul 23 16:45:29 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: [142B blob data] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: fence_apc_snmp_monitor_1[30692] error output [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: fence_apc_snmp_monitor_1[30692] error output [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: fence_apc_snmp_monitor_1[30692] error output [ 2021-07-23 16:45:35,820 ERROR: Please use '-h' for usage ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: fence_apc_snmp_monitor_1[30692] error output [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: [128B blob data] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: warning: fence_apc_snmp[30692] stderr: [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: warning: fence_apc_snmp[30692] stderr: [ ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: warning: fence_apc_snmp[30692] stderr: [ 2021-07-23 16:45:35,820 ERROR: Please use '-h' for usage ] Jul 23 16:45:35 an-a02n01.alteeve.com pacemaker-fenced[121629]: warning: fence_apc_snmp[30692] stderr: [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: [142B blob data] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: fence_apc_snmp_monitor_2[31341] error output [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: fence_apc_snmp_monitor_2[31341] error output [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: fence_apc_snmp_monitor_2[31341] error output [ 2021-07-23 16:45:42,979 ERROR: Please use '-h' for usage ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: fence_apc_snmp_monitor_2[31341] error output [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: [128B blob data] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: warning: fence_apc_snmp[31341] stderr: [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: warning: fence_apc_snmp[31341] stderr: [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: warning: fence_apc_snmp[31341] stderr: [ 2021-07-23 16:45:42,979 ERROR: Please use '-h' for usage ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: warning: fence_apc_snmp[31341] stderr: [ ] Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-fenced[121629]: notice: Operation 'monitor' [31341] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n01: error Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 178 aborted by operation apc_snmp_node1_an-pdu03_start_0 'modify' on an-a02n01: Event failed Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 178 action 32 (apc_snmp_node1_an-pdu03_start_0 on an-a02n01): expected 'ok' but got 'error' Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 178 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-198.bz2): Complete Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: (unset) -> INFINITY Jul 23 16:45:42 an-a02n01.alteeve.com pacemaker-attrd[121631]: notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: (unset) -> 1627083942 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Actions: Recover apc_snmp_node1_an-pdu03 ( an-a02n01 ) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Calculated transition 179, saving inputs in /var/lib/pacemaker/pengine/pe-input-199.bz2 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n01 at Jul 23 16:45:29 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n01 at Jul 23 16:35:04 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node1_an-pdu03 on an-a02n02 at Jul 23 16:45:16 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu03 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Unexpected result (error) was recorded for start of apc_snmp_node2_an-pdu04 on an-a02n02 at Jul 23 16:34:50 2021 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n01 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node2_an-pdu04 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: warning: Forcing apc_snmp_node1_an-pdu03 away from an-a02n02 after 1000000 failures (max=1000000) Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Actions: Stop apc_snmp_node1_an-pdu03 ( an-a02n01 ) due to node availability Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-schedulerd[121632]: notice: Calculated transition 180, saving inputs in /var/lib/pacemaker/pengine/pe-input-200.bz2 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Initiating stop operation apc_snmp_node1_an-pdu03_stop_0 locally on an-a02n01 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01 Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n01: ok Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: Transition 180 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-200.bz2): Complete Jul 23 16:45:43 an-a02n01.alteeve.com pacemaker-controld[121633]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE ==== ==== Jul 23 16:44:43 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: State transition S_NOT_DC -> S_PENDING Jul 23 16:44:43 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 16:44:43 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: State transition S_PENDING -> S_NOT_DC Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-attrd[3296858]: notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: INFINITY -> (unset) Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-attrd[3296858]: notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: 1627083488 -> (unset) Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-attrd[3296858]: notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: INFINITY -> (unset) Jul 23 16:44:47 an-a02n02.alteeve.com pacemaker-attrd[3296858]: notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: 1627083475 -> (unset) Jul 23 16:44:48 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n02: ok Jul 23 16:44:48 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Requesting local execution of monitor operation for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:44:48 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Result of monitor operation for apc_snmp_node1_an-pdu03 on an-a02n02: ok Jul 23 16:45:09 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:45:09 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n02: ok Jul 23 16:45:11 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: State transition S_NOT_DC -> S_PENDING Jul 23 16:45:11 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: State transition S_PENDING -> S_NOT_DC Jul 23 16:45:11 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: Added 'apc_snmp_node1_an-pdu03' to device list (5 active devices) Jul 23 16:45:16 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Requesting local execution of start operation for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: [144B blob data] Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: fence_apc_snmp_monitor_1[3377776] error output [ ] Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: fence_apc_snmp_monitor_1[3377776] error output [ ] Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: fence_apc_snmp_monitor_1[3377776] error output [ 2021-07-23 16:45:22,463 ERROR: Please use '-h' for usage ] Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: fence_apc_snmp_monitor_1[3377776] error output [ ] Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: [130B blob data] Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: warning: fence_apc_snmp[3377776] stderr: [ ] Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: warning: fence_apc_snmp[3377776] stderr: [ ] Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: warning: fence_apc_snmp[3377776] stderr: [ 2021-07-23 16:45:22,463 ERROR: Please use '-h' for usage ] Jul 23 16:45:22 an-a02n02.alteeve.com pacemaker-fenced[3296856]: warning: fence_apc_snmp[3377776] stderr: [ ] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: [144B blob data] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: fence_apc_snmp_monitor_2[3378394] error output [ ] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: fence_apc_snmp_monitor_2[3378394] error output [ ] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: fence_apc_snmp_monitor_2[3378394] error output [ 2021-07-23 16:45:29,627 ERROR: Please use '-h' for usage ] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: fence_apc_snmp_monitor_2[3378394] error output [ ] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: [130B blob data] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: warning: fence_apc_snmp[3378394] stderr: [ ] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: warning: fence_apc_snmp[3378394] stderr: [ ] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: warning: fence_apc_snmp[3378394] stderr: [ 2021-07-23 16:45:29,627 ERROR: Please use '-h' for usage ] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: warning: fence_apc_snmp[3378394] stderr: [ ] Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-fenced[3296856]: notice: Operation 'monitor' [3378394] using apc_snmp_node1_an-pdu03 returned -201 (Generic Pacemaker error) Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Result of start operation for apc_snmp_node1_an-pdu03 on an-a02n02: error Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-attrd[3296858]: notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: (unset) -> INFINITY Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-attrd[3296858]: notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n02]: (unset) -> 1627083929 Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Requesting local execution of stop operation for apc_snmp_node1_an-pdu03 on an-a02n02 Jul 23 16:45:29 an-a02n02.alteeve.com pacemaker-controld[3296860]: notice: Result of stop operation for apc_snmp_node1_an-pdu03 on an-a02n02: ok Jul 23 16:45:42 an-a02n02.alteeve.com pacemaker-attrd[3296858]: notice: Setting fail-count-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: (unset) -> INFINITY Jul 23 16:45:42 an-a02n02.alteeve.com pacemaker-attrd[3296858]: notice: Setting last-failure-apc_snmp_node1_an-pdu03#start_0[an-a02n01]: (unset) -> 1627083942 ==== -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
