Hi! I think the simple answer is that DLM wants to be shut down orderly, just as cLVM and OCFS2 do.
Regards, Ulrich >>> Patrick Whitney <[email protected]> schrieb am 11.09.2018 um 15:02 in Nachricht <CAE0zLk9OnX99vE0o9Xj+XtKLnP+uMUQFU6=2ygdnq4z_uxj...@mail.gmail.com>: > What I'm having trouble understanding is why dlm flattens the remaining > "running" node when the already fenced node is shutdown... I'm having > trouble understanding how power fencing would cause dlm to behave any > differently than just shutting down the fenced node. > > -Pat > > > On Thu, Sep 6, 2018 at 2:35 PM Vladislav Bogdanov <[email protected]> > wrote: > >> On September 6, 2018 2:36:03 PM UTC, Patrick Whitney < >> [email protected]> wrote: >>> >>> Good Morning Everyone, >>> >>> I'm hoping someone with more experience with corosync and pacemaker can >>> see what I am doing wrong. >>> >>> I've got a test setup of 2 nodes, with dlm and clvm setup as clones, and >>> using fence_scsi as my fencing agent. >>> >>> I've got it to the point where the cluster is up, and reports it is >>> happy. I then began testing fencing. When issuing 'pcs stonith fence' it >>> appears to work; that is, the scsi reservation is pulled and the output of >>> 'pcs status' looks sane, and I'm able to access resources on the un-fenced >>> node. >>> >>> Things go awry when I shutdown (init 0) the fenced node... my unfenced >>> node decides to fence itself (which looks like it was initiated by dlm due >>> to an abandoned lockspace). >>> >>> I suspect this is due to misconfiguration, since I'm new to the toolset, >>> but I'm not quite sure what I need to change. >>> >>> Any and all input is appreciated! >>> >>> Below is a chronology of events; my corosync config and cib.xml; command >>> output; and annotated logs. >>> >>> Again, any hints, suggestions, wild guesses, or premonitions are welcomed >>> -- I'm stuck! Please let me know if there is additional information which >>> would be helpful. >>> >>> Many thanks, >>> -Patrick W. >>> >>> Sep 6 08:54:14 -- Cluster is up and running; UI reports everything >>> healthy. >>> >>> Sep 6 08:55:44 -- 'pcs stonith fence' called against node 1 >>> (coro-test-1); >>> UI reports everything as expected -- that >>> is, resources show only running on unfenced node and they're available. >>> Oddly, although the UI says dlm is stopped >>> on fenced node, the dlm_controld is still running. >>> >>> Sep 6 09:03:38 -- node 1 is shutdown, and node 2 falls to pieces. >>> - First, corosync sees lost member -- >>> seems like this is appropriate, to me. >>> - Next, dlm_controld calls to fence >>> everything >>> - stonith-ng tries to fence node 1 (but >>> its already fenced!) >>> - dlm closes connection to "node 2" (does >>> dlm "nodes" map to cluster nodes? I'm not sure they do) >>> - clvmd dlm lockspace is now abandoned; >>> cluster attempts to fence the remaining node >>> (But can't because scsi_fence doesn't >>> work like that). >>> >>> *** >>> ****** -- Configuration -- >>> *** >>> root@coro-test-2:~# pcs --version >>> 0.9.149 >>> root@coro-test-2:~# pacemakerd --version >>> Pacemaker 1.1.14 >>> Written by Andrew Beekhof >>> root@coro-test-2:~# corosync -v >>> Corosync Cluster Engine, version '2.3.5' >>> Copyright (c) 2006-2009 Red Hat, Inc. >>> root@coro-test-2:~# >>> >>> # >>> ## COROSYNC >>> # >>> root@coro-test-1:~# more /etc/corosync/corosync.conf >>> totem { >>> version: 2 >>> secauth: off >>> cluster_name: my_first_cluster >>> transport: udpu >>> } >>> >>> nodelist { >>> node { >>> ring0_addr: coro-test-1 >>> nodeid: 1 >>> } >>> >>> node { >>> ring0_addr: coro-test-2 >>> nodeid: 2 >>> } >>> } >>> >>> quorum { >>> provider: corosync_votequorum >>> two_node: 1 >>> } >>> >>> logging { >>> to_logfile: yes >>> logfile: /var/log/corosync/corosync.log >>> to_syslog: yes >>> } >>> >>> ## >>> ### PACEMAKER CIB.XML >>> ## >>> root@coro-test-1:/var/lib/pacemaker/cib# cat cib.xml >>> <cib crm_feature_set="3.0.10" validate-with="pacemaker-2.4" epoch="30" >>> num_updates="0" admin_epoch="0" cib-last-written="Thu Sep 6 10:12:46 2018" >>> update-origin="coro-test-2" update-client="cibadmin" update-user="root" >>> have-quorum="1" dc-uuid="1"> >>> <configuration> >>> <crm_config> >>> <cluster_property_set id="cib-bootstrap-options"> >>> <nvpair id="cib-bootstrap-options-have-watchdog" >>> name="have-watchdog" value="false"/> >>> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" >>> value="1.1.14-70404b0"/> >>> <nvpair id="cib-bootstrap-options-cluster-infrastructure" >>> name="cluster-infrastructure" value="corosync"/> >>> <nvpair id="cib-bootstrap-options-cluster-name" >>> name="cluster-name" value="my_first_cluster"/> >>> <nvpair id="cib-bootstrap-options-no-quorum-policy" >>> name="no-quorum-policy" value="ignore"/> >>> <nvpair id="cib-bootstrap-options-stonith-enabled" >>> name="stonith-enabled" value="true"/> >>> <nvpair id="cib-bootstrap-options-stonith-action" >>> name="stonith-action" value="off"/> >>> </cluster_property_set> >>> </crm_config> >>> <nodes> >>> <node id="1" uname="coro-test-1"/> >>> <node id="2" uname="coro-test-2"/> >>> </nodes> >>> <resources> >>> <clone id="dlm-clone"> >>> <primitive class="ocf" id="dlm" provider="pacemaker" >>> type="controld"> >>> <instance_attributes id="dlm-instance_attributes"/> >>> <operations> >>> <op id="dlm-start-interval-0s" interval="0s" name="start" >>> timeout="90"/> >>> <op id="dlm-stop-interval-0s" interval="0s" name="stop" >>> timeout="100"/> >>> <op id="dlm-monitor-interval-30s" interval="30s" >>> name="monitor" on-fail="fence"/> >>> </operations> >>> </primitive> >>> <meta_attributes id="dlm-clone-meta_attributes"> >>> <nvpair id="dlm-clone-meta_attributes-interleave" >>> name="interleave" value="true"/> >>> <nvpair id="dlm-clone-meta_attributes-ordered" name="ordered" >>> value="true"/> >>> </meta_attributes> >>> </clone> >>> <clone id="clvmd-clone"> >>> <primitive class="ocf" id="clvmd" provider="heartbeat" >>> type="clvm"> >>> <instance_attributes id="clvmd-instance_attributes"/> >>> <operations> >>> <op id="clvmd-start-interval-0s" interval="0s" name="start" >>> timeout="90"/> >>> <op id="clvmd-stop-interval-0s" interval="0s" name="stop" >>> timeout="90"/> >>> <op id="clvmd-monitor-interval-30s" interval="30s" >>> name="monitor" on-fail="fence"/> >>> </operations> >>> </primitive> >>> <meta_attributes id="clvmd-clone-meta_attributes"> >>> <nvpair id="clvmd-clone-meta_attributes-interleave" >>> name="interleave" value="true"/> >>> <nvpair id="clvmd-clone-meta_attributes-ordered" name="ordered" >>> value="true"/> >>> </meta_attributes> >>> </clone> >>> <primitive class="stonith" id="scsi-shooter" type="fence_scsi"> >>> <instance_attributes id="scsi-shooter-instance_attributes"> >>> <nvpair id="scsi-shooter-instance_attributes-pcmk_host_list" >>> name="pcmk_host_list" value="coro-test-1 coro-test-2"/> >>> <nvpair >>> id="scsi-shooter-instance_attributes-pcmk_monitor_action" >>> name="pcmk_monitor_action" value="metadata"/> >>> <nvpair >>> id="scsi-shooter-instance_attributes-pcmk_reboot_action" >>> name="pcmk_reboot_action" value="off"/> >>> <nvpair id="scsi-shooter-instance_attributes-pcmk_delay_max" >>> name="pcmk_delay_max" value="15"/> >>> <nvpair id="scsi-shooter-instance_attributes-devices" >>> name="devices" value="/dev/sdb1"/> >>> </instance_attributes> >>> <meta_attributes id="scsi-shooter-meta_attributes"> >>> <nvpair id="scsi-shooter-meta_attributes-provides" >>> name="provides" value="unfencing"/> >>> </meta_attributes> >>> <operations> >>> <op id="scsi-shooter-monitor-interval-60s" interval="60s" >>> name="monitor"/> >>> </operations> >>> </primitive> >>> </resources> >>> <constraints> >>> <rsc_order first="dlm-clone" first-action="start" >>> id="order-dlm-clone-clvmd-clone-mandatory" then="clvmd-clone" >>> then-action="start"/> >>> <rsc_colocation id="colocation-clvmd-clone-dlm-clone-INFINITY" >>> rsc="clvmd-clone" score="INFINITY" with-rsc="dlm-clone"/> >>> </constraints> >>> </configuration> >>> </cib> >>> >>> *** >>> ****** -- Console Output -- Comments begin with "#" >>> *** >>> >>> # >>> ## >>> ### -- Moments before testing starts (Node 2)-- >>> ## >>> # >>> root@coro-test-2:~# date; pcs status >>> Thu Sep 6 08:54:14 EDT 2018 >>> Cluster name: my_first_cluster >>> Last updated: Thu Sep 6 08:54:14 2018 Last change: Wed Sep 5 16:28:53 >>> 2018 by root via cibadmin on coro-test-2 >>> Stack: corosync >>> Current DC: coro-test-1 (version 1.1.14-70404b0) - partition with quorum >>> 2 nodes and 5 resources configured >>> >>> Online: [ coro-test-1 coro-test-2 ] >>> >>> Full list of resources: >>> >>> Clone Set: dlm-clone [dlm] >>> Started: [ coro-test-1 coro-test-2 ] >>> Clone Set: clvmd-clone [clvmd] >>> Started: [ coro-test-1 coro-test-2 ] >>> scsi-shooter (stonith:fence_scsi): Started coro-test-1 >>> >>> PCSD Status: >>> coro-test-1: Online >>> coro-test-2: Online >>> >>> Daemon Status: >>> corosync: active/enabled >>> pacemaker: active/enabled >>> pcsd: active/enabled >>> root@coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1 >>> PR generation=0x41, 2 registered reservation keys follow: >>> 0x3ff20000 >>> 0x3ff20001 >>> root@coro-test-2:~# dlm_tool status >>> cluster nodeid 2 quorate 1 ring seq 496 496 >>> daemon now 58996 fence_pid 0 >>> node 1 M add 92 rem 0 fail 0 fence 0 at 0 0 >>> node 2 M add 92 rem 0 fail 0 fence 0 at 0 0 >>> root@coro-test-2:~# dlm_tool ls >>> dlm lockspaces >>> name clvmd >>> id 0x4104eefa >>> flags 0x00000000 >>> change member 2 joined 1 remove 0 failed 0 seq 1,1 >>> members 1 2 >>> >>> >>> # >>> ## >>> ### -- Node 1 is fenced -- >>> ## >>> # >>> root@coro-test-2:~# date; pcs stonith fence coro-test-1 >>> Thu Sep 6 08:55:44 EDT 2018 >>> Node: coro-test-1 fenced >>> root@coro-test-2:~# pcs status >>> Cluster name: my_first_cluster >>> Last updated: Thu Sep 6 08:56:40 2018 Last change: Wed Sep 5 16:28:53 >>> 2018 by root via cibadmin on coro-test-2 >>> Stack: corosync >>> Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum >>> 2 nodes and 5 resources configured >>> >>> Node coro-test-1: pending >>> Online: [ coro-test-2 ] >>> >>> Full list of resources: >>> >>> Clone Set: dlm-clone [dlm] >>> Started: [ coro-test-2 ] >>> Stopped: [ coro-test-1 ] >>> Clone Set: clvmd-clone [clvmd] >>> Started: [ coro-test-2 ] >>> Stopped: [ coro-test-1 ] >>> scsi-shooter (stonith:fence_scsi): Stopped >>> >>> PCSD Status: >>> coro-test-1: Online >>> coro-test-2: Online >>> >>> Daemon Status: >>> corosync: active/enabled >>> pacemaker: active/enabled >>> pcsd: active/enabled >>> root@coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1 >>> PR generation=0x42, 1 registered reservation key follows: >>> 0x3ff20001 >>> root@coro-test-2:~# dlm_tool ls >>> dlm lockspaces >>> name clvmd >>> id 0x4104eefa >>> flags 0x00000000 >>> change member 2 joined 1 remove 0 failed 0 seq 1,1 >>> members 1 2 >>> >>> root@coro-test-2:~# dlm_tool status >>> cluster nodeid 2 quorate 1 ring seq 496 496 >>> daemon now 59166 fence_pid 0 >>> node 1 M add 92 rem 0 fail 0 fence 0 at 0 0 >>> node 2 M add 92 rem 0 fail 0 fence 0 at 0 0 >>> root@coro-test-2:~# >>> >>> # >>> ## NOTE: DLM still running on other node, coro-test-1, at this point. >>> # >>> root@coro-test-1:~# ps -aef | grep dlm >>> root 1633 1 0 Sep05 ? 00:00:00 dlm_controld -s 0 >>> root 1752 2 0 Sep05 ? 00:00:00 [dlm_scand] >>> root 1754 2 0 Sep05 ? 00:00:00 [dlm_recv] >>> root 1755 2 0 Sep05 ? 00:00:00 [dlm_send] >>> root 1756 2 0 Sep05 ? 00:00:00 [dlm_recoverd] >>> root 29673 1850 0 08:58 pts/0 00:00:00 grep --color=auto dlm >>> root@coro-test-1:~# >>> >>> >>> # >>> ## >>> ### -- node 2 output after node 1 is shutdown >>> ## >>> # >>> #### NODE 1 SHUTDOWN #### >>> root@coro-test-1:~# pcs status; date; init 0 >>> Error: cluster is not currently running on this node >>> Thu Sep 6 09:03:38 EDT 2018 >>> >>> #### NODE 2 UI OUTPUT #### >>> root@coro-test-2:~# pcs status >>> Cluster name: my_first_cluster >>> Last updated: Thu Sep 6 09:04:16 2018 Last change: Wed Sep 5 16:28:53 >>> 2018 by root via cibadmin on coro-test-2 >>> Stack: corosync >>> Current DC: coro-test-2 (version 1.1.14-70404b0) - partition with quorum >>> 2 nodes and 5 resources configured >>> >>> Node coro-test-2: UNCLEAN (online) >>> OFFLINE: [ coro-test-1 ] >>> >>> Full list of resources: >>> >>> Clone Set: dlm-clone [dlm] >>> dlm (ocf::pacemaker:controld): FAILED coro-test-2 >>> Stopped: [ coro-test-1 ] >>> Clone Set: clvmd-clone [clvmd] >>> Started: [ coro-test-2 ] >>> Stopped: [ coro-test-1 ] >>> scsi-shooter (stonith:fence_scsi): Started coro-test-2 >>> >>> Failed Actions: >>> * dlm_monitor_30000 on coro-test-2 'unknown error' (1): call=17, >>> status=complete, exitreason='none', >>> last-rc-change='Thu Sep 6 09:04:09 2018', queued=0ms, exec=0ms >>> >>> >>> PCSD Status: >>> coro-test-1: Offline >>> coro-test-2: Online >>> >>> Daemon Status: >>> corosync: active/enabled >>> pacemaker: active/enabled >>> pcsd: active/enabled >>> root@coro-test-2:~# /usr/bin/sg_persist -n -i -k -d /dev/sdb1 >>> PR generation=0x42, 1 registered reservation key follows: >>> 0x3ff20001 >>> root@coro-test-2:~# dlm_tool status >>> cannot connect to dlm_controld >>> root@coro-test-2:~# dlm_tool ls >>> root@coro-test-2:~# >>> >>> *** >>> ****** -- Syslog Output -- Comments begin with "#" >>> *** >>> # >>> ### >>> ##### NODE 2 FENCES NODE 1 (pcs stonith fence coro-test-1) >>> ### >>> # >>> >>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-1' ] >>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: fence_scsi[27468] >>> stderr: [ ] >>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation >>> 'reboot' [27468] (call 2 from stonith_admin.27448) for host 'coro-test-1' >>> with device 'scsi-shooter' returned: 0 (OK) >>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Operation reboot >>> of coro-test-1 by coro-test-2 for [email protected]: >>> OK >>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: warning: >>> new_event_notification (1541-27448-12): Broken pipe (32) >>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition >>> S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK >>> origin=peer_update_callback ] >>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition >>> S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED >>> origin=election_timeout_popped ] >>> Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: crm_update_peer_proc: >>> Node coro-test-1[1] - state is now lost (was member) >>> Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Removing coro-test-1/1 >>> from the membership list >>> Sep 6 08:55:55 coro-test-2 attrd[1543]: notice: Purged 1 peers with >>> id=1 and/or uname=coro-test-1 from the membership cache >>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: >>> crm_update_peer_proc: Node coro-test-1[1] - state is now lost (was member) >>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Removing >>> coro-test-1/1 from the membership list >>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: notice: Purged 1 peers >>> with id=1 and/or uname=coro-test-1 from the membership cache >>> Sep 6 08:55:55 coro-test-2 cib[1540]: notice: crm_update_peer_proc: >>> Node coro-test-1[1] - state is now lost (was member) >>> Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Removing coro-test-1/1 >>> from the membership list >>> Sep 6 08:55:55 coro-test-2 cib[1540]: notice: Purged 1 peers with id=1 >>> and/or uname=coro-test-1 from the membership cache >>> Sep 6 08:55:55 coro-test-2 crmd[1545]: warning: FSA: Input >>> I_ELECTION_DC from do_election_check() received in state S_INTEGRATION >>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action >>> dlm:0_monitor_0 on coro-test-1 is unrunnable (pending) >>> Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action >>> clvmd:0_monitor_0 on coro-test-1 is unrunnable (pending) >>> Sep 6 08:55:55 coro-test-2 pengine[1544]: warning: Action >>> scsi-shooter_monitor_0 on coro-test-1 is unrunnable (pending) >>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Unfencing >>> coro-test-1: node discovery >>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Start >>> scsi-shooter#011(coro-test-2 - blocked) >>> Sep 6 08:55:55 coro-test-2 stonith-ng[1541]: error: Operation on of >>> coro-test-1 by <no-one> for [email protected]: No such >>> device >>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: Transition 0 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2, >>> Source=/var/lib/pacemaker/pengine/pe-input-107.bz2): Complete >>> Sep 6 08:55:55 coro-test-2 crmd[1545]: notice: State transition >>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL >>> origin=notify_crmd ] >>> Sep 6 08:55:55 coro-test-2 pengine[1544]: notice: Calculated >>> Transition 0: /var/lib/pacemaker/pengine/pe-input-107.bz2 >>> Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPREQUEST of 10.0.2.5 on >>> enp0s3 to 10.0.2.3 port 67 (xid=0x1c247c88) >>> Sep 6 08:58:22 coro-test-2 dhclient[1070]: DHCPACK of 10.0.2.5 from >>> 10.0.2.3 >>> Sep 6 08:58:22 coro-test-2 dhclient[1070]: bound to 10.0.2.5 -- renewal >>> in 466 seconds. >>> Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPREQUEST of 192.168.57.4 on >>> enp0s8 to 192.168.57.2 port 67 (xid=0x5dfb57d7) >>> Sep 6 08:59:20 coro-test-2 dhclient[860]: DHCPACK of 192.168.57.4 from >>> 192.168.57.2 >>> Sep 6 08:59:20 coro-test-2 dhclient[860]: bound to 192.168.57.4 -- >>> renewal in 599 seconds. >>> >>> # >>> ### >>> ##### NODE 1 IS SHUTDOWN >>> ### >>> # >>> >>> ## >>> ### Cluster appears to see the loss of node 1 >>> ## >>> Sep 6 09:03:38 coro-test-2 corosync[1505]: notice [TOTEM ] A new >>> membership (10.0.2.5:500) was formed. Members left: 1 >>> Sep 6 09:03:38 coro-test-2 corosync[1505]: [TOTEM ] A new membership ( >>> 10.0.2.5:500) was formed. Members left: 1 >>> Sep 6 09:03:38 coro-test-2 pacemakerd[1537]: notice: >>> crm_reap_unseen_nodes: Node coro-test-1[1] - state is now lost (was member) >>> Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 fence request 1 pid >>> 28247 procdown time 1536239018 fence_all dlm_stonith >>> Sep 6 09:03:38 coro-test-2 crmd[1545]: notice: crm_reap_unseen_nodes: >>> Node coro-test-1[1] - state is now lost (was member) >>> Sep 6 09:03:38 coro-test-2 dlm_controld[1657]: 59550 tell corosync to >>> remove nodeid 1 from cluster >>> >>> ## >>> ### DLM appears to close off connection to NODE 1 >>> ## >>> Sep 6 09:03:38 coro-test-2 kernel: [59552.844406] dlm: closing >>> connection to node 1 >>> Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Found 2 >>> entries for 1/(null): 0 in progress, 1 completed >>> Sep 6 09:03:38 coro-test-2 dlm_stonith: stonith_api_time: Node 1/(null) >>> last kicked at: 1536238555 >>> >>> ## >>> ### Stonith attempts to fence node 1 (again; it was already fenced) >>> ## >>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-1' ] >>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28248] >>> stderr: [ ] >>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation >>> 'reboot' [28248] (call 2 from stonith-api.28247) for host 'coro-test-1' >>> with device 'scsi-shooter' returned: 0 (OK) >>> Sep 6 09:03:42 coro-test-2 stonith-ng[1541]: notice: Operation reboot >>> of coro-test-1 by coro-test-2 for [email protected]: >>> OK >>> Sep 6 09:03:42 coro-test-2 stonith-api[28247]: stonith_api_kick: Node >>> 1/(null) kicked: reboot >>> Sep 6 09:03:42 coro-test-2 crmd[1545]: notice: State transition S_IDLE >>> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL >>> origin=abort_transition_graph ] >>> Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> >>> ## >>> ### I begin to get confused here: Why is dlm closing the connection >>> to the node 2 (itself??) >>> ## >>> Sep 6 09:03:42 coro-test-2 kernel: [59556.684699] dlm: closing >>> connection to node 2 >>> >>> ## >>> ### LOCKSPACE GOES ABANDONED!! why? WHY???? >>> ## >>> Sep 6 09:03:42 coro-test-2 dlm_controld[1657]: 59554 abandoned lockspace >>> clvmd >>> >>> ## >>> ### Alas, my cluster begins to topple itself >>> ## >>> Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Start >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:03:42 coro-test-2 pengine[1544]: notice: Calculated >>> Transition 1: /var/lib/pacemaker/pengine/pe-input-108.bz2 >>> Sep 6 09:03:42 coro-test-2 kernel: [59556.693578] dlm: dlm user daemon >>> left 1 lockspaces >>> Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Operation >>> scsi-shooter_start_0: ok (node=coro-test-2, call=20, rc=0, cib-update=56, >>> confirmed=true) >>> Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: Transition 1 >>> (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, >>> Source=/var/lib/pacemaker/pengine/pe-input-108.bz2): Complete >>> Sep 6 09:03:43 coro-test-2 crmd[1545]: notice: State transition >>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL >>> origin=notify_crmd ] >>> Sep 6 09:04:05 coro-test-2 controld(dlm)[28324]: ERROR: Uncontrolled >>> lockspace exists, system must reboot. Executing suicide fencing >>> Sep 6 09:04:07 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:07 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ ] >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ ] >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ ] >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ ] >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:08 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28344] >>> stderr: [ ] >>> Sep 6 09:04:09 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:09 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28357] >>> stderr: [ ] >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation >>> 'reboot' [28357] (call 2 from stonith_admin.controld.28343) for host >>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker >>> error) >>> Sep 6 09:04:09 coro-test-2 stonith-ng[1541]: error: Operation reboot >>> of coro-test-2 by coro-test-2 for >>> [email protected]: Generic Pacemaker >>> error >>> Sep 6 09:04:09 coro-test-2 crmd[1545]: notice: >>> coro-test-2-dlm_monitor_30000:17 [ Command failed: Generic Pacemaker >>> error\n ] >>> Sep 6 09:04:09 coro-test-2 crmd[1545]: notice: State transition S_IDLE >>> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL >>> origin=abort_transition_graph ] >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 2: /var/lib/pacemaker/pengine/pe-warn-68.bz2 >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:04:09 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 3: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:04:13 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:13 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ ] >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ ] >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ ] >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ ] >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:13 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28370] >>> stderr: [ ] >>> Sep 6 09:04:14 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:14 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28383] >>> stderr: [ ] >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28383] (call 3 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:04:14 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:04:14 coro-test-2 crmd[1545]: notice: Transition 3 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:04:14 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 4: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:04:19 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:19 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ ] >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ ] >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ ] >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ ] >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:19 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28432] >>> stderr: [ ] >>> Sep 6 09:04:20 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:20 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28445] >>> stderr: [ ] >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28445] (call 4 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:04:20 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:04:20 coro-test-2 crmd[1545]: notice: Transition 4 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:04:20 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 5: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:04:24 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:24 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ ] >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ ] >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ ] >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ ] >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:24 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28477] >>> stderr: [ ] >>> Sep 6 09:04:25 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:25 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28490] >>> stderr: [ ] >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28490] (call 5 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:04:25 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:04:25 coro-test-2 crmd[1545]: notice: Transition 5 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:04:25 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 6: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:04:39 coro-test-2 controld(dlm)[28520]: ERROR: Uncontrolled >>> lockspace exists, system must reboot. Executing suicide fencing >>> Sep 6 09:04:39 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:39 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ ] >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ ] >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ ] >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ ] >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:39 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28540] >>> stderr: [ ] >>> Sep 6 09:04:40 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:40 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28553] >>> stderr: [ ] >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28553] (call 6 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:04:40 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:04:40 coro-test-2 crmd[1545]: notice: Transition 6 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:04:40 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 7: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:04:49 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:49 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ ] >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ ] >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ ] >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ ] >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:49 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28568] >>> stderr: [ ] >>> Sep 6 09:04:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:50 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28581] >>> stderr: [ ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation >>> 'reboot' [28581] (call 2 from stonith_admin.controld.28539) for host >>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker >>> error) >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: error: Operation reboot >>> of coro-test-2 by coro-test-2 for >>> [email protected]: Generic Pacemaker >>> error >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: >>> new_event_notification (1541-28539-12): Broken pipe (32) >>> Sep 6 09:04:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:51 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28594] >>> stderr: [ ] >>> Sep 6 09:04:52 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:04:52 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28607] >>> stderr: [ ] >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28607] (call 7 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:04:52 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:04:52 coro-test-2 crmd[1545]: notice: Transition 7 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:04:52 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 8: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:05:01 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:01 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ ] >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ ] >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ ] >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ ] >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:01 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28643] >>> stderr: [ ] >>> Sep 6 09:05:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:02 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28656] >>> stderr: [ ] >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28656] (call 8 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:05:02 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:05:02 coro-test-2 crmd[1545]: notice: Transition 8 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:05:02 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 9: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:05:16 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:16 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ ] >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ ] >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ ] >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ ] >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:16 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28669] >>> stderr: [ ] >>> Sep 6 09:05:17 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:17 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28682] >>> stderr: [ ] >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28682] (call 9 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:05:18 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:05:18 coro-test-2 crmd[1545]: notice: Transition 9 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:05:18 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 10: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:05:21 coro-test-2 controld(dlm)[28695]: ERROR: Uncontrolled >>> lockspace exists, system must reboot. Executing suicide fencing >>> Sep 6 09:05:29 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:29 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ ] >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ ] >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ ] >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ ] >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:29 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28751] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28764] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28764] (call 10 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:05:30 coro-test-2 crmd[1545]: notice: Transition 10 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:05:30 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 11: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:05:30 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:30 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:30 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28777] >>> stderr: [ ] >>> Sep 6 09:05:31 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:31 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28790] >>> stderr: [ ] >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation >>> 'reboot' [28790] (call 2 from stonith_admin.controld.28714) for host >>> 'coro-test-2' with device 'scsi-shooter' returned: -201 (Generic Pacemaker >>> error) >>> Sep 6 09:05:31 coro-test-2 stonith-ng[1541]: error: Operation reboot >>> of coro-test-2 by coro-test-2 for >>> [email protected]: Generic Pacemaker >>> error >>> Sep 6 09:05:36 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:36 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ ] >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ ] >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ ] >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ ] >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:36 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28803] >>> stderr: [ ] >>> Sep 6 09:05:37 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:37 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28816] >>> stderr: [ ] >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28816] (call 11 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:05:37 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:05:37 coro-test-2 crmd[1545]: notice: Transition 11 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:05:37 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 12: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:05:50 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:50 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ ] >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ ] >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ ] >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ ] >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:50 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28829] >>> stderr: [ ] >>> Sep 6 09:05:51 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:05:51 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28842] >>> stderr: [ ] >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28842] (call 12 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:05:51 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:05:51 coro-test-2 crmd[1545]: notice: Transition 12 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: On loss of CCM >>> Quorum: Ignore >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Processing failed op >>> monitor for dlm:0 on coro-test-2: unknown error (1) >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Node coro-test-2 >>> will be fenced because of resource failure(s) >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:0 >>> with instance of dlm-clone >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Cannot pair clvmd:1 >>> with instance of dlm-clone >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Scheduling Node >>> coro-test-2 for STONITH >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop of failed >>> resource dlm:0 is implicit after coro-test-2 is fenced >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop >>> dlm:0#011(coro-test-2) >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop >>> clvmd:0#011(coro-test-2) >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: notice: Stop >>> scsi-shooter#011(coro-test-2) >>> Sep 6 09:05:51 coro-test-2 pengine[1544]: warning: Calculated >>> Transition 13: /var/lib/pacemaker/pengine/pe-warn-69.bz2 >>> Sep 6 09:06:01 coro-test-2 controld(dlm)[28874]: ERROR: Uncontrolled >>> lockspace exists, system must reboot. Executing suicide fencing >>> Sep 6 09:06:02 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:06:02 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ ] >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ ] >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ ] >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ ] >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:06:02 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28894] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28907] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation 'off' >>> [28907] (call 13 from crmd.1545) for host 'coro-test-2' with device >>> 'scsi-shooter' returned: -201 (Generic Pacemaker error) >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: error: Operation off of >>> coro-test-2 by coro-test-2 for [email protected]: Generic >>> Pacemaker error >>> Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition aborted: >>> Stonith failed (source=tengine_stonith_callback:748, 0) >>> Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: Transition 13 >>> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=9, >>> Source=/var/lib/pacemaker/pengine/pe-warn-69.bz2): Complete >>> Sep 6 09:06:03 coro-test-2 crmd[1545]: notice: State transition >>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL >>> origin=notify_crmd ] >>> Sep 6 09:06:03 coro-test-2 fence_scsi: Failed: keys cannot be same. You >>> can not fence yourself. >>> Sep 6 09:06:03 coro-test-2 fence_scsi: Please use '-h' for usage >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ WARNING:root:Parse error: Ignoring unknown option >>> 'port=coro-test-2' ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ ERROR:root:Failed: keys cannot be same. You can not fence >>> yourself. ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ Failed: keys cannot be same. You can not fence yourself. ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ ERROR:root:Please use '-h' for usage ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ Please use '-h' for usage ] >>> Sep 6 09:06:03 coro-test-2 stonith-ng[1541]: warning: fence_scsi[28922] >>> stderr: [ ] >>> Sep 6 09:06:04 >>> >> > > -- > Patrick Whitney > DevOps Engineer -- Tools _______________________________________________ Users mailing list: [email protected] https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
