Exactly the same parameters are set for my cluster:
[root@srv1 ~]# corosync-quorumtool -s
Quorum information
------------------
Date: Tue Mar 19 16:15:50 2019
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 1
Ring ID: 1/464
Quorate: Yes
Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 2
Quorum: 1
Flags: 2Node Quorate WaitForAll
Membership information
----------------------
Nodeid Votes Name
1 1 srv1cr1 (local)
2 1 srv2cr1
I was testing fencing (I’m using fence_vmware_soap) and followed
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-stonithtest-haarand
could in each case see that when:
a) Fencing the passive node the active remained active *
b) Fencing the active node caused the passive node to take over
the active role*
*in all cases pacemaker and corosync was not configured to start on
boot
Yes you are absolutely right regarding the fence condition when both
shoot at the same time and I even tried option 3 from your list that
is corosync qdevice. Here I followed
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-quorumdev-haar#s2-managequorum-HAAR
So here’s quorum runtime status before adding the qdevice:
[root@srv1 ~]# pcs quorum status
Quorum information
------------------
Date: Tue Mar 19 16:34:12 2019
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 1
Ring ID: 1/464
Quorate: Yes
Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 2
Quorum: 1
Flags: 2Node Quorate WaitForAll
Membership information
----------------------
Nodeid Votes Qdevice Name
1 1 NR srv1cr1 (local)
2 1 NR srv2cr1
And here’s the one after adding it:
[root@srv1 ~]# pcs quorum status
Quorum information
------------------
Date: Tue Mar 19 16:35:06 2019
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 1
Ring ID: 1/464
Quorate: Yes
Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate WaitForAll Qdevice
Membership information
----------------------
Nodeid Votes Qdevice Name
1 1 A,V,NMW srv1cr1 (local)
2 1 A,V,NMW srv2cr1
0 1 Qdevice
I got some while adding the quorum device to the cluster because as
mentioned I have on both nodes pacemaker and corosync set to not
start at boot:
[root@srv1 ~]# pcs quorum device add model net host=otrs_db
algorithm=ffsplit
Setting up qdevice certificates on nodes...
srv2cr1: Succeeded
srv1cr1: Succeeded
Enabling corosync-qdevice...
srv2cr1: not enabling corosync-qdevice: corosync is not enabled
srv1cr1: not enabling corosync-qdevice: corosync is not enabled
Sending updated corosync.conf to nodes...
srv2cr1: Succeeded
srv1cr1: Succeeded
Corosync configuration reloaded
Starting corosync-qdevice...
srv2cr1: corosync-qdevice started
srv1cr1: corosync-qdevice started
Is this a problem ? what can I do now, how can I test it ?
Thank you !
wt., 19.03.2019, 19:01 użytkownik Ken Gaillot <[email protected]>
napisał:
On Tue, 2019-03-19 at 15:55 +0100, Adam Budziński wrote:
Hello Ken,
Thank you.
But if I have a two node cluster and a working fencing mechanism
wouldn't it be enough to disable the corosync and pacemaker
service
on both nodes so when it fence they won't come up?
There's actually no problem when a fenced node comes back. Either
it
joins the remaining cluster normally, or (due to two_node /
wait_for_all in corosync.conf) waits until it can see the other
node
before doing anything.
Disabling or enabling the services is a personal preference based
on
whether you'd rather investigate why a node was shot before letting
it
back in the cluster, or get the cluster back to full strength as
quickly as possible.
The fence delay is for when both nodes are running and
communicating
correctly, then suddenly they lose communication with each other.
From
each node's point of view, the other node is lost. So, each will
attempt to fence the other. A delay on one node in this situation
makes
it less likely that they both pull the trigger at the same time,
ending
up with both nodes dead.
Thank you
pon., 18.03.2019, 16:19 użytkownik Ken Gaillot <
[email protected]>
napisał:
On Sat, 2019-03-16 at 11:10 +0100, Adam Budziński wrote:
Hello Andrei,
Ok I see your point. So per my understanding if the resource
is
started successfully in that case fence vmware it will be
monitored
indefinitely but as you sad it will monitor the current
active
node.
So how does the fence agent gets aware of problems with the
slave? I
The fence agent doesn't monitor the active node, or any node --
it
monitors the fence device.
The cluster layer (i.e. corosync) monitors all nodes, and
reports
any
issues to pacemaker, which will initiate fencing if necessary.
Pacemaker also monitors each resource and fence device, via any
recurring monitors that have been configured.
mean if in a two node cluster the cluster splits in to two
partitions
each of them will fence the other or does that happen because
both
will assume they are the only survivors and thus need to
fence
the
other node which is in a unknow state so to say?
If both nodes are functional but can't see each other, they
will
each
want to initiate fencing. If one of them is quicker than the
other
to
determine this, the other one will get shot before it has a
chance
to
do anything itself.
However there is the possibility that both nodes will shoot at
about
the same time, resulting in both nodes getting shot (a "stonith
death
match"). This is only a problem in 2-node clusters. There are a
few
ways around this:
1. Configure two separate fence devices, each targeting one of
the
nodes, and put a delay on one of them (or a random delay on
both).
This
makes it highly unlikely that they will shoot at the same time.
2. Configure a fencing topology with a fence heuristics device
plus
your real device. A fence heuristics device runs some test, and
refuses
to shoot the other node if the test fails. For example,
fence_heuristics_ping tries to ping an IP address you give it;
the
idea
is that if a node can't ping that IP, you don't want it to
survive.
This ensures that only a node that passes the test can shoot
(which
means there still might be some cases where the nodes can both
shoot
each other, and cases where the cluster will freeze because
neither
node can see the IP).
3. Configure corosync with qdevice to provide true quorum via a
third
host (which doesn't participate in the cluster otherwise).
4. Use sbd with a hardware watchdog and a shared storage device
as
the
fencing device. This is not a reliable option with VMWare, but
I'm
listing it for the general case.
Thank you and Best Regards,
Adam
sob., 16.03.2019, 07:17 użytkownik Andrei Borzenkov <
[email protected]> napisał:
16.03.2019 9:01, Adam Budziński пишет:
Thank you Andrei. The problem is that I can see with 'pcs
status'
that
resources are runnin on srv2cr1 but its at the same time
its
telling that
the fence_vmware_soap is running on srv1cr1. That's
somewhat
confusing.
Could you possibly explain this?
Two points.
It is actually logical to have stonith agent running on
different
node
than node with active resources - because it is the *other*
node
that
will initiate fencing when node with active resources
fails.
But even considering the above, active (running) state of
fence
(or
stonith) agent just determines on which node recurring
monitor
operation
will be started. The actual result of this monitor
operation
has no
impact on subsequent stonith attempt and serves just as
warning
to
administrator. When stonith request comes, agent may be
used by
any
node
where stonith agent is not prohibited to run by (co-
)location
rules. My
understanding is that this node is selected by DC in
partition.
Thank you!
sob., 16.03.2019, 05:37 użytkownik Andrei Borzenkov <
[email protected]>
napisał:
16.03.2019 1:16, Adam Budziński пишет:
Hi Tomas,
Ok but how then pacemaker or the fence agent knows
which
route
to take to
reach the vCenter?
They do not know or care at all. It is up to your
underlying
operating
system and its routing tables.
Btw. Do I have to add the stonith resource on each of
the
nodes
or is it
just enough to add it on one as for other resources?
If your fencing agent can (should) be able to run on any
node,
it should
be enough to define it just once as long as it can
properly
determine
"port" to use on fencing "device" for a given node.
There
are
cases when
you may want to restrict fencing agent to only subset on
nodes
or when
you are forced to set unique parameter for each node
(consider
IPMI IP
address), in this case you would need separate instance
of
agent
in each
case.
Thank you!
pt., 15.03.2019, 15:48 użytkownik Tomas Jelinek <
[email protected]>
napisał:
Dne 15. 03. 19 v 15:09 Adam Budziński napsal(a):
Hello Tomas,
Thank you! So far I need to say how great this
community
is,
would
never expect so much positive vibes! A big thank you
your
doing a great
job!
Now let's talk business :)
So if pcsd is using ring0 and it fails will ring1 not
be
used
at all?
Pcs and pcsd never use ring1, but they are just tools
for
managing
clusters. You can have a perfectly functioning cluster
without
pcs and
pcsd running or even installed, it would be just more
complicated to set
it up and manage it.
Even if ring0 fails, you will be able to use pcs (in
somehow
limited
manner) as most of its commands don't go through
network
anyway.
Corosync, which is the actual cluster messaging layer,
will of
course
use ring1 in case of ring0 failure.
So in regards to VMware that would mean that the
interface
should be
configured with a network that can access the
vCenter to
fence right?
But wouldn't it then use only ring0 so if that fails
it
wouldn't switch
to ring1?
If you are talking about pcmk_host_map, that does not
really
have
anything to do with network interfaces of cluster
nodes.
It
maps node
names (parts before :) to "ports" of a fence device
(parts
after :).
Pcs-0.9.x does not support defining custom node names,
therefore node
names are the same as ring0 addresses.
I am not an expert on fence agents / devices, but I'm
sure
someone else
on this list will be able to help you with configuring
fencing
for your
cluster.
Tomas
Thank you!
pt., 15.03.2019, 13:14 użytkownik Tomas Jelinek <
[email protected]
<mailto:[email protected]>> napisał:
Dne 15. 03. 19 v 12:32 Adam Budziński napsal(a):
> Hello Folks,____
>
> __ __
>
> Tow node active/passive VMware VM cluster.____
>
> __ __
>
> /etc/hosts____
>
> __ __
>
> 10.116.63.83 srv1____
>
> 10.116.63.84 srv2____
>
> 172.16.21.12 srv2cr1____
>
> 172.16.22.12 srv2cr2____
>
> 172.16.21.11 srv1cr1____
>
> 172.16.22.11 srv1cr2____
>
> __ __
>
> __ __
>
> I have 3 NIC’s on each VM:____
>
> __ __
>
> 10.116.63.83 srv1 and 10.116.63.84 srv2
are
networks used
to
> access the VM’s via SSH or any resource
directly
if
not via a
VIP.____
>
> __ __
>
> Everything with cr in its name is used for
corosync
communication, so
> basically I have two rings (this are two no
routable
networks
just for
> that).____
>
> __ __
>
> My questions are:____
>
> __ __
>
> __1.__With ‘pcs cluster auth’ which interface
/
interfaces
should
I use
> ?____
Hi Adam,
I can see you are using pcs-0.9.x. In that case
you
should do:
pcs cluster auth srv1cr1 srv2cr1
In other words, use the first address of each
node.
Authenticating all the other addresses should not
cause
any issues.
It
is pointless, though, as pcs only communicates
via
ring0
addresses.
>
> __2.__With ‘pcs cluster setup –name’ I would
use
the
corosync
interfaces
> e.g. ‘pcs cluster setup –name MyCluster
srv1cr1,srv1cr2
srv2cr1,srv2cr2’
> right ?____
Yes, that is correct.
>
> __3.__With fence_vmware_soap
inpcmk_host_map="X:VM_C;X:VM:OTRS_D"
which
> interface should replace X ?____
X should be replaced by node names as seen by
pacemaker.
Once you
set up
and start your cluster, run 'pcs status' to get
(amongs
other info)
the
node names. In your configuration, they should be
srv1cr1
and
srv2cr1.
Regards,
Tomas
> __ __
>
> Thank you!
>
>
>
_______________________________________________
> Users mailing list: [email protected]
<mailto:[email protected]>
>
https://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
_______________________________________________
Users mailing list: [email protected]
<mailto:
[email protected]>
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
