= Conversion to Active/Active =
== Requirements ==
The primary requirement for an Active/Active cluster is that the data
required for your services is available, simultaneously, on both
machines. Pacemaker makes no requirement on how this is achieved, you
could use a SAN if you had one available, however since DRBD supports
multiple Primaries, we can also use that.
The only hitch is that we need to use a cluster-aware filesystem. The
one we used earlier with DRBD, ext4, is not one of those. Both OCFS2
and GFS2 are supported, however here we will use GFS2 which comes with
Fedora 13.
We'll also need to use CMAN for Cluster Membership and Quorum instead
of our Corosync plugin.
== Adding CMAN Support ==
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Cluster_Suite_Overview/index.html#s2-clumembership-overview-CSO[CMAN v3] is a Corosync plugin that monitors the names and number of active
cluster nodes in order to deliver membership and quorum information to
clients (such as the Pacemaker daemons).
In a traditional Corosync-Pacemaker cluster, a Pacemaker plugin is
loaded to provide membership and quorum information. The motivation
for wanting to use CMAN for this instead, is to ensure all elements of
the cluster stack are making decisions based on the same membership
and quorum data.
footnote:[A failure to do this can lead to what is called 'internal
split-brain' - a situation where different parts of the stack disagree
about whether some nodes are alive or dead - which quickly leads to
unnecessary down-time and/or data corruption.]
In the case of GFS2, the key pieces are the dlm_controld and
gfs_controld helpers which act as the glue between the filesystem and
the cluster software. Supporting CMAN enables us to use the versions
already being shipped by most distributions (since CMAN has been
around longer than Pacemaker and is part of the Red Hat cluster
stack).
[WARNING]
=========
Ensure Corosync and Pacemaker are stopped on all nodes before continuing
=========
[WARNING]
=========
Be sure to disable the Pacemaker plugin before continuing with this
section. In most cases, this can be achieved by removing
/etc/corosync/service.d/pcmk and stopping Corosync.
=========
=== Installing the required Software ===
[source,Bash]
-----
# yum install -y cman gfs2-utils gfs2-cluster
Loaded plugins: auto-update-debuginfo
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package cman.x86_64 0:3.1.7-1.fc15 will be installed
--> Processing Dependency: modcluster >= 0.18.1-1 for package: cman-3.1.7-1.fc15.x86_64
--> Processing Dependency: fence-agents >= 3.1.5-1 for package: cman-3.1.7-1.fc15.x86_64
--> Processing Dependency: openais >= 1.1.4-1 for package: cman-3.1.7-1.fc15.x86_64
--> Processing Dependency: ricci >= 0.18.1-1 for package: cman-3.1.7-1.fc15.x86_64
--> Processing Dependency: libSaCkpt.so.3(OPENAIS_CKPT_B.01.01)(64bit) for package: cman-3.1.7-1.fc15.x86_64
--> Processing Dependency: libSaCkpt.so.3()(64bit) for package: cman-3.1.7-1.fc15.x86_64
---> Package gfs2-cluster.x86_64 0:3.1.1-2.fc15 will be installed
---> Package gfs2-utils.x86_64 0:3.1.1-2.fc15 will be installed
--> Running transaction check
---> Package fence-agents.x86_64 0:3.1.5-1.fc15 will be installed
--> Processing Dependency: /usr/bin/virsh for package: fence-agents-3.1.5-1.fc15.x86_64
--> Processing Dependency: net-snmp-utils for package: fence-agents-3.1.5-1.fc15.x86_64
--> Processing Dependency: sg3_utils for package: fence-agents-3.1.5-1.fc15.x86_64
--> Processing Dependency: perl(Net::Telnet) for package: fence-agents-3.1.5-1.fc15.x86_64
--> Processing Dependency: /usr/bin/ipmitool for package: fence-agents-3.1.5-1.fc15.x86_64
--> Processing Dependency: perl-Net-Telnet for package: fence-agents-3.1.5-1.fc15.x86_64
--> Processing Dependency: pexpect for package: fence-agents-3.1.5-1.fc15.x86_64
--> Processing Dependency: pyOpenSSL for package: fence-agents-3.1.5-1.fc15.x86_64
--> Processing Dependency: python-suds for package: fence-agents-3.1.5-1.fc15.x86_64
---> Package modcluster.x86_64 0:0.18.7-1.fc15 will be installed
--> Processing Dependency: oddjob for package: modcluster-0.18.7-1.fc15.x86_64
---> Package openais.x86_64 0:1.1.4-2.fc15 will be installed
---> Package openaislib.x86_64 0:1.1.4-2.fc15 will be installed
---> Package ricci.x86_64 0:0.18.7-1.fc15 will be installed
--> Processing Dependency: parted for package: ricci-0.18.7-1.fc15.x86_64
--> Processing Dependency: nss-tools for package: ricci-0.18.7-1.fc15.x86_64
--> Running transaction check
---> Package ipmitool.x86_64 0:1.8.11-6.fc15 will be installed
---> Package libvirt-client.x86_64 0:0.8.8-7.fc15 will be installed
--> Processing Dependency: libnetcf.so.1(NETCF_1.3.0)(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: cyrus-sasl-md5 for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: gettext for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: nc for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libnuma.so.1(libnuma_1.1)(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libnuma.so.1(libnuma_1.2)(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libnetcf.so.1(NETCF_1.2.0)(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: gnutls-utils for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libnetcf.so.1(NETCF_1.0.0)(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libxenstore.so.3.0()(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libyajl.so.1()(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libnl.so.1()(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libnuma.so.1()(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libaugeas.so.0()(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
--> Processing Dependency: libnetcf.so.1()(64bit) for package: libvirt-client-0.8.8-7.fc15.x86_64
---> Package net-snmp-utils.x86_64 1:5.6.1-7.fc15 will be installed
---> Package nss-tools.x86_64 0:3.12.10-6.fc15 will be installed
---> Package oddjob.x86_64 0:0.31-2.fc15 will be installed
---> Package parted.x86_64 0:2.3-10.fc15 will be installed
---> Package perl-Net-Telnet.noarch 0:3.03-12.fc15 will be installed
---> Package pexpect.noarch 0:2.3-6.fc15 will be installed
---> Package pyOpenSSL.x86_64 0:0.10-3.fc15 will be installed
---> Package python-suds.noarch 0:0.3.9-3.fc15 will be installed
---> Package sg3_utils.x86_64 0:1.29-3.fc15 will be installed
--> Processing Dependency: sg3_utils-libs = 1.29-3.fc15 for package: sg3_utils-1.29-3.fc15.x86_64
--> Processing Dependency: libsgutils2.so.2()(64bit) for package: sg3_utils-1.29-3.fc15.x86_64
--> Running transaction check
---> Package augeas-libs.x86_64 0:0.9.0-1.fc15 will be installed
---> Package cyrus-sasl-md5.x86_64 0:2.1.23-18.fc15 will be installed
---> Package gettext.x86_64 0:0.18.1.1-7.fc15 will be installed
--> Processing Dependency: libgomp.so.1(GOMP_1.0)(64bit) for package: gettext-0.18.1.1-7.fc15.x86_64
--> Processing Dependency: libgettextlib-0.18.1.so()(64bit) for package: gettext-0.18.1.1-7.fc15.x86_64
--> Processing Dependency: libgettextsrc-0.18.1.so()(64bit) for package: gettext-0.18.1.1-7.fc15.x86_64
--> Processing Dependency: libgomp.so.1()(64bit) for package: gettext-0.18.1.1-7.fc15.x86_64
---> Package gnutls-utils.x86_64 0:2.10.5-1.fc15 will be installed
---> Package libnl.x86_64 0:1.1-14.fc15 will be installed
---> Package nc.x86_64 0:1.100-3.fc15 will be installed
--> Processing Dependency: libbsd.so.0(LIBBSD_0.0)(64bit) for package: nc-1.100-3.fc15.x86_64
--> Processing Dependency: libbsd.so.0(LIBBSD_0.2)(64bit) for package: nc-1.100-3.fc15.x86_64
--> Processing Dependency: libbsd.so.0()(64bit) for package: nc-1.100-3.fc15.x86_64
---> Package netcf-libs.x86_64 0:0.1.9-1.fc15 will be installed
---> Package numactl.x86_64 0:2.0.7-1.fc15 will be installed
---> Package sg3_utils-libs.x86_64 0:1.29-3.fc15 will be installed
---> Package xen-libs.x86_64 0:4.1.1-3.fc15 will be installed
--> Processing Dependency: xen-licenses for package: xen-libs-4.1.1-3.fc15.x86_64
---> Package yajl.x86_64 0:1.0.11-1.fc15 will be installed
--> Running transaction check
---> Package gettext-libs.x86_64 0:0.18.1.1-7.fc15 will be installed
---> Package libbsd.x86_64 0:0.2.0-4.fc15 will be installed
---> Package libgomp.x86_64 0:4.6.1-9.fc15 will be installed
---> Package xen-licenses.x86_64 0:4.1.1-3.fc15 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================
Package Arch Version Repository Size
=============================================================================
Installing:
cman x86_64 3.1.7-1.fc15 updates 366 k
gfs2-cluster x86_64 3.1.1-2.fc15 fedora 69 k
gfs2-utils x86_64 3.1.1-2.fc15 fedora 222 k
Installing for dependencies:
augeas-libs x86_64 0.9.0-1.fc15 updates 311 k
cyrus-sasl-md5 x86_64 2.1.23-18.fc15 updates 46 k
fence-agents x86_64 3.1.5-1.fc15 updates 186 k
gettext x86_64 0.18.1.1-7.fc15 fedora 1.0 M
gettext-libs x86_64 0.18.1.1-7.fc15 fedora 610 k
gnutls-utils x86_64 2.10.5-1.fc15 fedora 101 k
ipmitool x86_64 1.8.11-6.fc15 fedora 273 k
libbsd x86_64 0.2.0-4.fc15 fedora 37 k
libgomp x86_64 4.6.1-9.fc15 updates 95 k
libnl x86_64 1.1-14.fc15 fedora 118 k
libvirt-client x86_64 0.8.8-7.fc15 updates 2.4 M
modcluster x86_64 0.18.7-1.fc15 fedora 187 k
nc x86_64 1.100-3.fc15 updates 24 k
net-snmp-utils x86_64 1:5.6.1-7.fc15 fedora 180 k
netcf-libs x86_64 0.1.9-1.fc15 updates 50 k
nss-tools x86_64 3.12.10-6.fc15 updates 723 k
numactl x86_64 2.0.7-1.fc15 updates 54 k
oddjob x86_64 0.31-2.fc15 fedora 61 k
openais x86_64 1.1.4-2.fc15 fedora 190 k
openaislib x86_64 1.1.4-2.fc15 fedora 88 k
parted x86_64 2.3-10.fc15 updates 618 k
perl-Net-Telnet noarch 3.03-12.fc15 fedora 55 k
pexpect noarch 2.3-6.fc15 fedora 141 k
pyOpenSSL x86_64 0.10-3.fc15 fedora 198 k
python-suds noarch 0.3.9-3.fc15 fedora 195 k
ricci x86_64 0.18.7-1.fc15 fedora 584 k
sg3_utils x86_64 1.29-3.fc15 fedora 465 k
sg3_utils-libs x86_64 1.29-3.fc15 fedora 54 k
xen-libs x86_64 4.1.1-3.fc15 updates 310 k
xen-licenses x86_64 4.1.1-3.fc15 updates 64 k
yajl x86_64 1.0.11-1.fc15 fedora 27 k
Transaction Summary
=============================================================================
Install 34 Package(s)
Total download size: 10 M
Installed size: 38 M
Downloading Packages:
(1/34): augeas-libs-0.9.0-1.fc15.x86_64.rpm | 311 kB 00:00
(2/34): cman-3.1.7-1.fc15.x86_64.rpm | 366 kB 00:00
(3/34): cyrus-sasl-md5-2.1.23-18.fc15.x86_64.rpm | 46 kB 00:00
(4/34): fence-agents-3.1.5-1.fc15.x86_64.rpm | 186 kB 00:00
(5/34): gettext-0.18.1.1-7.fc15.x86_64.rpm | 1.0 MB 00:01
(6/34): gettext-libs-0.18.1.1-7.fc15.x86_64.rpm | 610 kB 00:00
(7/34): gfs2-cluster-3.1.1-2.fc15.x86_64.rpm | 69 kB 00:00
(8/34): gfs2-utils-3.1.1-2.fc15.x86_64.rpm | 222 kB 00:00
(9/34): gnutls-utils-2.10.5-1.fc15.x86_64.rpm | 101 kB 00:00
(10/34): ipmitool-1.8.11-6.fc15.x86_64.rpm | 273 kB 00:00
(11/34): libbsd-0.2.0-4.fc15.x86_64.rpm | 37 kB 00:00
(12/34): libgomp-4.6.1-9.fc15.x86_64.rpm | 95 kB 00:00
(13/34): libnl-1.1-14.fc15.x86_64.rpm | 118 kB 00:00
(14/34): libvirt-client-0.8.8-7.fc15.x86_64.rpm | 2.4 MB 00:01
(15/34): modcluster-0.18.7-1.fc15.x86_64.rpm | 187 kB 00:00
(16/34): nc-1.100-3.fc15.x86_64.rpm | 24 kB 00:00
(17/34): net-snmp-utils-5.6.1-7.fc15.x86_64.rpm | 180 kB 00:00
(18/34): netcf-libs-0.1.9-1.fc15.x86_64.rpm | 50 kB 00:00
(19/34): nss-tools-3.12.10-6.fc15.x86_64.rpm | 723 kB 00:00
(20/34): numactl-2.0.7-1.fc15.x86_64.rpm | 54 kB 00:00
(21/34): oddjob-0.31-2.fc15.x86_64.rpm | 61 kB 00:00
(22/34): openais-1.1.4-2.fc15.x86_64.rpm | 190 kB 00:00
(23/34): openaislib-1.1.4-2.fc15.x86_64.rpm | 88 kB 00:00
(24/34): parted-2.3-10.fc15.x86_64.rpm | 618 kB 00:00
(25/34): perl-Net-Telnet-3.03-12.fc15.noarch.rpm | 55 kB 00:00
(26/34): pexpect-2.3-6.fc15.noarch.rpm | 141 kB 00:00
(27/34): pyOpenSSL-0.10-3.fc15.x86_64.rpm | 198 kB 00:00
(28/34): python-suds-0.3.9-3.fc15.noarch.rpm | 195 kB 00:00
(29/34): ricci-0.18.7-1.fc15.x86_64.rpm | 584 kB 00:00
(30/34): sg3_utils-1.29-3.fc15.x86_64.rpm | 465 kB 00:00
(31/34): sg3_utils-libs-1.29-3.fc15.x86_64.rpm | 54 kB 00:00
(32/34): xen-libs-4.1.1-3.fc15.x86_64.rpm | 310 kB 00:00
(33/34): xen-licenses-4.1.1-3.fc15.x86_64.rpm | 64 kB 00:00
(34/34): yajl-1.0.11-1.fc15.x86_64.rpm | 27 kB 00:00
-----------------------------------------------------------------------------
Total 803 kB/s | 10 MB 00:12
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : openais-1.1.4-2.fc15.x86_64 1/34
Installing : openaislib-1.1.4-2.fc15.x86_64 2/34
Installing : libnl-1.1-14.fc15.x86_64 3/34
Installing : augeas-libs-0.9.0-1.fc15.x86_64 4/34
Installing : oddjob-0.31-2.fc15.x86_64 5/34
Installing : modcluster-0.18.7-1.fc15.x86_64 6/34
Installing : netcf-libs-0.1.9-1.fc15.x86_64 7/34
Installing : 1:net-snmp-utils-5.6.1-7.fc15.x86_64 8/34
Installing : sg3_utils-libs-1.29-3.fc15.x86_64 9/34
Installing : sg3_utils-1.29-3.fc15.x86_64 10/34
Installing : libgomp-4.6.1-9.fc15.x86_64 11/34
Installing : gnutls-utils-2.10.5-1.fc15.x86_64 12/34
Installing : pyOpenSSL-0.10-3.fc15.x86_64 13/34
Installing : parted-2.3-10.fc15.x86_64 14/34
Installing : cyrus-sasl-md5-2.1.23-18.fc15.x86_64 15/34
Installing : python-suds-0.3.9-3.fc15.noarch 16/34
Installing : ipmitool-1.8.11-6.fc15.x86_64 17/34
Installing : perl-Net-Telnet-3.03-12.fc15.noarch 18/34
Installing : numactl-2.0.7-1.fc15.x86_64 19/34
Installing : yajl-1.0.11-1.fc15.x86_64 20/34
Installing : gettext-libs-0.18.1.1-7.fc15.x86_64 21/34
Installing : gettext-0.18.1.1-7.fc15.x86_64 22/34
Installing : libbsd-0.2.0-4.fc15.x86_64 23/34
Installing : nc-1.100-3.fc15.x86_64 24/34
Installing : xen-licenses-4.1.1-3.fc15.x86_64 25/34
Installing : xen-libs-4.1.1-3.fc15.x86_64 26/34
Installing : libvirt-client-0.8.8-7.fc15.x86_64 27/34
Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.
Installing : nss-tools-3.12.10-6.fc15.x86_64 28/34
Installing : ricci-0.18.7-1.fc15.x86_64 29/34
Installing : pexpect-2.3-6.fc15.noarch 30/34
Installing : fence-agents-3.1.5-1.fc15.x86_64 31/34
Installing : cman-3.1.7-1.fc15.x86_64 32/34
Installing : gfs2-cluster-3.1.1-2.fc15.x86_64 33/34
Installing : gfs2-utils-3.1.1-2.fc15.x86_64 34/34
Installed:
cman.x86_64 0:3.1.7-1.fc15 gfs2-cluster.x86_64 0:3.1.1-2.fc15
gfs2-utils.x86_64 0:3.1.1-2.fc15
Dependency Installed:
augeas-libs.x86_64 0:0.9.0-1.fc15
cyrus-sasl-md5.x86_64 0:2.1.23-18.fc15
fence-agents.x86_64 0:3.1.5-1.fc15
gettext.x86_64 0:0.18.1.1-7.fc15
gettext-libs.x86_64 0:0.18.1.1-7.fc15
gnutls-utils.x86_64 0:2.10.5-1.fc15
ipmitool.x86_64 0:1.8.11-6.fc15
libbsd.x86_64 0:0.2.0-4.fc15
libgomp.x86_64 0:4.6.1-9.fc15
libnl.x86_64 0:1.1-14.fc15
libvirt-client.x86_64 0:0.8.8-7.fc15
modcluster.x86_64 0:0.18.7-1.fc15
nc.x86_64 0:1.100-3.fc15
net-snmp-utils.x86_64 1:5.6.1-7.fc15
netcf-libs.x86_64 0:0.1.9-1.fc15
nss-tools.x86_64 0:3.12.10-6.fc15
numactl.x86_64 0:2.0.7-1.fc15
oddjob.x86_64 0:0.31-2.fc15
openais.x86_64 0:1.1.4-2.fc15
openaislib.x86_64 0:1.1.4-2.fc15
parted.x86_64 0:2.3-10.fc15
perl-Net-Telnet.noarch 0:3.03-12.fc15
pexpect.noarch 0:2.3-6.fc15
pyOpenSSL.x86_64 0:0.10-3.fc15
python-suds.noarch 0:0.3.9-3.fc15
ricci.x86_64 0:0.18.7-1.fc15
sg3_utils.x86_64 0:1.29-3.fc15
sg3_utils-libs.x86_64 0:1.29-3.fc15
xen-libs.x86_64 0:4.1.1-3.fc15
xen-licenses.x86_64 0:4.1.1-3.fc15
yajl.x86_64 0:1.0.11-1.fc15
Complete!
-----
=== Configuring CMAN ===
The first thing we need to do, is tell CMAN complete starting up even
without quorum. We can do this by changing the quorum timeout setting:
[source,Bash]
-----
# sed -i.sed "s/.*CMAN_QUORUM_TIMEOUT=.*/CMAN_QUORUM_TIMEOUT=0/g" /etc/sysconfig/cman
-----
Next we create a basic configuration file and place it in
/etc/cluster/cluster.conf. The name used for each clusternode should
correspond to that node's uname -n, just as Pacemaker expects. The nodeid
can be any positive mumber but must be unique.
.Basic cluster.conf for a two-node cluster
[source,XML]
-----
-----
=== Configuring CMAN Fencing ===
We configure the fence_pcmk agent (supplied with Pacemaker) to redirect
any fencing requests from CMAN components (such as dlm_controld) to
Pacemaker. Pacemaker's fencing subsystem lets other parts of the stack
know that a node has been successfully fenced, thus avoiding the need for
it to be fenced again when other subsystems notice the node is gone.
[WARNING]
=========
Warning
Configuring real fencing devices in CMAN will result in nodes being
fenced multiple times as different parts of the stack notice the node is
missing or failed.
=========
The definition should be placed in the fencedevices
section and contain:
[source,XML]
-----
-----
Each clusternode must be configured to use this device by adding a
fence method block that lists the node's name as the port.
[source,XML]
-----
-----
Putting everything together, we have:
.cluster.conf for a two-node cluster with fencing
[source,XML]
-----
-----
=== Bringing the Cluster Online with CMAN ===
The first thing to do is check that the configuration is valid
[source,Bash]
-----
# ccs_config_validate
Configuration validates
-----
Now start CMAN
[source,Bash]
-----
# service cman start
Starting cluster:
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Starting gfs_controld... [ OK ]
Unfencing self... [ OK ]
Joining fence domain... [ OK ]
# crm_mon -1
-----
Once you have confirmed that the first node is happily online, start the
second node.
[source,Bash]
-----
[root@pcmk-2 ~]# service cman startStarting cluster:
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Starting gfs_controld... [ OK ]
Unfencing self... [ OK ]
Joining fence domain... [ OK ]
# cman_tool nodes
Node Sts Inc Joined Name
1 M 548 2011-09-28 10:52:21 pcmk-1
2 M 548 2011-09-28 10:52:21 pcmk-2
# crm_mon -1
-----
You should now see both nodes online and services started.
== Create a GFS2 Filesystem ==
[[GFS2_prep]]
=== Preparation ===
Before we do anything to the existing partition, we need to make sure it
is unmounted. We do this by telling the cluster to stop the WebFS resource.
This will ensure that other resources (in our case, Apache) using WebFS
are not only stopped, but stopped in the correct order.
[source,Bash]
-----
# crm_resource --resource WebFS --set-parameter target-role --meta --parameter-value Stopped
# crm_mon
============
Last updated: Thu Sep 3 15:18:06 2009
Stack: openais
Current DC: pcmk-1 - partition with quorum
Version: 1.1.5-bdd89e69ba545404d02445be1f3d72e6a203ba2f
2 Nodes configured, 2 expected votes
6 Resources configured.
============
Online: [ pcmk-1 pcmk-2 ]
Master/Slave Set: WebDataClone
Masters: [ pcmk-1 ]
Slaves: [ pcmk-2 ]
ClusterIP (ocf::heartbeat:IPaddr): Started pcmk-1
-----
[NOTE]
=======
Note that both Apache and WebFS have been stopped.
=======
=== Create and Populate an GFS2 Partition ===
Now that the cluster stack and integration pieces are running smoothly,
we can create an GFS2 partition.
[WARNING]
=========
This will erase all previous content stored on the DRBD device. Ensure
you have a copy of any important data.
=========
We need to specify a number of additional parameters when creating a
GFS2 partition.
First we must use the -p option to specify that we want to use the the
Kernel's DLM. Next we use -j to indicate that it should reserve enough
space for two journals (one per node accessing the filesystem).
Lastly, we use -t to specify the lock table name. The format for this
field is clustername:fsname. For the fsname, we just need to pick
something unique and descriptive and since we haven't specified a
clustername yet, we will use the default (pcmk).
To specify an alternate name for the cluster, locate the service
section containing +name: pacemaker+ in corosync.conf and insert the
following line anywhere inside the block:
.....
clustername: myname
.....
Do this on each node in the cluster and be sure to restart them before
continuing.
[source,Bash]
-----
# mkfs.gfs2 -p lock_dlm -j 2 -t pcmk:web /dev/drbd1
This will destroy any data on /dev/drbd1.
It appears to contain: data
Are you sure you want to proceed? [y/n] y
Device: /dev/drbd1
Blocksize: 4096
Device Size 1.00 GB (131072 blocks)
Filesystem Size: 1.00 GB (131070 blocks)
Journals: 2
Resource Groups: 2
Locking Protocol: "lock_dlm"
Lock Table: "pcmk:web"
UUID: 6B776F46-177B-BAF8-2C2B-292C0E078613
-----
Then (re)populate the new filesystem with data (web pages). For now we'll
create another variation on our home page.
[source,Bash]
-----
# mount /dev/drbd1 /mnt/# cat <<-END >/mnt/index.html
My Test Site - GFS2
END
# umount /dev/drbd1
# drbdadm verify wwwdata#
-----
== Reconfigure the Cluster for GFS2 ==
[source,Bash]
-----
# crm
crm(live) # cib new GFS2
INFO: GFS2 shadow CIB created
crm(GFS2) # configure delete WebFS
crm(GFS2) # configure primitive WebFS ocf:heartbeat:Filesystem params device="/dev/drbd/by-res/wwwdata" directory="/var/www/html" fstype="gfs2"
-----
Now that we've recreated the resource, we also need to recreate all the
constraints that used it. This is because the shell will automatically
remove any constraints that referenced WebFS.
[source,Bash]
-----
crm(GFS2) # configure colocation WebSite-with-WebFS inf: WebSite WebFS
crm(GFS2) # configure colocation fs_on_drbd inf: WebFS WebDataClone:Master
crm(GFS2) # configure order WebFS-after-WebData inf: WebDataClone:promote WebFS:start
crm(GFS2) # configure order WebSite-after-WebFS inf: WebFS WebSite
crm(GFS2) # configure show
node pcmk-1
node pcmk-2
primitive WebData ocf:linbit:drbd \
params drbd_resource="wwwdata" \
op monitor interval="60s"
primitive WebFS ocf:heartbeat:Filesystem \
params device="/dev/drbd/by-res/wwwdata" directory="/var/www/html" fstype="gfs2"
primitive WebSite ocf:heartbeat:apache \
params configfile="/etc/httpd/conf/httpd.conf" \
op monitor interval="1min"
primitive ClusterIP ocf:heartbeat:IPaddr2 \
params ip="192.168.122.101" cidr_netmask="32" \
op monitor interval="30s"
ms WebDataClone WebData \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
colocation WebSite-with-WebFS inf: WebSite WebFS
colocation fs_on_drbd inf: WebFS WebDataClone:Master
colocation website-with-ip inf: WebSite ClusterIP
order WebFS-after-WebData inf: WebDataClone:promote WebFS:start
order WebSite-after-WebFS inf: WebFS WebSite
order apache-after-ip inf: ClusterIP WebSite
property $id="cib-bootstrap-options" \
dc-version="1.1.5-bdd89e69ba545404d02445be1f3d72e6a203ba2f" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
-----
Review the configuration before uploading it to the cluster, quitting the
shell and watching the cluster's response
[source,Bash]
-----
crm(GFS2) # cib commit GFS2
INFO: commited 'GFS2' shadow CIB to the cluster
crm(GFS2) # quit
bye
# crm_mon
============
Last updated: Thu Sep 3 20:49:54 2009
Stack: openais
Current DC: pcmk-2 - partition with quorum
Version: 1.1.5-bdd89e69ba545404d02445be1f3d72e6a203ba2f
2 Nodes configured, 2 expected votes
6 Resources configured.
============
Online: [ pcmk-1 pcmk-2 ]
WebSite (ocf::heartbeat:apache): Started pcmk-2
Master/Slave Set: WebDataClone
Masters: [ pcmk-1 ]
Slaves: [ pcmk-2 ]
ClusterIP (ocf::heartbeat:IPaddr): Started pcmk-2WebFS (ocf::heartbeat:Filesystem): Started pcmk-1
-----
== Reconfigure Pacemaker for Active/Active ==
Almost everything is in place. Recent versions of DRBD are capable of
operating in Primary/Primary mode and the filesystem we're using is
cluster aware. All we need to do now is reconfigure the cluster to take
advantage of this.
This will involve a number of changes, so we'll again use interactive
mode.
# crm
# cib new active
There's no point making the services active on both locations if we can't
reach them, so lets first clone the IP address. Cloned IPaddr2 resources
use an iptables rule to ensure that each request only gets processed by one of
the two clone instances. The additional meta options tell the cluster how
many instances of the clone we want (one "request bucket" for each node)
and that if all other nodes fail, then the remaining node should hold all
of them. Otherwise the requests would be simply discarded.
[source,Bash]
-----
# configure clone WebIP ClusterIP \
meta globally-unique="true" clone-max="2" clone-node-max="2"
-----
Now we must tell the ClusterIP how to decide which requests are
processed by which hosts. To do this we must specify the
clusterip_hash parameter.
Open the ClusterIP resource
[source,Bash]
-----
# configure edit ClusterIP
-----
And add the following to the params line
.....
clusterip_hash="sourceip"
.....
So that the complete definition looks like:
.....
primitive ClusterIP ocf:heartbeat:IPaddr2 \
params ip="192.168.122.101" cidr_netmask="32" clusterip_hash="sourceip" \
op monitor interval="30s"
.....
Here is the full transcript
[source,Bash]
-----
# crm crm(live)
# cib new active
INFO: active shadow CIB created
crm(active) # configure clone WebIP ClusterIP \
meta globally-unique="true" clone-max="2" clone-node-max="2"
crm(active) # configure shownode pcmk-1
node pcmk-2
primitive WebData ocf:linbit:drbd \
params drbd_resource="wwwdata" \
op monitor interval="60s"
primitive WebFS ocf:heartbeat:Filesystem \
params device="/dev/drbd/by-res/wwwdata" directory="/var/www/html" fstype="gfs2"
primitive WebSite ocf:heartbeat:apache \
params configfile="/etc/httpd/conf/httpd.conf" \
op monitor interval="1min"
primitive ClusterIP ocf:heartbeat:IPaddr2 \
params ip="192.168.122.101" cidr_netmask="32" clusterip_hash="sourceip" \
op monitor interval="30s"
ms WebDataClone WebData \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
clone WebIP ClusterIP \
meta globally-unique="true" clone-max="2" clone-node-max="2"
colocation WebSite-with-WebFS inf: WebSite WebFS
colocation fs_on_drbd inf: WebFS WebDataClone:Master
colocation website-with-ip inf: WebSite WebIPorder WebFS-after-WebData inf: WebDataClone:promote WebFS:start
order WebSite-after-WebFS inf: WebFS WebSiteorder apache-after-ip inf: WebIP WebSite
property $id="cib-bootstrap-options" \
dc-version="1.1.5-bdd89e69ba545404d02445be1f3d72e6a203ba2f" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
-----
Notice how any constraints that referenced ClusterIP have been updated
to use WebIP instead. This is an additional benefit of using the crm
shell.
Next we need to convert the filesystem and Apache resources into
clones. Again, the shell will automatically update any relevant
constraints.
[source,Bash]
-----
crm(active) # configure clone WebFSClone WebFS
crm(active) # configure clone WebSiteClone WebSite
-----
The last step is to tell the cluster that it is now allowed to promote
both instances to be Primary (aka. Master).
[source,Bash]
-----
crm(active) # configure edit WebDataClone
-----
Change master-max to 2
[source,Bash]
-----
crm(active) # configure show
node pcmk-1
node pcmk-2
primitive WebData ocf:linbit:drbd \
params drbd_resource="wwwdata" \
op monitor interval="60s"
primitive WebFS ocf:heartbeat:Filesystem \
params device="/dev/drbd/by-res/wwwdata" directory="/var/www/html" fstype="gfs2"
primitive WebSite ocf:heartbeat:apache \
params configfile="/etc/httpd/conf/httpd.conf" \
op monitor interval="1min"
primitive ClusterIP ocf:heartbeat:IPaddr2 \
params ip="192.168.122.101" cidr_netmask="32" clusterip_hash="sourceip" \
op monitor interval="30s"
ms WebDataClone WebData \
meta master-max="2" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
clone WebFSClone WebFSclone WebIP ClusterIP \
meta globally-unique="true" clone-max="2" clone-node-max="2"
clone WebSiteClone WebSitecolocation WebSite-with-WebFS inf: WebSiteClone WebFSClone
colocation fs_on_drbd inf: WebFSClone WebDataClone:Master
colocation website-with-ip inf: WebSiteClone WebIP
order WebFS-after-WebData inf: WebDataClone:promote WebFSClone:start
order WebSite-after-WebFS inf: WebFSClone WebSiteClone
order apache-after-ip inf: WebIP WebSiteClone
property $id="cib-bootstrap-options" \
dc-version="1.1.5-bdd89e69ba545404d02445be1f3d72e6a203ba2f" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
-----
Review the configuration before uploading it to the cluster, quitting the
shell and watching the cluster's response
[source,Bash]
-----
crm(active) # cib commit active
INFO: commited 'active' shadow CIB to the cluster
crm(active) # quit
bye
# crm_mon
============
Last updated: Thu Sep 3 21:37:27 2009
Stack: openais
Current DC: pcmk-2 - partition with quorum
Version: 1.1.5-bdd89e69ba545404d02445be1f3d72e6a203ba2f
2 Nodes configured, 2 expected votes
6 Resources configured.
============
Online: [ pcmk-1 pcmk-2 ]
Master/Slave Set: WebDataClone
Masters: [ pcmk-1 pcmk-2 ]
Clone Set: WebIP Started: [ pcmk-1 pcmk-2 ]
Clone Set: WebFSClone Started: [ pcmk-1 pcmk-2 ]
Clone Set: WebSiteClone Started: [ pcmk-1 pcmk-2 ]
-----
=== Testing Recovery ===
[NOTE]
=======
TODO: Put one node into standby to demonstrate failover
=======