How to Sync Cluster Configuration and Verify Failover Setup in Nodes – Part 4

Hello folks. First of all, my apologies for the delay of the last part of this cluster series. Let’s get on to work without getting any more delayed.

As we many of you have completed all three previous parts, I will brief you what we have completed so far. Now we already have enough knowledge to install and configure cluster packages for two nodes and enable fencing and failover in a clustered environment.

Sync Cluster Configuration and Verify FailOver

Sync Cluster Configuration and Verify FailOver – Part 4

You can refer my previous parts if you don’t remember since it took a little longer to post the last part.

Introduction to Linux Clustering and Advantages/Disadvanges of Clustering – Part 1

How to Install and Configure Cluster with Two Nodes in Linux – Part 2

Fencing and Adding a Failover to Clustering – Part 3

We will start by adding resources to the cluster. In this case we can add a file system or a web service as your need. Now I have /dev/sda3 partition mounted to /x01 which I wish to add as a file system resource.

1. I use below command to add a file system as a resource:

# ccs -h 172.16.1.250 --addresource fs name=my_fs device=/dev/mapper/tecminttest_lv_vol01 mountpoint=/x01 fstype=ext3
Add Filesystem to Cluster

Add Filesystem to Cluster

Additionally, if you want to add a service also, you can by using below methodology. Issue the following command.

# ccs -h 172.16.1.250 --addservice my_web domain=testdomain recovery=relocate autostart=1

You can verify it by viewing the cluster.conf file as we did in previous lessons.

2. Now enter following entry in cluster.conf file to add a reference tag to the service.

<fs ref="my_fs"/>
Add Service to Cluster

Add Service to Cluster

3. All set. No we will see how we can sync the configurations we made to cluster among the 2 nodes we have. Following command will do the needful.

# ccs -h 172.16.1.250 --sync --activate
Sync Cluster Configuration

Sync Cluster Configuration

Note: Enter passwords we set for ricci in the early stages when we were installing packages.

You can verify your configurations by using below command.

# ccs -h 172.16.1.250 --checkconf
Verify Cluster Configuration

Verify Cluster Configuration

4. Now it’s time to start the things up. You can use one of below commands as you prefer.

To start only one node use the command with relevant IP.

# ccs -h 172.16.1.222 start

Or if you want to start all nodes use --startall option as follows.

# ccs -h 172.16.1.250 –startall

You can use stop or --stopall if you needed to stop the cluster.

In a scenario like if you wanted to start the cluster without enabling the resources (resources will automatically be enabled when the cluster is started), like a situation where you have intentionally disabled the resources in a particular node in order to disable fencing loops, you don’t want to enable those resources when the cluster is starting.

For that purpose you can use below command which starts the cluster but does not enable the resources.

# ccs -h 172.16.1.250 --startall --noenable 

5. After the cluster has been started up, you can view the stats by issuing clustat command.

# clustat
Check Cluster Status

Check Cluster Status

Above output says there are two nodes in the cluster and both are up and running at the moment.

6. You can remember we have added a failover mechanism in our previous lessons. Want to check it works? This is how you do it. Force shutdown one node and look for cluster stats using clustat command for the results of failover.

I have shut down my node02server(172.16.1.223) using shutdown -h now command. Then executed clustat command from my cluster_server(172.16.1.250).

Check Cluster FailOver

Check Cluster FailOver

Above output clarifies you that node 1 is online while node 2 has gone offline as we shut it down. Yet service and the file system we shared are still online as you can see if you check it on node01 which is online.

# df -h /x01
Verify Cluster Node

Verify Cluster Node

Refer the cluster.conf file with whole config set relevant to our setup used for tecmint.

<?xml version="1.0"?>
<cluster config_version="15" name="tecmint_cluster">
        <fence_daemon post_join_delay="10"/>
        <clusternodes>
                <clusternode name="172.16.1.222" nodeid="1">
                        <fence>
                                <method name="Method01">
                                        <device name="tecmintfence"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="172.16.1.223" nodeid="2">
                        <fence>
                                <method name="Method01">
                                        <device name="tecmintfence"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman/>
        <fencedevices>
                <fencedevice agent="fence_virt" name="tecmintfence"/>
        </fencedevices>
        <rm>
                <failoverdomains>
                        <failoverdomain name="tecmintfod" nofailback="0" ordered="1" restricted="0">
                                <failoverdomainnode name="172.16.1.222" priority="1"/>
                                <failoverdomainnode name="172.16.1.223" priority="2"/>
                        </failoverdomain>
                </failoverdomains>
                <resources>
                        <fs device="/dev/mapper/tecminttest_lv_vol01" fstype="ext3" mountpoint="/x01" name="my_fs"/>
                </resources>
                <service autostart="1" domain="testdomain" name="my_web" recovery="relocate"/>
                <fs ref="my_fs"/>
       </rm>
</cluster>

Hope you’ll enjoyed the whole series of clustering lessons. Keep in touch with tecmint for more handy guides everyday and feel free to comment your ideas and queries.

Best Affordable Linux and WordPress Services For Your Business
Outsource Your Linux and WordPress Project and Get it Promptly Completed Remotely and Delivered Online.

If You Appreciate What We Do Here On TecMint, You Should Consider:

  1. Stay Connected to: Twitter | Facebook | Google Plus
  2. Subscribe to our email updates: Sign Up Now
  3. Get your own self-hosted blog with a Free Domain at ($3.45/month).
  4. Become a Supporter - Make a contribution via PayPal
  5. Support us by purchasing our premium books in PDF format.
  6. Support us by taking our online Linux courses

We are thankful for your never ending support.

Thilina Uvindasiri

I'm a BSc Special (Hons) graduate in Information Technology and works as an Engineer-Systems. Love to work, explore and research Linux and play rugby as a hobby.

Your name can also be listed here. Got a tip? Submit it here to become an TecMint author.

RedHat RHCE and RHCSA Certification Book
Linux Foundation LFCS and LFCE Certification Preparation Guide

You may also like...

55 Responses

  1. vinay says:

    I have configured all, but when i try to sync & proceed with next steps i get the below error for all commands

    Cluster.conf file specified is not a valid cluster.conf file (use -i to ignore this error)

    could anyone please help?

    thanks in advance.

    Vinay

  2. vinay says:

    After i added “ccs -f cluster.conf –addresource fs name=my_fs device=/dev/mapper/vg1-data_lv mountpoint=/x01 fstype=ext4” I don’t see it added in the cluster.conf file
    also when i run sync i get below error
    [[email protected] cluster]# ccs -h 192.168.0.102 –sync –activate
    Cluster.conf file specified is not a valid cluster.conf file (use -i to ignore this error)

  3. Suraj says:

    Hi,

    First of all, I would like to say thanks for providing this wonderful post.

    But I got an error message when I was trying to get the status of cluster.

    ***************************************
    [[email protected] mapper]# clustat
    -bash: clustat: command not found
    ****************************************

    • Thilina Uvindasiri says:

      Hi Suraj,

      This simply says that clustat command package is not installed properly. Try installing it using yum and try again. ;)

  4. Rahul Pal says:

    I am getting a following error

    service cman restart
    Stopping cluster:
    Leaving fence domain… [ OK ]
    Stopping gfs_controld… [ OK ]
    Stopping dlm_controld… [ OK ]
    Stopping fenced… [ OK ]
    Stopping cman… [ OK ]
    Unloading kernel modules… [ OK ]
    Unmounting configfs… [ OK ]
    Starting cluster:
    Checking if cluster has been disabled at boot… [ OK ]
    Checking Network Manager… [ OK ]
    Global setup… [ OK ]
    Loading kernel modules… [ OK ]
    Mounting configfs… [ OK ]
    Starting cman… Cannot find node name in cluster.conf
    Unable to get the configuration
    Cannot find node name in cluster.conf
    cman_tool: corosync daemon didn’t start Check cluster logs for details
    [FAILED]
    Stopping cluster:
    Leaving fence domain… [ OK ]
    Stopping gfs_controld… [ OK ]
    Stopping dlm_controld… [ OK ]
    Stopping fenced… [ OK ]
    Stopping cman… [ OK ]
    Unloading kernel modules… [ OK ]
    Unmounting configfs… [ OK ]

  5. Shail says:

    Hi, this is not working in my case, I am getting error
    ccs -h hostip –checkconf
    “Cluster.conf file specified is not a valid cluster.conf file (use -i to ignore this error)”

    service cman start
    Starting cman… Cannot find node name in cluster.conf
    Unable to get the configuration
    Cannot find node name in cluster.conf
    cman_tool: corosync daemon didn’t start Check cluster logs for details

    my file is exact replica of your cluster.conf file , please help

    • Thilina Uvindasiri says:

      Shail,

      Did you use IPs or hostnames in the conf? If used hostnames, can u check whether they are resolving correctly?

  6. Rahul says:

    Does it sync each and every service eg httpd, mysqld etc running on node01 into node02 ??

  7. Manikandaprabu S says:

    ccs -h 10.44.2.89 –addresource fs name=my_fs device=/dev/mapper/vgcluster-data mountpoint=/ex01 fstype=ext4
    Validation Failure, unable to modify configuration file (use -i to ignore this error).

    i’m getting above error . pls help selinux and iptables are disbled

  8. anil says:

    Wonderfully article is it possible to implement in ubuntu server 14.04 if yes please let me know the link

    • Thilina Uvindasiri says:

      Hi anil,

      Actually I didn’t use this in an Ubuntu environment. Theoretically it should work though. I’ll update you if I had a chace to try.

  9. shamal says:

    ccs -h 10.44.2.89 –addresource fs name=my_fs device=/dev/mapper/vgcluster-data mountpoint=/ex01 fstype=ext4
    Validation Failure, unable to modify configuration file (use -i to ignore this error).

    i’m getting above error . pls help

  10. sundar says:

    Hi, It is a useful post. how the ip address will sync if one node down, if i have a web server.

    • Thilina Uvindasiri says:

      Hi Sundar,

      That’ why we have setup a failover in the begining in case of one node becoming inactive. And since the webservice is synced across the both nodes, webservice won’t be down even if one node becomes down.

Got something to say? Join the discussion.

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.