As Theo’s post implied, setting up drbd between two servers is only “part 1” on the way to high availability (which is surely the only reason you’d be replicating filestore with drbd).

You probably need a “part 2” (apologies if you’ve already done this), where you explain how to make the storage highly available on your network using a floating virtual IP. As Theo said, this is typically done with corosync/pacemaker, but it’s non-trivial.

The aim is to be able to shut down the primary server and have the secondary server take over the primary role (including the virtual IP) within seconds of detecting the original primary server leaving the cluster. I know from experience that Linux VMs (via iSCSI to clustered SANs) will go read-only on their filesystems if they lose their filestore on the network for 25 seconds or more, requiring them to be restarted and fsck’ed afterward.

Note that bringing back the original shutdown server into the cluster should only make it a secondary that syncs any changes during its downtime from the new primary server. There are commands you can run after that sync is completed to switch the server to become the primary node again, but I’d do that manually myself.

You may also need a “part 3” where fencing of nodes is considered – this is mainly to avoid “split-brain” where a primary node that drops out of the cluster still thinks it’s the primary and tries to write to the filestore when it shouldn’t. The STONITH method Theo mentioned involves forcibly powering off the faulty primary node immediately before another node takes over its primary duties.

It is solved, it turned out that the mysqld was enabled, so it was started twice.

The second issue was solved by increasing the time outs.

We have configured the corosync/pacemaker cluster with 3 LV’s and 3 resources (our own application platform, MySQL and Nginx) and a virtual IP. Besides, we set up STONITH fencing; when the network connection dies, or a server runs in problems that server is powered down.

When we test the switching of the cluster from node 1 to node 2, we notice that MySQL has a tendency to always try to start on the node that is not active, and after a while stops when it doesn’t succeed. It s only that resource, the other don’t try it, and after it it does start on the active node. Did we do something with stickiness, or a preferred host for this MySQL resource?

Secondly, when a server powers down and the cluster is healthy and running on the active server, we boot up the second server. The first, active server remains active, but when the inactive server is up and the cluster status shows that both servers are back in the cluster config, the resources on the active host are stopped and restarted anyway.

Is this because the drbd setup now includes 2 nodes again and has to restart the resources? We would prefer the resources remain running when the second node is included but not promoted to active, as basically that should have no influence on the already active host.

I hope someone had this before and knows the magic trick ;)

Best regards, Theo

Aaron, it seems the drbd creates a partition over the existing partition. The existing partition cannot be formatted or used in any manner since it will be overwritten in favor of the drbd partition.

Then you format the drbd partition. From what I have researched, that is the only way it works. You cannot create a partition over an existing, in use partition. If I am wrong, I would love to be corrected if there is a way.

Please keep me posted.

@Eric

Great! An existing partition can work, if well configured as shown in the article. We simply considered a fresh setup for beginners. Thanks for the feedback.