How to recover with storage replication

It might happen that your single LXD server or all members of a cluster go offline or become unreachable. In such a disaster it is helpful to have another deployment of LXD running in a different location which can take over running the already existing workload.

If such an incident happens, we want to rely on the storage layer that all the instances and custom volumes got replicated to our backup location. We can then consolidate the storage layer and perform recovery of the resources to make them available to our LXD deployment in the backup location (see How to recover instances in case of disaster). This requires not only two individual deployments of LXD, but also the configuration of storage replication for the respective storage array.

Note

Recovery with storage replication is only possible with remote Storage drivers which support volume recovery (see Feature comparison). Configuring the replication on the storage array is out of scope for LXD and highly dependent on how replication is materialized by each of the vendors.

This how-to guide mainly covers the steps performed on LXD and mentions the necessary requirements on the storage array where applicable.

In this how-to we assume a primary and secondary LXD deployment. On both the primary and secondary location there exists an individual storage array. The LXD on the primary location gets configured to only interact with the primary storage array. The LXD on the secondary location gets configured to only interact with the secondary storage array. Both LXD deployments are operated independently of each other.

Ensure entities on both locations

Storage pool

Before we setup any replication on either of the storage arrays, we have to ensure that both the LXD on the primary and secondary location have a storage pool on the underlying storage array which can be leveraged later for replication.

If you don’t already have the respective storage pool created on any of the locations, see Create a storage pool.

Networks and profiles

There could be other entities like networks and profiles which you might also want to setup upfront on the secondary location so that in case of disaster you can concentrate on recovering the volumes.

When performing How to recover instances in case of disaster after a disaster, LXD will check if the required entities are present and notify in case something is missing. The recovery won’t create these entities.

Setup replication on the storage array

The replication has to be setup outside of LXD based on the specific concepts and constructs applied by the storage vendor. In the following list you can find steps published by some of the storage vendors:

After configuring the connection between both the primary and secondary storage array, follow the steps outlined by the storage vendor on how to setup the actual replication of volumes. Some vendors like Dell use a concept called replication consistency group (RCG) which allows replicating a group of volumes consistently. Such a RCG can contain an instances volume together with all of its attached custom volumes. Other vendors might use similar or different concepts.

Known storage array limitations

Consider the following limitations by storage array when setting up replication:

PowerFlex

Cannot replicate and recover volumes with snapshots

In PowerFlex a volume’s snapshot appears as its own volume but is still logically connected to its parent volume (vTree). When replicating a volume inside a RCG its snapshots won’t get replicated which causes inconsistencies on the secondary location. A volume’s snapshot can be replicated but will be placed inside a new vTree loosing the logical relation to its parent volume. During recovery LXD notices this inconsistency and raises an error.

Ceph RBD

Cannot use journaling mode

Mirroring can be configured using either journaling or snapshot mode. Only the latter is supported together with LXD as the volumes need to be mapped to the host for read access during recovery which might not be possible due to missing kernel features.

Verify replication

After configuring the replication check whether the volumes are successfully replicated to the secondary location.

Important

For a potential recovery it’s essential that the replication is running consistently. It’s recommended to check this regularly. As soon as the primary location experiences an outage you might not be able anymore to recover data which wasn’t yet replicated to the secondary location.

Promote secondary location after disaster

In case the primary location went down the secondary location can be promoted to the new source of truth. However it depends on the storage vendor how to promote the secondary storage array. See the guides linked in Setup replication on the storage array for the necessary steps based on the used storage array.

Important

Promoting the secondary array might cause data loss in case there is non-replicated data left on the primary location. Consult the storage vendors documentation for further information.

Recover resources

Once the secondary storage array is promoted, we can start recovering the workload. For this run the steps outlined in How to recover instances in case of disaster on the secondary LXD deployment. When being asked which pools should be scanned for unknown volumes, make sure to select the storage pool which was configured during the replication setup.

The instances and custom storage volumes are now present on the secondary LXD. Using lxc start you can now bring backup up instances which were originally running on the primary LXD.

Add missing pool

In case the LXD pool on the secondary site was only created in storage but not in LXD as outlined in Storage pool, you have to recover it first.

Use the lxc storage create command to add the storage pool. This works for both single and clustered LXD deployments.

Reuse Ceph RBD pool

LXD’s Ceph RBD driver uses a so called placeholder volume to occupy the storage pool and make sure that the same storage pool isn’t used more than once. In case of replication this can be ignored as we have to reuse the replicated pool at the secondary location. Therefore we need to set ceph.osd.force_reuse to allow ignoring the placeholder volume in case it also got replicated to the secondary location.

When creating the storage pool in a LXD cluster make sure to add the ceph.osd.force_reuse=true setting when running the final storage pool creation command. The setting doesn’t need to be provided when preparing the storage pool on each of the cluster members.

Demote secondary and return to primary location

Once the primary location is back online, the storage layer ensures there won’t be any data inconsistency as the secondary storage array now acts as the source of truth and doesn’t anymore receive updates from the primary storage array. As long as this doesn’t get changed, the running instances (and custom volumes) on the secondary location are protected.

Note

There is still the possibility for collisions in the network in case the primary location comes back up and LXD starts up any instances. This is out of scope for storage replication but proper measures have to be put into place to prevent conflicts.

Switching back service to the primary location can be performed in two ways. For both of them the operations on the storage layer are identical but choosing the right option depends on what has happened to the instances and custom volumes on the secondary location:

  1. Shutdown the resources on the secondary location and bring them back up on the primary This requires no modifications have been done to the configuration of both the instances and volumes on the secondary location after they got recovered. Those changes aren’t known to the database of the primary LXD and might cause side effects.

  2. Setup a fresh deployment of LXD on the primary location and redo the steps outlined in How to recover instances in case of disaster. This repeats the same steps performed for the actual disaster recovery but in this case the other way around.

After deciding for an approach, demote the storage array on the secondary location and promote the primary location. Refer to Setup replication on the storage array how this can be performed for each of the storage arrays. In a final step either bring up the instances using lxc start or recover them first and make them again known to the primary LXD before starting them up again.