Create a Global Cluster Disaster Recovery Pair

This page describes how to create a disaster recovery (DR) pair of global clusters, from planning to the first synchronization. Read it before you write the first install-config.yaml for either side.

SituationGo to
You are planning, or installing the two sidesThe sections below, from the top
You already installed a single global clusterDecide Before You Install
Both sides are installedStart Synchronization
The primary global cluster has failed, or you switch over on purposeFail Over to the Standby global Cluster

What a DR Pair Is

A DR pair consists of two global clusters: a primary global cluster and a standby global cluster. The primary global cluster serves the platform access address, which is a domain name. In normal operation, the domain points to the primary global cluster. The exceptions are while you install the standby global cluster and during a failover. The other cluster is the standby global cluster. After a failover, the former standby global cluster becomes the new primary global cluster.

You install each side as an ordinary installation with ACP CLI (ac), from its own bootstrap node and with its own install-config.yaml. After both sides are installed, Global Cluster Disaster Recovery (etcd-sync) runs on the standby global cluster and copies the etcd data of the primary global cluster to it. ac prepares every global cluster for DR. You do not need to edit any manifests for a DR pair. The infrastructure credentials of each side are not synchronized. The two sides can therefore use different infrastructure accounts.

This page applies to every provider. Steps that only Bare Metal needs are marked Bare Metal only.

Decide Before You Install

Decide on DR before you write the first install-config.yaml for either side. The standby global cluster can decrypt the Secrets of the primary global cluster only with the encryption key of the primary global cluster. An installed cluster keeps the key that it was installed with.

  • If a global cluster is already installed as a single cluster, contact technical support before you install a standby global cluster.
  • Do not install a standby global cluster with its own key. Such a standby global cluster cannot decrypt the Secrets of the primary global cluster.

Prepare for DR

Each side has its own:

  • Bootstrap node.
  • Infrastructure.
  • Control plane VIP.

Both sides must use the same global.registry address and account. Use one of these setups:

  • One external registry that both sides share.
  • One registry per side, behind one domain name.

Workload clusters that use the global registry keep this address and account after a failover. If the values differ between the two sides, those workload clusters cannot use the registry of the new primary global cluster.

Each global cluster keeps its own registry configuration, and synchronization does not copy it. When the registry password changes, update it on both global clusters.

Each side meets the common prerequisites and the prerequisites of its provider.

Also prepare:

  • The platform domain name. It is the platform access address. During normal operation, it resolves only to the primary VIP, and the standby global cluster reaches the platform of the primary global cluster through it. You point it at a VIP three times during the installation and once more at a failover; see Install the Primary and the Standby. Schedule these changes in advance.
  • A user-provided certificate and its private key for the domain, with the same files on both bootstrap nodes.
  • The registry domain name, if you use one registry per side. It is the value of global.registry.address. It resolves to the registry of the side that the platform domain points at. You switch both domains together.
  • The shared keys, generated once; see Generate the Shared Keys.
  • A copy of each side's installation directory, in particular output/kubeconfig. Store it where you can reach it when a site is down.
  • Violet v4.2.13 or later, on a machine that can reach the standby VIP and the standby registry. The etcd-sync package is a component of Global Cluster Disaster Recovery and is shipped in ACP Common Aligned Extensions; see Install etcd-sync.

Generate the Shared Keys

Generate each key once and write the same value in both install-config.yaml files. A key file holds a secret. Run chmod 600 on it and keep it private.

Run the commands on a Linux machine with OpenSSL. To generate the encryption key for Secrets at rest:

openssl rand -base64 32 > dr-encryption.key
chmod 600 dr-encryption.key

Add the key to both install-config.yaml files. Write the key itself on one line, not the file path. To print it, run cat dr-encryption.key.

install-config.yaml (both sides)
cluster:
  encryptionProviderConfig: <dr-encryption-key>

ac uses the value as it is, and checks it before it changes anything. The value must be the base64 encoding of a 16, 24, or 32 byte key, which the command above produces.

Bare Metal only: to generate the key that signs ServiceAccount tokens:

openssl genrsa -out sa.key 2048
base64 -w0 sa.key > sa.key.b64
chmod 600 sa.key sa.key.b64

You cannot change the signing key after installation. If the cluster already holds a different global-sa, ac stops instead of overwriting it.

Network

Before you install either cluster, ask your network administrator to allow the flows in Global Disaster Recovery, in both directions between the networks of the two sides.

The nodes of each side pull their images from the registry in global.registry. For the flows to that registry, see Image Registry.

Observability Placement

DR replicates only the etcd data of the global cluster. Metrics and logs are stored outside etcd, so they are never copied to the standby global cluster. Plan observability before you install either cluster:

  • Each global cluster collects, stores, and queries its own metrics and logs. The standby global cluster can therefore observe itself immediately after a failover.
  • Do not send the monitoring data and logs of workload clusters to either global cluster. Aggregate them in a dedicated workload cluster that runs the VictoriaMetrics Center and the log storage. If a network zone cannot reach that cluster, give the zone its own aggregation cluster.

For what is synchronized and what is excluded, see Global Cluster Disaster Recovery.

Write the Two install-config.yaml Files

Write the install-config.yaml of each side as for a single global cluster. Follow the Create the global Cluster page of your provider: Bare Metal, Huawei DCS, VMware vSphere, or Huawei Cloud Stack. Set these values:

FieldPrimaryStandby
global.platform.addressThe platform domainThe same domain
global.platform.certA user-provided certificate and key whose SANs cover the domainThe same certificate and key files
global.platform.username, global.platform.passwordThe platform administrator accountThe same values
global.platform.extraAddresses[<primary-vip>][<standby-vip>]
cluster.network.loadBalancer.address (provider.hcs.elb.vipAddress on Huawei Cloud Stack)The primary VIPThe standby VIP
global.registry.addressThe shared registry address, or the registry domainThe same value
global.registry.username, global.registry.passwordThe registry accountThe same values
cluster.encryptionProviderConfigThe encryption key from Generate the Shared KeysThe same value
provider.baremetal.serviceAccountSigningKey (Bare Metal only)The content of sa.key.b64The same value
  • Platform address: use a domain name, not an IP address. ac accepts an IP address here, but etcd-sync refuses to run when the platform address is not a domain name.
  • Certificate: do not omit global.platform.cert. DR does not support self-signed certificates. Without the field, each side generates its own self-signed certificate. After a failover, clients then reject the certificate of the other side. On Bare Metal, these clients include the agents of the workload hosts. For the accepted formats, see Common install-config.yaml Fields.
  • extraAddresses: each side lists its own VIP. You use that VIP to reach the side while the domain points at the other side. ac does not add the VIP.
  • Keys: write them before the first manifests or create run on each side. If you change a key afterward, an installed cluster keeps the key that it was installed with; see Rendered Manifests.
  • VM folder (VMware vSphere): when both sides share a vCenter, you can set a different provider.vsphere.folder on each side to keep their VMs apart. The same folder also works.

Applies to: every DR pair, before the first manifests or create run.

Run this command on each bootstrap node, in the installation directory:

v=$(awk '$1=="encryptionProviderConfig:"{gsub(/[\r"\047]/,"",$2);print $2}' install-config.yaml); [ -n "$v" ] && printf '%s' "$v" | sha256sum

On the machine where you keep the shared keys, also run the following command. It drops the trailing newline of the file and prints the hash of the key. Compare the hashes of both sides with this one:

tr -d '\r\n' < dr-encryption.key | sha256sum

Continue when both sides print the same hash as the key file. On a side that prints nothing or another hash, set cluster.encryptionProviderConfig in its install-config.yaml to the content of dr-encryption.key.

Bare Metal only. Also run this command on each bootstrap node, in the installation directory:

v=$(awk '$1=="serviceAccountSigningKey:"{gsub(/[\r"\047]/,"",$2);print $2}' install-config.yaml); [ -n "$v" ] && printf '%s' "$v" | sha256sum

On the machine where you keep the shared keys, also run the following command:

tr -d '\r\n' < sa.key.b64 | sha256sum

Continue when both sides print the same hash as the key file. On a side that prints nothing or another hash, set provider.baremetal.serviceAccountSigningKey in its install-config.yaml to the content of sa.key.b64.

Install the Primary and the Standby

While you install a side, the platform domain must point to that side. To reach the other side meanwhile, use its VIP and the kubeconfig in its installation directory.

StepWhat you do
1Point the domain at the primary VIP. On the primary bootstrap node, install the cluster as described on the Create the global Cluster page of your provider.
2When the primary global cluster is installed, point the domain at the standby VIP. On the standby bootstrap node, install the standby global cluster the same way.
3When the standby global cluster is installed, point the domain back at the primary VIP.

If you use one registry per side, point the registry domain at the same side as the platform domain in each step.

When both sides are installed and the domain points at the primary global cluster, continue with Start Synchronization.

Start Synchronization

After both sides are installed, install Global Cluster Disaster Recovery (etcd-sync) on the standby global cluster and verify the synchronization. The plugin copies the etcd data of the primary global cluster to the standby global cluster.

Before you start:

  • The platform domain points at the primary VIP.
  • Both sides were installed with the same cluster.encryptionProviderConfig; see Generate the Shared Keys. If they were not, contact technical support before you continue.
  • Applies to workload clusters on infrastructure other than that of the global cluster. Install the provider of that infrastructure on both global clusters, so that the standby global cluster can reconcile the same workload clusters after a failover. See Prepare. Upload the provider to the standby global cluster with --platform-address and --dest-repo set to the standby global cluster, as in step 1. If no workload cluster runs on other infrastructure, skip this item. ac has already installed the provider of its own platform on each side.

Conventions

The following steps run commands against both global clusters from one machine. Use a machine that has ac and can reach the VIPs of both sides, and copy the kubeconfig of each side to it. The kubeconfig of a side is output/kubeconfig in its installation directory and connects to the VIP of that side.

  • This page calls the two files <primary-kubeconfig> and <standby-kubeconfig>. Each command block starts with export KUBECONFIG=<primary-kubeconfig> or export KUBECONFIG=<standby-kubeconfig> and ends with unset KUBECONFIG.
  • To show which cluster a command talks to, run ac whoami --show-server.
  • In the parameter names of the plugin, Active means the primary global cluster.

The step marked Bare Metal only is not numbered.

1. Install etcd-sync

Install Global Cluster Disaster Recovery (etcd-sync) only on the standby global cluster, after both sides have completed create. Never install it on both sides. Use the version released with 4.5.

Reach the standby global cluster through its VIP.

  1. Get the plugin package from ACP Common Aligned Extensions and upload it to the standby global cluster with Violet v4.2.13 or later.

    • Violet signs in to the standby global cluster with an API token. Create the token in the Web Console of the standby global cluster (https://<standby-vip>): Profile → API Tokens.
    • Set both --platform-address and --dest-repo to the standby global cluster. Otherwise, the package goes to the primary global cluster.
    • --dest-repo is the address of the registry that the standby global cluster uses. With one shared registry, it is global.registry.address. With one registry per side, the registry domain now resolves to the primary registry, so use an address that reaches the standby registry instead.
    • If the standby registry needs no credentials, use --no-auth instead of the registry credentials.

    For the flags, see Upload Packages.

    read -rsp "Standby platform token: " STANDBY_TOKEN; printf '\n'
    read -rsp "Standby registry password: " STANDBY_REGISTRY_PASSWORD; printf '\n'
    violet push <etcd-sync-package>.tgz \
      --platform-address "https://<standby-vip>" --platform-token "${STANDBY_TOKEN}" \
      --dest-repo "<standby-registry-address>" --username "<registry-username>" --password "${STANDBY_REGISTRY_PASSWORD}"
    unset STANDBY_TOKEN STANDBY_REGISTRY_PASSWORD
  2. Create the token Secret on the standby global cluster. The plugin uses this token to read the primary global cluster; you do not log in with it. Its value is the bearer token of the cpaas-system/k8sadmin ServiceAccount on the primary global cluster. The commands do not print the token:

    export KUBECONFIG=<primary-kubeconfig>
    ac whoami --show-server   # the primary VIP
    PRIMARY_TOKEN=$(ac -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}' | base64 -d)
    export KUBECONFIG=<standby-kubeconfig>
    ac whoami --show-server   # the standby VIP
    printf '%s' "${PRIMARY_TOKEN}" | ac -n cpaas-system create secret generic etcd-sync-active-cluster-token \
      --from-file=token=/dev/stdin --dry-run=client -o yaml | ac apply -f -
    unset PRIMARY_TOKEN
    unset KUBECONFIG
  3. Open the Web Console of the standby global cluster at https://<standby-vip>, switch to the Administrator view, go to Marketplace → Cluster Plugins, select the global cluster, and install Global Cluster Disaster Recovery with these parameters.

    ParameterValue
    Active Global Cluster VIPControl plane VIP of the primary global cluster. Required
    Active Global Cluster ETCD EndpointsThe etcd addresses of the control plane nodes of the primary global cluster, https://<primary-control-plane-node-ip>:2379; separate several addresses with commas. Required when the global clusters use the Self-built VIP. With a load balancer, you can leave it empty if the load balancers of both global clusters forward 2379; see Global Disaster Recovery
    Standby Cluster ETCD EndpointsDefault value
    Active Global Cluster Token Secretetcd-sync-active-cluster-token
    Data Check Interval (unit: hour)Default value
    Print detail logsDisabled, unless you are troubleshooting

2. Start and Verify Synchronization

First check that the plugin is running on the standby global cluster:

export KUBECONFIG=<standby-kubeconfig>
ac get moduleinfo -l cpaas.io/module-name=etcd-sync
ac -n cpaas-system get pod -l 'app in (etcd-sync,etcd-sync-monitor)'
unset KUBECONFIG

Continue when the ModuleInfo shows STATUS Running and every Pod is Running. Otherwise, wait and run the commands again. If the results do not change, run the steps in Global Cluster Disaster Recovery Does Not Synchronize.

The mirror copies the data of the primary global cluster after its start delay. Wait until the initial synchronization has finished:

export KUBECONFIG=<standby-kubeconfig>
leader_pod=$(ac -n cpaas-system get lease etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}')
ac -n cpaas-system logs "$leader_pod" | grep "Start Sync update"
unset KUBECONFIG

Continue when the output contains Start Sync update. Otherwise, wait and run the commands again.

Then check the synchronization. On your machine, read the address of the monitor:

export KUBECONFIG=<standby-kubeconfig>
ac -n cpaas-system get svc etcd-sync-monitor -o jsonpath='{.spec.clusterIP}{"\n"}'
unset KUBECONFIG

Run the check on a control plane node of the standby global cluster. Put an IPv6 address in brackets, for example [fd00::10]:

curl -g "http://<monitor-cluster-ip>/check"
ResultAction
Zero missed and zero surplus keys (localMissedKeys [0] and localSurplusKeys [0])Continue.
HTTP 425 (check too frequent)Wait, then run the check again.
Missed keys are listed (localMissedKeys)Wait until the log of the current leader shows Start Sync update, then run the check again. If missed keys remain, restart the leader Pod with the commands below, wait for Start Sync update again, and run the check again.
Surplus keys are listed (localSurplusKeys)Contact technical support.
paused: trueSynchronization is paused for an upgrade. Contact technical support.
The command fails or prints anything elseRun the steps in Global Cluster Disaster Recovery Does Not Synchronize, then run the check again.

To restart the leader Pod, run the following commands. They read the name of the current leader Pod from the etcd-sync-mirror Lease on the standby global cluster and delete that Pod. Run them only when the table above tells you to:

export KUBECONFIG=<standby-kubeconfig>
leader_pod=$(ac -n cpaas-system get lease etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}')
ac -n cpaas-system delete pod "$leader_pod"
unset KUBECONFIG

Bare Metal Only: Check the Shared Agent Identity

Check the shared workload agent identity on each side:

# Run once with export KUBECONFIG=<primary-kubeconfig> and once with
# export KUBECONFIG=<standby-kubeconfig>; unset KUBECONFIG afterwards
ac -n cpaas-system get sa/baremetal-system-agent secret/baremetal-system-agent-token \
  role/baremetal-system-agent rolebinding/baremetal-system-agent

Continue when all four objects exist on both sides, or when none of them exists on either side. Otherwise, contact technical support.

3. Install the Cluster Plugins

Installing a global cluster does not install every cluster plugin, for example Advanced Monitoring. Install the cluster plugins that you need only now, on both global clusters, after both clusters are installed and synchronization runs.

Keep It Healthy

  • Run the check in step 2 regularly. The plugin also runs the check by itself, every hour by default, and reports the results as the metrics local_missed_keys_count and local_surplus_keys_count.
  • After each change to your workload clusters, save the output of ac get clusters.cluster.x-k8s.io -A and ac get machines.cluster.x-k8s.io -A from the primary global cluster. After a failover, you compare the new primary global cluster against this output; see Fail Over.
  • A DR-aware upgrade of the global clusters uninstalls the plugin and has its own consistency check; see Upgrade global Clusters in a DR Environment.

Bare Metal: Agents and the Platform Address

Bare Metal only. The agent of a global host talks only to the API server of its own global cluster, through the VIP of that cluster. The agent of a workload cluster host talks to the platform access address, not to the VIP of its cluster. After a failover, it therefore reaches the new primary global cluster without any change.

To add or replace a global host after the installation, register it on the global cluster that it belongs to. Follow Registering Hosts for the global Cluster, which describes the naming rule for the registration and the annotations that point the agent of the host at the local API server.

Next