Fail Over to the Standby global Cluster

Use this page when the primary global cluster has failed, or when you switch to the standby global cluster on purpose. In step 2, you uninstall Global Cluster Disaster Recovery (etcd-sync, "the plugin") from the standby global cluster. In step 3, the standby global cluster becomes the new primary global cluster. In step 6, the former primary global cluster becomes the new standby global cluster. For the definitions of primary and standby, see What a DR Pair Is.

While the plugin is uninstalled, there is no DR protection. Step 6 restores it. To make the former primary global cluster primary again later, fail over again; see step 6.

SituationWhat to do
You can reach the primary global clusterFollow steps 1 to 6. Steps 1 to 5 need no other page.
The primary global cluster is unreachableDo step 1. Then, before step 2, contact technical support. Do not uninstall the plugin or switch the platform domain until you have their answer.

Have the following ready:

  • A machine that has ac and can reach the standby global cluster.
  • <standby-kubeconfig>: output/kubeconfig in the installation directory of the standby global cluster.
  • SSH access to a control plane node of the standby global cluster.
  • The platform administrator account.
  • Someone who can change the DNS records of the platform domain and, if you use one registry per side, of the registry domain.
  • The list of workload clusters and Machine objects that you saved from the primary global cluster; see Keep It Healthy.

Through step 3, commands for the standby global cluster start with export KUBECONFIG=<standby-kubeconfig> and end with unset KUBECONFIG. To show which cluster a command talks to, run ac whoami --show-server.

Planned failover only. Before step 1, run this command on both global clusters:

# Primary: ac config use-cluster global
# Standby: export KUBECONFIG=<standby-kubeconfig>, and unset KUBECONFIG afterwards
ac get machines.cluster.x-k8s.io -A

Then, on the primary global cluster, check that no workload cluster upgrade is in progress:

ac config use-cluster global
ac -n cpaas-system get cvsh \
  -o custom-columns='CLUSTER:.metadata.name,REQUESTED:.spec.desiredUpdate.version,CURRENT:.status.current.version'

Continue when all of the following are true:

  • Every machine on both clusters is Running.
  • On every row of the cvsh list, REQUESTED is empty or equal to CURRENT, or the command reports the server doesn't have a resource type.
  • The flows in Global Disaster Recovery are allowed in both directions.

If REQUESTED differs from CURRENT on a row, wait until the upgrade of that workload cluster completes, then run the block again. If the upgrade does not advance, see Inspect Upgrade State. If any other condition is not true, contact technical support.

Procedure

1. Isolate the Primary

Isolate the primary global cluster so that it cannot change your workload clusters:

  • If the primary global cluster has failed, confirm that it is powered off, or that its VIP cannot be reached from the workload networks.
  • For a planned failover, block the VIP of the primary global cluster from the workload networks. Do not stop the control plane of the primary global cluster before step 2 is complete, because the check in step 2 needs it.

If you do not control the firewall, ask your network administrator. Continue when the VIP of the primary global cluster cannot be reached from the workload networks, or when the primary global cluster is powered off. Otherwise, repeat the isolation.

2. Check Consistency and Uninstall the Plugin

Verify consistency before uninstalling Alauda Container Platform Global Cluster Disaster Recovery

If the check reports missing or surplus keys, do not uninstall the plugin. Resolve the inconsistency or contact technical support. Workload nodes are backed by Cluster API Machine objects, and an inconsistent synchronization can delete those objects and destroy the machines behind them.

Applies to: a primary global cluster that you can still reach. If the primary global cluster is unreachable, the check cannot run, and this page cannot confirm that the standby global cluster holds all the data of the primary global cluster. In that case, contact technical support before you uninstall the plugin. Give them the time when the primary global cluster stopped, and the time and result of the last successful check.

On your machine, read the address of the monitor:

export KUBECONFIG=<standby-kubeconfig>
ac -n cpaas-system get svc etcd-sync-monitor -o jsonpath='{.spec.clusterIP}{"\n"}'
unset KUBECONFIG

Run the check on a control plane node of the standby global cluster. Put an IPv6 address in brackets, for example [fd00::10]:

curl -g "http://<monitor-cluster-ip>/check"
ResultAction
Zero missed and zero surplus keys (localMissedKeys [0] and localSurplusKeys [0])Continue.
HTTP 425 (check too frequent)Wait, then run the check again.
Missed keys are listed (localMissedKeys)Wait until the log of the current leader shows Start Sync update, then run the check again. If missed keys remain, restart the leader Pod with the commands below, wait for Start Sync update again, and run the check again.
Surplus keys are listed (localSurplusKeys)Contact technical support.
paused: trueSynchronization is paused for an upgrade. Contact technical support.
The command fails, hangs, or prints anything elseThe primary global cluster may be unreachable. See Applies to above.

To read the log of the leader and, if missed keys remain, restart the leader:

export KUBECONFIG=<standby-kubeconfig>
leader_pod=$(ac -n cpaas-system get lease etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}')
ac -n cpaas-system logs "$leader_pod" | grep "Start Sync update"
ac -n cpaas-system delete pod "$leader_pod"
unset KUBECONFIG

When the check passes, uninstall the plugin from the standby global cluster:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac delete moduleinfo -l cpaas.io/module-name=etcd-sync,cpaas.io/cluster-name=global
ac get moduleinfo -l cpaas.io/module-name=etcd-sync
ac -n cpaas-system get pod -l 'app in (etcd-sync,etcd-sync-monitor)'
unset KUBECONFIG

Continue when the last two commands report No resources found. Otherwise, run them again after one minute. If they still list objects, contact technical support.

Uninstall the plugin, do not just scale it down

Scaling the Deployments to zero is not enough. The ModuleInfo still exists, and at step 6 the new standby global cluster refuses to run with this error:

check failed, error: moduleinfo <name> is found, it shouldn't happen,
  please remove it or check the DNS resolution of platformURL: https://<platform-domain>

Run the ac delete moduleinfo command above on the new primary global cluster, which is the cluster you are switching to. The new standby global cluster then starts normally.

Check that the marker of the plugin is gone:

export KUBECONFIG=<standby-kubeconfig>
ac -n kube-public get configmap etcd-sync
unset KUBECONFIG

Continue when the command reports NotFound. Otherwise, wait and run it again. The providers do not reconcile workload clusters while the marker exists. They resume about one minute after the marker is gone. On Bare Metal, the provider also takes over the shared agent identity at that time. Until it does, new workload hosts cannot obtain their plans.

Do not change clusters until the former primary global cluster is the new standby global cluster

While the plugin is uninstalled, there is no DR protection. Until step 6 is complete, do not create, scale, or delete clusters or nodes on either global cluster, and do not change cluster configuration there. If the former primary global cluster cannot be recovered, contact technical support before you change any cluster.

3. Switch the Platform and Registry Domains

Switch the platform domain to the standby VIP. If you use one registry per side, also switch the registry domain to the registry of the standby global cluster. The standby global cluster is now the new primary global cluster. Check that the platform on the new primary global cluster resolves the domains to the new addresses:

export KUBECONFIG=<standby-kubeconfig>
ac exec -n cpaas-system deployments/sentry -- nslookup <platform-domain>
ac exec -n cpaas-system deployments/sentry -- nslookup <registry-domain>    # only with one registry per side
unset KUBECONFIG

Continue when the platform domain resolves to the standby VIP and the registry domain resolves to the registry of the standby global cluster. Otherwise, correct the DNS record and run the check again.

4. Log In Again and Restart the Dependent Pods

The platform domain now reaches the new primary global cluster. Log in again instead of reusing a session from before the switch. Then restart the following Pods, if they are installed, so that they reload the DR and endpoint configuration:

ac login https://<platform-domain> --name <session-name>
ac config use-cluster global
ac delete po -n cpaas-system -l 'service_name in (alertmanager,vmselect,vminsert)'
ac delete po -n cpaas-system -l service_name=cluster-transformer

If workload clusters send monitoring data to the global cluster, also restart warlock in each workload cluster:

ac config use-cluster <cluster-name>
ac delete po -n cpaas-system -l service_name=warlock
ac config use-cluster global

5. Check the New Primary

On the new primary global cluster, verify that every workload Cluster and Machine object is intact. If a Machine object is lost, its node is deleted.

ac config use-cluster global
ac get clusters.cluster.x-k8s.io -A
ac get machines.cluster.x-k8s.io -A

Continue when every workload cluster and Machine in the list that you saved before the failover is listed, and the provider checks below are met. Otherwise, contact technical support. To list the objects of a provider, run ac get <kind> -A.

ProviderContinue when
Bare MetalEach BaremetalMachine still references the expected MachineInventory, and the status.plan.state of each MachineInventory that a workload BaremetalMachine references (status.machineInventoryRef.name) is still Applied. At least 20 minutes after the DNS change, journalctl -u elemental-system-agent --since "5 min ago" on each workload host shows no errors; a warning about the plan Secret can appear and needs no action. global hosts still use the local VIP, local token, and local plan Secrets.
Huawei DCSThe DCSMachine objects are intact, and no VM was deleted. The workload clusters' credential Secrets exist in cpaas-system.
VMware vSphereThe VSphereMachine, VSphereVM, and VSphereMachineConfigPool objects are intact, and no VM was deleted. The slot and persistent disk status of the pools must have survived the synchronization. The workload clusters' vCenter credential Secrets exist in cpaas-system.
Huawei Cloud StackHCSCluster global reports its own side's VIP as the control plane endpoint, and the platform ELB listeners answer on it. The HCSMachine and HCSMachineConfigPool objects are intact, and no instance was deleted. The persistent disk status of the pools must have survived the synchronization. The workload clusters' HCS credential Secrets exist in cpaas-system.

6. Make the Former Primary the New Standby

Once the former primary global cluster is healthy, make it the new standby global cluster to protect the new primary global cluster again. Keep it isolated from the workload networks until the plugin runs on it. Until the marker of the plugin exists, the providers of a global cluster reconcile workload clusters.

If the former primary global cluster has to be installed again, contact technical support first.

Follow Start Synchronization with these values:

In that sectionAfter the failover
The primary global clusterThe new primary global cluster: the cluster the platform domain points to now
The standby global cluster and <standby-kubeconfig>The new standby global cluster (the former primary global cluster), with its own output/kubeconfig
--platform-address and --dest-repo of violet pushThe VIP and the registry of the new standby global cluster
Active Global Cluster VIPThe VIP of the new primary global cluster
Active Global Cluster ETCD EndpointsThe etcd addresses of the control plane nodes of the new primary global cluster, https://<control-plane-node-ip>:2379, separated by commas. Required with the Self-built VIP. With a load balancer, you can leave it empty if the load balancers of both global clusters forward 2379; see Global Disaster Recovery
The token for etcd-sync-active-cluster-tokenThe cpaas-system/k8sadmin token of the new primary global cluster

Bare Metal only. A failover changes no plan, so step 5 cannot show that the new primary global cluster delivers new plans. To check it after step 6, register a new host with a workload ISO whose URL uses the platform domain, as described in Register Hosts, and add one worker node to a workload cluster on it, as described in Manage Nodes on Bare Metal. The check passes when the new Machine reaches Running. Otherwise, contact technical support.

After step 6, the two sides are synchronized. To make the former primary global cluster primary again once it is healthy, fail over again. Follow this page again with the roles exchanged.

Troubleshooting

SymptomFirst check
The new standby's etcd-sync refuses to runThe warning in step 2.
Workload machines are not reconciled after a failoverWhether kube-public/etcd-sync still exists on the new primary global cluster.
Bare Metal: a workload host gets 401 or 403 right after the failoverWait about one minute after the plugin is gone. Then check the shared token Secret and the Role resourceNames.