Upgrade global Clusters in a DR Environment
Use this page when the platform has a primary and a standby global cluster connected by Alauda Container Platform Global Cluster Disaster Recovery, for every minor upgrade, and for a patch upgrade in which both global clusters are upgraded before the workload clusters. For a patch upgrade within the same minor version, from ACP 4.5.0 or later, when the standby is upgraded after the workload clusters, use Upgrade the Standby global Cluster After the Workload Clusters instead. Upgrade the standby, then the primary, and reinstall the plugin without delay; do not change clusters on either global cluster until the plugin is reinstalled. Upgrade no workload cluster until both global clusters run the target ACP version (Distribution Version). Synchronization stops from the plugin uninstall until the reinstall, so there is no DR protection in between; see Uninstall the Global Cluster Disaster Recovery Plugin from the Standby. In the plugin settings, the active_* parameters refer to the primary.
Before You Begin
- Prepare for the Upgrade is complete on both
globalclusters. These items are not synchronized between them, so each side has its own: published packages, including the Global Cluster Disaster Recovery package on the standby; Alauda OS image ConfigMaps or catalog entries; the administrator acknowledgement;ResourcePatchexemptions; and a preflight result that lets you continue. On the standby, every check passes exceptDRSplitUpgradeWindow, which may fail only withEtcdSyncNotPaused,TokenRotatorEnabled, or both, and passes after you uninstall the plugin below. For another reason, see DR Standby Checks. - On Bare Metal, the
globalclusters run ACP 4.5.0 or later. To upgrade a Bare Metal pair from an earlier version, contact technical support. - Each
globalcluster has a recent successful etcd backup, as described in Check the etcd Backup. - You are on the execution host, logged in with
ac loginas a platform administrator, in theinstaller/directory of the extracted target Core Package, with a kubeconfig of the standby that points to its VIP:<standby-kubeconfig>.
On the primary, ac commands use the ac login session, and upgrade.sh uses kubectl with the kubeconfig that ac login saved, after ac config use-cluster global; ac whoami --show-server prints https://<platform-address>/kubernetes/global/. Each command block for the standby starts with export KUBECONFIG=<standby-kubeconfig> and ends with unset KUBECONFIG, and ac whoami --show-server prints the standby VIP. Confirm the printed address before you continue with a block.
Upgrade Procedure
Verify Data Consistency
Do not uninstall the plugin until this check passes.
Run the check on a control plane node of the standby global cluster, then return to the execution host:
Check the Machines and Ongoing Upgrades
On the primary:
On the standby:
Uninstall the Global Cluster Disaster Recovery Plugin from the Standby
From now until you reinstall the plugin, do not change clusters on either global cluster: do not create, scale, or delete clusters or nodes, do not change cluster configuration, and do not request any upgrade other than the two global upgrades. A change made on one side can be reconciled back by the providers of the other side. Upgrade the standby, then the primary, and reinstall the plugin without delay.
The plugin settings are removed with the plugin. Record them first, without the token, in a file on the execution host:
Continue when active_cluster_vip shows the primary VIP; otherwise, do not uninstall the plugin, and contact technical support. Keep the file until the reinstall. Then uninstall the plugin:
Continue when the last two commands return nothing; otherwise, run them again after one minute, and if they still list objects, contact technical support.
Upgrade the Standby global Cluster
An upgrade cannot be rolled back. On Huawei DCS, Huawei Cloud Stack, and VMware vSphere, the request also replaces every node with a new VM, one at a time; on Bare Metal, it reinstalls every host, one at a time. This happens on every minor upgrade, and on a patch upgrade when the target changes the Kubernetes build or the Alauda OS image.
On the standby, deploy the Cluster Version Operator (CVO) and the target release information. <target-version> is the target ACP version, for example 4.5.0:
Then run the final preflight check:
Continue when Summary shows Passed or PassedWithWarnings. Otherwise, resolve the checks on the standby as described in Handle Preflight Blocks and run the check again; AdminAckRequired can appear for the first time here, see Handle Administrator Acknowledgement Gates. Then request the upgrade with the target version exactly as ac adm upgrade lists it:
Monitor the upgrade:
Run the block again until Current shows the target version and Ready is True, and on Alauda OS PHASE is Succeeded. While control plane nodes are replaced, a command can fail to connect; run it again. If Upgrade shows Blocked or Retrying, see Inspect Upgrade State. On Alauda OS, if the upgrade stays InProgress in the Kubernetes stage, read the Stalled condition of the InfraClusterVersion as described there. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support.
Check the Standby Before Upgrading the Primary
On the standby:
Continue with the primary when Current shows the target version and Ready is True, on Alauda OS every node is Ready and shows the target Kubernetes version in VERSION, and the last command returns nothing; otherwise, do not upgrade the primary, and contact technical support.
Upgrade the Primary global Cluster
An upgrade cannot be rolled back. On Huawei DCS, Huawei Cloud Stack, and VMware vSphere, the request also replaces every node with a new VM, one at a time; on Bare Metal, it reinstalls every host, one at a time. This happens on every minor upgrade, and on a patch upgrade when the target changes the Kubernetes build or the Alauda OS image.
On the primary, deploy CVO and the target release information:
Then run the final preflight check:
Continue when Summary shows Passed or PassedWithWarnings. Otherwise, resolve the checks on the primary as described in Handle Preflight Blocks and run the check again; AdminAckRequired can appear for the first time here, see Handle Administrator Acknowledgement Gates. Then request the upgrade with the target version exactly as ac adm upgrade lists it:
Monitor the upgrade:
Run the commands again until Current shows the target version and Ready is True, and on Alauda OS PHASE is Succeeded. While control plane nodes are replaced, a command can fail to connect; run it again. If Upgrade shows Blocked or Retrying, see Inspect Upgrade State. On Alauda OS, if the upgrade stays InProgress in the Kubernetes stage, read the Stalled condition of the InfraClusterVersion as described there. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support.
Reinstall the Global Cluster Disaster Recovery Plugin
Reinstall the plugin on the standby after both global clusters run the target ACP version. The standby reads the primary etcd on port 2379, through the primary VIP or, when the VIP does not forward that port, directly from the primary control plane nodes; etcd-sync-settings.txt shows which one the plugin used before. Create the token Secret on the standby from the primary token without printing it. Then list the plugin versions on the standby:
Install the plugin on the standby. Set version to the version that you pushed to the standby in Publish to the Standby global Cluster, the highest VERSION in the previous output. For active_cluster_vip, active_cluster_etcd_endpoints, dest_etcd_endpoints, monitor_check_interval, and detail, use the values recorded in etcd-sync-settings.txt instead of the example values:
Continue when the ModuleInfo shows STATUS Running and CURRENT_VERSION equal to the version you set, every Pod is Running, and the /check command from Verify Data Consistency, run again on a standby control plane node, shows localMissedKeys [0] and localSurplusKeys [0]. Otherwise, run the steps in Global Cluster Disaster Recovery Does Not Synchronize; if they do not resolve it, contact technical support.
Upgrade the Agnostic Extensions and Validate
The upgrade requests do not upgrade Extensions (Operators and Cluster Plugins) with the Agnostic lifecycle. Upgrade each Agnostic Extension installed on the primary, and each one installed on the standby, by following its own documentation; the plugin does not synchronize installed Extensions. For example, see Upgrade Alauda AI and Upgrade Alauda DevOps. Then complete Upgrade Validation on the primary and on the standby.
The upgrade of the two global clusters is complete only when every step on this page is done, including the synchronization check after the reinstall; do not start another upgrade of either global cluster before that.
Next Step
Upgrade the workload clusters. See Upgrade Workload Clusters.