Upgrade Overview

An upgrade moves each cluster to a new ACP version (Distribution Version), with one upgrade request per cluster: the global cluster first, then the workload clusters. You submit each request with ACP CLI (ac adm upgrade), and the Cluster Version Operator (CVO) on the global cluster carries it out. For the terms used in the upgrade pages, see Glossary.

Upgrade at a Glance

  1. Complete Prepare for the Upgrade before you upgrade the first cluster. Checks at a Glance lists the checks that apply to each cluster.
  2. Only for Huawei DCS and Huawei Cloud Stack clusters that keep platform data on machine template disks: Migrate to Pool-Managed Persistent Disks before the upgrade.
  3. Upgrade global: Upgrade the global Cluster, or in a DR environment, Upgrade global Clusters in a DR Environment or Upgrade the Standby global Cluster After the Workload Clusters, as chosen in Upgrade Order and Compatibility.
  4. Upgrade the workload clusters, one or more at a time: Upgrade Workload Clusters.
  5. Validate each cluster after its upgrade: Upgrade Validation.
  6. After the last cluster is upgraded, finish with After All Clusters Are Upgraded.

What an Upgrade Changes

Extensions (Operators and Cluster Plugins) with the Aligned lifecycle, Aligned Extensions for short, follow the ACP release. Extensions with the Agnostic lifecycle, Agnostic Extensions, are released independently. See Lifecycle Types.

PartUpgraded by
ACP CoreThe CVO, through the upgrade request of the cluster.
Aligned ExtensionsThe CVO, in the same request, for each Aligned Extension installed on the cluster.
Kubernetes, and Alauda OS on Alauda OS nodesDepends on the node operating system and the provider; see Who Upgrades Kubernetes.
Agnostic ExtensionsYou, separately for each Extension, after the cluster reaches the target ACP version, by following the documentation of the Extension.

The request stops if the package of an installed Aligned Extension is missing; preparation publishes these packages.

The infrastructure providers for Huawei DCS, Bare Metal, Huawei Cloud Stack, and VMware vSphere are Aligned Extensions installed on the global cluster, so they are upgraded with global. Their checks for a workload cluster are meaningful only after global is upgraded.

On an Alauda OS cluster upgraded from a version earlier than v4.5, the platform takes over Kube-OVN and CoreDNS once the cluster and global both run v4.5, and Kube-OVN is rolled out once at that point. Kube-OVN and CoreDNS are never upgraded by hand, and their takeover can finish after the upgrade request completes. A blocked takeover fails validation; if the takeover is blocked, contact technical support.

Who Upgrades Kubernetes

ClusterWho upgrades KubernetesHow
Alauda OSThe platform, in the same upgrade requestNodes are replaced one at a time, as described in What Happens on Alauda OS Nodes. Control-plane and worker nodes pass through each intermediate Kubernetes version; on Bare Metal, workers are reinstalled only at the target version. The request completes when every node runs the target version.
Non-Alauda OSThe platform, in the same upgrade requestIn place on the existing nodes; nodes are not replaced. Every node must already run Linux kernel 5.8 or later with cgroup v2. Complete a node OS update before the upgrade; it restarts the nodes.

One deployment can mix providers, for example global on Huawei DCS and workload clusters on Huawei DCS and VMware vSphere. Each cluster follows the row of its own provider.

ACP Version and Kubernetes Version

ACP Core goes straight from the current to the target ACP version. Kubernetes moves one minor version at a time. Each ACP minor version from 4.1 on currently upgrades clusters to one Kubernetes minor: 1.34 on ACP 4.3, 1.35 on ACP 4.4, and 1.36 on ACP 4.5. On Alauda OS, an upgrade from ACP 4.3 to 4.5 therefore passes through the intermediate Kubernetes version 1.35, and each intermediate Kubernetes version needs its own Alauda OS image, prepared before the upgrade. Preparation also publishes the images of the ACP release of each intermediate Kubernetes version, such as v4.4.1 for 1.35; you never request an upgrade to that release.

Patch upgrades follow the same rules: on Alauda OS, a patch upgrade also replaces every node, one at a time, when the target release changes the Kubernetes build or the Alauda OS image. When neither changes, no node is replaced, but the platform still runs its checks, so the preparation applies to every upgrade.

What Happens on Alauda OS Nodes

Alauda OS nodes are not upgraded in place. On Huawei DCS, Huawei Cloud Stack, and VMware vSphere, each node is replaced with a new virtual machine, one at a time. Bare Metal needs no additional machines: the existing hosts are reinstalled one at a time. The nodes of an Alauda OS global cluster are replaced the same way.

While a node is replaced, the cluster has one node fewer. Pods are evicted from a node before it is replaced and start again on other nodes; a workload with a single replica is unavailable until its Pod runs again. A worker node pool with a single node has no node for its Pods while that node is replaced.

DataDuring node replacement
Data on the node's own disks on Huawei DCS, Huawei Cloud Stack, and VMware vSphere, such as hostPath volumes and local PersistentVolumes, unless they are on pool-managed persistent disksNot kept. Move it to CSI storage before the upgrade.
Data on pool-managed persistent disks (Huawei DCS, Huawei Cloud Stack, VMware vSphere), such as the platform data in /var/cpaasKept when the replacement node takes over the pool entry of the node it replaces. On Huawei DCS and Huawei Cloud Stack, this is guaranteed only when the pool has no free entry, in control-plane and worker pools alike: for example, three entries for a three-node control plane. Remove the free entries from the pool before the upgrade, as described in Check the global Cluster and, for workload clusters, Before You Upgrade the Clusters. VMware vSphere reuses the released entry.
Data on the disks of a Bare Metal host: the platform data in /var/cpaas on the system disk, and the data on other disks, such as the raw disks that LVM Storage (TopoLVM) usesKept. The provider does not wipe the disks of a host when it reinstalls the host. The upgrade preflight still reports hostPath volumes and node-local PersistentVolumes; see Exempt a Confirmed Volume from NoHostPathWorkloads.
Other data on the system disk of a Bare Metal hostNot guaranteed. The reinstall writes the Alauda OS image to the system disk again. Move it to CSI storage before the upgrade.
External storage, such as volumes provided through CSINot affected.

Huawei DCS and Huawei Cloud Stack clusters that still keep platform data on machine template disks must first migrate to pool-managed persistent disks. Complete the migration before the upgrade; it replaces every node.

The time a cluster needs grows with its number of nodes multiplied by the number of Kubernetes versions on its path; on Bare Metal, workers are reinstalled once.

Upgrade Order and Compatibility

Upgrade the global cluster first, then the workload clusters. A workload cluster can be upgraded only to the ACP version that global currently runs. You can request the upgrade of several workload clusters at once; the platform upgrades a limited number of them at the same time, and the others wait. A cluster that is not upgraded yet must still be within the direct upgrade window and, if ACP created it, the Compatible Versions of the next global upgrade; otherwise upgrade it before that global upgrade.

Direct upgrade window. A cluster can be upgraded directly to ACP 4.5 from ACP 4.2, 4.3, or 4.4; preparation confirms that the platform offers the target for your exact patch version. On Alauda OS, the documented path starts at v4.3.2: for an Alauda OS cluster on an earlier version, contact technical support; see Check Cluster Versions. The window applies to every cluster registered on global: the ACP components on imported and registered third-party clusters must also be within it, otherwise the global upgrade is blocked. For a cluster on ACP 4.1 or earlier, contact technical support.

Compatible Versions. Before you upgrade global, every workload cluster created by ACP must run a Kubernetes version within the Compatible Versions of the target release, listed in the Kubernetes Support Matrix. Preflight does not check this rule; check it yourself as described in Check Cluster Versions. Upgrade a cluster outside this range first, to the ACP version that global runs now. The Kubernetes version of imported and registered third-party clusters is not covered by Compatible Versions and does not block the upgrade.

Global Cluster Disaster Recovery. With the Alauda Container Platform Global Cluster Disaster Recovery plugin, prepare both global clusters; the plugin does not synchronize the preparation. Two procedures are available; they differ in when the standby is upgraded:

Both global clusters firstStandby after the workload clusters
ProcedureUpgrade global Clusters in a DR EnvironmentUpgrade the Standby global Cluster After the Workload Clusters
UpgradesEvery minor and patch upgradeA patch upgrade within the same minor version, from ACP 4.5.0 or later
OrderThe standby, the primary, then the workload clustersThe primary, the workload clusters that you choose, then the standby; the other workload clusters after that
SynchronizationThe plugin is uninstalled from the standby and reinstalled without delay after both upgradesThe plugin stays installed; synchronization is paused, and resumed after the standby upgrade
Period without DR protectionFrom the uninstall until the reinstall, through the two global upgradesFrom the pause until the resume is checked, through the upgrades of the primary, of those workload clusters, and of the standby

In both, the upgrades and steps on the page are the only changes on either global cluster during that period, and the procedure is complete only when every step of its page is done, including the synchronization check. Do not start another upgrade of either global cluster before that.

Upgrading global again. You can upgrade global again, for example to a later patch, before every workload cluster is upgraded:

  • Do not start it while a workload cluster upgrade is running. The platform does not prevent it.
  • Workload clusters that lag behind then upgrade directly to the ACP version that global runs; preparation confirms that this version accepts their current version. For them, prepare the Alauda OS images of that version and of each intermediate Kubernetes version on their path. Images of the skipped ACP patch versions are not needed.
  • Each new global upgrade starts again with the preparation and uses the same global upgrade page; in a DR environment, use the page from the table above.

Risks and Recovery

Every upgrade request runs preflight checks before it applies the target version. If global has never been upgraded with the CVO, its first preflight runs during its upgrade, after you deploy the CVO. On Alauda OS, the platform also checks the cluster before it replaces nodes; when the provider on the global cluster does not support these node checks yet, they can run on it only after its ACP Core is upgraded, so preparation includes checking those items yourself.

An upgrade cannot be rolled back, on any provider or node operating system, and you cannot request a lower ACP version. Restoring an etcd backup does not return a cluster to an earlier ACP version. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support. For the reported causes, see Upgrade Troubleshooting.

With Global Cluster Disaster Recovery, uninstall the plugin only when the standby is fully synchronized with the primary: uninstalling it while the standby misses data can delete workload cluster machines and, on Alauda OS, the nodes behind them. The DR page checks this first. From the moment you uninstall the plugin until you reinstall it, there is no DR protection, and the two global upgrades are the only cluster changes allowed on either global cluster. When the standby is upgraded after the workload clusters, the plugin stays installed and synchronization is paused instead, so the standby remains a standby, and there is no DR protection from the pause until synchronization is resumed and checked.