Upgrade Workload Clusters

Use this page after the global cluster is upgraded; in a DR environment, after both global clusters are upgraded; when the standby is upgraded after the workload clusters, use this page once the primary is upgraded, then return to Upgrade the Workload Clusters. Each workload cluster moves to the ACP version (Distribution Version) that global runs now. Unless a step says otherwise, run the commands from the execution host, logged in with ac login, on the global cluster, which is the primary global cluster in a DR environment. Commands for a workload cluster itself run after ac config use-cluster <cluster-name>; switch back with ac config use-cluster global.

WARNING

Do not start another global upgrade while a workload cluster upgrade is running; the platform does not block it. The global upgrade pages check this before the upgrade. See Upgrade Order and Compatibility.

Before You Upgrade the Clusters

Complete these steps for each cluster that you plan to upgrade, before you request its upgrade.

Confirm the Standby global Cluster Is Upgraded

Applies to: DR environments. Skip it while synchronization is paused. Run Preflight Checks for the Cluster checks the primary.

On the standby, where ac whoami --show-server prints the standby VIP:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
ac adm upgrade status --cluster=global
unset KUBECONFIG

Continue when Current shows the target ACP version and the Ready condition is True; otherwise, finish the upgrade of the standby first.

Make Operator Packages Available

Applies to: a cluster that did not receive the target packages of its installed Operators with the Aligned lifecycle during preparation, for example a cluster added later. Skip it for other clusters.

From the execution host, push the target package of each Aligned Operator installed on the cluster. Set --dest-repo to the registry that the cluster uses, with its credentials; see Addresses and Credentials.

read -rsp "Platform token: " PLATFORM_TOKEN; printf '\n'
read -rsp "Registry password: " REGISTRY_PASSWORD; printf '\n'
violet push <operator-package>.tgz --platform-address "https://<platform-address>" \
  --platform-token "${PLATFORM_TOKEN}" --clusters "<cluster-name>" \
  --dest-repo "<registry-address>" --username "<registry-username>" --password "${REGISTRY_PASSWORD}"
unset REGISTRY_PASSWORD

Check the etcd Backup

Applies to: workload clusters created by ACP. Skip it for imported and registered clusters.

On the workload cluster, list its backup records:

ac config use-cluster <cluster-name>
ac whoami --show-server
ac get etcdbackupconfiguration etcd-backup-default \
  -o jsonpath='{range .status.records[*]}{.backupTimestamp}{"\t"}{.result}{"\n"}{end}'
ac config use-cluster global
ResultAction
The record with the latest time shows SuccessContinue.
The command reports NotFound or the server doesn't have a resource typeInstall Alauda Container Platform Cluster Enhancement (shown as Alauda Container Platform Cluster Enhancer on a cluster that runs ACP 4.4 or earlier) on the cluster, as described in etcd Backup and Restore, and run the command again after its first backup. If you cannot install it, contact technical support.
The record with the latest time does not show Success, or there is no recordContact technical support.

Run Preflight Checks for the Cluster

Run this block on the global cluster, the primary in a DR environment, in the installer/ directory of the extracted target Core Package; ac whoami --show-server prints an address that ends with /kubernetes/global/:

ac config use-cluster global
ac whoami --show-server
bash upgrade.sh --preflight --cluster=<cluster-name>
ResultAction
Summary shows Passed or PassedWithWarningsContinue.
GlobalVersionReady reports Retry with the reason GlobalVersionPendingFinish the upgrade of the global clusters, then run the check again.
InfraClusterVersionPreflight or an InfraClusterVersionPreflight/<check> row does not pass (Alauda OS)Take the action in the tab of the infrastructure provider in Check the Cluster by Type.
ResourcePatchUpgradeable failsReview each listed patch with technical support, then run the commands below on the workload cluster, and run the check again.
Another check failsFollow Resolve Other Preflight Blocks, and run the check again. If the check is not listed there, contact technical support.

On the workload cluster, list the patches, review each of them, then exempt it with the target version as the value; do not use *:

ac config use-cluster <cluster-name>
ac whoami --show-server
ac get resourcepatches \
  -o custom-columns='NAME:.metadata.name,EXEMPT:.metadata.annotations.config\.cpaas\.io/exempt-for-ver'
ac get resourcepatches <rp-name> -o yaml
ac annotate resourcepatches <rp-name> config.cpaas.io/exempt-for-ver=<target-version> --overwrite
ac config use-cluster global

Check the Cluster by Type

Applies to: every cluster that you upgrade, except imported clusters. Use the Non-Alauda OS tab for a cluster that ACP created on nodes without Alauda OS, and the tab of the infrastructure provider for an Alauda OS cluster.

Run each block on the execution host unless it says that it runs on a node or a host. Each table has a Continue row for each part of the output: continue when every part matches its Continue row; otherwise, take the action of each row that matches.

Non-Alauda OS
Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere

On each node of the cluster:

uname -r
stat -fc %T /sys/fs/cgroup/

Continue when the kernel is 5.8 or later and within Supported OS and Kernel Versions, and the second command prints cgroup2fs; otherwise, update the OS of the node before you upgrade the cluster, and if you cannot, contact technical support. The update restarts the node.

Check DR Synchronization

Applies to: DR environments. Skip it on a platform without a standby global cluster, and while synchronization is paused.

Run the check on a control plane node of the standby global cluster, then return to the execution host:

export KUBECONFIG=/etc/kubernetes/admin.conf
mirror_svc=$(kubectl get svc -n cpaas-system etcd-sync-monitor -o jsonpath='{.spec.clusterIP}')
if [[ $mirror_svc =~ ^[0-9a-fA-F:]+$ ]]; then mirror_host="[$mirror_svc]"; else mirror_host="$mirror_svc"; fi
curl -g "http://${mirror_host}/check"
ResultAction
localMissedKeys [0] and localSurplusKeys [0]Continue.
check too frequentWait one minute and run the check again; if it does not change, contact technical support.
Any other valueRun the steps in Global Cluster Disaster Recovery Does Not Synchronize, then run the check again.

Upgrade a Workload Cluster

Request the Upgrade

WARNING

An upgrade cannot be rolled back. On Huawei DCS, Huawei Cloud Stack, and VMware vSphere, the request also replaces every node with a new VM, one at a time; on Bare Metal, it reinstalls every host, one at a time. This happens on every minor upgrade, and on a patch upgrade when the target changes the Kubernetes build or the Alauda OS image.

Select the global cluster, list the available updates of the workload cluster, then request the version it lists as <target-version>, for example 4.5.0. Each cluster has its own request; repeat it for each cluster that you upgrade. The platform upgrades a limited number of clusters at the same time, and the rest wait.

ac config use-cluster global
ac whoami --show-server
ac adm upgrade --cluster=<cluster-name>
ac adm upgrade --cluster=<cluster-name> --to=<target-version>

Monitor the Upgrade

ac adm upgrade status --cluster=<cluster-name>
ac -n cpaas-system get infraclusterversion <cluster-name>   # Alauda OS

Run the commands again until Current shows the target version and the Ready condition is True; add --verbose to see every check and stage. If Upgrade shows Blocked or Retrying, see Inspect Upgrade State. On Alauda OS, if the upgrade stays InProgress in the Kubernetes stage, read the Stalled condition of the InfraClusterVersion as described there. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support.

On Alauda OS, CURRENT of the InfraClusterVersion advances through each intermediate Kubernetes version; also wait until CURRENT equals DESIRED and PHASE is Succeeded. Until then, do not edit the cluster's KubeadmControlPlane, MachineDeployment resources, or their templates.

Upgrade the Agnostic Extensions

The upgrade request does not upgrade Extensions (Operators and Cluster Plugins) with the Agnostic lifecycle. Upgrade each Agnostic Extension separately by following its own documentation; while synchronization is paused, upgrade them after it is resumed. For example:

Validate the Cluster

Complete Upgrade Validation for each workload cluster that you upgraded to finish its upgrade.