Prepare for the Upgrade

MySQL-PXC Removed in Alauda Data Services Relational Database E1 v4.3.0

If you have MySQL-PXC instances, they will become unmanaged after upgrading to ACP v4.3.0 or later (when the MySQL operator is also upgraded). See MySQL Operator Upgrade Guide for migration instructions.

Complete these steps before you upgrade. Prepare again for every upgrade, including a patch upgrade of the ACP version (Distribution Version) and an upgrade of global before every workload cluster has caught up.

In a DR environment with Alauda Container Platform Global Cluster Disaster Recovery, do every step that names global on both global clusters. If your clusters run on more than one provider, complete the steps for each of them, including the provider of the global cluster.

Checks at a Glance

CheckApplies to
Check Cluster VersionsEvery cluster
Check the Registry, Modules, and etcd Backup of globalEvery global cluster
Check by OS and ProviderEvery global cluster, in the tab of its OS or infrastructure provider
Check the Staged ImagesEvery global cluster on Alauda OS, in the tab of its infrastructure provider
Check DR SynchronizationDR environments
Run Preflight ChecksEvery global cluster
Before You Upgrade the ClustersEach workload cluster, before its upgrade

Prepare the Execution Host

Run every upgrade command from one execution host: a Linux machine, x86_64 or aarch64, that can reach ACP. Do not use a node of global or of any other cluster; the upgrade does not need one. On Alauda OS, the upgrade replaces the nodes, and files on a replaced node, such as the Core Package and the contents of /root and /home, are not guaranteed to be kept. The few steps that run on a node say so; return to the execution host after each of them.

The execution host needs:

  • Access to the platform address, and to the VIP and registry of each global cluster.
  • The target Core Package, downloaded as described in Download the Core Package, extracted on this host, and kept until every cluster is upgraded.
  • ACP CLI (ac). The Core Package contains it for both architectures, at installer/res/<arch>/ac, where <arch> is amd64 or arm64. Copy the one for this host to a directory in your PATH; for example, from the installer/ directory on an x86_64 host, run sudo cp res/amd64/ac /usr/local/bin/ac.
  • kubectl, and Violet v4.2.13 or later (violet version).
  • A platform account with the platform administrator role.
  • In DR, a kubeconfig of the standby global cluster that points to its VIP, <standby-kubeconfig>. For example, copy /etc/kubernetes/admin.conf, with any certificate files it references, from a control-plane node of the standby, and set its server to the VIP if it points elsewhere.

Log in once with the platform administrator account, with KUBECONFIG unset:

ac login https://<platform-address> --name <session>
ac config use-cluster <cluster-name>
ac whoami --show-server
ac config use-cluster global
  • The session is saved in ~/.kube/config, and ac commands run on global, the primary in DR. For a workload cluster, run ac config use-cluster <cluster-name> first and ac config use-cluster global afterwards; ac whoami --show-server then ends with /kubernetes/<cluster-name>/.
  • upgrade.sh uses kubectl with this session. Run ac config use-cluster global right before the script.
  • For the standby, run export KUBECONFIG=<standby-kubeconfig> first, confirm that ac whoami --show-server prints the standby VIP, and run unset KUBECONFIG afterwards.

Check Cluster Versions

Applies to: every cluster registered on global, global included, imported clusters included. In DR, run it on each global cluster; on the standby, read only its own version.

On global, list the ACP version (the CURRENT column of the first list), the Kubernetes version, and the infrastructure provider of every cluster. In the last list, DCSCluster, BaremetalCluster, HCSCluster, and VSphereCluster mark an Alauda OS cluster, UserProvisionedCluster a cluster that does not run Alauda OS, and <none> an imported cluster:

ac config use-cluster global
ac whoami --show-server
ac -n cpaas-system get cvsh
ac -n cpaas-system get clusters.clusterregistry.k8s.io -o custom-columns='NAME:.metadata.name,KUBERNETES:.status.version'
ac -n cpaas-system get clusters.cluster.x-k8s.io -o custom-columns='NAME:.metadata.name,INFRASTRUCTURE:.spec.infrastructureRef.kind'

If cvsh reports the server doesn't have a resource type, or does not list a cluster, take the ACP version of that cluster from VERSION:

ac get clustermodule -o custom-columns='NAME:.metadata.name,VERSION:.spec.version'

In DR, read the version of the standby, from CURRENT, or from VERSION when cvsh is not available:

ac --kubeconfig <standby-kubeconfig> -n cpaas-system get cvsh global
ac --kubeconfig <standby-kubeconfig> get clustermodule global -o custom-columns='NAME:.metadata.name,VERSION:.spec.version'

Upgrade Path and Compatibility

Compare each cluster with the table. Continue when every cluster matches a row whose action is Continue; otherwise, take the action of each row that matches.

Result for a clusterAction
ACP 4.2, 4.3, or 4.4 (on Alauda OS: ACP v4.3.2 or later); and, for a cluster created by ACP, a Kubernetes version within the Compatible Versions of the target in the Kubernetes Support MatrixContinue. The preflight check VersionUpgradePath confirms the exact patch version.
Already the target minor version: a patch upgrade, or global upgraded againContinue.
ACP 4.1 or earlierContact technical support.
On Alauda OS, an ACP version earlier than v4.3.2Contact technical support.
Created by ACP, with a Kubernetes version outside the Compatible Versions of the targetUpgrade the cluster on the current global first, as described in the documentation of its current ACP version.
No version is shown for the clusterContact technical support.

Check the Registry, Modules, and etcd Backup of global

Applies to: every global cluster. In DR, run each block on the standby too, as the comments on its first and last lines show: start with export KUBECONFIG=<standby-kubeconfig> and finish with unset KUBECONFIG.

Save this script as check-global-common.sh:

#!/usr/bin/env bash
ac whoami --show-server

echo "== Registry"
ac get productbase base -o jsonpath='external={.spec.registry.external}{"\n"}'

echo "== Modules"
ac get moduleinfo -l cpaas.io/cluster-name=global \
  -o custom-columns='MODULE:.metadata.labels.cpaas\.io/module-name,CURRENT:.status.version,TARGET:.spec.version,PHASE:.status.phase'

echo "== etcd backup records"
ac get etcdbackupconfiguration etcd-backup-default \
  -o jsonpath='{range .status.records[*]}{.backupTimestamp}{"\t"}{.result}{"\n"}{end}'

Then run it:

ac config use-cluster global        # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>
bash check-global-common.sh
# on the standby: unset KUBECONFIG

external= with an empty value or false is the platform built-in Registry. With it, on each control-plane node of global, run the line for the OS of the node:

df -h /var/cpaas/minio   # Alauda OS
df -h /cpaas/minio       # non-Alauda OS

Continue when every part of the output matches its Continue row; otherwise, take the action of each row that matches.

ResultAction
external=true, or each control-plane node has at least 120 GiB availableContinue.
A control-plane node has less than 120 GiB availableExpand that disk. If you cannot, contact technical support.
Every module shows PHASE Running and CURRENT equal to TARGETContinue.
A module shows CURRENT different from TARGETRun the script again later. If the output does not change, contact technical support.
A module shows PHASE Failed or BlockedRun the steps in Diagnose a Module That Does Not Advance. If the module does not recover, contact technical support.
A module shows any other PHASEContact technical support.
The etcd backup record with the latest time shows SuccessContinue.
The etcd command reports NotFound or the server doesn't have a resource typeInstall Alauda Container Platform Cluster Enhancement (shown as Alauda Container Platform Cluster Enhancer on a cluster that runs ACP 4.4 or earlier) on the cluster, as described in etcd Backup and Restore, and run the command again after its first backup. If you cannot install it, contact technical support.
The etcd backup record with the latest time does not show Success, or there is no recordContact technical support.

Check the global Cluster

Applies to: every global cluster. In DR, run each block on the standby too, as the comments on its first and last lines show: start with export KUBECONFIG=<standby-kubeconfig> and finish with unset KUBECONFIG.

Run each block on the execution host unless it says that it runs on a node. Each table has a Continue row for each part of the output: continue when every part matches its Continue row; otherwise, take the action of each row that matches. Check the Staged Images checks the images after you stage them.

Check by OS and Provider

Use the Non-Alauda OS tab when the nodes of global do not run Alauda OS; otherwise, use the tab of the infrastructure provider of global, shown in INFRASTRUCTURE in Check Cluster Versions.

WARNING

On Huawei DCS, Huawei Cloud Stack, and VMware vSphere, data on a node's own disks, such as hostPath volumes and local PersistentVolumes, is not kept when the node is replaced, unless it is on a pool-managed persistent disk. Move it to CSI storage before the upgrade.

On Bare Metal, the provider does not wipe the disks of a host when it reinstalls the host. Data on the system disk other than /var/cpaas is not guaranteed to be kept, because the reinstall writes the Alauda OS image to the system disk again. The preflight still reports hostPath volumes and node-local PersistentVolumes.

Non-Alauda OS
Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere

On each node of global:

uname -r
stat -fc %T /sys/fs/cgroup/

Continue when the kernel is 5.8 or later and within Supported OS and Kernel Versions, and the second command prints cgroup2fs; otherwise, update the OS of the node before the upgrade, and if you cannot, contact technical support. The update restarts the node.

Set the Rollout Settings

Use this when a check table or another page points here. Set <cluster-name> to the cluster of the check: global, or a workload cluster. Run the kubeadmcontrolplane patch when replicas is 3 or more and maxSurge is empty or not 0; when replicas is less than 3, the cluster cannot be upgraded. Run the machinedeployment patch for each MachineDeployment whose maxSurge is empty, <none>, or not 0, or whose maxUnavailable is empty, <none>, or not 1.

ac config use-cluster global        # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
KCP=$(ac -n cpaas-system get clusters.cluster.x-k8s.io <cluster-name> -o jsonpath='{.spec.controlPlaneRef.name}')
# The control plane, only when replicas is 3 or more
ac -n cpaas-system patch kubeadmcontrolplane "${KCP}" --type merge \
  -p '{"spec":{"rolloutStrategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":0}}}}'
# Each MachineDeployment to correct
ac -n cpaas-system patch machinedeployment <md-name> --type merge \
  -p '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}}}'
# on the standby: unset KUBECONFIG

Kubelet Patch File

Use this when a check table or another page points here. When a check shows a Secret name and a key for the file /etc/kubernetes/patches/kubeletconfiguration0+strategic.json, print the content from the Secret. Escape each . in <secret-key> as \.:

ac config use-cluster global        # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
ac -n cpaas-system get secret <secret-name> -o jsonpath='{.data.<secret-key>}' | base64 -d
# on the standby: unset KUBECONFIG

On Kubernetes 1.35 or later, the file needs this setting. Do not add it for Kubernetes 1.34 or earlier. In the Kubernetes upgrade, the platform adds it to the existing files.

{
  "imagePullCredentialsVerificationPolicy": "NeverVerify"
}

Check the etcd Backup of a Cluster

Use this when a check table or another page points here, for a cluster created by ACP; skip it for imported and registered clusters. During preparation, Check the Registry, Modules, and etcd Backup of global covers global, and Check the etcd Backup covers each workload cluster.

On the cluster:

ac config use-cluster <cluster-name>        # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
ac get etcdbackupconfiguration etcd-backup-default \
  -o jsonpath='{range .status.records[*]}{.backupTimestamp}{"\t"}{.result}{"\n"}{end}'
ac config use-cluster global        # on the standby, instead: unset KUBECONFIG
ResultAction
The record with the latest time shows SuccessContinue.
The command reports NotFound or the server doesn't have a resource typeInstall Alauda Container Platform Cluster Enhancement (shown as Alauda Container Platform Cluster Enhancer on a cluster that runs ACP 4.4 or earlier) on the cluster, as described in etcd Backup and Restore, and run the command again after its first backup. If you cannot install it, contact technical support.
The record with the latest time does not show Success, or there is no recordContact technical support.

Check Node Readiness for Kubernetes 1.35 or Later

Use this when a check table or another page points here. On a non-Alauda OS cluster created by ACP, run the node check of the Non-Alauda OS tab, on every node of the cluster:

Determine the Versions to Prepare

Applies to: Alauda OS clusters. Skip it for non-Alauda OS clusters, and go to Download the Packages.

Select the current version of each cluster, and prepare every version that the calculator lists for all paths, in the Alauda OS variant that each cluster runs today:

StepACP versionKubernetes versionetcd
Currentv4.5.0v1.36.4-acp.3v3.6.13-acp.5
Targetv4.5.1v1.36.4-acp.4v3.6.13-acp.6

Prepare the Alauda OS image of every Intermediate and Target row, using the same Alauda OS variant that the cluster runs today.

The Kubernetes version and the etcd tag of every ACP version are listed in Version Mapping.

Download the Packages

Download the Core Package and the Alauda OS Images

From the Customer Portal, download the Core Package of the target ACP version for your architecture, and extract it on the execution host. On Alauda OS, also download:

  • The Core Package of the ACP release of each intermediate Kubernetes version. You use it only to publish its images.
  • From the OS download page, the image of the target version and of each intermediate Kubernetes version, for the architecture and OS variant of your nodes: qcow2 for Huawei DCS and Huawei Cloud Stack, ova for VMware vSphere, and the archive of an OCI image layout for Bare Metal.

Download the Other Extension Packages

Export the Extensions installed on global to apps.yaml. Then let the Customer Portal check the file and download the target-version packages of the installed Extensions, including the infrastructure providers and the Cluster Bootstrap Provider. Set <arch> to amd64, arm64, or hybrid to match the Core Package, and <target-version> to the target ACP version, such as 4.5.0.

violet ac uses the address and access token of the Customer Portal; violet list uses the platform address and an ACP API token. See Addresses and Credentials.

read -rsp "Customer Portal token: " AC_TOKEN; printf '\n'
violet ac login --access-token="${AC_TOKEN}" --ac-url=<customer-portal-url>
read -rsp "Platform token: " PLATFORM_TOKEN; printf '\n'
violet list --platform-address "https://<platform-address>" --platform-token "${PLATFORM_TOKEN}" --output-file "./apps.yaml"
violet ac import-yaml --arch=<arch> --platformVersion=v<target-version> --path=./apps.yaml --download=true

Run violet ac on a machine that has access to the Customer Portal. If the execution host has none, run violet list on the execution host, copy apps.yaml to such a machine, run violet ac login and violet ac import-yaml there, and copy the downloaded packages to the execution host. If a command fails, see Troubleshooting.

In a DR environment, do the same for the standby, into its own file. The two global clusters can run different Extensions; for example, Global Cluster Disaster Recovery runs only on the standby:

read -rsp "Standby platform token: " STANDBY_TOKEN; printf '\n'
violet list --platform-address "https://<standby-vip>" --platform-token "${STANDBY_TOKEN}" --output-file "./standby-apps.yaml"
violet ac import-yaml --arch=<arch> --platformVersion=v<target-version> --path=./standby-apps.yaml --download=true

Download the Upgrade Scenario for ACP 4.3 or Earlier

Only when the source ACP version is 4.3 or earlier. With the Customer Portal login of the previous step, download the Upgrade from ACP 4.3 or earlier scenario:

violet ac scenarios --arch=<arch> --platformVersion=v<target-version>
violet ac download-app --arch=<arch> --platformVersion=v<target-version> --scenario="Upgrade from ACP 4.3 or earlier"

Publish the Packages

Run the steps from the installer/ directory of the extracted target Core Package, against global, and complete them before you upgrade global.

Stage the Upgrade Scenario Packages

Only when the source ACP version is 4.3 or earlier. Copy every package of the upgrade scenario into plugins/, then list the directory and compare it with the downloaded packages.

cp <path-to-scenario-package> ./plugins/
ls -1 ./plugins/

Publish Core and the Staged Packages

Read the registry type of global: the second value is empty or false for the platform built-in Registry, and true for an external registry. Every Aligned Extension installed on a cluster needs its target-version package before that cluster's upgrade; otherwise the upgrade stops until you publish it.

ac config use-cluster global
ac whoami --show-server
ac get productbase base -o jsonpath='{.spec.registry.address}{"\t"}{.spec.registry.external}{"\n"}'
Platform built-in Registry
External registry
ac config use-cluster global
ac whoami --show-server
bash upgrade.sh --only-sync-image

Publish the Intermediate Core Images

On Alauda OS, when the path has an intermediate Kubernetes version, publish the images of the Core Package of each intermediate Kubernetes version, from its installer/ directory, against the same global cluster. Do not request an upgrade to that release. Skip this step when there is no intermediate Kubernetes version.

Platform built-in Registry
External registry
ac config use-cluster global
ac whoami --show-server
bash upgrade.sh --only-sync-image

Publish the Other Extensions

Publish the packages downloaded for apps.yaml in Download the Other Extension Packages with violet push. The packages for the standby follow in the next step.

PackagesPublish toWhen
Cluster plugins, including the providersglobalBefore the global upgrade
Operatorsglobal and each workload cluster that runs the operatorBefore the upgrade of each such cluster

Set the registry of global and its credentials, by the registry type that you read in Publish Core and the Staged Packages. For the address and credentials, see Addresses and Credentials.

Platform built-in Registry
External registry

Set <global-vip> to the VIP of global, and <registry-port> to the port of the registry address printed in Publish Core and the Staged Packages.

ac config use-cluster global
REGISTRY_ADDRESS="<global-vip>:<registry-port>"
REGISTRY_USERNAME=$(ac -n cpaas-system get secret registry-admin -o jsonpath='{.data.username}' | base64 -d)
REGISTRY_PASSWORD=$(ac -n cpaas-system get secret registry-admin -o jsonpath='{.data.password}' | base64 -d)

Then push each package:

# A cluster plugin: the platform distributes it
violet push <cluster-plugin-package>.tgz --platform-address "https://<platform-address>" --platform-token "${PLATFORM_TOKEN}" \
  --dest-repo "${REGISTRY_ADDRESS}" --username "${REGISTRY_USERNAME}" --password "${REGISTRY_PASSWORD}"
# An operator: global and every workload cluster that runs it
violet push <operator-package>.tgz --platform-address "https://<platform-address>" --platform-token "${PLATFORM_TOKEN}" \
  --clusters "global,<cluster-name-1>,<cluster-name-2>" \
  --dest-repo "${REGISTRY_ADDRESS}" --username "${REGISTRY_USERNAME}" --password "${REGISTRY_PASSWORD}"
unset REGISTRY_USERNAME REGISTRY_PASSWORD

Violet pushes the images only to --dest-repo. When a workload cluster uses a registry other than that of global, push each of its operators again, with --clusters "<cluster-name>" and --dest-repo and the credentials of that registry.

Publish to the Standby global Cluster

In a DR environment only. Select the standby for the rest of this step, and read its registry type, as on the primary:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
ac get productbase base -o jsonpath='{.spec.registry.address}{"\t"}{.spec.registry.external}{"\n"}'
Platform built-in Registry
External registry

Publish the images to the registry of the standby: run the command from the installer/ directory of the target Core Package, and again from that of each intermediate Core Package. The script uses the registry of the cluster in KUBECONFIG.

bash upgrade.sh --only-sync-image

Then push the packages downloaded for standby-apps.yaml in Download the Other Extension Packages. Set <registry-port> to the port of the registry address printed above. Always set --platform-address and --dest-repo; without them, the packages go to the primary.

STANDBY_REGISTRY_USER=$(ac -n cpaas-system get secret registry-admin -o jsonpath='{.data.username}' | base64 -d)
STANDBY_REGISTRY_PASSWORD=$(ac -n cpaas-system get secret registry-admin -o jsonpath='{.data.password}' | base64 -d)
read -rsp "Standby platform token: " STANDBY_TOKEN; printf '\n'
violet push <package>.tgz --platform-address "https://<standby-vip>" --platform-token "${STANDBY_TOKEN}" \
  --dest-repo "<standby-vip>:<registry-port>" --username "${STANDBY_REGISTRY_USER}" --password "${STANDBY_REGISTRY_PASSWORD}"
# Repeat violet push for each package, then:
unset STANDBY_REGISTRY_USER STANDBY_REGISTRY_PASSWORD STANDBY_TOKEN

On the standby, list the versions of Global Cluster Disaster Recovery, which runs only on the standby, then return to the login session:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
ac get moduleconfigs -l cpaas.io/module-name=etcd-sync
unset KUBECONFIG

Continue when the target version is listed; otherwise, run violet push for the package again as above.

Stage the Alauda OS Images

On non-Alauda OS clusters, skip this section.

Stage the image of the target version and of every intermediate Kubernetes version for each provider in use. The commands that the tabs generate run on global; in DR, run them on both global clusters, on the standby after export KUBECONFIG=<standby-kubeconfig>, and run unset KUBECONFIG afterwards:

ac config use-cluster global        # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere

Upload the qcow2 image of each version to every DCS site that your clusters use, and register it as a VM template with the same name on every site. Then generate the VM template ConfigMaps and apply them on global. Select every architecture whose VM templates you register; each generated ConfigMap records its architecture. Register the VM templates of every architecture that your clusters use, and in a cluster that mixes architectures, set architecture on every ConfigMap, as described in VM Template ConfigMap.

Architectures
v4.5.1 · Kubernetes v1.36.4-acp.4
ac apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: dcs-vm-template-m55-x86-64-v4.5.1
  namespace: cpaas-system
  labels:
    cpaas.io/dcs-vm-template: "<vm-template-name-for-v4.5.1-x86_64>"
    cpaas.io/distribution-version: "v4.5.1"
data:
  kubernetesVersion: "v1.36.4-acp.4"
  etcdTag: "v3.6.13-acp.6"
  architecture: "x86_64"
EOF

Run the command on the global cluster. In a DR environment, run it on both global clusters: the ConfigMaps are not synchronized between them.

For each ACP version, register VM templates of only one OS per architecture on a global cluster. When VM templates of different OSes have the same Kubernetes version and architecture, an upgrade can move a cluster to the other OS.

ConfigMaps created from earlier versions of this document have no architecture key and no architecture in their name. Once the ConfigMaps of an ACP version exist for each architecture that you use, you can delete the earlier ConfigMaps of that version when no upgrade is in progress.

If you register VM templates of both architectures for a version on the upgrade path, also register the VM template that each cluster runs now, including global, when it has no ConfigMap yet. The upgrade takes the architecture of a node group from the ConfigMap of its current VM template. Without that ConfigMap, the upgrade cannot choose between the two architectures and stops with ImageCatalogMiss. Register it as described in VM Template ConfigMap, with the ACP version of the Alauda OS image that the VM template was made from.

Check the Staged Images

Applies to: every global cluster on Alauda OS, in the tab of its infrastructure provider, as in Check by OS and Provider. In DR, run each block on the standby too, as the comments on its first and last lines show.

Non-Alauda OS
Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere

No action is needed.

Check DR Synchronization

Applies to: DR environments. Skip it for a single global cluster.

Log in to a control-plane node of the standby global cluster, run the check there, and then return to the execution host:

export KUBECONFIG=/etc/kubernetes/admin.conf
mirror_svc=$(kubectl get svc -n cpaas-system etcd-sync-monitor -o jsonpath='{.spec.clusterIP}')
if [[ $mirror_svc =~ ^[0-9a-fA-F:]+$ ]]; then mirror_host="[$mirror_svc]"; else mirror_host="$mirror_svc"; fi
curl -g "http://${mirror_host}/check"
ResultAction
localMissedKeys [0] and localSurplusKeys [0]Continue.
check too frequentWait one minute and run the check again; if it does not change, contact technical support.
Any other valueRun the steps in Global Cluster Disaster Recovery Does Not Synchronize, then run the check again.

Run Preflight Checks

Applies to: every global cluster. In DR, run it on each global cluster. Skip it for workload clusters.

From the installer/ directory of the target Core Package, on each global cluster:

ac config use-cluster global        # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
unset REGISTRY_USERNAME REGISTRY_PASSWORD
ac get productmanifests
bash upgrade.sh --preflight
# on the standby: unset KUBECONFIG
ResultAction
Summary shows Passed or PassedWithWarnings, and no InfraClusterVersionPreflight or InfraClusterVersionPreflight/<check> row reports a warningContinue.
ac get productmanifests reports the server doesn't have a resource typeContinue without bash upgrade.sh --preflight. You run it in Run the Final Preflight Check, after you deploy the Cluster Version Operator (CVO).
On the standby, DRSplitUpgradeWindow is the only failed check, and its message names only EtcdSyncNotPaused, TokenRotatorEnabled, or bothContinue. Summary shows Blocked at this stage.
On the standby, DRSplitUpgradeWindow names another reasonSee DR Standby Checks.
On a global cluster on Alauda OS, InfraClusterVersionPreflight reports ProviderCapabilityNotDeclaredRun the script whose name ends with -no-preflight.sh in the provider tab of Check by OS and Provider, and the check in the provider tab of Check the Staged Images, and handle their results. Then continue.
InfraClusterVersionPreflight reports PreflightPendingRun the preflight check again after 5 minutes; if it does not change, contact technical support.
An InfraClusterVersionPreflight/<check> row reports a warningTake the action of each such <check> in Kubernetes Upgrade Checks. Then continue.
ResourcePatchUpgradeable failsRun the exemption commands in Resolve Preflight Blocks.
AdminAckRequired failsRun the acknowledgement commands in Resolve Preflight Blocks.
VersionUpgradePath failsFollow Resolve Other Preflight Blocks. Do not bypass the check with an explicit upgrade request.
Any other check failsFollow Resolve Other Preflight Blocks. If the check is not listed there, contact technical support.

Resolve Preflight Blocks

For ResourcePatchUpgradeable, review each listed patch with technical support, then exempt it on the global cluster, with the target version as the value; do not use *. In DR, do it on each side:

ac config use-cluster global        # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
ac get resourcepatches \
  -o custom-columns='NAME:.metadata.name,EXEMPT:.metadata.annotations.config\.cpaas\.io/exempt-for-ver'
ac get resourcepatches <rp-name> -o yaml
ac annotate resourcepatches <rp-name> config.cpaas.io/exempt-for-ver=<target-version> --overwrite
# on the standby: unset KUBECONFIG

For AdminAckRequired, copy the key from admin-gates, record the acknowledgement, and run preflight again. On non-Alauda OS, first complete the node check in the Non-Alauda OS tab of Check by OS and Provider. In DR, do it on each side:

ac config use-cluster global        # on the standby, instead: export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
ac -n cpaas-system get configmap admin-gates -o yaml
ACK_KEY='<key-from-admin-gates>'
ac -n cpaas-system patch configmap admin-acks --type merge -p "{\"data\":{\"${ACK_KEY}\":\"true\"}}"
bash upgrade.sh --preflight
# on the standby: unset KUBECONFIG

Next Step