Upgrade Troubleshooting
Use this page when an upgrade check fails or an upgrade does not advance. An upgrade cannot be rolled back. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support.
TOC
Inspect Upgrade StateHandle Preflight BlocksResolve Other Preflight BlocksDR Standby ChecksHandle Administrator Acknowledgement GatesCluster Plugin Affinity MismatchKubernetes Upgrade on Alauda OSKubernetes Upgrade ChecksImage Catalog ProblemsExempt a Confirmed Volume from NoHostPathWorkloadsRead the ProgressStalled ReasonsFree Space on COS_STATE (Bare Metal)Remove Partial SnapshotsHosts on a Different Image (Bare Metal)Node Replacement Symptoms (Bare Metal)vSphere Rollout Holds That Are Not FailuresKube-OVN and CoreDNS Takeover Is BlockedRecover a Missing Aligned Extension PackageAligned Cluster PluginsAligned OperatorsDiagnose a Module That Does Not Advancemachine-config-daemon Does Not StartGlobal Cluster Disaster Recovery Does Not SynchronizeCollect Evidence Before EscalationInspect Upgrade State
Each command block on this page names the cluster it runs on. Run it from the execution host after ac login. Commands run on global unless the block switches to another cluster with ac config use-cluster <cluster-name>; such a block ends with ac config use-cluster global. ac whoami --show-server shows the cluster you are on: the platform address followed by /kubernetes/<cluster-name>/. For the standby global cluster in a DR environment, skip ac config use-cluster and add --kubeconfig <standby-kubeconfig> to each other ac command; ac whoami --show-server then prints the VIP of the standby.
For example, on the global cluster, read the state of the cluster being upgraded. Use --cluster=global for a global cluster itself:
Upgrade shows the state of the request:
Blocked: a preflight check failed. Resolve the firstFailedcheck underPreflight.Retrying: a preflight check reportsRetry, and it runs again automatically.InProgress: the upgrade runs. A step that is being retried also showsInProgress.Completed: the cluster runs the target version.
Conditions (Ready, Reconciling, Stalled) and Preflight give the reason and message. Failed (non-blocking) does not stop the upgrade. Upgrade stages shows what is running. If Upgrade stays InProgress and Ready shows the reason ClusterPluginAffinityMismatch, see Cluster Plugin Affinity Mismatch.
On Alauda OS, when Upgrade stays InProgress in the Kubernetes stage, a node replacement that does not advance shows only on the InfraClusterVersion of the cluster. On the global cluster that manages the cluster, read its Stalled condition:
If it prints True, see Stalled Reasons. For the full progress, see Read the Progress.
Before the request, bash upgrade.sh --preflight prints the same checks; for a workload cluster, add --cluster=<cluster-name>. After the request, the Cluster Version Operator (CVO) retries automatically. Submitting the request again does not clear a block.
Handle Preflight Blocks
A ResourcePatch records a customization of a platform resource. ResourcePatchUpgradeable fails with the reason UnexemptResourcePatches until each ResourcePatch is exempted for the target version. The message names up to five of them. Run the commands on the cluster being upgraded: for a global cluster, use global as <cluster-name>; in a DR environment, run them on the primary and again on the standby global cluster, with --kubeconfig <standby-kubeconfig> instead of ac config use-cluster. List the patches and their exemptions, review each patch, then exempt it:
The value is the target version, for example 4.5.0, and covers every patch version of that minor version. An exemption keeps the customization in place; it does not confirm that the customization still works on the target version. Confirm that with the technical support contact who supplied it.
Resolve Other Preflight Blocks
Do not force an upgrade past a failed check. spec.desiredUpdate.force on the request turns every blocking failure of the whole preflight into a warning at once, not only the one you have reviewed; it releases nothing while any failure that cannot be bypassed remains. If a block cannot be resolved from this page, contact technical support.
DR Standby Checks
On a standby that has the DR marker, DRSplitUpgradeWindow passes only while synchronization is paused, the token rotator is disabled, and no object is being deleted, as in Upgrade the Standby global Cluster After the Workload Clusters. Without the marker, for example after the plugin is uninstalled in Upgrade global Clusters in a DR Environment, the check passes. In Upgrade global Clusters in a DR Environment, a failure on the standby before the uninstall that names only EtcdSyncNotPaused, TokenRotatorEnabled, or both is expected: continue as described there, and do not pause synchronization or disable the token rotator. For DeletingObjectsPresent, let the deletion finish on the primary, wait until the standby is synchronized, and run the preflight again. The table applies only to Upgrade the Standby global Cluster After the Workload Clusters. The message names the reason:
Handle Administrator Acknowledgement Gates
AdminAckRequired runs only for global clusters. In a DR environment, record the acknowledgement on the primary and again on the standby global cluster; it is not synchronized. On the global cluster, read the gates supplied by the target release:
Complete the action that the gate describes. On a global cluster on ACP 4.4 or earlier, AdminAckRequired reports Confirm Linux kernel 5.8+ and cgroup v2 readiness before upgrading to Kubernetes 1.35. (gate key ack-4.4-kubernetes-1.35-kernel-update); complete Check Node Readiness for Kubernetes 1.35 or Later for it. Then, on the same global cluster, record the acknowledgement with the key copied from admin-gates, and run the preflight again from the installer/ directory of the extracted target Core Package. On the standby, run export KUBECONFIG=<standby-kubeconfig> instead of ac config use-cluster global, and unset KUBECONFIG afterwards. Continue when Summary shows Passed or PassedWithWarnings:
Cluster Plugin Affinity Mismatch
When Upgrade stays InProgress and Ready shows the reason ClusterPluginAffinityMismatch, the labels of the cluster no longer match the cluster affinity of a Cluster Plugin installed on it, as defined in the target version. The message names the Cluster Plugins. No component of the cluster is upgraded until this is resolved; then the upgrade continues by itself. On the global cluster, read the labels of the cluster:
If a label of the cluster was removed or changed, restore it. Otherwise, contact technical support with the message.
Kubernetes Upgrade on Alauda OS
On Alauda OS, the upgrade request also upgrades Kubernetes and Alauda OS. The platform tracks this in an InfraClusterVersion named after the cluster, on the global cluster that manages it; a global cluster has its own, named global.
Kubernetes Upgrade Checks
The platform checks the whole path to the target, including every intermediate Kubernetes version, and evaluates the checks again every 5 minutes. A fixed problem passes on the next evaluation. A message about one version on the path starts with hop <version>: , for example hop v1.36.4-acp.3: DCS VM template <vm-template-name> not found.
Each check in the tables below that fails or reports a warning has its own row in the upgrade preflight, named InfraClusterVersionPreflight/<check>, with the reason of the check; for example, InfraClusterVersionPreflight/NoHostPathWorkloads. When every check passes, the preflight shows only InfraClusterVersionPreflight.
- Before the
globalupgrade installs the new provider,InfraClusterVersionPreflightis a non-blocking warning with the reasonProviderCapabilityNotDeclared. This is expected: the checks run again in theKubernetesstage, after ACP Core is upgraded. - If a blocking check fails in the
Kubernetesstage, the upgrade waits there and continues by itself once you fix the cause.
To read every check, run the commands below on the global cluster that manages the cluster. Replace <preflight-name> with the name at the start of the InfraClusterVersionPreflight/<check> message; in the list, it is the row whose TARGET is the target Kubernetes version:
Image Catalog Problems
On Huawei DCS and Huawei Cloud Stack, the platform picks the VM template or image of each Kubernetes version on the path from the ConfigMaps registered on global, separately for each architecture that the cluster uses. The architecture of a node pool comes from the ConfigMaps of the VM template or image it uses now. The reasons below appear in the check UpgradePathResolvable and as Stalled reasons; register or correct the ConfigMaps as described in VM Template ConfigMap or Image ConfigMap.
Exempt a Confirmed Volume from NoHostPathWorkloads
When you have confirmed that the data on a reported volume can be lost or survives node replacement, exempt that volume. On the global cluster, create the ConfigMap icv-hostpath-exemptions in cpaas-system, for example with ac apply -f <file>.
Set exactly one of hostPath, persistentVolume, and csiDriver in a rule.
Examples of rules in rules.yaml:
Logs and crash dumps of one Ceph cluster, on one cluster:
A rule exempts only the matching volume; other volumes on the same Pod are still reported. The check is evaluated again within 5 minutes, and when it passes, its message states how many volumes were exempted. Read the message with the command in Kubernetes Upgrade Checks.
If any rule is invalid, none of the rules apply, and the check fails with the reason HostPathExemptionsInvalid, naming the rule, even when no volume is reported. A ConfigMap without the rules.yaml key, or with an unknown field, is invalid too. Provider upgrades do not change this ConfigMap.
Read the Progress
On the global cluster that manages the cluster, read the summary, the conditions, the planned versions, the current step, and the events:
CURRENT advances after each intermediate Kubernetes version completes; DESIRED is the target. Until the provider takes a new target, PHASE is empty or Pending, or still shows Succeeded from an earlier upgrade. It then moves to Preparing (no node is replaced yet), Progressing (nodes are replaced one at a time; on Bare Metal, hosts are reinstalled), and Succeeded. There is no failed phase: a problem shows as Stalled=True, see Stalled Reasons. On Bare Metal, a worker node pool that is not replaced at an intermediate version shows the step phase Skipped.
The Kubernetes upgrade is complete when CURRENT equals DESIRED, PHASE is Succeeded, and Ready is True. The Ready message kube-proxy rollout pending also means complete: kube-proxy is still rolling out; check the kube-system/kube-proxy DaemonSet on the cluster. While the upgrade runs, do not edit the KubeadmControlPlane, the MachineDeployment resources, their templates, the VM template ConfigMaps (Huawei DCS and VMware vSphere), the image ConfigMaps (Huawei Cloud Stack), or elemental-image-catalog (Bare Metal).
Stalled Reasons
A problem shows as the InfraClusterVersion condition Stalled=True. Its message names the affected resources and the next action. In Preparing, no node has been replaced: fix the cause, and the platform retries automatically within 15 minutes. In Progressing, the upgrade waits where it is and continues as soon as the cause is cleared.
If PHASE does not move to Preparing and the upgrade status reports InfraClusterVersionUnconsumed, check that the Deployment cluster-api-provider-dcs-icv-controller (Huawei DCS), cluster-api-provider-baremetal-icv-controller (Bare Metal), cluster-api-provider-hcs-icv-controller (Huawei Cloud Stack), or cluster-api-provider-vsphere-icv-controller (VMware vSphere) in cpaas-system on the global cluster is running.
Before you change a Cluster API resource of the cluster by hand during an upgrade, stop this InfraClusterVersion controller on the global cluster that manages the cluster, and start it again when you are done; otherwise it continues the upgrade while you work. On Bare Metal, this includes deleting a Machine.
Stopping the InfraClusterVersion controller pauses the upgrade of every cluster of that provider on this global cluster. Start it again as soon as your change is done.
Free Space on COS_STATE (Bare Metal)
Each Bare Metal host needs at least 4 GiB free on its COS_STATE partition to install the new OS image. A host that runs short fails with no space left on device, and each failed attempt leaves a partial snapshot that reduces the free space further. Check every host of the affected pools, because any Available host can receive the new image. On each host, read the free space and the snapshots. Use btrfs, not df, because df does not count snapshots. Free (estimated) must be at least 4 GiB:
Remove Partial Snapshots
A snapshot left by a failed attempt shows no in the default and active columns and update-in-progress=yes in its userdata. On the host, list the snapshots, then delete only snapshots that match all three conditions:
Never delete a snapshot whose default or active column is yes. The host boots the default snapshot and runs the active one; deleting either leaves the host unable to boot.
Run btrfs filesystem usage -h / again to confirm that the space was reclaimed. If the deletion fails, or the host still has less than 4 GiB free, the partition is too small. A host whose COS_STATE has the default 8 GiB size needs a reinstall from an ISO built as described in Create the Registration and the ISO, which has a larger partition. Reinstall such hosts before you upgrade the cluster; a reinstall replaces the node. If a host cannot be reinstalled, contact technical support.
Hosts on a Different Image (Bare Metal)
If elemental-image-catalog changed during the upgrade, some reinstalled hosts can run another image than the catalog now lists. The upgrade still completes and reports a Warning event ImageMismatch that names the BaremetalMachine objects and both images. After the cluster's InfraClusterVersion is Succeeded, confirm that the catalog holds the image you want. Then, on the global cluster, start a new rollout of the affected node group; its hosts are reinstalled according to the rollout settings, from the image now in the catalog:
Node Replacement Symptoms (Bare Metal)
vSphere Rollout Holds That Are Not Failures
Some pauses during a vSphere rollout protect data. During such a hold, the InfraClusterVersion stays Progressing, its Ready message names the hold, and the upgrade continues by itself; no action is needed. Do not clear a hold by hand.
- Attachment-safety hold. While an old VM still has the slot's persistent disks attached, the provider keeps the deletion of the
VSphereVMpending. A rollout that seems stuck at the deletion of the first old machine is often this hold. Do not remove the finalizer. Reclaimedis a tombstone. Do not delete aphase: Reclaimedentry to unstick a rollout.SlotAvailable=Falseis expected. WithmaxSurge: 0and a fully allocated pool, no slot is free until an old machine is deleted.- Power state. A node that is powered off in vCenter stays off until you power it on there. A replacement VM is powered on, even if the node it replaces was off before the upgrade.
On the global cluster, read the disk handoff and the pending deletions. The normal handoff is Attached, then Available once the old VM is gone, then Attached again on the replacement:
Kube-OVN and CoreDNS Takeover Is Blocked
On an Alauda OS cluster upgraded from a version earlier than v4.5, the platform takes over Kube-OVN and CoreDNS after the cluster and global both run v4.5. On the global cluster that manages the cluster, read the state from the cluster resource of its provider, dcscluster, baremetalcluster, hcscluster, or vspherecluster:
True: the platform manages Kube-OVN and CoreDNS.FalsewithCoreAddonsHandoffPendingorCoreDNSSkipPhasePending: the takeover is waiting, for example for a node rollout to finish. Check again later; if it does not change, contact technical support.FalsewithCoreAddonsHandoffNotApplicable: the cluster is not on v4.5 yet; check again after its upgrade completes. If it still reports this reason then, contact technical support.FalsewithCoreAddonsHandoffBlocked: the message names the setting. Contact technical support.- No output: the cluster does not use Kube-OVN, and there is nothing to take over.
While the takeover is blocked, the cluster does not pass validation.
Recover a Missing Aligned Extension Package
Use this section when the target ProductManifest is not Ready after you deploy CVO, or when the upgrade status reports that an installed platform-aligned component has no ready ModulePluginConfig. A missing package does not change what is installed now. CVO can report it as Ready=False or Reconciling=True instead of Stalled=True.
On the global cluster, list the artifacts of the target release that are not Ready. Entries for optional components, for Extensions with the Agnostic lifecycle, or for components that are not installed can legitimately appear; compare the output with the installed Core and Aligned Extensions:
Aligned Cluster Plugins
artifactStatus: Absent means that the matching ModulePluginConfig is not fully Ready; it does not prove that the image is missing from the registry. On the global cluster, inspect the ModulePluginConfig named in the message, or identified by the component name and channel tag from the output above:
- For a plugin of the
Upgrade from ACP 4.3 or earlierscenario, copy its package into theplugins/directory of the target Core Package, then publish again as described in Publish Core and the Staged Packages. - For another Aligned Cluster Plugin, publish the target package with
violet pushto theglobalcluster whose upgrade reports it.
Aligned Operators
Aligned Operators do not use ModulePluginConfig. On the affected cluster, inspect the Subscription, the target InstallPlan, and the catalog; on the global cluster, inspect the ModuleInfo:
If the target Operator package is missing, publish it to every cluster where the Operator is installed; for one workload cluster, use violet push with --clusters "<cluster-name>". If the package exists, resolve the condition that CVO reports. Then watch the existing request with ac adm upgrade status --cluster=<cluster-name>; CVO retries automatically.
Diagnose a Module That Does Not Advance
On the global cluster, list the modules of the cluster, then read the conditions of the one that does not advance. Inspect the resource named in the condition and its Events; for an image pull failure, check registry access, CA trust, pull credentials, and the referenced manifest from the affected node:
machine-config-daemon Does Not Start
Use this section when an upgrade stays in the component stage and Kubernetes node replacement has not started. machine-config is not ready. machine-config-daemon is CrashLoopBackOff. The Pod log contains:
Handle one node at a time. Do not restart the node, delete a snapshot, or submit the upgrade request again. The overlay is stored on /run and disappears after any node restart. If the node restarts unexpectedly and /usr/local/bin is still read-only, mount the overlay again, then delete that node's machine-config-daemon Pod so it is recreated.
On the affected node, as root, mount a writable overlay on /usr/local/bin:
On the cluster being upgraded, delete only that node's daemon Pod. Wait until the new Pod is Running and the node stays Ready:
Repeat for every node whose log matches. While any machine-config-daemon Pod is CrashLoopBackOff, continue to watch the existing request:
Global Cluster Disaster Recovery Does Not Synchronize
Use this section when the Pods of the Alauda Container Platform Global Cluster Disaster Recovery plugin are not Running, or the /check command does not report localMissedKeys [0] and localSurplusKeys [0]. Run every command on the standby global cluster: export its kubeconfig, and confirm that ac whoami --show-server prints the VIP of the standby. The etcd-sync-bootstrap Job runs before the synchronization starts: check the Job and the credentials it creates (if the Job failed, its log names the failing step), then the synchronization Pods and the current leader:
In the /check output, the keys under localSurplusKeys (lines starting with +) exist on the standby but not on the primary; review them with your operations team before you delete them. The keys under localMissedKeys (lines starting with -) exist on the primary but not on the standby; this often resolves after you recreate the current leader Pod. Recreate it also when resources with ownerReference dependencies must be synchronized again. Wait until Start Sync update appears in its log, then run on the standby:
Run the /check command again. If it still does not report localMissedKeys [0] and localSurplusKeys [0], or a Pod is not Running, contact technical support.
Collect Evidence Before Escalation
Record the source and target ACP versions and the affected ModuleInfo or ModulePluginConfig. Do not include tokens, registry passwords, kubeconfig files, or Secret contents. On the global cluster that manages the cluster, also save the upgrade state and, on Alauda OS, the Kubernetes upgrade state: