Upgrade Troubleshooting

Use this page when an upgrade check fails or an upgrade does not advance. An upgrade cannot be rolled back. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support.

Inspect Upgrade State

Each command block on this page names the cluster it runs on. Run it from the execution host after ac login. Commands run on global unless the block switches to another cluster with ac config use-cluster <cluster-name>; such a block ends with ac config use-cluster global. ac whoami --show-server shows the cluster you are on: the platform address followed by /kubernetes/<cluster-name>/. For the standby global cluster in a DR environment, skip ac config use-cluster and add --kubeconfig <standby-kubeconfig> to each other ac command; ac whoami --show-server then prints the VIP of the standby.

For example, on the global cluster, read the state of the cluster being upgraded. Use --cluster=global for a global cluster itself:

ac config use-cluster global
ac whoami --show-server
ac adm upgrade status --cluster=<cluster-name> --verbose

Upgrade shows the state of the request:

  • Blocked: a preflight check failed. Resolve the first Failed check under Preflight.
  • Retrying: a preflight check reports Retry, and it runs again automatically.
  • InProgress: the upgrade runs. A step that is being retried also shows InProgress.
  • Completed: the cluster runs the target version.

Conditions (Ready, Reconciling, Stalled) and Preflight give the reason and message. Failed (non-blocking) does not stop the upgrade. Upgrade stages shows what is running. If Upgrade stays InProgress and Ready shows the reason ClusterPluginAffinityMismatch, see Cluster Plugin Affinity Mismatch.

On Alauda OS, when Upgrade stays InProgress in the Kubernetes stage, a node replacement that does not advance shows only on the InfraClusterVersion of the cluster. On the global cluster that manages the cluster, read its Stalled condition:

ac -n cpaas-system get infraclusterversion <cluster-name> \
  -o jsonpath='{range .status.conditions[?(@.type=="Stalled")]}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

If it prints True, see Stalled Reasons. For the full progress, see Read the Progress.

Before the request, bash upgrade.sh --preflight prints the same checks; for a workload cluster, add --cluster=<cluster-name>. After the request, the Cluster Version Operator (CVO) retries automatically. Submitting the request again does not clear a block.

Handle Preflight Blocks

A ResourcePatch records a customization of a platform resource. ResourcePatchUpgradeable fails with the reason UnexemptResourcePatches until each ResourcePatch is exempted for the target version. The message names up to five of them. Run the commands on the cluster being upgraded: for a global cluster, use global as <cluster-name>; in a DR environment, run them on the primary and again on the standby global cluster, with --kubeconfig <standby-kubeconfig> instead of ac config use-cluster. List the patches and their exemptions, review each patch, then exempt it:

ac config use-cluster <cluster-name>
ac get resourcepatches \
  -o custom-columns='NAME:.metadata.name,EXEMPT:.metadata.annotations.config\.cpaas\.io/exempt-for-ver'
ac get resourcepatches <rp-name> -o yaml
ac annotate resourcepatches <rp-name> config.cpaas.io/exempt-for-ver=<target-version> --overwrite
ac config use-cluster global

The value is the target version, for example 4.5.0, and covers every patch version of that minor version. An exemption keeps the customization in place; it does not confirm that the customization still works on the target version. Confirm that with the technical support contact who supplied it.

Resolve Other Preflight Blocks

CheckDetails
VersionUpgradePathVerifies: The ACP version (Distribution Version) of the cluster can be upgraded directly to the requested version. On a global cluster, every registered cluster, imported clusters included, is also within the direct upgrade window of the target.
Fix:
Message Upgrade path blocked: request a version listed under Available updates, or contact technical support.
Message upgrade path blocked for <cluster>(<version>): upgrade the named clusters first.
GlobalVersionReadyVerifies: Workload clusters only: the global cluster has reached the target version.
Fix: Reason GlobalVersionPending: finish the global upgrade first; in a DR environment, on both global clusters, or on the primary while synchronization is paused.
KubernetesVersionSupportedVerifies: The target release supports the Kubernetes version of the cluster.
Fix: Contact technical support.
ClusterRunningVerifies: The cluster status is Running.
Fix: Repair the cluster until its status is Running. The message shows the current status.
DockerRuntimeUnsupportedVerifies: No node uses the Docker runtime.
Fix: Migrate the affected nodes to containerd.
ClusterModuleStableVerifies: ACP Core of the cluster is not being upgraded and has not failed to deploy.
Fix:
Status Upgrading or empty: run the check again later; if it does not change, contact technical support.
Status DeployFailed or UpgradeFailed: follow Diagnose a Module That Does Not Advance.
ModuleInfoStableVerifies: ACP Core and the installed Extensions (Operators and Cluster Plugins) with the Aligned lifecycle are stable.
Fix: Follow the module rows of Check the Registry, Modules, and etcd Backup; for a workload cluster, use its name instead of global in the moduleinfo command.
InfraClusterVersionPreflight and InfraClusterVersionPreflight/<check>Verifies: Alauda OS: the Kubernetes upgrade can run. Each provider check that fails or reports a warning has its own row, named after the check.
Fix: See Kubernetes Upgrade Checks.
ResourcePatchConflictVerifies: No other ResourcePatch targets a resource that the upgrade patches itself. The message names the conflicting patches and their targets.
Fix: An exemption does not clear this check. Review each named patch with the technical support contact who supplied it, then delete it before the upgrade.
ClusterVersionUpgradeableVerifies: The request does not change the target to a patch version of the minor version that a running upgrade moves to.
Fix: Reason MinorVersionClusterUpdateInProgress or MajorVersionClusterUpdateInProgress: let the running upgrade to the version in the message complete, then request the patch version. If that upgrade cannot complete, contact technical support.
ControlPlaneStaticPodsPresentVerifies: Every control plane node runs the static Pods etcd-<node-name>, kube-apiserver-<node-name>, kube-controller-manager-<node-name>, and kube-scheduler-<node-name> in kube-system.
Fix: Restore the static Pods that the message names: on the node, check the manifests in /etc/kubernetes/manifests and read journalctl -u kubelet. If they cannot be restored, contact technical support.
CustomEtcdBackupCronJobsAbsentVerifies: Non-Alauda OS clusters created by ACP: cpaas-system has no CronJob with the label app=etcd-backup other than etcd-backup-default.
Fix: Delete the CronJobs that the message names. etcd-backup-default keeps backing up etcd; see Check the etcd Backup of a Cluster.
CRIUpgradePodsAbsentVerifies: Non-Alauda OS clusters created by ACP: no Pod whose name starts with cri-upgrade remains in kube-system.
Fix: Delete the Pods that the message names. If one of them is still running, contact technical support.
AdminAckRequiredVerifies: global clusters only: every administrator gate is acknowledged.
Fix: See Handle Administrator Acknowledgement Gates.
DRSplitUpgradeWindowVerifies: Standby global cluster, identified by the DR marker kube-public/etcd-sync: synchronization is paused, the marker exists, no listed object is being deleted, and the token rotator is disabled and rolled out.
Fix: See DR Standby Checks.
Any other checkContact technical support with the check name, reason, and message.

Do not force an upgrade past a failed check. spec.desiredUpdate.force on the request turns every blocking failure of the whole preflight into a warning at once, not only the one you have reviewed; it releases nothing while any failure that cannot be bypassed remains. If a block cannot be resolved from this page, contact technical support.

DR Standby Checks

On a standby that has the DR marker, DRSplitUpgradeWindow passes only while synchronization is paused, the token rotator is disabled, and no object is being deleted, as in Upgrade the Standby global Cluster After the Workload Clusters. Without the marker, for example after the plugin is uninstalled in Upgrade global Clusters in a DR Environment, the check passes. In Upgrade global Clusters in a DR Environment, a failure on the standby before the uninstall that names only EtcdSyncNotPaused, TokenRotatorEnabled, or both is expected: continue as described there, and do not pause synchronization or disable the token rotator. For DeletingObjectsPresent, let the deletion finish on the primary, wait until the standby is synchronized, and run the preflight again. The table applies only to Upgrade the Standby global Cluster After the Workload Clusters. The message names the reason:

ReasonAction
EtcdSyncNotPausedBefore the pause, in Prepare Both global Clusters, this reason is expected; continue with the pause. After the pause: if the message says etcd-sync is not paused yet, wait until status.phase is Paused and run the preflight again; otherwise do not pause or resume synchronization yourself, and contact technical support.
DRMarkerMissingThe DR marker kube-public/etcd-sync was removed. Do not continue the standby upgrade, do not reinstall the plugin, and contact technical support.
DeletingObjectsPresentThe message lists objects that are being deleted on the standby. Before synchronization is paused, let the deletion finish on the primary, wait until the standby is synchronized, and run the preflight again. While synchronization is paused, changes on the primary do not reach the standby: contact technical support.
TokenRotatorEnabledBefore Prepare Both global Clusters, this reason is expected; continue to that step. After it: disable the token rotator on the standby as described in Prepare Both global Clusters, wait until its rollout finishes, and run the preflight again.
DRSplitUpgradeWindowNotReadyMore than one item failed. The message lists each failed item with its reason. Handle each of them as described in this table.
DependencyReadFailedThe check could not read the standby. Confirm that its API server responds, for example with ac get nodes, and run the preflight again; a submitted upgrade request retries the check automatically. If it does not pass, contact technical support.

Handle Administrator Acknowledgement Gates

AdminAckRequired runs only for global clusters. In a DR environment, record the acknowledgement on the primary and again on the standby global cluster; it is not synchronized. On the global cluster, read the gates supplied by the target release:

ac config use-cluster global
ac -n cpaas-system get configmap admin-gates -o yaml

Complete the action that the gate describes. On a global cluster on ACP 4.4 or earlier, AdminAckRequired reports Confirm Linux kernel 5.8+ and cgroup v2 readiness before upgrading to Kubernetes 1.35. (gate key ack-4.4-kubernetes-1.35-kernel-update); complete Check Node Readiness for Kubernetes 1.35 or Later for it. Then, on the same global cluster, record the acknowledgement with the key copied from admin-gates, and run the preflight again from the installer/ directory of the extracted target Core Package. On the standby, run export KUBECONFIG=<standby-kubeconfig> instead of ac config use-cluster global, and unset KUBECONFIG afterwards. Continue when Summary shows Passed or PassedWithWarnings:

ac config use-cluster global
ac whoami --show-server
ACK_KEY='<key-from-admin-gates>'
ac -n cpaas-system patch configmap admin-acks --type merge \
  -p "{\"data\":{\"${ACK_KEY}\":\"true\"}}"
bash upgrade.sh --preflight

Cluster Plugin Affinity Mismatch

When Upgrade stays InProgress and Ready shows the reason ClusterPluginAffinityMismatch, the labels of the cluster no longer match the cluster affinity of a Cluster Plugin installed on it, as defined in the target version. The message names the Cluster Plugins. No component of the cluster is upgraded until this is resolved; then the upgrade continues by itself. On the global cluster, read the labels of the cluster:

ac config use-cluster global
ac -n cpaas-system get clusters.cluster.x-k8s.io <cluster-name> --show-labels

If a label of the cluster was removed or changed, restore it. Otherwise, contact technical support with the message.

Kubernetes Upgrade on Alauda OS

On Alauda OS, the upgrade request also upgrades Kubernetes and Alauda OS. The platform tracks this in an InfraClusterVersion named after the cluster, on the global cluster that manages it; a global cluster has its own, named global.

Kubernetes Upgrade Checks

The platform checks the whole path to the target, including every intermediate Kubernetes version, and evaluates the checks again every 5 minutes. A fixed problem passes on the next evaluation. A message about one version on the path starts with hop <version>: , for example hop v1.36.4-acp.3: DCS VM template <vm-template-name> not found.

Each check in the tables below that fails or reports a warning has its own row in the upgrade preflight, named InfraClusterVersionPreflight/<check>, with the reason of the check; for example, InfraClusterVersionPreflight/NoHostPathWorkloads. When every check passes, the preflight shows only InfraClusterVersionPreflight.

  • Before the global upgrade installs the new provider, InfraClusterVersionPreflight is a non-blocking warning with the reason ProviderCapabilityNotDeclared. This is expected: the checks run again in the Kubernetes stage, after ACP Core is upgraded.
  • If a blocking check fails in the Kubernetes stage, the upgrade waits there and continues by itself once you fix the cause.

To read every check, run the commands below on the global cluster that manages the cluster. Replace <preflight-name> with the name at the start of the InfraClusterVersionPreflight/<check> message; in the list, it is the row whose TARGET is the target Kubernetes version:

ac -n cpaas-system get infraclusterversionpreflight -l upgrade.alauda.io/cluster=<cluster-name>
ac -n cpaas-system get infraclusterversionpreflight <preflight-name> \
  -o jsonpath='{range .status.checks[*]}{.name}{"\t"}{.severity}{"\t"}{.state}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere
CheckEffectDetails
UpgradePathResolvableBlocksVerifies: The target and every intermediate Kubernetes version have a VM template ConfigMap for each architecture that the cluster uses, and the cluster topology is supported. The ConfigMaps of the VM template that each node pool uses do not specify different architectures. The message lists the planned path.
Fix:
Reason ImageCatalogMiss or ArchitectureUnknown: see Image Catalog Problems.
Reason UnsupportedTopology: contact technical support.
TargetImagePresentBlocksVerifies: Every VM template on the path exists on DCS, and the DCS credentials work.
Fix: Register the missing VM template, or fix the DCS credential or API access.
RolloutMaxSurgeZeroBlocksVerifies: maxSurge is 0 on each node group (KubeadmControlPlane or MachineDeployment) whose IP pool declares persistent disks. An unset value counts as 1.
Fix: Set maxSurge: 0 as described in Set the Rollout Settings.
KubeletPatchReadyBlocksVerifies: When any version on the path is Kubernetes 1.35 or later, the KubeadmControlPlane and every worker KubeadmConfigTemplate contain a readable /etc/kubernetes/patches/kubeletconfiguration0+strategic.json, and every Secret and key referenced by contentFrom.secret exists.
Fix:
Reason KubeletPatchFileMissing or KubeletPatchInvalid: contact technical support.
Reason KubeletPatchSourceUnavailable or FileSecretMissing: restore the Secret or key named in the message; if you cannot, contact technical support.
ControlPlaneTopologyBlocksVerifies: The control plane has at least 3 nodes.
Fix: None. A cluster with fewer than three control plane nodes cannot be upgraded.
PersistentDiskLayoutBlocksVerifies: No DCSMachineTemplate declares /var/cpaas as a template disk.
Fix: Migrate the cluster as described in Migrate to Pool-Managed Persistent Disks.
PersistentDiskTemplateVersionBlocksVerifies: When the IP pool declares persistent disks, the current and target VM templates are made from Alauda OS of ACP 4.2.1 or later.
Fix: Contact technical support.
NoHostPathWorkloadsBlocksVerifies: No Pod keeps data on its node, where it is lost when the node is replaced: through a hostPath volume, or a PVC bound to a node-local PersistentVolume (a local or hostPath PersistentVolume, or a CSI volume whose data stays on the node's disk). Not counted: Pods in kube-system, kube-public, kube-node-lease, and cpaas-system; DaemonSet, static, and finished Pods; node paths such as /dev, /run, and /var/lib/kubelet; persistent disk paths declared for the node in DCSIpHostnamePool.
Fix:
Reason HostPathWorkloadsFound: move the data to network storage, or, if you have confirmed that it is safe or can be lost, exempt the volume. The message names up to 10 Pods and, when there are more, the total; after you handle them, the next evaluation names the rest.
Reason HostPathExemptionsInvalid: no exemption applies; correct icv-hostpath-exemptions as the message says.
Reason WorkloadClusterUnreachable: the check could not read the cluster; fix the cause in the message. If you cannot, contact technical support.
IPPoolCapacityWarningVerifies: Each IP pool has as many free entries as the maxSurge of the node groups that use it.
Fix: Set maxSurge to 0 as described in Set the Rollout Settings. With maxSurge: 0, the pool needs exactly one entry for each node; do not add free entries.
AllNodesReadyWarningVerifies: Every node of the cluster is Ready.
Fix: Repair the nodes, or confirm that the upgrade can proceed.
KubeletPatchPolicyWarningVerifies: imagePullCredentialsVerificationPolicy in the kubelet patch file is NeverVerify or absent. The platform adds it when absent and never changes an existing value.
Fix: Set the value to NeverVerify, or confirm the existing value.
ArchitectureInferredWarningVerifies: The architecture of every node pool is known from the VM template ConfigMaps of the VM template it uses now. If not, and the next Kubernetes version on the path is registered for one architecture only, the platform uses that architecture; the message names the node pools, the architecture, and the ConfigMaps.
Fix: Before you request the upgrade, confirm the architecture of each named node pool. If it is wrong, register the VM template that the node pool uses now with its architecture; see Image Catalog Problems.

Image Catalog Problems

On Huawei DCS and Huawei Cloud Stack, the platform picks the VM template or image of each Kubernetes version on the path from the ConfigMaps registered on global, separately for each architecture that the cluster uses. The architecture of a node pool comes from the ConfigMaps of the VM template or image it uses now. The reasons below appear in the check UpgradePathResolvable and as Stalled reasons; register or correct the ConfigMaps as described in VM Template ConfigMap or Image ConfigMap.

Reason and messageCauseWhat to do
ImageCatalogMiss, naming a version, node pools, and an architectureThat Kubernetes version has no ConfigMap for an architecture that the cluster uses.Register the VM template or image of that version for that architecture, as described in Stage the Alauda OS Images.
ImageCatalogMiss, saying that the version has x86_64 and aarch64 entries and that the architecture of the named node pools is not inferredThe VM template or image that the named node pools use now is not registered with an architecture, and the next version is registered for both architectures.Register a ConfigMap with architecture for the VM template or image that the message names, or set architecture on its existing ConfigMap.
ImageCatalogMiss, listing invalid catalog entriesThe listed ConfigMaps lack a required label or data key, or have an invalid value.Correct the listed ConfigMaps.
ArchitectureUnknownThe ConfigMaps of one VM template or image specify different architectures.Correct architecture on the wrong ConfigMap that the message names, or delete it.

Exempt a Confirmed Volume from NoHostPathWorkloads

When you have confirmed that the data on a reported volume can be lost or survives node replacement, exempt that volume. On the global cluster, create the ConfigMap icv-hostpath-exemptions in cpaas-system, for example with ac apply -f <file>.

apiVersion: v1
kind: ConfigMap
metadata:
  name: icv-hostpath-exemptions
  namespace: cpaas-system
data:
  rules.yaml: |
    - clusters: [<cluster-name>]
      namespace: rook-ceph
      pod: rook-ceph-crashcollector-*
      hostPath: /var/lib/rook/rook-ceph/crash
      reason: Crash dumps can be lost; confirmed by <name> on <date>
FieldRequiredMeaning
clustersYesCluster names, or "*" for all clusters.
namespaceYesThe namespace of the Pod, or "*" for all namespaces.
podNoA glob for the Pod name. By default the rule covers every Pod in the namespace.
hostPathOne of threeAn absolute path of a hostPath volume that the Pod defines itself; it also covers its subdirectories. / is not allowed. It does not match PersistentVolumes.
persistentVolumeOne of threeA glob for the name of a node-local PersistentVolume that the Pod claims.
csiDriverOne of threeThe exact CSI driver name of a PersistentVolume that the Pod claims, such as topolvm.io. It does not match local or hostPath PersistentVolumes.
reasonYesWhy the exemption is safe.

Set exactly one of hostPath, persistentVolume, and csiDriver in a rule.

Examples of rules in rules.yaml:

Rook logs
Application cache
TopoLVM volumes
PersistentVolumes by name

Logs and crash dumps of one Ceph cluster, on one cluster:

- clusters: [prod-a]
  namespace: rook-ceph
  hostPath: /var/lib/rook/rook-ceph/log
  reason: Ceph logs can be lost; confirmed by the storage owner
- clusters: [prod-a]
  namespace: rook-ceph
  hostPath: /var/lib/rook/rook-ceph/crash
  reason: Crash dumps can be lost; confirmed by the storage owner

A rule exempts only the matching volume; other volumes on the same Pod are still reported. The check is evaluated again within 5 minutes, and when it passes, its message states how many volumes were exempted. Read the message with the command in Kubernetes Upgrade Checks.

If any rule is invalid, none of the rules apply, and the check fails with the reason HostPathExemptionsInvalid, naming the rule, even when no volume is reported. A ConfigMap without the rules.yaml key, or with an unknown field, is invalid too. Provider upgrades do not change this ConfigMap.

Read the Progress

On the global cluster that manages the cluster, read the summary, the conditions, the planned versions, the current step, and the events:

ac -n cpaas-system get infraclusterversion <cluster-name>
ac -n cpaas-system get infraclusterversion <cluster-name> \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
ac -n cpaas-system get infraclusterversion <cluster-name> \
  -o jsonpath='{range .status.hops[*]}{.kubernetesVersion}{"\t"}{.phase}{"\n"}{end}{range .status.steps[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}'
ac -n cpaas-system get events --field-selector involvedObject.kind=InfraClusterVersion,involvedObject.name=<cluster-name>
NAME        PROVIDER     CURRENT         DESIRED         PHASE         HOLDER                     AGE
cluster-1   DCSCluster   v1.35.6-acp.9   v1.36.4-acp.3   Progressing   cluster-api-provider-dcs   128d

CURRENT advances after each intermediate Kubernetes version completes; DESIRED is the target. Until the provider takes a new target, PHASE is empty or Pending, or still shows Succeeded from an earlier upgrade. It then moves to Preparing (no node is replaced yet), Progressing (nodes are replaced one at a time; on Bare Metal, hosts are reinstalled), and Succeeded. There is no failed phase: a problem shows as Stalled=True, see Stalled Reasons. On Bare Metal, a worker node pool that is not replaced at an intermediate version shows the step phase Skipped.

The Kubernetes upgrade is complete when CURRENT equals DESIRED, PHASE is Succeeded, and Ready is True. The Ready message kube-proxy rollout pending also means complete: kube-proxy is still rolling out; check the kube-system/kube-proxy DaemonSet on the cluster. While the upgrade runs, do not edit the KubeadmControlPlane, the MachineDeployment resources, their templates, the VM template ConfigMaps (Huawei DCS and VMware vSphere), the image ConfigMaps (Huawei Cloud Stack), or elemental-image-catalog (Bare Metal).

Stalled Reasons

A problem shows as the InfraClusterVersion condition Stalled=True. Its message names the affected resources and the next action. In Preparing, no node has been replaced: fix the cause, and the platform retries automatically within 15 minutes. In Progressing, the upgrade waits where it is and continues as soon as the cause is cleared.

Huawei DCS
Bare Metal
Huawei Cloud Stack
VMware vSphere
ReasonMeaningWhat to do
A check reason, such as ImageCatalogMiss, ImageNotFound, UnsupportedTopology, or RolloutMaxSurgeNotZeroPlanning found the same problem as the check.Fix it as described in Kubernetes Upgrade Checks.
TemplateConflictA machine or bootstrap template that the upgrade creates already exists with a different spec. The message names it.If no KubeadmControlPlane or MachineDeployment of the cluster references the named template, delete it; the upgrade creates it again. Otherwise, contact technical support.
TemplateCreateFailedThe API server rejected a template that the upgrade creates. The message quotes the error.Fix the cause in the message, such as an admission webhook or a quota. Otherwise, contact technical support.
NodeJoinFailedA new VM has not joined the cluster 20 minutes after it became ready. A common cause is an etcd tag in the VM template ConfigMap that does not match the images in the VM template.On the VM, read journalctl -u kubeadm. If kubeadm failed in its own preflight, delete the named Machine; a new one is created. If kubeadm got past its preflight, confirm that the machine is not an etcd member before you delete it.
RolloutFailedA Machine failed, a new machine was not Running and healthy 60 minutes after it was created, or the control plane reported unhealthy components or etcd for more than 60 minutes.Inspect the named Machine and DCSMachine resources. For unhealthy control plane components, check the static Pods on the new VMs with crictl ps -a and journalctl -u kubelet.

If PHASE does not move to Preparing and the upgrade status reports InfraClusterVersionUnconsumed, check that the Deployment cluster-api-provider-dcs-icv-controller (Huawei DCS), cluster-api-provider-baremetal-icv-controller (Bare Metal), cluster-api-provider-hcs-icv-controller (Huawei Cloud Stack), or cluster-api-provider-vsphere-icv-controller (VMware vSphere) in cpaas-system on the global cluster is running.

Before you change a Cluster API resource of the cluster by hand during an upgrade, stop this InfraClusterVersion controller on the global cluster that manages the cluster, and start it again when you are done; otherwise it continues the upgrade while you work. On Bare Metal, this includes deleting a Machine.

WARNING

Stopping the InfraClusterVersion controller pauses the upgrade of every cluster of that provider on this global cluster. Start it again as soon as your change is done.

CONTROLLER=cluster-api-provider-dcs-icv-controller   # or the controller of the provider, as listed above
ac -n cpaas-system scale deployment "${CONTROLLER}" --replicas=0
# ... manual changes ...
ac -n cpaas-system scale deployment "${CONTROLLER}" --replicas=1

Free Space on COS_STATE (Bare Metal)

Each Bare Metal host needs at least 4 GiB free on its COS_STATE partition to install the new OS image. A host that runs short fails with no space left on device, and each failed attempt leaves a partial snapshot that reduces the free space further. Check every host of the affected pools, because any Available host can receive the new image. On each host, read the free space and the snapshots. Use btrfs, not df, because df does not count snapshots. Free (estimated) must be at least 4 GiB:

btrfs filesystem usage -h /
snapper --no-dbus list

Remove Partial Snapshots

A snapshot left by a failed attempt shows no in the default and active columns and update-in-progress=yes in its userdata. On the host, list the snapshots, then delete only snapshots that match all three conditions:

WARNING

Never delete a snapshot whose default or active column is yes. The host boots the default snapshot and runs the active one; deleting either leaves the host unable to boot.

snapper --no-dbus --csvout list --columns number,default,active,userdata
snapper --no-dbus delete --sync <number>

Run btrfs filesystem usage -h / again to confirm that the space was reclaimed. If the deletion fails, or the host still has less than 4 GiB free, the partition is too small. A host whose COS_STATE has the default 8 GiB size needs a reinstall from an ISO built as described in Create the Registration and the ISO, which has a larger partition. Reinstall such hosts before you upgrade the cluster; a reinstall replaces the node. If a host cannot be reinstalled, contact technical support.

Hosts on a Different Image (Bare Metal)

If elemental-image-catalog changed during the upgrade, some reinstalled hosts can run another image than the catalog now lists. The upgrade still completes and reports a Warning event ImageMismatch that names the BaremetalMachine objects and both images. After the cluster's InfraClusterVersion is Succeeded, confirm that the catalog holds the image you want. Then, on the global cluster, start a new rollout of the affected node group; its hosts are reinstalled according to the rollout settings, from the image now in the catalog:

ac -n cpaas-system patch machinedeployment <md-name> --type merge \
  -p "{\"spec\":{\"rolloutAfter\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}}"
ac -n cpaas-system patch kubeadmcontrolplane <kcp-name> --type merge \
  -p "{\"spec\":{\"rolloutAfter\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}}"

Node Replacement Symptoms (Bare Metal)

SymptomWhat to check
The rollout stops at the first new nodeThe ImageResolved condition of the new BaremetalMachine: the key is in elemental-image-catalog, and the OS image can be pulled from the registry of the cluster; see Provider Conditions.
A host reboots but never joins the clusterMachineInventory.status.plan.state (Failed is final). Read the elemental upgrade errors on the host's serial console, and check that the registry of the cluster is reachable from the host.
A new control-plane node fails kubeadm joinThe control-plane VIP is reachable. Check ac -n kube-system get lease on the cluster being upgraded, and, on its control-plane nodes, ipvsadm -Ln and the keepalived logs of the static Pod.

vSphere Rollout Holds That Are Not Failures

Some pauses during a vSphere rollout protect data. During such a hold, the InfraClusterVersion stays Progressing, its Ready message names the hold, and the upgrade continues by itself; no action is needed. Do not clear a hold by hand.

  • Attachment-safety hold. While an old VM still has the slot's persistent disks attached, the provider keeps the deletion of the VSphereVM pending. A rollout that seems stuck at the deletion of the first old machine is often this hold. Do not remove the finalizer.
  • Reclaimed is a tombstone. Do not delete a phase: Reclaimed entry to unstick a rollout.
  • SlotAvailable=False is expected. With maxSurge: 0 and a fully allocated pool, no slot is free until an old machine is deleted.
  • Power state. A node that is powered off in vCenter stays off until you power it on there. A replacement VM is powered on, even if the node it replaces was off before the upgrade.

On the global cluster, read the disk handoff and the pending deletions. The normal handoff is Attached, then Available once the old VM is gone, then Attached again on the replacement:

ac -n <namespace> get vspheremachineconfigpool <pool-name> \
  -o jsonpath='{range .status.persistentDiskStatuses[*]}{.hostname}{" "}{.name}{" "}{.phase}{" owner="}{.ownerMachineName}{" err="}{.lastError}{"\n"}{end}'
ac -n <namespace> get vspherevm -o custom-columns='NAME:.metadata.name,DELETED:.metadata.deletionTimestamp,FINALIZERS:.metadata.finalizers'

Kube-OVN and CoreDNS Takeover Is Blocked

On an Alauda OS cluster upgraded from a version earlier than v4.5, the platform takes over Kube-OVN and CoreDNS after the cluster and global both run v4.5. On the global cluster that manages the cluster, read the state from the cluster resource of its provider, dcscluster, baremetalcluster, hcscluster, or vspherecluster:

ac -n cpaas-system get <provider-cluster-resource> <cluster-name> \
  -o jsonpath='{range .status.conditions[?(@.type=="CoreAddonsHandedOff")]}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
  • True: the platform manages Kube-OVN and CoreDNS.
  • False with CoreAddonsHandoffPending or CoreDNSSkipPhasePending: the takeover is waiting, for example for a node rollout to finish. Check again later; if it does not change, contact technical support.
  • False with CoreAddonsHandoffNotApplicable: the cluster is not on v4.5 yet; check again after its upgrade completes. If it still reports this reason then, contact technical support.
  • False with CoreAddonsHandoffBlocked: the message names the setting. Contact technical support.
  • No output: the cluster does not use Kube-OVN, and there is nothing to take over.

While the takeover is blocked, the cluster does not pass validation.

Recover a Missing Aligned Extension Package

Use this section when the target ProductManifest is not Ready after you deploy CVO, or when the upgrade status reports that an installed platform-aligned component has no ready ModulePluginConfig. A missing package does not change what is installed now. CVO can report it as Ready=False or Reconciling=True instead of Stalled=True.

On the global cluster, list the artifacts of the target release that are not Ready. Entries for optional components, for Extensions with the Agnostic lifecycle, or for components that are not installed can legitimately appear; compare the output with the installed Core and Aligned Extensions:

ac get productmanifest v<target-version> -o json | jq -r '
  .status.artifacts[] as $artifact
  | $artifact.channels[]
  | select(.artifactStatus != "Ready")
  | [$artifact.name, .channel, .tag, .artifactStatus]
  | @tsv'

Aligned Cluster Plugins

artifactStatus: Absent means that the matching ModulePluginConfig is not fully Ready; it does not prove that the image is missing from the registry. On the global cluster, inspect the ModulePluginConfig named in the message, or identified by the component name and channel tag from the output above:

ac get modulepluginconfig <modulepluginconfig-name> \
  -o jsonpath='{.spec.image}{"\n"}{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
EvidenceRecovery
The named ModulePluginConfig does not existInspect the conditions of the target ProductManifest. A missing object does not by itself prove that the package is absent.
ResolveDigestFailed reports manifest unknown or not found, or the registry confirms that .spec.image is absentPublish the exact target package again, as described below.
An authentication (unauthorized, denied), certificate (x509), DNS, routing, timeout, or connection errorCorrect the registry credentials, the certificate trust, or the network access from the component that resolves the image.
LoadModulePluginFailedInspect the package contents and module-plugin.yaml. Publish a corrected package only after you confirm that the package is invalid.
  • For a plugin of the Upgrade from ACP 4.3 or earlier scenario, copy its package into the plugins/ directory of the target Core Package, then publish again as described in Publish Core and the Staged Packages.
  • For another Aligned Cluster Plugin, publish the target package with violet push to the global cluster whose upgrade reports it.

Aligned Operators

Aligned Operators do not use ModulePluginConfig. On the affected cluster, inspect the Subscription, the target InstallPlan, and the catalog; on the global cluster, inspect the ModuleInfo:

ac config use-cluster <cluster-name>   # the affected cluster
ac get subscription -A
ac get installplan -A
ac config use-cluster global
ac get moduleinfo -l cpaas.io/cluster-name=<cluster-name>

If the target Operator package is missing, publish it to every cluster where the Operator is installed; for one workload cluster, use violet push with --clusters "<cluster-name>". If the package exists, resolve the condition that CVO reports. Then watch the existing request with ac adm upgrade status --cluster=<cluster-name>; CVO retries automatically.

Diagnose a Module That Does Not Advance

On the global cluster, list the modules of the cluster, then read the conditions of the one that does not advance. Inspect the resource named in the condition and its Events; for an image pull failure, check registry access, CA trust, pull credentials, and the referenced manifest from the affected node:

ac get moduleinfo -l cpaas.io/cluster-name=<cluster-name> \
  -o custom-columns='MODULE:.metadata.labels.cpaas\.io/module-name,CURRENT:.status.version,TARGET:.spec.version,AVAILABLE:.status.availableVersions[*].version,PHASE:.status.phase'
ac get moduleinfo <moduleinfo-name> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

machine-config-daemon Does Not Start

Use this section when an upgrade stays in the component stage and Kubernetes node replacement has not started. machine-config is not ready. machine-config-daemon is CrashLoopBackOff. The Pod log contains:

failed to re-exec: writing /rootfs/usr/local/bin/machine-config-daemon: open /rootfs/usr/local/bin/.machine-config-daemon: read-only file system

Handle one node at a time. Do not restart the node, delete a snapshot, or submit the upgrade request again. The overlay is stored on /run and disappears after any node restart. If the node restarts unexpectedly and /usr/local/bin is still read-only, mount the overlay again, then delete that node's machine-config-daemon Pod so it is recreated.

On the affected node, as root, mount a writable overlay on /usr/local/bin:

mkdir -p -m 0755 /run/usr-local-bin/upper /run/usr-local-bin/work
if command -v getenforce >/dev/null 2>&1 && [ "$(getenforce)" != "Disabled" ]; then
  chcon -t bin_t /run/usr-local-bin/upper
fi
if ! findmnt -M /usr/local/bin -no FSTYPE,OPTIONS | grep -Eq '^overlay rw(,|$)'; then
  mount -t overlay overlay \
    -o lowerdir=/usr/local/bin,upperdir=/run/usr-local-bin/upper,workdir=/run/usr-local-bin/work \
    /usr/local/bin
fi

On the cluster being upgraded, delete only that node's daemon Pod. Wait until the new Pod is Running and the node stays Ready:

ac config use-cluster <cluster-name>
ac -n cpaas-system delete pod -l app=machine-config-daemon --field-selector spec.nodeName=<node-name>
ac -n cpaas-system get pod -l app=machine-config-daemon -o wide
ac get node <node-name>
ac config use-cluster global

Repeat for every node whose log matches. While any machine-config-daemon Pod is CrashLoopBackOff, continue to watch the existing request:

ac config use-cluster global
ac adm upgrade status --cluster=<cluster-name> --verbose

Global Cluster Disaster Recovery Does Not Synchronize

Use this section when the Pods of the Alauda Container Platform Global Cluster Disaster Recovery plugin are not Running, or the /check command does not report localMissedKeys [0] and localSurplusKeys [0]. Run every command on the standby global cluster: export its kubeconfig, and confirm that ac whoami --show-server prints the VIP of the standby. The etcd-sync-bootstrap Job runs before the synchronization starts: check the Job and the credentials it creates (if the Job failed, its log names the failing step), then the synchronization Pods and the current leader:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server
ac get job -n cpaas-system etcd-sync-bootstrap
ac logs -n cpaas-system job/etcd-sync-bootstrap
ac get secret -n cpaas-system remote-etcd-ca remote-etcd-client
ac get issuer -n cpaas-system remote-etcd-issuer
ac get certificate -n cpaas-system remote-etcd-client
ac get po -n cpaas-system -l app=etcd-sync
ac get lease -n cpaas-system etcd-sync-mirror
leader_pod=$(ac get lease -n cpaas-system etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}')
ac logs -n cpaas-system "$leader_pod" | grep -E "Acquired leader lease|Start Sync update"
unset KUBECONFIG

In the /check output, the keys under localSurplusKeys (lines starting with +) exist on the standby but not on the primary; review them with your operations team before you delete them. The keys under localMissedKeys (lines starting with -) exist on the primary but not on the standby; this often resolves after you recreate the current leader Pod. Recreate it also when resources with ownerReference dependencies must be synchronized again. Wait until Start Sync update appears in its log, then run on the standby:

export KUBECONFIG=<standby-kubeconfig>
leader_pod=$(ac get lease -n cpaas-system etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}')
ac delete po -n cpaas-system "$leader_pod"
unset KUBECONFIG

Run the /check command again. If it still does not report localMissedKeys [0] and localSurplusKeys [0], or a Pod is not Running, contact technical support.

Collect Evidence Before Escalation

Record the source and target ACP versions and the affected ModuleInfo or ModulePluginConfig. Do not include tokens, registry passwords, kubeconfig files, or Secret contents. On the global cluster that manages the cluster, also save the upgrade state and, on Alauda OS, the Kubernetes upgrade state:

ac config use-cluster global
ac adm upgrade status --cluster=<cluster-name> --verbose > upgrade-status.txt
ac -n cpaas-system get infraclusterversion <cluster-name> -o yaml > infraclusterversion.yaml
ac -n cpaas-system get infraclusterversionpreflight -l upgrade.alauda.io/cluster=<cluster-name> -o yaml > infraclusterversionpreflight.yaml
ac -n cpaas-system get events --sort-by=.lastTimestamp > events.txt