Troubleshoot a Cluster Stuck in Deleting

Use this guide when a workload cluster that ACP created is being deleted, its machines and their VMs or hosts are already gone, but the Cluster resource on the global cluster stays in Phase: Deleting. It applies to any ACP-built workload cluster:

  • Huawei DCS (DCS Provider)
  • Huawei Cloud Stack (HCS Provider)
  • VMware vSphere (vSphere Provider)
  • Bare Metal (Bare Metal Provider)
  • Clusters created through the SSH UPI path managed by cluster-manager

This page does not cover:

Symptoms

Where to lookWhat you see
Cluster.status.phase on the global clusterDeleting, with no progress.
Cluster.metadata.finalizersStill contains capi.cpaas.io/imported.
Cluster.spec.infrastructureRef and Cluster.spec.controlPlaneRefBoth are set, for example DCSCluster and KubeadmControlPlane. A cluster onboarded as a Third-party Cluster has neither.
The provider Machine resources, the infrastructure cluster, KubeadmControlPlane, MachineDeployment, and Machine resources of the clusterNone remain.
The IaaS platform (DCS portal, HCS portal, vCenter, or the bare-metal inventory)None of the VMs of the cluster remain, or, on Bare Metal, the hosts have left the cluster.

Diagnosis

Check the Finalizer and the Cluster Type

ac config use-cluster global
ac get clusters.cluster.x-k8s.io <cluster-name> -n <namespace> \
  -o jsonpath='finalizers={.metadata.finalizers} infrastructureRef={.spec.infrastructureRef.kind} controlPlaneRef={.spec.controlPlaneRef.kind}{"\n"}'

<namespace> is the namespace of the cluster objects; for a cluster created with ac, it is cpaas-system.

ResultAction
finalizers contains capi.cpaas.io/imported, and both references are setContinue with the next check.
Both references are emptyThe cluster was onboarded as a Third-party Cluster; this page does not apply.
finalizers does not contain capi.cpaas.io/importedThis page does not apply. Contact technical support.

Check That No Cluster Resources Remain

ac config use-cluster global
ac get machines.cluster.x-k8s.io,machinedeployments,kubeadmcontrolplanes -n <namespace>
ac get dcsmachines,dcsclusters -n <namespace>          # Huawei DCS
ac get hcsmachines,hcsclusters -n <namespace>          # Huawei Cloud Stack
ac get vspheremachines,vsphereclusters -n <namespace>  # VMware vSphere
ac get baremetalmachines.infrastructure.cluster.x-k8s.io,baremetalclusters.infrastructure.cluster.x-k8s.io -n <namespace>  # Bare Metal

Run the first command and the line of your provider. Then confirm on the IaaS platform that none of the VMs of the cluster remain, or, on Bare Metal, that the hosts have been released from the cluster.

ResultAction
No resource of this cluster is listed, and no VM of the cluster remainsContact technical support. Provide the output of both checks.
A resource or VM of the cluster remainsThe deletion is still in progress or stopped on that resource. See the Troubleshooting page of your provider.
WARNING

Do not remove the finalizer from the Cluster resource yourself. Contact technical support.

See Also