Troubleshooting VMware vSphere Workload Clusters

Start from the stage that ac adm alpha cluster status reports. For failures that are common to all providers, see Troubleshooting.

Creation Stages

SymptomWhere to look first
create reports a vCenter object that it cannot findFix the name in install-config.yaml and run create again. ac reports only one missing object per run. Names given as inventory paths (they contain /) are checked only when the VMs are cloned.
[1/4] CAPI Cluster does not advanceThe Cluster and VSphereCluster conditions.
[2/4] KubeadmControlPlane does not advanceThe KubeadmControlPlane, Machine, VSphereMachine, and VSphereVM conditions, and the pool conditions and status.configStatuses.
The control plane does not come upThe endpoint and the registry: the health of the load balancer or VIP, DNS, the registry API, the ACP port rules, and whether the global cluster and the nodes can reach them.
Nodes stay uninitializedThe CPI delivery: the ClusterResourceSetBinding and the CPI Pods in the workload cluster.
[3/4] MachineDeployment does not advanceThe MachineDeployment rollout, the VM status, the CPI, and the CNI.
[4/4] ClusterView does not advanceThe platform ClusterView, the registry API, and the provider manager log.

Collect the state first. Run the following commands from a machine where ac is logged in to the platform:

ac config use-cluster global
ac -n cpaas-system describe clusters.cluster.x-k8s.io <cluster-name>
ac -n cpaas-system describe vspherecluster <cluster-name>
ac -n cpaas-system describe kubeadmcontrolplane <cluster-name>-control-plane
ac -n cpaas-system describe machinedeployment <cluster-name>-<group>
ac -n cpaas-system get clusters.cluster.x-k8s.io,vspherecluster,kubeadmcontrolplane,machinedeployment,machine,vspheremachine,vspherevm
ac -n cpaas-system logs deploy/capi-controller-manager

If the network plugin is not installed, also inspect the logs of the platform network controller and cluster lifecycle controller.

Common Causes

ResultAction
The CPI resources are not deliveredCheck that the ClusterResourceSet and its ClusterResourceSetBinding exist, as in CPI and Pools. If the ClusterResourceSet exists and nothing is applied, contact technical support.
A ClusterResourceSet exists, but no ClusterResourceSetBinding is createdCheck that the controller has delete permission on the referenced ConfigMap and Secret.
The network plugin is not installedCheck that the cluster annotations are present and that the platform controllers processed them.
The cpaas.io/registry-address annotation is missing or wrongSet it to the permanent <registry_address>, without a scheme or /v2/. Then check cpaas-system/public-registry-credential.
A machine is stuck in ProvisioningRead the MachineConfigPoolReady condition of the VSphereMachine. Then read the pool conditions MembersValid, MembersUnique, PersistentDisksReady, ClusterRefReady, and VCenterAvailable.
A machine never bootstraps; BootstrapReady is False with BootstrapSecretGetFailed or BootstrapSecretContentInvalidInspect the KubeadmConfig of that Machine and its bootstrap data Secret before you look at vCenter.
A node was powered off in vCenter and stays offPower the VM on in vCenter. The provider does not power on a VM that was powered off after its first boot. You do not need a rollout.
VSphereMachine clears Ready without an obvious causeRead the VSphereVM conditions. Ready is cleared while the VM, its providerID, or its network information is unavailable.
A VM waits for an IP addressCheck VMware Tools, the static IP settings, and VSphereVM.status.addresses.
Workload Node objects have no spec.providerIDCheck the CPI delivery first. Then look for an old VM with the same guest hostname in the same datacenter. Fix or remove the duplicate, and then restart the CPI Pods as shown below this table.
The datastore runs out of spaceList every VSphereMachineConfigPool, including terminating ones. Each pool owns its own persistent disks, so repeated deployments accumulate sets of disks. Read status.configStatuses[], status.persistentDiskStatuses[] (phase, lastError, retryAfter, taskRef), and the pool Events. Do not delete a VMDK, remove a pool finalizer, or edit status.persistentDiskStatuses[] by hand. If reclaim does not progress, contact technical support. See Persistent disk lifecycle.
The system disk keeps the template's size although rootDisk.sizeGiB is setThe VM is a linked clone, and rootDisk.sizeGiB takes effect only when the template has no snapshot (a full clone). See the global cluster template.
The control plane endpoint does not come upCheck TCP 6443 passthrough, all control plane backends, HTTPS /healthz, DNS and certificate SANs, and reachability from the global cluster and the control plane nodes.
The TLS connection to vCenter failsCheck the thumbprint, the vCenter address, and any proxy settings.

After you fix a duplicate hostname, restart the CPI Pods:

ac config sync
ac config use-cluster <cluster-name>
ac -n kube-system delete pod -l k8s-app=vsphere-cloud-controller-manager
ac config use-cluster global

Self-built VIP

Read the SelfBuiltLoadBalancerReady condition on the VSphereCluster, as in Verify the Self-built VIP. Then find the result in the following table:

ResultAction
False, reason SelfBuiltLoadBalancerReconcilingThe VIP is still being set up. Check again later.
False, reason SelfBuiltLoadBalancerNotReadyCheck the Alive Pods in kube-system and the alive AppRelease in cpaas-system of the workload cluster. Also check that the VIP is in the control plane Layer 2 domain, that the vrid is unique, and that VRRP and gratuitous ARP are allowed.
False, reason InvalidSelfBuiltLoadBalancerConfigurationThe loadBalancer values are invalid. You cannot change them on an existing cluster. Delete the cluster and create it again with corrected values.
healthz through the VIP does not print ok, and the cluster cannot be reached through the VIPCheck that the VIP is in the control plane Layer 2 domain, that the vrid is unique, and that VRRP and gratuitous ARP are allowed. Then check the Alive Pods.
No Alive Pods and no alive AppReleaseCheck the Alive plugin on the global cluster with the commands below.
ac config use-cluster global
ac get clustermodule global -o jsonpath='{.spec.version}{"\n"}'
ac get moduleplugin alive -o jsonpath='{.status.targetClusterVersions}{"\n"}'
ac get moduleplugin alive -o jsonpath='{.status.latestVersion}{"\n"}'
ac get moduleconfig alive-<alive-version> -o jsonpath='{.status.readyForDeploy}{"\n"}'

Replace <alive-version> with the targetClusterVersions entry for the version that the first command prints. If there is no such entry, use latestVersion. Continue when the last command prints true. Otherwise, the Core Package on this platform is incomplete; import the complete Core Package of your platform release.

Admission Rejections

The provider can reject an apply or edit before anything is created. Find a fragment of the rejection message in the first column of the following table. For the rules, see Provider Requirements.

Rejection containsCause and fix
maxSurge must be 0 when the infrastructure template references a machineConfigPoolRefSet maxSurge to the integer 0 on the KubeadmControlPlane or MachineDeployment. If you leave it unset, the Cluster API default 1 applies.
maxUnavailable must be at least 1Set maxUnavailable to 1 or more on the MachineDeployment. If you leave it unset, the Cluster API default 0 applies.
must have 1 or at least 3 replicasThe control plane has 2 replicas. Use 1, or 3 or more with maxSurge: 0.
also used by poolA hostname or primary IP is already used by a slot in another pool that is bound to the same Cluster in this namespace.
allocated slotThe slot is InUse or Released. The message names what is frozen: the primary IP or IPv6, or the sizeGiB or mountPath of a persistent disk. Or it reports that you cannot remove the slot or one of its persistent disks. See Allocated slot restrictions.
cannot delete pool whileThe pool still owns InUse slots. Delete the Machines that hold them first.
is already referenced by or is bound toTwo controllers reference one pool. Give each KubeadmControlPlane and MachineDeployment its own pool.
cannot change clusterRef while consumerRef is setThe pool is already bound to a controller. Create a new pool instead of changing the cluster that this pool refers to.
VSphereMachineTemplate spec.template.spec field is immutableCreate a new template under a new name instead of editing the existing one. See Roll out updated worker node configuration.
is immutable once controlPlaneLoadBalancer is setspec.controlPlaneLoadBalancer is write-once. See controlPlaneLoadBalancer fields.
cannot be set after the cluster is createdThe cluster was created without spec.controlPlaneLoadBalancer. Leaving the field out already means an external LoadBalancer. The endpoint is fixed when the cluster is created, so you cannot add the field later.