Manage Nodes on VMware vSphere
This document explains how to manage worker nodes on VMware vSphere after the baseline cluster is running. Node lifecycle operations are managed through VSphereMachineConfigPool, VSphereMachineTemplate, KubeadmConfigTemplate, and MachineDeployment resources.
To add another worker node group, see Add a worker node group.
Prerequisites
Before you begin, ensure the following conditions are met:
- The workload cluster was created successfully. See Create a Cluster on VMware vSphere.
- The worker machine config pool has enough available slots.
- The control plane is healthy and reachable.
- You know which manifest files currently define the worker nodes.
- You know the rollout and pool rules in Provider Requirements.
- To manage the nodes of the
globalcluster itself, the provider must be installed onglobalfirst; see Manage Nodes on Alauda OS Clusters.
A Day-2 apply can be rejected
The rollout and pool rules apply to every edit this page makes:
- Re-applying an archived
MachineDeploymentmanifest re-submits the whole object, including an apply whose only intended change isreplicas. Before you apply it, confirm the file setsstrategy.rollingUpdate.maxSurge: 0and amaxUnavailableof at least1, or the apply is rejected.strategy.type: OnDeleteneeds neither. - A
KubeadmControlPlanemust setspec.replicasto1, or to3or more withspec.rolloutStrategy.rollingUpdate.maxSurge: 0, or an edit to it is rejected. - A slot whose
status.configStatuses[].stateisInUseorReleasedcannot be removed, cannot have itsnetwork.primary.ipor.ipv6changed, and cannot lose a disk frompersistentDisks[]or change one'ssizeGiBormountPath. Adding a slot is accepted. Do not add a disk to such a slot; see Allocated slot restrictions.
Match a rejection message against Admission rejections.
ac adm alpha cluster create and ac adm alpha global create only create a cluster. To change a running cluster, change its objects as described on this page; do not change install-config.yaml and run create again. ac adm alpha cluster destroy deletes only the objects that ac rendered, so delete the templates that you created yourself.
Steps
Scale out worker nodes
When you add more worker nodes, update the worker machine config pool before you increase the replica count.
Worker scale-out depends on the relationship between MachineDeployment.spec.replicas and the available node slots in VSphereMachineConfigPool.spec.configs[]. The number of slots can be greater than replicas, and idle slots do not affect a running cluster. If replicas exceeds the number of available slots, CAPV cannot assign the new workers correctly.
- Add one or more new node slots to the worker
VSphereMachineConfigPool,<cluster_name>-<group>for a cluster created withac. - Update
replicasin theMachineDeploymentof the same name. - Add the slots first, then raise the replicas, as shown below.
The following example adds a worker slot with the required data disks:
The example uses the disk layout of a cluster created on provider v1.0.17. Match fsFormat to the existing slots: ac formats the standard disks as xfs. A pool created on an earlier provider has no ephemeralDisks[] and declares /var/cpaas, /var/lib/containerd, and /var/lib/etcd in persistentDisks[]; match the slots that pool already has instead of introducing ephemeralDisks[] into it.
To attach an additional raw disk without formatting or mounting, omit mountPath and fsFormat:
The raw disk is available inside the guest OS at /dev/disk/by-capv/app-data. On rolling updates, the same VMDK is re-attached to the replacement VM and the symlink is recreated. The application remains responsible for the disk contents and runtime mount.
Then increase the worker replicas:
Use the following order:
Note: Keep the same existing VSphereMachineConfigPool.metadata.name when scale-out or replacement must preserve the logical pool. Slots and VMDKs are not reused across differently named pool objects. releaseDelayHours is the reclaim grace period for an unused Released slot, not a delay before same-pool reuse.
Add a worker node group
Add workers that have their own slots, VM specification, or placement.
In the following steps, <group> is an existing worker node group and <new_group> is the new one, a DNS-1123 label that no other group uses. control-plane is reserved.
-
Read the values of an existing group:
The first command prints
<k8s_version>,<machine_template_name>, and<bootstrap_template_name>. -
Write the worker objects:
- Give the pool at least
replicasslots. Hostnames and primary IP addresses must be unique across every pool of the cluster, and eachMachineDeploymentneeds its own pool; see Provider Requirements. - Take
templatefrom<machine_template_name>, so that the VM template matches<k8s_version>. Takeserver,folder, andresourcePoolfrom it as well. - Keep
maxSurge: 0: a rollout with the Cluster API default stalls on a full pool. - When an existing object has fields that differ from the example, use its values: for example,
fsFormatand the disks of a slot, a second NIC, or a failure domain. When<bootstrap_template_name>differs fromspec.template.specabove, use its values too. For example, on Kubernetes 1.35 and later the kubelet configuration patch needs"imagePullCredentialsVerificationPolicy": "NeverVerify"; earlier versions do not accept this field.
- Give the pool at least
-
Apply the objects and monitor the group:
Continue when every
Machineof the group isRunning; then confirm that the nodes areReadyon the cluster. Match a rejection message against Admission rejections.
Roll out updated worker node configuration
When you need to change worker VM specifications, create a new VSphereMachineTemplate and update the MachineDeployment to reference it. This triggers a rolling update that replaces worker nodes with the new configuration.
Typical changes include:
- VM template name (
spec.template.spec.template) - CPU or memory sizing (
numCPUs,memoryMiB) - System disk sizing (
diskGiB), on a full clone only: a linked clone ignores it - VM network devices (
network.devices)
In these steps, change only the machine specification. Do not change spec.version of the KubeadmControlPlane or MachineDeployment, and do not switch to a VM template of another Kubernetes version.
Persistent disks are declared in VSphereMachineConfigPool slots rather than in VSphereMachineTemplate, so a new template does not change them. A slot that is InUse or Released rejects changing an existing disk's sizeGiB or mountPath, so a different geometry for an existing disk needs a new slot, not an edit of the current one. See Allocated slot restrictions.
-
Export the existing template
-
Modify the template
Edit
new-worker-template.yaml:- Set
metadata.nameto a new unique name (for example,<cluster_name>-worker-v2) - Update the desired specification fields
- Remove server-generated fields:
metadata.resourceVersion,metadata.uid,metadata.generation,metadata.creationTimestamp,metadata.managedFields,metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"], andstatus
- Set
-
Apply the new template
-
Update the MachineDeployment reference
If you also need to change bootstrap settings, see Updating Bootstrap Templates below.
-
Monitor the rolling update
To change the machine specification of control plane nodes, use the same steps and patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name instead.
Rolling back a failed update
If the rolling update fails (for example, new VMs fail to boot), revert the MachineDeployment reference back to the previous template name. This works only while the Kubernetes minor version is unchanged. The old template still exists and Cluster API will roll back to it.
Verify worker node status
Run the following commands to verify the management-cluster and workload-cluster status:
Confirm the following results:
- The target worker replica count is reached.
- Every new worker node joins the cluster.
- The nodes eventually become
Ready.
Updating Bootstrap Templates
To update bootstrap configuration, create a new KubeadmConfigTemplate and update the MachineDeployment reference.
-
Export the existing template
-
Modify the template
Edit
new-bootstrap-template.yaml:- Set
metadata.nameto a new unique name (for example,<cluster_name>-worker-config-v2) - Update the desired bootstrap configuration fields
- For Kubernetes 1.35 or later, add the required kubelet patch settings to
/etc/kubernetes/patches/kubeletconfiguration0+strategic.json - Remove the same server-generated fields listed in Roll out updated worker node configuration step 2
- Set
-
Apply the new template
-
Update the MachineDeployment reference
The Cluster API controller triggers a rolling update. Existing machines continue using the old bootstrap configuration until they are replaced.
Runtime Topology Changes
Networking, placement, and disk topology are part of the machine templates and machine config pool. Treat a change to these fields as a rollout of replacement machines, not as an in-place VM edit:
- Add the required slot definitions to the existing
VSphereMachineConfigPoolbefore increasing replicas or starting replacement. An allocated slot rejects some changes; see the warning in Prerequisites. - Create a new
VSphereMachineTemplatewhen its VM, network, placement, or system-disk fields change. - Create a new
KubeadmConfigTemplatewhen bootstrap files or node configuration change. - Update the
MachineDeploymentreferences only after the new pool capacity and templates exist. - Monitor replacement Machines and confirm that every node becomes
Readybefore removing old templates.
For the exact second-NIC and persistent-disk fields, use Optional Topology in Create a Cluster on VMware vSphere. Preserve the same cross-resource relationships during Day-2 changes:
- Keep
VSphereMachineConfigPool.spec.configs[].network.additional, andVSphereMachineTemplate.spec.template.spec.network.devicesconsistent when NIC topology changes. - Do not remove the required
var-cpaaspersistent disk, or the requiredvar-lib-kubeletandvar-lib-containerdephemeral disks, or the control-planevar-lib-etcdephemeral disk. - Keep a new slot's disk lists identical to the slots already in that pool.
ephemeralDisks[]exists from providerv1.0.17, but a pool created on an earlier provider declares/var/cpaas,/var/lib/containerd, and/var/lib/etcdinpersistentDisks[]; do not convert those slots, and do not mix layouts inside one pool. - Do not reshape or add disks to an allocated slot. A slot that is
InUseorReleasedrejects removing a persistent disk or changing itssizeGiBormountPath. Resizing means planning a new slot. - Do not treat
machineNamingStrategy,wipeFilesystem, orreleaseDelayHoursas a switch that moves disks between pools or accelerates datastore cleanup.
When you expand from one NIC to two NICs, update all of the following fields together:
- Append the second NIC entry to
VSphereMachineConfigPool.spec.configs[].network.additional; keepnetwork.primaryunchanged. - Add the second device to
VSphereMachineTemplate.spec.template.spec.network.devicesin the new template.
When you revert from two NICs to one NIC, remove the corresponding second entry from both fields. Leave network.additional empty or remove the key entirely, and roll out a new VSphereMachineTemplate; do not edit existing VMs directly.
Troubleshooting
Use the following checks first when worker node management fails:
- Check
VSphereMachineconditions forMachineConfigPoolReady. IfFalse, the reason indicates why slot allocation failed:PoolBoundToOtherConsumer: the pool is already bound to a differentKubeadmControlPlaneorMachineDeployment.NoAvailableSlots: no slot matches the required datacenter.
- Verify that the worker machine config pool still has free slots.
- Verify that the worker IP addresses, gateway, and DNS settings are correct.
- Verify that the worker VM template still matches the required Kubernetes version and guest-tools requirements.
- Check
VSphereVM.status.addresseswhen a node is waiting for IP allocation.
Next Steps
For cluster creation and persistent-disk diagnostics, see Create a Cluster on VMware vSphere and its Persistent disk lifecycle.