Manage Nodes on Bare Metal
This document explains how to scale worker nodes up and down, replace specific machines, and recover failed nodes on the bare-metal provider. Node management uses Cluster API Machine resources, which a MachineDeployment orchestrates. The provider binds each Machine to one MachineInventory from a pool and drives the host through the clean and reprovision plans.
TOC
PrerequisitesOverviewNode Management OperationsRegistering Hosts for the global ClusterScaling Worker NodesAdding Worker NodesRemoving Worker NodesAdding a Worker Node GroupReplacing a Single Failed NodeMoving Workers to a Different PoolUpdating Bootstrap TemplatesPool and Inventory ObservabilityCommon Failure ModesPrerequisites
Important Prerequisites
- The control plane must already be running. See Create Cluster.
- The worker
MachineInventoryPoolmust have enough free inventories (status.available) for the nodes that you add. WithmaxSurge: 0, a rollout needs no extra hosts. Take the new hosts from the available hosts of the cluster's OS. - The target
Machine.spec.versionmust be a key in theelemental-image-catalogConfigMap for the architecture and OS of the hosts. If the key is missing, add it as described in Add the Image Catalog Keys. - To manage the nodes of the
globalcluster itself, the provider must run onglobal. Aglobalcluster installed withacalready runs it; otherwise install it as described in Providers. Register its hosts as described in Registering Hosts for the global Cluster.
Configuration Guidelines
When you edit the configurations in this document:
- Change only the values enclosed in
<>brackets. - Replace placeholder values with the settings of your environment.
- Keep all other default configurations unless a step explicitly requires a change.
Overview
A worker node group consists of four resources:
MachineInventoryPool(<cluster-name>-worker-pool) — the set ofMachineInventorynames that the group may use. You created it in Create the Pools.BaremetalMachineTemplate(<cluster-name>-worker) — points to the worker pool. To move the workers to a different pool, see Moving Workers to a Different Pool.KubeadmConfigTemplate(<cluster-name>-worker-config) — the cloud-inituser-dataforkubeadm join. Each time the provider reinstalls a host, it sets the hostname,provider-id, andcriSocketin this user-data. Do not fill in those fields yourself.MachineDeployment— sets the replica count, the version, and the rollout strategy.
The names above belong to a node group named worker, as in the install-config.yaml example. For a group with another name, replace worker with that name in the commands on this page. To add a worker node group, see Adding a Worker Node Group.
ac adm alpha cluster create and ac adm alpha global create only create a cluster. To change a running cluster, change its objects as described on this page. Do not change install-config.yaml and run create again. ac adm alpha cluster destroy deletes only the objects that ac rendered. When you destroy the cluster, also delete the templates that you created yourself.
Node Management Operations
Registering Hosts for the global Cluster
Use Case: Add hosts to the global cluster, to scale it out or to replace a node. These hosts need a registration and an ISO of their own. The ISO built during the installation registers hosts with the bootstrap node, which no longer exists. The registration of workload cluster hosts does not connect a host to the global cluster.
Procedure:
-
Select the
globalclusterRun
ac config use-cluster global. In a disaster recovery pair, register the host on theglobalcluster that it joins. For the standby, runexport KUBECONFIG=<standby-kubeconfig>instead, andunset KUBECONFIGwhen you finish. -
Write the registration
Generate the Registration file for the OS and architecture of the hosts as described in Prepare Alauda OS for Hosts. Then change two things:
-
Rename it to a name that starts with
global-, for exampleglobal-alaudaos-m55-amd64. Replace the name inmetadata.name,machineName, theSeedImagename, and itsregistrationRef. A registration with any other name gives no registration URL. -
Add these annotations to the
MachineRegistration.<control-plane-vip>is the control plane VIP of theglobalcluster that the host joins:
-
-
Build the ISO
Run Create the registration and build the ISO with the new name, as described in Create the Registration and the ISO. If a
global-registration for this OS and architecture already exists, skip steps 2 and 3 and use its ISO. -
Register the hosts
Prepare, boot, and confirm the hosts as described in Register Hosts. They register as
global-*. Then add them to a pool as described in the sections below.
Scaling Worker Nodes
Adding Worker Nodes
Use Case: Increase cluster capacity.
Prerequisites:
- The cluster has a worker node group. A cluster created without
computesininstall-config.yamlhas none, and the commands below reportNotFound. In that case, add the workers as described in Adding a Worker Node Group. - The worker pool has enough free inventories (
status.available). If it does not, append more available hosts toMachineInventoryPool.spec.machineInventories[]. When there are no more available hosts, first register additional hosts as described in Register Hosts.
Procedure:
-
Check current state
Confirm that
MachineInventoryPool.status.availableis high enough for the nodes that you plan to add. -
Extend the worker pool when more inventories are needed
Append the names of the new
MachineInventoryobjects. They must already be registered and Ready. Save and exit, then confirm thatstatus.availableincreased by the number of names that you appended. -
Scale up the
MachineDeployment -
Monitor
New
BaremetalMachineobjects advance throughPending → Allocated → Reprovisioning → Running. Theavailablecounter of the pool decreases as new nodes are bound.
Removing Worker Nodes
Two strategies are supported. Both work the same way as in upstream Cluster API:
Inventory recycling
A clean plan stops kubelet, clears the CRI workload, and stops containerd. It does not wipe the Kubernetes persistent directories or reset the OS. The OS is reinstalled later: when the same or a different inventory is picked for a new BaremetalMachine, it runs the reprovision plan, which writes the OS image to the system disk again and does not wipe the disks of the host. In the meantime, the inventory returns to Available and can be allocated again.
Random Removal
CAPI selects the machines to delete in its standard order. For each selected machine, the provider writes a clean plan and sets the baremetal.alauda.io/allocation-state annotation of its MachineInventory to Preparing. When the plan reports Applied, the annotation returns to Available.
Targeted Removal
-
Identify the machines
-
Annotate the target machines
Repeat for every machine you want to remove.
-
Scale down by exactly the number of annotated machines
If you reduce the replica count by fewer than the number of annotated machines, some annotated machines stay in place. If you reduce it by more, random machines also go through the
cleanplan. -
Verify cleanup
The released inventory must show
baremetal.alauda.io/allocation-state=Available. Itsbaremetal.alauda.io/owner-*annotations must be cleared, and its pool annotation must remain. TheMachineInventoryitself is not deleted.
Adding a Worker Node Group
Use Case: Add workers that draw from their own pool. For the global cluster, <cluster-name> is global, and its hosts are registered as described in Registering Hosts for the global Cluster.
In the following steps, <group> is an existing worker node group, and <new-group> is the name of the new one. The new name must be a DNS-1123 label that no other group uses. The name control-plane is reserved.
Procedure:
-
Register the hosts and create the pool
Make sure that enough hosts are available; see Register Hosts. Then create the pool
<cluster-name>-<new-group>-poolas in Create the Pools. Each node group has its own pool. The pool must hold at leastreplicashosts. -
Read the values of an existing group
The first command prints
<kubernetes-version>and<bootstrap-template-name>.If the cluster has no worker node group yet, read the values from the control plane instead:
Use
spec.versionas<kubernetes-version>, and the keys inspec.kubeadmConfigSpec.users[].sshAuthorizedKeysas<ssh-public-key>. Ifspec.kubeadmConfigSpec.filescontains/etc/chrony.d/ntp-servers.conf, the cluster usescluster.ntpServers. Copy that file intofilesof the newKubeadmConfigTemplate, and addpreKubeadmCommandswithsystemctl restart chronyd.service. -
Write the worker objects
versionmust be the version of the existing group.- Keep
maxSurge: 0. Each old node is then removed before its replacement is created, so a rollout needs no spare host. The platform upgrade does not start while a node group has anothermaxSurge. An unset value counts as1. - If a field in
<bootstrap-template-name>differs fromspec.template.specabove, use the value from that template instead. For example, on Kubernetes 1.35 and later, the kubelet configuration patch needs"imagePullCredentialsVerificationPolicy": "NeverVerify". Earlier versions do not accept this field. - Leave out the fields that the provider sets itself: the hostname,
kubeletExtraArgs.provider-id, andnodeRegistration.criSocket.
-
Apply the objects and monitor the group
Continue when every
Machineof the group isRunning. Then confirm that the nodes areReadyon the cluster. If not, see Provider Conditions.
Replacing a Single Failed Node
If a BaremetalMachine ends up in Failed, the safest recovery is to delete the failed Machine. CAPI immediately creates a replacement Machine, because replicas did not change. The bare-metal provider then picks an Available inventory from the same pool. The replacement is usually a different MachineInventory. The provider does not guarantee that it picks the inventory that was just released.
Procedure:
-
Record why the machine failed
Before you delete the
Machine, record why it failed, because the deletion replaces the plan and clears its output.-
Read
BaremetalMachine.status.conditionsandMachineInventory.status.plan.state. The common root causes are listed in Common Failure Modes below. -
The plan Secret is on the
globalcluster, incpaas-system. Its name is inBaremetalMachine.status.planSecretRef.name. Itsfailed-outputkey holds output only on Alauda OS KU hosts. There, it holds the output of the check that the host has enough free disk space for the new OS image. The other steps save no output in the key. To read the key, run:The first command prints
<plan-secret>.<baremetalmachine-name>is the name inspec.infrastructureRef.nameof the failedMachine. Ifgzipreportsunexpected end of file, the key is empty. The command needs jq 1.6 or later. The second command prints the name of each step that saved its output, followed by that output. -
For any other failure, run
journalctl -u elemental-system-agenton the host. It shows the output of each step of the plan.
-
-
Delete the failed
MachineFor a node of the
globalcluster, the replacement host must be registered for theglobalcluster; see Registering Hosts for the global Cluster.
Moving Workers to a Different Pool
BaremetalMachineTemplate carries only the pool reference. It declares no CPU, memory, or disk.
Use Case: You registered new hosts, put them in a new MachineInventoryPool, and want the nodes to move onto them. Create a new BaremetalMachineTemplate that points to the new pool, then point the MachineDeployment at it. Nodes are replaced one by one.
Before you start:
-
The target pool is in the same namespace as the template.
-
The target pool has at least as many
Availableinventories as theMachineDeploymenthas replicas. Each moved node uses one host of the target pool. The hosts that the old nodes release return to the old pool, not to the target pool. Read the count fromstatus.available: -
The new hosts meet the same prerequisites as the current hosts: they are available (registered and ready), and the CNI network device is present. See Pool Capacity.
-
Keep the old template and the old pool until the move is finished.
Procedure:
-
Create a new
BaremetalMachineTemplatewith a newmetadata.namethat references the target pool. Use the format in Worker Objects. -
Apply the new template.
-
Patch the
MachineDeployment: -
Watch the rolling replacement (
maxSurge: 0, one node at a time) and the target pool'sstatus.available:
What to expect:
- The old node is deleted before its replacement is created. If the target pool runs out of hosts, the node pool stays one node short until you add hosts.
- Node names and IP addresses change.
- Node-local data is not carried over.
If the move stalls with no free host (BaremetalMachine shows InventoryAllocated=False / Reason=PoolExhausted), add Available hosts to the target pool. The provider picks them up automatically. Alternatively, roll back by patching infrastructureRef.name back to the old template. The nodes that were already moved are replaced again, one by one, so the old pool must still have enough Available hosts.
To move control plane nodes, use the same steps and patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name instead. Nodes are replaced one at a time. In addition:
- With
maxSurge: 0,replicasmust be 3 or more. - During each replacement, only two etcd members run. Start only when the control plane is healthy. Keep 3
Availablehosts ready in the target pool. - The new hosts must be in the same layer-2 network as the control plane VIP.
After the move is finished, you can delete the old template and the old pool. Hosts released by the old nodes stay in the old pool.
Updating Bootstrap Templates
Modifying an existing KubeadmConfigTemplate in place does not roll the change out to existing machines. Only newly created machines pick up the change.
To roll out a bootstrap change:
-
Export the existing template:
-
Change
metadata.name. Remove the server-generated fields (resourceVersion,uid,creationTimestamp,managedFields,kubectl.kubernetes.io/last-applied-configuration) and the entirestatus. Then edit the fields that you want to change. For Kubernetes 1.35 or later, add the required kubelet patch settings to/etc/kubernetes/patches/kubeletconfiguration0+strategic.json. -
Apply the new template:
-
Patch the
MachineDeploymentto reference the new template:This triggers a rolling replacement.
Pool and Inventory Observability
The annotations that the provider maintains on each MachineInventory show what each host is being used for right now. They are the authoritative source:
The default plan secret of each inventory also carries baremetal.alauda.io/plan.type. The normal node lifecycle uses the clean and reprovision plan types.
Common Failure Modes
For the phases, see Provider Overview → BaremetalMachine Phases.