Manage Nodes on Huawei DCS
This document explains how to manage worker nodes using Cluster API Machine resources.
TOC
PrerequisitesOverviewNode Management OperationsScaling Worker NodesAdding Worker NodesRemoving Worker NodesAdding a Worker Node GroupChanging Machine InfrastructureUpdating Bootstrap TemplatesTroubleshootingPrerequisites
Important Prerequisites
- The control plane must be deployed before performing node operations. See Create Cluster for setup instructions.
- Ensure you have proper access to the DCS platform and required permissions.
- To manage the nodes of the
globalcluster itself, the provider must be installed onglobalfirst; see Manage Nodes on Alauda OS Clusters.
Configuration Guidelines When working with the configurations in this document:
- Only modify values enclosed in
<>brackets - Replace placeholder values with your environment-specific settings
- Preserve all other default configurations unless explicitly required
- A cluster created with
acnames theMachineDeploymentand theDCSIpHostnamePoolof a node group<cluster-name>-<group>, where<group>iscomputes[].name. A cluster not created withacmay use other names; read them from itsMachineDeploymentresources
Overview
Worker nodes are managed through Cluster API Machine resources, providing declarative and automated node lifecycle management. A worker node pool is built from four resources:
- IP-Hostname Pool Configuration - Network settings for worker nodes
- Machine Template Setup - VM specifications
- Bootstrap Configuration - Node initialization and join settings
- Machine Deployment - Orchestration of node creation and management
ac adm alpha cluster create and ac adm alpha global create only create a cluster. To change a running cluster, change its objects as described on this page; do not change install-config.yaml and run create again. ac adm alpha cluster destroy deletes only the objects that ac rendered, so delete the templates that you created yourself.
To add another worker node group, see Adding a Worker Node Group.
Node Management Operations
This section covers common operational tasks for managing worker nodes, including scaling, machine infrastructure changes, and bootstrap template updates.
Cluster API Framework Node management operations are based on the Cluster API framework. For detailed information, refer to the official Cluster API documentation.
Scaling Worker Nodes
Worker node scaling allows you to adjust cluster capacity based on workload demands. The Cluster API manages the node lifecycle automatically through the MachineDeployment resource.
Adding Worker Nodes
Increase the number of worker nodes to handle increased workload or add new capacity.
Use Case: Scale up cluster to add more compute resources
Prerequisites:
- Verify the IP pool has sufficient available IP addresses for new nodes
- Ensure the DCS platform has adequate resources to provision new VMs
Procedure:
-
Check Current Node Status
View the current machines in the cluster:
-
Extend IP Pool
Before scaling up, add new IP configurations to the pool for the additional nodes.
INFOIP Pool Expansion The IP pool must contain at least as many entries as the desired replica count. Add new IP entries for each additional worker node you plan to deploy.
Add IP entries to the pool:
First, export the current pool configuration to preserve existing entries:
Then use the following command to add new IP configurations. The
poolarray must include all existing entries plus the new entries:WARNINGImportant Notes
- The
poolarray must include all existing entries plus the new entries you want to add - Copy the existing entries from the exported YAML to avoid data loss
ac patch --type='merge'replaces the entirespec.poolarray, so copy every existingpersistentDiskblock unchanged, includingisThinif it is present, unless you are intentionally adding new disks- Ensure each new entry has unique
ip,hostname, andmachineNamevalues - If new worker nodes also need the platform-required
/var/cpaasdisk, declare it in each new entry'spersistentDisk - Network parameters (
mask,gateway,dns) typically match existing entries
Example: Adding 2 new nodes to an existing pool of 3 nodes
- The
-
Verify IP Pool Capacity
After extending the IP pool, verify it has sufficient entries for the desired replica count:
Check that the pool contains at least as many entries as the desired replica count.
-
Scale Up the MachineDeployment
Update the
replicasfield to the desired number of nodes:Example: Scale from 3 to 5 nodes
-
Monitor the Scaling Progress
Watch the machine creation process:
The Cluster API controller will automatically create new machines based on the MachineDeployment template.
-
Verify Nodes Joined the Cluster
Verify the new nodes:
The new nodes should appear in the list and transition to
Readystatus.
Rolling Update Behavior When scaling up, new nodes are created immediately without affecting existing nodes. This ensures zero-downtime scaling.
Removing Worker Nodes
Decrease the number of worker nodes to reduce cluster capacity or remove underutilized resources. The Cluster API supports two removal strategies:
- Random removal: Reduce replicas, the platform randomly selects and deletes machines
- Targeted removal: Mark specific machines for deletion, then reduce replicas (recommended for IP recovery)
IP Recovery Scenario When you need to recycle specific machine IPs (e.g., for reassignment or IP pool management), use the targeted removal method. The deletion annotation ensures the platform deletes the marked machines, not random ones.
Data Loss Warning Scaling down removes nodes and their associated VMs. Ensure:
- Workloads can tolerate node loss through proper replication
- No critical data is stored only on the nodes being removed
- Applications are designed for horizontal scaling
Declared persistent disks in DCSIpHostnamePool.spec.pool[].persistentDisk are not deleted just because a Machine is replaced. They remain available for reuse while the corresponding IP slot stays in the pool. Removing the IP slot from the pool, deleting the pool, or deleting the cluster can trigger persistent-volume cleanup.
Random Removal
Use Case: Scale down cluster where any node can be removed (no specific IP requirements)
Procedure:
-
Identify Current Machine Status
View the current machines in the MachineDeployment:
-
Scale Down the MachineDeployment
Update the
replicasfield to reduce the node count:Example: Scale from 5 to 3 nodes
The Cluster API controller will randomly select and delete machines to match the desired replica count.
-
Monitor the Removal Progress
Watch the machine deletion process:
The Cluster API controller will:
- Drain the selected nodes (evict pods if possible)
- Delete the underlying VMs from the DCS platform
- Remove the machine resources
-
Verify Nodes Removed
Verify the nodes:
The removed nodes should no longer appear in the list.
Targeted Removal
Use Case: Remove specific machines (e.g., for IP recovery, replace unhealthy nodes)
Procedure:
-
Identify Machines to Remove
View the current machines:
Note the
<machine-name>of the machines you want to remove. -
Annotate Machines for Deletion
Mark the specific machines for deletion:
Repeat for each machine you want to remove.
Example: Remove two specific machines
-
Scale Down the MachineDeployment
After annotating the machines, reduce the replica count:
INFOReplica Count Must Match Annotated Machines Reduce replicas by exactly the number of annotated machines.
- If you reduce by fewer, not all annotated machines will be removed
- If you reduce by more, additional machines will be randomly selected for deletion
Example: If you annotated 2 machines, reduce replicas by exactly 2 (e.g., from 5 to 3)
The platform will delete the annotated machines, not randomly selected ones.
-
Monitor the Removal Progress
Watch the machine deletion process:
-
Verify Nodes Removed
Verify the nodes:
The removed nodes should no longer appear in the list.
Adding a Worker Node Group
Add workers that have their own IP pool or VM specification.
In the following steps, <group> is an existing worker node group and <new-group> is the new one, a DNS-1123 label that no other group uses. control-plane is reserved.
Procedure:
-
Read the values of an existing group
The first command prints
<kubernetes-version>,<machine-template-name>, and<bootstrap-template-name>. -
Write the worker objects
- Give the pool at least
replicasentries, one per node, with hostnames and addresses that no other pool uses. Repeat thepersistentDiskentry for each host. - Take
vmTemplateNamefrom<machine-template-name>, so that the VM template matches<kubernetes-version>. TakedvSwitchName,portGroupName, andcdRomDatastoreNamefrom it as well, and leave out the ones it does not set. For the disk fields, see Disks. - When
<bootstrap-template-name>has fields that differ fromspec.template.specabove, use its values. For example, on Kubernetes 1.35 and later the kubelet configuration patch needs"imagePullCredentialsVerificationPolicy": "NeverVerify"; earlier versions do not accept this field.
- Give the pool at least
-
Apply the objects and monitor the group
Continue when every
Machineof the group isRunning; then confirm that the nodes areReadyon the cluster.
Changing Machine Infrastructure
To change the machine specification of worker nodes (CPU, memory, disk, VM template), create a new machine template, then point the MachineDeployment at it. The worker nodes are replaced one by one.
-
Create a New Machine Template
Export the
DCSMachineTemplatereferenced by yourMachineDeploymentto a new file:In the new file:
- Set
metadata.nameto a unique name - Modify the required values (
spec.template.spec.vmTemplateName,spec.template.spec.vmConfig.dcsMachineCpuSpec.quantity,spec.template.spec.vmConfig.dcsMachineMemorySpec.quantity,spec.template.spec.vmConfig.dcsMachineDiskSpec) - Remove the server-generated metadata (
resourceVersion,uid,generation,creationTimestamp,managedFields, thekubectl.kubernetes.io/last-applied-configurationannotation) and the entirestatusfield - Leave
spec.template.spec.providerIDunset
Keep pool-managed persistent disks, including
/var/cpaas, inDCSIpHostnamePool.spec.pool[].persistentDiskand additional NICs inDCSIpHostnamePool.spec.pool[].additionNic[]. UsedcsMachineDiskSpeconly for the system disk and template-local disks.Apply the new template:
- Set
-
Update Machine Deployment
Point the
MachineDeploymentat the new template: -
Monitor the Rolling Update
-
Worker nodes are replaced one by one with the new specification
-
Any disks declared in
DCSIpHostnamePool.spec.pool[].persistentDiskare detached from the old VM and reattached to the replacement VM when the replacement takes over the IP entry of the old VM, which is guaranteed only when the pool has no free entry. If the pool uses persistent disks, keepspec.strategy.rollingUpdate.maxSurgeat0on theMachineDeployment, and do not leave free entries in the pool -
Monitor the progress:
-
To change the machine specification of control plane nodes, use the same steps and patch KubeadmControlPlane.spec.machineTemplate.infrastructureRef.name instead.
In these steps, change only the machine specification. Do not change spec.version of the KubeadmControlPlane or MachineDeployment, and do not switch to a VM template of another Kubernetes version.
If you are migrating an existing cluster from the old template-disk layout to pool-managed persistent disks, follow Migrate to Pool-Managed Persistent Disks before you rely on data preservation during node replacement.
Updating Bootstrap Templates
Bootstrap templates (KubeadmConfigTemplate) are used by MachineDeployment and MachineSet resources. Changes to existing templates do not automatically trigger rollouts of existing machines; only new machines use the updated template.
Update Process:
-
Export Existing Template
-
Modify Configuration
- Update the desired fields in the exported YAML
- For Kubernetes 1.35 or later, add the required kubelet patch settings to
/etc/kubernetes/patches/kubeletconfiguration0+strategic.json - Change the
metadata.nameto a new unique name - Remove extraneous metadata fields (
resourceVersion,uid,creationTimestamp, etc.)
-
Create New Template
-
Update MachineDeployment
- Modify the MachineDeployment resource
- Update
spec.template.spec.bootstrap.configRef.nameto reference the new template - Apply the changes to trigger a rolling update
Template Rollout Behavior Existing machines continue using the old bootstrap configuration. Only newly created machines (during scaling or rolling updates) will use the updated template.
Troubleshooting
A new or replacement Machine that stays in Provisioning while its DCS VM stays in creating is a DCS placement issue; see VM Stuck in creating. For control plane spread across hosts, see Control Plane HA.