Troubleshooting Huawei DCS Workload Clusters

Start from the preflight error or the stage that ac adm alpha cluster status reports. For failures that every provider shares, see Troubleshooting. Run the ac commands on this page from a machine where ac is logged in to the platform, after ac config use-cluster global.

SymptomAction
Preflight reports that the DCS infrastructure provider or the kubeadm control plane provider is not installedInstall them; see Providers.
Preflight reports that the credential cannot sign inCheck that authKey is the plain-text password and site the bare site ID; see DCS Credential Secret. Correct the Secret before you run create again: repeated failed sign-ins lock the DCS account.
Preflight reports a missing template, folder, switch, port group, or datastoreCorrect the name in install-config.yaml; for a port group, check that it belongs to the named switch.
[1/4] CAPI Cluster stays NotFoundCheck the cluster name: ac get clusters.cluster.x-k8s.io -n cpaas-system.
[2/4] KubeadmControlPlane does not become readyRun ac -n cpaas-system describe machine <name> and ac -n cpaas-system describe dcsmachine <name>, and check the credential, the template name, the capacity, the folder, and the port group.
A DCSMachine stays in Provisioning while the DCS VM stays in creatingSee VM Stuck in creating.
The control plane does not come upFor external, check that the load balancer forwards 6443 to the control plane nodes; for internal, see Self-built VIP.
ControlPlaneHAReady is not TrueSee Control Plane HA.
[3/4] MachineDeployment shows the wrong node groupsCheck computes[].name and the hosts of each group.
[4/4] ClusterView does not become readyCheck the provider log: ac -n cpaas-system logs deployment/cluster-api-provider-dcs-manager --tail=50.
The cluster reaches Provisioned but never becomes Ready, for example nodes stay NotReady because the CNI is not deployedSee Troubleshoot a Workload Cluster Stuck in Provisioned.
A DCSMachine stays in DeletingSee DCSMachine Stuck in Deleting.
The Cluster stays in Deleting after its machines are goneSee Troubleshoot a Cluster Stuck in Deleting.

Self-built VIP

ResultAction
type is not internal, or the address is not your VIPContact technical support.
No alive ModuleInfo, or the alive AppRelease is not synced and healthyContact technical support.
alive Pods are not Ready, or the cluster cannot be reached through the VIPCheck that the DCS port group allows VIP ownership, gratuitous ARP, and VRRP traffic, that the VIP is not a node address, and that vrid is unique in the Layer 2 domain; see Self-built VIP. If they are, contact technical support.

Control Plane HA

The Control Plane HA check prints the status, the reason, and a message:

ResultAction
True ControlPlaneHAReadyNone.
False ControlPlaneHAPendingRead the message. For placement, check DCS host capacity, DRS status, and whether anything blocks migration, such as an attached CD-ROM; the provider does not run DRS, so run it on the DCS side if the placement does not converge. To find the VMs on DCS, list the members: ac -n cpaas-system get dcscluster <cluster-name> -o jsonpath='{range .status.controlPlaneHA.members[*]}{.machineName}{" "}{.vmUrn}{"\n"}{end}'.
False ControlPlaneHAFailedRead the message, then check the DCS credential, the DRS settings, and rule-name conflicts on DCS. If they are correct, contact technical support.

VM Stuck in creating

A DCSMachine stays in Provisioning with status.cdRomFile set, so the sign-in and the ISO upload succeeded, while the DCS VM stays in creating with no host and its deploy task does not progress. The DCS scheduler cannot place the VM.

  1. In the DCS portal, check the free space of the datastores that the node group uses and the load of the hosts. A nearly full datastore, or hosts that are all heavily overcommitted, block placement; see DCS Objects, Capacity, and Addresses.
  2. Ask the DCS platform team to free space or rebalance the hosts. The DCS scheduler, not the provider, picks the host of each VM.
  3. If the deploy task does not respond to cancellation or to the VM deletion that the provider issues, a DCS administrator cancels it in the portal. Where DCS serializes VM deployment, one stuck task stalls every later one, so clear the first one.

DCSMachine Stuck in Deleting

Use this section when a DCSMachine stays in Deleting and blocks a scale-down, a rolling upgrade, or the deletion of the cluster. Before it deletes a VM, the provider stops it and detaches its persistent disks, and DCS must complete each step.

Read the VMStopPending condition of the DCSMachine:

ac config use-cluster global
ac -n cpaas-system describe dcsmachine <name>
ResultAction
VMStopPending with reason WaitingForStopWait. If the VM stays in stopping, see VM Stuck in stopping.
VMStopPending with reason VMMidTransitionRead the VM status in the message; see VM Stuck in Mid-Transition.
No VMStopPending condition, and the VM is still listed in the DCS portalSee Controller Cannot Reach DCS API.
No VMStopPending condition, and the VM is gone from DCSThe stop is complete; the deletion is held elsewhere, for example by a Cluster API drain. See the Cluster API troubleshooting documentation or contact technical support.

VM Stuck in stopping

The provider issued a safe stop, and the VM does not reach stopped. In the DCS portal, open the VM detail page:

ResultAction
pvDriverStatus is not running, or the guest OS is hungStop the VM with Force from the DCS portal (VM detail → Operations → Stop → Force). The provider then continues the deletion.
The VM was cloned from a template older than 4.2.1, whose guest tools cannot take a safe stop. Take the template name from vmTemplateName of the DCSMachine; the cpaas.io/distribution-version label of the VM template ConfigMap whose cpaas.io/dcs-vm-template label is that name shows its versionContact technical support.

VM Stuck in Mid-Transition

The provider waits until the VM is running or stopped:

Status in the messageAction
migratingWait; the deletion continues when the migration ends.
startingWait until running; the deletion then continues.
paused or hibernatedResume the VM from the DCS portal (Operations → Resume). If it cannot be resumed, contact technical support.

Controller Cannot Reach DCS API

Read the provider log:

ac config use-cluster global
ac -n cpaas-system logs deployment/cluster-api-provider-dcs-manager --tail=50
ResultAction
connection refused or timeoutCheck the paths from the global cluster nodes to DCS; see Network Connectivity.
errorCode: 10100116The DCS account is locked; see DCS Account Lockout.
401 UnauthorizedUpdate the credential Secret as described in DCS Credential Secret.

DCS Account Lockout

errorCode: 10100116 means that DCS has locked the account of the credential Secret.

  1. If the password changed, update the credential Secret as described in DCS Credential Secret: while it is wrong, the provider's sign-ins keep the account locked.
  2. Contact technical support to recover from the lockout.

Do not use the DCS portal admin account in the credential Secret: a lockout then locks everyone out. Use a dedicated account, as described in DCS Credential Secret.

FC Site Policy: errorCode 102212808

On a DCS site whose FusionCompute platform deletes only stopped VMs, the provider log can show DeleteVm rejected by FC site policy with errorCode=102212808. The provider stops the VM again and retries on its own.

ResultAction
The error appears once, and the Machine is then deletedNone.
The error repeatsSomething restarts the VM on DCS, such as an operator or another automation. Stop it. If the error still repeats, contact technical support.

Verify Recovery

ac config use-cluster global
ac -n cpaas-system get machines.cluster.x-k8s.io -l cluster.x-k8s.io/cluster-name=<cluster-name>

Continue when the Machine is no longer listed and the VM is gone from the DCS portal; otherwise contact technical support.