Troubleshooting

This page covers the installation progress and the failures that every provider shares. For failures specific to one provider, see the troubleshooting page of that provider.

Check the Progress

To see the state of all eight stages at once, run this command from another terminal on the bootstrap node:

<extract-dir>/res/<arch>/ac adm alpha global status

status changes nothing. It works only while the bootstrap cluster exists. It exits with a non-zero code when a stage has Failed. It cannot confirm stage 8. create confirms that stage.

logs/ac.log in the installation directory records every run. If create stops, run the same command again. Work that is already complete is skipped. See The create Command.

Installation Stages

#StageWhat happensDone when
1MiniAlaudaReadyac checks the registry and starts the bootstrap cluster on the bootstrap node.The bootstrap cluster and the platform installer are running.
2ProvidersReadyac installs the Cluster Bootstrap Provider and the infrastructure provider into the bootstrap cluster.Both providers are Ready.
3ClusterAcceptedac applies the rendered manifests.The cluster objects exist and none reports a failure.
4MachinesProvisionedThe infrastructure provider creates the machines.Every node group has its expected number of machines, all Running.
5ControlPlaneReadyThe control plane is initialized.The control plane endpoint accepts connections.
6PlatformInstalledThe platform is installed on the global cluster.The installer reports success.
7HandoffVerifiedac checks the global cluster after the handover.All nodes are Ready, the platform workloads are running, and the provider resources are on the global cluster.
8Finalizedac installs both providers on the global cluster, writes output/kubeconfig, collects the installer log, and removes the bootstrap cluster.The providers are Ready on the global cluster, the kubeconfig works, and the bootstrap cluster is gone.

Some providers add checks to stages 7 and 8. The troubleshooting pages of those providers list the checks.

Common Failures

Run the commands in the following table on the bootstrap node, unless the row says otherwise. Commands that target the bootstrap cluster use its kubeconfig, for example:

<extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig -n cpaas-system get pods

The installation removes /var/cpaas/data/alauda.kubeconfig some time after the platform is installed. If the file is missing, wait for create to stop, then run create again. It exports the file again and continues the installation.

ResultAction
setup.sh exits with an error in its preflight checksThe host is unchanged. Fix the cause it names, then run create again.
setup.sh exits with an error after its preflight checksIn the directory of setup.sh, run bash cleanup.sh -y, then run create again. cleanup.sh -y reboots the host and deletes /root/.kube.
create reports conflict with prepared input when it submits the installationglobal.registry (the address or the credentials) changed after the bootstrap cluster started. From that point on, the value is fixed, and the installation cannot continue with a new value. Restore the original values and run create again. To use the new values instead, run bash cleanup.sh -y in the directory of setup.sh, then run create again.
Stage 2 does not advanceCheck the provider releases and Pods with <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig get apprelease -A and <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig get pods -A. If an image or chart cannot be pulled, upload the payload again as described in Prepare the Target-Version Payload, then run create again. Otherwise, contact technical support.
Stage 4 does not advanceRun <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig -n cpaas-system describe machine <name>, then see the provider's troubleshooting page.
Stage 5 does not advanceIf the control plane endpoint is an external load balancer, check its health check first. On a host in the network from which the load balancer sends its health checks, run curl -kfsS https://<control-plane-node-ip>:6443/healthz, where <control-plane-node-ip> is the address of the first control plane node. If the output is not ok, correct the network path or firewall rules between that network and the control plane node. If the output is ok, or if you use a Self-built VIP, run <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig -n cpaas-system describe kubeadmcontrolplane global-control-plane and read its conditions. Then see the provider's troubleshooting page, or contact technical support.
Stage 5 shows readyReplicas: 0This is expected. The nodes stay NotReady until the platform installs the network plugin in stage 6. Continue.
Stage 6 stallsOn the bootstrap node, read the last lines and the latest error in /var/cpaas/data/installer.log, and fix the cause that the log names. The installer retries by itself. If the error does not name a cause that you can fix, contact technical support.
Stage 8 reports no running installer Podac removes the bootstrap cluster only after it has copied the installer log. Wait until <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig -n cpaas-system get pods shows the cpaas-installer Pod Running, then run create again. If the Pod does not return to Running, contact technical support.
Stage 8 waits for a providerOn a machine with ac (on the bootstrap node, <extract-dir>/res/<arch>/ac), run these commands in order: ac login https://<platform-address> --name <session-name> as the platform administrator, then ac config use-cluster global, then ac get moduleconfigs | grep cluster-api-provider. If both providers are listed, continue to wait. Otherwise, contact technical support.
Nodes cannot pull imagesCheck that the registry is reachable from the node network. If the payload is incomplete, upload it again as described in Prepare the Target-Version Payload.