Create a Cluster on VMware vSphere

ACP CLI (ac) renders the Cluster API, vSphere, and CPI objects from one install-config.yaml, applies them to the global cluster, and reports the creation stages. Every object lives in the cpaas-system namespace of the global cluster. For how the provider objects fit together, see How It Works.

Prerequisites

  • You meet the VMware vSphere prerequisites.
  • ac is logged in to the platform as a platform administrator (ac login https://<platform-address> --name <session-name>).

Procedure

1. Write install-config.yaml

Create an installation directory for the cluster and write install-config.yaml in it. ac writes manifests/ and logs/ac.log in the same directory.

Copy the following template and fill in the placeholders. The table below the template describes the numbered lines, which are specific to VMware vSphere. For the other fields, see Common install-config.yaml Fields. For the objects that ac renders from this file, see Parameters.

install-config.yaml
cluster:
  clusterName: <cluster-name>
  sshKeys:
    - <ssh-public-key>
  # registrySecretRef:
  #   name: <registry-secret-name>
  #   namespace: cpaas-system
  # encryptionProviderConfig: <encryption-key>
  ntpServers: []
  network:
    podCIDR: 10.199.0.0/16
    serviceCIDR: 10.96.0.0/12
    kubeOVN:
      joinCIDR: 100.64.0.0/16
    loadBalancer:
      type: external
      address: <control-plane-endpoint>
      port: 6443
      # vrid: <vrid>
      # interface: <nic-name>

provider:
  vsphere:
    server: <vcenter-address>
    datacenter: <datacenter>
    template: <template-name-or-path>
    datastore: <datastore>
    # folder: <folder-path>
    computeResource: <cluster-or-host>
    # rootDisk:
    #   sizeGiB: <root-disk-gib>
    releaseDelayHours: 24
    network:
      name: <portgroup-name>
      # deviceName: <nic-name>

controlPlane:
  replicas: 3
  cpuCore: 8
  memoryGiB: 16
  # vsphere:
  #   datastore: <datastore>
  #   folder: <folder-path>
  #   computeResource: <cluster-or-host>
  #   rootDisk:
  #     sizeGiB: <root-disk-gib>
  network:
    gateway: <cp-gateway>
    dns: [<dns-ip>]
    mask: <prefix-length>
    additions: []
    # additions:
    #   - mask: <prefix-length>
    #     vsphere:
    #       name: <portgroup-name>
    #       deviceName: <nic-name>
  disks:
    ephemeral:
      - name: var-lib-containerd
        sizeGiB: 100
      - name: var-lib-kubelet
        sizeGiB: 100
      - name: var-lib-etcd
        sizeGiB: 20
    persistent:
      - name: var-cpaas
        sizeGiB: 100
    rawDisks: [] 
    # rawDisks:
    #   - name: data-001
    #     sizeGiB: 50
    # vsphere:
    #   datastore: <datastore>
    #   storagePolicy: <storage-policy>
  hosts:
    - hostname: <cp-0-hostname>
      ip: <cp-0-ip>
      additionalIps: []
    - hostname: <cp-1-hostname>
      ip: <cp-1-ip>
      additionalIps: []
    - hostname: <cp-2-hostname>
      ip: <cp-2-ip>
      additionalIps: []

computes:
  - name: worker
    replicas: 3
    cpuCore: 8
    memoryGiB: 16
    # vsphere:
    #   datastore: <datastore>
    #   folder: <folder-path>
    #   computeResource: <cluster-or-host>
    #   rootDisk:
    #     sizeGiB: <root-disk-gib>
    network:
      gateway: <worker-gateway>
      dns: [<dns-ip>]
      mask: <prefix-length>
      additions: []
      # additions:
      #   - mask: <prefix-length>
      #     vsphere:
      #       name: <portgroup-name>
      #       deviceName: <nic-name>
    disks:
      ephemeral:
        - name: var-lib-containerd
          sizeGiB: 100
        - name: var-lib-kubelet
          sizeGiB: 100
      persistent:
        - name: var-cpaas
          sizeGiB: 100
      rawDisks: []
      # rawDisks:
      #   - name: data-001
      #     sizeGiB: 50
      # vsphere:
      #   datastore: <datastore>
      #   storagePolicy: <storage-policy>
    hosts:
      - hostname: <worker-0-hostname>
        ip: <worker-0-ip>
        additionalIps: []
      - hostname: <worker-1-hostname>
        ip: <worker-1-ip>
        additionalIps: []
      - hostname: <worker-2-hostname>
        ip: <worker-2-ip>
        additionalIps: []

credentialRef:
  name: <cluster-name>-credential
  namespace: cpaas-system
#FieldDescription
cluster.networkRequired · objectThe cluster network.
1cluster.network.loadBalancerRequired · objectThe control plane endpoint. Choose type as described in Control Plane Endpoint. You cannot change it after the cluster is created. external, the default, uses your own load balancer. internal uses a Self-built VIP that the provider runs on the control plane nodes. address is the address of the load balancer or the VIP. It is an IPv4 address that is not a node address and is outside the Pod, Service, and join CIDRs. port defaults to 6443. With internal, also uncomment vrid, which is required. It is the VRRP router ID, from 1 to 255, and it must be unique in the control plane Layer 2 domain. You can also uncomment interface, the NIC that holds the VIP. It defaults to the NIC of the node IP. vipMode is not supported.
provider.vsphereRequired · objectThe VMware vSphere settings.
2provider.vsphere.serverRequired · stringThe vCenter address.
3provider.vsphere.datacenterRequired · stringThe datacenter, as a name or an inventory path. If you give a nested path, such as /folder/dc, you cannot also set computeResource.
4provider.vsphere.templateRequired · stringThe name of the VM template that every node is cloned from, not an inventory path. The Alauda OS version of the template matches the platform version; see Version Mapping.
5provider.vsphere.datastoreConditional · stringThe datastore of the VMs, as a name or an inventory path. By default, their disks use it too. Required unless every node group sets vsphere.datastore.
6provider.vsphere.folderOptional · string · Default: the VM folder of the datacenterThe VM folder, as a name or an inventory path.
7provider.vsphere.computeResourceConditional · stringThe DRS cluster or standalone host, as a name. Do not use an inventory path. Required when the datacenter has more than one compute resource; without it, VM creation fails. When the datacenter has only one, you can delete the line.
8provider.vsphere.rootDiskOptional · object · Default: the system disk size of the templatesizeGiB is the size of the system disk, not counting the data disks. It must not be smaller than the system disk of the template. It takes effect only on a full clone. The provider makes a full clone when the template has no snapshot. A linked clone keeps the size of the template.
9provider.vsphere.releaseDelayHoursOptional · integer · Default: 24The number of hours that a released slot of a machine config pool keeps its persistent disks before the provider deletes them. A replacement machine takes a released slot immediately. See Persistent Disk Lifecycle.
10provider.vsphere.networkRequired · objectThe primary NIC of every node. name is the port group or distributed port group, as a name or an inventory path. deviceName is optional. It is the NIC name in the guest. deviceName defaults to eth0, eth1, and so on, in device order.
controlPlaneRequired · objectThe control plane node group.
11controlPlane.cpuCore, controlPlane.memoryGiBRequired · integerThe VM size of the node group: the number of CPU cores and the memory in GiB, each greater than 0. Keep both lines in every node group; you can change the values.
12controlPlane.vsphereOptional · object · Default: the values of provider.vsphereOverrides datastore, folder, computeResource, and rootDisk of provider.vsphere for this group. The same rules apply. To use it, uncomment the block and keep only the fields that differ.
13controlPlane.networkRequired · objectThe NICs of the node. gateway is the IPv4 gateway of the primary NIC. mask is the prefix length of the primary NIC, from 1 to 32. dns is required, and every entry is written into the resolv.conf of the node. additions is optional and defaults to []. It lists the additional NICs, which are matched in order to hosts[].additionalIps. To use it, uncomment the commented block. In that block, mask is from 1 to 32 and defaults to network.mask. See Add a Second NIC.
14controlPlane.network.additions[].vsphereOptional · object · Default: the port group of provider.vsphere.networkSet it when the additional NIC differs from the primary NIC. name is the port group, and it defaults to provider.vsphere.network.name. deviceName is the NIC name in the guest.
15controlPlane.disks.ephemeralRequired · []objectThe disks var-lib-containerd, var-lib-kubelet, and var-lib-etcd. They are formatted as xfs and are recreated with the VM. Keep the names. Change only sizeGiB.
16controlPlane.disks.persistentRequired · []objectThe disk var-cpaas. It is formatted as xfs and is kept when the VM is replaced. Keep the name. Change only sizeGiB.
17controlPlane.disks.rawDisksOptional · []object · Default: []Raw block volumes for TopoLVM. To add them, uncomment the commented block. Each volume needs a name of your choice and a sizeGiB greater than 0. It is attached unformatted at /dev/disk/by-capv/<name>. Other disk names are rejected, so add any extra volumes here. See Add Data Disks.
18controlPlane.disks.vsphereOptional · object · Default: the datastore of the group, no storage policySets where the data disks of this group are placed. datastore defaults to the datastore of the group. storagePolicy is the storage policy of the data disks. To use it, uncomment the block and keep only the fields you set.
19controlPlane.hostsRequired · []objectOne entry per node. List at least as many entries as replicas. Each entry becomes one slot of the machine config pool of the group. hostname is the host name. ip is the IPv4 address of the primary NIC, without a prefix length. ac writes it into the slot as <ip>/<prefix-length>, using network.mask. additionalIps lists one IPv4 address for each network.additions entry, in the same order. Its default is [].
computes[]Optional · []object · Default: []The worker node groups, one entry per group. The controlPlane fields above also apply to each entry. The rows below describe the fields whose rules differ for worker node groups.
20computes[].disks.ephemeralRequired · []objectThese nodes have no var-lib-etcd disk. Keep var-lib-containerd and var-lib-kubelet, and change only sizeGiB.

2. Create the Cluster

Run the following command from a machine where ac is logged in to the platform:

ac config use-cluster global
ac adm alpha cluster create --dir <install-dir> --provider vsphere

Before ac changes anything, it checks the following:

  • The provider CRDs on the global cluster.
  • The credential and the vCenter sign-in.
  • The vCenter objects named in install-config.yaml.
  • The registry.

If a check fails, nothing is changed, and the error names the failed item.

create returns as soon as the objects are applied. The cluster keeps building after that. To wait until all stages are ready, add --wait. --timeout (default 2h) limits only the wait. To render the objects without applying them, run ac adm alpha cluster manifests with the same flags.

3. Monitor the Creation

ac config use-cluster global
ac adm alpha cluster status --cluster <cluster-name>                      # prints the four stages once
ac adm alpha cluster wait-for install-complete --cluster <cluster-name>   # waits until all four are ready

Continue when all four stages are ready; otherwise see Troubleshooting.

4. Verify the Cluster

Run the following commands from a machine where ac is logged in to the platform:

ac config sync
ac config use-cluster <cluster-name>
ac get nodes
ac config use-cluster global

Continue when every node is Ready; otherwise see Troubleshooting.

CPI and Pools

ac config use-cluster global
ac -n cpaas-system get clusterresourceset,clusterresourcesetbinding
ac -n cpaas-system get vspheremachineconfigpool \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,TOTAL:.status.total,ALLOCATED:.status.allocated'
ac config sync
ac config use-cluster <cluster-name>
ac -n kube-system get daemonset
ac config use-cluster global

Continue when a ClusterResourceSetBinding exists for the cluster, vsphere-cloud-controller-manager is listed in kube-system, and each pool of the cluster shows READY True with ALLOCATED equal to the number of its machines; otherwise see Troubleshooting.

Persistent Disks

Run the following commands once for each pool, <cluster-name>-control-plane and <cluster-name>-<group>:

ac config use-cluster global
ac -n cpaas-system get vspheremachineconfigpool <pool-name> \
  -o jsonpath='{range .status.persistentDiskStatuses[*]}{.hostname}{" "}{.name}{" "}{.phase}{"\n"}{end}'

Continue when every persistent disk of an allocated slot reports Attached; otherwise see Troubleshooting.

Optional Topology

Applies to: node groups with network.additions or rawDisks. Skip it when you use neither.

ac config use-cluster global
ac -n cpaas-system get vspheremachineconfigpool <pool-name> -o yaml

On a node of that group:

ls -l /dev/disk/by-capv/

Continue when each slot lists the declared additional NICs and disks, and every raw disk appears in /dev/disk/by-capv/; otherwise contact technical support.

Self-built VIP

Applies to: type: internal. Skip it for an external load balancer.

ac config use-cluster global
ac -n cpaas-system get vspherecluster <cluster-name> \
  -o jsonpath='{.status.conditions[?(@.type=="SelfBuiltLoadBalancerReady")].status}{"\n"}'
ac config sync
ac config use-cluster <cluster-name>
ac -n kube-system get pods -l app=alive -o wide
ac config use-cluster global

Continue when the condition is True and one Alive Pod runs on each control plane node; otherwise see Troubleshooting.

Log in to a control plane node of the cluster as the boot user (ssh boot@<control-plane-node-ip>). Use the private key that matches one of the SSH public keys in cluster.sshKeys. Run curl with sudo, because only root can read the cluster CA file:

sudo curl --cacert /etc/kubernetes/pki/ca.crt https://<vip>:6443/healthz

Continue when it prints ok; otherwise see Troubleshooting.

Run create Again

create only creates a cluster. Do not run it on a cluster that already exists. To change a running cluster, see Manage Nodes.

If create stops before the cluster is created, run the same command again. It skips the work that is already done.

When you run create again on the same directory:

  • It recreates objects that were deleted, except the credential Secret, which ac does not render.
  • It does not overwrite a field that was changed with another tool. When the rendered value differs, create stops and reports that another manager owns the fields. Do not delete the object that the error names.
  • When manifests/ already exists, create asks whether to reuse it. Press Enter to reuse it, or answer n to render again after you change install-config.yaml. When standard input is not a terminal, pass --reuse-manifests=true or --reuse-manifests=false.
  • Reusing manifests/ ignores install-config.yaml and skips every check.

Do not edit the files in manifests/.

WARNING

Keep <install-dir>/manifests/, or set cluster.encryptionProviderConfig. A generated encryption key is kept only in manifests/. If you render again without doing either, a new key is generated, and manifests/ no longer records the key that the cluster uses.

Next Steps