Create the global Cluster on VMware vSphere

Prerequisites

Procedure

1. Write install-config.yaml

Create an installation directory on the bootstrap node, for example /root/global-install, and write install-config.yaml in it. The commands on this page call this directory <install-dir>; see Installation Directory. Keep the file private (chmod 600 install-config.yaml), because it holds the passwords.

Copy the following template and fill in the placeholders. The numbered lines are specific to VMware vSphere and are described in the table below the template. For the other fields, see Common install-config.yaml Fields. For the objects that ac renders from this file, see Rendered Manifests.

install-config.yaml
global:
  registry:
    address: <host>:<port>
    username: <user>
    password: <password>
  platform:
    address: <platform-domain-or-ip>
    username: admin@cpaas.io
    password: <password>
    extraAddresses: []
    cert:
      certificateFile: <console-cert-file>
      privateKeyFile: <console-key-file>
      # pkcs12CertificateFile: <console-pkcs12-file>
      # pkcs12Password: <password>

cluster:
  sshKeys:
    - <ssh-public-key>
  ntpServers: []
  network:
    podCIDR: 10.199.0.0/16
    serviceCIDR: 10.96.0.0/12
    kubeOVN:
      joinCIDR: 100.64.0.0/16
    loadBalancer:
      type: external
      address: <control-plane-endpoint>
      port: 6443
      # vrid: <vrid>
      # interface: <nic-name>

provider:
  vsphere:
    server: <vcenter-address>
    credential:
      username: <administrator@vsphere.local>
      password: <password>
    datacenter: <datacenter>
    template: <template-name-or-path>
    datastore: <datastore>
    # folder: <folder-path>
    computeResource: <cluster-or-host>
    # rootDisk:
    #   sizeGiB: <root-disk-gib>
    releaseDelayHours: 24
    network:
      name: <portgroup-name>
      # deviceName: <nic-name>

controlPlane:
  replicas: 3
  cpuCore: 8
  memoryGiB: 16
  deployMode: normal
  # vsphere:
  #   datastore: <datastore>
  #   folder: <folder-path>
  #   computeResource: <cluster-or-host>
  #   rootDisk:
  #     sizeGiB: <root-disk-gib>
  network:
    gateway: <cp-gateway>
    dns: [<dns-ip>]
    mask: <prefix-length>
    additions: []
    # additions:
    #   - mask: <prefix-length>
    #     vsphere:
    #       name: <portgroup-name>
    #       deviceName: <nic-name>
  disks:
    ephemeral:
      - name: var-lib-containerd
        sizeGiB: 100
      - name: var-lib-kubelet
        sizeGiB: 100
      - name: var-lib-etcd
        sizeGiB: 20
    persistent:
      - name: var-cpaas
        sizeGiB: 100
    rawDisks: [] 
    # rawDisks:
    #   - name: data-001
    #     sizeGiB: 50
    # vsphere:
    #   datastore: <datastore>
    #   storagePolicy: <storage-policy>
  hosts:
    - hostname: <cp-0-hostname>
      ip: <cp-0-ip>
      additionalIps: []
    - hostname: <cp-1-hostname>
      ip: <cp-1-ip>
      additionalIps: []
    - hostname: <cp-2-hostname>
      ip: <cp-2-ip>
      additionalIps: []

computes:
  - name: worker
    replicas: 3
    cpuCore: 8
    memoryGiB: 16
    deployMode: normal
    # vsphere:
    #   datastore: <datastore>
    #   folder: <folder-path>
    #   computeResource: <cluster-or-host>
    #   rootDisk:
    #     sizeGiB: <root-disk-gib>
    network:
      gateway: <worker-gateway>
      dns: [<dns-ip>]
      mask: <prefix-length>
      additions: []
      # additions:
      #   - mask: <prefix-length>
      #     vsphere:
      #       name: <portgroup-name>
      #       deviceName: <nic-name>
    disks:
      ephemeral:
        - name: var-lib-containerd
          sizeGiB: 100
        - name: var-lib-kubelet
          sizeGiB: 100
      persistent:
        - name: var-cpaas
          sizeGiB: 100
      rawDisks: []
      # rawDisks:
      #   - name: data-001
      #     sizeGiB: 50
      # vsphere:
      #   datastore: <datastore>
      #   storagePolicy: <storage-policy>
    hosts:
      - hostname: <worker-0-hostname>
        ip: <worker-0-ip>
        additionalIps: []
      - hostname: <worker-1-hostname>
        ip: <worker-1-ip>
        additionalIps: []
      - hostname: <worker-2-hostname>
        ip: <worker-2-ip>
        additionalIps: []
#FieldDescription
cluster.networkRequired · objectThe cluster network.
1cluster.network.loadBalancerRequired · objecttype: external, the default, uses your own load balancer, which must be ready before create. internal uses a Self-built VIP that the provider runs on the control plane nodes. You cannot change the type after the cluster is created. address: the load balancer address or the VIP. It must be an IPv4 address that is not a node address and is outside the Pod, Service, and join CIDRs. port: default 6443. With internal, uncomment vrid: a required VRRP router ID from 1 to 255 that is unique in the control plane Layer 2 domain. You can also uncomment interface: the NIC that holds the VIP. By default, this is the NIC of the node IP. vipMode is not supported.
provider.vsphereRequired · objectThe VMware vSphere settings.
2provider.vsphere.serverRequired · stringThe vCenter address.
3provider.vsphere.credentialRequired · objectThe vCenter username and password, written inline. They are rendered into the global-credential Secret. credentialRef is rejected.
4provider.vsphere.datacenterRequired · stringThe datacenter, by name or inventory path. A nested path, such as /folder/dc, cannot be combined with computeResource.
5provider.vsphere.templateRequired · stringThe name of the VM template that every node is cloned from, not an inventory path. Its Alauda OS version matches the platform version. See Version Mapping.
6provider.vsphere.datastoreConditional · stringThe datastore of the VMs and, by default, of their disks, by name or inventory path. Required unless every node group sets vsphere.datastore.
7provider.vsphere.folderOptional · string · Default: the datacenter's VM folderThe VM folder, by name or inventory path. A dedicated folder makes the global VMs easy to find.
8provider.vsphere.computeResourceConditional · stringThe DRS cluster or standalone host, by name, not by inventory path. Required when the datacenter has more than one compute resource, because VM creation fails without it. If the datacenter has only one compute resource, you can delete the line.
9provider.vsphere.rootDiskOptional · object · Default: the template's system disk sizesizeGiB: the system disk size, not counting the data disks. It must not be smaller than the template's system disk. It takes effect only on a full clone, which the provider makes when the template has no snapshot. A linked clone keeps the template's size.
10provider.vsphere.releaseDelayHoursOptional · integer · Default: 24How many hours a released slot of a machine config pool keeps its persistent disks before they are deleted. A replacement machine takes a released slot immediately.
11provider.vsphere.networkRequired · objectThe primary NIC of every node. name: the port group or distributed port group, by name or inventory path. deviceName: optional, the NIC name in the guest. deviceName defaults to eth0, eth1, and so on, in device order.
controlPlaneRequired · objectThe control plane node group.
12controlPlane.cpuCore, controlPlane.memoryGiBRequired · integerThe VM size of the node group: the number of CPU cores and the memory in GiB, each greater than 0. Keep both lines in every node group; you can change the values.
13controlPlane.vsphereOptional · object · Default: the values of provider.vsphereFor this group, overrides datastore, folder, computeResource, and rootDisk of provider.vsphere. The same rules apply. Uncomment the block and keep only the fields that differ.
14controlPlane.networkRequired · objectThe NICs of the node. gateway: the IPv4 gateway of the primary NIC. mask: its prefix length, 1 to 32. dns: required. Every entry is written into the node's resolv.conf. additions: optional additional NICs, matched in order to hosts[].additionalIps. additions defaults to []. Use the commented block. In each addition, mask is 1 to 32 and defaults to network.mask.
15controlPlane.network.additions[].vsphereOptional · object · Default: the port group of provider.vsphere.networkSet it when the additional NIC differs from the primary NIC. name: the port group, which defaults to provider.vsphere.network.name. deviceName: the NIC name in the guest.
16controlPlane.disks.ephemeralRequired · []objectvar-lib-containerd, var-lib-kubelet, and var-lib-etcd. They are formatted as xfs and recreated with the VM. Keep the names, and change only sizeGiB.
17controlPlane.disks.persistentRequired · []objectvar-cpaas. It is formatted as xfs and kept when the VM is replaced. Keep the name, and change only sizeGiB.
18controlPlane.disks.rawDisksOptional · []object · Default: []Raw block volumes for TopoLVM. Give each one a name of your choice and a sizeGiB greater than 0. Each volume is attached unformatted at /dev/disk/by-capv/<name>. Use the commented block. Add any extra volumes here, because other disk names are rejected.
19controlPlane.disks.vsphereOptional · object · Default: the datastore of the group, no storage policyPlaces the data disks of this group. datastore: defaults to the group's datastore. storagePolicy: the storage policy of the data disks. Uncomment the block and keep only the fields that you set.
20controlPlane.hostsRequired · []objectOne entry per node, and at least replicas entries. Each entry is one slot of the group's machine config pool. It sets hostname, ip (the IPv4 address of the primary NIC), and additionalIps (one IPv4 address for each network.additions entry, in order; default []).
computes[]Optional · []object · Default: []The worker node groups, one entry per group. The controlPlane fields above also apply to each entry. The rows below describe the fields whose rules differ for worker node groups.
21computes[].disks.ephemeralRequired · []objectThe list has no var-lib-etcd. Keep var-lib-containerd and var-lib-kubelet, and change only sizeGiB.

If you give a datastore, folder, or port group as an inventory path, make sure that the path exists.

2. Create the Cluster

On the bootstrap node:

<extract-dir>/res/<arch>/ac adm alpha global create --dir <install-dir> --provider vsphere

ac first checks the CPI image in the Core Package, the vCenter sign-in, and the vCenter objects named in install-config.yaml. If a check fails, nothing is changed, and the error names the first missing item. Make sure that an external load balancer is ready before you run create.

create returns when the installation is complete. If create stops, run the same command again. It skips the work that is already complete. See The create Command.

3. Monitor the Installation

From another terminal on the bootstrap node:

<extract-dir>/res/<arch>/ac adm alpha global status

For the stages and common failures, see Troubleshooting. For VMware vSphere failures, see Troubleshooting on VMware vSphere.

Verification

When create returns, the installation is complete, but some platform components can take a few more minutes to become ready. Before you judge the result, wait for them as described in Additional Checks for Alauda OS. Then verify the platform as described in Validation, including its checks for Alauda OS.

Applies to: a Self-built VIP (cluster.network.loadBalancer.type: internal). Skip this check with an external load balancer.

Log in to a control plane node of the global cluster as the boot user (ssh boot@<control-plane-node-ip>). Use the private key that matches an SSH public key in cluster.sshKeys. Run curl with sudo, because only root can read the cluster CA file:

sudo curl --cacert /etc/kubernetes/pki/ca.crt https://<control-plane-vip>:6443/healthz

Continue when it prints ok. Otherwise, see Troubleshooting on VMware vSphere.

Next Steps