Create the global Cluster on Bare Metal

On Bare Metal, you run create twice. The first run builds the installation ISO and stops. You then register the hosts with the ISO and assign them to pools. The second run installs the cluster.

Prerequisites

Procedure

1. Write install-config.yaml

Create an installation directory on the bootstrap node, for example /root/global-install, and write install-config.yaml in it. The commands on this page call this directory <install-dir>; see Installation Directory. Keep the file private (chmod 600 install-config.yaml).

Copy the following template and fill in the placeholders. The numbered lines are specific to Bare Metal and are described in the table below the template. For the other fields, see Common install-config.yaml Fields. For the objects that ac renders from this file, see Rendered Manifests.

install-config.yaml
global:
  bootstrapIP: <bootstrap-node-ip>
  registry:
    address: <host>:<port>
    username: <user>
    password: <password>
  platform:
    address: <platform-domain-or-ip>
    username: admin@cpaas.io
    password: <password>
    extraAddresses: []
    cert:
      certificateFile: <console-cert-file>
      privateKeyFile: <console-key-file>
      # pkcs12CertificateFile: <console-pkcs12-file>
      # pkcs12Password: <password>

cluster:
  sshKeys:
    - <ssh-public-key>
  ntpServers: []
  network:
    podCIDR: 10.199.0.0/16
    serviceCIDR: 10.96.0.0/12
    kubeOVN:
      joinCIDR: 100.64.0.0/16
    loadBalancer:
      type: internal
      address: <control-plane-endpoint>
      port: 6443
      vrid: <vrid>

provider:
  baremetal:
    # osType: alaudaos-k
    # architecture: arm64
    isoImageTag: <iso-image-tag>
    rootImageTag: <root-image-tag>
    serviceAccountSigningKey: <base64-pem-key>

controlPlane:
  replicas: 3
  deployMode: normal
  baremetal:
    machineInventoryPool: global-control-plane-pool

computes:
  - name: worker
    replicas: 3
    deployMode: normal
    baremetal:
      machineInventoryPool: global-worker-pool
#FieldDescription
globalRequired · objectThe settings for installing the global cluster.
1global.bootstrapIPRequired · stringThe bootstrap node address: the IPv4 address of the bootstrap node that the hosts route to. See Network. The hosts register through https://<bootstrapIP>:12443.
cluster.networkRequired · objectThe cluster network.
2cluster.network.loadBalancerRequired · objecttype: internal, the default, uses a Self-built VIP that the provider runs on the control plane nodes, on the NIC that holds the node address. external uses your own load balancer. You add its backends in Install the Cluster. address: the VIP or the load balancer address. It must be an IPv4 address that is not a node address and is outside the Pod, Service, and join CIDRs. port: default 6443. vrid: required with internal. Set a VRRP router ID from 1 to 255 that is unique in the control plane Layer 2 domain. With external, delete this field. interface and vipMode are not supported.
provider.baremetalRequired · objectThe Bare Metal settings: the host OS, the OS images, and the ServiceAccount signing key.
3provider.baremetal.osTypeOptional · string · Default: Alauda OS M55The host OS that you chose in Host OS and Architecture. For Alauda OS KU, uncomment the line, which sets alaudaos-k. A global cluster has one OS. This OS determines the two OS images, the OS label of the hosts, and the image catalog key. The generator in OS Images in the Registry prints this value.
4provider.baremetal.architectureOptional · string · Default: amd64The host architecture that you chose in Host OS and Architecture: amd64 or arm64. For arm64, uncomment the line. arm64 requires osType: alaudaos-k, because Alauda OS M55 is built for amd64 only. If the host architecture differs from the architecture of the bootstrap node, use the hybrid Core Package. The generator in OS Images in the Registry prints this value for arm64.
5provider.baremetal.isoImageTagRequired · stringThe ISO image tag, without a registry or repository. It is the tag of tkestack/baremetal-base-image-iso, or of tkestack/kubeos-iso with osType: alaudaos-k. The installation ISO is built from this image. Both image tags come from the OS archive whose Alauda OS version matches the platform version. See Version Mapping. The generator in OS Images in the Registry prints this value. To look it up later, see Look Up the Image Tags.
6provider.baremetal.rootImageTagRequired · stringThe root filesystem image tag, without a registry or repository. It is the tag of tkestack/baremetal-base-image, or of tkestack/kubeos-rootfs with osType: alaudaos-k. The hosts are installed from this image. The generator in OS Images in the Registry prints this value, which is the same tag as isoImageTag. To look it up later, see Look Up the Image Tags.
7provider.baremetal.serviceAccountSigningKeyOptional · string · Default: generatedThe base64-encoded PEM key that signs ServiceAccount tokens. Delete the line to have a key generated. Set it only for a disaster recovery pair, with the same value on both sides. See Generate the Shared Keys.
controlPlaneRequired · objectThe control plane node group.
8controlPlane.baremetal.machineInventoryPoolRequired · stringThe MachineInventoryPool that this node group takes its hosts from. You create it in Create the Pools. Two node groups cannot share a pool.
computes[]Optional · []object · Default: []The worker node groups, one entry per group.
9computes[].baremetal.machineInventoryPoolRequired · stringThe MachineInventoryPool of this worker group. Each node group has its own pool.

On Bare Metal, a node group takes only replicas, deployMode, and baremetal.machineInventoryPool, because the hosts supply the size, network, and disks of every node. Do not set cpuCore, memoryGiB, network, disks, hosts, or credentialRef. Any of them causes an error.

For the values of each side of a disaster recovery pair, see Create a Global Cluster Disaster Recovery Pair.

2. Build the Installation ISO

On the bootstrap node:

<extract-dir>/res/<arch>/ac adm alpha global create --dir <install-dir> --provider baremetal

The first run prepares the bootstrap cluster, builds the installation ISO, prints its download and checksum URLs, and then stops with an error:

The installation ISO is built. Boot every host of the global cluster from it:
  download: https://<bootstrapIP>:12443/...
  checksum: https://<bootstrapIP>:12443/...
...
error: the registration check did not pass, so nothing of the global cluster was applied:
  - pool missing: MachineInventoryPool cpaas-system/global-control-plane-pool, which controlPlane draws from, does not exist
  - pool missing: MachineInventoryPool cpaas-system/global-worker-pool, which computes[worker] draws from, does not exist

Register the hosts with the ISO and collect them into the pools, then run again:
  ac adm alpha global create --dir <install-dir> --provider baremetal --reuse-manifests
Everything done so far is kept and skipped

This error is expected on the first run. The pools do not exist yet, because no host has registered. No part of the global cluster has been applied yet. Continue with the steps below. You run the printed command in Install the Cluster.

Download the ISO. Verify it against the checksum URL, either on the bootstrap node or on the machine that downloaded it. The checksum file holds the SHA-256 hash and the original file name of the ISO. The original name contains colons, which Windows does not allow in file names. The saved file therefore often has a different name, and sha256sum -c cannot find it. Compare the hashes instead:

curl -sSk <checksum-url> | awk '{print $1}'    # expected hash
sha256sum <saved-iso-file> | awk '{print $1}'  # actual hash

Continue when the two hashes are the same. Otherwise, download the ISO again. On Windows, certutil -hashfile <saved-iso-file> SHA256 prints the actual hash.

3. Register the Hosts

  1. Boot every host from the ISO. Each host registers as a MachineInventory named global-<system-uuid> and reports its disks. It then waits in the live environment until you select its system disk. You can boot all hosts at once. If a host gets no DHCP address in the live environment, configure its network as prepared in Prerequisites. The download URL expires 120 minutes after the ISO is built. If it has expired, see The ISO Download URL Has Expired. Use this ISO only for this installation. Do not reuse it after the handover.

  2. On the bootstrap node, list the registered hosts and the disks that each waiting host reports. Every host waits for you to select a disk, even a host with a single disk:

    <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig \
      -n cpaas-system get machineinventories.elemental.cattle.io \
      -o custom-columns='NAME:.metadata.name,PHASE:.status.install.phase,IP:.metadata.labels.elemental\.cattle\.io/registration-ip'
    <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig \
      -n cpaas-system get machineinventories.elemental.cattle.io <inventory-name> \
      -o jsonpath='{range .status.install.candidates[*]}{.id}{"\t"}{.eligible}{"\t"}{.path}{"\t"}{.sizeBytes}{"\t"}{.reasons}{"\n"}{end}'

    A waiting host shows AwaitingTarget. IP is the address that the host uses to reach the bootstrap node, so that you can match an inventory to a host. Each candidate line prints the disk ID, whether the disk can be selected, its device path in the live environment, its size in bytes, and the reasons why it cannot be selected, if any. Identify the system disk by its size and path.

  3. While each host is still in the live environment, select its system disk by the disk ID:

    <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig \
      -n cpaas-system annotate machineinventories.elemental.cattle.io <inventory-name> \
      baremetal.alauda.io/install-target-device=<disk-id>
    • Select a disk that shows true, using its ID exactly as printed.
    • The phase changes to Installing, and the host installs Alauda OS to that disk and reboots. The selection cannot be changed after that.
    • If the annotation is rejected because the host was not seen recently, make sure that the host is still running the ISO, and run the command again.
  4. List the registered hosts again on the bootstrap node:

    <extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig \
      -n cpaas-system get machineinventories.elemental.cattle.io \
      -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,PHASE:.status.install.phase,PLAN:.status.plan.secretRef.name'

    You can use a host when READY is True, PHASE is Installed, and PLAN is not <none>. Otherwise, see Troubleshooting on Bare Metal.

4. Create the Pools

Create one MachineInventoryPool for each node group. Each pool must:

  • Have the name that install-config.yaml gives it.
  • Set spec.clusterName: global.
  • List at least as many hosts as the replicas of the node groups that take hosts from it.

A host can belong to only one pool.

pools.yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: MachineInventoryPool
metadata:
  name: global-control-plane-pool
  namespace: cpaas-system
spec:
  clusterName: global
  machineInventories:
    - name: <inventory-1>
      hostname: <hostname-1>
    - name: <inventory-2>
    - name: <inventory-3>
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: MachineInventoryPool
metadata:
  name: global-worker-pool
  namespace: cpaas-system
spec:
  clusterName: global
  machineInventories:
    - name: <inventory-4>
    - name: <inventory-5>
    - name: <inventory-6>
  1. hostname: optional. Defaults to the inventory name.

On the bootstrap node:

<extract-dir>/res/<arch>/ac --kubeconfig /var/cpaas/data/alauda.kubeconfig apply -f pools.yaml

5. Install the Cluster

With an external load balancer, configure its forwarding rules before you run create again. create does not check the load balancer, and the control plane is initialized through it. Forward every port marked Yes under Forward on load balancer in Platform Access Address and global VIP to every host in the pool of controlPlane.baremetal.machineInventoryPool. Use the address that each host had in the live environment. A host keeps that address after the installation, also when it came from DHCP.

On the bootstrap node, run create again, reusing the manifests that the first run rendered:

<extract-dir>/res/<arch>/ac adm alpha global create --dir <install-dir> --provider baremetal --reuse-manifests

ac first checks the pools named in install-config.yaml (their members and size) and the ISO URL. If a check fails, no part of the global cluster is applied, and the error names the item that failed.

create returns when the installation is complete. If create stops, run the same command again. It skips the work that is already complete. See The create Command.

6. Monitor the Installation

From another terminal on the bootstrap node:

<extract-dir>/res/<arch>/ac adm alpha global status

For the stages and common failures, see Troubleshooting. For Bare Metal failures, see Troubleshooting on Bare Metal.

Verification

When create returns, the installation is complete, but some platform components can take a few more minutes to become ready. Before you judge the result, wait for them as described in Additional Checks for Alauda OS. Then verify the platform as described in Validation, including its checks for Alauda OS.

Applies to: a Self-built VIP (cluster.network.loadBalancer.type: internal). Skip this check with an external load balancer.

Log in to a control plane node of the global cluster as the boot user (ssh boot@<control-plane-node-ip>). Use the private key that matches an SSH public key in cluster.sshKeys. Run curl with sudo, because only root can read the cluster CA file:

sudo curl --cacert /etc/kubernetes/pki/ca.crt https://<control-plane-vip>:6443/healthz

Continue when it prints ok. Otherwise, see Troubleshooting on Bare Metal.

Next Steps

  • A disaster recovery pair: continue with Install the Primary and the Standby.
  • To create workload clusters on Bare Metal, prepare the OS and register the hosts as described in Prepare Alauda OS for Hosts and Register Hosts, then follow Create a Cluster on Bare Metal. The global cluster already runs the Bare Metal provider. It also already has the OS images and the image catalog key for the osType and architecture in install-config.yaml. For hosts of that OS and architecture, create only the registration and its ISO. For hosts of another architecture or OS, also push the images and add the keys. See Add the Image Catalog Keys.
  • To create workload clusters on other infrastructure, install that provider as described in Prepare.