Upgrade the Standby global Cluster After the Workload Clusters

Use this page for a patch upgrade within the same minor version, from ACP 4.5.0 or later, when the standby is not upgraded together with the primary: you upgrade the primary and the workload clusters that you choose first, and the standby afterwards. The plugin stays installed and synchronization is paused, so the standby remains a standby. There is no DR protection from the pause until synchronization is resumed and checked, through the upgrades of the primary, of those workload clusters, and of the standby; in that time, make no changes on either global cluster other than those upgrades and the steps on this page. For a minor upgrade, or to upgrade both global clusters before the workload clusters, use Upgrade global Clusters in a DR Environment; Upgrade Order and Compatibility compares the two procedures.

Before You Begin

  • Both global clusters run the same ACP version (Distribution Version), 4.5.0 or later, and the target is a later patch version of the same minor version. <target-version> on this page is that target ACP version, for example 4.5.1.
  • Prepare for the Upgrade is complete on both global clusters. These items are not synchronized between them, so each side has its own: published packages, including the Global Cluster Disaster Recovery package on the standby; Alauda OS image ConfigMaps or catalog entries; the administrator acknowledgement; ResourcePatch exemptions; and a preflight result that lets you continue. On the standby, every check passes except DRSplitUpgradeWindow, which passes after you disable the token rotator and pause synchronization below.
  • Each global cluster has a recent successful etcd backup, as described in Check the etcd Backup.
  • You are on the execution host, logged in with ac login as a platform administrator, in the installer/ directory of the extracted target Core Package, with a kubeconfig of the standby that points to its VIP: <standby-kubeconfig>.

On the primary, ac commands use the ac login session, and upgrade.sh uses kubectl with the kubeconfig that ac login saved, after ac config use-cluster global; ac whoami --show-server prints https://<platform-address>/kubernetes/global/. Each command block for the standby starts with export KUBECONFIG=<standby-kubeconfig> and ends with unset KUBECONFIG, and ac whoami --show-server prints the standby VIP. Confirm the printed address before you continue with a block.

Upgrade the Primary and the Workload Clusters

Verify Data Consistency

WARNING

Do not pause synchronization until this check passes.

Run the check on a control plane node of the standby global cluster, then return to the execution host:

export KUBECONFIG=/etc/kubernetes/admin.conf
mirror_svc=$(kubectl get svc -n cpaas-system etcd-sync-monitor -o jsonpath='{.spec.clusterIP}')
if [[ $mirror_svc =~ ^[0-9a-fA-F:]+$ ]]; then mirror_host="[$mirror_svc]"; else mirror_host="$mirror_svc"; fi
curl -g "http://${mirror_host}/check"
ResultAction
localMissedKeys [0] and localSurplusKeys [0]Continue.
check too frequentWait one minute and run the check again; if it does not change, contact technical support.
Any other valueRun the steps in Global Cluster Disaster Recovery Does Not Synchronize, then run the check again.

Then check the machines and ongoing upgrades. On the primary:

ac config use-cluster global
ac whoami --show-server
ac get machines.platform.tkestack.io
ac get machines.cluster.x-k8s.io -A   # Alauda OS
ac -n cpaas-system get cvsh \
  -o custom-columns='CLUSTER:.metadata.name,REQUESTED:.spec.desiredUpdate.version,CURRENT:.status.current.version'

On the standby:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac get machines.platform.tkestack.io
ac get machines.cluster.x-k8s.io -A   # Alauda OS
unset KUBECONFIG
ResultAction
Every machine on both global clusters is Running, and on every row of the cvsh list REQUESTED is empty or equal to CURRENTContinue.
On a row of the cvsh list, REQUESTED differs from CURRENTRun the block on the primary again after that workload cluster upgrade completes. If that upgrade does not advance, see Inspect Upgrade State.
A machine is not RunningContact technical support.

Prepare Both global Clusters

On the standby, restart the cluster-transformer Pods, so that no work started before the pause keeps running on the standby, and disable the token rotator, which can otherwise revoke workload cluster tokens that the primary still uses. The set env command prints The operation is allowed, and ResourcePatch <name> is created; the block exempts that ResourcePatch for the target version, so that preflight does not block on it:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac -n cpaas-system delete pod -l service_name=cluster-transformer
ac -n cpaas-system set env deploy/cluster-api-provider-alauda-manager \
  -c cluster-api-provider-alauda CLUSTER_TOKEN_ROTATOR_ENABLED=false
ac -n cpaas-system rollout status deploy/cluster-api-provider-alauda-manager
RP=$(ac get resourcepatches \
  -o jsonpath='{range .items[?(@.spec.target.name=="cluster-api-provider-alauda-manager")]}{.metadata.name}{"\n"}{end}')
ac annotate resourcepatches "${RP}" config.cpaas.io/exempt-for-ver=<target-version> --overwrite
ac -n cpaas-system get pod -l service_name=cluster-transformer
ac -n cpaas-system get deploy cluster-api-provider-alauda-manager \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="cluster-api-provider-alauda")].env[?(@.name=="CLUSTER_TOKEN_ROTATOR_ENABLED")].value}{"\n"}'
unset KUBECONFIG

On the primary, disable the token rotator and exempt its ResourcePatch as well: the paused standby does not receive the tokens it writes, and the standby's copies stop working when it deletes the replaced tokens:

ac config use-cluster global
ac whoami --show-server   # ends with /kubernetes/global/
ac -n cpaas-system set env deploy/cluster-api-provider-alauda-manager \
  -c cluster-api-provider-alauda CLUSTER_TOKEN_ROTATOR_ENABLED=false
ac -n cpaas-system rollout status deploy/cluster-api-provider-alauda-manager
RP=$(ac get resourcepatches \
  -o jsonpath='{range .items[?(@.spec.target.name=="cluster-api-provider-alauda-manager")]}{.metadata.name}{"\n"}{end}')
ac annotate resourcepatches "${RP}" config.cpaas.io/exempt-for-ver=<target-version> --overwrite
ac -n cpaas-system get deploy cluster-api-provider-alauda-manager \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="cluster-api-provider-alauda")].env[?(@.name=="CLUSTER_TOKEN_ROTATOR_ENABLED")].value}{"\n"}'

Continue when, on both sides, the rollout completes, annotate reports the ResourcePatch as annotated, and the last command prints false, and on the standby every cluster-transformer Pod is Running. Then run the preflight check on the standby:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
bash upgrade.sh --preflight
unset KUBECONFIG
ResultAction
DRSplitUpgradeWindow is the only failed check, and its only reason is EtcdSyncNotPausedContinue.
DRSplitUpgradeWindow fails for another reasonSee DR Standby Checks, then run the check again.
Another check failsSee Handle Preflight Blocks, then run the check again.

Pause Synchronization

WARNING

From the pause until synchronization is resumed and checked, there is no DR protection. Do not fail over to the standby. Do not uninstall the plugin: uninstalling it removes the DR marker kube-public/etcd-sync, which the plugin creates, and the standby then acts on workload clusters. Do not delete or edit the marker or cpaas-system/etcd-sync-control except in Resume Synchronization. On either global cluster, make no changes other than the upgrades and steps on this page: the primary, the workload clusters, and the standby.

On the standby, pause synchronization, with a change reference of your choice as reason, then read the pause state, its message and self-check (status.pauseCheck), and the DR marker:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
if ac -n cpaas-system get configmap etcd-sync-control >/dev/null 2>&1; then
  ac -n cpaas-system patch configmap etcd-sync-control --type merge \
    -p '{"data":{"mode":"Paused","reason":"<change-reference>"}}'
else
  ac -n cpaas-system create configmap etcd-sync-control \
    --from-literal=mode=Paused --from-literal=reason="<change-reference>"
fi
ac -n cpaas-system get configmap etcd-sync-control \
  -o jsonpath='{.data.mode}{" "}{.data.status\.phase}{" "}{.data.status\.observedMode}{"\n"}{.data.status\.message}{"\n"}{.data.status\.pauseCheck}{"\n"}'
ac -n kube-public get configmap etcd-sync -o name
unset KUBECONFIG

Run the block again without the lines from if to fi until the second value on the line after the address is Paused. In the table, "the state line" is that line.

ResultAction
The state line is Paused Paused Paused, the status.pauseCheck line shows "unexpected":0 and no error, and the last command prints configmap/etcd-syncContinue.
On the state line, the second and third values stay emptyetcd-sync did not take the request. Do not upgrade the primary; contact technical support.
The state line stays Paused Pausing Paused or Paused Running Running when you run the block againRead the status.message line and contact technical support.
unexpected is not 0, error is set, or the marker is missingDo not upgrade the primary; contact technical support.

Upgrade the Primary global Cluster

WARNING

An upgrade cannot be rolled back. When the target changes the Kubernetes build or the Alauda OS image, the request also replaces every node with a new VM on Huawei DCS, Huawei Cloud Stack, and VMware vSphere, and reinstalls every host on Bare Metal, one at a time.

On the primary, deploy the Cluster Version Operator (CVO) and the target release information. <target-version> is the target ACP version:

ac config use-cluster global
ac whoami --show-server   # ends with /kubernetes/global/
bash upgrade.sh --skip-sync-image
ac get productmanifest v<target-version>
ResultAction
The script completes without an error, and READY is TrueContinue.
The script reports that the product manifest did not become ready, or READY is not TrueFollow Recover a Missing Aligned Extension Package.
The script fails with another errorContact technical support.

Then run the final preflight check:

ac config use-cluster global
ac whoami --show-server   # ends with /kubernetes/global/
bash upgrade.sh --preflight

Continue when Summary shows Passed or PassedWithWarnings. Otherwise, resolve the checks on the primary as described in Handle Preflight Blocks and run the check again; AdminAckRequired can appear for the first time here, see Handle Administrator Acknowledgement Gates. Then request the upgrade with the target version exactly as ac adm upgrade lists it:

ac adm upgrade
ac adm upgrade --cluster=global --to=<target-version>

Monitor the upgrade:

ac adm upgrade status --cluster=global
ac -n cpaas-system get infraclusterversion global   # Alauda OS

Run the commands again until Current shows the target version and Ready is True, and on Alauda OS PHASE is Succeeded. While control plane nodes are replaced, a command can fail to connect; run it again. If Upgrade shows Blocked or Retrying, see Inspect Upgrade State. On Alauda OS, if the upgrade stays InProgress in the Kubernetes stage, read the Stalled condition of the InfraClusterVersion as described there. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support.

Upgrade the Workload Clusters

First, on the primary, check that the token rotator is still disabled after the upgrade:

ac config use-cluster global
ac whoami --show-server   # ends with /kubernetes/global/
ac -n cpaas-system get deploy cluster-api-provider-alauda-manager \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="cluster-api-provider-alauda")].env[?(@.name=="CLUSTER_TOKEN_ROTATOR_ENABLED")].value}{"\n"}'

Continue when it prints false; otherwise, run the primary block of Prepare Both global Clusters again. Then complete Upgrade Validation on the primary.

Upgrade the workload clusters that you choose to upgrade now by following Upgrade Workload Clusters, with these differences, then return to this page:

  • Skip Confirm the Standby global Cluster Is Upgraded.
  • Skip Check DR Synchronization while synchronization is paused.
  • Skip Upgrade the Agnostic Extensions; upgrade them in Upgrade the Agnostic Extensions and Validate, after synchronization is resumed.

Continue when every workload cluster that you upgraded shows the target version in Current with Ready True and has passed its validation, and no workload cluster upgrade is running. Upgrade the other workload clusters after this page is complete.

Upgrade the Standby and Resume Synchronization

Check the State Before Upgrading the Standby

Start here when you return to upgrade the standby. Log in again with ac login if the session expired, and check that each global cluster still has a recent successful etcd backup, as in Before You Begin. Then check that the state is unchanged. On the primary:

ac config use-cluster global
ac whoami --show-server   # ends with /kubernetes/global/
ac adm upgrade status --cluster=global
ac -n cpaas-system get cvsh \
  -o custom-columns='CLUSTER:.metadata.name,REQUESTED:.spec.desiredUpdate.version,CURRENT:.status.current.version'
ac -n cpaas-system get deploy cluster-api-provider-alauda-manager \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="cluster-api-provider-alauda")].env[?(@.name=="CLUSTER_TOKEN_ROTATOR_ENABLED")].value}{"\n"}'

Continue when Current shows the target version and Ready is True, on every row of the cvsh list REQUESTED is empty or equal to CURRENT, and the last command prints false. Use this Current value as <target-version> for the standby.

On the standby, check the pause state and the marker, then check from the execution host that the token of the plugin still gives access to the primary. The block reads the primary VIP from the plugin settings and puts an IPv6 address in brackets; the token is passed to curl through standard input and is not printed:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac -n cpaas-system get configmap etcd-sync-control \
  -o jsonpath='{.data.mode}{" "}{.data.status\.phase}{" "}{.data.status\.observedMode}{"\n"}'
ac -n kube-public get configmap etcd-sync -o name
TOKEN_SECRET=$(ac get moduleinfo -l cpaas.io/module-name=etcd-sync,cpaas.io/cluster-name=global \
  -o jsonpath='{.items[0].spec.config.active_cluster_token_secret_ref}')
TOKEN_SECRET=${TOKEN_SECRET:-etcd-sync-token}
ACTIVE_VIP=$(ac get moduleinfo -l cpaas.io/module-name=etcd-sync,cpaas.io/cluster-name=global \
  -o jsonpath='{.items[0].spec.config.active_cluster_vip}')
if [[ $ACTIVE_VIP =~ ^[0-9a-fA-F:]+$ ]]; then ACTIVE_VIP="[$ACTIVE_VIP]"; fi
{ printf 'Authorization: Bearer '
  ac -n cpaas-system get secret "${TOKEN_SECRET}" -o jsonpath='{.data.token}' | base64 -d
  printf '\n'
} | curl -sk -H @- -o /dev/null -w '%{http_code}\n' \
  "https://${ACTIVE_VIP}:6443/api/v1/namespaces/cpaas-system/secrets/etcd-ca"
unset KUBECONFIG
ResultAction
Paused Paused Paused, configmap/etcd-sync, and 200Continue.
401 or 403The token of the plugin no longer gives access to the primary. Contact technical support.
The pause state is not Paused Paused Paused, or the marker is missingDo not upgrade the standby; contact technical support.
000The execution host does not reach the primary VIP on port 6443. Fix the connection from the execution host and run the block again.
Any other HTTP codeContact technical support.

Upgrade the Standby global Cluster

WARNING

An upgrade cannot be rolled back. When the target changes the Kubernetes build or the Alauda OS image, the request also replaces every node with a new VM on Huawei DCS, Huawei Cloud Stack, and VMware vSphere, and reinstalls every host on Bare Metal, one at a time.

On the standby, deploy CVO and the target release information:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
bash upgrade.sh --skip-sync-image
ac get productmanifest v<target-version>
unset KUBECONFIG
ResultAction
The script completes without an error, and READY is TrueContinue.
The script reports that the product manifest did not become ready, or READY is not TrueFollow Recover a Missing Aligned Extension Package.
The script fails with another errorContact technical support.

Then run the final preflight check. Check the marker first, because without the marker DRSplitUpgradeWindow passes without checking the standby:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac -n kube-public get configmap etcd-sync -o name
bash upgrade.sh --preflight
unset KUBECONFIG

If the first command does not print configmap/etcd-sync, do not upgrade the standby, and contact technical support. Continue when Summary shows Passed or PassedWithWarnings. If DRSplitUpgradeWindow fails, see DR Standby Checks. For other checks, resolve them on the standby as described in Handle Preflight Blocks and run the check again; AdminAckRequired can appear for the first time here, see Handle Administrator Acknowledgement Gates. Then request the upgrade with the target version exactly as ac adm upgrade lists it:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac adm upgrade
ac adm upgrade --cluster=global --to=<target-version>
unset KUBECONFIG

Monitor the upgrade:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac adm upgrade status --cluster=global
ac -n cpaas-system get infraclusterversion global   # Alauda OS
unset KUBECONFIG

Run the block again until Current shows the target version and Ready is True, and on Alauda OS PHASE is Succeeded. While control plane nodes are replaced, a command can fail to connect; run it again. If Upgrade shows Blocked or Retrying, see Inspect Upgrade State. On Alauda OS, if the upgrade stays InProgress in the Kubernetes stage, read the Stalled condition of the InfraClusterVersion as described there. Fix the reported cause and the upgrade continues; if it cannot be fixed, contact technical support.

Resume Synchronization

WARNING

Resuming makes the standby match the primary: objects that existed at the pause and were deleted on the primary since then are also deleted on the standby. Resume only after the checks below pass.

First, on the standby, check that synchronization is still paused, the marker exists, and the token rotator is still disabled after the upgrade:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac -n cpaas-system get configmap etcd-sync-control \
  -o jsonpath='{.data.mode}{" "}{.data.status\.phase}{" "}{.data.status\.observedMode}{"\n"}'
ac -n kube-public get configmap etcd-sync -o name
ac -n cpaas-system get deploy cluster-api-provider-alauda-manager \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="cluster-api-provider-alauda")].env[?(@.name=="CLUSTER_TOKEN_ROTATOR_ENABLED")].value}{"\n"}'
unset KUBECONFIG

Continue when the output is Paused Paused Paused, configmap/etcd-sync, and false. If the token rotator value is not false, run the standby block of Prepare Both global Clusters again. If the pause state or the marker differs, do not resume, and contact technical support. Then resume synchronization on the standby and read the result:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac -n cpaas-system patch configmap etcd-sync-control --type merge -p '{"data":{"mode":"Running"}}'
ac -n cpaas-system get configmap etcd-sync-control \
  -o jsonpath='{.data.status\.phase}{"\n"}{.data.status\.message}{"\n"}{.data.status\.lastResume}{"\n"}{.data.status\.deleteBlocked}{"\n"}'
unset KUBECONFIG

Run the block again without the patch command until the line after the address shows Running. Then run the /check command from Verify Data Consistency again on a standby control plane node.

ResultAction
status.phase is Running, status.lastResume contains "deleteSkipped":false, the status.deleteBlocked line is empty, and /check shows localMissedKeys [0] and localSurplusKeys [0]Continue.
status.phase stays Resuming, or stays PausedDo not edit etcd-sync-control; contact technical support.
"deleteSkipped":true, or the status.deleteBlocked line is not emptyContact technical support.
check too frequentWait one minute and run the check again; if it does not change, contact technical support.
/check shows any other valueRun the steps in Global Cluster Disaster Recovery Does Not Synchronize, then run the check again.

Restore the Token Rotator

On the primary, remove the setting, then check it and list the ResourcePatch objects for its Deployment:

ac config use-cluster global
ac whoami --show-server   # ends with /kubernetes/global/
ac -n cpaas-system set env deploy/cluster-api-provider-alauda-manager \
  -c cluster-api-provider-alauda CLUSTER_TOKEN_ROTATOR_ENABLED-
ac -n cpaas-system rollout status deploy/cluster-api-provider-alauda-manager
ac -n cpaas-system get deploy cluster-api-provider-alauda-manager \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="cluster-api-provider-alauda")].env[?(@.name=="CLUSTER_TOKEN_ROTATOR_ENABLED")].value}{"\n"}'
ac get resourcepatches \
  -o jsonpath='{range .items[?(@.spec.target.name=="cluster-api-provider-alauda-manager")]}{.metadata.name}{"\n"}{end}'

On the standby:

export KUBECONFIG=<standby-kubeconfig>
ac whoami --show-server   # the standby VIP
ac -n cpaas-system set env deploy/cluster-api-provider-alauda-manager \
  -c cluster-api-provider-alauda CLUSTER_TOKEN_ROTATOR_ENABLED-
ac -n cpaas-system rollout status deploy/cluster-api-provider-alauda-manager
ac -n cpaas-system get deploy cluster-api-provider-alauda-manager \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="cluster-api-provider-alauda")].env[?(@.name=="CLUSTER_TOKEN_ROTATOR_ENABLED")].value}{"\n"}'
ac get resourcepatches \
  -o jsonpath='{range .items[?(@.spec.target.name=="cluster-api-provider-alauda-manager")]}{.metadata.name}{"\n"}{end}'
unset KUBECONFIG

Continue when, on both sides, the rollout completes, the jsonpath command for the setting prints an empty line, and the last command lists no ResourcePatch: removing the setting also removes the ResourcePatch that it created. If a ResourcePatch is still listed, it holds another customization of the Deployment; keep it, and check that it does not contain CLUSTER_TOKEN_ROTATOR_ENABLED.

Upgrade the Agnostic Extensions and Validate

The upgrade requests do not upgrade Extensions (Operators and Cluster Plugins) with the Agnostic lifecycle. Upgrade each Agnostic Extension installed on the primary, on the standby, and on the workload clusters that you upgraded, by following its own documentation. For example, see Upgrade Alauda AI and Upgrade Alauda DevOps. Then complete Upgrade Validation on the standby; the primary was validated before the workload clusters.

The upgrade of the two global clusters is complete only when every step on this page is done, including the synchronization check after the resume; do not start another upgrade of either global cluster before that.

Next Step

Upgrade the remaining workload clusters. See Upgrade Workload Clusters.