Create an Image Registry with mirror-registry

An image registry provided by your organization is strongly recommended as the platform image registry. If you already have an image registry, skip this chapter and continue with Prepare the Target-Version Payload.

If no image registry is available, you can use mirror-registry. mirror-registry is a small-scale image registry that can serve as the target registry for platform images.

Important

mirror-registry is not a substitute for a production-grade, highly available image registry.

Prerequisites

  • A machine outside all clusters. Do not use a node of any cluster.
  • Linux with systemd and OpenSSL installed. You must be able to run commands as root.
  • 4 or more vCPUs.
  • 8 GB or more of RAM.
  • Local disk or S3-compatible object storage. With a local disk, the file system that holds the data directory needs at least 200 GB. LVM is recommended so that you can expand the storage later. See Deploy with Local Storage.
  • The registry port (443 by default) is reachable from the installation node and every cluster node that pulls platform images.
Important

Storage requirements depend on your environment. For example, you need more space to retain multiple versions or multiple CPU architectures. With a local disk, monitor the storage used by uploaded images.

About mirror-registry

Deploying requires an image registry that stores the images of platform components. Installing and later upgrading the platform depend on this registry. If you can provide an image registry as the platform image registry, you do not need mirror-registry.

If no image registry is available, you can use mirror-registry. mirror-registry is a small-scale image registry provided with , available on the Core Package download page. It is based on the native Distribution registry, runs as a single instance managed by systemd, and is accessible only over HTTPS. By default, it listens on port 443 and uses local storage and an automatically generated self-signed certificate. You must configure credentials when you deploy it. Optional settings include the port, custom TLS certificates, and S3-compatible object storage.

mirror-registry Limitations

The following limitations apply to mirror-registry:

  • mirror-registry is not a highly available registry. With S3-compatible object storage, the image data is stored off the machine, but the registry service remains a single instance. mirror-registry is not a substitute for a production-grade, highly available image registry.
  • mirror-registry is not recommended for production environments. It is a single-point image registry. If it goes down, Pods scheduled to other nodes might fail to start because they cannot pull images, which affects platform availability.
  • mirror-registry supports hosting only platform images.
  • Content built by users, including application images, should not be hosted in mirror-registry.
  • Sharing mirror-registry across multiple clusters is not recommended, because the clusters then depend on the same registry, which creates a single point of failure when you update them. Use a production-grade, highly available image registry provided by your organization for multiple clusters.

Install mirror-registry

This procedure installs mirror-registry on the local machine and starts an image registry, which listens on port 443 by default, to store platform images.

Note

The installation makes the following changes to the machine: it creates the /opt/mirror-registry directory, which contains the binaries, configuration, and local storage, and it creates the mirror-registry systemd service, which starts at boot. The installation script does not change firewall rules.

Supported Flags

The following flags apply to the install and upgrade commands. Default values apply to the first installation. When you run install again or run upgrade, unspecified flags keep their existing configuration.

General flags

FlagDescription
--listenThe listening port, on all IPv4 and IPv6 addresses of all interfaces. If the port is not 443, include it when you access the registry and when you specify the registry address. Defaults to 443.
--auth-fileThe path of the credentials file. Line 1 is the username and line 2 is the password. No default value. Required for the first installation.
--tls-certThe path of the TLS certificate, including the full chain. Specify it together with --tls-key. Defaults to an automatically generated self-signed certificate.
--tls-keyThe path of the TLS private key. Specify it together with --tls-cert. Defaults to an automatically generated private key.
--regen-certRegenerates the self-signed certificate. Defaults to keeping the existing certificate.

Storage flags

FlagDescription
--storageThe storage type, filesystem or s3, which cannot be changed after installation. Defaults to filesystem.
--data-dirThe image data directory for local storage, as an absolute path. Cannot be used with --storage s3. Defaults to /opt/mirror-registry/data.
--s3-endpointThe address of the S3-compatible service, without the bucket name or a path. Required with --storage s3. If it has no scheme, https:// is used. No default value.
--s3-regionThe region of the bucket. Required with --storage s3. No default value.
--s3-bucketThe name of the bucket, which must already exist. Required with --storage s3. No default value.
--s3-credentials-fileThe path of the S3 credentials file. Line 1 is the access key and line 2 is the secret key. Defaults to no credentials file.
--s3-root-dirThe path prefix inside the bucket, which can contain only letters, digits, ., _, -, and /. Defaults to the bucket root.

Prepare the Package and Credentials

  1. Download the mirror-registry-<version>.<arch>.tar.gz package that matches the machine architecture from the Core Package download page in the Customer Portal, and extract it:

    tar xzf mirror-registry-<version>.<arch>.tar.gz
    cd mirror-registry-<version>.<arch>
  2. Create the credentials file. Line 1 is the username and line 2 is the password:

    {
      read -rp "Registry username: " USERNAME
      read -rsp "Registry password: " PASSWORD; printf '\n'
      (umask 077; printf '%s\n%s\n' "$USERNAME" "$PASSWORD" > /root/registry.auth)
      unset USERNAME PASSWORD
    }

    The username cannot contain whitespace or :, and the password can be at most 72 bytes. You can delete this file after the installation.

Deploy with Local Storage

To use the default data directory, /opt/mirror-registry/data, make sure that its file system has at least 200 GB, and run:

./mirror-registry install --auth-file /root/registry.auth

To store the data on a dedicated data disk, prepare the disk with LVM, and then install mirror-registry with --data-dir:

  1. Create a logical volume on the data disk and mount it. The following example uses the empty disk /dev/vdb and creates a dedicated volume group, vg_registry, instead of reusing the volume group of the system disk, so that the image data does not share the root file system. The commands erase all data on the disk and require the lvm2 package. Replace the device name and the mount point for your environment. The mount point must be on a writable path.

    pvcreate /dev/vdb
    vgcreate vg_registry /dev/vdb
    lvcreate -l 100%FREE -n lv_data vg_registry
    mkfs.xfs /dev/vg_registry/lv_data
    mkdir -p /cpaas/registry
    echo "UUID=$(blkid -s UUID -o value /dev/vg_registry/lv_data) /cpaas/registry xfs defaults 0 0" >> /etc/fstab
    systemctl daemon-reload
    mount -a
    df -h /cpaas/registry

    The df output shows the logical volume mounted on /cpaas/registry. The /etc/fstab entry mounts it automatically after a reboot.

  2. Install mirror-registry with the mount point as the data directory. The following example also changes the listening port:

    ./mirror-registry install \
      --auth-file /root/registry.auth \
      --data-dir /cpaas/registry \
      --listen 8443

To expand the storage later, add a disk, for example /dev/vdc, to the volume group, and then extend the logical volume and its file system:

pvcreate /dev/vdc
vgextend vg_registry /dev/vdc
lvextend -r -l +100%FREE /dev/vg_registry/lv_data

Deploy with S3-Compatible Object Storage

  1. Create a bucket on the S3 service, and prepare an access key that can list, read, write, and delete objects and perform multipart uploads in the bucket. The script does not create the bucket. Paths inside the bucket do not need to exist in advance.

  2. Create the S3 credentials file. Line 1 is the access key and line 2 is the secret key:

    {
      read -rp "S3 access key: " S3_ACCESS_KEY
      read -rsp "S3 secret key: " S3_SECRET_KEY; printf '\n'
      (umask 077; printf '%s\n%s\n' "$S3_ACCESS_KEY" "$S3_SECRET_KEY" > /root/s3.cred)
      unset S3_ACCESS_KEY S3_SECRET_KEY
    }
  3. Set --s3-endpoint and --s3-region for your storage:

    Storage--s3-endpoint--s3-region
    MinIOhttps://<address>:9000us-east-1
    Ceph RGWhttps://<address>:<RGW-port>us-east-1
    AWS S3https://s3.<region>.amazonaws.comThe region of the bucket
  4. Install mirror-registry:

    ./mirror-registry install \
      --auth-file /root/registry.auth \
      --storage s3 \
      --s3-endpoint https://minio.example.com:9000 \
      --s3-region us-east-1 \
      --s3-bucket registry \
      --s3-credentials-file /root/s3.cred

    If the bucket has other content, add --s3-root-dir mirror/registry to store the data under a path inside the bucket.

    At the end of the installation, the script checks S3 connectivity. If S3 check failed appears, check the endpoint, bucket name, region, and keys.

Verify the Deployment

  1. Confirm that the registry is available:

    ./mirror-registry status
    curl -k -u <username> https://localhost/v2/_catalog

    A new registry returns {"repositories":[]}. If you changed the port, add it to the URL.

  2. Continue with Prepare the Target-Version Payload to upload the platform images to this registry. Set the registry address to <host-or-IPv4>[:port]. If the registry uses IPv6, use a domain name. Use the username and password from the credentials file.

Note

If images are affected by storage layer problems, you can upload the platform images again, or reinstall mirror-registry on more stable storage.

Deploy mirror-registry for Disaster Recovery

For global cluster disaster recovery, deploy a mirror-registry on both the primary side and the standby side, and use a dedicated domain name as the image registry address. The domain name normally resolves to the mirror-registry on the primary cluster side. If a failure occurs, switch its DNS resolution to the mirror-registry on the standby cluster side.

Important

The two mirror-registry instances do not synchronize data, so you must upload the platform images to both. Until the DNS resolution is switched, cluster nodes cannot pull images from mirror-registry.

Prerequisites

  • A machine on each of the primary side and the standby side that meets the prerequisites.
  • A domain name dedicated to the image registry that all cluster nodes can resolve. This domain name is independent of the domain name of the platform access address.

Procedure

  1. Install mirror-registry on both sides. Both sides must use the same username and password, so we recommend using the same credentials file. The listening port must also be the same.

  2. Resolve the image registry domain name to the mirror-registry on the primary cluster side.

  3. Upload the platform images to the mirror-registry on each side. When you upload, set the registry address to the address of the corresponding machine. For details, see Prepare the Target-Version Payload.

  4. When you install the primary and standby clusters, specify the same image registry address, username, and password on both sides.

If a failure occurs

  • Switch the DNS resolution of the image registry domain name to the mirror-registry on the standby cluster side.

Update mirror-registry

This procedure updates mirror-registry by using the upgrade command. Updates provide new features, bug fixes, and security vulnerability fixes.

Important

The service restarts during the update, so the registry is briefly unavailable.

Prerequisites

  • You have installed mirror-registry.

Procedure

  • Extract the new package and run the following command from its directory:

    ./mirror-registry upgrade

    Unspecified flags keep their existing configuration. If the service is not healthy after the update, it automatically rolls back to the previous version.

Replace the mirror-registry SSL/TLS Certificates

Use this procedure to replace the current certificate or to update it periodically.

Important

The service restarts after you run the following commands, so the registry is briefly unavailable.

Prerequisites

  • You have installed mirror-registry.

Procedure

  • To use your own certificate, run the following command. Specify both flags together. The private key cannot be passphrase-protected:

    ./mirror-registry install --tls-cert <full-chain-file> --tls-key <key-file>
  • If the IP address or host name of the machine changes, regenerate the self-signed certificate:

    ./mirror-registry install --regen-cert
    Note

    The automatically generated certificate covers the host name, localhost, 127.0.0.1, ::1, and the global IP addresses of the machine at generation time, and is valid for 10 years. If you access the registry through another domain name, use your own certificate.

Troubleshoot mirror-registry

mirror-registry installs the mirror-registry.service systemd service.

Prerequisites

  • You have installed mirror-registry.

Procedure

  1. Check the service status:

    systemctl status mirror-registry
  2. Check the service logs:

    journalctl -u mirror-registry