Skip to content

Under Cluster Setup

This guide covers preparation of the under cluster — the Kubernetes cluster that hosts the kMetal platform components and runs every tenant cluster's hosted control plane.

System Requirements

Hardware

Minimum (lab / evaluation):

  • 3 control-plane nodes
  • 8 CPU cores per node
  • 16 GB RAM per node
  • 100 GB SSD per node
  • 1 Gbps network

Recommended (production):

  • 3 control-plane + 3+ worker nodes
  • 16 CPU cores per node
  • 64 GB RAM per node
  • 500 GB NVMe per node
  • 10 Gbps network with redundancy

Network

Plan for four isolated VLANs, in addition to each machine's out-of-band IPMI/Redfish interface:

Network Carries Later configured as
Management OS access, under-cluster Kubernetes, console/UI, internet egress for the under-cluster spec.networking.loadBalancer.management (MetalLB mgmt-pool)
Tenant Tenant control-plane access (external kubeconfig endpoints) spec.networking.loadBalancer.tenantControlPlanes (MetalLB tenant-cp-pool)
Overlay Kube-OVN Geneve tunnel between under-cluster nodes spec.networking.kubeOVN.tunnelInterface
Provider Tenant egress (SNAT) and external IPs for tenant services spec.networking.providerNetworks[].interface
Storage (optional) Connects the machines to the storage backend, if it isn't node-local n/a — not part of spec.networking

All four required VLANs must be separate, isolated segments on your switch fabric — collapsing onto shared broadcast domains breaks the tenant isolation the platform is built to provide. They can optionally consolidate onto one or two bonds per machine — each bond aggregating two or more physical NICs for redundancy — with each VLAN riding as a tagged sub-interface on top. This documentation's examples elsewhere use .204 (management), bond0.205 (overlay), bond0.206 (provider) and .208 (tenant). Ideally each machine has two or more physical NICs.

IPMI/Redfish: every server needs its out-of-band management interface, kept on a network separate from the ones above — it gives console and power control independent of the node OS or data-plane network, and stays reachable even when Kubernetes itself is down.

See Networking Configuration for how each of these becomes a spec.networking field once the cluster is up.

  • A reserved IP range on the tenant network for MetalLB to advertise (tenant control-plane VIPs come from this range)
  • DNS resolution for the under cluster's API endpoint and any tenant FQDNs you plan to expose
  • Firewall: 6443 (Kubernetes API), 443 (HTTPS), 2379-2380 (etcd, internal), 10250 (kubelet), 8132 (Konnectivity from workers)

Storage

  • Storage you bring yourself. kMetal installs no CSI driver, provisioner or StorageClass. Two classes must exist before installing: one for the platform's own volumes, one handed to tenants — with an optional third for tenant control-plane etcd. The platform class's driver must ship a VolumeSnapshotClass, because golden OS images are snapshotted on it and every tenant machine's root disk is cloned from that snapshot. See Storage.

See Storage Configuration for details.

Node roles

kMetal separates the nodes that run the platform from the nodes that run tenant VMs, and it does so by label.

Label What lands there
node-role.kubernetes.io/control-plane The platform's own controllers, by default — see placement.
node-role.kubernetes.io/compute KubeVirt's virt-handler, and therefore every tenant machine and every image-builder installer VM.
node-role.kubernetes.io/network Kube-OVN's central nodes, if you dedicate nodes to them.

Label the compute nodes before installing:

kubectl label node <node> node-role.kubernetes.io/compute=""

No compute node means no tenant machine, and no error saying so

virt-handler is a DaemonSet pinned to that label, and KubeVirt starts a VM only on a node whose handler is running. With the label on nothing, the platform installs cleanly, every component reports healthy, and the first tenant cluster gets a control plane that comes up and workers that never appear. The golden-image pipeline stalls for the same reason — its installer is itself a VM.

The pinning is deliberate rather than incidental: unpinned, the DaemonSet covers every schedulable node, and a tenant VM lands on a control-plane node beside the platform that is supposed to survive it.

Commercial Access

Commercial Platform Access Required

kMetal is a commercial product. You need a registry username and token from Clastix to pull the chart and its container images.

  1. Request access: Contact Clastix.
  2. Receive a registry username and time-limited token (ghcr.io/clastix).

  3. Verify access before installation:

    helm registry login ghcr.io -u <username> -p <token>
helm show chart oci://ghcr.io/clastix/kmetal

Kubernetes Cluster Setup

Option 1: Existing Kubernetes Cluster

If you already have a Kubernetes cluster that meets the requirements above, verify it:

kubectl version
kubectl get nodes
kubectl get storageclass

Option 2: Create a new cluster

Bring up a 3-node HA control plane plus the worker nodes you need with YAKI, the CLASTIX node bootstrap script. It is a single Bash script, served at https://goyaki.clastix.io, that acts on one host per invocation: init on the first control-plane node, join on every other.

# On the first control-plane node
curl -sfL https://goyaki.clastix.io | \
  KUBERNETES_VERSION=v1.35.8 bash -s init

# On every other node — add JOIN_ASCP=true for a control-plane join
curl -sfL https://goyaki.clastix.io | \
  KUBERNETES_VERSION=v1.35.8 \
  JOIN_URL=<endpoint>:6443 \
  JOIN_TOKEN=<token> \
  JOIN_TOKEN_CACERT_HASH=<hash> \
  bash -s join

# The full variable reference
curl -sfL https://goyaki.clastix.io | bash -s help

The Getting Started walkthrough drives the same script through yctl, which orchestrates it over SSH — convenient for a lab, and the same mechanism for a fleet.

Why YAKI rather than driving kubeadm yourself

Not because of the install — kubeadm by hand produces the same cluster. Because of day 2.

kMetal installs yaki-operator, which upgrades the under cluster's nodes declaratively: you create a KubernetesNodeUpgrade, and the operator plans the order, moves the control plane first, bounds worker concurrency and verifies each node came back at the target version. The upgrade it runs on each host is YAKI's own upgrade flow, so a cluster bootstrapped with YAKI needs no Terraform run, no configuration-management pass and no Metal3/Ironic stack to stay aligned — the intent is an object in the cluster's own API.

See Node Upgrades.

Other installers work too — kMetal only needs a conformant Kubernetes cluster. Note what you give up: yaki-operator upgrades kubeadm nodes, so on a distribution that does not use kubeadm (RKE2, Talos, k0s) node upgrades go back to being that distribution's problem rather than a resource you create.

Install a primary CNI first

kMetal does not install a primary CNI, and will not start without one — pod networking has to work before the operator runs.

Install the CNI of your choice as part of bringing up the under cluster. Kube-OVN is installed later by kMetal as a non-primary CNI, through Multus: it never takes over pod eth0, and provides the tenant-facing substrate (VPCs, subnets, EIPs, SNAT) alongside whatever you chose.

Its overlay subnet must not overlap your primary CNI's pod network.

Next Steps

When the under cluster is up and you have registry credentials, continue to kMetal Installation.