Skip to content

Hosted Control Plane

kMetal uses Kamaji to implement hosted control planes. Control plane components run as pods in the under cluster rather than on dedicated master nodes.

Overview

Each tenant cluster's control plane (api-server, controller-manager, scheduler) runs as pods in the under cluster, alongside a per-tenant etcd. Tenant worker nodes connect to that control plane like they would to any Kubernetes cluster. No dedicated master nodes per tenant.

Core Concepts

KamajiControlPlane Resource

On kMetal a control plane is a Cluster API object. A Cluster names its control plane by a reference template (KamajiControlPlaneTemplate), and Cluster API materialises a KamajiControlPlane for it — the control-plane provider's resource, the same slot a KubeadmControlPlane would occupy on a cluster whose control plane runs on machines.

That is what puts hosted control planes on the standard path. The tenant declares a Cluster against the kMetal ClusterClass and the rest follows: Cluster API creates the KamajiControlPlane from the given KamajiControlPlaneTemplate, the Kamaji provider turns it into a running control plane, and every lifecycle operation — version upgrades, endpoint changes, scaling replicas — is a change to that object rather than an out-of-band procedure.

What it carries:

  • Control plane configuration: replicas, resources
  • Kubernetes version: what the control plane runs, and what workers are expected to join at
  • Network settings: API endpoint, service type and address, certificate SANs
  • Datastore: which backing store holds this cluster's state

Underneath, Kamaji reconciles it into a TenantControlPlane — Kamaji's own resource, and where the running pods and their status live. Reach for it when diagnosing a control plane that is not coming up; the KamajiControlPlane is what you edit.

See Kamaji for reference configuration.

Datastore Resource

Hosted control planes require persistent storage for cluster state. Kamaji supports multiple datastore backends via Datastore custom resource:

Shared etcd

Multiple tenant control planes share a single etcd cluster, with data isolated by namespace prefixes. This provides:

  • Cost efficiency with single etcd cluster serves multiple tenants
  • Operational simplicity since single datastore to manage and backup
  • Optimization for high-density cluster deployments

Dedicated etcd

Each tenant gets its own etcd instance for maximum isolation and performance:

  • Complete isolation with no shared resources between tenants
  • Custom performance tuning with etcd configured per tenant requirements
  • Enhanced security with physical separation of tenant data

High Availability

Multi Replica Deployment

Production control planes run with multiple replicas for availability. The platform:

  • Distributes replicas across different nodes
  • Maintains quorum for cluster operations
  • Handles rolling updates without downtime

See Kamaji for configuration details.

Load Balancer Integration

Control planes integrate with load balancers for:

  • High availability with multiple API server endpoints
  • Distributed client connections
  • Automatic failover for unhealthy instances

See Kamaji for configuration details.

Resource Monitoring

Platform monitoring tracks control plane resource usage:

  • CPU and memory utilization per control plane
  • Request latency and throughput metrics
  • Resource efficiency across the under cluster

See Kamaji for configuration details.

Security Model

There are no control plane nodes

In a conventional cluster, the control plane is software on machines. Those machines are the most valuable target on the cluster, and most of what hardening guidance asks of you is about them: file permissions on /etc/kubernetes/pki, the static pod manifests in /etc/kubernetes/manifests, the etcd data directory, the kubelet configuration on a control plane node, the admin.conf sitting on disk.

A tenant cluster on kMetal has none of those machines. kubectl get nodes returns worker nodes and nothing else, because the control plane is not on a node — it is pods on the under cluster, on hardware the tenant has no account on and no route to.

This changes the shape of a whole class of risk. A tenant's cluster-admin is genuinely an administrator of their Kubernetes API, and simultaneously has no host access to the layer that serves it — not because a control was applied, but because there is nothing there to access. A privileged pod, a hostPath mount, a compromised kubelet, an escaped container on a worker: each of these lands on a VM that runs tenant workloads and holds no control plane material. There is no lateral move to a control plane host, because the tenant's cluster does not contain one.

The same is true in the other direction. The control plane's configuration — its API server flags, its admission chain, how its state is stored and encrypted — is set by the platform, once, for every tenant. A tenant cannot weaken it, and does not have to be trusted to get it right.

What that means for security posture

Security posture tooling (KSPM, CIS Kubernetes Benchmark, and the audit that follows them) splits its checks into control plane configuration and workload configuration. On kMetal, the first half is not a per-cluster question.

  • Control plane controls are owned once. The platform team configures and evidences them for the whole platform, rather than every tenant re-implementing and every audit re-checking them per cluster.
  • Findings that cannot occur do not have to be managed. Node-level control plane findings have no surface to appear on: no manifest to tamper with, no key material on a reachable disk, no control plane kubelet to proxy to.
  • Posture is uniform by construction. Every tenant cluster's control plane is built the same way by the same operator, so a fleet does not drift into a hundred slightly different control planes with a hundred slightly different findings.
  • Tenants keep the half that is theirs. Workload posture — RBAC inside their cluster, pod security, network policy, image provenance — stays entirely with the tenant, where it belongs.

The practical effect is that adding a tenant cluster adds workload posture to review, not another control plane to harden and evidence.

Certificate Management

Each hosted control plane gets its own PKI infrastructure:

  • Certificates are created during cluster creation
  • Certificate are automatically renewed before expiry
  • Each tenant gets separate certificate authority and certificates

See Kamaji for reference configuration.

Konnectivity — Reaching the Workers

The api-server reaches kubelet (kubectl exec, kubectl logs, port-forward, metrics-server scrape, webhook callouts) over Konnectivity, a gRPC reverse tunnel that ships as a sidecar in every TenantControlPlane pod.

  • A konnectivity-agent runs as a DaemonSet on each tenant worker. On startup it dials out to the tenant's konnectivity-server (port 8132), which lives next to the api-server in the hosted control plane.
  • All api-server → kubelet traffic rides back down that already-established tunnel.
  • The connection is initiated by the worker, so no inbound firewall rule, no per-worker NAT, and no overlay route from the under cluster to the worker is required.

The tunnel uses TLS with tenant-specific certificates issued by the same per-tenant PKI that signs api-server certs.