Skip to content

Hosted Control Plane

kMetal uses Kamaji to implement hosted control planes. Control plane components run as pods in the under cluster rather than on dedicated master nodes.

Overview

Each tenant cluster's control plane (api-server, controller-manager, scheduler) runs as pods in the under cluster, alongside a per-tenant etcd. Tenant worker nodes connect to that control plane like they would to any Kubernetes cluster. No dedicated master nodes per tenant.

Core Concepts

TenantControlPlane Resource

The core abstraction is the TenantControlPlane custom resource, which defines a complete Kubernetes control plane for a tenant cluster. This resource specifies:

  • Control plane configuration: Replicas, resources, networking
  • Kubernetes version: Version and feature gates
  • Network settings: API endpoint, certificates, load balancing
  • Storage backend: etcd or external database connection

See Kamaji for reference configuration.

Datastore Resource

Hosted control planes require persistent storage for cluster state. Kamaji supports multiple datastore backends via Datastore custom resource:

Shared etcd

Multiple tenant control planes share a single etcd cluster, with data isolated by namespace prefixes. This provides:

  • Cost efficiency with single etcd cluster serves multiple tenants
  • Operational simplicity since single datastore to manage and backup
  • Optimization for high-density cluster deployments

Dedicated etcd

Each tenant gets its own etcd instance for maximum isolation and performance:

  • Complete isolation with no shared resources between tenants
  • Custom performance tuning with etcd configured per tenant requirements
  • Enhanced security with physical separation of tenant data

PostgreSQL

Uses CloudNativePG to provide PostgreSQL-based persistence:

  • Leverage existing PostgreSQL expertise
  • Point-in-time recovery, streaming replication
  • Integrate with existing database infrastructure

See Kamaji for reference configuration.

High Availability

Multi Replica Deployment

Production control planes run with multiple replicas for availability. The platform:

  • Distributes replicas across different nodes
  • Maintains quorum for cluster operations
  • Handles rolling updates without downtime

See Kamaji for configuration details.

Load Balancer Integration

Control planes integrate with load balancers for:

  • High availability with multiple API server endpoints
  • Distributed client connections
  • Automatic failover for unhealthy instances

See Kamaji for configuration details.

Resource Monitoring

Platform monitoring tracks control plane resource usage:

  • CPU and memory utilization per control plane
  • Request latency and throughput metrics
  • Resource efficiency across the under cluster

See Kamaji for configuration details.

Security Model

Certificate Management

Each hosted control plane gets its own PKI infrastructure:

  • Certificates are created during cluster creation
  • Certificate are automatically renewed before expiry
  • Each tenant gets separate certificate authority and certificates

See Kamaji for reference configuration.

Konnectivity — Reaching the Workers

The api-server reaches kubelet (kubectl exec, kubectl logs, port-forward, metrics-server scrape, webhook callouts) over Konnectivity, a gRPC reverse tunnel that ships as a sidecar in every TenantControlPlane pod.

  • A konnectivity-agent runs as a DaemonSet on each tenant worker. On startup it dials out to the tenant's konnectivity-server (port 8132), which lives next to the api-server in the hosted control plane.
  • All api-server → kubelet traffic rides back down that already-established tunnel.
  • The connection is initiated by the worker, so no inbound firewall rule, no per-worker NAT, and no overlay route from the under cluster to the worker is required.

The tunnel uses TLS with tenant-specific certificates issued by the same per-tenant PKI that signs api-server certs.