Skip to content

Control Plane External Access

Every tenant cluster needs an externally reachable endpoint for its api-server — that's how tenants run kubectl against their cluster. kMetal supports two patterns for publishing that endpoint, chosen based on tenant density.

Pattern 1 — LoadBalancer + NAT (default for ≤ 100 tenants)

Each tenant's TenantControlPlane Service is type: LoadBalancer. MetalLB allocates one VIP per tenant from the under cluster's provider pool. The tenant's kubeconfig points at that VIP on port 6443 (or the configured port).

Tenant kubectl ──▶ [ MetalLB VIP ] ──▶ [ TCP Service ] ──▶ [ TCP pod ]
   (over WAN)        (per-tenant)        (cluster IP)        (api-server)
  • One VIP per tenant. Straightforward to reason about, easy to firewall per-tenant if needed.
  • Scales to roughly 100 tenants before the per-VIP cost (IPs, MetalLB announcement footprint, edge-router routes) becomes operationally heavy.
  • Works with the existing MetalLB setup on the under cluster — no extra components.

This is the default mode; it's what a fresh kMetal install uses out of the box.

Pattern 2 — Ingress / SNI (for ≥ 100 tenants)

A single shared IP (or small handful of IPs) on the under cluster handles all tenants' api-server traffic. A TLS-passthrough ingress (HAProxy, NGINX, Envoy) routes by SNI hostname to the right TenantControlPlane Service. Each tenant's kubeconfig points at a tenant-specific FQDN that all resolve to the same shared IP.

                       ┌─▶ [ tenant-a TCP Service ]
[ tenant-a kubeconfig ─┤
  server: a.tcp.example ]      ┌────────────────────┐
                       └──▶  ──▶│ TLS-passthrough    │──▶ (by SNI)
[ tenant-b kubeconfig ─┐        │ ingress on shared  │
  server: b.tcp.example ]  ──▶  │ provider VIP       │
                       └─▶ [ tenant-b TCP Service ]
  • One under-cluster VIP serves N tenants. No per-tenant IP allocation.
  • TLS terminates inside the tenant's TCP pod (passthrough), so tenant CA + client certs remain end-to-end.
  • Scales past hundreds of tenants on a single ingress footprint.
  • Tenant kubeconfigs need a per-tenant FQDN, so a DNS plan is required (wildcard DNS to the ingress VIP works).

This is the right pattern when tenant density gets large enough that allocating MetalLB VIPs one-per-tenant becomes the bottleneck.

Konnectivity makes both possible

Worker → api-server traffic doesn't go through the externally exposed endpoint. The Konnectivity tunnel handles api-server ↔ kubelet traffic over a separate reverse-dialled connection. That means:

  • The external endpoint only carries human kubectl / CI / controller-manager-from-elsewhere traffic, not worker bootstrap traffic.
  • The same patterns above work for stretched topologies where workers and the api-server endpoint are on different network planes.
  • Tenants can lock down their api-server endpoint to specific source IPs (their own offices, their own CI) without breaking worker functionality.

Choosing between the patterns

Concern LoadBalancer + NAT Ingress / SNI
Tenant count ≤ ~100 Unlimited (single ingress scales)
IPs consumed 1 per tenant 1 shared
DNS plan Per-tenant or shared Per-tenant FQDN required
Edge router config Static route to provider pool Same
Kubernetes feature used type: LoadBalancer TLS passthrough ingress
Best for Few tenants, simple operations High-density tenant farms

Both patterns can coexist — a kMetal under cluster can serve some tenants via per-tenant LoadBalancer VIPs and others via SNI ingress on the same shared IP. Tenant kubeconfig is what binds a tenant to a pattern.