Scaling the kMetal

This guide provides comprehensive strategies for scaling the kMetal to handle increasing workloads, tenant clusters, and performance requirements.

Scaling Overview

Scaling Dimensions

1. Under Cluster: Node scaling, resource scaling
2. Platform Components: Horizontal scaling, vertical scaling
3. Tenant Clusters: Control plane scaling, worker node scaling

Scaling Strategies

Strategy	Use Case	Benefits	Considerations
Horizontal Scaling	Increased workload	Better fault tolerance	Network complexity
Vertical Scaling	Resource-intensive workloads	Simplified architecture	Single point of failure
Cluster Scaling	More tenant clusters	Workload isolation	Management overhead
Component Scaling	Specific bottlenecks	Targeted optimization	Dependency management

Under Cluster Scaling

Node Scaling

Adding Under Cluster Nodes

# Check current node capacity
kubectl get nodes -o wide
kubectl top nodes

# Add new nodes to cluster
# Example for kubeadm clusters
kubeadm token create --print-join-command

# On new node
kubeadm join <master-ip>:6443 --token <token> --discovery-token-ca-cert-hash <hash>

# Label nodes for specific workloads
kubectl label nodes <node-name> node-role.kubernetes.io/platform=true
kubectl label nodes <node-name> workload-type=control-plane

Node Affinity for Platform Components

# Platform component node affinity
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:

        - matchExpressions:
          - key: node-role.kubernetes.io/platform
            operator: In
            values: ["true"]
      preferredDuringSchedulingIgnoredDuringExecution:

      - weight: 100
        preference:
          matchExpressions:

          - key: workload-type
            operator: In
            values: ["control-plane"]

Resource Planning

Capacity Planning Formula

Total Resource Requirement = Base Platform + (Tenant Clusters * Per-Cluster Overhead)

Base Platform Resources:

- CPU: 4-8 cores
- Memory: 8-16 GB
- Storage: 100-500 GB

Per-Tenant Cluster Overhead:

- Control Plane: 1-2 cores, 2-4 GB RAM
- Monitoring: 0.5 cores, 1 GB RAM
- Network: 0.1 cores, 0.5 GB RAM

Resource Monitoring

# Monitor resource usage
kubectl top nodes
kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory

# Check resource requests vs limits
kubectl describe nodes | grep -A 5 "Allocated resources"

# Monitor storage usage
df -h /var/lib/kubelet
kubectl get pvc -A

Platform Component Scaling

Platform components are not scaled per component. Their replica counts and resource requests are pinned by the kMetal release, so there is no replicas value to raise — see Component Configuration.

What you do control is where they run. spec.placement moves every control-plane-workload component — Kamaji, cert-manager, Capsule, the CAPI operator, Sveltos, MetalLB's controller — onto a class of node you choose:

spec:
  placement:
    nodeSelector:
      node-role.kubernetes.io/platform: ""
    tolerations:
      - key: node-role.kubernetes.io/platform
        operator: Exists
        effect: NoSchedule

That is the lever for platform capacity: give the components more, or more powerful, nodes to sit on. Label the nodes before changing it, or every platform Pod sits Pending with no other symptom.

Placement deliberately does not reach the DaemonSets and per-node plugins. MetalLB's speaker has to answer for a VIP from any node that can carry it, and Kube-OVN's per-node workloads are placed by node label because they need the overlay interface present.

If the platform's own components are the bottleneck on a well-provisioned node pool, that is a sizing question for a kMetal release rather than a values edit — tell us what you are seeing.

The operator itself is separate from the platform it installs, and its chart does expose the usual controls:

replicaCount: 2                 # needs leaderElection.enabled; buys failover, not throughput
resources:
  requests: { cpu: 100m, memory: 128Mi }
  limits:   { memory: 512Mi }

Tenant Cluster Scaling

User Guide Content

Tenant cluster scaling operations (control plane, worker nodes, auto-scaling) are documented in the Tenant Guide: Scale Clusters.

This section focuses on platform-level scaling. For tenant-specific scaling, refer to the user guide.

Data Layer Scaling

Every hosted control plane keeps its etcd data on a Kamaji DataStore, and that data lives on the StorageClass named by spec.storage.etcdClassName — or by spec.storage.underclusterClassName, which it falls back to when unset.

Scaling this layer is therefore a storage question, not a kMetal one: as tenant count grows, so does the demand on that class. Watch it, and be deliberate about a node-local class — it ties the durability of every tenant control plane to the node holding the data.

See Hosted Control Plane for the model.

t.b.d. — Worked datastore sizing guidance per tenant tier is t.b.d. in this section.

Monitoring and Observability Scaling

kMetal does not install a monitoring stack. If you operate Prometheus / Alertmanager separately, scale it according to that stack's own guidance.

Auto-scaling the platform

There is none, by design. The platform's components are sized by the release and reconciled continuously, so an HPA or VPA attached to one of their Deployments fights the operator: it changes a field the operator owns, the operator changes it back, and the Pods roll every interval.

Tenant-count growth is absorbed by node capacity, not by scaling the controllers — see the placement lever above.

Autoscaling inside tenant clusters is a different mechanism entirely, and is supported — see Tenant Cluster Autoscaling below.

Scaling Monitoring

If you have a Prometheus stack deployed (operator's choice), useful scrape signals include:

• Per-node CPU/memory utilization (node_cpu_seconds_total, node_memory_*).
• Per-tenant control plane pod resource usage (container_memory_usage_bytes filtered by namespace="kmetal-kamaji").
• Tenant count growth (count of tenantcontrolplanes.kamaji.clastix.io resources over time).

Exact metric names depend on the Kamaji and exporter versions you've deployed; verify against your live stack before wiring alerts.

t.b.d. — A canonical PromQL ruleset for scaling-related alerts is t.b.d. in this section.

Scaling Alerts

Scaling Alerts Configuration

# Scaling alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: scaling-alerts
spec:
  groups:

  - name: scaling
    rules:

    - alert: HighNodeUtilization
      expr: node_cpu_utilization > 80
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Node {{ $labels.instance }} CPU utilization is high"
        description: "Node {{ $labels.instance }} has CPU utilization above 80%"

    - alert: HighTenantDensity
      expr: tenant_clusters_per_node > 20
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High tenant density on node {{ $labels.node }}"
        description: "Node {{ $labels.node }} has more than 20 tenant clusters"

    - alert: KamajiControllerMemoryHigh
      expr: kamaji_controller_memory_usage > 6000000000  # 6GB
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Kamaji controller memory usage is high"
        description: "Kamaji controller memory usage is above 6GB"

Tenant Cluster Autoscaling

Worker scaling inside a tenant cluster is driven by a per-tenant Cluster Autoscaler running in the under cluster, in the tenant's namespace. One autoscaler instance per tenant cluster, watching that tenant's kubeconfig.

The flow:

1. A workload in the tenant cluster goes Pending because no node has room.
2. The autoscaler, watching the tenant's api-server, sees the unschedulable pod.
3. It scales the matching CAPI MachineDeployment (in the under cluster) up by one replica.
4. CAPK provisions a new KubeVirt VM as a worker node.
5. The new worker joins the tenant cluster; the Pending pod schedules onto it.

Scaling down works the inverse way: when nodes are underutilized for the configured period, the autoscaler scales the MachineDeployment down and the corresponding worker VM is decommissioned.

Multiple node sizes per tenant

A tenant can expose multiple worker shapes by giving each one its own MachineDeployment with a distinct nodeSelector / taints profile. The autoscaler will scale each MachineDeployment independently based on which one matches the pending workload's scheduling constraints. Typical use: a small pool for control-plane-light workloads, a large pool for memory-heavy work, an optional gpu pool when GPU passthrough is in play.

Scale-from-zero

MachineDeployments can be configured with min: 0, so a tenant pool sits at zero VMs when no workload needs it and provisions on demand. This is the right setting for niche pools (rarely-used GPU pool, on-demand batch pool) — it eliminates the cost of idle VMs.

---