Multi-Tenancy

Configure secure isolation between tenants in kMetal.

Tenant Isolation Model

kMetal provides tenant isolation at multiple layers:

1. Control Plane Isolation: Each tenant has a dedicated Kubernetes control plane (TenantControlPlane) with its own etcd.
2. Compute Isolation: Tenant workers run as KubeVirt VMs on the under cluster — kernel-level isolation via KVM.
3. Network Isolation: Each tenant operates in an isolated OVN VPC; cross-tenant traffic is dropped by default.
4. Namespace Isolation: Tenant control plane and platform resources are separated by Kubernetes namespaces on the under cluster.
5. Resource Isolation: ResourceQuotas and LimitRanges on the under cluster prevent resource exhaustion across tenants.
6. RBAC Isolation: Role-based access control per tenant on the under cluster.

Tenant Namespace Configuration

Create Tenant Namespace

# tenant-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-acme
  labels:
    tenant: acme
    environment: production
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-acme-quota
  namespace: tenant-acme
spec:
  hard:
    requests.cpu: "100"
    requests.memory: "200Gi"
    requests.storage: "1Ti"
    persistentvolumeclaims: "50"
    tenantcontrolplanes.kamaji.clastix.io: "10"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-acme-limits
  namespace: tenant-acme
spec:
  limits:

  - max:
      cpu: "8"
      memory: "16Gi"
    min:
      cpu: "100m"
      memory: "128Mi"
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "250m"
      memory: "256Mi"
    type: Container

Tenant RBAC

# tenant-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-acme-admin
  namespace: tenant-acme
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kmetal-cluster-manager
subjects:

- kind: Group
  name: acme-admins
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-acme-viewer
  namespace: tenant-acme
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:

- kind: Group
  name: acme-viewers
  apiGroup: rbac.authorization.k8s.io

Tenant Control Plane Isolation

Dedicated Control Plane per Tenant

# tenant-cluster.yaml — Cluster CR; the ClusterClass auto-creates KamajiControlPlane + KubevirtCluster
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: acme-prod-cluster
  namespace: tenant-acme
  labels:
    tenant: acme
    environment: production
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["10.200.0.0/16"]
    services:
      cidrBlocks: ["10.201.0.0/16"]
  topology:
    classRef:
      name: kubevirt-kubeadm
      namespace: kmetal-capi-providers
    version: v1.34.1
    variables:
      - name: controlPlane
        value:
          dataStoreName: tenant-acme-etcd
          network:
            serviceAddress: <metallb-vip>
            serviceType: LoadBalancer
            serviceAnnotations:
              metallb.universe.tf/address-pool: tenant-acme-pool
      - name: machineSize
        value: medium
      - name: bootstrapConfig
        value:
          serverAddress: <bootstrap-server>
      - name: network
        value:
          subnet: tenant-acme
    workers:
      machineDeployments:
        - class: default-worker
          name: md-0
          replicas: 3

Dedicated Datastore per Tenant

# tenant-datastore.yaml
apiVersion: kamaji.clastix.io/v1alpha1
kind: DataStore
metadata:
  name: tenant-acme-etcd
spec:
  driver: "etcd"
  endpoints:

    - "https://etcd-acme-0.tenant-acme:2379"
    - "https://etcd-acme-1.tenant-acme:2379"
    - "https://etcd-acme-2.tenant-acme:2379"
  tlsConfig:
    certificateAuthority:
      certificate:
        secretRef:
          name: etcd-acme-ca
          namespace: tenant-acme
      privateKey:
        secretRef:
          name: etcd-acme-ca
          namespace: tenant-acme
    clientCertificate:
      certificate:
        secretRef:
          name: etcd-acme-client
          namespace: tenant-acme
      privateKey:
        secretRef:
          name: etcd-acme-client
          namespace: tenant-acme

Network Isolation

Tenant Network Policies

# tenant-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant-acme-isolation
  namespace: tenant-acme
spec:
  podSelector: {}
  policyTypes:

  - Ingress
  - Egress
  ingress:

  - from:
    - namespaceSelector:
        matchLabels:
          tenant: acme
  egress:

  - to:
    - namespaceSelector:
        matchLabels:
          tenant: acme

  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:

    - protocol: TCP
      port: 53

    - protocol: UDP
      port: 53

  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          app.kubernetes.io/name: kamaji

Control Plane Network Policies

# control-plane-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant-control-plane-isolation
  namespace: kmetal-kamaji
spec:
  podSelector:
    matchLabels:
      kamaji.clastix.io/name: acme-prod-cluster
  policyTypes:

  - Ingress
  - Egress
  ingress:

  - from:
    - namespaceSelector:
        matchLabels:
          tenant: acme
    ports:

    - protocol: TCP
      port: 6443

  - from:
    - namespaceSelector:
        matchLabels:
          name: kmetal-metallb
  egress:

  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:

    - protocol: TCP
      port: 2379

Resource Isolation

Pod Security Standards

# tenant-pss.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-acme
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Node Taints and Tolerations

# Taint nodes for specific tenants
kubectl taint nodes worker-1 tenant=acme:NoSchedule

# Tenant workload with toleration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: acme-app
  namespace: tenant-acme
spec:
  template:
    spec:
      tolerations:

      - key: "tenant"
        operator: "Equal"
        value: "acme"
        effect: "NoSchedule"
      nodeSelector:
        tenant-pool: "acme"

Tenant Monitoring Isolation

Tenant-Specific Monitoring

# tenant-service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tenant-acme-metrics
  namespace: tenant-acme
spec:
  selector:
    matchLabels:
      tenant: acme
  namespaceSelector:
    matchNames:

    - tenant-acme
  endpoints:

  - port: metrics
    interval: 30s

Tenant Cost Tracking

Label-Based Cost Allocation

# All tenant resources must include cost tracking labels
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: acme-prod-cluster
  namespace: tenant-acme
  labels:
    tenant: acme
    environment: production
    cost-center: "engineering"
    project: "platform"

Query Tenant Costs

# Get resource usage by tenant
kubectl top pods -n tenant-acme

# Get all resources for tenant
kubectl get all -n tenant-acme -o json | \
  jq '.items[] | select(.metadata.labels.tenant=="acme")'

# Calculate control plane costs
kubectl get tenantcontrolplanes -n tenant-acme -o json | \
  jq '.items[] | .spec.controlPlane.resources.requests'

Tenant Onboarding

Automated Tenant Provisioning

# tenant-onboarding.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: tenant-onboarding-template
data:
  tenant-template.yaml: |
    apiVersion: v1
    kind: Namespace
    metadata:
      name: tenant-{{TENANT_NAME}}
      labels:
        tenant: {{TENANT_NAME}}
    ---
    apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: tenant-{{TENANT_NAME}}-quota
      namespace: tenant-{{TENANT_NAME}}
    spec:
      hard:
        tenantcontrolplanes.kamaji.clastix.io: "{{MAX_CLUSTERS}}"
        requests.cpu: "{{MAX_CPU}}"
        requests.memory: "{{MAX_MEMORY}}"

Tenant Offboarding

# Offboard tenant safely
TENANT_NAME="acme"

# 1. Delete all tenant clusters
kubectl delete tenantcontrolplanes -n tenant-${TENANT_NAME} --all

# 2. Wait for clusters to be deleted
kubectl wait --for=delete tenantcontrolplane --all -n tenant-${TENANT_NAME} --timeout=300s

# 3. Delete tenant namespace
kubectl delete namespace tenant-${TENANT_NAME}

# 4. Clean up network policies
kubectl delete networkpolicy -l tenant=${TENANT_NAME} --all-namespaces

# 5. Remove RBAC
kubectl delete rolebinding -l tenant=${TENANT_NAME} --all-namespaces

Compliance and Auditing

Tenant Activity Logging

# audit-webhook-tenant.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: audit-webhook-config
data:
  webhook.yaml: |
    apiVersion: v1
    kind: Config
    clusters:

    - name: audit-webhook
      cluster:
        server: http://audit-service.monitoring:8080/audit/tenant/{{TENANT_NAME}}

Tenant Compliance Reports

# Generate tenant compliance report from the Cluster topology
kubectl get cluster -n tenant-acme -o json | \
  jq '{
    tenant: "acme",
    clusters: [.items[] | {
      name: .metadata.name,
      version: .spec.topology.version,
      controlPlaneEndpoint: .spec.controlPlaneEndpoint,
      variables: .spec.topology.variables
    }]
  }'