Multi-Tenancy Tasks¶

Onboarding tenants, giving them Environments, and bounding what they can consume.

For the model, see Tenant Layer. For the walkthrough of a first tenant, see Create First Tenant.

Make Capsule aware of the owner¶

Do this before the first Tenant, and again whenever tenants start coming from an identity source the platform has not seen yet.

Capsule only acts on requests from subjects it has been told about, and kMetal tells it about nobody: spec.multiTenancy on the KMetal object is opt-in throughout, with no defaults, because who owns a tenant is your identity provider's answer rather than the platform's.

apiVersion: clastix.io/v1
kind: KMetal
metadata:
  name: kmetal
spec:
  multiTenancy:
    users:
      # ServiceAccount owners are matched through the group covering their
      # namespace, never by their own name.
      - kind: Group
        name: system:serviceaccounts:kmetal-tenants
      # The provider group tenant owners come from.
      - kind: Group
        name: kmetal:tenant-owners

users is a list of candidates, not of grants. Listing a subject gives it nothing — a Tenant still has to name it in spec.owners — but it is the precondition for that ownership to mean anything.

An owner missing from users fails silently

A Tenant owner that Capsule was never told about is invisible to admission. The Tenant is accepted, the owner creates namespaces, and those namespaces land outside every tenant — no capsule.clastix.io/tenant label, no quota, no NetworkPolicy, none of the tenant's constraints. Nothing rejects the Tenant, and nothing reports the namespace as unmanaged.

The check is one command against a namespace the owner just created:

kubectl get namespace dynamo-prod -o jsonpath='{.metadata.labels}'

No capsule.clastix.io/tenant key means the owner is not in users. Add it, then delete and recreate the namespace — Capsule adopts at creation, so an already-created namespace is not picked up retroactively.

Names go in as the API server authenticates them, which is not always the name of an object:

Owner kind	What goes in `users`
`User`	The username itself.
`Group`	The group name itself.
`ServiceAccount`	The group `system:serviceaccounts:<namespace>` covering the namespace they live in — a bare ServiceAccount name matches nothing.

Nothing is added to what you set, so users is the whole set Capsule matches against. Keeping tenant owners in one or two provider groups is what stops this list growing per tenant.

Exempting the platform team¶

Two fields separate "subject to tenancy" from "operating the platform", and which one you want depends on how your provider issues groups.

administrators makes a subject an owner of every Tenant, present and future — for GitOps controllers and platform tooling that has to write inside tenants:

spec:
  multiTenancy:
    administrators:
      - kind: Group
        name: kmetal:platform-team

That is cluster-wide by construction, so it is never for someone who also owns a tenant of their own. Reaching a tenant stays explicit even so: Capsule reads a namespace operation as tenant-related only when the request carries the Capsule tenant label, so an administrator who omits it is working outside tenancy rather than across every tenant at once.

ignoreUserWithGroups takes a request out of Capsule's path entirely:

spec:
  multiTenancy:
    ignoreUserWithGroups:
      - kmetal:platform-team

This is for the provider that puts every account in one group. That group has to go in users for tenant owners to be matched at all, which also catches the platform team — naming their own group here is what separates the two. Without it a cluster administrator inherits tenant admission on every namespace they touch.

Letting a tenant promote its own ServiceAccounts¶

Off by default:

spec:
  multiTenancy:
    allowServiceAccountPromotion: true

With it on, a Tenant owner can label a ServiceAccount in one of the tenant's own Environments and make it an owner of that tenant — which is how a tenant wires up its own CI without asking you for a spec change.

It delegates entitlement downwards, and that is the whole decision: the promoted ServiceAccount is a tenant owner that nothing in the KMetal spec names. Capsule stops the chain at one step, so a promoted ServiceAccount cannot promote another.

Onboard a tenant¶

A Capsule Tenant, naming the identity that will own everything the tenant creates:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: dynamo
spec:
  owners:
    - kind: ServiceAccount
      name: system:serviceaccount:kmetal-tenants:dynamo
      clusterRoles:
        - admin
        - capsule-namespace-deleter
        - tenants-clusterrole
        - kmetal-networking-tenant

kubectl get tenant dynamo

The owner named here has to be one Capsule was told about — see Make Capsule aware of the owner. For this example that is the group system:serviceaccounts:kmetal-tenants, covering the namespace the ServiceAccount lives in.

clusterRoles is where you decide what this customer may do, and it is worth setting deliberately rather than inheriting — see Choose what the owner may do.

Choosing the owner kind¶

Kind	`name` is	Use when
`ServiceAccount`	`system:serviceaccount:<namespace>:<name>`	Automation, CI, or a first tenant on a cluster with no identity provider wired up.
`User`	The username your authenticator presents	One named human owns the tenant.
`Group`	The group name your authenticator presents	A team owns the tenant — the usual choice once an identity provider is in place, because membership changes without touching the `Tenant`.

A Tenant can carry several owners of mixed kinds.

Choose what the owner may do¶

Ownership is not a fixed set of permissions. Each owner carries its own list of ClusterRoles, and Capsule binds every one of them into every Environment the tenant owns — so what a tenant can do on the under cluster is a decision you make per customer, at onboarding:

spec:
  owners:
    - kind: ServiceAccount
      name: system:serviceaccount:kmetal-tenants:dynamo
      clusterRoles:
        - admin                      # Kubernetes' built-in, aggregated
        - capsule-namespace-deleter  # Capsule's
        - tenants-clusterrole        # yours: CAPI Clusters
        - kmetal-networking-tenant   # yours: the network claims

Omit clusterRoles and Capsule defaults the owner to admin and capsule-namespace-deleter. Set it and you replace that default rather than adding to it, which is what makes a narrower tenant possible as well as a wider one.

The list is three layers, and only the third is yours to write:

Role	Comes from	Grants, inside the tenant's Environments
`admin`, `edit`, `view`	Kubernetes	The aggregated built-ins. `admin` is the usual choice and it reaches further than applications — see what it actually carries.
`capsule-namespace-deleter`	Capsule	Deleting the tenant's own Environments. Without it a tenant can create namespaces and never remove them.
Anything else	You	The platform's own APIs. kMetal ships none of these roles — the reference set is `tenants-clusterrole` (CAPI `Cluster`s, read-only on the platform's `MachineHealthCheck`s, and self-service Sveltos `Profile`s), `kmetal-networking-tenant` (the network claims) and `kmetal-tenant-data-protector` (`ClusterBackup`s).

Nothing in kMetal requires a particular set. A tenant that only consumes clusters needs cluster.x-k8s.io/clusters and the claims they are allowed to make; everything else — raw VMs, Flux objects, direct DataVolumes — is a wider offering you choose to make.

Differentiate owners on the same tenant¶

The role list is per owner, so one Tenant can hold identities with different reach:

spec:
  owners:
    # The team, through the identity provider.
    - kind: Group
      name: dynamo-platform
      clusterRoles: [admin, capsule-namespace-deleter, tenants-clusterrole, kmetal-networking-tenant]
    # Their pipeline: creates clusters, cannot delete the Environment they live in.
    - kind: ServiceAccount
      name: system:serviceaccount:kmetal-tenants:dynamo-ci
      clusterRoles: [edit, tenants-clusterrole]

For subjects that should work inside a tenant's Environments without owning the tenant — a support rotation, a shared observability account — use additionalRoleBindings instead, which binds a ClusterRole to named subjects in every Environment without making them owners:

spec:
  additionalRoleBindings:
    - clusterRoleName: view
      subjects:
        - kind: Group
          name: platform-support
          apiGroup: rbac.authorization.k8s.io

An edit to either list re-syncs the bindings across the tenant's Environments, so widening or narrowing a tenant is a change to one object rather than a sweep.

# What the tenant's owners actually hold, per Environment
kubectl get rolebindings -n dynamo-prod \
  -o custom-columns='NAME:.metadata.name,ROLE:.roleRef.name,SUBJECT:.subjects[0].name'

proxySettings needs capsule-proxy, which kMetal does not install

owners[].proxySettings is how Capsule lets a tenant list cluster-scoped objects — ClusterClasses, StorageClasses, Nodes — through capsule-proxy. Without that component the field has no effect, and a tenant has no cluster-scoped reads at all: every value they must name in a Cluster is one you hand them.

For what each role adds up to in practice, and the three rules worth reconsidering in the reference set, see RBAC.

How a tenant authenticates¶

kMetal does not run an identity provider and does not add an authentication path of its own. Tenants authenticate to the under cluster's API server exactly as platform administrators do, so whatever that API server is already configured to trust is what a tenant uses.

In practice that is OIDC. Most under clusters are already wired to the organization's identity provider, and kMetal attaches to that mechanism rather than introducing a second one. The username and groups the provider presents in the token are the strings that go into spec.owners — User for one named human, Group for a team. No object needs to be created on the cluster for such an owner to exist — the identity is asserted by the token — but the platform still has to have declared it: the username or group goes in multiTenancy.users once, and every Tenant naming it is matched from then on.

What a tenant receives is therefore a kubeconfig pointing at the under cluster endpoint with the credential delegated to the provider — a client.authentication.k8s.io exec plugin, not an embedded token. Access is granted and revoked in the identity provider, and group membership changes without editing any Tenant.

ServiceAccount owners are for GitOps and automation: a Flux or Argo CD instance reconciling a tenant's Environment, or CI creating clusters. There is no login flow to redirect a controller through, so it gets a token. That token is a long-lived credential you have to store and rotate, and it carries no per-person identity in the audit log — which is why it belongs to a pipeline rather than to people.

A first tenant before an identity provider is wired

A ServiceAccount owner is also the way to onboard a tenant on a cluster that has no OIDC configuration yet, as Create First Tenant does. Treat it as a starting point: add the Group owner once the provider is in place, and drop the ServiceAccount unless something automated still uses it.

Give a tenant an Environment¶

An Environment is a namespace owned by the Tenant. Create it as the owner so Capsule adopts it:

kubectl create namespace dynamo-prod \
  --as=system:serviceaccount:kmetal-tenants:dynamo

kubectl get namespace dynamo-prod --show-labels     # capsule.clastix.io/tenant=dynamo

Then the Environment needs a network before a cluster can live in it — see Create First Environment.

Cap what a tenant can consume¶

Two mechanisms, and the difference matters.

spec.namespaceOptions.quota bounds how many Environments a tenant may create. spec.resourceQuotas bounds everything inside them — and with scope: Tenant it is aggregated across all of the tenant's Environments, so a tenant cannot escape a limit by spreading work over more of them.

Each example below is a fragment of the same Tenant object; they combine.

Cap Environments¶

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: dynamo
spec:
  namespaceOptions:
    quota: 3

Once reached, the owner cannot create more namespaces. Raising it is an edit to the Tenant; existing namespaces are untouched either way.

This is the cheapest cap to set and the one most worth setting early — every other limit is enforced across whatever Environments exist, so an unbounded count makes the rest harder to reason about.

Cap clusters¶

spec:
  resourceQuotas:
    scope: Tenant
    items:
      - hard:
          count/clusters.cluster.x-k8s.io: "10"

Ten clusters across every Environment the tenant owns, not ten per Environment.

A cluster is the tenant's unit of consumption: each one brings a hosted control plane on the under cluster, worker VMs on your hardware, and a subnet out of your address space. Capping clusters is the coarsest way to bound all three at once.

Cap external addresses¶

spec:
  resourceQuotas:
    scope: Tenant
    items:
      - hard:
          count/eipclaims.network.kmetal.io: "4"

Set this one deliberately on any platform where routable IPv4 is scarce.

External addresses are the resource that actually runs out. They are finite, shared, and expensive — and a tenant claiming them freely, entirely within their rights, can drain the pool every other tenant draws from. Unlike CPU or storage, you cannot buy more at short notice.

This quota only bites where tenants may claim addresses at all. If your platform grants external addresses rather than letting tenants take them, the bound belongs in the owner's role instead — see network.kmetal.io makes EIPs self-service.

A tenant at their limit is told when they create the claim, so the conversation happens at request time rather than after an address they expected never arrives.

Cap storage¶

spec:
  resourceQuotas:
    scope: Tenant
    items:
      - hard:
          tenant-storage-class.storageclass.storage.k8s.io/requests.storage: 500Gi

The key is derived from storage.tenantClassName in the KMetal spec — it changes if you change that.

It counts machine root disks too, if the platform shares that class

A tenant machine's root disk is provisioned in the tenant's own namespace, on storage.underclusterClassName. Name the same class under both fields and this quota counts those disks as well, so a two-worker cluster on a 30Gi boot tier spends 60Gi of the tenant's budget before a workload claims anything — and the number moves every time the tenant scales.

Give the platform its own StorageClass, even over the same driver and pool, and this quota goes back to meaning what it says. See Keep the platform class and the tenant class apart, and bound the machine side with the cluster and machine caps above instead.

This one is enforced inside the tenant cluster too

A PersistentVolumeClaim created inside a tenant cluster that would exceed this quota is rejected at creation by the kMetal webhook, with a message naming the quota — rather than accepted and left Pending with nothing to explain why.

See Storage.

Cap GPUs and other devices¶

spec:
  resourceQuotas:
    scope: Tenant
    items:
      - hard:
          requests.nvidia.com/A10: "2"
          limits.nvidia.com/A10: "2"

Set this deliberately, the same reasoning as external addresses: a GPU is expensive, physically finite per node, and not something a platform can provision more of on demand. Whitelisting a device only decides that it can be requested; it does not decide how many of it one tenant may hold at once, and skipping this quota leaves the first tenant to ask for one free to take all of it.

nvidia.com/A10 is not a fixed key

The key to target is requests.<resourceName>/limits.<resourceName>, where resourceName is the exact string set on the KMetal CR's permittedHostDevices or mediatedDevices entries, and the same string a tenant names as deviceName on their gpus variable.

See GPU & Device Passthrough for whitelisting devices and choosing resourceName, and Requesting a GPU for how this quota looks from the tenant's side.

Cap anything else your platform needs¶

The examples above are the ones with kMetal-specific reasoning behind them. Beyond them, spec.resourceQuotas is ordinary Kubernetes: any resource the API server counts can be bounded, aggregated the same way across the tenant's Environments.

spec:
  resourceQuotas:
    scope: Tenant
    items:
      - hard:
          limits.cpu: "64"
          limits.memory: 256Gi
      - hard:
          count/subnetclaims.network.kmetal.io: "8"
          count/vpcclaims.network.kmetal.io: "2"
          count/services.loadbalancers: "20"

count/<resource>.<group> works for any namespaced custom resource, so what a tenant can hold is a platform policy decision rather than a fixed list.

Start from what is scarce on your platform and cap that. A quota nobody hits is a quota that only produces surprise when someone finally does.

Checking consumption¶

kubectl get resourcequota -n dynamo-prod   -o custom-columns=NAME:.metadata.name,USED:.status.used,HARD:.status.hard

Read consumption from status.used. Capsule also mirrors each resource onto the ResourceQuota as quota.capsule.clastix.io/used-<resource> and hard-<resource>, but it md5-hashes the resource name whenever the key would exceed 63 characters — count/subnetclaims.network.kmetal.io lands past that limit and shows up as a hash, so those annotations are not something to grep for.

Offboard a tenant¶

Delete the Tenant:

kubectl delete tenant dynamo

That is the whole procedure. Capsule removes the Environments the Tenant owns, and everything inside them goes with the namespace: Cluster API tears down the clusters, their control planes and their machines, and the claim controllers release the VPC, the subnets and the addresses back to the platform.

Nothing has to be deleted in a particular order, and nothing needs to be cleaned up by hand afterwards — ownership is what makes it work, and it holds all the way down.

Deletion is not instant. Machines and volumes take as long as they take to tear down, so give it time before concluding something is stuck:

kubectl get namespaces -l capsule.clastix.io/tenant=dynamo
kubectl get clusters -A
kubectl get vpcclaims,subnetclaims,eipclaims -A

The last one is worth a look on a platform where addresses are scarce: it confirms the tenant's external addresses are back in the pool and available to the next tenant.

Finally, remove the owner identity if it existed only for this tenant — a ServiceAccount created for them is not owned by the Tenant and outlives it.