Secrets Management¶

Where credentials live, and how one reaches a tenant cluster.

kMetal is agnostic here: it installs no secret manager, and every credential it consumes is an ordinary Kubernetes Secret it reads by name. That leaves you two decisions — where the truth for a secret lives, and how a secret gets from the under cluster into a tenant cluster.

What the platform consumes¶

kMetal never holds a credential in its own API. The KMetal object is cluster-scoped and readable by anyone who can read it, so it carries Secret names only:

Secret	Named by	Shape	Lives in
Image pull credentials	`spec.registry.imagePullSecretName`	`kubernetes.io/dockerconfigjson`, used by kubelet	The operator's namespace
Chart pull credentials	`spec.registry.chartPullSecretName`	`username` / `password` keys, used by Flux's source-controller	The operator's namespace
Backup destination and encryption key	`ClusterBackup.spec.backend.s3.credentialsSecretRef` and `spec.encryption.secretKeyRef`	`accessKeyId` / `secretAccessKey`, and a 32-byte `key`	The tenant's Environment — see Data Protection
Datastore TLS	`DataStore.spec.tlsConfig`	cert-manager `Certificate` output	The etcd release's namespace
Tenant cluster kubeconfigs	Generated by Kamaji	`admin.conf`	The cluster's own namespace

The last row is the one to leave alone. Kamaji issues those, cert-manager renews the certificates behind them, and the Secret is regenerated — so a kubeconfig is something to hand out on demand rather than copy somewhere durable, as Hand a tenant a working kubeconfig describes.

Bring your own secret manager¶

Three shapes, none of them installed by kMetal, and the choice is about what a consumer can read:

Approach	What it produces	Fits when
Kubernetes Secrets, managed as such	The `Secret` itself is the source of truth	A small platform with encryption at rest enabled and a disciplined git tree. Everything on this page works, and rotation is manual.
External Secrets Operator	A real `Secret`, reconciled from an external store on `refreshInterval`	Almost always the right answer here: the store stays authoritative, and consumers still see an ordinary `Secret`.
Vault Agent Injector, or the Secrets Store CSI driver	A file inside a pod — no `Secret` object	Application workloads that read credentials from disk.

That third row carries a constraint worth knowing before choosing it. Most of what this platform does with secrets needs a Secret object, not a mounted file: imagePullSecrets, Flux HelmRelease valuesFrom, spec.registry, a ClusterBackup's credentials, and Sveltos policyRefs all resolve a Secret through the API. So a Vault-backed platform still wants ESO — or Vault's own sync — to materialise Secrets for the platform, with Agent Injector reserved for workloads.

An ESO ClusterSecretStore plus one ExternalSecret per credential is the whole pattern:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: registry-pull
  namespace: kmetal-operator-system      # where spec.registry expects it
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: platform-vault
    kind: ClusterSecretStore
  target:
    name: registry-pull                  # the name spec.registry.imagePullSecretName carries
    template:
      type: kubernetes.io/dockerconfigjson
  dataFrom:
    - extract:
        key: platform/registry

The target.name is the contract. Anything on this platform that names a Secret is naming that, so switching backends later is a change to the ExternalSecret and to nothing else.

Encrypt Secrets at rest on the under cluster¶

This is the under cluster's own api-server configuration, unchanged by kMetal, and it is worth doing before the first tenant: every tenant's etcd credentials, backup keys and cluster kubeconfigs are Secrets on this cluster.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded 32-byte key>
      - identity: {}

Pair it with the audit policy from RBAC, which is where reads of those Secrets show up.

Deliver a secret into tenant clusters¶

A tenant cluster is a separate API server, so a Secret on the under cluster is not visible there. The delivery mechanism is Sveltos, the same one that carries add-ons — see Fleet Management.

Sveltos policyRefs accept a Secret as well as a ConfigMap, with one requirement: the Secret must be of type addons.projectsveltos.io/cluster-profile, and its data keys hold the manifests to apply. So the object you deliver is a Secret wrapping the Secret:

apiVersion: v1
kind: Secret
metadata:
  name: tenant-registry-creds
  namespace: kmetal-system
type: addons.projectsveltos.io/cluster-profile     # required, or the ref is rejected
stringData:
  registry-secret.yaml: |
    apiVersion: v1
    kind: Secret
    metadata:
      name: registry-pull
      namespace: default
    type: kubernetes.io/dockerconfigjson
    data:
      .dockerconfigjson: <base64>
---
apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: tenant-registry-creds
spec:
  clusterSelector:
    matchExpressions:
      - key: cluster.x-k8s.io/cluster-name
        operator: Exists
  syncMode: ContinuousWithDriftDetection
  reloader: true                                   # roll workloads mounting it when it changes
  policyRefs:
    - kind: Secret
      name: tenant-registry-creds
      namespace: kmetal-system

Which gives a rotation path with no bespoke machinery:

rotate the credential in your store,
ESO refreshes the Secret on the under cluster,
update the delivered payload from it — a templating step, or an ExternalSecret that writes the cluster-profile Secret directly,
Sveltos re-applies into every matching cluster, and reloader: true restarts the Deployments, StatefulSets and DaemonSets that mount it.

ContinuousWithDriftDetection also means a tenant who deletes the delivered Secret gets it back, which is what makes this usable for something a cluster must have.

Everything delivered is readable by the tenant

A tenant holds cluster-admin of their own cluster, so a Secret you push into it is a Secret you have given them.

Deliver only credentials the tenant is entitled to — registry pull credentials, a shared observability token scoped to their cluster. Platform credentials stay on the under cluster, where the tenant has no reads: the backup S3 credential is the one to think twice about, since it sits in their Environment by design.

Tenants can run this themselves for their own secrets if you grant them Sveltos Profiles in their Environment — see What a tenant can do on the under cluster.

Keep secrets out of the git tree¶

The platform is reconciled from git by Flux, so the tree is a distribution channel. Nothing in kMetal requires a secret to be committed — the KMetal object references names, and both pull-secret fields point at Secrets created out of band.

Where a credential does have to travel through git, encrypt it there: SOPS, which Flux decrypts natively, or Sealed Secrets if you would rather the cluster hold the key. Either is preferable to the pattern this replaces, which is a Secret applied by hand and undocumented until someone needs to rotate it.

Verify¶

# What the platform is pointed at
kubectl get km <name> -o jsonpath='{.spec.registry}'

# Whether those Secrets exist where the operator looks
kubectl get secrets -n kmetal-operator-system

# ESO, if you use it
kubectl get clustersecretstores,externalsecrets -A
kubectl get externalsecret <name> -n <ns> -o jsonpath='{.status.conditions}'

# What Sveltos delivered, and to which clusters
kubectl get clustersummaries -A
kubectl get clustersummary <profile>-capi-<cluster> -n <environment> \
  -o jsonpath='{.status.featureSummaries[*].status}'

A delivered Secret that never appears in the tenant cluster is usually the type: a policyRefs entry naming a Secret that is not addons.projectsveltos.io/cluster-profile is refused, and the ClusterSummary says so.