Secrets Management¶
Secure handling of credentials, certificates, and sensitive data in kMetal.
External Secrets Operator¶
Operator's choice
External Secrets Operator (ESO) is not part of the kMetal umbrella chart. If your environment needs to source secrets from AWS Secrets Manager, HashiCorp Vault, or a similar backend, install ESO separately via its upstream Helm chart.
Configure Secret Store¶
AWS Secrets Manager
# aws-secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secrets
namespace: kmetal-flux
spec:
provider:
aws:
service: SecretsManager
region: us-west-2
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa
HashiCorp Vault
# vault-secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-secrets
namespace: kmetal-flux
spec:
provider:
vault:
server: "https://vault.company.com"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "kmetal-platform"
serviceAccountRef:
name: external-secrets-sa
Infrastructure Credentials¶
vSphere Credentials¶
# vsphere-credentials-external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: vsphere-credentials
namespace: kmetal-flux
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-secrets
kind: SecretStore
target:
name: vsphere-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: infrastructure/vsphere
property: username
- secretKey: password
remoteRef:
key: infrastructure/vsphere
property: password
AWS Credentials¶
# aws-credentials-external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: aws-credentials
namespace: kmetal-flux
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets
kind: SecretStore
target:
name: capa-manager-bootstrap-credentials
creationPolicy: Owner
dataFrom:
- extract:
key: kmetal/aws-credentials
Certificate Management¶
Self-Signed CA¶
# Generate self-signed CA
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
# Create CA secret
kubectl create secret tls ca-key-pair \
--cert=ca.pem \
--key=ca-key.pem \
-n kmetal-cert-manager
ClusterIssuer Configuration¶
# cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: ca-issuer
spec:
ca:
secretName: ca-key-pair
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@company.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: haproxy
Certificate Rotation¶
# Check certificate expiry
kubectl get certificates -A
# Force certificate renewal
cmctl renew <certificate-name> -n <namespace>
# Verify renewed certificate
kubectl get certificate <name> -n <namespace> -o jsonpath='{.status.notAfter}'
Tenant Cluster Secrets¶
Tenant Kubeconfig Management¶
# tenant-kubeconfig-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: tenant-cluster-kubeconfig
namespace: tenant-acme
type: Opaque
data:
kubeconfig: <base64-encoded-kubeconfig>
Sync Secrets to Tenant Clusters¶
# secret-sync.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: secret-sync
namespace: tenant-acme
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: secret-reader
namespace: kmetal-flux
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch"]
resourceNames: ["shared-registry-creds"]
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: sync-secrets
namespace: tenant-acme
spec:
schedule: "*/15 * * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: secret-sync
containers:
- name: kubectl
image: bitnami/kubectl:latest
command:
- /bin/bash
- -c
- |
kubectl get secret shared-registry-creds \
-n kmetal-flux -o yaml | \
sed 's/namespace: kmetal-flux/namespace: default/' | \
kubectl apply -f - \
--kubeconfig=/secrets/kubeconfig \
--context=tenant-cluster
volumeMounts:
- name: kubeconfig
mountPath: /secrets
volumes:
- name: kubeconfig
secret:
secretName: tenant-cluster-kubeconfig
restartPolicy: OnFailure
Sealed Secrets¶
Operator's choice
Sealed Secrets is not part of the kMetal umbrella chart. If your environment needs encrypted-at-rest secrets in git, install the upstream Sealed Secrets controller separately.
Create Sealed Secrets¶
# Install kubeseal CLI
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-linux-amd64
chmod +x kubeseal-linux-amd64
sudo mv kubeseal-linux-amd64 /usr/local/bin/kubeseal
# Create sealed secret
kubectl create secret generic db-credentials \
--from-literal=username=admin \
--from-literal=password=secret123 \
--dry-run=client -o yaml | \
kubeseal -o yaml > sealed-db-credentials.yaml
# Apply sealed secret
kubectl apply -f sealed-db-credentials.yaml
Secret Encryption at Rest¶
Enable Encryption Provider¶
# encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}
Apply to the tenant control plane¶
t.b.d.
Secret encryption-at-rest requires apiServer extra-args (--encryption-provider-config=...) plus additional volumes/mounts to surface the encryption config inside the api-server pod. The current kubevirt-kubeadm ClusterClass does not expose these as topology variables; wiring up encryption-at-rest is therefore t.b.d. until the ClusterClass is extended with encryption variables (or until a direct-patch escape hatch is documented).
Secret Scanning¶
Prevent Secret Commits¶
# Install git-secrets
git clone https://github.com/awslabs/git-secrets
cd git-secrets
sudo make install
# Configure git-secrets
cd /path/to/repo
git secrets --install
git secrets --register-aws
git secrets --add 'password.*=.*'
git secrets --add 'api[_-]?key.*=.*'
Scan Kubernetes Secrets¶
# Check for potential secret leaks
kubectl get secrets --all-namespaces -o json | \
jq -r '.items[] |
select(.metadata.name | test("token|password|key|secret")) |
"\(.metadata.namespace)/\(.metadata.name)"'
# Audit secret access
kubectl get events --all-namespaces | grep Secret
Monitoring Secret Access¶
Audit Secret Access¶
# audit-policy-secrets.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
verbs: ["get", "list", "watch"]
resources:
- group: ""
resources: ["secrets"]
- level: Metadata
resources:
- group: ""
resources: ["secrets"]
verbs: ["create", "update", "patch", "delete"]
Alert on Secret Access¶
# secret-access-alert.yaml
groups:
- name: secret-access
rules:
- alert: UnauthorizedSecretAccess
expr: |
rate(apiserver_audit_event_total{
verb=~"get|list",
objectRef_resource="secrets",
responseStatus_code!="200"
}[5m]) > 0
labels:
severity: warning
annotations:
summary: "Unauthorized secret access attempt"