Networking Configuration¶
This page covers networking configuration for the under cluster. For tenant-facing service exposure see Load Balancer; for the conceptual model (VPCs/Subnets) see Networking Concepts.
Under-cluster CNI¶
kMetal runs a dual-CNI setup on the under cluster: Flannel is the primary CNI (handles management-plane pod networking) and Kube-OVN is delivered as a secondary CNI via Multus (handles tenant network isolation, VPCs, subnets, and the provider network for tenant egress).
Flannel¶
network:
flannel:
enabled: true
podCidr: "10.93.0.0/16" # CIDR for under-cluster management-plane pods
backend: "host-gw"
Kube-OVN¶
network:
kubeOvn:
enabled: true
podCidr: "10.16.0.0/16" # Kube-OVN default pod subnet
podGateway: "10.16.0.1"
svcCidr: "10.96.0.0/16" # Must match kube-apiserver --service-cluster-ip-range
joinCidr: "100.64.0.0/16"
tunnelInterface: "data.205" # Physical NIC carrying inter-node Geneve traffic (REQUIRED, no default)
tunnelType: "geneve" # geneve | vxlan | stt
tunnelInterface is required
The chart has no sensible default for tunnelInterface — it must be set to the NIC name on each under-cluster node that carries inter-node tenant traffic.
Multus¶
MetalLB¶
MetalLB allocates LoadBalancer service IPs on the under cluster — specifically for tenant control-plane VIPs and any platform-facing services. Tenant workload LoadBalancer services do not come from MetalLB; they flow through Kube-OVN (see Load Balancer Configuration).
# IPAddressPool + L2Advertisement applied to the under cluster after install.
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: tenant-cp-pool
namespace: kmetal-metallb
spec:
addresses:
- 172.21.204.200-172.21.204.250 # adjust to a reserved range on your management VLAN
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: tenant-cp-l2
namespace: kmetal-metallb
spec:
ipAddressPools:
- tenant-cp-pool
Do not expose MetalLB on the provider VLAN
Define MetalLB pools only on the management VLAN. Putting a MetalLB pool on the provider VLAN gives anyone with a Service type: LoadBalancer direct access to the SNAT-egress and inter-tenant traffic plane that the platform reserves for OVN. Tenant CP VIPs (and platform services) live on the management pool; tenant workload IPs live on Kube-OVN EIPs.
For BGP advertisement instead of L2:
t.b.d. — A BGP example with the actual chart values shape is t.b.d. in this section. Refer to the upstream MetalLB BGP documentation for the underlying BGPPeer and BGPAdvertisement shapes; the chart passes them through.
Provider network (tenant egress / external IPs)¶
The provider network is a dedicated VLAN that carries tenant SNAT egress and hosts the External IPs (EIPs) clients connect to. It is configured by three Kube-OVN resources plus a small patch on the default VPC, applied to the under cluster after the chart is installed.
Step 1 — ProviderNetwork and Vlan¶
apiVersion: kubeovn.io/v1
kind: ProviderNetwork
metadata:
name: provider
spec:
defaultInterface: data.222 # NIC on every worker that carries provider-VLAN traffic
excludeNodes:
- c1 # CP nodes without a provider NIC
---
apiVersion: kubeovn.io/v1
kind: Vlan
metadata:
name: vlan222
spec:
id: 0 # MUST be 0 when defaultInterface is a kernel VLAN sub-interface (e.g. data.222)
provider: provider
Vlan id MUST be 0 with a kernel sub-interface
If ProviderNetwork.spec.defaultInterface is a kernel VLAN sub-interface like data.222, the host already tags. Setting Vlan.spec.id non-zero double-tags packets at the OVN localnet port and traffic is dropped silently. Use id: 0. If the NIC is a plain physical interface (no .NNN suffix), set id to the actual VLAN.
Step 2 — External Subnet on the default VPC¶
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
name: external # name "external" is REQUIRED by enableExternal reconcile
spec:
vpc: ovn-cluster # default VPC
vlan: vlan222
protocol: IPv4
cidrBlock: 77.220.68.96/27 # the publicly-routable / 1:1-NAT'd block
gateway: 77.220.68.97
gatewayType: distributed # every node SNATs/DNATs locally — no single chokepoint
excludeIps:
- 77.220.68.96 # network address
- 77.220.68.97 # gateway
- 77.220.68.127 # broadcast
# plus any IPs reserved for edge-NAT slots so OVN does not auto-allocate them
Custom tenant VPCs reference this subnet through enableExternal: true; Kube-OVN's controller looks for a subnet literally named external on the default VPC (ovn-cluster) before reconciling.
Step 3 — Gateway node labels¶
enableExternal on a tenant VPC creates a Logical Router Port (LRP) that needs to be hosted somewhere. Label the under-cluster worker nodes that should anchor those LRPs:
kubectl label node w1 ovn.kubernetes.io/external-gw=true
kubectl label node w2 ovn.kubernetes.io/external-gw=true
Apply this before creating any tenant VPC with enableExternal: true. Missing labels means tenant SNAT silently breaks.
Step 4 — Silence the default-VPC label stripper¶
Kube-OVN's kube-ovn-controller runs resyncExternalGateway() every second when --enable-eip-snat=true (the chart default). On controller restart it clears the ovn.kubernetes.io/external-gw=true label on every node, which breaks tenant VPC reconciliation.
The supported escape hatch is to set extraExternalSubnets on the default VPC. The reconcile loop then returns early without touching node labels, and no LRP is actually created on ovn-cluster because enable-eip-snat=true already short-circuits the default VPC:
apiVersion: kubeovn.io/v1
kind: Vpc
metadata:
name: ovn-cluster # the default VPC
spec:
extraExternalSubnets:
- external
Apply with server-side apply so only spec.extraExternalSubnets is owned by the operator — the rest of the auto-managed Vpc fields stay untouched.
Verifying¶
kubectl get providernetwork provider -o yaml | head -30
kubectl get vlan vlan222
kubectl get subnet external -o wide # expect gatewayType=distributed, vlan=vlan222
kubectl get nodes -l ovn.kubernetes.io/external-gw=true
kubectl get vpc ovn-cluster -o jsonpath='{.spec.extraExternalSubnets}'
Per-tenant network provisioning¶
Each tenant's networking is three Kube-OVN resources plus one Multus NetworkAttachmentDefinition per subnet — created in the tenant's under-cluster namespace.
VPC (one per tenant)¶
apiVersion: kubeovn.io/v1
kind: Vpc
metadata:
name: alpha-vpc
spec:
namespaces:
- alpha # under-cluster namespace the tenant owns
enableExternal: true # plumb the external LRP for egress and EIPs
staticRoutes:
- cidr: 0.0.0.0/0
nextHopIP: 77.220.68.97 # the external subnet's gateway
policy: policyDst
# Hard multi-tenancy: drop egress from this VPC to the platform external CIDR.
# Stops a compromised tenant from pivoting into other tenants' EIPs or the
# under-cluster apiserver via the provider VLAN.
policyRoutes:
- priority: 1500
match: "ip4.dst == 77.220.68.96/27"
action: drop
Subnet + NAD per tenant cluster¶
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: alpha-dev-net
namespace: alpha
spec:
config: |
{
"cniVersion": "0.3.1",
"name": "kube-ovn",
"type": "kube-ovn",
"server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
"provider": "alpha-dev-net.alpha.ovn"
}
---
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
name: alpha-dev-subnet # Subnet is cluster-scoped — <tenant>-<cluster> prefix is mandatory
spec:
vpc: alpha-vpc
cidrBlock: 10.100.0.0/24
gateway: 10.100.0.1
protocol: IPv4
provider: alpha-dev-net.alpha.ovn # MUST match the NAD's provider key
namespaces:
- alpha
excludeIps:
- 10.100.0.1
The CAPI Cluster for that tenant references the Subnet by name through the network.subnet topology variable on kubevirt-kubeadm.
SNAT rule (egress from tenant pods)¶
apiVersion: kubeovn.io/v1
kind: OvnSnatRule
metadata:
name: alpha-snat-dev
spec:
ovnEip: alpha-vpc-external # auto-created LRP EIP, name = <vpc>-external
vpcSubnet: alpha-dev-subnet # subnet whose pod traffic gets SNAT'd
Tenant pods on 10.100.0.0/24 egress as alpha-vpc-external (a 77.220.68.x address on the provider VLAN). Inbound LoadBalancer traffic uses separate OvnEip + OVN load balancer resources created by cloud-provider-ovn (see Load Balancer).
Verifying¶
# All tenant VPCs and their subnets
kubectl get vpc
kubectl get subnet -o wide
# A tenant's egress EIP and SNAT
kubectl get ovn-eips | grep alpha
kubectl get ovn-snat-rules | grep alpha
# NetworkAttachmentDefinitions per tenant
kubectl get net-attach-def -A
DNS¶
The under cluster's DNS is the standard CoreDNS deployed by kubeadm (or your Kubernetes installer). Tenant clusters get their own CoreDNS via their TenantControlPlane.
If tenant LoadBalancer services need external DNS records, configure ExternalDNS (or your DNS provider's automation) against your DNS zone separately — this is operator's choice and outside the chart's scope.
Troubleshooting¶
# Check Multus is healthy
kubectl get pods -n kube-system -l app=multus
# Check Kube-OVN
kubectl get pods -n kube-system -l app=ovs
kubectl get pods -n kube-system -l app=kube-ovn-controller
kubectl get pods -n kube-system -l app=kube-ovn-cni
kubectl get subnets
# Check Flannel
kubectl get pods -n kube-flannel -l app=flannel
# Check MetalLB
kubectl get pods -n kmetal-metallb
kubectl get ipaddresspool -n kmetal-metallb
kubectl get l2advertisement -n kmetal-metallb
# Verify a LoadBalancer service gets an IP
kubectl create service loadbalancer test-lb --tcp=80:80
kubectl get svc test-lb -w
kubectl delete svc test-lb
# Check MetalLB controller / speaker logs
kubectl logs -n kmetal-metallb -l app.kubernetes.io/name=metallb,app.kubernetes.io/component=controller --tail=100
kubectl logs -n kmetal-metallb -l app.kubernetes.io/name=metallb,app.kubernetes.io/component=speaker --tail=100
See Troubleshooting: Networking for additional diagnostic flows.