Running Cluster API in Production

Running Cluster API (CAPI) in production introduces a fundamental architectural shift: the management cluster becomes a mission-critical infrastructure control plane.

In single-cluster environments, an operational failure in the control plane affects only the workloads running within that boundary. With Cluster API, a failure in the management cluster decouples your ability to mutate, heal, scale, or secure the entire fleet of downstream workload clusters.

While running workload clusters continue executing application pods when the management plane is offline, all lifecycle operations freeze:

• Automated node remediation via MachineHealthCheck halts, leaving failed physical or virtual hosts unrecovered.
• Horizontal cluster autoscaling via cluster-autoscaler cannot allocate new compute instances during traffic spikes.
• Rolling security upgrades across worker pools and control planes stall midway.
• Ephemeral test or staging clusters cannot be provisioned or decommissioned, accumulating cloud costs.

Operating Cluster API in production is therefore not simply a matter of managing custom resource definitions (CRDs). It requires applying Tier-0 reliability engineering to the management cluster itself.

1. The Production Mental Model: The Fleet Control Plane

The management cluster functions as an infrastructure kernel. It converts declarative Kubernetes resource models (Cluster, Machine, MachineDeployment, KubeadmControlPlane) into cloud-provider API calls (AWS EC2 instances, Azure Virtual Machines, Google Cloud compute instances, load balancers, and security groups).

flowchart TD
    subgraph ManagementPlane["Management Cluster (Tier-0 Control Plane)"]
        KubeAPI["Kubernetes API Server (etcd HA)"]
        CertMgr["cert-manager (Webhook TLS)"]
        CAPICore["CAPI Core Controller"]
        BootstrapProvider["Bootstrap Provider (CABPK)"]
        ControlPlaneProvider["Control Plane Provider (KCP)"]
        InfraProvider["Infrastructure Provider (CAPA / CAPZ / CAPG)"]
        SecretsEngine["Cluster Secrets & Credentials"]

        KubeAPI --> CAPICore
        KubeAPI --> BootstrapProvider
        KubeAPI --> ControlPlaneProvider
        KubeAPI --> InfraProvider
        CertMgr -.->|"Injects Webhook Certs"| CAPICore
        CertMgr -.->|"Injects Webhook Certs"| InfraProvider
    end

    subgraph WorkloadA["Production Workload Cluster A"]
        CP_A["Control Plane Nodes (HA)"]
        Workers_A["Worker Pool (MachineDeployment)"]
        CP_A --- Workers_A
    end

    subgraph WorkloadB["Production Workload Cluster B"]
        CP_B["Control Plane Nodes (HA)"]
        Workers_B["Worker Pool (MachineDeployment)"]
        CP_B --- Workers_B
    end

    InfraProvider -->|"Reconciles Cloud APIs"| WorkloadA
    InfraProvider -->|"Reconciles Cloud APIs"| WorkloadB
    ControlPlaneProvider -->|"Kubeconfig Mutual TLS"| CP_A
    ControlPlaneProvider -->|"Kubeconfig Mutual TLS"| CP_B

The Separation of Concerns

1. Workload Clusters: Run stateless and stateful customer-facing applications. They require standard Kubernetes Day-2 operations (ingress, storage classes, service mesh, pod autoscaling).
2. Management Cluster: Runs only infrastructure automation controllers, admission webhooks, and the metadata state store. It must never host general application workloads. Mixing business applications with infrastructure controllers risks CPU/memory contention, noisy-neighbor evictions, and shared privilege escalation vectors.

2. Architectural Comparisons

Management Cluster vs. Workload Cluster Profiles

Operating the management plane requires different service level objectives (SLOs), capacity planning, and scaling policies than application-bearing workload clusters.

Disaster Recovery Strategies for Cluster API

If the management cluster loses its underlying etcd storage or suffers irreversible corruption, your disaster recovery (DR) strategy determines whether you recover in minutes or spend days reconciling orphan cloud infrastructure.

3. The Five Pillars of Production Cluster API

To transition Cluster API from a lab or proof-of-concept into a reliable enterprise platform, engineering teams must implement five operational pillars.

flowchart LR
    P1["1. High Availability & Sizing"]
    P2["2. Security & Credential Isolation"]
    P3["3. Provider Lifecycle Management"]
    P4["4. Infrastructure Quota & Capacity"]
    P5["5. Observability & Telemetry"]

    P1 --> Platform["Production CAPI Engine"]
    P2 --> Platform
    P3 --> Platform
    P4 --> Platform
    P5 --> Platform

Pillar 1: High Availability and Control Plane Sizing

A development management cluster running on KinD inside a single virtual machine is acceptable only during initial bootstrapping. Production management clusters must run on multi-node, high-availability Kubernetes deployments spanning at least three availability zones.

etcd Sizing and Disk I/O

Cluster API controllers continuously reconcile status subresources. Every transition (e.g., node bootstrapping progress, IP allocation, certificate renewal, health check pings) writes updates to etcd.

• Dedicated Disks: Host etcd on storage providing sustained high write IOPS with single-digit millisecond fsync latency (e.g., AWS io2 or high-performance NVMe).
• Compaction & Defragmentation: Configure aggressive etcd history compaction (--auto-compaction-retention=1h or 10000 revisions) and periodic defragmentation jobs to prevent storage exhaustion from rapid CRD status churn.

Controller Replica Count and Leader Election

Production CAPI controllers (CAPI core, KCP, CABPK, CAPx) run as Deployments with leader election enabled by default:

• Set replicas: 2 or replicas: 3 across different worker nodes to ensure warm standbys are ready if the active controller pod is evicted.
• Verify leader election lease settings to avoid rapid failover thrashing during temporary network partitions:
  • lease-duration: 15s
  • renew-deadline: 10s
  • retry-period: 2s

Pillar 2: Security Architecture and Credential Isolation

Cluster API providers hold high-privilege credentials capable of orchestrating networking, computing, and security primitives across cloud accounts.

flowchart TD
    subgraph ManagementNamespace["Management Namespace: tenant-prod"]
        ClusterCR["Cluster: prod-us-east-1"]
        AWSClusterCR["AWSCluster: prod-us-east-1"]
        KubeconfigSecret["Secret: prod-us-east-1-kubeconfig (Admin Access)"]
        CloudCredSecret["Secret: capz-manager-credentials (IAM Scoped)"]
    end

    subgraph RBACBoundary["Access Control Policies"]
        PlatformEngineers["Platform Admin Role"]
        DevEngineers["Tenant Dev Role"]
    end

    PlatformEngineers -->|"Full Access"| ManagementNamespace
    DevEngineers -.->|"Blocked by RBAC"| KubeconfigSecret
    DevEngineers -.->|"Blocked by RBAC"| CloudCredSecret
    DevEngineers -->|"Read-Only Status"| ClusterCR

Least-Privilege Cloud IAM

Do not run infrastructure providers with administrative cloud credentials (e.g., AWS AdministratorAccess or Azure Subscription Owner).

1. Scope IAM Permissions: Limit provider credentials to specific VPCs, subnets, and tagged resources.
2. Workload Identity: Avoid static API keys or long-lived service account secrets. Use cloud-native identity federation:
   • AWS: IAM Roles for Service Accounts (IRSA) or EKS Pod Identity.
   • Azure: Workload Identity.
   • GCP: Workload Identity Federation.
3. Multi-Tenancy Boundaries: Separate tenant clusters into distinct management namespaces. Configure provider credentials per namespace rather than sharing a global cluster-wide credential.

Workload Kubeconfig Protection

During cluster creation, Cluster API generates an administrative kubeconfig secret (<cluster-name>-kubeconfig) containing client certificates with system:masters cluster-admin rights.

• Restrict read access to these secrets using strict Kubernetes RBAC.
• Implement an automated secret management pattern: extract the generated CA, issue short-lived, role-bound OIDC tokens for engineering teams, and avoid distributing the root cluster-admin kubeconfig.

Pillar 3: Provider Lifecycle and Webhook Dependencies

Cluster API is a composite architecture of multiple independent controllers, each maintaining its own release cycle and API version compatibility matrix.

The Webhook Certificate Trap

Every CAPI provider relies on validating and mutating admission webhooks to enforce immutability rules, default field values, and validate referenced templates. These webhooks terminate TLS using certificates injected by cert-manager.

• If cert-manager fails, or if webhook CA certificates expire:
  • Every kubectl apply or controller mutation targeting CAPI resources is immediately rejected by the Kubernetes API server (Internal error occurred: failed calling webhook...).
  • Controllers cannot reconcile or remediate existing resources.
• Rule: Monitor cert-manager certificate validity (cert-manager.io/v1/Certificate) and configure automated alerts when certificate expirations approach 14 days.

Staged Provider Upgrades

Never upgrade management plane providers directly in production without running the upgrade against a staging management cluster. Use clusterctl upgrade plan to inspect the compatibility matrix before executing upgrades:

# 1. Check available provider upgrades and compatibility
clusterctl upgrade plan

# 2. Execute dry-run or targeted component upgrade
clusterctl upgrade apply \
  --core capi-system:v1.7.2 \
  --bootstrap capi-kubeadm-bootstrap-system:v1.7.2 \
  --control-plane capi-kubeadm-control-plane-system:v1.7.2 \
  --infrastructure capa-system:v2.5.1

Always execute provider upgrades in this exact order:

1. cert-manager (if managed separately).
2. Core Cluster API (capi-system).
3. Bootstrap Provider (cabpk-system).
4. Control Plane Provider (kcp-system).
5. Infrastructure Provider (capa-system, capz-system, etc.).
6. Add-on and IPAM providers.

Pillar 4: Infrastructure Quota, IPAM, and Surge Capacity

Declarative infrastructure can request resources faster than underlying cloud limits allow. A single misconfigured manifest can trigger infrastructure failure across multiple workload clusters.

The Upgrade Surge Trap

During a rolling upgrade of a MachineDeployment, CAPI provisions new machines before deleting outdated ones (governed by maxSurge):

$$\text{Total Required Nodes} = \text{Replicas} + \text{maxSurge}$$

If a 50-node cluster is configured with maxSurge: 20% (10 nodes), the cloud account must have:

• 10 spare compute instance quota allocations.
• 10 available private IP addresses in the target subnet.
• Sufficient Elastic IP or NAT gateway session capacity.

If IP addresses or cloud quotas are exhausted mid-rollout, the new machines enter an unrecoverable Failed state, the rollout halts, and the cluster remains in a degraded mixed-version state.

Cloud API Rate Limiting

CAPI controllers poll cloud APIs to detect infrastructure state changes. Across 100 clusters, hundreds of worker nodes generate thousands of Describe/List API calls per minute.

• Configure exponential backoff and rate-limiting flags on provider controllers (e.g., --aws-client-rate-limit, --max-concurrent-reconciles).
• Enable provider caching layers to serve read requests from in-memory informers rather than querying cloud provider endpoints directly.

Pillar 5: Observability and Controller Telemetry

Monitoring pod status (kubectl get pods -n capi-system) is insufficient. A controller pod can report Running while internal worker queues are stalled, failing, or starved of API rate limits.

Controller-Runtime Metrics

CAPI components expose Prometheus metrics over secure HTTPS diagnostic endpoints. Monitor these core metrics across all provider deployments:

# Controller workqueue depth: sustained increase indicates backlog or deadlocks
workqueue_depth{name="machine-controller"}

# Reconcile latency: spikes indicate slow external cloud API calls
controller_runtime_reconcile_time_seconds_bucket{controller="machine"}

# Reconciliation errors: spikes indicate authentication, quota, or network failures
controller_runtime_reconcile_total{controller="machine", result="error"}

# Cloud API client execution latency
capi_cloud_request_duration_seconds{provider="aws", operation="CreateInstance"}

Condition-Based Alerting

Rather than alerting on transient errors, trigger production alerts based on persistent status conditions. Resources should transition to Ready within bounded operational windows:

4. Disaster Recovery and Backup Architecture

To protect against catastrophic management cluster failure, implement automated etcd and custom resource backups using tools like Velero.

sequenceDiagram
    autonumber
    actor Admin as Platform Operator
    participant Backup as Velero Backup Engine
    participant S3 as Object Storage (S3 / GCS)
    participant NewKube as New Management Cluster
    participant Cloud as Cloud Infrastructure (Existing VMs)

    Note over Admin,S3: Scheduled Daily Backup
    Backup->>Backup: Pause CRDs (spec.paused = true)
    Backup->>S3: Export CAPI CRDs, Secrets, and ConfigMaps
    Backup->>Backup: Unpause CRDs

    Note over Admin,Cloud: Disaster Recovery Execution
    Admin->>NewKube: Initialize Fresh Management Cluster
    Admin->>NewKube: Install CAPI Core and Providers (Matching Versions)
    Admin->>NewKube: Install cert-manager
    Admin->>Backup: velero restore create --from-backup capi-backup
    Backup->>S3: Fetch Manifests and Secrets
    Backup->>NewKube: Apply CRDs, Provider Resources, Secrets
    NewKube->>Cloud: Reconcile Existing VMs via ProviderID (No Duplication)
    Note over NewKube,Cloud: Reconciliation Resumes with Zero Workload Downtime

Velero Backup Configuration for Cluster API

Configure Velero to back up all namespaces containing CAPI resources, including core namespaces and cluster secret stores:

# Create a dedicated backup of CAPI control plane resources
velero backup create capi-production-backup \
  --include-resources \
clusters.cluster.x-k8s.io,\
machinedeployments.cluster.x-k8s.io,\
machinesets.cluster.x-k8s.io,\
machines.cluster.x-k8s.io,\
kubeadmcontrolplanes.controlplane.cluster.x-k8s.io,\
kubeadmconfigs.bootstrap.cluster.x-k8s.io,\
secrets,\
configmaps \
  --include-namespaces default,capi-system,capa-system \
  --snapshot-volumes=false

[!IMPORTANT] When restoring to a new management cluster, install the exact same versions of CAPI providers and cert-manager before executing the restore. If CRD schemas differ or webhook endpoints are missing, the Kubernetes API server will reject the incoming backup objects.

5. Diagnostic Decision Tree & Failure Playbooks

When a workload cluster or machine stops progressing, use this structured decision tree to pinpoint the failure layer.

flowchart TD
    Start["Investigation: Cluster or Machine Stalled"] --> CheckWebhook{"Are admission webhooks healthy?"}
    
    CheckWebhook -->|"No"| FixCert["Check cert-manager pods and certificate expiration"]
    CheckWebhook -->|"Yes"| CheckCRDStatus{"Check kubectl describe on stalled resource"}

    CheckCRDStatus --> HasConditions{"Are Conditions reporting Failure?"}
    
    HasConditions -->|"Yes"| InspectCondition["Inspect FailureReason and FailureMessage"]
    InspectCondition --> CloudError{"Is error from Cloud Provider?"}
    CloudError -->|"Yes: Quota / IAM / Net"| FixCloud["Fix Cloud IAM, Increase Quotas, or Add Subnet CIDRs"]
    CloudError -->|"No: Bootstrap Failed"| InspectUserData["Inspect cloud-init / ignition logs on target node"]

    HasConditions -->|"No: Stuck in transitional condition"| CheckController{"Is Controller Pod running & reconciling?"}
    CheckController -->|"CrashLoopBackOff"| FixController["Inspect controller pod logs: OOMKilled or invalid flags"]
    CheckController -->|"Running"| CheckWorkqueue{"Check workqueue depth & rate limits"}
    CheckWorkqueue -->|"High depth / Backoff"| Throttled["Provider API rate-limited: tune concurrency / sync period"]
    CheckWorkqueue -->|"Normal"| Finalizers["Inspect resource metadata.finalizers for blocking deletion"]

Operational Diagnostic Playbooks

1. Safely Pausing a Cluster During Emergencies

If an external infrastructure incident or network partition occurs, immediately pause reconciliation to prevent automated controllers from taking destructive actions:

# Pause reconciliation for an entire cluster
kubectl patch cluster production-us-east \
  --type merge \
  -p '{"spec":{"paused":true}}'

When spec.paused is true, CAPI controllers skip reconciliation cycles for the cluster and all underlying machines, preventing remediation storms while you resolve upstream infrastructure failures.

2. Resolving Stuck Resource Deletions (Finalizers)

If a Machine or Cluster object remains stuck in Deleting indefinitely, a controller is failing to release its finalizer.

# 1. Inspect active finalizers on the stuck machine
kubectl get machine my-worker-node-abcde -o jsonpath='{.metadata.finalizers}'

# 2. Check provider controller logs to identify which cloud resource cannot be deleted
kubectl logs -n capa-system deployment/capa-controller-manager -c manager | grep my-worker-node-abcde

# 3. Only if the cloud resource has been confirmed manually deleted: remove finalizer
kubectl patch machine my-worker-node-abcde \
  --type json \
  -p '[{"op": "remove", "path": "/metadata/finalizers"}]'

[!CAUTION] Never manually remove a finalizer without verifying that the underlying cloud instance or load balancer has been terminated. Forcibly stripping finalizers abandons external infrastructure, causing persistent cloud resource leaks and billing waste.

6. Key Takeaways

• The Management Cluster is Tier-0: Decoupled workload data planes mean customer pods stay online during outages, but management cluster failures freeze all auto-scaling, healing, upgrades, and provisioning.
• Provider Health Determines Fleet Health: A failure in cert-manager or an infrastructure provider admission webhook halts cluster mutations across the entire organization.
• Guard Against Automation Storms: Configure rate limits, surge boundaries, and conservative MachineHealthCheck thresholds to avoid cascading cloud quota exhaustion or fleet-wide VM replacement during network partitions.
• Disaster Recovery Must Be Tested: clusterctl move is an interactive migration tool, not a disaster recovery mechanism. Production environments require scheduled etcd snapshots and automated CRD backup pipelines.
• Separate Management from Applications: Never schedule tenant workloads on the management cluster. Isolate management control planes with dedicated high-IOPS etcd storage and strict network boundaries.

Next in the series

Now that we have covered running Cluster API reliably in production, explore Part 15: Provider Architecture and Provider Contracts to examine how Cluster API decouples core reconciliation logic from cloud implementations through standardized provider contracts.

References & Further Reading

• Cluster API Book: Production Operational Recommendations
• Cluster API Book: Diagnostics and Telemetry
• Cluster API Book: Security Guidelines
• Cluster API Book: Upgrading Management and Workload Clusters
• Cluster API Book: Provider Specifications and Contracts
• Velero Documentation: Backup and Restore of Custom Resources