Upgrading a production Kubernetes cluster has historically been one of the most nerve-wracking operations in systems engineering. In procedural environments, upgrades rely on imperative automation scripts (e.g., Ansible or SSH loops) executing sequential package manager invocations: upgrading kubeadm, mutating local package versions for kubelet and kubectl, writing new manifests for static control plane pods, and hoping that intermediate failure states do not corrupt the local operating system or orphan cloud resources.

Cluster API replaces procedural, in-place host mutation with declarative, immutable rolling machine replacement.

Rather than modifying binaries on running operating systems, Cluster API reconciles a cluster to a new Kubernetes version by stamping out pristine machines provisioned with pre-baked target-version images, safely validating health and etcd quorum at each step, draining existing workloads respecting PodDisruptionBudgets, and gracefully retiring older instances.

Understanding this process requires examining the fundamental split between management plane and workload cluster upgrades, the strict order of operations enforced by the Kubernetes version skew policy, the internal mechanics of KubeadmControlPlane and MachineDeployment rollouts, and the single-knob workflow enabled by ClusterClass managed topologies.

1. Upgrades as Immutable Machine Replacement

The core architectural philosophy of Cluster API is that compute hosts are disposable and immutable. Mutating packages in place introduces configuration drift: leftover files from previous releases, mismatched kernel dependencies, and uncertain recovery paths if an upgrade script terminates halfway through.

In Cluster API, an upgrade is fundamentally a version-driven rolling replacement:

DimensionProcedural In-Place Upgrade (kubeadm via SSH/Ansible)Declarative CAPI Upgrade (Immutable Replacement)
Execution ModelImperative shell commands modifying live hostsDeclarative spec mutation reconciled by controllers
Host StateDirty host: old OS artifacts, stale configs remainClean host: pristine VM spun up from versioned image
Failure RecoveryManual troubleshooting of half-upgraded packagesDeterministic rollback or automatic retry of clean VMs
Drift ResistanceProne to divergence across long-lived nodesZero host drift: immutable golden images
Rollout CadenceSerial execution requiring custom orchestratorsGoverned natively by maxSurge and maxUnavailable
Quorum ProtectionOperator must manually verify etcd membershipAutomated etcd health gates built into KubeadmControlPlane

2. Two Upgrades That Must Never Be Confused

When operating Cluster API, teams frequently conflate two completely separate upgrade domains:

1. Management Plane Provider Upgrade:
   clusterctl upgrade apply --core cluster-api:v1.8.0 ...
   -> Upgrades the CAPI controllers running in the management cluster.
   -> Does NOT touch workload cluster Kubernetes versions.

2. Workload Cluster Kubernetes Upgrade:
   Patching KubeadmControlPlane / MachineDeployment version (or Cluster.spec.topology.version)
   -> Upgrades the actual Kubernetes version running on target control-plane and worker nodes.

Management plane controllers must support the target workload cluster version. Always verify your CAPI provider version support matrix before initiating a workload cluster upgrade.

3. The Strict Upgrade Sequence & Version Skew Policy

The Kubernetes project strictly enforces the Version Skew Policy. In any Kubernetes cluster:

  1. kube-apiserver must be upgraded before any worker node components (kubelet, kube-proxy).
  2. Worker nodes may lag the control plane by up to three minor versions ($n-3$), but worker nodes can never be newer than the control plane.
  3. Direct upgrades must proceed one minor version at a time (e.g., $1.29 \rightarrow 1.30 \rightarrow 1.31$). Skipping minor versions is unsupported because etcd storage schema migrations, API deprecations, and conversion webhooks depend on sequential version progression.

Cluster API enforces this order through its decoupled controller architecture:

flowchart TD
    UPGRADE["Operator updates desired version<br/>(e.g., v1.30.0 -> v1.31.0)"] --> KCP_START["1. KubeadmControlPlane Rollout"]
    
    subgraph ControlPlaneRollout["Phase 1: Control Plane Upgrade"]
        KCP_SURGE["Surge target-version CP Machine"] --> ETCD_JOIN["Join etcd quorum & verify health"]
        ETCD_JOIN --> CP_DRAIN["Cordon & drain oldest CP Machine"]
        CP_DRAIN --> ETCD_REMOVE["Remove old CP from etcd member list"]
        ETCD_REMOVE --> CP_TERM["Terminate old CP VM"]
        CP_REPEAT{"All CP replicas upgraded?"}
        CP_TERM --> CP_REPEAT
        CP_REPEAT -->|"No"| KCP_SURGE
        CP_REPEAT -->|"Yes"| CP_READY["Control Plane Stable & Converged"]
    end

    KCP_START --> ControlPlaneRollout
    CP_READY -->|"Enforces Version Skew: Kubelet <= Kube-apiserver"| MD_START["2. Worker Fleet Rollout"]

    subgraph WorkerRollout["Phase 2: Worker Fleet Upgrade"]
        MD_NEW_MS["MachineDeployment creates new MachineSet"] --> WORKER_SURGE["Surge worker capacity (maxSurge)"]
        WORKER_SURGE --> WORKER_DRAIN["Cordon & drain old worker Machine (PDB compliant)"]
        WORKER_DRAIN --> WORKER_TERM["Detach CSI volumes & terminate old VM"]
        WORKER_REPEAT{"All worker pools upgraded?"}
        WORKER_TERM --> WORKER_REPEAT
        WORKER_REPEAT -->|"No"| WORKER_SURGE
        WORKER_REPEAT -->|"Yes"| ALL_DONE["Cluster Upgrade Complete"]
    end

    MD_START --> WorkerRollout

4. Phase 1: Control Plane Upgrade Internals (KubeadmControlPlane)

Upgrading a high-availability (HA) control plane requires maintaining etcd quorum and API availability throughout the entire transition.

The KubeadmControlPlane (KCP) controller manages this lifecycle using an automated surge-and-retire state machine:

sequenceDiagram
    autonumber
    participant KCP as KCP Controller
    participant API as Management API
    participant etcd as etcd Quorum
    participant OldCP as Old CP Machine (v1.30)
    participant NewCP as New CP Machine (v1.31)

    KCP->>API: Detects spec.version = v1.31.0
    KCP->>etcd: Preflight check: Verify etcd cluster healthy & quorum = 3
    KCP->>API: Creates NewCP Machine (v1.31) with new MachineTemplate
    NewCP->>etcd: Boots, runs cloud-init, joins etcd quorum (members = 4)
    etcd-->>KCP: Quorum confirmed, new API server responding
    KCP->>OldCP: Cordon workload node (unschedulable = true)
    KCP->>etcd: etcdctl member remove <old-member-id> (members = 3)
    KCP->>OldCP: Evict static pods & drain node
    KCP->>OldCP: Delete OldCP Machine CR & terminate VM
    Note over KCP: Repeats for remaining control-plane machines

The Step-by-Step Control Plane Rollout

  1. Preflight Health Checks: Before taking any action, KCP confirms that all existing control plane machines are healthy, that the etcd cluster has an active leader, and that no unmanaged members exist.
  2. Surging New Capacity: KCP clones the infrastructure and bootstrap templates for the target version and creates a 4th control plane Machine.
  3. Quorum Expansion: The new control plane VM boots, cloud-init executes, and kubeadm join --control-plane registers the new node with the existing etcd cluster. The etcd cluster temporarily expands from 3 to 4 members.
  4. Health Verification: KCP verifies that the new machine’s API server is reachable, etcd synchronization is complete, and the Node reports Ready = True.
  5. Safe Member Retirement: KCP selects the oldest control plane machine. It executes etcdctl member remove to cleanly shrink the etcd cluster back to 3 members, cordons the node, drains any running workloads, deletes the InfrastructureMachine, and deletes the old Machine resource.
  6. Convergence: KCP repeats this sequence one machine at a time until all 3 control plane machines are running the target version.

5. Phase 2: Worker Fleet Upgrade Internals (MachineDeployment)

Once the control plane has converged on the target version, worker nodes can be upgraded.

Worker nodes are governed by MachineDeployment resources. As explored in worker hierarchy internals, infrastructure and bootstrap templates in Cluster API are immutable. You cannot edit an existing AWSMachineTemplate or KubeadmConfigTemplate to point to a new version.

Instead, the upgrade follows an immutable rotation protocol:

1. Create new InfrastructureMachineTemplate (e.g. prod-worker-v1-31-amt) with target image/AMI.
2. Patch MachineDeployment.spec.template:
   - spec.template.spec.version = "v1.31.0"
   - spec.template.spec.infrastructureRef.name = "prod-worker-v1-31-amt"
3. MachineDeployment detects template drift, derives a new hash, and creates a new MachineSet.
4. MachineDeployment shifts capacity between old and new MachineSets under RollingUpdate rules.

Rollout Capacity Controls: maxSurge and maxUnavailable

The worker rollout rate is governed by MachineDeployment.spec.strategy.rollingUpdate:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  • maxSurge: 1: Cluster API creates 1 new worker machine before terminating an old one. This preserves total cluster compute capacity, ensuring workloads have headroom to reschedule before old nodes are drained.
  • maxUnavailable: 0: Ensures that old worker nodes are never cordoned or drained until replacement nodes are verified as running and fully ready (minReadySeconds satisfied).

Graceful Node Drain During Upgrade

When an old worker machine is scheduled for retirement, Cluster API invokes the Kubernetes Eviction API:

  • Respects all active PodDisruptionBudgets (PDBs).
  • Waits for attached CSI persistent volumes to cleanly unmount and detach.
  • Honors spec.template.spec.nodeDrainTimeout to prevent stuck pods from stalling the cluster-wide upgrade.

6. Single-Knob Upgrades with Managed Topologies (ClusterClass)

In traditional Cluster API configurations, an operator must patch the KubeadmControlPlane, create new infrastructure templates, and patch each MachineDeployment individually.

With ClusterClass and Managed Topologies, the entire cluster upgrade collapses into modifying a single field on the root Cluster resource:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-cluster
spec:
  topology:
    class: production-cluster-class
    version: v1.31.0   # <- The single source of truth for cluster version
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
        - class: default-worker
          name: md-0
          replicas: 10

How the Topology Controller Drives the Upgrade

When you patch spec.topology.version:

  1. The Topology Controller intercepts the change.
  2. It automatically generates new versioned template objects for the control plane and workers from the underlying ClusterClass.
  3. It updates KubeadmControlPlane.spec.version first, initiating the control plane rolling replacement.
  4. It watches KCP conditions. While the control plane is upgrading, the topology controller deliberately holds worker MachineDeployments at the previous version to strictly enforce the version skew policy.
  5. Once KCP reaches status.version = v1.31.0 and all control plane replicas are available, the topology controller cascades the version update to all worker MachineDeployments.

Critical Rule: When using managed topologies, never directly edit generated KubeadmControlPlane or MachineDeployment resources. Any manual changes made to child objects will be automatically overwritten by the topology reconciliation loop.

7. Troubleshooting Stalled Upgrades

Upgrades can stall if underlying infrastructure, cloud quotas, or application constraints prevent controllers from making forward progress.

flowchart TD
    START["Upgrade Stalled"] --> PHASE{"Which Phase is Blocked?"}

    PHASE -->|"Control Plane"| CP_CHECK{"Where is KCP stuck?"}
    CP_CHECK -->|"Surge VM Pending"| CP_Q1["Check cloud quotas, subnet IPs,<br/>and new image AMI/VHD availability"]
    CP_CHECK -->|"etcd Not Joining"| CP_Q2["Inspect cloud-init logs & TLS certs<br/>for port 2379/2380 connectivity"]
    CP_CHECK -->|"Old CP Won't Delete"| CP_Q3["Verify etcd quorum: KCP will not remove<br/>member if remaining members < quorum"]

    PHASE -->|"Worker Fleet"| W_CHECK{"Where is MachineDeployment stuck?"}
    W_CHECK -->|"Surge Capacity Blocked"| W_Q1["Check maxSurge quota & cloud instance limits"]
    W_Q2["Check PDB violations blocking pod eviction<br/>or CSI volume detach locks"]
    W_CHECK -->|"Drain Hanging"| W_Q2
    W_CHECK -->|"Template Hash Drift"| W_Q3["Ensure new MachineTemplate created instead<br/>of mutating old template in place"]

Step-by-Step Diagnostic Inspection

1. Inspect Overall Rollout State

clusterctl describe cluster <cluster-name> --show-conditions

This command provides a real-time hierarchical view of the cluster tree, highlighting which machine is currently surging or draining and showing failure reasons.

2. Check Control Plane Rollout Status

kubectl describe kubeadmcontrolplane <cluster-name>-control-plane

Look for:

  • ReadyReplicas vs. UpdatedReplicas.
  • Condition MachinesCreated: Checks if cloud instances are being launched.
  • Condition Resized: Verifies whether KCP is surging new members or waiting for old member deletion.

3. Diagnose Worker Fleet Drain Failures

If the worker rollout is frozen with an old machine stuck in Deleting:

kubectl describe machine <stuck-worker-machine>

Look at the events for:

  • DrainNodeFailed: Indicates a pod cannot be evicted due to a restrictive PodDisruptionBudget (minAvailable: 100%) or a pod refusing to terminate within its grace period.
  • VolumeDetachFailed: Cloud storage provider has not released an attached volume.

8. Practical Upgrade Checklist

Pre-Upgrade Verification
[ ] Management cluster CAPI providers support the target Kubernetes version.
[ ] Target OS machine image (AMI / VHD / template) built, published, and accessible.
[ ] Workload cluster etcd cluster is healthy and all control plane machines are Ready.
[ ] Cloud provider quota verified: account has headroom for maxSurge instances.
[ ] Subnet CIDRs verified: sufficient IP addresses available for surge VMs.

Execution (Manual or Managed Topology)
[ ] Upgrade control plane first: wait for KCP to report all replicas updated and healthy.
[ ] Verify workload cluster API server health via kubectl.
[ ] Upgrade worker pools: rotate MachineDeployment templates and update versions.
[ ] Monitor node drains and confirm workloads reschedule cleanly.

Post-Upgrade Verification
[ ] kubectl get nodes reports all control plane and worker nodes on target version.
[ ] No orphaned Machine resources remain in the management cluster.
[ ] Storage CSI and Networking CNI plugins functioning normally on new nodes.

9. Key Takeaways

  • Immutable Replacement over In-Place Mutation: Cluster API never runs package upgrade scripts on running nodes. Upgrades execute via rolling machine replacement using pre-baked golden images.
  • Strict Version Skew Sequencing: The control plane must always upgrade and stabilize before worker nodes are touched.
  • Etcd Quorum Preservation: KubeadmControlPlane manages control plane rolling updates by temporarily expanding etcd quorum from 3 to 4, verifying health, and cleanly removing the old member.
  • Managed Topologies Simplify Upgrades: With ClusterClass, changing Cluster.spec.topology.version automatically orchestrates the entire two-phase rollout while maintaining version skew compliance.
  • Surge Headroom Matters: Ensure cloud instance quotas and network subnet IP address pools have sufficient headroom to accommodate maxSurge capacity during fleet rollouts.

Next in the series

Now that we understand how Cluster API orchestrates rolling upgrades across control planes and worker fleets, explore Part 11: ClusterClass and Managed Topologies to learn how platform teams standardize multi-cluster blueprints using reusable topologies and inline patches.

References & Further Reading