Machine, MachineSet, and MachineDeployment: Worker Node Hierarchy Internals

Learning Journey: Part 07 of 16

Module Goal: Master the internal mechanics of Cluster API’s worker node hierarchy: deconstruct how MachineDeployment orchestrates rolling revisions, how MachineSet enforces replica convergence, and how Machine coordinates the stateful, ordered lifecycle of individual infrastructure hosts.
Target Spec: Cluster API v1.14 (v1beta2 APIs).
Prerequisites: Part 06: Anatomy of Cluster Creation: What Actually Happens When You Create a Cluster? and Part 03: Understanding the Cluster API Object Model.

1. First-Principles Problem Framing: The Asymmetry of Compute Lifecycles

In standard Kubernetes application management, compute units are stateless and disposable. When you roll out a new version of an application, the Deployment controller manages ReplicaSets, which spawn and destroy Pods.

A Pod is lightweight:

• A collection of Linux cgroups, namespaces, and memory limits.
• Scheduling and execution occur in milliseconds to seconds.
• Storage volumes (if ephemeral) are discarded without external state coordination.
• Failure during creation simply prompts the scheduler to pick another node.

When managing infrastructure hosts through Kubernetes controllers, this mental model collapses. A worker node is not a lightweight container:

Application Plane (Pods)                  Infrastructure Plane (Machines)
------------------------                  ------------------------------
Instantiated in milliseconds              Instantiated in 2 to 10 minutes
cgroups and namespace primitives          Hypervisors, cloud VMs, physical servers
Discarded immediately on failure          Consumes quota, IP leases, and hardware budget
Detaches ephemeral mounts cleanly         Must flush disks, drain pods, detach SAN/EBS
No cloud provider API coordination        Heavy API rate limits, IAM bindings, VPC routing

The Breakdown of Traditional Host Management

Historically, teams maintained worker fleets through imperative automation (Ansible playbooks, Terraform pipelines, Packer golden images, or Bash scripts):

1. Configuration Drift via In-Place Mutation: Running configuration management across a fleet of 500 running VMs inevitably produces snowflake hosts. When an OS package upgrade fails on worker-042 due to a locked apt database or transient network timeout, the node remains in an inconsistent, half-mutated state.
2. Brittle Scripted Rollouts: An imperative rolling-update script that loops through hosts via SSH or cloud APIs cannot handle unexpected failures. If an instance termination hangs waiting for a detached EBS volume, the script aborts, leaving the cluster partitioned between old and new operating system kernels.
3. The Danger of Ungraceful Eviction: Abruptly terminating an infrastructure host without coordinating with the Kubernetes workload scheduler destroys stateful workloads. Unattached persistent volume claims (PVCs) remain locked, pod disruption budgets (PDBs) are violated, and stateful databases experience ungraceful failovers.

Cluster API (CAPI) resolves this through a declarative, three-tier worker reconciliation hierarchy:

MachineDeployment (Rollout & Revision Strategy)
       ↓
MachineSet (Replica Count Convergence)
       ↓
Machine (Individual Host Lifecycle & Node Identity)

By separating fleet-level rollouts from replica counting and host-level lifecycle mechanics, CAPI brings the safety of declarative convergence to physical and virtual compute instances.

2. Mental Model & Architectural Shifts: The Tripartite Worker Hierarchy

To understand CAPI’s worker plane, draw a direct architectural comparison between the core Kubernetes workload controller hierarchy and the Cluster API infrastructure hierarchy:

flowchart TD
    subgraph WorkloadHierarchy["Kubernetes Workload Hierarchy"]
        D["Deployment<br/>(Rollout Strategy & Revision Management)"]
        RS["ReplicaSet<br/>(Maintains Target Pod Count)"]
        P["Pod<br/>(Cgroups, Containers, Ephemeral IP)"]
        D ==>|Manages Revisions| RS
        RS ==>|Maintains Replicas| P
    end

    subgraph InfraHierarchy["Cluster API Infrastructure Hierarchy"]
        MD["MachineDeployment<br/>(Rollout Strategy & Revision Management)"]
        MS["MachineSet<br/>(Maintains Target Machine Count)"]
        M["Machine<br/>(Host Lifecycle, Provider Binding, Node Identity)"]
        MD ==>|Manages Revisions| MS
        MS ==>|Maintains Replicas| M
    end

While the structural symmetry is intentional, the responsibilities and failure costs diverge sharply.

The Separation of Concerns

Each tier in the CAPI worker hierarchy operates with a strictly bounded scope:

flowchart LR
    MD["MachineDeployment"] -->|"1. Reconciles Revisions"| MS["MachineSet"]
    MS -->|"2. Clones Templates"| M["Machine"]
    M -->|"3. Coordinates"| BC["BootstrapConfig (KubeadmConfig)"]
    M -->|"4. Coordinates"| IM["InfrastructureMachine (e.g. AWSMachine)"]
    IM -->|"5. Launches Compute"| HOST["Physical Host / Cloud VM"]
    HOST -->|"6. Kubelet Registers"| NODE["Workload Node"]
    M -.->|"7. Pairs Identity via providerID"| NODE

1. MachineDeployment: The Fleet-Level Rollout Controller

• Core Function: Owns fleet-wide desired state and orchestrates zero-downtime rolling upgrades.
• Reconciliation Scope: Monitors changes to spec.template. When the underlying machine image, Kubernetes version, or infrastructure template reference changes, MachineDeployment calculates a template hash, derives a new MachineSet, and gradually shifts capacity from the old MachineSet to the new one according to maxSurge and maxUnavailable budgets.
• Does Not: Directly create, delete, or manage individual VMs.

2. MachineSet: The Homogeneous Replica Controller

• Core Function: Enforces continuous convergence between desired and observed machine counts.
• Reconciliation Scope: Given a concrete template and a replica target (e.g., replicas: 5), the MachineSet controller creates or deletes Machine objects to balance the delta. It executes preflight safety checks (e.g., control plane version skew and etcd health) before launching new machines, and implements deterministic deletion ordering when scaling down.
• Does Not: Understand rollout strategies or manage multiple generations of templates.

3. Machine: The Host Lifecycle Coordinator

• Core Function: Coordinates the point-to-point provisioning, identity binding, and safe destruction of a single infrastructure host.
• Reconciliation Scope: Holds direct references to an infrastructure resource (infrastructureRef) and a bootstrap resource (bootstrapRef). It acts as a conductor: waiting for cloud-init secret generation, waiting for VM IP allocation, polling for providerID, observing workload cluster registration, and executing ordered draining upon deletion.
• Does Not: Know how many peers exist or care why it was created.

3. Comparison Tables & Contract Matrices

Workload Plane vs. Infrastructure Plane

The following matrix clarifies the operational and architectural differences between Kubernetes workloads and CAPI infrastructure abstractions:

Object Disambiguation: Machine vs. Node vs. InfrastructureMachine

Engineers new to Cluster API frequently conflate Machine, Node, and provider-specific infrastructure objects. They exist in different clusters and serve distinct roles:

Rollout Strategy Comparison: RollingUpdate vs. OnDelete

MachineDeployment supports two distinct rollout strategies configured in spec.strategy.type:

4. End-to-End Execution Walkthroughs: Deep Internal Mechanics

When a change is applied to a worker fleet, reconciliation flows through five distinct internal phases across three controller loops.

flowchart TD
    subgraph Phase1["Phase 1: Revision Management & Hashing"]
        MD_APPLY["User updates MachineDeployment spec.template"]
        MD_HASH["Controller hashes spec.template to machine-template-hash"]
        MD_REV["Allocates new revision number (e.g. rev 2)"]
        MD_NEW_MS["Instantiates new MachineSet with hash label"]
    end

    subgraph Phase2["Phase 2: MachineSet Reconciliation & Preflight"]
        MS_DELTA["MachineSet evaluates desired vs actual replicas"]
        MS_PRE["Runs preflight checks: ControlPlane ready? Version skew valid?"]
        MS_CLONE["Clones InfrastructureMachineTemplate & KubeadmConfigTemplate"]
        MS_CREATE_M["Instantiates concrete Machine object"]
    end

    subgraph Phase3["Phase 3: Machine Provisioning & Identity Binding"]
        M_COORD["Machine controller observes bootstrapRef & infrastructureRef"]
        BOOT_SEC["Bootstrap controller renders cloud-init Secret"]
        INFRA_VM["Infrastructure controller boots VM with Secret"]
        NODE_JOIN["Kubelet starts on VM, joins workload cluster"]
        BIND_ID["Machine controller matches spec.providerID to Node.spec.providerID"]
    end

    subgraph Phase4["Phase 4: Workload-Aware Rolling Update"]
        SURGE_CHK["MachineDeployment monitors maxSurge & maxUnavailable"]
        AVAIL_CHK["Wait for new Machine Node to pass minReadySeconds"]
        SCALE_OLD["Scale down old MachineSet by 1 replica"]
    end

    subgraph Phase5["Phase 5: Ordered Deletion Protocol"]
        HOOK_PRE["Execute pre-drain lifecycle hooks"]
        CORDON["Cordon workload Node (Unschedulable)"]
        DRAIN["Evict Pods respecting PodDisruptionBudgets"]
        VOL_WAIT["Wait for PVC volume detachment"]
        HOOK_TERM["Execute pre-terminate lifecycle hooks"]
        VM_DEL["Delete InfrastructureMachine and cloud VM"]
        FIN_CLEAN["Remove finalizer, garbage collect Machine"]
    end

    Phase1 ==> Phase2
    Phase2 ==> Phase3
    Phase3 ==> Phase4
    Phase4 ==> Phase5

Phase 1: Revision Management and Template Hashing

A MachineDeployment definition specifies worker configuration using a pod-like template:

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  name: prod-worker-md
  namespace: default
spec:
  clusterName: prod-cluster
  replicas: 3
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: prod-cluster
  template:
    metadata:
      labels:
        role: worker
    spec:
      clusterName: prod-cluster
      version: v1.31.0
      bootstrap:
        configRef:
          apiGroup: bootstrap.cluster.x-k8s.io
          kind: KubeadmConfigTemplate
          name: prod-worker-kct-v1
      infrastructureRef:
        apiGroup: infrastructure.cluster.x-k8s.io
        kind: AWSMachineTemplate
        name: prod-worker-amt-v1

How the Controller Detects Template Drift

1. Hashing the Template: The MachineDeployment controller computes a deterministic 32-bit FNV-1a hash over spec.template. This hash is assigned to the label machine-template-hash (e.g., machine-template-hash: "746f889b9d").
2. Revision Annotation: The controller maintains the deployment.kubernetes.io/revision annotation on owned MachineSets. If the active MachineSet has a matching template hash, the controller detects a scaling-only event and updates the existing MachineSet.spec.replicas.
3. Triggering a Rollout: If spec.template has changed (for example, pointing to AWSMachineTemplate/prod-worker-amt-v2 or upgrading version: v1.31.1), the controller calculates a new hash, creates a brand new MachineSet, and begins rolling upgrade reconciliation.

[!IMPORTANT] Infrastructure Templates Are Immutable: Cluster API contract rules mandate that InfrastructureMachineTemplate objects must never be mutated in place. To change instance types, subnets, or security groups, you must create a new template object (prod-worker-amt-v2) and update the MachineDeployment.spec.template.spec.infrastructureRef pointer. In-place mutation of template objects causes indeterminate reconciliation states across existing machines.

Phase 2: MachineSet Reconciliation and Preflight Safety Gates

Once created, the new MachineSet reconciles its replica count:

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineSet
metadata:
  name: prod-worker-md-746f889b9d
  namespace: default
  ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta2
      kind: MachineDeployment
      name: prod-worker-md
      uid: 91b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d
spec:
  clusterName: prod-cluster
  replicas: 1 # Set dynamically by MachineDeployment during surge
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: prod-cluster
      machine-template-hash: "746f889b9d"
  template:
    # Machine blueprint...

Preflight Safety Gates

Before creating any Machine object, the MachineSet controller executes preflight checks to protect cluster integrity:

1. Control Plane Health: Verifies that the workload cluster control plane is fully initialized and reporting healthy status. Launching workers while the control plane is degraded risks orphaned join attempts.
2. Kubernetes Version Skew Enforcement: Verifies that the requested worker version is not newer than the control plane version, adhering strictly to Kubernetes version skew policy (a worker node can never exceed the API server version).
3. Kubeadm Skew Validation: Ensures the version of kubeadm bundled in the bootstrap specification is compatible with the target Kubernetes release.

If preflight checks fail, the controller blocks Machine creation and records an event on the MachineSet. Operators can bypass these checks in emergency recovery scenarios using the annotation machineset.cluster.x-k8s.io/skip-preflight-checks, though this is not recommended in standard operations.

Template Instantiation

Once cleared, the MachineSet instantiates concrete resources for each replica:

• It clones KubeadmConfigTemplate into a unique KubeadmConfig object.
• It clones AWSMachineTemplate into a unique AWSMachine object.
• It creates a Machine object, setting its bootstrapRef and infrastructureRef to these newly instantiated objects, and injects ownerReferences pointing back to itself.

Phase 3: Machine Provisioning and Identity Binding

The Machine controller reconciles the individual host:

sequenceDiagram
    autonumber
    participant MC as Machine Controller
    participant BC as Bootstrap Controller (CABPK)
    participant IC as Infrastructure Controller (CAPA)
    participant VM as Cloud VM / Kubelet
    participant W_API as Workload Cluster API

    MC->>BC: Observes KubeadmConfig (needs user-data)
    BC->>MC: Renders cloud-init script into Kubernetes Secret
    MC->>IC: Observes AWSMachine, passes Secret name
    IC->>VM: Launches EC2 instance with user-data Secret
    VM->>VM: Executes cloud-init, runs kubeadm join
    VM->>W_API: Kubelet registers Node object
    IC->>MC: Writes spec.providerID (e.g. aws:///us-east-1/i-0123)
    MC->>W_API: Queries workload cluster for Node with matching providerID
    W_API-->>MC: Found Node ip-10-0-1-50
    MC->>MC: Sets status.nodeRef to Node name, marks Machine Running

The Identity Bridge: How Machine Binds to Node

The management cluster and workload cluster are physically separate control planes. The Machine controller links them using spec.providerID:

1. The cloud provider assigns a global hardware identifier when the instance boots (e.g., aws:///us-east-1a/i-0a1b2c3d4e5f67890).
2. The infrastructure controller reads this identifier from the cloud API and writes it into InfrastructureMachine.spec.providerID.
3. The core Machine controller copies it into Machine.spec.providerID.
4. When kubelet starts on the host, its cloud provider integration populates Node.spec.providerID.
5. The Machine controller queries the workload API server for any Node matching the provider identity:

   Machine.spec.providerID == Node.spec.providerID
   

6. When a match is discovered, the Machine controller populates Machine.status.nodeRef. The management plane now knows the exact Kubernetes Node representing that physical host.

Phase 4: Rollout Execution Algorithms (RollingUpdate)

The MachineDeployment controller orchestrates the transition between the old MachineSet (MS-old) and the new MachineSet (MS-new) using a rolling update algorithm.

The Rollout Invariants and Budget Bounds

The rollout algorithm enforces two mathematical bounds during every reconciliation loop:

Total Machines Across All Sets <= desiredReplicas + maxSurge
Available Machines Across All Sets >= desiredReplicas - maxUnavailable

Where:

• desiredReplicas: Target worker replica count (e.g., 3)
• maxSurge: spec.strategy.rollingUpdate.maxSurge (default: 1)
• maxUnavailable: spec.strategy.rollingUpdate.maxUnavailable (default: 0)

Step-by-Step Rollout Progression (Replicas = 3, maxSurge = 1, maxUnavailable = 0)

Step 0: Steady State (v1)
---------------------------------------------------------------
MS-v1: replicas=3, ready=3, available=3
MS-v2: replicas=0, ready=0, available=0
Total Machines: 3 (Allowed Max: 3 + 1 = 4)

Step 1: Surge Scale-Up (Scale MS-v2 to 1)
---------------------------------------------------------------
MS-v1: replicas=3, ready=3, available=3
MS-v2: replicas=1, ready=0, available=0 (Provisioning VM...)
Total Machines: 4

Step 2: New Machine Becomes Available
---------------------------------------------------------------
MS-v1: replicas=3, ready=3, available=3
MS-v2: replicas=1, ready=1, available=1 (Node Ready + minReadySeconds elapsed)
Total Available: 4 (Minimum Required: 3 - 0 = 3)

Step 3: Scale Down Old MachineSet (Scale MS-v1 to 2)
---------------------------------------------------------------
MS-v1: replicas=2, ready=2, available=2 (1 Machine entering deletion/drain)
MS-v2: replicas=1, ready=1, available=1
Total Available: 3

Step 4: Surge Next Replica (Scale MS-v2 to 2)
---------------------------------------------------------------
MS-v1: replicas=2, ready=2, available=2
MS-v2: replicas=2, ready=1, available=1 (Second v2 VM provisioning...)
Total Machines: 4

[... Process repeats until MS-v1 reaches 0 and MS-v2 reaches 3 ...]

Readiness vs. Availability: The minReadySeconds Buffer

A host reporting VM Running in AWS or vSphere is not ready to accept production traffic. Even when Node.status.conditions reports Ready=True, application daemonsets, service meshes, or telemetry collectors may still be initializing.

MachineDeployment provides spec.minReadySeconds (e.g., minReadySeconds: 120). A newly ready machine is not counted as “Available” by the rollout controller until it has remained continuously healthy for this duration. This prevents the rollout controller from tearing down an old machine while the new machine is flapping or failing to start ingress pods.

Phase 5: The Graceful Node Deletion and Teardown Workflow

Deleting a Machine triggers a sophisticated multi-stage teardown protocol designed to prevent data loss and workload downtime.

sequenceDiagram
    autonumber
    actor MS as MachineSet Controller
    participant MC as Machine Controller
    participant W_API as Workload Cluster API (Node)
    participant IC as Infrastructure Controller
    participant CP as Cloud Provider (AWS/GCP/vSphere)

    MS->>MC: Deletes Machine (Sets metadata.deletionTimestamp)
    MC->>MC: Evaluates pre-drain lifecycle hooks
    Note over MC,W_API: Stage 1: Cordoning
    MC->>W_API: Patches Node spec.unschedulable = true
    Note over MC,W_API: Stage 2: Eviction & Drain
    MC->>W_API: Issues Pod Eviction API requests
    W_API-->>MC: Pods terminate respecting PodDisruptionBudgets
    Note over MC,W_API: Stage 3: Volume Detachment
    MC->>W_API: Polls VolumeAttachments until all PVCs are detached
    Note over MC,IC: Stage 4: Pre-Terminate Hooks & VM Teardown
    MC->>MC: Evaluates pre-terminate lifecycle hooks
    MC->>IC: Deletes InfrastructureMachine CR
    IC->>CP: Calls Cloud API: TerminateInstance
    CP-->>IC: Instance destroyed
    Note over MC,W_API: Stage 5: Node Cleanup & Finalizer Release
    MC->>W_API: Deletes Node object from Workload API
    MC->>MC: Removes finalizer: machine.cluster.x-k8s.io
    MC-->>MS: Machine CR vanishes from management etcd

Timeout Configuration Controls

To prevent hung nodes from stalling cluster operations indefinitely, CAPI exposes granular timeout controls on the Machine specification:

spec:
  nodeDrainTimeout: 10m
  nodeVolumeDetachTimeout: 5m
  nodeDeletionTimeout: 10m

• nodeDrainTimeout: Maximum time allowed for all non-daemonset pods to be successfully evicted. If this timeout expires, the controller logs an error and forces the deletion forward to avoid freezing rollouts.
• nodeVolumeDetachTimeout: Maximum time allowed to wait for CSI storage providers to cleanly detach network storage volumes before terminating compute.
• nodeDeletionTimeout: Maximum duration allowed for the workload Node object to be deregistered from the workload API server.

MachineSet Deletion Ordering Algorithms

When a MachineSet must scale down, it selects machines for termination using one of several deterministic strategies configured in spec.deletePolicy:

1. Random (Default): Selects machines with no preference.
2. Newest: Deletes the most recently created machines first, useful when rolling back an erroneous scale-out event.
3. Oldest: Deletes the longest-running machines first, useful for proactive node recycling to prevent memory fragmentation and OS drift.
4. Explicit Target Annotation (cluster.x-k8s.io/delete-machine): If an operator or automated remediator places this annotation on a specific Machine, the MachineSet controller prioritizes that machine for eviction during the next scale-down reconciliation, bypassing the default policy.

5. Mental Experiments & Diagnostics: Top-Down Troubleshooting Playbook

When a worker fleet fails to scale or a rollout becomes stuck, use a systematic top-down diagnostic methodology.

flowchart TD
    START["Rollout or Scaling Issue Reported"] --> MD_CHK{"1. Inspect MachineDeployment<br/>kubectl get md"}
    
    MD_CHK -->|Replicas != Desired| MD_COND["Check MD Conditions & Strategy<br/>Is maxSurge / maxUnavailable blocking?"]
    MD_CHK -->|Replicas == Desired| MS_CHK{"2. Inspect MachineSets<br/>kubectl get ms -l ..."}
    
    MS_CHK -->|Multiple MS active| MS_ROLL["Rollout in Progress.<br/>Which MS is not progressing?"]
    MS_CHK -->|New MS not scaling| PRE_CHK["Check MS Events & Preflight Checks<br/>kubectl describe ms &lt;new-ms&gt;<br/>Control plane degraded or version skew?"]
    
    MS_ROLL --> M_CHK{"3. Inspect Machines<br/>kubectl get machines -l ..."}
    PRE_CHK --> M_CHK
    
    M_CHK -->|Phase: Provisioning| INFRA_CHK["Check InfrastructureMachine<br/>Cloud quota exhausted? Subnet IPs full?"]
    M_CHK -->|Phase: Running, Node not ready| BOOT_CHK["Check Bootstrap Secret & Cloud-Init<br/>Did kubelet fail to join? CNI missing?"]
    M_CHK -->|Phase: Deleting stuck| DRAIN_CHK["Check Node Drain & PVCs<br/>PDB violation blocking eviction? Stuck volume?"]

Diagnostic Scenarios & Root Cause Analysis

Scenario A: Rollout Stalled at Surge Creation (New MachineSet Replicas = 0)

• Symptom: You update MachineDeployment to reference a new AWSMachineTemplate, but no new worker machines appear.
• Diagnostic Inspection:

  kubectl describe machineset <new-machineset-name>
  

• Observed Event:

  Warning  FailedPreflightChecks  1m  machineset-controller
  Preflight check failed: Kubernetes version "v1.32.0" cannot be higher than ControlPlane version "v1.31.0"
  

• Root Cause: CAPI preflight checks blocked MachineSet expansion because the operator attempted to upgrade workers before upgrading the control plane.
• Resolution: Revert the worker version field to match the control plane version, or upgrade the control plane (KubeadmControlPlane) first.

Scenario B: Rollout Stalled Mid-Way (New Machine Provisioned, Old Not Draining)

• Symptom: New machine is in Running phase, but the old machine remains active and the rollout does not proceed to the next node.
• Diagnostic Inspection:

  kubectl get machinedeployment prod-worker-md
  # Shows: REPLICAS: 4, READY: 4, AVAILABLE: 3
  
  kubectl get machine <new-machine-name> -o yaml
  

• Check Conditions:

  status:
    conditions:
      - type: NodeReady
        status: "True"
      - type: MachinesAvailable
        status: "False"
        reason: WaitingForMinReadySeconds
  

• Root Cause: The machine’s node is ready, but spec.minReadySeconds has not yet elapsed. The controller is intentionally waiting to guarantee stability before terminating old capacity.
• Resolution: Normal operation. If stuck indefinitely, verify whether application pods on the new node are crash-looping or failing readiness gates.

Scenario C: Machine Deletion Frozen in Deleting Phase

• Symptom: A machine targeted for scale-down remains stuck in Deleting for hours.
• Diagnostic Inspection:

  kubectl describe machine <stuck-machine-name>
  

• Observed Event:

  Warning  DrainNodeFailed  42s  machine-controller
  Cannot evict pod "order-processor-0" due to PodDisruptionBudget "order-pdb": allowed disruptions is 0
  

• Root Cause: A PodDisruptionBudget configured with minAvailable: 100% prevents the eviction API from terminating the pod. Because spec.nodeDrainTimeout was omitted (defaulting to infinite wait), the machine controller refuses to terminate the host.
• Resolution: Scale up the deployment owning the blocked pod, adjust the PDB, or set a reasonable nodeDrainTimeout on the MachineDeployment spec to bound future teardown delays.

Command Cheat Sheet for Worker Fleet Inspection

# 1. View complete worker hierarchy across all revisions
kubectl get machinedeployment,machineset,machine -o wide

# 2. Correlate Machine template hashes to active MachineSets
kubectl get machinesets -o custom-columns=\
NAME:.metadata.name,\
HASH:.metadata.labels."machine-template-hash",\
REPLICAS:.spec.replicas,\
READY:.status.readyReplicas,\
AVAILABLE:.status.availableReplicas

# 3. Trace providerID binding between Machine and Node
kubectl get machines -o custom-columns=\
NAME:.metadata.name,\
PHASE:.status.phase,\
PROVIDERID:.spec.providerID,\
NODEREF:.status.nodeRef.name

# 4. Trigger an immediate rolling restart of a worker fleet
clusterctl alpha rollout restart machinedeployment/prod-worker-md

# 5. Prioritize a specific malfunctioning machine for the next scale-down
kubectl annotate machine prod-worker-abcde \
  cluster.x-k8s.io/delete-machine="yes"

6. Key Learning Takeaways

• Deliberate Separation of Concerns: MachineDeployment manages rollout strategies and template revisions; MachineSet maintains exact replica counts; Machine coordinates point-to-point host lifecycle and provider resource bindings.
• The Asymmetry of Host Lifecycle: Replacing a node takes minutes and involves disk flushes, network teardown, and cloud API coordination. CAPI balances this using nodeDrainTimeout and nodeVolumeDetachTimeout to prevent data loss.
• Immutable Templates Prevent Drift: Never mutate infrastructure or bootstrap template objects in place. Create new versioned templates and rotate the references in the MachineDeployment spec to trigger deterministic, traceable rollouts.
• Identity Bridge via providerID: Management Machine objects and workload Node objects exist in separate clusters and are matched purely by their shared, globally unique spec.providerID.
• Preflight Safety Gates: MachineSet controllers enforce version skew and control plane health before instantiating new worker machines, preventing broken cluster states during upgrades.
• Readiness vs. Availability: Cloud VMs running does not equal worker readiness. CAPI leverages minReadySeconds and condition bubbling to verify true application capacity before retiring older nodes.

7. Self-Check Questions

Test your comprehension of worker node hierarchy internals before proceeding:

1. Why does mutating an existing AWSMachineTemplate in place fail to trigger a rolling update across an existing MachineDeployment?
2. If a MachineDeployment is configured with replicas: 5, maxSurge: 2, and maxUnavailable: 1, what is the maximum number of total machines that can exist simultaneously during a rollout, and what is the minimum number of available machines that must be preserved?
3. What is the exact sequence of events executed by the Machine controller when a machine enters the deletion lifecycle, and what happens if a PodDisruptionBudget prevents a pod from being evicted?
4. How does the Machine controller in the management cluster know which Node object in the workload cluster corresponds to a specific Machine custom resource?
5. Why is cluster.x-k8s.io/delete-machine a safer mechanism for removing a problematic node than running kubectl delete machine <name> directly?

Next Milestone in the Learning Journey

In Part 08: Scaling Kubernetes Clusters with Cluster API, we will explore dynamic cluster scaling: manual replica expansion, failure-domain distribution across cloud availability zones, cluster autoscaler integration, scale-from-zero semantics, and specialized GPU/heterogeneous worker pools.

References & Further Reading

• Cluster API Book: MachineDeployment Controller Implementation
• Cluster API Book: MachineSet Controller Implementation
• Cluster API Book: Machine Lifecycle and Controller
• Cluster API Book: Updating Machine Templates & Rollout Strategies
• Cluster API Specification: v1beta2 CRD Reference
• Kubernetes Core Documentation: Version Skew Policy
• Kubernetes Core Documentation: Node Draining and Eviction