Self-Healing with MachineHealthCheck

When a software process inside a container crashes, the local kubelet restarts the container in seconds. When a node’s physical hardware fails, the kernel deadlocks, or the network interface drops, the problem transcends container orchestration: the underlying compute instance itself must be diagnosed, isolated, and remediated.

In unmanaged cloud environments, operators often rely on cloud autoscaling groups (ASGs, MIGs) for instance health. However, cloud health checks only monitor basic hypervisor liveness. They are completely blind to Kubernetes-level failures: corrupt Container Network Interface (CNI) state, deadlocked storage drivers, unmounted kubelet partitions, or silent etcd communication partitions.

Cluster API solves this through the MachineHealthCheck (MHC) controller. MHC bridges workload cluster Node health signals into declarative management cluster lifecycle operations.

Crucially, MachineHealthCheck is a detection and gating engine, not a repair script. It evaluates whether machines violate health policies and authorizes remediation, delegating actual machine replacement or power-cycling to higher-level controllers under strict safety limits.

1. The Detection vs. Remediation Architectural Split

A common misconception is that MachineHealthCheck deletes VMs or runs reboot commands. In idiomatic Kubernetes controller design, responsibilities are cleanly decoupled:

flowchart TD
    NODE["Workload Node<br/>(Reports conditions: Ready, DiskPressure)"] -->|"Informer watches Node status"| MHC["MachineHealthCheck Controller<br/>(Detection & Safeguard Gating)"]
    M["Machine CR<br/>(Binds providerID to Node)"] -->|"Monitored by"| MHC

    MHC -->|"Condition: RemediationAllowed=True"| DECIDE{"Remediation Strategy"}

    DECIDE -->|"Owner-Based (Default)"| OWNER["Owner Controller<br/>(MachineSet / KubeadmControlPlane)"]
    DECIDE -->|"External Template"| EXT["External Remediation Controller<br/>(Metal3, AWS, SelfNodeRemediation)"]

    OWNER -->|"Drains, deletes & replaces"| NEW_M["New Machine & Cloud VM"]
    EXT -->|"In-place power-cycle / reboot"| BMC["Hardware BMC / Cloud API"]

Anchor on this division of responsibilities:

By separating health observation from remediation execution, Cluster API ensures that broken machines are replaced according to the same declarative blueprints used for routine provisioning.

2. Anatomy of a MachineHealthCheck Resource

A production MachineHealthCheck manifest specifies which machines to watch, the conditions that define unhealthiness, and the timeout thresholds required before triggering remediation:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineHealthCheck
metadata:
  name: worker-healthcheck
  namespace: default
spec:
  clusterName: prod-cluster

  # Target specific worker pools using label selectors
  selector:
    matchLabels:
      nodepool: general-workers

  # Maximum duration to wait for a newly provisioned Machine to register a Node
  nodeStartupTimeout: 10m

  # Criteria defining an unhealthy workload node
  unhealthyConditions:
    - type: Ready
      status: "False"
      timeout: 300s
    - type: Ready
      status: "Unknown"
      timeout: 300s
    - type: DiskPressure
      status: "True"
      timeout: 10m

  # Blast-radius safeguard: halt remediation if too many nodes are failing
  maxUnhealthy: "20%"

The Three Core Elements

1. Target Selector (selector): Identifies the pool of Machine resources to monitor. Never combine control plane and worker machines in the same health check policy; control plane nodes have distinct failure characteristics and quorum requirements.
2. Health Conditions (unhealthyConditions): Maps directly to Node.status.conditions. A machine is marked for remediation only when an observed condition matches the target status continuously for the entire duration of timeout.
3. Startup Timeout (nodeStartupTimeout): Defines how long a Machine can remain in the management cluster without a corresponding Node registering in the workload cluster.

3. Detecting Failure: Two Critical Signals

Cluster API monitors two distinct categories of failure: running host degradation and initialization stall (zombie hosts).

sequenceDiagram
    autonumber
    participant Node as Workload Node
    participant M as Machine Controller
    participant MHC as MachineHealthCheck Controller
    participant MS as MachineSet Controller

    Note over Node,MHC: Scenario A: Node Degrades in Service
    Node->>M: Kubelet stops sending heartbeats
    M->>M: Node condition transitions to Ready=Unknown
    MHC->>MHC: Starts unhealthy timer (timeout: 300s)
    Note over MHC: If Ready recovers before 300s, timer aborts
    MHC->>MHC: 300s elapsed: Evaluates maxUnhealthy guards
    MHC->>M: Sets condition: RemediationAllowed=True
    MHC->>M: Sets annotation: cluster.x-k8s.io/remediate-machine
    MS->>M: Initiates node drain and machine deletion
    MS->>M: Creates replacement Machine from template

    Note over M,MHC: Scenario B: Startup Initialization Failure
    M->>M: Machine provisioned, cloud VM launched
    Note over M: No Node registers with matching providerID
    MHC->>MHC: nodeStartupTimeout (10m) timer elapses
    MHC->>M: Sets condition: RemediationAllowed=True
    MS->>M: Deletes stuck VM, provisions fresh replacement

Signal 1: The Node Condition Timeout

The Kubernetes node lifecycle controller marks a node Ready=Unknown when it stops receiving heartbeats (defaulting to after 40 seconds of missed heartbeats). However, taking down a cloud VM immediately after 40 seconds is dangerous: temporary network congestion or a routine kubelet binary upgrade would trigger catastrophic node replacement.

MachineHealthCheck introduces an intentional dampening period via timeout:

• If the node recovers to Ready=True before the timeout expires, the timer resets and no action is taken.
• If the condition persists across the full window (e.g., 300 seconds), MHC transitions the machine to an unhealthy state.

Signal 2: The Startup Timeout (Zombie Hosts)

A common cloud infrastructure failure mode is a VM that launches successfully in AWS/Azure/GCP, but never becomes a Kubernetes node. Common causes include:

• Exhausted subnet IP addresses during CNI initialization.
• Corrupted cloud-init userdata or expired kubeadm join tokens.
• Security group rules blocking outbound traffic to the control plane on port 6443.

In this scenario, the Machine status shows Running (the VM exists), but Machine.status.nodeRef remains nil. Without nodeStartupTimeout, this machine would sit idle forever, consuming cloud budget while starving application capacity. Once nodeStartupTimeout expires without a bound node, MHC marks the machine unhealthy and requests a replacement.

4. Remediation Mechanisms: Owner-Based vs. External

Once a machine is declared unhealthy, how is it remediated? Cluster API supports two architectural paradigms:

flowchart TD
    UNHEALTHY["Unhealthy Machine Confirmed"] --> GATE{"Evaluate maxUnhealthy Guard"}
    GATE -->|"Threshold Exceeded"| PAUSE["Pause Remediation & Emit Warning Event"]
    GATE -->|"Within Threshold"| ROUTE{"Configured Remediation Mode"}

    ROUTE -->|"Default (Owner-Based)"| WORKER_CHECK{"Is Worker or Control Plane?"}
    WORKER_CHECK -->|"Worker (MachineSet)"| MS_REM["1. Cordon & Drain Node<br/>2. Delete Machine CR<br/>3. MachineSet spawns replacement"]
    WORKER_CHECK -->|"Control Plane (KCP)"| KCP_CHECK{"Is etcd Quorum Safe?"}
    KCP_CHECK -->|"Quorum Safe (3/3 healthy)"| KCP_REM["1. Remove etcd member<br/>2. Delete Machine CR<br/>3. KCP launches replacement"]
    KCP_CHECK -->|"Quorum at Risk"| KCP_BLOCK["Block Remediation to Prevent etcd Split-Brain"]

    ROUTE -->|"External (Template)"| EXT_REM["1. Instantiate External CR<br/>2. External controller reboots host / fences BMC<br/>3. Clears remediation annotation on recovery"]

1. Owner-Based Remediation (Default)

In the default configuration, MHC does not specify a remediationTemplate. It delegates the remediation lifecycle directly to the object that owns the machine:

Worker Nodes (MachineSet Owner)

The MachineSet controller sees the remediation signal. It initiates the standard machine deletion sequence:

1. Cordons the workload node.
2. Invokes the Pod Eviction API respecting PodDisruptionBudgets.
3. Detaches CSI volumes.
4. Deletes the cloud VM.
5. Decrements observed replicas and immediately provisions a brand-new Machine from the template.

Control-Plane Nodes (KubeadmControlPlane Owner)

Remediating a control plane node is high risk because etcd state is stateful. If a control plane member is deleted while etcd quorum is degraded, the cluster can lose its state store entirely.

KubeadmControlPlane implements strict preflight safety checks before executing remediation:

• The control plane must have at least 3 replicas.
• All other control plane machines must report Ready=True.
• Etcd membership checks confirm that removing the failed member will not cause quorum loss.
• If quorum is at risk, KCP blocks remediation until healthy nodes recover or an operator intervenes.

2. External Remediation

For bare-metal environments or edge deployments, deleting and reprovisioning a host is slow and disruptive. External remediation allows plugging in specialized controllers:

spec:
  remediationTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: Metal3RemediationTemplate
    name: metal3-remediation

Instead of deleting the Machine, MHC instantiates a concrete Metal3Remediation CR. An external controller intercepts this object and performs out-of-band recovery: triggering an IPMI power cycle, resetting a hardware Baseboard Management Controller (BMC), or rebooting the hypervisor. If the host recovers, the external controller clears the remediation condition, avoiding VM recreation entirely.

5. Blast-Radius Safeguards: The maxUnhealthy Guard

The most dangerous operational failure in automated infrastructure self-healing is cascading remediation storms.

Consider a top-of-rack network switch failure or an accidental DNS misconfiguration that isolates an entire availability zone. Instantly, 40 out of 50 worker nodes transition to Ready=Unknown.

If an automated controller indiscriminately deleted all 40 machines simultaneously:

• Workloads would face catastrophic cluster-wide eviction.
• Cloud APIs would be hammered with dozens of simultaneous VM provisioning calls, triggering rate limits and quota lockouts.
• Any stateful volume mounts would become trapped in unmount storms.

To prevent this, MachineHealthCheck enforces strict blast-radius guards:

spec:
  # May be specified as an absolute integer (e.g., 2) or a percentage (e.g., "20%")
  maxUnhealthy: "20%"

How the Guard Evaluates

1. MHC continuously tallies the total number of machines matched by its selector.
2. It calculates the current number of unhealthy machines.
3. If the percentage of unhealthy machines exceeds maxUnhealthy, remediation is immediately suspended for all machines in the pool:

Pool size: 10 machines
maxUnhealthy: 20% (threshold: 2 machines)

Case A: 1 machine unhealthy  → 1 <= 2  → Remediation executes.
Case B: 3 machines unhealthy → 3 > 2   → Remediation HALTED. Alert emitted.

When suspended, MHC sets the condition RemediationAllowed = False on the health check resource and emits a Kubernetes Warning event: RemediationRestricted. This forces automated recovery to pause, leaving the cluster in place for operator investigation rather than destroying surviving nodes.

6. Practical Inspection and Diagnostic Workflows

When implementing automated self-healing, you must be able to verify whether health checks are tracking correctly and diagnose why remediation might be blocked.

flowchart TD
    START["Machine Unhealthy but Not Remediated"] --> C1{"Does MHC Selector Match?"}
    C1 -->|"No"| F1["Fix Label Selector on MHC or Machine"]
    C1 -->|"Yes"| C2{"Has Timeout Elapsed?"}
    C2 -->|"No"| F2["Normal behavior: Awaiting timeout window"]
    C2 -->|"Yes"| C3{"Is maxUnhealthy Exceeded?"}
    C3 -->|"Yes"| F3["Remediation Blocked: Circuit breaker tripped<br/>Inspect cluster-wide network/infrastructure"]
    C3 -->|"No"| C4{"Is Owner Controller Supported?"}
    C4 -->|"Standalone Machine"| F4["Unsupported: Machine requires MachineSet or KCP owner"]
    C4 -->|"Supported Owner"| C5{"Is Control Plane Quorum Degraded?"}
    C5 -->|"Yes"| F5["KCP Protection: etcd quorum at risk<br/>Restore other CP nodes first"]
    C5 -->|"No"| F6["Inspect Owner Controller Logs for Drain/Cloud API Errors"]

Essential Diagnostic Commands

1. Inspect Health Check Status

kubectl get machinehealthchecks -o wide

Output displays:

• EXPECTEDMACHINES: Count of machines matched by the selector.
• CURRENTHEALTHY: Number of nodes currently reporting healthy conditions.
• MAXUNHEALTHY: Configured threshold limit.

2. Detailed Condition Inspection

kubectl describe machinehealthcheck <mhc-name>

Look at the Status conditions:

• HealthCheckSucceeded = True: All evaluated nodes are currently passing checks.
• RemediationAllowed = False with reason TooManyUnhealthy: Alerts that the circuit breaker has tripped and automated replacement is disabled.

3. Inspect the Target Machine

kubectl describe machine <machine-name>

Look for:

• Conditions: Check NodeReady and MachineHealthCheckSucceded.
• Annotations: Look for cluster.x-k8s.io/remediate-machine. If present, MHC has signaled the owner to initiate replacement.

7. Key Takeaways

• Decoupled Responsibilities: MachineHealthCheck detects unhealthy states and gates remediation safety. The machine’s owner controller (MachineSet or KubeadmControlPlane) executes the actual drain and replacement.
• Node Conditions as Telemetry: MHC bridges workload Node.status.conditions (Ready, DiskPressure) to management Machine lifecycles, using mandatory timeout dampening to prevent flapping.
• Startup Timeout Prevents Zombies: nodeStartupTimeout ensures that machines whose infrastructure provisioned but whose OS bootstrap or CNI failed are automatically purged and retried.
• Circuit Breaker Protection: The maxUnhealthy guard prevents automated self-healing from converting a partial infrastructure blip into a catastrophic cluster-wide deletion cascade.
• Control Plane Quorum Safety: Control plane machine remediation is gated by etcd quorum availability; Cluster API refuses to remediate a control plane node if doing so risks losing cluster consensus.

Next in the series

Now that we understand how Cluster API automates node failure detection and remediation, explore Part 10: Kubernetes Upgrades with Cluster API to learn how CAPI orchestrates zero-downtime, rolling control plane and worker fleet upgrades across Kubernetes versions.

References & Further Reading

• Cluster API Book: Health Checking
• Cluster API Book: MachineHealthCheck Controller
• Cluster API Specification: MachineHealthCheck v1beta1 API
• Kubernetes Core Documentation: Node Lifecycle Controller