Static infrastructure sizing is inherently inefficient. Over-provisioning compute capacity to survive peak traffic spikes wastes substantial cloud expenditure during idle hours. Conversely, under-provisioning risks capacity starvation: incoming workloads sit stalled in Pending with scheduling errors when user demand surges.

In cloud environments, engineers often reach for cloud-native autoscaling mechanisms like AWS Auto Scaling Groups (ASGs) or Azure Virtual Machine Scale Sets (VMSS). However, cloud autoscalers operate strictly on hypervisor-level metrics (e.g., average CPU or memory utilization of the virtual machine). This creates a fatal blind spot in Kubernetes: a cluster may have 80% idle hypervisor CPU, yet a new pod requiring 32 GB of memory or an attached GPU remains unschedulable due to memory fragmentation, specific node affinity rules, taints, or storage topology zone constraints.

The Kubernetes Cluster Autoscaler (CA) solves this by observing the cluster from the inside out: it monitors the kube-scheduler queue for unschedulable pods rather than watching VM metrics.

When integrated with Cluster API through the Cluster API cloud provider (--cloud-provider=clusterapi), Cluster Autoscaler brings cross-cloud, declarative autoscaling to any infrastructure provider without vendor lock-in.

1. The Separation of Concerns: Sizing vs. Provisioning

The foundational mental model for autoscaling in Cluster API is the strict architectural boundary between capacity calculation and infrastructure reconciliation:

flowchart TD
    subgraph WorkloadCluster["Workload Cluster (Workload & Scheduling Plane)"]
        PENDING["Pending Pods<br/>(Reason: Unschedulable)"] --> CA["Cluster Autoscaler<br/>(Evaluates Scheduling Feasibility)"]
        UNUSED["Underutilized Nodes<br/>(Request sum < 50% for 10m)"] --> CA
        SCHED["kube-scheduler"] -.-> PENDING
    end

    subgraph ManagementCluster["Management Cluster (Declarative Infrastructure Plane)"]
        CA -->|"1. Mutates spec.replicas<br/>(over Management API)"| MD["MachineDeployment<br/>(Annotated with min/max bounds)"]
        MD -->|"2. Reconciles replica drift"| MS["MachineSet"]
        MS -->|"3. Manages Machine lifecycles"| M["Machines"]
        M -->|"4. Launches cloud instances"| INFRA["Infrastructure Provider<br/>(CAPA, CAPZ, CAPV)"]
    end

    INFRA -.->|"5. New Nodes join cluster"| WorkloadCluster

To reason about autoscaling, anchor on what each system owns:

ComponentOperational DomainPrimary ResponsibilityInput Signal
kube-schedulerWorkload ClusterAssigns pending pods to existing nodes with matching resourcesPod resource requests, taints, tolerations, affinities
Cluster AutoscalerWorkload / ManagementCalculates how many nodes of a specific shape are neededPending unschedulable pods, underutilized nodes
Cluster APIManagement ClusterExecutes the asynchronous machine lifecycle (VM creation, cloud-init, node join, graceful drain)MachineDeployment.spec.replicas drift
Infrastructure ProviderCloud / HypervisorInterfaces with cloud APIs to allocate or destroy compute, disks, and network interfacesInfrastructureMachine custom resources

Cluster Autoscaler never provisions a virtual machine directly. It simply patches the desired spec.replicas on a MachineDeployment. Cluster API then executes the concrete infrastructure lifecycle.

2. Deployment Topologies & API Access

Cluster Autoscaler requires access to two distinct Kubernetes APIs:

  1. Workload Cluster API: To watch Pod scheduling states, Node resource requests, and evaluate eviction feasibility.
  2. Management Cluster API: To discover scalable MachineDeployment resources and mutate their spec.replicas.

Depending on your security and operational requirements, Cluster Autoscaler can be deployed in two topologies:

flowchart LR
    subgraph InWorkload["Topology A: Autonomous (In-Workload Cluster)"]
        W_CA["Cluster Autoscaler Pod"]
        W_SA["In-Cluster ServiceAccount<br/>(Watches Workload Pods/Nodes)"]
        W_SEC["Kubeconfig Secret<br/>(Talks to Management Cluster API)"]
        W_CA --> W_SA
        W_CA --> W_SEC
    end

    subgraph InManagement["Topology B: Centralized (In-Management Cluster)"]
        M_CA["Cluster Autoscaler Pod"]
        M_SA["In-Cluster ServiceAccount<br/>(Mutates local CAPI CRDs)"]
        M_SEC["Target Kubeconfig Secret<br/>(Watches remote Workload Cluster)"]
        M_CA --> M_SA
        M_CA --> M_SEC
    end

Auto-Discovery by Cluster Name

Cluster Autoscaler discovers scalable worker pools dynamically without hardcoding static resource names. When started with --node-group-auto-discovery, it targets only the MachineDeployments that belong to the specific cluster:

cluster-autoscaler \
  --cloud-provider=clusterapi \
  --node-group-auto-discovery=clusterapi:clusterName=prod-cluster,namespace=default \
  --scale-down-unneeded-time=10m \
  --scale-down-delay-after-add=10m \
  --balance-similar-node-groups=true

3. Node Group Contract: Annotations & Configuration

To enable Cluster Autoscaler on a MachineDeployment, annotate the resource with minimum and maximum size boundaries:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: prod-worker-general
  namespace: default
  annotations:
    # Define autoscaling bounds for Cluster Autoscaler discovery
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "2"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "15"
spec:
  clusterName: prod-cluster
  replicas: 2
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: prod-cluster
  template:
    # Machine template specification...

The Critical ClusterClass / Managed Topology Gotcha

In Part 11, we established that in managed topologies, Cluster.spec.topology is the single source of truth. This introduces a critical operational conflict if not configured correctly:

THE DRIFT CONFLICT:
1. Operator sets: Cluster.spec.topology.workers.machineDeployments[0].replicas: 2
2. Pending pods arrive -> Cluster Autoscaler scales MachineDeployment.spec.replicas: 2 -> 5
3. Topology Controller detects drift between Cluster.spec.topology (2) and MachineDeployment (5)
4. Topology Controller overwrites MachineDeployment.spec.replicas back to 2!

The Golden Rule for Autoscaling Managed Topologies: When a MachineDeployment is managed by Cluster Autoscaler, leave spec.topology.workers.machineDeployments[].replicas completely UNSET in the Cluster resource.

# Correct Managed Topology Configuration for Autoscaling:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-cluster
spec:
  topology:
    class: production-aws-class
    version: v1.31.0
    workers:
      machineDeployments:
        - class: general-purpose
          name: md-general
          # replicas: intentionally omitted!

When omitted, the Topology Controller delegates replica management to the Cluster Autoscaler, preventing state flapping.

4. Scale-From-Zero Mechanics & GPU Workloads

A major cost optimization pattern is maintaining specialized worker pools (e.g., expensive GPU compute, high-memory instances, or FPGA nodes) configured with min-size: 0.

However, scaling from zero presents an architectural challenge:

THE COLD-START DILEMMA:
When replicas = 0, no Node exists in the workload cluster.
How does Cluster Autoscaler know whether scaling this empty MachineDeployment
will satisfy a pending pod requiring 8 vCPUs, 32GB RAM, and an NVIDIA A10G GPU?

In typical cloud providers, CA queries cloud instance APIs to discover VM specifications. In Cluster API, the autoscaler inspects the MachineDeployment spec, infrastructure templates, and capacity annotations:

sequenceDiagram
    autonumber
    participant Pod as Pending Workload Pod
    participant CA as Cluster Autoscaler
    participant MD as MachineDeployment (replicas: 0)
    participant CAPI as Cluster API
    participant Node as Workload Node

    Pod->>Pod: Requires nvidia.com/gpu: 1 & memory: 32Gi
    Pod-->>CA: Status: Pending (Unschedulable)
    CA->>MD: Inspects node group annotations & InfrastructureMachineTemplate
    Note over CA: Simulates scheduling using capacity metadata
    CA->>CA: Confirms scaling md-gpu from 0 to 1 satisfies Pod
    CA->>MD: Patches spec.replicas = 1
    MD->>CAPI: Stamped out Machine & cloud VM instance
    CAPI->>Node: Boots VM, runs GPU driver bootstrap, registers Node
    Node-->>CA: Node Ready with allocatable nvidia.com/gpu: 1
    CA->>Pod: kube-scheduler binds Pod to new GPU Node

Declaring Capacity for Scale-From-Zero

To enable Cluster Autoscaler to simulate scheduling accurately without running nodes, annotate the MachineDeployment with capacity definitions:

metadata:
  name: gpu-worker-pool
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "8"

    # Explicit capacity annotations for scale-from-zero simulation:
    capacity.cluster-autoscaler.kubernetes.io/cpu: "16"
    capacity.cluster-autoscaler.kubernetes.io/memory: "64Gi"
    capacity.cluster-autoscaler.kubernetes.io/gpu-type: "nvidia-a10g"
    capacity.cluster-autoscaler.kubernetes.io/gpu-count: "1"

Many modern infrastructure providers (like CAPA for AWS) automatically expose instance type specifications to CAPI, allowing the autoscaler to extract CPU and memory allocations directly from the referenced AWSMachineTemplate.

5. Scale-Down & The Graceful Deletion Handshake

Scale-down is where Cluster Autoscaler and Cluster API coordinate closely to prevent workload disruption:

flowchart TD
    EVAL["1. CA evaluates underutilized node<br/>(Request sum < 50% for scale-down-unneeded-time)"] --> CHECK{"2. Can pods be safely evicted?"}

    CHECK -->|"No: PDB violation / Local storage"| BLOCK["Abort scale-down for this node"]
    CHECK -->|"Yes: All pods can reschedule"| MARK["3. Annotate Machine CR:<br/>cluster.x-k8s.io/delete-machine=yes"]

    MARK --> DECR["4. Decrement MachineDeployment.spec.replicas"]
    DECR --> MS["5. MachineSet Controller prioritizes<br/>annotated Machine for scale-down"]

    MS --> CORDON["6. Cordon Workload Node<br/>(spec.unschedulable = true)"]
    CORDON --> DRAIN["7. Evict pods via Eviction API<br/>(Honors PodDisruptionBudgets)"]
    DRAIN --> DETACH["8. Detach Persistent Volumes"]
    DETACH --> DESTROY["9. Terminate Cloud VM & Delete Machine CR"]

The Coordinated Deletion Handshake

  1. Candidate Identification: CA identifies a node whose utilization has remained below the threshold (default 50%) for the duration of --scale-down-unneeded-time (default 10 minutes).
  2. Eviction Simulation: CA verifies that all pods currently running on the candidate can be scheduled on remaining nodes without violating any PodDisruptionBudgets.
  3. The Victim Mark: CA does not delete the node randomly. It adds the annotation:
    cluster.x-k8s.io/delete-machine: "yes"
    directly to the target Machine resource in the management cluster.
  4. Replica Decrement: CA decrements MachineDeployment.spec.replicas by 1.
  5. CAPI Deletion Priority: When the MachineSet controller reconciles the reduced replica count, its selection algorithm specifically picks the machine marked with cluster.x-k8s.io/delete-machine="yes".
  6. Drain Execution: CAPI executes the multi-phase drain sequence: cordoning the node, invoking the Eviction API, detaching CSI volumes, and destroying the cloud VM.

6. Diagnostic Workflows: Decision Trees & Debugging

When autoscaling fails to expand or contract capacity, isolate whether the problem is an Autoscaler Decision Failure or a Cluster API Provisioning Failure:

flowchart TD
    START["Autoscaling Problem Detected"] --> SYM{"What is the Symptom?"}

    SYM -->|"Pending Pods but Replicas Not Increasing"| CA_FAIL{"Inspect Cluster Autoscaler"}
    CA_FAIL -->|"Node group missing"| CA_Q1["Verify min-size & max-size annotations<br/>and --node-group-auto-discovery flags"]
    CA_FAIL -->|"Max size reached"| CA_Q2["Current replicas == max-size: Increase max bound"]
    CA_FAIL -->|"Simulation failed"| CA_Q3["Pod requests (CPU/Memory/GPU/Affinity) exceed<br/>the capacity of defined worker classes"]

    SYM -->|"Replicas Increased but Nodes Not Joining"| CAPI_FAIL{"Inspect Cluster API Layer"}
    CAPI_FAIL -->|"Machine stuck in Pending"| CAPI_Q1["Check Cloud Quota / VPC subnet IP exhaustion<br/>in InfraMachine status"]
    CAPI_FAIL -->|"Machine stuck in Provisioning"| CAPI_Q2["Inspect cloud-init bootstrap logs on VM<br/>for join token or port 6443 timeouts"]
    CAPI_FAIL -->|"Node NotReady"| CAPI_Q3["Verify CNI plugin is functional on the node"]

    SYM -->|"Scale-Down Blocked"| SD_FAIL{"Inspect Scale-Down Blockers"}
    SD_FAIL -->|"PDB violations"| SD_Q1["Inspect PodDisruptionBudgets blocking eviction"]
    SD_FAIL -->|"Unreplicated pods"| SD_Q2["Check for pods not managed by Deployment/ReplicaSet"]

Essential Diagnostic Commands

1. Inspect Autoscaler Status ConfigMap

Cluster Autoscaler writes its internal decision state to a ConfigMap inside the workload cluster:

kubectl --kubeconfig workload.kubeconfig -n kube-system get configmap cluster-autoscaler-status -o yaml

Look for:

  • Health: Confirms the autoscaler loop is executing.
  • ScaleUp: Lists whether node groups are ready, unready, or reached maximum size.
  • ScaleDown: Lists candidate nodes and reasons why nodes are not currently scalable.

2. Check Autoscaler Annotations on MachineDeployments

kubectl get machinedeployments -o custom-columns=\
NAME:.metadata.name,\
REPLICAS:.spec.replicas,\
MIN:.metadata.annotations."cluster\.x-k8s\.io/cluster-api-autoscaler-node-group-min-size",\
MAX:.metadata.annotations."cluster\.x-k8s\.io/cluster-api-autoscaler-node-group-max-size"

3. Inspect Autoscaler Pod Logs

kubectl --kubeconfig workload.kubeconfig -n kube-system logs -l app=cluster-autoscaler -f

Filter for:

  • Scale-up: Shows simulated scheduling calculations and node group selection.
  • Not scheduleable: Details which constraints (CPU, memory, affinity) prevented a pending pod from fitting into an existing node group.

7. Key Takeaways

  • Decoupled Roles: Cluster Autoscaler observes pod scheduling pressure in the workload cluster and determines capacity sizing. Cluster API executes the declarative machine lifecycle on the management cluster.
  • Unified Cross-Cloud Interface: The Cluster API cloud provider (--cloud-provider=clusterapi) allows the same autoscaler binary and configuration to scale AWS, Azure, GCP, VMware, and bare-metal fleets consistently.
  • Managed Topology Rule: When using ClusterClass, leave spec.topology.workers.machineDeployments[].replicas unset so the Topology Controller does not fight the Autoscaler’s replica changes.
  • Scale-From-Zero Requires Capacity Metadata: For 0-replica node groups (especially GPU pools), use capacity annotations or provider metadata so CA can simulate scheduling without running nodes.
  • The Victim Handshake: During scale-down, CA marks victim machines with cluster.x-k8s.io/delete-machine="yes", ensuring that CAPI safely drains and terminates the exact node chosen by the autoscaler.

Next in the series

Now that we understand how Cluster Autoscaler dynamically drives worker fleet sizing, explore Part 13: Bootstrap Clusters, Management Clusters, and clusterctl move to understand how management planes are bootstrapped from local Kind clusters and pivoted into production management clusters.

References & Further Reading