Building Your First Cluster API Environment

Learning Journey: Part 04 of 16

Module Goal: Build a local Cluster API management environment using kind and the Docker infrastructure provider (CAPD), observe real-time custom resource expansion, and trace declarative cluster reconciliation.
Target Spec: Cluster API v1.14 (v1beta2 APIs).
Prerequisites: Part 03: Cluster API Object Model: Mapping the Custom Resource Graph (Three-layer resource model, contract references, and template blueprints).

1. The Hands-On Objective: Grounding Abstractions in Real State

In Parts 01 through 03, we established the core concepts of Cluster API (CAPI): declarative reconciliation, management versus workload cluster separation, and the three-layer object graph (Cluster, ControlPlane, MachineDeployment).

Now we translate those theoretical abstractions into running software.

The goal of this module is not simply executing a sequence of commands until a terminal prints:

cluster created successfully

Instead, the objective is opening up the management cluster control plane during execution to observe how controllers transform high-level declarative state into running Kubernetes clusters.

By the end of this lab, you will trace the full expansion cycle:

Cluster Intent (YAML)
   ↓
Management Cluster etcd
   ↓
CAPI Controllers
   ↓
MachineSet & Machine Objects
   ↓
Provider Infrastructure (Docker Containers)
   ↓
Workload Kubernetes Nodes

2. Target Architecture: Dual-Cluster Environment with Docker (CAPD)

To run Cluster API without incurring cloud provider costs, we use CAPD (Cluster API Provider Docker).

CAPD is an infrastructure provider plugin that creates Kubernetes cluster nodes as Docker containers on a local host system.

Our laboratory setup consists of two distinct Kubernetes clusters:

1. Management Cluster: A local kind cluster configured to communicate with the host Docker daemon. It hosts the CAPI provider controllers and etcd state store.
2. Workload Cluster: A multi-node Kubernetes cluster provisioned and managed by the management cluster. Its control plane and worker nodes run as sibling containers on the host Docker engine.

flowchart TB
    HOST["Host OS & Docker Engine"]

    subgraph MGMT["kind Management Cluster"]
        API["Kubernetes API Server"]
        CORE["Cluster API Core Controller"]
        BOOT["Kubeadm Bootstrap Controller"]
        CP["Kubeadm Control Plane Controller"]
        CAPD["Docker Infrastructure Provider (CAPD)"]
    end

    subgraph WORKLOAD["Workload Cluster"]
        WCP["Control Plane Node (Container)"]
        WORKERS["Worker Node (Container)"]
    end

    HOST --> MGMT
    API <--> CORE
    API <--> BOOT
    API <--> CP
    API <--> CAPD

    CAPD -->|Docker API via Socket Mount| HOST
    HOST -->|Provisions Node Containers| WCP
    HOST -->|Provisions Node Containers| WORKERS

On a cloud platform (e.g. AWS or Azure), the infrastructure provider issues API requests to provision EC2 instances or Azure VMs. In our local setup, CAPD issues API requests to the host Docker socket to create node containers.

The upper-level reconciliation loops remain identical across all providers.

Resource Allocation Note: CAPD provisions multiple Docker containers simulating full Kubernetes nodes. On macOS running Docker Desktop, ensure Docker is allocated at least 6 GB of RAM to prevent container OOM-kills during node initialization.

3. Prerequisites & Local Environment Validation

Ensure the following client binaries are installed on your workstation:

Validate binary versions before proceeding:

docker version
kind version
kubectl version --client
clusterctl version

On macOS systems using Homebrew, install clusterctl via:

brew install clusterctl

4. Step 1: Provisioning the Management Cluster

CAPD needs access to the host Docker daemon to create workload cluster containers. A standard kind cluster does not expose the host Docker socket (/var/run/docker.sock) inside its control plane container.

To grant socket access, create a kind configuration file with an extraMounts directive:

Create kind-cluster-with-extramounts.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4

networking:
  ipFamily: dual

nodes:
  - role: control-plane
    extraMounts:
      - hostPath: /var/run/docker.sock
        containerPath: /var/run/docker.sock

Boot the management cluster:

kind create cluster \
  --name capi-management \
  --config kind-cluster-with-extramounts.yaml

Verify context and node status:

kubectl cluster-info --context kind-capi-management
kubectl get nodes

At this stage, we have a standard kind Kubernetes cluster. It is not a Cluster API management cluster yet.

Standard Kubernetes Cluster != Cluster API Management Cluster

A cluster becomes a management cluster only after installing Cluster API CRDs and provider controller managers into its control plane.

5. Step 2: Initializing Cluster API Controllers

To configure CAPD quick-start templates, enable Cluster Topology support:

export CLUSTER_TOPOLOGY=true

Initialize Cluster API with the Docker infrastructure provider:

clusterctl init --infrastructure docker

While this command appears simple, clusterctl init performs a complex installation sequence:

flowchart TD
    INIT["clusterctl init --infrastructure docker"] --> CERT["cert-manager Deployment"]
    INIT --> CORE["Core Provider (capi-system)"]
    INIT --> BOOT["Bootstrap Provider (capi-kubeadm-bootstrap-system)"]
    INIT --> CP["Control Plane Provider (capi-kubeadm-control-plane-system)"]
    INIT --> INFRA["Docker Infrastructure Provider (capd-system)"]

clusterctl automatically fetches release manifests, applies required CRDs, deploys cert-manager for webhook TLS certificates, and launches controller deployments for all four CAPI provider pillars.

Inspecting Installed Infrastructure

Verify the newly created system namespaces:

kubectl get namespaces

You will observe four dedicated operational namespaces:

• capi-system: Core CAPI controllers (Cluster, Machine, MachineDeployment)
• capi-kubeadm-bootstrap-system: Kubeadm bootstrap provider (CAPBK)
• capi-kubeadm-control-plane-system: Kubeadm control plane provider (KCP)
• capd-system: Docker infrastructure provider (CAPD)
• cert-manager: Webhook certificate infrastructure

Verify the registered Custom Resource Definitions (CRDs):

kubectl get crds | grep cluster.x-k8s.io

Prior to clusterctl init, querying kubectl get clusters fails because the API server lacks the CRD schema. After initialization, the management cluster API server natively understands CAPI resources.

Inspecting Installed Providers & Controllers

When managing a Cluster API setup, distinguish between inspecting installed provider resources, checking controller manager deployments, and evaluating provider upgrade status:

1. Installed Providers in the Management Cluster

To list all installed Cluster API providers registered in the cluster, query the Provider CRD:

kubectl get providers.clusterctl.cluster.x-k8s.io -A

Or use the shorter equivalent:

kubectl get providers -A

Output:

2. Provider Controller Deployments

To inspect the running controller manager deployments handling reconciliation across all namespaces:

kubectl get deployments -A | grep -E 'capi|capd|kubeadm'

You can also inspect the individual controller pods:

kubectl get pods -A | grep -E 'capi|capd|cert-manager'

3. Provider Upgrade & Version Information

To check available updates or version upgrade plans across installed providers, use clusterctl:

clusterctl upgrade plan

This displays your currently installed provider versions alongside any newer releases available in provider repositories.

6. Step 3: Generating and Deconstructing the Workload Manifest

Now we generate the declarative manifest for our target workload cluster using clusterctl generate.

Execute:

clusterctl generate cluster capi-quickstart \
  --flavor development \
  --kubernetes-version v1.37.0 \
  --control-plane-machine-count=1 \
  --worker-machine-count=1 \
  > capi-quickstart.yaml

Note: For laboratory inspection, configuring 1 control plane node and 1 worker node minimizes local CPU and RAM consumption while keeping the object graph straightforward.

Generated Manifest vs Runtime Execution

A common misconception is assuming clusterctl generate cluster creates the cluster.

It does not. clusterctl generate is a client-side template expander. It reads pre-packaged YAML templates and outputs a unified manifest containing desired API objects.

The boundary between CLI generation and controller runtime is strict:

flowchart LR
    C1["clusterctl generate"] -->|1. Render YAML| FILE["capi-quickstart.yaml"]
    FILE -->|2. kubectl apply| API["Management API Server (etcd)"]
    API -->|3. Event Triggers| CTRL["CAPI Controller Managers"]
    CTRL -->|4. Continuous Reconciliation| INFRA["Host Docker Engine"]

Deconstructing the Generated Objects

Inspect the custom resource kinds defined inside capi-quickstart.yaml:

grep '^kind:' capi-quickstart.yaml

The output reveals six discrete custom resources across our three structural layers:

kind: Cluster
kind: DevCluster
kind: KubeadmControlPlane
kind: DevMachineTemplate
kind: MachineDeployment
kind: KubeadmConfigTemplate

Note on Provider Resource Names: Cluster API providers define concrete Custom Resources to fulfill generic infrastructure contracts (InfrastructureCluster, InfrastructureMachine, InfrastructureMachineTemplate). In current CAPI v1.14.x CAPD development flavors, these are represented by Dev* resources (DevCluster, DevMachineTemplate). Older DockerCluster and DockerMachineTemplate resources are deprecated.

Map each generated resource to its structural role:

7. Step 4: Applying Desired State & Tracing Graph Expansion

Submit the generated workload cluster manifest to the management API server:

kubectl apply -f capi-quickstart.yaml

To monitor multiple resource types periodically as CAPI provisions them, use the terminal watch command (or watch a single resource type using kubectl get <resource> -w):

watch -n 2 "kubectl get cluster,kubeadmcontrolplane,machinedeployments,machines"

Tip: You can also watch a single resource type in streaming mode (e.g., kubectl get machines -w), or view the live cluster hierarchy in real time using watch -n 2 "clusterctl describe cluster capi-quickstart".

The Object Graph Expansion Sequence

When you executed kubectl apply, you submitted high-level intent objects (Cluster, KubeadmControlPlane, MachineDeployment). You did not submit MachineSet or Machine manifests directly.

Controller managers continuously reconcile state and expand the object graph automatically:

flowchart TD
    subgraph Applied["Applied Manifest (User Intent)"]
        C["Cluster"]
        KCP["KubeadmControlPlane"]
        MD["MachineDeployment"]
        IT["DevMachineTemplate"]
        BT["KubeadmConfigTemplate"]
    end

    subgraph Derived["Derived Objects (Controller Generated)"]
        MS["MachineSet (Worker Pool)"]
        M_CP["Machine (Control Plane)"]
        M_W["Machine (Worker)"]
        DC_CP["DevMachine (Control Plane)"]
        DC_W["DevMachine (Worker)"]
        KC_CP["KubeadmConfig (Control Plane)"]
        KC_W["KubeadmConfig (Worker)"]
    end

    KCP -->|Creates & Owns| M_CP
    MD -->|Creates & Owns| MS
    MS -->|Creates & Owns| M_W

    M_CP -->|Instantiates| DC_CP
    M_CP -->|Instantiates| KC_CP

    M_W -->|Clones IT| DC_W
    M_W -->|Clones BT| KC_W

Verifying Garbage Collection Ownership

Verify who created the MachineSet by inspecting its metadata:

kubectl get machineset -o yaml | grep -A 5 ownerReferences

The output displays an explicit ownerReferences pointer linking back to the MachineDeployment:

ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta2
    kind: MachineDeployment
    name: capi-quickstart-md-0

Next, inspect a worker Machine:

kubectl get machine -l cluster.x-k8s.io/deployment-name=capi-quickstart-md-0 -o yaml | grep -A 5 ownerReferences

The ownership pointer leads back to the MachineSet.

This confirms that controller-driven garbage collection and lifecycle ownership mirror our API object graph.

8. Step 5: Dissecting Asynchronous Bootstrap & Infrastructure Handoff

Select one running Machine object and inspect its functional contract references:

kubectl get machine -o yaml

Locate the spec reference blocks:

spec:
  bootstrap:
    configRef:
      apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
      kind: KubeadmConfig
      name: capi-quickstart-md-0-x8z9p
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: DevMachine
    name: capi-quickstart-md-0-v4l2k

To quickly determine which provider-specific resource a Machine references without parsing full YAML specs, run:

kubectl get machine <machine-name> -o jsonpath='{.spec.infrastructureRef.kind}{"\n"}'

This is the most reliable way to determine whether a Machine points to DevMachine, DockerMachine, AWSMachine, or another provider-specific resource.

These reference pointers drive asynchronous handoffs between independent controllers:

flowchart LR
    subgraph Core["Core CAPI"]
        M["Machine"]
    end

    subgraph Bootstrap["Bootstrap Provider (CAPBK)"]
        BC["KubeadmConfig"] -->|Generates| SEC["Kubernetes Secret\n(cloud-init script)"]
    end

    subgraph Infra["Infrastructure Provider (CAPD)"]
        IM["InfrastructureMachine\n(DevMachine)"] -->|Consumes Secret| DOCKER["Host Docker API"]
    end

    M -->|spec.bootstrap.configRef| BC
    M -->|spec.infrastructureRef| IM
    SEC -. Read by .-> IM

1. The Core Machine Controller reconciles the Machine object and verifies reference targets exist.
2. The Bootstrap Controller (CAPBK) watches KubeadmConfig, generates the kubeadm join cloud-init script, and writes it to a Kubernetes Secret.
3. The Infrastructure Controller (CAPD) watches the concrete InfrastructureMachine (DevMachine), waits until the bootstrap Secret is ready, reads the payload, and invokes host Docker APIs to launch the node container.

Inspecting Infrastructure Resources & Host Engine Nodes

Because provider-specific custom resource names can change across CAPI versions and flavors, start by discovering the infrastructure API resources registered in your management cluster:

kubectl api-resources --api-group infrastructure.cluster.x-k8s.io

For current Cluster API v1.14.x CAPD development-flavor environments, inspect installed Dev* infrastructure resources:

kubectl get devclusters -A
kubectl get devmachines -A
kubectl get devmachinetemplates -A

Note: Older DockerCluster, DockerMachine, and DockerMachineTemplate resources are deprecated in current CAPD development APIs, which favor DevCluster, DevMachine, and DevMachineTemplate. Concrete CRD names vary depending on your provider and provider version, but all fulfill standard InfrastructureCluster, InfrastructureMachine, and InfrastructureMachineTemplate contracts.

Next, open a shell terminal on your host machine and query Docker directly:

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Status}}"

You will observe running container instances created by CAPD alongside the original capi-management-control-plane container:

CONTAINER ID   NAMES                                        STATUS
a1b2c3d4e5f6   capi-quickstart-control-plane-9x2lm          Up 3 minutes
f7e8d9c0b1a2   capi-quickstart-md-0-v4l2k                   Up 2 minutes
9876543210ab   capi-management-control-plane                Up 10 minutes

This makes the infrastructure abstraction tangible:

Machine  →  InfrastructureMachine (DevMachine)  →  CAPD Controller  →  Docker Container  →  Kubernetes Node

9. Step 6: Accessing & Networking the Workload Cluster

Instead of running multiple kubectl get commands to evaluate status, use clusterctl describe:

clusterctl describe cluster capi-quickstart

This command renders an visual tree of cluster health, control plane quorum, machine replica counts, and underlying provider readiness:

Cluster/capi-quickstart
├─ClusterInfrastructure - DevCluster/capi-quickstart-infra
├─ControlPlane - KubeadmControlPlane/capi-quickstart-control-plane
│ └─Machine/capi-quickstart-control-plane-9x2lm [Running]
└─Workers
  └─MachineDeployment/capi-quickstart-md-0
    └─MachineSet/capi-quickstart-md-0-7988587d6
      └─Machine/capi-quickstart-md-0-v4l2k [Running]

Extracting Workload Cluster Credentials

Retrieve the kubeconfig for the newly provisioned workload cluster:

kind get kubeconfig --name capi-quickstart > capi-quickstart.kubeconfig

Compare API server outputs between management and workload clusters:

Management Cluster Nodes:

kubectl get nodes

Workload Cluster Nodes:

kubectl --kubeconfig ./capi-quickstart.kubeconfig get nodes

Notice that these command targets hit completely separate Kubernetes API servers.

Management Cluster API Server  !=  Workload Cluster API Server
(Stores CAPI CRDs & Specs)         (Runs Application Workloads)

Resolving NotReady Node Status via CNI Installation

Inspecting workload cluster nodes reveals a status of NotReady:

NAME                                   STATUS     ROLES    AGE     VERSION
capi-quickstart-control-plane-9x2lm   NotReady   control-plane   5m      v1.37.0
capi-quickstart-md-0-v4l2k            NotReady   <none>          4m      v1.37.0

Nodes remain NotReady because Cluster API intentionally decouples cluster lifecycle management from workload CNI plugins. CAPI successfully provisions VMs and bootstraps kubelet, but pod networking requires a CNI.

Apply a CNI plugin (e.g. Calico) directly to the workload cluster:

kubectl --kubeconfig ./capi-quickstart.kubeconfig \
  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

Monitor node transition until status reports Ready:

kubectl --kubeconfig ./capi-quickstart.kubeconfig get nodes -w

Correlating Machine to Node via ProviderID

To verify how the management cluster correlates a Machine CR with a registered Node in the workload cluster, inspect their providerID fields.

Query providerID from the management cluster Machine:

kubectl get machine -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.providerID}{"\n"}{end}'

Query providerID from the workload cluster Node:

kubectl --kubeconfig ./capi-quickstart.kubeconfig \
  get nodes -o custom-columns=NAME:.metadata.name,PROVIDER_ID:.spec.providerID

Output:

NAME                                   PROVIDER_ID
capi-quickstart-control-plane-9x2lm   docker:////capi-quickstart-control-plane-9x2lm
capi-quickstart-md-0-v4l2k            docker:////capi-quickstart-md-0-v4l2k

Matching providerID strings establish the binding bridge across management and workload control planes.

10. End-to-End Trace: Tracking a Single Worker Node

To consolidate this workflow, trace the full dependency chain of a single worker node from top-level declaration down to host container execution:

flowchart TD
    MD["1. MachineDeployment\n(Spec: Replicas=1)"] -->|Creates| MS["2. MachineSet\n(Spec: Replicas=1)"]
    MS -->|Instantiates| M["3. Machine\n(Spec: v1.37.0)"]

    M -->|spec.bootstrap.configRef| BC["4a. KubeadmConfig"]
    M -->|spec.infrastructureRef| DM["4b. InfrastructureMachine\n(DevMachine)"]

    BC -->|CAPBK Generates Secret| SEC["5. Bootstrap Secret\n(kubeadm join script)"]
    SEC -. Injected into .-> DM

    DM -->|CAPD Issues API Request| DOCKER["6. Host Docker Daemon"]
    DOCKER -->|Launches Container| CONT["7. Container Instance"]

    CONT -->|Executes kubeadm join| APISERVER["8. Workload API Server"]
    APISERVER -->|Registers Node| NODE["9. Workload Node (Ready)"]

11. Diagnostic Playbook: Isolating Failure Boundaries

When provisioning fails, avoid inspecting arbitrary log streams. Use status conditions to isolate the responsible controller boundary:

flowchart TD
    START["Cluster Provisioning Issue"] --> C1{"kubectl get cluster<br/>status.infrastructureReady == true?"}
    
    C1 -- No --> F1["Check CAPD Controller Logs<br/>kubectl logs -n capd-system deployment/capd-controller-manager"]
    C1 -- Yes --> C2{"kubectl get kcp<br/>status.ready == true?"}

    C2 -- No --> F2["Check Control Plane Provider Logs<br/>kubectl logs -n capi-kubeadm-control-plane-system ..."]
    C2 -- Yes --> C3{"kubectl get kubeadmconfig<br/>status.ready == true?"}

    C3 -- No --> F3["Check Bootstrap Provider Logs<br/>kubectl logs -n capi-kubeadm-bootstrap-system ..."]
    C3 -- Yes --> C4{"kubectl get devmachine<br/>status.ready == true?"}

    C4 -- No --> F4["Check Docker Socket & Host Resource Allocation (OOM/RAM)"]
    C4 -- Yes --> F5["Check Workload Cluster CNI Plugin & Kubelet Logs"]

Controller Responsibility Matrix

12. Lifecycle Cleanup & Teardown Protocol

Deleting a Cluster API workload cluster requires adhering to proper deletion ordering.

Deleting the management cluster directly (e.g. executing kind delete cluster) without deleting the workload cluster first leaves orphaned containers or cloud VMs running on the infrastructure provider.

The correct teardown sequence initiates cascading deletion via Kubernetes finalizers:

kubectl delete cluster capi-quickstart

Cascading Deletion Workflow

flowchart TD
    DEL["kubectl delete cluster capi-quickstart"] --> FIN["CAPI Finalizers Triggered"]
    FIN --> W_DRAIN["1. Worker Nodes Cordoned & Drained"]
    W_DRAIN --> W_TERM["2. Worker Docker Containers Terminated"]
    W_TERM --> CP_DRAIN["3. Control Plane Nodes Drained"]
    CP_DRAIN --> CP_TERM["4. Control Plane Containers Terminated"]
    CP_TERM --> INFRA_CLEAN["5. Docker Network Infrastructure Removed"]
    INFRA_CLEAN --> GONE["6. Cluster Object Removed from etcd"]

Monitor cluster deletion progress:

kubectl get clusters

Once kubectl get clusters returns no resources, safely delete the host management cluster:

kind delete cluster --name capi-management

13. Key Learning Takeaways

• Socket-Mounted Management: Local CAPD labs require mounting /var/run/docker.sock into the management cluster to allow node container provisioning.
• clusterctl init vs clusterctl generate: init installs long-running background controller managers into the management cluster; generate is a client-side CLI template expander.
• etcd as Single Source of Truth: Cluster API maintains state directly inside Kubernetes etcd custom resources. No external database exists.
• Decoupled Asynchronous Handoff: Core Machine controllers, Bootstrap providers, and Infrastructure providers coordinate exclusively via Kubernetes secrets and custom resource status fields.
• Node Readiness Depends on CNI: CAPI manages machine provisioning and node registration, but workload nodes remain NotReady until a CNI plugin is deployed to the workload cluster.
• Cascading Finalizer Teardown: Deleting a Cluster CR triggers finalizer loops that safely drain nodes and tear down underlying infrastructure before removing API records.

14. Self-Check Questions

1. Why does executing kind create cluster without extra mounts prevent CAPD from provisioning workload clusters?
2. What is the specific role of cert-manager inside a CAPI management cluster during clusterctl init?
3. If a worker node container launches in Docker but the Node status inside the workload cluster remains NotReady, which component is missing?
4. Why must you delete the workload cluster resource (kubectl delete cluster <name>) before deleting the management cluster (kind delete cluster)?

Next Milestone in the Learning Journey

In Part 05: clusterctl Explained: Managing the Cluster API Management Plane, we will dissect the internal mechanics of clusterctl: repository discovery, provider contract validation, template rendering algorithms, and state movement operations (clusterctl move).

References & Further Reading

• Cluster API Book: Quick Start Guide
• Cluster API Provider Docker (CAPD) Documentation
• Cluster API Book: clusterctl Command Reference
• Kubernetes KinD: Extra Mounts Specification