Anatomy of Cluster Creation: What Actually Happens When You Create a Cluster?

Learning Journey: Part 06 of 16

Module Goal: Trace the end-to-end lifecycle of cluster creation across management controllers, understanding how independent control loops orchestrate shared infrastructure, bootstrap data, VM provisioning, providerID binding, and workload node registration.
Target Spec: Cluster API v1.14 (v1beta2 APIs).
Prerequisites: Part 03: Cluster API Object Model: Mapping the Custom Resource Graph and Part 05: clusterctl Explained: Managing the Cluster API Management Plane.

1. First-Principles Problem Framing: The Illusion of the Single “Create” Command

In traditional infrastructure management, provisioning a Kubernetes cluster is treated as an imperative, procedural pipeline:

Run Terraform / Ansible / shell script
   ↓ (step 1: create VPC and subnets)
   ↓ (step 2: launch VMs)
   ↓ (step 3: run kubeadm init via SSH)
   ↓ (step 4: retrieve kubeconfig)
   ↓ (step 5: run kubeadm join on workers)
   ↓ (step 6: exit 0)

If step 3 fails due to a network glitch, the procedural script terminates with a non-zero exit code, leaving orphaned VMs or half-configured networks behind. Re-running the script often fails because intermediate resources already exist.

When you begin using Cluster API (CAPI), you initiate cluster creation with a standard Kubernetes command:

kubectl apply -f cluster.yaml

A few minutes later, querying the management cluster reveals a fully realized, multi-tier custom resource tree:

Cluster
KubeadmControlPlane
MachineDeployment
MachineSet
Machines
Nodes

It is tempting to adopt a flawed mental model:

“There is a monolithic ‘Cluster Creation Engine’ in CAPI that reads cluster.yaml and executes an end-to-end provisioning workflow from top to bottom.”

There is no monolithic cluster creation engine.

There is no single controller that owns the cluster creation process. Instead, multiple independent controllers run concurrently in the management cluster. Each controller watches a strictly bounded slice of the Kubernetes API, reconciles its own state machine, updates status conditions, and exits its reconciliation loop.

The running workload cluster emerges not from a top-down procedure, but from the eventual convergence of independent, asynchronous control loops.

2. Mental Model & Architectural Shifts: Asynchronous Choreography vs. Monolithic Orchestration

To understand what actually happens when you create a cluster, contrast orchestration with choreography:

• Orchestration (Procedural / Scripted): A central coordinator tells each subsystem what to do, waits for the response, and then executes the next instruction.
• Choreography (Declarative / Controller-Driven): Independent actors react to changes in shared state (etcd). When Actor A completes its work and publishes a status condition, Actor B observes the update and begins its own reconciliation.

sequenceDiagram
    autonumber
    actor Engineer as Platform Engineer
    participant API as Management API Server (etcd)
    participant CC as Cluster Controller
    participant InfraC as InfraCluster Controller
    participant KCP as KubeadmControlPlane Controller
    participant MC as Machine Controller
    participant BootC as Bootstrap Controller
    participant InfraM as InfraMachine Controller
    participant Workload as Workload Node / Kubelet

    Engineer->>API: kubectl apply -f cluster.yaml
    Note over API: Applied Intent: Cluster, KCP, MachineDeployment
    
    CC->>API: Watches Cluster, adds finalizers, waits for Infra
    InfraC->>API: Watches InfraCluster, provisions VPC/LB, sets controlPlaneEndpoint
    API-->>CC: controlPlaneEndpoint available
    
    KCP->>API: Watches KCP, creates initial control-plane Machine
    MC->>API: Watches Machine, coordinates bootstrapRef & infrastructureRef
    
    BootC->>API: Reconciles KubeadmConfig, generates Secret (cloud-init)
    InfraM->>API: Reconciles InfraMachine, consumes Secret, provisions VM
    
    Workload->>Workload: Host boots, executes kubeadm init
    Workload->>API: Node registers, InfraMachine sets spec.providerID
    MC->>API: Matches Machine.spec.providerID == Node.spec.providerID
    KCP->>API: Control plane reports Initialized & Available
    CC->>API: Cluster status.ready = true

Applied Intent vs. Derived Objects

When you examine the YAML rendered by clusterctl generate cluster or your GitOps repository, notice the disparity between what you apply and what eventually runs:

flowchart TD
    subgraph AppliedManifest["Applied High-Level Intent (cluster.yaml)"]
        C["Cluster"]
        IC["InfrastructureCluster (e.g. AWSCluster / DevCluster)"]
        KCP["KubeadmControlPlane"]
        MD["MachineDeployment"]
        IMT["InfrastructureMachineTemplate"]
        BCT["KubeadmConfigTemplate"]
    end

    subgraph DerivedRuntime["Derived Objects (Instantiated Dynamically by Controllers)"]
        MS["MachineSet (Created by MD Controller)"]
        M_CP["Machine: Control Plane (Created by KCP Controller)"]
        M_W["Machine: Worker (Created by MS Controller)"]
        BC["KubeadmConfig (Cloned from BCT)"]
        IM["InfrastructureMachine (Cloned from IMT)"]
        SEC["Secret: Bootstrap Payload (Rendered by Bootstrap Controller)"]
        NODE["Node (Registers in Workload Cluster via Kubelet)"]
    end

    C -. References .-> IC
    C -. References .-> KCP
    MD -. References .-> IMT
    MD -. References .-> BCT

    MD ==>|Reconciles| MS
    MS ==>|Reconciles| M_W
    KCP ==>|Reconciles| M_CP

    M_CP -->|Coordinates| BC
    M_CP -->|Coordinates| IM
    M_W -->|Coordinates| BC
    M_W -->|Coordinates| IM

    BC ==>|Generates| SEC
    SEC -. Injected into .-> IM
    IM ==>|Boots VM/Container| NODE

The initial YAML does not contain a single Machine, MachineSet, KubeadmConfig, or InfrastructureMachine. Those resources are derived at runtime by controllers reacting to the higher-level blueprints.

3. The Resource Graph & Controller Responsibility Matrix

Before tracing the execution lifecycle, map the controllers responsible for each stage of cluster creation:

Resource Generation Timeline

Controller Responsibility Matrix

4. End-to-End Walkthrough: The 6 Lifecycle Phases of Cluster Creation

When kubectl apply -f cluster.yaml executes, cluster creation unfolds across six discrete, chronological phases.

flowchart LR
    P1["Phase 1<br/>Root Cluster & Shared Infra"] --> P2["Phase 2<br/>Control Plane & Bootstrap Secret"]
    P2 --> P3["Phase 3<br/>Host Provisioning & Boot"]
    P3 --> P4["Phase 4<br/>providerID Binding & CP Init"]
    P4 --> P5["Phase 5<br/>Worker Fleet Expansion"]
    P5 --> P6["Phase 6<br/>In-Cluster CNI & Ready State"]

Phase 1: Cluster Root & Shared Infrastructure Reconciliation

The entry point of the entire hierarchy is the Cluster custom resource:

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: production-cluster
  namespace: default
spec:
  clusterNetwork:
    services:
      cidrBlocks: ["10.96.0.0/12"]
    pods:
      cidrBlocks: ["192.168.0.0/16"]
    serviceDomain: "cluster.local"
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: AWSCluster # or DevCluster / DockerCluster
    name: production-cluster-infra
  controlPlaneRef:
    apiGroup: controlplane.cluster.x-k8s.io
    kind: KubeadmControlPlane
    name: production-cluster-control-plane

flowchart TD
    C["Cluster CR Applied"] --> CC["Cluster Controller"]
    CC --> FIN["1. Adds finalizers to Cluster CR"]
    CC --> OWN["2. Sets ownerReferences on InfrastructureCluster"]
    CC --> WAIT["3. Waits for InfrastructureCluster status.ready == true"]

    IC["InfrastructureCluster CR"] --> IPC["Infra Controller (e.g. CAPA/CAPD)"]
    IPC --> NET["Allocates VPC, Subnets, Gateways, Security Groups"]
    IPC --> LB["Provisions Control Plane Load Balancer"]
    IPC --> EP["Writes spec.controlPlaneEndpoint into InfraCluster status"]
    
    EP --> SYNC["Cluster Controller copies Endpoint into Cluster.spec"]

What Happens Internally:

1. Finalizer Registration: The Core Cluster Controller adds cluster.cluster.x-k8s.io to metadata.finalizers to guarantee that external cloud infrastructure cannot be orphaned if the object is accidentally deleted.
2. Ownership Linking: The controller injects metadata.ownerReferences pointing from InfrastructureCluster to Cluster.
3. Shared Infrastructure Provisioning: The infrastructure provider (e.g. AWS, Azure, GCP, or Docker) reconciles the InfrastructureCluster. It provisions shared networking components: VPCs, subnets, routing tables, security group firewall rules, and the control-plane load balancer.
4. Control Plane Endpoint Discovery: As soon as the load balancer or virtual IP is provisioned, the provider writes the control-plane endpoint to status.controlPlaneEndpoint:

   status:
     ready: true
     controlPlaneEndpoint:
       host: "production-cluster-apiserver-12345.us-east-1.elb.amazonaws.com"
       port: 6443
   

5. Upward Propagation: The Core Cluster Controller observes status.ready == true, copies controlPlaneEndpoint onto Cluster.spec.controlPlaneEndpoint, and unblocks downstream control plane controllers.

Phase 2: Control Plane Initialization & Bootstrap Secret Generation

With the control plane endpoint established, the KubeadmControlPlane (KCP) controller begins provisioning the control plane:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlane
metadata:
  name: production-cluster-control-plane
spec:
  replicas: 1
  version: v1.37.0
  machineTemplate:
    infrastructureRef:
      apiGroup: infrastructure.cluster.x-k8s.io
      kind: AWSMachineTemplate
      name: production-cluster-control-plane-template
  kubeadmConfigSpec:
    clusterConfiguration:
      controllerManager:
        extraArgs:
          cluster-name: production-cluster
    initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          cloud-provider: external

flowchart TD
    KCP["KubeadmControlPlane Controller"] --> CHECK{"Control Plane<br/>Machines exist?"}
    CHECK -- No --> GEN_CERT["1. Generates PKI Certificates & CA Secrets"]
    GEN_CERT --> INST_M["2. Instantiates first control-plane Machine CR"]
    
    INST_M --> GEN_BC["3. Clones KubeadmConfig from spec"]
    INST_M --> GEN_IM["4. Clones InfraMachine from template"]

    GEN_BC --> BC_CTRL["Bootstrap Controller (CAPBK)"]
    BC_CTRL --> SCRIPT["Renders cloud-init / ignition payload with kubeadm init"]
    SCRIPT --> SEC["Writes payload to Kubernetes Secret: <cluster>-ca-bootstrap"]
    SEC --> BC_READY["KubeadmConfig status.ready = true"]

What Happens Internally:

1. Certificate Authority Initialization: If external certificates are not provided, KCP generates the cluster CA keys and certificates, storing them in Kubernetes Secrets inside the management cluster (<cluster>-ca, <cluster>-etcd-ca, <cluster>-sa).
2. First Machine Instantiation: Because replicas: 1 and zero machines exist, KCP creates the first control-plane Machine custom resource:

   apiVersion: cluster.x-k8s.io/v1beta2
   kind: Machine
   metadata:
     name: production-cluster-control-plane-7h9xk
     labels:
       cluster.x-k8s.io/cluster-name: production-cluster
       cluster.x-k8s.io/control-plane: ""
   spec:
     clusterName: production-cluster
     version: v1.37.0
     bootstrap:
       configRef:
         kind: KubeadmConfig
         name: production-cluster-control-plane-7h9xk
     infrastructureRef:
       kind: AWSMachine
       name: production-cluster-control-plane-7h9xk
   

3. Cloning Concrete Objects: KCP does not link machines directly to templates. It creates a concrete KubeadmConfig and a concrete AWSMachine (or DevMachine), setting owner references directly to the new Machine.
4. Bootstrap Payload Rendering: The Kubeadm Bootstrap Controller (capi-kubeadm-bootstrap-controller-manager) reconciles the KubeadmConfig. It combines the cluster CA certificates, cloud-init templates, and kubeadm init parameters into a complete bootstrap script.
5. Secret Storage: The script is written to a standard Kubernetes Secret (<machine-name>-bootstrap-data). The bootstrap controller sets:

   status:
     ready: true
     dataSecretName: production-cluster-control-plane-7h9xk-bootstrap-data
   

Phase 3: Infrastructure Machine Provisioning & Host Boot

Now, the Infrastructure Provider Controller reconciles the concrete InfrastructureMachine:

flowchart TD
    IM["InfraMachine CR (e.g. AWSMachine / DevMachine)"] --> IPC["Infrastructure Provider Controller"]
    IPC --> READ_SEC["1. Reads Bootstrap Secret from management etcd"]
    READ_SEC --> API_REQ["2. Calls Cloud API (EC2 / Docker / vSphere)"]
    API_REQ --> VM["3. Launches Compute Instance with Bootstrap as User-Data"]
    
    VM --> BOOT["4. Host Operating System Boots"]
    BOOT --> USER_DATA["5. cloud-init consumes bootstrap payload"]
    USER_DATA --> RUN_KUBEADM["6. Executes kubeadm init --config kubeadm.yaml"]
    RUN_KUBEADM --> CP_PODS["7. Starts etcd, apiserver, controller-manager, scheduler"]
    CP_PODS --> KUBELET["8. kubelet initializes on host"]

What Happens Internally:

1. Asynchronous Hand-off via Secrets: The Infrastructure Controller does not call the Bootstrap Controller directly. It waits until Machine.spec.bootstrap.dataSecretName is populated, then reads the Secret from the management cluster etcd.
2. Cloud API Invocation: The controller encodes the bootstrap script into the provider’s native format:
   • For AWS: Passed to the EC2 API as UserData (MIME multipart cloud-init).
   • For vSphere: Injected via guestinfo parameters.
   • For Docker (CAPD): Injected into the container filesystem before starting systemd.
3. Instance Launch: The cloud VM or local container is created.
4. Host Boot & Execution: As the operating system finishes kernel initialization, cloud-init runs. It unpacks the certificates generated in Phase 2, writes /etc/kubernetes/admin.conf, and executes:

   kubeadm init --config /etc/kubernetes/kubeadm-init.yaml
   

5. Static Pod Creation: kubeadm generates static pod manifests in /etc/kubernetes/manifests/. The host kubelet brings up kube-apiserver, etcd, kube-controller-manager, and kube-scheduler.

Phase 4: ProviderID Binding & Workload Node Registration

At this stage, an instance is running in the cloud, but the management cluster does not yet know how to connect its Machine record with the new virtual machine.

flowchart TD
    subgraph CloudLayer["Infrastructure Layer"]
        VM["Cloud Instance Running (Instance ID: i-0abc123)"]
    end

    subgraph MgmtCluster["Management Cluster"]
        IM["InfraMachine CR"]
        M["Machine CR"]
        MC["Machine Controller"]
    end

    subgraph WorkloadCluster["Workload Cluster"]
        KUBELET["Host Kubelet"]
        W_API["Workload API Server"]
        NODE["Node: ip-10-0-1-50"]
    end

    VM -->|EC2 Reports Metadata| IM
    IM -->|1. Writes spec.providerID: aws:///us-east-1a/i-0abc123| IM
    IM -->|2. Copies to Machine| M

    KUBELET -->|3. Registers Node| W_API
    W_API -->|4. Creates Node with matching providerID| NODE

    MC -->|5. Queries Workload API Server| NODE
    MC -->|6. Matches Machine.spec.providerID == Node.spec.providerID| M
    M -->|7. Sets status.nodeRef to Node name| M
    M -->|8. Sets status.phase = Running| M

What Happens Internally:

1. Assigning providerID to InfraMachine: Once the VM launches, the infrastructure controller reads the unique cloud instance ID and writes it to InfraMachine.spec.providerID:

   spec:
     providerID: "aws:///us-east-1a/i-0abc12345def6789a"
   

2. Assigning providerID to Machine: The Core Machine Controller copies this value into Machine.spec.providerID.
3. Workload Node Registration: Inside the workload VM, kubelet registers itself with the newly initialized workload API server. Because the cloud provider is enabled, kubelet populates its own Node.spec.providerID:

   apiVersion: v1
   kind: Node
   metadata:
     name: ip-10-0-1-50.ec2.internal
   spec:
     providerID: "aws:///us-east-1a/i-0abc12345def6789a"
   

4. The providerID Matching Bridge: The Core Machine Controller connects to the workload cluster API server (using the cluster credentials stored in <cluster>-kubeconfig). It iterates through workload Node objects looking for a match:

   Machine.spec.providerID == Node.spec.providerID
   

5. Establishing status.nodeRef: Once matched, the Machine Controller writes status.nodeRef:

   status:
     nodeRef:
       name: ip-10-0-1-50.ec2.internal
     phase: Running
   

6. Control Plane Initialization: The KCP controller verifies that the workload API server is responding and that etcd quorum is healthy. It updates its own status:

   status:
     initialized: true
     ready: true
   

Core Architectural Insight: Initialized vs. Available

• Initialized = true: A functional workload Kubernetes API server exists and etcd is answering requests. Worker nodes can now join the cluster.
• Available = true: The desired control-plane replica count is fully reached and all control-plane machines are passing health checks. For a 3-replica control plane, the cluster is Initialized when machine 1 succeeds, but only Available when all 3 machines have joined and synchronized etcd.

Phase 5: Worker Fleet Expansion & MachineDeployment Hierarchy

With the control plane initialized, worker node provisioning is unblocked. Worker nodes follow a three-tier lifecycle mirroring Kubernetes application deployments:

flowchart TD
    MD["MachineDeployment<br/>spec.replicas: 2"] --> MDC["MachineDeployment Controller"]
    MDC --> MS["MachineSet<br/>spec.replicas: 2"]
    
    MS --> MSC["MachineSet Controller"]
    MSC --> M1["Machine: worker-01"]
    MSC --> M2["Machine: worker-02"]

    M1 --> BCT1["Clones KubeadmConfig<br/>(from KubeadmConfigTemplate)"]
    M1 --> IMT1["Clones InfraMachine<br/>(from InfraMachineTemplate)"]

    BCT1 --> BOOT_W["Bootstrap Controller:<br/>Generates kubeadm join token Secret"]
    BOOT_W --> SEC_W["Secret: worker-join-script"]

    SEC_W --> INFRA_W["Infra Controller provisions VM"]
    INFRA_W --> JOIN["Host boots & executes kubeadm join"]
    JOIN --> WORKLOAD_NODE["Registers as worker Node in Workload Cluster"]

What Happens Internally:

1. MachineSet Creation: The MachineDeployment controller observes replicas: 2. It computes a template hash and creates a MachineSet custom resource.
2. Machine Instantiation: The MachineSet controller instantiates two worker Machine objects.
3. Template Cloning: Unlike the control plane (which derives configuration from KubeadmControlPlane), workers clone their configuration from:
   • KubeadmConfigTemplate -> KubeadmConfig
   • InfrastructureMachineTemplate -> InfrastructureMachine (e.g. AWSMachineTemplate -> AWSMachine)
4. Join Script Generation: The Bootstrap Controller reconciles the worker’s KubeadmConfig. Because the control plane is already Initialized, the bootstrap controller retrieves a join token from the workload cluster, renders a kubeadm join payload, and stores it in a Secret.
5. VM Provisioning & Joining: The infrastructure provider boots the worker instance with the join script. The worker executes:

   kubeadm join production-cluster-apiserver:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>
   

6. Matching Worker Nodes: When the worker kubelet registers, the Machine Controller binds the worker Machine to the workload Node via providerID.

Phase 6: In-Cluster Network Convergence (The CNI Boundary)

At this point, querying both management and workload clusters reveals an apparent contradiction:

Management Cluster:

kubectl get machines

NAME                                       PHASE     NODENAME
production-cluster-control-plane-7h9xk     Running   ip-10-0-1-50.ec2.internal
production-cluster-md-0-v4l2k              Running   ip-10-0-2-120.ec2.internal

Workload Cluster:

kubectl --kubeconfig ./workload.kubeconfig get nodes

NAME                         STATUS     ROLES           AGE   VERSION
ip-10-0-1-50.ec2.internal   NotReady   control-plane   6m    v1.37.0
ip-10-0-2-120.ec2.internal  NotReady   <none>          3m    v1.37.0

flowchart LR
    subgraph ManagementPlane["Management Cluster State"]
        M["Machine CR<br/>Phase: Running<br/>status.nodeRef: Set"]
    end

    subgraph WorkloadPlane["Workload Cluster State"]
        N["Node Object<br/>Status: NotReady<br/>NetworkUnavailable: True"]
    end

    subgraph MissingPiece["Network Boundary"]
        CNI["Container Network Interface (CNI)<br/>(Calico / Cilium / Flannel)"]
    end

    M -.->|providerID Matched| N
    CNI ==>|Deployed to Workload Cluster| N
    N -->|Network Plugin Initialized| READY["Node Status: Ready"]

Why are Nodes NotReady when Machines are Running?

• Machine Running means: The infrastructure exists, the bootstrap script executed, kubelet started, and the Machine record is successfully paired with a Node via providerID.
• Node NotReady means: Kubelet has evaluated its local node conditions. Because Cluster API deliberately does not install a Container Network Interface (CNI) by default, the node reports NetworkPluginNotReady (NetworkReady=False).

Cluster API manages the lifecycle of the machines and control plane, but respects the operational boundary of in-cluster software. Once you deploy a CNI plugin (such as Calico or Cilium) to the workload cluster, kubelet detects the CNI binary and converts node status to Ready.

kubectl --kubeconfig ./workload.kubeconfig \
  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

Once the CNI pods start, all nodes transition to Ready:

NAME                         STATUS   ROLES           AGE   VERSION
ip-10-0-1-50.ec2.internal   Ready    control-plane   8m    v1.37.0
ip-10-0-2-120.ec2.internal  Ready    <none>          5m    v1.37.0

5. Status Propagation, Conditions, and Eventual Consistency

Because Cluster API is distributed and asynchronous, objects update at different speeds. State updates propagate strictly upward through the object tree:

flowchart BT
    subgraph Compute["Infrastructure Layer"]
        IM["InfrastructureMachine"]
    end

    subgraph MachineLayer["Machine Layer"]
        M["Machine"]
        MS["MachineSet"]
        MD["MachineDeployment"]
    end

    subgraph ClusterLayer["Cluster Layer"]
        IC["InfrastructureCluster"]
        KCP["KubeadmControlPlane"]
        C["Cluster"]
    end

    IM -->|status.ready & providerID| M
    M -->|status.phase & nodeRef| MS
    MS -->|status.readyReplicas| MD

    IC -->|status.controlPlaneEndpoint| C
    KCP -->|status.initialized & status.ready| C
    MD -->|Replica status| C

Temporary Discrepancies During Convergence

During cluster creation, you will routinely observe transient states:

1. InfrastructureMachine Exists, Machine Still Pending: The cloud VM is running, but the infrastructure provider has not yet finished writing spec.providerID back to the management API.
2. Node Registered, Machine Not Running: The workload API server has accepted the node registration, but the Machine Controller has not yet executed its poll loop to bind spec.providerID.
3. Control Plane Initialized, Cluster Not Ready: KubeadmControlPlane has marked status.initialized = true, but the Core Cluster Controller is waiting for workers or infrastructure conditions before reporting full cluster readiness.

These transient states are not errors. They are normal characteristics of an eventually consistent distributed system.

6. Diagnostic Playbook: Tracing Creation Failures Across Controller Boundaries

When cluster creation stalls, running kubectl describe cluster <name> only shows high-level symptoms. To diagnose the root cause, follow the condition hierarchy down the tree:

flowchart TD
    START["Cluster Creation Stalled"] --> C1{"kubectl get cluster<br/>status.infrastructureReady == true?"}

    C1 -- No --> F1["1. Inspect InfraCluster<br/>kubectl describe <infracluster> <name><br/>Check Cloud Quotas, VPC limits, Security Groups"]
    C1 -- Yes --> C2{"kubectl get kcp<br/>status.initialized == true?"}

    C2 -- No --> F2{"Check Control-Plane Machine<br/>kubectl get machines -l cluster.x-k8s.io/control-plane<br/>Phase = Running?"}
    
    F2 -- No --> F3{"Check KubeadmConfig & Bootstrap Secret<br/>kubectl get kubeadmconfig<br/>status.ready == true?"}
    F3 -- No --> LOG_BOOT["Inspect Bootstrap Controller Logs:<br/>kubectl logs -n capi-kubeadm-bootstrap-system ..."]
    F3 -- Yes --> LOG_INFRA["Inspect Infra Provider Logs (VM creation / Quota error):<br/>kubectl logs -n <infra-system> ..."]

    F2 -- Yes --> F4["Check Workload API Server & kubeadm init logs<br/>SSH into Control Plane host / check cloud-init-output.log"]

    C2 -- Yes --> C3{"kubectl get machinedeployment<br/>status.readyReplicas == desired?"}
    C3 -- No --> F5["Inspect MachineSet & Worker Machine conditions<br/>Check join token validity, subnet routing, or CNI plugin"]
    C3 -- Yes --> OK["Cluster Fully Functional"]

Common Failure Scenarios and Investigation Steps

Real-Time Inspection Commands

To observe the controllers actively constructing the object graph, open three terminals:

Terminal 1 (High-Level State Tree):

clusterctl describe cluster production-cluster --show-conditions all

Terminal 2 (Multi-Resource Watch):

watch -n 2 \
  'kubectl get clusters,kubeadmcontrolplanes,machinedeployments,machinesets,machines'

Terminal 3 (Dynamic Provider Resource Discovery):

kubectl api-resources --api-group infrastructure.cluster.x-k8s.io

7. Post-Creation Reality: Continuous Reconciliation & Lifecycle State

Cluster creation is not a one-time provisioning script that exits. Once all objects converge to Ready:

1. Continuous Drift Correction: If someone deletes an EC2 instance in the AWS console, the infrastructure provider detects the missing VM, marks the InfrastructureMachine unhealthy, and the MachineSet controller deploys a replacement instance.
2. Rolling Upgrades: If you update spec.version: v1.38.0 on the KubeadmControlPlane or MachineDeployment, the respective controllers orchestrate an automated rolling upgrade without downtime.
3. Cascading Deletion: If you execute kubectl delete cluster production-cluster, the deletion follows the exact reverse of the creation sequence: worker nodes are drained and deleted first, followed by control plane nodes, and finally shared infrastructure (VPCs and load balancers) is torn down.

Cluster creation is simply the initial convergence point of a system that runs forever.

8. Key Learning Takeaways

• Choreography Over Orchestration: There is no master script or single creation engine. Creation is the convergence of independent, asynchronous controllers watching etcd.
• Applied vs. Derived Objects: You apply high-level abstractions (Cluster, KCP, MachineDeployment, and templates); controllers derive concrete operational resources (MachineSet, Machine, KubeadmConfig, InfrastructureMachine).
• The Secret as the Provider Contract: Bootstrap providers and Infrastructure providers coordinate asynchronously through Kubernetes Secrets containing initialization scripts (cloud-init or Ignition).
• Identity Bridge via providerID: A Machine (management cluster) and a Node (workload cluster) are completely independent API objects paired together by matching spec.providerID.
• Initialized vs. Available: A control plane is Initialized when its first API server responds, but only Available when its desired replica quorum is healthy.
• The CNI Boundary: CAPI manages compute instances and control plane registration. Workload nodes remain NotReady until a CNI plugin is installed on the workload cluster.

9. Self-Check Questions

Before moving to Part 07, test your comprehension of the creation lifecycle:

1. Why does the initial YAML generated by clusterctl generate cluster contain InfrastructureMachineTemplate instead of InfrastructureMachine objects?
2. If an EC2 instance launches successfully and kubelet starts, but the Machine object in the management cluster remains stuck in Provisioning, which specific field has the infrastructure controller failed to report?
3. What is the architectural reason for storing the output of the Kubeadm Bootstrap Controller in a Kubernetes Secret rather than passing it via a direct Go function call to the Infrastructure Controller?
4. If all worker nodes in the workload cluster report status NotReady, what command and check should you perform before assuming Cluster API failed?
5. How does the KubeadmControlPlane controller determine whether to run kubeadm init versus kubeadm join --control-plane on a newly provisioned control-plane machine?

Next Milestone in the Learning Journey

In Part 07: Machine, MachineSet, and MachineDeployment: Worker Node Hierarchy Internals, we will dive into the worker node hierarchy: template cloning mechanisms, rolling upgrade surge strategies, immutable infrastructure replacement, and how worker fleets self-heal when host instances fail.

References & Further Reading

• Cluster API Book: The Cluster Controller
• Cluster API Book: Machine Controller Implementation
• Cluster API Book: Kubeadm Control Plane Controller
• Cluster API Book: Provider Contracts Overview
• Cluster API Book: BootstrapConfig Provider Contract
• Cluster API Book: ControlPlane Provider Contract
• Kubernetes Documentation: Creating a Cluster with kubeadm