Bootstrap Clusters, Management Clusters, and clusterctl move

Cluster API introduces a fundamental chicken-and-egg paradox:

If Kubernetes manages Kubernetes clusters declaratively through Custom Resource Definitions and controllers, what cluster runs the controllers that create the very first management cluster?

You cannot provision a high-availability, production-grade management cluster in AWS, Azure, or GCP without an active Kubernetes control plane running CAPI controllers (capi-controller-manager, capa-controller-manager, etc.). Yet spinning up that permanent cloud cluster requires an operational API server.

Cluster API resolves this paradox through the Bootstrap and Pivot pattern.

An operator creates a temporary, ephemeral bootstrap cluster (typically running locally via KinD). That bootstrap cluster provisions the permanent management cluster in the target cloud. Once the cloud cluster is healthy, Cluster API executes a pivot: transferring the entire Custom Resource graph and reconciliation authority from the ephemeral cluster to the permanent cluster using clusterctl move. Finally, the ephemeral bootstrap cluster is destroyed.

1. The Bootstrap Cluster vs. Management Cluster Separation

To master the pivot workflow, contrast the three types of clusters in a Cluster API ecosystem:

2. The Architectural Mental Model: What Moves During a Pivot?

A common point of operational confusion is misunderstanding what clusterctl move actually migrates:

flowchart TD
    subgraph PhaseBefore["Phase A: Before Pivot"]
        BC_A["Bootstrap Cluster (KinD on Laptop/CI)"]
        MC_A["Permanent Cloud Management Cluster<br/>(VMs, VPC, Load Balancer in AWS/Azure)"]
        
        BC_A -->|"Hosts CAPI CRD Graph & Reconciles"| MC_A
    end

    subgraph PhasePivot["Phase B: clusterctl move"]
        TRANS["clusterctl move transfers:<br/>• Cluster, KCP, MachineDeployments<br/>• Provider CRDs (AWSCluster, AWSMachine)<br/>• Secrets (Certificates, Tokens, Credentials)"]
    end

    subgraph PhaseAfter["Phase C: After Pivot"]
        BC_C["Bootstrap Cluster<br/>(Deleted via kind delete)"]
        MC_C["Permanent Cloud Management Cluster<br/>(Self-Managing or Managing Fleet)"]
        WC_C["Workload Clusters"]

        MC_C -->|"Reconciles its own nodes & workload fleet"| WC_C
    end

    PhaseBefore --> PhasePivot
    PhasePivot --> PhaseAfter

The Fundamental Rule: clusterctl move never moves physical or cloud infrastructure. The virtual machines, networks, storage volumes, and control plane endpoints remain running untouched in the cloud. What moves is the declarative management state: the etcd custom resource records, owner references, and reconciliation authority.

3. The 8-Stage Bootstrap and Pivot Lifecycle

Transitioning from an empty workstation to a production management cluster follows an eight-stage sequence:

flowchart TD
    S1["1. Create Ephemeral KinD Cluster<br/>(kind create cluster --name capi-bootstrap)"] --> S2["2. Initialize CAPI Providers on KinD<br/>(clusterctl init --infrastructure aws)"]
    S2 --> S3["3. Apply Cloud Management Cluster Manifest<br/>(kubectl apply -f management-cluster.yaml)"]
    S3 --> S4["4. Await Infrastructure & Control Plane Stability<br/>(clusterctl describe cluster ...)"]
    S4 --> S5["5. Extract Target Admin Kubeconfig<br/>(clusterctl get kubeconfig ...)"]
    S5 --> S6["6. Initialize CAPI Providers on Target Cluster<br/>(clusterctl init --kubeconfig target.kubeconfig)"]
    S6 --> S7["7. Execute clusterctl move<br/>(Transfers CRD graph & pauses/resumes reconciliation)"]
    S7 --> S8["8. Teardown Ephemeral KinD Cluster<br/>(kind delete cluster --name capi-bootstrap)"]

Detailed Execution Walkthrough

Stage 1: Ephemeral Bootstrap Cluster

Spin up a local containerized Kubernetes cluster:

kind create cluster --name capi-bootstrap

Stage 2: Initialize Bootstrap Providers

Install the Core, Bootstrap, Control Plane, and Infrastructure provider controllers onto the KinD cluster:

clusterctl init --infrastructure aws

Stage 3: Provision the Future Management Cluster

Generate and apply the manifest describing the target management cluster. The target cluster must include worker nodes (or untainted control plane nodes) to host the CAPI controller manager pods after the move:

clusterctl generate cluster prod-mgmt \
  --kubernetes-version v1.31.0 \
  --control-plane-machine-count 3 \
  --worker-machine-count 2 \
  > prod-mgmt.yaml

kubectl apply -f prod-mgmt.yaml

Stage 4: Verify Stability

Wait until the control plane reports Initialized = True, all machines report Running, and nodes report Ready = True:

clusterctl describe cluster prod-mgmt

Stage 5: Extract Target Kubeconfig

Retrieve the admin credentials for the newly created cloud cluster:

clusterctl get kubeconfig prod-mgmt > prod-mgmt.kubeconfig

Stage 6: Initialize Providers on the Target Cluster

Before migrating objects, the target cluster must have identical (or newer) versions of all CAPI CRDs and controller webhooks installed:

clusterctl init \
  --kubeconfig prod-mgmt.kubeconfig \
  --infrastructure aws

Stage 7: Execute clusterctl move

Transfer the custom resource graph:

# Always execute dry-run first to inspect discovered resources
clusterctl move \
  --to-kubeconfig prod-mgmt.kubeconfig \
  --dry-run -v 5

# Execute the live migration
clusterctl move \
  --to-kubeconfig prod-mgmt.kubeconfig

Stage 8: Teardown Bootstrap Cluster

Verify that the target management cluster is actively reconciling its own resources, then delete the local KinD cluster:

kind delete cluster --name capi-bootstrap

4. Under the Hood of clusterctl move

The clusterctl move command is not a simple kubectl get -o yaml | kubectl apply. It implements a specialized, multi-phase migration protocol designed to prevent split-brain dual-reconciliation storms:

sequenceDiagram
    autonumber
    participant CLI as clusterctl CLI
    participant SrcAPI as Source API (Bootstrap)
    participant Cloud as Cloud Infrastructure (AWS/Azure)
    participant TgtAPI as Target API (Management)
    participant TgtCtrl as Target CAPI Controllers

    CLI->>SrcAPI: 1. Discovers full CRD graph via owner references
    CLI->>SrcAPI: 2. Sets Cluster.spec.paused = true (Split-Brain Prevention)
    Note over SrcAPI,Cloud: Source controllers freeze reconciliation loops
    CLI->>SrcAPI: 3. Exports CRDs, Secrets (CAs, tokens), ConfigMaps
    CLI->>TgtAPI: 4. Applies resources to target cluster
    CLI->>TgtAPI: 5. Verifies all objects established and webhooks validate
    CLI->>SrcAPI: 6. Deletes objects from source (suppressing infrastructure deletion)
    CLI->>TgtAPI: 7. Sets Cluster.spec.paused = false
    TgtCtrl->>Cloud: 8. Target controllers adopt instances & rebuild status from reality

Phase 1: Dependency Graph Discovery

clusterctl identifies all root Cluster resources in the target namespace and recursively traverses their ownerReferences. It builds an in-memory graph containing Cluster, InfrastructureCluster, KubeadmControlPlane, MachineDeployment, MachineSet, Machine, InfrastructureMachine, and all referenced KubeadmConfig, Secret, and ConfigMap objects.

Phase 2: Reconciliation Pausing (Split-Brain Prevention)

If controllers in both the source and target clusters reconciled the same cloud resources simultaneously, they would issue conflicting API calls (e.g., competing rolling updates or duplicate VM teardowns). To prevent this, clusterctl sets:

spec:
  paused: true

on all Cluster resources in the source cluster. Source controllers immediately halt reconciliation loops.

Phase 3: Object Transformation & Secret Preservation

clusterctl strips runtime metadata from the exported objects:

• Removes metadata.resourceVersion, metadata.uid, and metadata.selfLink.
• Preserves all CA certificates, bootstrap tokens, and cloud credential Secrets.
• Deliberately strips the .status stanza.

Phase 4: Safe Source Object Purge

clusterctl deletes the custom resources from the source cluster. Crucially, CAPI controllers are programmed to respect the pause state: because spec.paused is true, deleting a Machine or Cluster from the source cluster does not invoke cloud API calls to delete the underlying VMs or load balancers.

Phase 5: Unpausing & Status Re-Derivation

Once the resources exist on the target cluster, clusterctl unpauses the Cluster (spec.paused = false). The target controllers adopt the resources, query the cloud provider using the recorded spec.providerID, and rebuild .status and condition trees from live infrastructure.

5. Why Status Is Not Transferred

A fundamental design principle of Kubernetes and Cluster API is:

Desired state (spec) is declared; observed state (status) is derived from reality.

clusterctl move intentionally does not migrate .status blocks. The target management cluster’s controllers are expected to inspect the external cloud provider, query workload node APIs, evaluate certificate expiration dates, and rebuild the status condition tree from scratch.

Immediately after running clusterctl move, running kubectl get machines on the target cluster may show machines briefly in an uninitialized phase while informers populate and cloud provider queries resolve. Within one or two reconciliation loops, status converges cleanly.

6. Management Topologies: Self-Managing vs. Dedicated

Once pivoted, how should the management plane be organized?

flowchart TD
    subgraph SelfManaging["Pattern A: Self-Managing Management Cluster"]
        SM_CTRL["CAPI Controllers"]
        SM_NODES["Management Nodes (AWS)"]
        SM_CTRL -->|"Reconciles its own MachineDeployments"| SM_NODES
        SM_CTRL -->|"Reconciles external Workload Clusters"| WC1["Workload Cluster 1"]
    end

    subgraph Dedicated["Pattern B: Dedicated Management Cluster (Hub-and-Spoke)"]
        D_MGMT["Dedicated Management Cluster (EKS / Bare-Metal)"]
        D_CTRL["CAPI Controllers"]
        D_MGMT --- D_CTRL
        D_CTRL -->|"Only manages external clusters"| WC_A["Workload Cluster A"]
        D_CTRL -->|"Only manages external clusters"| WC_B["Workload Cluster B"]
        D_CTRL -->|"Only manages external clusters"| WC_C["Workload Cluster C"]
    end

Pattern A: Self-Managing Cluster

The management cluster’s own nodes are represented as CAPI Machine objects inside its own etcd. When the management cluster scales or upgrades itself, its local controllers provision and replace its own host VMs.

• Pros: Minimal resource footprint; zero external cluster dependencies.
• Cons: High blast radius; a corrupted controller upgrade can lock operators out of the management cluster.

Pattern B: Dedicated Management Cluster (Hub-and-Spoke)

The management cluster is hosted on a managed Kubernetes service (e.g., Amazon EKS, Azure AKS, Google GKE) or dedicated hardware. It never hosts business workloads and only manages external CAPI workload clusters.

• Pros: Complete failure isolation; workload cluster churn cannot starve management plane controllers.
• Cons: Requires running a dedicated cluster purely for management plane operations.

7. Diagnostic Workflows: Troubleshooting Pivot Failures

When clusterctl move fails or stalls, use a structured decision tree to diagnose the failure:

flowchart TD
    START["clusterctl move Fails or Hangs"] --> CHECK{"Where Did Failure Occur?"}

    CHECK -->|"Dry-Run / Preflight"| PRE{"Preflight Error"}
    PRE -->|"Provider Missing"| P1["Target cluster lacks required CAPI providers.<br/>Run clusterctl init on target first"]
    PRE -->|"Version Incompatible"| P2["Target provider versions < source provider versions.<br/>Upgrade providers on target"]

    CHECK -->|"During Move Execution"| MOV{"Execution Error"}
    MOV -->|"Blocked by Finalizers"| M1["Inspect source resources for custom non-CAPI finalizers<br/>preventing deletion"]
    MOV -->|"Active Rollout Conflict"| M2["Cluster was mid-upgrade or mid-scale.<br/>Wait for cluster to reach steady-state"]

    CHECK -->|"Post-Move on Target"| PST{"Target Reconciliation Stuck"}
    PST -->|"Controllers Not Running"| T1["Check target worker capacity: pods stuck in Pending<br/>due to control-plane NoSchedule taints"]
    PST -->|"Cloud Credentials Missing"| T2["Ensure cloud credential Secrets (e.g. AWS credentials)<br/>were copied to target namespace"]

Essential Diagnostic Commands

1. Pre-Move Verification (Always Dry-Run First)

clusterctl move \
  --to-kubeconfig target.kubeconfig \
  --dry-run \
  -v 5

Inspect the output to verify:

• All Clusters, Machines, and MachineDeployments are discovered.
• No dangling external resources block migration.

2. Verify Provider Parity

# Check providers on source
kubectl get providers -A

# Check providers on target
kubectl --kubeconfig target.kubeconfig get providers -A

Target provider versions must be greater than or equal to source provider versions.

3. Inspect Stalled Move Events

If clusterctl move hangs while deleting from source, inspect source resources for blocking finalizers:

kubectl get clusters,machines -o custom-columns=NAME:.metadata.name,FINALIZERS:.metadata.finalizers

8. Key Takeaways

• The Chicken-and-Egg Solution: Ephemeral bootstrap clusters (KinD) exist solely to break the paradox of declarative cluster creation. They instantiate the real management cluster and are discarded after the pivot.
• Infrastructure Does Not Move: clusterctl move transfers CRD objects, etcd state, and reconciliation authority. The underlying virtual machines and load balancers continue running without interruption.
• Split-Brain Prevention: clusterctl move pauses source clusters (spec.paused = true) before exporting objects, ensuring two controller planes never compete to manage the same cloud resources.
• Status Is Re-Derived: .status is never migrated. Target controllers adopt resources and derive health conditions from the actual cloud infrastructure.
• clusterctl move Is Not Disaster Recovery: move is an interactive migration protocol designed for the pivot workflow. True disaster recovery requires etcd snapshots and backup tools like Velero.

Next in the series

Now that we understand how management clusters are bootstrapped and pivoted, explore Part 14: Running Cluster API in Production to examine multi-tenant management architectures, disaster recovery strategies, controller high-availability, and GitOps workflows at scale.

References & Further Reading

• Cluster API Book: The clusterctl move Command
• Cluster API Book: Bootstrap & Pivot Architecture
• Cluster API Book: Provider Contracts for Move Operations
• KinD Documentation: Quick Start & Multi-Node Clusters