Provider Architecture and Provider Contracts

Cluster API manages Kubernetes clusters across radically different environments: AWS, Microsoft Azure, Google Cloud, VMware vSphere, OpenStack, Nutanix, Equinix Metal, and local Docker containers.

Yet, the core Cluster API controller (capi-controller-manager) contains zero lines of cloud-specific code. It imports no cloud provider SDKs, constructs no EC2 instances, mounts no EBS volumes, and makes no calls to Azure Resource Manager.

How does a generic controller coordinate concrete infrastructure across heterogeneous clouds without compiling provider code into its binary?

The answer lies in Provider Contracts: a decoupled, distributed interface architecture built on Kubernetes Custom Resource Definitions (CRDs), dynamic controller-runtime clients, and asynchronous status handshakes.

1. First-Principles Problem Framing: The Polymorphism Dilemma in Distributed Systems

In traditional software design, polymorphism is implemented via language-level interfaces:

// The in-process monolithic pattern (anti-pattern for multi-cloud controllers)
type InfrastructureProvider interface {
    CreateCluster(ctx context.Context, cluster *Cluster) error
    CreateMachine(ctx context.Context, machine *Machine) (*Instance, error)
    DeleteMachine(ctx context.Context, machine *Machine) error
}

If Cluster API had adopted this monolithic in-process interface pattern, the entire project would have collapsed under operational and architectural weight:

1. Dependency Hell and Binary Bloat: Compiling AWS SDK v2, Azure SDK for Go, Google Cloud client libraries, VMware govmomi, and OpenStack gophercloud into a single binary results in hundreds of megabytes of compiled machine code and constant dependency resolution conflicts.
2. Release Coupling: A critical bug or API deprecation in AWS would require re-releasing the entire Cluster API core, forcing vSphere and Azure operators to update an unrelated binary.
3. Ecosystem Lock-in: Hardware vendors, telcos, and internal enterprise private clouds could not build custom infrastructure drivers without upstreaming proprietary code into the Kubernetes project.

flowchart TD
    subgraph MonolithicModel["The Flawed Monolithic Approach (Tight Coupling)"]
        CoreAPI["CAPI Core Engine"]
        AWSLib["AWS SDK"]
        AzureLib["Azure SDK"]
        VSphereLib["vSphere SDK"]
        BareMetalLib["Bare Metal Drivers"]

        CoreAPI --> AWSLib
        CoreAPI --> AzureLib
        CoreAPI --> VSphereLib
        CoreAPI --> BareMetalLib
    end

    subgraph ContractModel["The Cluster API Contract Approach (Loose Coupling)"]
        GenericCore["CAPI Core (Generic State Machine)"]
        CRDInterface["Kubernetes API & Contract Handshake"]
        CAPA["CAPA Controller (AWS)"]
        CAPZ["CAPZ Controller (Azure)"]
        CAPV["CAPV Controller (vSphere)"]
        Custom["Custom Private Cloud Controller"]

        GenericCore -->|"Watches Unstructured CRDs"| CRDInterface
        CRDInterface <-->|"Reconciles AWSCluster / AWSMachine"| CAPA
        CRDInterface <-->|"Reconciles AzureCluster / AzureMachine"| CAPZ
        CRDInterface <-->|"Reconciles VSphereCluster / VSphereMachine"| CAPV
        CRDInterface <-->|"Reconciles CustomCluster / CustomMachine"| Custom
    end

The Solution: Distributed Contracts via Duck Typing

Instead of Go interfaces, Cluster API relies on distributed custom resource contracts.

• CAPI Core manages high-level intent (Cluster, Machine, MachineDeployment).
• Dedicated provider controllers run in separate pods and namespaces, managing provider-specific CRDs (AWSCluster, AWSMachine, KubeadmConfig, KubeadmControlPlane).
• Core and provider controllers communicate asynchronously through the Kubernetes API server using duck typing: CAPI Core reads and writes provider CRDs dynamically via controller-runtime unstructured clients, expecting specific standardized fields and status subresources to exist.

2. Mental Model: The Four Provider Domains

Cluster API decomposes cluster management into four independent operational domains. Each domain is governed by a distinct provider contract:

flowchart TD
    subgraph CoreDomain["1. Core Provider (CAPI)"]
        ClusterCR["Cluster (Desired Topology)"]
        MachineCR["Machine (Node Intent)"]
        ClusterCR --> MachineCR
    end

    subgraph ControlPlaneDomain["2. Control Plane Provider (e.g., KCP)"]
        KCP["KubeadmControlPlane"]
        KCP -->|"Orchestrates CP Quorum"| MachineCR
    end

    subgraph BootstrapDomain["3. Bootstrap Provider (e.g., CABPK)"]
        BootstrapConfig["KubeadmConfig"]
        BootstrapSecret["Secret (cloud-init / Ignition)"]
        BootstrapConfig -->|"Renders Payload"| BootstrapSecret
    end

    subgraph InfraDomain["4. Infrastructure Provider (e.g., CAPA, CAPZ, CAPV)"]
        InfraCluster["AWSCluster (VPC, Subnets, LB)"]
        InfraMachine["AWSMachine (EC2, Disks, ENIs)"]
    end

    ClusterCR -.->|"spec.controlPlaneRef"| KCP
    ClusterCR -.->|"spec.infrastructureRef"| InfraCluster
    MachineCR -.->|"spec.bootstrap.configRef"| BootstrapConfig
    MachineCR -.->|"spec.infrastructureRef"| InfraMachine
    BootstrapSecret -.->|"Injected via user data"| InfraMachine

1. Core Provider (CAPI)

• Scope: Orchestrates the lifecycle state machine across the fleet.
• Responsibilities: Evaluates replica counts, computes rollout strategies (RollingUpdate), tracks machine health checks, coordinates deletion order, and manages node binding.

2. Infrastructure Provider (CAPx)

• Scope: Environment-specific compute, networking, and storage primitives.
• Implementations: AWS (CAPA), Azure (CAPZ), GCP (CAPG), vSphere (CAPV), Bare Metal (CAPM3), Docker (CAPD).
• Responsibilities: Provisions VPCs, subnets, route tables, load balancers, security groups, virtual machine instances, network interfaces, and block storage.

3. Bootstrap Provider (CABPx)

• Scope: Initializing the operating system into a functional Kubernetes node.
• Implementations: Kubeadm (CABPK), MicroK8s, Talos.
• Responsibilities: Generates ignition configs, cloud-init scripts, certificates, token authentication files, and kubelet configuration flags. Writes the output to a Kubernetes Secret.

4. Control Plane Provider (KCPx)

• Scope: Control plane quorum, initialization, and sequential upgrade orchestration.
• Implementations: Kubeadm Control Plane (KCP), Managed Control Planes (EKS, AKS, GKE).
• Responsibilities: Manages etcd clustering, distributes CA certificates across master nodes, performs single-node rolling replacements during Kubernetes upgrades, and tracks apiserver health.

3. Contract Specifications and Field Matrices

A provider contract is an operational agreement: if a provider CRD exposes specific standardized fields and behaves according to lifecycle rules, CAPI Core guarantees seamless orchestration.

The Four Core Provider Contracts

4. Architectural Deep Dive: Anatomy of Core Contracts

The InfrastructureCluster Contract

The InfrastructureCluster represents the shared networking and control plane access layer for a Kubernetes cluster.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSCluster
metadata:
  name: prod-cluster
  namespace: default
spec:
  # Standard Contract Requirement: Control Plane Endpoint
  controlPlaneEndpoint:
    host: "prod-cluster-apiserver-12345.us-east-1.elb.amazonaws.com"
    port: 6443
  # Provider-specific configurations
  networkSpec:
    vpc:
      cidrBlock: "10.0.0.0/16"
status:
  # Standard Contract Requirement: Readiness Handshake
  ready: true
  # Standard Contract Requirement: Failure Domain Topology
  failureDomains:
    us-east-1a:
      controlPlane: true
    us-east-1b:
      controlPlane: true
    us-east-1c:
      controlPlane: true

Key Contract Behaviors:

1. Control Plane Endpoint Discovery: CAPI Core will not allow worker machines to boot until spec.controlPlaneEndpoint is populated and status.ready is true. The bootstrap script requires this endpoint so kubeadm join knows where to find the API server.
2. Failure Domain Surfacing: The provider discovers and populates status.failureDomains. CAPI’s MachineSet and KubeadmControlPlane controllers read this map dynamically to spread control plane and worker nodes evenly across physical data center zones without knowing cloud-specific availability zone names.

The InfrastructureMachine Contract

The InfrastructureMachine represents a single compute host (virtual machine, bare-metal server, or container).

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachine
metadata:
  name: prod-worker-01
  namespace: default
spec:
  # Standard Contract Requirement: Assigned by provider upon creation
  providerID: "aws:///us-east-1a/i-0a1b2c3d4e5f67890"
  instanceType: "m5.large"
status:
  # Standard Contract Requirement: Ready when VM is provisioned
  ready: true
  # Standard Contract Requirement: Surfacing Node Network Identity
  addresses:
    - type: InternalIP
      address: "10.0.12.44"
    - type: Hostname
      address: "ip-10-0-12-44.ec2.internal"

The Crucial Role of providerID:

The providerID string is the global bridge between Cluster API’s management plane and the workload Kubernetes cluster.

• When the cloud VM launches, the infrastructure provider reads the cloud instance ID and formats it as a URI: aws:///<zone>/<instance-id>.
• The provider writes this URI into InfrastructureMachine.spec.providerID.
• Core CAPI copies spec.providerID from InfrastructureMachine to Machine.spec.providerID.
• Meanwhile, the cloud-provider integration (or kubelet) running inside the workload cluster registers the Node with the identical spec.providerID.
• Node Matching: The CAPI Machine controller watches the workload cluster. When it finds a Node whose spec.providerID matches Machine.spec.providerID, it binds them together: Machine.status.nodeRef = Node.metadata.name.

The BootstrapConfig Contract

The bootstrap provider transforms machine configuration into an executable bootstrapping script:

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfig
metadata:
  name: prod-worker-01
  namespace: default
status:
  # Standard Contract Requirement: Data Secret is generated
  ready: true
  dataSecretName: "prod-worker-01-bootstrap-secret"

The Secret Passing Protocol:

1. When a Machine is created, it references a BootstrapConfig (e.g., KubeadmConfig).
2. The bootstrap controller evaluates cluster certificates, generates a bootstrap token (kubeadm join --token ...), formats cloud-init YAML or Ignition JSON, and saves it into a Kubernetes Secret.
3. The bootstrap controller sets status.dataSecretName = "<secret-name>" and status.ready = true.
4. CAPI Core detects status.ready = true, extracts status.dataSecretName, and injects it into the InfrastructureMachine spec.
5. The infrastructure provider reads the secret and injects its contents into the VM’s launch template as user data.

5. End-to-End Walkthrough: The Multi-Provider Machine Creation Handshake

To understand how contracts enable real-time collaboration between independent controllers, trace the sequence of events when a new worker node is provisioned:

sequenceDiagram
    autonumber
    actor Admin as Platform Engineer
    participant Core as CAPI Machine Controller
    participant Bootstrap as Kubeadm Bootstrap Controller
    participant Infra as AWS Infrastructure Controller
    participant AWS as AWS EC2 Cloud API
    participant Node as Workload Cluster Kubelet

    Admin->>Core: 1. Creates Machine referencing KubeadmConfig and AWSMachine
    Core->>Bootstrap: 2. Reconciles KubeadmConfig resource
    Core->>Infra: 3. Reconciles AWSMachine without dataSecretName
    
    Note over Bootstrap: Reconciles tokens and CA certificates
    Bootstrap->>Bootstrap: 4. Generates cloud-init Secret
    Bootstrap->>Core: 5. Sets status.ready = true and status.dataSecretName
    Core->>Infra: 6. Injects dataSecretName into AWSMachine spec
    
    Note over Infra,AWS: Infrastructure Provisioning
    Infra->>AWS: 7. Calls ec2:RunInstances with user data from Secret
    AWS-->>Infra: 8. Instance i-012345 launched in us-east-1a
    Infra->>Core: 9. Populates spec.providerID and sets status.ready = true

    Note over AWS,Node: Node Boot and Registration
    AWS->>Node: 10. Instance boots and executes cloud-init
    Node->>Node: 11. Executes kubeadm join against control plane
    Node->>Core: 12. Registers Node with providerID aws:///us-east-1a/i-012345

    Core->>Core: 13. Matches Machine.spec.providerID == Node.spec.providerID
    Core->>Core: 14. Binds Machine.status.nodeRef to Node name

Notice that:

• The Bootstrap controller never talks to AWS.
• The AWS controller never inspects kubeadm configurations or tokens.
• CAPI Core coordinates the entire handshake through standard field transitions (status.ready, dataSecretName, providerID).

6. Contract Versioning vs. CRD API Versioning

A frequent point of confusion for platform engineers is the distinction between a CRD API Version and a Cluster API Contract Version.

CRD API Version (e.g., v1beta2) != Cluster API Contract Version (e.g., v1beta1)

How CAPI Discovers Compatible Contracts

When CAPI Core initializes, it scans all CRDs registered in the management cluster. Providers declare which CAPI contract versions their CRDs implement using labels:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: awsmachines.infrastructure.cluster.x-k8s.io
  labels:
    # Advertises compliance with CAPI runtime contract
    cluster.x-k8s.io/contract: v1beta1
    cluster.x-k8s.io/provider: infrastructure-aws

If you deploy a new provider version whose CRD drops the contract label supported by your CAPI core controller, CAPI will log an admission error and refuse to reconcile the resources, preventing split-brain or schema mismatch panics.

7. The clusterctl Provider Contract

In addition to the runtime reconciliation contracts, Cluster API defines a packaging and distribution contract for clusterctl.

flowchart LR
    Repo["GitHub / OCI Release"] --> Metadata["metadata.yaml"]
    Repo --> Components["components.yaml"]
    
    CLI["clusterctl CLI"] -->|"1. Reads release metadata"| Metadata
    CLI -->|"2. Downloads provider components"| Components
    CLI -->|"3. Applies manifests to management cluster"| ManagementAPI["Management API Server"]

To allow clusterctl init and clusterctl upgrade to automatically discover, fetch, install, and wire up a provider, the provider repository must provide:

1. metadata.yaml: Declares release mappings, supported contract versions, and upgrade paths:

   apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
   kind: Metadata
   releaseSeries:
     - major: 2
       minor: 4
       contract: v1beta1
   

2. components.yaml: A consolidated multi-resource manifest containing the provider’s CRDs, RBAC Roles, ServiceAccounts, Deployments, and Webhook Configurations.

8. Mental Experiments & Diagnostics: Diagnosing Contract Failures

When a machine or cluster stops progressing, the failure is almost always a violation of a provider contract. Use this diagnostic decision tree to identify which contract is broken:

flowchart TD
    Start["Machine Stuck in 'Provisioning' or 'Pending'"] --> CheckBootstrap{"Is KubeadmConfig status.ready == true?"}

    CheckBootstrap -->|"No"| BootstrapBroken["Bootstrap Contract Violation: CABPK failed to create Secret. Check CABPK controller logs & cluster CA secrets."]
    CheckBootstrap -->|"Yes"| CheckSecretRef{"Did Machine controller pass dataSecretName to InfraMachine?"}

    CheckSecretRef -->|"No"| CoreBroken["Core Controller Reconcile Stall: Check capi-controller-manager logs."]
    CheckSecretRef -->|"Yes"| CheckInfraReady{"Is InfraMachine status.ready == true?"}

    CheckInfraReady -->|"No"| InfraBroken["Infra Contract Stall: Cloud VM failed to launch. Check cloud IAM, quotas, or CAPx logs."]
    CheckInfraReady -->|"Yes"| CheckProviderID{"Is spec.providerID populated on Machine?"}

    CheckProviderID -->|"No"| ProviderIDMissing["Contract Violation: Infra provider marked ready=true but forgot to populate spec.providerID."]
    CheckProviderID -->|"Yes"| CheckNodeMatch{"Does a Node in Workload Cluster have matching spec.providerID?"}

    CheckNodeMatch -->|"No"| NodeInitFailed["Bootstrap Execution Failure: VM launched, but cloud-init / kubelet failed. Inspect /var/log/cloud-init-output.log on the VM."]
    CheckNodeMatch -->|"Yes"| NodeRefBound["Contract Satisfied: Machine bound to Node (status.nodeRef)."]

Practical Inspection Commands

1. Inspecting Duck-Typed Provider Fields Across the Fleet

Verify that all infrastructure machines have satisfied the contract requirements:

kubectl get infrastructuremachines -A -o custom-columns=\
NAMESPACE:.metadata.namespace,\
NAME:.metadata.name,\
READY:.status.ready,\
PROVIDER_ID:.spec.providerID,\
DATA_SECRET:.spec.dataSecretName

2. Checking CRD Contract Compliance Labels

Verify that your installed provider CRDs correctly advertise supported contract versions:

kubectl get crd -l cluster.x-k8s.io/contract -o custom-columns=\
CRD:.metadata.name,\
CONTRACT:.metadata.labels."cluster\.x-k8s\.io/contract",\
PROVIDER:.metadata.labels."cluster\.x-k8s\.io/provider"

3. Tracing Node Matching Discrepancies

If a Machine has a providerID but never binds to a Node, compare the strings directly:

# Get Machine Provider ID
kubectl get machine my-worker-01 -o jsonpath='{.spec.providerID}'

# Get Workload Cluster Node Provider IDs
kubectl --kubeconfig=my-cluster.kubeconfig get nodes -o custom-columns=NAME:.metadata.name,PROVIDER_ID:.spec.providerID

If the prefixes differ (e.g., aws:///us-east-1a/i-xyz vs aws://us-east-1a/i-xyz), the Machine controller will never bind the node, leaving the machine in Provisioning indefinitely.

9. Key Takeaways

• Decoupled by Design: CAPI Core contains zero cloud-specific dependencies. It interacts with providers via Kubernetes CRD contracts and duck typing.
• Contract Over Interface: Because controllers run in separate processes and pods, contracts are enforced through OpenAPI schema conventions, dynamic clients, and asynchronous status transitions.
• The Core Contract Quadrant: InfrastructureCluster (endpoints & failure domains), InfrastructureMachine (providerID & readiness), BootstrapConfig (secret generation), and ControlPlane (quorum initialization).
• The providerID Bridge: The spec.providerID field is the fundamental anchor that connects the management plane’s declarative Machine to the workload cluster’s physical Node.
• API Version vs. Contract Version: Provider CRD versions (v1beta2) reflect schema stability, while CAPI contract versions (v1beta1) reflect core interaction protocols.

Next in the series

Now that we understand how Cluster API coordinates cloud infrastructure through standardized contracts, explore Part 16: Building a Cluster API Infrastructure Provider to build a functional, custom infrastructure provider from scratch using Kubebuilder and the CAPI contract framework.

References & Further Reading

• Cluster API Book: Provider Contracts Overview
• Cluster API Book: InfraCluster Contract
• Cluster API Book: InfraMachine Contract
• Cluster API Book: BootstrapConfig Contract
• Cluster API Book: ControlPlane Contract
• Cluster API Book: clusterctl Provider Contract