Building a Cluster API Infrastructure Provider

Cluster API’s modular architecture allows platform teams to bring Kubernetes lifecycle management to any environment: specialized bare-metal hardware, custom internal hypervisors (Proxmox, bhyve, Firecracker), regional cloud providers, or isolated on-premises data centers.

To support new infrastructure, you do not modify Cluster API Core. Instead, you build an Infrastructure Provider: a specialized Kubernetes operator that translates generic CAPI resources into concrete compute, network, and storage primitives.

Writing a functional provider is not just about generating Custom Resource Definitions (CRDs). The core engineering challenge lies in reconciliation mechanics: building idempotent state machines, implementing non-blocking status handshakes, managing asynchronous hardware lifecycles, and ensuring leak-free infrastructure teardowns through finalizers.

1. First-Principles Problem Framing: The Role of an Infrastructure Adapter

At its core, a Cluster API Infrastructure Provider acts as a declarative bidirectional adapter between two state machines:

1. The Kubernetes Management API: Where desired state is expressed through declarative objects (Cluster, Machine, ProxmoxCluster, ProxmoxMachine).
2. The Infrastructure Target API: Where actual state is queried, allocated, updated, and terminated across physical servers, hypervisors, or cloud endpoints.

flowchart TD
    subgraph ManagementPlane["Management Cluster (Kubernetes API)"]
        ClusterCR["Cluster (Core CAPI)"]
        MachineCR["Machine (Core CAPI)"]
        CustomCluster["ProxmoxCluster (CRD)"]
        CustomMachine["ProxmoxMachine (CRD)"]

        ClusterCR -.->|"spec.infrastructureRef"| CustomCluster
        MachineCR -.->|"spec.infrastructureRef"| CustomMachine
    end

    subgraph ProviderOperator["Infrastructure Provider Controller Manager"]
        ClusterReconciler["Cluster Reconciler Loop"]
        MachineReconciler["Machine Reconciler Loop"]
        PatchHelper["Status & Finalizer Patch Helper"]

        CustomCluster <-->|"Watches & Updates"| ClusterReconciler
        CustomMachine <-->|"Watches & Updates"| MachineReconciler
        ClusterReconciler --> PatchHelper
        MachineReconciler --> PatchHelper
    end

    subgraph InfrastructurePlane["Target Infrastructure (Hypervisor / Cloud)"]
        NetworkVIP["Virtual IP / Load Balancer"]
        ComputeHost["Virtual Machines / Bare Metal Hosts"]

        ClusterReconciler <-->|"HTTPS / REST API"| NetworkVIP
        MachineReconciler <-->|"HTTPS / REST API"| ComputeHost
    end

The provider must fulfill four non-negotiable architectural invariants:

• Idempotency: A reconciliation triggered ten times in succession must produce the identical infrastructure footprint as a single invocation.
• Asynchrony: Physical and virtual hardware provisioning takes minutes. The reconciler must never block worker threads; it must requeue with backoff until external operations complete.
• Convergence: If external resources diverge (e.g., an operator terminates a VM out-of-band), the reconciler must detect drift and re-converge reality toward desired state.
• Deletion Safety: Deleting a Kubernetes CR must reliably deprovision underlying cloud or hypervisor resources before the Kubernetes object is purged from etcd.

2. Mental Model: The Minimal Provider Topology

A production-ready infrastructure provider requires two core custom resource pairs and their template variants:

Cluster Level:
├── ProxmoxCluster               (Defines shared VPC, subnets, VIP endpoint)
└── ProxmoxClusterTemplate       (Allows parameterized generation in ClusterClass)

Machine Level:
├── ProxmoxMachine               (Defines CPU, memory, disk, and MAC addressing)
└── ProxmoxMachineTemplate       (Enables MachineDeployment worker pool rollouts)

The Architectural Boundary

• ProxmoxCluster fulfills the InfraCluster Contract. It allocates shared networking and publishes the Kubernetes API server endpoint (spec.controlPlaneEndpoint).
• ProxmoxMachine fulfills the InfraMachine Contract. It receives bootstrap data, launches a single compute host, assigns a unique spec.providerID, and signals readiness when the VM is responsive.

3. Comparison: Cluster vs. Machine Reconciler Responsibilities

The two reconcilers execute concurrently in separate goroutines within the same controller binary, each bounded by strict contract expectations:

4. Step-by-Step Implementation Blueprint

To illustrate the architecture concretely, we will walk through the construction of cluster-api-provider-proxmox (CAPP).

Step 1: Repository Initialization with Kubebuilder

Cluster API providers are standard Kubernetes operators. Initialize the project using Kubebuilder:

# 1. Initialize Go module
mkdir cluster-api-provider-proxmox
cd cluster-api-provider-proxmox
go mod init github.com/example/cluster-api-provider-proxmox

# 2. Initialize Kubebuilder scaffold using the standard CAPI domain
kubebuilder init --domain cluster.x-k8s.io --repo github.com/example/cluster-api-provider-proxmox

# 3. Create the Cluster API
kubebuilder create api \
  --group infrastructure \
  --version v1alpha1 \
  --kind ProxmoxCluster \
  --resource --controller

# 4. Create the Machine API
kubebuilder create api \
  --group infrastructure \
  --version v1alpha1 \
  --kind ProxmoxMachine \
  --resource --controller

Step 2: Defining the Go API Types and Contract Compliance

Open api/v1alpha1/proxmoxcluster_types.go. To satisfy the InfraCluster contract, the struct must expose spec.controlPlaneEndpoint and status.ready:

package v1alpha1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
)

// ProxmoxClusterSpec defines the desired state of ProxmoxCluster
type ProxmoxClusterSpec struct {
    // ControlPlaneEndpoint represents the public endpoint for the cluster.
    // MANDATORY CONTRACT FIELD: Populated by this provider or user-supplied.
    ControlPlaneEndpoint clusterv1.APIEndpoint `json:"controlPlaneEndpoint"`

    // TargetNode represents the Proxmox virtualization server node.
    TargetNode string `json:"targetNode"`
}

// ProxmoxClusterStatus defines the observed state of ProxmoxCluster
type ProxmoxClusterStatus struct {
    // Ready denotes that the cluster infrastructure is fully provisioned.
    // MANDATORY CONTRACT FIELD: Signals CAPI Core to begin machine provisioning.
    // +kubebuilder:default=false
    Ready bool `json:"ready"`

    // FailureDomains surfaces the available virtualization nodes for scheduling.
    // +optional
    FailureDomains clusterv1.FailureDomains `json:"failureDomains,omitempty"`

    // Conditions defines current service state of the cluster.
    // +optional
    Conditions clusterv1.Conditions `json:"conditions,omitempty"`
}

Next, open api/v1alpha1/proxmoxmachine_types.go. To satisfy the InfraMachine contract, the struct must expose spec.providerID, status.ready, and status.addresses:

package v1alpha1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
)

// ProxmoxMachineSpec defines the desired hardware configuration of a host
type ProxmoxMachineSpec struct {
    // ProviderID is the unique identifier assigned by the provider.
    // MANDATORY CONTRACT FIELD: Must match the format used by the workload kubelet.
    // Format: proxmox://<node-name>/<vm-id>
    // +optional
    ProviderID *string `json:"providerID,omitempty"`

    VCPUs    int    `json:"vcpus"`
    MemoryMB int    `json:"memoryMB"`
    DiskGB   int    `json:"diskGB"`
    Template string `json:"template"`
}

// ProxmoxMachineStatus defines the observed state of ProxmoxMachine
type ProxmoxMachineStatus struct {
    // Ready indicates that the VM has been created and is active.
    // MANDATORY CONTRACT FIELD: Signals CAPI that the VM exists.
    // +kubebuilder:default=false
    Ready bool `json:"ready"`

    // Addresses contains the IP addresses assigned to the VM.
    // +optional
    Addresses []clusterv1.MachineAddress `json:"addresses,omitempty"`

    // FailureReason indicates that an unrecoverable error has occurred.
    // +optional
    FailureReason *string `json:"failureReason,omitempty"`

    // FailureMessage provides human-readable failure context.
    // +optional
    FailureMessage *string `json:"failureMessage,omitempty"`
}

5. The Machine Reconciliation Engine

The Reconcile loop for ProxmoxMachine must coordinate five distinct lifecycle phases:

flowchart TD
    Start["Reconcile(req)"] --> FetchMachine["1. Fetch ProxmoxMachine & owning CAPI Machine"]
    FetchMachine --> CheckPaused{"Is Cluster or Machine Paused?"}
    
    CheckPaused -->|"Yes"| StopReconcile["Return ctrl.Result{}, nil"]
    CheckPaused -->|"No"| CheckDeletion{"Is DeletionTimestamp set?"}

    CheckDeletion -->|"Yes"| HandleDelete["Execute ReconcileDelete: Terminate VM & Remove Finalizer"]
    CheckDeletion -->|"No"| AddFinalizer["Ensure Finalizer present: proxmoxmachine.infrastructure.cluster.x-k8s.io"]

    AddFinalizer --> CheckBootstrap{"Is Machine.spec.bootstrap.dataSecretName present?"}
    CheckBootstrap -->|"No: Wait for CABPK"| RequeueWait["Return ctrl.Result{RequeueAfter: 15s}, nil"]
    
    CheckBootstrap -->|"Yes"| CheckVMExists{"Does VM exist in Proxmox API?"}
    CheckVMExists -->|"No"| LaunchVM["Fetch Secret, inject Cloud-Init, call POST /api2/json/nodes/vm/clone"]
    LaunchVM --> RequeuePending["Return ctrl.Result{RequeueAfter: 30s}, nil"]

    CheckVMExists -->|"Yes: VM Running"| PopulateContract["Read VM ID, Set spec.providerID & status.ready = true"]
    PopulateContract --> PatchStatus["Save updates using CAPI patch.Helper"]

Writing the Core Reconciler (Go Implementation)

Here is the production reconciliation structure implementing the CAPI contract pattern:

package controllers

import (
    "context"
    "fmt"
    "time"

    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"

    clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
    "sigs.k8s.io/cluster-api/util"
    "sigs.k8s.io/cluster-api/util/annotations"
    "sigs.k8s.io/cluster-api/util/patch"

    infrav1 "github.com/example/cluster-api-provider-proxmox/api/v1alpha1"
)

const MachineFinalizer = "proxmoxmachine.infrastructure.cluster.x-k8s.io"

type ProxmoxMachineReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

func (r *ProxmoxMachineReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // 1. Fetch the target ProxmoxMachine
    proxmoxMachine := &infrav1.ProxmoxMachine{}
    if err := r.Get(ctx, req.NamespacedName, proxmoxMachine); err != nil {
        if apierrors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }

    // 2. Fetch the owning CAPI Machine using CAPI utility helpers
    machine, err := util.GetOwnerMachine(ctx, r.Client, proxmoxMachine.ObjectMeta)
    if err != nil {
        return ctrl.Result{}, err
    }
    if machine == nil {
        // Machine controller has not yet set OwnerReference
        return ctrl.Result{}, nil
    }

    // 3. Check if the cluster is paused
    cluster, err := util.GetClusterFromMetadata(ctx, r.Client, machine.ObjectMeta)
    if err != nil {
        return ctrl.Result{}, err
    }
    if annotations.IsPaused(cluster, proxmoxMachine) {
        return ctrl.Result{}, nil
    }

    // 4. Initialize CAPI Patch Helper to safely handle concurrent status updates
    patchHelper, err := patch.NewHelper(proxmoxMachine, r.Client)
    if err != nil {
        return ctrl.Result{}, err
    }
    defer func() {
        if err := patchHelper.Patch(ctx, proxmoxMachine); err != nil {
            // Log patch failure
        }
    }()

    // 5. Handle Deletion Lifecycle (Finalizers)
    if !proxmoxMachine.ObjectMeta.DeletionTimestamp.IsZero() {
        return r.reconcileDelete(ctx, proxmoxMachine)
    }

    // Ensure finalizer is registered before creating physical resources
    controllerutil.AddFinalizer(proxmoxMachine, MachineFinalizer)

    // 6. Contract Requirement: Verify bootstrap data has been rendered
    if machine.Spec.Bootstrap.DataSecretName == nil {
        // Bootstrap provider has not finished; wait and requeue
        return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
    }

    // 7. Reconcile Infrastructure (Idempotent External API Calls)
    return r.reconcileNormal(ctx, proxmoxMachine, *machine.Spec.Bootstrap.DataSecretName)
}

func (r *ProxmoxMachineReconciler) reconcileNormal(
    ctx context.Context,
    pm *infrav1.ProxmoxMachine,
    secretName string,
) (ctrl.Result, error) {
    // If providerID is already set, verify VM health and exit
    if pm.Spec.ProviderID != nil {
        pm.Status.Ready = true
        return ctrl.Result{}, nil
    }

    // Fetch bootstrap Secret contents
    // Call Hypervisor API: clone VM template, inject user data
    // Example: vmID := client.CloneVM(pm.Spec.Template, secretName)
    vmID := 105
    nodeName := "pve-node-01"

    // Contract Requirement: Populate providerID matching kubelet format
    providerID := fmt.Sprintf("proxmox://%s/%d", nodeName, vmID)
    pm.Spec.ProviderID = &providerID
    pm.Status.Ready = true

    return ctrl.Result{}, nil
}

func (r *ProxmoxMachineReconciler) reconcileDelete(
    ctx context.Context,
    pm *infrav1.ProxmoxMachine,
) (ctrl.Result, error) {
    // 1. Call Hypervisor API: Stop and delete VM
    // Example: client.DeleteVM(pm.Spec.ProviderID)

    // 2. Once external deletion is confirmed, release finalizer
    controllerutil.RemoveFinalizer(pm, MachineFinalizer)
    return ctrl.Result{}, nil
}

6. End-to-End Walkthrough: Building and Packaging for clusterctl

To allow operators to install your custom provider using standard clusterctl init --infrastructure proxmox, the repository must adhere to the clusterctl packaging specification.

sequenceDiagram
    autonumber
    actor Dev as Provider Developer
    participant Git as GitHub / Git Repository
    participant CLI as clusterctl CLI
    participant Kube as Management Cluster API

    Dev->>Git: Push Git Tag v0.1.0 with Release Assets
    Note over Git: Assets: metadata.yaml & infrastructure-components.yaml

    CLI->>Git: clusterctl init --infrastructure proxmox:v0.1.0
    Git-->>CLI: Download metadata.yaml (contract compatibility)
    Git-->>CLI: Download infrastructure-components.yaml (CRDs & Deployment)

    CLI->>Kube: Verify contract compliance labels
    CLI->>Kube: Apply CRDs (ProxmoxCluster, ProxmoxMachine)
    CLI->>Kube: Apply RBAC Roles, ServiceAccounts & Deployments
    Kube-->>CLI: Controllers established and running

1. The metadata.yaml File

This file declares which CAPI contract version your provider release series implements:

apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Metadata
releaseSeries:
  - major: 0
    minor: 1
    contract: v1beta1

2. Advertising the Contract on CRD Manifests

In your generated CRD YAML (config/crd/bases/infrastructure.cluster.x-k8s.io_proxmoxclusters.yaml), ensure the standard CAPI contract label is present:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: proxmoxclusters.infrastructure.cluster.x-k8s.io
  labels:
    # Signals CAPI Core that this CRD implements the v1beta1 runtime contract
    cluster.x-k8s.io/contract: v1beta1
    cluster.x-k8s.io/provider: infrastructure-proxmox

3. Configuring Local clusterctl Overrides

During local development, test your provider without publishing public GitHub releases by defining a provider override in ~/.cluster-api/clusterctl.yaml:

providers:
  - name: "proxmox"
    url: "/Users/developer/workspace/cluster-api-provider-proxmox/config/default/infrastructure-components.yaml"
    type: "InfrastructureProvider"

Initialize your local provider directly against a test cluster:

clusterctl init --infrastructure proxmox

7. Diagnostics and Failure Modes in Custom Providers

During provider development, unexpected stalls and reconciliation loops are common. Use this diagnostic tree to locate the failure:

flowchart TD
    Start["Investigation: Machine Creation Stalled"] --> CheckFinalizer{"Is Finalizer Added?"}
    
    CheckFinalizer -->|"No"| FixFinalizer["Check Reconciler initialization: controllerutil.AddFinalizer"]
    CheckFinalizer -->|"Yes"| CheckBootstrapSecret{"Is Machine.spec.bootstrap.dataSecretName present?"}

    CheckBootstrapSecret -->|"No"| WaitBootstrap["Normal Wait: CABPK has not rendered secret. RequeueAfter: 10s"]
    CheckBootstrapSecret -->|"Yes"| CheckExternalAPI{"Did call to hypervisor / cloud API succeed?"}

    CheckExternalAPI -->|"Error / Timeout"| HandleBackoff["Return ctrl.Result{}, err (triggers exponential backoff)"]
    CheckExternalAPI -->|"Success: VM created"| CheckProviderID{"Is spec.providerID set on ProxmoxMachine?"}

    CheckProviderID -->|"No"| FixContractSet["Contract Error: Reconciler must set spec.providerID and status.ready"]
    CheckProviderID -->|"Yes"| CheckPatchHelper{"Did patchHelper.Patch() succeed?"}

    CheckPatchHelper -->|"OptimisticLock Error"| FixPatch["Concurreny bug: Reconciler mutated stale struct; re-fetch before patch"]
    CheckPatchHelper -->|"Success"| NodeMatching{"Does Node join workload cluster with matching ProviderID?"}

    NodeMatching -->|"No"| FixKubeletURI["URI Mismatch: Node.spec.providerID != ProxmoxMachine.spec.providerID"]
    NodeMatching -->|"Yes"| Success["Complete: Machine bound to Node"]

The Three Most Common Provider Implementation Traps

Trap 1: The Optimistic Concurrency Patch Collision

If a controller modifies a resource while another routine updates its status, calling r.Update(ctx, obj) will fail with: Operation cannot be fulfilled on ...: the object has been modified; please apply your changes to the latest version.

Solution: Always use the CAPI patch.Helper (sigs.k8s.io/cluster-api/util/patch). It calculates an atomic two-way JSON merge patch between the original object and your modifications, isolating status updates from metadata changes.

Trap 2: Orphaned Cloud Resources on Delete

If your reconciler strips the finalizer before the external hypervisor confirms that the virtual machine has been destroyed, a network failure mid-deletion will orphan the compute instance.

Solution: The deletion function must verify external deletion:

1. Issue hypervisor delete call.
2. Query hypervisor API to confirm resource no longer exists.
3. If still terminating, return ctrl.Result{RequeueAfter: 10 * time.Second}, nil.
4. Only call controllerutil.RemoveFinalizer once 404/NotFound is returned by the external API.

Trap 3: The providerID Formatting Mismatch

If your provider sets spec.providerID = "proxmox://node1/105", but the workload node’s kubelet initializes with --provider-id="proxmox://105", CAPI’s core Machine controller will fail to match them. The Machine will remain in Provisioning phase forever despite the node being completely healthy.

Solution: Ensure the cloud-init script or cloud controller manager (CCM) passes the exact identical string format to kubelet’s --provider-id flag.

8. Key Takeaways

• Providers Are Adapters: Cluster API Core contains no cloud logic. Infrastructure providers translate generic Cluster and Machine declarations into hypervisor or cloud API operations.
• Contract Enforcement via Duck Typing: Core CAPI communicates with your custom CRDs through standardized fields (spec.controlPlaneEndpoint, spec.providerID, status.ready) rather than in-process Go interfaces.
• Always Use Finalizers: Every physical or virtual resource creation must be protected by a finalizer to prevent abandoned infrastructure when Kubernetes objects are deleted.
• Rely on the CAPI Patch Helper: Concurrency conflicts are standard in Kubernetes. Using patch.Helper prevents lost status updates and race conditions.
• The Packaging Contract: Proper integration with clusterctl requires advertising the cluster.x-k8s.io/contract label on CRDs and providing structured metadata.yaml and infrastructure-components.yaml release assets.

Series Conclusion & Next Steps

This module concludes the 16-part Kubernetes Cluster API study journey:

The Complete CAPI Learning Journey:
01. First Principles: Kubernetes Managing Kubernetes
02. Architecture & The Multi-Controller Pattern
03. The Object Model & CRD Hierarchy
04. Building Your First Environment (KinD & Docker Provider)
05. clusterctl Explained: The Management Plane Engine
06. Anatomy of Cluster Creation: End-to-End Traces
07. Worker Hierarchy: Machine, MachineSet, and MachineDeployment
08. Horizontal & Vertical Scaling Dynamics
09. Self-Healing & Automated Remediation with MachineHealthCheck
10. Rolling Kubernetes Upgrades & Immutability Patterns
11. ClusterClass & Parameterized Managed Topologies
12. Dynamic Scaling with Cluster Autoscaler
13. Bootstrap Clusters, Management Clusters, and clusterctl move
14. Running Cluster API in Production: Sizing, HA, and DR
15. Provider Architecture & Provider Contracts
16. Building a Custom Infrastructure Provider

Return to the Kubernetes Cluster API: Study & Learning Journey Index to review earlier modules, explore reference architectures, or revisit operational runbooks.

References & Further Reading

• Cluster API Book: Developing Cluster API Providers
• Cluster API Book: Provider Contracts Specification
• Kubebuilder Book: Building Kubernetes Operators in Go
• Controller-Runtime Documentation: Client, Cache, and Reconciler
• Cluster API Utilities: The Patch Helper Package