ai-infrastructure

Kubernetes has spent years getting very good at answering one fundamental question: Where should this Pod run? For a large class of traditional applications, that is exactly the right question. A stateless Deployment with ten API replicas usually does not care if replica seven starts 20 seconds after replica six. Each Pod makes progress independently. The scheduler evaluates them one at a time, finds a feasible Node, binds the Pod, and moves to the next item in its queue....

Walk into any AI infrastructure discussion and you’ll hear the same question: What’s your GPU utilization? It’s become the infrastructure equivalent of asking a web service for its CPU utilization. The assumption is simple: higher utilization is better. After all, GPUs are expensive, and a cluster running at 90% utilization sounds far more impressive than one running at 50%. For a long time, I believed that too. Then I spent more time working on GPU scheduling and multi-tenant AI workloads....