Inferact: Building the Infrastructure That Runs Modern AI

Video summary

Inferact founders explain vLLM’s rise, 500,000-GPU scale, and a universal open-source inference layer

Inferact’s founders trace vLLM from a slow 2022 Meta OPT demo at UC Berkeley into a major open-source inference engine. Its core breakthroughs addressed dynamic LLM workloads through scheduling, memory management, and PagedAttention, unlike static image-model batching. The project now has more than 2,000 GitHub contributors, over 50 regular full-time contributors, and runs on an estimated 400,000–500,000 GPUs continuously. Amazon uses vLLM for its Rufus assistant, while LinkedIn and Character AI have adopted cutting-edge features early. The founders say inference is becoming harder as models exceed one trillion parameters, hardware and architectures diversify, and agents create long-lived, tool-using sessions. Their new company, Inferact, aims to steward vLLM and build a universal runtime spanning models, chips, applications, and deployment environments.

Chapters

1. 0:00Introduction: Inferact’s Universal Open-Source Inference Layer
2. 6:01Introduction: From OPT’s Slow Demo to vLLM’s Scheduling Breakthrough
3. 11:41Community and Collaboration in vLLM: 2,000 Contributors and $100K Monthly CI
4. 19:19Understanding Inference Engines: Tokenized Output, Scheduling, and KV Cache
5. 24:27Cluster Scale and GPU Deployment: 400K–500K GPUs and Agentic Cache Challenges
6. 31:19Belief in Open Source AI: Diversity Across Models, Chips, and Use Cases
7. 35:45Founding of Inferact: A Universal Inference Layer Built Around vLLM
8. 40:00Future of Inference at Scale: Universal Runtime for GB200 and GB300 Systems