Why Inference Matters 12x More Than Training | YC Paper Club

Video summary

YC’s inference panel says serving can use 12× more FLOPs than training, while SGLang’s cache cuts time to first token by 56%.

At YC Paper Club, speakers argued inference deserves far more attention: DeepSeek’s reported serving workload uses about 6× the training FLOPs per year, or 12× across a two-year model lifetime, and inference advances have created $220 billion in market cap. Together AI’s Dan Fu mapped the request path from routing and KV caching through prefill and decode, then detailed kernel optimization, GPU parallelism, and quantization. He said Together’s training-free NVFP4 recipe restored MiniMax M3 performance, enabling cheaper serving. Ollama CEO Jeff Morgan warned that provider quality can vary more than the gap between open and closed models, and that small errors compound across agent tasks. Luminal’s Joe Fioti described compiler search using equality saturation and real-hardware measurements. SGLang’s Liangsheng Yin explained radix-tree cache sharing, overlapped scheduling, and unified memory pools, reporting a 56% reduction in time to first token.

Chapters

1. 0:00Why inference matters more than training: DeepSeek’s inference uses 6× annual FLOPs, or 12× over two years
2. 5:17Why inference matters more than training: cached prefill reaches 1.5 million tokens per second per GPU
3. 8:24Three kinds of inference innovation: MOE, attention kernels, and routing cut serving costs
4. 12:14Dan Fu: the lifecycle of an inference request, from Together’s 400 trillion monthly tokens to GPU serving
5. 16:01Dan Fu: the lifecycle of an inference request, KV-cache routing and prefill-versus-decode scheduling
6. 21:30Dan Fu: the lifecycle of an inference request, from 1.5-terabyte models to speculative decoding
7. 24:23Dan Fu: the lifecycle of an inference request, agent kernel optimization and 72-GPU parallelism
8. 29:26Dan Fu: Together validates quantization and helps MiniMax match FP8 quality with NVFP4
9. 33:35Dan Fu: MiniMax NVFP4 matches original performance; inference spans routing to GPU racks
10. 38:31Dan Fu: Training-free quantization and once-unlikely architectures reach major models
11. 42:18Jeff Morgan: Ollama Cloud tackles provider differences as open models reach 1 trillion parameters
12. 46:27Jeff Morgan: Provider variation can exceed the model gap and compound to 79% task errors
13. 51:17Jeff Morgan: Monitoring open-model endpoints with scoreboards, log probabilities, and router signals
14. 55:32Jeff Morgan: DeepSeek’s DeepInfer and log-probability checks beyond 200,000 tokens
15. 1:00:14Jeff Morgan: private customer evals and Ollama’s local-cloud routing future
16. 1:04:31Joe Fiaty: Luminal’s equality-saturation compiler targets 14–15 ops and 80–90T tokens monthly
17. 1:09:04Joe Fiaty: E-graph equality saturation searches equivalent programs and times them
18. 1:12:51Joe Fiaty: multi-level IR maps GPU operations and selectively builds mega-kernels
19. 1:17:37Joe Fiaty: Luminal benchmarks real hardware across 80T tokens and diverse models
20. 1:21:53Joe Fioti: genetic search profiles kernels in under 10 milliseconds
21. 1:26:22Joe Fioti: forward-pass search fuses kernels while vLLM and SGLang handle decoding
22. 1:30:30Liangsheng Yin: SGLang’s RadixAttention and overlap scheduling for agents
23. 1:37:06Liangsheng Yin: 56% lower TRFT and unified memory for hybrid attention
24. 1:44:22Liangsheng Yin: unified memory pools, compaction, and Miles asynchronous RL

What the full analysis includes

24 chapters · 130 key points · 24 section summaries · mind map

The remaining 127 key points, section summaries and mind map are in the full analysis. Analyze this video to see everything →

This video is 1.9 hours. The Free plan analyzes videos up to 30 min; plans from Plus cover videos up to 2 hours.