Why Inference Matters 12x More Than Training | YC Paper Club

By Y Combinator

Categories: VC, Startup, Design

Summary

Inference costs dominate AI economics 12x more than training over a model's lifetime, yet most papers optimize for training efficiency instead. DeepSeek's 80% margins prove that inference optimization—not raw model size—determines SaaS profitability for AI companies.

Key Takeaways

  1. Inference flops matter 6x more annually than training flops (43,000 H800 GPU hours/day vs 2.8M hours to train DeepSeek V3), scaling to 12x over a 2-year model lifetime. Most research papers focus on training optimization, but should prioritize inference flop reduction.
  2. KV cache hit rates dramatically impact throughput: DeepSeek achieves 1.5M tokens/sec with 96% cache hits vs 60K tokens/sec without caching. Traffic patterns matter—agent workloads have high cache reuse while new chat sessions require full prefill restarts.
  3. Decode (token generation) is the actual bottleneck, not prefill. Throughput drops 100x from prefill (1.5M tokens/sec) to decode (14K tokens/sec), making decode optimization the primary lever for serving cost reduction and latency improvements.
  4. GPU utilization (Model FLOPs Utilization/MFU) is far lower than theoretical capacity: ~50% during training/prefill, dropping to ~5% during decode. This gap creates massive optimization opportunities for inference infrastructure engineers.
  5. Track four inference KPIs religiously: time-to-first-token (prefill latency), time-per-output-token (decode speed), throughput (tokens/GPU hour), and goodput (tokens at SLA). DeepSeek's metrics (400ms TTFT, 1s P99, 14K tokens/sec decode) set competitive benchmarks.

Related topics

Transcript Excerpt

[music] >> How's everyone doing today? Woo! All right, it's inference night. Come on, you got to get excited. Okay, I didn't really know how to uh to represent an inference in a pictorial, so I did Halloween. So, I made it uh spookier, as my 3-year-old would say, pookie. Um this is my wife over here. Okay, so why inference? Who cares about inference? Um well, I think we do a lot of inference investments at YC. We look at this a ton. Um I used to run an AI company where I would train and we would do a lot of inference, and inference was always the bottleneck. It's so hard to do inference well at scale and without losing your shirt on cost of goods sold, uh a term called COGS, as some people in software are unaware what that word means. Um and so far, in just the last 10 years, it's produced…

More from Y Combinator