Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club
Categories: VC, Startup, Design
Summary
AI chip specialization is exploding—training and inference require completely different hardware architectures, and companies are already splitting workloads across specialized chips (Nvidia for prefill, Cerebras for decode). The real untapped opportunity is batch-size-one inference for latency-critical applications like voice agents, where current GPU-optimized designs fail.
Key Takeaways
- Training and inference chips need opposite specs: training requires all-to-all GPU interconnects for gradient passing but doesn't need user proximity (can literally orbit the sun), while inference needs low latency and can shard workloads more efficiently.
- Batch-size-one inference is an overlooked chip design problem—when latency matters critically (voice agents waiting 8 seconds), current GPU designs either waste compute prioritizing throughput or become prohibitively expensive, creating a specialized hardware opportunity.
- The ASIC specialization trend (TPU v8 Zebra/Sunfish) is just the beginning—we're already seeing stack splitting (Nvidia prefill + Cerebras decode), and demand is now sufficient to justify full product introduction cycles for specialized inference chips.
- Intelligence-per-watt and intelligence-per-joule metrics are becoming primary design constraints for hardware, shifting from pure throughput optimization to efficiency-first architectures for cost-constrained deployments.
- There's still massive untapped optimization across three layers: algorithm-side (do we need full inference for simple queries like 1+1?), software/kernel side (CUDA optimization juice), and data center architecture design.
Related topics
Transcript Excerpt
[music] Welcome to uh YC Mountain View special edition of YC Paper Club. I'm thinking about some of the feedback was like kind of making it theme related. So today it's the YC kernel and chip club. So the trend that I see in lab with Stu and John that we talk about all the time is this like specialization that's happening. Uh we saw it with um TPU8 the I and the T version zebra fish and sunfish. Is that what it's called? It's like to make an ASIC you need such activation energy. Um and now and there wasn't enough demand for there to be such a split. But we just see that at the chip level there's going to be massive specialization. The specs that you would need for a training data center are going to be very different than the specs for a uh inference data center. You don't need any bandwid…