What If We Stopped Using GPUs? | YC Paper Club

By Y Combinator

Categories: VC, Startup, Design

Summary

GPU efficiency has stalled for two years despite exponential memory demands from transformers; the speaker argues we need fundamentally different compute architectures inspired by the brain's 20-watt efficiency, moving away from backpropagation toward feed-forward learning models that don't require massive VRAM.

Key Takeaways

  1. GPU cost-per-gigaflop improvements have plateaued since 2022, while the human brain operates at 20 watts doing vastly more—creating a $trillion opportunity for alternative compute architectures.
  2. Transformers created hardware-software lock-in: post-2020, Nvidia pivoted from compute efficiency to memory bandwidth/capacity due to attention's O(n²) complexity, creating suboptimal designs for non-transformer workloads.
  3. The brain uses massive cortical inhibition and feed-forward cycles instead of backpropagation—suggesting alternative learning algorithms (Hebbian learning, cyclic graphs) could replace backprop within 10 years.
  4. Training and inference chips are now bifurcating; builders should focus on inference-specific architectures optimized for feed-forward operations rather than universal designs supporting both.
  5. The weight transfer problem in biologically-plausible learning requires weights to be identical across synaptic pairs—solvable with activation functions where both paths see identical signals, enabling non-backprop training.

Related topics

Transcript Excerpt

Okay, welcome to Alternative Compute Club. So, this night started with a dinner with a close friend of mine who um posed this prompt to me. I thought it was a really interesting prompt that you guys can take with you to your dinner parties. Um it was like, if you could only ask the aliens one question, these super intelligent aliens, one question, what would you ask? And so we're thinking about, you know, maybe how do you solve power? And I was kind of thinking, I think we kind of know Dyson spheres or I nuclear or something like that. Um but I thought it was more interesting about how do you how do they compute their flops, their floating point operations. And so, um, I think that we're so far down this rabbit hole of hardware, software co-adaptation, or more specifically, hardware archit…

More from Y Combinator

Featured in