Self-Play for LLMs, AI for Biology, Formal Verification, and More | YC Paper Club
Categories: VC, Startup, Design
Summary
Alpha Zero-style self-play may be the key to AGI because training only on human solutions (the set H) can't feasibly sample the full solution space F—no amount of test-time compute can overcome this fundamental limitation. The real bottlenecks are intelligence per sample and intelligence per watt, not scale alone.
Key Takeaways
- In-context learning (ICL) performance is non-monotonic: adding more samples causes performance to degrade, hit cliffs at context length limits, then plateau. This suggests humans use a fundamentally different learning algorithm that maintains continuous improvement.
- Different learning strategies are optimal at different data regimes: ICL works best with 1-2 samples, LoRA at low rank for moderate samples, then full SFT becomes optimal. Current systems use one fixed algorithm instead of adapting the learning procedure.
- Training on human-generated solutions creates an upper bound (set H) on achievable intelligence. Even with infinite test-time compute, you cannot feasibly sample F minus H (the solution space humans haven't explored), making unbiased search algorithms more promising than scaling supervised learning.
- Alternatives to backpropagation (SPSA, biologically-plausible learning) may be critical because the brain lacks evidence of transposing weight matrices. These could unlock higher intelligence per sample, matching how humans improve monotonically through repeated experience.
- Intelligence per sample and intelligence per watt are the two major unsolved problems in AI, not pure scale. Smaller models may outperform larger ones from an efficiency perspective, suggesting research should focus on sample efficiency and energy efficiency over parameter count.
Related topics
Transcript Excerpt
Thank you guys so much for coming. This one will have much a much more applied bent based on the feedback. We have a bunch of really cool people that I'll introduce in a second, but we're covering AI for uh biology by my favorite one of my favorite co- researchers, Yas Beg. We have Luke um out of Tatsu's lab talking about selfplay, Alpha Zero style selfplay for LLMs. Super excited about that. Arnob will be uh presenting he's a researcher at Giga on stream rag uh super different application you know thinking about uh real realtime voice uh agents uh Robert George working on lean for science super exciting and then the AI token maxer himself Luke Worthwine cool so I want to introduce some like call for presentations you know maybe inspire some of my interest and maybe inspire some some of yo…