Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

By Sequoia Capital

Categories: VC, Startup

Summary

Core Automation's founders argue the transformer architecture itself is the bottleneck to AGI, not compute or scale—after 6 years of parameter scaling through MoE and attention, the next breakthrough requires rethinking model fundamentals rather than making existing systems cheaper.

Key Takeaways

  1. The industry has mastered two algorithms at scale (pre-training and reinforcement learning) but is trapped in incremental optimization. Focus on architectural innovation rather than efficiency gains to unlock next-generation capabilities.
  2. Use task automation gaps as a research signal: if you successfully use AI for a task but still perform parts manually, that's evidence of fundamental capability limitations worth investigating architecturally.
  3. Distinguish between learning modalities—reinforcement learning (trial-and-error iteration) differs fundamentally from conceptual learning (deep thinking until connections click). Better approaches beyond RL will emerge for leveraging training data.
  4. Understand transformer strengths and weaknesses deeply before attempting replacements. Most architecture work focuses on making transformers cheaper; competitive advantage comes from understanding what problems they solve poorly.
  5. The real tension: models trained in controlled lab environments deployed in complex real-world conditions creates a fundamental mismatch that architecture redesign must address.

Related topics

Transcript Excerpt

If I play football, for example, it looks very very closely to reinforcement learning. I get all a lot of times and every time I adjusted a little bit and I see if it roughly matches what I what I wanted and there are there are there are some self-reinforcement happening. When I learn mathematics, it's a very different type of thing. It's it's like reading about hard concepts and thinking about them very deeply inside my head until things click and I until until I have them connected. And both of those in some way are learning from experience. They are just very different. We probably are spending the most compute than ever on learning from experience, but there reinforcement learning is is not the end of learning from experience and there will be better approaches that researchers will be…

More from Sequoia Capital