Andrej Karpathy Automated His Home

By No Priors Podcast

Categories: AI, VC

Summary

Andrej Karpathy built an AI agent that autonomously discovered, reverse-engineered, and integrated all smart home systems on his local network without manual configuration. The agent created APIs and dashboards to control lights, HVAC, shades, pool, spa, and security—demonstrating how AI agents can solve real-world automation problems by combining web search, API discovery, and natural language commands.

Key Takeaways

  1. AI agents can autonomously discover connected devices on a local network by searching for API documentation and endpoints, eliminating manual integration work that typically requires developer time.
  2. Reverse-engineering device APIs through agentic behavior is now practical—agents can identify endpoints, test them, and create abstraction layers without hardcoded integrations.
  3. Natural language commands tied to automation routines (e.g., 'sleepy time') create superior UX compared to traditional smart home interfaces, showing product design opportunity.
  4. Agent-generated dashboards and command centers enable visibility across fragmented smart home ecosystems, solving the multi-vendor integration problem at the UI layer.
  5. Local network automation with AI agents reduces privacy/security risks compared to cloud-dependent smart home platforms, a potential competitive advantage for products.

Related topics

Transcript Excerpt

Is there something that you have done personally with your claws beyond software engineering that you think is fun or interesting? >> I have a claw basically that takes care of my home and I call him Dobby the elf claw. Um, [music] and uh, basically I used the agents to find all of the smart home subsystems of my home on the local area network which I was kind of surprised that worked out of the box. Like I just told it that I think I have Sonos at home like can you try to find it? I just logged in and it's like oh yeah you have these Sonos systems installed. I let me try to reverse engineer how it's working. It does some web searches and it finds like okay these are the API endpoints and then it's like do you want to try it? And I'm like whoa like you just did that. And I'm like, "Yeah, c…

More from No Priors Podcast