Andrej Karpathy Automated His Home

By No Priors Podcast

Categories: AI, VC

Summary

Andrej Karpathy built an AI agent that autonomously discovered, reverse-engineered, and integrated all smart home systems on his local network without manual configuration. The agent created APIs and dashboards to control lights, HVAC, shades, pool, spa, and security—demonstrating how AI agents can solve real-world automation problems by combining web search, API discovery, and natural language commands.

Key Takeaways

  1. AI agents can autonomously discover connected devices on a local network by searching for API documentation and endpoints, eliminating manual integration work that typically requires developer time.
  2. Reverse-engineering device APIs through agentic behavior is now practical—agents can identify endpoints, test them, and create abstraction layers without hardcoded integrations.
  3. Natural language commands tied to automation routines (e.g., 'sleepy time') create superior UX compared to traditional smart home interfaces, showing product design opportunity.
  4. Agent-generated dashboards and command centers enable visibility across fragmented smart home ecosystems, solving the multi-vendor integration problem at the UI layer.
  5. Local network automation with AI agents reduces privacy/security risks compared to cloud-dependent smart home platforms, a potential competitive advantage for products.

Topics

Transcript Excerpt

Is there something that you have done personally with your claws beyond software engineering that you think is fun or interesting? >> I have a claw basically that takes care of my home and I call him Dobby the elf claw. Um, [music] and uh, basically I used the agents to find all of the smart home subsystems of my home on the local area network which I was kind of surprised that worked out of the box. Like I just told it that I think I have Sonos at home like can you try to find it? I just logged...