I'm Obsessed With Local AI. Here's Why

By Greg Isenberg

Categories: Startup, Product

Summary

Local AI running on your own hardware creates massive business opportunities over the next 24 months that most founders are missing. The key isn't whether local models match cloud AI capabilities—it's whether they're "good enough" for the job while solving privacy, latency, and offline needs that make the product better.

Key Takeaways

  1. Stop asking 'Is this model smarter than cloud AI?' and start asking 'Is this model good enough for the job and does running it locally make the product better?' This reframe unlocks hidden business opportunities.
  2. Local AI makes sense for five specific use cases: private/sensitive data workflows, offline usage, field work, low-latency requirements, and internal workflows that repeat. These are the product validation signals to build around.
  3. The local AI stack has four core pieces: models (Gemma, Llama, Mistral), model warehouse (Hugging Face), software runners (LM Studio, Ollama), and the product workflow you build. Start with LM Studio for non-technical users or Ollama for builders.
  4. When evaluating models on Hugging Face, focus on five questions: What is it for? How big is it? What license? What hardware runs it? What capabilities (text, images, audio, embeddings)? Quantized versions are already available to ease local deployment.
  5. A smaller model in the right place can be very valuable. For Apple silicon, use MLX. For Google ecosystem on-device apps, use Google AI Edge and Lite RTLM. For general inference, llama.cpp powers most local model inference.

Related topics

Transcript Excerpt

I think local AI and open models are going to create a ridiculous number of business opportunities over the next 24 months and I don't think most people actually have the map yet. >> [music] >> They've used ChatGPT, they've used Claude, but when they hear local AI, Hugging Face, Ollama, LM Studio, AI Edge, it sounds like it's for this developer world and that normal founders are just not supposed to touch it. And I think that's a mistake because the opportunity here is actually pretty endless. By the end of today's episode, you're going to understand what local AI is, when it matters, [music] how to run open models at work, where Hugging Face fits in here, which Gemma model I'd start with, how I'd run a model locally with LM Studio or Ollama, and how this turns into real business ideas. An…

More from Greg Isenberg

Featured in