Uber’s AI infra, from the inside

By Pragmatic Engineer

Categories: Product, Startup

Summary

Uber built ~20 internal AI systems stacked on battle-tested infrastructure, revealing that production-ready agentic code requires far more than just an AI coding harness—the real complexity lies in permissions, monitoring, and traffic replay. This approach enables thousands of engineers to ship AI features safely.

Key Takeaways

  1. Unified API gateway abstracts all model vendors (OpenAI, Anthropic, open-weight models) behind one compatible interface, simplifying multi-vendor AI adoption at scale across thousands of engineers.
  2. MCP gateway (130+ edge types, 100+ node types) unifies all tool integrations similarly to model gateway, reducing context switching and enabling standardized agentic workflows across the organization.
  3. Capture-replay system enables production traffic testing by intercepting real data with middleware, allowing engineers to validate AI agents against actual user patterns without manual test case creation.
  4. AI Guard prevents sensitive data leakage to third-party APIs—critical compliance layer that most companies skip when rushing to integrate external AI models, especially in regulated industries.
  5. Retrofit strategy: layering AI on existing infrastructure (RT API, M3DB, Cadence) proves easier than rebuilding; this suggests founders should prioritize AI-first architecture design from day one to avoid technical debt.

Related topics

Transcript Excerpt

What does development look like today? What kind of tools do you use? >> We expose one OpenAI compatible API or anthropic compatible API and all of the models across all the vendors open weight or closed weight. They're all behind this gateway. >> What is AI guard? >> You know, we wanted to make sure we are not sending any sensitive content back to the user. Around April of 2025, we started working on MCP gateway. Similar to model gateway, this is one gateway for all of the tools. We have 100 plus node types, 130 plus edge types and this is live data as of say like 3 days ago. We call it slate. Essentially you can create your own instance that deploys the branch that you're working on. For some of our most complex platforms we call it capture replay like essentially you can capture real pr…

More from Pragmatic Engineer