Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

By ai.engineer

Categories: AI, Tools

Summary

Uber Eats built a closed-loop multimodal agent system to improve food photography at scale across 90 billion run-rate marketplace without triggering user distrust of AI-generated content. The key: agents with constrained creativity, continuous self-correction loops, and comprehensive logging for cross-functional diagnosis.

Key Takeaways

  1. Design agents on a spectrum between deterministic rules-based systems (brittle but controlled) and unconstrained creative agents (scalable but risky). Balance using safety guardrails and clear decision boundaries.
  2. Implement three-stage evaluation: image understanding routing agent (describe + structure), editing agent with QA feedback loops for self-correction, then final post-processing QA before publishing.
  3. Log all agent outputs in flat JSON structure accessible to non-technical and technical teams alike, enabling both individual case diagnosis and aggregate trend analysis across the entire pipeline.
  4. Solve the authenticity trust problem by preserving original image fidelity, avoiding prompt uniformity across items (prevents marketplace visual collapse), and selectively improving quality rather than blanket enhancement.
  5. Operate at global scale with long-tail quality distribution—require agents to handle everything from poor sharpness/composition to diverse user-generated content without cannibalizing smaller merchants.

Related topics

Transcript Excerpt

[music] >> My name is Jay and I'm here with Sonya. We are part of the computer vision team at Aruba. We're going to talk to you about a real world production use. Oh, my son done. Okay. Try again. Okay. Don't worry. I'll I'll manage. You hear me now? Okay, so we're going to talk to you today about a real world production use case and specifically we're going to dive into how we design the e-bows and the e-bow loops. So All right, cool. So just before we get into the agent design, we're going to talk about a little bit about the use case. So our delivery marketplace Uber Eats, we do about 90 billion run rate per year at the moment. We were adding millions of items to the marketplace each and every year. Sorry, every every month. We're growing at 20% year-on-year and and we operate in 10,000…

More from ai.engineer