Uber Eats Uses AI Agents to Enhance Food Photos at Scale

In the competitive world of food delivery, visual appeal is paramount. For platforms like Uber Eats, the quality of food photography directly impacts user experience and conversion rates. However, ensuring high-quality, authentic-looking images across a vast global marketplace presents a significant challenge, especially for smaller, independent merchants who often lack the time, expertise, or budget for professional photoshoots.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with founding year, headquarters, and a short description from our database.

Uber

Global mobility platform offering ride-sharing, delivery, and logistics services.

Founded

2009

Location

San Francisco, United States

Valuation

$82.0B

At the recent AI Engineer World's Fair, Jay Chopra and Soumya Gupta from Uber's computer vision team shared insights into how they are tackling this problem using AI agents. Their presentation, "Building Closed-Loop Evals for a Multimodal Agent at Uber Scale," detailed a sophisticated system designed to enhance food imagery at scale while maintaining crucial qualities like authenticity and brand integrity.

The Uber Eats Challenge: Visuals at Scale

Uber Eats operates on a massive scale, processing around $90 billion in annual run rate and adding millions of new items to its marketplace each month. Visual content, particularly photos, serves as the first impression for customers. High-quality images can drive clicks and orders, while poor quality can lead to lost business.

The core problem identified by Uber is that many smaller merchants struggle with photo quality due to a lack of time, know-how, or the cost associated with professional photography. This is further complicated by the fact that consumers often distrust AI-generated content, making it a delicate balancing act to improve images without alienating users.

The goals for Uber's agent design were clear: preserve authenticity and trust, selectively improve quality, optimize globally without collapsing marketplace diversity, ship safely, learn continuously, and operate cost-efficiently. Agents, with their ability to generalize and adapt, proved to be a well-suited solution.

Agent Orchestration for Image Enhancement

The system involves a pipeline of agents. It begins with an "Image Understanding and Routing Agent." This agent uses a multimodal approach, leveraging LLMs to describe the image content. This description is then used by a router to decide whether an image needs enhancement or can be skipped. If an image is routed for enhancement, it moves to the "Image Editing Agent."

This editing agent can operate in a loop, receiving feedback from a "QA Agent" to self-correct and refine the image. If an image fails to meet quality standards after several iterations, it is not published. Successful images then proceed to a final "Post-Processing and QA Step" before being published to the menu.

Crucially, Uber emphasizes comprehensive logging throughout the process. This end-to-end logging, structured in a flat JSON format, provides invaluable data for diagnosing issues and rolling up aggregate performance metrics, enabling continuous optimization and the setup of self-learning loops.

Routing Evals and Human Alignment

The routing agent's performance is evaluated using a confusion matrix, measuring precision and recall against human-labeled data. These human labels serve as the golden source of truth, guiding the model's alignment. The team collects representative datasets across various geographies and dish types, providing objective guidelines to human labelers to minimize bias.

Guardrail metrics for routing, such as recall, are critical to ensure that low-quality images are not mistakenly passed through the system. Examples of failures include over-processing high-quality inputs or, conversely, failing to identify mismatches between image content and dish descriptions, which could lead to issues like hallucinated ingredients.

Continuous Learning and Adaptation

To address the dynamic nature of the marketplace and potential model drift, Uber implements online tuning focused on drift detection. Production data is sampled at regular intervals, re-labeled by human annotators, and compared against the agent's output. A "Diagnoser Agent" then localizes mismatches and triggers an auto-tuning pipeline.

This auto-tuning process involves a "Prompt Optimizer Agent" with "Reflect" and "Synthesize" sub-agents. The Reflect agent identifies systemic issues in mismatches, while the Synthesize agent updates the agent's configuration based on this feedback. The newly tuned agent is then benchmarked against the golden dataset before being registered in an agent store, ensuring a continuously improving system without constant human intervention.

The Importance of Pairwise Comparison and Failure Analysis

For generation and enhancement tasks, Uber utilizes pairwise comparison for evaluations, assessing whether a generated image is a better version of the original. This involves criteria such as faithfulness, completeness, naturalness, and realism. The output is a simple yes, no, or unsure, helping to identify and rectify issues like adding unintended elements (e.g., shrimp to a crab dish) or removing crucial components (e.g., sauce).

The team also highlighted the challenge of "reward hacking," where agents might optimize for a metric in a way that doesn't truly improve the user experience or product quality. Analyzing failure modes, including reward hacking and issues with object coherence and physics plausibility, is an ongoing process that involves close collaboration with frontier model teams.

The Future of AI-Powered Visuals

By building these closed-loop evaluation and tuning systems, Uber Eats aims to continuously improve the visual quality of its marketplace, ensuring that merchants' offerings are presented in the most appealing and accurate way possible. This approach, combining intelligent agents, rigorous evaluation, and a commitment to continuous learning, is key to scaling AI solutions effectively in real-world applications.