Tesla ML Engineer on Enterprise Agent Problems

Enterprise agents, powered by advanced AI models, are increasingly being adopted across organizations. However, according to Ishita Daga, a Machine Learning Engineer at Tesla, these agents often falter due to fundamental structural issues rather than limitations in model size or data availability. In her talk, "Enterprise Agents Have a Structural Problem," Daga outlines three key challenges: ambiguity, staleness, and preference, and proposes solutions for each.

The Core Problems with Enterprise Agents

Daga begins by debunking the common assumption that a larger or more context-rich model, or simply adding more data sources like documents and plugins, will automatically improve an enterprise agent's performance. She identifies three primary issues:

• Ambiguity: Agents struggle to determine the correct data source, table, or column to query when multiple options exist, and cannot easily identify the definitive source of truth.
• Staleness: The rapid pace of change in business definitions, KPIs, and processes leads to outdated information within agents, making their responses inaccurate over time.
• Preference: Capturing individual or team-specific preferences for metrics, query logic, or filters is a significant challenge, as different groups may interpret the same request differently.

Daga stresses that while these issues are complex, they can be addressed through better structural design.

Structuring the Source of Truth

To combat ambiguity, Daga proposes a hierarchical approach to data sources. The idea is to prioritize cleaner, less flexible sources over messier, more dynamic ones when answering queries. This hierarchy consists of three layers:

• Semantic Layer: This is the most curated and reliable source, containing single, governed definitions for all metrics and KPIs, along with canonical queries. An agent would first consult this layer.
• Canonical Tables: This layer includes a list of parametric tables or queries that offer more flexibility. The agent can use these to construct its own queries or apply filters when a direct semantic match isn't found.
• Database Graph: The most complex layer, involving connecting tables to metrics and vice versa to create an extensive graph. This provides the most flexibility but requires significant effort to build and maintain.

Daga recommends starting with the semantic layer and canonical tables, as they are easier to set up and can solve a majority of the problems, with the database graph addressing the remaining 20%.

Tackling Staleness with a Context Lifecycle

The problem of staleness is addressed by implementing a robust context lifecycle. This involves two key components:

• Embedding Live Data Sources: Utilizing data sources that are continuously updated and curated, such as GitHub repositories, CRM tools, or BI platforms like Tableau and DBT.
• A Feedback Loop: Establishing a system to capture and log user feedback on agent responses. This includes noting when data or definitions are incorrect, outdated, or when new calculation methods or filters emerge.

This feedback is then used to evaluate the agent's performance, either through human annotation or automated comparison of past and current answers. This iterative process of logging, evaluating, and updating is crucial for maintaining agent accuracy.

The Unsolved Challenge of Preference

The final problem, preference, is described as the most challenging and open-ended. Daga illustrates this with an example of calculating 'average milestone time,' where different teams might use distinct methodologies, leading to different results. The core issue is understanding the user's intent and preferred calculation method.

While potential solutions like storing preferences in a semantic layer or utilizing agent memory exist, they do not fully address the problem. Daga suggests that the ultimate goal is a system that can route agents to the appropriate metric or definition based on the user's identity or team. This, she notes, requires further research and could involve creating a 'hive mind' for data agents.

In conclusion, Daga emphasizes that the key to improving enterprise agents lies not just in providing more data, but in building a better structure for managing that data, ensuring its accuracy through continuous feedback, and ultimately, understanding and adapting to user preferences.