Enterprise Agents Have a Structure Problem - Ishita Daga, Tesla

The talk, “Enterprise Agents Have a Structural Problem,” by Ishita Daga, a Senior Machine Learning Engineer at Tesla, delves into the fundamental reasons why enterprise AI agents frequently fail in production environments. Daga argues that common attempts to resolve these failures—such as employing larger models, increasing context windows, or adding more knowledge bases (RAG)—are often misguided. Instead, the core issue lies in the agent’s inability to understand the nuances of business-defined data, necessitating a robust semantic retrieval infrastructure and machine-readable metadata layer.

Core Problems Leading to Enterprise Agent Failure

Daga identifies three primary structural challenges for enterprise agents:

1. Ambiguity: Agents often struggle to identify the correct data source, table, column, or knowledge base. This arises because agents typically treat all available information equally, lacking a structured hierarchy to determine the most authoritative or relevant “source of truth.” This leads to confusion about which data is most accurate or appropriate for a given query.
   • Proposed Solution: Hierarchical Sources of Truth: To combat ambiguity, Daga suggests implementing a hierarchical system for knowledge sources, prioritizing from the cleanest and least flexible to the messiest yet most dynamic.
     • Semantic Layer: Positioned as the highest-quality source, this layer comprises highly curated KPI definitions, metric calculations, and critical business definitions. An agent leverages this layer to identify the most relevant KPI and derive accurate, business-aligned answers.
     • Canonical Tables/Parametric Queries: This intermediate layer offers more flexibility. Agents are provided with a collection of parametric queries or table structures, allowing them to construct more tailored queries and filters, thus addressing a broader spectrum of questions.
     • Database Graph (Metadata Graph): While offering the highest flexibility, this is also the most complex to build and maintain. It involves intricately mapping each database table to the specific metrics it can answer, and vice versa. This creates an expansive graph that agents can navigate for highly complex queries. Daga advises a phased implementation, starting with the easier-to-establish Semantic Layer and Canonical Tables to solve approximately 80% of problems, before investing in the maintenance-heavy Database Graph.

2. Staleness: The rapid evolution of business context, including updated KPIs, definitions, and processes, quickly renders agent knowledge obsolete. Relying on static documentation (e.g., .md files) is unsustainable for maintaining accuracy.
   • Proposed Solution: Context Life Cycle: Daga proposes a dynamic system comprising two key elements:
     • Embedding Live Data Sources: Integrating data sources that are continuously updated, curated, and reviewed. Examples include GitHub, CRM systems, Tableau dashboards, or dbt semantic layers. These sources provide the most current and reliable information.
     • Feedback Loop: A critical, yet frequently overlooked, component. This involves logging every pertinent event—such as corrections to data definitions, new metric calculation methods, or required filter updates. These logged events are then used to dynamically update the agent’s context. Coupled with this, a robust evaluation mechanism (either human-annotated test suites or automated comparisons of agent outputs against ground truth) is essential to monitor agent performance and guide continuous context updates.

3. Preference: This is identified as a highly subjective and unresolved challenge. Different teams or individuals often have unique preferences for calculating metrics, defining terms, or applying filters, even for the same underlying concept. For example, two teams might calculate “average milestone time” using different start and end points.
   • Current Limitations and Future Directions: While a semantic layer can store various calculation methods, it doesn’t inherently understand which method an individual user prefers. Agent memory mechanisms can store preferences but struggle to differentiate between similar metrics or contextual application. Daga emphasizes that the ultimate solution requires routing the agent to the appropriate metric or definition based on the specific user or team interacting with it, effectively embedding individual preferences into the agent’s reasoning. This problem remains an active area of research.

In conclusion, Daga underscores that achieving reliable, production-grade enterprise agents hinges on addressing these structural issues through sophisticated context management, a well-defined semantic infrastructure, and innovative approaches to incorporating individual user preferences.

• Tags:
• AI Agents
• RAG
• Enterprise AI
• Context Management
• Data Quality
• Semantic Layer
• LLM Applications
• Data Governance
• Machine Learning Engineering
• Metadata Management
• Business Logic