The Semantic Layer and AI Agents // David Jayatillake // MLOps Podcast #343

The real battle in the data world isn’t just about AI, but about who controls the semantics. Every organization uses a semantic layer, whether it’s codified in software or exists within a person’s knowledge—a “human semantic layer.” The goal is to move from an inefficient human-in-the-loop process to a governed, automated system.

The Problem with Proprietary Semantic Layers

A common mistake is baking the semantic layer into proprietary Business Intelligence (BI) tools like Looker or PowerBI. While this might work for small companies whose only use case is reporting within that single tool, it creates significant problems as an organization scales.

• Vendor Lock-in: Business logic is defined in a proprietary DSL (like LookML) and becomes trapped within that ecosystem.
• Duplicated Logic: As soon as you need to use this logic elsewhere—for an ML pipeline, a data application, or another BI tool—you are forced to duplicate it.
• Multiple Versions of the Truth: Duplicating logic inevitably leads to inconsistencies, defeating the primary purpose of having a semantic layer to provide a single source of truth.

This approach builds technical debt and restricts data from being open and reusable across the organization.

The Headless, Open-Source Approach

The superior strategy is to abstract the semantic layer out, making it a standalone, “headless” component. Using an open-source semantic layer, like the one from Cube, allows it to be a central hub for data definitions that can be consumed by any tool.

If that logic is locked into one platform then you either have to duplicate the logic across every other platform or you just maintain that separately and then you have different versions of the truth. Either way you end up with different versions of the truth and then you’ve kind of defeated the object of having a semantic layer.

This decoupled architecture enables:

• React applications
• BI tools (Tableau, PowerBI, etc.)
• ML pipelines
• AI-driven applications

While BI vendors have little incentive to integrate with external semantic layers because they prefer customer lock-in, workarounds exist, such as emulating the vendor’s native API (e.g., pretending to be a DAX or MDX API for Microsoft tools).

Data Modeling vs. The Semantic Layer

Data modeling and the semantic layer are tightly linked but distinct concepts.

• Data Modeling: This is the process of structuring and arranging organizational data to clearly describe business entities and activities. Methodologies include Kimble, Data Vault, or even a “one big table” approach common in data science.
• The Semantic Layer: This layer sits on top of the data model. It describes the model in business terms: how entities join, how metrics are aggregated, and which fields are dimensions. It exposes a simple API (e.g., REST, GraphQL) that allows users to request data using business terms without writing SQL or understanding the underlying complexity of the data model.

Good data modeling makes building a semantic layer much easier. Conversely, thinking about how you want to query the data (the semantic layer) should dictate how the underlying data model is built.

The Rise of AI Agents in the Data Pipeline

AI agents have the potential to automate large portions of the data engineering workflow, creating a “Henry Ford model” of specialized agents for each stage of data production.

• Ingestion: Building data connectors is a repetitive task. An agent can be instructed to “connect to Salesforce and pull the data into an S3 bucket,” automating a process that data engineers repeatedly perform manually. The primary challenge is permissions, which can be handled by a one-time, human-guided setup.
• Transformation: Much of the transformation work, especially from raw to staging data (deduplication, casting, handling nulls), follows a standard recipe and can be automated by an agent.
• Semantic Layer Generation: It is highly feasible to use an agent to build a semantic layer from a data model. This has been done successfully even with poorly defined models, where the agent learns the relationships by observing existing queries. A key finding was that parsing raw SQL into a structured format (like JSON) that explicitly defines joins and entities provides a significant performance lift for the LLM compared to feeding it raw SQL.
• Querying: Once the semantic layer is built, an agent can query it to answer natural language questions. This is preferable to having an LLM write raw SQL for every query, as the semantic layer provides a governed, reviewed, and reusable interface.

Organizational Intelligence and LLMs

As companies roll out enterprise LLMs like ChatGPT, there is a growing demand to connect them to internal data sources. The semantic layer is the ideal interface for this.

• Governed Access: By connecting an LLM to a semantic layer, employees can query and analyze data using natural language within a governed framework. This prevents the proliferation of random CSV uploads and ensures everyone is working from the same definitions of truth.
• Metadata and Insights: Analyzing the interactions between users and the semantic layer via the LLM can provide valuable metadata. You can discover what questions are being asked, what data points are being joined, and how different departments (e.g., Marketing vs. Finance) define the same metric, like “revenue.”
• Shared Memory: The concept of “memory” in LLMs can be leveraged across an organization. If one user refines a vague question into a precise, useful query, that refined logic can be suggested to the next person who asks a similar question, fostering transparency and a shared understanding of data.

Semantic Layers vs. Feature Stores

While semantic layers and feature stores are conceptually similar in that they both describe entities, they serve different purposes and have a key distinction.

• The Key Difference is Temporality: A feature store is fundamentally concerned with point-in-time correctness. It must be able to provide the state of a feature at a specific historical moment to prevent data leakage during model training. This is not a primary concern for a BI-focused semantic layer.
• Other Differences:
  • Scope: A semantic layer aims for a canonical, 360-degree view of an entity. A feature store is more selective, containing only the specific features needed for a model.
  • Clarity: Metrics in a semantic layer are designed to be obvious and human-readable. Features in a feature store can be abstract and unintelligible without context (e.g., E79).
  • Mutability: A semantic layer strives for one evolving, canonical definition of a metric. In a feature store, you might have multiple versions of a feature (E79, E80) being used simultaneously by different models.

• Tags:
• AI Agents
• LLM
• Feature Store
• Data Engineering
• Semantic Layer
• Data Modeling
• BI
• Cube.dev