How AI covered a human’s paternity leave // Quinten Rosseel

The Communication Channel is a First-Class Citizen

Adoption of an AI agent is not just about the quality of its answers, but how it integrates into the team’s workflow. The initial plan for a web UI was abandoned in favor of a direct Slack integration, which proved to be a key driver of adoption.

• Group Dynamics: Operating within a group chat like Slack fosters a sense of community. When a few curious users ask creative and successful questions, it creates peer pressure and a positive feedback loop, encouraging others to engage.
• Onboarding and Discoverability: Slack’s UI elements allow for custom onboarding workflows, guiding new users on how to effectively query the agent.
• Debugging in the Open: Having customer support present in the channel allows for real-time observation of failures. When the agent provides a wrong answer, the team can immediately see the context, capture it as an evaluation case, and debug the issue, which builds trust with skeptical business users.

Business Logic: Benchmarks Fail, Semantics Succeed

Initial development over-indexed on acing Text-to-SQL benchmarks like BIRD and Spider. However, these benchmarks often contain broken or ambiguous queries, and even top models stall at around 80% accuracy, which is unacceptable for a production system where one in five queries being wrong is a non-starter. The real challenge is not SQL syntax, but encoding the business context surrounding the data.

An early approach using RAG over a repository of “golden” parameterized SQL query templates failed. It resulted in over 100 similar-looking queries, creating more review work for analysts and not aligning with their daily BI process.

The key realization was the need for a semantic layer. This logical layer sits between the user and the physical database, defining metrics, dimensions, and relationships in a way that encodes business logic. This led to a multi-agent system:

• Quick & Deep Agents: These analytics agents interact with the logical layer, not the raw database, to answer user questions.
• Steward Agent: This agent interacts with the physical layer (the raw database) and helps business users build out the semantic layer by suggesting new metrics, dimensions, and KPIs.

Building a semantic layer, once an ambitious project, is now feasible by leveraging open-source projects like Cube or MetricFlow for inspiration. The core lesson is that for Text-to-SQL, context is everything, making the semantic layer the most critical component for accuracy.

Architecture: Pragmatism Over Frameworks

When development began, agentic frameworks were immature. After negative experiences with early versions of LangChain’s abstractions and breaking APIs, the team opted for a “clean code” approach based on Uncle Bob’s principles. This provided more direct control and flexibility, which was crucial as responsibilities were split between the agent and the application layer.

Useful Abstractions:

• Concurrency Control: An event loop and semaphores to prevent agents from being swamped.
• Standardized History: Using the OpenAI message format (system, tool, assistant) for consistency.
• State Management: Storing state in message metadata rather than maintaining separate state objects.
• Factories & Wrappers: Using the factory pattern for agent creation and building wrappers around external APIs to insulate from upstream changes.
• WebSockets: A “necessary evil” for achieving fast response times.

Less Useful Abstractions:

• MCP (Multi-agent Collaboration Protocol): A separate server for tool execution was deemed overly complex.
• ReAct Prompting: Found to be not useful for this specific use case.
• Multi-agent Networks: Rejected due to adding more context and making debugging even harder.

Testing vs. Evals: A Layered Approach

A hot take from the talk is that evals are overhyped for early-stage projects. The focus should first be on building a robust suite of traditional tests.

If you have a failure mode, put it in a test case. If you have a happy path, put it in a test case. A test case ensures your code is cleaner and more testable, and it can be run automatically in your CI pipeline.

The “Swiss Cheese Model” was adopted to visualize testing in layers, where each layer is designed to catch different types of bugs. Simpler bugs should be caught by early, easy-to-implement layers (like unit tests), while more complex, nuanced issues are left for the later, more expensive layers (evals).

For evals, the team built a simple Streamlit app. A key innovation was the Overall Agent Effectiveness score, a composite metric inspired by Overall Equipment Effectiveness in manufacturing. Since different agent trajectories can have different valid end states, this weighted score combines multiple signals into a single, interpretable number. This allows for consistent performance tracking across all possible agent paths, which is crucial for an offline evaluation task like validating SQL correctness.

Latency and Prompting Philosophy

The team moved away from rigid, deterministic workflows towards putting more logic directly into prompts. As LLMs become better at instruction following, prompts become a portable and easier way to manage logic compared to complex, hard-to-test workflows.

Latency Optimization Tricks:

• Progressive Disclosure: Instead of a fixed intent detection step, this logic is incorporated into the main agent prompt.
• Compression: Compressing large schemas and JSON outputs.
• History Pruning: Keeping the message history concise.
• Caching: Caching system prompts and tool definitions is critical.
• Interleaved Thinking: Prompting the agent to include its “thinking” process within the content of a tool call response, saving a separate think tool call.

• Tags:
• Evaluation
• MLOps
• Agents
• Semantic Layer
• System Design
• Text-to-SQL