Build Hour: AgentKit

The Evolution of Agent Development

Building agentic workflows has historically been a complex, code-intensive process fraught with challenges. Orchestration required manual coding, version updates could introduce breaking changes, and securely connecting tools demanded custom solutions. Furthermore, evaluating agent performance was a disjointed process of daisy-chaining separate systems, and UI development added weeks or months to timelines.

AgentKit is a comprehensive suite of tools designed to streamline this entire lifecycle. It introduces a visual, versioned workflow builder (Agent Builder), a secure connector registry for data and tools, and a customizable chat UI (ChatKit). Crucially, it integrates a powerful evaluation platform (Evals) directly into the development process, featuring automated prompt optimization and support for third-party models.

Building a Go-to-Market Agent with Agent Builder

The core of the development experience is Agent Builder, a visual canvas for orchestrating agentic workflows. A powerful architectural pattern demonstrated is the use of a triage agent to create specialized, more effective systems.

1. Triage and Routing

A “Question Classifier” agent acts as the entry point. Its sole purpose is to analyze the user’s query and route it to the appropriate sub-agent.

• Prompting: The agent is instructed to classify a query into a predefined category (e.g., data, qualification, email).
• Structured Output: Instead of free-form text, the agent is forced to output a structured enum value. This ensures a reliable, machine-readable output that can be used for routing.
• State Management: A “Set State” node captures the classifier’s output (e.g., category = 'data') into a variable. This state is then used by “Condition” nodes to branch the workflow, directing the query to the correct downstream agent.

2. Data Analysis Agent

For queries classified as data, a dedicated agent connects to external data sources.

• Tool Connectivity: Agent Builder can connect to external tools like a Databricks MCP (Message Channeling Protocol) server. Authentication is handled securely, supporting methods like personal access tokens for federated access to protected resources.
• Tool Usage: The agent is given access to specific functions from the connected tool (e.g., a fetch tool). By providing a high-level instruction like “show me the top 10 accounts,” the agent can autonomously construct the necessary SQL query, execute it via the tool, and return the results.
• Iterative Debugging: The visual builder allows for real-time testing and debugging. You can see the model step through the workflow, observe the tools it decides to use, and provide inline feedback to refine the output format (e.g., “I would like the results to be in natural language”).

3. Information Gathering and Email Generation

For qualification or email tasks, the workflow combines multiple agents.

• Information Gathering Agent: This agent uses a built-in web search tool to find publicly available information about a company. It leverages a structured JSON output schema to ensure it extracts specific fields like legal name, employee count, and annual revenue.
• Email Generation Agent (RAG): This agent demonstrates Retrieval-Augmented Generation (RAG). It’s equipped with a file search tool connected to a vector store containing uploaded documents, such as:
  • Standard Operating Procedures (SOPs) for email composition.
  • Marketing materials about a specific campaign or promotion. When tasked with drafting an email, the agent searches this knowledge base to generate a highly relevant and context-aware message.

4. Rich UI Outputs with Widgets

Agents are not limited to text or JSON outputs. They can render rich, interactive widgets. A pre-built widget file can be uploaded to an agent node, allowing it to display complex UI elements like charts, graphs, or formatted email drafts directly in the chat interface.

Deploying with ChatKit

ChatKit is an embeddable and highly customizable chat UI for deploying agents built with Agent Builder.

• Customization: Developers can fully align the UI with their brand guidelines, configuring color schemes, font families, and starter prompts.
• Rendering Widgets: ChatKit seamlessly renders the rich widgets output by Agent Builder, enabling sophisticated user experiences like displaying data visualizations or interactive forms.
• Web Page Interaction: ChatKit can interact with the host webpage’s JavaScript, enabling powerful use cases. The demo showed an agent controlling a 3D globe on a website based on natural language commands, showcasing its portability and deep integration capabilities.

Optimizing and Evaluating with Evals

Evals are critical for ensuring agent reliability and performance in production. The platform is deeply integrated into Agent Builder.

An agent is only as good as its weakest link. You need every single component to be dialed in.

1. Node-Level Evaluation

• Datasets: You can evaluate a single agent node by creating a dataset with test cases (inputs and ground truth values).
• Annotations & Grading: After running generations, you can manually add annotations (e.g., thumbs up/down, free-text feedback). More powerfully, you can create LLM-based graders—prompts that define a rubric for what constitutes a good response (e.g., “the analysis must contain upside and downside arguments”). The grader then automatically evaluates the outputs.
• Automated Prompt Optimization: The platform features an “Optimize” button that takes all the generations, human annotations, and grader feedback to automatically rewrite and improve the agent’s system prompt, saving significant manual engineering time.

2. End-to-End Trace Grading

• Traces: Every agent run emits a detailed trace, showing the execution path and the inputs/outputs of each node (span).
• Grading Rubrics: By inspecting traces, you can identify failure patterns. You can then build a “grader rubric” with multiple criteria (e.g., “web search sources must be first-party,” “end result must contain a buy/sell rating”).
• Evaluation at Scale: This rubric can be run across thousands of traces, allowing you to programmatically identify problematic runs without manually inspecting each one.

Best Practices for Evals

1. Start Simple, Start Early: Begin with a handful of inputs and a simple grader. Adopt an “eval-driven development” mindset.
2. Use Human Data: Real user data is messy and invaluable for capturing edge cases that synthetic data often misses.
3. Invest in Grader Alignment: Spend time annotating and refining your LLM graders to ensure they accurately encode your domain-specific requirements.

• Tags:
• RAG
• Feature
• OpenAI
• Evals
• Agentic Workflows
• Tool Calling
• AgentKit
• Agent Builder
• ChatKit