Build Hour: Built-In Tools

Built-in tools for the Responses API provide models with new capabilities out-of-the-box, allowing them to interact with data, perform complex tasks, and take actions without requiring developers to write or host custom functions. This approach minimizes coding effort and leverages OpenAI’s infrastructure and expertise.

Built-in Tools vs. Function Calling

A key distinction lies in where the execution happens.

• Function Calling is a three-step process managed by the developer:

  1. The model decides to call a function and returns the function name and parameters to your application.
  2. Your code executes the function on your own infrastructure using the provided parameters.
  3. You send the result of the function call back to the model, adding it to the conversation history so it can generate a final response.

• Built-in Tools streamline this process significantly:

  1. You specify which built-in tools the model can use in your API call.
  2. When the model decides to use a tool, it is executed automatically on OpenAI’s infrastructure. The results are added to the conversation context transparently.
  3. The model immediately generates a final response using the tool’s output, all within a single turn.

This removes the middle execution step for the developer, reducing complexity and latency.

Available Built-in Tools

As of the presentation, six tools are available:

• Web Search: Gives the model real-time access to the internet, overcoming the knowledge cut-off date (May 2024 for current models). This allows the model to answer questions about recent events and access up-to-date information.
• File Search: Provides a managed Retrieval-Augmented Generation (RAG) system. Developers can upload files to a vector store, and OpenAI handles the entire RAG pipeline, including pre-processing (chunking, embedding) and retrieval (finding and reranking relevant information to inject into the model’s context).
• MCP (Model Context Protocol): Allows the model to interact with any remote MCP server, granting access to hundreds of third-party applications and functions (e.g., Shopify, Stripe, Zapier). It acts as a standardized way for APIs to expose their capabilities to LLMs.
• Code Interpreter: Enables the model to generate and execute Python code in a sandboxed environment on OpenAI’s infrastructure. It is highly effective for data analysis, solving mathematical problems, and generating outputs like charts and files.
• Computer Use: A unique tool that allows the model to interact with computer interfaces visually, suggesting actions like “click on that button” or “scroll down.” The developer is still responsible for executing these actions in their own computer environment.
• Image Generation: Integrates image creation capabilities.

Practical Application: Building a Data Exploration Dashboard

A live demo showcased how to integrate these tools to build a powerful data exploration dashboard for a fictional wellness store. The goal was to unify and query data from multiple sources: online sales from a Stripe account and in-person sales from a CSV file.

Architecture and Implementation

The application used a Next.js frontend and a backend that called the Responses API, configured with multiple tools. The process highlighted several key concepts:

1. Rapid Prototyping with the Playground: The initial experimentation was done in the OpenAI Playground, which allows for visual configuration of tools. The “View Code” feature was then used to directly copy the necessary tool configuration into the application code, bridging the gap between testing and implementation.

2. Hybrid Tool Approach: The demo illustrated a powerful pattern of combining built-in tools with custom function calling:

   • Built-in Tools for Data Handling:
     • The mcp tool was used to connect to a Stripe MCP server to fetch live sales data.
     • The code_interpreter was given access to the store’s offline sales CSV file, allowing it to perform analysis and calculations.
     • The web_search tool was enabled to find external data, such as local competitors.
   • Function Calling for UI Integration: A custom function named generate_component was defined. The model was instructed to call this function to display data in the frontend. Using structured outputs, the function defined schemas for different UI components (e.g., a card for a single metric, a chart for time-series data, a table for lists). When the model analyzed data using Code Interpreter or fetched it via MCP, it would then call generate_component with the correctly formatted data to populate the dashboard dynamically.

3. Multi-Turn, Multi-Tool Orchestration: The final application demonstrated the model’s ability to handle complex, multi-step queries. For instance, a user could ask for a summary of the last week’s payments, and the model would:

   1. Call the Stripe mcp tool to list recent payment intents.
   2. Use the code_interpreter to process this data, calculating totals and failure rates.
   3. Call the custom generate_component function multiple times to display the results in different cards and tables in the UI.

This showed how the Responses API manages the conversation state, allowing the model to intelligently chain multiple tools together to fulfill a complex user request.

Customer Spotlight: Hebbia

Will, the technical lead at Hebbia, explained how they leverage built-in tools for their information retrieval platform for financial and legal services.

• Web Search for the “Explore-Exploit” Strategy: Web search is critical for overcoming the model’s knowledge cutoff. Hebbia employs an “explore-exploit” strategy: first, they use web_search to broadly explore a topic and gather up-to-date context (e.g., on new government tariffs). Then, they use this fresh context to “exploit” their private, internal knowledge bases with more targeted and informed deep searches.
• Custom MCP Implementation: While they find MCPs powerful, Hebbia uses a custom implementation. They build their own high-performance, opinionated data indexing systems and then wrap them in an MCP server for the model to interact with, rather than relying on off-the-shelf MCPs. This gives them fine-grained control while still benefiting from the standardized protocol.

• Tags:
• Agentic AI
• RAG
• Feature
• MCP
• Code Interpreter
• Function Calling
• Responses API
• Built-in Tools
• Web Search
• File Search