Tokenless

Build Hour: Agentic Tool Calling

A deep dive into building agentic systems using OpenAI's latest APIs. The session covers the core concept of 'agentic tool calling' (reasoning + tools), outlines a four-part framework (Agent, Infrastructure, Product, Evaluation) for designing long-horizon tasks, and provides a hands-on demonstration of building a non-blocking task processing system with a real-time progress UI.

4 min read
Build Hour: Agentic Tool Calling
source: https://www.youtube.com/watch?v=7E-qdsVEoB8

The paradigm for building with AI is shifting from chat-based interactions to goal-oriented, long-horizon tasks. This evolution is powered by agentic tool calling, a powerful concept that combines a model’s ability to reason with its ability to execute tools and take actions.

The Core Concept: Agentic Tool Calling

Agentic tool calling is the fusion of two key components: reasoning and tools.

  1. Reasoning: Modern models like 01 were trained using reinforcement learning, not on the specific steps to solve a problem, but on the final correctness of the solution. This approach allows a “chain of thought” or reasoning to emerge organically. The model learns to strategize and find its own path to a correct answer.

  2. Tools: By integrating function and tool calling into this emergent reasoning process, models can move beyond thinking to doing. They can fetch information, interact with external systems, and perform actions.

The combination results in agents that are goal-oriented, resourceful, and robust. They can handle failures by course-correcting and maintain consistency over long sequences of actions, sometimes involving tens or hundreds of function calls.

A New Primitive: The Task

This agentic capability introduces the “task” as a new, fundamental abstraction. Building a robust task-based system requires considering four interconnected components:

  • Agent: This defines what the task does. It involves specifying the end-state or goal, rather than the intermediate steps. You provide the agent with a set of tools (functions) it can use to interact with your systems and resources. This is also where you can design for delegation, allowing one agent to kick off other tasks.
  • Infrastructure: This is how the task runs. It involves managing state, handling parallel execution for multiple tasks, providing a secure runtime environment (similar to how Codex has a local repo copy), and implementing strategies for retries and failure handling.
  • Product: This defines the user’s interaction with the task. A crucial aspect is surfacing progress to the user. A user who sees what the agent is doing is less impatient and trusts the output more. This component also involves gathering the necessary context for the agent to succeed, either explicitly from the user or implicitly from the application’s state.
  • Evaluation: This determines how well the task was performed. Unlike traditional chat models evaluated turn-by-turn, task-based agents are evaluated on the final result. This often involves using LLM-based graders with a rubric to assess the quality of the outcome. Tracing and online monitoring are also essential for continuous improvement.

Practical Implementation: Building a Task System

The presentation demonstrated building an agentic system to process a backlog of customer support tickets. The architecture was designed to be non-blocking and scalable.

The chosen architecture consists of:

  1. A front end that initiates tasks and listens for updates.
  2. A back end (built with Flask) with two key endpoints:
    • A POST /tasks endpoint that receives the task description, creates a task object, and kicks off a background worker to run the agent. It immediately returns a task ID.
    • A GET /events endpoint that uses Server-Sent Events (SSE) to maintain an open connection with the front end, streaming real-time updates for all active tasks.
  3. An agent worker that uses the Agents SDK. The SDK simplifies the process of running the agent’s reasoning loop, calling tools, and managing the conversation state.

This decoupled architecture allows the user to start a task and immediately receive confirmation, while the agent runs asynchronously in the background. The UI is kept in sync by listening to the event stream.

A Key Pattern: Surfacing Agent Progress with To-Dos

A powerful pattern for enhancing the user experience is to give the agent tools to report its own progress. In the demo, the agent was given two meta-functions: add_todo and complete_todo.

The agent’s prompt was updated to instruct it to:

  1. First, analyze the user’s request and create a plan by calling add_todo for each step it intends to take.
  2. Then, as it executes each step, call complete_todo to mark it as done.

These function calls update a task object on the back end, and the changes are immediately streamed to the front end via the SSE connection. This creates a dynamic to-do list in the UI, giving the user clear, real-time insight into the agent’s plan and progress.

Advanced Concepts: Delegation

Delegation is the idea of one agent kicking off another, potentially more specialized agent. For example, a primary user-facing agent could delegate a complex research task to a deep_research agent. This is implemented by having the primary agent call a tool that starts a background task. The user can continue interacting with the primary agent in a non-blocking manner while the delegated task runs, receiving an update once it’s complete.

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

Too much lock-in for too little gain: agent frameworks are a dead-end // Valliappa Lakshmanan

Too much lock-in for too little gain: agent frameworks are a dead-end // Valliappa Lakshmanan

Lak Lakshmanan presents a robust architecture for building production-quality, framework-agnostic agentic systems. He advocates for using simple, composable GenAI patterns, off-the-shelf tools for governance, and a strong emphasis on a human-in-the-loop design to create continuously learning systems that avoid vendor lock-in.

Why Language Models Need a Lesson in Education

Why Language Models Need a Lesson in Education

Stephanie Kirmer, a staff machine learning engineer at DataGrail, adapts her experience as a former professor to address the challenge of evaluating LLMs in production. She proposes a robust methodology using LLM-based evaluators guided by rigorous, human-calibrated rubrics to bring objectivity and scalability to the subjective task of assessing text generation quality.