AI traces are worth a thousand logs

The architecture of an AI agent can be simplified to three core components:

• Instructions: The system prompt that dictates how the agent should behave.
• Tools: External functions the agent can call to perform actions beyond text generation, requiring the agent to understand a specific tool-calling syntax.
• Model: The underlying Large Language Model (LLM) that powers the agent’s reasoning.

This simplified model is implemented across virtually all agent frameworks through a fundamental execution loop. The process begins with an initial history of messages (instructions and the user prompt). Then, in a while loop, the LLM is called with the current history. The system checks if the model’s output is a request to use a tool. If so, the tool is executed, its result is appended to the message history, and the loop continues. This cycle repeats until the LLM generates a final answer without a tool call, at which point the loop terminates.

Even advanced concepts like memory, knowledge retrieval (RAG), and multi-agent systems can be seamlessly integrated into this paradigm. Instead of being complex, internal framework features, they are best implemented and exposed as standard tools, allowing the agent to decide when and how to use them.

What is a Trace?

A trace provides a complete, structured narrative of an agent’s execution, moving beyond simple, unstructured logs. Following the OpenTelemetry standard for generative AI, a trace is defined as a sequence of JSON objects. Each object, called a “span,” represents a single, distinct operation within the agent’s execution loop.

For example, a span is created for each LLM call or tool execution. This span captures critical data:

• The input arguments to the operation (e.g., the prompts sent to the LLM).
• The output of the operation (e.g., the LLM’s response or the tool’s result).
• Additional metadata relevant to the step.

The goal of projects like any-agent is to standardize this process, ensuring that a trace and its constituent spans have a consistent structure regardless of the underlying framework (e.g., LangChain, LlamaIndex, Google’s SDK).

From Visualization to Programmatic Control

While many platforms like Weights & Biases or LangSmith offer powerful UIs to visualize these traces, their real power lies in programmatic access. The fancy dashboards are merely rendering the underlying JSON trace data.

The key insight is to return the complete trace object directly to the developer in their Python code upon an agent’s run. This allows for direct inspection and manipulation of the agent’s entire execution history within the same context it was run, without relying on external services.

This approach unlocks powerful new workflows for building reliable agents. By having the trace object available, developers can write precise and robust tests for agent behavior. You can move from simple input/output checks to asserting the agent’s internal process:

# Pseudocode for testing with a trace object
trace = agent.run("What is the capital of France?")

# Assert that a specific tool was used
assert trace.did_call_tool("search_web")

# Assert the arguments passed to a tool
assert trace.get_tool_call("search_web").arguments == {"query": "capital of France"}

This capability is transformative for debugging and regression testing. It enables the creation of “golden datasets” where not just the final answer, but the entire chain of thought and tool usage is validated. This ensures that as the agent, its tools, or the underlying models evolve, its behavior remains predictable and correct.

• Tags:
• AI Agents
• MLOps
• Testing
• Observability
• Debugging
• Tracing
• OpenTelemetry