Tokenless

Codex, Behind the Harness — Dominik Kundel, OpenAI

Once GPT 5.3 Codex Spark achieved 1000 tokens/sec on Cerebras, network latency superseded inference as the bottleneck for agents. This talk details how the Codex harness addresses this and other agentic challenges through innovations like WebSocket mode for stateful context, deferred tools for efficient context construction, robust sandboxing (Seatbelt, Bubblewrap, custom Windows solution), and an auto-review subagent to mitigate approval fatigue while ensuring security. It also covers structured actions via 'apply patch' for file edits, shell tools for system interaction, and sophisticated long-horizon goal management, with most distinct features exposed through the open Responses API.

4 min read
Codex, Behind the Harness — Dominik Kundel, OpenAI
source: https://www.youtube.com/watch?v=shRR1e2HXMk

This presentation delves into the advanced architectural features and optimizations of the Codex harness, designed for building robust and efficient AI agents. The speaker, Dominik Kundel, emphasizes that the harness is open source (Apache 2 licensed, written in Rust) and serves as both a blueprint and a usable framework for agent development.

Communication Protocols and Ecosystem

The Codex agent utilizes two primary communication protocols:

  • App Server Protocol: Manages communication between the UI and the harness. It’s an open protocol, powering the official Codex app and enabling third-party integrations like Theos T3 code, Remote X, and Cloud code. This allows developers to build custom UIs on top of the Codex harness.
  • Responses API: Handles communication between the harness and the language model inference. Redesigned from the chat completions API for an “agentic world,” it incorporates advanced capabilities such as web search and image generation. OpenAI has collaborated with partners like Ollama, LM Studio, and Nvidia to codify an open Responses API schema and governance, fostering an ecosystem where any compatible model provider can be plugged into the Codex harness.

Context Construction and Management

Context construction is a critical step, balancing three key concerns:

  • Size: Minimizing token usage and preventing contradictory information in the context window.
  • Flexibility: Ensuring a consistent experience regardless of the number of skills or plugins installed.
  • Cachability: Optimizing for performance and cost-effectiveness.

To manage context size and cost, especially with dynamic elements like tool registries and available skills, the harness employs two main strategies:

  • Deferred Tools: Tools can be marked as deferred, meaning they are not immediately added to the context window. Instead, they are discovered and included only when the model explicitly requires them via a “tool search” mechanism. This feature is available through the Responses API, particularly useful since GPT-5.4.
  • Capping Available Skills: The list of available skills is capped at 2% of the maximum context window. If the list exceeds this limit, descriptions of the skills are progressively trimmed to conserve token space.

Agent Actions and Execution

The harness provides sophisticated mechanisms for agents to perform actions:

  • Asynchronous Actions: The agent can delegate tasks to sub-agents or background terminals, continuing its main work. It uses a spawn agent tool to create new agent instances and a send input tool to interact with them (send content, wait for completion, or shut down). Similarly, it can spin up background terminals and interact with them via standard input.
  • Computer Use: Initially limited, computer use has evolved to allow agents to script their own interactions using code execution. This means agents can write code (e.g., JavaScript or Python) to automate tasks. For browser interactions, Codex leverages a persistent Node.js REPL to execute Playwright-like JavaScript code against a Chromium instance, enabling efficient web scraping and dynamic interaction across turns.
  • File System Interactions: All recent models (starting with GPT-5) are trained to use an apply patch tool for file editing, which accepts diffs to modify or create files. For other file system navigation and searches, the agent uses a shell tool, often employing ripgrep (shipped with the harness) due to its training. On Windows, models are trained to use PowerShell natively.

Sandboxing and Security

All file system interactions are rigorously sandboxed:

  • macOS: Uses Seatbelt.
  • Linux: Uses Bubblewrap.
  • Windows: A custom, open-source sandbox built by the team due to the limitations of existing alternatives.

Addressing Approval Fatigue with Auto-Review

To combat

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

5 Best Practices for Building AI Agent Skills

5 Best Practices for Building AI Agent Skills

This video outlines five essential best practices for developing reliable, secure, and effective AI agent skills. It covers optimizing skill triggering through descriptive metadata, leveraging real-world domain expertise over generic LLM output, managing context windows efficiently by writing lean skills and using progressive disclosure, implementing deterministic logic with scripts for fragile operations, and critically vetting all skills for security vulnerabilities before deployment. These practices are crucial for professionals building robust agentic systems.

Anthropic's CCA Exam as a Field-Guide for Agentic Engineering — Frank Coyle, UC Berkeley

Anthropic's CCA Exam as a Field-Guide for Agentic Engineering — Frank Coyle, UC Berkeley

Frank Coyle demystifies the Claude Certified Architect exam by dissecting key scenarios and highlighting common anti-patterns in Agentic AI design. He provides actionable best practices, emphasizing effective tool use, context management, specialized agent architectures, and cost-saving techniques, all centered on understanding what to avoid to build robust LLM applications.