Code Mode - Sunil Pai, Cloudflare
Sunil Pai from Cloudflare introduces "Code Mode," a paradigm where AI agents generate and execute code (like JavaScript) instead of using traditional JSON-based tool calling. This approach enables more efficient, stateful, and complex interactions with large-scale systems by leveraging the inherent capabilities of programming languages.
Traditional tool calling, based on exchanging JSON objects, becomes inefficient and cumbersome at scale. When an AI agent needs to interact with hundreds of tools, the context window quickly fills up, composition becomes difficult, and the constant back-and-forth communication with the model introduces significant latency.
A Shift to “Code Mode”
A more efficient and powerful approach is to have the model generate executable code, such as JavaScript, that runs against a defined environment. This method, which we call “Code Mode,” leverages several key advantages:
- Leverages Existing Training: Large language models are already trained on vast amounts of code, making them naturally proficient at generating it.
- Efficiency: Instead of multiple round trips for a series of actions, the model can generate a single script that accomplishes a complex task—including loops, state management, and parallelization—in one execution.
- Robustness: Code comes with inherent benefits like typed APIs and syntax checking, which can be validated before execution.
Case Study: Taming the Cloudflare API
The Cloudflare API surface is immense, with approximately 2,600 endpoints. Representing each as a distinct tool would require about 1.2 million tokens in the initial prompt, making it completely infeasible.
To solve this, we exposed just two tool calls: search and execute. Both accept a string of code as input.
search: The model writes code to query the entire OpenAPI specification to find the relevant endpoints.execute: The model then writes another piece of code to call the discovered endpoints with the necessary parameters.
This approach reduced the initial context from over a million tokens to just around 1,000—a 99.9% reduction. For a user experiencing a DDoS attack, they can simply state their problem in natural language (“We are getting DDoS’d. Find every offending IP and block them.”), and the agent can generate and execute the necessary code in a single, efficient shot, bypassing the need to navigate a complex dashboard during a crisis.
Deeper Implications: Inhabiting the State Machine
This is more than just an optimization; it represents a new way for AI to interact with systems. LLMs can now act as a “programmer buddy” for every user, translating natural language intent into executable actions.
A powerful example of this emerged from an internal experiment. Kenton Varda, creator of Cloudflare Workers, prompted an agent to create a drawing canvas. After it generated the canvas, he drew a tic-tac-toe board and asked the model to play with him. Instead of generating a separate tic-tac-toe application, Kenton instructed it to work with the existing system. The model’s instructions were: “You have access to the entire state of the system… an array of strokes… Inspect that and play it with me.”
The model analyzed the array of stroke data, recognized the pattern as a tic-tac-toe board, and proceeded to play by generating the stroke data for its own moves (drawing a circle). No tic-tac-toe game logic was ever explicitly programmed. The model was not running a program; it was inhabiting the state machine of the existing application.
The “Harness”: An Architecture for Safe Execution
This new paradigm requires a specific architecture: a “harness” or sandbox where AI-generated code can be executed safely. This sandbox must have specific attributes:
- Zero-Capability Start: It begins with no permissions—no network access, no file system access.
- Explicitly Granted Capabilities: Functionality is granted explicitly through APIs passed into the environment. All outgoing network calls should be controlled.
- Fast and Ephemeral: The environment must be able to start up and shut down almost instantly. V8 isolates are an excellent candidate for this, offering speed and a decade of security hardening.
- Full Observability: You must have complete insight into what code was run and why, enabling you to trace any action back to its origin.
The Future: Generative UI and Agent-Centric DX
With this architecture, we can build far more ambitious systems.
- Long-Running Workflows: Agents can manage stateful processes that last for days, months, or even years.
- Generative UI: Instead of building one-size-fits-all interfaces, we can generate a perfectly customized UI for every single user, based on their context, history, and immediate needs.
- Designing for Agents: The “users” interacting with our backend systems will increasingly be AI agents. This necessitates a shift in focus to a new kind of Developer Experience (DX). Systems should be designed with machine-to-machine interaction in mind, featuring markdown-based documentation, actionable error messages, and search-based API discoverability.
This heralds a resurgence of concepts like capability-based security. The fundamental distinction between programmers who use code and users who get buttons is dissolving. In this new world, you must let the code do the talking.