Inside OpenAI DevDay: Superhuman Computer Use, Decisions API, and the AI Cloud — Ari & Nikunj
OpenAI has unveiled significant advancements in its AI agents and developer APIs, transforming Computer Use into a self-debugging system capable of superhuman speeds. Key innovations include multimodal interfaces, the Agents API for building on OpenAI's harness, and a new API stack featuring async tool calling, UltraFast inference, and the rapidly developed Decisions API for fast classification. These advancements, coupled with robust caching and context compaction, pave the way for an "AI cloud" with higher-level primitives for developers.
OpenAI has dramatically advanced its AI agents, particularly in “Computer Use,” transitioning from rudimentary task execution to sophisticated, self-debugging systems that can complete real-world tasks faster than humans. These advancements, revealed at OpenAI DevDay, include new capabilities, APIs, and performance optimizations aimed at making AI agents “superhuman” at operating software.
Computer Use: From Delegation to Superhuman Operation
Ari Weinstein, Head of Product & Engineering for Computer Use, highlights a profound shift, stating that Computer Use is now “180 degrees different.” Previously, models struggled with tasks after initiation, but current agents excel at debugging, introspecting, and recovering from failures. This capability is attributed to more advanced models and techniques, including:
- Code Generation: Agents now write JavaScript code to execute multiple actions simultaneously, significantly improving speed and capability.
- Multimodal Interfaces: Combining screenshots, accessibility data, the DOM, and Playwright allows models to “see” and operate software more comprehensively. This overcomes limitations like needing to scroll repeatedly, enabling agents to perceive an entire page or application context.
- App Shots: These provide models with rich context beyond ordinary screenshots, capturing metadata and full text, not just visual pixels. This allows agents to understand elements like link destinations or truncated text, leveraging accessibility technology originally designed for humans.
Playwright is an open-source automation library developed by Microsoft, launched in 2020. It enables reliable web automation for testing, scripting, and AI agents across Chromium, Firefox, and WebKit browsers.
Block 2: WebSockets
ANCHOR: WebSockets BLOCK: WebSockets are a computer communications protocol providing a bidirectional, full-duplex communication channel over a single TCP connection. [1] [2]
This improved capability enables Computer Use to complete tasks faster than the average human in many cases, with the next frontier being “literally superhuman” performance, outpacing even expert computer users. This will facilitate real-time experiences and lower the activation energy for delegating tasks to agents. While challenges like network latency (e.g., waiting for doordash.com to load) remain, future efforts will focus on making operations even more event-driven and responsive.
Practical Applications and Trust
Agents are now capable of complex delegations, such as managing meal prep services (saving 2 hours by performing a 15-minute order), automating YouTube tasks not exposed via API, handling customer service, configuring DNS, and paying bills. For software development, Computer Use agents can build and test software, closing the loop on the software development life cycle and acting as automated QA.
However, with increased agency comes the need for trust, permissions, and safety. Developers integrating Computer Use via the Agents API must prioritize building reliable systems, implementing safety checks, seeking user consent for consequential actions (like payments), and restricting agent access to only necessary applications or websites. The Agents API allows developers to leverage OpenAI’s specialized Computer Use implementation, offering potential advantages in speed, cost, and accuracy due to model training alignment.
API Stack and Performance Enhancements
Nikunj Handa, Head of Product for API at OpenAI, detailed significant advancements in the API stack, particularly with GPD6.
Asynchronous Capabilities and UltraFast Inference
Key new features include:
- Async Function Calling: Models no longer pause execution while tools run. They can initiate a tool call, continue reasoning, and then check back, significantly improving efficiency.
- Mid-Turn Steering: Developers can inject messages into the model’s reasoning process while it’s active, enabling more responsive and dynamic interactions. These capabilities are underpinned by WebSockets, facilitating bidirectional communication.
- UltraFast Inference: This pushes frontier models towards much lower latency. OpenAI’s inference team actively uses coding agents to optimize performance, shifting focus from pure cost efficiency to maximizing speed. This allows models like Astra to operate at unprecedented speeds, leveraging the same underlying technology that drove significant cost reductions.
Decisions API: Rapid Classification and Structured Outputs
The Decisions API was developed internally in just four weeks, inspired by the market’s excitement around Jev. It’s designed for fast classification and structured outputs at low latency.
- Mechanism: It leverages existing Luna weights but optimizes the inference stack through constraints and parallel processing (running multiple questions as a batch). It is explicitly not a new model but an optimized use of an existing one.
- Capabilities: While not at the same intelligence level as Astra for long-horizon tasks, it offers significant speed for classification and has the benefit of vision capabilities inherently available with Luna.
- Use Cases: Internal applications include classifying support tickets, enhancing GPD Live for snappier tool calls by controlling a computer, and potentially aiding in evaluating LLMs.
Performance Optimization: Caching and Compaction
Optimizing for long-running agent threads is crucial:
- Prompt Caching: OpenAI now offers guarantees for cache hits (e.g., 30 minutes, with 12-hour previews for some users) and pre-warming capabilities. This allows developers to pay a cash write fee to prepare prompts, ensuring fast responses for subsequent uses, which is critical for continuous agent interactions.
- Context Compaction: For managing large contexts, OpenAI offers two methods:
- Server-side compaction: Automatic context reduction when a token threshold is met within the Agents API harness.
- Manual compaction (
/compact): Developers retain full control to compact context at specific times, a method favored by complex coding agents (e.g., in the open-source Codex harness). New file-based compaction techniques are also being explored.
The Vision of an AI Cloud
Looking ahead, OpenAI envisions itself building an “AI cloud”, moving beyond raw model APIs to offer higher-level primitives for agents. This involves exploring solutions for memory vaults, state management, and other abstractions that simplify building complex AI applications, drawing parallels to the evolution of cloud services like AWS providing EC2 and S3. OpenAI actively seeks developer feedback on these new APIs and performance aspects to guide its roadmap toward this future.