Computer use in Codex
Ari Weinstein discusses how Codex's 'computer use' feature allows the AI agent to operate local Mac applications in the background by combining multimodal vision with accessibility data, enabling non-intrusive, parallel task execution.
Codex has evolved from a coding assistant into a comprehensive agent capable of performing a wide range of tasks by interacting directly with local, graphical applications on a user’s computer. This “computer use” functionality extends the agent’s capabilities beyond the command line and into the GUI-driven workflows that define modern computing.
Non-Intrusive Background Operation
A key differentiator of Codex’s approach is its ability to operate applications in the background without taking over the user’s session. Unlike other implementations that monopolize the system’s input, Codex uses its own cursor, visually distinct from the user’s. This allows the user to continue their work uninterrupted while the agent performs tasks in parallel.
This architecture enables true multitasking, where a user can delegate multiple, distinct tasks to Codex across different applications simultaneously. For example, a user can ask Codex to:
- Set up a new macOS virtual machine in UTM.
- Play a specific type of music on Spotify.
- Add a task to the Reminders app.
Codex can execute these tasks concurrently, each with its own cursor, while the user works on something else entirely. The motion of the agent’s cursor is intentionally designed with natural, whimsical curves, making it easier and more “delightful” for the user to follow and understand the agent’s actions on screen.
Technical Implementation: A Hybrid Approach
The technical foundation of computer use combines two critical data sources:
- Screenshots (Multimodal Vision): Like many vision-based agents, Codex can “see” the application’s interface, allowing it to identify and interact with elements by their visual location.
- Accessibility Framework Data: This is a crucial enhancement. By tapping into the macOS accessibility framework, Codex pulls structured, textual information describing the UI elements, their roles, and their hierarchy.
This hybrid approach provides significant advantages over relying on screenshots alone:
- Deeper Understanding: The model gains a more profound, semantic understanding of the interface. It knows an element is a “button” or a “text field,” not just a collection of pixels.
- Beyond the Viewport: It can perceive and interact with elements that are scrolled off-screen, as they are still present in the accessibility tree.
- Increased Accuracy: The combination of visual and textual data makes the model far more accurate and robust in performing tasks.
Performance and Model Efficiency
A major benefit of this dual-data approach is the ability to use faster, more efficient language models that are not multimodal. Because the accessibility framework provides a rich, textual representation of the UI, the agent isn’t always dependent on processing images. This allows for the use of models like Codex Spark, which can execute tasks at superhuman speeds—often faster than a human could manually operate the software. This speed makes the feature practical for quick, everyday tasks, such as sending a message in the Messages app.
Safety and User Trust
To ensure user comfort and security, Codex’s computer use is built on a foundation of explicit, granular permissions. The system is designed to prevent broad, unrestricted access to the user’s computer.
- App-by-App Permissions: Codex must ask for permission the first time it needs to access a new application.
- Sandboxed Access: When permission is granted for an app (e.g., UTM), Codex can only see and interact with that specific app. It remains completely unaware of and unable to interact with any other applications, ensuring that sensitive information in other apps remains private.
This opt-in, per-application model is designed to build user trust by giving them precise control over what the agent can and cannot do.