How Google DeepMind Runs Agents at Scale — KP Sawhney & Ian Ballantyne, Google DeepMind
KP Sawhney from Google DeepMind discusses the internal strategies for scaling agentic AI, including managing token-hungry workflows, curating a 'Darwinian' skills library, and evolving the Deep Research pipeline from large context blobs to a collaborative file system.
Scaling Agents at Google: Challenges and Strategies
At Google DeepMind, scaling agentic systems for thousands of internal power users presents unique challenges, primarily centered around resource management and system design. Agentic workflows are incredibly “token hungry,” which makes quota management a top priority. The internal system for managing this is currently a “brute force” approach, where teams monitor usage spikes 24/7 and will directly ask a user or team to stop a job that is consuming too many resources on a cluster. Interestingly, internal Google employees often have stricter limits than paying customers, as customer-facing services are prioritized.
To mitigate costs and manage resource constraints, one key strategy is to mix and match models. For example, an agentic system might use a smaller, more efficient model like Gemma for tasks that don’t require cutting-edge reasoning, reserving more advanced models for critical components of the workflow. This hybrid approach helps balance performance with the high cost of computation.
The “Antigravity” Harness and Internal Workflows
DeepMind utilizes an internal agentic framework called “Antigravity,” which functions as a Visual Studio-style IDE with an integrated agent manager. This allows developers to spawn and manage multiple agents working on different projects. The framework includes:
- A chat panel for issuing commands.
- A planning system where the agent outlines its implementation plan.
- A “human in the loop” feedback mechanism, allowing developers to review and edit the agent’s plan before it proceeds.
- The ability to control a browser, inspect the DOM, and capture screenshots or videos of its interactions.
- A “scratch pad” that provides a trace of the agent’s thoughts and actions for debugging.
A “Darwinian” Approach to a Shared Skills Library
To enhance agent capabilities across the organization, DeepMind fosters a vast, internally-contributed library of skills. Engineers who are experts in a specific domain can contribute skills, which then become available for other agents (and their users) to leverage. This democratizes expert knowledge.
However, in an organization the size of Google, this can lead to an uncontrolled sprawl of skills. To manage this, the team has adopted an almost “Darwinian” approach, where processes are in place to ensure only the most effective and well-maintained skills survive and are promoted. This curation process is essential for maintaining a high-quality, reliable toolset.
The Future: Evolving the Deep Research Pipeline
A major focus for the future is evolving the architecture of complex agents like the Deep Research agent. Currently, the pipeline involves passing “huge huge blobs of of text” between different stages, which is inefficient, costly, and consumes a large portion of the context window.
The vision is to replace this with a shared file system where each component of the agentic system can collaborate like human researchers working in a shared workspace. This paradigm shift is expected to be faster, cheaper, and yield better results. It also opens up the potential for producing richer supporting artifacts, such as infographics and supplementary documents, which the current architecture cannot easily generate. This evolution moves towards a future of more efficient agent-to-agent communication, with the human user acting as a “supervisor on a digital assembly line.”
Observability and Evaluation
To debug and understand these complex systems, DeepMind has built custom internal tools for observability. This includes a web application that allows engineers to drill down into an agent’s operations, from the high-level query to the raw prediction requests made to the model. For coding tasks, an “agent trajectory store” is used to diagnose exactly where an agent went off track or entered a loop.
Evaluation remains one of the hardest problems. It requires spinning up sandboxed environments and developing novel datasets. While there are open-source benchmarks, evaluating specific, custom skills often requires the skill’s author to create a corresponding test suite. In a meta-level approach, teams are also experimenting with using agents to design the evaluation tests for other agents.