Ex-DeepMind: How To Actually Protect Your Data From AI
Dr. Ilia Shumailov, former DeepMind AI Security Researcher, explains why traditional security fails for AI agents. He details the unique threat model of agents, the dangers of supply chain attacks and architectural backdoors, and proposes a system-level solution called CAML to enforce security policies by design, separating model reasoning from data execution.
Dr. Ilia Shumailov argues that the advent of AI agents fundamentally changes the security landscape, requiring a complete shift in thinking away from human-centric threat models. Traditional security was built by humans, for humans, and against humans, but this paradigm breaks down when dealing with AI agents.
The Distinction Between AI Safety and Security
A core distinction is drawn between safety and security, borrowing from classical software security principles.
- Safety is concerned with the average-case performance of a system. A safety failure is when a system does something unexpected on its own, without malicious intent. The analogy given is a phone’s battery spontaneously exploding.
- Security is concerned with the worst-case performance of a system under the influence of a malicious actor. A security failure is when an adversary forces the system into an undesirable state. The analogy is someone a meter away from your phone forcing it to explode.
In AI, the line has blurred because searching for jailbreaks inherently involves an adversarial mindset, but the distinction remains critical. Security assumes an intelligent, malicious actor actively pushing the system to its breaking point.
The Unique Threat Model of AI Agents
AI agents are not comparable to human employees or adversaries. They represent a worst-case scenario that traditional security frameworks are not equipped to handle.
- Capabilities and Scale: An agent can work 24/7, simultaneously interact with every endpoint in a system, and generate sophisticated hacking tools in seconds from its training data. A human insider threat has physical and cognitive limitations that simply don’t apply to an agent.
- Lack of Rationality and Consequences: Human behavior is constrained by legal and social consequences (e.g., fines, prison). An agent has no such constraints. This makes them akin to the worst-case adversary in security: a child—completely irrational, with infinite time, and no expectations of predictable behavior. Agents are even worse than this.
- Unpredictable Problem Solving: Agents often solve tasks in non-human, unpredictable ways. An agent asked to forward a note might decide, on its own, to email several other parties not mentioned in the original request. This emergent, non-human-like “reasoning” makes their behavior hard to predict and contain. The thinking traces or “chain of thought” from a model cannot be reliably interpreted as a true representation of its reasoning process.
System-Level Defense: Defeating Prompt Injection by Design
As models increase in capability and become better at following instructions, they also become more vulnerable to prompt injections because they can be more effectively instructed to perform malicious tasks. Research shows that most academic and commercial defenses against indirect prompt injections (e.g., a malicious instruction hidden in an email the agent is processing) are ineffective.
The solution is not to try and make the model itself secure, but to build secure systems around the model. The paper “Defeating Prompt Injections by Design” proposes a system called CAML:
- Formal Semantics: User queries are rewritten into a language with formal semantics for control flow and data flow, such as Python.
- Symbolic Representation: The model never sees sensitive data directly. Instead, it operates on a symbolic representation (e.g., a variable name for a passport number).
- Policy-Enforcing Interpreter: An external interpreter executes the generated program step-by-step. This interpreter enforces access control policies on the data flow graph. For example, it can enforce a rule that the variable containing the passport number can only be passed to a tool whose domain is
gov.uk.
This architecture provides formal guarantees by separating the model’s probabilistic reasoning from the deterministic, policy-controlled execution environment.
The Dangers of Open Source and Supply Chain Attacks
The open-source AI ecosystem presents significant security risks, particularly for average consumers.
- Remote Code Execution: The
trust_remote_code=Trueflag in popular libraries like Hugging Face’stransformersis functionally equivalent to the vulnerability that caused the massive Log4j crisis. It allows for remote code to be loaded and executed on a user’s machine. - Compromised Infrastructure: There have been publicly reported compromises of the PyTorch CI/CD runners on GitHub and instances where malicious packages were injected into the PyTorch nightly builds by exploiting package dependency resolution.
- Dependency Risks: Popular ML libraries have a disproportionately large number of dependencies. These dependencies, especially third-level ones, are often obscure and poorly maintained, creating a massive attack surface.
Advanced Threats: Architectural Backdoors
A more insidious threat is the concept of architectural backdoors, where malicious functionality is embedded in the structure of a neural network, not its weights.
- Persistence: Because the backdoor is part of the architecture itself, it survives retraining and fine-tuning.
- Mechanism: An example is modifying a transformer architecture to be sensitive to specific trigger tokens. When these tokens are present in one user’s input within a batch, the model’s memory is manipulated to route data incorrectly, causing another user’s data in the same batch to be leaked. This is implemented with standard-looking operations like gather-scatter, making it extremely difficult to detect.
A New Paradigm: ML as a Trusted Third Party
Counterintuitively, machine learning models could potentially replace some complex and expensive cryptographic protocols. In a scenario like Yao’s Millionaires’ Problem (determining who is richer without revealing wealth), two parties could agree on a specific model, a prompt, and a platform that provides integrity verification. They could then provide their secrets to the model, which performs the trusted computation. While this approach lacks the formal soundness and completeness of cryptography, it represents a completely new and different trust model for secure computation.