Hacking AI Systems: How to (Still) Trick Artificial Intelligence • Katharine Jarmul • GOTO 2025
To build secure AI systems, we must first learn to break them. Katharine Jarmul explores the landscape of adversarial AI, detailing how attackers exploit fundamental weaknesses in deep learning models—from poisoned training data and overparameterization to the attention mechanism itself. This talk provides a practical taxonomy of attacks and a primer on building robust defenses.
Thinking Like an Adversary
To effectively secure AI systems, it’s essential to adopt the mindset of an attacker. An adversary’s primary goals are to disrupt, exploit, or exfiltrate valuable assets, which can include data, services, software, or infrastructure. They aim to infiltrate systems, move laterally, and cause harm to a brand or business, all while ensuring the cost of the attack to the victim outweighs their own effort.
When targeting modern AI systems, an attacker analyzes the architecture, which typically falls into one of three categories:
- Large-Scale Vendor Systems: Highly distributed hardware (cloud or on-prem) with a complex software stack, including the main deep learning model, monitoring, logging, and auxiliary models like guardrails.
- API-Based Systems: A common setup where a company’s software, running in the cloud, makes API calls to an external AI vendor. This creates an entry point to the larger vendor system.
- Local Systems: An AI agent or model running directly on a user’s machine, providing access to the local operating system and data.
Across these architectures, the common exploitable elements are the AI model itself, sensitive data it can access, proprietary code, and the underlying critical infrastructure.
Fundamental Weaknesses in AI Models
The core vulnerabilities of modern deep learning models can be traced back to their data, architecture, and training process.
Training Data
The “joyous place to start” for any attacker is the training data. Large models are trained on vast datasets scraped from the internet, which are inherently filled with problematic content:
- Sensitive and personal information, such as medical photos or financial details.
- Watermarked images, advertisements, and copyrighted material.
- Malware, hate speech, and content from the dark web. An attacker who understands the training data’s composition can craft inputs to elicit unintended or malicious outputs.
Embeddings and Decision Boundaries
Deep learning models don’t process raw data; they convert it into dense mathematical representations called embeddings. This process, rooted in information and coding theory, places similar concepts close to each other in a high-dimensional space. The model then learns by creating “decision boundaries” to separate different classes or concepts within this space.
The robustness of a model is correlated with the margin, or space, between these boundaries. Attackers can create inputs that lie very close to a decision boundary, causing the model to misclassify the input with high confidence. This is the principle behind adversarial examples.
Overparameterization and Attention
Modern models like the GPT series are overparameterized, meaning they have more parameters than data points in their training set. This gives the model an enormous capacity, allowing it to not just learn patterns but to literally memorize and store training data. This makes it highly susceptible to data extraction attacks.
The attention mechanism is another key architectural feature. It allows a model to hold information in an active state over very long sequences, like a key-value lookup system. An attacker can design a query (part of a prompt) that activates a specific key, causing a highly weighted, malicious value to be held and used later in the generation process, effectively hijacking the model’s focus.
A Taxonomy of Attacks by Attacker Goal
Instead of focusing on every possible attack, it’s more effective to categorize them by the adversary’s objective.
Goal: Exfiltrate Data
- Training Data Extraction: By providing partial data (e.g., a cropped image) and asking the model to “fill in the blank,” attackers can reconstruct original training samples, verifying their presence and extracting them.
- RAG Data Extraction: Models connected to Retrieval-Augmented Generation (RAG) systems can be tricked into revealing the contents of the documents they have access to.
- Prompt Leakage: It is trivial to make a model repeat its system prompt, exposing confidential instructions, rules, or keywords. Never put secrets in a prompt.
- Lateral Movement: AI agents with access to other systems (like Google Drive or calendars) can be used as a vector for lateral movement to exfiltrate data from those connected services.
Goal: Change AI Output & Disrupt Services
- Adversarial Examples: These are specially crafted inputs that are imperceptible to humans but cause the model to produce an incorrect or targeted output. A famous example is a 3D-printed turtle that is consistently classified as a rifle from every angle due to a subtle pattern on its shell. Prompt injection is a form of this.
- Data Poisoning & Backdoors: While poisoning the initial training data is a threat, a more practical risk is poisoning documents in a RAG system that an AI uses for context. It’s also mathematically possible to create undetectable backdoors in a model that trigger specific behavior when a secret input is provided.
- System and Brand Disruption:
- Prompt Hijacking: Attackers can exploit public-facing endpoints or steal API keys to run up massive bills for the victim.
- Jailbreaking: Coercing a model into generating offensive or brand-damaging content, which is then screenshotted and shared publicly.
- System Hijacking: In agentic workflows, an adversarial prompt hidden in a document (like a meeting invite) can be used to trigger actions in connected systems, such as controlling smart home devices.
A Primer on Defense Strategies
Protecting against these threats requires a multi-layered approach rooted in security fundamentals.
- Start with Architecture: Treat the AI model as a potentially dangerous piece of software. Isolate and segregate it from critical systems and sensitive data as much as possible.
- Use Appropriate Models: Question the need for a massive, general-purpose model. Smaller, locally-run, or fine-tuned models can offer a smaller attack surface and more predictable behavior.
- Implement Guardrails (with caution): Use input/output filters and algorithmic guardrails, but understand that they are not a complete solution and can often be bypassed.
- Practice Interdisciplinary Threat Modeling: Bring together people from different teams (security, engineering, legal, product) to identify realistic threats and brainstorm solutions.
- Conduct Continuous Red Teaming and Testing: Go beyond vendor-provided security reports. Build custom evaluations that test the model’s security within your specific use case. Integrate privacy and security testing into your MLOps lifecycle as a standard practice.