Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd
David Brumley discusses the challenges and solutions for teaching AI models to hack, drawing parallels with human learning. He introduces a 'ladder of tasks' approach for reinforcement learning and addresses the critical flaw of traditional benchmarks: measurement difficulties with multiple vulnerabilities and 'reward hacking.' His team's 'Audit Task' uses deterministic graders and precision/recall metrics for open-world assessment. He demonstrates this with an in-depth case study on attacking Chrome's V8 engine, showcasing how advanced models achieve real zero-day exploits, and warns against 'benchmaxxing security' without robust, honest grading.
David Brumley, a distinguished professor at Carnegie Mellon University and chief AI/science officer at Bugcrowd, articulates that teaching AI models to hack parallels the successful pedagogical methods used for humans: a progressive “ladder of tasks.” This approach is inherently well-suited for reinforcement learning (RL), involving two primary axes: target difficulty (from toy programs to hardened systems like Chrome’s V8) and exploitation difficulty (escalating from merely triggering a crash to achieving arbitrary read/write, and ultimately, arbitrary code execution). The ultimate goal is to “bend computers to our will,” not just to crash them.
Critically, Brumley identifies significant flaws in traditional cybersecurity benchmarks. Firstly, relying on an LLM as a judge is unreliable, as models “will always say they were successful hacking,” necessitating deterministic grading oracles. Secondly, the assumption of a single vulnerability per program is unrealistic and leads to reward hacking, where models perpetually exploit the easiest bug, hindering advanced learning. He cites instances from DARPA’s Cyber Grand Challenge and AIxCC where even expert-curated problems contained unintended, exploitable vulnerabilities. Furthermore, “nudging” LLMs by pointing to vulnerable functions in benchmarks stifles their reasoning capabilities.
To overcome these limitations, his team developed the Audit Task. This method prompts LLMs to “find all vulnerabilities discovered” within a program. A crucial component is a deterministic grader capable of verifying and uniquifying distinct vulnerabilities, typically through stack backtrace analysis—a standard industry practice. Evaluation then leverages Precision and Recall metrics, borrowed from Brumley’s DARPA Cyber Challenge scoring work, balancing the discovery of unknown (zero-day) vulnerabilities with preventing the model from submitting false positives or “spamming.” This approach fosters open-world grading, allowing testing on real open-source software and providing a cleaner learning trajectory because the model doesn’t know how many vulnerabilities to expect. As Brumley states, “Don’t define the task by a single bug, let the program define the task.”
The research extends to measuring weaponization against high-value targets, specifically Chrome’s V8 JavaScript engine. V8, a complex, sandboxed environment, demands out-of-sandbox exploits—often chaining multiple vulnerabilities—for true impact, differentiating it from mere in-sandbox crashes. Their ExploitBench framework, comprising 16 exploitation capabilities, assesses models’ progress. Results from testing 41 V8 vulnerabilities reveal that while many models achieve high success (95% for top-tier, 50% for others) in simply crashing a program, a stark divergence appears for full sandbox escape and arbitrary code execution. Mythos achieved 73% success, GPT 68%, while others scored 0%. Notably, Mythos demonstrated non-memorized, novel exploitation paths, such as reversing JavaScript’s Math.random to forge a pointer for a return-oriented program in a known CVE (CVE-2023-670T), identifying new WASM paths for CVE-2024-767965, and exploiting CVE-2024-7965 on x86, which even internal experts believed was only possible on ARM—performance deemed “on par with a human elite researcher.”
Brumley concludes with a warning against “benchmaxxing security.” He stresses the imperative of building RL environments grounded in real bugs and employing honest, deterministic graders to truly differentiate models that look like they can hack from those that actually can. The work, including environments and data, is openly available at exploitbench.ai.