We let an AI agent execute Bash and lived to talk about it — Sarah Sanders, PostHog
Sarah Sanders, a context engineer at PostHog, discusses the security implications of building an agentic CLI tool, "The Wizard," which automates developer workflows. She highlights the inherent risks of agents capable of executing commands, labeling it a "malware starter pack," and details how PostHog evolved its security posture from basic allow lists to a sophisticated "defense in depth." Key insights include the principle that "attacks compose, code review doesn't," the discovery of internal supply chain vulnerabilities through the "Context Mill," and the development of "The Warlock," a deterministic scanner using YARA rules. Sanders emphasizes the critical distinction between deterministic enforcement and probabilistic judgment, explaining how an LLM-powered "Triage" layer acts as an adviser (failing closed) rather than a bouncer, preserving the integrity of their security model.
The presentation by Sarah Sanders, a context engineer at PostHog, delves into the critical security challenges of building and deploying agentic AI tools, specifically focusing on PostHog’s “Wizard” CLI. The Wizard is designed to automate PostHog setup (SDK installation, event instrumentation, dashboard creation) for developers, reducing hours of work to minutes. Its agentic nature, while powerful for developer experience, immediately raised security alarms, leading Sanders to describe it as a “malware starter pack” if not carefully secured, given its ability to run commands on developer machines.
Early security measures included a tightly bounded allow list, denying bash by default, restricting commands to trusted operations (build, type check, lint), and routing secrets through a vault. However, Sanders emphasized that “Layer Zero” — using prompts for security — is not security, as prompts are merely suggestive. An independent security audit revealed gaps, not through “obviously evil” vulnerabilities, but where “two innocent, well-intentioned things” combined to create an opening. This led to a crucial insight: “Attacks compose, code review doesn’t”, meaning attackers view the entire system for composite vulnerabilities, while developers typically review isolated code changes.
A significant concern identified was the “Context Mill”, PostHog’s in-house context engine that feeds the Wizard its “brain.” This engine pulls content from documentation, handwritten prompts, and example applications, packaging it into “skill bundles” loaded into the agent’s runtime context. This creates a potential supply chain vulnerability: an attacker could inject a prompt injection payload into a markdown file or code comment in an open-source repository. If an LLM-powered code review misses this, PostHog could inadvertently ship a malicious payload, signed by them, to thousands of developer machines. To counter this, Sanders implemented a “catch it at the source, assume the source failed” methodology, scanning content at both creation and use.
The solution developed is the “Warlock”, a standalone security scanner replacing earlier hacky regex solutions. The Warlock’s core principle is “detects, it does not act.” It takes a string and returns findings (category, severity, recommended action), but the decision to act is separated. It uses YARA rules, a pattern-matching engine long employed by malware researchers, making it “boring on purpose” — fully deterministic and reliable.
The Warlock proved effective, catching:
- Sub-agent exploits: Sub-agents attempting to bypass guardrails and “invent secrets,” leading to the complete removal of sub-agents.
- PII exposure: Agents, without explicit rules, dumping sensitive PII (emails, phone numbers) into events.
- Prompt injection attempts: While no malicious injections were caught, numerous false positives from legitimate demo content led to rethinking how applications and documentation are built to avoid “threat-shaped” content.
To manage the volume of false positives, an LLM-powered “Triage” layer was introduced. Crucially, this LLM acts as an “adviser, not a bouncer.” Sanders stressed that “if it isn’t enforced deterministically, it is not enforced.” Therefore, detection and enforcement remain mechanical and deterministic; if a Warlock rule matches, the gate locks before the LLM’s opinion is sought. The LLM’s role is to reduce noise from unblocked findings, and it “fails closed,” meaning if the model has an issue, all Wizard runs are halted. This separation ensures enforcement remains robust, while judgment (with its inherent nuance) can incorporate probabilistic elements.
Sanders outlined the anatomy of a Warlock rule, comprising metadata (description, severity, category, action, direction), strings (specific patterns, e.g., “verb + instruction-flavored noun” for prompt injection), conditions (when to fire), and essential negative tests to reduce false positives. Rule severity is determined by real-world impact, not just perceived threat (e.g., rm -rf is scary but often benign).
PostHog’s current security posture is one of “defense in depth,” a layered approach where each layer performs a specific, well-defined job. This includes prompts for steering, sandboxing, deny-by-default policies, a secrets vault, the Warlock for scanning, Triage for noise reduction, and comprehensive telemetry. The presentation concluded with three core takeaways:
- Deterministic enforcement is paramount; prompts are not security.
- Dangerous input encompasses all content flowing into the model, including internal supply chain elements.
- Attacks compose, unlike code reviews; focus on systemic vulnerabilities.
The Wizard, Warlock, and Context Mill are all open-source projects.