LLM codegen fails and how to stop 'em — Danilo Campos, PostHog
Danilo Campos of PostHog details the common failure modes of LLM-based code generation—from model rot to security risks—and shares the practical, prose-driven strategies his team uses to make their autonomous coding agent reliable for thousands of users.
Danilo Campos, from PostHog, shares a playbook for making autonomous code generation agents more reliable, drawing from his experience building the PostHog Wizard, a tool that successfully automates product integration for over 15,000 users monthly. The key is not to write more code, but to shift focus towards providing high-quality context and guidance in plain text.
Common Failure Modes and Mitigation Strategies
Building a reliable agent involves anticipating and preventing common ways they can fail. The PostHog team has identified several key failure modes and developed effective strategies to counter them.
1. Model Rot
Models are trained on a snapshot of the world, which quickly becomes outdated. For a fast-moving software project, this “model rot” means the LLM will generate incorrect code, invent non-existent APIs, and use deprecated patterns.
- Problem: The model’s knowledge of a library or framework is stale.
- Solution: Instead of relying solely on the model’s internal knowledge, provide fresh, up-to-date context at inference time. While RAG is a common approach, for this use case, it’s highly effective to simply “shove a bunch of markdown files into the context.” The PostHog Wizard uses tools to select and inject the latest documentation relevant to the user’s specific framework and language, effectively patching the model’s knowledge gaps.
2. Weird Architectural Patterns
LLMs have been trained on a vast amount of code from the internet, not all of which follows best practices. This can lead to agents producing code that is technically functional but architecturally strange or difficult to maintain.
- Problem: The agent generates a correct but non-idiomatic or poorly structured integration.
- Solution: Guide the agent with “model airplanes.” These are lightweight, template projects that demonstrate the ideal shape of an integration for various frameworks. They are not full production applications but a “simulacrum of a real application” — for example, an auth system that accepts any password but has the correct structure. By providing these token-efficient examples, the agent learns the correct patterns for placing code, such as where to track login and identity events.
3. Unconstrained Improvisation
Given a complex task, an agent might find thousands of unique paths to a solution. While this demonstrates flexibility, it creates an enormous support burden, akin to a “sorcerer’s apprentice” scenario where every integration is a unique, hard-to-debug creation.
- Problem: The agent produces too many variations of the solution, making support and maintenance at scale impossible.
- Solution: Guide the agent with “breadcrumbs” instead of giving it the full, complex goal upfront. The process is broken down into sequential, manageable steps:
- Discover: First, the agent is asked to find files with “interesting business value” (e.g., login, payments, churn indicators) without even mentioning PostHog.
- Ideate: Next, it suggests potential events to track within those files, creating a list of event names and descriptions.
- Implement: Only after the high-level plan is established does the agent begin to implement the code, using the fresh documentation and patterns from the model airplanes.
4. Human Error
The biggest threat to an agent’s performance can be its creators. Developers can inadvertently introduce contradictory instructions, forget to grant tool permissions, or provide context for the wrong programming language.
- Problem: The human team introduces bugs, inconsistencies, or missing components in the agent’s instructions and tools.
- Solution: Conduct “inference time interrogation.” At the end of every run, the system performs a bit of user research on the agent itself, asking a simple question: “What could we have done better to set you up for success in this run?” This feedback loop is invaluable for discovering issues like contradictory prompts, missing tools, or incorrect context that would otherwise go unnoticed.
5. Security and Trust (“Shenanigans”)
Running an agent on a user’s machine requires a huge amount of trust. The agent must not perform destructive or insecure actions, such as reading sensitive information from .env files and sending it to a cloud service.
- Problem: The agent might access sensitive user data or perform unwanted actions.
- Solution: Implement fine-grained tool control. Instead of giving the agent broad permissions like
readFile, create highly specific, secure tools. For handling environment variables, the PostHog team built a tool that only allows two actions: checking for the presence of a key and writing a new value to a key. This prevents sensitive data from ever being read or exposed during inference.
The Paradigm Shift: From Code to Prose
The most significant lesson is to resist the engineering impulse to solve problems by writing more code. In the age of LLMs, code is a depreciating asset, while well-crafted prose (prompts, documentation, examples) is an appreciating one. As models improve, they can derive more value and capability from the same high-quality text.
The wizard that makes everybody so happy is 90% markdown files, 8% tools for delivering and processing markdown files, and then the rest is like agent harness stuff. Plain text pros is where so much of our value now lives.
The goal is not to over-constrain the agent with rigid code structures but to step back, provide it with the right information in the right sequence, and let it maneuver and solve the problem effectively.