From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization - May Walter, Hud
May Walter, CTO of Hud, details a real-world case study on integrating AI agents into the SDLC for continuous performance optimization. The talk covers Hud's runtime intelligence layer, which uses production context to identify and fix high-ROI performance issues like N+1 queries and missing database indexes. It highlights the technical approach, challenges, and the development of a human-friendly reporting system that delivers measurable P90 latency improvements, enabling proactive optimization in mature codebases.
This talk addresses the pervasive issue of performance degradation in mature codebases, where optimization is often deferred due to unpredictable investigation efforts and the difficulty of justifying proactive work amidst feature development. The speaker, May Walter, CTO of Hud, introduces a novel solution: a runtime intelligence layer for coding agents designed for continuous performance optimization in production environments.
The core problem stems from a “leaky bucket” cycle: performance issues are ignored, degrade to crisis levels, are reactively fixed, and then the cycle repeats. Investigating these issues is a black box, making prioritization challenging as the return on investment (ROI) is unknown. Current trends, as highlighted by DORA metrics for 2026, show that while AI adoption increases individual developer effectiveness, it also contributes to software delivery instability, without significantly boosting team throughput. The presenter argues that if AI can help build and fix faster, these gains can be realized.
Hud’s approach automates this investigation process, performing weekly analyses with real production context to identify high-ROI optimization opportunities that are both easy to implement and impactful. This effectively automates the “performance sprint” teams typically conduct every few months.
The technical architecture for these agentic workflows emphasizes several key requirements:
- Vendor Neutrality: The system must be adaptable across different compute platforms, harnesses, and large language models (LLMs) to avoid vendor lock-in.
- Security: Robust mechanisms for tool calls, permissions, and authentication are crucial.
- Trigger Systems: Support for scheduled runs (e.g., weekly) and event-driven triggers (e.g., SLO breaches).
- Maintainability: Workflows must be easy to update and evolve over time, allowing for continuous feedback loops to build trust and reliability.
A specific implementation example involves GitHub Actions running weekly with Cloud Code to capture runtime intelligence via MCP (Micro-Container Platform), analyzing function-level overhead, and sending reports to Slack. The agent-driven workflow proceeds as follows: it analyzes production context (traces, queries, latencies), scores and flags high-impact opportunities (e.g., those affecting hot paths or critical business flows), and then the agent fixes the issue, reruns tests, and verifies the impact on the optimized flow. Only after this verification is the proposed fix presented to a human for review, transforming an idea into a proven solution with demonstrated business impact.
Initial challenges in developing this system included:
- Plausible Unverified Fixes: Agents proposing theoretically sound but practically ineffective solutions not grounded in actual production behavior.
- Complex Queries: Difficulty in handling specific database query patterns (e.g., ClickHouse’s nuances).
- Lazy Fixes: Agents suggesting superficial solutions (e.g., catching an exception instead of addressing its root cause).
To overcome these, Hud focused on “Prod-to-Code” context translation. Recognizing that agents reason at the function and file level, while traditional metrics often exist at the service/endpoint level, they created a system to explain production events (like slow endpoints) in function-level terms. This involves capturing forensic evidence only when needed (e.g., for requests exceeding P99 latency) and linking function invocations to their originating endpoints or events. This is facilitated by a “HUD Query Language” (ClickHouse queries over structured function/endpoint data), a “Skills Layer” (reusable capabilities for common tasks like analyzing HTTP 500s or memory spikes), and subsequent “Automations” (e.g., auto-fixing N+1 queries, missing indexes, artificial delays, or eliminating dead code).
A critical aspect is integrating humans effectively. Instead of overwhelming engineers with numerous automated pull requests, Hud developed human-friendly reports. These reports prioritize opportunities based on business impact, hot path relevance, and risk, aiming for the highest ROI. Each report provides a concise explanation of the problem, its impact, the proposed fix, and verified evidence of its effectiveness, encouraging engineers to adopt and trust the system incrementally.
Key takeaways from this journey include:
- Defining what matters: Robust scoring and guardrails are essential for reliable automation, ensuring that only truly valuable and vetted issues are surfaced.
- Automating what didn’t happen: Agents can unlock entirely new optimization processes that teams previously couldn’t undertake (e.g., continuous proactive checks).
- Context over cleverness: Providing agents with precise, function-level production context and specialized skills yields better results than relying solely on advanced models.
- High confidence for autonomous agents: Unlike interactive coding with an agent (where 80% accuracy is often sufficient), autonomous agentic engineering requires a much higher confidence level (80-90% trust) because the agent acts independently. The fix must be thoroughly verified in runtime before a human reviews it, ensuring the effort is worthwhile.
This continuous performance optimization framework demonstrates a practical, real-world application of AI agents in the Software Development Life Cycle, delivering measurable performance improvements in production.