AI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, Netflix

Rajat Shah, a Staff Software Engineer in Netflix’s AI platform organization, discusses how integrating AI agents into performance engineering workflows significantly improves throughput and reduces infrastructure costs. The core problem addressed is the inefficiency of traditional, manual performance tuning, which is exacerbated by AI-assisted coding generating code optimized for speed of writing rather than runtime performance.

The Challenge of Manual Performance Engineering

Traditional performance engineering involves a human engineer manually triggering profiling, downloading data, visualizing call stacks (often flame graphs), and spending considerable time identifying bottlenecks. This process is tedious, requires a steep learning curve, and is usually only performed reactively when critical issues arise, leading to prolonged inefficiencies in production.

AI Agent for Automated Performance Tuning

The talk introduces an AI agent designed to automate this performance tuning loop. The agent’s capabilities include:

• Reading Profiling Data: LLMs can interpret raw profiling data (call stacks, self/inclusive CPU time) consistently across different languages (Java, Python, Go) as profilers output well-structured, language-agnostic data.
• Pattern Recognition: Leveraging common anti-patterns learned from training on vast codebases (e.g., O(N^2) loops, loop invariants, repetitive object allocations, contention points).
• Code Tracing and Fix Proposal: The agent can take a call stack from profiling data, locate the exact method in the codebase, clone the Git repository at the production commit, and trace the call path to identify the problematic code section. Given sufficient instructions, it can then propose an optimized fix.

Case Study: An O(N^2) Quadratic Inefficiency

Shah presents a real-world example where the agent identified an O(N^2) quadratic inefficiency within a tensor merge method in a live Netflix service. This single fix, proposed and validated by the agent, resulted in an 8.8% CPU time reduction. The agent demonstrated its ability to not only identify problems but also generate actionable code reviews within minutes.

Scaling with a Shared Pattern Catalog

Crucially, the agent’s ability to identify anti-patterns can be scaled. The same anti-pattern (e.g., inefficient counter object creation in hot paths) was found and fixed across seven different Netflix services, leading to 0.5% to 4.6% CPU savings per service. To enable this, Netflix developed a centralized, ever-growing “pattern catalog” — effectively a long-term memory for AI agents. This catalog, initially a simple Markdown-based Git repository, stores identified patterns and anti-patterns. This allows agents profiling new services to leverage existing knowledge, avoiding redundant discovery.

• Bootstrapping the Catalog: The catalog can be bootstrapped from public sources (e.g., Jeff Dean’s C++ optimization blog, PyTorch’s torch-fix repo) and internal performance playbooks.
• Catalog Entry: Each entry details hints for LLMs, relevant symbols, confirmed services, and confidence levels, alongside both anti-pattern and good-pattern code examples.

Verification and Human Oversight

While the agent automates much of the process, human approval remains critical due to the risk associated with modifying production code. Before reaching human review, the agent performs several automated checks:

• Functional Correctness: Running unit and integration tests to ensure business logic is not broken.
• Canary Validation: Deploying the fix to a small percentage of real production traffic (canary) alongside the old code. The agent compares observability reports (CPU reduction, latency, error rate) to validate performance gains and absence of regressions. The mantra is: “Profiler gives the estimate, canary gives ground truth.”

Shifting Left: Reactive to Proactive Performance Engineering

The ultimate goal is to move from reactive (fixing issues in production) to proactive performance engineering. The catalog plays a vital role here:

• Code Review Phase: Reviewer agents can provide inline comments during code reviews, flagging potential anti-patterns based on the catalog.
• Code Authoring Phase: Integrating the catalog directly into coding agents allows them to reference known anti-patterns during code generation, preventing inefficient code from being written in the first place. This requires a well-indexed catalog to avoid overwhelming the agent’s context window.

Foundational Investments and Autonomy Levels

Successful implementation requires robust foundational investments:

• Automation: Seamless integration for triggering profiling, downloading data, cloning Git repos, and managing canary deployments.
• Testing: Rock-solid unit and integration test coverage.
• Pattern Catalog: A well-maintained, human- and agent-readable catalog.

The speaker outlines a progression of autonomy levels for AI agents:

• Level 0 (Manual): Human-intensive, hours spent finding problems.
• Level 1 (LLM for Identification): LLM identifies hot paths from profiling data, still manual for other steps.
• Level 2 (Orchestrated Agent): Full, predefined workflow (profiling, analysis, fix proposal, canary validation). This is the focus of the talk, providing maximum benefits with controlled risk. Allows for scheduled, continuous performance checks.
• Level 3 (AI for Planning and Reasoning): Higher autonomy, where the agent plans and acts without fixed workflows. Requires significant investment in evaluation, sandboxing, and security to prevent issues like prompt injection.

The key takeaway is to build strong foundations, start with reactive approaches to populate the pattern catalog, and gradually move towards proactive integration throughout the software development lifecycle to ensure optimal code from inception.

• Tags:
• AI Agents
• Feature
• Software Development Lifecycle
• Performance Engineering
• Profiling
• Machine Learning Operations
• Code Optimization
• Canary Deployments
• Antipatterns
• Netflix