What OpenAI & Google engineers learned deploying 50+ AI products in production

Building AI products presents a fundamental departure from traditional software development, primarily due to two key characteristics: non-determinism and the agency-control trade-off. Understanding these differences is crucial for creating reliable and effective AI systems.

Key Differences in AI Product Development

1. Non-Determinism: Unlike traditional software with well-defined decision paths, AI products operate with non-determinism on both the input and output sides.

   • Input: Users interact via natural language, a fluid and unpredictable interface. There are countless ways a user can express intent, making it impossible to anticipate every interaction.
   • Output: The Large Language Models (LLMs) at the core of these systems are probabilistic and sensitive to prompt phrasing, making their responses inherently unpredictable. This creates a challenging environment where the input, the process, and the output are all variable.

2. The Agency-Control Trade-off: As we grant AI systems more agency—the ability to make autonomous decisions—we inherently relinquish human control. This trade-off is central to building agentic systems. It’s a mistake to jump directly to full autonomy. Instead, an AI system must earn trust over time. The goal is to build a flywheel of improvement by starting with high human control and low AI agency, gradually shifting the balance as the system proves its reliability.

A Framework for Iterative AI Development: Start Small and Scale

The most common pitfall is attempting to build a fully autonomous, all-encompassing agent from day one. This approach often leads to overwhelming complexity and failure. A more effective strategy is to start small and scale iteratively, focusing on a problem-first approach rather than a solution-first one.

This iterative process can be visualized as moving up a ladder of autonomy:

• V1: High Control, Low Agency (e.g., Suggestions): The AI acts as an assistant, providing suggestions or completing boilerplate tasks that a human must review and approve.

  • Example (Customer Support): The AI classifies and routes a support ticket to the correct human agent. Even if it errs, a human can easily correct it.
  • Example (Coding Assistant): The AI suggests inline code completions for a developer to accept or reject.

• V2: Balanced Control and Agency (e.g., Co-pilot): The AI handles more complex, multi-step tasks but still operates under human supervision, generating drafts or plans for review.

  • Example (Customer Support): The AI drafts a full response to a customer query based on standard operating procedures, which a human agent can then edit and send. This process also generates valuable data on what humans change, feeding the improvement flywheel.
  • Example (Coding Assistant): The AI generates larger blocks of code, like unit tests or refactoring plans, for a developer to review before implementation.

• V3: Low Control, High Agency (e.g., Autonomous Agent): After building sufficient trust and reliability, the AI can execute entire workflows autonomously.

  • Example (Customer Support): The AI handles the entire ticket resolution process, from drafting the response to closing the ticket.
  • Example (Coding Assistant): The AI applies changes, opens pull requests, and even launches and A/B tests campaigns autonomously.

The Continuous Calibration, Continuous Development (CC/CD) Lifecycle

To manage this iterative process, a new development lifecycle is needed, analogous to CI/CD in traditional software. This is the Continuous Calibration, Continuous Development (CC/CD) framework.

• Continuous Development Loop: This is the initial build phase.

  1. Scope Capability & Curate Data: Define the specific task and create a dataset of expected inputs and ideal outputs. This alignment exercise is critical for the entire team (PMs, engineers, SMEs).
  2. Set up Application & Design Evals: Build the application and define specific evaluation metrics to measure performance against the curated dataset.
  3. Deploy & Run Evals: Deploy the system and test it.

• Continuous Calibration Loop: This is the post-deployment feedback loop for handling the unexpected.

  1. Analyze Behavior & Spot Errors: Once live, users will interact in unpredictable ways. This loop focuses on analyzing production traces to spot new error patterns that the initial evals didn’t cover.
  2. Apply Fixes & Design New Evals: Fix specific issues (e.g., a poorly defined tool call) and, more importantly, design new evaluation metrics to catch newly discovered classes of errors (emerging patterns).
  3. Iterate: This calibrated understanding feeds back into the development loop, informing the next version and guiding the progression toward greater autonomy.

The Role of Evals and Production Monitoring

There is a false dichotomy between offline evaluations (“evals”) and online production monitoring. Both are essential.

• Evals are your trusted product knowledge codified into datasets. They are crucial for pre-deployment testing and ensuring the system doesn’t regress on known, critical behaviors. However, the term “evals” has suffered from “semantic diffusion,” now meaning everything from data labeling to LLM-as-a-judge, causing confusion.
• Production Monitoring is necessary to catch the “unknown unknowns.” You cannot predict every failure mode. Monitoring implicit user signals (like re-generating an answer in ChatGPT) and explicit feedback provides the raw data for the calibration loop, helping you discover emerging patterns of failure.

The team behind OpenAI’s Codex uses a balanced approach: they have core evaluation suites to prevent regressions but rely heavily on customer feedback, social media monitoring, and internal “vibes-based” testing to understand how the product performs in the real world.

Patterns of Successful AI Teams

Technology is only one part of the equation. Success in AI is often determined by people and culture.

1. Hands-On Leadership: Leaders cannot rely on decade-old intuitions. They must get hands-on, relearn, and be comfortable being “the dumbest person in the room.” A leader who spends time every day learning about AI and experimenting with the tools can guide their organization effectively.
2. Culture of Empowerment: AI should be framed as a tool for augmentation, not replacement. Subject matter experts are critical for building effective AI, but they will not contribute if they fear for their jobs. An empowering culture where employees see AI as a way to 10x their impact is vital.
3. Persistence and Problem Obsession: Building successful AI is a painful process of iteration. This “pain is the new moat.” Teams that succeed are those who endure the struggle of learning, implementing, and understanding what works. They are obsessed with the customer’s workflow and problem, not just the technology, and they build the flywheels necessary to improve over time.

• Tags:
• Agentic AI
• Product Management
• LLMs
• OpenAI
• Evals
• Google
• AI Product Development
• Software Development Lifecycle