Shipping AI That Works: An Evaluation Framework for PMs – Aman Khan, Arize

Aman, an AI Product Manager at Arise, presents a practical framework for product managers to rigorously evaluate and ship reliable AI products. The talk addresses the changing expectations of the PM role in the age of AI, coining the term “AIPM” (AI Product Manager).

From Vibe Coding to Thrive Coding

Aman argues against relying on gut-feel “vibe checks” for evaluating AI features. He introduces the concept of “Thrive Coding,” which emphasizes using concrete data and repeatable evaluation strategies to build confidence in AI-driven features. This is crucial because, as leaders from OpenAI and Anthropic admit, LLMs hallucinate and are not inherently reliable. Rigorous evaluation is becoming a key differentiator and a competitive “moat” for AI startups.

What is an Eval?

An evaluation, or “eval,” is analogous to software testing but for non-deterministic AI systems. Aman breaks down an “LLM-as-a-judge” eval into four key components:

1. Role: Defining the task for the evaluating agent.
2. Context: Providing the text or data to be evaluated.
3. Goal: Specifying the objective of the evaluation (e.g., determining if text is toxic).
4. Terminology/Label: Providing examples and a clear set of output labels (e.g., toxic, not-toxic). A key insight shared is to use text labels instead of numerical scores, as LLMs are notoriously poor with numbers.

Live Demo: Building and Evaluating an AI Trip Planner

The core of the talk is a hands-on demonstration using a multi-agent AI trip planner built with LangGraph. The system uses several agents (budget, local experiences, research) that feed into a final itinerary agent.

Key steps in the demo include:

• Observability: Using Arise to visualize the agent system as a graph of traces and spans. This helps PMs understand the underlying mechanics of their AI product.
• Prompt Iteration: Taking a real production trace into a prompt playground to A/B test changes. For example, modifying a prompt to be less verbose and to include a marketing offer.
• Running Evals at Scale: Moving beyond single-example tests, Aman demonstrates creating a dataset from production data and running experiments across it. He runs custom evals like is_friendly and offers_discount to programmatically score the outputs of his prompt A/B test.
• Evals for Your Evals: The crucial final step is validating the evaluator itself. Aman shows how the LLM-as-a-judge’s output can disagree with human labels. He introduces a “match” eval to measure this disagreement, highlighting the need to iterate and improve the evaluation prompts themselves by adding few-shot examples or making them stricter. This meta-evaluation loop of Human -> LLM -> Human is essential for building trust in the evaluation system.

The Future Role of the AIPM

The talk concludes by framing evals as the new form of requirement specifications. Instead of a traditional PRD, an AIPM can provide an engineering team with a curated dataset and a set of evals that define the acceptance criteria for a feature. This empowers PMs to have more control and confidence in the final product, fostering a more collaborative and efficient development cycle.

• Tags:
• Product Management
• Prompt Engineering
• LLM Evaluation
• Observability
• LangGraph
• LLM as a Judge
• AI Products
• AIPM
• Agent Evaluation
• Arise
• A/B Testing