How to look at your data — Jeff Huber (Choma) + Jason Liu (567)

The key to making systematic, deterministic improvements to AI applications is to rigorously measure and analyze your data. This involves a two-pronged approach: evaluating the inputs to your system, particularly for retrieval, and analyzing the outputs, such as conversational histories, to understand user behavior and product value.

Part 1: How to Look at Your Inputs

The quality of a Retrieval-Augmented Generation (RAG) system is fundamentally limited by its retrieval performance. While methods like guessing, using expensive LLM-as-a-judge frameworks, or relying on public benchmarks like MTEB exist, a more effective strategy is to use Fast Evals.

A Fast Eval is a simple, rapid, and inexpensive evaluation method based on a “golden dataset” of query-and-document pairs. The process involves:

1. Creating a Golden Dataset: Assemble a set of queries where for each query, you define which document(s) should be retrieved.
2. Running the Evaluation: Input the queries into your system and measure the recall—how often the correct documents are returned within the top k results.

This process is fast and cheap, enabling rapid experimentation with different components like chunking strategies or embedding models.

Generating High-Quality Queries If you don’t have existing user queries, you can use an LLM to generate them. However, naive prompting often produces queries that are overly clean and specific to the document content, which doesn’t reflect real-world usage. A better approach is to teach the LLM to generate queries that are semantically aligned with the specificity of real user questions. This ensures that your evaluation isn’t based on an artificially easy task.

Case Study: Weights & Biases Chatbot An analysis of the Weights & Biases chatbot demonstrates the power of this approach. By running a Fast Eval on their data, the following insights were discovered:

• The original embedding model used, text-embedding-3-small, was the worst performer among the models tested for their specific dataset.
• The Gina-embeddings-v3 model, which performs exceptionally well on public English benchmarks, was not the top performer in this application.
• The voyage-3-large model was empirically proven to be the best for their specific data and queries.

This highlights a critical point: public benchmarks are not a substitute for evaluating models on your own data. The North Star should be the success rate on queries that are representative of your users.

Part 2: How to Look at Your Outputs

While inputs are critical, the outputs—such as agent execution logs and conversation histories—contain a wealth of information about user intent, frustration, and satisfaction. Manually reviewing thousands of conversations is not scalable. The solution is to extract structured data from these unstructured conversations and perform traditional data analysis.

Just as a single marketing KPI is not actionable without segmenting the audience, a single evaluation metric like “factuality is 0.6” offers little direction. By segmenting conversations, you can uncover specific areas of success and failure.

A Pipeline for Conversational Analysis A practical pipeline for analyzing outputs involves three steps:

1. Extract Structure: Use an LLM to process each conversation and extract a structured object containing metadata like a summary, topics discussed, tools used, errors encountered, and inferred user frustration or satisfaction.
2. Cluster and Segment: Embed these structured objects and use clustering algorithms to identify cohesive groups of conversations. This reveals emergent usage patterns, such as “data visualization requests,” “SEO content generation,” or “authentication errors.” These clusters can be further grouped into higher-level themes like “technical support.”
3. Compare KPIs Across Clusters: With conversations segmented, you can now compare your evaluation metrics across different clusters. You might find that factuality is very high for “contract search” queries but very low for queries that require “time filters.” This provides a clear, actionable insight into what needs to be fixed or improved.

From Analysis to a Data-Driven Roadmap This segmentation allows for impact-weighted decision-making. By creating a simple matrix of usage frequency versus performance, you can prioritize your work:

• High Usage, Low Performance: This is the most critical area to fix.
• High Usage, High Performance: Maintain and build upon this strength.
• Low Usage, High Performance: This is a discovery problem. Educate users about this capability.
• Low Usage, Low Performance: Consider deprecating the feature or explicitly stating it’s unsupported.

This process creates a continuous feedback loop where research and data analysis directly inform the product roadmap. The solution is often not just “making the AI better” but providing the right infrastructure: better tools, refined prompts, improved data sources, or specialized routers for specific intents. By building a high-quality hypothesis and testing it quickly, you can systematically build a better product.

• Tags:
• LLM
• Synthetic Data
• Evaluation
• Embeddings
• RAG
• Retrieval
• Clustering
• Data Analysis
• Product Development