LLMOps for eval-driven development at scale

The Eval-Centric Approach to LLMOps

While LLM application development has three core components—the model, the data, and the evaluation—our experience at Mercari shows that an evaluation-centric approach is the most critical for success. It’s easy to build a demo, but productionizing an application is hard, primarily because it’s difficult to define and measure what is “good” or “bad.” Evals are the enablers that help you build better products.

The fundamental goal of any “Ops” discipline, from DevOps to MLOps and now LLMOps, is to enable faster iteration. A streamlined process for evaluating quality, debugging issues, and changing behavior is key. Many teams focus on changing behavior (e.g., prompt engineering, fine-tuning) first, but without a robust evaluation pipeline, you cannot improve a product beyond the demo stage.

LLM Applications at Mercari

Our team works with core business units to deploy LLM-powered features across the Mercari platform. Our approach and architecture are tailored to the specific problem and its constraints.

• Content Moderation: We use a hybrid system where traditional ML models flag potentially suspicious items. Human operators then make the final decision, aided by a knowledge base. We deployed a simple LLM app using a system prompt and RAG over our internal documents to assist the operators. This allows the LLM to use the same knowledge base as the humans, significantly decreasing overall costs without any model fine-tuning.
• AI Listing: This feature generates an item’s title, description, pricing, and metadata from an uploaded image. We found that for this real-time, synchronous feature, the performance gain from a fine-tuned model did not justify the increased infrastructure cost and latency. We opted to use a standard LLM API instead, proving that performance is not always the only factor.
• Metadata Extractor: In contrast, this asynchronous feature suggests metadata improvements for items that haven’t sold. Since we can control the traffic flexibly, we were able to use a fine-tuned LLM to maximize quality.
• Internal Agents & Failed Projects: We successfully use LLM agents for internal Q&A over company documents. However, a customer-facing chatbot for recommendations did not go to production because we struggled to build a reliable evaluation framework for its open-ended nature.

A Tiered and Iterative Evaluation Pipeline

For our AI Listing feature, we developed a multi-stage evaluation pipeline that allows us to iterate quickly and effectively.

1. Initial Vibe Check: We start with simple scripts to test various models and prompts against a small dataset to get a basic sense of performance.
2. Heuristic and Automated Metrics: We filter the initial candidates using a combination of traditional metrics (BLEU, ROUGE) and custom binary classifiers (e.g., a “Japanese detector” to ensure outputs are in the correct language). This step is a cheap and effective way to narrow down the options.
3. LLM-as-a-Judge: For the most promising candidates, we use a more sophisticated and costly evaluation where another LLM scores the output for quality. This involves custom prompts for scoring and classification.
4. Grounding and Fact-Checking: After discovering hallucinations for certain item categories with sparse training data (like trading cards), we integrated a fact-checking step using an external API (Vertex AI) to ensure the generated text is grounded in reality.
5. Domain Expert Feedback: The final and most crucial step is always human evaluation. We provide intuitive UI tools to our subject matter experts (SMEs) so they can provide qualitative feedback, which is the ultimate source of truth.

Key Learnings: Tooling and Iteration

• Start Simple, but Start with a Foundation: You don’t need a complex MLOps stack from day one. Begin with scripts, Git, and a small dataset. The key is to establish a process.
• Embrace Tooling for Observability and Reproducibility: Tools for experiment tracking are essential for scaling. They provide observability into prompts, models, and data, and ensure that any experiment is reproducible. In one week, a single engineer was able to run over 20,000 experiments, get rapid feedback, and iterate towards a production-ready solution because every run was tracked and visualized.
• Use Tiered Evaluations: Don’t run expensive evaluations on every experiment. Use cheap heuristics to filter out bad ideas first, then apply more expensive methods like LLM-as-a-Judge to the best candidates.
• Evaluate Your Evals: If your metrics don’t seem to align with human judgment, it’s time to evaluate the evaluation metric itself. The goal is to achieve alignment between your automated metrics and human feedback.

Ultimately, attention is not all you need; sometimes, evals are all you need. Our experience has shown that a good evaluation strategy can be more impactful than advanced techniques like fine-tuning or RAG. The entire industry, including foundation model developers, is suffering from a lack of good evals, which means building better evaluation systems is one of the most valuable things we can do.

• Tags:
• Evaluation
• LLMOps
• Mercari
• Observability
• Reproducibility
• Production ML
• AI Listing