Scaling Enterprise-Grade RAG: Lessons from Legal Frontier - Calvin Qi (Harvey), Chang She (Lance)

Harvey, an AI assistant for legal professionals, tackles the unique challenges of building enterprise-grade Retrieval-Augmented Generation (RAG) systems for the legal, tax, and compliance domains. This requires handling massive, complex datasets with a strong emphasis on accuracy, privacy, and performance.

The Challenge: Scale and Complexity in Legal RAG

Building for the legal industry presents a unique set of problems beyond typical RAG implementations.

• Data Scale and Variety: Harvey processes data at multiple scales:

  • On-demand: Small batches of 1-50 documents for immediate analysis.
  • Vaults: Larger project-based collections, such as data rooms for M&A deals or litigation documents.
  • Data Corpuses: Massive knowledge bases, like the complete legislation and case law of a country, often containing tens of millions of lengthy, dense documents.

• Query Complexity: Legal queries are not simple semantic searches. They are often multi-faceted and require a system that can deconstruct them. For example, a typical query might be:

  “What is the applicable regime to covered bonds issued before 9 July 2022 under the directive EU 2019/2062 and article 129 of the CR?”

  This single query combines:

  • Semantic search: “applicable regime”
  • Keyword/ID match: “EU 2019/2062”, “article 129”
  • Implicit filtering: Documents before a specific date.
  • Domain-specific jargon: “CR” (Capital Requirements Regulation).
  • Multi-part logic: Relating a directive to a separate article.

• Domain Specificity: The data is deeply complex. Understanding legal structures and nuances is critical for effective preprocessing, indexing, and retrieval. This requires close collaboration with legal domain experts to translate their knowledge into data representation and system logic.

• Security and Privacy: Given the sensitive nature of legal work (e.g., confidential IPO filings, M&A deals), data security, segregation, and strict retention policies are non-negotiable.

Evaluation-Driven Development is Key

With such complex requirements, ensuring system quality is paramount. Harvey invests heavily in “eval-driven development” as the key to building robust systems in a domain where engineers are not the experts. There is no single silver bullet for evaluation; instead, a spectrum of methods is used:

• High-Fidelity / High-Cost: Direct review of outputs by expert lawyers, providing the highest quality feedback.
• Expert-Labeled Criteria: Experts create labeled datasets and criteria, which can then be used for more automated or synthetic evaluation. This is more scalable than direct review.
• Automated Metrics: For rapid iteration, quantitative metrics are used, such as retrieval precision and recall, or deterministic checks like verifying if a retrieved document is from the correct folder or contains specific keywords.

The Need for New Data Infrastructure: LanceDB

The unique combination of massive scale, diverse workloads (search, analytics, training), and multimodal data requires a new kind of data infrastructure. This is where LanceDB comes in, positioning itself not just as a vector database but as an “AI-native multimodal lakehouse.”

The core of LanceDB is the open-source Lance format, designed from the ground up for AI workloads. It can be thought of as “Parquet + Iceberg + secondary indices, but for AI data.” Its key advantages include:

• Unified Storage: It stores multimodal data—text, images, videos, audio, embeddings, and tabular data—in a single table. This eliminates the need for multiple data copies and the complex pipelines required to keep them in sync.
• High Performance: The format provides fast random access (critical for search and shuffle operations) and fast scans (for analytics and model training).
• Optimized for AI Data: It is uniquely suited for storing a mix of large blob data (like document pages) and small scalar data (like metadata).
• Ecosystem Compatibility: Built on Apache Arrow, it integrates seamlessly with existing data tools like Spark, Ray, PyTorch, and Polars.

LanceDB’s distributed architecture builds on this format to deliver a system that separates compute, memory, and storage. This allows it to serve queries with low latency from cloud object storage, drastically reducing costs while handling billions of vectors and enabling both online serving and offline batch processing. This foundation provides the flexibility and power needed to support Harvey’s complex, large-scale RAG applications.

Key Takeaways

1. Creative Solutions for Domain Challenges: Domain-specific problems in fields like law demand creative solutions that go beyond standard RAG techniques, focusing on understanding the data’s structure and the user’s implicit needs.
2. Build for Iteration and Flexibility: The AI landscape is evolving rapidly. Grounding development in a strong evaluation framework allows for rapid, confident iteration as new models and techniques emerge.
3. Modern Infrastructure for Modern AI: The era of multimodal data and diverse AI workloads requires new infrastructure. A lakehouse architecture like LanceDB’s, which unifies data for search, analytics, and training, is essential for building scalable and cost-effective AI products.

• Tags:
• LLM
• RAG
• Feature
• Enterprise AI
• Vector Database
• LanceDB
• Legal Tech
• Retrieval
• Data Infrastructure