How Harvey Built a Research Lab on a Budget | Gabe Pereyra

Gabe Pereyra, co-founder and president of Harvey, outlines a strategic playbook for application-layer companies to compete with well-resourced frontier AI labs. The core philosophy is to leverage the frontier ecosystem rather than attempting to build everything in-house. This approach centers on developing specialized intelligence within a specific domain.

The playbook begins with building robust benchmarks and training data. Harvey, focusing on the legal domain, has released three significant datasets:

• Legal Agent Bench: A taxonomy of complex tasks for legal associates, covering multiple practice areas like drafting fund formation documents and case law research.
• Contracting Dataset: Designed to teach agents negotiation skills for in-house legal departments.
• Large Diligence Dataset: Harvey’s most recent and largest release, an RL environment with data rooms up to 80 million tokens, enabling research on long-context, complex legal tasks. This is considered one of the largest RL environments released to date.

A critical challenge in legal AI is the sensitivity of client data. To overcome the inability to train on proprietary customer data, Harvey pioneered using domain experts to guide synthetic data generation. Lawyers, trained by individuals like Gabe’s brother, an expert in using coding models, generate incredibly realistic synthetic data. This process starts from a rubric where experts plant specific issues (e.g., missing contracts, inconsistencies) within a mock “data room.” This allows for the creation of input data sets against which models can generate outputs (like diligence memos), and evaluation can check if the planted issues were caught. Companies like Mercor and Snorkel are then used to scale up this process and refine the realism of the generated contracts.

Open-sourcing these datasets was a deliberate and initially controversial decision. The motivation was to validate the datasets’ quality, as widespread use by the research community (similar to ImageNet, CIFAR, MNIST) helps identify issues and generate improvements through community contributions. It also encourages frontier labs to benchmark their new models against Harvey’s datasets, fostering broader adoption and recognition.

Next, the playbook addresses post-training models. Historically, post-training wasn’t worthwhile due to rapid advancements in pre-trained models. However, with the emergence of powerful open-source models like Kimmy 3, GLM 5.2, NeMo-Megatron, and Inkling, it is now feasible to post-train these base models to achieve “frontier intelligence” for specific tasks. Harvey recommends starting by working with multiple “Neo Labs” (e.g., Fireworks.ai, Base 10, Ngram, Trajectory, Applied Compute). This multi-partner strategy allows Harvey to pursue more research projects than internal bandwidth permits, leverage diverse expertise and research philosophies from each lab, and learn more effectively from various approaches to different open-source models. Harvey is also increasingly building in-house post-training capabilities, inspired by efforts like Cursor, to create models like “Composer 1” that integrate synthetic data and neo-lab work.

A crucial component is robust model serving and evaluation infrastructure. Harvey operates in 60 countries with multiple product surface areas and diverse customer preferences, necessitating a complex “model serving matrix.” This infrastructure must handle fallbacks across providers to meet SLAs and now integrates open-source models alongside closed-source ones.

Pre-production evaluation involves:

• Automated Evals: Using proprietary benchmarks (like the legal labs benchmark) to quickly assess model strength and domain-specific performance.
• Human Testing: Side-by-side comparisons with existing models.
• Automated Product Tests: Critical user journeys tailored to specific product surfaces.
• Human Product Testing: Ensuring models fit specific product needs. These signals, combined with cost, latency, and regional availability heuristics, determine if a model enters production.

In-production monitoring includes:

• AB testing for major changes.
• Engagement tracking to monitor expected performance.
• Metrics like uptime and token efficiency.
• Product feedback, including “angry customer emails,” to identify issues.

Harvey also emphasizes implementing “simple open-source switches” and model routing. The first involves identifying areas where open-source models can naively replace larger models (e.g., citation generation) for cost/performance benefits, building internal muscle for managing diverse models. The second involves intelligently routing certain queries to open-source models when direct swaps aren’t feasible. This establishes the foundation for a “post-training flywheel” where feedback (carefully collected, without customer data) informs future data sets and model improvements.

Pereyra concludes with a “Moneyball” analogy, asserting that if application-layer companies can “win on this budget with this team,” they will “change the game.” He envisions a future where “every company will eventually need some version of this playbook” to become an AI company.

Q&A Highlights:

• Training with Sensitive Data (Elaboration): The process for generating synthetic legal data involves lawyers defining a rubric of problems and then “planting” those issues within a large, generated data room (e.g., 10,000 contracts that fit together). This allows for controlled testing and proof-of-concept for law firms without using their sensitive data.
• Competing for Research Talent: Harvey initially struggled to compete with the compensation packages of frontier labs. However, the talent landscape is changing, with more PhDs and individuals seeking alternatives to large labs. Furthermore, the availability of robust API-based infrastructure (e.g., Fireworks, Tinker) for training and serving models reduces the need for specialized “training infrastructure” talent, broadening the hiring pool.
• Designing Challenging Rubrics: The focus is on creating realistic representations of complex legal work. In domains like legal, even frontier models struggle with genuine client matters. Domain experts, having worked with models for years, develop strong intuition for crafting challenging data rooms and rubrics that expose model weaknesses.
• Pipeline Breakdowns (Data, Research, Infra): There’s no single point of failure. Harvey’s strategy involves having product-market fit with existing closed-source models first, building confidence in the end-to-end system. Post-training is then integrated as a small input. Each stage (benchmarks, product fit, user feedback) acts as a gate, making it evident where issues arise. Models that fail benchmarks don’t proceed; those that pass but perform poorly in product (e.g., due to overfitting) are identified by user feedback.
• Tension in Open-Sourcing Benchmarks: Harvey acknowledges the tension of allowing frontier labs to “catch up” but sees strategic advantages. Open-sourcing helps customers understand model capabilities, validates the benchmark’s quality through community engagement, and enables external companies with new techniques to easily test their solutions. Strategically, Harvey aims to differentiate by helping law firms customize AI systems on their private data, while open-sourcing synthetic data for general improvement.
• Biggest Remaining Open Problems:
  1. Bridging Synthetic-to-Production Data Gap: Synthetic data, while realistic, doesn’t perfectly match production distributions, especially for future-facing use cases. The challenge is connecting synthetic data to current product needs without accessing sensitive customer data.
  2. Long-Context Management: Models still struggle to effectively manage extremely long contexts, such as 80 million tokens in diligence data rooms. Training models to operate in these complex environments remains a significant performance gap.
  3. Continual Learning for Customization: The ultimate goal is not a single “best legal model,” but rather helping individual law firms and enterprises customize AI systems to their specific work. Operationalizing continual learning where an AI system improves with every client matter, while protecting client data, presents a massive technical, operational, and AI challenge.
• Competing with Horizontal Products (e.g., Coda): Harvey competes by hyper-verticalization, shifting focus from individual productivity (like general workflow tools) to organizational productivity. For large law firms, the problem is managing thousands of client projects, coordinating teams (20-30 people), orchestrating humans and agents, and ensuring profitability across an entire firm. This involves complex project management and resource allocation at a scale horizontal products don’t address, making Harvey’s solutions deeply embedded in domain-specific workflows.

• Tags:
• Synthetic Data
• Feature
• AI Strategy
• Benchmarking
• Legal AI
• Continual Learning
• Open Source Models
• Long Context AI
• Machine Learning Applications
• Model Post-Training
• Model Serving
• Evaluation Infrastructure
• Hyper-Verticalization