How to Leverage Domain Expertise — Chris Lovejoy, Notius Labs

Winning in vertical AI is fundamentally an organizational problem, not a technological one. The system for incorporating domain insights is more important than the sophistication of your models. The primary gap in building successful AI products is not the power of foundation models, but how organizations can effectively operationalize the expert judgment required to guide them.

The most common mistakes organizations make are:

1. Not hiring domain experts, or hiring them too late.
2. Hiring the wrong type of domain expert.
3. Failing to integrate them correctly into the organization.

Appraising AI quality requires judgment, and the best judgment comes from domain expertise. This expertise can be formal (e.g., a doctor for a healthcare product) or informal (e.g., a writer with a “taste” for good meeting notes).

Three Archetypes for Domain Experts

A framework for integrating domain experts into an AI organization can be understood through three primary roles: the Oracle, the Evaluator, and the Architect. These roles define how an expert participates in the two core stages of product development: assessing AI performance and improving it.

• The Oracle: The domain expert directly performs both assessment and improvement. They review AI outputs, analyze where they go wrong, and then directly make changes, often by tweaking prompts, adding documentation, or adjusting tools. This is a hands-on, iterative loop owned entirely by the expert.

• The Evaluator: The domain expert focuses on assessment by defining and measuring quality. They establish objective metrics, set up review systems (e.g., using other domain experts, customer feedback, or LLM-as-judge), and analyze performance data. They then feed these insights to an engineering team, which is responsible for implementing the improvements.

• The Architect: The domain expert steps back to design a system that automates both assessment and improvement. Instead of being in the loop, they build the loop. Their goal is to create a product that learns and adapts from user interactions and other feedback mechanisms with minimal manual intervention.

Choosing the Right Model

The appropriate model depends on your use case and scale. The decision process can be guided by a few key questions:

1. Can performance be measured with objective metrics?
   • If No, and quality is a matter of “taste,” the Oracle model is most suitable. The main consideration is whether one person is sufficient or if you need multiple Oracles for different segments. .
2. If Yes, is manual iteration by engineers fast enough?
   • If Yes, the Evaluator model works well. The expert defines quality, and engineers iterate on the product.
   • If No, and you need the system to adapt more dynamically, the Architect model is necessary to build self-improving systems.

These roles often evolve. A startup might begin with an Oracle, but as it scales, the expert may need to transition into an Evaluator or Architect to handle the increased complexity and volume.

Case Studies in Practice

• Granola (AI Meeting Notes): Granola exemplifies the Oracle model. Its first employee was a writer who, after extensive research, established the standard for a “good” meeting note. Even at scale, she remains the primary gatekeeper of AI quality, personally reviewing outputs and refining prompts. This works because the quality of a meeting note is subjective and not easily captured by objective metrics.

• Tandem (Medical AI Scribe): Tandem started with a single doctor as an Oracle, reviewing medical notes and updating prompts. As the company expanded to numerous medical specialties, countries, and use cases, this approach didn’t scale. They evolved to a Decentralized Oracle model, building a platform that allows many different doctors to customize prompts for their specific contexts, effectively managing the long tail of variations.

• Anterior (Healthcare Prior Authorization): Anterior’s journey shows the full evolution from Oracle to Architect.

  1. Oracle: Initially, as the first technical employee and a doctor, Chris Lovejoy reviewed AI-driven authorization decisions and updated the prompts and code himself.
  2. Evaluator: As volume grew, this became untenable. The role evolved to defining metrics (correct vs. incorrect approval), building a review dashboard, and hiring other clinicians to assess outputs at scale. This data was then used to guide the engineering team.
  3. Architect: Manual engineering iteration also proved too slow to handle the vast variation in how different insurance companies interpret medical policies. The final step was to become an Architect, designing systems for automated improvement that could learn these rules dynamically from usage.

How to Leverage Your Domain Expert

To successfully integrate a domain expert, follow three principles:

1. Define a Principal Domain Expert: Appoint a single individual who is ultimately accountable for the quality of the AI’s performance. This avoids slow, committee-based decisions and empowers someone to develop a deep, holistic understanding of the AI’s behavior.

2. Give Them Ownership: Treat the domain expert as a core owner of the product, not a consultant. They need to be in the room where decisions are made. A failure mode is to have experts in an advisory role, which slows progress and often leads to them leaving the company, taking valuable context with them.

3. Hire for Breadth: The ideal candidate has the necessary domain expertise as a baseline, but also a breadth of adjacent skills.

   • Oracle: Requires prompting skills, attention to detail, and customer communication.
   • Evaluator: Benefits from data science intuition, statistical skills, and product management experience.
   • Architect: Needs experience with LLM-powered products and engineering skills.

Hiring for breadth allows the expert to grow with the company and evolve their role as needed, ensuring continuity and leveraging the deep insights gained in earlier stages.

• Tags:
• LLMs
• Feature
• Vertical AI
• Organizational Structure
• Domain Expertise
• AI Products
• Case Study