Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo

This talk addresses the critical issue of AI-generated errors in clinical notes, particularly in high-stakes environments like healthcare, and proposes an adaptive, continuous evaluation framework. Sebastian Fox, a medical doctor and co-founder of Composo.ai, highlights that these errors often go unnoticed and can have severe consequences.

The Unseen Dangers of AI in Healthcare

Fox begins with a compelling example: a clinical note generated by AI that describes a headache as “routine tension type,” recommending paracetamol. What the note omits, however, is the patient’s additional symptom of jaw pain when chewing. For a new headache in a patient over 50, jaw pain is a red flag for giant cell arteritis (GCA), a condition that can lead to blindness within days if untreated. The danger lies not in what the note says, as “nothing in the note is technically wrong,” but “what isn’t there.”

Beyond these subtle omissions, more egregious errors are also common. Examples include an AI misdiagnosing a sore throat as suspected angina, prescribing diabetes medications the patient doesn’t take, or referring to a non-existent hospital. While such blatant errors are usually caught, the subtle ones that “sit in the record uncalled” pose a greater challenge and potential for harm.

Prevalence and Impact of AI-Generated Errors

In the largest real-world study of AI-generated clinical notes, approximately 1 in 20 notes contained an error serious enough to cause significant patient harm. Expanding the scope, nearly 1 in 5 had an important omission, and over 1 in 10 contained a hallucination. These are production errors, not theoretical test cases. Despite ambient scribes now being deployed in about a third of US practices, these errors rarely surface as reported incidents, leading to a situation where we are “flying blind.” While healthcare provides a visceral example due to life-and-death stakes, this problem extends to all high-stakes AI applications.

Anatomy of AI Failures

Analyzing outputs from three leading production ambient scribes, Fox identifies various failure modes:

• Misinterpretations/Inferences: An AI might infer “abrupt sudden onset” from a patient stating a headache “just happened,” which is a red flag for a brain bleed, even if the patient never used those exact words.
• Contradictory Plans: The AI might record an initial treatment plan discussed, rather than the final, agreed-upon decision (e.g., keeping “arrange tests today” when the team decided to try antibiotics first).
• Transcriptional Errors: Although not the primary focus, misheard words can lead to critical errors, such as “Humalog” becoming “Humulin” (different insulins) or “no evidence of cancer” becoming “evidence of cancer.”
• Core LLM Failures: Even with a perfect transcript, models add information never said, change what was said meaningfully, or omit crucial details. The challenge is distinguishing a “slight over-inference versus the dangerous fabrication” or a “harmless rephrase versus the meaningful edit.”

At the heart of these failures is the AI’s lack of “taste” or judgment regarding “what matters.” This judgment is tacit (experts possess it but can’t fully write it down), contextual (the same detail can be critical or noise depending on the situation), and moving (guidelines, models, and medical consensus evolve). Since the standard is not fixed, models, despite their capabilities, struggle to understand the significance of information for a specific example.

Why Traditional Verification Fails

A common approach to mitigating generator errors is to place a checker after the AI model, leveraging the asymmetry of verification (it’s often easier to check an answer than to generate it perfectly). State-of-the-art verification systems might include detailed faithfulness rubrics with worked examples, automated rubric optimization, and deterministic concept counting.

However, Fox’s research, using such advanced systems, revealed that even the “best version of a judge” still waved through 1 in 5 notes containing serious errors, particularly omissions. These judges are not unintelligent; they are frontier models with sophisticated engineering. The problem is that while they are “faithful to the words,” they can be “a lie about the intent.” The checker only finds what’s wrong if the standard of “good” was written down in a way it can read. For complex, high-stakes tasks, “the standard of good was never written down anywhere that the judge can read it.” The rubric only captures the “taste you could write down,” not the critical, unwritten judgment.

This is exemplified by the “France versus Lake Malawi” scenario: two patients both present with blood in their urine, and their notes both drop the detail of where they vacationed. For one, a trip to France is irrelevant noise. For the other, a trip to Lake Malawi in sub-Saharan Africa is the critical diagnostic clue for schistosomiasis, completely altering the management plan. The same dropped line is noise in one context and the answer in another—a distinction impossible to fully codify in advance.

A Continuous, Adaptive Evaluation Loop

Inspired by methods like Reinforcement Learning from Human Feedback (RLHF) which learn reward functions from examples, Fox proposes that the solution is not to bake the standard into weights or static prompts, but to keep the “taste as the examples themselves.” This involves a continuous, three-step loop:

1. Discover: Instead of guessing potential failures, put the system into production and discover real failure modes from actual outputs. Cluster these errors to build a “failure mode ontology” – a map of how the system genuinely goes wrong, including unforeseen issues.
2. Capture: Engage experts to provide detailed judgment on these real outputs. This involves not just assigning a score but also documenting their reasoning and corrections. This process builds a rich record of “how your experts actually judge.”
3. Calibrate: For each new output, the judging agent dynamically assembles a case-specific standard. It pulls in similar past judgments, expert corrections, and relevant guidelines, effectively performing “context engineering per output.” This ensures the model is judged against a living, evolving standard tailored to the specific case, rather than a static rubric or a frequently retrained model.

This adaptive loop constantly sharpens its judgment with every new output and expert correction. Revisiting the initial headache example, a capable model, given the right context from this loop (e.g., similar cases of red flags misfiled as routine, criteria for GCA), would easily identify the missed red flag. It wasn’t hard to catch; the AI simply “didn’t know what mattered.”

Results and Broader Implications

Comparing different judging systems on the same dataset, an off-the-shelf judge misses most critical errors, and even a sophisticated system with rubrics and deterministic checks still misses a significant portion. However, the adaptive loop judge, which discovers failure modes and calibrates against expert-judged outputs, performs substantially better. The key difference is not more compute or a better prompt, but how the judge learns what truly matters. This approach avoids fighting “taste” and instead embraces its dynamic nature.

This methodology is applicable to any high-stakes AI where “being confidently wrong has a cost,” such as contract review that misses critical clauses or support agents promising unoffered refunds. Fox concludes by emphasizing that “evaluation isn’t something you have, it’s something that you do continuously over time.” The easiest starting point is collecting free-form expert comments on real AI outputs, which provides the raw material for building this evolving standard.

• Tags:
• Feature
• Model Reliability
• Machine Learning
• AI Evaluation
• Continuous Learning
• Medical AI
• Healthcare AI
• Domain Expertise
• LLM Safety
• Clinical Notes
• Ambient Scribes
• Error Detection
• Contextual Reasoning
• Sebastian Fox
• Composo.ai