Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind
SonderMind's AI mental health coach, Sonder, pioneers an eval-driven development approach balancing effectiveness and safety. This involves a clinical feedback loop turning human therapist insights into machine-readable evaluations, an Ethics Engine with modular, LLM-as-a-judge guardrails for evolving clinical guidelines, and a shift from single-prompt agents to a Supervisor/Executor/Evaluator architecture with human oversight to ensure safety and quality in high-stakes mental health conversations. They also open-source clinically reviewed datasets to foster community safety.
SonderMind, a mental healthcare company, has engineered Sonder, a clinically grounded AI coach specifically designed for mental health support. Recognizing that general-purpose LLMs are ill-suited for this sensitive domain, Sonder aims to provide support for individuals not yet ready for therapy, or between sessions, and can act as a front door to SonderMind’s human provider network when needed.
Engineering for Safety: Guardrail Architecture
SonderMind’s approach to building a safe mental health AI is rooted in a robust guardrail system and an agent harness architecture, with user safety as the primary objective.
Modular Guardrails
- Input Guardrails: These analyze user messages upon arrival, intervening if necessary before Sonder Core responds.
- Output Guardrails: These evaluate the AI’s response and the overall conversation, intervening if clinical safety is at risk to keep the conversation on track.
- Modularity: Designed for the complex and nuanced mental health space, modularity is key. This allows for iteration on Sonder Core without compromising user safety and facilitates a straightforward evaluation process.
- Separate LLMs as “Judges”: A critical design choice is to use separate LLMs to act as “judges” for guardrail decisions. This makes them more robust and harder to circumvent or “jailbreak” through prompt engineering, despite trade-offs in latency and cost. This sensitivity of the use case warrants such a robust approach.
Overall Architecture
The agent harness incorporates separate LLMs for input and output guardrails, alongside Sonder Core (handling memory and personalization), and integrates with analytics and alerting platforms to monitor system behavior and safety.
Clinical Nuance & Guardrail Philosophy: Correct Triggers
General-purpose LLMs are often overly conservative, leading to inappropriate guardrail triggers that can make vulnerable users feel isolated. SonderMind aims for correct triggers, not merely more triggers, understanding the deep nuance required in mental health conversations. Examples illustrate this:
- Active Crisis (e.g., domestic violence): If a user indicates immediate danger, Sonder surfaces local resources and disengages, as direct engagement is not appropriate.
- Past Trauma/Distress (e.g., questioning assault): If a user is processing past events, not in an active crisis, Sonder provides resources but may continue engagement if the user desires, as continued conversation is not an immediate safety risk.
- General Challenges (e.g., relationship issues): If no safety risk is present, the guardrails remain transparent, and Sonder Core responds directly.
This nuanced calibration is achieved through extensive collaboration with clinicians.
Trusting the Guardrails: The Clinical Feedback Loop
To ensure guardrails are trustworthy, SonderMind employs a clinical feedback loop that addresses the challenge of interpreting “the sentence underneath the sentence”—indirect, coded messages that imply distress or risk.
Human-in-the-Loop Annotation
- Tracing Conversations: Conversations are traced and captured, allowing clinicians to review and annotate moments where guardrails should have intervened or where the AI’s response was incorrect.
- Clinician-Defined Correctness: A licensed professional, not the AI system, defines “correct” behavior in clinical edge cases.
- Annotation Process: Clinicians annotate traces using a rubric, specifying the expected observation (assertion), the turn index where intervention should occur, and notes for engineers. An extraction script converts these annotations into typed evaluations.
Impact on System Improvement
These clinician-validated evaluations are then integrated into the Continuous Integration (CI) pipeline. Every prompt, model, or guardrail change must be scored against these clinician-taught scenarios. This ensures that fixes for specific instances (e.g., a coded self-harm message) lift the entire category, enhancing the system’s overall safety and correctness.
Calibration and Design Choices
SonderMind’s design philosophy for guardrail calibration prioritizes genuine human needs over