Human in the loop

Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind

Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind

SonderMind's AI mental health coach, Sonder, pioneers an eval-driven development approach balancing effectiveness and safety. This involves a clinical feedback loop turning human therapist insights into machine-readable evaluations, an Ethics Engine with modular, LLM-as-a-judge guardrails for evolving clinical guidelines, and a shift from single-prompt agents to a Supervisor/Executor/Evaluator architecture with human oversight to ensure safety and quality in high-stakes mental health conversations. They also open-source clinically reviewed datasets to foster community safety.

Loop Engineering from First Principles — Kyle Mistele, HumanLayer

Loop Engineering from First Principles — Kyle Mistele, HumanLayer

Kyle Mistele, in this presentation, argues that fixing the problem of unreviewable 40,000-line pull requests generated by coding agents doesn't lie in better prompts but in better, control theory-inspired loops. He details how to build agentic control loops with distinct sensors, controllers, and actuators, ensuring incremental, verifiable code changes. Using a practical example of RPC API migration, he demonstrates how to implement these loops with tools like ASGrep, integrate them into CI/CD, enable human feedback, and manage workflow to prevent overwhelming reviewers, ultimately leading to more robust and readable agent-generated code.

Don't Build Agents You Can't Answer For — Addy Osmani

Don't Build Agents You Can't Answer For — Addy Osmani

Addy Osmani discusses the evolving role of software engineers in the age of AI agents, emphasizing a shift from code production to human judgment, accountability, and system ownership. He highlights new challenges like cognitive debt and the "orchestration tax," proposing that the modern engineer's value lies in "answerability," taste, and discerning which paths warrant human investment and responsibility.

Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

The talk addresses the critical flaw in human-in-the-loop AI: humans often surrender cognitive effort, leading to "automation bias" rather than true discernment. Through a Duolingo English Test case study and various interaction design examples, it demonstrates how engineering the human-AI interface—rather than just the model or oversight—can elicit critical thinking, generate high-quality data, and foster a virtuous cycle of AI improvement.

What Is AI Code Review? Fixing Slow PRs & Broken Workflows with AI

What Is AI Code Review? Fixing Slow PRs & Broken Workflows with AI

Anna Gutowska explains how AI code review enhances software development by addressing the slowness and inconsistency of traditional methods. The video delves into the benefits of AI in accelerating reviews, improving code quality, fostering developer learning, and reducing technical debt. It covers the underlying technologies like static/dynamic analysis and LLMs, discusses critical considerations such as over-reliance and context, and provides best practices for integrating AI while emphasizing the indispensable role of human oversight.

The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen

The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen

This talk exposes "Miranda distortion," a critical flaw in AI personas where models, influenced by modern cultural narratives, produce convincing but anachronistic outputs. Current evaluations fail to detect this, prioritizing fluency over fidelity. The speaker proposes "epistemic simulation"—a new paradigm grounded in corpus-bounded, temporally-anchored, and expert-evaluated reasoning—and introduces the "Prism Experiment." This rigorous, pre-registered protocol uses Abraham Lincoln to demonstrate how a weighted rubric, created by historians and de-emphasizing rhetorical fluency, can detect anachronism. It advocates for the "humanist in the loop" as a technical requirement to ensure AI personas are true to their documentary records, not just convincing.