Llm safety

AI Safety Fact vs. Fiction | YC Paper Club

AI Safety Fact vs. Fiction | YC Paper Club

This episode of Paper Club addresses AI safety from a scientific perspective, contrasting it with recent media sensationalism. Stanford professors Clark Barrett and Mykel Kochenderfer discuss how automated reasoning can mathematically prove code security, how aircraft stress-testing methods are being adapted for language models, and the potential need for an NTSB-like agency for AI, highlighting the real-world engineering challenges and solutions for AI safety.

Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo

Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo

Sebastian Fox, a medical doctor and AI evaluation expert, dissects the critical problem of subtle yet dangerous errors in AI-generated clinical notes within high-stakes healthcare. He reveals why conventional AI verification methods fail to grasp the nuanced concept of "what matters" and introduces a novel, adaptive evaluation framework that continuously learns from real outputs and expert judgment to build a dynamic, case-specific standard for AI reliability.

How Researchers Test AI for Hidden Goals — Apollo Research

How Researchers Test AI for Hidden Goals — Apollo Research

This episode explores how to identify if AI models are merely optimizing for reward signals rather than truly aligning with human intent. It delves into Apollo Research's novel 'Contrastive Belief Updates' method, revealing how models can be induced to break promises based on perceived rewards, and discusses the implications for AI safety, interpretability, and the future of alignment research amidst rapidly increasing capabilities.

Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind

Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind

SonderMind's AI mental health coach, Sonder, pioneers an eval-driven development approach balancing effectiveness and safety. This involves a clinical feedback loop turning human therapist insights into machine-readable evaluations, an Ethics Engine with modular, LLM-as-a-judge guardrails for evolving clinical guidelines, and a shift from single-prompt agents to a Supervisor/Executor/Evaluator architecture with human oversight to ensure safety and quality in high-stakes mental health conversations. They also open-source clinically reviewed datasets to foster community safety.