Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab

Felipe Blanes from the Amazon AGI Lab shares lessons from working directly with customers of Nova Act, Amazon's browser agent service, from research preview to general availability on AWS. He argues that static benchmarks leave teams hoping their evals match real usage, and proposes a customer-driven eval flywheel, the trust cliff around 90% reliability, the value of being open about limits, and principles for keeping evals fresh.