An overview of OpenAI's approach to AI evaluation, covering the GDP-val benchmark for frontier models and the practical tools available for developers to evaluate their own custom agents and applications.
Developers from OpenAI showcase "Storyboard", a custom tool built in 48 hours that transforms creative workflows. They detail how, in partnership with a film production, they used AI-assisted coding to rapidly prototype a tool that takes artists from initial sketch to high-fidelity images and even full-motion video, drastically reducing production timelines.
Distinguished Engineer Chris Hay demonstrates how to run and program Large Language Models (LLMs) locally in just two lines of Python code. The tutorial covers setting up a local environment with Ollama and UV, using a custom library for simplified interaction, and explores advanced topics like asynchronous streaming, persona customization with system prompts, and managing multi-turn conversations.
Sam Altman discusses OpenAI's strategy, the path to AGI through world models like Sora, the importance of societal co-evolution with AI, and the massive infrastructure and energy requirements for future models. He covers topics from AI safety and regulation to monetization and the future of scientific discovery driven by AI.
Drawing from their work fine-tuning an ASR model in Wolof and building a stereotype detection dataset, researchers from YUX share a practical toolbox for evaluating the cultural relevance of AI models and products. The session covers methods for data collection, model benchmarking, user testing, and introduces LOOKA, a platform for scalable human evaluation in the African context.
Qinge Xie from Georgia Tech presents a large-scale evaluation of modern website privacy policies using a novel LLM-based framework. The research systematizes privacy practices from 10 major US and EU regulations into 34 clauses and analyzes over 100,000 websites to reveal current trends in data collection, sharing, and consumer rights disclosure.