Three experts discuss the evolving landscape of data in AI, covering its critical role in model performance, challenges in sourcing and evaluating expert data, the development of advanced data generation techniques for diffusion models, and the complexities of multilingual pre-training.
The talk by Sangwha Lee discusses Krea 2's open-source medium variant, emphasizing stylistic diversity and rapid iteration over the consistency-focused approach of larger models. A significant portion details their robust data curation pipeline, including unique methods for deduplication, filtering out AI-generated images, utilizing sparse autoencoders for unsupervised tagging, and ensuring world knowledge coverage. He outlines an LLM-inspired multi-stage training process, culminating in a prompt expander, and shares insights on fast iteration and future directions for image generation, highlighting the increasing integration of VLM advancements.
Matthieu Wyart, a statistical physicist, argues that deep networks discover abstractions by recovering hidden data hierarchies, allowing them to escape the curse of dimensionality. He explains how this mechanism, combined with predicting latent representations instead of raw tokens, can significantly improve sample efficiency. The discussion also covers the physics of rough loss landscapes, machine creativity, diffusion models, and a theoretical framework for neural scaling laws.
This Paper Club delves into the current state of robotics, addressing roadblocks like the sim-to-real gap and embodiment drift. Speakers present advancements in multi-scale memory for long-horizon tasks, self-supervised embodied reasoning, zero-shot dexterous manipulation via massive simulation, and the economic imperative of teleoperation-first robotics companies, concluding with optimizations for efficient, real-time World Action Models.
Chai Discovery is revolutionizing drug discovery by treating biology as an engineering problem, leveraging AI—particularly diffusion models and the "bitter lesson" of scaling—to design molecules rather than merely discover them. Their approach has boosted antibody design hit rates from 0.1% to 16%, aiming for a "Molecular CAD" suite that collapses discovery timelines from months to days. They partner with pharma, building infrastructure and creating a data flywheel to develop higher-quality, more targeted medicines for previously undruggable diseases.
Ziv Ilan from NVIDIA details how latency in video diffusion models can be drastically reduced to achieve real-time generation. He presents a layered approach combining dynamic quantization for memory and speed, chunk-based caching to skip redundant denoising computations, and, most critically, step distillation—training models to achieve high-quality output in significantly fewer steps. These techniques, packaged in the open-source FastGen repository, offer additive performance gains, enabling real-time video on a single Blackwell B200 GPU.