Jul 29, 2026

Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

Jerry Tworek and Rohan Anil, founders of Core Automation, argue that the transformer architecture has reached its limits and the primary bottleneck to smarter AI systems is now architectural. They contend that current models lack continual learning and test-time adaptation capabilities, critical for real-world deployment. They advocate for new architectures that learn from experience more efficiently than current reinforcement learning, optimize pre-training and RL end-to-end, and build a highly automated lab focused on accelerating innovation through kernel generation, aiming for models that can improve themselves without human intervention.

Jul 26, 2026

The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

Poolside discusses their innovative approaches to synthetic data generation, pre-training validation, and distributed training challenges. They highlight how modular data pipelines, rigorous replica hash checks, and numerical stability fixes enabled them to scale their LLMs, culminating in the 118B parameter Laguna S model designed for agentic coding, which shows strong early results against leading open-weight models.

May 08, 2026

How Transformers Finally Ate Vision – Isaac Robinson, Roboflow

Isaac Robinson from Roboflow explains why Vision Transformers (ViTs), despite their initial disadvantages in computational complexity and lack of inductive bias, ultimately surpassed Convolutional Neural Networks (CNNs) for computer vision tasks. The talk covers the critical roles of massive, ViT-specific pre-training methods like MAE and DINO, the architectural evolution through models like Swin, ConvNeXt, and Hiera, and optimizations borrowed from the LLM ecosystem. It culminates in a discussion on the practical deployment challenges of large foundation models like SAM and how Neural Architecture Search can bridge the gap.

Sep 30, 2025

How To Train An LLM with Anthropic's Head of Pretraining

Anthropic's Head of Pre-training, Nick Joseph, details the immense engineering and infrastructure challenges behind training frontier models like Claude. He covers the evolution from early-stage custom frameworks to debugging hardware at massive scale, balancing pre-training with RL, and the strategic importance of data quality and team composition.