Interactive discovery
Follow the connections between themes, people, and ideas across the Tokenless archive in an interactive topic modeling map.
SWE-Marathon introduces a benchmark for long-horizon autonomous software engineering, pushing coding agents from bug fixes to full project ownership. It highlights the critical need for robust, multi-layered verification and anti-cheat mechanisms to prevent reward hacking in tasks spanning hundreds of millions of tokens, revealing that current agents achieve only a 26% success rate.
RL Nabors discusses the significant costs associated with using frontier AI models, covering security, latency, and financial implications. She introduces a framework for right-sizing AI solutions by leveraging smaller, task-specific models and Small Language Models (SLMs). The framework details how to prove task feasibility, establish success criteria with golden datasets, conduct capability evaluations (using tools like Phoenix), and select the most appropriate "Small And Good Enough" (SAGE) model. Nabors further demonstrates how prompt engineering, particularly few-shot prompting, and post-processing can close performance gaps with larger models, while advocating for continuous regression evaluations to maintain performance integrity. The overarching message is to "prototype big, deploy small" to optimize AI deployments.
Vaidas Razgaitis, Senior Research Engineer at Higharc, shares three tactical tips to accelerate the transition of novel AI/ML research into production-ready features. He emphasizes addressing the critical handoff challenge between ML researchers and software engineers through structured documentation (Research Prototype Taxonomy Document), a well-organized monorepo utilizing decoupled microservices, and a systematic approach to code decomposition and PR review. These strategies aim to improve legibility, maintainability, and delivery speed for ML-driven products.
This video tackles the overwhelming choice of agentic AI frameworks by categorizing projects into five types: linear workflows, autonomous multi-agent systems, role-based AI, production orchestration, and rapid prototyping. It details each type with examples and recommends specific frameworks like LangChain, AutoGen, and CrewAI, guiding developers to select the optimal tool based on their system design and real-world needs.
Booking Holdings CEO Glenn Fogel discusses his journey from Priceline's early struggles to leading a global travel giant. He details Booking's strategic adoption of AI for agentic travel planning and customer service, exemplified by Priceline's 'Penny,' and its significant capital investment in technology. Fogel emphasizes continuous innovation over "moats" and shares insights on AI's impact on job displacement, advocating for proactive employee upskilling to navigate the rapidly changing technological landscape.
Modal CTO Akshat Bubna discusses the company's shift from developer to agent experience, highlighting why traditional cloud infrastructure fails for bursty AI workloads. He details Modal's primitives like elastic inference with GPU snapshotting and speculative decoding, agent sandboxes for RL rollouts, multi-node training with RDMA, and a "supercloud" strategy across 17 providers. The conversation also covers the importance of observability, hard guardrails for production agents, and AI's role in making infrastructure exciting again.
Zipline co-founder Keller Rinaudo Cliffton and Eric Watson discuss how their autonomous logistics system evolved from addressing critical needs in Rwanda to becoming the largest commercial autonomous system globally. They highlight that the drone is only 15% of the solution, emphasizing the deep integration of software, vertical hardware design, advanced safety protocols like compute failover, and extensive testing required. The discussion also covers the immense market potential for autonomous delivery, the impending cost-effectiveness over traditional methods, and the necessary transformation of air traffic control to support a future of pervasive aerial autonomy.
Trisha Gee delivers a compelling talk on Developer Productivity Engineering (DPE) for testing, dissecting common pain points in writing, troubleshooting, and running tests. She advocates for strategic use of IDEs, advanced tooling like build caches and predictive test selection (leveraging ML), and a disciplined approach to test design to overcome these challenges, emphasizing that good tests serve as crucial living documentation.
This episode delves into Q-Day, the anticipated future when quantum computers can break public key cryptography, and the U.S. Executive Order accelerating the transition to post-quantum cryptography. Experts discuss why Q-Day is a gradual process rather than a sudden event, the critical importance of "crypto-agility" as a long-term strategy, and the necessity for organizations to begin immediate discovery and planning to secure data against "collect now, decrypt later" threats. The discussion also touches upon the broader, transformative benefits of quantum computing beyond just security.
David Karam, formerly of Google Search, presents a pragmatic framework for enhancing RAG systems, advocating a "quality engineering" approach. The talk progresses through a ladder of techniques, from in-memory retrieval and BM25 to custom embeddings, re-ranking, and advanced orchestration, emphasizing that the choice of technique should be driven by empirical analysis of system failures ("loss analysis") and balanced by a "complexity-adjusted impact" mindset.
Jared Kaplan, co-founder of Anthropic, explains how the discovery of predictable, physics-like scaling laws in AI training provides a clear roadmap for progress. He details the two main phases of model training (pre-training and RL), discusses how scaling compute predictably unlocks longer-horizon task capabilities, and outlines the remaining challenges—memory, nuanced oversight, and organizational knowledge—on the path to human-level AI.
OpenSAFELY offers a new paradigm for medical research by bringing analysis code to sensitive patient data, rather than exporting the data itself. This interview explores its secure, privacy-preserving platform, its custom DSL `ehrql`, and how this model enforces reproducible, hypothesis-driven science while navigating the complexities of real-world clinical data.
Will Bryk, CEO of Exa, explains why traditional keyword-based search is insufficient for AI agents and introduces a new paradigm of neural, semantic search. He demonstrates how a hybrid approach, combining neural for discovery and keyword for precision, enables AI agents to perform complex, multi-step information retrieval tasks that were previously impossible.
Researcher Robert Ness discusses the practical implementation of Causal AI, distinguishing it from correlation-based machine learning. He covers the essential role of assumptions about the data-generating process, key Python libraries like DoWhy and Pyro, the intersection with LLMs, and a step-by-step workflow for tackling causal problems.
![[Full Workshop] Building Metrics that actually work — David Karam, Pi Labs (fmr Google Search)](https://img.youtube.com/vi/jxrGodnopHo/maxresdefault.jpg)
This workshop, led by former Google product directors, introduces a methodology for building reliable and tunable evaluation metrics for LLM applications. It details how to create granular 'scoring systems' that break down complex evaluations into simple, objective signals, and then use these systems for model comparison, prompt optimization, and online reinforcement learning.
Get updates straight to your inbox. No spam, just useful content.