A product and go-to-market lead at NVIDIA examines the economics of AI inference as agentic AI multiplies token demand. The talk covers matching premium versus efficient models to use cases, model routing, open models, sizing token demand, and extreme co-design across mixture-of-experts models, NVLink systems, KV caching, software stacks, and CPUs to maximize tokens per watt and minimize cost per token.
Charles Frye explains the architecture, performance engineering, and operational challenges of LLM inference engines. He dissects workload types, core components like schedulers and kernel backends, and advanced techniques such as KV caching, CUDA graphs, and speculative decoding, concluding with essential strategies for observability and debugging in production environments.
Independent model evaluation is crucial for AI progress, as public benchmarks are saturated and models optimize for them. Rayan Krishnan discusses VALS's role in providing continuously evolving, third-party evaluations, highlighting the Llama 4 'disaster' and the need to measure increasingly agentic systems. The conversation covers the complexities of evaluation beyond capability to include cost and latency, the looming financial impact of token spend on enterprises, and the critical role of evaluations in informing AI policy and international coordination for aligning AI with human intent and managing geopolitical risks, especially concerning recursive self-improvement.
This episode explores the dynamic landscape of the AI industry, dissecting IBM's strategic alliance with OpenAI for enterprise AI integration, Stripe's acquisition of OpenRouter to capitalize on AI token routing, and insights from Ramp's AI Index revealing evolving AI spend and the rise of open-source models. It highlights the shift from model-centric to infrastructure and governance-centric AI, and touches on the ethical implications of AI-drafted legislation.
Denny Lee discusses the industry's shift to meta-harnesses like Omnigent, which enables hot-swappable AI models and agents, illustrated by his personal project of using debating agents to plan a matcha farm in Taiwan. He highlights how "tokenomics" is replaying the CapEx-to-OpEx cost shift, emphasizing the need for developer visibility, central governance, and auto-model selection to manage AI spend. The conversation also touches on the importance of databases for agent memory and accountability in AI-assisted workflows.
Discover the AI Engineering World's Fair 2026, the largest iteration yet, offering an unparalleled deep dive into AI engineering with expanded tracks on auto research, GPU specialization, and new verticals like finance and healthcare. Highlights include an innovative expo experience, exclusive leadership initiatives like the "Token Billionaires Program," and unique side events fostering community, including "Posters on AI" where attendees can defend their tweets. This event is designed to be a curated hub for practical, cutting-edge insights and networking in the AI/ML professional landscape.