Speculative decoding

Sep 17, 2026

OpenAI Built Its Own Chip in 9 Months

Richard Ho, VP of Hardware at OpenAI, details the development of 'Jalapeño,' OpenAI's first custom AI accelerator. The chip aims to drastically lower inference costs and empower full stack control, achieving a remarkable 9-month tape-out by leveraging AI in chip design. Jalapeño's blank-slate architecture brings memory and compute closer, delivering both high throughput and low latency, outperforming competitors even without speculative decoding, and proving that AI can supercharge expert productivity in complex engineering fields.

Sep 08, 2026

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

A workshop by Harshul Jain and Tanmay Sah breaks down LLM inference optimization from first principles, addressing memory, TTFT, and throughput issues. It covers foundational concepts like KV cache mathematics and the prefill/decode phases, then dives into model-side optimizations (quantization, attention variants like GQA, FlashAttention) and serving-side techniques (Paged Attention, continuous batching, prefix caching, KV quantization). The session concludes with benchmarks of popular inference engines like vLLM and SGLang, highlighting their strengths for different workloads, and outlines future research directions.

Sep 01, 2026

Building AI That Optimizes AI

Wafer (S24) is a fast AI inference cloud that runs open-source models at industry-leading speeds by using AI agents to optimize GPUs across the entire stack. This unique approach allows them to deliver the low cost of open source with the latency of much smaller models. The company recently raised a $40 million Series A after achieving rapid growth by making GLM 5.2 run 2-3x faster than competitors, driving significant customer adoption due to speed, not just cost, which is crucial for latency-sensitive applications like voice agents.

Jul 28, 2026

Green AI: Making Machine Learning Environmentally Sustainable • Charles Humble • YOW! 2025

Charles Humble explores the significant environmental impact of AI, particularly generative AI, on global carbon emissions. He offers practical, actionable strategies across the entire AI lifecycle—from project planning and data collection to training and deployment—to reduce this footprint. Key recommendations include questioning the necessity of AI solutions, choosing open-source models, leveraging carbon-aware computing for demand shifting, employing model compression techniques like distillation and quantization, and integrating sustainability as a fundamental architectural principle in software engineering.

Jul 28, 2026

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

Explore the strengths and optimal use cases of `llama.cpp` and `vLLM` for local LLM inference. `llama.cpp` excels on consumer hardware with optimizations like quantization and CPU support, while `vLLM` is designed for production-scale efficiency with features like continuous batching and speculative decoding on high-performance accelerators.

Jul 08, 2026

The 100,000 Sandbox Problem — Akshat Bubna, Modal CTO

Modal CTO Akshat Bubna discusses the company's shift from developer to agent experience, highlighting why traditional cloud infrastructure fails for bursty AI workloads. He details Modal's primitives like elastic inference with GPU snapshotting and speculative decoding, agent sandboxes for RL rollouts, multi-node training with RDMA, and a "supercloud" strategy across 17 providers. The conversation also covers the importance of observability, hard guardrails for production agents, and AI's role in making infrastructure exciting again.