Speculative decoding

Jul 28, 2026

Green AI: Making Machine Learning Environmentally Sustainable • Charles Humble • YOW! 2025

Charles Humble explores the significant environmental impact of AI, particularly generative AI, on global carbon emissions. He offers practical, actionable strategies across the entire AI lifecycle—from project planning and data collection to training and deployment—to reduce this footprint. Key recommendations include questioning the necessity of AI solutions, choosing open-source models, leveraging carbon-aware computing for demand shifting, employing model compression techniques like distillation and quantization, and integrating sustainability as a fundamental architectural principle in software engineering.

Jul 28, 2026

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

Explore the strengths and optimal use cases of `llama.cpp` and `vLLM` for local LLM inference. `llama.cpp` excels on consumer hardware with optimizations like quantization and CPU support, while `vLLM` is designed for production-scale efficiency with features like continuous batching and speculative decoding on high-performance accelerators.

Jul 08, 2026

The 100,000 Sandbox Problem — Akshat Bubna, Modal CTO

Modal CTO Akshat Bubna discusses the company's shift from developer to agent experience, highlighting why traditional cloud infrastructure fails for bursty AI workloads. He details Modal's primitives like elastic inference with GPU snapshotting and speculative decoding, agent sandboxes for RL rollouts, multi-node training with RDMA, and a "supercloud" strategy across 17 providers. The conversation also covers the importance of observability, hard guardrails for production agents, and AI's role in making infrastructure exciting again.

Jun 30, 2026

How KV Cache Speeds Up LLMs for Faster AI Models on GPUs

LLMs often slow down under heavy traffic due to inefficient GPU memory management during inference. This overview explains how KV cache and Paged Attention, implemented in VLLM, optimize memory usage across prefill and decode phases, significantly boosting LLM throughput, reducing latency, and improving GPU utilization through advanced context handling and specific tuning techniques like prefix caching and speculative decoding.

May 20, 2026

The Future of AI – Key Trends Shaping What’s Next • Ekaterina Sirazitdinova • YOW! 2025

Ekaterina Sirazitdinova from NVIDIA provides a high-level overview of the latest trends shaping the future of AI, covering the evolution from early deep learning to the rise of agentic and physical AI, and diving deep into the critical optimization techniques required to deploy these powerful models efficiently.

Jul 26, 2025

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

A deep dive into SGLang, an open-source serving framework for LLMs. This summary covers its core features, history, performance optimization techniques like CUDA Graph and Eagle 3 speculative decoding, and how to contribute to the project.