Oct 02, 2026

Stop Renting Your AI's Memory — Dylan Couzon, Qdrant

Dylan Couzon, developer relations engineer at Qdrant, argues that frontier-class models now run on hardware people can own, but the memory that makes an assistant personal still lives in someone else's cloud. He covers why owning inference gives autonomy while owning memory gives continuity, the CPU, RAM, and disk model for LLMs, why retrieval beats stuffing everything into the prompt, and a live offline demo of a drone building searchable memory with Qdrant Edge, plus opt-in shared memory.

Sep 19, 2026

What's New in Inference Engineering — Philip Kiely, Baseten

Philip Kiely provides an update on recent advancements in inference engineering, covering new techniques in quantization (TurboQuant's specific application), KV cache compaction (introducing 'Still' for learned, compressed memory), and speculative decoding (from Eagle 3 to Dlash and DSpark, including continuous retraining). He emphasizes the blurring lines between training and inference and differentiates between local and data center optimization strategies.

Sep 08, 2026

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

A workshop by Harshul Jain and Tanmay Sah breaks down LLM inference optimization from first principles, addressing memory, TTFT, and throughput issues. It covers foundational concepts like KV cache mathematics and the prefill/decode phases, then dives into model-side optimizations (quantization, attention variants like GQA, FlashAttention) and serving-side techniques (Paged Attention, continuous batching, prefix caching, KV quantization). The session concludes with benchmarks of popular inference engines like vLLM and SGLang, highlighting their strengths for different workloads, and outlines future research directions.

Aug 07, 2026

Compression at the Edge — Chris Alexiuk, NVIDIA

This panel discussion explores the critical role of model compression, particularly quantization, in democratizing AI. It delves into how massive models like GLM 5.2 can be shrunk by over 80% without equivalent performance loss, thanks to techniques like mixed-precision quantization and understanding uneven layer importance. The discussion covers NVIDIA's NVFP4 format, challenges posed by new model architectures, the preference for KL divergence over accuracy benchmarks, and the vision of future AI running efficiently on all local devices.

Jul 28, 2026

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

Explore the strengths and optimal use cases of `llama.cpp` and `vLLM` for local LLM inference. `llama.cpp` excels on consumer hardware with optimizations like quantization and CPU support, while `vLLM` is designed for production-scale efficiency with features like continuous batching and speculative decoding on high-performance accelerators.

Jul 25, 2026

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Cormac Brick from Google AI Edge discusses how the increasing constraint of DRAM cost on edge devices necessitates the development and deployment of increasingly smaller AI models. He outlines the work of his team in optimizing models like Gemma, achieving 2.9 bits per weight for a 2 billion parameter model capable of running on a Raspberry Pi at 7.6 tokens/second, or on an NPU at 31 tokens/second decode for vision tasks. The talk delves into 'tiny models' (50M-500M parameters) that extend AI to older devices and enable features like robust voice-to-function calling via fine-tuning with synthetic data, exemplified by an offline voice dictation app.