Tokenless

Llm serving

The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Byung-Gon Chun from FriendliAI details how agentic inference fundamentally differs from traditional chat-based LLM usage, requiring a specialized infrastructure to optimize for end-to-end task completion rather than single request latency. He highlights the economic advantage of open-weight models for agentic workflows and explains FriendliAI's innovative inference stack, featuring prefix caching, advanced KV cache management, cache-aware routing, and agent-aware optimization, which together deliver significant speedups and cost savings for production AI agents.

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

A deep dive into SGLang, an open-source serving framework for LLMs. This summary covers its core features, history, performance optimization techniques like CUDA Graph and Eagle 3 speculative decoding, and how to contribute to the project.