Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

A deep dive into SGLang, an open-source serving framework for LLMs. This summary covers its core features, history, performance optimization techniques like CUDA Graph and Eagle 3 speculative decoding, and how to contribute to the project.

July 26, 2025 4 min read

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten — source: https://www.youtube.com/watch?v=Ahtaha9fEM0

SGLang is an open-source, high-performance serving framework for Large Language Models (LLMs) and Vision Language Models (VLMs), positioned as a powerful alternative to other frameworks like vLLM and TensorRT-LLM. It is used in production by companies such as xAI (for their Grok models), AMD, and Meituan, handling trillions of tokens daily.

Why SGLang?

The primary advantages of adopting SGLang for model serving include:

High Performance: It offers excellent throughput on a wide variety of GPUs.
Production-Ready: The framework is stable and ready for production workloads out of the box.
Day-Zero Model Support: SGLang provides immediate support for new model releases from labs like Quen and DeepSeek.
Strong Open-Source Ethos: With a vibrant community, users are empowered to fix issues and contribute directly to the codebase, avoiding potential bottlenecks waiting for core developers.

The project has seen impressive growth, going from a paper in December 2023 to nearly 15,000 GitHub stars and wide industry adoption in a very short period.

Getting Started and Core Mechanics

At its core, using SGLang involves running a server command with a set of configuration flags. The key to effective use is understanding the available flags, their supported configurations, and how they interact. For instance, combining a major speculation algorithm with a significantly increased batch size might not yield optimal results, whereas quantization can pair well with other optimizations.

The workshop demonstrated deploying a model using truss to package the SGLang dependencies and server command into a YAML file, which is then deployed to a GPU.

Performance Optimization Techniques

Two key techniques were highlighted for boosting serving performance.

1. Tuning CUDA Graph Batch Size

SGLang uses CUDA Graphs by default to optimize the decoding process. However, the default cuda_graph_max_bs (maximum batch size for CUDA graph) might be too conservative for your specific hardware and workload.

Problem: During a benchmark on an L4 GPU, the default max batch size was 8. When the number of concurrent requests exceeded this, the cuda_graph flag in the server logs turned to false, indicating that the optimization was disabled, leading to lower throughput.
Solution: By manually setting a higher limit, for example, cuda_graph_max_bs=32, the CUDA Graph optimization remained active (true) even with a larger number of concurrent requests (e.g., 13), resulting in better performance.
Recommendation: Engineers should monitor their server logs during benchmarking and adjust this parameter to ensure CUDA Graph is enabled for their typical batch sizes.

2. Eagle 3 Speculative Decoding

SGLang supports Eagle 3, a recent and powerful speculative decoding algorithm. Unlike traditional methods that use a smaller, separate draft model, Eagle works by creating a draft model derived from the intermediate layers of the target model itself. This can lead to a much higher token acceptance rate.

To use Eagle, you need to configure several parameters:

--speculative-decoding eagle
--draft-model-path (pointing to the target model, as Eagle derives the draft from it)
--num-spec-tokens (number of tokens to speculate)
--eagle-top-k
--num-spec-tokens

To find the optimal settings for these parameters, SGLang provides a benchmarking script (bench_spec_decoding.py) in its repository. This script iterates through different combinations of batch sizes, depths, and top-K values to identify the configuration that provides the best speed and acceptance rate for a given workload. It is crucial to use prompts representative of your production traffic for this benchmark, as speculation performance is highly content-dependent.

Community and Contributions

The SGLang project encourages community involvement. The best ways to contribute are:

Engage on GitHub: Star the repository, file detailed bug reports, and request features. The maintainers use good first issue and help wanted tags to highlight opportunities for new contributors.
Join the Slack: The community Slack (slack.sglang.ai) is the central hub for discussions and announcements.
Contribute to the Codebase: The repository is structured into several components:
- sgl_kernel: The CUDA kernel library for low-level operations.
- sgl_router: The scheduler and routing logic.
- srt: The core Python runtime that handles features like function calling, constrained decoding, and the OpenAI-compatible server. Adding support for a new model is often a matter of referencing an existing implementation (like Llama) and making minor modifications.

Q&A Highlights

On choosing SGLang vs. vLLM: Baseten uses multiple runtimes. SGLang is favored for its configurability and extensibility. Its open-source nature allows Baseten’s team to contribute fixes directly, unblocking themselves without waiting on core developers. vLLM is often used for its broad model compatibility, especially when a new model is first released.
On Security: The choice of serving runtime (SGLang, vLLM, etc.) does not inherently change the security model. Security is managed at a higher, runtime-agnostic level through containerization, network isolation, and least-privilege principles. For high-security or air-gapped environments, SGLang’s open-source nature is an advantage, as the code can be fully audited and version-locked before deployment.