SGLang — Fast LLM Serving with RadixAttention

What it is

SGLang is a serving framework optimized for running large language models and multimodal models at high throughput. Its signature feature is RadixAttention, a prefix caching mechanism that reuses KV cache across requests sharing common prompt prefixes.

The framework targets ML engineers and platform teams who need to serve LLMs in production with low latency. It supports NVIDIA, AMD, Intel GPUs, and Google TPUs, making it hardware-flexible.

How it saves time or tokens

RadixAttention caches KV states for shared prompt prefixes across requests. When multiple users send requests with the same system prompt, SGLang avoids recomputing attention for the shared prefix. Speculative decoding further reduces latency by predicting multiple tokens at once. These optimizations translate to higher throughput per GPU dollar.

How to use

1. Install SGLang via pip with your target hardware backend.
2. Launch the server pointing to a model checkpoint.
3. Send requests via the OpenAI-compatible API endpoint.

# Install
pip install sglang[all]

# Launch server with a model
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3-8B-Instruct \
  --port 30000

# Query the server
curl http://localhost:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "meta-llama/Llama-3-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'

Example

import sglang as sgl

@sgl.function
def qa_pipeline(s, question):
    s += sgl.system('You are a helpful assistant.')
    s += sgl.user(question)
    s += sgl.assistant(sgl.gen('answer', max_tokens=256))

runtime = sgl.Runtime(model_path='meta-llama/Llama-3-8B-Instruct')
sgl.set_default_backend(runtime)

state = qa_pipeline.run(question='What is RadixAttention?')
print(state['answer'])

Related on TokRepo

• Local LLM tools — Compare SGLang with vLLM for self-hosted inference
• AI gateway solutions — Route requests across multiple serving backends

Common pitfalls

• RadixAttention benefits diminish when requests have no shared prefixes; batch requests with common system prompts to maximize cache hits.
• GPU memory must accommodate both the model weights and the KV cache; underprovisioning causes OOM errors on long-context workloads.
• Structured output mode (JSON schema enforcement) adds decoding overhead; disable it for free-form generation tasks.