6.7. SGLang Acceleration¶

SGLang is a fast serving engine for large language models, OpenAI-compatible like vLLM but with a different set of operational strengths. It runs as a GPU container and serves full Hugging Face safetensors repositories on NVIDIA GPUs.

What is SGLang?¶

• Repository: github.com/sgl-project/sglang

• Core Innovation: RadixAttention, an automatic prefix-cache that reuses the KV cache across requests that share a prompt prefix.

Key Benefits¶

• Automatic Prefix Caching: RadixAttention reuses shared prompt prefixes across requests, which speeds up workloads with long system prompts or many similar requests (agents, few-shot prompting, multi-turn chat).

• Structured Outputs: First-class constrained decoding (JSON Schema, regex, EBNF), so a request can be guaranteed to return valid, schema-conforming output.

• OpenAI-Compatible API: Exposes /v1/chat/completions, /v1/completions, and /v1/models, so the chat UI and the Management API talk to it the same way they talk to vLLM.

Role in Backend.AI GO¶

SGLang sits alongside vLLM as a container engine for users with NVIDIA GPUs (on Linux or Windows WSL). vLLM is the default engine for Hugging Face safetensors repositories; SGLang is an explicit alternative you can select when its prefix caching or structured-output strengths fit your workload.

Note: SGLang support in Backend.AI GO is currently in Beta.

When to use SGLang?¶

You should consider SGLang if:

1. Your requests share prompt prefixes: Agents and chat sessions with a large, stable system prompt benefit most from RadixAttention's automatic prefix caching.

2. You need guaranteed-valid structured output: SGLang's constrained decoding (JSON Schema / regex / EBNF) returns output that conforms to the schema you specify.

3. You have a high-VRAM NVIDIA GPU: Like vLLM, SGLang targets cards such as the RTX 5090 (32GB), professional RTX Pro 6000 (96GB), or DGX Spark (GB10, 128GB) workstations.

Host Requirements¶

SGLang runs inside a Linux container, so the Engines page performs the same readiness check it performs for vLLM before offering the SGLang option.

The check detects:

• Whether Docker (or another supported container runtime) is installed.
• Whether your GPU can be passed into a container (NVIDIA and AMD GPUs can; Apple Metal and Intel GPUs cannot).
• Whether the selected model is likely to fit in available VRAM.

If a blocking condition is found, the Engines page shows a banner with a plain-language explanation and a one-step remediation. The option is disabled until the condition is resolved.

Supported Model Formats¶

SGLang works directly with Hugging Face format: standard .safetensors weights (a full repository, not a single GGUF file). This is the same safetensors-repo format vLLM serves.

How Backend.AI GO launches SGLang¶

Backend.AI GO owns the container launch, so you do not run docker run yourself. When you start a downloaded Hugging Face safetensors repository on the SGLang engine, Backend.AI GO:

1. Pulls the lmsysorg/sglang image on first use. Like vLLM's image, this is large, so the first start spends time pulling and warming up before the model loads. Subsequent starts reuse the cached image.

2. Runs the container with the GPU passed through (--gpus all), a large shared-memory segment (--shm-size 32g, which SGLang uses instead of vLLM's --ipc=host), the model mounted read-only, and the host port published on loopback. SGLang's in-container port is 30000 (vLLM uses 8000).

3. Launches the server as a Python module, python3 -m sglang.launch_server, with the flags that keep routing correct:

   • --served-model-name matches the alias the router exposes, so a request for your model id reaches the engine.
   • --trust-remote-code lets repositories with custom modeling code load.

   SGLang's tool-call parser is model-specific (qwen25, llama3, mistral, and others) with no universal default, so Backend.AI GO does not enable one automatically; enabling tool calling for a specific model is handled by a downloaded engine variant that overrides the built-in manifest.

Once the container reports healthy at /health, Backend.AI GO registers it with the router as a self-hosted, OpenAI-compatible backend (the router's generic backend type, so router-managed web search still applies and the model owner is not relabeled). The model then answers /v1/chat/completions from both the chat UI and the Management API. Stopping the model tears the container down and deregisters the route; an unexpected crash is reaped the same way, so no stale route is left behind.

SGLang adds automatic prefix caching and guaranteed structured outputs to your high-end NVIDIA workstation.