6.6. vLLM Acceleration¶

vLLM is a high-throughput and memory-efficient LLM serving engine. While llama.cpp focuses on broad compatibility and low-resource environments, vLLM is designed for production-grade performance on high-end GPUs.

What is vLLM?¶

• Repository: github.com/vllm-project/vllm

• Core Innovation: PagedAttention.

Traditional attention algorithms require contiguous memory blocks, which leads to significant fragmentation and waste (up to 60-80% of memory). vLLM introduces PagedAttention, an algorithm inspired by virtual memory paging in operating systems. It allows KV cache blocks to be stored in non-contiguous memory spaces.

Key Benefits¶

• Higher Throughput: Can handle many more concurrent requests than standard Hugging Face Transformers.

• Efficient Memory: Near-zero waste of GPU VRAM, allowing for larger batch sizes or longer context windows.

• Fast Feature Adoption: Often first to ship new serving features such as continuous batching and quantization (AWQ, GPTQ, SqueezeLLM).

Role in Backend.AI GO¶

In Backend.AI GO, vLLM serves as the "Pro" engine, primarily for users with NVIDIA GPUs (on Linux or Windows WSL).

Note: vLLM support in Backend.AI GO is currently in Beta.

When to use vLLM?¶

You should consider switching to the vLLM backend if:

1. You have a high-VRAM NVIDIA GPU: vLLM performs best on cards like the RTX 5090 (32GB), professional RTX Pro 6000 (96GB), or DGX Spark (GB10, 128GB) personal AI workstations.

2. Concurrency Matters: You are using Backend.AI GO as a server for multiple users or agents simultaneously.

3. Throughput over Latency: You need to process massive amounts of text (e.g., summarizing hundreds of documents) where total completion time is more important than the "time to first token".

Host Requirements¶

vLLM runs inside a Linux container, so the Engines page performs a readiness check before offering the vLLM option.

The check detects:

• Whether Docker (or another supported container runtime) is installed.
• Whether your GPU can be passed into a container (NVIDIA and AMD GPUs can; Apple Metal and Intel GPUs cannot).
• Whether the selected model is likely to fit in available VRAM.

If a blocking condition is found, the Engines page shows a banner with a plain-language explanation and a one-step remediation (for example, "Install Docker" or "Install the NVIDIA Container Toolkit"). The option is disabled until the condition is resolved.

Configuration¶

To use vLLM in Backend.AI GO:

1. Go to Settings > Inference.

2. Change the default backend for CUDA from llama.cpp to vLLM.

3. Ensure you have the appropriate NVIDIA drivers and CUDA toolkit installed (usually version 11.8 or higher).

Supported Model Formats¶

Unlike llama.cpp which uses .gguf, vLLM works directly with:

• Hugging Face format: Standard .safetensors or .bin weights.

• AWQ / GPTQ: Pre-quantized models for faster inference.

How Backend.AI GO launches vLLM¶

Backend.AI GO owns the container launch, so you do not run docker run yourself. When you start a downloaded Hugging Face safetensors repository on the vLLM engine, Backend.AI GO:

1. Pulls the vllm/vllm-openai image on first use. This image is large (roughly 9–10 GB), so the first start spends time pulling and warming up before the model loads. Subsequent starts reuse the cached image.

2. Runs the container with the GPU passed through (--gpus all), shared-memory transport (--ipc=host), the model mounted read-only, and the host port published on loopback.

3. Sets the launch flags that a hand-run vLLM often forgets, so the common failures never happen:

   • --served-model-name matches the alias the router exposes, so a request for your model id reaches the engine.
   • --max-model-len is derived from your context setting, so a request whose max_tokens exceeds the model length is avoided.
   • --enable-auto-tool-choice --tool-call-parser hermes makes tool_choice: auto work, so tool-enabled requests do not hit a 400 error.
   • --gpu-memory-utilization defaults to a safe 0.90 fraction (overridable later from settings).

Once the container reports healthy at /health, Backend.AI GO registers it with the router, and the model answers /v1/chat/completions from both the chat UI and the Management API. Stopping the model tears the container down and deregisters the route; an unexpected crash is reaped the same way, so no stale route is left behind.

vLLM brings data-center class inference performance directly to your high-end workstation.