vLLM Acceleration¶

vLLM is a high-throughput and memory-efficient LLM serving engine. While llama.cpp focuses on broad compatibility and low-resource environments, vLLM is designed for production-grade performance on high-end GPUs.

What is vLLM?¶

• Repository: github.com/vllm-project/vllm

• Core Innovation: PagedAttention.

Traditional attention algorithms require contiguous memory blocks, which leads to significant fragmentation and waste (up to 60-80% of memory). vLLM introduces PagedAttention, an algorithm inspired by virtual memory paging in operating systems. It allows KV cache blocks to be stored in non-contiguous memory spaces.

Key Benefits¶

• Higher Throughput: Can handle many more concurrent requests than standard Hugging Face Transformers.

• Efficient Memory: Near-zero waste of GPU VRAM, allowing for larger batch sizes or longer context windows.

• Fast Feature Adoption: Often first to ship new serving features such as continuous batching and quantization (AWQ, GPTQ, SqueezeLLM).

Role in Backend.AI GO¶

In Backend.AI GO, vLLM serves as the "Pro" engine, primarily for users with NVIDIA GPUs (on Linux, x86_64 or arm64).

When to use vLLM?¶

You should consider switching to the vLLM backend if:

1. You have a high-VRAM NVIDIA GPU: vLLM performs best on cards like the RTX 5090 (32GB), professional RTX Pro 6000 (96GB), or DGX Spark (GB10, 128GB) personal AI workstations.

2. Concurrency Matters: You are using Backend.AI GO as a server for multiple users or agents simultaneously.

3. Throughput over Latency: You need to process massive amounts of text (e.g., summarizing hundreds of documents) where total completion time is more important than the "time to first token".

Host Requirements¶

vLLM runs inside a Linux container, so the Engines page performs a readiness check before offering the vLLM option.

The check detects:

• Whether Docker (or another supported container runtime) is installed.
• Whether an NVIDIA GPU can be passed into the container.
• Whether the selected model is likely to fit in available VRAM.

If a blocking condition is found, the Engines page shows a banner with a plain-language explanation and a one-step remediation (for example, "Install Docker" or "Install the NVIDIA Container Toolkit"). The option is disabled until the condition is resolved.

Configuration¶

To use vLLM in Backend.AI GO:

1. Open Engines and install the vLLM engine if it is not already installed.

2. Open Settings > Models and review Default engines by format.

3. Use Models > Browse Hugging Face to download a Hugging Face repository that the app classifies as safetensors-repo.

4. On the model you want to run, choose the vLLM engine when running it, or set vLLM as the default engine for that format.

5. Ensure Docker and the required NVIDIA container stack are available on the host.

Supported Model Formats¶

Unlike llama.cpp which uses .gguf, vLLM works directly with:

• Hugging Face format: Standard .safetensors or .bin weights.

• AWQ / GPTQ: Pre-quantized models for faster inference.

How Backend.AI GO launches vLLM¶

Backend.AI GO owns the container launch, so you do not run docker run yourself. When you run a downloaded Hugging Face safetensors repository on the vLLM engine, Backend.AI GO:

1. Pulls the vllm/vllm-openai image on first use, at the tag described under Image version. This image is large (roughly 9–10 GB), so the first run spends time pulling and warming up before the model loads. Subsequent runs reuse the cached image.

2. Runs the container with the GPU passed through (--gpus all), shared-memory transport (--ipc=host), the model mounted read-only, and the host port published on loopback.

3. Sets the launch flags that a hand-run vLLM often forgets, so the common failures never happen:

   • --served-model-name matches the alias the router exposes, so a request for your model id reaches the engine.
   • --max-model-len is derived from your context setting, so a request whose max_tokens exceeds the model length is avoided.
   • --enable-auto-tool-choice with a tool-call parser makes tool_choice: auto work, so tool-enabled requests do not hit a 400 error. The parser is hermes unless the model sets another (see Per-model serve settings).
   • --gpu-memory-utilization defaults to a safe 0.90 fraction, which you can change in Settings for every container engine or per model.

Once the container reports healthy at /health, Backend.AI GO registers it with the router, and the model answers /v1/chat/completions from both the chat UI and the Management API. Stopping the model tears the container down and deregisters the route; an unexpected crash is reaped the same way, so no stale route is left behind.

Per-model serve settings¶

Current model cards often ask for vllm serve flags beyond the ones above: a tool-call parser that matches the model's tool format, a reasoning parser, a MoE kernel for your GPU, or a speculative-decoding draft model. Set them per model in the vLLM tab of the model settings dialog, which appears for a downloaded Hugging Face safetensors repository. Each setting maps to one flag, and a setting left unset adds no flag, so a model you never configured runs exactly as before.

The dropdowns list the names vLLM v0.31.0 ships with, and the app refuses any other value both when you save and when the model starts, so a setting cannot add an arbitrary flag to the container command. There is no free-form arguments field. With an older image tag, a name that release does not know makes vLLM exit at startup with its own error message.

vLLM downloads the draft model inside the container, and the container is removed when the model stops, so the download repeats every time the model starts. The GPU memory fraction, tensor parallel size, MoE backend, and Mamba backend depend on the host's GPUs, so a .baimodel export leaves them out.

For example, the DGX Spark (GB10) recipe on the nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 model card maps to these settings:

These settings apply only when the vLLM engine runs the model. If SGLang runs it, they are ignored.

Image version¶

This release pins vllm/vllm-openai:v0.31.0, which is published for both x86_64 (amd64) and arm64, so the same engine runs on a GB10 DGX Spark or a GH200 host. The default tag is a CUDA 13 build and needs an NVIDIA driver from the R580 branch or newer. On a host with an older driver, set the image tag to v0.31.0-cu129, the CUDA 12.9 build vLLM publishes for the same release, as described below.

To run a different tag, open Settings > Engines, enter the tag in the Image tag field on the vLLM card, and select Save. Enter only the tag, not the vllm/vllm-openai: part. Leaving the field empty returns to the built-in tag. The saved tag is used for the image status on the card, for Pull image and Update image, and for every launch, so the image you pull is the image the model runs on. The same override is available from the CLI:

# Save v0.31.0-cu129 as the vLLM image tag, then pull it
aigo engine container image pull vllm --tag v0.31.0-cu129

# Return to the built-in tag
aigo engine container image pull vllm --reset-tag

# The readiness report names the image a start will pull
aigo engine container readiness --json '{"engineId":"vllm"}'

The readiness report's image field shows the reference (imageReference), the built-in tag (defaultTag), and whether your override chose it (tagOverridden). aigo engine container image inspect vllm shows the same two fields.

vLLM brings data-center class inference performance directly to your high-end workstation.