Skip to content

Local Model Acceleration

Backend.AI GO uses different inference engines for different hardware, model formats, and workloads. The app resolves them from the installed engine set, your host capabilities, and the selected model format.

The local inference stack

When you load a model, Backend.AI GO starts a dedicated local or containerized serving process for that session.

  • binary engines such as llama.cpp, mlxcel-server, and stable-diffusion.cpp run directly on the host
  • container engines such as vLLM and SGLang run inside managed containers
  • the app health-checks the engine, then registers the session so Chat, Sessions, and the Management API can use it

Current engine families

llama.cpp

The cross-platform default for GGUF models.

  • Formats: gguf
  • Best for: Windows, Linux, Intel Macs, and Apple Silicon users who prefer GGUF
  • Accelerators: CPU, Metal, CUDA, ROCm, SYCL, Vulkan, depending on the package

MLX via mlxcel

The preferred MLX path for MLX repositories.

  • Formats: mlx (safetensors aliases to this in the local MLX path)
  • Best for: Apple Silicon Macs, plus supported Linux CUDA13 mlxcel packages
  • Preferred engine: mlxcel-server
  • Legacy fallback: mlx-lm, when installed

vLLM

A beta container engine for high-throughput serving on large NVIDIA systems.

  • Formats: safetensors-repo
  • Best for: high-VRAM NVIDIA GPUs, multi-user or agent-heavy workloads
  • Strengths: PagedAttention, continuous batching, high throughput

SGLang

A beta container engine for safetensors repositories on NVIDIA GPUs.

  • Formats: safetensors-repo
  • Best for: structured outputs and prefix-cache-heavy workloads
  • Strengths: RadixAttention prefix caching, constrained decoding

stable-diffusion.cpp

The local image-generation engine.

  • Formats: diffusion model files such as .safetensors, .ckpt, and supported .gguf variants
  • Best for: on-device image generation

Format names you will see in the app

Backend.AI GO currently normalizes local-model formats as:

  • gguf
  • mlx
  • safetensors-repo

The important compatibility note is that legacy safetensors local-model labels are treated as mlx for the Apple Silicon MLX path. Container engines use safetensors-repo for full Hugging Face repositories.

Where to manage engines and defaults

  • Engines: install, update, and inspect engine packages
  • Settings > Models: choose default engines by format and app-level model defaults
  • Models > Local: load downloaded models and open per-model settings
  • Models > Browse: download new models from Hugging Face

Choosing the right engine

Choose llama.cpp when you need the broadest cross-platform support or a GGUF quantization.

Choose mlxcel when the model is available as an MLX repository and you use Apple Silicon Metal or a supported Linux CUDA13 package.

Choose vLLM when throughput and concurrency matter more than the smallest footprint.

Choose SGLang when you want container serving plus structured outputs or strong prefix-cache reuse.

Choose stable-diffusion.cpp for local image generation.