Engine & Runtime Management¶

The Engines menu is your central hub for managing inference engines and their runtime dependencies. Here you can download, update, and configure different versions of llama.cpp, MLX, and other inference backends optimized for your specific hardware.

Understanding the Engines Page¶

The Engines page is divided into three main sections:

1. Installed Engines - Engines currently available on your system
2. Installed Runtimes - Runtime libraries (CUDA, ROCm, etc.) that engines depend on
3. Available Engines - New engines you can download from the official registry

Why Manage Engines?¶

Hardware-Specific Optimization¶

Different GPUs require different engine builds:

CUDA 13 on arm64: two GPU-specific builds¶

On Linux arm64, llama.cpp publishes two CUDA 13 builds. They are compiled for different NVIDIA GPU architectures, so only one of them runs on any given host:

Installing the wrong one succeeds, and then every model load fails inside CUDA:

CUDA error: no kernel image is available for execution on the device

Backend.AI GO detects the host GPU and marks only the matching build Recommended. The other one is listed but disabled, with a note naming the build to install instead, so the wrong choice is refused before the download rather than at the first model load. To check what the app detected on the command line:

aigo engine available

The VARIANT column shows gb10 or gh200 and the NOTE column shows recommended or unsupported.

GB200 and GB300 are not covered by either build. They are the same Blackwell generation but report compute capability 10.0, and neither package targets it, so they are offered both builds and blocked from neither.

When the host GPU cannot be identified, both builds stay installable and both keep the Recommended badge, because a detection gap should not leave a CUDA host with nothing it can install. In that case, read the compute capability yourself and pick the matching row:

nvidia-smi --query-gpu=name,compute_cap --format=csv,noheader

A DGX Spark prints NVIDIA GB10, 12.1, which is the GB10 build.

Supported Inference Engines¶

Backend.AI GO supports multiple inference engines:

Version Control¶

Keep multiple engine versions installed side-by-side:

• Test new releases before committing
• Roll back if a new version has issues
• Compare performance between versions

Dependency Management¶

Some engines require runtime libraries (like CUDA or ROCm). The Engines page automatically detects and manages these dependencies, installing them when needed.

Installing an Engine¶

From the Registry¶

1. Navigate to the Engines page from the sidebar menu.

2. Scroll to the Available Engines section.

3. Find the engine you want (e.g., llama.cpp).

4. Click the Download button.

5. If multiple variants are available (e.g., CUDA 13, Metal, CPU), a dialog will appear:

   • Recommended variants are marked based on your detected hardware
   • Each variant shows its download size
   • Select the variant that matches your GPU

6. The download begins immediately. Watch the progress in the floating Download Queue panel.

Download Progress Stages¶

The installation process goes through several stages:

Offline Installation¶

For air-gapped environments or when you prefer manual downloads:

1. Download .baiengine package files from Backend.AI GO Releases.

2. Place them in the incoming directory:

   macOS / LinuxWindows

   ~/Library/Application Support/ai.backend.go/engines/incoming/ (macOS) or ~/.local/share/ai.backend.go/engines/incoming/ (Linux)
   

   %APPDATA%\ai.backend.go\engines\incoming\
   

3. Open the Engines page. A Pending Packages banner will appear.

4. Click Import to install the detected packages.

Alternatively, you can drag and drop .baiengine files directly onto the Engines page.

Managing Installed Engines¶

Engine Cards¶

Each installed engine is displayed as a card showing:

• Engine name and version (e.g., llama.cpp 1.0.0)
• Accelerator badge (Metal, CUDA 13, CPU, etc.)
• Status badges:
  • 🟢 Active - Currently running a model
  • 🟠 Update Available - Newer version in registry
• Supported formats (GGUF, MLX, etc.)
• Installation size

Actions¶

Setting a Default Engine¶

When multiple installed engines support the same model format (e.g., both llama.cpp-metal and llama.cpp-cpu support GGUF), each format has its own default:

1. Find the engine you want and click Make default for GGUF (or the relevant format name).
2. The button is replaced by a Default — GGUF badge, and any previous default for that format is cleared.
3. Backend.AI GO uses this engine whenever it loads a model of that format.

Engine Details Drawer¶

Click on any engine card to open the details drawer with three tabs:

Overview Tab¶

• Basic Information: ID, version, accelerator, installation date
• Manifest Details: Format version, upstream version, platform compatibility
• Accelerator Information: Specific backend details and optimizations

Files Tab¶

Browse the installed files:

• Directory structure
• File sizes
• Useful for troubleshooting or verifying installation

Dependencies Tab¶

View runtime dependencies:

• Required vs. optional dependencies
• Installation status of each dependency
• Version requirements

Runtime Libraries¶

What are Runtimes?¶

Runtime libraries are shared dependencies that engines need to function. The most common are:

Automatic Installation¶

When you install an engine that requires a runtime:

1. Backend.AI GO detects the missing dependency.
2. The runtime is downloaded automatically.
3. Both progress indicators appear in the download queue.
4. The runtime is installed before the engine finishes.

Viewing Installed Runtimes¶

The Installed Runtimes section shows:

• Runtime name and version
• Installation date
• Which engines depend on this runtime

Click a runtime card to see the full list of dependent engines.

Runtime Persistence¶

Runtimes are shared across engine versions. If you:

• Update an engine: The runtime remains intact
• Uninstall all engines using a runtime: The runtime stays (for future use)
• Manually delete a runtime: Dependent engines may stop working

Hardware Detection¶

Backend.AI GO automatically detects your system hardware to recommend the best engine variants.

What's Detected¶

• GPU Vendor: NVIDIA, AMD, Intel, or Apple
• GPU Model: Specific card name (e.g., RTX 4090, RX 7900 XTX)
• Driver Version: CUDA version, ROCm version, etc.
• VRAM: Available video memory
• Disk Space: Available storage for engine installation

Viewing System Capabilities¶

The system capabilities are shown when selecting engine variants:

• Recommended badge on the best variant for your hardware
• Available accelerators listed in the install dialog
• Disk space warnings if storage is low

Updating Engines¶

Checking for Updates¶

Backend.AI GO periodically checks the registry for new engine versions. Every check, including the one the Engines page's refresh button runs, reads the registry again, so a version published while the app is running shows up without a restart. When an update is available:

• An orange Update Available badge appears on the engine card
• The version number of the update is displayed

Applying Updates¶

1. Click the refresh icon on the engine card.
2. The new version downloads and its checksum is verified. The installed version stays in place while this happens.
3. The new version replaces the old one. If the download or the install fails, or the app quits partway, the previous version is kept (or restored at the next start).
4. Your settings and default preferences are preserved.

Troubleshooting¶

Engine Won't Install¶

Missing System Libraries¶

An engine package ships its own libraries in lib/, and AI:GO puts that directory (plus any runtime dependency) on the library search path before it starts the server. Anything the package was linked against that is in neither place has to come from the operating system.

The install step now asks the dynamic loader whether the packaged binaries can actually start on this machine, and refuses the install when they cannot. The message names both the binary and the library, for example:

ENGINE_MISSING_SHARED_LIBRARY: bin/llama-server cannot start on this host. The dynamic loader
cannot find libnccl.so.2, which is shipped in neither the engine package nor its runtime
dependencies. Install the missing library with your system package manager, then install the
engine again.

Install the named library with your distribution's package manager and install the engine again. There is nothing to repair in the package itself.

Engine Won't Start¶

Clearing Stuck Downloads¶

If a download appears stuck:

1. Click the Cancel button in the download queue.
2. Wait for cleanup to complete.
3. Try the installation again.

Best Practices¶

Choose the Right Variant¶

• For maximum performance: Match the engine variant to your GPU
• For compatibility: CPU variants work everywhere but are slower
• For memory-constrained systems: Some variants are more memory-efficient

Keep Engines Updated¶

New versions often include:

• Performance improvements
• Bug fixes
• Support for new model architectures
• Security patches

Manage Disk Space¶

Engine packages can be large (100MB - 1GB+). Periodically:

• Remove unused engine variants
• Check the Files tab to see installation sizes
• Keep only the variants you actively use

Integration with Model Loading¶

The Engines page works closely with the model loading system:

1. Model Format Detection: When you load a model, Backend.AI GO checks its format (GGUF, Safetensors, etc.).

2. Engine Resolution: The app finds an installed engine that supports that format.

3. Format-Specific Defaults: Check your configured default engine for the format in Settings.

4. Priority-Based Selection: If no default is set, engines are selected based on priority:

   • For MLX format: MLXcel > MLX LM
   • For GGUF format: llama.cpp (with GPU acceleration preferred)

5. Hardware Optimization: Within the same priority level, GPU-accelerated variants are preferred over CPU-only. When multiple GPU families are installed (e.g., CUDA and SYCL on the same machine), the resolver picks the engine whose accelerator family matches your detected hardware — so on an NVIDIA host with both a CUDA and SYCL build installed, the CUDA build wins automatically. If multiple CUDA versions are installed, the highest version your driver supports is chosen.

For more details on model loading, see:

• Running Models
• llama.cpp Acceleration
• MLX Acceleration

Technical Details¶

Engine Package Format¶

Engine packages use the .baiengine format:

• ZIP archive containing binaries and metadata
• manifest.json describes the engine and its requirements
• Checksums ensure file integrity
• Platform-specific builds for each OS/architecture

Directory Structure¶

The app-data root is ~/Library/Application Support/ai.backend.go/ on macOS, %APPDATA%\ai.backend.go\ on Windows, and ~/.local/share/ai.backend.go/ on Linux. The old ~/.backend-ai-go/ root is legacy-only.

<app-data>/
├── engines/
│   ├── incoming/              # Offline package staging
│   ├── installed.json         # Installed engines registry
│   ├── llama-cpp-metal/       # Engine: llama.cpp with Metal
│   ├── llama-cpp-cuda13/      # Engine: llama.cpp with CUDA 13
│   └── mlxcel/                # Engine: MLXcel
├── runtimes/
│   ├── installed.json         # Installed runtimes registry
│   ├── cuda13-runtime/        # Runtime: CUDA 13
│   └── hip-runtime/           # Runtime: AMD HIP
├── logs/                      # Application logs
└── config/                    # Configuration files

Supported Accelerators¶

The Engines system ensures you always have the optimal inference backend for your hardware, making local AI both accessible and performant.