Headless Mode¶

"Headless Mode" refers to running Backend.AI GO primarily as a background service or server, without relying on the graphical user interface (GUI) for daily interactions. This is particularly useful for setting up a dedicated inference server on a spare machine or managing the application remotely.

Concept¶

Although Backend.AI GO ships as a desktop app, its core runtime is shared:

• Shared Rust runtime: Handles model inference, process orchestration, the Management API, and the Continuum Router.
• Desktop transport: Tauri IPC from the embedded WebView.
• Headless transport: REST and SSE exposed by aigo-server, plus the WebUI served over HTTP.

Headless WebUI and desktop UI therefore execute the same runtime logic; only features that genuinely depend on desktop integration (such as the system tray or native windows) behave differently.

Operation¶

System Tray¶

The simplest form of "headless-like" operation is closing the main window. * By default, closing the window minimizes Backend.AI GO to the System Tray (Menu Bar on macOS). * The API server and model inference continue running in the background.

CLI Control¶

You can use the bundled aigo CLI to manage the application without opening the window.

# List loaded models
aigo loaded list

# Load a model
aigo loaded load MODEL_ID

# Check system stats
aigo system info

See the CLI Reference for full documentation.

Dedicated Headless Server (aigo-server)¶

Backend.AI GO provides a standalone headless binary:

aigo-server

In this mode:

• The tauri crate is not part of the aigo-server dependency graph.
• The Management API becomes the primary control plane.
• The WebUI connects over HTTP/SSE instead of Tauri IPC.
• Model pools, router management, scheduling, agents, memory, and provider/runtime coordination reuse the same shared runtime managers used by the desktop app.

Remote Access (Server Mode)¶

To turn your local machine into a headless node for others, bind aigo-server externally. The Management API default is 8001:

aigo-server --external --port 8001

Use --host for an explicit bind address. Run aigo-server --help for the current flags, and allow only the required ports through the host firewall.

The Continuum Router is separate from the Management API and has its own listener. By default it talks to local clients over a Unix domain socket (see Continuum Router & API) and binds no TCP port, so there is nothing to open in the firewall for it. Since v1.13 you can give it one:

aigo-server --external --port 8001 --router-tcp --router-port 39080

--router-tcp (env AIGO_ROUTER_TCP, config key [router] api_server_enabled) binds the OpenAI-compatible /v1 surface on TCP as well. --router-port (env AIGO_ROUTER_PORT, config key [router] api_server_port) sets the port; without it the port saved in settings.json is used, which is 39080 on a fresh install. The listener binds 127.0.0.1 unless --external is also passed, in which case it binds 0.0.0.0 and the router starts requiring an API key on /v1: see Who Can Reach the Inference API. The same three settings are on the API page in the WebUI, and either way they are persisted, so a restart keeps them.

Two details worth knowing. --router-port also feeds the startup port-collision check and the banner, but neither now claims a listener that does not exist: the banner names a URL only when the listener is explicitly enabled, and a busy port refuses to start only in that case, since otherwise the router may never bind it. And a config file that sets [router] api_server_enabled wins over what the API page saved, while --external can only open the listener: it is an opt-in flag with no negative form, so closing an open listener is done in the UI or by removing the setting.

Other Backend.AI GO instances, the WebUI, and API clients can then connect to the host address.

Web UI¶

The released aigo-server binary carries the web UI inside it, and serves it on every path the REST API and /api/docs do not claim. A packaged install needs no configuration for this: the deb, the systemd unit, and the container image all leave AIGO_STATIC_DIR unset, and the UI is there.

When the deb finds the exact active assignment AIGO_STATIC_DIR=/var/lib/aigo/static left by the legacy 1.12 system installer, its post-install step comments out that line so the embedded frontend takes effect. Every other value is preserved. If /var/lib/aigo/static is an intentional custom build, enable it again after the upgrade. Whenever any external directory overrides the embedded bundle, the startup log warns with both its path and the aigo-server version because the two builds can drift independently.

Two knobs change that. --static-dir (env AIGO_STATIC_DIR, config server.static_dir) serves a frontend build of your own from disk instead. A path that does not exist stops the server: under a container or a unit file that almost always means a mount did not materialize, and quietly serving the embedded build in its place would hide the mismatch. A directory that exists but has no index.html is a content problem, so it warns and the embedded UI still answers. server.enable_static = false turns off both sources, leaving only the REST API and /api/docs; the startup banner then points there instead of telling you to open a browser.

Deep links such as /login, /models, and /chat resolve to the single-page application in both cases. Paths under /api/, /admin/, and the legacy /setup prefix never do: an unmatched one returns a JSON 404, so a client parsing the response cannot receive HTML where it expects JSON.

First-Run Sign-in (aigo-server)¶

When aigo-server starts for the first time with no existing data, it serves the Initial Setup screen at http://<host>:8001. Navigate there in a browser, choose a username and password for the admin account, and the server logs you in immediately. No second step is required. Subsequent visits show the Sign in screen.

If the server is bound to a non-local interface, first-run setup also requires a setup token. Managed per-user services should generate a random AIGO_SETUP_TOKEN, start the server with it, and open the user's browser with /#setupToken=<token> so the setup form is prefilled. Operators can also copy the one-time setup token printed in the aigo-server startup log. Localhost-only first-run setup does not require this token.

SDK clients (curl, Python, etc.) use X-API-Key or Authorization: Bearer headers with access keys created from API > Access keys after signing in.

Troubleshooting¶

Moving the Models Directory Under systemd¶

[models] directory in config.toml and AIGO_MODELS_DIR in the environment file move where models are stored. The deb and install.sh --system ship AIGO_MODELS_DIR=/var/lib/aigo/models active in /etc/aigo/environment, so that is where a fresh install of either puts models; the rpm does not install that file at all, so an rpm install uses the service account's home-directory default until you copy the template from /usr/share/doc/aigo-server/examples/environment. Moving models somewhere else needs one more step, because the unit does not read your configuration.

[engines] directory and AIGO_ENGINES_DIR are the engine equivalent and work the same way: engines are downloaded to, and loaded from, the directory you name. The deb and install.sh --system ship AIGO_ENGINES_DIR=/var/lib/aigo/engines active as well, and an upgrade from a package that predates it moves an existing engine tree from the home-directory default into that path once, with the service stopped. If you had already named a different directory yourself, in either the environment file or [engines] directory, that value is left alone and so is your tree: the upgrade names both paths and leaves the rename to you, because the path you chose can be a mount that is not up yet when the package is configured. The startup banner and log name the models and engines directories the server actually resolved, so read those rather than inferring from configuration.

Moving an engines directory that already holds engines is a rename, with the service stopped. This is exactly what the packaged upgrade does for you; do it by hand when you are pointing the variable somewhere else, or when the upgrade declined the move and said so:

sudo systemctl stop aigo-server
sudo mv /var/lib/aigo/.local/share/ai.backend.go/engines /var/lib/aigo/engines
sudo chown -R aigo:aigo /var/lib/aigo/engines
sudo systemctl start aigo-server

installed.json records each engine's absolute path, so a rename would otherwise leave every record naming a directory that is gone. On the next start the server re-anchors any record whose recorded directory has disappeared onto the matching directory under the current engines directory, logs one line per record it moved, and writes the corrected registry back, so the repair does not repeat. A record with no matching directory under the new path is left alone rather than pointed at some other engine.

The packaged migration declines in four cases, each without deleting anything: you have already named a different directory yourself, in /etc/aigo/environment or under [engines] in config.toml; both directories already hold data; the target is a symlink you created; or the service is running and could not be stopped. It names both paths when it declines, and so does the start that follows: if you set the directory but leave the engines where they were, startup warns and names both paths. Nothing is moved for you there either, because the tree can be large and the two paths can be on different filesystems. Until you do the rename the server starts with no installed engines. The path must also be absolute; a relative one is rejected at startup rather than leaving the engine subsystem quietly disabled.

The service runs with ProtectSystem=strict, which makes the whole filesystem read-only except the paths listed in ReadWritePaths=. The shipped unit lists /var/lib/aigo and /var/log/aigo. A directory outside those stays read-only no matter who owns it, so chown does not help. The service also orders itself after the mounts covering those two paths through RequiresMountsFor=, which a relocated directory on a separate volume does not get either.

Grant both with a drop-in rather than editing the unit, since the packages replace the unit on upgrade:

# /etc/systemd/system/aigo-server.service.d/10-storage.conf
[Unit]
RequiresMountsFor=/mnt/models

[Service]
ReadWritePaths=/mnt/models

The drop-in grants the sandbox exception and nothing else; ordinary permissions still apply on top of it. The service runs as aigo, and the entry carries no - prefix, so the path has to exist and be writable by that user before the restart. A path that does not exist yet fails namespace setup with 226/NAMESPACE before ExecStart runs, which is a systemd error rather than one of the server's:

sudo mkdir -p /mnt/models
sudo chown aigo:aigo /mnt/models
sudo systemctl daemon-reload
sudo systemctl restart aigo-server

ReadWritePaths= in a drop-in appends to the unit's list rather than replacing it, so /var/lib/aigo and /var/log/aigo are kept.

--data-dir / AIGO_DATA_DIR needs the same drop-in when it points outside /var/lib/aigo. It moves the whole application data directory: settings, access keys, installed engines, and the default location of the models directory.

GPU Access Under systemd¶

aigo-server under the packaged unit sees no GPU unless a drop-in gives it one, and the way it fails is easy to misread. The Engines page reports No GPU detected on this host, nvidia-smi works fine for you at a shell prompt, and the device nodes are world-readable:

crw-rw-rw- 1 root root 195,   0 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 /dev/nvidiactl
crw-rw-rw- 1 root root 500,   0 /dev/nvidia-uvm

This is not a permissions problem and not a driver problem. The unit sets PrivateDevices=true, which gives the service a private /dev holding pseudo-devices only, so /dev/nvidia*, /dev/kfd and /dev/dri are not in its mount namespace at all. Confirm it with the same command run both ways:

sudo systemd-run --quiet --wait --pipe --uid=aigo -p PrivateDevices=true  /bin/sh -c 'nvidia-smi -L'
sudo systemd-run --quiet --wait --pipe --uid=aigo -p PrivateDevices=false /bin/sh -c 'nvidia-smi -L'

Both the .deb and .rpm packages and the --system installer place the drop-in below for you, but only on a host that has an accelerator they can identify at install time: an NVIDIA or ROCm compute node, or a DRM render node belonging to a display-class PCI device from NVIDIA, AMD or Intel with a driver bound to it. A virtual display in a VM does not qualify, and neither does a card with no driver loaded. Add a GPU to a machine that had none, install into an image that is later given one, or install the driver after the package, and you have to place the drop-in yourself:

# /etc/systemd/system/aigo-server.service.d/10-gpu.conf
[Service]
PrivateDevices=false
DevicePolicy=closed
SystemCallFilter=~@raw-io
DeviceAllow=char-nvidia rw
DeviceAllow=char-nvidia-frontend rw
DeviceAllow=char-nvidia-uvm rw
DeviceAllow=char-nvidia-caps r
DeviceAllow=char-drm rw
DeviceAllow=/dev/kfd rw

sudo systemctl daemon-reload
sudo systemctl restart aigo-server

PrivateDevices=true bundles three controls: the private /dev, DevicePolicy=closed on the cgroup, and a seccomp filter blocking the @raw-io syscalls. Only the first blocks GPU access, so the drop-in turns that off and restates the other two, which makes the allow list the actual boundary rather than a formality. The rest of what PrivateDevices= implies is already written out in the unit and is untouched: NoNewPrivileges=true, and a CapabilityBoundingSet holding CAP_NET_BIND_SERVICE alone, so the service has neither CAP_MKNOD nor CAP_SYS_RAWIO and cannot create a device node for anything the list omits.

The entries name device classes rather than paths on purpose. char-nvidia resolves through /proc/devices, so it covers /dev/nvidia0 through /dev/nvidiaN on a multi-GPU host from one line, and char-nvidia-uvm and char-nvidia-caps follow NVIDIA's dynamically allocated majors, which differ between hosts. char-drm covers /dev/dri/renderD*, which ROCm, Vulkan and the Intel SYCL engine need; /dev/kfd is ROCm's compute node and is a misc device with no class name of its own. systemd ignores an entry naming a device class this host does not have.

char-nvidia-uvm is the one worth keeping even if you trim the list. nvidia-smi -L can report a GPU without it on some drivers, so the readiness panel goes green and then every CUDA context allocation fails at model load.

sudo usermod -aG render aigo
sudo systemctl restart aigo-server

Where the Sockets Live¶

aigo-server talks to the router and to every local model server over Unix domain sockets in a short per-user runtime directory, never under the data directory (see Local Socket Location). Under systemd the shipped unit declares RuntimeDirectory=aigo, so systemd creates /run/aigo before the server starts and exports it as $RUNTIME_DIRECTORY; the server then keeps its sockets in /run/aigo/sockets. Keep that directive if you write your own unit. Without it, and with no login session providing $XDG_RUNTIME_DIR, the server falls back to /tmp/aigo-{uid} (under PrivateTmp=true that is the unit's private /tmp, which is fine for sockets).

The server also checks who owns that runtime root and how open it is: it must belong to the aigo user and stay closed to writes from anyone else, which the shipped unit's RuntimeDirectoryMode=0755 already satisfies. A custom unit that widens the mode, or an $XDG_RUNTIME_DIR you point at a directory shared with other users, is skipped with the reason logged, and the server falls through to the next candidate in the order above rather than using it.

A container image with no writable /tmp and no runtime directory runs with loopback TCP only, which is what a containerized server wants anyway since its clients are not on the same kernel. The startup log records the chosen directory, or every candidate that was rejected and why.

Where Uploaded Files Live¶

The bundled continuum-router serves an OpenAI-compatible Files API at /v1/files and keeps what clients upload on disk. aigo-server writes that location into the router config as an absolute path: a router-files directory beside the router_config.yaml it generates, which on a --system install is /var/lib/aigo/.local/share/ai.backend.go/router-files. Both files.enabled and files.storage_path are rewritten on every save, so editing them in router_config.yaml does not stick.

Before v1.13 no files: section was written at all, and the router fell back to its own default of ./data/files, resolved against the working directory systemd gives a service. That is /, which ProtectSystem=strict makes read-only, so file storage failed to initialize on every start and the Files API was disabled for the whole boot:

ERROR continuum_router::server::init: Failed to initialize File service: storage error: failed to create storage directory ./data/files: Read-only file system (os error 30). File resolution will be disabled.

The shipped unit now also sets WorkingDirectory=/var/lib/aigo, which is already inside its ReadWritePaths=, so any other relative path the router resolves lands somewhere writable rather than under /. Keep both if you write your own unit: the absolute files.storage_path comes from the generated config, and the working directory has to be a path your ReadWritePaths= grants.

Two things this does not change. The Files API requires Authorization: Bearer <key> with the files scope; the generated config now declares router API keys (see below), but no access key is given that scope, so /v1/files still answers 401 even where /v1/chat/completions is open. What changed is that file storage initializes and the routes are mounted. And nothing deletes uploaded files: the router's retention_days default of 0 keeps them forever, so the directory above grows until you remove files from it yourself.

Who Can Reach the Inference API¶

There are two doors onto the same models, and they are protected differently.

The Management API's inference proxy (/api/v1/inference/* on port 8001) requires an access key and checks its scopes: Inference Read for the model list, Inference Write for a completion. That is the door the Access Keys page is about, and it works exactly as the scope names say.

The router's own OpenAI-compatible surface (/v1/models, /v1/chat/completions) is the other door. aigo-server gives the router a Unix domain socket and no TCP port at all, so on a stock install the socket's file permissions are what protects /v1: reaching it means already having an account on the machine that can open that file. The socket lives under the runtime directory described in Where the Sockets Live, which the shipped unit creates as /run/aigo owned by the aigo user.

Since v1.13, aigo-server also writes an api_keys section into the router config, so the router knows about the same keys the Access Keys page issues:

api_keys:
  mode: permissive
  api_keys_file: /var/lib/aigo/.local/share/ai.backend.go/router-api-keys.yaml

router-api-keys.yaml sits beside router_config.yaml and holds one entry for each access key carrying Inference Read or Inference Write. It contains the key secrets in the clear, because that is the form the router compares an incoming Authorization: Bearer against. On Linux and macOS it is created mode 0600; on Windows it inherits the data directory's ACL, exactly as router_config.yaml does, and that file already carries your provider API keys. Treat the two the same way: do not copy either out of the data directory, and do not widen their permissions. Both api_keys fields are rewritten on every router start and every settings save, so editing them in router_config.yaml does not stick.

The mode follows the bind address, and nothing else sets it. A socket, or a TCP listener on loopback, gets permissive: it recognizes and attributes a request that presents a valid key, and serves a request that presents none, because reaching that listener already means having an account on the machine. A request presenting an invalid key is served too, not rejected: to the router, an unrecognized token and no token are the same thing.

A listener another host can reach gets blocking instead. That is what --router-tcp --external produces, and what the API page's "Allow external access" toggle produces, and it is not a separate switch you can decline: permissive on such a listener publishes an unauthenticated inference API, which is the whole point of the section you are reading. Under blocking, /v1 answers 401 without a valid key and 401 with an invalid or expired one, /health stays open, and every route the router carries is covered, including /anthropic/v1/messages, which the Claude Code integration uses. So issue an access key with Inference Read or Inference Write before you open the listener, and give it to your clients.

Two things the router does not do at the version shipped here. It does not distinguish read from write on /v1, so a key holding only Inference Read can still post a completion; the scope is carried for attribution and for when the router grows a per-route check. And it has no exemption for loopback or for the Unix socket: blocking means blocking on every transport. That is why the app carries a credential of its own, aigo-internal-inference, which you will see in router-api-keys.yaml as an environment placeholder rather than a secret. It is minted per installation, injected into the router process at startup, and presented by the Management API's inference proxy, the agent runtime, and desktop chat, so turning the listener on does not take the product's own inference offline. Do not remove it and do not give it to a client.

Editing mode in router_config.yaml does nothing lasting: api_keys is rewritten from scratch on every router start and on every settings save, so the edit lasts until the next restart at most, and the server logs that it replaced it. Change the listener instead.

Bridging the socket to a TCP port by hand, which before v1.13 was the only way to reach /v1 from another machine, is still possible and is still unauthenticated:

# The socket file name carries a per-instance suffix. API > General shows the
# current path, and the startup log records it.
socat TCP-LISTEN:39080,fork,reuseaddr UNIX-CONNECT:/run/aigo/sockets/continuum-router-<suffix>.sock

A bridge like that publishes the inference API to everyone who can reach the port, with no key required. The router still sees a local socket connection, so it still applies permissive; the file-permission boundary is gone and nothing replaces it. Use --router-tcp instead, which opens the port and requires a key for it in one step. If you keep the bridge, put an authenticating reverse proxy in front of the port, or restrict it to a trusted network, or both.

Key changes are pushed to a running router through its Admin API, so creating, rotating, disabling, or deleting an access key normally applies without a restart. That push is best effort: if the router is down, starting, or does not answer, the change is only written to the keys file and takes effect the next time the router starts, with the reason in the log.

Where API Keys Are Stored¶

In headless mode, Backend.AI GO stores API keys in an encrypted file (encrypted_keys.json) instead of the OS keychain, which requires a GUI. Keys persist across server restarts. That file is the store; router-api-keys.yaml described above is a derived copy of the subset the router needs, and it is not encrypted, because the router has to compare a presented bearer token against the value itself.

curl http://localhost:8001/api/v1/providers