Continuum Router & API¶

Backend.AI GO works as an AI API Gateway for your local network.

What is Continuum Router?¶

The Continuum Router is an internal component that acts as a traffic controller. It provides a single, unified entry point for all your AI models—whether they are running locally on your machine or in the cloud.

For more detailed technical documentation, please refer to docs.continuum.lablup.ai.

OpenAI-Compatible API¶

Backend.AI GO exposes an API that mimics the OpenAI API standard. This means you can use Backend.AI GO as a drop-in replacement for any application that supports OpenAI.

• Endpoint: http://localhost:39080/v1 (Default)
• Authentication: Optional (configurable in Settings)

Example: Using with curl¶

curl http://localhost:39080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Request Timeouts and Health Checks¶

API > Health holds two related groups of settings: how often the router probes each backend, and how long it waits on a request before giving up.

Backend.AI GO writes the timeouts: section on every install, shaped exactly the way the router reads it, so a fresh install runs on these budgets rather than on the router's own defaults:

timeouts:
  connection: 10s
  request:
    standard:          # Non-streaming requests
      first_byte: 30s
      total: 300s
    streaming:         # Streaming (SSE) requests
      first_byte: 120s
      chunk_interval: 30s
      total: 600s
    image_generation:
      first_byte: 60s
      total: 180s
  health_check:
    timeout: 4s
    interval: 5s

Four details are worth knowing:

• Only the streaming first_byte is a real deadline. It bounds the wait for the first chunk of a streaming response, and the router enforces it; once the first chunk arrives, chunk_interval takes over. It ships at 120 seconds because a reasoning model routinely spends longer than a minute before its first token.
• The other two first_byte values do nothing. A non-streaming response is delivered in one piece once the whole answer has been generated, so for standard and image_generation the time to the first byte is the total generation time, and total is the only budget that applies. Both fields must still be present in the file, so they cannot simply be deleted, and changing them has no effect on anything. Leave them at their defaults; if a non-streaming request is being cut short, total is the value to raise.
• health_check is derived, not edited. The router takes its live health-check interval and per-probe timeout from timeouts.health_check rather than from the health_checks section, so the app rebuilds that block from your Check interval and Timeout values every time it writes the file. The router also requires the timeout to be shorter than the interval, so when both are set to the same value the timeout is shortened by a second and the interval you asked for is the one that takes effect.
• The router enforces its own limits. A non-streaming total above 9 minutes, a streaming total above 20 minutes, a streaming first_byte above 8 minutes, a chunk interval outside 1 to 60 seconds, or a connection timeout above 15 seconds is refused when the router loads the file, and the router then does not start at all. Backend.AI GO checks the same limits before saving, so a value past one of them is rejected in this page with the field named rather than surfacing later as a router that will not start.

Local Socket Location¶

Inside your machine, Backend.AI GO talks to the router over a Unix domain socket rather than a TCP port. The socket file, and the sockets of every local model server, live in a short per-user runtime directory:

The directory is chosen once at launch and the router's current socket path is shown on API > General under Socket path. None of the automatic locations contain your user name. A Unix socket path is limited to 103 bytes, and earlier versions built it under the application data directory, which includes your home directory: a macOS account name of 16 bytes or more (6 Hangul syllables) made the router fail to start with path must be shorter than SUN_LEN, and model servers silently switched to TCP from 11 bytes.

Each candidate directory is created with owner-only permissions and checked before use (not a symlink, owned by you, not group- or world-accessible, and a test socket can actually be bound there). A directory that fails any check is skipped with the reason in the log, and the walk continues to the next one; the old application data sockets/ directory is the last resort. Leftover socket files under the old location are cleaned up once at launch.

If no directory at all can be used (for example an unwritable /tmp, or a sandboxed build whose container path is too long), the router still starts, bound to a loopback TCP port, and a notification explains why. This substitute binds 127.0.0.1 regardless of the Allow External Access setting below: it exists to stand in for the local socket, not to publish the API server you left off, so turning on external access does not put it on your network. In that mode the router Admin API token is the only protection for the port, since every local process can reach a loopback port. To restore socket mode, set a shorter Socket directory in Settings > Advanced (62 bytes or less; the change takes effect after a restart) or make /tmp writable.

External Access¶

By default, the API is only accessible from your own computer (127.0.0.1). If you want to access your models from other devices on your local network (like a tablet or another laptop):

1. Go to API > General.
2. Enable Allow External Access.
3. Open API > Security and API > Access keys to configure authentication before accepting remote clients.
4. Use your computer's local IP address (for example, http://192.168.1.10:39080/v1) in your other apps.