Skip to content

Distributed Inference Routing

When load balancing is enabled, Backend.AI GO transparently routes inference requests to network nodes. If the requested model is not running locally, the inference proxy looks it up in the distributed model index and forwards the request to a network node that has the model available.

How It Works

The inference proxy uses a local-first strategy:

  1. Local resolution: Check whether the requested model is running locally, either through the Continuum Router port or directly on a model server port.

  2. Network node fallback: If the model is not found locally and load balancing is enabled, the InferenceRouter resolves the model to a reachable network node.

  3. Authenticated forwarding: The proxy forwards the request to the network node's API, adding an Authorization: Bearer header with the node's API key.

When load balancing is active, connected network nodes are also synchronized to the local continuum-router instance as OpenAI cloud backends (node-{fingerprint}). Local chat and proxy requests route directly through continuum-router using the configured load balancing strategy and node weights. If requests bypass continuum-router, the in-process InferenceRouter serves as the fallback routing layer.

This process is transparent to the caller. The request and response format is unchanged, and the routing layer is fully invisible to clients using the OpenAI-compatible API.

Enabling Distributed Routing

Distributed routing is controlled by the load balancing configuration. To enable it:

  1. Open Settings > Nodes > Connect other computers, or the Distributed pool tab under Models. The Load Balancing section appears once at least one computer is connected, and both places show the same section.
  2. Turn on the switch in the Load Balancing heading. The rest of the section shows while it is on.
  3. Choose a routing strategy (see below).

Every change in the section is saved as you make it, so there is no save button: the strategy and the switches when you change them, the node order when you drop a node, and the health check interval when you leave the field or press Enter. While load balancing is off, the Distributed pool tab says so at the top, with a button that turns it on.

When load balancing is disabled, all inference requests are handled locally, node-* cloud backends are removed from continuum-router, and node synchronization is paused. When load balancing is enabled, registered nodes are synchronized into continuum-router with weights and selection strategy mapped to the chosen load balancing strategy.

Routing Strategies

The InferenceRouter supports four strategies:

Strategy Behavior
priority Send requests to nodes in the order you arrange them (see Node Order). Maps to WeightedRoundRobin with descending priority weights in continuum-router.
round-robin Cycle through available nodes in order. Maps to RoundRobin in continuum-router.
least-loaded Route to the node with the fewest active in-flight requests (real-time load). Maps to LeastLatency in continuum-router.
fastest Route to the node with the lowest measured latency. Maps to LeastLatency in continuum-router.

Configure the strategy in the Load Balancing section, in Settings > Nodes > Connect other computers or on the Distributed pool tab.

Node Order

Under the priority strategy, Load Balancing lists the connected computers in the order they get requests, and the computer at the top gets them first. Drag a computer up or down the list to change the order; it is saved when you drop it. With the keyboard, focus a computer's handle, press Space to pick it up, move it with the up and down arrow keys, and press Space again to drop it. Each computer in the list has a switch for whether it takes part in routing, and a computer's own Routing tab shows its place in the order.

The order is stored as each computer's priority in nodeConfigs: dropping a computer gives the top computer the highest value and the last one 1, so no two computers tie. The server accepts priority from 1 to 255.

Circuit Breaker

Each network node has an associated circuit breaker that protects the pool from cascading failures.

  • Threshold: After 3 consecutive failures to a node, the circuit opens and that node is excluded from candidate selection.
  • Reset: After 60 seconds, the circuit transitions to half-open, allowing a single probe request through. A successful probe closes the circuit; another failure reopens it immediately.
  • Effect on routing: Open-circuit nodes are skipped during endpoint resolution. If all candidates have open circuits, the request returns 503.

The circuit breaker is independent of the failover mechanism: a node with an open circuit will not receive requests even as a failover target.

Automatic Failover

When failoverEnabled is true in the load balancing configuration (the default), the proxy will retry failed requests on an alternate node.

Failover triggers when:

  • A connection error occurs reaching the selected node, or
  • The node responds with a 5xx status code.

On trigger, the proxy re-resolves the endpoint excluding the failed node and retries the request once. The original request body is replayed in full.

Streaming Failover Limitation

Failover is only possible before the first SSE chunk is sent to the client. Once streaming has begun, the response is committed and the error propagates directly to the client. Non-streaming requests support full retry on an alternate node.

Merged Model List

The GET /api/v1/inference/models (OpenAI-compatible) endpoint returns a merged list:

  • Local models: reported directly by the local inference server.
  • Network node models: discovered via the distributed model index, marked with "owned_by": "remote".

Duplicate model IDs are deduplicated; local models always take precedence.

{
  "object": "list",
  "data": [
    { "id": "llama-3-8b-instruct", "object": "model", "owned_by": "llama.cpp" },
    { "id": "mistral-7b-instruct", "object": "model", "owned_by": "remote" }
  ]
}

Distributed Pool Status

Two API endpoints expose the state of the distributed pool.

Inference Status (summary)

GET /api/v1/inference/status returns a distributed field when load balancing is enabled:

{
  "available": true,
  "routerRunning": true,
  "routerPort": 39080,
  "loadedModels": 1,
  "llmModels": 1,
  "diffusionModels": 0,
  "distributed": {
    "enabled": true,
    "strategy": "priority",
    "totalNodes": 2,
    "onlineNodes": 2,
    "remoteModels": 3
  }
}

Full Distributed Status

GET /api/v1/pool/distributed-status returns the complete pool view including per-node details:

{
  "enabled": true,
  "strategy": "priority",
  "totalNodes": 2,
  "onlineNodes": 2,
  "models": [
    {
      "id": "mistral-7b-instruct",
      "availableOn": ["node-fingerprint-abc"]
    }
  ],
  "endpoints": [
    {
      "fingerprint": "node-fingerprint-abc",
      "baseUrl": "http://192.168.1.50:39080",
      "status": "online",
      "latencyMs": 12,
      "loadedModelCount": 2,
      "priority": 5
    }
  ]
}

This endpoint is also accessible from the frontend via the get_distributed_pool_status Tauri IPC command.

Routing Statistics

GET /api/v1/pool/routing-stats (and GET /api/v1/nodes/routing-stats) returns per-node request metrics and circuit breaker state:

{
  "nodes": [
    {
      "fingerprint": "node-fingerprint-abc",
      "totalRequests": 150,
      "successCount": 147,
      "failureCount": 3,
      "avgLatencyMs": 42.5,
      "activeRequests": 2,
      "circuitOpen": false,
      "consecutiveFailures": 0
    }
  ],
  "totalRequests": 150,
  "totalSuccesses": 147,
  "totalFailures": 3,
  "circuitBreaker": {
    "failureThreshold": 3,
    "resetTimeoutSecs": 60
  }
}

When continuum-router is active, telemetry from /admin/stats is merged into these routing statistics. Request counts, successes, failures, and latency measurements reflect both traffic dispatched through continuum-router and local fallback requests. If a backend is marked unhealthy in continuum-router, its circuit state is opened.

The same merged data is also available via the get_routing_stats Tauri IPC command and the nodeStore.getRoutingStats() action in the frontend store.

Network Node Authentication

When a request is routed to a network node, the proxy adds an Authorization: Bearer <api_key> header. The API key is stored securely in the local node registry and is never exposed in logs or the UI.

Ensure that the network node has an API key configured. Requests to nodes without an API key will fail if the network node requires authentication.

Fallback Behavior

Condition Behavior
Load balancing disabled Serve locally only; return 503 if the model is not running.
Load balancing enabled, model available locally Serve locally.
Load balancing enabled, model on network node Forward to network node.
Load balancing enabled, model not found anywhere Return 503 Service Unavailable.
Network node unreachable, failover enabled Retry once on an alternate node.
Network node unreachable, failover disabled Return the upstream error to the client.
Node circuit open Skip node during routing; use alternate if available.