Router Fallback and Upstream Routing¶

A request that reaches a cloud model through Backend.AI GO can be retried by two independent layers. The bundled Continuum Router retries across the models you list in its fallback chains. Some providers also pick models and retry on their own before they answer the router. Neither layer knows what the other did, so when both are active for the same model their attempts multiply, and no response says which layer retried or switched the model.

This page describes both layers, how they combine, and how to keep one of them in charge.

Two retry layers¶

Router fallback¶

The router's fallback settings are on Settings > API Server > Advanced. They map to these blocks of the router configuration:

Setting	Router configuration	What it does
Fallback chains	`fallback.fallback_chains`	For a primary model, the ordered list of models to try when a request to it fails.
Trigger conditions, Max fallback attempts, Timeout multiplier	`fallback.fallback_policy`	Which failures move the request along the chain (listed HTTP status codes, timeouts, connection errors, unknown models, an open circuit breaker, unhealthy backends), how many times one request may move, and how much longer each later attempt may run.
Mid-stream fallback	`streaming.mid_stream_fallback`	Whether a streaming response that fails after output has started is continued or restarted on the next model in the chain, bounded by its own Max fallback attempts.
Answer quality check	`fallback.quality_check`	Whether a non-streaming reply that arrived without error but is unusable also moves the request along the chain. See Answer quality check.

Router fallback is model-based. Each model in a chain is resolved through the normal backend pool, so a chain can move a request from one provider to another. A hop swaps the request's model field and forwards the rest of the body as the client sent it, apart from a few adjustments the router makes for the target, such as fitting a reasoning effort that model cannot accept.

When Notify on fallback is on for the primary model, a fallback that answers a non-streaming request, or a stream that had not started yet, is reported in the X-Fallback-Used, X-Original-Model, X-Fallback-Model, X-Fallback-Reason, and X-Fallback-Attempts response headers. A recovery after a stream has started cannot add headers, because they were already sent.

Answer quality check¶

The triggers above are all failures the transport or the provider reports. A model that answers with a refusal, an empty or cut-off reply, or an answer to a different question returns success, so the router serves that reply as it is. The answer quality check closes that gap for non-streaming /v1/chat/completions replies: a System One decision model (TypeSafe's hosted Jev, or a self-hosted laya-serve) reads the completed reply, and when the reply is probably unusable the router moves the request to the next model in the same chain.

Turn it on with the switch in the Answer quality check section of Settings > API Server > Advanced, which appears while fallback is on. The section has these settings:

Setting	Router configuration	What it does
Decision endpoint	`target.endpoint`	URL of the System One server, ending in `/v1/systemone`.
API key environment variable	`target.api_key_env`	Name of the environment variable that holds the server's bearer key. Leave it empty for a server without authentication.
Mode	`mode`	Shadow (the default) judges replies and records the moves it would have made, but always serves the first reply. Enforce moves the request.
Threshold	`threshold`	Probability of an unusable reply at which the check triggers, above 0 and at most 1. The default is 0.8.
Quality fallbacks per request	`max_extra_hops`	How many times the check may move one request, from 1 to 5. Each move also counts toward Max fallback attempts.
Judged models	`models`	Primary models whose replies are judged. Only models with a fallback chain are offered; selecting none judges every model that has a chain.

Start in Shadow. The 0.8 default is a starting point, not a value measured on your traffic, and in Enforce a false positive changes what a client receives. Compare how often the router would have moved requests against how often replies passed before you switch.

The decision model reads only the text of the last user message and the reply, each shortened to a fixed size by keeping its beginning and end. System prompts, earlier turns, tool calls, and tool results are never sent, which matters when the endpoint is a hosted service. A reply that calls a tool is not judged, and neither is a streaming reply.

When a move happens, the response carries x-continuum-quality-fallback: 1, and if Notify on fallback is on, X-Fallback-Reason is quality_check. Both replies were real generations, so both are counted in usage. A failed or slow decision call never blocks a request: the router serves the reply it already has.

Backend.AI GO does not store the API key

API key environment variable takes the variable's name, never the key. Backend.AI GO does not store or pass this key: the router reads the variable from the environment the app was started in. Set it before starting Backend.AI GO, for example in the shell that starts the headless server. On macOS, a desktop app opened from Finder or the Dock does not see variables exported in a shell profile.

The router configuration file can also name a decision target by backend (target.backend) instead of by URL. Backend.AI GO keeps and checks such an entry but does not offer it in the settings, because it rewrites the backend list whenever a model is run or stopped, and the router refuses the whole configuration when the named backend is missing or cannot answer decision calls. When that happens, Backend.AI GO leaves the quality check out of the router configuration instead, logs a warning, and keeps your setting, including across a restart. The check is written again as soon as the backend is back. While it is left out, the section shows a warning naming the backend.

Upstream routing¶

Some providers are aggregators that front many models and route each request themselves. OpenRouter is the one Backend.AI GO recognizes. The router forwards the fields OpenRouter reads on /v1/chat/completions without changing them, so a client can use all of its routing controls:

openrouter/auto lets OpenRouter pick the model for each request.
A models array is an ordered list of models OpenRouter tries in turn when one fails, for example on rate limiting or downtime.
provider.allow_fallbacks decides whether OpenRouter moves a request to another host serving the same model when the preferred host fails. It defaults to true, and it applies to every model OpenRouter serves, including an explicitly named one such as openai/gpt-5.6-sol.

OpenRouter returns the model that finally answered in the response's model field and bills at that model's rate. It does not report the attempts it made on the way. See OpenRouter's guides on model fallbacks, the Auto Router, and provider selection for the details of each control.

How the layers multiply¶

The router sees one upstream call per model attempt. When that call goes to OpenRouter, it can already contain several attempts: other hosts for the same model, then other models from a models array. The router sees a failure only after OpenRouter gives up, and then it moves to the next model in its own chain. If that model is also served through OpenRouter, the same inner sequence runs again.

For example, take the default Max fallback attempts of 3 and a chain that names three models, all served through OpenRouter. One request can reach OpenRouter three times. If OpenRouter tries two hosts for each model before it returns an error, the request has used six upstream attempts, and a stream that failed after its output began can go around again through mid-stream fallback. The response names only the model that answered.

What each layer reports does not add up to the whole path either. The router's X-Fallback-* headers describe router hops only, and OpenRouter's response describes its final choice only. A request that costs several times what it should, or that answers from a model nobody picked, gives no direct sign of which layer caused it.

The forwarded body has a second effect. A router hop does not remove OpenRouter's fields, so a models array or provider object sent for OpenRouter travels to the next model in the chain even when that model is served by a different provider, and a strict endpoint can reject fields it does not recognize.

Let one layer own fallback¶

For a model served through a provider that routes upstream, pick one layer to own fallback:

Let the upstream own it. Keep the model out of the router's fallback chains, both as a primary model and as a fallback in other chains. Express the fallback you want per request instead, with openrouter/auto, a models array, or provider.allow_fallbacks. This suits a deployment that uses OpenRouter for its catalog-wide routing.
Let the router own it. Name explicit models in the chain, and have clients ask OpenRouter not to substitute: no models array, no openrouter/* routers, and provider.allow_fallbacks set to false. This suits a chain that moves between OpenRouter and other providers, or a deployment that needs the router's headers and fallback metrics to describe every hop.

Running both can still be a deliberate choice, for example a router chain that reaches a different provider only after OpenRouter has run out of options. The app allows it and warns instead of refusing.

The upstream routing warning¶

When a fallback chain on Settings > API Server > Advanced names a model served through a provider that routes upstream, the chain is marked Upstream routing and the affected models and provider are listed under it. The chain can still be edited and saved.

A provider counts as routing upstream when the host of its base URL is openrouter.ai or a subdomain of it. The check reads only the base URL, so it also covers an OpenRouter provider added as Custom OpenAI-Compatible with https://openrouter.ai/api/v1, and it covers every model that provider serves rather than only the openrouter/* routers, since host-level fallback applies to all of them. A lookalike host such as openrouter.ai.example.com does not match.

Which provider serves which model comes from the backends the router currently has, so the warning follows the providers you have registered and the models selected for them.

Model IDs with a slash¶

OpenRouter's model IDs use the vendor/model form. The bundled router accepts / and : in fallback chain model names (since lablup/continuum-router#1585), next to letters, digits, ., _, and -, so a chain can name an OpenRouter model such as openai/gpt-5.6-sol or an Ollama model such as llama3:8b directly. The overlap on this page then applies to chains in full, and the warning above covers it. Whitespace, control characters, and other punctuation are still refused, and a name may be at most 128 bytes.