Skip to content

OpenAI-Compatible Services

Backend.AI GO's architecture is built around the OpenAI API standard. This means you can connect not just to OpenAI itself, but to any service or local server that speaks the "OpenAI dialect."

What works with this?

You can connect Backend.AI GO to a vast ecosystem of tools and providers, including:

  • Local Inference Servers: Ollama, LocalAI, LM Studio, Text Generation WebUI (Oobabooga).

  • Third-Party APIs: Groq, DeepSeek, Together AI, Perplexity, Anyscale.

  • Enterprise Gateways: MLflow, Azure OpenAI (via proxy), and internal corporate gateways.

How to Connect

  1. Go to the Providers tab.

  2. Click Add provider.

  3. Select Custom OpenAI-Compatible as the provider type. It is listed directly below OpenAI (GPT).

    Do not use OpenAI (GPT) for these endpoints. That type has no Base URL field and always checks the key against https://api.openai.com, so a token for any other server is rejected. When validation fails, the error message starts with the URL that was checked, and the setup form points you to Custom OpenAI-Compatible when the key does not look like an OpenAI key.

  4. Enter the details:

    • Base URL: The full URL to the API endpoint (usually ending in /v1).

      • Example (Ollama): http://localhost:11434/v1
      • Example (DeepSeek): https://api.deepseek.com/v1
      • Either form works. A base URL that already ends in an API root (/v1, /openai) is detected, so the path is not doubled into /v1/v1/models.

    • API Key: The authentication token. Optional for custom endpoints: many self-hosted servers (like Ollama) accept no authentication at all, so you can leave the field empty and still use Test Connection to confirm the endpoint is reachable. If your server does require a key but does not actually validate it, any string works (e.g., sk-dummy).

  5. Model Discovery: Backend.AI GO will attempt to list available models from the endpoint. If auto-discovery fails, you may need to manually enter the model names.

  6. Select Models: The wizard's model selection step lists the models the endpoint reported. Matching known defaults are pre-checked; adjust the selection, then click Finish. The chosen models appear in the chat model selector right away, without opening the provider card. Choose Later adds the provider without a selection (Backend.AI GO then seeds any known defaults it can detect, and you can pick models on the provider card later). When no catalog is available, the step says so and you can still finish.

Run Cost and the Model That Answered

Some OpenAI-compatible services report what each completion cost. OpenRouter, for example, returns usage.cost in US dollars on every response. When a completion carries that figure, an agent run uses it in two places:

  • Spending limits: the reported cost counts toward any spending limit rule. For a model the service picks per request (such as openrouter/auto), no static price exists, so the reported cost is the only figure that can make a spending rule fire.

  • Run Details: the cost shown for the run is the reported figure. Hover over it to see whether the provider reported it or it was estimated from model pricing.

A completion without a reported cost is priced from the model pricing table, as before. A negative or malformed figure is ignored rather than counted as zero.

When the service answers with a different model than the one requested, Run Details lists the models that actually served the run under Served by. This is how a run that asked for openrouter/auto shows which model answered it.

Request Extensions

Some endpoints need extra request fields or headers on every call: chat_template_kwargs for a vLLM server, provider routing preferences and the HTTP-Referer and X-Title headers for OpenRouter, or a tenant header for an API gateway. After the provider is added, click Edit on its card and open Request extensions at the bottom of the form. The router adds these fields and headers to every request it sends to this provider.

  • Extra body (defaults): a JSON object merged into the request body. A field the client already sends keeps the client's value.

  • Body overrides: a JSON object merged into the request body that replaces whatever the client sent. Use it for a policy a client must not relax, such as {"provider": {"data_collection": "deny"}} on OpenRouter.

  • Extra headers: one Name: value per line. Header values are stored like API keys and are never shown again after saving. When you edit the provider, a saved header appears masked; leave that line as it is to keep the saved value.

Nested objects are merged key by key, so an override of provider.data_collection keeps the client's provider.sort. Arrays and single values are replaced as a whole.

The router checks the block before it is saved and rejects fields it reserves for itself (such as model, messages, or stream) and headers it sets itself (such as Authorization). Its message is shown in the form and names the field it refused. The router must be running to save request extensions. A change to the headers restarts the router so it picks up the new values.

From the command line, the same fields are set with aigo provider add or aigo provider update:

aigo provider update <id> \
  --extra-body '{"chat_template_kwargs": {"enable_thinking": false}}' \
  --header "X-Title: Backend.AI GO"

--extra-body-override sets body overrides, --extra-body-file and --extra-body-override-file read the JSON object from a file (- reads standard input), and --header can be repeated.

Examples

Connecting to Ollama

If you have Ollama running on another machine (or locally on a port):

  • Base URL: http://<ip-address>:11434/v1
  • API Key: ollama (or leave empty if allowed)

Connecting to Groq

For ultra-fast inference using Groq's LPU:

  • Base URL: https://api.groq.com/openai/v1
  • API Key: Your Groq API Key

This flexibility turns Backend.AI GO into a universal frontend for all your AI needs.