10.7. Pipeline Serving¶

Backend.AI GO can serve a single MLX model that is too large for any one machine by splitting it across nodes with pipeline parallelism. Each node holds a contiguous block of transformer layers (a "stage"); only the activation tensor crosses a node boundary, once per stage boundary per token. This unlocks capacity, the ability to run a model no single node can hold, over a low-bandwidth interconnect. It does not speed up inference: a token still traverses every layer in sequence.

This page covers the full end-to-end flow: pick a model, stage it on every node, launch the cluster, run chat against it, watch live health, and stop it. For the read-only planning preview, see Pipeline-Parallel Planning.

How it differs from distributed routing¶

Backend.AI GO has two independent scaling axes that coexist:

A running pipeline cluster is itself just one more backend the horizontal router can load-balance against.

Requirements¶

• At least two registered peer nodes (see Manual Registration or Auto-Discovery).
• A homogeneous cluster: every node shares the same accelerator family (all Apple Silicon, or all Linux CUDA) and the same mlxcel version.
• An MLX-format model whose weights plus KV cache do not fit on a single one of the selected nodes.
• Direct node-to-node TCP reachability on the inter-node transport port range (see Network and firewall requirements).
• The "Enable distributed serving" setting turned on (off by default; see Settings).

The serve flow¶

Open the Models tab and select the Distributed sub-tab. When distributed serving is enabled, the Serve across cluster section drives the whole flow:

1. Pick a model. Choose an MLX model from the dropdown. Only MLX-format models are eligible.
2. Select nodes. Toggle two or more registered nodes. Each node shows a capability badge and its reported free memory.
3. Preview the shard plan. Select Preview shard plan. The plan shows the layer range per node, a per-node memory-fit bar, and any warnings. See Pipeline-Parallel Planning for how to read it.
4. Stage the model. Select Stage on all nodes. Each node is classified as already present, needs download, or mismatched; nodes that are missing the model download it on their own download path. Per-node progress is shown. Staging must complete on every node before launch.
5. Launch the cluster. Select Launch cluster. A launch timeline shows the ordered bring-up: stages start first, become transport-ready, then the coordinator comes up and registers with the router. Launch is all-or-nothing (see below).
6. Use it. Once the cluster is healthy, the model is available through the normal model selector. Run chat or completions against it exactly as you would a single-node model; the router proxies requests to the coordinator.
7. Monitor and stop. The section switches to a live monitor: an aggregate health badge plus one card per stage (node, layer range, process liveness, transport-port state). Select Stop cluster to tear it down cleanly.

The cluster lifecycle¶

Ordered launch is mandatory¶

The coordinator connects out to the stages, so every stage worker must be listening on its transport address before the coordinator starts. The orchestrator therefore:

1. starts every remote stage in ascending stage order;
2. waits until each stage's transport port is accepting connections;
3. only then starts the local coordinator and waits for its health check to pass;
4. registers the coordinator's endpoint with continuum-router as one backend.

If any step fails, everything already started is torn down so there is never a half-started cluster or an orphan process.

The coordinator is the only entry point¶

One coordinator process binds the OpenAI-compatible HTTP listener, receives requests, drives the pipeline, and streams results back. Stage workers serve no client HTTP; they bind only their inter-node transport address. continuum-router sees a single mlxcel backend and never learns it is a sharded cluster.

Failure is all-or-nothing¶

A pipeline cannot survive the loss of a stage. If any stage worker dies, the whole cluster is marked unhealthy and torn down. In the UI this surfaces as the aggregate health badge turning unhealthy and the cluster being removed; the router backend is deregistered so no stale backend is left behind.

Sessions integration¶

A running cluster appears in the Sessions surface as a single session (the coordinator's projection), marked with a Cluster badge. Open its detail drawer and select the Cluster stages tab to see the per-stage breakdown and aggregate health, the same view as the live monitor. The Stop action in Sessions reuses the same lifecycle command as the Distributed Models tab.

Network and firewall requirements¶

Pipeline serving needs direct node-to-node TCP on a dedicated inter-node transport port range, separate from the OpenAI HTTP listener. Each node's stage binds one port from this range (the TOML address), and the coordinator connects out to those ports.

• The default range is 39600-39629. Change it under Settings -> Advanced -> Distributed serving -> Transport port range.
• Open this range between nodes in any host or network firewall. The ports only need to be reachable node-to-node on your local network, not from the public internet.
• Wi-Fi or Ethernet latency caps practical cluster size. For all-Apple-Silicon local clusters, Thunderbolt is a faster transport; RDMA is an opt-in fast path that falls back to TCP.

Settings¶

Under Settings -> Advanced -> Distributed serving:

• Enable distributed serving turns the Serve across cluster flow on. It is off by default.
• Default transport backend chooses the inter-node transport the planner proposes: tcp (default, works everywhere), thunderbolt (all-Apple-Silicon local clusters), or rdma (opt-in, falls back to TCP).
• Transport port range sets the inter-node transport port pool described above.

Limitations¶

• Pipeline parallelism only. Tensor parallelism and disaggregated serving are not part of this flow.
• Homogeneous clusters only. Mixed accelerator families in one cluster are not supported.
• Identical weights on every node. Each node loads its own on-disk copy of the same model at the same quantization; staging verifies presence and triggers per-node download where missing.
• Upstream validates pipeline parallelism best on Llama-family text models and two-stage topologies. The planner gates on model family and warns beyond two stages.