10.6. Pipeline-Parallel Planning¶
Backend.AI GO can distribute a single oversized MLX model across multiple peer nodes using pipeline parallelism. The planning step turns your node selection into a concrete cluster plan: how many layers each node holds, whether each node's free memory is sufficient, and what inter-node transport ports will be used. No servers are launched during planning; planning is a pure preview step.
Status: Planning (this page) is the preview step. To stage, launch, monitor, and stop the cluster end to end, see Pipeline Serving.
When to use pipeline parallelism¶
Pipeline parallelism is for models that do not fit on any single node. If a model fits on one node, the planner will report an error and suggest loading it directly on that node instead.
Requirements:
- At least two registered peer nodes (see Manual Registration or Auto-Discovery)
- All nodes must be running the same mlxcel version
- All nodes must use the same accelerator family (e.g. all Apple Silicon)
- The total free memory across all selected nodes must exceed the model's estimated weight size
Opening the plan preview¶
- Go to the Models tab and select the Distributed sub-tab.
- Choose an MLX model from the dropdown. The planner only considers models that the
DistributedServingcapability gate accepts (Apple Silicon, mlxcel engine). - Select two or more registered nodes from the node list. Use the checkboxes to include or exclude nodes.
- The plan updates automatically as your selection changes.
Reading the plan¶
The plan preview shows one row per selected node:
| Column | Meaning |
|---|---|
| Node | Registered node name |
| Layers | Contiguous layer range assigned to this stage (e.g. 0-15) |
| Memory fit | Bar showing estimated weight + KV cost against the node's reported free memory |
| Warnings | Per-node soft issues (see below) |
A summary at the top shows total stages, total layers, and the --pp-layers spec string that the cluster will use.
Errors¶
If the planner cannot produce a valid plan, it shows one of these errors:
| Error | Cause |
|---|---|
| Heterogeneous accelerator | Selected nodes use different GPU families |
| Mismatched mlxcel version | Nodes report different mlxcel versions |
| Incapable node | A selected node does not meet the DistributedServing gate |
| Too few nodes | Fewer than two nodes are selected |
| Model fits on one node | The model's total memory fits on a single node; use direct loading instead |
| Insufficient total memory | The combined free memory across all nodes is less than the model's estimated weight size |
| Too few layers | The model has fewer layers than selected nodes; each stage needs at least one layer |
Warnings¶
Warnings appear inside a valid plan. The plan is still usable, but review each warning before launching:
| Warning | Meaning |
|---|---|
| Non-Llama family | The model architecture is outside the best-validated set for pipeline parallelism; results may vary |
| More than two stages | mlxcel upstream documentation has only been validated for one or two stages |
| Node below memory floor | A node's estimated stage memory exceeds 90% of its reported free memory; that node may OOM at runtime |
| Model not yet staged | A node does not have the model files staged locally; use the Stage button in the Distributed tab to stage the model across all selected nodes before launching |
| Unreachable transport port | A node's inter-node transport port did not respond during planning; check network reachability before launching |
Network and firewall requirements¶
Each stage node in a pipeline cluster needs a dedicated inter-node transport port. By default, Backend.AI GO allocates ports from the range 39600-39629. You can change this range in Settings > Inference > Distributed Transport Port Range.
For the cluster to function, every node in the plan must be able to reach every other node on its assigned transport port. On most LAN setups this works without changes. On managed networks or when nodes are on different subnets:
- Open TCP ports 39600-39629 (or your configured range) between all participating nodes in both directions.
- The coordinator node (the first stage) also needs one additional port from this pool for its own transport listener.
If a node reports a transport-port warning during planning, check that your firewall allows the port and that the node is reachable from the other nodes at that address.
Layer distribution algorithm¶
The planner splits the model's L transformer layers across N stages using a memory-proportional largest-remainder algorithm:
- Compute each node's share of total free memory.
- Multiply each share by
Lto get a fractional layer count per stage. - Start each stage at
floor(fractional count)layers. - Distribute remaining layers one by one to the stages with the largest remainders.
- Guarantee each stage at least one layer.
The embedding table is pinned to the first stage and the output head to the last stage, matching mlxcel's distributed tensor layout.
The planner uses the same memory estimator as the model browser (per-layer weight cost, embedding size, and per-stage KV cache). Numbers shown in the plan preview match what the runtime will attempt.
Changing the transport port range¶
Go to Settings > Inference and adjust Distributed Transport Port Start and Distributed Transport Port End. The range must be at least as wide as the number of stages plus one (for the coordinator). Changes take effect at the next planning request.