Parallel Request Slots¶

A loaded model serves requests through a fixed number of parallel request slots. The slot count is the Parallel requests setting in the Model Config Drawer's Context tab. Backend.AI GO passes it to llama-server as the --parallel flag, adding --kv-unified for more than one slot so the slots share one context. MLX models currently run with one slot; see MLX models.

What the setting does¶

Each in-flight request occupies one slot. Requests beyond the slot count wait in the engine's internal queue, where the delay surfaces only as longer first-token latency.

Agent features routinely send several requests to the same loaded model at once: a chat turn that @mentions sub-agents, Cowork sub-agents working in parallel, or multiple Squad agents sharing one local model. With a single slot, all of those calls serialize at the engine even though the application issued them concurrently.

Range and default¶

The default of 2 keeps the smallest realistic agent workload, a primary agent plus one sub-agent, from serializing. A value you set explicitly is always preserved, including 1 for deliberately single-slot operation.

The ceiling is a fixed constant, not derived from the loaded model. llama-server itself has no hard upper bound (production server deployments run 8 to 32 slots on server-class hardware), but on the consumer laptops Backend.AI GO targets, a context length sized for one conversation holds only a few long requests at a time, so slots beyond 8 mostly add requests that run out of room.

Requests share the context length¶

Context Length is the limit for one request. The parallel slots share one pool of that size: a single request can use all of it, and requests running at the same time together cannot exceed it.

Raising the slot count neither shrinks any request's window nor grows KV-cache memory. It lets more requests run at once against the same pool. When the requests running at the same time need more than the context length between them, llama.cpp fails the in-flight requests with Context size has been exceeded. If your agents run long requests in parallel, raise the context length (KV-cache memory grows roughly linearly with it) or lower the slot count.

The context length shown in Sessions and in the chat status bar is the one the engine reports it enforces, read from its /props endpoint after the model loads. It can be slightly larger than the configured value, because llama.cpp rounds the context up to a multiple of 256.

MLX models¶

mlxcel-server currently divides the context across its slots, so Backend.AI GO launches MLX models with one slot whatever this setting holds. An MLX model therefore serves one request at a time with the full context length, as models without batched decode already do. Parallel requests on MLX models will use the shared context once a released mlxcel supports it.

An explicit Max batch size in the MLX Scheduler tab still divides the context across concurrently decoded sequences, and that tab warns when a sequence would get fewer than 1,024 tokens. Sessions then shows the smaller window the engine reports, while the Context tab keeps the launched context length on its slider and notes the smaller window under it.

Choosing a value¶

• Plain chat, one conversation at a time: 1 or 2 slots. There is nothing to parallelize.
• Agent workflows (Cowork, Squad, sub-agent mentions): match the slot count to the number of agents that realistically run at once, and size the context length for their combined load. Four agents that each hold about 4K tokens need a 16K context between them.
• Memory-constrained machines (8 to 16 GB): KV-cache memory grows with the context length and stays the same whatever the slot count. Pick the largest context length that fits, then set the slot count to the number of requests that actually run together.

Verifying concurrency on your hardware¶

You can confirm that slots actually serve in parallel rather than serializing, and that each one sees the full context.

llama-server¶

1. Load a small model (for example Qwen3-4B-Instruct at Q4_K_M) with Context Length 8192 and Parallel requests 4.
2. Check /props on the inference port: default_generation_settings.n_ctx should report the full 8192.
3. Fire 4 overlapping streaming requests against /v1/chat/completions, keeping the prompts short enough that all four together stay within 8192 tokens.
4. Check /slots on the inference port: all 4 slots should report processing at the same time, and each request's first token should arrive long before the others finish.

mlxcel-server (macOS)¶

MLX models run with one slot until mlxcel supports a shared context, so there is no concurrency to verify yet. To confirm that a request gets the full context, load an MLX model with Context Length 8192 and check that /props on the inference port reports default_generation_settings.n_ctx as 8192.