Supervisor Agent¶

The Supervisor Agent is a centralized decision-making system that continuously monitors the health and resource usage of all running models, and takes coordinated actions to ensure reliability and efficiency. It automatically restarts failed models, evicts idle ones to free resources, and even predicts potential issues before they occur.

Concept¶

Backend.AI GO can run multiple models simultaneously, each consuming GPU memory, system RAM, and CPU. Without supervision, a crashed model might go unnoticed, or idle models might waste valuable resources indefinitely.

The Supervisor Agent solves this by introducing a three-layer architecture:

┌─────────────────────────────────────────────────────┐
│  Intelligence Layer                                  │
│  Predictive Analytics · Adaptive Tuning · Webhooks   │
└──────────────────────┬──────────────────────────────┘
┌──────────────────────▼──────────────────────────────┐
│  Decision Layer                                      │
│  Supervisor Agent · Policy Engine · Resource Arbiter  │
└──────────────────────┬──────────────────────────────┘
┌──────────────────────▼──────────────────────────────┐
│  Foundation Layer                                     │
│  Event Bus · Heartbeat Monitor · Hook Registry        │
└─────────────────────────────────────────────────────┘

• Foundation Layer: Detects problems — heartbeat checks, idle tracking, and lifecycle events.
• Decision Layer: Decides what to do — evaluates policies, resolves conflicts, and executes actions.
• Intelligence Layer: Anticipates issues — usage forecasting, adaptive parameter tuning, and external notifications.

Getting Started¶

Enabling the Supervisor¶

1. Go to Settings > Supervisor.
2. Toggle the Supervisor Agent switch to enable it.
3. Select a Configuration Preset (see below).

Once enabled, the Supervisor begins its multi-tier decision loop, checking model health and system resources at regular intervals. The loop does not start on its own at app launch: nothing runs until you turn the switch on, and stopping the app stops it.

Configuration Presets¶

Three built-in presets let you quickly tune the Supervisor's behavior:

• Conservative: Minimal intervention. Only restarts failed models and protects pinned models. Best for stable, long-running setups.
• Balanced: Moderate resource management, with the idle eviction policy and optimization enabled. Recommended for most users. The policy being enabled is not the same as the Supervisor being allowed to act on it; see the note below the list.
• Aggressive: Proactive resource reclamation. Short idle timeouts, lower pressure thresholds, and more restart attempts. Best for resource-constrained machines.

Unattended idle eviction is off in every preset, including Aggressive. It is a separate opt-in (Settings > Supervisor > Configuration > Allow unattended idle eviction) rather than a preset value, so switching presets never turns it on by itself. Turning it on lets the Supervisor unload one of your models without asking; read the caveat under Idle Tracking below before you do.

Health Monitoring¶

The Supervisor runs a per-model heartbeat monitor that periodically checks whether each loaded model's inference server is responding.

Health States¶

Each model transitions through four health states:

Unknown → Healthy → Degraded → Dead
                  ↑           ↓
                  └───────────┘
                   (recovery)

• Healthy: The model responds to health checks normally.
• Degraded: Several consecutive health checks have failed, but the model hasn't reached the dead threshold yet.
• Dead: The model has stopped responding entirely.

When a model transitions to Dead, the Supervisor can automatically restart it (up to the configured maximum attempts).

Pinning Models¶

You can pin a model to prevent it from being evicted by idle tracking or resource optimization policies. A pinned model is kept running: it is never evicted for inactivity, and when it fails the Supervisor restarts it (or reloads it, if the failure took it out of the pool) regardless of what the other policies want.

Recovery is bounded, though. A pin gets its own attempt budget, Max pinned recovery attempts in the preset table above, which is larger than the ordinary Max restart attempts in every preset because a pin is a stronger statement about the model than the Availability policy's default handling. It is not unlimited: a model that cannot come back at all (the file was moved, the engine binary is missing, the GPU is out of memory) would otherwise have the Supervisor spawning an engine process every few seconds for as long as it ran, with no setting able to stop it.

• Each failure gets a fresh budget. The count is per failure, not per session. Once the model reports healthy again, the next failure starts from zero.
• A restart in progress does not refund an attempt. A model reads as Unknown for a cycle after a restart, and that cycle is neither an attempt nor a recovery.
• Running out is announced. When the budget is exhausted the Supervisor stops retrying and raises one critical alert naming the model, which appears in the application log (Logs, filtered to ERROR) and as an emit_alert entry in Settings > Supervisor > Recent decisions. It is raised once per failure, not once per tick.
• The pin itself is untouched. The Supervisor never unpins a model on your behalf. Fix the model and load it manually, or unpin it; either brings the budget back.

To pin a model, right-click it in the model list and select Pin Model, or use the API:

curl -X POST http://localhost:8090/api/v1/lifecycle/pin \
  -H "Content-Type: application/json" \
  -d '{"model_id": "my-important-model"}'

Idle Tracking¶

The Supervisor tracks how long each model has been idle. Models that exceed the configured idle timeout are candidates for eviction, freeing up GPU memory and RAM for other models.

• The idle clock starts at load, not at the last request: inference traffic reaches the model server through the router, which the app does not observe, so a model that has been serving requests all afternoon still reports as idle. This is why unattended eviction is a separate opt-in and is off in every preset.
• Pinned models are exempt: Pinned models are never evicted due to inactivity, whatever the opt-in is set to.
• Two switches, both required: the Idle Eviction policy has to be enabled and Allow unattended idle eviction has to be on before the Supervisor proposes unloading an idle model. With the shipped defaults it proposes nothing.

Policy Engine¶

The Supervisor uses a priority-based Policy Engine to decide what actions to take. Five built-in policies are evaluated on every decision cycle:

When two policies propose conflicting actions (e.g., one wants to keep a model loaded while another wants to evict it), the higher-priority policy always wins. Every conflict is recorded in the audit log for transparency.

Audit Log¶

Every decision the Supervisor makes is recorded in a detailed Audit Log. Each entry includes:

• Timestamp and decision ID
• System snapshot at the time of the decision (loaded models, health status, resource usage)
• Policies evaluated and their proposed actions
• Conflicts resolved (which policy won and why)
• Actions taken and their outcomes (success, failed, or skipped)

Viewing the Audit Log¶

Go to Settings > Supervisor and scroll to the Recent Decisions section. Click any entry to expand its details, including the full snapshot summary, action list, and conflict records.

You can also query the audit log via the Management API:

curl "http://localhost:8090/api/v1/supervisor/audit?limit=20"

Fallback Routing¶

The Supervisor integrates with the Continuum Router to provide automatic failover. When a primary model becomes unresponsive:

1. The Supervisor detects the failure via heartbeat monitoring.
2. It activates a fallback route, redirecting inference requests to a designated backup model.
3. When the primary model recovers, the fallback route is deactivated and traffic returns to normal.

This ensures that your API clients experience minimal disruption, even during model failures.

Predictive Analytics¶

The Supervisor can analyze historical usage patterns to make proactive decisions:

• Usage Forecasting: Predicts how many requests each model will receive in the next hour using exponential weighted moving average (EWMA) analysis.
• Failure Prediction: Detects early warning signs of instability, memory leaks, thermal throttling, or latency degradation.
• Demand-Based Preloading: Automatically loads models that are predicted to be needed soon based on usage trends.

Viewing Forecasts¶

In the Supervisor settings page, the Predictive Analytics section displays:

• A table of usage forecasts per model (predicted requests/hour, trend direction, confidence level)
• Risk indicators showing failure predictions with recommended actions

Adaptive Tuning¶

The Supervisor can learn from its own decisions by tracking outcomes:

• Outcome Feedback: After executing an action (e.g., restarting a model), the system checks whether the action achieved its goal.
• Parameter Adjustment: Based on success rates, the Supervisor adjusts its internal parameters (within safe bounds) to improve future decisions.
• Rate-Limited Changes: Adjustments are capped at a configurable maximum percentage per cycle to prevent oscillation.

View the tuning history in Settings > Supervisor to see what parameters were adjusted, their old and new values, and the reasons for each change.

External Integrations¶

Webhooks¶

Register webhook endpoints to receive real-time notifications about Supervisor events:

• Health state changes
• Auto-restarts
• Model evictions and preloads
• Fallback activations
• Resource alerts
• Failure predictions

Each webhook delivery includes an HMAC-SHA256 signature (if a signing secret is configured) for payload verification.

To add a webhook:

1. Go to Settings > Supervisor.
2. Scroll to the Webhooks section.
3. Enter a name, URL, and select the event types you want to receive.
4. Click Add Webhook.

You can test a webhook using the Test button to verify connectivity.

Prometheus Metrics¶

Enable the Prometheus metrics endpoint to scrape Supervisor and model health data:

curl http://localhost:8090/metrics

Available metrics include:

• aigo_model_health_status — Per-model health (0=dead, 1=degraded, 2=healthy)
• aigo_model_request_total — Total requests per model
• aigo_model_inference_latency_seconds — Inference latency percentiles (p50, p95, p99)
• aigo_supervisor_decisions_total — Decision count by action type
• aigo_resource_gpu_memory_usage_ratio — GPU memory utilization
• aigo_resource_system_memory_usage_ratio — System RAM utilization

OpenTelemetry (OTLP)¶

For environments using OpenTelemetry collectors, enable OTLP export in the Supervisor configuration and specify your collector endpoint.

Lifecycle Events (SSE)¶

For headless or web clients, the Supervisor streams lifecycle events via Server-Sent Events (SSE):

curl -N http://localhost:8090/api/v1/lifecycle/events

Events include model loads/unloads, health changes, idle timeouts, auto-restarts, and resource pressure alerts. This allows external tools to react to system changes in real time without polling.

API Reference¶

Supervisor Endpoints¶

Lifecycle Endpoints¶

Fail-Safe Design¶

The Supervisor is designed with a fail-open philosophy:

• Foundation hooks remain active: The base heartbeat and lifecycle hooks from the Foundation Layer are never disabled. If the Supervisor Agent itself crashes, these hooks continue to provide basic health monitoring and auto-restart.
• Agent self-heartbeat: The Supervisor emits its own heartbeat. If the frontend or Management API detects the Supervisor's heartbeat is lost, it falls back to the Foundation Layer's hook-based behavior.
• Continuum Router fallback: Even during Supervisor downtime, the Continuum Router's fallback model configuration ensures that API requests continue to be served.

This layered approach ensures that no single component failure can take down the entire system.