Container Troubleshooting Guide¶

This guide covers common issues with container execution and multi-channel messaging in Backend.AI GO.

Container Runtime Issues¶

Runtime Not Detected¶

Symptom: Container Runtime in Settings > Squad shows Not Available.

Diagnosis:

curl http://localhost:8001/api/v1/container/runtime
# Expected: "available": true

Solutions:

Docker Does Not Answer¶

Symptom: The runtime status reads Docker CLI found but the daemon did not answer within 10s.

The Docker CLI is installed, but its daemon accepted the connection and never replied. This happens when Docker Desktop stops partway through starting, or when DOCKER_HOST points at a socket that nothing serves. Each runtime check waits 10 seconds and then reports containers as unavailable, so the app and aigo-server still start.

Solutions:

• Quit Docker Desktop and open it again, or restart the daemon on Linux (sudo systemctl restart docker). Then check that docker version returns.
• Unset DOCKER_HOST, or point it at a running daemon.
• Restart Backend.AI GO or aigo-server before you use Hermes. Hermes picks its runtime once, when the app starts, so it does not use a daemon that recovers later.

Runtime Detected but Commands Fail¶

Symptom: Runtime shows as available but container operations fail.

Solutions:

• Restart the container daemon: Docker Desktop → Quit and reopen; container system stop && container system start for Apple Container
• Check available disk space (containers need disk to run)
• Review the Backend.AI GO application logs for detailed error messages

Image Build Issues¶

Build Fails Immediately¶

Symptom: Image build starts but fails right away.

Diagnosis:

curl http://localhost:8001/api/v1/container/image/status

Solutions:

• Ensure the container runtime is running before attempting a build
• Check available disk space (the agent runner image is approximately 1-2 GB)
• Look at the output field in the build status response for the error

Build Hangs Indefinitely¶

Symptom: The build button keeps showing Building... for a long time with no result.

Solutions:

• Network issues can cause base image downloads to hang; check your internet connection
• If using a proxy, ensure Docker/Apple Container is configured to use it
• The app has no control to cancel a running build. After verifying network connectivity, build again once the current build ends

Image Becomes Outdated¶

Symptom: Container agents fail with missing tools or incompatible dependencies.

Solution: Rebuild the image after a Backend.AI GO update:

• Settings > Squad > Agent Runner Image → Click Rebuild Image

Squad Task Fails With an Agent Runner Protocol Error¶

Symptom: Every squad task assigned to a container-mode agent fails, and the task error or the container status shows one of:

• Container agent error: ... (the image predates agent runner protocol 2; rebuild aigo-agent-runner:latest or update the custom image)
• Container agent speaks protocol version <v>; this app speaks 2 (rebuild aigo-agent-runner:latest or update the custom image)

Cause: The agent runner image and the app speak different versions of the agent runner protocol. The first message comes from an image built before the protocol existed, which cannot read the task the app sends. The second comes from an image built for a different protocol version, such as a version 1 image built by an earlier app, which would otherwise ignore the agent's model and call its own default.

Solutions:

• Rebuild the image so it matches this app: Settings > Squad > Agent Runner Image → Click Rebuild Image, or run ./container/build.sh from a source checkout
• If the agent uses a custom image, update it to read the version 2 input, call the model it is given, and write one output block, then rebuild it
• After the rebuild, run the squad request again: the image is read when each task starts, so tasks that already failed stay failed

Mount Security Issues¶

Mount Rejected: "Path contains blocked pattern"¶

Symptom: Mount validation fails with a blocked pattern error.

Cause: The host path contains a blocked pattern (e.g., .ssh, .env, .aws).

Solution: Use a different path that does not contain sensitive directory names, or copy only the necessary files to an approved location.

Mount Rejected: "Path not under any allowed root"¶

Symptom: Mount validation fails even though the path does not contain blocked patterns.

Cause: The path is not under any configured allowed root.

Solution:

1. Open the Cowork Settings drawer (in a Cowork conversation, click Cowork settings in the workspace rail's Context section) and select the Mount Security tab. See Adding Allowed Roots.

2. Add the parent directory of the path you want to use as an allowed root.

3. Be specific — avoid adding broad roots like your home directory.

Symlink Traversal Blocked¶

Symptom: A path is rejected even though the literal path looks safe.

Cause: The path resolves via symlinks to a blocked location.

Solution: Use the canonical (real) path, not a symlink. The mount validator canonicalizes all paths before checking.

Credential Proxy Issues¶

A squad container reaches its model through the credential proxy's router route, which the app starts on its own when a container task needs it. Squad tasks need no credential mapping: a mapping is only for a custom image that calls an external API itself. See Model Route.

Container Task Cannot Reach Its Model¶

Symptom: A container-mode task fails before or during its model call.

Diagnosis Steps:

1. While a container task runs, verify the proxy is running:

   curl http://localhost:8001/api/v1/container/credential-proxy/status
   # Expected: "running": true
   

2. Check the running task container's environment (docker inspect <container>):

   • ANTHROPIC_BASE_URL should be http://host.docker.internal:<proxy port>/router/anthropic on Docker, and http://localhost/router/anthropic with AIGO_INFERENCE_SOCKET=/tmp/aigo-inference.sock on Apple Container
   • ANTHROPIC_API_KEY should be a cpt_ token. It is valid only while that task runs.

3. Match the task error:

   Error	Cause	Solution
   Container mode under Apple Container needs the credential proxy's Unix socket, which this process could not create	The proxy runs, but on TCP only: it could not create its Unix socket, which is the only route an Apple Container guest has to the host. The sockets directory is usually the cause, for example one owned by another user or open to group or other access.	Look for the credential proxy socket warning in the app log and fix what it names, then stop and start the proxy (POST /api/v1/container/credential-proxy/stop, then start) or restart the app.
   Container agents need the credential proxy, which this runtime did not provide	This runtime was started without a credential proxy.	Run the agent in-process, or use the desktop app or a headless server that provides the proxy.
   Container inference route unavailable: <error>	The proxy could not start. The port (3001 by default) is taken, or on Linux there is no docker0 bridge address to listen on (rootless Docker, Docker Desktop for Linux).	Free the port or set another one with PUT /api/v1/container/credential-proxy/port, or use Docker with its default bridge.
   Router unreachable: <error> inside Container agent error: ...	The proxy reached no router: it stopped or restarted while the task ran.	Check that inference is running, then retry the task.
   Request must carry a valid router access token	The request came with no token, or with one whose task already ended.	Tokens do not outlive their task. Retry the task rather than reusing a token.
   HTTP 401 from the router inside Container agent error: ...	The router requires a key (external access is on) and the app has none available for it.	Restart the app so it can read its router key, or turn external access off.

Proxy Starts But Requests Fail¶

Symptom: Proxy status shows running, but model calls from containers still fail with a connection error.

Solutions:

• Verify how the container reaches the host:
  • Docker on Linux: the container gets --add-host=host.docker.internal:<docker0 address>, and the proxy listens on that same address. Compare ip -4 addr show docker0 with the proxy status bindAddress.
  • Docker Desktop (macOS, Windows): host.docker.internal is built in, and the proxy listens on 127.0.0.1.
  • Apple Container: the proxy's Unix socket is relayed into the container at /tmp/aigo-inference.sock. The proxy status does not report the socket; check that credential-proxy-<instance>.sock exists in the app sockets directory while the proxy runs.
• Check firewall rules are not blocking the proxy port (3001 by default)

IPC Communication Issues¶

Container Does Not Respond to Follow-Up Messages¶

Symptom: A steering message sent during a squad run does not reach an agent that runs in a container.

Cause: This is expected. One container runs one task: the host sends no follow-up messages, and the bundled aigo-agent-runner image exits as soon as it writes its result. See Agent Runner Protocol.

What to do:

• Put the change in a new task or a new request to the squad.
• Run the agent in-process when it needs steering during a task.

Container Sends Messages But They Are Not Delivered¶

Symptom: Containers write to ipc/messages/ but messages do not appear in the channel.

Solutions:

• Verify the channel is connected: its card under Channels in Settings > Squad shows Connected
• Check the message router status in the application logs
• Verify the chat JID in the message matches a connected channel's JID format

Channel Connection Issues¶

Telegram Bot Does Not Respond¶

Solutions:

1. Check that the Telegram Bot card under Channels in Settings > Squad shows Connected. The form checks only the token's format, so if you are not sure the token is current, get it again from @BotFather and reconnect.

2. Check the connection status:

   curl http://localhost:8001/api/v1/channels/telegram
   

3. In group chats, ensure the bot is a member of the group and has message reading permissions.

4. Test with a direct message to the bot first.

Slack Bot Not Receiving Messages¶

Solutions:

1. Verify both tokens are valid (bot token xoxb-... and app token xapp-...)

2. Verify Socket Mode is enabled in the Slack app configuration

3. Check that the required bot scopes are granted:

   • app_mentions:read, channels:history, chat:write, groups:history, im:history, im:read, im:write

4. Ensure the bot is installed to your workspace and added to the channels you want it to monitor

Discord Bot Not Responding in Channels¶

Solutions:

1. Verify the Message Content Intent is enabled in the Discord Developer Portal

2. Ensure the bot is invited to the server with the correct permissions

3. In text channels, the bot only responds to @mentions — verify you are mentioning the bot

4. Check the gateway connection status:

   curl http://localhost:8001/api/v1/channels/discord
   

WhatsApp Webhook Verification Fails¶

Solutions:

1. Verify the webhook URL is publicly accessible (HTTPS required):

   curl https://your-domain.com/api/v1/channels/whatsapp/webhook?hub.mode=subscribe&hub.verify_token=your-token&hub.challenge=test
   # Should return: test
   

2. Ensure the verify token in Backend.AI GO matches the one you entered in Meta's developer portal

3. Check that your HTTPS certificate is valid (self-signed certificates are not accepted by WhatsApp)

Task Scheduling Issues¶

Scheduled Tasks Do Not Run¶

Diagnosis:

curl http://localhost:8001/api/v1/container/schedules
# Check "enabled": true and "nextRun" timestamp

Solutions:

• Verify enabled is true on the container task
• Check that the container runtime is available

• Review the container task's run logs for errors:

  curl http://localhost:8001/api/v1/container/schedules/{id}/logs
  

Cron Tasks Fire at Wrong Times¶

Solutions:

• Verify your system timezone is configured correctly
• Use a cron expression validator (e.g., crontab.guru) to check your expression
• Remember that cron uses local system time, not UTC

Interval Tasks Drift Over Time¶

Solutions:

• The scheduler uses drift-prevention: next_run is computed from the scheduled time, not execution time
• Verify system clock is accurate (sync with NTP)
• Check for excessively long task execution times that might delay the scheduler

Security Event Issues¶

Unexpected "violation" Events in Audit Log¶

Symptom: The audit log shows permission violations for container operations you expect to be allowed.

Diagnosis:

curl "http://localhost:8001/api/v1/container/audit-log?severity=violation"

Solutions:

• Check if the container is in a non-main group and is trying to send messages to a different group's chat JID (this is expected behavior and the block is correct)
• If the violation is unexpected, review the group namespace configuration

Security Audit Marks Containers as Non-Compliant¶

Symptom: Periodic security audits report containers as non-compliant.

Solutions:

• Verify you have not modified the container environment variables (especially ANTHROPIC_API_KEY)
• Check that mounts have not changed since the container started
• Review the audit log details for the specific check that failed

Getting Help¶

If you cannot resolve an issue using this guide:

1. Set Logging level to Debug in Settings > Advanced

2. Collect the application logs:

   aigo log --last 500
   

3. Export the container audit log:

   curl "http://localhost:8001/api/v1/container/audit-log?limit=500" > audit.json
   

4. Report the issue with the collected logs at the Backend.AI GO GitHub repository.