2.7. Tool Calling (Function Calling)¶

Tool Calling is the feature that transforms an AI model from a passive text generator into an active problem solver. It gives the AI "hands" to interact with the digital world: your file system, the internet, and your computer's internal state.

What is Tool Calling?¶

Normally, an LLM is limited to the knowledge it was trained on. It can't know what time it is, what files are on your desktop, or what happened in the news today.

Tool Calling bridges this gap. Instead of hallucinating an answer, the model can say: "I need to use the web_search tool to find the current stock price of Apple."

Backend.AI GO intercepts this request, executes the tool securely, and feeds the result back to the model. The model then uses this real data to answer your question.

Why is it Important?¶

• Real-Time Knowledge: Access the latest news, weather, and financial data via web search.

• System Interaction: Read log files, organize folders, or check system health directly through chat.

• Accuracy: Perform precise mathematical calculations using a calculator instead of relying on the model's approximate arithmetic.

• Agentic Behavior: It is the foundation of AI Agents. By chaining multiple tool calls (Search -> Read -> Write), the AI can complete complex workflows autonomously.

Compatible Models¶

Not all models support Tool Calling. You need a model that has been fine-tuned to understand tool definitions and output structured tool requests.

• Look for the "Tool" Tag: In the Backend.AI GO Model Library, compatible models are marked with a Tool or Function Calling chip.

• Recommended Models: Gemma 4, Qwen3.6, Llama 4, Mistral Large, and (via cloud integration) GPT-5.4, Gemini 3 Pro, Claude Sonnet 4.6 are excellent choices.

Security & Permissions¶

Because tools can access your files and run code, security is paramount. Backend.AI GO implements a Risk-Based Permission System.

Risk Levels¶

Every tool is assigned a risk level:

• 🟢 Safe (No Risk): Actions that simply read public data or perform calculations.

  • Examples: Calculator, Get Current Time.
  • Behavior: Executed automatically without interrupting you.

• 🟡 Moderate (Read Access): Actions that read your personal files or access external websites.

  • Examples: Read File, Web Search, List Directory.
  • Behavior: Requires one-time approval per session. Once approved, the agent can use it freely for the rest of the conversation.

• 🔴 Critical (Write/Execute): Actions that modify data or execute code.

  • Examples: Write File, Delete File, Run Shell Command.
  • Behavior: Requires explicit approval for every single call. The AI cannot delete a file without you clicking "Approve".

Built-in Tools¶

Backend.AI GO ships 31 built-in tools across eight categories. The chat tool selector is derived from the same registry the agent runtime uses, so tool definitions and behavior are identical on both surfaces. The legacy names execute_python, list_files, and run_shell resolve to run_python, list_directory, and run_command, so saved tool selections using the old names keep working via backward-compatible aliases.

1. File System Tools¶

2. Web Tools¶

fetch_url Parameters¶

3. Utility & System Tools¶

4. Data Query Tools¶

5. Code Execution Tools¶

Python sandbox configuration¶

When the Python executable is not on your application's PATH (common for desktop apps launched from Finder or the Windows Start menu, or for pyenv/conda environments), the run_python tool will report a "Python interpreter not found" error. Set the interpreter path in Settings → Tools & Extensions → Python sandbox → Python interpreter path to point directly at the executable, for example /usr/local/bin/python3 or C:\Python312\python.exe.

A configured path is validated when code actually runs (the interpreter must report a "Python 3" version string), not at save time.

When submitted code fails (syntax error, runtime exception, non-zero exit, or timeout), the tool result carries success: false and a classified error message, and the error together with captured stderr are displayed prominently in the tool result block in the chat interface.

Code execution filesystem sandbox¶

run_python and run_command run model-supplied code in a real subprocess. The import blocklist restricts which modules the code can load, but it does not by itself restrict the open builtin or general file I/O. To prevent a tool call from reading, writing, or deleting arbitrary host files, all file access these tools perform is confined to a single sandbox folder:

• run_python runs with the sandbox folder as its working directory, and the interpreter's file primitives (open, io.open, os.open, and the path-taking os.* functions) are wrapped so any path that resolves outside the sandbox folder (after collapsing .. and following symlinks) raises PermissionError.
• run_command runs with the sandbox folder as its working directory and rejects path arguments that resolve outside it (for example cat /etc/passwd is denied).
• Temporary files created with Python's tempfile are redirected into the sandbox folder, so legitimate scratch-file usage keeps working.

Set the folder in Settings → Tools & Extensions → Python sandbox → Code execution sandbox folder. It must be an absolute path. Leave it empty to use the default folder (code-exec-sandbox under the application data directory), which is created automatically the first time a code-execution tool runs.

The same enforcement applies on every surface that runs these tools, the in-app chat, the agent runtime, and the REST POST /api/v1/tools/execute endpoint, because they all dispatch through one shared implementation.

6. Audio Tools¶

audio_transcribe is reported with available_in_headless: false by /api/v1/tools and is not executable through /api/v1/tools/execute. Use the dedicated /api/v1/audio/transcriptions endpoint instead.

audio_transcribe Parameters¶

7. Image Generation Tools¶

generate_image is reported with available_in_headless: false by /api/v1/tools and is not executable through /api/v1/tools/execute. Use the dedicated /api/v1/diffusion/generate endpoint instead.

8. Desktop Integration Tools¶

These tools require the desktop app's process context (system clipboard, native notifications, the agent memory bank) and are not available in headless server mode. The REST /api/v1/tools endpoint still lists these tools, but marks each one with available_in_headless: false, and /api/v1/tools/execute rejects them with a descriptive error. Headless clients filter on this flag instead of receiving the tools as normally executable.

9. Interactive Tools (Agent Runtime Only)¶

select_option is intentionally not offered in the one-shot chat tool selector. It requires the agent loop's interactive approval channel to resolve its result, so it cannot function on the chat or REST execution path. It remains fully available to the agent runtime.

The agent calls select_option when it needs a decision that should come from the user rather than be inferred from context. When the tool fires, the chat shows a widget with labeled buttons; clicking one resumes the run with the chosen value. Cancelling the prompt returns an empty result and the agent continues.

select_option Parameters¶

The LLM never selects the image model; the model is always resolved from Settings → Models → Default Image Model. When the configured model id starts with cloud: (a cloud Provider image model), the request is routed through continuum-router to the registered provider instead of the local sd-server pool. Local diffusion models are auto-loaded into the pool when not already in memory. When no default model is configured, the tool returns an explicit error that the model can relay verbatim, prompting you to configure it.

generate_image Parameters¶

Tool Results in Chat¶

When a tool finishes executing, the result is displayed inline in the chat conversation. You can expand the result block to see the full output, including structured data like file listings, search results, or command output. This transparent display ensures you always know what information the AI is working with.