Skip to content

2.6. Tool Calling (Function Calling)

Tool Calling is the feature that transforms an AI model from a passive text generator into an active problem solver. It gives the AI "hands" to interact with the digital worldβ€”your file system, the internet, and your computer's internal state.

What is Tool Calling?

Normally, an LLM is limited to the knowledge it was trained on. It can't know what time it is, what files are on your desktop, or what happened in the news today.

Tool Calling bridges this gap. Instead of hallucinating an answer, the model can say: "I need to use the web_search tool to find the current stock price of Apple."

Backend.AI GO intercepts this request, executes the tool securely, and feeds the result back to the model. The model then uses this real data to answer your question.

Why is it Important?

  • Real-Time Knowledge: Access the latest news, weather, and financial data via web search.

  • System Interaction: Read log files, organize folders, or check system health directly through chat.

  • Accuracy: Perform precise mathematical calculations using a calculator instead of relying on the model's approximate arithmetic.

  • Agentic Behavior: It is the foundation of AI Agents. By chaining multiple tool calls (Search -> Read -> Write), the AI can complete complex workflows autonomously.

Compatible Models

Not all models support Tool Calling. You need a model that has been fine-tuned to understand tool definitions and output structured tool requests.

  • Look for the "Tool" Tag: In the Backend.AI GO Model Library, compatible models are marked with a Tool or Function Calling chip.

  • Recommended Models: Gemma 3, Qwen3, Llama 3.1, Mistral Large, and (via cloud integration) GPT-5.2, Gemini 3 Pro, Claude Sonnet 4.5 are excellent choices.

Security & Permissions

Because tools can access your files and run code, security is paramount. Backend.AI GO implements a robust Risk-Based Permission System.

Risk Levels

Every tool is assigned a risk level:

  • 🟒 Safe (No Risk): Actions that simply read public data or perform calculations.

    • Examples: Calculator, Get Current Time.
    • Behavior: Executed automatically without interrupting you.

  • 🟑 Moderate (Read Access): Actions that read your personal files or access external websites.

    • Examples: Read File, Web Search, List Directory.
    • Behavior: Requires one-time approval per session. Once approved, the agent can use it freely for the rest of the conversation.

  • πŸ”΄ Critical (Write/Execute): Actions that modify data or execute code.

    • Examples: Write File, Delete File, Run Shell Command.
    • Behavior: Requires explicit approval for every single call. The AI cannot delete a file without you clicking "Approve".

Built-in Tools

Tools panel in chat Tools panel in chat

Backend.AI GO comes with 19 built-in tools across six categories.

1. File System Tools

Tool Description Risk Level
Read File (read_file) Read the contents of a file at the specified path. 🟒 Low
Write File (write_file) Write content to a file at the specified path. 🟑 Medium
List Directory (list_directory) List files and directories at the specified path. 🟒 Low
Create Directory (create_directory) Create a new directory at the specified path. 🟑 Medium
Delete File (delete_file) Delete a file at the specified path. Requires approval for every call. πŸ”΄ High
Move File (move_file) Move or rename a file from one location to another. Requires approval for every call. πŸ”΄ High
Search Files (search_files) Search for files matching a glob pattern (e.g., *.pdf, **/*.ts). 🟒 Low
Search Content (search_content) Search file contents using regex patterns with recursive directory traversal. Automatically skips binary files and large files (>10 MB). 🟒 Low

2. Web Tools

Tool Description Risk Level
Web Search (web_search) Search the web using Brave Search or Google Search (via Serper). Returns relevant results with titles, URLs, and snippets. Requires an API key configured in Settings. 🟒 Low
Fetch URL (fetch_url) Fetch content from a URL. HTML pages are automatically converted to Markdown for efficient context usage. 🟑 Medium

3. Utility & System Tools

Tool Description Risk Level
Calculator (calculator) Evaluate mathematical expressions. Supports basic operations (+, -, *, /, ^, %), parentheses, and functions (sqrt, sin, cos, log, exp, etc.). 🟒 Safe
Get Current Time (get_current_time) Get the current date and time. Supports ISO 8601, Unix timestamp, human-readable, and custom strftime formats. 🟒 Safe
System Info (get_system_info) Get system information including CPU, memory, disk, and OS details. Filter by category for specific data. 🟒 Safe

4. Data Query Tools

Tool Description Risk Level
JSON Query (json_query) Query JSON data using JSONPath syntax. Supports path access, array operations, and filtering. Can read from a string or a file. 🟒 Low
CSV Reader (csv_reader) Read and analyze CSV files with configurable parsing options. Supports column selection, custom delimiters, and row limiting (up to 10,000 rows). 🟒 Low

5. Code Execution Tools

Tool Description Risk Level
Run Python (run_python) Execute Python code and return the output. πŸ”΄ Critical
Run Shell (run_shell) Execute a shell command and return the output. Dangerous commands are blocked by the built-in shell security validator. πŸ”΄ Critical

6. Audio Tools

Tool Description Risk Level
Audio Transcribe (audio_transcribe) Transcribe audio files to text using Whisper speech-to-text. Supports MP3, WAV, M4A, FLAC, OGG, WebM, MP4, MPEG, MPGA, OGA, and Opus formats. Language can be auto-detected or specified explicitly (e.g., en, ko, ja). Requires a running whisper-server with a loaded Whisper model. 🟑 Medium

audio_transcribe Parameters

Parameter Type Required Description
path string Yes Path to the audio file to transcribe
language string No Language code (e.g., en, ko, ja). Auto-detects if not specified.
timestamps boolean No Include segment timestamps in the output. Defaults to false.
model_size string No Whisper model size: tiny, base, small, or medium.

Tool Results in Chat

Tool result displayed in chat

When a tool finishes executing, the result is displayed inline in the chat conversation. You can expand the result block to see the full output, including structured data like file listings, search results, or command output. This transparent display ensures you always know what information the AI is working with.