2.3. Running Models & Your First Chat¶

Once you have downloaded a model, it's time to put it to work. Backend.AI GO provides a modern chat interface that feels familiar and responsive.

Loading a Model¶

Before you can chat, the model must be "loaded" from your disk into your computer's memory (RAM or VRAM).

1. Navigate to the Models tab.
2. Your downloaded models will appear here as cards.
3. Click the Load button on the model you want to use.
4. Advanced Settings: Before loading, you can click the settings icon on the model card to adjust parameters like Context Length and GPU Offloading.
   • For a complete guide on all available options, see Model Settings & Parameters.
5. Watch the progress bar. Once it turns green and says "Loaded," you are ready!

Model Structure¶

Each model card includes a Model Structure viewer that provides a detailed breakdown of the model's internal architecture. Click the structure icon on a model card to open the modal.

Overview¶

The overview section displays:

• Model Overview: Architecture type (e.g., GEMMA3N), tensor count, and number of layers.
• Quantization: Compression method (e.g., Q4KM), compression ratio, original vs. quantized bit depth, and quality-vs-size trade-off visualization.
• Dimensions: Embedding size, vocabulary size, and context length shown as proportional bars.

Model Flow & Layer Stack¶

The model flow section visualizes:

• Model Flow: The data pipeline from Input Tokens through Embedding, Transformer layers, Output, and Vocabulary.
• Layer Stack: The layer hierarchy including Input Embedding, individual transformer layers, and Output Head.
• KV Cache: Context capacity, estimated KV cache size, and memory estimation details (per-layer size, head dimensions, precision).

Transformer Layer Details¶

Clicking on a transformer layer reveals:

• Multi-Head Attention: Number of Q/V heads, KV heads, head dimension, and GQA ratio.
• Grouped Query Attention (GQA): A visual diagram showing how query heads are grouped and share KV heads, helping you understand the model's attention efficiency.

Position Encoding & Normalization¶

This section shows the model's positional encoding and normalization parameters:

• Position Encoding (RoPE): Explains how Rotary Position Embedding works, with a position-based rotation visualization and frequency spectrum display.
• Normalization: RMS epsilon value used for layer normalization.

The Chat Interface¶

Click the Chat icon in the sidebar to enter the main interface.

Creating Conversations¶

• New Chat: Click the "+" button in the sidebar to start a fresh conversation.
• History: Your previous chats are automatically saved in the sidebar for easy access.
• Search: Use the search bar in the sidebar to find past conversations by keyword.

Interaction Features¶

• Markdown Support: The model can format responses with bold text, lists, and tables.
• Code Highlighting: Programming code in responses is beautifully highlighted with a "Copy" button.
• LaTeX Support: Mathematical formulas are rendered cleanly.
• Thinking Blocks: Some models (like DeepSeek or specialized reasoning models) can show their internal "thinking" process. Backend.AI GO displays these in a dedicated collapsible block.

Understanding Chat Parameters¶

In the chat interface, you can find a "Parameters" drawer (usually a gear icon on the top right) to fine-tune the model's behavior:

• Temperature: Controls "creativity." Lower (0.1) is more focused and predictable; higher (0.8+) is more creative and random.
• Top P: Another way to control randomness.
• Repeat Penalty: Prevents the model from getting stuck in a loop.
• System Prompt: Give the model a "personality" or specific instructions (e.g., "You are a helpful coding assistant" or "Speak like a pirate").

Model Status in Header¶

When a model is loaded, the header displays a Model Status Pill showing:

• Model Name: The display name of the currently loaded model
• Memory Usage: How much RAM/VRAM the model is using (e.g., "2.3 GB")
• Context Usage: A visual bar showing context token usage (e.g., "0/8K")

Quick Actions Popover¶

Click the status pill to open a detailed popover with:

• Full Model Path: Where the model file is located on disk
• Memory Details: Memory usage with a progress bar relative to system total
• Context Details: Token usage with percentage
• Load Time: When the model was loaded (relative time like "2 hours ago")
• Uptime: How long the model has been running
• Unload Model: Quickly free resources without navigating to the Models tab
• Model Settings: Jump directly to model configuration

This provides a convenient way to monitor and manage your loaded model from anywhere in the application.

Unloading Models¶

When you are finished, or want to switch to a different model:

1. Go back to the Models tab.
2. Click Unload.
3. Alternatively, click the Model Status in the header and select Unload Model from the popover.

This frees up your system RAM/VRAM for other tasks.

Batch Operations¶

When you have many models, Backend.AI GO provides batch operations to manage multiple models at once.

Entering Selection Mode¶

1. Go to the Models tab.
2. Click the Select button in the page header to enter selection mode.
3. Model cards will now show checkboxes for selection.

Selecting Models¶

• Click on a model card to toggle its selection.

• Shift+Click to select a range of models (from the last selected model to the clicked one).

• Cmd/Ctrl+Click (on macOS/Windows/Linux) to toggle individual model selection.

• Use Select all to select all visible models.

• Use Deselect all to clear your selection.

Batch Delete¶

1. Select the models you want to delete.

2. Click the Delete button in the floating action bar at the bottom.

3. A confirmation dialog will appear showing the list of models to be deleted.

4. Click Delete to confirm. A progress bar shows the deletion status.

5. If any deletions fail, an error summary is displayed.

Exiting Selection Mode¶

Click Exit selection or press Escape to leave selection mode and return to normal view.

Model Package Export and Import¶

Backend.AI GO supports a portable .baimodel package format that allows you to export and import models with all their metadata intact. This is useful for:

• Transferring models between computers
• Sharing models with colleagues
• Backing up models with their configuration

Exporting a Model¶

1. Go to the Models tab.

2. Find the model you want to export.

3. Right-click (or long-press on touch devices) to open the context menu.

4. Select Export as Package.

5. In the export dialog:

   • Review the model information and file sizes.

   • For vision models, optionally include the mmproj (multimodal projector) file.

   • Choose a save location for the .baimodel package.

6. Click Export to begin. A progress bar shows the packaging status.

The exported package contains:

• The model file(s) in their original format
• Package manifest with model metadata
• SHA256 checksums for integrity verification

Importing a Package¶

1. Go to the Models tab.

2. Click the Import Package button in the header.

3. Select the .baimodel file you want to import.

4. The import dialog shows:

   • Validation status of the package
   • Model information (name, format, size)
   • Any warnings or errors

5. Click Import to extract the package.

6. The model will be placed in your models directory and appear in the model list.

Package Features¶

• Integrity Verification: SHA256 checksums are calculated during export and verified during import to ensure data integrity.

• Security Checks: Packages are validated for path traversal attacks, symlinks, and ZIP bomb attempts.

• Progress Tracking: Both export and import operations show detailed progress including phase, speed, and estimated time remaining.

• Atomic Operations: Export uses atomic file writes to prevent partial packages on failure.