Benchmarking¶

The Benchmark page, in the System group of the sidebar, is where you measure models: to see how fast one is on this computer, find faster settings for it, compare runs, or read raw llama-bench numbers. It has four tabs: Measure, Compare models, Advanced measurement, and History.

Tune, which turns a measurement into a model's settings, opens on Measure (see Tune). Settings > Performance lists the same models and holds Tune settings automatically, which tunes them while the computer is idle (see Settings). Both use the same measurement service and the same records, so a check run on either one shows up in both. Performance settings at the top of the page opens that tab, and a result that may be slow for chat offers Tune when the model can be tuned here (a GGUF model, or an MLX model on a Mac with Apple Silicon).

Measure¶

Measure lists every local GGUF and MLX model with one result line each, for example:

Answers 45 tok/s · First response 0.3 s · Enough for chat · October 2

A line may end with engine changed or settings changed: the engine was updated after the measurement, or you changed the model's settings since. A model never measured reads Not measured yet. Turn on ⋯ > Show provider models at the top right to list cloud models too. It is off by default because measuring one sends a paid request to the provider, and it is the same option as in Settings.

Quick check measures a model in about 15 seconds. It sends one short request (a prompt of about 300 tokens, an answer of up to 128 tokens) and records:

• Answer speed: how fast the model writes, in tokens per second.
• First response: how long it takes before the first word appears.
• Prompt processing speed: how fast it reads, which decides how long a long document takes to start answering.

A local model is measured directly at its engine, and the numbers come from the engine's own timings when it reports them (llama.cpp does). A provider model goes through the router, and the app times the streamed answer itself. Each request starts with a unique line, so a cached prompt does not make a model look faster than it is. One measurement runs at a time: while a tune, a model comparison, an advanced run, or another quick check runs, from this window or from another client of the same server, the check does not start and the button says Available once the current measurement finishes.

A local model that is not running is asked about first. Run this model and measure it? names the model to measure and, while other models run, the models running now, and asks what to do with them: Stop them for now (chosen at first: the check is accurate, and they run again when it ends) or Keep them running (they share memory with the check, so the numbers may come out lower). Start check goes ahead, and the choice is kept for the next check while the app runs. Either way the model runs on its saved settings only for the check and stops again when it ends; if a model stopped for the check does not come back, the result's notification names it. Cancel measures nothing. If the running models change while the dialog is open, it shows the new list and says they changed. A model that is still loading is not asked about: the button says A model is still loading. Try again once it has loaded. The same dialog opens from Settings > Performance, a model's detail, and the slowdown notice.

When a check finishes, or when you select a model's name, the result opens under its row: the numbers (answer speed, the same against reading speed, first response, and the first response to a three-page document), then one sentence that says whether that is enough for chat and how it compares with the model's previous quick check, such as That is enough for chat. 15% faster than the check on October 2. This may come from the llama.cpp update. A difference under 10% reads as about the same, and a likely cause is named only beside a difference: an engine update, or the settings that changed. When the speed may be slow for chat, the sentence ends with the next step: Tune can find faster settings. for a model that can be tuned here and has no applied tune, or Try a lighter model. otherwise. A result more than 10% slower than the previous one links Show difference to the two records in History. The notification center keeps a speed check entry, and its View result opens the result on this page again. A check started from Settings opens the model's page in Settings instead.

Reading the numbers¶

Answer speed is the one raw number the app shows outside the advanced view, and it is set against reading speed. How many tokens a reader takes in per second depends on the language, because the same text costs a different number of tokens in each. The app uses one value per interface language, measured on the tokenizers of three models it recommends (Gemma 4, Qwen 3.5, and Ministral 3) over Wikipedia and news text, together with published silent-reading rates:

These values are estimates. The Japanese and Chinese reading rates rest on few studies, and the page size assumes the default A4 layout of Word (Hancom Hangul for Korean). They change only the wording of a sentence, such as "about 8 times reading speed". The verdict below does not depend on them and is the same in every language.

First response is reported in seconds. Where prompt processing speed is known, the result also says how long a three-page A4 document takes before the answer starts. The page size in the table sets the length of that document.

Tune¶

Tune finds faster settings for one use with llama-bench. It opens on Measure in place of the list: from a result that may be slow for chat, from Find automatically in the Hardware tab of a model's settings, or from a tune's notification. Pick Chat, Long documents, or Agents; the screen estimates the time and names the models that stop while it measures (they run again afterward). The tune measures the current settings first, then searches in stages, fixing each step's best value before the next: the CPU/GPU split (only when the model does not fit in GPU memory), the prompt processing batch, threads (only when the CPU does part of the work), flash attention in a long chat (Chat only: answer speed with about 4,000 tokens of conversation in context, flash attention as the model launches now against on), speculative decoding (with a drafter installed, or for Chat draft-free n-gram speculation, kept only when the engine drafted and answered faster), and parallel slots (agents only). A skipped step says why. When the chat tune could only change prompt processing, which a chat turn barely waits for (every MLX model, or a GGUF model with flash attention already on, nothing to speculate with, and every layer on the GPU), the screen says there is nothing to tune for chat in place of Start tuning, and the Benchmark page suggests a lighter model instead of tuning. It takes a few minutes and stops after 10 at most. While it measures, no model can be run, and leaving the screen does not stop it; Stop asks first, changes no setting, and records the run as Stopped in the history.

The tune stops every running local model while it measures, the tuned one included, which on a server can be other people's models, so when any is running Start tuning asks first. The Stop running models dialog names them and says they run again when the tune ends, and the tune starts only from Stop and tune. If a model started or stopped in the meantime, the dialog shows the new list under The running models changed. and starts nothing until you confirm it too. Try again after a failed tune asks the same way. With nothing running, the tune starts at once, and while a model is still loading it does not start: try again once the model has loaded.

The result sets the current settings against the found ones and lists every setting that would change, in the Hardware and Context tabs' own labels. Apply writes those settings into the model and, if the model is running, runs it again (the banner says so if it does not run again); Don't apply keeps the result as a saved profile. Show raw numbers opens the run's record in the Benchmark page's history. When nothing is at least 1.1 times faster, the result says no faster settings were found, names the settings the tune checked, and changes nothing. For Chat, a change that makes a long chat's answers at least 1.1 times faster counts too, and the result then says how much faster answers get in a long chat. Measure above the title returns to the tab with the model open, where an apply leaves its banner with Undo. Each model keeps one profile per use under Profiles in its detail in Settings > Performance (see Settings). Apply and Undo are unavailable while another measurement runs, since both run the model again. A headless server has no llama-bench, so it offers the quick check only.

On a Mac with Apple Silicon, Tune works on an MLX model too, in four steps instead of seven. There is no llama-bench for MLX, so each value is measured by starting the bundled mlxcel-server with it, sending the quick check's request, and stopping the server again. The steps are the KV cache (FP16 or INT8, for Long documents only, and not when you chose a Turbo mode or per-side cache types yourself), the prefill chunk (512, 1024, or 2048 tokens, measured with a long prompt), and the decode batch (Agents only). mlxcel divides the context among the requests it decodes together, so the decode batch is tried only when the model's saved context still leaves each request 16K; otherwise the step says that a larger batch would split the context. The result, Apply, Undo, profiles, the 10-minute limit, Stop, and the question before running models stop work as for a GGUF model. Elsewhere, the tune of an MLX model says that tuning is available for GGUF models and for MLX models on Apple Silicon.

Compare models¶

Compare models answers which of your models is fastest for one use. Pick the use (Chat, Long documents, or Agents) and two to four models as chips; a fifth is refused with a line saying so. The line beside Start comparison estimates the time. Start comparison then, for each model in turn, runs it with its saved settings, gives it the quick check with the use's prompt length, and stops it again. Every local model running beforehand stops during the comparison and runs again at the end.

Because of that, Start comparison asks first when local models are running, which on a server can be other people's models. The Stop running models dialog names them and says they run again when the comparison ends, and the comparison starts only from Stop and compare. If a model started or stopped in the meantime, the dialog shows the new list under The running models changed. and starts nothing until you confirm it too. With nothing running, the comparison starts at once. While a model is still loading, the comparison does not start: start it once the model has loaded.

Because every model gets the same quick check over HTTP, GGUF, MLX, and provider models compare side by side, and the tab works on a headless server too. Provider models appear only while ⋯ > Show provider models is on, and the estimate line then adds that they incur request costs.

While it runs, a Comparing models dialog fills in model by model: a spinner on the one being measured, a check with its numbers once it is, and the reason if it fails, with the count beside the use. Stop ends it (finished rows are kept) and Close only hides the dialog; the comparison goes on, and Show progress beside the start brings the dialog back. When it ends, the page moves to the result: its title names the fastest model for the use, with its answer speed and first response large and Use as default model under them, or says the comparison was stopped. Compare other models or the back control returns to the form. ⋯ > View latest comparison at the top right opens the most recent result again later.

The ranking under it orders the models by how long one turn of the use takes: the first response plus a 128-token answer, so a model with a fast answer but a slow first response does not automatically win. Each row reads like a Measure line with the judgement for the use: Enough for chat or Slow for chat for chat, and Fast for long documents too or Slow for long documents for long documents, from the first response to a 10-page document. Agents rows carry no judgement, since a quick check does not send four requests at once. A model that failed shows Failed and the reason. A bar chart under the list shows answer speed, first response, and prompt processing per model, each on its own scale, with a longer bar always meaning faster.

Once a comparison has finished, Use as default model under the fastest model makes it a default, and every other measured row in the ranking has the same action. A chat or long-documents comparison sets the Chat default model (long documents are read in a chat), and an agents comparison sets the Agent default model: the settings under Settings > Models > Default Models, not new ones. A row whose model already is that default shows the setting's name instead of the action. Setting the chat default makes new chats start with that model, while a chat that is open keeps its own. The confirmation names the setting and the model it replaced, and Undo puts that model back, or clears the setting when there was none, along with the model new chats started with before (unless you have picked another one in a chat since). A result with a row measured on an engine build that has changed since says earlier engine after its date.

The launcher reads the result too. On its Chat tab, when the newest chat comparison finished within the last 14 days, measured at least two models, and measured none of them on an engine build that has changed since, a line under the composer names its fastest model with Fastest in a recent comparison. Use this model chooses it in the model picker the way picking it there does (a stopped local model starts running), and Close hides the line for that comparison until the app restarts. The line does not appear when the picker already holds that model, when the model is no longer installed, or while the first-run speed check is showing. A later comparison for long documents or agents does not replace it.

Stop does not ask first: each row is a 15-second check, and the rows already measured are kept. A stopped comparison says how many models it measured instead of naming the fastest, and the rows it did not reach read Not measured. While a comparison runs, a model cannot be run from elsewhere, and no other measurement can start. The status bar shows Comparing ⅔, and the notification center keeps one entry for the whole run. Each row is saved as a record of the kind Comparison, so History lists them and Show difference works on any two.

Advanced measurement¶

Advanced measurement runs llama-bench directly on a GGUF model. The result is saved to the history; no setting changes. MLX models are not offered, because llama-bench measures GGUF files only, and the form says how many MLX models it left out.

What do you want to measure? comes first, as three choices, and the form then shows only the rows that choice uses:

• Speed by length: prompt and answer lengths, conversation depth, and repetitions. It starts from a 128, 512, and 2048-token prompt with a 128-token answer.
• Hardware combinations: GPU layers, threads, flash attention, and repetitions, at a 512-token prompt and a 128-token answer. It starts from threads 1, 2, 4, and 8 against every GPU layer chip and flash attention on and off, with 2 repetitions (thread counts above this computer's cores are left out), and ends in Apply fastest settings. Its line points to Tune, which finds settings for a use instead of trying every combination.
• Custom: every row, starting from whatever the form holds.

A row a choice hides is left to llama-bench.

Each condition is a row of chips, and any number of them can be pressed. Custom at the end of a row opens a field for values the chips do not offer, separated by commas; the typed values join the pressed chips and count only while Custom is on. A row with nothing selected says what runs instead: llama-bench decides a setting, an empty prompt or answer length runs 512 or 128 tokens, and an empty depth is an empty conversation. Typing 0 as a length skips that kind of test.

A setting with more than one value becomes an axis, and llama-bench runs every combination in one process: a custom sweep. The line under the form counts the tests and estimates the time from the model's size, for example 48 tests × 3 repetitions · about 6 min. A run over 400 tests or about 30 minutes cannot start; the line then reads Too many tests. Reduce the values. in place of Run, and the server refuses such a run from the CLI as well.

The result table has one row per test, a column for each setting the rows differ in, Prompt processing (pp) and Answer generation (tg) speeds, and the spread across repetitions; the fastest row for the metric shown is marked Fastest, which for a hardware profile is the fastest combination of threads, GPU layers, and flash attention. A run on every GPU layer reads All in the GPU layers column. For a sweep, a line chart above the table plots the chosen metric over one setting (Axis), with one line per value of a second setting when there is one, named in a legend under the chart, in tok/s; Show switches between answer generation and prompt processing. Under a GGUF model's result, the What changes line names the settings of the row marked Fastest that differ from the model's settings now, in the Hardware and Context tabs' labels, and Apply fastest settings writes them to the model as its chat profile and runs the model again if it is running. Nothing asks first, because the line already says what changes; the notice that follows offers Undo, which asks before restoring the settings from before. Only the settings the run varied or set are written (GPU layers, CPU expert layers, threads, batch sizes, and flash attention, where Auto clears the saved choice), never the KV cache types, and threads are left alone when every layer runs on the GPU. When the fastest settings are already in place, the line says there is nothing to change and no button shows. A run that lasts more than a few seconds appears in the notification center and the status bar, so you can leave the page while it runs, and View result brings you back to this tab. Stop asks first, and a stopped run is recorded as Stopped. One measurement runs at a time: Run is unavailable while a tune, a model comparison, or a quick check runs, and the server refuses a run during a tune, a comparison, or a quick check too. Advanced records export from History with the kind set to Advanced.

History¶

History lists every measurement, newest first, with its model, kind (Quick check, Tuning, Comparison, or Advanced), engine build, and settings. Statuses read Completed, Stopped, or Failed. Filter by model or kind, and expand a record to see its settings, the hardware, and any raw llama-bench rows.

• Add to compare on two records, then Show difference, shows them side by side in place of the list: the engine, every setting that differs (marked changed), and the change in answer speed and first response. When the two records come from different models or different computers, the comparison says so above the table. History at the top goes back to the list.
• ⋯ > Show difference from the previous record compares a record with the same model's previous one.
• ⋯ > Use as baseline makes an older record the one the model card, the run dialog, and Settings show.
• ⋯ > Delete asks before it deletes. Deleting a baseline leaves the model without one.
• Export writes the filtered records as CSV, JSON, or Markdown.

A tune AI:GO ran on its own (Tune settings automatically) reads Automatic, and its details name the steps it measured and, for a run that stopped early, why it stopped.

Records older than the retention set in Settings > Performance (90 days by default) are deleted once a day. Baselines are kept regardless of age.

Headless server and the CLI¶

A headless server (aigo-server) has no llama-bench, so the page has no Advanced measurement tab there. Quick checks, the history, baselines, and export all work over HTTP, and the aigo CLI covers the same ground:

aigo bench quick --model qwen3-8b          # measure a model (asks to run it if it is not running)
aigo bench quick --model gemma-3-12b --load --keep-running   # run it for the check beside the others
aigo bench baselines                       # each model's baseline, and whether it is stale
aigo bench history --model qwen3-8b -n 20
aigo bench show <record-id>
aigo bench baseline <record-id>            # make a record the baseline
aigo bench export --format csv
aigo bench cleanup                         # apply the retention now
aigo bench run --model-path ~/models/qwen3-8b.gguf --model qwen3-8b \
  --prompt-tokens 512,2048 --gen-tokens 128 --flash-attention off   # desktop only
aigo bench compare-models qwen3-8b gemma-3-12b --use long-documents --stop-running
aigo bench compare-models-state --use chat   # the newest chat comparison, with engineChanged rows
aigo bench tune --model qwen3-8b --use chat --stop-running   # GGUF: desktop only; MLX: Apple Silicon; see Tune above
aigo bench profiles --model qwen3-8b
aigo bench apply --model qwen3-8b --use chat
aigo bench undo --model qwen3-8b --use chat
aigo bench apply-fastest <record-id> --metric tg --preview   # what an advanced run's fastest row would change
aigo bench apply-fastest <record-id> --metric tg             # apply it as the chat profile (undo with bench undo)
aigo bench power                               # whether the host runs on battery

On a headless server aigo bench run says that llama-bench is not part of the server build and points to aigo bench quick. aigo bench quick on a model that is not running asks on a terminal whether to run it for the check, and when other models run, whether to stop them or keep them running; --load runs it without asking, --stop-running or --keep-running decides for the others, and with neither a terminal nor the flags the command fails with the list. A REST client sends load in POST /api/v1/benchmarks/quick, with stopSessionIds or keepRunning when others run; without load the check answers PERFORMANCE_MODEL_NOT_RUNNING followed by the model's name and the running models as JSON, and a stop list that no longer matches answers PERFORMANCE_MEASUREMENT_STOPS_MODELS with the fresh one. The answer waits for the whole check, the load and the restore included. aigo bench compare-models and aigo bench tune list the running models the measurement would stop (for a tune, the tuned model too) and ask on a terminal; --stop-running stops them without asking, and with neither a terminal nor the flag the command fails with the list. A REST client starts the same way: POST /api/v1/benchmarks/model-comparison and POST /api/v1/benchmarks/tune answer PERFORMANCE_MEASUREMENT_STOPS_MODELS followed by the list as JSON, and the start succeeds once stopSessionIds names exactly the listed sessions. GET /api/v1/benchmarks/model-comparison?use=chat returns the newest comparison for one use, the read behind the launcher's line, and marks each row whose engine build changed since with engineChanged. POST /api/v1/benchmarks/sweep/apply with runId, metric (tg or pp), and preview is the same apply: the server picks the fastest row itself, answers changes (empty when nothing would change), and with preview false adds applied, the profile apply's answer with its undo. A record that is not a completed advanced run of a GGUF model is refused, and so is one whose model file changed since.

Hiding it by policy¶

An enterprise policy that lists the /benchmark page id in hiddenPages hides this page and Settings > Performance, and the server rejects the benchmark routes and commands. Listing performance in hiddenSettingsTabs hides only the Settings tab, with the model card line, the Overview group, the run dialog line, and the first-run check that belong to it; this page stays. See Policy Schema Reference.