Data Hub¶

The Data Hub is the document corpus behind the desktop Data page: an immutable original, a converted Markdown body, and an annotation card for every document, plus the collections, tags, retrieval presets, ingest pipelines, and generated wiki built on top of them. The Management API exposes all of it: 66 endpoints under /data, with full Tauri parity.

Every endpoint on this page is reachable at the Management API base URL, http://127.0.0.1:8001/api/v1 by default, and appears in Swagger UI at /api/docs under the Data Documents, Data Organization, Data Retrieval, Data Pipelines, Data Maintenance, and Data Wiki tags. The aigo data command group wraps the same endpoints; the flags are in the CLI reference.

How this surface behaves headless¶

Three things are worth knowing before you build against it.

The whole corpus works headless. aigo-server initializes the Data Hub with the same shared event emitter the desktop app installs, so ingestion, conversion, search, retrieval, folder imports, watch scans, embeddings, maintenance, and the wiki all run on a headless server, and the five data:* events below reach the SSE stream rather than only the desktop WebView. There is no desktop-only endpoint in this family.

Paths are the API host's paths. POST /data/folder-import, POST /data/folder-import-runs, and POST /data/watch-folders take a path that the server resolves on its own filesystem. On a remote aigo-server that is the server's disk, not the caller's, and a path that exists on your workstation usually does not exist there. restrictToPermittedFolders narrows the root to a cowork permitted folder when the caller wants that check; it defaults to false, because choosing a folder in a desktop dialog is itself the consent act and requiring pre-registration would make a first import impossible. It is a FolderImportRequest field, so it reaches only the two folder-import routes. CreateWatchFolderRequest has no such field and does not deny unknown ones, so sending it to POST /data/watch-folders is dropped rather than refused.

Model-backed operations report unavailability as state, not as failure. Summary drafting, wiki drafting, embedding runs, and the hybrid re-rank all need a model through the inference router. When none is available the request still succeeds: POST /data/documents/{id}/generate-summary answers 202 with an already-failed job carrying the reason, GET /data/wiki/status reports hasModel: false, and GET /data/search answers from the lexical index alone. A client renders those; it does not retry them as transport errors.

Authentication and scopes¶

Requests authenticate the same way as the rest of the Management API: an X-API-Key header, an Authorization: Bearer token, or the aigo_session cookie obtained from POST /api/v1/auth/login.

Access keys carry scopes, and this surface uses two:

• data_read for every GET.
• data_write for every POST, PUT, and DELETE, with two exceptions.

The two exceptions are POST routes that mutate nothing and are deliberately data_read. POST /data/grounding-context builds a retrieval block, exactly as GET /memory/context does, and its lastReferencedAt stamp is a usage counter on the read path rather than a content change. POST /data/citations/export renders card metadata into a bibliography. Both are POST only because the payload (a whole user message, or a document id list) does not belong in a URL. The authoritative table is ROUTE_MANIFEST in src-tauri/crates/aigo-rest/src/route_scope.rs; the scope column in the tables below is copied from it.

The scoped Data Hub stream has SSE and WebSocket transports. GET /data/events and GET /data/ws carry only data:* and require data_read; the global /events pair carries every domain and requires admin. A desktop app that also runs the embedded management API carries both, since issue #5040: every desktop subsystem emits through one shared sink, and that server attaches its event bus as a second surface while it is listening, so the desktop UI and the stream receive the same events.

In a managed install, an administrator can hide the Data page with the /data page id in features.hiddenPages. That gate covers the /data, /memory, /text-creations, and /artifact-creations route prefixes and the matching Tauri commands, so hiding the page refuses the API too. The extra three prefixes are there because the Data page owns those surfaces: hiding the separate /memory page removes that page without weakening the Data page's own gate.

Resource model¶

Document¶

A document is the triple the whole surface is built on: an immutable original as uploaded or fetched, a converted Markdown body, and an annotation card. Its id is a UUID; its handle is a short numeric label (D42) that [[handle]] links inside cards refer to and that GET /data/documents/by-handle/{handle} resolves. A document carries a kind, a source, a status, a content hash used for deduplication, its collection memberships, its tags, and the timestamps, including lastReferencedAt, which grounding stamps when it includes the document.

Body and card¶

The body (GET /data/documents/{id}/body) is the converted Markdown and nothing else. The card (GET /data/documents/{id}/card) is the annotation layer: a summary, key points, notes, and a free-form metadata object holding the bibliographic fields the citation export reads. summaryGenerated and keyPointsGenerated say whether a model wrote those fields, which is what lets a later draft fill only what a human has not edited.

Revision¶

Every card write and every body reconversion records a revision. GET /data/documents/{id}/revisions lists them newest first, and each carries a revertible flag: a body revision holds no card snapshot, so POST /data/documents/{id}/revisions/{revision_id}/revert refuses it. Reverting restores a card, never a body.

Collection and tag¶

A collection is a named grouping with a description, a live-document count, and a sensitive flag that keeps it out of implicit grounding and out of tool reads that do not name it. Deleting a collection detaches its members and deletes nothing. A tag is free-form and has no lifecycle of its own: GET /data/tags lists the tags that at least one live document still carries.

Chunk and retrieval preset¶

Chunks are the retrieval unit. Each carries an ordinal, its heading path, its text, a token count, a UTF-16 character offset into the body, and the embedding models it already has a vector for. A retrieval preset is the scope and budget retrieval runs under: a collection list (empty means every collection, not none), a mode of lexical or hybrid, a wikiPages policy of include, exclude, or prefer, a topK, a tokenBudget, and an optional embedding model. One preset is the built-in default and cannot be deleted.

Ingest job¶

Every asynchronous unit of work is an ingest job: a conversion, a summary draft, or a wiki operation, distinguished by kind (ingest, summary, wiki). A job carries a status (queued, converting, indexing, completed, failed, duplicate), fractional progress, the document it produced or operated on, an error when it failed, and a message for "finished, nothing to do". error and message are separate fields and can be set together, so a wiki run that a hard failure stopped reports both what it wrote and why it stopped.

A job carries no attempt counter. Retries are driven by POST /data/jobs/{id}/retry, which is idempotent because the immutable original is the input, and there is no field on the wire that says how many times a job has run.

Folder import run¶

A run is a durable, resumable, batched folder import. Its state is scanning, running, paused, completed, completed_with_errors, cancelled, or interrupted (the enum is snake_case on the wire), and it carries per-outcome counters (queued, completed, duplicate, failed, skipped, refused), the batch it is on, a bounded sample of failures and skipped entries, moreFiles when it stopped at a ceiling, and resumeBlocked when it hit the absolute safety ceiling that resume refuses.

Watch folder and maintenance schedule¶

A watch folder is a directory the maintenance schedule re-scans, with its own recursion and hidden-file settings, tags and collections applied to what it imports, an enabled flag, a consecutive-failure count, and a next-attempt time for backoff. The maintenance schedule holds four independent cron-driven tasks (watch scan, URL refetch, trash purge, wiki update), each with its own enabled flag, cron expression, and last-run timestamp, plus the numeric settings those tasks need and the timezone the expressions are evaluated in. The four last-run timestamps are owned by the runner and are not writable.

Wiki page¶

A wiki page is an ordinary document with source: "wiki", generated by the wiki pipeline from clusters of the corpus. GET /data/wiki/pages returns a summary per page with its type, staleness, citation count, and backlink count. Wiki pages are excluded from the ordinary document listing whenever the caller does not ask for source=wiki explicitly, which is what keeps generated pages out of the Library view.

Wire enum values¶

All snake_case on the wire, and rejected rather than coerced when misspelled.

• DocumentStatus: processing, ready, warning, failed, trashed.
• DataSource: upload, url, folder, chat, agent, wiki.
• DocumentKind: markdown, text, html, pdf, docx, spreadsheet, code, presentation, workbook, word_processing, ebook, other.
• IngestPhase: queued, converting, indexing, completed, failed, duplicate.
• DataJobKind: ingest, summary, wiki.
• Retrieval mode: lexical, hybrid. wikiPages: include, exclude, prefer.
• Citation format: bibtex, csl_json.

Endpoints¶

Paths below omit the /api/v1 prefix. {id} is the document id except where the surrounding path says otherwise.

Documents¶

DELETE /data/documents/{id} is a soft delete: it returns the trashed document, and POST /data/documents/{id}/restore brings it back. Permanent deletion is POST /data/trash/purge and nothing else. PUT /data/documents/{id} replaces the tag set and the collection set wholesale, so name every value the document should keep; POST /data/documents/bulk-organize is the delta form.

PUT /data/documents/{id}/card accepts an optional expectedUpdatedAt. When present, the write is refused unless the stored card still carries that timestamp, so a concurrent edit is reported rather than overwritten. The refusal is a 400 whose message contains DATA_CARD_CONFLICT; match on that token rather than on the status, which the endpoint also uses for ordinary validation failures.

Collections and tags¶

POST /data/collections/{id}/generate-summaries drafts one bounded round of the collection's documents that have no summary. The response carries nextOffset; feed it back through startOffset to drain a collection larger than the scan window. That bound is also what keeps a document whose draft just failed from being picked up again immediately.

Retrieval, embeddings, grounding, and citations¶

A presetId that names no stored preset is a 404 on GET /data/search, POST /data/embeddings/run, and GET /data/embeddings/status. Resolution never falls back to the built-in preset for an unknown id, because a preset's collection list is a privacy boundary and quietly substituting a different scope would widen retrieval past what the caller asked for.

Embedding progress is both an event and a poll: data:embedding-progress carries the same EmbeddingRunStatus the status endpoint returns, so a client that misses an event and polls sees an identical value. Cancelling keeps the vectors already written, and a later run resumes from them.

Ingest jobs, URL ingest, and folder import¶

POST /data/documents and POST /data/jobs take the same IngestDocumentRequest; the first answers with the document once it exists, the second answers with the queued job. POST /data/ingest-url accepts http:// and https:// only, caps the URL at 4096 characters, and passes the fetch through the shared SSRF guard.

The two folder-import forms are not interchangeable. POST /data/folder-import walks the whole folder inside the request and returns every skipped entry, leaving no record behind. POST /data/folder-import-runs starts a durable batched run you can list, poll, cancel, and resume. A run that stopped at a file ceiling reports moreFiles: true and continues where it stopped on resume; one that stopped at the absolute safety ceiling reports resumeBlocked: true, and resume refuses it in favour of a new import.

Maintenance, watch folders, trash, doctor, and metrics¶

A watch folder's path is not patchable. A different folder is a different watch, and rewriting the path in place would re-key the already-imported set that a scan resumes from. DELETE /data/watch-folders/{id} and DELETE /data/presets/{id} both answer { "deleted": <boolean> } rather than 404ing on an id that does not exist, so that boolean is the only signal that nothing was removed.

POST /data/trash/purge is the only permanent deletion in this surface. It removes the original, the body, the card, the chunks, the vectors, the revisions, and the links, and its PurgeTrashResult counts each of those. One call purges at most 500 documents and sets truncated when more were eligible, so emptying a large trash takes several calls even with purgeAll. Its three scopes are resolved by precedence rather than refused as a combination, so send exactly one of documentIds, purgeAll, and retentionDays.

POST /data/doctor reports without repairs and repairs with them. The repair names are reindex_search, prune_orphan_rows, quarantine_orphan_files, fail_stuck_jobs, complete_interrupted_purges, resync_frontmatter, and backfill_wiki_meta. Two of them delete data: prune_orphan_rows removes derived rows whose owner is gone, and complete_interrupted_purges finishes purges already under way.

GET /data/metrics reads process-local counters and is development-build-only. A release build answers with structurally zero counts and a disabled recording flag rather than pretending to have measured anything.

Wiki¶

Every mutating wiki verb queues a job and answers with it, because drafting a page is a model call and a corpus build on a local model is minutes of the machine's attention. Asking for an operation that is already running for the same scope is not an error: the response is the job already in flight, so a retry is idempotent rather than a second build. All four bodies are optional, and the two request types are not interchangeable: build and continue take WikiBuildRequest, which has collectionId, while update and refresh take WikiRefreshRequest, which does not. build redrafts every planned page in a scope, update redrafts only the pages whose cited sources changed, and continue drafts the planned clusters the last run left without a page. GET /data/wiki/status is what says whether a model is available at all, so a build that would produce nothing is visible before it is started.

Event stream¶

See Events.

What the request types do and do not accept¶

Several of these are places where the obvious request field does not exist. A client that sends it gets no error, because most of these types do not reject unknown fields; the value is simply dropped. Check this section before assuming a filter is honoured.

Bulk selection is one of three kinds, and two endpoints accept only the first. selection is a tagged union: {"kind": "document_ids", "documentIds": [...]}, {"kind": "all_live"}, or {"kind": "matching_query", "query": {...}}. POST /data/documents/bulk-trash accepts all three. POST /data/documents/bulk-restore and POST /data/documents/bulk-organize refuse all_live and matching_query by name, with an error saying which kind was rejected, so restore and organize take an explicit id list and nothing else. Restore rebuilds each document's search projection from its card and body on disk, which is why it has no unbounded form; organize applies per-document deltas, which a corpus-wide selection cannot express usefully.

The selection query is not the listing query. DocumentSelectionQuery is deny_unknown_fields and carries only status, collectionId, tag, and includeTrashed. It has no limit, no offset, and no source. Posting a listing query verbatim is therefore rejected loudly rather than having its paging dropped in silence, which would have acted on a much larger set than the caller wrote down. The missing source is deliberate too: resolution always applies the Library's wiki exclusion, so a query-scoped selection cannot reach a generated wiki page however it is spelled.

Grounding takes a preset, not a filter. POST /data/grounding-context deserializes GroundingContextRequest, whose fields are query, presetId, memoryBudgetTokens, groundingFraction, and recordReference. It is not a retrieval request: there is no collectionId, no tag, and no limit, and the type does not reject unknown fields, so a caller that sends them has them dropped without a word. Put the collection scope on the preset instead. memoryBudgetTokens is what the caller would spend on memory injection this turn, and the server subtracts it from the joint budget; recordReference: false builds a preview without stamping the documents as referenced.

Citation export ignores collectionId whenever documentIds is non-empty. ExportCitationsRequest carries both, but the resolver returns the named documents directly and never reaches the collection-scoped listing. Sending both is not an error and produces the id list alone, so treat the two as mutually exclusive. With documentIds empty the export covers every live document, optionally narrowed by collectionId, and stops at 2000 documents with truncated: true.

Folder import clamps rather than refuses, and the durable run ignores maxFiles outright. maxDepth is clamped to at most 24 on both folder-import routes. maxFiles is clamped into the range 1 to 500 on POST /data/folder-import only: a request asking for maxFiles: 0 gets 1 and one asking for 10000 gets 500, in both cases with a success response that does not say the value was changed. POST /data/folder-import-runs does not clamp it, it replaces it: the run assigns its own batch size before the request is persisted and again on every resume, so a maxFiles sent there is not a ceiling on the import at all and the run continues in batches until the folder is exhausted or something cancels it. Read the run's counters to see what actually happened. recursive defaults to true on the wire; includeHidden defaults to false, and the standard ignored-directory list still applies when it is set.

Two refusals carry a stable token in a 400. Anything that claims the shared lifecycle refuses while a document ingest, restore, folder import, or watch scan is running, and the message contains DATA_BULK_LIFECYCLE_BUSY. Every endpoint that ingests, converts, imports, scans, drafts, or otherwise moves a document through its lifecycle takes that same claim, so the set is wider than the three bulk endpoints: it also covers POST /data/documents, POST /data/jobs, POST /data/ingest-url, POST /data/documents/{id}/refetch, POST /data/backfill, POST /data/folder-import, POST /data/folder-import-runs and its resume, all four wiki verbs rather than refresh alone, DELETE /data/documents/{id}, POST /data/documents/{id}/restore, /reconvert, /generate-summary, POST /data/collections/{id}/generate-summaries, POST /data/jobs/{id}/retry, POST /data/folder-import-runs/{id}/cancel, and POST /data/watch-folders/scan. Branch on the token, not on a list of paths. A stale card write contains DATA_CARD_CONFLICT. Neither is a 409: this surface returns 400 for both, so match the token, not the status. GET /data/document-counts reports allLiveTrashBlocked and nonTerminalJobCount so a client can tell in advance that a corpus-wide trash would be refused.

Three endpoints take less than they look like they should. GET /data/jobs has no query extractor at all: no filters, no paging, at most 200 rows, newest first, with no truncation flag, so a full page means "at least this many". POST /data/backfill takes no body: it queues a re-ingest for every document whose body is missing or failed, at most 100 jobs per call, and is resumable, so re-running it after those finish continues where it stopped. POST /data/wiki/pages/{id}/refresh and POST /data/wiki/update both take a WikiRefreshRequest of language and model only, with no scope field: for refresh the page id in the path is the whole scope, and update is always corpus-wide. A collectionId sent to either is dropped without a word, because only WikiBuildRequest has one.

Events¶

Five events, all delivered through the runtime-neutral event emitter, so the desktop app receives them over Tauri and a headless client receives the identical payload over SSE.

GET /data/events and GET /data/ws are the scoped stream's SSE and WebSocket transports. They require data_read, the same scope as GET /data/documents, and carry only data:*: nothing from memory, the squads, or the scheduler crosses onto them. GET /events still carries the same five events among everything else and still requires admin, so a key that only reads the Data Hub should use a scoped route.

?types= narrows to a comma-separated subset, for example ?types=data:ingest-progress,data:documents-changed. A name outside the data: family is a 400 rather than a connection that silently delivers nothing. It honours Last-Event-ID and ?since= and reports a truncated replay with stream:gap and a lagging consumer with stream:lagged, exactly as GET /events does; replayed events pass the same domain filter the live stream applies. See Event Streams.

curl -N -H "X-API-Key: $AIGO_API_KEY" \
  "http://127.0.0.1:8001/api/v1/data/events"

This stream works on the desktop app's own embedded Management API too. Every desktop subsystem emits through one shared event sink, and that server attaches its own event bus as a second surface when it starts and releases it when it stops (issue #5040), so a restart moves delivery to the new server's bus and the desktop UI keeps receiving throughout. Before that fix this stream stayed open and silent there, and so did GET /events for these events.

IngestProgressEvent carries documentId, jobId, folderImportRunId, kind, phase, progress, fileName, and an optional message, plus error on a failed phase and duplicateOfDocumentId on a duplicate one. folderImportRunId is the discriminator that keeps bulk per-file work out of an ordinary upload progress display.

Every event has a polling equivalent, and the payloads match the endpoint responses field for field, so a client that cannot hold an SSE connection loses timeliness and nothing else.

Walkthroughs¶

Each assumes AIGO=http://127.0.0.1:8001/api/v1 and an access key in KEY.

Ingest a file¶

Send the text inline for a text format, or contentBase64 for a binary one such as PDF or DOCX. filename is what the server infers the kind and the original extension from.

curl -sS -X POST "$AIGO/data/documents" \
  -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{
        "title": "Routing notes",
        "content": "# Routing notes\n\nUpstream selection is ...",
        "filename": "routing-notes.md",
        "source": "upload",
        "tags": ["routing"]
      }'

The response is the created Document (201). Draft its card with a model, then read the card back once the job completes:

DOC=$(curl -sS "$AIGO/data/documents?limit=1" -H "X-API-Key: $KEY" | jq -r '.[0].id')
curl -sS -X POST "$AIGO/data/documents/$DOC/generate-summary" \
  -H "X-API-Key: $KEY" -H 'Content-Type: application/json' -d '{}'
curl -sS "$AIGO/data/documents/$DOC/card" -H "X-API-Key: $KEY"

A 202 with a job whose status is already failed means no model was available. That is a state to render, not a request to retry.

Search and ground a query¶

Lexical search first, then the grounding block a chat turn would inject for the same message:

curl -sS -G "$AIGO/data/search" -H "X-API-Key: $KEY" \
  --data-urlencode 'query=upstream selection' --data-urlencode 'limit=5'

curl -sS -X POST "$AIGO/data/grounding-context" \
  -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{"query": "how does upstream selection work?", "recordReference": false}'

GroundingContext.content is the block itself; sources, budget, mode, and degraded say what went into it and why it may be smaller than asked for. To scope it to a collection, create a preset and pass presetId: the request type has no collection field, and one sent there is dropped without an error.

Run a folder import and poll it¶

The path is resolved on the API host:

RUN=$(curl -sS -X POST "$AIGO/data/folder-import-runs" \
  -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{"path": "/srv/corpus/handbook", "recursive": true, "tags": ["handbook"]}' \
  | jq -r '.id')

until curl -sS "$AIGO/data/folder-import-runs/$RUN" -H "X-API-Key: $KEY" \
  | jq -e '.state as $s | ["completed","completed_with_errors","cancelled","paused","interrupted"] | index($s)' >/dev/null; do
  sleep 2
done
curl -sS "$AIGO/data/folder-import-runs/$RUN" -H "X-API-Key: $KEY" | jq '{state, queuedCount, completedCount, failedCount, moreFiles, resumeBlocked}'

The loop above waits for a state the run will not leave on its own, which includes paused and interrupted: neither advances until something calls resume, so a loop that waited only for the three finished states would never return. moreFiles: true means the run stopped at its file ceiling; POST /data/folder-import-runs/{id}/resume continues it. resumeBlocked: true means it stopped at the absolute safety ceiling and a new import is the way forward. aigo data folder-import show <RUN_ID> --follow does this loop for you and exits 3 when the run ends in a failed or stopped state.

Build the wiki¶

Check for a model before starting anything, since a build without one produces no pages:

curl -sS "$AIGO/data/wiki/status" -H "X-API-Key: $KEY" | jq '{hasModel, pageCount, unwrittenClusters, activeJobId}'

curl -sS -X POST "$AIGO/data/wiki/build" \
  -H "X-API-Key: $KEY" -H 'Content-Type: application/json' -d '{}'

An empty body builds over the whole corpus, which excludes sensitive collections; pass collectionId to narrow it. The response is the queued job, and GET /data/wiki/status reports activeJobId while it runs. When a build stops with clusters still unwritten, POST /data/wiki/continue drafts them without redoing the pages that already exist.

Limits¶

Enforced server-side, and the same on both transports.

• Request body: 68 MiB for the whole /data router, with a 50 MiB ceiling on one ingested file.
• Explicit bulk selection: 500 document ids per request. Per-document failure samples in the result: 25.
• Document listing page: 1000 rows.
• Job listing: 200 rows, newest first, no paging. Folder-import run listing: limit clamped to the range 1 to 50, defaulting to 50.
• Search: limit clamped to 100 hits.
• Backfill: 100 jobs queued per call, resumable.
• Folder import: depth at most 24, at most 500 files per run, at most 50000 entries walked per scan.
• Citation export: 2000 documents, with truncated: true past that. Trash purge: 500 documents per call, with truncated past that.
• URL ingest: 4096-character URL, http and https only, behind the shared SSRF guard.

See also¶

• Data Hub: the desktop page these endpoints sit behind.
• Data Wiki: what the wiki pipeline generates and how it decides what to write.
• CLI reference: the aigo data command group, which covers all 65 endpoints on this page.