The Workspace

How Kubemoot stores every tool result as its own file in the discussion’s workspace, puts the file’s description in the discussion, and posts a result as well when it is small enough.

When a tooler fetches data from a live system, the result can be large: a full namespace inventory, a Prometheus time-series dump, a JSON response carrying hundreds of rows. The NATS discussion bus and the LLM context window were not designed to carry datasets. Dumping bulk data onto the bus truncates it, floods downstream agents’ context, and degrades the quality of reasoning that follows.

Kubemoot treats this as plain mechanics, with no setting to turn on:

  1. Every tool result is written to a file, always, one file per call.
  2. The discussion carries the file’s description: the tool, its arguments, rows, bytes, content type and columns.
  3. A result small enough to read is posted in the discussion as well, whole, after its description. A larger result is only in its file. How small is computed from the context window of the models that read the discussion (Small results go in the discussion). A result is never cut: it is posted whole or not at all.
  4. Agents read files by key (workspace_query, the artifact_* tools, execute_code) and add the facts they find to the discussion. Keys are short and relative to the thread (Short keys).
  5. The synthesizer answers from the discussion only. A fact no agent added and no posted result holds is reported as unknown, and the file descriptions in the discussion let the answer say which data existed. When tools ran and the discussion holds no fact at all, the coordinator reads the stored results of the calls that succeeded (cut to the room of the discussion; a cut result is reported as cut, one that holds the discussion’s own markers is not shown) and gives them to the synthesizer; when nothing could be shown it does not call the model and says, in the asker’s terms, that the question could not be checked (No facts: read the stored results, else no model).

The remedy for a result that is too large is a tool call that asks for less (fewer fields, a filter, a limit, a shorter time range). Storing the result as a file keeps the bus and the context lean; it does not replace asking for less.

The producer pattern

Every agent with tools writes each tool call’s result as its own object in the NATS JetStream Object Store bucket kubemoot_workspace, under the thread’s key prefix:

{namespace}/{crew}/{threadId}/{agent}/{nnn}-{tool}

nnn is the agent’s call number in the thread, three digits and counting up (001-pods_list, 002-events_list). A tool name is cut to letters, digits, _, . and -. The body is the tool’s own result text, with no [tool] header: JSON where the tool returned JSON. A result over the runtime’s size cap (1,000,000 characters) is cut and ends with ...[truncated]. An error result is stored too, and flagged in its headers. Each object is announced to the materializer, so a compute sandbox finds it as a file under /kubemoot/workspace/{key}.

Call numbers keep counting after an agent restarts: the next number is found from the objects already stored, so a call never overwrites an earlier one.

A loop with no discussion thread (a direct chat to an agent) has nowhere to store files and works as a plain chat.

Object headers

The call is described by NATS object headers on the object. Names that the OpenTelemetry GenAI semantic conventions define use the convention’s name; the rest sit in the kubemoot. namespace.

HeaderValue
gen_ai.operation.nameAlways execute_tool.
gen_ai.tool.nameThe tool called.
gen_ai.tool.call.idThe model’s tool call id, or the object’s last key segment (001-pods_list) when the model sent none.
gen_ai.tool.call.argumentsThe arguments as the model sent them, cut at 1,000 characters.
gen_ai.agent.nameThe agent that made the call.
error.typetool_error, present only on an error call.
kubemoot.startedWhen the call started, RFC 3339 UTC.
kubemoot.duration_msHow long the call took.
kubemoot.errortrue or false.
kubemoot.bytesSize of the stored body.
kubemoot.rowsItems in a JSON list, or in the items list of a JSON object; absent when unknown (text).
Content-TypeWhat the stored body is, decided once when it is stored: application/json, text/csv, text/markdown (a Markdown table), application/yaml, text/plain; profile=kubectl-table, or text/plain for anything else. Every reader believes it and looks at the bytes only when it is missing (an object stored before the header existed). See Content type.
kubemoot.truncatedtrue only when the result was cut at the size cap.

Header values that are not plain ASCII

A NATS header value can hold printable ASCII only, but arguments, tool names and agent names can carry any text (é, CJK, emoji). The runtime writes a value that is plain ASCII as it is. Any other value, and any ASCII value that starts with utf8-pct:, is written as utf8-pct: followed by the text’s UTF-8 bytes, with every byte outside printable ASCII, and every %, as %XX. A value without the utf8-pct: prefix is the text itself, so an object stored before the encoding existed reads as before, and ASCII that merely looks percent-encoded (caf%C3%A9) is left alone. Every reader decodes before it uses a value: the catalog and the read tools show the original text. Percent encoding was chosen over RFC 2047 because it needs no charset or length-limited word framing, one rule decodes any value, and a value stays readable in nats object info. The Java encoder (HeaderValue) and the Go decoder (store.DecodeHeaderValue) both run test/fixtures/header-values.json.

The 1,000-character cap on gen_ai.tool.call.arguments applies before encoding, so an encoded value is longer than the cap: up to 9 characters for each character that needs three UTF-8 bytes (a CJK character), 12 for an emoji, so at worst about 9,000 to 12,000 characters, far below a NATS header limit. A line break in the arguments is kept (encoded as %0A), so pretty-printed arguments read back as written; the catalog and the call columns show a control character (an escape, a line separator) as a space. A plain value that was stored before the encoding and starts with utf8-pct: followed only by valid escapes reads as encoded, the one case where an old value is read differently.

What an agent posts: the descriptions of its files, and its small results

A contribution is what the agent states, followed by the description of each file it produced, built in code from what was stored. A raw-output tooler (discussRole: tooler) states nothing: its contribution is the descriptions alone. Any other agent with tools posts its own words first, then the descriptions after a blank line. The descriptions stay under 4,000 characters. A result small enough for the discussion follows its description whole, between [RESULT] and [END RESULT]; a larger result is only in its file, so nothing large goes through the bus. A successful call that returned nothing has an [EMPTY RESULT] block instead, which counts against the 4,000 characters.

Pod a-1 uses the most CPU, 0.9 cores (workspace_query over 001-pods_list).

[WORKSPACE agent=k8s-workloads calls=4 errors=0 prefix=k8s-workloads/]
[CALL key=k8s-workloads/001-pods_list tool=pods_list args={"namespace":"kube-system"} type=text/plain; profile=kubectl-table bytes=15210 rows=unknown error=false columns=name,ready,status,restarts,age]
[CALL key=k8s-workloads/002-nodes_list tool=nodes_list args={} type=application/json bytes=2210 rows=3 error=false fields=metadata,spec,status]
[CALL key=k8s-workloads/003-events_list tool=events_list args={} type=text/plain bytes=15 rows=unknown error=false]
[RESULT]
No events found
[END RESULT]
[CALL key=k8s-workloads/004-workspace_grep tool=workspace_grep args={"key": "k8s-nodes/001-resources_list", "pattern": "DiskPres... (args cut at 60 of 91 chars) type=text/plain bytes=0 rows=unknown error=false]
[EMPTY RESULT]
(empty: the call succeeded and returned nothing)
[END RESULT]
The results are stored under these keys; a result not shown above is only in its file. Read a key with workspace_query (SQL; the columns are listed on the call line, and workspace_describe lists every one), the artifact_* tools, or execute_code over the file /kubemoot/workspace/<key>; the key k8s-workloads/*-<tool> reads every call of one tool as one table.

The keys in the lines are relative to the thread: {agent}/{nnn}-{tool} (Short keys).

Each call line says what the stored body is, so a reader does not guess it. type= is the object’s Content-Type header (see Content type); it replaces the format= field of earlier versions, and its values are content types, not json, text-table or text. For a table, columns= lists its column names; for a JSON list or object, fields= lists the keys of its first row (the items list of a Kubernetes list). Every name is listed, up to 60, each cut at 60 characters, then ,+Nmore counts the rest; only when an agent’s descriptions near their size cap does a line list its first 12 names, and then none, before calls are dropped. workspace_describe lists every column of an artifact with its type and examples. These are the names workspace_query reads the table by, so a query can use them as written. A workspace_query that names a column the table does not have fails with no such column and the error lists the actual columns of every table in the query. The list has the shape of the columns= list (name,status,+5more, the same rule in the runtime and in workspace-access). When the name shares a word with real columns, the error also names the closest ones (closest to status_phase: pods.status), and when a table was read as plain text lines (line_no, line) it says so. A name is never turned into a column silently: a query that names a column the table lacks fails, so a wrong column is never returned.

Content type

The type of a result is decided once, when the runtime stores it, and recorded as the standard Content-Type object header. The runtime tells them apart with one rule:

ResultContent-Type
One JSON object or list (the whole text)application/json
A Markdown pipe table (a pipe row, then its separator row, in the first five lines)text/markdown
A kubectl style table (a first line of upper-case column titles separated by two or more spaces)text/plain; profile=kubectl-table
YAML that parses as a mapping, a list or several documents, of at most 1,000,000 bytes; comment lines before the first YAML line are skipped only when the text is a list of mappings or a mapping that nests (a Markdown title over a bullet list stays text), and the first YAML line must be among the first five non-blank lines, comments includedapplication/yaml
CSV (the first five lines hold the same number of fields, two or more)text/csv
Anything else, and any result cut at the size captext/plain

Every reader (workspace_query, artifact_select) believes the header. It looks at the bytes only when the header is missing or names a type it does not read, and when the bytes cannot be read as the type the header names (a JSON type over text that is not JSON). A result cut at the size cap is stored as text/plain, a reader ignores the header of an object marked kubemoot.truncated (artifact_select then reads a cut table by what it holds), and workspace_query mends the cut text and reads it again section by section.

The detection rule lives in two places because the runtime (Java) writes the header and workspace-access (Go) reads objects stored without one. Both run the cases in test/fixtures/result-types.json and 1,500 seeded random inputs in test/fixtures/result-types-generated.json (regenerate it with UPDATE_RESULT_CORPUS=1 go test ./internal/workspace in workspace-access), the way both run test/fixtures/column-names.json for column names, so they cannot drift apart. Objects stored without a Content-Type are detected from their content.

Column names

A table’s column names are SQL-safe, so a query never needs quotes: the printed title is lowercased, every run of other characters becomes one underscore, and none is left at either end. NODE SELECTOR is node_selector, UP-TO-DATE is up_to_date, CPU(cores) is cpu_cores. A name that is empty after this is col<position> (the first column is col1), a name that starts with a digit gets c_ in front (1st is c_1st), and names that collide get _2, _3 (NAME, name is name, name_2). Non-ASCII characters count as other characters.

The rule applies to kubectl tables, Markdown tables and CSV files, in workspace_query, in the columns= list on the call line, in the no such column error and in artifact_select, which also accepts the printed title and any case as a request. artifact_select finds and reads the table the way workspace_query does, so it takes a kubectl table, a Markdown table or CSV and returns CSV with the normalized headers. When a column is missing, its error lists the columns the table has. A result that is not a table (JSON, YAML, plain text) gets an error that names the tool that fits: artifact_jq for JSON, artifact_grep or artifact_rows for text. Cell values are never changed. The keys of a JSON result are not renamed: they are already identifiers in nearly every result, SQL ignores their case, and artifact_jq and the fields= list show the same keys the file holds. artifact_head, artifact_rows, artifact_grep and artifact_tail print the stored text, so a kubectl table shows its original titles there.

Arguments are cut to 60 characters. Descriptions that would pass the limit drop the arguments, then the column names, then replace the last lines with a count ([N more calls under the prefix]). Square brackets in the summarized arguments show as parentheses so a list cannot end the call line early.

A statement is cut at 24,000 characters; the descriptions and posted results after it are never cut. An agent that answers NOTHING_TO_ADD or TOOL_GAP: posts no descriptions (the files are still stored and still listed in the catalog).

An agent’s own model keeps its normal behavior while it works: a result too large for its window is shown to a tooler’s model as a short stand-in that names the call’s key (Stored as workspace artifact {key}.), and to any other agent as an error asking for a narrower call. In both cases the whole result is in the file.

A write that fails is not hidden. The call gets a line [NOT STORED tool=<tool> args=... error=... - the result could not be written to the workspace and is not available] in place of its [CALL ...] line, and the synthesizer treats that data as unknown. The result is not posted in the contribution in its place: nothing backs it, so it is not offered as data.

Small results go in the discussion

A tool result is posted after its description when it fits in whole. Both limits follow from the context window of the models that read the discussion. The window used is the smallest one among the servers that hold the agent’s model, so the reader with the least room can still take the discussion in; when no window is published, the room is a fixed 8,000 characters.

Rule32,768-token window131,072-token windowNo window known
Room for results in the discussionA quarter of the window, at 3 characters a token24,57698,3048,000
One resultAn eighth of the room3,07212,2881,000
All the results of one contributionA quarter of the room, at most 20,0006,14420,0002,000

The 20,000 cap keeps a contribution (descriptions of at most 4,000 characters plus results) inside the 24,000 characters an agent’s statement may take on the bus. The catalog uses the same room.

Results are admitted in call order. A result that is over its share, or does not fit what is left of the contribution’s, stays in its file only, and a smaller result after it can still go in. Rules that keep the posted text trustworthy:

  • A result is posted exactly as the tool returned it, or not at all. It is never cut, and a result that would have been cut at the storage cap (1,000,000 characters) is not posted.
  • A result that holds the text [END RESULT] itself and a result that could not be stored are not posted. An agent that stores nothing posts its results by the same rules (Calls of an agent that stores nothing).
  • A successful call whose result is empty (no text, or only white space) always posts the block [EMPTY RESULT], (empty: the call succeeded and returned nothing), [END RESULT], whatever room is left, so an empty result is never read as a result that is only in its file. A failed call with no text posts nothing.
  • Arguments longer than 60 characters are cut on the call line (on a character boundary) and the line says so: args={"key": "k8s-nodes/001-res... (args cut at 60 of 91 chars), where 91 is the length of the one-line form of the arguments. The stored call holds the arguments (up to 1,000 characters). The [CATALOG-CALL ...] lines cut and mark arguments the same way.
  • A failed call posts its error text between [FAILED RESULT] and [END RESULT] when it fits. That text says why the call failed; it is not data about the system and does not count as a fact.
  • A long manifest that has to drop lines drops the calls without a posted result with data (an empty result or a large one) first, so a posted result keeps the line that describes it.

The example from a Kubernetes crew: a CronJobs table of 856 bytes and an events result of 15 bytes are posted, and a Jobs table of 25,662 bytes stays in its file on a 32,768-token window (and on a 131,072-token window, where one result may take 12,288 characters). The agents that read files can still query the Jobs table; the synthesizer reads the CronJobs and the events from the discussion and reports the Jobs as unknown.

Short keys

Agents see and type keys relative to the thread: {agent}/{nnn}-{tool}, for example k8s-workloads/002-resources_list. The long key still exists in the store and in the reference envelope, but no agent has to copy a 36-character thread id, which models mistype.

The agent runtime names the thread in the _meta of each call to a tool that reads stored results (workspace_query, the artifact_* tools, execute_code), as the entry workspacePrefix ({namespace}/{crew}/{threadId}/). It is the same MCP _meta that carries the requester’s time zone and date, and the MCP gateway forwards it after checking it is three plain name parts. The tool resolves each key inside that thread:

Key givenResult
k8s-workloads/002-resources_listRead as {namespace}/{crew}/{threadId}/k8s-workloads/002-resources_list.
k8s-workloads/*-resources_listThe pattern, inside the thread. */*-pods_list is every pods_list call of every agent.
The full key of this threadRead as given.
A key of another thread, or a full key with a mistyped thread idRefused with a message that gives the short form.
A key with an empty, . or .. partRefused, so a key cannot climb out of the thread.

Without a thread in _meta (a call from outside a discussion), keys are used as given. execute_code runs in a sandbox that knows no thread, so the runtime writes the thread into each /kubemoot/workspace/{agent}/{nnn}-{tool} path in the code before the call; a path that already names a thread is left alone. Short keys are resolved for the files of the thread the call runs in only; code that builds a path to another thread’s file by hand can still read it from the sandbox’s mount.

The catalog: the complete list of files

When the review begins, the coordinator builds the catalog of the thread in code from the object headers under the thread’s key prefix (not from what any model wrote), and puts it in the review_ready message that wakes the agents that can read files, after the contributions. Nothing stored is missing from it, and the totals line shows if a line were:

[CATALOG calls=11 errors=1 bytes=198940 groups=3]
[GROUP agent=proxmox-qm tool=get_vm_rrd_data calls=9 errors=0 bytes=198000 rows=540 pattern=proxmox-qm/*-get_vm_rrd_data]
[CATALOG-CALL key=proxmox-qm/002-get_vm_rrd_data tool=get_vm_rrd_data args={"node":"pve","vmid":101} type=text/plain rows=60 bytes=22000 error=false]
...
Read one call by its key, or every call of a group by its pattern, with workspace_query (SQL) or the artifact_* tools, or with execute_code over the file /kubemoot/workspace/<key>. A pattern reads the calls as one table with the columns _call (the call's key), _args (its arguments) and _arg_<name> for an argument that differs between the calls (for example _arg_vmid).

One [GROUP ...] line per agent and tool carries the call, error and byte counts, the sum of the rows when every call reports them (unknown otherwise) and the key pattern that reads the group. An object stored before the Content-Type header existed shows type=unknown. One [CATALOG-CALL ...] line per call carries the tool, the arguments (cut at 60 characters, with the same “(args cut at N of M chars)” marker as a call line), the content type and rows from the headers, the bytes and whether it was an error. A catalog line never begins with [CALL key=, so it is not read as an agent’s file description. When the call lines do not fit a quarter of the smallest window any server publishes for the model (about 3 characters a token; 8,000 characters when no window is known), the catalog is rendered one level coarser: the totals and the group lines only, with a note that SELECT DISTINCT _call, _args over a group’s pattern lists its calls. It is never cut: no group line is dropped.

An analyst reads files by key or pattern and puts the facts it finds into its own contribution, for example “pod a-1 uses the most CPU, 0.9 cores”. Counting, summing and ranking belong to the analysts’ tools (workspace_query, execute_code), never to the synthesizer.

The synthesis: the discussion only

The tool-free synthesizer cannot read the workspace and is not given a file. Its input is:

  1. the signal counts and the original question,
  2. a few ADL rules for file descriptions (agent-runtime/src/main/resources/prompts/synthesis-files.adl), present only when a contribution describes a file,
  3. the thread: every contribution, each with the descriptions of its files and the results posted with them, with the [WORKSPACE ...] header and the closing “stored under these keys” line removed (they speak to an agent that can read files).

The rules say that a [CALL ...] line describes a file and carries no values; that a [RESULT] block is the whole result of the call above it and is data; that a count, total or list is stated as the agent stated it; that nothing is derived from rows and bytes; and that a value no agent stated and no posted result holds is reported as unknown, naming the files that existed for it (tool, arguments, rows). A [NOT STORED ...] line is unavailable data and also unknown. Because the synthesizer sees the discussion only, the cost of an answer is bounded by the room computed above, not by the size of the files.

When tools ran and the discussion holds no fact, the coordinator reads the successful stored results for the synthesizer, or answers without it when none can be read (No facts, read the stored results, else no model).

The final answer to the user never carries a file description line or a posted result block.

Per-call objects are deleted with their thread (by the thread’s key prefix), and the 48-hour bucket TTL bounds anything that is missed. See Garbage collection.

The reference envelope

Each stored file is announced on the artifact subject (kubemoot.artifacts.<namespace>.<crew>.<threadId>) with a small JSON structure so the materializer can put it on disk and a consumer can decide before opening it:

FieldPurpose
bucketObject store bucket name.
keyFull object key (namespace/crew/thread/agent/nnn-tool).
contentTypeThe stored object’s Content-Type header, for example application/json.
bytesTotal object size in bytes.
rowsRow count when known.
toolName of the MCP tool that produced the data.
agentName of the agent that ran the tool.
createdAtISO-8601 timestamp of the write.

The rows and byte counts let many consumers answer without fetching at all.

Consumer paths

Not every agent can reach the object store directly, so there are two consume paths depending on the agent’s network posture.

Network-capable agents (sip from the bus)

Toolers, analysts, and the coordinator can reach NATS directly. They read a file’s description (tool, arguments, rows, bytes, columns), decide which slice they need, and issue a targeted read through the read-ops MCP service (workspace-access in mcp mode; an MCPServer with kubemootImage: workspaceAccess runs it without naming a version, see the MCPServer guide). That service exposes output-bounded tools that extract only the slice an agent needs:

ToolWhat it returns
headFirst N lines of the object.
tailLast N lines.
grepLines matching a pattern.
selectNamed column projection for CSV objects.
countRow count (no data transfer).
rowsRows at a given offset/limit.
jqA JQ expression applied to a JSON object.

Each tool is output-bounded: it returns only the filtered slice, never the whole object, so the LLM context receives only what it asked for.

The discipline is “sip, don’t slurp”: never pull a whole blob into memory or into an LLM context window. A grep for a pod name, a select for two columns, or a jq path expression is always cheaper than a full fetch, and keeps the context tight enough for reliable reasoning.

Query with SQL

workspace_query gives every agent one query language, SQL, for reading the evidence a discussion has already gathered. Agents call tools, and every result becomes a file in the discussion’s workspace. Those files arrive in many formats: kubectl text tables, JSON (kubectl -o json, search results), YAML, Markdown tables, CSV, and the sectioned output older Kubemoot versions wrote. With workspace_query an agent counts, filters, groups, sorts and totals any of them without choosing a tool for the format.

A call works in four steps:

  1. The agent names the artifacts as table names and writes one SELECT.
  2. The read-ops service reads each artifact by its recorded content type (detecting it from the content when there is none) and turns it into a table: columns from a kubectl table’s headers, keys from JSON items, a tooler’s sections read one by one (see How each artifact becomes a table).
  3. It loads the tables into a new in-memory SQLite database, runs the read-only SELECT, and returns the rows.
  4. It discards the database.
artifacts: {"pods": "<key>"}
sql: SELECT namespace, COUNT(*) FROM pods GROUP BY namespace ORDER BY 2 DESC LIMIT 5

Why one language. Without it, an agent chooses among format-specific read tools (select reads CSV only, jq reads JSON only, and so on) or writes code for the compute sandbox. With it, the counting is done by a deterministic engine instead of the model, and models write SQL well. The read-ops tools above remain available for fetching a slice of an object.

What it does not do. It reads only artifacts already in the workspace. It does not reach the cluster, Prometheus or any other source, so it counts exactly what the toolers gathered, and incomplete data stays incomplete. It runs one SELECT, with no writes, no files outside the named artifacts and no network (Limits and safety).

What persists

ThingPersists?
The artifact in the NATS object store (kubemoot_workspace)Yes. This is the only persistence for artifacts, until garbage collection removes it.
The files under /kubemoot/workspaceNo. A local copy that workspace-access reads from. It is lost with the pod and fetched again from the store on next use (Workspace directory).
The SQLite databaseNo. It exists for one query.

SQLite stores nothing. Crew memory is separate and lives in NATS KV; see Agent Context.

Configuration

The tool is on by default, backed by SQLite. The operator chart value artifacts.queryEngine chooses the engine: sqlite (the default) and duckdb offer workspace_query; none turns it off and the service offers no query tool. A KubemootConfig written by hand that leaves out spec.artifacts.queryEngine, or the whole artifacts object, also gets sqlite. To turn the tool off, set artifacts.queryEngine: none in the chart values (or spec.artifacts.queryEngine: none in the KubemootConfig). Turning the engine on changes no CPU or memory request or limit of the workspace-access pod: an MCPServer that sets no spec.resources keeps the operator’s default MCPServer resources (see the MCPServer reference), which hold the growth the memory watch allows for one query. The only addition is the 1Gi /kubemoot/workspace volume described below. If you set spec.resources yourself, keep the memory limit above the memory setting plus the base process. See Query engines for why and when. The engine sits behind a seam: another engine can replace these through the chart value without a change to crews or prompts. The engines speak their own SQL; the tool description tells the model which one it has.

InputMeaning
artifactsA map from a table name you choose to an artifact key. Up to 8 per query. A name is letters, digits and underscores. The tool also accepts a string that holds that JSON object; any other string is rejected with the expected shape, {"<table>": "<artifact key>"}.
sqlOne SELECT (or WITH ... SELECT) over those table names.

The result is plain text a small model reads easily: a header line, one a | b line per row, then (N rows). A result longer than the row limit (100 by default) is cut and says so.

artifacts: {"pods": "default/pilot/t1/k8s/pods_list-1"}
sql: SELECT "STATUS", count(*) AS n FROM pods GROUP BY 1 ORDER BY 1

STATUS | n
Pending | 1
Running | 2
(2 rows)

How each artifact becomes a table

The format is the object’s recorded Content-Type; an object without one is detected from its content (object keys carry no file extension):

ContentTable
A JSON list of objectsOne row per object, one column per key.
A JSON object with an items list (kubectl get -o json)One row per item.
A kubectl get or kubectl top table (titles may end in a unit, CPU(cores), MEMORY(bytes))One row per line, columns named by the header. The table ends at a totals rule (a line of underscores, as kubectl top --sum prints), so the total row under it is not a row; it stays readable in workspace_tail.
YAML (several documents become a list)Read as JSON.
A Markdown pipe tableOne row per table row.
CSV with a header rowOne row per line.
Anything elseOne row per line, columns line_no and line.

The key of one call (k8s-workloads/001-pods_list, or the full default/pilot/t1/k8s-workloads/001-pods_list) is one table, and its Content-Type header decides how it is read; an object without one is detected from its content as above. A key with a * reads every stored call that matches as one table. * stands for any characters except /. A short key (k8s-workloads/*-pods_list) is read inside the current thread (Short keys); a full key must name the namespace, crew and thread (ns/crew/thread/), and the namespace must be the workspace-access pod’s own (KUBEMOOT_NAMESPACE). k8s-workloads/*-pods_list is every pods_list call of one agent in the thread; */*-pods_list is every pods_list call of every agent. Calls of the same shape are stacked as the rows of one table, a different shape is name_2, and a call that returned an error is left out and named in a note. At most 100 objects match, and together they stay inside the artifact size limit.

Every row of a table stacked from a * key carries the call that produced it, so rows can be grouped, joined and counted per call (both engines):

ColumnValue
_callThe object key of the call (the full key).
_argsThe call’s arguments as the runtime recorded them (JSON text, cut at 1,000 characters); empty when the call has none.
_arg_<name>An argument that is a plain value (text, number or flag) in every stacked call and differs between them, one column per such argument, at most 6 in name order, with the argument’s name made of letters, digits and _. For nine get_vm_rrd_data calls that differ only in vmid, that is _arg_vmid.
artifacts: {"rrd": "proxmox-qm/*-get_vm_rrd_data"}
sql: SELECT _arg_vmid, max(cpu) AS peak_cpu FROM rrd GROUP BY _arg_vmid ORDER BY peak_cpu DESC

A column the data already has under that name (compared without regard to case) is never replaced; the call column is then left out. The columns are added to a * key even when it matches one call. A key that names one call is unchanged, and so is every artifact that is not a stored tool call. An argument that is the same in every call, is missing from one of them, is a list or object, or comes from arguments cut at the header limit has no _arg_ column; _args still holds the text, and SELECT DISTINCT _call, _args FROM rrd lists the calls. artifact_count on a call whose result is a JSON list returns the list’s item count from the kubemoot.rows header.

An artifact written by an older Kubemoot version holds a tooler’s raw output as one section per tool call: a [tool_name] line, the result, then a blank line. A query reads such an artifact one section at a time, so a kubectl table or a JSON result behind a [pods_list] line is read as the table or the JSON it is. Sections of the same format and columns (the same tool called twice) are the rows of one table, the name you chose. A section with other columns is its own table, name_2, name_3 and so on, in order of appearance. When a table is made of several sections, a note after the rows names the tools behind it. An artifact whose sections are all prose is read as one text. A result the runtime cut at its size limit loses its last partial row (a JSON list keeps its complete items), and a note says so. An artifact that does not start with a [tool_name] line is read as above.

A column whose every value is a number is stored as numbers, so sum, avg and ORDER BY work on it (RESTARTS of a kubectl table, a Markdown count column). A kubectl table or Markdown table column whose every non-empty value is a Kubernetes quantity, with at least one carrying a suffix, is read as a quantity: stored as numbers in base units, so ORDER BY memory_bytes DESC and sum(memory_bytes) compare values and not text. The suffixes are binary (Ki, Mi, Gi, Ti, Pi, Ei), decimal (k, M, G, T, P, E), milli (m) and exponents (12e3); an exponent alone, like a plain number, does not make a column a quantity, and the n and u suffixes are not read, so such a column stays as it is. Binary and decimal suffixes give bytes (65Mi is 68157440); m gives cores (250m is 0.25) and is read that way only under a column name with the word cpu or core(s), because 5m in an AGE column is minutes. A column that mixes quantities with other text, or bytes with cores, stays text, and so does a column of plain numbers without a suffix (that one is already numbers). workspace_describe types a quantity column quantity, bytes or quantity, cores. The stored result is unchanged: workspace_head, workspace_grep and workspace_tail still show 65Mi. Stacked calls (k8s-metrics/*-pods_top) are read once over all their rows, so the column is a quantity only when every call agrees.

A nested object or list in JSON or YAML is read per engine. In SQLite it is a text column holding JSON: status->>'phase' reads a field, json_array_length(tags) counts a list. In DuckDB it is a struct: status.phase. A JSON true or false is 1 or 0 in SQLite and a boolean in DuckDB.

Markdown beyond pipe tables is read as lines of text. DuckDB’s Markdown reader is a community extension that downloads at run time, which a server with no network cannot do.

Limits and safety

  • Read-only. The SQL must be exactly one SELECT (or WITH or VALUES). Writes, ATTACH, PRAGMA, VACUUM, COPY, INSTALL, SET and a second statement are refused before the engine sees them. The query then runs as a sub-select of a wrapper query, so no other statement can be spelled around it. SQLite adds a connection set to query-only with ATTACH switched off; DuckDB locks its configuration.
  • Only the named artifacts. Each query gets its own in-memory database holding only the tables it names. SQLite has extension loading disabled and no file or network functions. DuckDB runs in its own short-lived process with external access off, so a path outside the named artifacts (/etc/passwd, another artifact) and any http, https or s3 address are refused and extensions cannot load or download.
  • Bounded. A query that runs past the time limit (10 seconds by default) is stopped. An artifact over 32 MiB is refused, and the result is cut at the row limit and at the tool output cap. DuckDB’s memory is capped at the memory setting (256 MB by default). The pure-Go SQLite build has no heap limit of its own, so workspace-access watches the size of its process: a query that grows it by more than the memory setting is stopped, a single value cannot exceed a quarter of the setting, and at most two SQLite queries run at once. The watch reads /proc; where that is unavailable the time limit and the value ceiling still apply. Give the pod a container memory limit as the last bound.
  • Errors say what to fix. A missing artifact, an unknown column or a refused statement comes back as a tool error naming the cause.

The service reads these variables; the chart value sets only the engine. Each one is also read under its old ARTIFACT_ name (ARTIFACT_QUERY_ENGINE, ARTIFACT_DIR, ARTIFACT_MODE, ARTIFACT_SUBJECT, ARTIFACT_BUCKET), and the WORKSPACE_ name wins. The ARTIFACT_ names keep working for one release and each logs a warning the first time it supplies a value; set the new names in new manifests. The operator sets WORKSPACE_QUERY_ENGINE on every server that selects the workspace image; a server that sets either name itself keeps its own value.

VariableDefaultMeaning
WORKSPACE_QUERY_ENGINEnonenone, sqlite or duckdb. The service itself defaults to none when run on its own; the operator sets it from the KubemootConfig spec.artifacts.queryEngine (sqlite when unset) unless the MCPServer sets it.
WORKSPACE_DUCKDB_BINduckdbThe duckdb program. The workspace-access image that serves duckdb must carry it; when it is missing the service logs the reason and offers no query tool.
WORKSPACE_QUERY_TIMEOUT10sTime limit per query.
WORKSPACE_QUERY_MAX_ROWS100Rows per result.
WORKSPACE_QUERY_MEMORY256MBDuckDB memory limit. For SQLite, the growth of the process that stops a query, and a quarter of it is the ceiling on any one value.
WORKSPACE_QUERY_MAX_ARTIFACT_BYTES33554432Largest artifact a query may read.
WORKSPACE_DIR/kubemoot/workspaceWhere artifacts are materialized; the server needs a writable volume there.

An artifact the sandbox sidecar has not materialized yet is fetched from the object store on first use.

Workspace directory

The path is /kubemoot/workspace, not /workspace: a Buildpacks image keeps its program in /workspace, and a volume mounted there would hide the binary.

When the engine is sqlite or duckdb, the operator gives the MCPServer pod an emptyDir mounted at /kubemoot/workspace with a size limit of 1Gi, and sets TMPDIR=/kubemoot/workspace, so a pod with a read-only root filesystem has a writable place for the materialized artifacts and the engine’s scratch files (the DuckDB program’s home and temporary directory). One query names at most 8 artifacts of at most 32 MiB each and DuckDB may spill up to 512 MiB, so 1Gi holds the most one query writes; the kubelet evicts a pod that exceeds it. With none nothing is added. A volume the MCPServer already mounts at /kubemoot/workspace, or a TMPDIR it sets, is kept. Raising WORKSPACE_QUERY_MAX_ARTIFACT_BYTES raises what a query writes, so declare a larger /kubemoot/workspace emptyDirVolumes entry with it.

Query engines

sqlite is the recommended engine and duckdb is the option for large data.

Why SQLite is the recommended engine.

  • It runs inside the workspace-access process. There is no extra image, no extra process to start and no program to install, and it needs a few MB of memory.
  • Its database is a new in-memory one for each query and holds nothing afterward; see What persists.
  • SQLite is public domain and written in C. Kubemoot uses a pure-Go translation of it, so workspace-access stays a single static binary on its existing image.
  • “Lite” means embedded and serverless, not limited. SQLite is the most widely deployed database engine, and it has full SQL: joins, grouping, window functions, common table expressions and JSON functions.
  • Workspace data is capped in size and is usually kilobytes. For that, both engines answer in milliseconds; with DuckDB the start of its process dominates the time.

When DuckDB fits.

  • DuckDB is a column store, built for analytical scans over large tables, and it reads JSON and CSV files directly.
  • It is a C library and a separate program, so the workspace-access image must carry the duckdb program. When duckdb is chosen and the program is missing, workspace-access logs why and offers no query tool.
  • Choose it when artifacts are large (many thousands of rows and up) and queries scan them whole.

Measured with the same kind of query on both, a group-by over a JSON list:

CaseSQLite, in the workspace-access processDuckDB, separate process
SELECT 1about 3 MB more than an idle process, 1 to 2 ms42 MiB peak, 0.07 s
50 rowsabout 3 MB more, 1 to 2 ms45 MiB peak, 0.07 s
200,000 rows, about 14 MBabout 30 MB more, 1.3 s127 MiB peak, 0.24 s

SQLite figures are the added peak resident memory of a process running the query in-process, over the same process idle (about 9 MB). DuckDB figures are the peak resident memory of its whole process. SQLite loads the file into a table before it queries, so its time grows with the size of the artifact. DuckDB scans the file as it queries and is faster on the large case.

Workspace access

An Agent declares what it may do with the workspace in spec.workspace.access; a Crew sets a default for its agents in spec.workspace.defaultAccess and a KubemootConfig sets a cluster default in workspace.defaultAccess. The closest declaration wins.

AccessRead toolsStores its resultsNATS publish on the bucket (not yet applied)
nonenono, results return inline under the budgetno
readyesnono
writenoyesyes
readWriteyesyesyes

A crew that declares nothing keeps today’s behavior: every result is stored and the read tools are present when enabledTools names them. A tooler that fetches and hands off declares write: it stores what it fetched and gets no read tools. The operator reports the resolved access in the Agent’s status.workspace, and a WorkspaceAccessResolved condition when a declared access cannot work as written.

The agent-runtime reads the declared access from KUBEMOOT_WORKSPACE_ACCESS. With none or read it stores nothing: it does not write a file for any tool call, so its calls have no key, and every tool result returns to the model inline under the tool result budget, with a large result cut, exactly as a tool loop does outside a discussion. The agent still lists its calls in its contribution (Calls of an agent that stores nothing), so the coordinator and the other agents see what it asked and what it got. A raw-output tooler with none or read has no such list: its contribution is the tool output it gathered. With write or readWrite, or when the variable is absent, it stores every result as before. A value the runtime does not recognize is logged once and treated as absent, so a typo cannot silence an agent that used to store. An agent with none or read still receives the coordinator’s catalog of stored keys in the discussion; the access limits what it stores and which tools it has, not what the coordinator posts.

Read is enforced through the agent’s tool list, not per caller, and write by the agent-runtime. NATS enforcement lands with NATS user provisioning; the NATS column above is the grant that provisioning will give, and it is not applied yet. The workspace server scopes reads by discussion, so access does not isolate two agents of one crew. The code sandbox’s /kubemoot/workspace mount, the coordinator’s catalog of stored keys, and direct MCP mode (tool names with a server. prefix) reach the workspace whatever the tool list says; see Agent.

Calls of an agent that stores nothing

An agent with read or none access lists its tool calls in its contribution the same way a storing agent does, and writes nothing to the workspace. The [WORKSPACE ...] header says stored=no in place of a prefix. Each call has one line with no key (nothing was stored to read), and its result follows when it fits the same room as a stored agent’s:

[WORKSPACE agent=obs-advisor calls=3 errors=0 stored=no]
[CALL tool=artifact_select args={"q": "events"} type=text/plain bytes=15 rows=unknown error=false stored=no]
[RESULT]
node-a NotReady
[END RESULT]
[CALL tool=workspace_grep args={"pattern": "DiskPres... (args cut at 60 of 91 chars) type=text/plain bytes=0 rows=unknown error=false stored=no]
[EMPTY RESULT]
(empty: the call succeeded and returned nothing)
[END RESULT]
[NOT STORED tool=pods_list args={} error=false - the agent does not store results and this one is not shown here, so it is not available]

A call that failed with no text has the block [FAILED RESULT] (the call failed and returned no text), so every line of such an agent has a block or a [NOT STORED ...] line, and a line without one never needs a workspace read. A result that is too large for the room has the [NOT STORED ...] line shown last above, which says it was not stored and not shown: nothing is “only in its file”, and the synthesizer treats that data as unknown. When the manifest must leave calls out, it says [N more calls not listed; this agent stores no results, so theirs are not available] instead of pointing at a prefix. The closing “The results are stored under these keys” line is not written, because no key reads anything. Readers treat the lines as follows: a posted result is a fact, like a stored agent’s; a [CALL tool=...] line shows that tools ran; no reader looks for a stored object for a line without key=. The fitness runner’s invented-name check adds the call lines and posted results of these findings to the evidence it read from the store. It does not check a run that holds the “calls not listed” line or a [NOT STORED ...] line for a raw tool (or a failed write), since a name could be in a result nobody kept; a [NOT STORED ...] line for a workspace read tool does not stop the check, because that tool read data the store evidence already holds. The shared fixture test/fixtures/unstored-manifests.json holds the manifests both the agent-runtime and the runner test.

Names

The discussion’s artifacts are the discussion’s workspace, and the names follow. Both sets work for one release; the old names are removed in the next.

OldNewState
artifact_head, artifact_tail, artifact_grepworkspace_head, workspace_tail, workspace_grepBoth are served and answer the same. The old tools say they are deprecated in their descriptions, and the first call of each logs a warning.
artifact_count, artifact_rows, artifact_select, artifact_jqworkspace_queryThe four still work. Their descriptions say to use workspace_query, and the first call of each logs a warning.
image and component artifact-access (its image ghcr.io/kubemoot/artifact-access, its release tags)workspace-accessThe workspace server is built, tagged and published as workspace-access. Name the new image wherever a crew or a registry mirror names the old one.
ARTIFACT_MODE, ARTIFACT_DIR, ARTIFACT_SUBJECT, ARTIFACT_QUERY_*, ARTIFACT_DUCKDB_BINthe same names with WORKSPACE_The service reads the new name first and the old name second.
MCPServer spec.kubemootImage: artifactAccessworkspaceAccessBoth select the same image. The server’s status carries a DeprecatedNames condition while it uses the old name.
KubemootConfig spec.images.artifactAccess, chart value kubemootConfig.images.artifactAccessspec.images.workspaceAccess, kubemootConfig.images.workspaceAccessThe chart renders only the new field. kubemootConfig.images.workspaceAccess wins; a value under the old name is the workspace image only when the new one is empty, and the install notes say when it is ignored. A KubemootConfig written by hand that sets only the old field is still read.
KubemootConfig spec.artifacts.queryEngine, chart value artifacts.queryEnginespec.workspace.queryEngine, workspace.queryEngineThe new value wins; the old one is read when the new one is not set. The chart renders only the new field.
mount /artifactsmount /kubemoot/workspaceOn a workspace server and on a pod with sidecars (the code sandbox with its materializer), a volume mounted at either path is also mounted at the other in the same containers unless another volume holds that path, so crews and sandbox code that name either path find the files. The agent runtime writes /kubemoot/workspace in the paths it hands out and reads both in code.
bucket kubemoot_discussion_artifactsbucket kubemoot_workspaceEvery writer stores into the new bucket; the agent runtime creates it with the 48-hour TTL, and nothing else creates it. Readers (the workspace server, and the agent runtime’s catalog, stored-result read and clean-up) look in the new bucket first and the old one second, skipping a bucket that does not exist, so a discussion that spans the change keeps its calls, and call numbers continue across both buckets. Data in the old bucket expires within 48 hours; nothing is copied. Start the change with no discussion running.

A crew that declares workspace access governs both sets of tool names: a tool the access does not grant is disabled under either name.

List and describe the workspace

Two read-ops tell an agent what the discussion has gathered, before it writes a query. Both work on the calling thread only: the thread comes from the call’s _meta (Short keys), a call without one is refused, a key of another thread is refused, and so is a key that does not lie under the thread once resolved (a bare *, a one-part key such as x).

workspace_list returns the catalog of the thread on request, so an agent can check it has the full picture before it counts. It is the object store’s own listing of the thread’s key prefix with each object’s headers; nothing else stores it. It is offered whenever the store can list objects.

[CATALOG calls=3 errors=1 bytes=44060]
[CATALOG-CALL key=k8s-workloads/001-pods_list tool=pods_list args={"namespace":"kube-system"} type=application/json rows=3 bytes=2210 error=false started=2026-11-15T10:00:00Z]
...

The call lines are the ones the coordinator’s catalog uses (both render test/fixtures/catalog-lines.json), with one more field, started=, the time the call started. A header an object lacks is unknown (an object stored before the Content-Type header has type=unknown). The optional tool argument lists the calls of one tool. At most 200 call lines are shown, then a line counts the rest. A key of a line reads as it is shown, short.

workspace_describe takes the same artifacts map as workspace_query (table name to key; a key may be short or end in *) and returns, for each table the artifacts become, its columns and the type the query engine gives each, so a query uses real names:

[TABLE pods rows=3 columns=4]
- ns: text; examples: "kube-system", "default"
- restarts: integer; 1 null; examples: "0", "2"
- labels: text (JSON); examples: "{\"app\":\"x\"}"
- _call: text; meaning: the key of the stored call the row came from; examples: "k8s-workloads/001-pods_list"

The data dictionary is what the data and the engine show: the type, how many values are null, and up to three example values from the first 50 rows (each cut at 40 characters). A column that holds nested JSON is marked text (JSON); SQLite reads such a column with ->>. A meaning is given only for the columns Kubemoot adds to a table of stacked calls (_call, _args, _arg_<name>); a unit only for a quantity column (see above), and no other unit and no meaning is invented for any other column. Types are the engine’s own (SQLite: integer, real, text, blob), and the result says that the types and examples are read from the data. It is offered when the engine can describe; it asks the engine for typeof and a UNION ALL, so any engine behind the Engine seam that has them can.

A column name that an earlier column already has is described once, and a line says how many were left out. If the engine reports more types for a column than were asked for, a line says so.

Each table is read twice, once for its columns and examples and once for its types, so describing a large artifact costs about two queries. Tables of a call pattern carry the same notes a query shows.

Compute sandbox agents (materialised to disk)

Counting, summing, sorting, and filtering are unreliable when a model performs them by reading a long listing, particularly for small local models; asked how many pods are running in each namespace, a model can return a confident, plausible, and wrong number. Kubemoot routes that work to a compute agent instead: another agent gathers the raw data into the artifact store described above, the compute agent writes a short Python or bash program that reads it back, and the sandbox described below runs the program and returns the number the compute agent reports. The runtime enforces the contract: a compute agent may not state a number that did not come from running code.

The compute sandbox is the code sandbox (code-sandbox in the repository), an MCP stdio server that exposes execute_code and validate_code for Python (standard library only) or bash. The pod is the sandbox: the container that runs programs holds no credentials and talks to nothing itself, and the crew’s network policy limits the pod’s egress to the message bus. A materializer sidecar (the same workspace-access binary in its default mode) is the only part of the pod that reads NATS. It fetches referenced objects from the store and writes them into a shared local volume under /kubemoot/workspace. The sandbox reads the materialised file and computes over it locally, then publishes its result through the normal discussion signal path.

Each run happens in a fresh temporary directory that is removed afterwards, with no stdin: data goes in the code itself or in a file under /kubemoot/workspace, never piped in. A time limit kills the program’s whole process group, and stdout and stderr are each capped.

The materializer runs as an init step per discussion turn, so the sandbox always sees files that match the current turn’s reference envelopes.

Garbage collection

Every object stored here is temporary. Three layers of garbage collection ensure no blobs survive their discussion.

Layer 1: lifecycle-tied deletion. When a discussion thread closes and leaves the coordinator’s in-memory state, the coordinator issues a prefix-delete for that thread’s artifact key prefix ({namespace}/{crew}/{threadId}/). All objects for the thread are removed immediately.

Layer 2: bucket TTL. The kubemoot_workspace bucket carries a 48-hour object TTL. Any object that Layer 1 misses (coordinator restart mid-thread, exceptional close paths) is reaped by the NATS server automatically.

Layer 3: orphan reaper. A periodic background job scans the bucket for objects whose thread prefix is no longer in the active-thread registry and that are older than a configurable grace window. Objects that clear both conditions are deleted. The reaper runs independently of the per-thread lifecycle so it cleans up across restarts.

The three layers together mean there are no immortal blobs and no orphaned objects accumulating in the store.

Why this design

The pattern mirrors how a CLI agent (or a human analyst) works: data lands on a filesystem, reasoning tools sip from it selectively, and the working memory holds only the conclusions. It applies the same discipline to the Kubemoot discussion: keep the bus and the LLM context lean, let the object store hold weight, and provide precise retrieval tools so agents fetch only what they need.