KubemootConfig Guide
Overview
KubemootConfig is a cluster-scoped singleton CRD that centralizes default images and configuration for all Kubemoot components. It eliminates hardcoded image versions in the operator source code and enables GitOps-friendly version management.
Every Kubemoot controller reads from KubemootConfig via the ConfigCache - a thread-safe in-memory cache that the KubemootConfigReconciler populates on startup and updates on every change.
Why a singleton
Image references and operator defaults live in one cluster-scoped resource instead of the
operator source, operator environment variables, or every custom resource. An image bump
is a patch to one resource: no recompile, no operator restart, no Helm upgrade, and no
chance of two namespaces running different runtime images. The scope is cluster-wide on
purpose; a per-namespace config would bring the drift back. New cluster-wide defaults
(scheduler strategy, retention windows) become new fields on the same resource rather
than new CRDs. If default is missing, controllers fail their first reconcile until it
exists, which is why the operator chart installs it.
Architecture
KubemootConfig CR ("default")
│
▼
KubemootConfigReconciler
│
▼
ConfigCache (thread-safe, in-memory)
│
├── RAGSourceReconciler → indexer, queryService, doclingServe images
├── AgentReconciler → agentRuntime and agentSidecar images
├── MCPServerReconciler → mcpBridge image
└── MCPGatewayReconciler → mcpGateway image
The singleton must be named default. The operator creates it via the Helm chart on initial install.
Spec Reference
Images
| Field | Type | Default | Description |
|---|---|---|---|
images.indexer | string | ghcr.io/kubemoot/indexer:latest | RAGSource indexer job image |
images.queryService | string | ghcr.io/kubemoot/query-service:latest | RAGSource query service image |
images.agentRuntime | string | ghcr.io/kubemoot/agent-runtime:latest | Agent deployment image |
images.agentSidecar | string | ghcr.io/kubemoot/agent-sidecar:latest | Moot Mesh sidecar image, added to the pod of an Agent whose spec.harness.mode is sidecar. The operator chart stamps a pinned tag from the release |
images.harnessGoose | string | ghcr.io/kubemoot/harness-goose:latest | The goose seat image: the harness shim and the goose binary. An Agent whose spec.harness.name is goose and that names no image runs it. The operator chart stamps a pinned tag from the release |
images.mcpGateway | string | ghcr.io/kubemoot/mcp-gateway:latest | MCPGateway deployment image |
images.mcpBridge | string | ghcr.io/kubemoot/mcp-bridge:latest | MCP bridge sidecar image |
images.fitnessRunner | string | ghcr.io/kubemoot/fitness-runner:latest | CrewFitness test job image |
images.workspaceAccess | string | none (stamped by the operator chart) | workspace-access image. An MCPServer selects it with spec.kubemootImage: workspaceAccess instead of naming an image. No default: an unset value leaves such an MCPServer in Error rather than floating a tag |
images.doclingServe | string | quay.io/docling-project/docling-serve-cpu:latest | Docling document converter image |
Defaults
| Field | Type | Default | Description |
|---|---|---|---|
defaults.imagePullSecrets | []LocalObjectReference | Default pull secrets for all Kubemoot workloads | |
defaults.vectorStoreType | enum | pgvector | Default vector store: pgvector, qdrant, milvus, chroma |
defaults.vectorStoreEndpoint | string | Default vector store endpoint for tool indexing. When empty, MCPServer tool indexing is disabled | |
defaults.embeddingModel | string | nomic-embed | Default EmbeddingModel CR name for RAGSources |
defaults.agentResources | ResourceRequirements | requests cpu 25m, memory 96Mi; limit memory 256Mi (goose: 100m/128Mi, limit 512Mi) | Default resources for the agent container of every Agent. Merged per resource name over the built-in default and under the Agent’s own spec.deployment.resources or spec.harness.resources. No CPU limit by default. See Agent resources |
defaults.agentRuntimeClassName | string | Default RuntimeClass for the pods of every Agent, for example gvisor. Empty means the pods set no runtimeClassName. An Agent’s spec.deployment.runtimeClassName wins, and "" there turns the default off for that Agent. Covers Agent pods only. See Isolation | |
defaults.otelCollectorEndpoint | string | Default OpenTelemetry collector for all agents. Moot Mesh sidecars export spans to it over OTLP/HTTP |
Artifacts
Every tool result is always stored as its own file in the discussion workspace; there is no setting for it (see The workspace). The field artifacts.perCallArtifacts no longer exists: a KubemootConfig already in the cluster that holds it keeps working (the API server drops the unknown field and the operator never reads it), and the chart ignores the value artifacts.perCallArtifacts. A client that validates strictly (kubectl apply --validate=strict) rejects a manifest that still sets the field, so remove it from manifests you apply yourself.
| Field | Type | Default | Description |
|---|---|---|---|
workspace.queryEngine | enum | sqlite | The engine behind the workspace_query tool of the workspace-access read-ops server. sqlite is the default, also when the field or the whole workspace object is left out (the old artifacts.queryEngine is read when this one is not set): it runs inside the workspace-access process, so nothing is added to the image, and it answers small artifacts in milliseconds with a few MB of memory. none is the explicit way to turn the tool off: no query tool is offered. duckdb is the option for large data: it is faster on large scans, but the workspace-access image must carry the duckdb program, and without it the server logs why and offers no query tool. See Query engines. The operator sets WORKSPACE_QUERY_ENGINE on every MCPServer that selects spec.kubemootImage: workspaceAccess; a server that sets the variable itself keeps its own value. For sqlite and duckdb (everything but none) the operator also adds a 1Gi emptyDir at /kubemoot/workspace and sets TMPDIR to it (see Workspace directory). The operator chart sets it from the value workspace.queryEngine |
Workspace
| Field | Type | Default | Description |
|---|---|---|---|
workspace.defaultAccess | enum | The workspace access (none, read, write, readWrite) of every agent whose Agent and Crew declare none. Empty leaves such an agent implicit, which keeps today’s behavior. See Agent |
Coordinator
| Field | Type | Default | Description |
|---|---|---|---|
coordinator.resumeMaxAge | string | 30m | With coordinator.resume, how old a thread’s last logged event may be for a restarted coordinator to resume it, as a Go duration such as 30m or 1h30m. A thread older than that is closed with reason expired, and its request, handed again, starts a new thread. 0 or empty turns the limit off, and then a thread of any age is resumed. The chart refuses a value that is not a duration. The operator sets KUBEMOOT_COORDINATOR_RESUME_MAX_AGE on every coordinator Agent when coordinator.resume is on; an Agent that sets the variable itself in spec.deployment.env keeps its own value. See Resume |
coordinator.lease | bool | false | Opt in; leases pay off only with more than one coordinator replica. More than one coordinator replica can run, each owning the threads it holds a lease for. A replica takes a lease on a thread before it acts, renews it while the thread is open and gives it up at close; a thread whose owner stopped is taken over by another replica through resume, and a request handed to any replica finds its thread in the log. Needs coordinator.resume and coordinator.eventLog, both on by default (the chart refuses lease when you turned either off; a coordinator started with the variable set and no resume stays off and says so). With it on the chart creates the kubemoot_coordinator_leases KV bucket, the coordinator Agents may set spec.deployment.replicas above 1, and they roll with RollingUpdate instead of Recreate. With it off, the operator runs every coordinator Agent with one replica and the Recreate strategy whatever its spec.deployment.replicas asks for, and sets the Agent’s ReplicasLimited status condition to say the replicas were limited to 1 because leases are off. The operator sets KUBEMOOT_COORDINATOR_LEASE=true on every Agent whose discussRole is coordinator, and a coordinator Agent’s strategy follows the same setting; an Agent that sets the variable itself in spec.deployment.env keeps its own value. See Leases and more than one replica |
coordinator.resume | bool | true | On by default, also when the field is unset. A restarted coordinator picks up the threads it had in flight: it rebuilds each from its log, reads the messages that arrived while it was down, and carries on from the last decision. Needs coordinator.eventLog: a realm that turns the log off must turn resume off too (the chart refuses resume without the log, and the operator treats resume as off while the log is off; a coordinator started with the variable set and no log stays off and says so). Run one coordinator replica unless coordinator.lease is on. See Resume. The operator sets KUBEMOOT_COORDINATOR_RESUME=true on every Agent whose discussRole is coordinator; an Agent that sets the variable itself in spec.deployment.env keeps its own value. The operator chart sets it from the value coordinator.resume |
coordinator.eventLog | bool | true | On by default, also when the field or the whole spec.coordinator object is left out of a hand-written KubemootConfig. Every coordinator writes each decision it makes about a thread to the KUBEMOOT_COORDINATOR stream before it acts on it, keeps other coordinators off its threads, and checks the log against the thread’s live state when the thread closes. See The coordinator log. The operator sets KUBEMOOT_COORDINATOR_EVENT_LOG=true on every Agent whose discussRole is coordinator; an Agent that sets the variable itself in spec.deployment.env keeps its own value. The operator chart sets it from the value coordinator.eventLog. Set it to false (with coordinator.resume false) to keep the behavior of releases before the log became the default |
Status
| Field | Type | Description |
|---|---|---|
ready | bool | Configuration applied successfully |
lastUpdated | Time | Last configuration update timestamp |
message | string | Additional status information |
release | string | The Kubemoot release the cluster runs: the version of the operator chart, set by the operator from what the chart build stamps. kmctl version and the dashboard’s Kubemoot components panel show it. Empty until an operator that reports it has reconciled the object |
Example
Reference KubemootConfig
The Helm chart creates this on install:
apiVersion: kubemoot.ai/v1alpha1
kind: KubemootConfig
metadata:
name: default
spec:
images:
indexer: "ghcr.io/kubemoot/indexer:0.3.1"
queryService: "ghcr.io/kubemoot/query-service:0.2.0"
agentRuntime: "ghcr.io/kubemoot/agent-runtime:0.8.5"
mcpGateway: "ghcr.io/kubemoot/mcp-gateway:0.5.2"
mcpBridge: "ghcr.io/kubemoot/mcp-bridge:0.4.0"
doclingServe: "quay.io/docling-project/docling-serve-cpu:latest"
defaults:
vectorStoreType: pgvector
embeddingModel: nomic-embed
defaults.imagePullSecrets is optional and empty by default: public images are published to
ghcr.io/kubemoot and need no pull secret. Set it only if a private mirror requires one, for
example:
defaults:
imagePullSecrets:
- name: my-registry-pull-secret
Secret replication
The operator copies Secrets into crew namespaces: image pull secrets, vector-store
credentials, and Secrets named in a crew’s kubemoot.ai/replicate-secrets annotation. It
copies only from its own namespace plus the namespaces listed in the operator chart’s
secretReplication.allowedSourceNamespaces value; a request to copy from any other
namespace is refused.
secretReplication:
allowedSourceNamespaces:
- shared-credentials
Secret use grants
The chart value security.secretUseGrants installs ValidatingAdmissionPolicies that reject
an MCPServer or Agent referencing a Secret or ServiceAccount the requester holds no use
grant on. The grant, the rejection message and the exempt cases are in
Secrets and tools.
| Value | Default | Meaning |
|---|---|---|
security.secretUseGrants.enabled | false | Install the policies and their bindings. Needs Kubernetes 1.30 or later. |
security.secretUseGrants.exemptUsers | [] | User names the check skips, as the API server names them. The operator’s ServiceAccount is always skipped. |
security.secretUseGrants.validationActions | ["Deny"] | Deny rejects. ["Audit", "Warn"] records and warns without rejecting; use it first when rolling out. |
security.secretUseGrants.namespaceSelector | {} | Label selector limiting the check to matching namespaces; empty means every namespace. |
Network policies
The chart value security.networkPolicies makes the operator generate NetworkPolicy objects
for crew namespaces, tool servers, gateways, and the workloads that declare spec.egress.
It is cluster-wide: there is no per-crew switch. What is generated and what each CNI
enforces are in Network policies.
| Value | Default | Meaning |
|---|---|---|
security.networkPolicies.enabled | false | Master switch. Off creates nothing and deletes only the objects the operator generated (label kubemoot.ai/generated-by=network-policy). Needs a CNI that enforces NetworkPolicy. |
security.networkPolicies.ingress | true | The ingress set, with enabled. |
security.networkPolicies.egress.mode | Off | Off, or Declared to restrict only the MCPServers and Agents that set spec.egress. |
security.networkPolicies.cilium | auto | auto emits CiliumNetworkPolicy objects when that kind is installed (read when the operator starts); on always; off never. |
security.networkPolicies.dns | kube-system, k8s-app=kube-dns | Namespace and pod labels of cluster DNS. |
security.networkPolicies.metricsScraper | none | Namespace and pod labels of the metrics scraper, admitted to the pods it scrapes. |
security.networkPolicies.trustedIngressNamespaces | [] | Without Cilium: namespaces of the Gateway API proxy pods, admitted to the discussion gateways. |
security.networkPolicies.controlPlaneCIDRs | [] | Without Cilium: source ranges of the API server’s service proxy (kubectl ask). kmctl uses a port-forward and needs no range. |
security.networkPolicies.clusterCIDRs | [] | Pod and service ranges that are not private addresses; the fallback for undeclared host enforcement keeps them blocked. |
schedulingCrew.allowedIngress | [] | Namespace selectors copied to the scheduling crew’s spec.network.allowedIngress, so a web server in another namespace that lists the crews can reach its discussion gateway, for example - matchLabels: {kubernetes.io/metadata.name: homelab-pilot}. Empty leaves the network key off the Crew. |
Caching
The operator keeps a thread-safe, in-memory cache of the default KubemootConfig’s
images and defaults, shared across every controller. The cache is populated on startup
and updated whenever the default KubemootConfig changes; controllers read from it on
every reconciliation, so no controller needs its own watch on KubemootConfig.
Image Version Management
CI/CD Flow
- CI builds a component (e.g., mcp-bridge) and pushes a versioned image tag
- The Flux HelmRelease in your GitOps repository overrides chart defaults with specific versions
- Helm renders KubemootConfig with the pinned versions
- All controllers pick up the new image via ConfigCache on next reconcile
Manual Override
After building a new image version outside of the normal CI flow, manually patch KubemootConfig:
kubectl patch kubemootconfig default --type=merge -p \
'{"spec":{"images":{"mcpBridge":"ghcr.io/kubemoot/mcp-bridge:0.4.1"}}}'
Why Not :latest?
Using :latest tags with IfNotPresent pull policy causes stale images - Kubernetes caches the image and never pulls a newer version even when the tag is updated. Always use versioned tags. The Flux HelmRelease values file is the correct place to pin versions.
Troubleshooting
Controller using wrong image
Check the KubemootConfig:
kubectl get kubemootconfig default -o yaml
If the values are wrong, check the Flux HelmRelease values:
kubectl get helmrelease -n flux-system kubemoot-operator -o jsonpath='{.spec.values}' | jq .
imagePullSecrets not applied
KubemootConfig’s defaults.imagePullSecrets must use the secret name that exists in the target namespace. It is empty by default, since public images pull from ghcr.io/kubemoot without a secret. Set it to a secret you create in each namespace only if you point global.imageRegistry at a private mirror.
ConfigCache stale
The cache updates on KubemootConfig reconciliation. If a controller seems to use old values, check that the KubemootConfigReconciler is running:
kubectl logs deploy/kubemoot-operator | grep KubemootConfig