KubemootConfig Guide

KubemootConfig is the cluster-scoped singleton that holds default images and settings for every Kubemoot component, keeping versions out of the operator source.

Overview

KubemootConfig is a cluster-scoped singleton CRD that centralizes default images and configuration for all Kubemoot components. It eliminates hardcoded image versions in the operator source code and enables GitOps-friendly version management.

Every Kubemoot controller reads from KubemootConfig via the ConfigCache - a thread-safe in-memory cache that the KubemootConfigReconciler populates on startup and updates on every change.

Why a singleton

Image references and operator defaults live in one cluster-scoped resource instead of the operator source, operator environment variables, or every custom resource. An image bump is a patch to one resource: no recompile, no operator restart, no Helm upgrade, and no chance of two namespaces running different runtime images. The scope is cluster-wide on purpose; a per-namespace config would bring the drift back. New cluster-wide defaults (scheduler strategy, retention windows) become new fields on the same resource rather than new CRDs. If default is missing, controllers fail their first reconcile until it exists, which is why the operator chart installs it.

Architecture

KubemootConfig CR ("default")
    │
    ▼
KubemootConfigReconciler
    │
    ▼
ConfigCache (thread-safe, in-memory)
    │
    ├── RAGSourceReconciler     → indexer, queryService, doclingServe images
    ├── AgentReconciler         → agentRuntime and agentSidecar images
    ├── MCPServerReconciler     → mcpBridge image
    └── MCPGatewayReconciler    → mcpGateway image

The singleton must be named default. The operator creates it via the Helm chart on initial install.

Spec Reference

Images

FieldTypeDefaultDescription
images.indexerstringghcr.io/kubemoot/indexer:latestRAGSource indexer job image
images.queryServicestringghcr.io/kubemoot/query-service:latestRAGSource query service image
images.agentRuntimestringghcr.io/kubemoot/agent-runtime:latestAgent deployment image
images.agentSidecarstringghcr.io/kubemoot/agent-sidecar:latestMoot Mesh sidecar image, added to the pod of an Agent whose spec.harness.mode is sidecar. The operator chart stamps a pinned tag from the release
images.harnessGoosestringghcr.io/kubemoot/harness-goose:latestThe goose seat image: the harness shim and the goose binary. An Agent whose spec.harness.name is goose and that names no image runs it. The operator chart stamps a pinned tag from the release
images.mcpGatewaystringghcr.io/kubemoot/mcp-gateway:latestMCPGateway deployment image
images.mcpBridgestringghcr.io/kubemoot/mcp-bridge:latestMCP bridge sidecar image
images.fitnessRunnerstringghcr.io/kubemoot/fitness-runner:latestCrewFitness test job image
images.workspaceAccessstringnone (stamped by the operator chart)workspace-access image. An MCPServer selects it with spec.kubemootImage: workspaceAccess instead of naming an image. No default: an unset value leaves such an MCPServer in Error rather than floating a tag
images.doclingServestringquay.io/docling-project/docling-serve-cpu:latestDocling document converter image

Defaults

FieldTypeDefaultDescription
defaults.imagePullSecrets[]LocalObjectReferenceDefault pull secrets for all Kubemoot workloads
defaults.vectorStoreTypeenumpgvectorDefault vector store: pgvector, qdrant, milvus, chroma
defaults.vectorStoreEndpointstringDefault vector store endpoint for tool indexing. When empty, MCPServer tool indexing is disabled
defaults.embeddingModelstringnomic-embedDefault EmbeddingModel CR name for RAGSources
defaults.agentResourcesResourceRequirementsrequests cpu 25m, memory 96Mi; limit memory 256Mi (goose: 100m/128Mi, limit 512Mi)Default resources for the agent container of every Agent. Merged per resource name over the built-in default and under the Agent’s own spec.deployment.resources or spec.harness.resources. No CPU limit by default. See Agent resources
defaults.agentRuntimeClassNamestringDefault RuntimeClass for the pods of every Agent, for example gvisor. Empty means the pods set no runtimeClassName. An Agent’s spec.deployment.runtimeClassName wins, and "" there turns the default off for that Agent. Covers Agent pods only. See Isolation
defaults.otelCollectorEndpointstringDefault OpenTelemetry collector for all agents. Moot Mesh sidecars export spans to it over OTLP/HTTP

Artifacts

Every tool result is always stored as its own file in the discussion workspace; there is no setting for it (see The workspace). The field artifacts.perCallArtifacts no longer exists: a KubemootConfig already in the cluster that holds it keeps working (the API server drops the unknown field and the operator never reads it), and the chart ignores the value artifacts.perCallArtifacts. A client that validates strictly (kubectl apply --validate=strict) rejects a manifest that still sets the field, so remove it from manifests you apply yourself.

FieldTypeDefaultDescription
workspace.queryEngineenumsqliteThe engine behind the workspace_query tool of the workspace-access read-ops server. sqlite is the default, also when the field or the whole workspace object is left out (the old artifacts.queryEngine is read when this one is not set): it runs inside the workspace-access process, so nothing is added to the image, and it answers small artifacts in milliseconds with a few MB of memory. none is the explicit way to turn the tool off: no query tool is offered. duckdb is the option for large data: it is faster on large scans, but the workspace-access image must carry the duckdb program, and without it the server logs why and offers no query tool. See Query engines. The operator sets WORKSPACE_QUERY_ENGINE on every MCPServer that selects spec.kubemootImage: workspaceAccess; a server that sets the variable itself keeps its own value. For sqlite and duckdb (everything but none) the operator also adds a 1Gi emptyDir at /kubemoot/workspace and sets TMPDIR to it (see Workspace directory). The operator chart sets it from the value workspace.queryEngine

Workspace

FieldTypeDefaultDescription
workspace.defaultAccessenumThe workspace access (none, read, write, readWrite) of every agent whose Agent and Crew declare none. Empty leaves such an agent implicit, which keeps today’s behavior. See Agent

Coordinator

FieldTypeDefaultDescription
coordinator.resumeMaxAgestring30mWith coordinator.resume, how old a thread’s last logged event may be for a restarted coordinator to resume it, as a Go duration such as 30m or 1h30m. A thread older than that is closed with reason expired, and its request, handed again, starts a new thread. 0 or empty turns the limit off, and then a thread of any age is resumed. The chart refuses a value that is not a duration. The operator sets KUBEMOOT_COORDINATOR_RESUME_MAX_AGE on every coordinator Agent when coordinator.resume is on; an Agent that sets the variable itself in spec.deployment.env keeps its own value. See Resume
coordinator.leaseboolfalseOpt in; leases pay off only with more than one coordinator replica. More than one coordinator replica can run, each owning the threads it holds a lease for. A replica takes a lease on a thread before it acts, renews it while the thread is open and gives it up at close; a thread whose owner stopped is taken over by another replica through resume, and a request handed to any replica finds its thread in the log. Needs coordinator.resume and coordinator.eventLog, both on by default (the chart refuses lease when you turned either off; a coordinator started with the variable set and no resume stays off and says so). With it on the chart creates the kubemoot_coordinator_leases KV bucket, the coordinator Agents may set spec.deployment.replicas above 1, and they roll with RollingUpdate instead of Recreate. With it off, the operator runs every coordinator Agent with one replica and the Recreate strategy whatever its spec.deployment.replicas asks for, and sets the Agent’s ReplicasLimited status condition to say the replicas were limited to 1 because leases are off. The operator sets KUBEMOOT_COORDINATOR_LEASE=true on every Agent whose discussRole is coordinator, and a coordinator Agent’s strategy follows the same setting; an Agent that sets the variable itself in spec.deployment.env keeps its own value. See Leases and more than one replica
coordinator.resumebooltrueOn by default, also when the field is unset. A restarted coordinator picks up the threads it had in flight: it rebuilds each from its log, reads the messages that arrived while it was down, and carries on from the last decision. Needs coordinator.eventLog: a realm that turns the log off must turn resume off too (the chart refuses resume without the log, and the operator treats resume as off while the log is off; a coordinator started with the variable set and no log stays off and says so). Run one coordinator replica unless coordinator.lease is on. See Resume. The operator sets KUBEMOOT_COORDINATOR_RESUME=true on every Agent whose discussRole is coordinator; an Agent that sets the variable itself in spec.deployment.env keeps its own value. The operator chart sets it from the value coordinator.resume
coordinator.eventLogbooltrueOn by default, also when the field or the whole spec.coordinator object is left out of a hand-written KubemootConfig. Every coordinator writes each decision it makes about a thread to the KUBEMOOT_COORDINATOR stream before it acts on it, keeps other coordinators off its threads, and checks the log against the thread’s live state when the thread closes. See The coordinator log. The operator sets KUBEMOOT_COORDINATOR_EVENT_LOG=true on every Agent whose discussRole is coordinator; an Agent that sets the variable itself in spec.deployment.env keeps its own value. The operator chart sets it from the value coordinator.eventLog. Set it to false (with coordinator.resume false) to keep the behavior of releases before the log became the default

Status

FieldTypeDescription
readyboolConfiguration applied successfully
lastUpdatedTimeLast configuration update timestamp
messagestringAdditional status information
releasestringThe Kubemoot release the cluster runs: the version of the operator chart, set by the operator from what the chart build stamps. kmctl version and the dashboard’s Kubemoot components panel show it. Empty until an operator that reports it has reconciled the object

Example

Reference KubemootConfig

The Helm chart creates this on install:

apiVersion: kubemoot.ai/v1alpha1
kind: KubemootConfig
metadata:
  name: default
spec:
  images:
    indexer: "ghcr.io/kubemoot/indexer:0.3.1"
    queryService: "ghcr.io/kubemoot/query-service:0.2.0"
    agentRuntime: "ghcr.io/kubemoot/agent-runtime:0.8.5"
    mcpGateway: "ghcr.io/kubemoot/mcp-gateway:0.5.2"
    mcpBridge: "ghcr.io/kubemoot/mcp-bridge:0.4.0"
    doclingServe: "quay.io/docling-project/docling-serve-cpu:latest"
  defaults:
    vectorStoreType: pgvector
    embeddingModel: nomic-embed

defaults.imagePullSecrets is optional and empty by default: public images are published to ghcr.io/kubemoot and need no pull secret. Set it only if a private mirror requires one, for example:

  defaults:
    imagePullSecrets:
      - name: my-registry-pull-secret

Secret replication

The operator copies Secrets into crew namespaces: image pull secrets, vector-store credentials, and Secrets named in a crew’s kubemoot.ai/replicate-secrets annotation. It copies only from its own namespace plus the namespaces listed in the operator chart’s secretReplication.allowedSourceNamespaces value; a request to copy from any other namespace is refused.

secretReplication:
  allowedSourceNamespaces:
    - shared-credentials

Secret use grants

The chart value security.secretUseGrants installs ValidatingAdmissionPolicies that reject an MCPServer or Agent referencing a Secret or ServiceAccount the requester holds no use grant on. The grant, the rejection message and the exempt cases are in Secrets and tools.

ValueDefaultMeaning
security.secretUseGrants.enabledfalseInstall the policies and their bindings. Needs Kubernetes 1.30 or later.
security.secretUseGrants.exemptUsers[]User names the check skips, as the API server names them. The operator’s ServiceAccount is always skipped.
security.secretUseGrants.validationActions["Deny"]Deny rejects. ["Audit", "Warn"] records and warns without rejecting; use it first when rolling out.
security.secretUseGrants.namespaceSelector{}Label selector limiting the check to matching namespaces; empty means every namespace.

Network policies

The chart value security.networkPolicies makes the operator generate NetworkPolicy objects for crew namespaces, tool servers, gateways, and the workloads that declare spec.egress. It is cluster-wide: there is no per-crew switch. What is generated and what each CNI enforces are in Network policies.

ValueDefaultMeaning
security.networkPolicies.enabledfalseMaster switch. Off creates nothing and deletes only the objects the operator generated (label kubemoot.ai/generated-by=network-policy). Needs a CNI that enforces NetworkPolicy.
security.networkPolicies.ingresstrueThe ingress set, with enabled.
security.networkPolicies.egress.modeOffOff, or Declared to restrict only the MCPServers and Agents that set spec.egress.
security.networkPolicies.ciliumautoauto emits CiliumNetworkPolicy objects when that kind is installed (read when the operator starts); on always; off never.
security.networkPolicies.dnskube-system, k8s-app=kube-dnsNamespace and pod labels of cluster DNS.
security.networkPolicies.metricsScrapernoneNamespace and pod labels of the metrics scraper, admitted to the pods it scrapes.
security.networkPolicies.trustedIngressNamespaces[]Without Cilium: namespaces of the Gateway API proxy pods, admitted to the discussion gateways.
security.networkPolicies.controlPlaneCIDRs[]Without Cilium: source ranges of the API server’s service proxy (kubectl ask). kmctl uses a port-forward and needs no range.
security.networkPolicies.clusterCIDRs[]Pod and service ranges that are not private addresses; the fallback for undeclared host enforcement keeps them blocked.
schedulingCrew.allowedIngress[]Namespace selectors copied to the scheduling crew’s spec.network.allowedIngress, so a web server in another namespace that lists the crews can reach its discussion gateway, for example - matchLabels: {kubernetes.io/metadata.name: homelab-pilot}. Empty leaves the network key off the Crew.

Caching

The operator keeps a thread-safe, in-memory cache of the default KubemootConfig’s images and defaults, shared across every controller. The cache is populated on startup and updated whenever the default KubemootConfig changes; controllers read from it on every reconciliation, so no controller needs its own watch on KubemootConfig.

Image Version Management

CI/CD Flow

  1. CI builds a component (e.g., mcp-bridge) and pushes a versioned image tag
  2. The Flux HelmRelease in your GitOps repository overrides chart defaults with specific versions
  3. Helm renders KubemootConfig with the pinned versions
  4. All controllers pick up the new image via ConfigCache on next reconcile

Manual Override

After building a new image version outside of the normal CI flow, manually patch KubemootConfig:

kubectl patch kubemootconfig default --type=merge -p \
  '{"spec":{"images":{"mcpBridge":"ghcr.io/kubemoot/mcp-bridge:0.4.1"}}}'

Why Not :latest?

Using :latest tags with IfNotPresent pull policy causes stale images - Kubernetes caches the image and never pulls a newer version even when the tag is updated. Always use versioned tags. The Flux HelmRelease values file is the correct place to pin versions.

Troubleshooting

Controller using wrong image

Check the KubemootConfig:

kubectl get kubemootconfig default -o yaml

If the values are wrong, check the Flux HelmRelease values:

kubectl get helmrelease -n flux-system kubemoot-operator -o jsonpath='{.spec.values}' | jq .

imagePullSecrets not applied

KubemootConfig’s defaults.imagePullSecrets must use the secret name that exists in the target namespace. It is empty by default, since public images pull from ghcr.io/kubemoot without a secret. Set it to a secret you create in each namespace only if you point global.imageRegistry at a private mirror.

ConfigCache stale

The cache updates on KubemootConfig reconciliation. If a controller seems to use old values, check that the KubemootConfigReconciler is running:

kubectl logs deploy/kubemoot-operator | grep KubemootConfig