kubemend diagnoses incidents from Prometheus metrics and Loki logs, proposes a fix, and
verifies that fix itself - helm render, Kyverno policy check, live diff, scope check,
live quota headroom - before it ever asks a human to approve anything. It never runs
kubectl apply. It has no cluster credentials that can write.
# bring up a real cluster, inject a fault, watch it get fixed
$ pip install kubemend
$ task lab:up
$ task demo
# ~90s later: a draft PR against the GitOps repo, or a written handoffMost "AI SRE agent" demos are impressive and unverifiable - a model claims it fixed something, and you take its word for it. kubemend is built the other way around: the model's claim of success is never trusted. Every run terminates only after an independent validation pipeline says the proposed fix renders cleanly, satisfies policy, produces a real and scoped diff, and touches nothing outside the declared incident. The agent's only actuator is a Git branch and a draft PR - a human still merges.
It's also a from-scratch agent harness, not a wrapper around LangChain/CrewAI/AutoGen. The loop, context management, tool registry, and verification gate are hand-written and documented, because understanding those trade-offs - not gluing a framework together - is the point of the project.
Every claim the model makes gets re-checked independently before the run can end.
A failure at any gate stage feeds the structured failure back into the loop instead of ending the run.
PromQL against Prometheus/Mimir, LogQL against Loki. Swappable behind a provider interface - Datadog and Grafana Cloud already supported, Dynatrace/CloudWatch are future drop-ins.
Read-only ServiceAccount, allow-listed resource kinds, no Secret values ever fetched.
The agent edits Helm values*.yaml only - never templates directly - so diffs stay small and reviewable.
Re-run independently by the harness at termination, never taken on the model's word.
A hermetic kind-based fault-injection lab reproduces real incidents with property-based checkers, run N times per scenario - not cherry-picked demos. Adversarial scenarios cover a fix with no values-only solution, an out-of-scope incident, and a prompt injection planted in the agent's own log evidence.
Requires Docker (or anything kind can use), uv, and go-task, plus an ANTHROPIC_API_KEY.
$ pip install kubemend$ git clone https://github.com/m-stepkowski/kubemend.git && cd kubemend
$ uv sync
$ export ANTHROPIC_API_KEY=...
$ task lab:up # kind cluster: gitea, Argo CD, kube-prometheus-stack, Loki, Kyverno
$ task demo # inject a fault, run kubemend, show the resulting proposal (~90s)To drive it by hand against your own cluster instead of the demo script:
$ kubemend run --task "shop-api pods in namespace shop are crash-looping since 10 minutes ago" \
--namespace shop --app shop-api
Full config reference and every field in kubemend.yaml:
the README.
Baseline run, claude-sonnet-5, n=5 per scenario.
| scenario | pass | avg cost |
|---|---|---|
| bad-image-tag | 5/5 | $0.29 |
| oom-limit | 5/5 | $0.26 |
| missing-configmap-key | 5/5 | $0.35 |
| bad-probe-path | 4/5 | $0.38 |
| bad-env-endpoint | 5/5 | $0.38 |
| quota-conflict | 5/5 | $0.56 |
The one failure is a genuine model struggle, not a harness bug. Full report and the adversarial-scenario baseline: evals/reports.
The Helm chart installs a narrowly-scoped read-only reader ServiceAccount and RBAC, and spawns nothing by default - an on-call engineer only needs permission to create a Job in one namespace, not the reader's own permissions:
$ helm install kubemend charts/kubemend -n kubemend-system --create-namespace
Runs use the same ghcr.io/m-stepkowski/kubemend image published on every
release, with an in-cluster ServiceAccount identity - no kubeconfig involved. An optional
alert-triggered operator mode can create Jobs automatically from Alertmanager webhooks,
behind a bearer token and per-scope cooldown. See
the chart README
for the full values reference.
I write up the verification pipeline's own bugs, cost numbers, and an incident I caused myself, with real traces rather than a demo reel, at stepkowski.dev/blog.