A Go agent that investigates a real Kubernetes cluster through read-only tools, names the root cause with evidence, and never touches the cluster without a human in the loop.
Read-only by default, human-gated to act, and measurably correct.
The project is a failure sandbox coupled to a diagnostic agent. The sandbox is a real two-node k3s cluster running synthetic workloads that can be broken on demand — each injected failure has a known-correct root cause. The agent is a Go tool-calling loop that drives an LLM, lets it investigate the cluster through read-only tools, and reports a root cause with evidence.
The coupling is the whole idea: because every failure has a known answer, the agent's accuracy is measurable — the sandbox doubles as the agent's test harness. It's an exhibit of one skill, agent engineering on real infrastructure, and its thesis is the safety engineering around it.
Each failure breaks the cluster one specific, reproducible way — and each hides its evidence somewhere different, so diagnosing the set demonstrates range rather than one trick.
The dependency failure is the load-bearing one: the symptom (frontend errors) sits on a different object than the cause (the NetworkPolicy), so a naive "find the unhealthy pod" agent fails it. The system prompt trains explicitly for that — a symptom can have its cause elsewhere.
A single Go binary, ~2,100 lines, with a hand-written OpenAI-compatible client instead of an SDK. That choice buys two things: the backend is swappable with one env var — local Ollama / LM Studio or any hosted endpoint, nothing bundled — and the tool-calling loop stays fully visible in one file. For an exhibit of agent engineering, the mechanism being legible is a feature.
The model investigates through eleven read-only tools whose plain-language descriptions are load-bearing — they're the model's entire basis for choosing what to call. Two fixes came straight out of the eval, both about a model over-investigating: forced finalization (when the iteration budget runs out mid-investigation, one final tools-disabled call converts a dead end into a best-effort diagnosis) and a system-prompt stop rule. Secrets are never dereferenced — the tools confirm a referenced Secret exists and has the right keys without ever reading a value.
The allowlist is the difference between an LLM that can run arbitrary commands and one that can request one of seven reviewable, dry-runnable, reversible actions.
Remediation is off by default — the diagnosis loop never mutates the cluster. When it's enabled, the chain is propose → human approve → dry-run → apply. The model proposes exactly one action through a single structured tool whose action field is an enum of seven allowlisted operations — never a free-form command.
Because each action runs through client-go, it can be server-side dry-run: the API server validates and computes the result without persisting, and the agent renders a real before→after diff. The blast radius is bounded by enumeration — the agent literally cannot propose anything outside the seven.
Because every failure has a known-correct cause, accuracy is measurable. The eval injects each of the six failures, runs the agent, and scores whether the diagnosis names the real cause — the latest run scores 6/6 on a local model. That harness is what makes the real risk (a confident, wrong diagnosis) visible, and it's why the agent is read-only first. Every LLM and tool call is a span on one OpenTelemetry trace, so a diagnosis is also an audit trail.