| 9 min read

I let an AI agent into my infrastructure. What it takes in guardrails.

AI Governance Agentic Development platform-engineering GitOps

Also available in German: Ich habe einen KI-Agenten in meine Infrastruktur gelassen. Was es an Leitplanken braucht.

TL;DR: An AI agent has been working on my infrastructure for months. Two real mistakes happened, both could have hurt, and nothing broke. The order made the difference: first a gateway in front of the models so there is a single way in, then policies in audit mode until you can see what they would block, then enforce them, and only after that let the agent get to work.

It started with a question I couldn’t shake: how far do I actually get if I let an AI agent into my own infrastructure? Not as a chat window on the side, but working alongside me on manifests, on configuration, on documentation. And how do you introduce something like that step by step, without waking up to a cluster that no longer feels like mine?

The same question goes unanswered in plenty of companies: AI arrives faster than any approval process, and nobody quite knows what people work with. What drove me first, though, was plain curiosity.

My test lab in the basement

My test lab sits in the basement: a handful of mini PCs, a NAS, a UPS, solar power. It is my lab for my own experiments. I try patterns out here before I suggest them anywhere else, and if something falls over, it falls over at my place.

Two questions interested me. Do the principles people preach in big projects also hold at small scale? And can an AI agent seriously work on infrastructure without costing me sleep? After a few months I would say yes to both. But only because I set limits from day one.

The architecture

The base is a Proxmox cluster with six nodes and a highly available k3s cluster on top. Everything is described declaratively in Git and applied through ArgoCD. Nobody changes the cluster directly, myself included. Storage and backups come from the NAS, and a copy sits offsite in the cloud, for the same reason the secrets live outside the house: what I need to rebuild must not live inside the thing that just went away.

Every model access goes through one gateway, every change through Git A human and an AI agent change the Git repository, and ArgoCD applies the changes to the k3s cluster. Inside the cluster, applications reach Ollama only through the LiteLLM gateway; direct access is blocked by Kyverno. Secrets come from the external Bitwarden Secrets Manager via the External Secrets Operator. Human and AI agent own user, wrapper scripts Git repository declarative, versioned ArgoCD applies changes k3s cluster on Proxmox Applications Open WebUI, n8n LiteLLM gateway only way to the models Ollama local models, GPU nodes direct access blocked by Kyverno Secrets Secrets sync External Secrets Operator writes Kubernetes secrets Bitwarden Secrets Manager external, end-to-end encrypted
Changes reach the cluster only through Git, model access only through the gateway, and secrets come from outside.

The models run locally with Ollama. Small ones up to about 8 billion parameters on an ordinary mini PC, larger ones on a dedicated GPU node with an NVIDIA RTX 5060 Ti (16 GB) and a GTX 1660 Super (6 GB). Part of the power comes from my own solar panels, because compute in the basement should cost the environment as little as possible.

Governance: one gateway, clear policies

No service talks to a model directly. Everything goes through LiteLLM as the single gateway, and direct access to Ollama is blocked by a Kyverno policy. Kyverno checks every resource before it enters the cluster and rejects what does not fit.

The benefit is mundane and that is exactly why it matters: I decide in one place which models exist and who may use them. Every service has its own key. The chat bot for the radio network gets a small chat model, the workflow automation only embeddings and extraction. And I can see who consumes how much without stitching together logs from five services.

The gateway does both: local compute in the basement and, when a task calls for it, the APIs of the big frontier models. Same guardrails, same way in, just a different route behind it. When migrating, one rule applies that I recommend to everyone: only the transport path changes, never the model at the same time.

Most policies still run in audit mode and only report what they would block. They get enforced once they break nothing legitimate. One detail I did not know beforehand: generate rules take effect even in audit mode, so the block on direct model access is already live.

Secrets: deliberately external

Secrets live in Bitwarden Secrets Manager, connected through the External Secrets Operator. In an otherwise local setup that is on purpose. A secrets manager inside the cluster depends on the cluster, and exactly when I have to rebuild the data center, I need the secrets first. The dependency is smaller than it sounds: the values land as regular Kubernetes secrets, so running services keep going through a short outage.

An AI agent in infrastructure operations

The agent is Claude Code. It writes manifests, Ansible roles and OpenTofu code, hunts for errors and documents, always in phases: one block, one report, I read, decide and give the go for the next step. It sounds more cumbersome than it is.

A nice example is NetBox, which I am introducing as the source of truth for network and hardware. Typing the inventory by hand would have cost me entire evenings. The agent evaluated documentation, OpenTofu state and running systems, and I checked and corrected.

Where the agent got it wrong

Two mistakes stuck with me, because they say more about working together than any success report.

The key in the report. After the first migration phase, the middle of an otherwise clean report contained n8n’s virtual key in plain text. Reachable internally only, so no drama, but it was valid for all models. I regenerated it and banned the agent from printing secret values. That was not enough for me: a rule it has to follow is weaker than one it cannot break. The Bitwarden migration is therefore built so that it never gets to see the values.

The DNS trap. The agent noticed that full service names in the cluster land at the ingress instead of the service. It documented that cleanly and used short names as a workaround. Well spotted, but weighted wrongly: the places using full names were the critical ones, the tunnels for all public sites and the single sign-on. It only worked because the pods were older than the DNS change. A power outage would have taken everything down at once.

The agent is strong at finding things, sometimes stronger than I am. Judging what hangs off it in the worst case stays my job.

Security: what holds, and what I leave open

An agent reads whatever lands in front of it: documentation, files in a repository, web pages, output from tools. Any of that can contain a sentence that looks like an instruction but did not come from me. The technical term is prompt injection, and OWASP ranks it first among the risks for LLM applications. A residual risk remains. Better prompts do not solve it, safety has to come from the architecture.

Here is what I have in place: a dedicated user without special rights. Access to the secrets manager and Ansible only through prepared wrapper scripts. No direct cluster changes, only Git and ArgoCD, every step versioned and reversible with a revert. Kyverno checks regardless of who wrote a manifest. Plus the phase model: I approve, I read the reports, and I grant command permissions one at a time.

One limit belongs on the table. The agent itself is a cloud service, and its context of files read and output seen leaves the house. Data sovereignty applies to the local models. Not to the agent, and not to requests that go through the gateway to frontier APIs. Blur that line and you are selling a sense of safety that does not exist.

Observability: how I plan to approach it. Traceability should come in three layers. First Git and ArgoCD: every change the agent makes is a commit that points back to the session, and ArgoCD shows what of it actually arrived in the cluster. Second the gateway’s metrics: LiteLLM sees every model request with key, model and tokens. I want to evaluate that with Prometheus and Grafana, which already run here for power and UPS, and have budgets and per-key limits raise an alarm when something looks off. Third the cluster audit log together with Kyverno’s policy reports in one central log target. What is still missing after that is the view inside the agent itself. Next up, then, is a hook that writes every tool call into a log the agent is not allowed to change.

Where I want to look closer:

  • write down a formal threat model
  • test systematically for prompt injection, ideally from the outside
  • have the wrapper scripts and sudo rights reviewed independently
  • limit the agent’s outbound network traffic
  • set up and test those three observability layers, alarm rules included
  • the legal picture and data protection, with specialists

What I take from it

The order was the decisive part. First a gateway, then traceability through Git and the gateway’s logs, then the policies enforced, and only after that an agent that works alongside me. Swap the steps and you end up migrating every service one by one later, grumbling as you go. I have been there myself.

Just as important is saying the limits out loud. Local stays local, anything that goes through the gateway to a cloud API leaves the house. Anyone who signs something off needs to know that line, otherwise data sovereignty is just a word on a slide. That the same gateway decouples applications from models and keeps me out of lock-in is a bonus I happily take, but it was never the main reason. And that logged access is the basis for the records the EU AI Act asks about is a point I name and then leave to specialists. This is not legal advice.

I now treat an agent like a new colleague from outside: its own access, only the rights it needs, everything traceable. It works faster than I do, and therefore makes mistakes faster too.

If you have already been down this road, I’d like to hear where it creaked for you.

Sources and further reading

Transparency: This post was created with AI assistance. I reviewed it and take responsibility for the content. More under AI Transparency.