AI observability with Grafana: what we learned from a real incident

We knew something had broken. What we didn’t know was why nobody had noticed.

JOSÉ GINER
DevOps

You can have dashboards for almost everything and still not find out that something is broken. That happened to us recently: one weekend, when a firewall did its job so well — blocking traffic before it reached any service — that not a single error fired, and precisely because of that, nobody noticed. What I’m telling you here about AI observability with Grafana started that weekend.

We work on an infrastructure, for a client, that mixes Azure with self-hosted services: Kubernetes, a RabbitMQ broker, and two firewalls in front of every domain. When something breaks in there, looking at one isolated panel isn’t enough: you need to see the whole picture. That’s where Grafana comes in — the tool that took us from having metrics scattered across half a dozen sources to being able to ask an agent what happened in a service.

Why Grafana and not Azure’s native alerts

Grafana is a web interface where you build dashboards, define alerts on metrics, and plug in hundreds of plugins. We manage the Kubernetes cluster on Azure, but we’d rather not hand Azure the job of alerting: we do that with Grafana, for cost and configuration reasons, and because it lets us keep a close eye on the most critical pieces, like the messaging broker and the services running on Kubernetes.

Azure’s cost and configuration didn’t convince us

Azure’s native alerts, properly configured, get expensive — which is why we ruled them out early on. And while they do offer plenty of metrics, they don’t expose them at the level of granularity you need to debug a real incident; configuration was the other reason Grafana won out over Azure.

RabbitMQ and Kubernetes, the two things we watch most closely

Beyond alerting, there are two pieces we watch especially closely: the messaging broker and the services on Kubernetes. The first is RabbitMQ, the system our services use to talk to each other, and it’s one of the most critical points: we have a Grafana folder just for that, with a general cluster dashboard (two nodes) and a queues dashboard we check whenever someone asks to purge a queue, and we want to confirm it has actually dropped to zero.

It’s a similar story with Kubernetes: each project has its own namespace and its own metrics, and we lean hard on the time range selector to ask what happened between Friday and Sunday and come away with an exact snapshot of that moment.

Day to day, this solves two things at once:

• Seeing the state of a service or cluster in real time, without jumping from console to console.
• Querying any past time window to reconstruct what happened, not just what’s happening right now.

The incident that showed us the real problem

To understand why we needed more than a handful of disconnected dashboards, it helps to explain how traffic gets into our client’s systems. Any connection arriving through a domain passes through three layers before it reaches the service:

1. First, the WAF or Web Application Firewall, with a set of rules to detect scraping and attack patterns, increasingly common now that AI is in the mix.
2. Then the network firewall, the classic Fail2ban: if your IP fails too many times, it gets blocked.
3. And only then does the request reach Kubernetes, and the service that actually has to handle it.

We’re used to the idea that when something fails, step one is jumping into K9s, kubectl or any Kubernetes UI. That weekend, that was exactly what didn’t work: one of our teams reported a problem and, when we looked, we found nothing — not one error, not one odd trace. That silence is the worst possible scenario in an incident.

Everything was being stopped at the Web Application Firewall. The requests never reached the cluster, so from the service’s point of view nothing had happened. It wasn’t that we were missing data: we’d had WAF dashboards from day one. The problem was that the information was spread across sources that didn’t talk to each other, so we couldn’t tell, in the moment, whether a spike belonged to one particular project or service.

That was the real starting point for everything that follows: we didn’t need more telemetry; we needed to cross-reference it. We had plenty of data sitting in different corners, but nobody was looking at it together.

Grafana is already, in itself, a viewer for aggregated information. Once you connect an MCP on top of that, the agent gets access to everything inside it.

DANIEL ALARCÓN
Software Architect

AI observability with Grafana: what changes when you connect Claude via MCP

From there, we started cross-referencing sources and built a hybrid dashboard that joins the cluster of the project where we spotted the error with the WAF, so we can tell exactly when a specific URI triggered something. In parallel, we started connecting Claude to Grafana through MCP so it can take care of generating the alerts.

The flow we use now is simple:

┌────────────────────────────────────────────────────┐
│                metric out of range                 │
└────────────────────────────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────┐
│               the alert is generated               │
└────────────────────────────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────┐
│     the dashboard is captured, at that instant     │
└────────────────────────────────────────────────────┘
                           │
                           ▼
┌────────────────────────────────────────────────────┐
│   published to Slack: service channel + context    │
└────────────────────────────────────────────────────┘

That flow has given us per-service alert channels where, at any moment, you get a snapshot of what’s going on: when someone kicks off a reindex and things start failing, we watch the queued requests grow in real time; when they stop, we see it go green.

Our colleague Daniel Alarcón, a software architect, pointed out during the session why this becomes so powerful: Grafana is already, in itself, a viewer for information aggregated from every source you connect to it, both infrastructure and business logic. Once you connect an MCP on top of that, the agent gets access to everything inside it: logs, CPU, business metrics, any KPI wired to a panel, even if it comes straight from the production database.

The same applies to Azure resources: if something fails there, the agent goes straight to those logs without us having to build a specific integration for each piece, because Grafana is already looking there.

A real case: asking Claude to review how a PR behaved in production

During the session, we ran a live demo so the rest of the team could see clearly where the limit sits today on what we can ask the agent. To do that, we asked it to analyse the production impact of a recent pull request using the context available through MCP.

The request to Claude

Daniel had merged and deployed a pull request the day before with changes to the ingestions. Instead of going through logs and metrics by hand, he asked Claude — with access to the repository and to Grafana over MCP — to analyse the deployment.

The agent replicated the work any of us would have done: it built queries against the cluster metrics, found the application’s name in Loki without anyone telling it where to look, compared timestamps before and after the deployment and computed durations, flagging anomalies. Without that even being the goal, it has already found us real bugs in ingestions.

Daniel stressed that the more specific the question, the more precise the answer it returns. Logs are heavy and eat a lot of context, so without narrowing the access down, the agent reads more than it needs to. That’s why we’ve started to stop writing sensitive data in the clear into our logs.

What we still don’t have: traces

Felipe Polo, CEO of Orbitant, raised a point that’s still open: investigating properly requires seeing both the physical trace of the requests and the logical trace — that is, what the system did, step by step. Of the three pillars of observability we have two: metrics, with Grafana and Prometheus where it’s already deployed; and logs, currently being cleaned up so they don’t expose sensitive data. The third one is missing: traces. The next step is to bring in Grafana Tempo and reconstruct exactly what happened when one specific client message came in.

A change in how we investigate incidents

The next technical step is already clear: add Tempo, carry on with the migration of logs to JSON that’s already under way, and roll Prometheus out to more services so we get finer-grained internal metrics.

The real change is in the order of the questions: we used to look at Kubernetes first and guess which layer had failed; now the starting point is asking an agent with access to every source what happened, and letting it drill down as far as it needs to. Daniel already points to where this is heading: agents that continuously watch the health of the system, not just when someone asks — though that one isn’t built yet.

Frequently asked questions

Do I need traces (Tempo) before I can connect an AI agent to Grafana over MCP?
No. With metrics and logs you can already analyse quite a lot: that’s exactly what we’ve been doing so far. Traces add another layer of precision (the “why” at the code level), but they aren’t a requirement to get started.

Why not just use Azure’s native alerts if the cluster is already there?
In our case, cost and configuration flexibility. Azure’s native alerts have improved, but we still need the level of correlation between sources that Grafana gives us, especially when the problem isn’t in a single layer but at the boundary between several.

Is it safe to give an AI agent access to raw logs?
Not by default. Logs can contain sensitive data, so before granting that access it’s worth reviewing what’s being written there. That’s exactly what we’ve started doing: we’ve stopped dumping sensitive data in the clear into our logs, precisely because we now know that access exists.