Before Opening Grafana: How Azure Databricks Uses an AI Agent to Investigate Its Own Incidents

Any on-call engineer who has ever been woken up by an alert in the middle of the night knows the routine: open the dashboard, correlate logs from three different services, check whether there was a recent deployment, review metrics from upstream dependencies, and only after spending twenty minutes putting all that context together actually start investigating the root cause.

Azure Databricks measured this internally and found an uncomfortable number: 60% to 80% of incident investigation time is not spent finding the root cause, it is spent gathering the context required to start looking for it.

AI SRE, an internal agent recently documented by Azure Databricks, targets exactly this inefficiency.

The core idea is not to replace engineering judgment. Instead, it eliminates the mechanical work of gathering context before that judgment can happen, running in parallel and within seconds what would otherwise take a human several minutes to assemble manually.

The Mechanism: Three Parallel Tracks, Evidence Before Conclusions

When an incident is triggered, AI SRE does not wait for an engineer to request information. It immediately starts three investigations in parallel.

Platform health checks: It verifies cloud infrastructure, networking, and upstream dependencies, quickly ruling out causes outside the team's code, such as an availability zone experiencing issues or an external provider being unavailable.

Service-level analysis: It examines logs, metrics, traces, recent deployments, and configuration changes related to the specific service involved in the alert.

Runbook execution: It runs workflows that the team has already documented as "what an expert would check for this type of failure," converted into agentic runbooks.

The design principle connecting these three tracks is explicitly stated by the team:

"Structured checks before open-ended reasoning."

This means AI SRE first performs deterministic platform checks and runbook steps, and only then passes the raw results to the LLM layer for synthesis and explanation, never the other way around.

Data collection is not left to the model's judgment. It happens first, following a predefined process.

Every final recommendation is tied to the evidence supporting it and can be traced back to the specific check that generated it, rather than being a loose model inference about what "probably" happened.

My take is that this is the kind of architectural decision that becomes obvious only after someone has experienced the opposite approach.

An agent that immediately starts "open-ended reasoning" over logs and metrics without deterministic checks first can sound convincing even when it is wrong. And during an incident, a convincing but incorrect root-cause hypothesis can delay the actual resolution.

Making "data first, interpretation second" an architectural rule rather than an optional best practice is what makes this type of agent reliable enough to operate without constant supervision.

Hands-On: Turning an On-Call Checklist into an Agentic Runbook

One of the most reusable ideas outside Azure Databricks' internal environment is the runbook mechanism itself.

It is built on top of Genie Code's public skills system (.assistant/skills/), the same mechanism used to teach business logic to data agents.

Any team using Azure Databricks can apply the same approach to its own incident-response process by converting an informal checklist into a skill:

Workspace/.assistant/skills/incidente-fila-kafka-atrasada/
└── SKILL.md

---
name: incidente-fila-kafka-atrasada
description: Runbook para lag alto no consumer group de streaming. Use quando o alerta mencionar consumer lag, offset atrasado ou fila de eventos acumulando.
---

Checagem, na ordem:
1. Consultar `lag_by_partition` no painel de métricas do consumer group; lag acima de
   500 mil mensagens em qualquer partição é o limiar de atenção.
2. Verificar se houve deploy do consumer nas últimas 2 horas (causa mais comum:
   handler novo mais lento que o anterior).
3. Se não houve deploy, checar throughput do broker de origem; partição com lag
   isolado numa única partição indica hot partition, não problema de consumer.
4. Mitigação padrão: escalar réplica do consumer group primeiro, nunca aumentar
   partição em produção sem aprovação, isso reembaralha o particionamento existente.

Once documented this way, the knowledge no longer lives only in the head of the engineer who has solved that incident before.

It becomes executable automatically the next time the alert fires.

That is exactly the effect of the agentic runbook approach used internally by AI SRE, but the underlying pattern can already be applied by other teams without waiting for access to Azure Databricks' internal tooling.

The Work Nobody Sees: Giving the Agent Access Without Creating a Security Risk

The least glamorous part of the project, and the one the Azure Databricks team says required more engineering effort than designing the agent itself, was rebuilding the API layer that gives the agent access to the company's observability systems.

The reason is straightforward.

An agent that can freely query logs, metrics, and traces can also, if poorly configured, generate enough query volume to overload the very observability infrastructure supporting critical production alerts.

There would be a certain irony in a reliability agent becoming the cause of a reliability incident.

The team therefore had to implement rate limits, permission scopes, and sufficient guardrails to keep the agent fast without allowing it to become a source of incidents itself.

This lesson extends far beyond AI SRE.

Any agent project with broad access to production systems carries the same second-order risk, and designing the necessary guardrails often requires more engineering effort than implementing the agent behavior they are meant to protect.

Underestimating this part is one of the most common reasons internal agent projects get stuck during the security phase after already working well as proofs of concept.

What This Does Not Solve

AI SRE does not eliminate the need for well-written runbooks.

It simply executes more quickly what the team already knows how to do.

If nobody has documented the checklist for a new type of incident, there is no runbook to execute. The agent then falls back to generic platform and service-level checks without the shortcut provided by the team's specific operational knowledge.

The architecture also assumes broad access to the company's observability layer.

According to the Azure Databricks team, redesigning the API to provide this access with sufficient guardrails to avoid impacting critical monitoring infrastructure required more engineering effort than building the agent itself.

It is also important to remember that AI SRE is an internal Azure Databricks tool.

There is currently no equivalent public product that can simply be installed and enabled.

What teams can replicate today is the architectural pattern:

structured checks before reasoning, and runbooks implemented as version-controlled skills.

Final Thoughts

The real improvement here does not come from having a smarter model interpret an incident.

It comes from eliminating the time engineers spend assembling context before they can even begin interpreting what happened.

For any platform team that already maintains a wiki full of "how to solve X" procedures, often known only by the people who have encountered that incident before, the natural next step is similar to what Azure Databricks describes internally:

turn that knowledge into version-controlled, executable runbooks and let deterministic checks run before any open-ended reasoning layer enters the process.

References

• Databricks Blog - How Databricks Uses AI to Accelerate Incident Investigation
• Databricks Documentation - Extend Genie Code with agent skills