Blog Post

Azure Observability Blog
6 MIN READ

Inside the Observability Agent: How Deep Investigations and Reasoning Work

yalavi's avatar
yalavi
Icon for Microsoft rankMicrosoft
Jun 15, 2026

Today, investigating incidents often means navigating multiple tools, forming hypotheses manually, and guessing which signals matter. Deep investigation replaces that process with a structured system that explores multiple explanations, correlates signals across layers, and produces a data-backed root cause analysis.

Incidents are rarely explained by one chart. A spike in request failures might be caused by a dependency timeout. A dependency timeout might come from a connection pool issue. And that connection pool issue might trace back to a deployment or a resource health event that started minutes earlier.

From the outside, Azure Copilot observability agent presents this as a focused troubleshooting experience that correlates signals such as metrics, logs, alerts (and alert context), tracing data, and resource and service health.

Under the hood, the implementation is more than a chat prompt with a few queries attached. It is a dedicated investigation runtime designed to build hypotheses, gather data across layers, and return an explanation that is meant to be understandable and data-backed.

Deep Investigation Is a Dedicated Investigation Run

A deep investigation runs as its own scoped investigation session anchored to a clear goal, a fixed incident time, a set of target resources, and the user’s Azure context, so it can query the right telemetry and platform data without losing the problem frame. Beyond goal, time, and scope, the agent also folds in the prior conversation, such as symptoms shared, suspected causes, and alerts that came up, so the run starts already aware of context.

That isolation matters for two reasons:

  • Stable problem frame: Anchored to a specific incident window and resource scope, which keeps analysis from drifting into generic commentary.
  • Room to do real work: Can take multiple steps to collect data, explore alternatives, and stream progress while the focused run continues.

In other words, the product feels conversational, but the runtime behaves like an agentic investigation engine.

Learning the Resource Context

Today the agent builds and persists a resource context per resource, learning each resource’s configuration, normal behavior, and relationships and reusing that context across investigations, so it does not start from scratch every time.

The Core Method: Think, Act, Review

Deep investigation is organized around a think-act-review loop.

The agent starts by generating multiple plausible leads instead of jumping to a single answer. It is explicitly instructed to consider several hypotheses, investigate each one, and rule out alternatives with evidence. That matters in observability, because the first suspicious signal is often a symptom rather than the cause.

The loop works like this:

  1. Think: Form hypotheses about what could explain the incident.
  2. Act: Use the available tools to gather telemetry and platform data.
  3. Review: Critique progress, identify gaps, and decide whether to continue digging.
  4. Repeat: Refine the next set of questions until the root cause is well supported, or uncertainty is clearly stated.

This is a meaningful design choice. Many AI troubleshooting experiences fail because they are optimized for fluency instead of investigation discipline. Here, the agent is pushed toward structured reasoning: identify leads, test them, reject weak ones, and only then conclude.

Success is defined as more than a plausible answer. The investigation aims to produce:

  • Incident framing: onset time, impact window, and blast radius.
  • Affected components: services, operations/endpoints, and dependencies.
  • Alternative hypotheses: what was considered and why it was ruled out.
  • Mitigation steps: recommended actions and how to validate recovery.
  • Support data: metric anomalies, resource health, configuration changes, and alerts as standard inputs.

Signals from Each Layer

The most important thing deep investigation does is correlate signals across layers that usually live in different tools.

Application layer

At the application layer, the agent can query Application Insights for requests, exceptions, dependencies, traces, custom events, and performance counters. That gives it the immediate symptom picture:

  • What failed?
  • Which operations or endpoints were affected?
  • Which dependencies slowed down or timed out?
  • How many users were impacted?
  • Did the issue hit one role or several?

This is where the incident becomes concrete. Instead of saying “latency increased,” the system can identify the deployment version, operation, the failing dependency, the time window, and the scale of impact.

Infrastructure & Platform layer

At the infrastructure & platform layer, the agent looks at Azure Monitor metrics, Log Analytics, resource changes, alerts, Azure Resource Graph metadata, and Azure Resource Health.

This is where the investigation shifts from “what broke” to “what changed” and “what else moved with it.”

  • A quick ML-based anomaly screen for resource metrics.
  • Deeper metric inspection using discovered metric definitions, metric values, and dimension values.
  • Resource change analysis to identify configuration, deployment, scale, or network changes.
  • Alerts as supporting data.
  • Azure Resource Graph queries for topology and configuration context.
  • Azure Resource Health and Azure Service Health to detect resource-level degradations and broader service incidents.

Together, these checks help answer not just “what moved,” but whether that movement plausibly explains the symptom.

That is an important characteristic of a mature investigation system: it does not confuse “not flagged” with “healthy.”

Correlating Signals Across Layers

The differentiator is not any single tool. It is the way the system combines signals from different layers into a single causal story.

Deep investigation is designed to align data along three axes:

  • Time: Did the signal move at the same time as the incident?
  • Scope: Did it affect the same resource, dependency, operation, or region?
  • Type: Does the signal match the observed symptoms?

That may sound simple, but it is what turns observability data into an explanation.

  • Application telemetry: Dependency timeout pattern.
  • Infrastructure logs: Connection pool exhaustion.
  • Platform data: Deployment or configuration change shortly before failures began.
  • Resource Health: No degradation reported.

Individually, each signal is suggestive. Together, they can support a much stronger conclusion: the problem likely came from an application or configuration change, not from a broader Azure platform outage.

The same logic works in the other direction. If application failures start at the same time a resource becomes degraded in Resource Health, metrics show a sharp availability drop, and alerts fire on the same target, the agent has a much stronger basis for identifying a platform-level contributing factor.

This is also why alerts are treated as supporting data instead of root cause by themselves: they indicate a threshold breach, but not why it happened. Deep investigation uses alerts as one input into a larger causal picture.

Explainability Is Built into the Runtime

The observability agent is built to explain its reasoning.

Layer 1: Live progress. As the deep investigation runs, the runtime captures reasoning updates and tool outputs and streams lightweight progress steps back into the parent conversation, where users can follow them in the Chain-of-Thought (CoT) view as the investigation unfolds.

Layer 2: Final narrative. The investigation produces a customer-facing report that separates What happened, Analysis, and What can be done next, which keeps symptoms, causes, and mitigations distinct.

Layer 3: Dashboard contract. The narrative is transformed into a structured dashboard object (summary, impact time, severity, findings, supporting data). Data cards are expected to be telemetry-backed artifacts that explain timeline, blast radius, dependency health, error mix, saturation signals, change correlation, and mitigation validation.

This structure matters because it keeps symptoms, causes, and mitigations distinct, making the outcome easier to read, easier to validate, and easier to act on.

In other words, explainability is not just “show the answer.” It is “show the data structure behind the answer.”

A Concrete Example

Imagine an incident where an API’s latency jumps from hundreds of milliseconds to several seconds.

Deep investigation reconstructs the chain of events:

  • Application layer: Latency spike concentrated in one operation, coinciding with dependency timeouts to a downstream service.
  • Infrastructure layer: Downstream logs show connection pool exhaustion messages or restart behavior.
  • Platform layer: A configuration or deployment change appears a few minutes before the incident; detailed metrics and Resource Health checks help confirm timing and rule out broader platform degradation.

Now the system can do something more useful than saying “latency is high.” It can tell a connected story:

  1. The incident began at a specific time.
  2. It affected a specific set of operations and users.
  3. The API slowdown was downstream-driven.
  4. The downstream failures started after a configuration change reduced available capacity.
  5. No platform health event explains the timing, which makes an application or config-level cause more likely.
  6. The recommended next step is to revert the change, validate recovery, and monitor the affected metrics.

That is the difference between telemetry retrieval and root cause analysis.

What Makes This Approach Valuable

The interesting part of deep investigation is not that it can query logs, metrics, or alerts. Plenty of systems can do that.

The interesting part is that it treats investigation as a control loop with data gates:

  • Form multiple leads.
  • Check across layers.
  • Correlate by time and scope.
  • Rule out healthy resources.
  • Treat weak data as weak.
  • Turn the result into an explainable output.

That combination is what makes the experience useful for SREs and developers under pressure. During an incident, teams do not just need more telemetry. They need a system that can reduce ambiguity without hiding data.

That is what the deep investigation runtime in the observability agent is trying to deliver.

Learn More

Updated Jun 10, 2026
Version 1.0
No CommentsBe the first to comment