Voice Live and Observability for Production Agent Systems Part 5/5

This is the fifth and final post in our series on the Microsoft agent platform. We cover two critical production concerns: Azure AI Voice Live for accessible, hands-free agent interaction, and observability, the tracing, evaluation, and monitoring infrastructure that keeps autonomous systems accountable.

All examples reference the FibreOps repository, demonstrated in Microsoft Build BRK241.

Azure AI Voice Live Integration

Voice Live brings spoken status updates to agent systems. In a Network Operations Center, operators may be focused on screens, coordinating on radio, or moving between stations, spoken updates provide an accessible, ambient awareness channel without requiring visual attention.

How FibreOps Uses Voice

The system speaks status updates through Azure AI Voice Live integration with Foundry Agent Service. Each update is built as an SSML utterance with voice and prosody chosen per severity level:

# src/fibreops/tools/voice.py — simplified
def speak_status_update(
    incident_id: str,
    node_id: str,
    severity: str,
    message: str,
    *,
    phrase_type: str = "outage_detected",
) -> dict:
    """Speak a status update through Azure AI Voice Live.
    
    Voice and prosody are selected based on severity:
    - critical: en-GB-RyanNeural, rate slow, pitch low
    - major: en-GB-SoniaNeural, rate medium
    - minor: en-GB-LibbyNeural, rate normal
    
    Falls back to state/voice_outbox.jsonl when endpoint is unset.
    """
    ssml = build_ssml(message, severity)
    
    if config.azure_voice_live_endpoint:
        response = requests.post(
            config.azure_voice_live_endpoint,
            json={"voice": get_voice(severity), "ssml": ssml, "text": message},
            headers={"Ocp-Apim-Subscription-Key": config.azure_voice_live_api_key},
        )
        return {"status": "spoken", "incident_id": incident_id}
    else:
        append_to_voice_outbox(incident_id, severity, ssml, message)
        return {"status": "queued_offline", "incident_id": incident_id}

Voice Behaviour in the System

• The "Speak status" button in the NOC console speaks the latest incident — uses the engineer_dispatched phrase when dispatch is complete, otherwise outage_detected.
• Setting FIBREOPS_VOICE_UPDATES=1 causes the NetOps and Field Dispatch agents to emit voice updates automatically at each milestone.
• The Voice Live pane in the UI shows the rolling outbox (voice, transcript, incident id, timestamp).

Configuration

Design Principles for Voice in Agent Systems

• Severity-appropriate delivery — Critical incidents use slower speech with lower pitch to convey urgency without panic. Minor issues use conversational tone.
• Concise utterances — Voice updates are short and structured: incident ID, node, severity, action taken. No verbose explanations.
• Non-blocking — Voice is always fire-and-forget. If the endpoint is unavailable, the update is queued locally.
• Offline capability — The voice outbox (state/voice_outbox.jsonl) captures everything for replay or review.

Observability Architecture

Autonomous agents must be observable. When an agent makes a decision — classifying an incident as critical, dispatching a specific engineer, or escalating to human review — that decision must be traceable, auditable, and evaluable.

The Observability Stack

FibreOps uses a layered observability approach:

1. Structured JSON logs — Every component emits structured logs with correlation IDs.
2. OpenTelemetry spans — Each agent decision, tool call, and external service interaction produces a span.
3. Local trace persistence — Spans are written to state/traces.jsonl for offline inspection.
4. Application Insights — Set APPLICATIONINSIGHTS_CONNECTION_STRING to ship everything to Azure Monitor.

# src/fibreops/observability.py — simplified
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

tracer = trace.get_tracer("fibreops")

class JsonFileExporter:
    """Export spans to state/traces.jsonl for offline inspection."""
    def export(self, spans):
        with open("state/traces.jsonl", "a") as f:
            for span in spans:
                f.write(json.dumps(span_to_dict(span)) + "\n")

# When Application Insights is configured, add the Azure exporter
if config.applicationinsights_connection_string:
    from azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter
    provider.add_span_processor(
        SimpleSpanProcessor(AzureMonitorTraceExporter(
            connection_string=config.applicationinsights_connection_string
        ))
    )

What Gets Traced

Every meaningful operation produces a span:

KQL Queries for Agent Operations

The repository includes paste-ready KQL queries in docs/KQL.md. These are designed for the operations team to answer common questions about agent behaviour in production.

Agent Decision Timeline

// Full decision timeline for a specific incident
traces
| where customDimensions.incident_id == "INC-2026-001"
| project timestamp, name, 
    agent = tostring(customDimensions.agent),
    decision = tostring(customDimensions.decision),
    duration_ms = duration
| order by timestamp asc

Per-Agent Latency

// Latency percentiles by agent role
traces
| where name startswith "agent."
| summarize 
    p50 = percentile(duration, 50),
    p95 = percentile(duration, 95),
    p99 = percentile(duration, 99),
    count = count()
    by agent = tostring(customDimensions.agent)
| order by p95 desc

Optimizer Score Trend

// Track optimizer scores over time to detect regression
traces
| where name == "optimiser.score"
| project timestamp, 
    score = todouble(customDimensions.score),
    run_id = tostring(customDimensions.run_id)
| render timechart

Dispatch SLA Compliance

// Percentage of dispatches within SLA by severity
traces
| where name == "agent.field_dispatch"
| extend severity = tostring(customDimensions.severity),
    eta_minutes = todouble(customDimensions.eta_minutes),
    sla_minutes = case(
        severity == "critical", 30.0,
        severity == "major", 60.0,
        120.0
    )
| summarize 
    total = count(),
    within_sla = countif(eta_minutes <= sla_minutes)
    by severity
| extend compliance_pct = round(100.0 * within_sla / total, 1)

Full Per-Incident Trace Replay

// Complete trace for incident replay and post-mortem
traces
| where customDimensions.run_id == "run-abc-123"
| project timestamp, name, duration,
    agent = tostring(customDimensions.agent),
    tool = tostring(customDimensions.tool),
    input = tostring(customDimensions.input),
    output = tostring(customDimensions.output)
| order by timestamp asc

The NOC Console: Operational Visibility

The FibreOps NOC console (python -m fibreops.demo ui) provides a real-time operational dashboard backed by the same trace and state files:

• KPI wallboard — 8 tactical tiles: incidents 24h, critical count (with alarm pulse), customers impacted, engineers dispatched, Foundry IQ lookups, Teams cards posted, optimizer average score, system health.
• Runs panel — Live list of agent runs with severity LED, node ID, engineer, ETA.
• Detail panel — Full agent decision timeline (Incident Analysis → NetOps → Field Dispatch).
• Topology grid — Nodes coloured by severity with dispatched outline.
• Optimizer panel — Average rubric score, per-criterion bars, top suggestions.
• Teams panel — Flattened Adaptive Card preview.
• Voice panel — Voice Live outbox (utterance, voice, severity).
• IQ panel — Foundry IQ, Web IQ, and Work IQ grounding lookups.

Responsible AI in Production

Operating autonomous agents in production requires deliberate governance:

Evaluation and Scoring

The optimizer evaluates every run against a rubric. This is not optional — it runs automatically after each batch. Criteria include:

• Classification accuracy — Did the agent correctly identify severity and root cause?
• Dispatch appropriateness — Was the right engineer selected for the fault type?
• SLA compliance — Is the estimated resolution time within service level targets?
• Communication quality — Are notifications clear, actionable, and appropriate?

Human-in-the-Loop Escalation

The Routine decision logic includes explicit escalation paths. When severity exceeds thresholds or the agent's confidence is low, the system hands off to a human operator rather than proceeding autonomously.

Audit Trail

Every decision is traced — who (which agent), what (which tool calls), why (the reasoning context), and when (timestamped spans). This trail is immutable once written to Application Insights, providing a compliance-ready audit log.

Putting It All Together: Production Deployment Checklist

1. Deploy infrastructure — azd up provisions all Azure resources.
2. Grant managed identity roles — Run scripts/grant-mi-roles.ps1 (requires Owner).
3. Publish hosted agents — python -m fibreops.demo publish or set FIBREOPS_DEPLOY_HOSTED=true.
4. Configure observability — Set APPLICATIONINSIGHTS_CONNECTION_STRING.
5. Configure voice — Set AZURE_VOICE_LIVE_ENDPOINT if voice updates are desired.
6. Configure Teams — Set TEAMS_WEBHOOK_URL for real-time notifications.
7. Publish to M365 — python -m fibreops.demo publish-m365 and upload to Teams Admin Center.
8. Monitor — Use the KQL queries and NOC console to track agent performance.
9. Optimise — Review optimizer suggestions and iterate on prompts and tool logic.

Key Takeaways

• Azure AI Voice Live provides accessible, severity-aware spoken updates for agent systems.
• OpenTelemetry tracing captures every agent decision for auditing and debugging.
• Application Insights + KQL gives operations teams paste-ready queries for common questions.
• The optimizer provides continuous, automated evaluation — not just logging, but scoring.
• The NOC console aggregates all observability data into a single tactical dashboard.
• Responsible AI requires evaluation, escalation paths, and immutable audit trails — not just good intentions.

Series Summary

Across five posts, we have walked through the complete Microsoft agent platform:

1. Overview — The Build → Run → Distribute story and FibreOps as reference implementation
2. Build — Microsoft Agent Framework, GitHub Copilot SDK, tool design, and testing
3. Run — Hosted Agents, Optimizer, Routines, Memory, Toolboxes, and Tracing
4. Distribute — Publishing to Teams, M365 Copilot, declarative agents, and Autopilots
5. Operate — Voice Live, observability, KQL, responsible AI, and production readiness

The platform is now GA. The FibreOps repository provides a complete, runnable reference for every feature discussed.

Next Steps

• Explore the FibreOps repository on GitHub
• Microsoft Foundry Agent Service documentation
• Azure AI Speech documentation
• Application Insights documentation
• KQL reference