Voice agents are changing how companies interact with customers. From service questions to technical support, the expectation is the same: a natural conversation that leads to useful, reliable answers. But responding quickly and resolving a complex request are not always the same task.

Some questions can be answered directly. Others require information from backend systems, specialized analysis, or several steps to determine what should happen next. The challenge is to support that deeper work without putting the conversation on hold, or giving an answer before the necessary information is available.

In this blog, I will walk through a dual-mode voice-agent pattern in Microsoft Foundry. A frontline agent manages the real-time conversation, while a backend agent handles specialized work. We will explore how these responsibilities work together, how the frontline can continue interacting while backend processing runs, and how to configure and verify the experience. The pattern is broadly applicable, with particular relevance to telecom customer service and support.

You will learn how to:

• Separate real-time conversation from specialized backend work.
• Configure a native voice subagent

• Understand how Voice Agents build on Voice Live

• Test delayed responses, interruptions, missing information, and backend failures.

How voice agents work

Before we look at our full workflow. Lets have a look at the voice experience in Microsoft Foundry. A voice application needs to carry audio, manage the conversation, and connect that conversation to useful information or actions. Those responsibilities work together, but they are not the same layer.

• A browser or contact-center connector handles audio capture and playback
• The voice runtime manages streaming and turn-taking
• Instructions, knowledge, and authorized tools determine what the agent can say and do.

In Microsoft Foundry, Voice Live provides the real-time speech foundation. Voice Agents build on it with a named, versioned agent definition and capabilities such as tools, tracing, and evaluation. The diagram below shows the underlying voice pipeline. 

1. Audio arrives from the client
2. Before anything else, Voice Live cleans up that audio. It uses conversational enhancements that can reduce background noise and echo and help detect when a speaker has finished a turn. The exact combination depends on the selected model, pipeline, and audio settings; not every session uses all the processing stages shown

3. The conversation can then either

   1. follow a native speech-to-speech path. A model in the GPT Realtime family processes spoken input directly and produces a spoken response. Supported configurations can also use Azure Speech voices for output, so choosing a realtime model does not mean we must use its native output voice.

   2. Alternatively, a cascaded pipeline converts speech to text, sends that text to a language model or agent, and synthesizes the response as speech. This separates the recognition, conversation model, and voice choices. Voice Live can manage that pipeline without us writing the streaming and turn-taking loop ourselves.

4. In either path, useful answers may also require knowledge sources or external tools. These connections give the agent access to information and actions beyond the conversation itself. Configure the required permissions and safety controls for the selected model and application; a tool or content-safety box in the diagram is not a substitute for those decisions.

5. On the return path, speech streams back to the client for playback. Where supported, Azure Speech provides standard and HD voices, with custom voice subject to access requirements.

These choices describe how the spoken conversation runs. The dual-mode pattern adds a different separation: the frontline owns that conversation, while a backend agent handles a focused task. It does not require two speech-to-speech models, and the backend does not speak directly to the caller.

How the frontline and backend work together

To keep the conversation responsive while specialized work runs, we separate the caller-facing interaction from the backend task. Each agent has a clear responsibility:

• The frontline agent is responsible for the interaction. It listens, takes turns, answers supported questions, and explains what happens next.
• The backend agent has a different job: work through a focused task, such as reviewing incident evidence or recommending a troubleshooting sequence.

The important part is that delegation does not have to take the caller out of the conversation. The frontline can remain active while the backend works. In the native subagent pattern, this behavior is configured through the agent definitions, instructions, and response policy rather than a separate routing model

The following diagram shows how those responsibilities fit together:

 

1. The voice client captures and streams the caller’s audio: The client connects the caller to the frontline voice agent, sends microphone audio, and plays returned speech. It can also display transcripts and runtime events. The client can send audio and receive events concurrently.

2. The frontline voice agent decides whether to answer directly or delegate: The frontline owns the live conversation. It handles straightforward requests within its configured knowledge and permissions without invoking the backend. When a request needs specialized work, the frontline uses its instructions and the configured subagent’s description to determine when to delegate. Direct answers return through the voice client without a backend invocation.

3. Foundry invokes the configured backend subagent. The voice agent’s subagent_config references a backend in the same Foundry project by name. Foundry invokes that backend for the delegated task. The frontline retains ownership of the voice conversation; the caller is not transferred to the backend.

4.  The backend performs the task and uses its tools. A prompt or hosted agent follows its own instructions and can retrieve data, call APIs, or execute supported functions. Access to service information, incident records, or diagnostics must be explicitly configured and authorized. 

5. Meanwhile, the frontline keeps the voice interaction active. Its response_policy controls an immediate acknowledgment, gap-filling speech, and supported progress updates. The basic sample enables only immediate_ack=True; additional behavior requires configuration. The frontline can receive further caller input, but it must not invent backend findings while waiting. Further caller input does not automatically update an in-flight backend task.

6. Results and supported progress return through Foundry. The backend returns its output to the frontline through managed delegation. Where supported and enabled, progress can also inform the conversation before completion. The diagram’s return arrow does not mean every backend automatically streams intermediate findings.

7. The frontline presents the result to the caller. Following its instructions, it explains the findings, communicates uncertainty, and identifies appropriate next actions. Speech streams back through the client for playback. The backend does not speak directly to the caller, and its output is not necessarily identical to the spoken response.

Note: The GPT Live samples demonstrate an alternative implementation of the conversation/backend split. They use VoiceAgentLiveDefinition and live.connect(), with direct Responses or managed prompt-agent delegation. 

The design goal is responsiveness without sacrificing depth. A fast acknowledgment improves perceived waiting time; it does not shorten the underlying investigation or guarantee an accurate answer. Measure conversational responsiveness and backend task quality separately.

Decide when the frontline should ask for help

With the flow in place, the next question is when the frontline should ask for help. Define clear boundaries in the frontline agent’s instructions and the subagent’s capability description.

• Rule 1: Answer directly when the frontline has sufficient grounded information and permission to respond.
• Rule 2: Delegate when the backend owns the required capability. This might include combining information from several systems, assessing service impact, investigating symptoms, or developing a troubleshooting plan. 
• Rule 3: Escalate when the request exceeds the automated workflow’s boundaries. Define what should happen when an action requires human approval, evidence remains insufficient, or a tool failure prevents progress. Implement and test the human-handoff or failure path; instructions alone do not provide it.

From the caller’s perspective, this should feel like one conversation, not a handoff between agents. The frontline listens, clarifies the request, and brings in specialized help when needed. The caller does not need to understand which backend agent or tool is doing the work.

When backend processing begins, an enabled acknowledgment can make that transition clear: “Let me check that for you.” If gap-filling speech or supported progress updates are configured, the frontline can keep the caller informed while the task runs. Those updates should reflect what is actually known, without implying that a check has completed or a result has been confirmed.

Once the backend returns, the frontline explains the outcome in language suited to the conversation. That might be a confirmed finding, a recommended next step, or a clear statement that more information is needed. The goal is not to fill every pause with speech. It is to keep the caller oriented while specialized work happens in the background, and to make the eventual answer useful, understandable, and grounded in evidence.

Build the native voice-subagent workflow

Now that we have defined the conversation flow, let’s configure the agents behind it. We will use native voice subagents throughout this walkthrough: a VoiceAgentDefinition for the frontline, a prompt agent for the backend, and subagent_config to connect them. 

Voice subagents are currently in preview. Microsoft does not recommend features in preview for production workloads.

Step 1: Give the backend one focused responsibility

Start with the work the frontline should delegate. A useful backend has a clear responsibility. The subagent can be a Foundry prompt agent or a hosted agent in the same project.

For our example here we will create a prompt agent that can analyze information supplied to it. It can be connected to knowledge bases and different tools to help.

You can also create a subagent in the Microsoft Foundry portal or deploy a code-based hosted agent.

Keep the instructions specific to the backend’s actual capabilities. If it can only analyze supplied information, say so. If you later add an incident lookup, update the instructions and capability description to reflect that change.

Instructions guide model behavior; they do not enforce permissions or guarantee an output schema. Restrict access in the tools and services themselves, and validate any structured fields your application relies on. Separate read-only investigation from actions that change a customer account or service.

Step 2: Connect the backend to the frontline

Next, we can create the caller-facing voice-based agent. We can create this using the SDK. In its instructions we should explain when to delegate to the backend agent and we should add the backend agent to its subagent_config. In the subagent_config, you identify the backend agent by name and describe when the voice-based agent should delegate a request to it in the agent_capabilities, which describes the specialist’s responsibility. 

You can use the following code to spin up such an agent (Also see: Use a subagent in a Voice-based agent - Microsoft Foundry | Microsoft Learn) 

import os

from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
    RealtimeAudioFormatsAudioPcm,
    VoiceAgentAudioConfig,
    VoiceAgentAudioInputConfig,
    VoiceAgentAudioOutputConfig,
    VoiceAgentDefinition,
    VoiceAgentInputTranscription,
    VoiceAgentServerVadTurnDetection,
    VoiceAgentSubagent,
    VoiceAgentSubagentConfig,
    VoiceAgentSubagentResponsePolicy,
)
from azure.identity import DefaultAzureCredential

endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
model = os.environ.get("FOUNDRY_VOICE_MODEL", "gpt-realtime-2.1")

definition = VoiceAgentDefinition(
    model_type="managed",
    model=model,
    instructions=(
        "You are a real-time customer support assistant for service incidents. "
        "Handle straightforward factual and support-process questions directly. "
        "For complex troubleshooting, SLA-risk triage, root-cause hypotheses, "
        "or multi-step remediation planning, call forward_to_subagent for "
        "the configured telco support agent. "
        "After the subagent returns, provide a concise, customer-safe summary "
        "and next actions. "
        "Do not present uncertain reasoning as confirmed fact."
    ),
    store=True,
    output_modalities=["audio"],
    audio=VoiceAgentAudioConfig(
        input=VoiceAgentAudioInputConfig(
            format=RealtimeAudioFormatsAudioPcm(rate=24000),
            turn_detection=VoiceAgentServerVadTurnDetection(
                threshold=0.5,
                prefix_padding_ms=300,
                silence_duration_ms=1000,
            ),
            transcription=VoiceAgentInputTranscription(
                model="azure-speech",
            ),
        ),
        output=VoiceAgentAudioOutputConfig(
            format=RealtimeAudioFormatsAudioPcm(rate=24000),
            voice="en-US-JennyNeural",
            voice_type="azure-standard",
        ),
    ),
    subagent_config=VoiceAgentSubagentConfig(
        subagents=[
            VoiceAgentSubagent(
                agent_name="[BACKEND AGENT NAME]",
                agent_capabilities=(
                    "Performs deeper SLA-risk triage and troubleshooting "
                    "reasoning, including structured cause hypotheses "
                    "and remediation planning."
                ),
                response_policy=VoiceAgentSubagentResponsePolicy(
                    immediate_ack=True,
                ),
            )
        ],
    ),
)

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(
        endpoint=endpoint,
        credential=credential,
        allow_preview=True,
    ) as project_client,
):
    voice_agent = project_client.agents.create_version(
        agent_name="frontline-voice-agent",
        definition=definition,
        description="Voice frontline with subagent delegation",
    )
    print(
        f"Created voice agent '{voice_agent.name}', "
        f"version {voice_agent.version}"
    )

 

While you currently can't link these subagents through the portal You can see the subagents linked and its instructions in the portal by clicking on the YAML button. 

Step 3: Configure what the caller hears while waiting

With the basic connection in place, we can decide how the frontline should support the caller during backend work.  

Use the optional response_policy object to control what the voice agent says while it waits for a subagent:

• immediate_ack: Specifies whether the voice agent immediately acknowledges that it forwarded the request.

• ack_instructions Provides instructions for generating the immediate acknowledgment.

• enable_delta_progress Emits progress incrementally instead of waiting for the subagent invocation to finish. The default is false.

• gap_filling_interval Sets the period without subagent content or user input before a gap-filling response. Use a value from 5 through 120 seconds.

• gap_filling_instructions Provides instructions for speech generated while the voice agent waits for progress.

• progress_instructions Provides instructions for summarizing streamed subagent progress as speech.

• progress_update_interval Sets the minimum time between spoken progress updates. Use a value from 5 through 300 seconds.

Step 4: Start a conversation with the voice agent

With the frontline and its subagent configured, we can now test the spoken experience. The quickest starting point is the Foundry playground: open the project containing both agents, go to Build → Agents, select frontline-voice-agent, and open it in the playground. Start a voice session and allow microphone access. This tests the saved agent configuration without building a local audio client.

For a custom application, we can use the shared native microphone client linked here (Native voice client: dependencies, launch commands, and event handling) or the complete client in the Microsoft Learn walkthrough. Configure it with the same Foundry project endpoint and the name created in Step 2: frontline-voice-agent. The client connects through beta.voice_agents.realtime.connect(), sends microphone audio, and receives speech and events concurrently.

The example uses 24 kHz PCM audio and plays the returned audio chunks as they arrive. Follow the setup and readiness sequence of the client we choose rather than combining snippets from different clients. Voice activity detection identifies speech boundaries; the frontline instructions and subagent configuration still determine when to delegate.

Start with a general support question, then ask for analysis that should use the backend. Check the execution evidence as well as the spoken answer: an acknowledgment alone does not prove that the subagent ran. In our current configuration, only immediate_ack is enabled, so additional gap filling or progress narration should not be assumed. Use headphones to prevent speaker audio from feeding back into the microphone.

Then introduce a delayed response, a caller correction, missing evidence, and a backend failure. Check how the frontline responds and what the execution trace shows. Test interruptions separately: stopping audio playback does not prove that the backend task was canceled.

From a microphone to a contact-center call

The microphone client helps us validate the agent. Connecting it to a contact center adds another layer: a channel integration that bridges call audio and call-control events to the voice runtime. The division between frontline and backend remains useful, but a working local conversation does not automatically provide telephony routing or human transfer. 

The Azure-Samples Call Center Voice Agent Accelerator documents browser testing and multiple telephony providers, including Genesys Cloud, Azure Communication Services, and Twilio. It provides a reference for the channel-integration layer.

There are also call-center-specific accelerators e.g. This Genesys Voice Live Connector sample. This integration option uses Genesys Cloud AudioConnector and a connector service between Genesys and Azure Voice Live. The connector handles the channel protocol and audio conversion; the agent handles the conversation and its configured tools.

Conclusion

A dual-mode voice agent separates responsibilities that operate on different timelines. The frontline keeps the conversation clear and responsive; the backend performs specialized work. Native subagent delegation connects them without handing the caller over to another speaking agent.

The value is not constant speech or an instant answer to every request. It is an interaction that stays understandable while the necessary work happens, then delivers a response grounded in what that work actually established.

Next Steps

Ready to go deeper? 

• Use a subagent in a Voice-based agent (preview) – Follow the end-to-end walkthrough to create a backend subagent, configure delegation, and test the spoken experience.
• Configure a voice agent – Explore model selection, audio settings, turn detection, voices, and agent behavior.
• Native voice-subagent samples – Explore working examples that connect a frontline voice agent to prompt or hosted backend agents.
• Voice Live overview – Learn about the underlying speech capabilities, supported models, and regional availability.
• Genesys Cloud Voice Live Connector – Explore a sample audio bridge between Genesys Cloud and Azure Voice Live, with a simulator for local testing.
• Call Center Voice Agent Accelerator – Explore deployment guidance and reference integrations for browser-based conversations and supported telephony providers.