Choose the right speech and voice model in Microsoft Foundry

The practical approach to selecting the right model is simple: do not ask which speech model is best. Ask what the speech system must guarantee. Once you know whether the job is conversation, translation, transcription, diarization, synthesis, or managed voice-agent orchestration, the model choice becomes much clearer.

Choosing the wrong speech model usually does not fail in a benchmark. It fails in production: a customer interrupts and the agent keeps talking, a meeting transcript has no speaker labels, a translation workflow adds too much delay, or a text-to-speech pipeline sounds good but cannot use the voice your product needs.

Microsoft Foundry now gives builders several speech and voice paths across Azure OpenAI, Voice Live API, Azure Speech in Foundry Tools, and Microsoft AI (MAI) speech models. This guide helps you choose the right surface for the job before you commit to region, quota, pricing, or architecture assumptions.

TL;DR: pick by job, not by model name

• Use Azure OpenAI Realtime API when you want direct, model-native speech-to-speech control over WebSocket, WebRTC, or SIP.
• Use Voice Live API when you want a managed voice-agent stack with speech recognition, model reasoning, text-to-speech, turn detection, interruption handling, voices, tools, and optional avatars.
• Use Azure Speech realtime or batch speech-to-text (STT) when the transcript is the artifact of record.
• Use a diarize or Azure Speech batch when speaker labels matter (Who said what).
• Use Azure Speech TTS, Custom Neural Voice, gpt-4o-mini-tts, or MAI-Voice-2 when voice quality, identity, language coverage, or long-form narration matters.
• Use mai-transcribe-1.5 when world-class, low-cost multilingual STT with phrase/entity biasing is useful and diarization is not required.
• Validate region, deployment type, lifecycle, quota, and price in your tenant before treating any model choice as final.

A quick glossary

• STT: speech-to-text, also called transcription.
• TTS: text-to-speech, also called speech synthesis.
• VAD: voice activity detection, used to decide when a speaker has started or stopped talking.
• SIP: Session Initiation Protocol, commonly used for telephony integration.
• PTU: provisioned throughput units, a reserved-capacity deployment option where supported.

Start with the user experience

The first decision is not which model is smartest. It is what the system must produce and how fast it must respond.

If a user speaks to an agent and expects the agent to listen, reason, be interrupted, and speak back in real time, you are building a speech-to-speech system. Choose between direct Azure OpenAI Realtime API and managed Voice Live API.

If a user uploads or records audio and needs a transcript, you are building STT. Use transcription APIs or Azure Speech batch, not a conversational voice model.

If text goes in and speech comes out, you are building TTS. Use a TTS surface, not a speech-to-speech model.

If speech must be translated while the speaker is still talking, you are building live speech translation. That is different from file translation to English with Whisper.

If the app can wait for a normal request/response turn, turn-based audio chat is often simpler than a realtime session.

Scenario map

Interactive experiences

Speech artifacts and workflows

How the main surfaces work

Azure OpenAI Realtime API: direct model-native voice

Use the Azure OpenAI Realtime API when you want low-latency speech-to-speech and your app can manage the live session. The GA WebSocket path is:

wss://{resource}.openai.azure.com/openai/v1/realtime?model={deployment}

WebRTC uses /openai/v1/realtime/client_secrets and /openai/v1/realtime/calls. SIP call control uses /openai/v1/realtime/calls/{call_id}/.... Do not add date-based api-version parameters to the GA /openai/v1/realtime path. The Realtime API supports up to 32,000 input tokens and 4,096 output tokens.

Choose realtime path when you need tight control over audio buffering, session state, function calls, and telephony. Choose Voice Live instead when you want Azure Speech voices, semantic turn detection, custom speech, custom voice, avatar output, or broader model selection without wiring those pieces yourself.

Voice Live API: managed voice agents

Voice Live API is the managed speech-to-speech surface in Azure Speech in Foundry Tools. It combines STT, model reasoning, TTS, turn detection, interruption handling, noise suppression, echo cancellation, voices, avatars, and function calling behind one realtime interface.

The WebSocket endpoint shape is:

wss://{resource}.services.ai.azure.com/voice-live/realtime?api-version=2026-04-10&model={model}

Older resources can use:

wss://{resource}.cognitiveservices.azure.com/voice-live/realtime?api-version=2026-04-10&model={model

Voice Live can also connect to Microsoft Foundry Agent Service with agent_id and project_id parameters. Microsoft Entra authentication is recommended. API keys are also supported.

A typical Voice Live session starts with session.update:

{
  "type": "session.update",
  "session": {
    "instructions": "You are a concise customer support voice agent.",
    "modalities": ["text", "audio"],
    "turn_detection": {
      "type": "azure_semantic_vad",
      "silence_duration_ms": 500,
      "interrupt_response": true
    },
    "input_audio_noise_reduction": { "type": "azure_deep_noise_suppression" },
    "input_audio_echo_cancellation": { "type": "server_echo_cancellation" },
    "voice": {
      "name": "en-US-AvaNeural",
      "type": "azure-standard"
    }
  }
}

Voice Live is useful for contact centers, automotive assistants, accessibility experiences, public-service agents, learning companions, and any app where conversation quality depends on turn-taking as much as model intelligence.

Voice Live supports a broad model set: GPT-Realtime, GPT-4o, GPT-4.1, GPT-5 family models, Phi preview models, azure-realtime, Agent Service, and BYOM patterns.

Key tradeoffs:

• Voice Live has separate quota and pricing from Azure OpenAI deployment tables.
• Pricing is tiered by selected model: Pro, Basic, and Lite.
• Custom speech, custom voice, and avatar features can add training, hosting, or usage charges.
• Custom voice and avatar access can be limited or approval-gated.
• WebRTC support is preview.
• SIP is documented for Azure OpenAI Realtime API, but not directly supported in Voice Live, yet.

Azure Speech: the production speech service layer

Azure Speech in Foundry Tools is more than a model list. It is a service layer for realtime STT, batch STT, TTS, translation, custom speech, custom voice, privacy controls, and containers.

Use Azure Speech when the speech artifact matters: a durable transcript, a brand voice, a tuned recognizer, a controlled storage boundary, or a Speech SDK workflow.

For STT, Azure Speech supports realtime streaming and batch patterns. It can also support capabilities such as language detection, phrase hints, custom speech models, punctuation, capitalization, keyword spotting, pronunciation assessment, and diarization. Exact diarization limits and feature combinations should be verified in the current Speech docs for the selected API and region.

For TTS, Azure Speech is the broader production path for voice catalog depth, locale coverage, pronunciation control, and branded voice. Custom Neural Voice lets teams create a unique voice under limited-access controls.

For privacy-sensitive workloads, Azure Speech can help define the data boundary. Batch jobs can use customer-controlled Azure Storage for input and output. Speech containers can support customer-managed or on-premises processing for supported features. Validate the exact feature, container image, compliance scope, and region before making regulated production commitments.

Capability Matrix

Transcription, diarization, and captions

For files, use /audio/transcriptions rather than a speech-to-speech model. The core Azure OpenAI choices are whisper, gpt-4o-transcribe, gpt-4o-mini-transcribe, and gpt-4o-transcribe-diarize.

Use gpt-4o-transcribe when accuracy matters most, and gpt-4o-mini-transcribe when cost matters more. Use gpt-4o-transcribe-diarize when a bounded file needs speaker-aware output. It is not a Realtime API model.

Use Azure Speech batch when files are long, files exceed synchronous limits, word-level timestamps matter, or mature diarization workflows are required. This is also the right place to evaluate customer-controlled storage and retention.

Use gpt-realtime-whisper or Azure Speech realtime STT for live captions. Use Azure Speech realtime STT when the product needs phrase hints, custom speech tuning, language detection, punctuation, or other Speech service features.

Use mai-transcribe-1.5 when preview multilingual STT with phrase/entity biasing is useful and diarization is not required. Current research records 43 languages, WAV/MP3/FLAC input under 300 MB, optional language forcing, phrase lists, and a transcribeStyle choice for readable or verbatim output.

Text-to-speech and voice assets

If text goes in and speech comes out, start with a TTS surface.

Use gpt-4o-mini-tts when style instructions are important. It supports instructions for accent, emotion, intonation, speed, and similar output qualities. OpenAI docs list voices such as alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse, marin, and cedar.

Use legacy tts and tts-hd only with care.

Use Azure Speech TTS or Custom Neural Voice when voice identity, broad locale support, pronunciation control, or brand voice matters. Use Voice Live when speech output is part of a realtime agent. Use MAI-Voice-2 when expressive quality, long-form content, emotion/style, multi-speaker scenarios, or approved voice prompting matter more than ultra-low latency.

MAI-Voice-2 has 18+ languages and 5-60 second recommended reference clips.

For enterprise design, validate five things in the target tenant: region, deployment type, lifecycle, quota, and price. For Azure Speech features, also validate privacy posture, storage ownership, container support, and compliance scope.

Practical architecture patterns

Multilingual customer support

Use gpt-realtime-translate for live translated speech and translated transcript. Run gpt-realtime-whisper or Azure Speech realtime STT when you also need the original source transcript. Route complex resolution to gpt-realtime-2 when deeper reasoning matters. Use Azure Speech batch for durable compliance transcripts, diarization, or word-level timestamps.

Choose Voice Live when the agent needs managed turn detection, noise suppression, echo cancellation, Azure Speech voices, custom voice, function calling, or avatar output in one surface.

Contact-center voice agent

Use Voice Live API with GPT-Realtime, GPT-4o, GPT-4.1, GPT-5 family, Phi, or azure-realtime models. Configure semantic VAD, barge-in, noise suppression, echo cancellation, voice output, and tool calls for account lookup or workflow actions.

Use WebSocket for server-to-server integration. Evaluate WebRTC preview for browser or mobile client audio. Validate default quota, model tier, custom voice/avatar access, and region.

Meeting transcript and summary

Do not use a speech-to-speech model as the source of record. For live captions, use gpt-realtime-whisper or Azure Speech realtime STT. For the durable transcript, use Azure Speech batch when files are large, diarization is needed, or word-level timestamps matter. Use gpt-4o-transcribe-diarize for bounded files when its limits fit.

Voice-enabled chat feature

If the user records audio, submits it, and receives an audio answer after a normal turn, start with gpt-audio-mini or gpt-audio-1.5. It is simpler than a realtime session. Do not use this path if users expect live interruption.

Branded speech asset pipeline

Use gpt-4o-mini-tts when style instructions are useful. Use Azure Speech TTS, Custom Neural Voice, HD voices, or MAI-Voice-2 when voice identity, language coverage, pronunciation control, or long-form quality matters. Validate consent, access controls, voice availability, and pricing before launch.

Regulated transcript or on-prem speech

Start from the data boundary, not the model name. Azure Speech batch can keep input and output files in customer-controlled Azure Storage. Speech containers can support customer-managed or on-premises processing for supported features. Validate container availability, feature coverage, region, pricing, and compliance scope.

Next steps

If you are choosing a speech architecture now:

1. Prototype the transport first. Build the smallest WebSocket, WebRTC, Speech SDK streaming, REST transcription, TTS, container, or batch proof of concept that matches the user interaction.
2. Validate availability in your tenant. Check model region, deployment type, lifecycle, quota, price, resource type, and compliance posture.
3. Test the failure mode that matters. For voice agents, test interruption and noisy audio. For transcripts, test diarization and timestamps. For TTS, test pronunciation, voice identity, and long-form quality.

Source-of-truth docs

• Use the GPT Realtime API for speech and audio
• Voice Live API overview
• How to use the Voice Live API
• Voice Live language support
• Voice Live WebRTC preview
• Azure OpenAI audio generation quickstart
• Azure OpenAI REST API reference
• Azure OpenAI Whisper quickstart
• Create a batch transcription
• Speech translation overview
• MAI-Transcribe in LLM Speech API
• MAI-Voice
• Azure OpenAI pricing

Get started

• MAI Playground | Microsoft AI
• Foundry | Microsoft Foundry
• Spech Studio | Legacy Portal