Blog Post

Microsoft Foundry Blog
3 MIN READ

Audio Evaluation for Voice Agents

nemo-nano's avatar
nemo-nano
Icon for Microsoft rankMicrosoft
Nov 18, 2026

A transcript can look complete while still missing what happened in a voice conversation. Transcription errors can alter words or omit details, so text alone might not fully represent the interaction. Evaluating the recording directly gives teams a view of the original conversation, including details the transcript may miss.

Authors: Sheng Liu, Justin Gonzalez, Shivank Goel, Morteza Ziyadi

Four audio evaluators for complete conversations

To address the need to evaluate spoken interactions directly, Microsoft Foundry introduces four conversation-level audio evaluators for existing call recordings: Task Completion Audio, Groundedness Audio, Customer Satisfaction Audio, and Coherence Audio. They evaluate the complete spoken interaction rather than isolated turns.

See audio evaluation in action

In this demo video, we show how to evaluate voice-agent conversations in Microsoft Foundry, from selecting an evaluator and audio judge to reviewing the results.

Watch the demo video

How audio evaluation works

Unlike a text evaluator, which scores the transcript and structured messages supplied to it, an audio evaluator can also consume the call recording. To use an audio evaluator, configure a supported audio-capable judge deployment, such as gpt-realtime-2.1, and set evaluation Scope to Full conversations. The evaluation input can also include additional textual messages, such as tool calls and tool results, when they are available.

Science validation

We measured two representative evaluators from this set to cover both task outcome and user experience.

Evaluator

What it measures

Output

Task Completion Audio

Whether the requested task was successfully completed end to end

Boolean: 0 or 1

Customer Satisfaction Audio

Holistic satisfaction across helpfulness, completeness, clarity, resolution, and adaptability

Ordinal: 1 to 5

We evaluated more than 1,000 calls from the τ-Voice dataset, covering airline, retail, and telecom tasks across four voice model or agent configurations. We removed “invisible” cases, where the ground-truth pass label could not be determined reliably from the evidence available to the evaluator. We used gpt-realtime-1.5 as the audio-capable judge model.

Experiment results

Task completion.

We evaluated Task Completion Audio evaluator under two input settings: audio with tool calls and tool results, and audio with transcript (including tool calls and tool results). After visibility screening, each setting had more than 1,000 calls.

We compared the evaluator's predictions with the ground-truth pass labels in the τ-Voice dataset. We report accuracy, the overall percentage of correct predictions, and balanced accuracy, the average accuracy across ground-truth pass and fail cases so both outcomes receive equal weight. Audio plus tools achieved 79.8% accuracy and 79.7% balanced accuracy; audio plus transcript achieved 79.5% and 79.0%, respectively. The two settings produced similar aggregate results, with the audio-plus-tools setting slightly higher on both metrics.

Repeatability. To assess repeatability, we ran the Task Completion Audio evaluator twice under identical settings. Across more than 1,000 paired calls, the two runs agreed on 91.9% of verdicts, while 8.1% changed. Cohen's κ was 0.84. κ measures agreement after accounting for agreement that could occur by chance; it ranges from -1 to 1, and values closer to 1 indicate stronger repeatability. Accuracy was nearly unchanged at 79.7% in run 1 and 79.6% in run 2. Overall, these results suggest relatively high repeatability under the tested setup.

Customer Satisfaction. Customer satisfaction is subjective, and τ-Voice does not provide human-labeled satisfaction scores. We therefore compared the Customer Satisfaction Audio evaluator with the text Customer Satisfaction evaluator, which used gpt-6-luna as its judge. The two configurations used different judges and evaluator versions, so this analysis measures agreement between the tested configurations rather than an isolated audio-versus-text effect.

Across more than 1,000 paired calls, the two configurations assigned exactly the same 1 to 5 rating in 37.4% of cases and were within one point in 84.0%. Mean absolute error was 0.84, meaning their ratings differed by less than one point on average. Spearman ρ was 0.74 and Kendall τ-b was 0.67. Both are rank-correlation measures: they show whether the same conversations tend to receive higher or lower ratings in both configurations, and values closer to 1 indicate stronger positive correlation. Overall, the results suggest that the two configurations generally ranked conversations similarly.

Get started with audio evaluation

In Microsoft Foundry, set Scope to Full conversations; select Task Completion Audio, Groundedness Audio, Customer Satisfaction Audio, or Coherence Audio; and choose a supported audio-capable judge such as gpt-realtime-2.1. Use the demo above for a quick walkthrough. The conversation-level evaluation guide describes the general run workflow. Review each score together with its explanation.

Disclaimer: These benchmark results reflect the tested dataset and evaluation setup and are not a guarantee of real-world performance.

Updated Nov 18, 2026
Version 2.0
No CommentsBe the first to comment