A transcript can look complete while still missing what happened in a voice conversation. Transcription errors can alter words or omit details, so text alone might not fully represent the interaction. Evaluating the recording directly gives teams a view of the original conversation, including details the transcript may miss.
Authors: Sheng Liu, Justin Gonzalez, Shivank Goel, Morteza Ziyadi
Four audio evaluators for complete conversations
To address the need to evaluate spoken interactions directly, Microsoft Foundry introduces four conversation-level audio evaluators for existing call recordings: Task Completion Audio, Groundedness Audio, Customer Satisfaction Audio, and Coherence Audio. They evaluate the complete spoken interaction rather than isolated turns.
See audio evaluation in action
In this demo video, we show how to evaluate voice-agent conversations in Microsoft Foundry, from selecting an evaluator and audio judge to reviewing the results.
How audio evaluation works
Unlike a text evaluator, which scores the transcript and structured messages supplied to it, an audio evaluator can also consume the call recording. To use an audio evaluator, configure a supported audio-capable judge deployment, such as gpt-realtime-2.1, and set evaluation Scope to Full conversations. The evaluation input can also include additional textual messages, such as tool calls and tool results, when they are available.
Science validation
We measured two representative evaluators from this set to cover both task outcome and user experience.
|
Evaluator |
What it measures |
Output |
|
Task Completion Audio |
Whether the requested task was successfully completed end to end |
Boolean: 0 or 1 |
|
Customer Satisfaction Audio |
Holistic satisfaction across helpfulness, completeness, clarity, resolution, and adaptability |
Ordinal: 1 to 5 |
We evaluated more than 1,000 calls from the τ-Voice dataset, covering airline, retail, and telecom tasks across four voice model or agent configurations. We removed “invisible” cases, where the ground-truth pass label could not be determined reliably from the evidence available to the evaluator. We used gpt-realtime-1.5 as the audio-capable judge model.
Experiment results
Task completion.
We evaluated Task Completion Audio evaluator under two input settings: audio with tool calls and tool results, and audio with transcript (including tool calls and tool results). After visibility screening, each setting had more than 1,000 calls.
We compared the evaluator's predictions with the ground-truth pass labels in the τ-Voice dataset. We report accuracy, the overall percentage of correct predictions, and balanced accuracy, the average accuracy across ground-truth pass and fail cases so both outcomes receive equal weight. Audio plus tools achieved 79.8% accuracy and 79.7% balanced accuracy; audio plus transcript achieved 79.5% and 79.0%, respectively. The two settings produced similar aggregate results, with the audio-plus-tools setting slightly higher on both metrics.
Repeatability. To assess repeatability, we ran the Task Completion Audio evaluator twice under identical settings. Across more than 1,000 paired calls, the two runs agreed on 91.9% of verdicts, while 8.1% changed. Cohen's κ was 0.84. κ measures agreement after accounting for agreement that could occur by chance; it ranges from -1 to 1, and values closer to 1 indicate stronger repeatability. Accuracy was nearly unchanged at 79.7% in run 1 and 79.6% in run 2. Overall, these results suggest relatively high repeatability under the tested setup.
Customer Satisfaction. Customer satisfaction is subjective, and τ-Voice does not provide human-labeled satisfaction scores. We therefore compared the Customer Satisfaction Audio evaluator with the text Customer Satisfaction evaluator, which used gpt-6-luna as its judge. The two configurations used different judges and evaluator versions, so this analysis measures agreement between the tested configurations rather than an isolated audio-versus-text effect.
Across more than 1,000 paired calls, the two configurations assigned exactly the same 1 to 5 rating in 37.4% of cases and were within one point in 84.0%. Mean absolute error was 0.84, meaning their ratings differed by less than one point on average. Spearman ρ was 0.74 and Kendall τ-b was 0.67. Both are rank-correlation measures: they show whether the same conversations tend to receive higher or lower ratings in both configurations, and values closer to 1 indicate stronger positive correlation. Overall, the results suggest that the two configurations generally ranked conversations similarly.
Get started with audio evaluation
In Microsoft Foundry, set Scope to Full conversations; select Task Completion Audio, Groundedness Audio, Customer Satisfaction Audio, or Coherence Audio; and choose a supported audio-capable judge such as gpt-realtime-2.1. Use the demo above for a quick walkthrough. The conversation-level evaluation guide describes the general run workflow. Review each score together with its explanation.
Disclaimer: These benchmark results reflect the tested dataset and evaluation setup and are not a guarantee of real-world performance.