First published on TECHNET on Dec 07, 2012
Have you ever wanted to figure out why a user experienced poor call quality, but didn’t have access to the Lync Sever Monitoring Server reports to dig into the data? This article shows you how to use UCCAPI logging, Snooper, and XML Notepad to log, review, and analyze call data, when Lync Server 2010 or Lync Server 2013 monitoring reports aren’t available.
Author : Aaron Steele , Microsoft Senior Consultant
Publication date : December 10, 2012
Product version : Lync 2010 Client, Lync 2013 Client
Recently, I was helping a colleague troubleshoot a client’s audio quality issues and discovered that we didn’t have access to the Lync Server Monitoring reports. The first course of action was to install Monitoring Server Reports —Monitoring Server Reports is the best place to get diagnostic information and troubleshoot call quality issues.
For one-offs, it’s possible to gather the data locally and use Lync troubleshooting tools to diagnose call quality issues. This article explains how to gather the automatically generated Quality of Experience data found in client logs, and analyze that data to determine the source of the call quality problems.
To be clear, we’re not talking about tracing failed calls. If a call fails, use Lync client logs, diagnostics, and tools to troubleshoot those issues.
Let’s assume you are the administrator of a branch office. Michelle Martin comes to you and says that she has a scheduled call with the CIO every Tuesday. She can’t use Lync for these calls, because the call always sounds bad—it cuts in and out, and is choppy. She uses her Lync client for all other calls and the call quality is excellent. Why is she having an issue with this particular call and can you fix the problem?
As the branch office administrator, you don’t have full access to the Lync Server Monitoring reports. I could argue that you should, but that’s a different discussion. Assuming you don’t have access, you need to wait for Michelle to initiate her next call with the CIO and gather her client log when the call is completed.
To capture, analyze, and diagnose poor quality audio, do the following:
Here are the tools you’ll need to do the job:
To enable client logging in your Lync 2010 or Lync 2013 client:
Best Practice : To ensure you capture the correct call and QoE data in the UCCAPI log, remove all client UCCAPI logs from the path before starting Lync and replicating the issue.
The Lync 2010 client tracing logs are located at: %userprofile%\tracing .
The Lync 2013 client tracing logs located at: % userprofile%\appdata\Local\Microsoft\Office\15.0\Lync\Tracing.
The Lync 2010 client creates logs in the Communicator-uccapi-0.uccapilog format, while Lync 2013 creates logs in the Lync-UccApi-0.UccApilog format.
Note : the Lync 2010 client, runs the binary communicator.exe which harkens back to the days of Microsoft Live Communication Server 2005. The Lync 2013 client runs the binary named lync.exe.
After client logging is enabled, recreate the issue:
If necessary install Snooper. By default Snooper installs at C:\Program Files\Lync Server 2010\ResKit\Tracing . Open Snooper and navigate to the location where you saved the copy of Michelle’s client UCCAPI log.
Important : Lync Server 2010 or Lync Server 2013 ResKit tools are 64-bit. If you have a 32-bit OS, you need to install the Office Communication Server 2007 R2 version of the tool.
In Snooper, open the UCCAPI log and locate the call with the CIO. At the end of the call locate the Service type SIP message sent from the user to the Front End pool.
Figure 1. Searching for vqreportevent in Snooper
1. Search for vqreportevent and in the right hand pane (Figure 1).
2. Select the bottom XML blob that starts with <?xml version=”1.0”?> and ends with </VQReportEvent>.
3. Copy the entire XML blob into your buffer.
4. Paste the data into a new text file, and save it as a XML file.
5. Open XML Notepad 2007.
Figure 2. XML file open in XML Notepad 2007
First, open the XML file in XML Notepad
1. Click tree view (Figure 2).
2. Open the VQSessionReport node.
3. Navigate to the MediaLine area (Figure 3).
Figure 3. Locating the MediaLine node in the XML
The MediaLine folder contains a great deal of information. This article doesn’t detail each item or provide an authoritative list of thresholds for all values. Information and trending from Lync Server Monitoring reports can be used to trend call quality— as well as dig into individual calls—and comes with built in thresholds to alert the report consumer to issues.
Inbound and Outbound Network Mean Option Score (NMOS) scores, Jitter, Packet Loss, Delay, Connectivity, and the Capture and Render devices are the common cause of perceived call quality issues and are discussed in the article.
Note : Some of the same thresholds mentioned in the article, Voice Quality Improvements in Lync Server 2010 , are related to the visual feedback mechanisms of the Lync client and are useful when troubleshooting quality.
In our example we learned that when Michelle talked to Patrick, she could speak to him without a problem. The problem was poor quality audio she received from him. With that information, we dug into the data from her call.
Figure 4. The Inbound and Outbound Streams in XML Notepad
The Network Mean Option Score (NMOS) represents a machine approximation of a user’s perceived call quality as it relates to the network’s impact on that call quality.
The exact specifics of what we expect any Lync 2010 or Lync 2013 codec to provide during a call, can impact the possible number. Therefore, it isn’t always as simple as saying that a four is good, and that a one is bad.
In the Lync Monitoring Server Reports, NMOS degradation is a drop from the ideal for that codec. The ideal values for the Codec vary by codec. In general, one should look at the degradation of a NMOS score, and it’s average and maximum values rather than looking at the raw NMOS score presented.
Figure 5. Degradation Average of NMOS score
The InboundStream and OutboundStream documents the last codec used for the conversation. There are cases when a codec used at the beginning of the call is replaced by a higher quality codec later in the call because overall bandwidth improved. PayloadType is a numeric value, but the PayloadDescription is text that should help you determine which codec was used.
Figure 6. Outbound Stream Audio Payload type in XML Notepad
Please refer to the Microsoft Lync Server 2010: Work Smart Guide for Monitoring Server Reports for more information about diagnosing bad calls with Monitoring Server Reports data.
The downloadable document titled Understanding QoE Alerting , provides a reference table that shows parameter values that when exceeded, cause the Lync client to raise a visual alert to the end user.
See the classification metrics for calls are documented here .
To summarize: calls that exceeded 10% average packet loss, 50% max packet loss, 500ms latency or 30ms Jitter, or have a healer metric exceeding .007 for concealing, should be investigated. Look at each parameter in the XML data and investigate parameters that exceed the acceptable range, to determine the root cause of the issue.
In most cases, the Lync client is not the cause of the poor call quality. Client call quality is often affected by network conditions. Jitter occurs when packets arrive inconsistently for processing by the audio engine. As the name implies, packet loss occurs when packets are lost in transit. Because Lync leverages UDP for audio media, the Lync client doesn’t ask for lost packets to be resent. If packets arrive out of sequence, Lync just ignores them.
The Healer metrics are contained within the Network node of the InboundStream and OutboundStreams of our XML blob See Figure 7.
Figure 7. Healer Metrics from XML Notepad
Healer metrics help you understand possible network conditions and how well the codec and Lync client are able to fix and conceal these issues.
When reviewing the XML metrics in Excel to establish trends and if you have access to the Lync CDR and QoE databases, there’s a great TechNet article by Ilse Van Criekinge titled Extending your Lync monitoring data using PowerPivot and Power View .
Remember, with all this process, you are only reviewing half of the call stream. You can review the sending side of this conversation and its interpretation of the received data, but you can’t review the official record of what the other side sent. To do that, you would need to get the other side of the conversation or rely on the Lync monitoring reports.
Investigating call quality issues often leads to some interesting finds. Call quality can be impacted by variables such as the quality of the headset or audio device, the processor utilization of the machine making the call, and logical things such as packet loss and limited bandwidth availability.
Please let us know if you find the article helpful in your troubleshooting call quality issues and remember that Lync Server Monitoring Reports is the best source for all QoE issues.
To learn more, check out the following articles:
Keywords : Lync, Snooper, Logging, Media, Troubleshooting, Monitoring
Have you ever wanted to figure out why a user experienced poor call quality, but didn’t have access to the Lync Sever Monitoring Server reports to dig into the data? This article shows you how to use UCCAPI logging, Snooper, and XML Notepad to log, review, and analyze call data, when Lync Server 2010 or Lync Server 2013 monitoring reports aren’t available.
Author : Aaron Steele , Microsoft Senior Consultant
Publication date : December 10, 2012
Product version : Lync 2010 Client, Lync 2013 Client
Recently, I was helping a colleague troubleshoot a client’s audio quality issues and discovered that we didn’t have access to the Lync Server Monitoring reports. The first course of action was to install Monitoring Server Reports —Monitoring Server Reports is the best place to get diagnostic information and troubleshoot call quality issues.
For one-offs, it’s possible to gather the data locally and use Lync troubleshooting tools to diagnose call quality issues. This article explains how to gather the automatically generated Quality of Experience data found in client logs, and analyze that data to determine the source of the call quality problems.
To be clear, we’re not talking about tracing failed calls. If a call fails, use Lync client logs, diagnostics, and tools to troubleshoot those issues.
Let’s assume you are the administrator of a branch office. Michelle Martin comes to you and says that she has a scheduled call with the CIO every Tuesday. She can’t use Lync for these calls, because the call always sounds bad—it cuts in and out, and is choppy. She uses her Lync client for all other calls and the call quality is excellent. Why is she having an issue with this particular call and can you fix the problem?
As the branch office administrator, you don’t have full access to the Lync Server Monitoring reports. I could argue that you should, but that’s a different discussion. Assuming you don’t have access, you need to wait for Michelle to initiate her next call with the CIO and gather her client log when the call is completed.
To capture, analyze, and diagnose poor quality audio, do the following:
- Enable Lync client logging.
- After the call is complete, open the client log using Lync Server 2010 Resource Kit Tool: Snooper .
- Copy the call quality data into a XML file.
- Parse that XML data with a free tool called XML Notepad .
- Trend on that data in Excel if needed.
Here are the tools you’ll need to do the job:
To enable client logging in your Lync 2010 or Lync 2013 client:
- In the upper right corner of the Lync main window, click Options (gear icon).
- In the Lync - Options dialog box, click General .
- Under Logging , select the Turn on logging in Lync and Turn on Windows Event logging for Lync check boxes.
- Click OK .
- Restart Lync.
Best Practice : To ensure you capture the correct call and QoE data in the UCCAPI log, remove all client UCCAPI logs from the path before starting Lync and replicating the issue.
The Lync 2010 client tracing logs are located at: %userprofile%\tracing .
The Lync 2013 client tracing logs located at: % userprofile%\appdata\Local\Microsoft\Office\15.0\Lync\Tracing.
The Lync 2010 client creates logs in the Communicator-uccapi-0.uccapilog format, while Lync 2013 creates logs in the Lync-UccApi-0.UccApilog format.
Note : the Lync 2010 client, runs the binary communicator.exe which harkens back to the days of Microsoft Live Communication Server 2005. The Lync 2013 client runs the binary named lync.exe.
After client logging is enabled, recreate the issue:
If necessary install Snooper. By default Snooper installs at C:\Program Files\Lync Server 2010\ResKit\Tracing . Open Snooper and navigate to the location where you saved the copy of Michelle’s client UCCAPI log.
Important : Lync Server 2010 or Lync Server 2013 ResKit tools are 64-bit. If you have a 32-bit OS, you need to install the Office Communication Server 2007 R2 version of the tool.
In Snooper, open the UCCAPI log and locate the call with the CIO. At the end of the call locate the Service type SIP message sent from the user to the Front End pool.
Figure 1. Searching for vqreportevent in Snooper
1. Search for vqreportevent and in the right hand pane (Figure 1).
2. Select the bottom XML blob that starts with <?xml version=”1.0”?> and ends with </VQReportEvent>.
3. Copy the entire XML blob into your buffer.
4. Paste the data into a new text file, and save it as a XML file.
5. Open XML Notepad 2007.
Figure 2. XML file open in XML Notepad 2007
First, open the XML file in XML Notepad
1. Click tree view (Figure 2).
2. Open the VQSessionReport node.
3. Navigate to the MediaLine area (Figure 3).
Figure 3. Locating the MediaLine node in the XML
The MediaLine folder contains a great deal of information. This article doesn’t detail each item or provide an authoritative list of thresholds for all values. Information and trending from Lync Server Monitoring reports can be used to trend call quality— as well as dig into individual calls—and comes with built in thresholds to alert the report consumer to issues.
Inbound and Outbound Network Mean Option Score (NMOS) scores, Jitter, Packet Loss, Delay, Connectivity, and the Capture and Render devices are the common cause of perceived call quality issues and are discussed in the article.
Note : Some of the same thresholds mentioned in the article, Voice Quality Improvements in Lync Server 2010 , are related to the visual feedback mechanisms of the Lync client and are useful when troubleshooting quality.
In our example we learned that when Michelle talked to Patrick, she could speak to him without a problem. The problem was poor quality audio she received from him. With that information, we dug into the data from her call.
Figure 4. The Inbound and Outbound Streams in XML Notepad
The Network Mean Option Score (NMOS) represents a machine approximation of a user’s perceived call quality as it relates to the network’s impact on that call quality.
The exact specifics of what we expect any Lync 2010 or Lync 2013 codec to provide during a call, can impact the possible number. Therefore, it isn’t always as simple as saying that a four is good, and that a one is bad.
In the Lync Monitoring Server Reports, NMOS degradation is a drop from the ideal for that codec. The ideal values for the Codec vary by codec. In general, one should look at the degradation of a NMOS score, and it’s average and maximum values rather than looking at the raw NMOS score presented.
Figure 5. Degradation Average of NMOS score
The InboundStream and OutboundStream documents the last codec used for the conversation. There are cases when a codec used at the beginning of the call is replaced by a higher quality codec later in the call because overall bandwidth improved. PayloadType is a numeric value, but the PayloadDescription is text that should help you determine which codec was used.
Figure 6. Outbound Stream Audio Payload type in XML Notepad
Please refer to the Microsoft Lync Server 2010: Work Smart Guide for Monitoring Server Reports for more information about diagnosing bad calls with Monitoring Server Reports data.
The downloadable document titled Understanding QoE Alerting , provides a reference table that shows parameter values that when exceeded, cause the Lync client to raise a visual alert to the end user.
See the classification metrics for calls are documented here .
To summarize: calls that exceeded 10% average packet loss, 50% max packet loss, 500ms latency or 30ms Jitter, or have a healer metric exceeding .007 for concealing, should be investigated. Look at each parameter in the XML data and investigate parameters that exceed the acceptable range, to determine the root cause of the issue.
In most cases, the Lync client is not the cause of the poor call quality. Client call quality is often affected by network conditions. Jitter occurs when packets arrive inconsistently for processing by the audio engine. As the name implies, packet loss occurs when packets are lost in transit. Because Lync leverages UDP for audio media, the Lync client doesn’t ask for lost packets to be resent. If packets arrive out of sequence, Lync just ignores them.
The Healer metrics are contained within the Network node of the InboundStream and OutboundStreams of our XML blob See Figure 7.
Figure 7. Healer Metrics from XML Notepad
Healer metrics help you understand possible network conditions and how well the codec and Lync client are able to fix and conceal these issues.
When reviewing the XML metrics in Excel to establish trends and if you have access to the Lync CDR and QoE databases, there’s a great TechNet article by Ilse Van Criekinge titled Extending your Lync monitoring data using PowerPivot and Power View .
Remember, with all this process, you are only reviewing half of the call stream. You can review the sending side of this conversation and its interpretation of the received data, but you can’t review the official record of what the other side sent. To do that, you would need to get the other side of the conversation or rely on the Lync monitoring reports.
Summary
Investigating call quality issues often leads to some interesting finds. Call quality can be impacted by variables such as the quality of the headset or audio device, the processor utilization of the machine making the call, and logical things such as packet loss and limited bandwidth availability.
Please let us know if you find the article helpful in your troubleshooting call quality issues and remember that Lync Server Monitoring Reports is the best source for all QoE issues.
Additional Resources
To learn more, check out the following articles:
- Microsoft Lync Server 2010: Work Smart Guide for Monitoring Server Reports
- Extending your Lync monitoring data using PowerPivot and Power View
Lync Server Resources
- Lync Server 2010 Documentation Library
- DrRez blog
- NextHop blog
- Lync Server and Communications Server resources
We Want to Hear from You
Keywords : Lync, Snooper, Logging, Media, Troubleshooting, Monitoring
Updated May 20, 2019
Version 2.0