How to Use TSS to Collect Data and Analyze to Solve High CPU Issues.
Hello everyone, this is Denzel with the Windows Performance Team. I found a tool that actively collects different data based on scenarios and streamlines the data collection process. Drumroll – introducing TSS (Troubleshooting Support Script). In my job, I see a lot of High CPU cases and collecting an ETL trace using TSS with Xperf aka WPR for high CPU has been fundamental in resolving issues.
I’d like to share some instructions, methods, and just insight on the tools in general that should be able to empower IT professionals resolve issues. This post will show how the TSS tool can work with the Windows Performance Recorder. TSS also works with several tools as it is very powerful but will focus on collecting a WPR trace using TSS when regarding a case of High CPU. I can even give you a great clue as to how to collect data for Intermittent High CPU cases as well! Once you have the data, I’ll then show you how to analyze it. Lastly, I’ll provide some additional resources on WPA Analysis for High CPU.
Data Collection Tools:
TSS
TSS (TroubleShootingScript) is a code signed, PowerShell based Tool and Framework for rapid flexible data collection with a goal to resolve customer support cases in the most efficient and secure way. TSS offers an extensible framework for developers and engineers to incorporate their specific tracing scenarios.
WPR/Xperf
“Windows Performance Recorder (WPR) is a performance recording tool that is based on Event Tracing for Windows (ETW). It records system and application events that you can then analyze by using Windows Performance Analyzer (WPA). You can use WPR together with Windows Performance Analyzer (WPA) to investigate particular areas of performance and to gain an overall understanding of resource consumption.”
*Xperf is strictly a command line tool, and it can be used interchangeably with the WPR tool.*
_________________________________________________________________________________________________________________________________________________
Let’s Dig in!
You notice your server or device is running with 90% high CPU. Your users are complaining of latency and poor performance. You have checked task manager, resource monitor or even downloaded and opened process explorer but there is still no exact root resource glaring you in the face. No worries, a WPR will break down the high CPU processes a bit more. You could even skip straight to this step in the future when you get comfortable working with these tools.
Setup TSS
Running a TSS troubleshooting script with the parameters for either WPR or Xperf gathers some granular Performance data on machines showing the issue. In the example below, I’m saving the TSS the script to 😧 (note the default data location is c:\MS_Data). In your web browser, download TSS.zip found here: https://aka.ms/getTSS or you can Open an Administrative PowerShell Prompt and paste the following commands.
The commands below will automatically prepare the machine to run TSS by taking the following actions in the given order:
- Create D:\TSS folder
- Set the PowerShell script execution policy to RemoteSigned for the Process level (process level changes only affect the current PowerShell window)
- Set TLS type to 1.2 and download the TSS zip file from Microsoft
- Expand the TSS.zip file into D:\TSS folder
- Change to D:\TSS folder
Ex: Commands used below
md D:\TSS
Set-ExecutionPolicy -scope Process -ExecutionPolicy RemoteSigned -Force
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Start-BitsTransfer https://aka.ms/getTSS -Destination D:\TSS\TSS.zip
Expand-Archive -LiteralPath D:\TSS\TSS.zip -DestinationPath D:\TSS -force
cd D:\TSS
The result will be a folder named TSS on drive D.
_________________________________________________________________________________________________________________________________________________
Gathering Data using TSS
Open an elevated PowerShell window (or start PowerShell with elevated privileges) and change the directory to this folder:
cd D:\TSS\
*WARNING* Data collection grows rather large quickly. You should have at least 30% of your overall RAM available as hard drive space. (Example, if you have 8 GB of RAM – then the file can grow to 2.5GB or larger in c:\MS_Data.)
What are some of the scenarios you might have? Maybe you want to manually collect the trace. Or, once you start the trace, let it automatically stop.
How about limiting the file size? There are several parameters you can adjust for your needs.
Below you will find variations of using TSS to collect WPR data in high CPU occurrences. You have an option of using either WPR or Xperf commands. Please review all of them before deciding which trace to take for your environment.
1. Scenario In State needing user intervention to stop the trace: The issue is currently occurring, and the following example needs user intervention to stop the trace. The WPR can grow to 80% of the memory with the example commands listed below.
.\Tss.ps1 -WPR General *** (run it for 60 seconds to no longer than 3 minutes)
.\Tss.ps1 -Xperf CPU ***(run it for 60 seconds to no longer than 3 minutes)
Default location of saved data will be C:\MS_Data.
The prompt will tell you when to reproduce the issue, just simply enter “Y” will END the trace at that time and the machine in question experiencing high CPU will then finish running the data collection.
2. Scenario In State but you wanted to limit the size and length of time: The issue is currently occurring; the following example does NOT need user intervention to stop the trace. Default location of saved data will be C:\MS_Data. The Xperf can grow to 4GB of memory and runs for 5 minutes with the setting below:
.\TSS.ps1 -Xperf CPU -XperfMaxFileMB 4096 -StopWaitTimeInSec 300
Note: you can modify the size and length of the trace by increasing or decreasing -XperfMaxFileMB and -StopWaitTimeInSec when it is initially run.
3. Scenario In State but you wanted to limit the size and length of time with data saved on Z:\Data drive instead of C:\MS_DATA (default): The issue is currently occurring; the following example does NOT need user intervention to stop the trace. The Xperf can grow to 4GB of the memory and runs for 5 minutes with the setting below and this time the resulting data will be saved on Z:\Data. You simply need to add -LogFolderPath Z:\Data to the command.
.\TSS.ps1 -Xperf CPU -XperfMaxFileMB 4096 -StopWaitTimeInSec 300 -LogFolderPath Z:\Data
4. Scenario Intermittent High CPU and having a tough time capturing data: These commands will wait for the CPU to reach 90%, start a trace and will stop the file from growing larger than 4GB while running for 5 minutes.
.\TSS.ps1 -Xperf CPU -WaitEvent HighCPU:90 -XperfMaxFileMB 4096 -StopWaitTimeInSec 300
5. Scenario Intermittent High CPU and having a tough time capturing data: These commands will wait for the CPU to reach 90%, start a trace and will stop the file from growing larger than 4GB while running for 100 seconds (1.5 minutes roughly).
.\TSS.ps1 -Xperf CPU -WaitEvent HighCPU:90 -StopWaitTimeInSec 100
Pro Tip: You can check for additional Xperf/WPR commands by doing a search on the help command files in TSS by typing
.\TSS.ps1 -help at the prompt. When prompted to enter a number or keyword, type xperf or wpr, hit enter, and you will see the options.
Ex: Finding help with keyword ‘xperf’
Be sure to wait for the TSS script to finish, it can take some time (even an hour to finish writing out). The PowerShell will return to type line and the folder in C:\MS_Data should zip itself when complete. The location of the script does not determine the location of the data collected. Wait for trace to finish before exiting PowerShell.
Reminder: Just like in the first trace, you learned data collection grows rather large quickly. You should have at least 30% of your overall RAM available as hard drive space. (Example, if you have 8 GB of RAM – then the file can grow to 2.5GB or larger on c:\MS_Data.)
_________________________________________________________________________________________________________________________________________________
You have the Data – Now Let’s look at it!
Download the Windows ADK (Windows Assessment and Deployment Kit) from this location: Download and install the Windows ADK | Microsoft Learn. Once you download the Windows ADK, you want to install the Windows Performance Toolkit. Double click on the executable (.exe) to start the installation process.
Uncheck everything except Windows Performance Toolkit, then click Install.
Opening the data in the C:\MS_DATA folder
When complete, the WPR general TSS command should have placed all collected data into this folder in a zipped file. You will know the trace ran all the way without stopping prematurely when you see the zipped file in C:\MS_DATA. There will also be a message in the PowerShell window when the diagnostic completes stating the name of and the location of the zipped file.
You will need to unzip the zipped file to analyze the WPR trace (.etl file). After unzipping, you will see several data collections that can be helpful with analysis. However, what you mainly want to look at is the .etl file which is usually the biggest file located in the folder.
If you double click the .ETL file it should open in WPA, but if not, you can manually open the newly installed application and navigate to your file.
Example:
You can open the .ETL file to view the WPR trace with WPA (Windows Performance Analyzer) by clicking File, Open and then browsing to the file that ends with the .ETL extension.
Step 1. Open WPR trace in WPA and load the Public Symbols. You may also see symbols listed from the NGEN folder (NGEN is part of the folder name) collected at the time the WPR trace was run.
Select Trace, select Configure Symbol Paths
Click + sign (highlighted in yellow in screenshot below), then enter Public Symbol Path: srv*c:\symbols*https://msdl.microsoft.com/download/symbols
More Information: (Symbol path for Windows debuggers - Windows drivers | Microsoft Learn)
Once symbols are configured simply click Load Symbols
Step 2. Once open you should see a window similar to the screenshot below. Expand Computation on the left and drag CPU Usage (Sampled) to the right side of the Window to load. You can also double click CPU Usage (Sampled) for it appear on the right side.
You will then see space on the top graph showing, “Trace Rundown”. That is not needed as it is the part of the trace where the script was finishing up. To get rid of the trace rundown, highlight the area before trace rundown, right click, then select “Zoom”.
You can now filter down each of your processes deeper and deeper to try to locate a potential root cause of what is spiking the CPU. You can look to see which processes have the highest weight over on the right-hand columns to help pinpoint the highest consumers. It may be a specific kernel driver, application, process, etc. but this should help point you in the right direction of what process is exhausting resources.
These are the columns you will want to focus on:
Left of Gold Bar:
Process
Stack
Right of Gold Bar:
Count
Weight
% Weight
You can see the CPU usage is the highest due to CPUSTRESS.EXE in this example. As you filter down you can see the threads that contribute to the max CPU spike which is visible in % weight sum. This can be helpful to find out which threads, functions and modules are called for the root cause.
Step 3. You can open graph, "Utilization By CPU" for additional analysis.
By adding the process column on the left side of the gold bar you can confirm which processes are the highest consumer % weight wise on each CPU. To add the process column, right click on one of the column titles and select more columns as instructed in screenshot below. You can then filter down each individual processor and view the CPU percentages per process.
Conclusion:
Once again this is not the only use for the TSS tool. But as you can see, the WPR/Xperf trace is a very complex tool that gathers data from a simple PowerShell command. This can be very efficient for troubleshooting. This article is not meant to cover all scenarios. However, I highly recommend taking some time to learn more about what TSS can accomplish as this tool will only continue to get better.
If at any point you get stuck don’t hesitate to open a support case with Microsoft.
Additional Information:
Information on TSS and alternative download site:
Introduction to TroubleShootingScript toolset (TSS) - Windows Client | Microsoft Learn
Information about Windows Performance Toolkit
Windows Performance Toolkit | Microsoft Learn
For Reference:
Download Windows Assessment Toolkit which contains Windows Performance Analyzer
Download and install the Windows ADK | Microsoft Learn
How to setup public symbols
Symbol path for Windows debuggers - Windows drivers | Microsoft Learn
Microsoft
